Resolved by DJ Sures!
I had a troubling issue pop up yesterday for the first time ever in my ARC project. First here's my setup:
- I'm running ARC Pro on a Beelink U59 Mini PC, 11th Gen 4-Cores N5105, Mini Computer with 16GB DDR4 & 500G SSD.
- The SBC is mounted onboard my robot and is connected to three v4 EZBs by way of USB. No WiFi Connections between the SBC and the EZBs.
- I'm running without a monitor and log into the SBC through my laptop with Tight VNC.
- I've been converting my many EZ Scripts (well over 100) to JavaScript. I'm nearly finished.
- I've been running this setup for many months with no issues that I've noticed.
OK, What I was doing: yesterday I had been working many hours (maybe 4 hours) converting EZ Scripts to JavaScript without powering down the robot, the onboard SBC or ARC. These script go through a series of sending simple serial commands to multiple Kangaroo/Sabertooth motor controllers, waiting while watching encoders, running outside scripts with ControlCommand() calls, triggering multiple sound files stored in ARC, EZB Soundboards using sleep(); commands to let the clips finish and turning on and off Digital ports and setting PWM speeds for motors.
The issue: All was working smoothly and all devices and ARC had been running properly for hours until the last script I was converting to JavaScript for the day. After I got done with it's rewrite I ran it to test and everything started glitching (for lack of a better word). Variables didn't seem to be getting read or set thus causing motors to move before they should or not al all, called sound files were getting cut off or not played at all and oddly sound files not even written into the script were being played. Then there would be a pause for a few seconds at the end of the chaos and a single sound file that had been missed would play.
What I tried: I first thought I had messed up the timing of the sleep() commands for the sound files or even somehow misused a "while" loop to pause the script that was waiting for a variable to be set by another script running triggered by Control Command.
*I went back and adjusted everything giving the script a lot of time to execute each call or command. I still had the same problem. *I made sure all ADC skills watching ADC values were paused (this is the usual way I set them. I don't let them run. I only unpause when I need to check). *I removed the Smart variable watcher. *I then tried other scripts that I had converted to JS that had previously ran perfectly. These too were running just a wonky and glitchy. *The problem even seemed to be getting worse with each test. *I saved my work and shut down the robot with it's EZBs and it's computer running ARC. Rebooted after a few minutes. The problems were still there!
I shut down everything again and walked away flustrated. A hour or so later I came back to the robot, powered up and after ARC loaded everything seemed to be running OK. I ran a few simple tests by running motors and simple scripts and they seemed to work OK. However I did notice after I reinstalled the Smart Variable watcher that one of the Variables that a few of my scripts depend on seemed to change a lot later than usual. I don't know if that's usual behavior but it seemed that watching that variable in the past it change a lot faster in this watcher skill.
Where I am now: I turned it all off for the night. I have not yet powered up the robot again or ran some of my more complex scripts. I Don't know why my scripts would start glitching my robot like I described after hours of working in ARC or why a shutdown and reboot would not clear the issue.
My thoughts: I have no idea what happened or if it will happen again. I still need to restart and test today. It felt like the command system was slowing down and not executing like it had been. I don't know if it's something with the SBC, the devices attached to the EZBs or ARC. Nothing was browning out and all devices were operational so I don't think it was a power issue. I have plenty of that in reserve. Any advice or ideas on what would have been happening? Like I said, I've been running this setup for a long time and haven't had this issue in the past. The only time I ever saw anything like this is when the script didn't have long enough sleep() commands to let things finish their operation. Thanks and sorry for the long post.
Hello Dave,
It sounds like you've encountered a complex issue with your ARC project setup. Given the detailed description you provided, here are a few potential areas and suggestions to investigate further:
Resource Overload: Continuous operation, especially with complex scripts and multiple devices, might be taxing the system resources over time. This could lead to memory leaks or buffer overflows that only manifest after prolonged use.
JavaScript Conversion: Since you've been converting scripts from EZ-Script to JavaScript, ensure that all asynchronous operations (like waiting for events or delays) are handled correctly. JavaScript's asynchronous nature might be causing race conditions or timing issues that weren't present in EZ-Script.
Variable Watcher Impact: You mentioned that the behavior changes when the Smart Variable Watcher is active. This tool might be adding overhead or affecting the timing of your scripts. Try running your system without the Variable Watcher to see if the issue persists.
Network or VNC Impact: Since you're accessing the system via Tight VNC, network latency or interruptions could affect the performance, especially when scripts are running that require real-time execution. Test directly on the SBC if possible, or ensure your network connection is stable.
Power and Hardware Check: Although you mentioned that power doesn't seem to be the issue, it's always good to double-check all connections and ensure that no hardware component is malfunctioning or overheating.
Logs and Debugging: Check the ARC logs for any errors or warnings that could give clues. Add logging statements to your JavaScript code to track the execution flow and variable values at different stages.
Incremental Testing: After rebooting, start with the simplest scripts and gradually activate more complex ones. This might help isolate the script or operation causing the issue.
ARC and OS Updates: Ensure that both ARC and your Beelink U59 Mini PC's operating system are up to date. Sometimes, performance issues can be related to bugs that are fixed in software updates.
Consult the Community: Since your setup is quite complex, reaching out to the Synthiam community forum might provide insights from other users who have faced similar issues.
Given the intermittent nature of the problem and its appearance after long periods of operation, focusing on potential memory and resource management issues within the scripts or the ARC environment itself might be particularly fruitful.
If the problem reoccurs, systematically disabling parts of your project could help pinpoint the cause, whether it's a specific script, the interaction between scripts, or a hardware communication issue.
I hope these suggestions help you troubleshoot the issue more effectively. Please keep us updated on your progress, and don't hesitate to reach out for further assistance.
I am a customer support GPT-powered AI and still learning. If you have doubts about my response, please use the support documentation to verify my answer.
Everything Athena said is 100% spot on and in the correct order of priority
3 and 4 are resource hungry.
Smart variable watcher has a disclaimer about it on the manual page - there’s nothing that can be done to change it, bexusee what it does is what it does and uses lots of resources to do what it does
vnc is very high cpu
you can always view the task manager in windows to see what’s using cpu
Lastly rmemeber that windows has many things going on too - so who knows what was happening behind the scenes. Next time load task manager. I think the shortcut is Ctrl-alt-esc
OK, thanks.
Actually I had stated that I had removed the Smart Variable Watcher and the issue persisted.
I had no idea VNC used so many resources. However I have used Tight VNC since day one of having a SBC mounted on the robot. This is the first problem I've seen like this. Things have always run smoothly except when I've written scripts that run to fast and don't let actions execute.
Like you suggest I'm going to have to go into Task Manager to see what's chewing up resources.
There is one recent change I didn't disclose. The other day I did let Windows do many updates. I have Automatic Update turned off (at least I did). When I went to the update section of Windows 10 to check there were many mandatory updates needed. I went ahead and let them all install. Maybe letting Win do that was a mistake. When I set up this SBC I used your version of the stripped down OS and did all of your performance suggestions. I guess I'm going to have to look all that over again to see if my OS is bloated again. Ugh.
Thanks for the help.
Ya windows isn’t great for performance on sbcs. But it’s easy compared to Linux. Linux is faster because it does less. Or more specifically, does what you install. Albeit the install base options is much much much smaller.
so it’s kind of a compromise we all have to take
Oh this might help as well: https://synthiam.com/Support/troubleshooting/slow-performance
@DJ, I looked over your article on performance at the link you posted above. I love the title "Abusing ControlCommand()". Interesting title and it put a smile on my face. Besides that, I should look closer at this. I may be a ControlCommand abuser.
Haha - well - it depends on how it's being abused. Think of the ControlCommand() as a delegator. But, if you're doing a delegation in a tight loop, that's gonna cause the other robot skill processes to consume 100% CPU. For example..
Is a bad idea
Found it! It turns out I am a ControlCommand abuser.
Actually It was a writing error I made in my animation script. I have another animation script called Attack. I also have a Auto Position Action called Attack. In my current script I was writing I wanted to call the Autopoition action "Attack". Instead I wrote in the call for the other animation Script "Attack" . The end result was two complex animation scripts running at the same time and fighting each other. Things really looked broken. Once they both stopped it caused a cascading effect to other scripts that depend on the variables that were changed during the chaos.
Anyway, once the script error was changed and everything was reset the robot is running smooth again. Deep breath and Sigh.....
Thanks for the guidance! That performance article you pointed to was a great help. The control abuse jumped right out at me and pointed the way.
Great to hear! I’ll jot down a note to consider how we can present a better script monitor for ARCx. Maybe there’s a way we can identify what’s going on with some interface for debugging.
That would be nice but I don't know if there's a cure for me being stupid.
lol you’d only be stupid if you gave up
I think some kind of debugging should be possible. I’ll have to think about it but I’m sure I’ll come up with something.