You only look once (YOLO) is a state-of-the-art, real-time object detection system. using Tiny YOLOv3 a very small model as well for constrained environments (CPU Only, NO GPU)
How to add the Darknet YOLO (Obj Detection) robot skill
- Load the most recent release of ARC (Get ARC).
- Press the Project tab from the top menu bar in ARC.
- Press Add Robot Skill from the button ribbon bar in ARC.
- Choose the Camera category tab.
- Press the Darknet YOLO (Obj Detection) icon to add the robot skill to your project.
Don't have a robot yet?
Follow the Getting Started Guide to build a robot and use the Darknet YOLO (Obj Detection) robot skill.
How to use the Darknet YOLO (Obj Detection) robot skill
You only look once (YOLO) is a state-of-the-art, real-time object detection system. using Tiny YOLOv3 a very small model as well for constrained environments (CPU Only, NO GPU)Darket YOLO website: https://pjreddie.com/darknet/yolo/
Requirements: You only need a camera control, the detection is done offline (no cloud services).
1) start the camera.
2) check the Running (check box)
The detection will run continuously when the detection results change an On Changes script is executed (check the configuration area):
1) Press config
2) Edit the on changes script
3) on changes Javascript script
you can run the detection on demand, javascript:
Code:
controlCommand("Darknet YOLO", "Run");
The above command runs the configured on demand script.
An example of script:
Code:
var numberOfRegions=getVar('$YOLONumberOfRegions');
if (numberOfRegions==0)
{
Audio.sayWait('No regions found');
}
else
{
Audio.sayWait('Found ' + numberOfRegions + ' regions');
var classes = getVar('$YOLOClasses');
var scores = getVar('$YOLOScores');
for(var ix=0; ix {
Audio.sayWait('Found ' + classes[ix] + ' with score: ' + (classes[ix]*100) + '%');
}
}
Please download the following file:
And copy to the plugin folder:Code:
Expected plugin folder content:Code:
var numberOfRegions=getVar('$YOLONumberOfRegions');
if (numberOfRegions==0)
{
// Audio.sayWait('No regions found');
}
else
{
// Audio.sayWait('Found ' + numberOfRegions + 'regions');
//Audio.sayWait('Found ');
var classes = getVar('$YOLOClasses');
for(var ix=0; ix {
//Audio.sayWait("I see a " + (classes[ix])); or
Audio.sayWait(classes[ix]);
}
}
So the audio only comes out with the item it detects
EzAng
Code:
I'm guessing the detection is done in a new task (new thread)? If so, you'll have to make a copy of the bitmap if it's not being manipulated in the OnNewFrame event. Working with any camera image has to be done either in the new frame event, or a copy of itself needs to be made to work in another thread.
EzAng
:
Maybe is not enough to copy the bitmap ?Code:
Do you recommend another method to copy the bitmap ?
An old version of ARC (when ARC days) used to create a new bitmap and dispose it for every frame. But that was super expensive on garbage collection. With the new method, the memory is reused - and that's also why a camera image can exist on many skill controls without a ton of cpu being used or memory. It's because every instance of that bitmap is actually referencing the same memory location and only the screen needs to be refreshed when the memory updates
So the solution for yours is actually quite easy - i would recommend taking the Camera's bitmap and "draw" it to a bitmap for your own detection thread. That way, you can keep your own bitmap or dispose of it how you wish etc etc and it lives for long as you want it to.
Something like...
Code:
OR fastest way is....... you can use memcpy and get an instance of your Bitmap's BitmapData - and memcpy the Camera.GetCurrentBitmapUnmanaged.Data to your bitmap's Data0
quick search:
https://docs.microsoft.com/en-us/dotnet/api/system.drawing.bitmap.-ctor
and I'm guessing is a shallow copy not a deep copy, so the "shell" object is different but the byte buffer is the same.
So soon or later becomes an issue. I'll change the code, looking for elegant e.g. (less boiler plate code) to generate a deep clone.
EDITED *** I did not see the previous post **
Thanks!
Code:
I love Yolo...seems to work pretty well in near dark lighting conditions too. I've been using the 80 class version. My favorite is when it recognizes my cats, plants, phones, and tvs. I don't know why but it continues to amuse me, I think because I know I would never be able to do it without a NN. It feels like magic. I keep hoping someone in the industry will build some more Yolo-like models with a lot more classes. It seems like I read about a Yolo9000 but was never was able to find anything I could use. If anyone finds a model with a lot more classes, I'd love to hear about it. I haven't tried v4 or v5 yet...I don't think anyone has published one beyond v3 that will deploy on a Movid.
it uses a worldwide database of trained stuff. And you get a description of the scene that’s neat. You can feed that into nlp for topics of the surroundings.
@Guys:
Thanks for the good words
@Martin:
Short answer: Visual Studio Community/Enterprise 2019
ARC plugins are .NET
The main plugin dll is a visual studio c# .NET class project and I've two other additional projects in c++. ARC is a 32 bits application so when you combine low level code (c++) or external native libraries and .NET you need to take that in consideration. Sometimes you need to compile from source, and fix or tweak the open source code to use msft building environment.
If you need more details my email is in the profile.
When I start playing with object detection, I wanted something to monitor a live video feed and trigger actions based on objects. All the cloud APIs have limits so is not feasible to use the online services.
I presume your skill has a limit cap ?
Q) What is the limit (number of request) per ARC account ?
Also the model is a Tiny version optimized for CPU, so the accuracy is lower than the full models (Nvidia GPUs).
The biggest challenge is to find optimized models for our needs, for example I'm using this model to track the deliver man. I don't expect a train, horse, sheep, cow, elephant, bear, zebra, giraffe in the camera
But the model supports those categories.
The solution is to train your model with your images. I'm capturing pictures of the deliver guy, and I want to expand the capture to the trucks.
Maybe later I can train a model to detect UPS, Fedex, USPS, DHL, Amazon trucks
Until then... I've a trigger to alert me if an Elephant arrives at the door.
For me, I am proceeding along the following path with darknet object detection:
1. I implemented Darknet as you know, and it is returning good bounding boxes and probabilities for the 80 classes. When I mentioned having a better model...I meant I wanted more classes.
2. I implemented a skill to get the robot to tell me what it sees by saying "What do you see?".
3. I am wondering if there is a way to augment DarkNet with AlexNet. To this ends, I first implemented the AlexNet model (1000 classes). The problem here is AlexNet classifies an image as a single thing, so I need to figure out an algo for picking subsets of an image that might contain single interesting objects. Until I figure out now to do that, AlexNet is not all that useful to me unless the bot is leaning over, staring directly at something and trying to identify or pick it up. Also, a huge amount of the 1000 classes are still biology or other fairly useless classes like you pointed out. There are a lot of alternatives to AlexNet that all have these same issues...single object and too many useless classes like species. Species aside, does anyone know a good way to segment an image so parts of it can be classified?
4. Here are the use cases I want to focus on next... more verbal questions and answers (about what is seen) like "Where is the cat?", "How far away is the cat?" (depth sensor), "How big is the cat." (some trig with distance and bounding box), "What color is the cat?" (image processing, tougher one for me), "Shoot the cat." (lasers), "Go to/chase the cat" (nav/drive), "Point at the cat." (servos), "What is next to the cat", "How many cats do you see?", "Look at the cat", "What is the cat on top of?" (table, tv, etc.) and others. You get the idea. While some of these sound challenging or error prone, almost all of these are achievable. I'd like to make a vid when I get some of these going.
I have been using this skill for some time and I am very glad that you created it! I do have a couple of questions:
1. is there any EASY way to REMOVE objects from its list? I want to remove objects that my robot will never encounter.
2. is there any way from within ARC to define new objects? I understand that defining new objects (using thousands of photos) is a very CPU intensive operation.
3. is it possible to add other DarkNet/Yolo objects from other datasets?
If your are asking to remove classes e.g. elephants, horses, zebras to speed up the detection, that it's not possible.
ARC has a camera control that allows training new custom objects using a camera:
https://synthiam.com/Support/Skills/Camera/Camera-Device?id=16120#objectTracking
It's possible to improve the plugin to specify a custom Yolo dataset, there are a few Yolo framework implementation versions i.e. v3, v2, v4,v5. If you have one in particular please share the URL and I can try to see if is compatible with the plugin.
It's not possible to use multiple datasets in a single inference process.
2. Does the ARC camera control support YOLO objects?
3. I'll take a look. Thanks!
Thomas Messerschmidt
Scroll to the bottom, and you can read that relevant section of the page. You can use the support section to find additional information about using ARC.
*edit: or this step of the getting started guide is quite popular: https://synthiam.com/Support/Get-Started/how-to-make-a-robot/choose-skill-level
Thanks.