Thumbnail

Train Vision Object By Script

by Synthiam

Enable real-time object vision training using scripts

How to add the Train Vision Object By Script robot skill

  1. Load the most recent release of ARC (Get ARC).
  2. Press the Project tab from the top menu bar in ARC.
  3. Press Add Robot Skill from the button ribbon bar in ARC.
  4. Choose the Camera category tab.
  5. Press the Train Vision Object By Script icon to add the robot skill to your project.

Don't have a robot yet?

Follow the Getting Started Guide to build a robot and use the Train Vision Object By Script robot skill.

How to use the Train Vision Object By Script robot skill

In order to train objects to the camera device, the Train Object menu is used in the camera device skill. This menu normally requires human intervention to enter the object name and use the mouse. This skill allows your robot to learn objects in real-time triggered by ControlCommand() scripts programmatically. Take a look at the Cheat Sheet within the Script skill settings to see what ControlCommand() this skill accepts.

User-inserted image

Main Window

User-inserted image

1. Beginning Learning Button This button will attach/detach the camera configured in the camera device skill to the program with pictures skill window.

2. Attach/Detach Camera Button This button will attach/detach the camera configured in the camera device skill to the Train Vision Object by Script skill window.

3. Train Status Bar This will show the object training process time remaining. During this time move the object around to train it at different angles.

4. Status Display This field will display the status and name of the learned object and any errors that occur.

How To Use the Train Vision Object by Script Skill

  1. Add a camera device skill (Project -> Add Skill -> Camera -> Camera Device).

  2. Connect to a camera.

  3. Add this skill (Project -> Add skill -> Camera -> Train Vision Object by Script).

  4. Press the ATTACH button. This will attach this skill to the camera device skill. You will now see a mirror of the camera's video stream in this skill.

  5. Center the onscreen pink box on the object that will be trained. Have the object fill up the pink box completely. Press the BEGIN LEARNING button, then physically move the object around during the training process to get multiple angles.

  6. Wait until the Train Status bar has completed and make note of the object name in the Status display so that you can use it in your scripts.

Script Sample

Here's an script sample that learns an object titled DJ FACE via ez-script:


# Check if the camera is active. If not, exit the script and notify user
if (!$IsCameraActive)
  say("The camera is not active")
  halt()
endif

# This is the name of the object that will be stored
$LearnedObject = "DJ FACE"

# Attach to the camera for the control
ControlCommand("Train Vision Object By Script", Attach)

# Introduce verbally out of the PC speaker what it is about to happen
SayWait("Prepare to learn " + $LearnedObject + ". 3, 2, 1, learning")

# Instruct the skill to begin learning
ControlCommand("Train Vision Object By Script", Learn, $LearnedObject)

# Wait until the skill has completed learning the object
WaitFor($CameraIsLearning == 0)

# Speak what the object has been learned
Say("I have learned " + $LearnedObject)

# Enable object tracking in the camera control
ControlCommand("Camera", CameraObjectTracking)

Video

Resources

You can also download the project here: trainobjectplugintest.EZB


ARC Pro

Upgrade to ARC Pro

ARC Pro is your passport to a world of endless possibilities in robot programming, waiting for you to explore.

PRO
USA
#1   — Edited

Do you have to make numerous scripts for each item learned or can one script handle it all?

Where are the learned items stored?

EzAng

PRO
USA
#2  

has anyone used this control?

PRO
Synthiam
#3  

Watch the tutorial video or read the manual would be two good places to start

PRO
USA
#4  

I will do it, had to get to a place to watch video, farm no internet

Thanks

EzAng

PRO
Belgium
#5  

dj

your picture off sleep&power is diff then mine. yours has a slider .here's mine. do i have an older version win10? User-inserted image

PRO
USA
#6   — Edited

Great tutorial - I learned the java script code in "train object by script". Now  my robot can learn many things :-)

EzAng

#7  

Great Video DJ. Its bene awhile but I am finally getting around to trying it. I am trying to recreate all the script that you showed in the video, but the video would get blurry at times as you'd type away so I am not able to see clearly some of the syntax being entered, esp around the "if" statement. This is my first attempt to use the JavaScript feature also. You wouldn't happen to have that file that was created handy? :-)

PRO
USA
#8   — Edited

Hey RobHappy

try DJ script above, it works for me, my camera recognizes all the objects I trained it to see, you can change the items in the script:

This is the name of the object that will be stored

$LearnedObject = "cup"

example, cup, bottle, face etc... "

add BingSpeech, add a Camera (in the Camera script add : Audio.say("I see the "); Audio.say (getVar("$CameraObjectName"));

Add this is a script

if (!$IsCameraActive) say("The camera is not active") halt() endif

This is the name of the object that will be stored

$LearnedObject = "cup"

Attach to the camera for the control

ControlCommand("Train Vision Object By Script" , "Attach")

Introduce verbally out of the PC speaker what it is about to happen

SayWait("Prepare to learn " + $LearnedObject + ". 3, 2, 1, learning")

Instruct the plugin to begin learning

ControlCommand("Train Vision Object By Script", Learn, $BingSpeech)

Wait until the plugin has completed learning the object

WaitFor($CameraIsLearning == 0)

Speak what the object has been learned

Say("I have learned " + $LearnedObject)

Enable object tracking in the camera control

ControlCommand("Camera", CameraObjectTracking)

when you start the script, make sure the pink square in the Camera is totally covered with the object you are training

Good to see you also

EzAng

#9  

Hi EzAng

Thanks for this.  I tried it out, but it gave an error about using the "." for the $LearnedObject = line.

Also, just wanted to clarify, the first part of the program, the Audio.say(   ) gets entered in the Bing Speech script or in the camera ?

PRO
USA
#10   — Edited

follow the one below

PRO
USA
#11   — Edited

did it work for you?

#12  

HI EzAng, Thanks for the quick reply :)

I must be missing something, Still cannot get it to work here. Its late so I will try again tomorrow hopefully.

ttyl :)

PRO
USA
#13   — Edited

Hi, good morning,  try this:

DJ's code Add:

Camera code in camera

Audio.sayWait("I see the ");

Audio.say (getVar("$CameraObjectName"));

Conversational Menu Bing Speech Recognition Train Object By Script Speech Recognition

In Speech Recognition add:

learn a new object

code:   Audio.sayWait("what is the name of the new object you want me to learn");

setVar("$BingSpeech", "" );

ControlCommand("Bing Speech Recognition", "StartListening");

while (getVar("$BingSpeech") == "" );

sleep(250);

if (getVar("$BingSpeech") == "cancel"){

Audio.sayWait("cancelling");

return; }

Audio.sayWait("Let me know when you are ready to teach me what a " + getVar("$BingSpeech") + "is." );

var resp = Audio.waitForSpeech(60, "ok", "cancel");

if (resp != "ok"){ Audio.sayWait("ok, I will not learn the "+ getVar("$BingSpeech"));

return; }

ControlCommand("Camera", "CameraObjectTrackingDisable");

ControlCommand("Train Vision Object By Script", "Learn", getVar("$BingSpeech"));

sleep(1000);

while (getVar("$CameraIsLearning"));

sleep(250);

Audio.SayWait("Great, I know what a " + getVar("$BingSpeech") + "is.");

ControlCommand("Camera", "CameraObjectTracking");

It works here, let me know

#14  

HI EzAng,:) thanks again. I will hopefully try it later on tonight. One question, that Im still stuck on. The First part, Where you say code in camera , is to add the Audio.say command script into the Camera?  Do I enter this under the Scripts&variables tab inside the camera config? (add to camera start under Activity tag?) Thats what Im a bit lost with :-) Yeah I'm a lil bit out of practice with this stuff. And its a Java thing right?  Rest of this I think I got.

Will def let you know if I get it working here. Look forward to it. :D

Ps. Hopefully soon to start another bot build. I have a wheeled platform kit coming from SuperDroid on the way to start things off.

PRO
USA
#15  

Great,

Yes, add Camera,  then in the  Scripts&variables tab inside the camera config

add: Audio.sayWait("I see the  "); Audio.say (getVar("$CameraObjectName"));

correct the rest is java script is the "Speech Recognition" control

include the: Conversational Menu Bing Speech Recognition Train Object By Script

and you are set to go

#16  

EzAng, What does the Conversational Menu do for this script? I assume that when I press START in the Bing Speech I then ask it to "learn a new Object", then I click to STOP after I state name ofobject before it continues on?

PRO
USA
#17   — Edited

Add nothing to the Conversational Menu Add nothing to the BingSpeech control Add nothing to t "Train Vision Object by Script" control

All the java script code goes into "Speech Recognition" control

User-inserted image

save

then say into your computer mic, (the Speech Recognition) "learn a new object"

sometimes I have to click "stop recording" in Bing

then it will be saved to your camera control, objects, trained images

you can keep adding other objects

that's it

PRO
USA
#18  

RoboHappy, did you get it going yet?

#19  

Hi EzAng, Sadly no :( I have not and I am not sure exactly why.  I have been either running into microphone sensitivity issies , which I have been playing with or it seems the program comes back with variable or syntax issues. Still not sure i did put the audio.say in the right place in the camera. I am still learning this java stuff too.

I did find an earlier EZscript version around here, which seems to work but its not fancy, it doesn't let you verbally say the objects name prior to it learning the object. have to type the name in.

Ive also tried to repeat the setup that DJ did with his inmoov via a livehack episode, and pretty much ran into many errors with that as well. Maybe its my setup? I am using windows 10 and am using latest and greatest ARC and updated all my plugins.

I am bummed yes, this is one of the coolest features one can do with the camera and really need to find where my issue is. Any suggestions at this point would be greatly appreciated. I wish I could spend more time learning too, but that not in the cards for me right now. I just do what I can.

PRO
USA
#20   — Edited

typing is never ask good as talking, what is your number, I will call you or:

step 1: open ARC add:

Conversational Menu BingSpeech control Train Vision Object by Script" control

Add nothing to the them

step 2: add a Camera

then in the Scripts&variables tab inside the camera config

add the java script below

Audio.sayWait("I see the "); Audio.say (getVar("$CameraObjectName"));

step 3: Add

Speech Recognition

in the config - phrases:

add a new phrase - " Learn a new object"

next to that, under the Command - click on it and add the java script below

User-inserted image

copy and paste it

under java script

Audio.sayWait("what is the name of the new object you want me to learn");

setVar("$BingSpeech", "" );

ControlCommand("Bing Speech Recognition", "StartListening");

while (getVar("$BingSpeech") == "" );

sleep(250);

if (getVar("$BingSpeech") == "cancel"){

Audio.sayWait("cancelling");

return; }

Audio.sayWait("Let me know when you are ready to teach me what a " + getVar("$BingSpeech") + "is." );

var resp = Audio.waitForSpeech(60, "ok", "cancel");

if (resp != "ok"){ Audio.sayWait("ok, I will not learn the "+ getVar("$BingSpeech"));

return; }

ControlCommand("Camera", "CameraObjectTrackingDisable");

ControlCommand("Train Vision Object By Script", "Learn", getVar("$BingSpeech"));

sleep(1000);

while (getVar("$CameraIsLearning"));

sleep(250);

Audio.SayWait("Great, I know what a " + getVar("$BingSpeech") + "is.");

ControlCommand("Camera", "CameraObjectTracking");

save

that is it

#21   — Edited

Hi EzAng

Thanks again. I will try the program again. :)

User-inserted image But first, Just wanted to clarify adding the Audio  scripts in the camera setting again.

step 2: add a Camera

then in the Scripts&variables tab inside the camera config

add the java script below

Audio.sayWait("I see the "); Audio.say (getVar("$CameraObjectName"));

As you can se, there are two additonal boxes to add script(start and stop as shown) So I wanted to verify this part again in case I have something different than yours. :D

PRO
USA
#22  

yes in Tracking start

#23  

Ok, So I got the program to where it does ask me the name of the object to learn, but after that, nothin happens. It never begins to learn or even tell me if the camera is not activated. Just nothing happens,

PRO
USA
#24   — Edited

In the Camera control, click on tracking, highlight object, then under that, you will see Object, click that.

make sure your "train Vision Object by script" control sees your camera, click "attach Camera"

After you say "learn a new object" give Bing a few seconds, then click "stop recording" in Bing - sometimes you have to do this.

In Speech Recognition I put the confidence to 0.55, see below so it hears me.

check the java script you put in "speech recognition", should end up looking like this User-inserted image

#25  

Hi EzAng, Thank you again for the help:) But I am going to give up on this for now. I cannot get your code or even the code from DJs live hack to work properly. I Have looked over it and corrected any and all my syntax errors several times now but still getting the results I wanted. At least the old fashion way of doing it manually still works, somewhat. One of the biggest hurtles I keep running into with this is that after the program ask me for the name of the object, the program just seems to stop (Bingspeech will just timeout) or if it does take my answer, then when is responds that it will learn name of new object, Bing responds with "learning a new object" as the name of the object instead. this has baffled me to no end, so I will take a break from this and tray again some other time.

#26   — Edited

@RobotHappy.

Try putting this code in Speech Recognition add a new phrase - " Learn a new object" next to that, under the Command line , click on it and add the java script below...

Audio.sayWait("Okay, what is the name of the new object you want me to learn");

setVar("$BingSpeech", "" );

ControlCommand("Bing Speech Recognition", "StartListening");

while (getVar("$BingSpeech") == "" )

sleep(250);

if (getVar("$BingSpeech") == "cancel"){

Audio.sayWait("cancelling");


return;
}


Audio.sayWait("Excellent. Let me know when you are ready to teach me what a " + getVar("$BingSpeech") + "is." );

var resp = Audio.waitForSpeech(60, "ok", "cancel");

if (resp != "Okay"){
Audio.sayWait("Okay, I will not learn the "+ getVar("$BingSpeech"));

return;
}
  
ControlCommand("Camera", "CameraObjectTrackingDisable");

ControlCommand("Train Vision Object By Script", "Learn", getVar("$BingSpeech"));
sleep(1000);

while (getVar("$CameraIsLearning"))
  sleep(250);
  
  Audio.SayWait("Great, I know what a " + getVar("$BingSpeech") + "is.");
   ControlCommand ("camera", "CameraObjecTracking");

This is work on my robot...

EDIT: This is not complete...

PRO
USA
#27   — Edited

merne

here is DJ's code from the train object by script video

Put this code in Speech Recognition add a new phrase - " Learn a new object" next to that, under the Command line , click on it and add the java script below

Audio.sayWait("what is the name of the new object you want me to learn"); setVar("$BingSpeech", "" ); ControlCommand("Bing Speech Recognition", "StartListening"); while (getVar("$BingSpeech") == "" ); sleep(250); if (getVar("$BingSpeech") == "cancel"){ Audio.sayWait("cancelling"); return; }

Audio.sayWait("Let me know when you are ready to teach me what the " + getVar("$BingSpeech") + "is." );

var resp = Audio.waitForSpeech(60, "ready", "cancel");

if (resp != "ready"){ Audio.sayWait("ok, I will not learn the "+ getVar("$BingSpeech"));

return; }

ControlCommand("Camera", "CameraObjectTrackingDisable");

ControlCommand("Train Vision Object By Script", "Learn", getVar("$BingSpeech"));

sleep(1000);

while (getVar("$CameraIsLearning"));

sleep(250);

Audio.SayWait("Great, I know what a " + getVar("$BingSpeech") + "is.");

ControlCommand("Camera", "CameraObjectTracking");

*Camera then in the Scripts&variables tab inside the camera config

add the java script below

Audio.sayWait("I see the "); Audio.say (getVar("$CameraObjectName"));

Sometimes you might have to stop Bing from recording

that's it, this works great here, In Camera, objects, add trained images then Camera recognizes them

PRO
Synthiam
#28  

Here you go: https://synthiam.com/Community/Apps/train-object-by-script-19553

#29  

Thank you DJ.

Is they a way to only delete selected objects without deleting all object created?  Cheers

Thank you too @EzAng.

PRO
USA
#30  

You are welcome, merne,

Quote:

Is they a way to only delete selected objects without deleting all object created? Cheers
In Camera, Object, trained images, that would be a nice feature to be able to delete one image you don't want instead of all of them like it is now.