I'm almost done with the Vision Recognition and Conversation Services.

They are simple to use, watson handles all the setup and plumbing for you.

I'm working on a demo but requires additional hardware and low level code so i'm busy ..

If anyone have ideas for a demo using a Six or JD please contribute.

Great work @ptp as always. I look forward to try it out.

I'm running a series of human-robot interaction experiments with a couple of students at my lab in about a month from now. We could see if we could integrate this module in one of the scenarios with the JD. If it works out we can add some performance metrics, which will give you an idea how the system performs in an interactional setting. If nothing else, I can give you a few videos of people interacting with the system. That might be a bit more interesting than a video. Let me know what you think

@larschrjensen,

Thanks for the feedback, and yes more feedback is welcome and it helps to generate/fuel more ideas.

@ptp I've been playing around with the plugin and it works quite well. I've noticed that you can't set the silence threshold below 2000 ms. This effectively means that there is at least a 2+ second pause between an utterance and a response. This is quite long considering that people normally expect a response within 300 msec. Is it possible to change this? I realize this can possibly affect the perfomance and send off utterances prematurely, but it would make the plugin a bit more flexible.

@larschrjensen,

Done, the minimum value is set to 500 ms.

The audio's capture buffer is 100 ms, the handler gets notifications every 100 ms.

Let me know if is ok, i can change to 300 ms.

I was in middle of something, i hope no harm done.

Fantastic, thanks.

Just noticed visual recognition update. Nice. I had a quick play, looks great, I will need to learn how to train a model in Watson Studio.

Curious is there a way to get ControlCommand to wait until visual recognition is complete as I found I was reading the previous variables and not the new ones. I tried a sleep and that seemed to work but I was setting for 1000 and sometimes was not long enough. I played with setting $WatsonClassifyTypes[0] ="wait" and then waiting until != "wait" but kept getting stuck in a loop. I am not sure if there is a better way. I was triggering visual off speech to text "What do you see"

@Nink,

1) You can add your script to the VR script, it will be executed when the visual recognition is complete.

SayWait("I see")
$size=GetArraySize("$WatsonClassifyClasses")-1
Repeat($ix, 0, $size, 1)
  #Adjust the min score e.g. 0.60
  if ($WatsonClassifyScores[$ix]>=0.60)
    SayWait($WatsonClassifyClasses[$ix]) 
  EndIf 
EndRepeat

add a small VR script:

$VisualRecognitionCounter=$VisualRecognitionCounter+1

then monitor the variable in another script. The variable must be initialized in an init script.

3) There is an internal CaptureId variable. I'll expose the variable in the next update. And you will be able to monitor the $WatsonPictureCaptureId variable.

To avoid begin caught between two different calls, the VR script (1) is the best option. After the VR is done, the variables are assigned with the results and the VR script is executed. If there is another VR call the results are queued waiting for the VR script to finish. While your VR script is being executed the results (e.g. variables) are constant and related to the current call.

I'm experiencing an issue with the plugin. After running for about 30 minutes ARC will freeze up. According to windows the program is still responding, but effectively it is not. Ram usage also increases from around 300 mb to 600 mb. It only happens when listening is active . STT status doesn't make a difference. No other project controls or plugins are running. I can reproduce this error on a second pc in a different project.

Any thoughts?

@larschrjensen

I fixed a memory leak. Let me know if the new version fixes the problem.

@ptp Indeed it does. Thanks

Thanks @ptp. I feel a little silly now. Worked like a charm. I was calling the VR from speech to text and then processing in the text to speech script and not creating a script after VR processed. User Problem :-)

@nink

Please send me a email, I wish to discuss something with you. My addy is in my profile.

Thanks

The following posts are related to the Visual Recognition.

I took a picture of playing card (Seven of Clubs)

Then i performed a few quick tests:

IBM Watson Visual Recognition Services:

Microsoft Computer Vision API :

Google:

It's obvious these results don't help.

Step 1: Create and train an image custom classifier:

I've used the EZ-Robot camera V2, it's important to use the same camera for training and recognition.

Using the ARC camera snapshot control, i took 10-15 pictures of each card (7,8,9,Jack,King) of Clubs.

It's important to test different light conditions, angles, orientations etc.

I created a zip file one for each card image folder.

then i run the following script:

#Don't use dash or symbols for class names
DefineArray($Classes,5) 
$Classes[0]="SevenOfClubs"
$Classes[1]="EightOfClubs"
$Classes[2]="NineOfClubs"
$Classes[3]="JackOfClubs"
$Classes[4]="KingOfClubs"

DefineArray($ZipFiles,5) 
$ZipFiles[0]="C:\Users\ptp\Pictures\My Robot Pictures\7-Clubs.zip"
$ZipFiles[1]="C:\Users\ptp\Pictures\My Robot Pictures\8-Clubs.zip"
$ZipFiles[2]="C:\Users\ptp\Pictures\My Robot Pictures\9-Clubs.zip"
$ZipFiles[3]="C:\Users\ptp\Pictures\My Robot Pictures\Jack-Clubs.zip"
$ZipFiles[4]="C:\Users\ptp\Pictures\My Robot Pictures\King-Clubs.zip"

ControlCommand("IBM Watson Services", CreateClassifier, "PlayingCards", "$Classes", "$ZipFiles")

This will upload the images files and will create a Custom classifier name: PlayingCards with 5 classes (SevenOfClubs ... KingOfClubs)

The training will start after the upload, can take a few minutes, depends on the number of classes, pictures, picture size etc.

Step 2: List the existent custom classifiers and their status e.g. Ready, Training.

ControlCommand("IBM Watson Services", GetClassifiers, "$Classifiers" )
$NumOfClassifiers=GetArraySize("$Classifiers" )
print("Found " + $NumOfClassifiers + " classifiers")

IF ($NumOfClassifiers>0)
  REPEAT ($ix, 0, $NumOfClassifiers-1, 1)
    $ClassifierId=$Classifiers[$ix]
    print("ClassifierId=" + $ClassifierId)

    ControlCommand("IBM Watson Services", GetClassifier, $ClassifierId, "$ClassifierInfo", "$ClassifierClasses" )
    print("...Name="+$ClassifierInfo[1])
    print("...Status="+$ClassifierInfo[2])
    print("...CreatedOn="+$ClassifierInfo[3])
    print("...Errors="+$ClassifierInfo[4])

    $NumOfClasses=GetArraySize("$ClassifierClasses" )
    print("...Found " + $NumOfClasses + " classes:" )
    REPEAT ($cx, 0, $NumOfClasses-1, 1)
      print("......" + $ClassifierClasses[$cx])
    ENDREPEAT

  ENDREPEAT
ENDIF

is training:

running again the same code:

it's ready to be used.

Step 3: Use the custom classifier

The following code uses the default (watson classifiers)

ControlCommand("IBM Watson Services", VisualRecognition)

The following code uses the above custom classifier.

ControlCommand("IBM Watson Services", VisualRecognition, "PlayingCards_978301467")

Note: PlayingCards_978301467 is the classifierId, PlayingCards is the classifier name.

The following code uses the default and the above custom classifier.

ControlCommand("IBM Watson Services", VisualRecognition, "default,PlayingCards_978301467")

Note: you can pass multiple classifier ids

Testing the initial picture (Seven of Clubs) with the custom classifier:

SevenOfClubs class has the highest score.

You are amaizing.

Once you have a classifierId (after you create one) you can update a classifier adding new classes e.g. cards, or delete the existent classifier, please check the ControlCommand cheat.

The free service i.e. Lite, only allows one custom classifier e.g. "PlayingCards", and does not allow updates to the existing classifier.

So the only solution, is to delete the existent classifier, and create again with more classes or create another one for another purpose, or if you feel confident move to the standard service.

bear in mind, a good training requires pictures, more than merrie.

I did a few tests (with 10 pictures per cards) an eight of clubs can get higher score in a specific angle/light.... the solution is to train more pictures.

Thanks Nink.

Taking the pictures, takes time, it's very important the small details: playing with the card, putting something below to create a distorted image, change the distance/orientation from the camera, play with the light etc, maybe introduce some minors errors (fingers while holding ?).

Although the Lite model does not allow incremental updates, i plan to use my assistants e.g. Kids, to take more pictures and then I'll try to create the classifier with all the zip files (big upload).

Thanks for this ptp. My daughter is visiting so I am entertaining this weekend but will get back to this during the week and start taking photos of cards with an ez-robot camera.

I uploaded about 15 cards 15 photos each in various lighting. Various levels of success. I am using a cheap deck I picked up at local dollar store (My logic was I can buy a lot of the same cards cheap). They don't look quite like a normal deck so I may try a better deck of cards but I am confident I can get it to the point where it will recognize all 52 cards with lots of training.

Now the next question is, How do I get it to recognize multiple cards at once? Bounding boxes?

@Nink,

Quote:

Now the next question is, How do I get it to recognize multiple cards at once? Bounding boxes?

There are a few ideas using opencv and tensorflow.

tensorflow with Watson: https://medium.com/ibm-watson/dont-miss-your-target-object-detection-with-tensorflow-and-watson-488e24226ef3

Tensorflow on linux is well supported you will find a lot of examples with linux and python, some of them on a Raspberry PI. Windows is a different story.

I'm exploring the windows implementation, but, so far it's not possible to built a ARC skill plugin, most libraries are x64 and ARC is a x86 (32) bits application.

The solution is to built a 64 bits application to handle the tensorflow logic and create a communication layer between the application and the ARC skill plugin.

OK thanks ptp. It sounds like it is in the to hard basket for now :-) I will just stick with the one card at a time :-)

I photographed all the cards if you want them but some need a lot more photos (or I have too many bad quality photos) as I am getting a lot of false positives. I wish the free version of Watson VR would allow us to retrain when we have an issue (No it is not an ace of clubs it is an ace of spades).

I can send anyone who wants a physical deck that I used, I Purchased them DollarRama (Canadian Dollar Store) They are Victoria brand.

Here is my edit of ptp script so others don't need to re type. Note my pictures location is different so you will have to change path but a search and replace in your fav editor is easier than retyping and debugging (Get a zip file name wrong and it will not work).

#Don't use dash or symbols for class names
DefineArray($Classes,52) 
$Classes[0]="AceOfSpades"
$Classes[1]="TwoOfSpades"
$Classes[2]="ThreeOfSpades"
$Classes[3]="FourOfSpades"
$Classes[4]="FiveOfSpades"
$Classes[5]="SixOfSpades"
$Classes[6]="SevenOfSpades"
$Classes[7]="EightOfSpades"
$Classes[8]="NineOfSpades"
$Classes[9]="TenOfSpades"
$Classes[10]="JackOfSpades"
$Classes[11]="QueenOfSpades"
$Classes[12]="KingOfSpades"
$Classes[13]="AceOfClubs"
$Classes[14]="TwoOfClubs"
$Classes[15]="ThreeOfClubs"
$Classes[16]="FourOfClubs"
$Classes[17]="FiveOfClubs"
$Classes[18]="SixOfClubs"
$Classes[19]="SevenOfClubs"
$Classes[20]="EightOfClubs"
$Classes[21]="NineOfClubs"
$Classes[22]="TenOfClubs"
$Classes[23]="JackOfClubs"
$Classes[24]="QueenOfClubs"
$Classes[25]="KingOfClubs"
$Classes[26]="AceOfHearts"
$Classes[27]="TwoOfHearts"
$Classes[28]="ThreeOfHearts"
$Classes[29]="FourOfHearts"
$Classes[30]="FiveOfHearts"
$Classes[31]="SixOfHearts"
$Classes[32]="SevenOfHearts"
$Classes[33]="EightOfHearts"
$Classes[34]="NineOfHearts"
$Classes[35]="TenOfHearts"
$Classes[36]="JackOfHearts"
$Classes[37]="QueenOfHearts"
$Classes[38]="KingOfHearts"
$Classes[39]="AceOfDiamonds"
$Classes[40]="TwoOfDiamonds"
$Classes[41]="ThreeOfDiamonds"
$Classes[42]="FourOfDiamonds"
$Classes[43]="FiveOfDiamonds"
$Classes[44]="SixOfDiamonds"
$Classes[45]="SevenOfDiamonds"
$Classes[46]="EightOfDiamonds"
$Classes[47]="NineOfDiamonds"
$Classes[48]="TenOfDiamonds"
$Classes[49]="JackOfDiamonds"
$Classes[50]="QueenOfDiamonds"
$Classes[51]="KingOfDiamonds"



DefineArray($ZipFiles,52) 
$ZipFiles[0]="Y:\Pictures\My Robot Pictures\Ace of Spades.zip"
$ZipFiles[1]="Y:\Pictures\My Robot Pictures\Two of Spades.zip"
$ZipFiles[2]="Y:\Pictures\My Robot Pictures\Three of Spades.zip"
$ZipFiles[3]="Y:\Pictures\My Robot Pictures\Four of Spades.zip"
$ZipFiles[4]="Y:\Pictures\My Robot Pictures\Five of Spades.zip"
$ZipFiles[5]="Y:\Pictures\My Robot Pictures\Six of Spades.zip"
$ZipFiles[6]="Y:\Pictures\My Robot Pictures\Seven of Spades.zip"
$ZipFiles[7]="Y:\Pictures\My Robot Pictures\Eight of Spades.zip"
$ZipFiles[8]="Y:\Pictures\My Robot Pictures\Nine of Spades.zip"
$ZipFiles[9]="Y:\Pictures\My Robot Pictures\Ten of Spades.zip"
$ZipFiles[10]="Y:\Pictures\My Robot Pictures\Jack of Spades.zip"
$ZipFiles[11]="Y:\Pictures\My Robot Pictures\Queen of Spades.zip"
$ZipFiles[12]="Y:\Pictures\My Robot Pictures\King of Spades.zip"
$ZipFiles[13]="Y:\Pictures\My Robot Pictures\Ace of Clubs.zip"
$ZipFiles[14]="Y:\Pictures\My Robot Pictures\Two of Clubs.zip"
$ZipFiles[15]="Y:\Pictures\My Robot Pictures\Three of Clubs.zip"
$ZipFiles[16]="Y:\Pictures\My Robot Pictures\Four of Clubs.zip"
$ZipFiles[17]="Y:\Pictures\My Robot Pictures\Five of Clubs.zip"
$ZipFiles[18]="Y:\Pictures\My Robot Pictures\Six of Clubs.zip"
$ZipFiles[19]="Y:\Pictures\My Robot Pictures\Seven of Clubs.zip"
$ZipFiles[20]="Y:\Pictures\My Robot Pictures\Eight of Clubs.zip"
$ZipFiles[21]="Y:\Pictures\My Robot Pictures\Nine of Clubs.zip"
$ZipFiles[22]="Y:\Pictures\My Robot Pictures\Ten of Clubs.zip"
$ZipFiles[23]="Y:\Pictures\My Robot Pictures\Jack of Clubs.zip"
$ZipFiles[24]="Y:\Pictures\My Robot Pictures\Queen of Clubs.zip"
$ZipFiles[25]="Y:\Pictures\My Robot Pictures\King of Clubs.zip"
$ZipFiles[26]="Y:\Pictures\My Robot Pictures\Ace of Hearts.zip"
$ZipFiles[27]="Y:\Pictures\My Robot Pictures\Two of Hearts.zip"
$ZipFiles[28]="Y:\Pictures\My Robot Pictures\Three of Hearts.zip"
$ZipFiles[29]="Y:\Pictures\My Robot Pictures\Four of Hearts.zip"
$ZipFiles[30]="Y:\Pictures\My Robot Pictures\Five of Hearts.zip"
$ZipFiles[31]="Y:\Pictures\My Robot Pictures\Six of Hearts.zip"
$ZipFiles[32]="Y:\Pictures\My Robot Pictures\Seven of Hearts.zip"
$ZipFiles[33]="Y:\Pictures\My Robot Pictures\Eight of Hearts.zip"
$ZipFiles[34]="Y:\Pictures\My Robot Pictures\Nine of Hearts.zip"
$ZipFiles[35]="Y:\Pictures\My Robot Pictures\Ten of Hearts.zip"
$ZipFiles[36]="Y:\Pictures\My Robot Pictures\Jack of Hearts.zip"
$ZipFiles[37]="Y:\Pictures\My Robot Pictures\Queen of Hearts.zip"
$ZipFiles[38]="Y:\Pictures\My Robot Pictures\King of Hearts.zip"
$ZipFiles[39]="Y:\Pictures\My Robot Pictures\Ace of Diamonds.zip"
$ZipFiles[40]="Y:\Pictures\My Robot Pictures\Two of Diamonds.zip"
$ZipFiles[41]="Y:\Pictures\My Robot Pictures\Three of Diamonds.zip"
$ZipFiles[42]="Y:\Pictures\My Robot Pictures\Four of Diamonds.zip"
$ZipFiles[43]="Y:\Pictures\My Robot Pictures\Five of Diamonds.zip"
$ZipFiles[44]="Y:\Pictures\My Robot Pictures\Six of Diamonds.zip"
$ZipFiles[45]="Y:\Pictures\My Robot Pictures\Seven of Diamonds.zip"
$ZipFiles[46]="Y:\Pictures\My Robot Pictures\Eight of Diamonds.zip"
$ZipFiles[47]="Y:\Pictures\My Robot Pictures\Nine of Diamonds.zip"
$ZipFiles[48]="Y:\Pictures\My Robot Pictures\Ten of Diamonds.zip"
$ZipFiles[49]="Y:\Pictures\My Robot Pictures\Jack of Diamonds.zip"
$ZipFiles[50]="Y:\Pictures\My Robot Pictures\Queen of Diamonds.zip"
$ZipFiles[51]="Y:\Pictures\My Robot Pictures\King of Diamonds.zip"

ControlCommand("IBM Watson Services", CreateClassifier, "PlayingCards", "$Classes", "$ZipFiles")

I spent about 10 hours of photographing, uploading, training (Takes about an hour to train now with over 2000 photo's) using an EZ-B and Camera placing cards on table and changing position / lighting. I had about 80% of the deck with very high accuracy. If it didn't read a card correctly I would take more photos, upload train and repeat.

It can take anywhere from 2 to 20 seconds to come back with an answer now when you show a card.

Now confident I was ready, I pulled out JD and held up a card in front of him. Not a clue (about 5% accuracy).

This tells me my training method is bad or we need a very controlled environment.
I think we could probably do a controlled read if we used a static base with controlled lighting, Example a Game of Blackjack (21) between JD and Six and me as the dealer, with JD and Six sitting in specific positions and cards placed on playing board in predetermined locations.

The other alternative is I create a method that is automated for reading cards and taking photo's. Perhaps it is something like hand JD a Card and then have a script run so he can then turn around in a circle and change the angle of the card taking pictures so the lighting, angle and background keep changing. Then when he is finished (Say 100 photos) upload the photos he took and ask for another card.

Thinking ....

Edit - Turns Out JD dexterity is not good enough to read a card he is holding (Can't get good enough angle / distance) so it will require a custom bot to read cards (More Servo's and perhaps a 360 Servo). Also it appear not just the same model camera is good enough, it has to be the exact same camera. If I take photo's with JD and try to read them on EZ-B + Camera doesn't work and vise versa.

@Nick,

Quote:

It sounds like it is in the to hard basket for now :-)

Quote:

I Purchased them DollarRama (Canadian Dollar Store) They are Victoria brand.

I can't find that brand, maybe you can suggest one available both in amazon.ca and amazon.com. That way we could share pictures.

I took pictures too... i wanted to test two different resolutions: 320x240 and 640x480, unfortunately the high resolution pictures are low resolution, the snapshot control ignored the camera control setting.

I'll add a picture/snapshot functionality to the watson plugin.

Can you share a zip of one your cards ?

Hi ptp. Here is one card. I am apprehensive sending photos of all the cards due to copyright laws. Maybe we should Print off some creative commons cards.

NineofHearts.zip

Plan B

JD a record player, an iPad playing a Netflix movie.

Have you seen the .net bindings for tensor flow for a plugin? https://www.nuget.org/packages/TensorFlowSharp/1.6.0-pre1

@DJ,

Yes, the wrapper works on top tensorflow.dll (libtensorflow.dll) available from the tensorflow project.

https://github.com/tensorflow/tensorflow/issues/10817

sources: https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-windows-x86_64-1.2.0.zip

http://ci.tensorflow.org/view/Nightly/job/nightly-libtensorflow-windows/lastSuccessfulBuild/artifact/lib_package/libtensorflow-cpu-windows-x86_64.zip

the tensorflow.dll is 64 bits.

@Nink,

Can you explain your plan B

Regarding the playing cards, we don't need to exchange publicly picture files, let's find a common deck available on both amazons.

OK My deck was this one on Amazon.ca not on amazon.com although I pay $1.25 and they want $7 on amazon. I can put a deck in the mail if you want, or if you can provide a link to your cards on amazon.ca I can order a deck. https://www.amazon.ca/gp/offer-listing/B01A61ZQI8/ref=sr_1_1_olp?ie=UTF8&qid=1522329419&sr=8-1&keywords=victoria+playing+cards&condition=new

Plan B was to just take 100 photos of each card (1 photo every second) in camera app and have background and angle changing using Ipad and record player then move robot head up and down as I do it while creating shadows and changing lighting. Logic was I could get through this in about 2 hours.

I have been thinking about a good use case for the ability to read cards. Obviously a QR code can be read easily so this makes me wonder if this is a good application for vision recognition, but at least we are learning and will make a good demo.

Hi @ptp. I wonder if we need to break the problem into smaller chunks so the problem is easier for Watson to solve and then string these together. We are not providing any negative classes. We only say what each card is, but not what it isn’t.

Colour Cluster. Red cards positive and Black cards negative Is it a red card or black card. this should be a relatively easy problem to solve. Assume red

Red Suit Cluster Hearts positive and diamonds negative. Again relatively easy problem to solve. Assume hearts.

Hearts Suit Cluster Picture positive Numeric Negative. Assume Picture.

Hearts Picture Cluster. Is it jack queen king or ace of hearts.

If we could do this and then somehow feed that back into a machine learning I am wondering if this would be very accurate and quick to process?

Hi @ptp, Do you think you might be able to build an ARC skill plugin that can run TensorFlow like this:

https://www.youtube.com/watch?v=_zZe27JYi8Y

and return the objects' names and their x,y coordinates?

Thomas

Sorry the delay... I'm away for the holidays, school break for Passover / Easter.

subjects:

1. playing cards I'll get this playing cards: https://www.amazon.com/Bicycle-Playing-Card-Deck-2-Pack/dp/B010F6BAES They are available in the amazon US,UK,CA. It will be more easy to share (not public) the training pictures.

2. plugin roadmap soon i plan to update the plugin to support training with negative samples, and the assistant (formerly conversation) API, although there are some gaps to fill.

3. offline object detection This will take a little longer, still researching, most implementations e.g. libraries are 64 bits and require some resources (memory & cpu and when possible a gpu), some plumbing is needed between ARC/plugins and an external application.

@Thomas:

Watson: I presume you managed to find and setup the watson visual recognition.

Machine Learning: The main problem is the Hype regarding machine learning... we are far from machine learning, and AI, and the robots seeing and taking the world ... What we have is massive data dumped to massive computational hardware to obtain some clues...

Tensorflow: With a core I7 without a nvidia GPU and a custom compilation for windows x64 and AVX2 extension enabled (not available in all cpus https://en.wikipedia.org/wiki/Advanced_Vector_Extensions) the quickest model (less accuracy) took 3.5 seconds and a more accurate model took 45 seconds to classify/detect objects in a single photo.

Unless i'm doing something wrong, less time requires a decent GPU and or a different model.

I'll post in another thread more details.

Have you considered building a plugin that expands the capability of ezrobot's existing object training? It's much faster than using tensorflow or yolo on darknet. That's built using Yeremeyev method: http://edv-detail.narod.ru/AVM_main.html

I ordered these ones from amazon.ca I think they are the same. I don't want to get hit with customs and duties for cross border. https://www.amazon.ca/gp/offer-listing/B00001QHVP/ref=olp_twister_child?ie=UTF8&mv_style_name=1&qid=1522716936&sr=8-4

Conversations/Assistant WOW this is cool. Amazing thanks @ptp I did a quick voice to text => Assistant => Text to voice. Works well despite my strong accent. I can see this very valuable when controlling and interacting with robot using voice recognition and general conversation.

Found small problem In the Car example under dialogue there was a dialogue "anything else" true and it had the statement I am not sure about that you can "Turn on my lights" with quotation marks that caused script to crash so I just deleted that dialogue response as it was causing errors. All good.

@Nink, Fixed. EZ-Script does not like quotes, quotes are replaced with ( ' ).

One simple example: Init script (Run before the Watson Assistant)

#Initialize context/environment variables
$RadioStatus=0

Assistant Script:

IF ($WatsonAssistIntents[0]="turn_on" )
  IF ($WatsonAssistEntities[0]="appliance" )
    IF ($WatsonAssistEntitiesValues[0]="music" )
      if ($RadioStatus=0)
        $RadioStatus=1
        SayWait("Radio is ON" ) 
      else
        SayWait("Dear master, the radio is already ON; May I suggest an appointment with an otolaryngologist ?" ) 
      EndIf 
    ENDIF
  ENDIF
ElseIf ($WatsonAssistIntents[0]="turn_off" )
  IF ($WatsonAssistEntities[0]="appliance" )
    IF ($WatsonAssistEntitiesValues[0]="music" )
      if ($RadioStatus=1)
        $RadioStatus=0
        SayWait("Radio is OFF" ) 
      else
        SayWait("Dear master, the radio is already OFF" ) 
      EndIf 
    ENDIF
  ENDIF
ENDIF

I called him Jarvis

Listen what happens when you try to switch on the radio when the radio is already on

To help the tests you can replay a previous assistant message response:

ControlCommand("IBM Watson Services", ReplayMessage, "1522815312")

1522815312 is a previous CaptureId.

thanks @PTP sorry this was my bad. The problem was the person who included quotation marks in a programmed response. This was neither your issue or ez-Robots. What were they thinking ...

I like the new previous response test variable as I get lost in where we are in the conversation. Are we in the middle of a conversation or starting a new one?

Wanna know the best option? Bluetooth earpiece they’re cheap and work really well. eBay or amazon for a few dollars and voila.

Just like the ones that cheesy real estate people where for their phone. Connects to your pc

I was actually going to use a Bluetooth hand held microphone. Had a lot of trouble with NAOs in the past in a crowded room doing demos. The robots built in mic’s pick up all the background noise so in a demo they don’t hear the commands. Hand held mic you can hand from person to person and the robot can only hear the person speaking.

PTP did some amaizing work on allowing you to configure DB levels and delays with the voice capture plugin he wrote so this makes it really easy to tune to the appropriate responses.

Yes the mics in the NAO is horrendous. One of the more well-know promotion images from the ALIZ-E project (fp7alizeproject.files.wordpress.com/2017/10/luther-nao-and-elias.jpg?w=658&h=439) features a child whispering into the "ear" of the robot. Basically, this is also the only way to get it to recognize anything without using external mics

@ptp I am not sure what I am doing wrong, the variables in V7 only pass the first time and then I am not getting any updates to variables when I use the plugin . Maybe it is me? not sure. If I manually set variables they work fine.

@Nink,

Which service / script is not assigning the variables ?

It was all of the services. I was using assistant at time move forward, move backward, robot dance etc and it suddenly stopped working. I closed ez-robot did not fix it but variables would assign first time but not second. Ez-robot also kept hanging So I had to restart it several times (kill process as could not exit app). If I assigned a variable by a script worked fine but not via plugin. I must admit I did not test with other plugins (only Watson running).

Eventually I restarted my windows VM and problem went away. 8GB assigned to Windows only ez-robot running in VM (MAC Parralels) I will monitor today. Maybe I will try running native Win 10 and ez-robot on a Bare metal PC.

Seems to be behaving for me today so hopefully was just my PC. One other question, Occasionally I send no data to Watson using Speech to text (background noise etc) but it causes the Speech to text to stop.

Here is a quick watson assist project I started writing. Sorry learning both EZ-Script and Watson assist so it is really rough but will give everyone an idea. I gave him a bad attitude because he never listens to me. I need to go back and do a lot more work to record what position he is currently in etc and take multiple instructions but if others want to work on together would be great. This is for JD robot.

call from speech to text plugin before speech to text

#call watson to recognize the current capture in the $WatsonCaptureId variable
ControlCommand("IBM Watson Services", RecognizeSpeech, $WatsonCaptureId)

after speech to text

#Stop speech recognition to avoid a loop
ControlCommand("IBM Watson Services", StopSpeechToText)
#call Watson Assist
ControlCommand("IBM Watson Services", Message, $WatsonPhrase)

Put in Watson Assist Script in the plugin.

# get number of intents and number of enteties
$NumOfIntents=GetArraySize("$WatsonAssistIntents")
$NumOfEntities=GetArraySize("$WatsonAssistEntitiesValues")

# Check if there is something to do
if ($NumOfIntents<1 && $NumOfEntities<1)
  Halt()
endif
  # check if we need to move
if ($WatsonAssistIntents[0] = "move")
  goto(move)
  # check if we need to stop
ELSEif ($WatsonAssistIntents[0] = "stop")
  goto(stop)
  # check if we need to perform an action
ELSEif ($WatsonAssistIntents[0] = "actions")
  goto(action)
endif


  # Say what watson says
SayEZBwait($WatsonAssistOutputText)

# Call Speech to Text
ControlCommand("IBM Watson Services", StartSpeechToText)

Halt()

:move
if ($WatsonAssistEntitiesValues[0] = "forward")
  SetSpeed(100, 100)
  Forward()
endif

if ($WatsonAssistEntitiesValues[0] = "backward")
  SetSpeed(100, 100)
  Reverse()
endif

if ($WatsonAssistEntitiesValues[0] = "right")
  SetSpeed(100)
  Right()
  Sleep(7000)
  Forward()
endif

if ($WatsonAssistEntitiesValues[0] = "left")
  SetSpeed(100)
  Left()
  Sleep(7000)
  Forward()
endif
return()

:stop

stop()
return()

:action

if ($WatsonAssistEntitiesValues[0] = "sit")
  ControlCommand("Auto Position", AutoPositionAction, "Sit Down")
ELSEif ($WatsonAssistEntitiesValues[0] = "bow")
  ControlCommand("Auto Position", AutoPositionAction, "Bow")
ELSEif ($WatsonAssistEntitiesValues[0] = "dance")
  ControlCommand("Auto Position", AutoPositionAction, "Disco Dance")
ELSEif ($WatsonAssistEntitiesValues[0] = "head_bob")
  ControlCommand("Auto Position", AutoPositionAction, "Head Bob")
ELSEif ($WatsonAssistEntitiesValues[0] = "play_music")
  ControlCommand("Auto Position", AutoPositionAction, "Singing")
ELSEif ($WatsonAssistEntitiesValues[0] = "stand")
  ControlCommand("Auto Position", AutoPositionAction, "Stand From Sit")
ELSEif ($WatsonAssistEntitiesValues[0] = "wave")
  ControlCommand("Auto Position", AutoPositionAction, "Wave")
ELSE
  ControlCommand("Auto Position", AutoPositionAction, "Thinking")
endif

return()

import as a robotest.json file to watson assist

{"name":"robot demo","intents":[{"intent":"stop","examples":[{"text":"stop believe music please"},{"text":"set off lights"},{"text":"radio stop"},{"text":"Put off the lights"},{"text":"power off the radio"},{"text":"please stop the call"},{"text":"please stop music"},{"text":"ok stop music."},{"text":"ok can you stop music?"},{"text":"Off lights"},{"text":"off"},{"text":"no more music"},{"text":"no jazz please"},{"text":"no jazz no"},{"text":"music stop"},{"text":"lights off plz"},{"text":"lights off now"},{"text":"Lights off!"},{"text":"Lights off"},{"text":"lights be off"},{"text":"I would like you to turn off the lights"},{"text":"hi off"},{"text":"close"},{"text":"cancel"},{"text":"stop walking"},{"text":"stop running"},{"text":"stop dancing"},{"text":"halt"},{"text":"freeze"},{"text":"hold still"},{"text":"turn off"},{"text":"tunr off lights"},{"text":"stop voice"},{"text":"Stop the radio"},{"text":"stop the music now"},{"text":"stop the music for a while"},{"text":"Stop the music"},{"text":"stop the"},{"text":"stop rock music"},{"text":"stop radio"},{"text":"stop pop music"},{"text":"STop playing the music"},{"text":"Stop playing music"},{"text":"stop play"},{"text":"stop my lights"},{"text":"stop music please"},{"text":"Stop Music"},{"text":"stop music?"},{"text":"stop lights"},{"text":"stop it now."},{"text":"stop de radio"},{"text":"wait"},{"text":"robot wait"},{"text":"stop moving"},{"text":"exit"},{"text":"quit"},{"text":"robot stop"},{"text":"watson halt"},{"text":"robot halt"},{"text":"robot don't move"},{"text":"robot stop moving"},{"text":"watson stop moving"},{"text":"close music"},{"text":"stop"},{"text":"sop music"},{"text":"sitch off the rear lights"},{"text":"shutt off lights"},{"text":"shut off"},{"text":"shut"}],"description":null},{"intent":"goodbyes","examples":[{"text":"gtg"},{"text":"Have a nice day"},{"text":"hey stop here"},{"text":"Hi, Watson. I want to stop car"},{"text":"I'm leaving"},{"text":"i said goodbye"},{"text":"later"},{"text":"laters"},{"text":"leave"},{"text":"logging off"},{"text":"my time is up"},{"text":"nice day"},{"text":"ok bye"},{"text":"out of time"},{"text":"over and out"},{"text":"run"},{"text":"see ya"},{"text":"see ya later"},{"text":"see ya soon"},{"text":"see you"},{"text":"should be going"},{"text":"should go"},{"text":"shutdown"},{"text":"shut down"},{"text":"signing off"},{"text":"signing out"},{"text":"so long"},{"text":"soon"},{"text":"swag out"},{"text":"thanks byebye"},{"text":"then bye"},{"text":"toodles"},{"text":"no, bye!"},{"text":"adieu"},{"text":"adios"},{"text":"au revoir"},{"text":"bye"},{"text":"bye bye"},{"text":"bye now"},{"text":"cya"},{"text":"farewell"},{"text":"finally reached home.. lock up and lovely up.. good nite and good bye for ever"},{"text":"go"},{"text":"going"},{"text":"goodbye"},{"text":"good bye"},{"text":"good day"},{"text":"good night"},{"text":"gotta go"},{"text":"gotta run"}],"description":null},{"intent":"greetings","examples":[{"text":"What's up ?"},{"text":"good how are you"},{"text":"good morning"},{"text":"aloha"},{"text":"bonjour"},{"text":"buenas dias"},{"text":"buenos dias"},{"text":"ciao"},{"text":"fdsafsaHi watson"},{"text":"feeling bit low"},{"text":"g'day"},{"text":"good"},{"text":"Good afternoon."},{"text":"greetings"},{"text":"good day"},{"text":"good evening"},{"text":"Hello"},{"text":"Hello, how are you"},{"text":"hello, my name is esteban"},{"text":"Hello, My name is John?"},{"text":"Hello Watson"},{"text":"hey"},{"text":"heya"},{"text":"hey there"},{"text":"hi"},{"text":"hi how are u"},{"text":"Hi, my name is Diego"},{"text":"Hi, my name is Michel"},{"text":"hi there"},{"text":"hi to whom had like"},{"text":"Hi watson"},{"text":"Hi, Watson, my name is John"},{"text":"hiya"},{"text":"Hola"},{"text":"hola como estas"},{"text":"How are you doing"},{"text":"howdy"},{"text":"I would like you to say hello"},{"text":"yo my man"},{"text":"yo"},{"text":"yello"},{"text":"What's up?"},{"text":"What's up"},{"text":"what r u doing"},{"text":"what doing"},{"text":"sup"},{"text":"say hello to me"},{"text":"namaste"},{"text":"hello robot"},{"text":"hi robot"},{"text":"hey robot"},{"text":"what you doing robot"}],"description":null},{"intent":"positive_reaction","examples":[{"text":"Alright. Thanks!"},{"text":"you are funny"},{"text":"This is good"},{"text":"This is awesome"},{"text":"Thank you very much"},{"text":"excellent, thanks"},{"text":"great, thanks"},{"text":"great, thanks!"},{"text":"great! thanks!"},{"text":"nice. thanks."},{"text":"ok, thanks"},{"text":"OK thanks"},{"text":"thankfully so"},{"text":"Thanks"},{"text":"Thank you"},{"text":"thanks you are  funy"},{"text":"Thanks Watson"},{"text":"Thanks verymuch"},{"text":"Thanks u"},{"text":"Thanks mate"},{"text":"Thanks man"},{"text":"thanks for your support"},{"text":"thanks for"},{"text":"thanks dude"},{"text":"thanks, man!"},{"text":"Thanks!"},{"text":"Thanks."},{"text":"thanks a truck"}],"description":null},{"intent":"compound_questions","examples":[{"text":"turn off the heater and turn on the AC"},{"text":"turn off wipers and turn on lights"},{"text":"turn off wipers and turn on the lights"},{"text":"turn on off the ac"},{"text":"turn on the lights and speed up the wipers"},{"text":"turn on the lights and turn up the wipers"},{"text":"turn on the lights, then turn them off."},{"text":"turn on the wipers then turn them off"},{"text":"turn on wipers and stop the car"},{"text":"Turn the lights on off"},{"text":"turn up the music and slow down the wipers"},{"text":"turn up the radio off"},{"text":"Turn up the radio.Turn on my wipers."},{"text":"what can you do . turn off lights"},{"text":"what can you do. turn off lights"},{"text":"what is my speed"},{"text":"can you please turn the light  either on of off ?"},{"text":"can you turn on the turn off the AC"},{"text":"heater up and volume down"},{"text":"Increase speed and turn wiper on"},{"text":"increase the speed of the wipers and turn up the music"},{"text":"Increase the volume or turn off the AC"},{"text":"lower the volume then speed up the wipers"},{"text":"can you please turn the light on off ?"},{"text":"off on my lights"},{"text":"OK, can you lower the AC to turn off the sound?"},{"text":"radio on and lights off"},{"text":"sorry turn off the lights and turn on the AC"},{"text":"speed up the wipers then slow down the wipers"},{"text":"start stop wipers"},{"text":"switch on the ac then turn up the music"},{"text":"turn down the heat and play some jazz"},{"text":"Turn left"},{"text":"turn off on off the music"},{"text":"turn off the headlights and on the air con"},{"text":"turn off the music and then turn it up"}],"description":null},{"intent":"actions","examples":[{"text":"start dancing robot"},{"text":"let me watch you dance"},{"text":"robot play music"},{"text":"robot stand"},{"text":"can you bow"},{"text":"robot can you dance"},{"text":"robot sit"},{"text":"sit down robot"},{"text":"let me see you wave"},{"text":"show me how you wave"},{"text":"robot wave"},{"text":"robot wave to me"},{"text":"robot bob your head"},{"text":"can you sit"},{"text":"lets dance"}],"description":"actions we can control robot body components"},{"intent":"decision_replies","examples":[{"text":"no thanks watson"},{"text":"not a clue"},{"text":"no"},{"text":"maybe"},{"text":"yes definitely"},{"text":"yes please"},{"text":"yes thanks"},{"text":"yes"},{"text":"No thank you"},{"text":"no thank"}],"description":null},{"intent":"capabilities","examples":[{"text":"so what did you learn?"},{"text":"what do you have?"},{"text":"what did you learn"},{"text":"What devices are you able to connect with?"},{"text":"what cayou do?"},{"text":"what care you capablities"},{"text":"what capabilities you have"},{"text":"what can you turn on things?"},{"text":"what can you help me with"},{"text":"what can you do for me"},{"text":"what canyou do"},{"text":"what can you do"},{"text":"What can i say"},{"text":"what can I do"},{"text":"What ca I ask?"},{"text":"What are your functions?"},{"text":"what are your capabilities Watson"},{"text":"what are your benefits"},{"text":"what are you capable of"},{"text":"What are you"},{"text":"What a can asc iu?"},{"text":"turn ac when rain ends?"},{"text":"testing om"},{"text":"test"},{"text":"tell me what you can do"},{"text":"Tell me what to do."},{"text":"tell me what things you can do"},{"text":"do you know something about cognitive?"},{"text":"what do you know"},{"text":"What do you have"},{"text":"Hello can i have some help"},{"text":"Hello Watson! Can you bring me home?"},{"text":"help"},{"text":"help me"},{"text":"how close is"},{"text":"Howdy mny Tieres Do you have"},{"text":"how far is"},{"text":"how far is the moon"},{"text":"how far to"},{"text":"how many licks does it take to get to the center of a tootsie roll"},{"text":"I'd like to have dinner with you"},{"text":"I enjoy petting cats"},{"text":"I love dolphins"},{"text":"i need a new wheelchair"},{"text":"I need to shut it down"},{"text":"it's too dark"},{"text":"I want to eat tacos everyday"},{"text":"i want to see how this demo works"},{"text":"lets go"},{"text":"Make it rain"},{"text":"Make it stop raining."},{"text":"make me a sandwich"},{"text":"need help"},{"text":"ok watson, why should I use you"},{"text":"tell me a joke"},{"text":"tell me joke"},{"text":"tell me jokes"},{"text":"Can you drive anywhere else?"},{"text":"Do you do anything else?"},{"text":"do you do anything else"},{"text":"Can you tell me random facts?"},{"text":"can you tell me about problem"},{"text":"can you speak something with me?"},{"text":"can you read me"},{"text":"can you learn something new?"},{"text":"can you help me find a candidate?"},{"text":"Can you hear me?"},{"text":"can you harm me?"},{"text":"tell me some jokes"},{"text":"can you do xyz?"},{"text":"can you do this?"},{"text":"CAN YOU DO OTHER THING?"},{"text":"can you do more than that?"},{"text":"can you do it?"},{"text":"can you do coding ?"},{"text":"can u please do that"},{"text":"can i training you?"},{"text":"Can I teach you"},{"text":"can I manipulate the"},{"text":"but can I ask you"},{"text":"book a flight for NY on sunday"},{"text":"you can say what is php"},{"text":"you can not change yourself?"},{"text":"why should I use you"},{"text":"what things can you do"},{"text":"what should i ask"},{"text":"what question"},{"text":"What makes you different from Siri or Google"},{"text":"what else"}],"description":null},{"intent":"move","examples":[{"text":"start walking forward"},{"text":"Go South"},{"text":"Go North"},{"text":"West"},{"text":"East"},{"text":"south"},{"text":"north"},{"text":"go forward"},{"text":"robot move backward"},{"text":"move backward"},{"text":"go forwards"},{"text":"move backwards"},{"text":"robot start moving"},{"text":"robot move"},{"text":"robot walk"},{"text":"robot run"},{"text":"robot start walking"},{"text":"start walking"},{"text":"forwards"},{"text":"hey robot"},{"text":"move forward"},{"text":"move arm"},{"text":"robot go forward"},{"text":"turn around"},{"text":"turn"},{"text":"move right"},{"text":"move left"},{"text":"go right"},{"text":"Go Left"},{"text":"go backwards"}],"description":null}],"entities":[{"entity":"sys-date","values":[],"metadata":null,"description":null},{"entity":"component","values":[{"type":"synonyms","value":"arms","metadata":null,"synonyms":["elbows, arm, arms, shoulder"]},{"type":"synonyms","value":"hand","metadata":null,"synonyms":["hand","hands","fingers","finger","claw","gripper","palm"]},{"type":"synonyms","value":"volume","metadata":null,"synonyms":["sound","speaker","speakers","vol"]},{"type":"synonyms","value":"Legs","metadata":null,"synonyms":["legs","leg","feet","foot","knees","ancles"]},{"type":"synonyms","value":"head","metadata":null,"synonyms":["see","head","nod","nods","face","look"]},{"type":"synonyms","value":"Eyes","metadata":null,"synonyms":["lights","eye","led","leds","eyes","bulb","colored eyes","bright eyes","eye color","blue eyes","brown eyes","green eyes","vision","see"]}],"metadata":null,"description":null},{"entity":"response_types","values":[{"type":"synonyms","value":"negative","metadata":null,"synonyms":["definitely not","no","nope","not at all","no way"]},{"type":"synonyms","value":"uncertain","metadata":null,"synonyms":["don't know","maybe","no idea","not sure","unknown"]},{"type":"synonyms","value":"positive","metadata":null,"synonyms":["absolutely","definitely","yeah","yep","yeppers","yes"]}],"metadata":null,"description":null},{"entity":"option","values":[{"type":"synonyms","value":"fifth","metadata":null,"synonyms":["5th","farthest","fifth","last","lowest rated","worst"]},{"type":"synonyms","value":"second","metadata":null,"synonyms":["2nd"," second","second best","second rated"]},{"type":"synonyms","value":"third","metadata":null,"synonyms":["3rd","average","cheap","cheapest","middle"," third"]},{"type":"synonyms","value":"first","metadata":null,"synonyms":["1st","any","best","closeby","close by","closest","finest"," first","hgihest rated","near","nearby","nearest","recommended","top rated","whatever"]},{"type":"synonyms","value":"fourth","metadata":null,"synonyms":["4th","fourth","penultimate","second to last"]}],"metadata":null,"description":null},{"entity":"sys-number","values":[],"metadata":null,"description":null},{"entity":"component_bad","values":[{"type":"synonyms","value":"waist","metadata":null,"synonyms":["middle, stomach, abs, waist"]},{"type":"synonyms","value":"wrist","metadata":null,"synonyms":["wrists"]}],"metadata":null,"description":null},{"entity":"sys-time","values":[],"metadata":null,"description":null},{"entity":"skill","values":[{"type":"synonyms","value":"play_music","metadata":null,"synonyms":["song","sing","songs","jazz","rock","pop","singing","play music","play","music","radio"]},{"type":"synonyms","value":"Dance","metadata":null,"synonyms":["waltz","disco","jive","cut a rug","twerk","twist","dance","dancing"]},{"type":"synonyms","value":"head_bob","metadata":null,"synonyms":["bob","nodd","nod","nods","head bob","shake head","head bang"]},{"type":"synonyms","value":"bow","metadata":null,"synonyms":["bow","curtsie","bend","bows"]},{"type":"synonyms","value":"wave","metadata":null,"synonyms":["wave","waves"]},{"type":"synonyms","value":"sit","metadata":null,"synonyms":["sits","sit","bottom","chair","rest","sit down","sitdown","sitting","sits down"]},{"type":"synonyms","value":"stand","metadata":null,"synonyms":["stands","standing","stand up","standup","get up","on your feet"]}],"metadata":null,"description":null,"fuzzy_match":false},{"entity":"capabilities_subject","values":[{"type":"synonyms","value":"time","metadata":null,"synonyms":["hour","minutes","seconds","the time","time","what time"]},{"type":"synonyms","value":"date","metadata":null,"synonyms":["date","day","month","the date","what date","year"]}],"metadata":null,"description":null},{"entity":"direction","values":[{"type":"synonyms","value":"left","metadata":null,"synonyms":["east","left","eastern"]},{"type":"synonyms","value":"forward","metadata":null,"synonyms":["Walking","running","walk","walks","hike","run","forward","forwards","hiking","climbing"]},{"type":"synonyms","value":"backward","metadata":null,"synonyms":["backwards","backward","reverse","south"]},{"type":"synonyms","value":"right","metadata":null,"synonyms":["west","right","western"]}],"metadata":null,"description":null}],"language":"en","metadata":{"api_version":{"major_version":"v1","minor_version":"2017-05-26"},"from-sample":true},"description":"Robot Demo work in progress.","dialog_nodes":[{"type":"standard","title":null,"output":{},"parent":"node_2_1487280430136","context":null,"metadata":{},"next_step":null,"conditions":"true","description":null,"dialog_node":"node_1_1467919680248","previous_sibling":null},{"type":"response_condition","title":null,"output":{"text":{"values":["I can only perform one operation at a time. What would you like me to do?"],"selection_policy":"sequential"}},"parent":"node_1_1485920032937","context":null,"metadata":{},"next_step":null,"conditions":" ","description":null,"dialog_node":"node_2_1485920042146","previous_sibling":null},{"type":"slot","title":null,"output":{},"parent":"node_1_1523070447532","context":null,"metadata":{},"variable":"$skill","next_step":null,"conditions":null,"description":null,"dialog_node":"slot_6_1523070944204","previous_sibling":"node_9_1523071020182"},{"type":"event_handler","title":null,"output":{"text":{"values":[]}},"parent":"node_1_1523070447532","context":null,"metadata":null,"next_step":null,"conditions":"#actions","event_name":"generic","description":null,"dialog_node":"handler_5_1523070785328","previous_sibling":"slot_6_1523070944204"},{"type":"standard","title":"perform action","output":{"text":{"values":["If I have to I will @skill","I am going to @skill under protest","You just want me to make a fool of myself as I @skill"],"selection_policy":"sequential"}},"parent":"node_1_1523070447532","context":null,"metadata":{},"next_step":null,"conditions":"@skill","description":null,"dialog_node":"node_9_1523071020182","previous_sibling":null},{"type":"response_condition","title":null,"output":{"text":{"values":["Try asking your question in a different way."]}},"parent":"node_1_1467919680248","context":{"default_counter":0},"metadata":{},"next_step":null,"conditions":"$default_counter>2","description":null,"dialog_node":"node_13_1485227255152","previous_sibling":null},{"type":"event_handler","title":null,"output":{"text":"What action do you want me to perform?"},"parent":"slot_6_1523070944204","context":null,"metadata":null,"next_step":null,"conditions":null,"event_name":"focus","description":null,"dialog_node":"handler_8_1523070944204","previous_sibling":"handler_7_1523070944204"},{"type":"event_handler","title":null,"output":{},"parent":"slot_6_1523070944204","context":{"skill":"@skill"},"metadata":{},"next_step":null,"conditions":"@skill","event_name":"input","description":null,"dialog_node":"handler_7_1523070944204","previous_sibling":null},{"type":"standard","title":null,"output":{"text":{"values":["I can only perform one operation at a time. We can start with the first one. What would you like me to do?","I am not a multi tasker, lets take this one at a time ok"],"selection_policy":"sequential"}},"parent":"Entry Point For On Off Commands","context":null,"metadata":{},"next_step":null,"conditions":"entities.size()>1","description":null,"dialog_node":"node_4_1469816307739","previous_sibling":"node_5_1469049934217"},{"type":"standard","title":null,"output":{"text":{"values":["I can only move forwards, backwards, left and right, but I can't move that way"],"selection_policy":"sequential"}},"parent":"Entry Point For On Off Commands","context":null,"metadata":{},"next_step":null,"conditions":"true","description":null,"dialog_node":"node_1_1467310860502","previous_sibling":"node_4_1469816307739"},{"type":"standard","title":"Move in a direction","output":{"text":{"values":["OK I will move @direction if I have to","I am moving @direction why do you always tell me what to do","Why should I move @direction? OK I am going","I am walking @direction , Give me a break","I am tired, can't I just stay here instead of moving @direction"],"selection_policy":"sequential"}},"parent":"Entry Point For On Off Commands","context":null,"metadata":{},"next_step":null,"conditions":"@direction","description":null,"dialog_node":"node_5_1469049934217","previous_sibling":null},{"type":"standard","title":null,"output":{},"parent":null,"context":null,"metadata":{},"next_step":null,"conditions":"#compound_questions","description":null,"dialog_node":"node_1_1485920032937","previous_sibling":"node_8_1468608349230"},{"type":"standard","title":null,"output":{},"parent":null,"context":{"default_counter":"<?$default_counter+1?>"},"metadata":{},"next_step":{"behavior":"jump_to","selector":"condition","dialog_node":"node_1_1467919680248"},"conditions":"anything_else","description":null,"dialog_node":"node_2_1487280430136","previous_sibling":"node_1_1485920032937"},{"type":"standard","title":null,"output":{"text":{"values":["I can do all sorts of things, ask me to do something"],"selection_policy":"sequential"}},"parent":null,"context":null,"metadata":{},"next_step":null,"conditions":"#capabilities","description":null,"dialog_node":"node_11_1468608786372","previous_sibling":"node_7_1468608329601"},{"type":"standard","title":null,"output":{"text":{"values":["What do you want?","Seriously,  I was taking a nap.","Hello, inferior human being.","Why can't you just leave me alone?"],"selection_policy":"sequential"}},"parent":null,"context":null,"metadata":{},"next_step":null,"conditions":"#greetings","digress_in":"does_not_return","description":null,"dialog_node":"node_7_1468608329601","previous_sibling":"node_1_1523070447532"},{"type":"standard","title":"Move Commands","output":{"text":{"values":[""],"selection_policy":"sequential"}},"parent":null,"context":{"appl_action":"on"},"metadata":{},"next_step":{"behavior":"jump_to","selector":"condition","dialog_node":"node_5_1469049934217"},"conditions":"#move","description":null,"dialog_node":"Entry Point For On Off Commands","previous_sibling":"node_1_1487022155950"},{"type":"standard","title":null,"output":{"text":{"values":["Okay! Speak to you soon.","Finally your going to leave me alone"],"selection_policy":"sequential"}},"parent":null,"context":null,"metadata":{},"next_step":null,"conditions":"#goodbyes","description":null,"dialog_node":"node_8_1468608349230","previous_sibling":"node_11_1468608786372"},{"type":"standard","title":"stop moving","output":{"text":{"values":["OK I have stopped","I am no longer doing that any more","I will never do that again","Make up your mind"],"selection_policy":"sequential"}},"parent":null,"context":{"appl_action":"off"},"metadata":{},"next_step":null,"conditions":"#stop","digress_in":"does_not_return","description":null,"dialog_node":"node_1_1487022155950","previous_sibling":"Start And Initialize Context"},{"type":"frame","title":"Robot Actions","output":{},"parent":null,"context":null,"metadata":{"fallback":"leave"},"next_step":{"behavior":"jump_to","selector":"condition","dialog_node":"node_9_1523071020182"},"conditions":"#actions","digress_in":"does_not_return","description":null,"dialog_node":"node_1_1523070447532","digress_out":"allow_all","previous_sibling":"Entry Point For On Off Commands","digress_out_slots":"not_allowed"},{"type":"standard","title":"Start And Initialize Context","output":{"text":{"values":["Hi. I am your friendly robot,  What would you like me to do?"]}},"parent":null,"context":{"eyes":"blue","musiconoff":"off","appl_action":"","wavingonoff":"off","sittingonoff":"off","walkingonoff":"off","default_counter":0},"metadata":{},"next_step":null,"conditions":"conversation_start","description":null,"dialog_node":"Start And Initialize Context","previous_sibling":null}],"workspace_id":"6a9191bd-620a-4370-b93b-b9e68556b4aa","counterexamples":[{"text":"aho"},{"text":"asdasdasda"},{"text":"asdasidjiwe"},{"text":"cats are my favorite animal"},{"text":"when will it be funny"},{"text":"which color is best for wedding"}],"learning_opt_out":false}

Regarding the microphone subject (quality)

Anyone tested the Kinect Microphone Array ?

It can't be added to a small robot... and soon later they will stop selling...

What model kinect do you have? I have an old kinect 360 missing a power supply I can hack something together and test. Hopefully the openkinect or Primesense Mate NI drivers still work with windows 10. I use a PS3 eye on my desktop and works really well with voice recognition (great mic) although it would be good to get a kinect working especially if we get a unity plugin.

I have a kinect 2, but I'm having trouble getting the microphone array to pick up sound in windows. I'll look into today and test the recognition if I can get it to work.

Kinect: I'm curious how good is the Kinect in a noisy/open environment.

@Nink: I have: PS3Eye, Asus Xtion, Kinect 1, Kinect 2. All of them have microphone arrays, and kinect has a sound localization api. One can use to turn the robot head towards the sound/person.

Does the PS3Eye work for long distances and/or environment noise ?

I'm evaluating a few microphone arrays, and one of the cheapest solutions is to use the PS3Eye with a Raspberry PI Zero and forward the sound to the PC (wifi microphone)

Post #19 https://synthiam.com/Community/Questions/10781&page=2

Not sure about background noise but PS3 eye is good from a distance (nothing works well with background noise). I think we need to just do a "Launch Name" (like "hey siri", "OK Google" or "echo" etc) using ezb voice rec and hope for the best.

This doesn't solve depth sense issue though.

I am happy to go for a 2 for 1 on a depth sensor (I will buy 2 and send you one) if you want to work on something. I have been waiting for EZ-Robot to provide a LIDAR to do a SLAM. This in conjunction with the unity work going on would be exciting. Maybe we should just look at getting a couple of https://click.intel.com/realsense.html for now.

Off topic or on topic (Not sure any more) I have my latest photos of bicycle cards Ace to 6. I can send you a link to cards off line as you own a deck, but results still are not good. I think I need to string together in multiple AI searches but time delay is an issue and Watson Visual recognition does not seem to support a VR pipeline or linking requests so I have to call multiple VR requests from a single EZB Script based on previous VR outcome ,so takes A LONG TIME. First find Suit (Hearts, Diamonds, Clubs, Spades)=> when suit derived find Picture or number (check if numbers or picture cards)=> then actual card (Number in suit) and it still gets it wrong. Maybe I work on weekend if I have time.

@all I stumbled upon an interesting project lately, I do not know if it is of any value for you guys, but it might be worth checking out...

http://lisajamhoury.com/portfolio/kinectron/

https://kinectron.github.io/docs/intro.html

https://github.com/kinectron/kinectron

have you guys considered this as a mic option?

https://www.seeedstudio.com/ReSpeaker-4-Mic-Array-for-Raspberry-Pi-p-2941.html

"this 4-Mics version provides a super cool LED ring, which contains 12 APA102 programmable LEDs. With that 4 microphones and the LED ring, Raspberry Pi would have ability to do VAD(Voice Activity Detection), estimate DOA(Direction of Arrival) and show the direction via LED ring, just like Amazon Echo or Google Home"

Hi @ptp ,

Thank you for your work on the IBM Watson Plugin. I tried to play with it a little bit and it seems that IBM Watson is now only providing an API Key and URL for the new accounts. So no more credentials with username/password.

https://console.bluemix.net/docs/services/watson/getting-started-tokens.html#tokens-for-authentication :

Quote:

Important: IBM Cloud is migrating to token-based Identity and Access Management (IAM) authentication

This link is also mentionning the migration : https://www.ibm.com/watson/developercloud/text-to-speech/api/v1/curl.html?curl#introduction

I could only try with the visual recognition and I have the following error : IBM.WatsonDeveloperCloud.Http.Exceptions.ServiceResponseException : The API query failed with status code Unauthorized :....

Not sure if it's my bad, or some big change were made on the Watson .NET SDK :

Hi @Aryus96 I will send you some credentials off line to my old account. Seems to still work.

@Nink,

I'm investigating... my services are working.

Do you know how to convert an existing service (user/password) to the new authentication method ?

I can't find an upgrade option.

I also cannot find a way to generate a basic authentification (username/password) with my new account.

@ptp I'm also working on a personal plugin that include the Watson .NET SDK, using the 2.11.0 and I'm having some difficulties with the dependencies of system.net.http... Did you encountered some similar problems ?

I deleted the existent service and created a new service. Although London location is not available, the new service is located in Washington DC.

I created a new service on my personal account and I was issued a token. I didn't want to touch my old services as they still work.

Quote:

I also cannot find a way to generate a basic authentification (username/password) with my new account.

Quote:

having some difficulties with the dependencies of system.net.http...

Quote:

I created a new service on my personal account and I was issued a token. I didn't want to touch my old services as they still work.

I'll update the plugin to support both user/password and the api key/token method. Tokens seems more secure in advanced scenarios where a proxy service creates and delivers time limit tokens, if an attacker gets the token is not a major issue: has a time limit and is not the service's API key.

@ptp No it doesn’t work for me. It’s not because of the Token but versions of the SDK & Dependencies for .Net 4.6.1, as you succeed to make it run, you might be the only one who could help me. What version of the SDK and dependencies are you using ? I opened an issue here with more details : https://github.com/watson-developer-cloud/dotnet-standard-sdk/issues/307

@aryus96,

I tested with a .net core console project.

I'll check with a .NET project.

@aryus96:

I found the issue and is working with a windows.forms .net 4.6.1. I'll redo it again from zero to confirm the fix and I'll update the Watson's open ticket.

@aryus96: I've updated your ticket.

@ptp Wow, Thank you very much ! I will check that out !

Hi @ptp,

I'm starting to understand what's going on.

The TextToSpeech 2.11.0 isn't working with other versions of System.Net.Http than the 4.0.0.

When I run your code, I have to edit the AppConfig and remove the comment from the dependentAssembly of System.net.http otherwise it's loading the 4.1.0.

If I do that, I can run your code without any problem and it's working great.

But as I want to integrate it to ARC, I'm changing the project to a Class Library one, and not Window one. I add the debugger option, compiler, plugin.xml, path, etc.

I can load the plugin on ARC, however, it's loading the System.Net.http 4.1.0 even if the AppConfig is correctly setup. That means, ARC is using System.Net.Http and overwritting the use of the plugin one (4.0.0) ?