I have been working in API.AI for a while and jumped back into it yesterday some. I want to describe what API.AI is and what it isn't so that there is a clearer picture of it. I am attaching a sample api.ai project that you can use to start building from if you would like to do so.
First, what is it? API.AI is great at taking speech and breaking it down to give you the intent behind the speech. For example, in the example, you can say something like "Hey Robot, I want you to raise your right arm 19 degrees". The returned text (what would be returned to EZ-Builder) is "[Movement] [RightArm] [up] [19] [deg]".
You can use a script to break down what the robot is being asked to do from the information above. The phrase "yo, raise your right arm 1 degree" would also work and return "[Movement] [RightArm] [up] [1] [deg]" for you to parse and use.
There are some chatbot features called "SmallTalk". This works like any other chatbot and takes questions or statements from the user and returns text that you want it to return. This is the simplest form of using API.AI and probably is the easiest part of it, but is also not the most useful.
There are some prebuilt agents. These agents each use their own API key. Because of this, I don't recommend using them because the plugin only allows one API key, and you will quickly run out of allowable uses. It is far better to build a single customized agent which contains everything that you want your robot to use.
The use of this tool is to break apart and parameterize language. This allows you to use completely different speech for specific commands in EZ-Builder. Currently, the plugin only sets 2 variables in EZ-Builder. This requires that you pass the parameters in API.AI back in the Text response field that would match the layout of what you want to parse.
In addition, you can add what are called integrations. This is where you could tie into many different things like Actions on Google, Microsoft Cortana, Amazon Alexa skills, telegram, skype, twitter, facebook messenger and many other one-click integrations. There are also SDK's for Android, IOS, Ruby, HTML5, JavaScript, Node.JS, Cordova, Unity, C#, C++, Xamarin, Python, PHP, Java, Botkit and Epson Moverio that allow you to write whatever you want with whatever language you want for the most part. These integrations allow you to run code instead of simply returning the meaning of what was stated back to EZ-Builder.
The example here doesn't use integrations, but is designed more to have the information sent back to ARC for you to do something with.
This is a very scaled down version of the Rafiki client that I had been working on.
Rafiki-Client.zip
You can take this and import or restore it into your api.ai project.
I hope this example helps people see what API.AI can be used for and better understand where it fits in your robot brain.
Asked
— Edited
"Hey Robot, I want you to raise your right arm 19 degrees" and "yo, raise your right arm 1 degree".
These statements are
Code:
and
Code:
It isn't google's search engine, although you could call the search engine from the results returned by api.ai.
It isn't a great chatbot. If you want to do that, I would recommend other options.
It isn't something that won't require you to do more work to use the data returned.
It isn't a complete AI that can just be dropped in and used.
It isn't something that requires no learning and effort
It isn't something that you can just let someone else develop things in it for you.
You will have to take the initiative to learn about it and make it customized for your robot. Not every robot has arms. Moving forward 5 feet is different for every type of robot with different types of sensors. Some wheeled robots have encoders, some don't.
It is a powerful tool for breaking apart speech and then formatting it in a way that you can use it. This allows many people to use the same robot with different speech styles and ways of saying something. It will help to prevent you from having to remember exact phrases that you have programmed to do things. It will make you a much better script writer if you use it.
David, your such an asset and help to this forum. Personally, I really appreciate the time and expert knowledge you're giving. Thanks!
It's also a great starting point for everyone not yet familiar with the api.ai platform.
You did a great job in pointing out what api ai is, and what it is not...I could not agree more on all of your points!
When it comes to returned parameters, we could also ask @DJSures to be so kind and include all of them to be exposed within ARC, the first version of the plugin was only returning speech and later on he also included the action parameter, I do agree that it would make the work in ARC a lot simpler if we would not have to break down the returned text and the variables are directly accessible!
If you would want to use api.ai as a seach engine, you could do this in ARC or using api.ai fulfillment...same goes for smalltalk! If eg a smalltalk action is returned you could pass the speech to your chatbot within ARC, I would also guess you could use api.ai fulfillment to handle this, but I did not look into this yet!
Great intro, lets start digging into this...its worth it!
This is an example ARC project that leverages the Bing Speech API, API.AI and Pandora Bot.
It could be cleaner, just wanting to put out something that could be used to start things off for people if they wanted to use it.
I will discuss this project and API.AI on this weeks Technology Today episode.
After installing your project I said "Robot move backwards" when the proper phrase should have been "Robot Reverse". It understood what I wanted and responded.
Very cool.
It already involves a great deal of coding, and having the speech recognition already triggered and ready to go is a cool thing!
To see how you coded the response parse is great! I still did not get pandora bot to respond, but I guess I can figure that out somehow...your example should send the speech from Bing to PandoraBot if the returned action is blank right?
I think it would be kind of cool to have your api.ai client access token so the testing enviroment will be level for everyone...that should be save right?
I already deleted an EZ-Robot project that I posted because I read your thread on the Bing Speech key being included...but for the api.ai key this should not matter right?
Good starting point, I am really happy to have this, since you already worked with api.ai for quiet a bit, it is so cool seeing how you manage to piece it all together in ARC! Yaaaaay!
*edit*
I was just not getting the point that the provided example was obviously to be run on the Rafiki Client which can be installed from the .zip file that was uploaded earlier in this thread, it provides some interesting approaches on api.ai robot control and can be tailored to your own needs...just stating this in case someone else might also be confused about it!
Here is the jist of the example project
The init script just pauses the PandorBot control and the Bing Speech Recognition plugin, along with making sure that the Speech Recognition control is not paused.
In the Speech Recognition control there are a couple of different phrases that will unpause the Bing Speech Recognition control. They are "Hey Robot", "I want to ask you something" and "Hey Spock". Feel free to change these to whatever you need. They all run the same script which causes the Bing Speech Recognition plugin to start listening.
After the user says something, the Bing Speech Recognition plugin pauses the Speech Recognition control, calls ControlCommand("API.AI", Send, $BingSpeech) and then pauses the Bing Speech Recognition plugin. The converted text is returned to the $BingSpeech variable and then launches the API.AI plugin to send the $BingSpeech variable to API.AI.
The API.AI plugin starts the ResponseParse script which then decides what to do with the variables returned from API.AI. If it doesnt know what to do with the text, it sends the $BingSpeech variable to PandoraBot after unpausing Pandoria bot. After two seconds, Pandora Bot is paused again by the script.
If API.AI returns values that can be used, it calls the appropriate script to execute the actions necessary.
Okay, I will make sure that everything is working as intended this afternoon and publish it.
Example project here. You will have to supply your own API.AI key and Bing Speech key.
Thanks for your work David.
I'm trying to understand who this works, on David's API-AI example it has this in the script .
Code:
I think I understand the >0 means if no text or nothing heard, right?
just not sure what the 0,5 means.
Thanks,
Any help would be most appreciated when you have time of course.
Thank you, mike
Code:
Substring command is looking for a specific set of characters in a longer string of characters like a sentance... In this case the word "small" starting at character 0 and reading for 5 characters (the length of the word "small")...
Code:
All these commands are listed in ez script including examples on how to use them...
In api.ai terms the zip file contains the actions that I used in the example.
I re read your tread and still confused. Maybe I just don't get the stuff because I see Dave s. has it installed and working. So I must be missing something.
I have the zip file not sure where to unzip it to.
Thanks David,
I think I still have to do something I'm not sure, when I say hey robot robot move forward it's starts the response script that you made but it does not start the movie script.
I will hunt and peck like I said I'm a visual person I have a hard time reading and understanding some things especially this techie stuff but thanks again for pointing me to the right direction David I appreciate it !
This is what I have in the api.ai example that David created. When I say robot moveforward it still gives an "input.unknown" in the $APIAction variable watch.
Code:
I add the forward() command in David's move script as you see above. When I say Robot moveforward it starts the script ResponseParse but it does not start the moverobot script.
Do I need to do something or add something on David's example code in the api.ai web site, for this below?
Code:
I was hoping Dave S. wound respond as I know David is very busy and has stated we need to do the leg work. I just need a start how ever small, as this is how I have learned what little I know by others examples. I am not asking for someone else to do my work, just asking for a visual example. I hope this makes sense.
Thanks in advance,
I was looking at them parameters on the API.AI website and thought I had to change something there after your last post. It now makes sense thank you very much David. It is now working. Thank you very much for all your time and sharing your project with others David. This is very cool!
Thanks and cheers!
@RichardR, thanks Richard for your reply.