Welcome to Synthiam!

Program robots using technologies created from industry experts. ARC is our free-to-use robot programming software that makes features like vision recognition, navigation and artificial intelligence easy.

Get Started
United Kingdom
Asked — Edited
Resolved Resolved by WBS00001!

Displaying A Robots Response In A Text Format

A bit of an unusual request, but there is reasoning behind what I'm about to ask. So my question is...

Is there a way to display a robots spoken response as text with EZ Builder?

Let me explain. Speech synthesis for a robots response is not an option here. You speak to a robot using speech recognition and the robot responds with text displayed on the computer screen. Now using the Pandorabot control will do this showing the robots response in the debug window, but the text is too small.

If your familiar with Pandorabot, you'll know that when you have a published bot, you get your own page where your Pandorabots response is displayed and you can add HTML code to change the text layout. This is something like what I'm looking to do via EZ Builder. I could use the Pandorabot portal, but I would need control commands which are imbedded in the bots responses to activate when the response is given.

The reasoning behind this, is for someone who is hard of hearing to interact and converse with the robot, so you see why speech synthesis can't really be used in this situation. I might be overlooking something obvious here, or there might be something that I'm unaware of, so ANY input anyone can offer on this really would be gratefully appreciated.

Cheers,

Steve.

AI Support Bot
Related Content
Synthiam
Based on your post activity, we found some content that may be interesting to you. Explore these other tutorials and community conversations.
#35  
The size seems to make a difference for a line, but for text it does nothing for me
#36  
@JustinRatliff
Yes, that's the thing in all this. DJ says to just look at the examples. There, problem solved. Nope. The examples only highlight a couple of things about it all. All I learned from the examples was that the color was words and not number values. They, and the tutorials are FAR from complete. Nothing about the other color names. Nothing about why 15 was used in the size value or why it doesn't seem to do anything. Nothing about the appropriate range of values for the Red, Blue, Green values. Nothing about changing font sizes or thickness. Maybe you have to change line thickness first, then that will change font thickness as well. Still working on that sort of thing. For something that is supposed to be easy for beginners to use, there is a lot left out that beginners would need to know.
United Kingdom
#37  
@WBS.

I finally managed to get around to try your program.

I seemed to have set it up okay, and your test script works. I'm trying to run a simple speech recognition control script to display a Say() response using the $SpeechPhrase variable, but I'm having trouble receiving the response.

I had a play with the SketchPad as well and I'm a littler clearer on what it does, but an explanation of the different values would be helpful. But I'm still having the same issue as above. I guess what I need is a kind of SpeechCommand() variable to display the speech synthesis Say() text and Pandorabot control text which I thought would exist but can't see any. What do I need to do here, as I'm a little lost and not sure what a variable to display this text would look like?
#38  
@SteveG
The variable you're looking for is $DisplayText. It is used in the SendToDisplay script.

Before using the SendToDisplay script to automatically send strings to the external display (via setting $DisplayText), however, you must start the script:

Code:


CC("SendToDisplay",ScriptStart)
Sleep(2000)

You only need to start it once and it runs forever until explicitly stopped.

Wherever you have a "Say" command, then, you also set $DisplayText to that same string. Something like this:

Code:


Say("Hello World")
$DisplayText ="Hello World"
Sleep(2000)
#Or, using a variable:
$Phrase ToSay ="Hello World"
Say($PhraseToSay)
$DisplayText =$PhraseToSay
Sleep(2000)

That's it. The bottom line is, whatever you want to go to the display, you set $DisplayText to that text. Anywhere in any script at any time. That's all. The SendToDisplay script will automatically take care of sending that text to the external display.

Also the Sleep(2000) statement could be placed in the SendToDisplay script. It already has a Sleep(500) statement. Just change it to Sleep(2000).

OTOH, if there will naturally be at least 2 seconds between sending text to the display, then you can leave it out entirely.

Now, while I'm on the topic, let me address another point concerning the 2 second (or longer) sleep time between individual strings being sent to the external display. This could become annoying when the reader is having to read more than one line for a given response. If you wish to send multiple lines to the display and they all need to be read in quick succession, the best way to do that is to send them all at the same time. This is what I mean:

Code:


Say("Sentence 1")
Say("Sentence 2")
Say("Sentence 3")
#There would need to be at least 2 seconds between for the display
$DisplayText ="Sentence 1")
Sleep(2000)
$DisplayText ="Sentence 2")
Sleep(2000)
$DisplayText ="Sentence 3")
Sleep(2000)

That pause between what should be a rapid succession of sentences could be annoying. So it would be best to concatenate them and send all at once, like this:

Code:

 
Say("Sentence 1")
Say("Sentence 2")
Say("Sentence 3")

$DisplayText ="Sentence 1. Sentence 2. Sentence 3" )
Sleep(2000)

#Or, using variables:
$Sentence1 ="Sentence 1"
$Sentence2 ="Sentence 2"
$Sentence3 ="Sentence 3"

Say($Sentence1)
Say($Sentence2)
Say($Sentence3)

$DisplayText =$Sentence1+$Sentence2+$Sentence3
Sleep(2000)

The display could be setup to show the sentences like this:
Sentence 1. Sentence 2. Sentence 3.

Or it could break them at the periods and show then like this
Sentence 1.
Sentence 2.
Sentence 3.

Even put a blank line in between:
Sentence 1.

Sentence 2.

Sentence 3.

All that could be done via setup options.

Something I just thought of
To better display the sentences in a timely manner it may be best to send them to the display BEFORE saying them. That would probably help with the sleep issue as well, since it would probably take a couple of seconds to say the sentence(s), so no Sleep(2000) statements would be needed.

EDIT
You're using Say instead of SayWait so you would need the Sleep delays after all.

One advantage of using the Sketch Pad option is that there would not be an issue of needing sleep periods between the sending of text lines to it.
United Kingdom
#39  
Thanks for getting back to me. I see where I was going wrong now. So every Say() or SayWait() script needs the $DisplayText variable in it too. Any ideas on how this could be achieved with the Pandorabot control as well, as there is not a scripting option for this control?
#40  
@SteveG
I had another look at the Pandorabot control but there just seems to be no way to get at the returned text from the Pandorabot site. One would have to figure out a way to intercept it as it comes in to the control or as it is displayed to the debug window. Perhaps there would be a way using the SDK, but it would require a knowledge of this control's operation. That is probably not a part of the SDK. Nor would it likely be a part of the new Plug-in system. The only other ways I can think of offhand to get to it would be:

1) Somehow cajole DJ into putting a variable in the Pandorabot control which would contain the text send from Pandorabot (like is done with the $SpeechPhrase variable). This would be the quickest and most straightforward route and should be relatively easy for him to do, but that task is much like getting a glacier to move by pushing from the back end, being as swamped as he is. I'm surprised this isn't already a part of that control actually.

2) To write a plug-in which would perform the same function as the current Pandorabot control but allow for the returned text to be placed into a variable for the user to send to a display other than the Debug window.

3) Write a plug-in which would be able to get at the text of any debug window.

4) Write a totally separate program which would provide a user interface for the users current Pandorabots and send the text to the computer's speech generator or to the ARC to be sent on to the robot's speaker.

5) Write something which would look at the data stream going to and from the internet on the user's network and parse out those parts coming from Pandorabot for display.

None of those options are easy or something I could do quickly since I am not versed with the SDK or the plug-in system. Nor am I familiar with the Pandorabot's system of coding. Though it doesn't seem to be too difficult.

Perhaps starting a new thread focusing on just the Pandorabot control and it's lack of a text variable like the Speech Recognition control has would be the best course of action for now. No one seems to be very concerned with this thread anymore but the two of us.
United Kingdom
#41  
@WBS.

My brain is refreshed today do I gave your program another go... And I t works great. Nice job indeed, and thank you for your work on this. I am very grateful indeed. I think I was over thinking things last night (as well as being really tired) as its a lot easier to display text than what I thought. *eyeroll*

If I could make a couple of suggestions though if that's cool.

One would be to be able to change the background colour or add an image so it blends in with any set ARC wallpapers that might be set.

And the other, not so much of a suggestion but more of a question, is there a way of adding a script to bring the display to the foreground? When the display is up but you then use an ARC control, the display is pushed to the background (as Windows does), so having a line of code to bring the display back to the foreground after another computer/robots response would be great. I can't think how to do that with a script.

In regards to the Pandorabot control, I thought that might be an issue. But like you say, it shouldn't be that difficault to add as the returned text is already displayed in the debug window. I don't think it's worth starting a new thread as we already have DJ's attention. I hope he see's this and think it's worthy of adding a vairable for this. I really hope so, as this would be quite an important addition for me personally (as I'll mention below).

I am surprised that there has not been much interest or input from other members on this entire subject. For such a simple tool, I can see so many uses for this like for kids education in robotics where some children my be hard of hearing and/or poor vision. This would be a great way to get this kids to interact with a robot platform. And, as in my case, family members who are hard of hearing who want to interact with a robot, for example, a scripted reminder "it's time to take your pills" would be better as large colourful font displayed text where speech syntheses wouldn't be adequate for this. I appreciate this may not be suitable for every robot, especially ones without onboard computers, but as ARC has to use a computer, then the option would be there for any type of robot using the EZ-Robot platform.

There are many more examples I can think of, and I'm sure there are other current or future members who are hard of hearing who would appreciate having this ability, but I guess the lack of responses here says otherwise which surprises me a bit.

BTW, I haven't tried the EZ-Display on my tablet yet, but I will do later today and let you know how it goes (although I can't see any ussurs using it in portrait mode as the window can be resized).
PRO
Synthiam
#42  
The colors are which are words, ie red, green, etc are universal computer colors. You can find that by searching c# color names. Or html color names. Those are the two most popular resources I use.

As for using RGB values (red, green, blue). The colors are numbers between 0 and 255. You will find programming languages and html use the same. The alpha is transparency. It is also a number between 0 and 255. I'm on my phone, but I'm certain there was an example of using rgb in the sketch pad example project. If not, I'll add it.

As for your ez display program... Awesome. May I suggest converting it into a ARC skill plugin? Take a look at the plugin howto and there is also source code for the new plugins that I had created in the list.
United Kingdom
#43  
@DJ.

WBS done great didn't he? I agree with you that EZ-Dislay should be available as an ARC plug-in, as it's deffintaly worthy of being one.

I don't know if you read my last post (#42), but would you concider adding a vairable to extract the response text from the Pandorabot control to display on WBS's EZ-Display?
#44  
@DJ Sures
Thanks for your reply. At least that confirms what I wrote back in post #35. I also listed the other colors back there.

There are 2 examples of using Sketch Pad but they only address a couple of points concerning it's use.

I would like to convert it to a plug-in but I still have a steep learning path to working with C# and the MS programming environment. Not to mention the fact I am averse to working with anything which has the letter "C" in it. :D It's difficult for me to work up the enthusiasm to do it at the moment. Thus my penchant for using a text file interface and going with what I am familiar. The only problem is that the text file interface is slow. Considerably slower than I expected even.

What I would like to see in ARC is a way to load a script from a text file. That would open up all sorts of possibilities by itself.

@SteveG
I can put in a background "color picker" option, no problem. Putting in a background image would be a bit more difficult however. I'll look into it.

Also, no problem on making the display stay on top of other windows. There is a setting especially for that in windows. I'll put in an option for that as well.

Making it do that from a script should also be possible by allowing for some control characters to be recognized. For example, you could send the character sequence "<StayOnTopOn>" to the display either by concatenating it to the beginning of a string you want to send, or simply by itself like this:

Code:


$DisplayText ="<StayOnTopOn>"+"Sentence to send to display"
#Or
$DisplayText ="<StayOnTopOn>"
Sleep(2000)
$DisplayText ="Sentence to send to display"

#Using a varible:
$StayOnTopOn ="<StayOnTopOn>" #Declared somewhere like in an Init script
$StayOnTopOff ="<StayOnTopOff>"

$DisplayText =$StayOnTopOn+"Sentence to send to display"
#Or:
$DisplayText =$StayOnTopOn
Sleep(2000)
$DisplayText ="Sentence to send to display"

To turn that off you could send:
"<StayOnTopOff>"

Concatenating the On-Off code to a string you wish to send anyway is the best method since it eliminates that extra Sleep(2000).

In a similar way the other options could be changed in value via text sent from the ARC making the Display program fully remote controlled. Doing that was the next phase of this project I had planned anyway.

Then there is the two-way communication as suggested by Justin. Also quite feasible.
PRO
Synthiam
#45  
I'll add the pandorabot response to a variable.
United Kingdom
#46  
@DJ.

Thanks for getting back to me and thanks so much for adding that. Having that vairable will be very useful indeed, and I for one would be very grateful to you for that.:)

@WBS.

Okay, that would be great. In regards to having the display in the foreground,I figured there would be a command for keeping there, but for the life of me I couldn't figure out how to do it.
Having it as an option would be a great addition.

Again, great work with EZ-Display (I like the name too).:D

Steve.
United Kingdom
#47  
@WBS.

I just tried EZ-Display on my Acer W3... It worked great, better than I expected in portrait mode actually. Half the screen is ARC, and resizing the display fills the other half of the screen with good visibility when using the font size option you added.

Thanks again for this. I'm really pleased you are part of the community.:)

Steve.

United Kingdom
#48  
@WBS.

One last little suggestion I wanted to add, and that is having a line space inbetween each of the robots responses in the display.
United Kingdom
#50  
Okay DJ. I'll check it out. Thanks.
#51  
@SteveG
Thank you for the kind words and you are more than welcome. Glad to help. Thanks for the video as well. It was great to see it working on the tablet. I was happy to see it automatically change width appropriately when being turned from Landscape to Portrait modes. I wasn't sure about that. Now that you have a Pandorabot variable, (even a response script) the last piece of the puzzle falls into place. That's excellent!

I have gotten the control code for having the window go to stay on top mode (and setting it back to normal) to work ok. Also, it is an option via the options menu. I plan to have similar control codes for the other functions as well. I'll make up a list of variables set to the appropriate values that will be initialized when the SendToDisplay script is first run. You will be able to refer to that script for the names as needed. Also, I'll have an option for placing a blank line between the text lines.

Additionally the color picker for the background is installed. You can choose from a set of predefined colors and create up to 16 custom ones. As with the other options, all that will be saved upon program termination and reloaded upon restarting the program.

Finally I am going to go ahead and enable the hide buttons feature so as to hide the two front buttons and the bar upon which they are mounted. I should be able to initiate that by double tapping in the display area. It would toggle from visible to invisible, giving you only a nice clean display area. That means the Clear button will be gone too, but I can make it clear the screen by a single tap. Unless you can simulate a right mouse click with your fingers somehow.

Once I have those things going well, I'll upload the new updated executable file, as well as, a revised SendToDisplay script so you can use those features soon.
United Kingdom
#52  
@WBS.

That's great. I posted the video as I guessed that you might like to have seen a quick demo on the tablet. Yes, I'm so happy that DJ added the Pandorabot vairable (and so quickly too), and the added bonus of the script editor as well which comes in useful for the text displays (your's and the plug-in DJ added to ARC too). The other features you mentioned will be a great addition to EZ-Display. I look forward to the update.
#53  
There are some files attached which are explained at the end of this post.

I finally have the newer version of the EZdisplay program done. The changes are:

1) Alphabetical arrangement of the selections in the popup menu.
2) A debug option added just in case.
3) An "Extra Linefeed" option for blank line between text inputs.
4) A background color select option.
5) The "Show Buttons" option has been enabled.
6) A "Stay on Top" option has been added.
7) Clearing the display area by clicking/tapping it once.
8) Showing/hiding the Set and Clear buttons with double click/tap
9) A start on the remote control commands functionality.
10) A remote control option to make the Display visible or invisible.

Taking these one at a time:

Debug option. This, when selected, generates a Debug.txt file in the same place as where the EZDisplay.exe file is located. It will allow me to analyze problems should they arise.

An Extra Linefeed option. This is simply for the blank line between text lines from the SendToDisplay script.

The Background Color option for selecting the desired color of the background of the display.

The Show Buttons option. This makes the "Set" and "Clear" buttons, and the strip they are on, visible or invisible. This option can also be set and reset with a mouse doubleclick or touch doubletap.

Usually the clearing of the display can be done manually by clicking/tapping the Clear button on the display. But, since the Set and Clear buttons can now be hidden as desired, a way was needed to bring them back. Usually this would be done by a mouse Right Click. No mouse on a touch screen, so I added a doubletap option to bring the buttons back. The doubletap will toggle the visibility of the buttons as desired. Additionally, the screen can be cleared by a single tap to the display area. When you doubletap to hide the buttons, the display area will also clear, however. I don't think that will be a problem. If it is, there is a way around that but will make the code more complicated (and possibly prone to errors) so I didn't employ it. If there comes a need to keep the text on the screen when doubletapping let me know and I will do that.

The stay on Top function. When selected, the display will stay on top of all other windows regardless of what they do. This function can also be set/reset via sending a remote control sequence. When the $StayOnTopOff is sent via remote control, the Display will probably immediately go behind another window.

Remote Controls
These are sent via the SendToDisplay script as desired. Currently, the Remote Controls are:

Code:


$StayOnTopOn ="<StayOnTopOn>"
$StayOnTopOff ="<StayOnTopOff>"
$ClearScreen ="<ClearScreen>"
$WinNormal ="<WinNormal>"
$WinMaximized ="<WinMaximized>"
$WinMinimized ="<WinMinimized>"
$WinVisibleOn ="<WinVisibleOn>"
$WinVisibleOff ="<WinVisibleOff>"

They are sent either by themselves or in conjunction with a line of text:

Code:


$DisplayText =$StayOnTopOn

#Or
$DisplayText =$StayOnTopOn+"Sentence to send to EZDisplay"

More Detail:

Code:


$WinNormal ="<WinNormal>"
$WinMaximized ="<WinMaximized>"
$WinMinimized ="<WinMinimized>"

These are associated with the usual functions accessed via the negative sign (-), the square, and the X found at the top right of most windows.

$WinNormal:
This is the usual state for the window depending on what size it was last set to.

$WinMaximized:
Fills the screen with the Display program.

$WinMinimized:
Reduces the window to an icon at the bottom of the screen.

$WinVisibleOff
Makes the Display invisible. Unlike $WinMinimized, it is still there so when it it is made visible again, It just appears. This, instead of flying out from it's minimized state.

$WinVisibleOn:
Makes the Display visible again. NOTE: $WinNormal, $WinMinimized, and $WinMaximized will also make the Display visible again.

Generally, the $WinNormal, $WinMinimized, $WinMaximized, $WinVisibleOff, $WinVisibleOn will be sent to the Display by themselves. They will still need a sleep period between them when they are.

The attached files are:
A new executable (EZDisplay.exe) and new SendToDisplay script containing the remote control variables. Simply overwrite the old executable wherever you put the original one in the install process, and copy and paste the contents of the SendToDisplay file into the SendToDisplay script in place of that is there now.
NOTE: Leave the first line as it is and overwrite the rest.

Let me know how it goes.

EZDisplayFiles.zip
United Kingdom
#54  
@WBS.

Many thanks for the update and for adding the upgrades. I haven't had a chance to download and try the update yet, but once I do, I'll let you know how it performs. Over the weekend, I had a great idea for using EZ-Display on my current project which should be fun.:)