Streaming Audio From Other Program Through ARC

Question

Hi all,

Has anyone been able to Stream audio triggered from other programs through ARC over wifi to an EZB? I know the PC soundboard will play sound files located on your laptop. But what I want to do is play audio on an EZB that would normally play through my PC's speaker. For example maybe an internet radio show, a youtube video audio track, sound files triggered and used by another program on the laptop other then ARC.

Thanks in advance.

DJ Sures · Answer

Hi Dave - that may be a feature request or plugin. The WASAPI can be used as input for something like the microphone control. I'll think on it.

Dave Schulpius · Answer

Thanks DJ. Wish I knew how to write plug in or even what would need to be used to make this work.

ptp · Answer

I just finished publishing a plugin to relay sound to EZB. @DJ: it's my first plugin i think i missed some point, the zip contains the DLL but ARC can't load the plugin, do you know if there are logs to help troubleshoot the issue ? Thanks

ptp · Answer

I solved the issue, compilation issue (Debug vs Release). Please let me know if it works for you.

DJ Sures · Answer

Oooh fancy! I'm excited to try it out.

CochranRobotics · Answer

I just messed with this plugin. Really cool PTP, and useful!

ptp · Answer

@DJ, Please try, there is a glitch every time (red light) between the PlayData actions, i tried different methods, but no success, i think i'll need an aditional PlayData method to allow adding data to the current EZB .NET Stack PlayData Buffer. If you have time i can open & share the code in Github to improve the Plugin. @David does it work ? I trusted some old code from an old project (Broadcasting music to multiple wifi ARM/Arduino Clones), then i spent a couple nights porting to an ARC skill plugin, and then i crossed the fingers

CochranRobotics · Answer

It worked well. There was some lag issues where the audio cut out for a few seconds but nothing too annoying. Played youtube audio from a web browser through the EZ-B V4 speaker.

CochranRobotics · Answer

Take that logo you used and include it in the zip file calling it title.jpg and you will have a nice graphic to find the plugin by.

ptp · Answer

now i understand why there was a title.jpg in one of the DJ's projects. If DJ's is reading this, he will say there is a "Tutorial...." i know it's my fault I do the same thing with books, I look the Index, i browse some chapters, and then i tell everyone i read the book

Dave Schulpius · Answer

I'm overwhelmed with grateatude and excitement. Thanks so very much. I've been hoping for this function useing ez robot for years now. If I don't end up working another long day today at the day job I'll give this a test run tonight. I'll be useing the remote mouse plug in to trigger a button in another program to play current time and date phrases. Ive already been able to do this. I expect with this new plug in the answer will now be played through the ezb on the robot instead of the computer speaker. Thanks again and I'll keep you posted.

DJ Sures · Answer

I'm flying today and will land in a few more hours. I'll take a look at it then. Also, yeah it's in the Tutorial for the title didn't want to disappoint you by not saying it hahah. I recommend using PNG files over JPG. Because png also has transparent background and the compression is lossless so you don't get those big blurry pixels.

Dave Schulpius · Answer

I must be doing something wrong. Cant get audio to stream to my EZB from anything playing on the computer after installing the plugin. Perhaps I don't understand how to script properly to get this to work. I installed successfully and am using the latest ARC. Added the plugin into a new project and connected to a V4 EZB. I opened the settings and configured it by clicking on the only audio choice in the drop down (my default audio out driver) I then wrote two scripts with EZ Script manager by copying your example scripts and making sure they pointed to the proper device number. I added a sleep command to the end of each because without it the scrip wouldn't stay actave while I played the pc's audio. One is the capture audio:
ControlCommand("AudioToolbox plugin", "StartCapture", 1) Sleep(10000)

Second is the EZB Listen:
ControlCommand("AudioToolbox plugin", "StartEZBListen", 0, 1, 75) Sleep(10000)

Then I triggered either one or both scripts and played various audio sources like youtube and sound files on my computer. All I heard was the laptop speakers play the audio. Am I missing something here. I can get the EZ v4 Soundboard to work with the EZB in the same project. Do I need to do something else to get EZB to play the sounds? Do I need to use both of these commands at the same time to get this to work? It kinda seems like I would. Will the computer's speakers continue to also play the sound? That's all I'm hearing now. Nothing out of the EZB. I've tried this on two Windows 10 computers. One old Dell that was upgraded and a newer HP that was upgraded.

ptp · Answer

@Dave, When you open the configuration, there are 3 dropdowns #0, #1, #2, did you select the device "Out" for the 2 dropdown #1 ?

ptp · Answer

the following command starts the capture on device #1, the second dropdown:
 ControlCommand("AudioToolbox plugin", "StartCapture", 1)

Dave Schulpius · Answer

Hi ptp, First I want to say thanks for the hard work on this and being willing to hold my hand here. Anyway, I'm pertty sure I have the correct drop down's selected and matched in the script. In fact I even tried choosing the same device in each of the three dropdowns. I'll post my simple project for you to see if you wish.

Dave Schulpius · Answer

Here's my little test project: test.EZB

ptp · Answer

@Dave, Your script has two spaces between AudioToolbox Plugin, change to one.

Dave Schulpius · Answer

Humm, OK, Thanks! I copied these from your examples in the plugin section. LOL. I'll have to give it a try tomorrow. Off to bed now. Got to be back to work in a few hours.

ptp · Answer

@Dave,

It's a glitch, i just checked the plugin information, when i use the editor there is only one space between "AudioToolbox" and "plugin" but when the content is rendered as HTML, some black magic runs and one space becomes 2....

I thing the culprit is the HTML gadget used to highlight the keywords.

maybe DJ can perform some white magic to fix

DJ Sures · Answer

Dave, use the cheat sheet.

Dave Schulpius · Answer

I love the cheat sheet DJ. I use it all the time. Last night I was feeling lazy and just cut and pasted. Guess I won't do that again. Lol. Looking forward to cleaning this up and getting this running tonight after work. Never enough time to play and have fun. While I wait could you tell me the way the Listen and Capture commands should be used? What does each one do. Why do I want to capture and listen? Do I need to use both to get the sound to stream to the ezb speaker? Sorry if this sounds like a dumb question.

ptp · Answer

@Dave, Short Answer: Yes, you need both commands, because capturing sound and forwarding (listen) are two different actions. Long Answer (Next Post)

Dave Schulpius · Answer

Kinda what I thought. I'm guessing you need to initiate a device in windows and then start ezb listening to it.

ptp · Answer

Long answer: When you capture sound or video, there is a delay until you fill a buffer, and then you start doing something at specific intervals. Buffering introduces a delay between the real data (sound) and whatever you do next with the captured data (play on the EZB). The capture process introduces a buffer of 5 seconds, this means when a capture action is received, 5 seconds are captured and only then are ready to be forward to a device, and then repeating a cycle of 5 seconds It could be less time. why is needed a buffer ? all the code running between capture, re-sampling, sending to EZB, takes cpu time. When capturing sound the encoding will vary per device, generally the output device (Speakers) generates: Stereo, 48Khz, 32 Bits EZB accepts Mono, 14.7Khz, 8 Bits. The re-sampling is also done by the capture action. As an example: 1 second of sound captured represents: 48000 x 4 (32bits) x 2 (Stereo) = 384K 1 second of sound sent to EZB represents: 14700 x 1 (8bit) x 1 (Mono) = 14.7 K Each capture action, is a different thread and is responsible for capturing, resample, record to file (if specified), and forward to all EZBs listening.

ptp · Answer

The Listen command adds an EZB to the capture thread, even if you add two EZBs at different intervals, both will be in sync. The capture thread forwards the sound at specific intervals (5 seconds). BUT if you do this:
 #device 0 = Speakers(Out) ControlCommand("AudioToolbox plugin", "StartCapture", 0) #ezb 0 listen device #0 ControlCommand("AudioToolbox plugin", "StartEZBListen", 0, 0, 75) Sleep(1000) #device 1 = Speakers(Out) ControlCommand("AudioToolbox plugin", "StartCapture", 1) #ezb 1 listen device #1 ControlCommand("AudioToolbox plugin", "StartEZBListen", 1, 1, 75)

EZB 1 will have a delay of a 1 second or more compared to EZB 0 If you want 1 or more EZBs synchronized you should add them to the same capture. Sorry the "mambo jambo" stuff, but sometimes it's necessary to explain the implementation details.

Dave Schulpius · Answer

It's working for me now. Thanks very much. It sounds great! It was just as simple as the extra space. Also thanks for the info on how to run the scripts. As great as this works I was wondering if you had a couple ideas. You mentioned that the buffering delay could be shortened. How would I do that and how short could we go? Also is there any way to stop the speakers on the computer from playing when streaming to EZB when using this plugin? It would be nice to have the only sound coming from the EZB like hen you use the V4 Soundboard. ? Perhaps I could install another audio device? I did notice that if I raise or lower the volume or even mute it on the computer the EZB speaker will adjust at the same time. If non of that is possible then as it is, this plugin you made is fabulous anyway.

ptp · Answer

@Dave, I ll need to investigate. i have some other projects on queue (Kinect is one of them), i ll come back to this as soon as possible. but i'm always open to brainstorm ideas. One question before you mention you capture sounds from an APP to stream to EZB, what kind of app is that ?

Dave Schulpius · Answer

PTP, sorry if I wasn't clear. I don’t have an app installed yet that I could configure your plugin towards that I could use. I was just wondering if there was one out there that I could install on my computer that I could point the plugin towards that would allow me to turn off the speakers but still stream to ezb. All I have listed now for available devices to configure to is my laptops sound and speaker system. When I mentioned volume and mute I was talking about the laptops sound system. Anyway, there's no big hurry. What you have done so far is amazing and much appreciated. Take care of your other projects. They sound awesome also. In the meantime I have lot of other things to do including play with this new plugin of yours.

thetechguru · Answer

There are (fairly inexpensive now) USB sound card devices. You could plug one in and jsut not connect speakers to it. Here is an example: https://www.amazon.com/Sabrent-External-Adapter-Windows-AU-MMSA/dp/B00IRVQ0F8?ie=UTF8&ref_=zg_bs_3015427011_1 If the app you are using to play sounds doesn't allow selection of the sound device, the Windows 10 Mixer does allow you to specify which device is used for which applications. It would probably be possible to create a "dummy" sound driver that did the same without hardware, but I doubt it exists today since the use cases are pretty small and there is no way you could get someone to write one for less than the hardware solution costs. Alan

Dave Schulpius · Answer

Thanks Alan! This was just what I was thinking and looking for. If I can install this usb sound card and map the plugin to it I may not need PTP to do any adjustments to his coding.

ptp · Answer

@Alan, Good info, the first time i played with windows sound api, i used DirectSound, the plugin has some direct sound api code, but, is not being used. A funny fact, i'm using code from 1999, so still working today With Direct Sound you need to rely on the hardware board to provide a Mix or loopback device. If not available the solution would be to wire the speakers output to a line IN to capture sound. but the game changed, when Vista was launched, a new API WASAPI the "Loopback" channel is done by the new API, so does not matter what kind of hardware driver. I believe the wasapi loopback is like a Y cable, so you are dependent of the speakers volume, you don't have control over the output. I noticed that with the sound format, you can't instruct the API to capture the sound with a specific format. Alan suggestion makes sense, although i think it's possible to create a software Endpoint, but it can't be hosted inside an ARC skill plugin. @Dave, what kind of Windows you have ?

Dave Schulpius · Answer

Win 10

Dave Schulpius · Answer

I have one of these USB sound board adapters on it's way from Amazon Prime. Wow, can't beat $6 USD and free 2 day shipping. Not to mention Sunday "free" delivery. Due to arrive tomorrow and I just ordered late last night. This should work just fine. I plugged in a set of head phones into the laptop jack just so I couldn't hear the first stanza from the computer speaker before the 5 second buffering delay out of the EZB. Other then faintly hearing the Munchkin like voice out of the head phones it took care of the problem. As for the 5 second delay; Did I mention I'm using Speech Recognition to trigger Scripts that run the Remote Mouse Plugin so I can click on a phrase in an outside program to say the time and/or date in the robot's orignal Lost in Space TV voice? I'm using the I-B9 voice interaction program available through the B9 Robot Builders Club here: Craig Reinbrecht's I-B9 program I have that scripted now but there is that 5 second delay between when I ask for the time and date answers and the responce. What I've doe to deal with that is to place a command in the script to have the V4 Soundboard play a voice file or two while I wait; "Please Stand By", "Checking". and things like that. Well, that's my setup so far. I only have hit one issue; often while this above process is happening the EZB will disconnect from ARC and the robot will lockup, turn on a lot of his motors and lights triggered from controls attached to EZB's Digital Ports. Also (and more annoying) the sound file will keep repeating a few words of the file like "Please Stand, Please Stand, Please Stand, Please Stand". Well you get the picture. I'll be troubleshooting this issue today hopefully. I'm pretty sure it's not a low power or WIFI signal strength issue. I have lots of other load on the system at other times with no issues and the WiFi signal strength is very strong. I'll be looking at the script and what is being done at the time of disconnect. Really only a few things are happening; *The Audio Toolbox is turned on. *The V4 Soundboard runs a sound file or two and then stops. *The Remote Mouse plugin clicks on the I-B9 software icon on the Windows toolbar to switch over to that program and clicks on a phrase over there. *The Audio Toolbox plugin is running during this and is buffering, then speaks the phrase when it appears in the buffer. *The Remote Mouse then clicks on the icon on Windows 10 toolbar to switch back to ARC. *The Audio Toolbox plugin is turned off. What stands out to me above are two things: *The V4 Soundboard is playing at the same time the Audio Toolbox plug is running (Conflict?). *The Mouse is clicking around all over the place (Com Port conflict like the old Window days? Don't know if this is still an issue now a days. If I recall the mouse and sound card or video display used to have issues with each other sometimes). Anyway, just some rambling thoughts here to help me work through this. I'll report back later. Feel free to interject any advice. Thanks!

Dave Schulpius · Answer

I've been working on this today and I think I've got most of the bugs worked out of my setup and scripts. I've got a stable connection now and have smooth transitions between switching programs and between streaming audio from the I-B9 software and ARC's V4 Soundboard. Seems seamless now. I think the problem I had with ARC disconnecting from wifi had something to do with script timing of the sleep commands. I was not giving things enough time to process. Also it's possible I was not properly stopping the Capture and Listen processes before I tried to restart them again. After some testing I found that not giving commands enough time to execute and not stopping the capture and Listening before restarting them was the only things that caused disconnections. Thanks to PTP, DJ and Alan for the plugins and help! Here's a video I took of the end result (so far). It's a little low quality. Sorry but I hope you enjoy:

Dave Schulpius · Answer

@Alan and @PTP, Sadly the USB Sound card adapter doesn't solve the problem. I am able to get all the sound from the computer to channel through it and play. However the plugin PTP wrote doesn't seem to pick up the audio when running through the USB adapter. I did make sure all settings in windows 10 that I'm aware of point to this adapter for sound playback. Maybe I missed something. When I change back to the laptops default driver and laptop speakers the plugin works like I show in my video. I hear the phase played on the laptop speakers and then 5 seconds later when the buffering catches up I hear it through the EZB in the robot. I even tried using the default laptop driver with the USB Sound Card adapter but no go (Windows Sound Troubleshooter suggested I try this and did it for me). So for now the way I've eliminated the double talking from the laptop and then from the robot is to attach a set of headphones with it's own volume control in it's cord. That works well. When I plug these headphones in to the laptop's jack the main speaker shuts off. I then just use it's adjustable volume switch to turn the volume all the way down. Walla! I can live with this.

ptp · Answer

@Dave, 1) Can you capture a screen shoot of the Device list i.e. Plugin's ComboBox, to see what's being list there. 2) Can you share the amazon link of the usb device ?

Aceboss · Answer

Hi Dave, As obvious as this probably sounds, plugging a 3.5mm jack which is not connected to anything into the audio socket would cause your laptop speakers to mute just as well, and you wouldn't have to fiddle with the volume control on your headphones. If you don't have a spare 3.5mm jack laying around, they can be purchased for just a few cents from virtually any electronics or audio store. Regards

Dave Schulpius · Answer

Holy cow. Am I a dope. PTP, I never reconfigured the capture list to point to the new sound card. blush I'll go down to the lab and check that out now. Boy do I feel dumb. eyeroll Aceboss, Thanks for that tip. I didn't even think about that. tired If pointing to the proper device in PTP's plug in doesn't work I'll just do that.

Dave Schulpius · Answer

OK, After re-configuring the audio toolbox plug in to point to the new USB sound card all is well. It works as expected and without speakers plugged into the adapter I get no double talk. Aceboss, I just may give your tip a try. Just a wire hanging off the side of the laptop seems much better then a dongle. Thanks again for the help.

thetechguru · Answer

I assumed you still wanted the ability to have other PC sounds from the PC, which is why I suggested the USB card and discussed using Windows 10 mixer to select which apps use which device. If you want all sound to go to the EZ-B, then yeah, just plugging in a disconnwplug, either unwired or just cut off an old headset (careful not to short the wires) would work fine. Alan

Dave Schulpius · Answer

Thanks Alan and everyone else. You all have been great help. I would have been stuck in the mud without you all.

solasasa · Answer

Quote: I solved the issue, compilation issue (Debug vs Release).  @ptp hello, I am having the same issue. I did the steps in the tutorial but I didn't find the plugin I made in the ARC add controls window. How did you solve it?

DJ Sures · Answer

You’re missing a step in the very excellent tutorial: https://synthiam.com/Tutorials/UserTutorials/146/1 One of the steps I assume you’re missing could be: 1) output of compile not being put in the plugin folder with folder name of your guid 2) plugin.xml not being copied to output folder 3) plugin.xml not containing the dll filename

Dave Schulpius

Upgrade to ARC Pro