Virtual Reality Robot

Awesome plugin! I've been trying to do this for some time . Thank you Dj and all that worked on it.

Virtual Reality Robot Plugin

Glad you enjoyed it I'll add a few additional features, such as controllers support and such in future versions

Check mark off, on to do list! Thanks DJ!

EDIT: Is it possible to use the virtual servos ie dynamixels with this plug in?

https://synthiam.com/Community/Questions/11135

You can use any servos with any plugin. All plugins are compatible with all features of ARC.

Here’s a link on how to select servos in ARC: https://synthiam.com/Tutorials/Lesson/49?courseId=4

All controls that use servos in ARC share the same servo selection interface, this includes plugins.

Does it also support opentrack (/trackir)?

Thing is, I get motion sickness with VR even with 50+ FPS and virtually no input latency. Given the <20 FPS of the robot camera and the inherent massive latency involved in moving a camera with servos, I hope this plug-in comes with sick bags.

FWIW, Ive been using FPV (first person view) cameras and goggles on RC planes and drones for many years. My goggles have motion sensors and I can use them to control servos to move the camera, and using analog video transmission, there is zero latency in the video, but even so I learned a long time ago they only way this is bearable for longer than 2 minutes, is keeping my head absolutely motionless and controlling the camera servos with a stick.

I know, VR is all the rage, but in reality a trackir (or any of the DIY/opensource alternative headtrackers) would provide most of the benefits of being able to move a robot head/camera with head gestures, without the inherent downsides.

All recent headsets (i.e. vive, oculus, windows mixed reality) use head tracking. This is controlled by SteamVR and has nothing to do with the plugin or a game. Games do not write code for the VR headset, it's done by Steam VR, which interfaces with the headset. The video feed framerate is what ever you're source is, USB camera is 30 fps, wifi ezrobot camera is 20fps and usb ezrobot camera is 30 fps. The FPS has no correlation with the VR fps. The VR FPS is dependent on your GPU hardware. On my 1070, I get 90 FPS with this plugin.

What GPU and headset are you using? That will make the difference.

You misunderstand. I would be interested in controlling a robot head with my own head, but without putting on vr goggles.

In PC flight/race simulators we use trackir/opentrack for that, its typically 3 ir leds clipped to your headset, a webcam that tracks those LEDs and software that calculates your head motion: https://www.youtube.com/watch?v=XHNwQY9iMrw

That motion is then increased through configurable curves, so a smal head movement, say 10 degree or turning your head towards the edge of your monitor, results in say 90 degree change in perspective in the game.

This is a very cheap ($30 or so) and popular way to get some "VR" effect, allowing you to look around in your virtual cockpit, and compared to real VR sets has added benefits like higher resolution monitors, you can see your hands, see your keyboard, controllers, beer, ..

what you dont get is the 3D effect and immersion. But you dont get that with a ezrobot cam anyway, even when using a VR set.

Opentrack is the most popular software solution for this. Its opensource and supports all kinds of in- and outputs, not just tracking leds. It will also work with things like Wii controllers, Arduino sensor boards, eye/face tracking etc. And could be useful to do more than only control a camera, it could be used to control a robot hand or whatever.

As for what I was saying earlier; I am using FPV goggles, not on PC, but for remote control aircraft and racing drones. And I have built RC aircraft where the FPV camera has pan/tilt servos that I can control by moving the goggles, ie my head, much like a VR set. But I chose not to, because unlike a VR game on PC, where moving my head pretty much instantly moves the rendered camera, servos take 100s of milliseconds to reach their new position and the lag between moving my head and the servos and thus the image in my goggles responding makes anyone sick in minutes.

I fear it will be the exact same problem with robots. Possibly even worse because the video lag will be slightly worse (digital video feed that needs encoding/decoding versus analogy and thus zero latency video on my rc craft),

Using an opentrack or trackir, you dont have this problem, because you dont wear goggles that obscure everything except for the video feed, you can still see "the real world" move instantly along with your head, so your brain doesnt get confused.

If you want to control a camera on servos (perhaps a gimbal) with a wii controller, simply add the wii skill control to ARC, here's the link: https://synthiam.com/Support?id=154

Have fun!

I have been using the old Vuzix plugin but only has 2DOF and I don't believe it will work with SteamVR. I have a fatshark Dominator but don't believe it works with steamvr either.

That said I really don't want to pay for yet another VR headset. Has anyone tried this with Trinus or ivry etc with an iphone in a google cardboard thingy.

>a fatshark Dominator

If you really wanted to, you should be able to connect the fatsharks headtracking output (which is just a PWM signal) to an arduino or a USB simulator cable/dongle, and use that as input to a virtual joystick .

No need to use any third party stuff. There’s an iPhone and android client for this plugin waiting in the AppStore for approval. That takes about a week or so. You can use any google cardboard compatible device with your phone to control servos if your ezb and view the camera output, just like the video.

hey thanks Vertigo I need to buy the Head tracking module for the fatshark but now DJ says he has an iphone APP WOOT I will wait for that !

Yeah. The iPhone/android app is a client for this plugin. It uses the phones accelerometer and gyro for head tracking. It natively controls servos through ARC and doesn’t require any third party hacks - it just works.

If you wish to control the arms of your robot, I recommend wii controllers. They also magically work with ARC natively and don’t need any hacks. Simply follow the instructions in the wiimote tutorial to add the controllers to your pc via Bluetooth, configure the servos, and it works.

Here’s the link for wiimote: https://synthiam.com/Tutorials/Help.aspx?id=154

Here's the android app: https://play.google.com/store/apps/details?id=com.ezrobot.virtualrealityrobot

iOS is still waiting review

Thanks DJ looking forward to this app. Wii remote works well but would be good if we could get this working with VR controllers and inverse kinematics so robot arm moved with human arm and robot hand positioned with human hand location.

You can do the kinematics. Learn more about configuring multiple servos and adding ratios for joints, such as elbow and shoulder etc. It's easy to do with the servo control editor that's universal across ALL ARC skill controls. It doesn't make sense for me to create a plugin specific to my application, so instead there's features for you to customize it yourself. My effort is to make very universal controls that you can all expand and build with.

In the meantime, if you want to connect to other controllers for your application, take a look at creating your own plugin with this tutorial: https://synthiam.com/Community/Tutorials/146/1

And, here's instructions on how to configure servos for ALL ARC skill controls that accept servo commands. The UI is universal across the entire software application: https://synthiam.com/Community/Tutorials/49

Once you familiarize yourself with how to add more than one servo to features, you'll unlock a whole new power. Best example is how the Six Hexapod can rotate and follow objects with the camera. Take a look at that project for examples of how the servos are configured.

I do have on my list to add onto the plugin for hand controllers - for the present i'm using wiimote controllers.

Can someone tell me how Steam is used in using this VR App? Do I load the Steam app and select something?

I am very excited in using this app. I have a Samsung Mixed Reality Headset and have loaded its software and the Mixed Reality Headset software.

Ellis

You must be specific with names of things, specially in technology because things are similarly named but entirely different. Steam and SteamVR are different

Yes you need steam for steamvr. But steamvr is the dependency. Install steam, then install steamvr inside of steam.

Because you have windows mixed reality, you also have to install the windows mixed reality in steam.

And when you load the plugin, the rest just happens automatically. You just need steamvr installed, that’s all

My question is, if you have windows mixed reality, what have you been using it for if you don’t have steamvr?!? That’s where ALL the vr titles worth mentioning exist. without steamvr, having a virtual reality headset is a paperweight. You’ll thank me lol!

So. I haven't even tried the pmugin/android app yet and I already have a feature request. Likely to only be something a small number of users care about, so feel free to say no....

I would like to define the left and right cameras seperatly so that I could have true stereo vision with depth perception by using 2 cameras set at eye distance apart. Probably way more complicated than I am thinking (like need to be able to adjust centerpoint, etc....) but would be cool.

Alan

Thanks for the clarification. I already have installed steam to windows and windows to steam. As I understand all I need to do now is install tha ARC skill plugin.

I sometimes do not understand what is being said so I ask for some clarifications. The community is very helpfull and I thank the people on it and ARC for providing it. Thanks

You need steamvr that installs into steam. You also need to install windows mixed reality that installs in stream.

1. download and install stream

2. inside of steam, search their store and install SteamVR

3. inside of steam, search their store and install Windows Mixed reality

4. install Virtuak reality robot plugin into ARC

Thanks DJ, I will get right to it.

I already have stream and mixed reality installed but I don't think I have SteamVR installed unless I have and don't know it.

Look at your installed list...

This is great news...I cannot wait to try this!

Thanks I will do this.

Did IOS version of this get approved? I cannot find the app in the app store.

Unfortunately not yet - it's been rejected but we're working on the changes required.

Will this set up work with a EZB USB/serial connected to the computer? Or Wi Fi mode only?

Everything works with either usb or WiFi. There’s no difference across the entire ARC software. I don’t understand the question maybe?

Ok yeah I prolly need to explore the plug in a bit. Didn’t want to invest the dwindling time if i couldn’t use in the CES environment. Thank you.

Will, you do need WiFi between the phone and the PC even if the EZ-B's are connected by USB.  Probably not the best option for CES based on our previous discussion

Alan

I assume for ces he’d be using a steam vr headset because it’s wired. A phone would last only a few minutes with battery life. And it would just get so hot it would shut down in the headset anyway.

I sure wouldn’t recommend a phone lol

Looks like the Steam headset it is! lol.....I found an Acer Windows Mixed Reality head set on amazon...This particular model uses the HDMI port to connect to the graphics card, is that an issue?

I finally got Stream VR installed and Windows mixed reality and the VR Robot plugs installed. I cannot get get steam and the plug in to talk. Steam sees the headset and it says using ezrobot.exe, but no data is exchanged when i hit connect to client

Last issue... the plug settings all to default positions to 90 when you activate, this puts my dynamixel positions not facing forward. IE the dynamixel plug has all my settings on those servos 1-4096 positions. 90 should be 2048 on the dynamixels. Some how its not doing the math to convert 1-180 positions?

If i can get both the headset and the plug in to talk i can grab a video to show dynamixel issue.

Is there a longer description of setting up the plug in?

Does the headset work with steam vr? Do you have supported hardware? Did you verify your computers requirements?

Yes, yes, yes. Green check marks for everything. Headset came with windows mixed reality and it works great as stand alone.

Does STEAM VR work? Load STEAM VR from STEAM. Use the STEAM VR instructions to learn how to use your headset with STEAM VR. You have to install Windows Mixed Reality support into STEAM. You have to get your headset working with STEAM VR.

Have you read the comments in this plugin to learn from the effort i put into others for help? Do you have Windows Mixed Reality support added to STEAM VR?

1. Ensure a Camera Device is loaded and connected to a camera in your project

2. Press ATTACH TO CAMERA (any errors?)

3. Press Start VR Client (Steam VR should automatically load, does it?)

4. Press Robot View in the Steam VR Client window that loads

that's it

As for your servo settings - All servo controls are the same. Given the type of headset, the INVERT button may need to be used to invert the direction of the servos. If the servos work in any other control of ARC, they'll work in the plugin. Notice how they're all the same interface? Notice my comment on your other thread about asking about servos? They all use the exact same control which is the exact same code. It's all the same. Nothing is different - the entire servo interface in all of ARC is standardized to be identical.

OK Just rebooted the system, got the camera to transfer data to the head set. But I cant click activate servos, it wont let me check mark it.

What do you mean? You click on it and the checkbox doesn't change? Where are you clicking? On the words or on the checkbox?

Also, i notice you have the Config window open while trying to click on something in ARC? It doesn't work that way. You have to close teh config window because it's a dialog widow. ARC has always been like that, and so are all windows programs. When you open a dialog window in a windows porgram, you have to close it before using the real window.

I open the config to take a picture of the correct set up of the dynamixels, not during testing.

I will take a video to show you that check box is not clicking for me...seems to hang the steam VR as i detach camera and close the VR Client Steam VR says unresponsive.....

Final item is still the dynamixels. Please watch this video. Also sorry for the lethargy ...days of prepping for CES...exhausted.

Then set the servo ranges to match the head position. Move the positions until the vertical head angle is looking forward. You will have to experiment with the values, the software can't guess what they should be. Figure it out with experimentation...

No worries. I'm gonna pass on this. Too many other things to do. Thanks for the help.

I watched a documentary years ago about how our brains solve problems. They had these people play an arcade game for skiing. The people would play all day and have to keep beating their high score. Then, they’d get a full nights rest. The next day, on their first attempt, they all beat their previous high score.

It’s because our brain processes information of the previous day while we sleep - it’s a process similar to defragging a hard drive. But, it’s also solving problems and sorting the day’s information into knowledge.

1. organize your TO DO list and prioritize items between NEEDS and WANTS

2. when you’re feeling defeated, move to another task and revisit it the next day after a full night of rest

We have 48 hours until we step on a plane to Vegas. I've managed to wear all the hats (as usual) and produced all the flyers, business cards, banners in photoshop, dismantle the robots, design, build and install a bluetooth speaker systems, cast new faces in silicone and build shipping containers, created a pitch deck, made a press kit, etc. And alot to do in the remaining time. (No investor can deny my sweat equity.)

The reality, I only had 12 hours to get this plugin to work with my robots. That's all the time available in my schedule. It was a last minute decision to try and demonstrate telepresence, something not planned for the event. If I had a week, to mess with it then maybe I could figure out the dynamixel issue. The reality is I just don't.  As of this morning we have 84 companies who have scheduled to see us. Hopefully none of them request a telepresence demonstration Maybe we fake it til we make it

Will - have you spent any more time with this control? It's so much fun!

Ha not yet, I have to pry the headset from my kids hands. They are on it everyday!

I hope to revisit soon!

Any news on the IOS port of the app? Sorry my entire family switched to apple years ago so we are locked into their eco system.  :-(  https://play.google.com/store/apps/details?id=com.ezrobot.virtualrealityrobot

Thanks for the reminder - i'll take a look and see where it's at

It there a app I can use my iPhone to use the  EZ-Robot Virtual Reality Viewer?

is there an app for am iPhone to use the  EZ-Robot Virtual Reality Viewer? I don't have a Android

Is there an iPhone app for the Virtual Reality Viewer? I don't have an android phone like this: https://play.google.com/store/apps/details?id=com.ezrobot.virtualrealityrobot

This is what I am interested in doing this summer outside when my robot goes on patrol outside. See what the robot see in 360 or 180 degree VR 3d and take control of robot drive and arm movements. I just got the Oculus Go and it is a fantastic VR device to try this, will the plugin support all VR head sets?

Oculus Go is great, but using my iPhone seems a little more in my range  :-)

@EZang60, the iOS iPhone app is not available at this time

@RobotRad, the supported VR headset requirement is in the plugin description. Steam compatible headsets are supported - or google cardboard

Thanks for your response,

Hey thanks DJ I do have the steam client working with my GO ,will try out the ARC skill plugin as well. Glad to know you are working on this cutting edge technology way before anyone else, I bet you are already working on adding EZ chip to bionic arms and legs for disabled people ten years in the future!

Is their a way that I can move my robots head when using my htc vive as it is not working right now.  Never mind got it had to activate servos

Here's a tutorial that I highly recommend watching that walks you through the process: https://synthiam.com/Community/Tutorials/Control-robot-with-Virtual-Reality-headset-18905

You can find related tutorials to this plugin above under the description and manual, including a link to that tutorial

Thank DJ for the help.

@DJ_Sures Can you make this plugin allow people to select what camera to tie too as I have two camera's and they are not named Camera so it does not tie to them.

Use the ControlCommand to specify the camera name

Updated the VR projection in the client to be far more stable to reduce motion sickness. Also it has a bunch of new UI enhancements and bug fixes

I only have an iPhone and no Android or a device with StreamVR support, so I can't test, although I did like the old vuzix one you made years ago that also turns the camera servo's when you turn your head.

I might be tempted to buy an Oculus. Do you support two cameras with this plugin so you can get stereo 3D view ?

The iOS should be approved in a few weeks

I tried it with 2 cameras last year and has terrible results. Aligning the share of each camera physically was challenging. I couldn’t get it to not make my eyes go buggy.

I can always add it as an option if anyone and a use case where they were interested in tackling that physical challenge

Can you please add a 2 camera option? It would really help with my project!

That’s a tough one - I’ve tried to consider how to do that a number of times. I’m not quite sure how. It’s mainly because unity is already rendering the scene as a vr 3D space. I can’t find a way to provide it to video streams that give it stereo view.

Hmm, well I thought that because you said you could add it as an option for someone trying to tackle the physical challenge that you would be able to add it relatively easily. How were you able to produce the unenjoyable experience with 2 cameras if it only allows one camera? I am a bit confused. Thanks for your reply!

I originally thought it should be done. After looking into it further, I’m unable to figure out how it can be done with two cameras. The unity vr system generates the 3D space. I’m unable to place different objects in each eye of the vr glasses because unity is doing it within the engine. Maybe there’s a way if someone has an idea.

Hmmm, yeah I'm not really sure. Do you think that a 2 camera method would even be possible on a one screen headset like the rift s?

Also what did you do in order to use 2 cameras when the experience was unpleasant?

That was using directx and rendering two cameras in an oculus dk a number of years ago before steam vr. There was an oculus only plug-in that worked with the dk we had. It’s been deprecated when the oculus dk was discontinued.

at the time, they distributed an sdk to access the video and sensor data, not requiring unity or any other AAA engine.

these days you can’t talk to the headset directly without A LOT of effort. Which is why we rely on existing 3D engines like unity.

I did find some information dating back to a 2016 unity version where someone wanted to generate an object in each eye. There’s a little hope it’s possible to do today if I continue researching a bit further. It’s such a rarity that there isn’t much reliable info to go on.

I did add your request to the dev list. There’s just no eta on it until we have better ideas of a direction to take.

Well thank you so much for doing that! Please let me know if you find anything that will help. I'll do some research too.

I did some research and I found a page on github that used 2 cameras to create a 3D image on the Oculus CV1. I have linked it. https://github.com/feferna/dual_cameras_with_oculus_cv1

Good hope that it can be done - that's using the oculus sdk in unity so we'd need to look deeper into how it can be done with the unity vr engine.

Currently, I'm not aware of the differences between using the oculus sdk in unity vs using the unity vr engine on oculus, so I will learn how they are different and then will do some more research.  The github page shows that it uses a lot of Classic ASP language in the code. Could that be a problem with getting it to run within ARC?

Also, thanks for your continued engagement and effort. It is a rarity that a company's CEO is so engaged and helpful with their community. You have my sincere appreciation and respect.

I have done some research and I have a few questions. The VR client uses the Unity XR engine, right? I updated the build from github to the latest unity version and I was wondering if you could get the VR client to run that. Also the guthub page mentions the robot needs to be running linux ubuntu and ROS igloo to use GSCAM. Would that prevent ARC from working with the EZ v4 controller? Also, I'm not quite sure what operating system the EZ v4 controller runs, or if it even has an operating system. Could you explain?

thanks for the kind words.

Adapting those code examples to jerry-rig into ARC will limit the flexibility. We have a belief that anything we create must be agnostic - and not locked to a specific use-case. This is why I’m using that codebase as an example to reverse engineer rather than duplicate.

I have found some interesting code examples that demonstratest the direction I’d need to take

Well it looks like I will leave it up to the experts! Thank you so much! Do you have any idea how long it may take for the skill or updated VR robot skill to be accessible? I am working on prototyping my project this month and it would be very helpful to have it availible sometime this month. Lastly, I found another project on github that you many find find useful or could provide some inspiration. I have linked it. https://github.com/jleppert/3dfpv

Any updates on that project?

Not yet - but it is on the list to be looked at one day

v26

Updated with Roll servo option

v27

• Fix to selecting the roll servos in the config

This is by far my favorite skill. Appreciate the quick updates (especially the Roll servo config).

I'm getting some new errors I didn't get before though in the stable version:

Disconnection on tty1 VR client connected on tty0 from [My IP ADDRESS HERE] Streaming video to VR client... Disconnection on tty0

Any ideas? The servos work fine for about 3-5 seconds, then stutter, freeze, and it disconnects from the VR server.

???

I'm working with the team on this robot skill. It appears to work correctly in our tests. Can you verify that the disconnection is not from a power brown-out due to the additional servo being moved?

Is the EZB also disconnecting when the VR client disconnects?

It could be a power brown out, but the X and Y axis were working consistently in v26. I'm also running up to 10a power brick (at 7.4v). They're certainly acting like a brown out though - stuttering, freezing and then disconnecting.

The EZB does not disconnect when the VR client disconnects. Seems to be something with client server and IP address. Maybe just on my side.

Is it possible to have version control access? IE Can I install the stable v26 again?

I should also highlight, I didn't even add an additional servo. I have it set to N/A; so there shouldn't be any additional workload (yet).

Hey - I just gave it a shot to see what changes were made. After reviewing the code, the only added thing was the Roll request. So I tried this robot skill on a robot with no trouble. Because your robot is disconnecting, I'm confident that the issue is with power and your servos.

One way to test is to try the robot skill without an EZB connection. Do not connect to an EZB.

Run the robot skill, configure it for servos, and select the Activate Servos checkbox. You should see the values change.

You can confidently determine its power brown-out if the robot skill shows those values moving without an EZB connection.

This was a great idea - just tried it. Involuntarily disconnected from server in 60 seconds or less (no servos involved). This is definitely a problem only I'm seeming to have, so maybe it's a settings issue internally.

So weird.

Also just tried running exact same servo profiles with a joystick and they ran fine for 3+ minutes.

Okay, with no servos involved, that could mean something silly with the robot skill that wasn't caught before. I wonder if the Windows Defender is getting involved? Or the Windows Firewall?

When you check the Task Manager, do you see Anti Malware using a lot of CPU during the runtime?

I'll dive in deeper and see if I can reproduce it on my end.

I verified the robot's skill is working. I even made a few optimizations to it for performance.

However, I noticed an issue with the VR client app - that you might be experiencing. That's the unity program that connects to the headset. It was never caught before because the camera framerate wasn't as efficient. Over the last number of ARC releases, the video camera process has gone through several performance improvements. I believe one or all of those improvements has caused a blocking call between two threads in the VR client unity app.

I can resolve it and publish a new update to this robot skill tonight. Unity has changed a lot since this skill was created, so it needs several updates.

Any updates that address this issue would be sincerely appreciated!

My boy cant wait to get his neck software fixed!

v28

• fix for unity app closing connection

v29

• Few changes from DJ's update for increasing the versioning on the unity app and locking another object to prevent cross-thread errors

Thanks Synthiam team - this version is awesome! Super stable!

@hardy,  your boy is looking pretty good! Have you made any videos?

https://youtube.com/shorts/GD-VjxTWpEU?feature=share

v30

Minor performance improvement

Loads faster