Virtual Reality Robot

Plugin for all SteamVR supported virtual reality headsets (Google Cardboard, Oculus Rift, Windows Mixed Reality, HTC Vive, etc)

Get Version 25 by Synthiam
+ How To Add This Control To Your Project (Click to Expand)
  1. Make sure you have the latest version of ARC installed.
  2. Select the Get button in this page to download the archive file.
  3. Double click the downloaded archive file to execute installer.
  4. The installer will add this control to ARC.
  5. Load ARC and press the Project -> Add Control button from the menu.
  6. Choose the Virtual Reality category tab.
  7. Press the Virtual Reality Robot icon to add the control to your project.


Use Virtual Reality with your robot to see what the robot sees, and control servos by moving your head. This plugin supports all virtual reality headsets through Steam VR (OpenVR), Google Play and iOS, including Google Cardboard, Oculus, Windows Mixed Reality and HTC Vive. The plugin does two things, it allows you to see what the robot sees and control the servos with the Pitch and Yaw of the headset. This means when you move your head, the robot head can move as well, mimicking your movement. 

In this video, it was a test of the plugin on a Synthiam JD Humanoid. Sadly, he had no arms during the test so it's a bit funny to see. Professor E from The Robot Program was controlling the robot's head and viewing the robot's camera through the VR headset. In this example, we were using OpenVR in STEAM with a Samsung Odyssey headset.

In this video below, we were controlling an Synthiam Six Hexapod with a joystick and the VR headset. When we moved the head left to right, the legs of the hexapod will rotate the body so we can look around. This also applied to looking up and down, the legs would tilt the body so we can look up and down as well. In this example we were using an Android phone with Google Cardboard Glasses.

Here is another interesting example where we combined this VR Headset plugin with 2 WiiMotes to create custom animations. Each time a button was pressed on the WiiMote, a new FRAME was created with the current position of all servos and appended to an ACTION. DJ keeps pressing the button and the FRAMES keep getting appended to the ACTION until he stops. One he presses the trigger on the WiiMote, the code instructs the ACTION to be played back. And thus, all FRAMES are played back because they are part of the action.

Here is a screenshot of the plugin running on an ARC workspace with a camera.

User-inserted image

What Hardware Does It Support?
There are two methods that connect to the plugin which open a wide variety of hardware. 

1. SteamVR (OpenVR)
On PC, the supported headsets use Steam VR (OpenVR). This means any headset supporting Steam VR will work with this plugin, given the headset is directly connected to the PC. The most popular SteamVR headsets are Oculus, HTC Vive and Windows Mixed Reality.

2. Google Cardboard (iOS and Android)
The Google Cardboard is both a product and standard that turns your mobile device into a virtual reality headset. This means you can either purchase the cardbooard version from Google, or you can shop online and purchase a more robust plastic/foam padded version from amazon.

Download the Android App here:

User-inserted image

Allow Firewall Access
User-inserted image

When the plugin is loaded, you may receive a message to allow firewall access. Be sure to check Private and Public networks. If you do not check both, you may end up in a situation in the future where you're connected to a public network and the plugin isn't working. In this case, it's because ARC doesn't have firewall permission. Also, if you do not see this message when the plugin loads, it could be because you've already answered this in the past. In which case, you should visit the firewall rules to see if ARC has correct access.

1) Press the Windows button and type FIREWALL to search for Windows Defender Firewall
2) Select "Allow an app or feature through windows defender firewall"
User-inserted image

User-inserted image

4) Ensure both PRIVATE and PUBLIC are checked for ARC and press OK
User-inserted image

Attach/Detach Camera Button
User-inserted image

This plugin requires a camera, whether on your robot or a USB device. Add the camera device to your project and connect to the camera. Once you have a successful camera operational, press the Attach to Camera button on the plugin. This will connect the plugin to the camera device so the video stream is sent to the VR headset.

Start/Close VR Client Button (SteamVR only)
User-inserted image

If you are have a Steam VR compatible headset connected to your PC, this button will allow you to use that headset. This button will start the VR client, which connects to the VR Headset. The client can be launched either full screen or in a window. By default, the setting is to launch in the plugin window. This setting can be altered for your project in the configuration dialog.

Google Cardboard Users
For those using a google cardboard (iOS or Android) compatible device and headset, your device needs to be on the same network as the ARC PC. This is because your mobile device will connect to the ARC plugin by typing in the IP address of the PC. The IP Address for your ARC PC is displayed in the plugin. Simply load the Virtual Reality Robot mobile app on your device and enter the ip address of the ARC PC.

User-inserted image

4K Display Issues
There is an issue with 4K displays and running the local client in the robot skill window. If you have a 4k display and press the "Start Local Client" button, you will notice two things...
1) The client is very small
2) The client is offset from the display window

This is known issue with Windows & Unity with 4k displays. The way around this is to set the client to run in full screen, which is configurable in this skill's configuration window.
User-inserted image

Virtual Reality Caution
Read the warnings on your device before using a virtual reality (VR) viewer with this plugin to reduce the risk of personal injury, discomfort or property damage. Ensure you use this plugin with adequate space to avoid hurting yourself or others by walking into something. Nausea and motion sickness may be experienced and worse for some users. Do not use this plugin for extended periods of time to avoid experiencing nausea.:D

Updated the VR projection in the client to be far more stable to reduce motion sickness. Also it has a bunch of new UI enhancements and bug fixes
#65   — Edited
I only have an iPhone and no Android or a device with StreamVR support, so I can't test, although I did like the old vuzix one you made years ago that also turns the camera servo's when you turn your head. 

I might be tempted to buy an Oculus. Do you support two cameras with this plugin so you can get stereo 3D view ?
The iOS should be approved in a few weeks 

I tried it with 2 cameras last year and has terrible results. Aligning the share of each camera physically was challenging. I couldn’t get it to not make my eyes go buggy. 

I can always add it as an option if anyone and a use case where they were interested in tackling that physical challenge
Unknown Country
Can you please add a 2 camera option? It would really help with my project!
That’s a tough one - I’ve tried to consider how to do that a number of times. I’m not quite sure how. It’s mainly because unity is already rendering the scene as a vr 3D space. I can’t find a way to provide it to video streams that give it stereo view.
Unknown Country
Hmm, well I thought that because you said you could add it as an option for someone trying to tackle the physical challenge that you would be able to add it relatively easily. How were you able to produce the unenjoyable experience with 2 cameras if it only allows one camera? I am a bit confused. Thanks for your reply!
I originally thought it should be done. After looking into it further, I’m unable to figure out how it can be done with two cameras. The unity vr system generates the 3D space. I’m unable to place different objects in each eye of the vr glasses because unity is doing it within the engine. Maybe there’s a way if someone has an idea.
Unknown Country
Hmmm, yeah I'm not really sure. Do you think that a 2 camera method would even be possible on a one screen headset like the rift s?
Unknown Country
Also what did you do in order to use 2 cameras when the experience was unpleasant?
That was using directx and rendering two cameras in an oculus dk a number of years ago before steam vr. There was an oculus only plug-in that worked with the dk we had. It’s been deprecated when the oculus dk was discontinued. 

at the time, they distributed an sdk to access the video and sensor data, not requiring unity or any other AAA engine. 

these days you can’t talk to the headset directly without A LOT of effort. Which is why we rely on existing 3D engines like unity. 

I did find some information dating back to a 2016 unity version where someone wanted to generate an object in each eye. There’s a little hope it’s possible to do today if I continue researching a bit further. It’s such a rarity that there isn’t much reliable info to go on. 

I did add your request to the dev
 list. There’s just no eta on it until we have better ideas of a direction to take.
Unknown Country
Well thank you so much for doing that! Please let me know if you find anything that will help. I'll do some research too.
Good hope that it can be done - that's using the oculus sdk in unity so we'd need to look deeper into how it can be done with the unity vr engine.
Unknown Country
Currently, I'm not aware of the differences between using the oculus sdk in unity vs using the unity vr engine on oculus, so I will learn how they are different and then will do some more research.  The github page shows that it uses a lot of Classic ASP language in the code. Could that be a problem with getting it to run within ARC?

Also, thanks for your continued engagement and effort. It is a rarity that a company's CEO is so engaged and helpful with their community. You have my sincere appreciation and respect.
Unknown Country
I have done some research and I have a few questions. The VR client uses the Unity XR engine, right? I updated the build from github to the latest unity version and I was wondering if you could get the VR client to run that. Also the guthub page mentions the robot needs to be running linux ubuntu and ROS igloo to use GSCAM. Would that prevent ARC from working with the EZ v4 controller? Also, I'm not quite sure what operating system the EZ v4 controller runs, or if it even has an operating system. Could you explain?
thanks for the kind words. 

Adapting those code examples to jerry-rig into ARC will limit the flexibility. We have a belief that anything we create must be agnostic - and not locked to a specific use-case. This is why I’m using that codebase as an example to reverse engineer rather than duplicate. 

I have found some interesting code examples that demonstratest the direction I’d need to take
Unknown Country
Well it looks like I will leave it up to the experts! Thank you so much! Do you have any idea how long it may take for the skill or updated VR Robot skill to be accessible? I am working on prototyping my project this month and it would be very helpful to have it availible sometime this month. Lastly, I found another project on github that you many find find useful or could provide some inspiration. I have linked it.
Not yet - but it is on the list to be looked at one day:)