Virtual Reality Robot

by Synthiam

View your robot’s camera using any SteamVR supported virtual reality headsets (Google Cardboard, Oculus Rift, Windows Mixed Reality, HTC Vive, etc)

How to add the Virtual Reality Robot robot skill

  1. Load the most recent release of ARC (Get ARC).
  2. Press the Project tab from the top menu bar in ARC.
  3. Press Add Robot Skill from the button ribbon bar in ARC.
  4. Choose the Virtual Reality category tab.
  5. Press the Virtual Reality Robot icon to add the robot skill to your project.

Don't have a robot yet?

Follow the Getting Started Guide to build a robot and use the Virtual Reality Robot robot skill.

How to use the Virtual Reality Robot robot skill

Use Virtual Reality with your robot to see what the robot sees, and control servos by moving your head. This plugin supports all virtual reality headsets through Steam VR (OpenVR), Google Play, and iOS, including Google Cardboard, Oculus, Windows Mixed Reality, and HTC Vive. The plugin does two things; it allows you to see what the robot sees and control the servos with the Pitch and Yaw of the headset. This means when you move your head, the robot head can also move, mimicking your movement. 

This video was a test of the plugin on a Synthiam JD Humanoid. Sadly, he had no arms during the trial, so it's funny to see. Professor E from The Robot Program controlled the robot's head and viewed the robot's camera through the VR headset. In this example, we used OpenVR in STEAM with a Samsung Odyssey headset.

In this video below, we controlled a Synthiam Six Hexapod with a joystick and a VR headset. When we move the head from left to right, the legs of the hexapod will rotate the body so we can look around. This also applied to looking up and down; the legs would tilt the body so we could look up and down. In this example, we used an Android phone with Google Cardboard Glasses.

Here is another exciting example where we combined this VR Headset plugin with 2 Wiimotes to create custom animations. Each time a button was pressed on the WiiMote, a new FRAME was created with the current position of all servos and appended to an ACTION. DJ keeps pressing the button, and the FRAMES append to the ACTION until he stops. Once he squeezes the trigger on the WiiMote, the code instructs the ACTION to be played back. And thus, all FRAMES are played back because they are part of the action.

Here is a screenshot of the plugin running on an ARC workspace with a camera.

User-inserted image

What Hardware Does It Support?
Two methods connect to the plugin, which opens a wide variety of hardware. 

1. SteamVR (OpenVR)
On PC, the supported headsets use Steam VR (OpenVR). This means any headset supporting Steam VR will work with this plugin, given the headset is directly connected to the PC. The most popular SteamVR headsets are Oculus, HTC Vive, and Windows Mixed Reality.

2. Google Cardboard (iOS and Android)
The Google Cardboard is a product and standard that turns your mobile device into a virtual reality headset. This means you can either purchase the cardboard version from Google or shop online and purchase a more robust plastic/foam padded version from amazon.

Download the Android App here:

User-inserted image

Allow Firewall Access
User-inserted image

When the plugin is loaded, you may receive a message to allow firewall access. Be sure to check Private and Public networks. If you do not check both, you may end up in a situation in the future where you're connected to a public network and the plugin isn't working. In this case, it's because ARC doesn't have firewall permission. Also, if you do not see this message when the plugin loads, it could be because you've already answered this in the past. In which case, you should visit the firewall rules to see if ARC has correct access.

1) Press the Windows button and type FIREWALL to search for Windows Defender Firewall
2) Select "Allow an app or feature through windows defender firewall."
User-inserted image

User-inserted image

4) Ensure both PRIVATE and PUBLIC are checked for ARC and press OK
User-inserted image

Attach/Detach Camera Button
User-inserted image

This plugin requires a camera, whether on your robot or a USB device. Add the camera device to your project and connect to the camera. Once you have a successful camera operational, press the Attach to Camera button on the plugin. This will connect the plugin to the camera device, sending the video stream to the VR headset.

Start/Close VR Client Button (SteamVR only)
User-inserted image

If you have a Steam VR-compatible headset connected to your PC, this button will allow you to use that headset. This button will start the VR client, connecting to the VR Headset. The client can be launched either full screen or in a window. By default, the setting is to launch in the plugin window. This setting can be altered for your project in the configuration dialog.

Google Cardboard Users
Your device needs to be on the same network as the ARC PC for those using google cardboard (iOS or Android) compatible device and headset. This is because your mobile device will connect to the ARC plugin by typing in the PC's IP address. The IP Address for your ARC PC is displayed in the plugin. Load the Virtual Reality Robot mobile app on your device and enter the IP address of the ARC PC.

User-inserted image

4K Display Issues
There is an issue with 4K displays and running the local client in the robot skill window. This is a known issue with Windows & Unity with 4k displays. If you have a 4k display and press the "Start Local Client" button, you will notice two things...

1) The client is minimal
2) The client is offset from the display window

There are two solutions...

1) Change your screen resolution to 1080

2) Set the client to run in full screen, configurable in this skill's configuration window.
User-inserted image

Virtual Reality Caution
Read the warnings on your device before using a virtual reality (VR) viewer with this plugin to reduce the risk of personal injury, discomfort, or property damage. Ensure you use this plugin with adequate space to avoid hurting yourself or others by walking into something. Nausea and motion sickness may be experienced and worse for some users. Do not use this plugin for extended periods to avoid experiencing nausea.:D


Upgrade to ARC Pro

ARC Early Access will give you immediate updates and new features needed to unleash your robot's potential!

Not yet - but it is on the list to be looked at one day:)

- Fix to selecting the roll servos in the config
This is by far my favorite skill. Appreciate the quick updates (especially the Roll servo config). 

I'm getting some new errors I didn't get before though in the stable version:

Disconnection on tty1
VR client connected on tty0 from [My IP ADDRESS HERE]
Streaming video to VR client...
Disconnection on tty0
Any ideas? The servos work fine for about 3-5 seconds, then stutter, freeze, and it disconnects from the VR server.
I'm working with the team on this robot skill. It appears to work correctly in our tests. Can you verify that the disconnection is not from a power brown-out due to the additional servo being moved?

Is the EZB also disconnecting when the VR client disconnects?
It could be a power brown out, but the X and Y axis were working consistently in v26. I'm also running up to 10a power brick (at 7.4v). They're certainly acting like a brown out though - stuttering, freezing and then disconnecting.

The EZB does not disconnect when the VR client disconnects. Seems to be something with client server and IP address. Maybe just on my side.

Is it possible to have version control access? IE Can I install the stable v26 again?
I should also highlight, I didn't even add an additional servo. I have it set to N/A; so there shouldn't be any additional workload (yet).
#89   — Edited
Hey - I just gave it a shot to see what changes were made. After reviewing the code, the only added thing was the Roll request. So I tried this robot skill on a robot with no trouble. Because your robot is disconnecting, I'm confident that the issue is with power and your servos. 

One way to test is to try the robot skill without an EZB connection. Do not connect to an EZB.

Run the robot skill, configure it for servos, and select the Activate Servos checkbox. You should see the values change.

User-inserted image

You can confidently determine its power brown-out if the robot skill shows those values moving without an EZB connection.
This was a great idea - just tried it. Involuntarily disconnected from server in 60 seconds or less (no servos involved). This is definitely a problem only I'm seeming to have, so maybe it's a settings issue internally.

So weird.

User-inserted image
Also just tried running exact same servo profiles with a joystick and they ran fine for 3+ minutes.
Okay, with no servos involved, that could mean something silly with the robot skill that wasn't caught before. I wonder if the Windows Defender is getting involved? Or the Windows Firewall?

When you check the Task Manager, do you see Anti Malware using a lot of CPU during the runtime?

I'll dive in deeper and see if I can reproduce it on my end.
I verified the robot's skill is working. I even made a few optimizations to it for performance.

However, I noticed an issue with the VR client app - that you might be experiencing. That's the unity program that connects to the headset. It was never caught before because the camera framerate wasn't as efficient. Over the last number of ARC releases, the video camera process has gone through several performance improvements. I believe one or all of those improvements has caused a blocking call between two threads in the VR client unity app.

I can resolve it and publish a new update to this robot skill tonight. Unity has changed a lot since this skill was created, so it needs several updates.
Any updates that address this issue would be sincerely appreciated!

- fix for unity app closing connection

- Few changes from DJ's update for increasing the versioning on the unity app and locking another object to prevent cross-thread errors
Thanks Synthiam team - this version is awesome! Super stable!
@hardy,  your boy is looking pretty good! Have you made any videos?