Virtual Reality Robot

How to use the Virtual Reality Robot robot skill

Use Virtual Reality with your robot to see what the robot sees, and control servos by moving your head. This plugin supports all virtual reality headsets through Steam VR (OpenVR), Google Play, and iOS, including Google Cardboard, Oculus, Windows Mixed Reality, and HTC Vive. The plugin does two things; it allows you to see what the robot sees and control the servos with the Pitch and Yaw of the headset. This means when you move your head, the robot head can also move, mimicking your movement. 

This video was a test of the plugin on a Synthiam JD Humanoid. Sadly, he had no arms during the trial, so it's funny to see. Professor E from The Robot Program controlled the robot's head and viewed the robot's camera through the VR headset. In this example, we used OpenVR in STEAM with a Samsung Odyssey headset.




In this video below, we controlled a Synthiam Six Hexapod with a joystick and a VR headset. When we move the head from left to right, the legs of the hexapod will rotate the body so we can look around. This also applied to looking up and down; the legs would tilt the body so we could look up and down. In this example, we used an Android phone with Google Cardboard Glasses.




Here is another exciting example where we combined this VR Headset plugin with 2 Wiimotes to create custom animations. Each time a button was pressed on the WiiMote, a new FRAME was created with the current position of all servos and appended to an ACTION. DJ keeps pressing the button, and the FRAMES append to the ACTION until he stops. Once he squeezes the trigger on the WiiMote, the code instructs the ACTION to be played back. And thus, all FRAMES are played back because they are part of the action.




Here is a screenshot of the plugin running on an ARC workspace with a camera.




What Hardware Does It Support?
Two methods connect to the plugin, which opens a wide variety of hardware. 

1. SteamVR (OpenVR)
On PC, the supported headsets use Steam VR (OpenVR). This means any headset supporting Steam VR will work with this plugin, given the headset is directly connected to the PC. The most popular SteamVR headsets are Oculus, HTC Vive, and Windows Mixed Reality.

2. Google Cardboard (iOS and Android)
The Google Cardboard is a product and standard that turns your mobile device into a virtual reality headset. This means you can either purchase the cardboard version from Google or shop online and purchase a more robust plastic/foam padded version from amazon.

Download the Android App here: https://play.google.com/store/apps/details?id=com.ezrobot.virtualrealityrobot




Allow Firewall Access


When the plugin is loaded, you may receive a message to allow firewall access. Be sure to check Private and Public networks. If you do not check both, you may end up in a situation in the future where you're connected to a public network and the plugin isn't working. In this case, it's because ARC doesn't have firewall permission. Also, if you do not see this message when the plugin loads, it could be because you've already answered this in the past. In which case, you should visit the firewall rules to see if ARC has correct access.

1) Press the Windows button and type FIREWALL to search for Windows Defender Firewall
2) Select "Allow an app or feature through windows defender firewall."



3) Press CHANGE SETTINGS



4) Ensure both PRIVATE and PUBLIC are checked for ARC and press OK





Attach/Detach Camera Button



This plugin requires a camera, whether on your robot or a USB device. Add the camera device to your project and connect to the camera. Once you have a successful camera operational, press the Attach to Camera button on the plugin. This will connect the plugin to the camera device, sending the video stream to the VR headset.

Start/Close VR Client Button (SteamVR only)



If you have a Steam VR-compatible headset connected to your PC, this button will allow you to use that headset. This button will start the VR client, connecting to the VR Headset. The client can be launched either full screen or in a window. By default, the setting is to launch in the plugin window. This setting can be altered for your project in the configuration dialog.

Google Cardboard Users
Your device needs to be on the same network as the ARC PC for those using google cardboard (iOS or Android) compatible device and headset. This is because your mobile device will connect to the ARC plugin by typing in the PC's IP address. The IP Address for your ARC PC is displayed in the plugin. Load the Virtual Reality Robot mobile app on your device and enter the IP address of the ARC PC.






4K Display Issues
There is an issue with 4K displays and running the local client in the robot skill window. This is a known issue with Windows & Unity with 4k displays. If you have a 4k display and press the "Start Local Client" button, you will notice two things...

1) The client is minimal
2) The client is offset from the display window

There are two solutions...

1) Change your screen resolution to 1080

2) Set the client to run in full screen, configurable in this skill's configuration window.





Virtual Reality Caution
Read the warnings on your device before using a virtual reality (VR) viewer with this plugin to reduce the risk of personal injury, discomfort, or property damage. Ensure you use this plugin with adequate space to avoid hurting yourself or others by walking into something. Nausea and motion sickness may be experienced and worse for some users. Do not use this plugin for extended periods to avoid experiencing nausea.