View your robot’s camera using any SteamVR supported virtual reality headsets (Google Cardboard, Oculus Rift, Windows Mixed Reality, HTC Vive, etc)
Get
Version
30
How to add the Virtual Reality Robot robot skill
- Load the most recent release of ARC (Get ARC).
- Press the Project tab from the top menu bar in ARC.
- Press Add Robot Skill from the button ribbon bar in ARC.
- Choose the Virtual Reality category tab.
- Press the Virtual Reality Robot icon to add the robot skill to your project.
Don't have a robot yet?
Follow the Getting Started Guide to build a robot and use the Virtual Reality Robot robot skill.
How to use the Virtual Reality Robot robot skill
Use Virtual Reality with your robot to see what the robot sees, and control servos by moving your head. This plugin supports all virtual reality headsets through Steam VR (OpenVR), Google Play, and iOS, including Google Cardboard, Oculus, Windows Mixed Reality, and HTC Vive. The plugin does two things; it allows you to see what the robot sees and control the servos with the Pitch and Yaw of the headset. This means when you move your head, the robot head can also move, mimicking your movement.This video was a test of the plugin on a Synthiam JD Humanoid. Sadly, he had no arms during the trial, so it's funny to see. Professor E from The Robot Program controlled the robot's head and viewed the robot's camera through the VR headset. In this example, we used OpenVR in STEAM with a Samsung Odyssey headset.
In this video below, we controlled a Synthiam Six Hexapod with a joystick and a VR headset. When we move the head from left to right, the legs of the hexapod will rotate the body so we can look around. This also applied to looking up and down; the legs would tilt the body so we could look up and down. In this example, we used an Android phone with Google Cardboard Glasses.
Here is another exciting example where we combined this VR Headset plugin with 2 Wiimotes to create custom animations. Each time a button was pressed on the WiiMote, a new FRAME was created with the current position of all servos and appended to an ACTION. DJ keeps pressing the button, and the FRAMES append to the ACTION until he stops. Once he squeezes the trigger on the WiiMote, the code instructs the ACTION to be played back. And thus, all FRAMES are played back because they are part of the action.
Here is a screenshot of the plugin running on an ARC workspace with a camera.
What Hardware Does It Support?
Two methods connect to the plugin, which opens a wide variety of hardware.
1. SteamVR (OpenVR)
On PC, the supported headsets use Steam VR (OpenVR). This means any headset supporting Steam VR will work with this plugin, given the headset is directly connected to the PC. The most popular SteamVR headsets are Oculus, HTC Vive, and Windows Mixed Reality.
2. Google Cardboard (iOS and Android)
The Google Cardboard is a product and standard that turns your mobile device into a virtual reality headset. This means you can either purchase the cardboard version from Google or shop online and purchase a more robust plastic/foam padded version from amazon.
Download the Android App here: https://play.google.com/store/apps/details?id=com.ezrobot.virtualrealityrobot
Allow Firewall Access
When the plugin is loaded, you may receive a message to allow firewall access. Be sure to check Private and Public networks. If you do not check both, you may end up in a situation in the future where you're connected to a public network and the plugin isn't working. In this case, it's because ARC doesn't have firewall permission. Also, if you do not see this message when the plugin loads, it could be because you've already answered this in the past. In which case, you should visit the firewall rules to see if ARC has correct access.
1) Press the Windows button and type FIREWALL to search for Windows Defender Firewall
2) Select "Allow an app or feature through windows defender firewall."
3) Press CHANGE SETTINGS
4) Ensure both PRIVATE and PUBLIC are checked for ARC and press OK
Attach/Detach Camera Button
This plugin requires a camera, whether on your robot or a USB device. Add the camera device to your project and connect to the camera. Once you have a successful camera operational, press the Attach to Camera button on the plugin. This will connect the plugin to the camera device, sending the video stream to the VR headset.
Start/Close VR Client Button (SteamVR only)
If you have a Steam VR-compatible headset connected to your PC, this button will allow you to use that headset. This button will start the VR client, connecting to the VR Headset. The client can be launched either full screen or in a window. By default, the setting is to launch in the plugin window. This setting can be altered for your project in the configuration dialog.
Google Cardboard Users
Your device needs to be on the same network as the ARC PC for those using google cardboard (iOS or Android) compatible device and headset. This is because your mobile device will connect to the ARC plugin by typing in the PC's IP address. The IP Address for your ARC PC is displayed in the plugin. Load the Virtual Reality Robot mobile app on your device and enter the IP address of the ARC PC.
4K Display Issues
There is an issue with 4K displays and running the local client in the robot skill window. This is a known issue with Windows & Unity with 4k displays. If you have a 4k display and press the "Start Local Client" button, you will notice two things...
1) The client is minimal
2) The client is offset from the display window
There are two solutions...
1) Change your screen resolution to 1080
2) Set the client to run in full screen, configurable in this skill's configuration window.
Virtual Reality Caution
Read the warnings on your device before using a virtual reality (VR) viewer with this plugin to reduce the risk of personal injury, discomfort, or property damage. Ensure you use this plugin with adequate space to avoid hurting yourself or others by walking into something. Nausea and motion sickness may be experienced and worse for some users. Do not use this plugin for extended periods to avoid experiencing nausea.
Related Tutorials
tutorial
Control Robot With Virtual Reality Headset
Learn how to control your robot with a virtual reality headset in the synthiam software platform
tutorial
Create Auto Position Actions From EZ-Script
This tutorial uses the ControlCommand() EZ-Script command that instructs the Auto Position control to create a frame and...
Related Hack Events
Related Questions
question
How To Extract VR Goggle Data That Is Used To Move JD...
Hi Currently, I'm able to use an HTC Vive VR goggle to control and move JD humanoid's head. The tutorial was really...
question
Unable To Locate The Vr Client App.This Is Installed With...
when i click on start vr client then dialogue appears unable to locate the vr client app.this is installed with the...
question
Using Vive Controllers For Arms?
Hi, ive successfully managed to get Virtual Reality running with my HTC Vive and the JD Camera, very nice :) Is anyone...
Upgrade to ARC Pro
Your robot can be more than a simple automated machine with the power of ARC Pro!
Virtual Reality Robot Plugin
EDIT: Is it possible to use the virtual servos ie dynamixels with this plug in?
https://www.ez-robot.com/Community/Forum/Thread?threadId=11135
Here’s a link on how to select servos in ARC: http://www.ez-robot.com/Tutorials/Lesson/49?courseId=4
All controls that use servos in ARC share the same servo selection interface, this includes plugins.
Thing is, I get motion sickness with VR even with 50+ FPS and virtually no input latency. Given the <20 FPS of the robot camera and the inherent massive latency involved in moving a camera with servos, I hope this plug-in comes with sick bags :).
FWIW, Ive been using FPV (first person view) cameras and goggles on RC planes and drones for many years. My goggles have motion sensors and I can use them to control servos to move the camera, and using analog video transmission, there is zero latency in the video, but even so I learned a long time ago they only way this is bearable for longer than 2 minutes, is keeping my head absolutely motionless and controlling the camera servos with a stick.
I know, VR is all the rage, but in reality a trackir (or any of the DIY/opensource alternative headtrackers) would provide most of the benefits of being able to move a robot head/camera with head gestures, without the inherent downsides.
What GPU and headset are you using? That will make the difference.
In PC flight/race simulators we use trackir/opentrack for that, its typically 3 ir leds clipped to your headset, a webcam that tracks those LEDs and software that calculates your head motion:
https://www.youtube.com/watch?v=XHNwQY9iMrw
That motion is then increased through configurable curves, so a smal head movement, say 10 degree or turning your head towards the edge of your monitor, results in say 90 degree change in perspective in the game.
This is a very cheap ($30 or so) and popular way to get some "VR" effect, allowing you to look around in your virtual cockpit, and compared to real VR sets has added benefits like higher resolution monitors, you can see your hands, see your keyboard, controllers, beer, ..
what you dont get is the 3D effect and immersion. But you dont get that with a ezrobot cam anyway, even when using a VR set.
Opentrack is the most popular software solution for this. Its opensource and supports all kinds of in- and outputs, not just tracking leds. It will also work with things like Wii controllers, Arduino sensor boards, eye/face tracking etc. And could be useful to do more than only control a camera, it could be used to control a robot hand or whatever.
I fear it will be the exact same problem with robots. Possibly even worse because the video lag will be slightly worse (digital video feed that needs encoding/decoding versus analogy and thus zero latency video on my rc craft),
Using an opentrack or trackir, you dont have this problem, because you dont wear goggles that obscure everything except for the video feed, you can still see "the real world" move instantly along with your head, so your brain doesnt get confused.
Have fun!
That said I really don't want to pay for yet another VR headset. Has anyone tried this with Trinus or ivry etc with an iphone in a google cardboard thingy.
If you really wanted to, you should be able to connect the fatsharks headtracking output (which is just a PWM signal) to an arduino or a USB simulator cable/dongle, and use that as input to a virtual joystick .
If you wish to control the arms of your robot, I recommend wii controllers. They also magically work with ARC natively and don’t need any hacks. Simply follow the instructions in the wiimote tutorial to add the controllers to your pc via Bluetooth, configure the servos, and it works.
Here’s the link for wiimote: http://www.ez-robot.com/Tutorials/Help.aspx?id=154
iOS is still waiting review
In the meantime, if you want to connect to other controllers for your application, take a look at creating your own plugin with this tutorial: https://www.ez-robot.com/Tutorials/UserTutorials/146/1
And, here's instructions on how to configure servos for ALL ARC skill controls that accept servo commands. The UI is universal across the entire software application: https://www.ez-robot.com/Tutorials/Lesson/49
Once you familiarize yourself with how to add more than one servo to features, you'll unlock a whole new power. Best example is how the Six Hexapod can rotate and follow objects with the camera. Take a look at that project for examples of how the servos are configured.
I do have on my list to add onto the plugin for hand controllers - for the present i'm using wiimote controllers.
I am very excited in using this app. I have a Samsung Mixed Reality Headset and have loaded its software and the Mixed Reality Headset software.
Ellis
Yes you need steam for steamvr. But steamvr is the dependency. Install steam, then install steamvr inside of steam.
Because you have windows mixed reality, you also have to install the windows mixed reality in steam.
And when you load the plugin, the rest just happens automatically. You just need steamvr installed, that’s all
My question is, if you have windows mixed reality, what have you been using it for if you don’t have steamvr?!? That’s where ALL the vr titles worth mentioning exist.
I would like to define the left and right cameras seperatly so that I could have true stereo vision with depth perception by using 2 cameras set at eye distance apart. Probably way more complicated than I am thinking (like need to be able to adjust centerpoint, etc....) but would be cool.
Alan
I sometimes do not understand what is being said so I ask for some clarifications. The community is very helpfull and I thank the people on it and ARC for providing it. Thanks
1) download and install stream
2) inside of steam, search their store and install SteamVR
3) inside of steam, search their store and install Windows Mixed reality
4) install Virtuak reality robot plugin into ARC
I already have stream and mixed reality installed but I don't think I have SteamVR installed unless I have and don't know it.
Alan
I sure wouldn’t recommend a phone lol
Last issue... the plug settings all to default positions to 90 when you activate, this puts my dynamixel positions not facing forward. IE the dynamixel plug has all my settings on those servos 1-4096 positions. 90 should be 2048 on the dynamixels. Some how its not doing the math to convert 1-180 positions?
If i can get both the headset and the plug in to talk i can grab a video to show dynamixel issue.
Is there a longer description of setting up the plug in?
Have you read the comments in this plugin to learn from the effort i put into others for help? Do you have Windows Mixed Reality support added to STEAM VR?
1) Ensure a Camera Device is loaded and connected to a camera in your project
2) Press ATTACH TO CAMERA (any errors?)
3) Press Start VR Client (Steam VR should automatically load, does it?)
4) Press Robot View in the Steam VR Client window that loads
that's it
As for your servo settings - All servo controls are the same. Given the type of headset, the INVERT button may need to be used to invert the direction of the servos. If the servos work in any other control of ARC, they'll work in the plugin. Notice how they're all the same interface? Notice my comment on your other thread about asking about servos? They all use the exact same control which is the exact same code. It's all the same. Nothing is different - the entire servo interface in all of ARC is standardized to be identical.
Also, i notice you have the Config window open while trying to click on something in ARC? It doesn't work that way. You have to close teh config window because it's a dialog widow. ARC has always been like that, and so are all windows programs. When you open a dialog window in a windows porgram, you have to close it before using the real window.
I will take a video to show you that check box is not clicking for me...seems to hang the steam VR as i detach camera and close the VR Client Steam VR says unresponsive.....
Final item is still the dynamixels. Please watch this video. Also sorry for the lethargy ...days of prepping for CES...exhausted.
It’s because our brain processes information of the previous day while we sleep - it’s a process similar to defragging a hard drive. But, it’s also solving problems and sorting the day’s information into knowledge.
1) organize your TO DO list and prioritize items between NEEDS and WANTS
2) when you’re feeling defeated, move to another task and revisit it the next day after a full night of rest
The reality, I only had 12 hours to get this plugin to work with my robots. That's all the time available in my schedule. It was a last minute decision to try and demonstrate telepresence, something not planned for the event. If I had a week, to mess with it then maybe I could figure out the dynamixel issue. The reality is I just don't.
As of this morning we have 84 companies who have scheduled to see us. Hopefully none of them request a telepresence demonstration
I hope to revisit soon!
@RobotRad, the supported VR headset requirement is in the plugin description. Steam compatible headsets are supported - or google cardboard
You can find related tutorials to this plugin above under the description and manual, including a link to that tutorial
I might be tempted to buy an Oculus. Do you support two cameras with this plugin so you can get stereo 3D view ?
I tried it with 2 cameras last year and has terrible results. Aligning the share of each camera physically was challenging. I couldn’t get it to not make my eyes go buggy.
I can always add it as an option if anyone and a use case where they were interested in tackling that physical challenge
at the time, they distributed an sdk to access the video and sensor data, not requiring unity or any other AAA engine.
these days you can’t talk to the headset directly without A LOT of effort. Which is why we rely on existing 3D engines like unity.
I did find some information dating back to a 2016 unity version where someone wanted to generate an object in each eye. There’s a little hope it’s possible to do today if I continue researching a bit further. It’s such a rarity that there isn’t much reliable info to go on.
I did add your request to the dev
list. There’s just no eta on it until we have better ideas of a direction to take.
Also, thanks for your continued engagement and effort. It is a rarity that a company's CEO is so engaged and helpful with their community. You have my sincere appreciation and respect.
Adapting those code examples to jerry-rig into ARC will limit the flexibility. We have a belief that anything we create must be agnostic - and not locked to a specific use-case. This is why I’m using that codebase as an example to reverse engineer rather than duplicate.
I have found some interesting code examples that demonstratest the direction I’d need to take
Updated with Roll servo option
- Fix to selecting the roll servos in the config
I'm getting some new errors I didn't get before though in the stable version:
Disconnection on tty1
VR client connected on tty0 from [My IP ADDRESS HERE]
Streaming video to VR client...
Disconnection on tty0
-
Any ideas? The servos work fine for about 3-5 seconds, then stutter, freeze, and it disconnects from the VR server.
-
???
Is the EZB also disconnecting when the VR client disconnects?
The EZB does not disconnect when the VR client disconnects. Seems to be something with client server and IP address. Maybe just on my side.
Is it possible to have version control access? IE Can I install the stable v26 again?
One way to test is to try the robot skill without an EZB connection. Do not connect to an EZB.
Run the robot skill, configure it for servos, and select the Activate Servos checkbox. You should see the values change.
You can confidently determine its power brown-out if the robot skill shows those values moving without an EZB connection.
So weird.
When you check the Task Manager, do you see Anti Malware using a lot of CPU during the runtime?
I'll dive in deeper and see if I can reproduce it on my end.
However, I noticed an issue with the VR client app - that you might be experiencing. That's the unity program that connects to the headset. It was never caught before because the camera framerate wasn't as efficient. Over the last number of ARC releases, the video camera process has gone through several performance improvements. I believe one or all of those improvements has caused a blocking call between two threads in the VR client unity app.
I can resolve it and publish a new update to this robot skill tonight. Unity has changed a lot since this skill was created, so it needs several updates.
My boy cant wait to get his neck software fixed!
- fix for unity app closing connection
- Few changes from DJ's update for increasing the versioning on the unity app and locking another object to prevent cross-thread errors
Minor performance improvement
Loads faster