Use Meta Quest 2 VR headsets to control a real robot by viewing their cameras and manipulating servos with natural hand gestures or controllers. This skill brings a futuristic touch to remote robot operation, enhancing safety and efficiency in various work environments.
Hardware InfoHow to add the Oculus Quest Robot robot skill
- Load the most recent release of ARC (Get ARC).
- Press the Project tab from the top menu bar in ARC.
- Press Add Robot Skill from the button ribbon bar in ARC.
- Choose the Virtual Reality category tab.
- Press the Oculus Quest Robot icon to add the robot skill to your project.
Don't have a robot yet?
Follow the Getting Started Guide to build a robot and use the Oculus Quest Robot robot skill.
How to use the Oculus Quest Robot robot skill
Transform your robot interaction experience with the Oculus Quest robot skill for Synthiam ARC. This skill enables immersive control of robot cameras and servos using the Meta Quest 2 VR headset. This innovative skill offers a hands-on approach to robotics, complete with a remote desktop feature for seamless ARC control. It promises a future where remote work and safety in hazardous environments are enhanced through virtual reality.
Why Control Robots From VR?
While it is exciting to use this robot skill and have a robot mimic your movements with VR accessories, this is also a preview of future technology in the workplace. Robots can perform many jobs, but the computing power to automate the task is unavailable. Today, VR technology is advanced enough to allow humans to control robots remotely and accomplish tasks. This means, for example, that warehouse workers can work from the comfort and safety of their homes while improving outcomes by remotely controlling robots. As artificial intelligence improves, robots will require less human intervention, but humans will still be needed to help the robot when it gets stuck. This means one person can control many robots, complementing Synthiam's approach with our Exosphere product.Imagine people working from home while operating robots remotely in dangerous locations, such as mining, disaster recovery, or space exploration. With Synthiam's agnostic approach, our platform can program any robot. With the addition of virtual reality remote control, we are saving lives and improving job safety. There is no better time to familiarize yourself with this technology at home, school, or the workplace.
Downloading Quest Robot Controller App
Get the Oculus Quest Robot App: Oculus Quest Robot.apk (v0.10 June 25, 2022).
*Note: Due to Meta's review process complications, this app must be side-loaded while we work with Meta to have the app accepted for their store. Meta is currently disallowing this app because it connects to a PC. We're working to convince Meta that this useful VR app should be accepted for their review process so that students, robot builders, and inventors can use it. Stay tuned!
Meta Refuses To Accept Our App
We apologize for the method of installing the app rather than making it available in the Oculus app store. No matter how hard we plead, Meta will not accept our app in the store. The Meta Oculus Store denies our Oculus Quest Robot App because they do not like that it controls physical hardware. We continue to plead to their process, but they do not respond, and we continue to receive this same response.
Stereoscopic Camera View
This robot skill and Oculus Quest VR app optionally support two cameras (left and right eye), which provide the user with a 3D stereoscopic view. If two cameras are not required, a single camera can be used instead. The position of the two cameras is critical to prevent eye strain for the viewer. It is generally advised to measure the distance between your pupils. Use that measurement when distancing each camera from the center of each lens.Using Oculus Meta Quest App with Virtual Reality
When the Oculus Meta Quest robot control app is loaded, you will find yourself on the bridge of a spaceship. The left controller can move your view around the ship to explore. The right controller can interact with the menu connecting to the robot.
Enter the IP Address of the Synthiam ARC software. The robot skill will show public and private IP addresses you can enter in this field. If you use this robot skill on your local network, use the local IP address. If you connect over the internet through a port on your router, use the public IP address.
The port can be edited, but the default value is always 8082.
Pressing the connect button will load the robot control view.
About The Oculus Meta Quest 2
The Oculus Meta Quest 2 (marketed since November 2021 as Meta Quest 2) is a low-cost virtual reality (VR) headset developed by Facebook Reality Labs (formerly Oculus). It is the successor to the company's previous headset, the Oculus Quest. The Quest 2 was unveiled on September 16, 2020, during Facebook Connect 7.As with its predecessor, the Quest 2 can run as a standalone headset with an internal, Android-based operating system that does not require a PC or high-cost graphics card. It is a refresh of the original Oculus Quest with a similar design but lighter weight, updated internal specifications, a display with a higher refresh rate and per-eye resolution, and updated Oculus Touch controllers.
Main Window
Controller Inputs
Configuration
Press the configure button on the robot skill to view the configuration screen for this robot skill.
Headset
The servos for X (horizontal), Y (vertical), and Z (tilt) can be assigned. These servos will move with your headset to look around. Generally, you will want these servos attached to a camera so you can look around at the environment with the headset.
There is an option to pause servos when the robot moves and pause the headset servos while any Movement Panel is being moved. When checked, the headset servos are only usable when the robot is not moving. This is useful to reduce motion sickness while the robot is moving.
Left Hand & Right Hand
The position tracking group will bind servos to the hand's position. The X is horizontal, Y is vertical, and Z is the distance outward from your body.
The finger tracking group allows assigning servos to each finger. Gripping each finger into a fist will move the respective servos. This is useful when using a robot hand, such as the InMoov, to control each finger individually.
Left Controller & Right Controller
The position tracking will bind servos to the hand's position. The X is horizontal, Y is vertical, and Z is the distance outward from your body.
Triggers and analog sticks can also control servos. The triggers are the index trigger and the hand trigger. The analog stick for each controller can be assigned servos.
Scripts can be assigned to the A and B buttons when pressed or released.
General
1) The TCP ports for servo Server and Remote Desktop are read-only. They cannot be changed,
Enable/Disable the remote desktop feature. This feature has no password protection, so be cautious if you enable it. Do not enable this feature on public networks.
The number of frames to skip when transmitting servo positions. If too many servo positions are set, the EZB may brown out due to a lack of power supply.
Left Camera (or single camera if using only one camera)
Right camera (optional if using only the left/single camera)
Remote Desktop
The remote desktop lets you move the mouse with the right controller. You can use the trigger to left-click and the hand grab to right-click. The remote desktop feature does not provide authentication, so only use this on private networks.
Press the menu button on the left controller to exit the remote desktop mode.
The analog stick on the left controller can move around the floating screenshot window.
Robot Control
When viewing the robot control, the menu button on the left controller will load the menu to enable and disable specific servos.
In the popup menu, you can adjust the camera depth for comfort.
The return to lobby option will do just that: return you to the main menu lobby.
The remote desktop can remotely control ARC through the VR headset if the robot skill configuration is enabled.
Understood, I hope they change their strategy in the near future as this is a great field to explore and develop.
Hey I was just wondering if someone could help me with an issue that I'm running into. I'm trying to use this skill with the Quest 2 and I got it to connect but am running into some issues.
When I use the Connect Single option, I can see out of the Camera device but in the Main Window, I can't see any of the positional information updating.
When I use the Connect Dual option, I can't see anything out of the camera (just a gray screen) but the positional information on the Main Window updates, allowing me to control motors.
When I try to save the script on the button presses, they don't save properly. This one is more minor but thought it would be good to point out.
The main issue that I'm running into is that some of the features work separately but not together (controls as well as vision). Do you think you could help me troubleshoot the issues? Thanks
That's an interesting scenario. I must load and test it to see what could cause that - nothing comes to mind as a cause right now as it works for everyone else who uses it. I do know the position data isn't sent until the PAUSE checkbox is unchecked in the robot skill.
As for the grey screen with dual camera. Do you have both cameras selected in the robot skill configuration screen?
So I've got the dual camera servo to work (just had to add a second camera) but I'm having some issues with trying to control the fingers. Essentially, I'm running into an issue where when I close my entire fist the fingers aren't reading the maximum angle (ie a finger is fully retracted at 150 deg but the skill only reads like 70 deg). I found a setting to set the servo ratio to a different value but for some reason it's not saving. I want to save the ratio to something like 2 or 1.5 so that 70=140 but when I save it, it just reverts back to 1. Do you know if I'm doing anything wrong?
Servo ratio only works with slave servos: synthiam.com/Support/ARC-Overview/Servo-Controls
Increase your MAX or decrease your MIN servo position. That will move the servo further for the range of the finger. The oculus robot skill scales the MIn and MAX values. At least, they should be internally. I know this robot skill is getting an overhaul in the next 2 months, so there's probably several changes being done.
Is there support for wrist movement in this skill? I was able to control a hand with an Meta Quest 2, but I was just curious if this skill also allows for wrist movement or if there is another skill for that/I'll have to script my own.
There isn't but we can look into it in the future.
v11 has been updated with a fix for saving scripts attached to the controllers' A and B buttons. The servo ranges have also been changed to move the entire specified range.