ARC Pro

Upgrade to ARC Pro

Your robot can be more than a simple automated machine with the power of ARC Pro!

Thumbnail

Oculus Quest Robot

by Meta

Control a robot with a Meta Quest 2 VR Headset. View the robot’s camera and control servos with your hands or Quest controllers.

How to add the Oculus Quest Robot robot skill

  1. Load the most recent release of ARC (Get ARC).
  2. Press the Project tab from the top menu bar in ARC.
  3. Press Add Robot Skill from the button ribbon bar in ARC.
  4. Choose the Virtual Reality category tab.
  5. Press the Oculus Quest Robot icon to add the robot skill to your project.

Don't have a robot yet?

Follow the Getting Started Guide to build a robot and use the Oculus Quest Robot robot skill.

How to use the Oculus Quest Robot robot skill

User-inserted image

Use the Oculus Meta Quest 2 VR Headset to view any robot's camera and control any robot’s servos with your hands or controllers. Servos can be attached to finger movements, hand movements, controllers, and buttons. Additionally, there is a built-in remote desktop to control ARC without having to remove the VR headset. This robot skill requires the Meta Quest 2 app installed from the Oculus Store to control a robot. 

Why Control Robots From VR?


While it is exciting to use this robot skill and have a robot mimic your movements with VR accessories, this is also a preview of future technology in the workplace. Robots can perform many jobs, but the computing power to automate the task is unavailable. Today, VR technology is advanced enough to allow humans to control robots remotely and accomplish tasks. This means, for example, that warehouse workers can work from the comfort and safety of their homes while performing improved outcomes by controlling robots remotely. As artificial intelligence improves, robots will require less human intervention, but humans will still be needed to help the robot when it gets stuck. This means many robots can be controlled by one person, complementing Synthiam's approach with our Exosphere product.

Imagine people working from home while operating robots remotely in dangerous locations, such as mining, disaster recovery, or space exploration. With Synthiam's agnostic approach, our platform can program any robot. With the addition of virtual reality remote control, we are saving lives and improving job safety. There is no better time to familiarize yourself with this technology at home, school, or the workplace. 


Oculus Meta Quest App


User-inserted image


Get the Oculus Quest Robot App here: https://www.oculus.com/experiences/quest/5097609970275847

When the oculus meta quest robot control app is loaded, you will find yourself on the bridge of a spaceship. The left controller can move your view around the ship to explore. The right controller can interact with the menu that will connect to the robot. 

1) Enter the IP Address of the Synthiam ARC software. The robot skill will show public and private IP addresses you can enter in this field. Using this robot skill on your local network, use the local IP address. If connecting over the internet through a port opened on your router, use the public IP address.

2) The port can be edited, but the default value is always 8082.

3) Pressing the connect button will load the robot control view.


About The Oculus Meta Quest 2


The Oculus Meta Quest 2 (marketed since November 2021 as Meta Quest 2) is a low-cost virtual reality (VR) headset developed by Facebook Reality Labs (formerly Oculus). It is the successor to the company's previous headset, the Oculus Quest. The Quest 2 was unveiled on September 16, 2020, during Facebook Connect 7.

As with its predecessor, the Quest 2 can run as a standalone headset with an internal, Android-based operating system not requiring a PC or high-cost graphics card. It is a refresh of the original Oculus Quest with a similar design but lighter weight, updated internal specifications, a display with a higher refresh rate and per-eye resolution, and updated Oculus Touch controllers.


Main Window


User-inserted image

The main window displays real-time tracking information of Quest input devices. Either the controllers or hand detection can be used. If the hand tracking is being used, each finger position starting with the thumb, will be displayed and the hand's position. The controller location, analog stick, buttons, and trigger positions will be displayed if controllers are used.


Controller Inputs


User-inserted image

Both hand tracking and controller tracking can control their respective servos. When the controllers are set aside, and hand tracking is detected, the configured hand servos will enable. The hand servos will be disabled when used, and the controller servos will become enabled. This switches automatically based on what control method you use. Switching control methods can be done on the fly.


Configuration


Press the configure button on the robot skill to view the configuration screen for this robot skill.
User-inserted image



Headset

User-inserted image


1) The servos for X (horizontal), Y (vertical), and Z (tilt) can be assigned. These servos will move with your headset to look around. Generally, you will want these servos attached to a camera so you can look around at the environment with the headset. 

2) The option to pause servos when the robot is moving will pause the headset servos while any Movement Panel is being moved. When this is checked, the headset servos are only usable when the robot is not moving. This is useful to reduce motion sickness while the robot is moving.


Left Hand & Right Hand
User-inserted image

1) The position tracking group will bind servos to the hand's position. The X is horizontal, Y is vertical, and Z is the distance outward from your body.

2) The finger tracking group allows assigning servos to each finger. Gripping each finger into a fist will move the respective servos. This is useful when using a robot hand, such as the InMoov, to control each finger individually.

Left Controller & Right Controller
User-inserted image

1) The position tracking will bind servos to the hand's position. The X is horizontal, Y is vertical, and Z is the distance outward from your body.

2) Triggers and analog sticks can also control servos. The triggers are index trigger and hand trigger. The analog stick for each controller can be assigned servos.

3) Scripts can be assigned to the A and B buttons when pressed or released.


General
User-inserted image

1) The TCP ports for servo Server and Remote Desktop are read-only. They cannot be changed,

2) Enable/Disable the remote desktop feature. There is no password protection on this feature, so be cautious if you enable it. Do not enable this feature on public networks. 


Remote Desktop


User-inserted image

1) The remote desktop allows you to move the mouse with the right controller. You can use the trigger to left-click and the hand-grab to right-click. The remote desktop feature does not provide any authentication, so only use this on private networks.

2) Press the menu button on the left controller to exit the remote desktop mode.

3) The analog stick on the left controller can be used to move around the floating screenshot window.


Robot Control


User-inserted image

1) When viewing the robot control, pressing the menu button on the left controller will load the menu to enable and disable specific servos.

2) In the popup menu, you can adjust the camera depth for your comfort.

3) The return to lobby option will do just that, return you to the main menu lobby.

4) If enabled in the robot skill configuration, the remote desktop can be used to control ARC remotely through the VR headset.

ARC Pro

Upgrade to ARC Pro

Unleash your creativity with the power of easy robot programming using Synthiam ARC Pro

PRO
Canada
#16  
Right on! I'll try to sideload the app tonight. I believe I already activated developer mode so it shouldn't be too difficult to install.

Do you guys have a tutorial that you use to sideload quest apps? Figured I'd ask before I go looking.
PRO
Synthiam
#17  
I just press the upload button in the oculus developer hub.
#18  
To expand on DJ's response, the developer hub can be downloaded from here: https://developer.oculus.com/downloads/package/oculus-developer-hub-win/

Synthiam does not support adding apps to the Quest using this method, so there will not be a formal tutorial. The wait for Meta to review and accept the app will allow the app to be added to your quest from the store.

In the meantime, if you use the developer hub software, the apk can be installed by selecting your Quest from the devices menu. There will be an upload button that will allow selecting an APK to be added. I hope that helps!
#19  
Robot skill updated to v4

- Fix for hand servo configuration not saving

- Added remote desktop server (configurable in General tab)
#20  
Robot skill updated to v6
- enhanced remote desktop features

Meta Quest APK updated to v0.8 (Oculus Quest Robot.apk)
- support for remote desktop
- improved robot control resolution
- improved depth control menu
PRO
Canada
#21  
I'm excited, I got the Synthiam app sideloaded onto my quest last night! I used the SideQuest app on my laptop and loaded the APK into the headset via a USB-C cable. I found the app in the "Unknown Sources" section of the app menu on the Quest.

I used this skill and the camera skill in ARC to check out the camera view. It's so weird to be looking at yourself when you use a laptop camera!!xD A true out-of-body experience!

Is anyone else trying this out? I'd love to compare notes! 

I'm wondering which robot would be the quickest to make/buy that has fingers? Is it still the inMoov?
PRO
Canada
#22  
Cool I still haven't purchased a quest yet was holding off until someone other than dev had it working. Did you try with JD and his grippers?
PRO
Canada
#23  
Right on @Nink, No I haven't tried that yet but I guess I could. I was just thinking that full finger movement would definitely be next level, but I guess I could start off small:D
PRO
Synthiam
#24   — Edited
I used JD with the grippers. I assigned my index finger to the gripper, and it worked well. 

I'd like to 3d print only an inmoov arm and attach it to a mount or something. I don't need the full inmoov but just an arm. What would be really wild is an inmoov arm attached to a base that you can drive around. Then you could remotely operate a hand from anywhere in the world with it.
#25  

Quote:

What would be really wild is an inmoov arm attached to a base that you can drive around.
Creepy. LOL;) I have nightmares about stuff like that. Sounds like a bad "B" horror movie.:D
PRO
Canada
#26  
Picked up the oculus then sideloaded the ARC APK. Connected JD configured head track and that seemed to work but when I went to do controller track and then connect it kept locking up.  I just did a couple of servos on one arm.  I set min max on the servos to conservative movement. 

If anyone has a working profile for a JD with Oculus would be good since this is a common robot for this community we can all use as a great starting point.
PRO
Synthiam
#27  
What do you mean it locked up?
PRO
Canada
#28  
Screen just froze and camera disconnected, head track stopped working JD stopped responding. I went back to lobby and then disconnected EZB in ARK and reconnected.  Arm control still didn’t work. I disabled servos and repeated disconnected reconnect and head track worked.  I can try another JD etc and try and play again this weekend if the honey do list isn’t to long.
PRO
Canada
#29  
I had one lock up once as well. The meta app was still functional except the Remote Desktop and camera screens were white. The ARC software froze and then when I clicked on the screen an error briefly showed up and ARC closed.
PRO
Synthiam
#30  
Hmmmm I wonder if the commands are being sent too fast and flooding the channel. I’ll take a look this weekend.
PRO
Canada
#31  
ARC never died on me. I just lost coms. I am using an overclocked gaming PC hardwired to an 802.11ac WIFI router in the same room so hopefully no bottle necks.  My hunch is I need to do a starter pose that the robot and I stand in so our joints and servo's are at least close to aligned and not have everything try to move at once.
PRO
Synthiam
#32  
You’re thinking the jd ezb browned out?
#33   — Edited
updated to v8

We have updated this robot skill with an option to skip servo commands being sent to the EZB. This is a setting that you can configure. If you are experiencing lockups with the EZB or disconnects, increase this value to skip sending every received frame command. 

User-inserted image
PRO
Canada
#34  
Cool, gonna try this when I get home tonight! I'll try it with JD.
PRO
Canada
#35  
Unfortunately, I've been too busy with hardware development to test this out. BUT, I did stumble across a open-source 3D printable hand design by Open Bionics that seems promising. Although the servos are like $70 a piece :(. The hand does seem more solid than the inMoov hand.

If you're curious, check it out here: https://www.thingiverse.com/thing:3000641