Thumbnail

Virtual Reality Robot

by Synthiam

View your robot’s camera using any SteamVR supported virtual reality headsets (Google Cardboard, Oculus Rift, Windows Mixed Reality, HTC Vive, etc)

How to add the Virtual Reality Robot robot skill

  1. Load the most recent release of ARC (Get ARC).
  2. Press the Project tab from the top menu bar in ARC.
  3. Press Add Robot Skill from the button ribbon bar in ARC.
  4. Choose the Virtual Reality category tab.
  5. Press the Virtual Reality Robot icon to add the robot skill to your project.

Don't have a robot yet?

Follow the Getting Started Guide to build a robot and use the Virtual Reality Robot robot skill.

How to use the Virtual Reality Robot robot skill

Use Virtual Reality with your robot to see what the robot sees, and control servos by moving your head. This plugin supports all virtual reality headsets through Steam VR (OpenVR), Google Play, and iOS, including Google Cardboard, Oculus, Windows Mixed Reality, and HTC Vive. The plugin does two things; it allows you to see what the robot sees and control the servos with the Pitch and Yaw of the headset. This means when you move your head, the robot head can also move, mimicking your movement.

This video was a test of the plugin on a Synthiam JD Humanoid. Sadly, he had no arms during the trial, so it's funny to see. Professor E from The Robot Program controlled the robot's head and viewed the robot's camera through the VR headset. In this example, we used OpenVR in STEAM with a Samsung Odyssey headset.

In this video below, we controlled a Synthiam Six Hexapod with a joystick and a VR headset. When we move the head from left to right, the legs of the hexapod will rotate the body so we can look around. This also applied to looking up and down; the legs would tilt the body so we could look up and down. In this example, we used an Android phone with Google Cardboard Glasses.

Here is another exciting example where we combined this VR Headset plugin with 2 Wiimotes to create custom animations. Each time a button was pressed on the WiiMote, a new FRAME was created with the current position of all servos and appended to an ACTION. DJ keeps pressing the button, and the FRAMES append to the ACTION until he stops. Once he squeezes the trigger on the WiiMote, the code instructs the ACTION to be played back. And thus, all FRAMES are played back because they are part of the action.

Here is a screenshot of the plugin running on an ARC workspace with a camera.

User-inserted image

What Hardware Does It Support? Two methods connect to the plugin, which opens a wide variety of hardware.

1. SteamVR (OpenVR) On PC, the supported headsets use Steam VR (OpenVR). This means any headset supporting Steam VR will work with this plugin, given the headset is directly connected to the PC. The most popular SteamVR headsets are Oculus, HTC Vive, and Windows Mixed Reality.

2. Google Cardboard (iOS and Android) The Google Cardboard is a product and standard that turns your mobile device into a virtual reality headset. This means you can either purchase the cardboard version from Google or shop online and purchase a more robust plastic/foam padded version from amazon.

Download the Android App here: https://play.google.com/store/apps/details?id=com.ezrobot.virtualrealityrobot

User-inserted image

Allow Firewall Access User-inserted image When the plugin is loaded, you may receive a message to allow firewall access. Be sure to check Private and Public networks. If you do not check both, you may end up in a situation in the future where you're connected to a public network and the plugin isn't working. In this case, it's because ARC doesn't have firewall permission. Also, if you do not see this message when the plugin loads, it could be because you've already answered this in the past. In which case, you should visit the firewall rules to see if ARC has correct access.

  1. Press the Windows button and type FIREWALL to search for Windows Defender Firewall

  2. Select "Allow an app or feature through windows defender firewall." User-inserted image

  3. Press CHANGE SETTINGS User-inserted image

  4. Ensure both PRIVATE and PUBLIC are checked for ARC and press OK User-inserted image

Attach/Detach Camera Button User-inserted image

This plugin requires a camera, whether on your robot or a USB device. Add the camera device to your project and connect to the camera. Once you have a successful camera operational, press the Attach to Camera button on the plugin. This will connect the plugin to the camera device, sending the video stream to the VR headset.

Start/Close VR Client Button (SteamVR only) User-inserted image

If you have a Steam VR-compatible headset connected to your PC, this button will allow you to use that headset. This button will start the VR client, connecting to the VR Headset. The client can be launched either full screen or in a window. By default, the setting is to launch in the plugin window. This setting can be altered for your project in the configuration dialog.

Google Cardboard Users Your device needs to be on the same network as the ARC PC for those using google cardboard (iOS or Android) compatible device and headset. This is because your mobile device will connect to the ARC plugin by typing in the PC's IP address. The IP Address for your ARC PC is displayed in the plugin. Load the Virtual Reality Robot mobile app on your device and enter the IP address of the ARC PC.

User-inserted image

4K Display Issues There is an issue with 4K displays and running the local client in the robot skill window. This is a known issue with Windows & Unity with 4k displays. If you have a 4k display and press the "Start Local Client" button, you will notice two things...

  1. The client is minimal
  2. The client is offset from the display window

There are two solutions...

  1. Change your screen resolution to 1080

  2. Set the client to run in full screen, configurable in this skill's configuration window. User-inserted image

Virtual Reality Caution Read the warnings on your device before using a virtual reality (VR) viewer with this plugin to reduce the risk of personal injury, discomfort, or property damage. Ensure you use this plugin with adequate space to avoid hurting yourself or others by walking into something. Nausea and motion sickness may be experienced and worse for some users. Do not use this plugin for extended periods to avoid experiencing nausea. :D


ARC Pro

Upgrade to ARC Pro

Stay on the cutting edge of robotics with ARC Pro, guaranteeing that your robot is always ahead of the game.

PRO
Synthiam
#2  

Glad you enjoyed it:D I'll add a few additional features, such as controllers support and such in future versions

PRO
Synthiam
#4  

You can use any servos with any plugin. All plugins are compatible with all features of ARC.

Here’s a link on how to select servos in ARC: https://synthiam.com/Tutorials/Lesson/49?courseId=4

All controls that use servos in ARC share the same servo selection interface, this includes plugins.

Belgium
#5  

Does it also support opentrack (/trackir)?

Thing is, I get motion sickness with VR even with 50+ FPS and virtually no input latency. Given the <20 FPS of the robot camera and the inherent massive latency involved in moving a camera with servos, I hope this plug-in comes with sick bags :).

FWIW, Ive been using FPV (first person view) cameras and goggles on RC planes and drones for many years. My goggles have motion sensors and I can use them to control servos to move the camera, and using analog video transmission, there is zero latency in the video, but even so I learned a long time ago they only way this is bearable for longer than 2 minutes, is keeping my head absolutely motionless and controlling the camera servos with a stick.

I know, VR is all the rage, but in reality a trackir (or any of the DIY/opensource alternative headtrackers) would provide most of the benefits of being able to move a robot head/camera with head gestures, without the inherent downsides.

PRO
Synthiam
#6  

All recent headsets (i.e. vive, oculus, windows mixed reality) use head tracking. This is controlled by SteamVR and has nothing to do with the plugin or a game. Games do not write code for the VR headset, it's done by Steam VR, which interfaces with the headset. The video feed framerate is what ever you're source is, USB camera is 30 fps, wifi ezrobot camera is 20fps and usb ezrobot camera is 30 fps. The FPS has no correlation with the VR fps. The VR FPS is dependent on your GPU hardware. On my 1070, I get 90 FPS with this plugin.

What GPU and headset are you using? That will make the difference.

Belgium
#7  

You misunderstand. I would be interested in controlling a robot head with my own head, but without putting on vr goggles.

In PC flight/race simulators we use trackir/opentrack for that, its typically 3 ir leds clipped to your headset, a webcam that tracks those LEDs and software that calculates your head motion: https://www.youtube.com/watch?v=XHNwQY9iMrw

That motion is then increased through configurable curves, so a smal head movement, say 10 degree or turning your head towards the edge of your monitor, results in say 90 degree change in perspective in the game.

This is a very cheap ($30 or so) and popular way to get some "VR" effect, allowing you to look around in your virtual cockpit, and compared to real VR sets has added benefits like higher resolution monitors, you can see your hands, see your keyboard, controllers, beer, ..

what you dont get is the 3D effect and immersion. But you dont get that with a ezrobot cam anyway, even when using a VR set.

Opentrack is the most popular software solution for this. Its opensource and supports all kinds of in- and outputs, not just tracking leds. It will also work with things like Wii controllers, Arduino sensor boards, eye/face tracking etc. And could be useful to do more than only control a camera, it could be used to control a robot hand or whatever.

Belgium
#8  

As for what I was saying earlier; I am using FPV goggles, not on PC, but for remote control aircraft and racing drones. And I have built RC aircraft where the FPV camera has pan/tilt servos that I can control by moving the goggles, ie my head, much like a VR set. But I chose not to, because unlike a VR game on PC, where moving my head pretty much instantly moves the rendered camera, servos take 100s of milliseconds to reach their new position and the lag between moving my head and the servos and thus the image in my goggles responding makes anyone sick in minutes.

I fear it will be the exact same problem with robots. Possibly even worse because the video lag will be slightly worse (digital video feed that needs encoding/decoding versus analogy and thus zero latency video on my rc craft),

Using an opentrack or trackir, you dont have this problem, because you dont wear goggles that obscure everything except for the video feed, you can still see "the real world" move instantly along with your head, so your brain doesnt get confused.

PRO
Synthiam
#9  

If you want to control a camera on servos (perhaps a gimbal) with a wii controller, simply add the wii skill control to ARC, here's the link: https://synthiam.com/Support?id=154

Have fun!

PRO
Canada
#10  

I have been using the old Vuzix plugin but only has 2DOF and I don't believe it will work with SteamVR. I have a fatshark Dominator but don't believe it works with steamvr either.

That said I really don't want to pay for yet another VR headset. Has anyone tried this with Trinus or ivry etc with an iphone in a google cardboard thingy.

Belgium
#11  

>a fatshark Dominator

If you really wanted to, you should be able to connect the fatsharks headtracking output (which is just a PWM signal) to an arduino or a USB simulator cable/dongle, and use that as input to a virtual joystick .

PRO
Synthiam
#12  

No need to use any third party stuff. There’s an iPhone and android client for this plugin waiting in the AppStore for approval. That takes about a week or so. You can use any google cardboard compatible device with your phone to control servos if your ezb and view the camera output, just like the video.

PRO
Canada
#13  

hey thanks Vertigo I need to buy the Head tracking module for the fatshark but now DJ says he has an iphone APP WOOT I will wait for that !

PRO
Synthiam
#14  

Yeah. The iPhone/android app is a client for this plugin. It uses the phones accelerometer and gyro for head tracking. It natively controls servos through ARC and doesn’t require any third party hacks - it just works.

If you wish to control the arms of your robot, I recommend wii controllers. They also magically work with ARC natively and don’t need any hacks. Simply follow the instructions in the wiimote tutorial to add the controllers to your pc via Bluetooth, configure the servos, and it works.

Here’s the link for wiimote: https://synthiam.com/Tutorials/Help.aspx?id=154

PRO
Synthiam
#15  

Here's the android app: https://play.google.com/store/apps/details?id=com.ezrobot.virtualrealityrobot

iOS is still waiting review

PRO
Canada
#16  

Thanks DJ looking forward to this app. Wii remote works well but would be good if we could get this working with VR controllers and inverse kinematics so robot arm moved with human arm and robot hand positioned with human hand location.

PRO
Synthiam
#17  

You can do the kinematics. Learn more about configuring multiple servos and adding ratios for joints, such as elbow and shoulder etc. It's easy to do with the servo control editor that's universal across ALL ARC skill controls. It doesn't make sense for me to create a plugin specific to my application, so instead there's features for you to customize it yourself. My effort is to make very universal controls that you can all expand and build with.

In the meantime, if you want to connect to other controllers for your application, take a look at creating your own plugin with this tutorial: https://synthiam.com/Community/Tutorials/146/1

And, here's instructions on how to configure servos for ALL ARC skill controls that accept servo commands. The UI is universal across the entire software application: https://synthiam.com/Community/Tutorials/49

Once you familiarize yourself with how to add more than one servo to features, you'll unlock a whole new power. Best example is how the Six Hexapod can rotate and follow objects with the camera. Take a look at that project for examples of how the servos are configured.

I do have on my list to add onto the plugin for hand controllers - for the present i'm using wiimote controllers.

#18  

Can someone tell me how Steam is used in using this VR App? Do I load the Steam app and select something?

I am very excited in using this app. I have a Samsung Mixed Reality Headset and have loaded its software and the Mixed Reality Headset software.

Ellis

PRO
Synthiam
#19  

You must be specific with names of things, specially in technology because things are similarly named but entirely different. Steam and SteamVR are different :)

Yes you need steam for steamvr. But steamvr is the dependency. Install steam, then install steamvr inside of steam.

Because you have windows mixed reality, you also have to install the windows mixed reality in steam.

And when you load the plugin, the rest just happens automatically. You just need steamvr installed, that’s all

My question is, if you have windows mixed reality, what have you been using it for if you don’t have steamvr?!? That’s where ALL the vr titles worth mentioning exist.:) without steamvr, having a virtual reality headset is a paperweight. You’ll thank me lol!

#20  

So. I haven't even tried the pmugin/android app yet and I already have a feature request. Likely to only be something a small number of users care about, so feel free to say no....

I would like to define the left and right cameras seperatly so that I could have true stereo vision with depth perception by using 2 cameras set at eye distance apart. Probably way more complicated than I am thinking (like need to be able to adjust centerpoint, etc....) but would be cool.

Alan

#21  

Thanks for the clarification. I already have installed steam to windows and windows to steam. As I understand all I need to do now is install tha ARC skill plugin.

I sometimes do not understand what is being said so I ask for some clarifications. The community is very helpfull and I thank the people on it and ARC for providing it. Thanks

PRO
Synthiam
#22  

You need steamvr that installs into steam. You also need to install windows mixed reality that installs in stream.

  1. download and install stream

  2. inside of steam, search their store and install SteamVR

  3. inside of steam, search their store and install Windows Mixed reality

  4. install Virtuak reality robot plugin into ARC

#23  

Thanks DJ, I will get right to it.

I already have stream and mixed reality installed but I don't think I have SteamVR installed unless I have and don't know it.

PRO
Synthiam
#24  

Look at your installed list...

User-inserted image

#25  

This is great news...I cannot wait to try this! :D

#26  

Thanks I will do this.

PRO
USA
#27  

Did IOS version of this get approved? I cannot find the app in the app store.

PRO
Synthiam
#28  

Unfortunately not yet:( - it's been rejected but we're working on the changes required.

PRO
USA
#29  

Will this set up work with a EZB USB/serial connected to the computer? Or Wi Fi mode only?

PRO
Synthiam
#30  

Everything works with either usb or WiFi. There’s no difference across the entire ARC software. I don’t understand the question maybe?

PRO
USA
#31  

Ok yeah I prolly need to explore the plug in a bit. Didn’t want to invest the dwindling time if i couldn’t use in the CES environment. Thank you.

#32  

Will, you do need WiFi between the phone and the PC even if the EZ-B's are connected by USB.  Probably not the best option for CES based on our previous discussion

Alan

PRO
Synthiam
#33  

I assume for ces he’d be using a steam vr headset because it’s wired. A phone would last only a few minutes with battery life. And it would just get so hot it would shut down in the headset anyway.

I sure wouldn’t recommend a phone lol

PRO
USA
#34   — Edited

Looks like the Steam headset it is! lol.....I found an Acer Windows Mixed Reality head set on amazon...This particular model uses the HDMI port to connect to the graphics card, is that an issue?

PRO
USA
#35   — Edited

I finally got Stream VR installed and Windows mixed reality and the VR Robot plugs installed. I cannot get get steam and the plug in to talk. Steam sees the headset and it says using ezrobot.exe, but no data is exchanged when i hit connect to client

Last issue... the plug settings all to default positions to 90 when you activate, this puts my dynamixel positions not facing forward. IE the dynamixel plug has all my settings on those servos 1-4096 positions. 90 should be 2048 on the dynamixels. Some how its not doing the math to convert 1-180 positions?

If i can get both the headset and the plug in to talk i can grab a video to show dynamixel issue.

Is there a longer description of setting up the plug in?

PRO
Synthiam
#36  

Does the headset work with steam vr? Do you have supported hardware? Did you verify your computers requirements?

PRO
USA
#37  

Yes, yes, yes. Green check marks for everything. Headset came with windows mixed reality and it works great as stand alone.

PRO
Synthiam
#38   — Edited

Does STEAM VR work? Load STEAM VR from STEAM. Use the STEAM VR instructions to learn how to use your headset with STEAM VR. You have to install Windows Mixed Reality support into STEAM. You have to get your headset working with STEAM VR.

Have you read the comments in this plugin to learn from the effort i put into others for help? Do you have Windows Mixed Reality support added to STEAM VR?

  1. Ensure a Camera Device is loaded and connected to a camera in your project

  2. Press ATTACH TO CAMERA (any errors?)

  3. Press Start VR Client (Steam VR should automatically load, does it?)

  4. Press Robot View in the Steam VR Client window that loads

that's it

As for your servo settings - All servo controls are the same. Given the type of headset, the INVERT button may need to be used to invert the direction of the servos. If the servos work in any other control of ARC, they'll work in the plugin. Notice how they're all the same interface? Notice my comment on your other thread about asking about servos? They all use the exact same control which is the exact same code. It's all the same. Nothing is different - the entire servo interface in all of ARC is standardized to be identical.

PRO
USA
#39  

OK Just rebooted the system, got the camera to transfer data to the head set. But I cant click activate servos, it wont let me check mark it. User-inserted image

PRO
Synthiam
#40   — Edited

What do you mean? You click on it and the checkbox doesn't change? Where are you clicking? On the words or on the checkbox?

Also, i notice you have the Config window open while trying to click on something in ARC? It doesn't work that way. You have to close teh config window because it's a dialog widow. ARC has always been like that, and so are all windows programs. When you open a dialog window in a windows porgram, you have to close it before using the real window.

PRO
USA
#41  

I open the config to take a picture of the correct set up of the dynamixels, not during testing.

I will take a video to show you that check box is not clicking for me...seems to hang the steam VR as i detach camera and close the VR Client Steam VR says unresponsive.....

PRO
USA
#42  

Well two outta three problems solved. As things do..they all of a sudden start working with out doing anything to them.

Final item is still the dynamixels. Please watch this video. Also sorry for the lethargy ...days of prepping for CES...exhausted.

PRO
Synthiam
#43   — Edited

Then set the servo ranges to match the head position. Move the positions until the vertical head angle is looking forward. You will have to experiment with the values, the software can't guess what they should be. Figure it out with experimentation...

PRO
USA
#44  

No worries. I'm gonna pass on this. Too many other things to do. Thanks for the help.

PRO
Synthiam
#45  

I watched a documentary years ago about how our brains solve problems. They had these people play an arcade game for skiing. The people would play all day and have to keep beating their high score. Then, they’d get a full nights rest. The next day, on their first attempt, they all beat their previous high score.

It’s because our brain processes information of the previous day while we sleep - it’s a process similar to defragging a hard drive. But, it’s also solving problems and sorting the day’s information into knowledge.

  1. organize your TO DO list and prioritize items between NEEDS and WANTS

  2. when you’re feeling defeated, move to another task and revisit it the next day after a full night of rest

PRO
USA
#46   — Edited

We have 48 hours until we step on a plane to Vegas. I've managed to wear all the hats (as usual) and produced all the flyers, business cards, banners in photoshop, dismantle the robots, design, build and install a bluetooth speaker systems, cast new faces in silicone and build shipping containers, created a pitch deck, made a press kit, etc. And alot to do in the remaining time. (No investor can deny my sweat equity.)

The reality, I only had 12 hours to get this plugin to work with my robots. That's all the time available in my schedule. It was a last minute decision to try and demonstrate telepresence, something not planned for the event. If I had a week, to mess with it then maybe I could figure out the dynamixel issue. The reality is I just don't.  As of this morning we have 84 companies who have scheduled to see us. Hopefully none of them request a telepresence demonstration:) Maybe we fake it til we make it :D

PRO
Synthiam
#47  

Will - have you spent any more time with this control? It's so much fun!

PRO
USA
#48  

Ha not yet, I have to pry the headset from my kids hands. They are on it everyday!

I hope to revisit soon!

PRO
Canada
#49  

Any news on the IOS port of the app? Sorry my entire family switched to apple years ago so we are locked into their eco system.  :-(  https://play.google.com/store/apps/details?id=com.ezrobot.virtualrealityrobot

PRO
Synthiam
#50  

Thanks for the reminder - i'll take a look and see where it's at

PRO
USA
#51  

It there a app I can use my iPhone to use the  EZ-Robot Virtual Reality Viewer?

PRO
USA
#52  

is there an app for am iPhone to use the  EZ-Robot Virtual Reality Viewer? I don't have a Android

PRO
USA
#53  

Is there an iPhone app for the Virtual Reality Viewer? I don't have an android phone like this: https://play.google.com/store/apps/details?id=com.ezrobot.virtualrealityrobot

#54  

This is what I am interested in doing this summer outside when my robot goes on patrol outside. See what the robot see in 360 or 180 degree VR 3d and take control of robot drive and arm movements. I just got the Oculus Go and it is a fantastic VR device to try this, will the plugin support all VR head sets?

PRO
USA
#55  

Oculus Go is great, but using my iPhone seems a little more in my range  :-)

PRO
Synthiam
#56   — Edited

@EZang60, the iOS iPhone app is not available at this time

@RobotRad, the supported VR headset requirement is in the plugin description. Steam compatible headsets are supported - or google cardboard

PRO
USA
#57  

Thanks for your response,

#58  

Hey thanks DJ I do have the steam client working with my GO ,will try out the ARC skill plugin as well. Glad to know you are working on this cutting edge technology way before anyone else, I bet you are already working on adding EZ chip to bionic arms and legs for disabled people ten years in the future!

#59   — Edited

Is their a way that I can move my robots head when using my htc vive as it is not working right now.  Never mind got it had to activate servos

PRO
Synthiam
#60   — Edited

Here's a tutorial that I highly recommend watching that walks you through the process: https://synthiam.com/Community/Tutorials/Control-robot-with-Virtual-Reality-headset-18905

You can find related tutorials to this plugin above under the description and manual, including a link to that tutorial

#62  

@DJ_Sures Can you make this plugin allow people to select what camera to tie too as I have two camera's and they are not named Camera so it does not tie to them.

PRO
Synthiam
#64  

Updated the VR projection in the client to be far more stable to reduce motion sickness. Also it has a bunch of new UI enhancements and bug fixes

PRO
Canada
#65   — Edited

I only have an iPhone and no Android or a device with StreamVR support, so I can't test, although I did like the old vuzix one you made years ago that also turns the camera servo's when you turn your head.

I might be tempted to buy an Oculus. Do you support two cameras with this plugin so you can get stereo 3D view ?

PRO
Synthiam
#66  

The iOS should be approved in a few weeks

I tried it with 2 cameras last year and has terrible results. Aligning the share of each camera physically was challenging. I couldn’t get it to not make my eyes go buggy.

I can always add it as an option if anyone and a use case where they were interested in tackling that physical challenge

Unknown Country
#67  

Can you please add a 2 camera option? It would really help with my project!

PRO
Synthiam
#68  

That’s a tough one - I’ve tried to consider how to do that a number of times. I’m not quite sure how. It’s mainly because unity is already rendering the scene as a vr 3D space. I can’t find a way to provide it to video streams that give it stereo view.

Unknown Country
#69  

Hmm, well I thought that because you said you could add it as an option for someone trying to tackle the physical challenge that you would be able to add it relatively easily. How were you able to produce the unenjoyable experience with 2 cameras if it only allows one camera? I am a bit confused. Thanks for your reply!

PRO
Synthiam
#70  

I originally thought it should be done. After looking into it further, I’m unable to figure out how it can be done with two cameras. The unity vr system generates the 3D space. I’m unable to place different objects in each eye of the vr glasses because unity is doing it within the engine. Maybe there’s a way if someone has an idea.

Unknown Country
#71  

Hmmm, yeah I'm not really sure. Do you think that a 2 camera method would even be possible on a one screen headset like the rift s?

Unknown Country
#72  

Also what did you do in order to use 2 cameras when the experience was unpleasant?

PRO
Synthiam
#73  

That was using directx and rendering two cameras in an oculus dk a number of years ago before steam vr. There was an oculus only plug-in that worked with the dk we had. It’s been deprecated when the oculus dk was discontinued.

at the time, they distributed an sdk to access the video and sensor data, not requiring unity or any other AAA engine.

these days you can’t talk to the headset directly without A LOT of effort. Which is why we rely on existing 3D engines like unity.

I did find some information dating back to a 2016 unity version where someone wanted to generate an object in each eye. There’s a little hope it’s possible to do today if I continue researching a bit further. It’s such a rarity that there isn’t much reliable info to go on.

I did add your request to the dev list. There’s just no eta on it until we have better ideas of a direction to take.

Unknown Country
#74  

Well thank you so much for doing that! Please let me know if you find anything that will help. I'll do some research too.

Unknown Country
#75  

I did some research and I found a page on github that used 2 cameras to create a 3D image on the Oculus CV1. I have linked it. https://github.com/feferna/dual_cameras_with_oculus_cv1

PRO
Synthiam
#76  

Good hope that it can be done - that's using the oculus sdk in unity so we'd need to look deeper into how it can be done with the unity vr engine.

Unknown Country
#77  

Currently, I'm not aware of the differences between using the oculus sdk in unity vs using the unity vr engine on oculus, so I will learn how they are different and then will do some more research.  The github page shows that it uses a lot of Classic ASP language in the code. Could that be a problem with getting it to run within ARC?

Also, thanks for your continued engagement and effort. It is a rarity that a company's CEO is so engaged and helpful with their community. You have my sincere appreciation and respect.

Unknown Country
#78  

I have done some research and I have a few questions. The VR client uses the Unity XR engine, right? I updated the build from github to the latest unity version and I was wondering if you could get the VR client to run that. Also the guthub page mentions the robot needs to be running linux ubuntu and ROS igloo to use GSCAM. Would that prevent ARC from working with the EZ v4 controller? Also, I'm not quite sure what operating system the EZ v4 controller runs, or if it even has an operating system. Could you explain?

PRO
Synthiam
#79  

thanks for the kind words.

Adapting those code examples to jerry-rig into ARC will limit the flexibility. We have a belief that anything we create must be agnostic - and not locked to a specific use-case. This is why I’m using that codebase as an example to reverse engineer rather than duplicate.

I have found some interesting code examples that demonstratest the direction I’d need to take

Unknown Country
#80  

Well it looks like I will leave it up to the experts! Thank you so much! Do you have any idea how long it may take for the skill or updated VR robot skill to be accessible? I am working on prototyping my project this month and it would be very helpful to have it availible sometime this month. Lastly, I found another project on github that you many find find useful or could provide some inspiration. I have linked it. https://github.com/jleppert/3dfpv

PRO
Synthiam
#82  

Not yet - but it is on the list to be looked at one day :)

PRO
Synthiam
#84  

v27

  • Fix to selecting the roll servos in the config
#85  

This is by far my favorite skill. Appreciate the quick updates (especially the Roll servo config).

I'm getting some new errors I didn't get before though in the stable version:

Disconnection on tty1 VR client connected on tty0 from [My IP ADDRESS HERE] Streaming video to VR client... Disconnection on tty0

Any ideas? The servos work fine for about 3-5 seconds, then stutter, freeze, and it disconnects from the VR server.

???

#86  

I'm working with the team on this robot skill. It appears to work correctly in our tests. Can you verify that the disconnection is not from a power brown-out due to the additional servo being moved?

Is the EZB also disconnecting when the VR client disconnects?

#87  

It could be a power brown out, but the X and Y axis were working consistently in v26. I'm also running up to 10a power brick (at 7.4v). They're certainly acting like a brown out though - stuttering, freezing and then disconnecting.

The EZB does not disconnect when the VR client disconnects. Seems to be something with client server and IP address. Maybe just on my side.

Is it possible to have version control access? IE Can I install the stable v26 again?

#88  

I should also highlight, I didn't even add an additional servo. I have it set to N/A; so there shouldn't be any additional workload (yet).

PRO
Synthiam
#89   — Edited

Hey - I just gave it a shot to see what changes were made. After reviewing the code, the only added thing was the Roll request. So I tried this robot skill on a robot with no trouble. Because your robot is disconnecting, I'm confident that the issue is with power and your servos.

One way to test is to try the robot skill without an EZB connection. Do not connect to an EZB.

Run the robot skill, configure it for servos, and select the Activate Servos checkbox. You should see the values change.

User-inserted image

You can confidently determine its power brown-out if the robot skill shows those values moving without an EZB connection.

#90  

This was a great idea - just tried it. Involuntarily disconnected from server in 60 seconds or less (no servos involved). This is definitely a problem only I'm seeming to have, so maybe it's a settings issue internally.

So weird.

User-inserted image

#91  

Also just tried running exact same servo profiles with a joystick and they ran fine for 3+ minutes.

PRO
Synthiam
#92  

Okay, with no servos involved, that could mean something silly with the robot skill that wasn't caught before. I wonder if the Windows Defender is getting involved? Or the Windows Firewall?

When you check the Task Manager, do you see Anti Malware using a lot of CPU during the runtime?

I'll dive in deeper and see if I can reproduce it on my end.

PRO
Synthiam
#93  

I verified the robot's skill is working. I even made a few optimizations to it for performance.

However, I noticed an issue with the VR client app - that you might be experiencing. That's the unity program that connects to the headset. It was never caught before because the camera framerate wasn't as efficient. Over the last number of ARC releases, the video camera process has gone through several performance improvements. I believe one or all of those improvements has caused a blocking call between two threads in the VR client unity app.

I can resolve it and publish a new update to this robot skill tonight. Unity has changed a lot since this skill was created, so it needs several updates.

#94  

Any updates that address this issue would be sincerely appreciated!

PRO
Synthiam
#96  

v28

  • fix for unity app closing connection
#97  

v29

  • Few changes from DJ's update for increasing the versioning on the unity app and locking another object to prevent cross-thread errors
#98  

Thanks Synthiam team - this version is awesome! Super stable!

Canada
#99  

@hardy,  your boy is looking pretty good! Have you made any videos?

#100  

https://youtube.com/shorts/GD-VjxTWpEU?feature=share