Thumbnail

Intel Realsense D435i

Connect the D435i Intel Realsense depth cameras to the ARC navigation messaging system (NMS) for distance detection and mapping.

+ How To Add This Control To Your Project (Click to Expand)
  1. Make sure you have the latest version of ARC installed.
  2. Select the Get button in this page to download the archive file.
  3. Double click the downloaded archive file to execute installer.
  4. The installer will add this control to ARC.
  5. Load ARC and press the Project -> Add Control button from the menu.
  6. Choose the Navigation category tab.
  7. Press the Intel Realsense D435i icon to add the control to your project.

Manual

Connect the D435i Intel Realsense camera to the ARC Navigation Messaging System (NMS) for distance detection and mapping. This should also work with other Intel RealSense depth camera models because the FOV and parameters are used from the device.

User-inserted image



Main Screen


User-inserted image

1) Select the device by the serial number. This allows multiple Intel Realsense devices to be added to a project.

2) START/STOP button to connect that connects to the RealSense device.

3) Log window for status and errors.


Image Align


The Image Align tab allows selecting the area of the image that you wish to detect for the point cloud. For example, having the detection too low will result in detecting the floor at a close distance. Ideally, set the bar high enough to avoid detecting the floor.
User-inserted image

1) Adjust the top of the detected image range.

2) Adjust the bottom of the detected image range.

3) detected image range is highlighted in purple.


Configuration


User-inserted image



1) Offset of the T265 from the center of the robot. The positive number is toward the rear, a negative number is toward the front from the center of the robot. The sensor must sit aligned with the center of the robot, not toward the left or right. The sensor must face 0 degrees heading.

2) Sensor resolution during initialization. If the values are to be changed from the default, obtain them from the Intel RealSense viewer. Changing these values may be necessary if using a different sensor than the D435i. Set both width & height to 0 for auto-size, which isn't the most efficient for performance.

3) The number of depth pixels to skip when processing the depth data. This is useful for lower CPU PCs or when high-resolution depth is not required. Since there is a limited field of view (FOV) on the sensor, it may not be necessary to process every depth pixel. If the sensor resolution is 1280x720, the CPU would be processing 921,600 depth points for every scan of the specified framerate. At 1280x720 @ 15 FPS, that's 13,824,000 depth points per second. If the Skip Scan Points for X & Y are set to 10 for that resolution/fps, the CPU would only need to process 9,216 depth points per scan (or 138,240 per second).

The editor will recommend a value for the best balance between performance and detection based on the sensor FOV.

4) The camera video from the RealSense can be pushed into a selected Camera Device. In the selected camera device, choose CUSTOM as the device type. This will accept the video feed from this robot skill. Ensure START is pressed on the selected camera device as well.


Variables


This skill will create 4 global variables that can be referenced by scripts.
User-inserted image


$D435IsRunning - status of the D435 robot skill if it is connected or not to the sensor.

$D435FurthestDistanceCM - furthest distance detected

$D435FurthestDistanceDegree - furthest degree of the furthest distance CM variable.

$D435NearestDistanceCM - nearest distance detected

$D435NearestDistanceDegree - nearest degree of the nearest distance CM variable.


ARC Navigation Messaging System


This skill is part of the ARC navigation messaging system. It is encouraged to read more about the messaging system to understand available skills HERE. This skill is in level #3 group #1 of in the diagram below. This skill contributes telemetry positioning to the cartesian positioning channel of the NMS. Combining this skill with Level #3 Group #2 skills for obstacle avoidance. And for Level #1, The Navigator works well.
User-inserted image

PRO
Synthiam
#1   — Edited
Fixed Intel's bug so the device can be started and stopped

There is a small performance improvement as well, which will be noticeable on SBCs

Also, this only displays depth sensors in the drop-down
PRO
Synthiam
#2   — Edited
Pretty big bug fix that correctly calculates the point-cloud distances into distance scans. You may have noticed that flag objects, such as walls were rounded. This is no longer the case.

Here you can see the wall, corner of the wall, and the edge of the monitor. 
User-inserted image


And here's a low scan of just the table in front - where it's flat and no longer curved.
User-inserted image
PRO
Synthiam
#3   — Edited
Updated with a huge performance improvement. I didn't realize the Intel sample code was copying the depth data into a managed array on every frame *ouch*. So I'm referencing the unmanaged memory pointer directly. Should have known better than to copy that bit of code from their sample:D

Should experience a 10 times improvement. I went from 27% cpu to 2% cpu

User-inserted image
PRO
Synthiam
#4  
Like The Navigator, there are variables for the nearest and furthest object detected... just in case this is being used without an NMS skill like the navigator

User-inserted image
PRO
Synthiam
#21  
Ya 640x480 is 4:3 which is a square. The 435i uses widescreen 16:9 or something close. So it’s probably an unsupported resolution.
#22  
I didn't get a chance to test this weekend, but will after work today.  Thanks.  the resolution thing makes sense.  I was planning on using the RGB camera as the only camera on the bot, so set it to a fairly low res so I wouldn't have an issue with object recognition, but I can stick an EZ-B camera on for that if 16:9 uses too much CPU for object recognition and tracking.

Alan
PRO
Synthiam
#23  
Updated to allow specifying resolution. Read the manual above for instructions

User-inserted image
#24  
I got it to work only at the very highest resolution which only supports 6 frames per second, but it was taking 20 seconds or more to detect a distance change in the skill (again, it worked fine in the Intel viewer), so the robot was running into walls.  Most of the resolutions gave the same error as before, without your added text about checking the USB port and supported resolutions.  640x480 gave an error pointing to a particular line of code, but I ran out of troubleshooting time, so I didn't catch it for you.  Will try again tonight.

I'll gather more detailed info for you tonight.  If you want me to try a debug build with extra logging, I am always happy to do that.
PRO
Synthiam
#25  
when you "test" in the realsense app, are the RGB camera and Sensors both at the same resolution?
#26  
I have set them the same and different resolutions, and it works regardless.

Alan
PRO
Synthiam
#27   — Edited
Your tests in the realsense app would need to be the same resolution for both the sensor and rgb. Otherwise, there image and distance detected would be out of sync for mapping in ARC.

What is the resolution and format for both rgb and sensor in realsense app that provide the best cpu usage?
PRO
Synthiam
#28   — Edited
Also, try setting the width & height in the latest version to 0. That will use the auto-size, or the last size used (i.e. from intel realsense viewer)
#29  
Will try after work tonight and let you know.

I am sure we'll get this figured out.

Should I run the calibration app that came with the driver download too?   I didn't both since you didn't mention it in the documentation.  Also, there are several profiles in the viewer app for what you are expecting to see (ie, hand signals, far distances close, etc...).  Is there a preferred one and does the device remember it after it is used in the app to when the skill accesses it?

Alan
PRO
Synthiam
#30   — Edited
I don't know anything about a calibration app. That's most likely a question for the intel company that makes the product. You're somewhat exceeding my knowledge of the device haha:)
#31  
I have tried everything I could think of, I removed the Intel drivers I had installed and just tried whatever Windows installed.  Still worked in the viewer, but the same issues in ARC.  I tried multiple resolutions.  The only 2 that the RGB and the Stereo have in common are 640x 480 which gives this errors (same as 0x0 no matter what the resolution in the viewer was:

User-inserted image


or 1280 x 720 which works but only when set to 6 frames per second and actually only updates the distance about every 20 seconds so the robot drives into walls.

All other resolutions give some variation of:

User-inserted image



Here are screen shots of the supported resolutions:

User-inserted image

 and

User-inserted image



I also tried removing the Intel driver and letting Windows find its own driver in case it was a driver issue, with the same results.   If this is not what you see in the viewer, I will return the device and get a 435i.  Otherwise, I am going to go back to Lidar for a while so I can work on navigation scripts that work until we have a 435 skill that actually works.  (that sounded more pissy than I meant.  My mom is in the hospital with Pneumonia,  Thank G-D it is not Covid, but I am stressed).  

As always, if you want to provide a debug version with high or highest logging level set to figure out what is causing the issue, I am glad to install and test, although depending on my Mom's condition and if they send her home, I may be offline for a week or so...  They won't let me visit the hospital despite the fact I am fully vaccinated, so at least for the next few days I should be available to test.
#32  
I have a question, about the Intel Device. Can it work for outdoors as well? would beinteresting to have an indoor/outdoor sensoe on the robot.
PRO
Synthiam
#33  
That’s super weird - wonder what they’re doing differently. I’m using their examples in the sdk. 

did you set width and height to 0? And same error?

I know you keep asking about a debug version - but you have it already :). the error message that you see is the most verbose there is. 

This is why I dislike working with any intel products... discontinued products with discontinued support and incomplete sdk.
PRO
Synthiam
#34  
One more thing ....

I noticed your screenshots do not have the rgb sensor or depth sensor active. Can your future screenshots include real-world test scenario. Sometimes things can be missed if liberties are taken. 

I’d like to know the these things when you get an ideal resolution working with the realsense viewer. Make sure the rgb and depth resolutions & framerate are identical and both sensors are active. Once that’s done, tell me these things...

- the common resolution 
- the common frame rate 
- the depth format (ie z16)
- the rgb format (ie rgb24)
#35  
I'll test again tonight.  Yes, I tried 0x0.  It gave the same error as the 640x480 test (first screen shot in my last message)

There are only 2 common resolutions available. 640x480 and 1280x720.  1280x720 is the only one that kind of works in ARC, but not in a usable way.
PRO
Synthiam
#36  
Understood. Let me know the frame rate and formats for 640x480
PRO
Synthiam
#37   — Edited
In the latest version, you may be able to use 1280x720 and specify higher skip depth points. try 10 or even 20 and see what your performance is like (or use the recommended performance values). This isn't a long-term solution, but necessary for even slow PCs with low depth resolutions.

*Note: Read about this setting in the manual at the top of this page.

User-inserted image
#38  
Thanks for the update DJ.  Have a family emergency going on.  I couldn't test last night.  Might be able to this weekend, but then I will probably be unable to do anything for a week or so.

Alan
PRO
Synthiam
#39  
Sorry to hear that Alan - take care of your fam:)
#40  
Thanks DJ.  Family first always. I'll share more (in a different thread) after I know more about what is really going on.   Hearing what my Mom thinks she hears vs what the doctors actually say is not always the same thing, but the situation is not good, just may not be as dire as I first thought.