Intel Realsense

How to use the Intel Realsense robot skill

Connects any Intel RealSense depth camera to ARCx: an obstacle scan for the Navigation Messaging System (NMS) with the floor removed and drops found, colour, depth and infrared video for the camera skills, a measured distance for every detection a vision skill makes, the camera's tilt from its inertial sensor, recordings and point clouds.

The skill drives the RealSense D400 series (D401, D405, D410, D415, D435, D435i, D435f, D436, D455, D456, D457) and the D500 series (D555, D585) on Windows, Linux (Ubuntu 24.04, x64 and ARM64) and macOS (Apple silicon). Intel's librealsense library is included for each of them, so there is nothing else to install on Windows. Several cameras can be used at once, one robot skill each.

A depth camera sees far more than a lidar. It measures a whole picture of distances, so it finds obstacles at every height the robot could hit: a table top, a chair seat, a shelf, a person's legs and a toy on the floor alike. Knowing how high the camera is mounted, the skill tells the floor from what stands on it, and can see where the floor drops away at the top of a staircase. All of that becomes a scan the navigator reads like a lidar's.

The window

• Start / Stop - open the camera and start streaming, or stop and close it.
• Camera - which RealSense to use. "First camera found" suits a robot with one camera. The refresh button looks for cameras again. When a recording is being played, its file name shows here instead.
• Status - Stopped, Connecting, Streaming, Playing, Recording or Reconnecting, and Obstacle while something is inside the obstacle zone.
• Readouts - the camera's model, serial number, firmware, USB connection and chip temperature; the frame rate, how much of the picture has a depth reading, and the distance to whatever the middle of the picture sees; the nearest obstacle and its direction, the nearest drop, and the camera's tilt; and what the skill is sending to the NMS. The last line is the librealsense version, or why the library did not load.
• Color, Depth, Infrared - the camera's picture. Depth is coloured red when near through yellow and green to blue when far, and black where the camera has no reading. Detections other skills make on the colour video are drawn on it.
• Scan view - the robot seen from above, facing up the screen. The blue lines are the edges of what the camera sees, green dots are the nearest obstacle in each direction, purple dots are drops in the floor, the red circle marks the nearest obstacle, the orange triangle is the robot and the grey rings are distance markers. The obstacle zone is outlined in orange, red while something is in it. The mouse wheel zooms, and clicking reports the direction and distance of the spot in the log.
• Log - what the skill is doing: the camera it found, the streams it started, settings it changed, obstacles and drops, and anything that went wrong.

Nothing in the window costs anything while it is not on screen.

How the scan is made

Each depth picture is turned into points in the room, using where the camera is mounted (its height, tilt and position on the robot). Every point is then sorted by how high it is off the floor:

• Floor - lower than the lowest obstacle height (5 cm). Ignored, so the floor in front of the robot is never mistaken for a wall.
• Obstacle - between the lowest and highest obstacle heights (5 cm to 150 cm). The nearest one in each direction goes into the scan.
• Overhead - higher than the highest obstacle height: a table top the robot fits under, the ceiling. Ignored.
• Drop - more than the drop depth (8 cm) below the floor, when drop detection is on. Reported where the floor ends, and sent to the NMS as an obstacle so the navigator steers round the top of the stairs.

The result is one reading per degree across what the camera sees, sent to the NMS ten times a second. A D435 sees about 87 degrees across, so a navigator gets a wedge in front of the robot rather than a full circle. Mounting the camera a little higher than the robot's obstacles and tilting it 10 to 20 degrees down lets it see low things and drops just in front.

Without a mount height the skill cannot tell the floor from an obstacle and reports everything it sees, as a lidar would. Then the rows used setting (Navigation tab) can limit the scan to a band of the picture, as the ARC version did.

Like a lidar, the camera's scans only reach a navigator when something reports the robot's position. With no wheel encoders or other position source in the project, the skill publishes a placeholder position after each scan and stands aside as soon as another skill reports one.

Measuring detections

With colour streaming, depth aligned to colour and Measure detections on (they all are by default), every detection any vision skill makes on the colour video source is measured from the depth. Face tracking, colour tracking, glyph recognition and object tracking all report how far away things are, without knowing anything about the RealSense:

• DistanceM - the straight-line distance from the camera, in metres.
• ForwardM, LeftM, HeightM - where the object is from the robot's centre: ahead, to the left (negative to the right) and above the floor.
• RangeM and Angle - its distance along the floor and its direction in radians, counter-clockwise from straight ahead.

The Vision Detection Tracking Script skill turns these into its object variables (ObjectDistance, ObjectForwardM and so on), and can stop the robot when the object it follows gets close. Any other depth camera skill built on ARCx's depth service does the same.

The inertial sensor

The D435i, D455, D457 and D555 have an accelerometer and gyroscope. The skill uses them three ways:

• The camera's tilt - measured from gravity, so the mounting tilt and roll do not have to be measured by hand. Measure the tilt now on the Mounting tab copies the camera's current tilt into the settings; Measure tilt with the IMU can instead measure it every time the camera starts (with the robot standing on level floor), or continuously for a camera on a tilting head.
• The robot's pitch, roll and turn rate - as global variables, for a robot that wants to know it is on a slope or spinning.
• The depth service - other skills read the readings in the robot's frame.

Global variables

With Set global variables on, the skill publishes these, each starting with the variable prefix (RealSense by default). Distances are in metres and angles in radians counter-clockwise from straight ahead, like every robot skill in ARCx.

Variable	Meaning
IsRunning	True while the camera is streaming
NearestDistance, NearestAngle	The nearest obstacle, along the floor from the robot's centre
FrontDistance	The nearest obstacle within 10 degrees of straight ahead
LeftDistance, RightDistance	The nearest obstacle 10 to 60 degrees to the left or right
CenterDistance	Straight-line distance from the camera to what the middle of the picture sees
Cliff, CliffDistance, CliffAngle	Whether a drop in the floor is seen, and where the nearest one is
Obstacle, ObstacleDistance, ObstacleAngle	Whether something is in the obstacle zone, and where it was when it entered
CameraTilt, CameraRoll	The camera's tilt down and roll, from gravity
RobotPitch, RobotRoll, YawRate	The robot's pitch and roll, and how fast it is turning (positive left), from the inertial sensor
FPS, ValidPercent	Depth frames per second, and how much of the picture has a reading

Only changed values are written, at most every 200 ms.

The obstacle zone

The obstacle zone watches a wedge in front of the robot (60 degrees wide, closer than 40 cm by default). When something enters it, the skill can stop the robot, pause a navigation in progress (and carry on once the zone has been clear for half a second), and run a script. A drop in the floor inside the zone counts too. Because the depth camera sees things at every height, this catches what a lidar or ultrasonic sensor at one height would miss.

Recording and playback

Everything the camera streams can be recorded to a .bag file with the StartRecording ControlCommand (and stopped with StopRecording). The skill plays a recording instead of the live camera when one is chosen on the Camera tab: navigation, measured detections, variables and the obstacle zone all work as if the camera were connected. That makes it possible to try settings at a desk, to reproduce a problem the robot met, and to try the skill without a camera at all using one of Intel's sample recordings. Recordings from Intel's RealSense Viewer play too.

Point clouds

The SavePointCloud ControlCommand and the save_point_cloud AI tool save the next depth frame as a 3D point cloud (.ply) in metres in the robot's frame, coloured from the colour camera when depth is aligned to it. MeshLab, CloudCompare, Blender and Open3D open it. Point clouds and recordings go to My Documents/ARCx Data/RealSense unless another folder is chosen.

AI assistants (MCP)

An AI assistant connected to ARCx gets three tools from the skill: get_depth_scan summarizes what is in front of the robot as sectors with the nearest obstacle and drop in each, plus the camera's tilt and the obstacle zone; measure_point says how far away the thing at a point of the picture is and where it is relative to the robot; save_point_cloud saves a point cloud. With ARCx's own get_camera_image tool on the colour or depth video source, an assistant can look at the picture and then measure what it sees.

Configuration

Camera

Setting	Meaning
Camera	Which RealSense to use, by serial number, or the first one found
Start when the project loads	Open the camera and start streaming as soon as the project has loaded
Reconnect automatically	Keep trying when the camera is unplugged, stops sending frames or cannot be opened
Reconnect every, Lost after	How often to try again, and how long without a frame before the camera counts as lost
Play a recording	A .bag file to play instead of the live camera; empty uses the camera
Repeat the recording	Start the recording again when it ends
Save folder	Where point clouds and recordings go
librealsense library	Empty uses the bundled librealsense 2.59.1. Intel dropped the L515, SR305 and T265 after release 2.53.1; to use one, install librealsense 2.53.1 and enter the full path of its realsense2.dll (or librealsense2.so) here, then restart ARCx

On Linux a camera can only be opened once Intel's udev rules are installed. The Camera tab shows the one command to run when they are missing.

Streams

Setting	Meaning
Depth resolution, frames per second	Automatic lets the camera choose its best depth resolution. 15 frames a second is plenty for navigation and costs half the processor time of 30
Stream colour, resolution, frames per second	The colour camera, needed for the colour video source, coloured point clouds and measured detections. 640 x 480 suits ARCx's vision skills
Stream infrared	The left infrared camera as a greyscale picture, which works in the dark
Stream the inertial sensor	The accelerometer and gyroscope on models that have them
Stream tracking pose	A T265 tracking camera's position, reported to the NMS as the robot's (needs librealsense 2.53.1)

The lists show the resolutions and frame rates the chosen camera offers once it has been found. When the exact streams cannot be had (a USB 2 port offers less), the skill falls back to what the camera can give and says so in the log.

Depth

Setting	Meaning
Visual preset	How the camera trades coverage for certainty. High Accuracy drops doubtful pixels, which suits obstacle detection; High Density fills more of the picture
Laser projector, laser power	The infrared dot pattern that lets the camera measure blank walls. Leave it on indoors; turn it off when two RealSense cameras face each other
Automatic exposure	Let the depth sensor choose its own exposure
Align depth to colour	Line the depth up with the colour picture, pixel for pixel. Needed for measured detections
Shrink depth by	2 halves each side of the picture, cutting the work four times over
Spatial filter, temporal filter	Smooth depth along surfaces, and over the last few frames
Fill holes	Fill pixels with no reading. Off is the honest answer for navigation
Ignore closer than, further than	Depth outside this range is discarded

Mounting

Setting	Meaning
Height off the floor	From the floor to the camera's lenses. Needed to tell the floor from obstacles and to find drops
Tilted down	How far the camera looks down; negative when it looks up
Forward of centre, left of centre	Where the camera sits from the point the robot turns around
Turned left	Which way the camera faces: 0 ahead, 90 left, -90 right, 180 backwards
Roll	Positive when the camera's right side is lower
Measure tilt with the IMU	Off, at start, or continuously (see The inertial sensor)

Navigation

Setting	Meaning
Publish to the NMS	Send the scan to the navigation messaging system
Scans per second	The most scans a second sent
Resolution	Degrees per reading
Nearest, furthest	The distances along the floor the scan covers
Lowest obstacle, highest obstacle	The heights between which something is an obstacle
Rows used	The band of the picture used, for a camera without a mount height
Find drops, drop deeper than, look for drops within	Drop detection
Drops are obstacles	Send drops to the NMS as obstacles
Position placeholder	When to publish the placeholder position a navigator needs

Obstacle zone

Setting	Meaning
Watch for obstacles	Turn the zone on
Closer than, zone width	The size of the wedge in front of the robot
Drops count	A drop inside the zone is an obstacle too
Clear after	How long the zone must stay empty before the obstacle is over
Pause navigation, stop moving	What happens when something enters
When an obstacle appears, when the zone is clear	Scripts to run

Video and variables

Setting	Meaning
Publish colour, depth, infrared video	Video sources named after the robot skill's window title ("Intel Realsense Color", "Intel Realsense Depth", "Intel Realsense Infrared") for the camera and vision skills, recording and the Custom UI
Measure detections	Measure the detections made on the colour video source
Set global variables, prefix, update every	The global variables
Scan view range, window frame interval	How far the scan view shows when it opens, and how often the window's pictures update

Custom UI

The skill offers a RealSense view widget for Custom UI screens: the colour, depth or infrared picture, or the top down scan, with a start/stop button and the nearest obstacle or drop. Its View setting chooses which (Color, Depth, Infrared or Scan).

Troubleshooting

• No camera found - use a USB 3 port and a USB 3 cable; a RealSense on USB 2 is often not found at all, or limited. On Linux, install the udev rules (the Camera tab shows the command), then unplug and replug the camera. On Linux, librealsense also needs libusb (sudo apt install libusb-1.0-0).
• Black holes in the depth - glass, mirrors, shiny and very dark surfaces, sunlight, and anything closer than the camera's minimum range (about 20 cm for a D435, 7 cm for a D405) have no depth. Turning the laser projector on helps indoors.
• The floor shows as obstacles - the mount height or tilt is wrong. Measure the tilt with the IMU, or raise the lowest obstacle height on an uneven floor.
• Drops everywhere - the same: a tilt set too far down makes the floor look like it falls away.
• Old firmware - the log says when the camera's firmware is older than librealsense recommends. Update it with Intel's RealSense Viewer.
• L515, SR305 or T265 - these need librealsense 2.53.1; see the librealsense library setting.

Control Commands for the Intel Realsense robot skill

There are Control Commands available for this robot skill which allows the skill to be controlled programmatically from scripts or other robot skills. These commands enable you to automate actions, respond to sensor inputs, and integrate the robot skill with other systems or custom interfaces. If you're new to the concept of Control Commands, we have a comprehensive manual available here that explains how to use them, provides examples to get you started and make the most of this powerful feature.

Control Command Manual

This robot skill has 23 control commands, listed below. Send them from a JavaScript or Python script, from Blockly, or from another robot skill.

The first value is always the window title, which is the name in the title bar of the robot skill on your workspace. It starts out as Intel Realsense, which is what every example below uses. If you rename the robot skill, or add a second copy of it to the project, use the title as it appears on that window instead.

ControlCommand() can be shortened to CC(), and both spellings work in JavaScript and in Python. The examples are JavaScript; Python is the same call, except that true and false are written True and False.

GetCameraTilt

The camera's tilt measured from gravity by its inertial sensor, as [tilt down, roll] in radians (positive looking down, positive right side lower). [0, 0] for a camera without an inertial sensor.

Syntax

ControlCommand("Intel Realsense", "GetCameraTilt")

Returns: Array of Float

Example

var result = ControlCommand("Intel Realsense", "GetCameraTilt");

GetCenterDistance

The straight-line distance in metres from the camera to whatever the middle of its picture sees. 0 when there is no reading there.

Syntax

ControlCommand("Intel Realsense", "GetCenterDistance")

Returns: Float [1.5e45 to 3.4e38]

Example

var result = ControlCommand("Intel Realsense", "GetCenterDistance");

GetDeviceInfo

The camera's model, serial number, firmware, USB connection and the streams running, plus the librealsense version.

Syntax

ControlCommand("Intel Realsense", "GetDeviceInfo")

Returns: String

Example

var result = ControlCommand("Intel Realsense", "GetDeviceInfo");

GetDistance

The distance in metres along the floor from the robot's centre to the nearest obstacle in a direction, from the latest depth scan. angle is in radians counter-clockwise from straight ahead (0 ahead, PI/2 left, -PI/2 right). 0 when nothing is seen there or the direction is outside the camera's view.

Syntax

ControlCommand("Intel Realsense", "GetDistance", (Number)angle)

Parameters

#	Name	Type	What to pass
1	angle	Number	A number, without quotes

Returns: Float [1.5e45 to 3.4e38]

Example

var result = ControlCommand("Intel Realsense", "GetDistance", 0);

GetDistanceAtPixel

The straight-line distance in metres from the camera to what a point of the picture sees. x and y are fractions of the picture's width and height from the top left (0.5, 0.5 is the middle), the same for the colour, depth and infrared pictures. 0 when there is no reading there.

Syntax

ControlCommand("Intel Realsense", "GetDistanceAtPixel", (Number)x, (Number)y)

Parameters

#	Name	Type	What to pass
1	x	Number	A number, without quotes
2	y	Number	A number, without quotes

Returns: Float [1.5e45 to 3.4e38]

Example

var result = ControlCommand("Intel Realsense", "GetDistanceAtPixel", 0, 0);

GetNearest

The nearest obstacle the camera sees as [distance in metres along the floor, direction in radians counter-clockwise from straight ahead]. [0, 0] when nothing is seen.

Syntax

ControlCommand("Intel Realsense", "GetNearest")

Returns: Array of Float

Example

var result = ControlCommand("Intel Realsense", "GetNearest");

GetPointAtPixel

Where the thing a point of the picture sees is, as [forward, left, height] in metres from the robot's centre on the floor (+forward ahead, +left to the robot's left, height off the floor). x and y are fractions of the picture from the top left. [0, 0, 0] when there is no reading there.

Syntax

ControlCommand("Intel Realsense", "GetPointAtPixel", (Number)x, (Number)y)

Parameters

#	Name	Type	What to pass
1	x	Number	A number, without quotes
2	y	Number	A number, without quotes

Returns: Array of Float

Example

var result = ControlCommand("Intel Realsense", "GetPointAtPixel", 0, 0);

GetSectorDistance

The distance in metres to the nearest obstacle between two directions, going counter-clockwise from startAngle to endAngle, in radians from straight ahead (-0.35 to 0.35 is the 40 degrees around straight ahead). 0 when nothing is seen there.

Syntax

ControlCommand("Intel Realsense", "GetSectorDistance", (Number)startAngle, (Number)endAngle)

Parameters

#	Name	Type	What to pass
1	startAngle	Number	A number, without quotes
2	endAngle	Number	A number, without quotes

Returns: Float [1.5e45 to 3.4e38]

Example

var result = ControlCommand("Intel Realsense", "GetSectorDistance", 0, 0);

HardwareReset

Power cycle the camera over USB, for a camera that has stopped answering. Streaming carries on once it is back when reconnecting is on. Returns false when not running.

Syntax

ControlCommand("Intel Realsense", "HardwareReset")

Returns: Boolean [true or false]

Example

var result = ControlCommand("Intel Realsense", "HardwareReset");

IsCliffDetected

Whether the camera sees a drop in the floor (stairs, a ledge) right now. Needs drop detection turned on and the mount height set.

Syntax

ControlCommand("Intel Realsense", "IsCliffDetected")

Returns: Boolean [true or false]

Example

var result = ControlCommand("Intel Realsense", "IsCliffDetected");

IsObstacleDetected

Whether something is inside the obstacle zone right now. Always false when the obstacle zone is turned off.

Syntax

ControlCommand("Intel Realsense", "IsObstacleDetected")

Returns: Boolean [true or false]

Example

var result = ControlCommand("Intel Realsense", "IsObstacleDetected");

IsRunning

Whether the skill is running (connecting, streaming or waiting to reconnect).

Syntax

ControlCommand("Intel Realsense", "IsRunning")

Returns: Boolean [true or false]

Example

var result = ControlCommand("Intel Realsense", "IsRunning");

ListCameras

The RealSense cameras connected to this computer, one "model (serial number)" per item.

Syntax

ControlCommand("Intel Realsense", "ListCameras")

Returns: Array of String

Example

var result = ControlCommand("Intel Realsense", "ListCameras");

MeasureTilt

Measure the camera's tilt and roll now with its inertial sensor and save them as the mounting tilt and roll. Stand the robot on level floor first. Returns the tilt down in degrees.

Syntax

ControlCommand("Intel Realsense", "MeasureTilt")

Returns: Float [1.5e45 to 3.4e38]

Example

var result = ControlCommand("Intel Realsense", "MeasureTilt");

SavePointCloud

Save the next depth frame as a 3D point cloud (.ply, opens in MeshLab, CloudCompare, Blender), in metres in the robot's frame, coloured when depth is aligned to colour. filename is optional: empty saves a dated file in the save folder. Returns the file's full path.

Syntax

ControlCommand("Intel Realsense", "SavePointCloud", (String)filename)

Parameters

#	Name	Type	What to pass
1	filename	Text	Your text, in quotes

Returns: String

Example

var result = ControlCommand("Intel Realsense", "SavePointCloud", "filename");

SetEmitter

Turn the laser projector on or off while streaming. Off gives a clean infrared picture but worse depth on blank surfaces. Returns false when the camera has no projector or is not running.

Syntax

ControlCommand("Intel Realsense", "SetEmitter", (Bool)on)

Parameters

#	Name	Type	What to pass
1	on	True or false	true or false, without quotes

Returns: Boolean [true or false]

Example

var result = ControlCommand("Intel Realsense", "SetEmitter", true);

SetLaserPower

Set the laser projector's power in milliwatts while streaming (0 to 360 on a D435). Returns false when the camera has no projector or is not running.

Syntax

ControlCommand("Intel Realsense", "SetLaserPower", (Number)milliwatts)

Parameters

#	Name	Type	What to pass
1	milliwatts	Number	A number, without quotes

Returns: Boolean [true or false]

Example

var result = ControlCommand("Intel Realsense", "SetLaserPower", 0);

SetPublishToNMS

Turn sending the depth scan to the Navigation Messaging System on or off. Returns the new setting.

Syntax

ControlCommand("Intel Realsense", "SetPublishToNMS", (Bool)publish)

Parameters

#	Name	Type	What to pass
1	publish	True or false	true or false, without quotes

Returns: Boolean [true or false]

Example

var result = ControlCommand("Intel Realsense", "SetPublishToNMS", true);

SetVisualPreset

Choose the depth sensor's visual preset by name while streaming: "High Accuracy", "High Density", "Medium Density", "Hand" or "Default" on D400 cameras. Returns false when not running.

Syntax

ControlCommand("Intel Realsense", "SetVisualPreset", (String)preset)

Parameters

#	Name	Type	What to pass
1	preset	Text	Your text, in quotes

Returns: Boolean [true or false]

Example

var result = ControlCommand("Intel Realsense", "SetVisualPreset", "preset");

Start

Open the RealSense camera (or play the recording) and start streaming. Returns false if it was already running.

Syntax

ControlCommand("Intel Realsense", "Start")

Returns: Boolean [true or false]

Example

var result = ControlCommand("Intel Realsense", "Start");

StartRecording

Record everything the camera streams to a .bag file until StopRecording (the camera restarts to begin). filename is optional: empty saves a dated file in the save folder. Returns the file's full path. Recordings grow by hundreds of megabytes a minute.

Syntax

ControlCommand("Intel Realsense", "StartRecording", (String)filename)

Parameters

#	Name	Type	What to pass
1	filename	Text	Your text, in quotes

Returns: String

Example

var result = ControlCommand("Intel Realsense", "StartRecording", "filename");

Stop

Stop streaming and close the camera. Returns false if it was not running.

Syntax

ControlCommand("Intel Realsense", "Stop")

Returns: Boolean [true or false]

Example

var result = ControlCommand("Intel Realsense", "Stop");

StopRecording

Stop recording and carry on streaming. Returns the recorded file's path, or empty when nothing was being recorded.

Syntax

ControlCommand("Intel Realsense", "StopRecording")

Returns: String

Example

var result = ControlCommand("Intel Realsense", "StopRecording");

---

This list is read out of the robot skill itself each time its package is uploaded, so it matches the published version (v2).