Robot Skill Services And Core

What a robot skill can call: the robot controllers (servos, digital, ADC, PWM, I2C, UART), video, audio, speech, movement, kinematics, navigation, game controllers and HID devices, browser windows on the robot's own screens, neural networks with ONNX Runtime, scripting, variables, logging and configuration.

This is the companion to RobotSkill-UI-Components.md, which covers the visual side. Both sit beside ARCx.exe, ARCx.dll and ARCx.xml. ARCx.xml is the authoritative reference for every public member; this manual tells you what exists, which of several overloads to reach for, and the rules that are not obvious from a signature.

Contents

1. The worker: lifecycle and overrides
2. Getting hold of a service
3. EZBManager: the robot controllers
4. Servos
5. Digital, ADC, PWM
6. I2C, UART and ping sensors
7. Serial COM ports
8. Video routing
9. Audio
10. Text to speech
11. Speech to text
12. Movement
13. Kinematics
14. Navigation messaging (NMS)
15. Auto Position
16. Scripting, control commands and variables
17. Talking to other robot skills
18. The project
19. Configuration and the JSON schema
20. Logging
21. Persistence and runtime state
22. Custom UI and robot configuration services
23. Utilities
24. Gamepads, joysticks and HID devices
25. Browser windows: web pages, dashboards, video calls and kiosks
26. Depth cameras
27. Robot Bus: topics, services and frames
28. Rules that will bite you

1. The worker: lifecycle and overrides

ARCx.Services.ProjectTools.RobotSkillWorkerAbstractEntity, which derives from ARCx.Core.Disposable.DisposableBase.

The worker holds the skill's logic and state. One instance per skill instance in the project, shared by every browser.

Lifecycle, in order

Between Setup() and Initialize() ARCx assigns Properties (the window title made unique), creates the never-rendered prototype UI instances and the speech engine, and checks the single-instance rules: a second TTS engine, a second movement panel, or a second instance of a skill whose Setup() returned AllowMultipleInstances = false is refused there. During Initialize() the skill is not yet in the project list, so other skills cannot see it. Throwing from Initialize() aborts the add and unloads the assembly.

How each way into a project calls the worker

Exceptions from OnSetConfiguration and OnProjectLoadCompleted during a load are caught, logged and reported as warnings; the project still loads. A programmatic configuration change does not set AllWorkersPaused.

Loading and removal

Each robot skill instance gets its own collectible AssemblyLoadContext. Dependencies and native libraries resolve through the skill's .deps.json, so two skills can ship different versions of one package. ARCx, MudBlazor, Microsoft.AspNetCore.Components and Microsoft.JSInterop always come from ARCx's own copy, whatever the skill ships.

Two instances of one skill mean two copies of every DLL it bundles. A library that finds its own assembly by name across the AppDomain (the OPC UA stack does) can pick up the other instance's copy and fail with a cast error. Suspect this whenever a bundled library works with one instance and fails with two.

ARCx takes the first type in the assembly assignable to each base class: RobotSkillWorkerAbstractEntity and RobotSkillUIMainBaseEntity (required), RobotSkillUIConfigBaseEntity and SpeechToTextEngineAbstractEntity (optional). Each is created with ActivatorUtilities.CreateInstance, which is why constructors can take services.

On removal (or clearing the project) ARCx, in order: removes the worker from the project list, clears its logs and canvases, disposes the UI components, raises ProjectToolsService.OnRobotSkillRemoved, calls DisposeOverride(), unloads the load context and runs a garbage collection. Anything still subscribed keeps the context alive and the DLL locked.

State to check

WindowTitle is the unique per-instance name the user sees and the address scripts use. InstanceGuid survives a rename, so a configuration that points at another skill is better off remembering the guid - but a user-facing setting is normally the window title, marked with [RobotSkillReference].

Redraw

Other overrides

Interfaces a worker can implement

Speech recognition engines are wired differently - see section 11.

2. Getting hold of a service

ARCx services are registered as singletons (one robot, not one per browser tab). Inject them through the worker's constructor:

public class RobotSkillWorker : RobotSkillWorkerAbstractEntity {

  readonly VideoRouteService _videoRouteService;
  readonly ProjectToolsService _projectToolsService;

  public RobotSkillWorker(VideoRouteService videoRouteService, ProjectToolsService projectToolsService) {

    // Constructor only stores references. Objects are created in Initialize()
    _videoRouteService = videoRouteService;
    _projectToolsService = projectToolsService;
  }
}

The injectable services:

Not injected - these are static and called directly:

EZBManager, VariableManager, ARCLogService, ComPortManager, FileManagerService, ConfigSchemaBuilder, UCLogBuffered.AddEntry, UCSkillCanvas.TryBeginDraw, ProjectToolsService.CurrentProjectProperties.

3. EZBManager: the robot controllers

ARCx.Core.EZBManager (static). The EZB classes live in ARCx.Core.EZBController.

An "EZB" is a robot controller. A project always has at least one, at index 0. The list is contiguous - there is never a gap between 0 and EZBSlotCount - 1 - and removing one shifts the rest down.

The EZB object

Peripheral sub-objects on the EZB: Servo, Digital, ADC, PWM, I2C, Uart, HC_SR04, Audio, Power, MusicSynth, Recorder, ConfigurationManager.

The pattern for every hardware call:

var ezb = EZBManager.TryGetEZB(_Configuration.Port.BoardIndex);

if (ezb == null || !ezb.IsConnected)
  return;

var value = ezb.ADC.GetADCValue(_Configuration.Port.Port);

4. Servos

Port enums

Servo.ServoPortEnum covers D0..D255 (physical digital ports), then V0..V255 (virtual ports, which robot skills such as a Dynamixel or Feetech driver hook onto and drive over their own bus), then NA for "not assigned".

Constants: Servo.SERVO_MIN (1), Servo.SERVO_OFF (0), Servo.SERVO_MAX (the project's MaxServoPositions, default 180), Servo.SERVO_CENTER, Servo.SERVO_PWM_VELOCITY_FASTEST (0), Servo.SERVO_PWM_VELOCITY_SLOWEST (20).

The descriptor helpers (preferred)

ARCx.Config.Helpers.ServoDescriptor is what a skill stores in its configuration, normally inside a ServoCollection edited by UCServoSelector.

ServoCollection.Servos is a List<ServoDescriptor>, defaulting to one unassigned servo. Servos[0] is the master; every following servo derives its position from the first using its own Multiplier.

The EZBManager helpers apply all of that for you:

Moving many servos in sync

One command per board rather than one per servo, which keeps a multi-servo move together:

var buffers = EZBManager.CreateServoBuffers();

EZBManager.TryBufferServoPosition(buffers, boardIndex, new ServoPositionItem(port, position));
// ... buffer more ...

EZBManager.DispatchServoBuffers(buffers);

TryBufferServoPosition silently ignores a board index that is out of range, so a project referencing a missing controller moves the servos it can rather than throwing.

The raw servo API

ezb.Servo when you need a specific port directly. Classes are in ARCx.Core.EZBController.Classes.

Events on ezb.Servo, for a skill that mirrors or logs servo traffic: OnServoMove, OnServoMoveDetailed, OnServoMoveDetailed2 (includes the EZB), OnServoVelocity, OnServoAcceleration, OnServoRelease, OnServoGetPosition.

5. Digital, ADC, PWM

Store the port in a descriptor (DigitalPortDescriptor, ADCPortDescriptor, PWMPortDescriptor in ARCx.Config.Helpers) edited by the pickers in the UI manual.

Digital - ezb.Digital

DigitalPortEnum: D0..D255, plus NA.

ADC - ezb.ADC

ADCPortEnum: ADC0..ADC255, plus NA.

PWM - ezb.PWM

PWMPortEnum: D0..D99 only (the digital ports that can output PWM). PWM.PWM_MIN is 0, PWM.PWM_MAX is 100.

6. I2C, UART and ping sensors

I2C - ezb.I2C

UART - ezb.Uart

Two families. Bit-banged serial out of any digital port (send only):

ezb.Uart.SendSerial(Digital.DigitalPortEnum.D5, Uart.BAUD_RATE_ENUM.Baud_9600, "hello");

Overloads take string, char[], byte, or byte[]. SetBaudClock(BAUD_RATE_ENUM, int clockSpeed) adjusts the timing.

The hardware UART expansion ports (bidirectional, buffered):

HC-SR04 ultrasonic - ezb.HC_SR04

EZB speaker - ezb.Audio

ARCx.Core.EZBController.Audio. Streams 8 bit mono PCM to an EZ-B v4 speaker.

ezb.MusicSynth makes synthesized music on the same speaker.

7. Serial COM ports

ARCx.Core.Scripting.Common.ComPortManager (static). A shared registry of open COM ports, so a skill and a script can share one port and the Script Manager can close them all.

A skill that owns a device on a port (a servo bus, a lidar) normally opens its own SerialPort in Initialize() and closes it in DisposeOverride(). Use ComPortManager when the port should be shared with scripts.

Files have an equivalent registry: ARCx.Core.Scripting.Common.FileManagerService (FileReadLine, FileReadAll, FileReadByte, SetFileReadPosition, FileClose, FileCloseAll, and an OnCollectionChanged event).

8. Video routing

ARCx.Services.VideoRouting.VideoRouteService. The meeting point between skills that produce frames (cameras) and skills that consume them (vision, tracking, recording).

Producing frames

// Initialize()
_sourceGuid = _videoRouteService.AddSource("My Camera", "USB camera on /dev/video0", 30);

// per frame, with an OpenCV Mat
_videoRouteService.NotifySubscribers(this, _sourceGuid, matImage);

// DisposeOverride()
_videoRouteService.RemoveSource(_sourceGuid);

VideoRouteSource carries SourceName, SourceDescription, FPS (as declared, not measured), FrameCount and LastUpdated.

Save the source name in a configuration, never the guid - guids change every session.

Consuming frames

Each subscriber is invoked on its own, so one skill that throws cannot stop the others from seeing the frame or take the capture thread down.

VideoRouteSourceEventArg:

The images are borrowed. The service disposes the argument after both events and the browser refresh; subscribers must not dispose it and must clone anything they need to keep beyond the handler.

VideoSourceMarkupItem - let ARCx draw the overlay rather than doing it yourself:

Following detections

A detection skill only detects and adds markup items. Following the object with servos or the movement panel, scripts that run when something appears, and the variables describing it belong to the Object Detection Script robot skill, which works for every detection skill. Do not duplicate them in a detection skill: fill in TrackingType, TrackingTypeValue and, where you have them, Confidence and Attributes, and it takes care of the rest.

A skill of your own that reacts to detections subscribes to OnObjectsDetected. It is raised for every frame of every source, after OnNewImageBeforeProcess, with an empty list when nothing was detected so you can tell an object has gone. Every markup item with an area that names its TrackingType or is flagged TreatAsObjectDetected becomes a DetectedObject:

The argument also carries VideoSource, FrameWidth, FrameHeight, FrameNumber and Time (Environment.TickCount64). It holds no images and never changes, so keep it as long as you like. It is raised on the capture thread, so hand servo moves and scripts to a thread of your own.

Enumerating cameras

ARCx.Core.ComputerVision.VideoDeviceManager.GetDeviceList() returns the cameras attached to this computer with the modes each supports. Store the device Identity to find it again on reload, and the mode's VideoDeviceResolution.ID (which carries FPS, width and height). Enumeration is synchronous and can take a moment. Windows and Linux only; anywhere else it logs and returns an empty array rather than throwing.

9. Audio

ARCx.Services.AudioServices.AudioService. Audio in and out, mixed across every skill.

Everything you play must match OUTPUT_WAVE_FORMAT, or it is noise. The microphone runs at a different rate from the speaker: 16kHz is what speech recognition and voice activity detection are built for, so a speech engine can usually take OnAudioInput as it is without resampling. Never assume the two rates are the same.

Where it comes out

Do not write a "play on the PC" path and a "play on the EZB" path. There is one output mix, and where it goes is a setting: this machine's preference in Options, Preferences, overridden by the project's setting on the EZB Connection robot skill. Queue your audio in the format above and it plays out of the computer's speakers or is converted, resampled and streamed to an EZB's speaker without your skill knowing which. A skill that offers its own speaker choice is asking the user the same question twice and will disagree with the rest of the project.

Read them if you want to show the user where the sound is going. Do not branch on them to decide how to play something.

Playing a finite clip

Register a source, play through it, deregister on dispose:

// Initialize()
_audioGuid = _audioService.RegisterInputSource();

// Play
using var stream = ...;
using var waveReader = new WaveFileReader(stream);

var resampled = new WdlResamplingSampleProvider(waveReader.ToSampleProvider(), AudioService.OUTPUT_SAMPLE_RATE);

_audioService.StopAudio(_audioGuid);
_audioService.AppendPlayAudioForGUIDSource(_audioGuid, resampled);

// DisposeOverride()
_audioService.DeregisterInputSource(_audioGuid);

Streaming live audio

AddSourceToMixer(ISampleProvider) / RemoveSourceFromMixer(ISampleProvider). The mixer pulls from the provider as needed, in packets, so nothing is read into memory up front. Use this for live streams and for very large files.

Input and devices

Saving audio files

A WAV is NAudio's WaveFileWriter. For MP3, ARCx has its own fully managed encoder, ARCx.Services.AudioServices.Encoders.Mp3Encoder. It needs no native library, so the same code works on every platform ARCx runs on; never ship LAME or call the operating system's encoder.

using ARCx.Services.AudioServices.Encoders;

// A whole clip: float samples, -1 to +1, interleaved when stereo
byte[] mp3 = Mp3Encoder.Encode(samples, 22050, channels: 1, bitrateKbps: 64);

// Or a stream, written as the audio arrives
using (var encoder = new Mp3Encoder(fileStream, AudioService.INPUT_SAMPLE_RATE, 1)) {

  encoder.Write(frame);          // float samples
  encoder.WritePcm16(pcmBytes);  // or 16 bit little endian PCM, such as a TTS result
}                                // Dispose() calls Finish(), which sends the tail

// Anything with an end, resampled when MP3 cannot have its rate
Mp3Encoder.Encode(waveReader.ToSampleProvider(), outputStream);

The result is constant bitrate MPEG-1, 2 or 2.5 layer III (whichever the sample rate belongs to) that every player and browser plays. It is a simple encoder, with no psychoacoustic model and no joint stereo: speech and robot sounds are clean at the default bitrates, but music at a low bitrate sounds worse than from LAME. The decoded audio starts about 1057 samples (a few tens of milliseconds) late and ends with a little silence, because there is no LAME tag to tell players to trim it. An instance is not thread safe.

NAudio types (ISampleProvider, WaveFileReader, WdlResamplingSampleProvider) come from the copy of NAudio that ARCx ships. Add NAudio.Core to the References item group of the skill's .csproj to use them (see rule 16); note that NAudio.Core 3.x Read() takes a Span<float>.

10. Text to speech

ARCx.Services.TextToSpeech.TextToSpeechService.

Speaking

There is no PC version and EZB version of these. Speech goes into the output mix like anything else, and the project's audio output setting decides which speaker it reaches - see 9. Nothing that speaks needs an EZB index, and nothing that speaks should offer the user a choice of speaker.

The project's TTS engine is whichever robot skill implements TextToSpeechEngineInterface; if there is none, the Say methods throw.

Following speech

A robot skill that moves with the voice - a talking mouth, lights that pulse while the robot speaks - subscribes to OnSpeaking. It is raised for every Say and SayWait, from any script, robot skill or AI agent, once the speech has been synthesized and queued, and it carries the speech itself, so the skill works from the real sound rather than guessing from the text.

TextToSpeechSpeakingEventArgs: Text (as passed to Say, possibly SSML), IsSsml, Samples (mono floats, -1 to +1, shared by every subscriber - read, do not change), SampleRate, StartPosition (the GetPlaybackPosition() value at which the first sample is heard; in the future when earlier speech is still playing) and Duration.

GetPlaybackPosition() - StartPosition is how many samples of that speech have been played: negative before it starts, past Samples.Length once it has finished. When IsSpeaking is false and the position has stopped moving short of the end, the speech was stopped. What is heard trails the position by the sound card's or EZB's own buffering, so give the user an offset to tune. The Talk Servo robot skill is a complete example.

Being the engine

Implement TextToSpeechEngineInterface on the worker:

public TextToSpeechResultCls OnTextToSpeech(TextToSpeechRequestCls request) {

  // request.SampleRate, request.SampleSize (Bits8 / Bits16), request.Channels (always 1),
  // request.TextToSpeak (plain or SSML - use TextToSpeechService.IsSsml())

  return new TextToSpeechResultCls() {
    Success = true,
    Audio = pcmBytes,       // RAW PCM, no file header, in exactly the format requested
  };
}

Called synchronously on the caller's thread. Return Success = false (or null Audio) when the text could not be synthesized. Honour request.SampleRate and request.SampleSize exactly; any other format produces noise. In practice speech is always asked for as 16 bit at AudioService.OUTPUT_SAMPLE_RATE, because it goes into the output mix and the mix is converted to whatever the speaker needs - an engine has nothing to say about where its voice comes out.

Only one TTS engine skill is allowed in a project.

11. Speech to text

ARCx.Services.SpeechToText.SpeechToTextService routes between skills that provide recognition and skills that consume it. The service does no recognition itself.

An engine says what it can do, not what it is for. Whether an engine is the wake word listener, the command listener or the one behind Wait For Speech is decided by whoever wires the project together. Everything an engine cannot do natively is filled in by the service:

Those first two are separate, and an engine billed per phrase wants both. CanListenContinuously false alone still leaves the service re-arming it forever, which is exactly how a metered service burns through a daily allowance on an empty room.

SpeechToTextModeEnum is a property of the request, not of the engine:

Providing an engine

Add one subclass of SpeechToTextEngineAbstractEntity to the skill's assembly (not on the worker itself). ProjectToolsService finds it when the skill is added, creates it with dependency injection, and assigns it to the worker's SpeechToTextEngine property.

SpeechToTextCapabilitiesCls:

Report results back through the service. There is no session to track - the service stamps each message with whichever session the engine is running for:

_speechToTextService.UpdateSpeechDetectedMessage(worker, speechResultCls);
_speechToTextService.UpdateSpeechHintMessage(worker, speechHintCls);     // "speak louder", etc
_speechToTextService.UpdateAudioStateMessage(worker, audioStateCls);      // silence / speech / stopped

Report failures as well as successes (Success = false, ErrorString). The service closes a Once session, re-arms and puts gated engines to sleep on the strength of these messages, and a consumer waiting on one utterance would otherwise wait out its whole timeout.

Triggers - the wake word

A trigger belongs to the gated engine's own robot skill, so the wiring follows that skill around the project and every consumer of the engine inherits it.

1. Put a SpeechToTextTriggerCls on the skill's Configuration and save it as usual.
2. Drop <UCSpeechToTextTrigger @bind-Trigger="_configuration.Trigger" EngineWindowTitle="@(CurrentRobotSkillWorker?.Properties?.WindowTitle)" /> into the skill's config dialog. The component lists only engines whose capabilities report CanListenContinuously, because a one-utterance engine cannot listen for a wake phrase all day.
3. Return it from GetTrigger().
4. Call SpeechToTextService.NotifyTriggerChanged(windowTitle) from OnConfigWindowClosed, so a wake word the user just set takes effect on recognition that is already running rather than at the next project load.

That is the whole feature on the engine's side. The service arms the trigger source, starts the gated engine when the phrase is heard, refreshes the awake window on every result, and stops it when StayAwakeMs lapses. StayAwakeMs of 0 means one utterance per wake, which is the setting for an engine with a daily allowance. An engine may be its own trigger source, in which case the service swaps the grammar back and forth.

An engine that reports RequiresTrigger with no usable trigger gets a gate that never wakes by itself: it stays quiet for Continuous requests, and TriggerWake() still works so a listen button or a Wake control command can run it for one utterance.

Waking on any voice. SpeechToTextTriggerCls.WakeOnAnySpeech wakes the gated engine as soon as anybody talks, with no phrase to say. The trigger listener then asks its source for dictation rather than a one-phrase grammar, and any successful result wakes the gate.

That is what makes a voice activity detector useful. Write one as an ordinary speech engine with CanListenContinuously = true and DetectsSpeechOnly = true, and signal either way:

• publish a result with Success = true (the text may be empty), or
• publish UpdateAudioStateMessage with AudioStateEnum.Speech.

Either wakes whatever the detector is the trigger for, so a detector that never produces text at all still works. The service forces WakeOnAnySpeech for a detector source whatever the saved trigger says, since there is nothing for a phrase to match, and GetSpeechToTextTriggerSources() is the list that includes detectors - GetSpeechToTextRobotSkills() leaves them out, because they have no text to give a consumer.

Consuming recognition

Filter on SessionId, never on OriginWindowTitle alone. Two consumers can be sharing one engine, and the window title does not tell you whose speech a result is.

Do not build a wake word state machine in a consumer. Ask for Continuous and subscribe to OnTriggered if you want to know when the robot starts and stops listening.

The engine the Wait For Speech popup and script commands use is named in ProjectToolsService.CurrentProjectProperties.WaitForSpeechEngineWindowTitle.

12. Movement

ARCx.Services.Movement.MovementManagerService. The abstraction between "go forward" and the motors.

Anything can call Go*(). The movement panel - the one skill in the project implementing MovementPanelInterface - subscribes to the events and actually drives the hardware.

MAX_SPEED is 255, MIN_SPEED is 0.

MovementDirectionEnum: Stop, Forward, Reverse, Left, Right, Up, Down, RollRight, RollLeft, Takeoff, Land, Emergency, Custom.

Events:

13. Kinematics

ARCx.Services.Kinematics.KinematicsService. A registry and router: a provider skill publishes named chains ("Left Arm", "Camera Gimbal") and does the maths; everything else moves them by name without knowing who owns them. The service does no kinematics itself.

Consumer side

// Configuration: UCKinematicChainSelector, save the name
var result = _kinematicsService.MoveTo(_Configuration.ChainName, x, y, z);

if (!result.Accepted || !result.Reachable)
  UCLogBuffered.AddEntry(this, $"Move refused: {result.Message}");

An unknown chain name throws - that is a configuration error the caller needs to see. A provider that cannot reach the target does not throw; check KinematicMoveResult.Accepted and .Reachable.

KinematicMoveResult: Accepted, Reachable, Message, SolvedPose, JointAnglesDegrees, ResidualErrorMm, plus the factories Ok(...), Rejected(...), Unreachable(...).

KinematicPose: X, Y, Z, Roll, Pitch, Yaw, HasOrientation, Timestamp, and DistanceTo(), Offset(), Clone(), ToPositionArray(), ToPoseArray().

KinematicMoveModeEnum selects Joint (default) or linear interpolation.

Provider side

• AddChain(KinematicChainRegistration) in Initialize() or after your configuration loads. Keep the returned Guid. Throws on an empty name, a duplicate name (case-insensitive) or a missing required handler - catch per chain so one bad name does not sink the others.
• RemoveChain(guid) in DisposeOverride(). Safe with an unknown or already-removed guid.
• UpdateChainDetails(guid, name, description, providerName, jointCount, supportsOrientation, supportsLinearMoves) - leave a parameter null (or -1 for the joint count) to keep it.
• NotifyPoseChanged(guid, pose) whenever the end effector moves, including every step of an interpolated move and when you notice another skill moved your servos. Displays, the published variables and OnPoseChanged all hang off this.
• NotifyMoveStateChanged(guid, bool isMoving) when a move starts and ends, or WaitForMoveComplete() never returns.

KinematicChainRegistration handlers: OnMoveRequested (Func<KinematicMoveRequest, KinematicMoveResult>), OnPoseRequested (Func<KinematicPose>), OnStopRequested (Action), OnIsMovingRequested (Func<bool>), OnJointAnglesRequested (Func<double[]>). Plus Name, Description, ProviderName, JointCount, SupportsOrientation, SupportsLinearMoves, PublishVariables (default true).

With PublishVariables, poses are published as global script variables prefixed Kinematic + the chain name with non-alphanumerics removed ("Left Arm" becomes KinematicLeftArmX, KinematicLeftArmY, ... KinematicLeftArmIsMoving).

14. Navigation messaging (NMS)

ARCx.Services.NMS.NMS2Dv1.NMSService. Routes 2D positioning and scanning data between sensor skills and a navigator (SLAM) skill.

Three levels: sensors report, the service collates, a navigator consumes.

Reporting

ScanPoint: Range (metres, 0 means nothing detected), Angle (radians relative to the robot, -PI to PI: 0 straight ahead, growing counter-clockwise, so PI/2 is left and -PI/2 is right), Confidence (0 = none, 255 = certain).

Pose: X, Y (metres), Yaw (radians, -PI to PI), Confidence, plus Frame and TimeStamp which the service assigns.

The NMS is ROS's frame and units (REP 103): metres and radians. +X is forward (the way the robot faced when tracking began), +Y is left, and Yaw grows counter-clockwise, as the robot turns left, from 0 at the start. ScanPoint.Angle grows the same way from straight ahead, so a scan point at range r lands on the map at bearing h = Yaw + Angle: X + r * cos(h), Y + r * sin(h). A ROS pose or LaserScan maps straight across with no conversion at all. Report angles normalised to -PI..PI (NMSAngles.NormalizeRadians()), as ROS does, and accept any angle you are given. NMSAngles.ToRadians()/ToDegrees() convert for display.

The old Pose.DegreesHeading, ScanPoint.Distance (cm) and ScanPoint.Degree remain as [Obsolete] members that convert, so robot skills built before this change keep loading. Do not use them in new code.

Wheel odometry in this frame, per reading, with each wheel's travel dL, dR (metres) and the distance between the wheels b: the yaw changes by (dR - dL) / b radians (positive turning left), and the robot moves (dL + dR) / 2 along the yaw half way through that change.

Every robot skill uses this frame for any angle or position around the robot, not only NMS data, and uses metres and radians for anything a script reads or passes (variables, control command parameters and results): Turn(1.5708) turns a quarter turn left, and an arc "from A to B" runs counter-clockwise. Settings a person types and readouts on screen stay in degrees (and whatever distance unit reads naturally), converted at the UI.

Use the same names as the other robot skills for the same things, so a script written for one reads naturally against another:

Consuming

A navigator that works its position out from the scans alone still needs poses to arrive, because that is what releases the scan points. A distance sensor skill can publish a placeholder pose (the last known X, Y and Yaw) after each sweep when it notices no other skill is reporting one - the RPLidar skill does this, and compares FrameCnt before and after its own UpdatePose() to tell whether anything else is posting.

15. Auto Position

ARCx.Core.AutoPosition.AutoPositionService. The engine behind the Auto Position skill: named frames (servo poses) combined into actions (sequences).

Derives from DisposableBase - dispose it when the skill closes, which stops any movement and unhooks it from the EZBManager connection events.

16. Scripting, control commands and variables

Control commands: exposing your skill to scripts

Declare them, then handle them.

readonly string cmd_setCounter = "SetCounter";

public override List<ControlCommandItem> GetSupportedControlCommands() {

  var items = new List<ControlCommandItem>();

  items.Add(
    new ControlCommandItem() {
      CommandName = cmd_setCounter,
      Description = "Set the counter to the specified value",
      ReturnType = typeof(int),
      Parameters = [
        ControlCommandParameter.AddParameter(ControlCommandParameter.ParameterTypeEnum.Number, "value"),
      ],
    });

  return items;
}

public override object ControlCommand(ControlCommandArgs args) {

  if (args.Command.Equals(cmd_setCounter, StringComparison.InvariantCultureIgnoreCase)) {

    // The user's script decides what is actually passed, so always verify
    if (args.Parameters.Length != 1)
      throw new Exception($"The {cmd_setCounter} control command expects 1 parameter (value)");

    _Cntr = Convert.ToInt32(args.Parameters[0]);

    RefreshUI();

    return _Cntr;
  }

  // The base raises the "no matching control command" exception for the calling script
  return base.ControlCommand(args);
}

ControlCommandItem:

ControlCommandParameter.AddParameter(ParameterTypeEnum, string name) and AddDefaultValue(ParameterTypeEnum, object defaultValue).

ControlCommandArgs: Command, Parameters (object[]), CancellationToken - observe the token in a long-running command.

Running scripts

Create executors through ScriptManagerService rather than new Executor(), so the Script Manager UI can list, monitor and stop them, and so they are disposed when the project is cleared.

// Initialize()
_executor = _scriptManagerService.CreateExecutor($"{Properties.WindowTitle} script");

// Read back CompilerName - a duplicate name gets a number appended
_executor.EnableConsoleDebugLogging();

// Run
_executor.StartScriptASync(_Configuration.OnDetectedScript);

// DisposeOverride()
_scriptManagerService.RemoveAndDisposeExecutor(_executor);

ScriptManagerService: CreateExecutor(string compilerName), RemoveAndDisposeExecutor(Executor) (safe with null or an already-removed executor), Executor[] GetExecutors, events OnAdded / OnRemoved.

ScriptDefinition (ARCx.Config.Helpers) holds BlocklyXML, Javascript and Python slots - only one at a time; the editors clear the others when the user switches language. IsEmpty, GetScriptEditType (checked Blockly, then JavaScript, then Python), Title, Description.

Global variables

ARCx.Core.Scripting.Common.VariableManager (static). One program-wide set of named values shared by robot skills, JavaScript, Python, Blockly, Custom UI screens and the AI agent. It is how a skill publishes a reading that anything else can use without a control command: FrontPingDistance, KinematicLeftArmX, and so on.

Names are plain (Direction, not $Direction) and case-insensitive. A leading $ - the old EZ-Script convention - is accepted and removed, so "$Direction" still finds Direction, but do not use it in new names. NormalizeName(name) gives the canonical form.

Values are stored as the raw object. Numbers, strings and booleans (real bool, not 1/0) are stored as they are; an enum is stored as its name. Arrays, lists and dictionaries are stored by reference - scripts use setVarObject() to store a copy.

OnVariableChanged is raised on the thread that made the change - often a script or another skill's worker thread - on every set, even when the value is the same. Filter first and keep the handler cheap; marshal to the UI with InvokeAsync. An exception thrown by a handler is logged and does not reach the code that set the variable.

// When the skill starts: describe it once
VariableManager.SetVariable("FrontPingDistance", 0, "Distance in cm from the front ping sensor");

// Every reading: the description is kept
VariableManager.SetVariable("FrontPingDistance", cm);

// Reading one another skill publishes, with a fallback
var tracking = VariableManager.GetVariable("CameraIsTracking", false);

// Watching one
VariableManager.OnVariableChanged += e => {

  if (!e.Affects("FrontPingDistance"))
    return;

  var cm = e.Value;   // null when it was deleted or cleared
};

Scripts see the same variables: getVar("Name"), getVar("Name", default), setVar("Name", value, "optional description"), setVarObject(...) (stores a copy), varExists("Name") and deleteVar("Name"), identically in JavaScript and Python. In a Blockly workspace, a variable whose name starts with $ is a global variable (the $ is removed in the generated code); other Blockly variables are local to the script. Custom UI labels show one with {Name}.

MCP tools and resources: exposing your skill to AI clients

ARCx is a Model Context Protocol (MCP) server (ARCx.Services.Mcp.McpServerService, at /mcp). AI clients the user connects - Claude, Cursor, VS Code Copilot, ChatGPT, agent frameworks - see the project through it. Every control command is already offered to them through the generic send_control_command tool, with the description and parameters you declared, and the configuration is readable and editable through the schema of section 19. Nothing is needed for a skill to work with an AI.

Override GetMcpTools() when a command deserves a first class tool: named and typed arguments instead of a positional list, a longer description, or a result that is not a string - an image above all, which lets the AI see what the robot sees. Override GetMcpResources() for data the AI reads by URI (a latest reading, a status document, an image).

using ARCx.Services.Mcp;

public override List<McpToolItem> GetMcpTools() {

  return [
    new McpToolItem() {
      Name = "look",
      Description = "The next camera frame as an image you can look at. Nothing is saved.",
      ReadOnly = true,
      InputSchemaJson = "{\"type\":\"object\",\"properties\":{\"maxWidth\":{\"type\":\"integer\",\"description\":\"Widest, in pixels. Defaults to 640.\"}},\"additionalProperties\":false}",
      Handler = args => Task.Run(() => {

        using var mat = waitForFrame(args.CancellationToken);

        return McpToolResult.Image(encodeJpeg(mat, args.GetInt("maxWidth", 640)), "image/jpeg", $"{mat.Width}x{mat.Height}");
      }),
    },
    new McpToolItem() {
      Name = "set_counter",
      Description = "Set the counter and optionally start or stop its timer.",
      InputSchemaJson = McpToolItem.InputSchemaFromType(typeof(SetCounterArguments)),
      Handler = args => {

        var a = args.GetArguments<SetCounterArguments>();

        _Cntr = a.Value;

        return Task.FromResult(McpToolResult.Json(new { counter = _Cntr }));
      },
    },
  ];
}

public override List<McpResourceItem> GetMcpResources() {

  return [
    new McpResourceItem() {
      Uri = "status",
      Name = "Counter status",
      MimeType = "application/json",
      Reader = ct => Task.FromResult(McpResourceContent.Json(new { counter = _Cntr })),
    },
  ];
}

McpToolItem:

McpToolCallArgs: Arguments (JsonElement), CancellationToken (the client gave up, the 120 second call timeout elapsed, or the skill is being removed), ToolName, and the tolerant getters GetString, RequireString, GetInt, GetDouble, GetBool, GetArray, Has, GetArguments<T>().

McpToolResult: Text(string), Json(object) (text and structured content; anonymous types work), Image(byte[] jpegOrPng, mimeType, caption), Error(message) (the AI reads it and tries something else, which beats an exception for anything the model can fix), .WithText(...) to add a caption. McpContent.FromText/FromImage/FromAudio for anything else.

McpResourceItem: Uri (relative; ARCx prefixes arcx://skill/<slug>/), Name, Description, MimeType, Reader returning McpResourceContent.Text, .Json(object) or .Blob(bytes, mimeType).

Rules:

• Both overrides are called whenever a client lists tools or calls one, so keep them quick and return the same tools each time. When what you offer changes at run time (a tool per trained object, say), call NotifyMcpChanged() so connected clients list again.
• Never put an API key or anything secret in a tool result or a resource; it goes to a model.
• Keep results small: they are tokens in somebody's conversation. Scale images down (640 pixels wide is plenty), cap lists.
• A tool that calls a paid model or moves the robot is not ReadOnly, however harmless it seems.

McpCoreTools.GrabFrameJpegAsync(videoRouteService, sourceName, maxWidth, processed, ct) (a static on the DI singleton ARCx.Services.Mcp.McpCoreTools) returns the next frame of a video source as JPEG bytes and is what get_camera_image uses; a skill that wants the same thing can call it rather than subscribing to the video events itself.

The user controls all of this in Options → Preferences → AI connections: the server on or off, an access token, and whether a connected AI may control the robot and change the project. See Manuals\mcp.md for the user's side.

17. Talking to other robot skills

ARCx.Services.ProjectTools.ProjectToolsService.

Look up another robot skill only in OnProjectLoadCompleted(). In Initialize() or OnSetConfiguration() the other skills may not be loaded yet.

Reading and writing another skill's configuration programmatically (this is how Athena does it):

18. The project

ProjectToolsService.CurrentProjectProperties (static) is ProjectProperties.

ARCx.Core.Constants holds the well-known folders: DEFAULT_FOLDER (My Documents\ARCx Data, or ~/ARCx Data on Linux and macOS), DEFAULT_APPLICATION_FOLDER, DEFAULT_PROJECT_FOLDER, ROBOT_SKILL_PROJECT_FOLDER, PLUGINS_FOLDER, SERVO_PROFILES_FOLDER, PYTHON_MODULES_FOLDER, DEFAULT_USER_LOGS_FOLDER, and the file extensions (.arcx, .AutoPositionX, .servoX, .blocklyX).

An .arcx project file is a zip:

projectProperties.json
robotskills/<WindowTitle>/properties.json    RobotSkillProperties
robotskills/<WindowTitle>/config.json        whatever GetConfigOnProjectSave wrote

A skill whose GetConfigOnProjectSave throws aborts the whole save, so catch what you can and log it. GetConfigOnProjectSave is also called at any time by the AI agent to inspect the configuration, so keep it cheap and free of side effects.

19. Configuration and the JSON schema

Save and load with whatever format you like - the stream is yours - but JSON is the convention:

public override void OnSetConfiguration(OnSetConfigurationParams configurationParams) {

  if (configurationParams.IncludesConfig)
    _Configuration = JsonSerializer.Deserialize<Configuration>(configurationParams.ConfigStream)!;
}

public override void GetConfigOnProjectSave(Stream configStream, JsonSerializerOptions suggestedJSONSerializerOptions) {

  JsonSerializer.Serialize(configStream, _Configuration, suggestedJSONSerializerOptions);
}

public override Type GetConfigurationType() {

  return typeof(Configuration);
}

GetConfigurationType() is what lets the Athena AI agent (and any other programmatic editor) read and change the skill's settings - "make my robot track faces" works because the agent can see what each setting means. ConfigSchemaBuilder builds an annotated JSON Schema from the type:

• Descriptions come from, in order: a [Description] attribute, a [RobotSkillReference] attribute, or the /// summary comment on the property. That is why the template project builds with GenerateDocumentationFile on: the XML ships beside the DLL and the schema reads it. Write a /// summary on every configuration property, for someone who has never seen the skill.
• [Range] becomes minimum / maximum.
• Integer-backed enums list their names compactly ("D0..D255=0..255, NA=512"), because the JSON holds the number and the number alone means nothing.
• Byte arrays and get-only properties are marked so an editor leaves them alone.

[RobotSkillReference("kind")] (ARCx.Config.Schema) marks a string property that holds another robot skill's window title:

/// <summary>
/// The camera robot skill this skill takes its video frames from.
/// </summary>
[RobotSkillReference("camera source")]
public string SourceName { get; set; } = string.Empty;

It adds "x-arcx-reference": "robotSkillWindowTitle" to the schema, so an agent knows the value must match an existing skill rather than being free text. The constructor without arguments means any robot skill.

ConfigSchemaBuilder.Build(Type) returns a JsonNode; BuildJson(Type, bool indented = false) returns a string. Schemas are cached per type. A skill can call it on any type.

Return null from GetConfigurationType() to opt out of programmatic editing.

20. Logging

Two destinations, for two audiences.

ARCLogService - errors and diagnostics

ARCx.Core.ARCLog.ARCLogService (static). Goes to the console, the Log Monitor, and the user's usage log file on disk.

try {

  doTheWork();
} catch (Exception ex) {

  ARCLogService.AddLogEntry(this, LogEntryCls.SeverityEnum.Error, MethodBase.GetCurrentMethod(), ex);
}

Overloads:

AddLogEntry(RobotSkillWorkerAbstractEntity origin, SeverityEnum type, MethodBase m, Exception ex = null)
AddLogEntry(RobotSkillWorkerAbstractEntity origin, SeverityEnum type, MethodBase m, string message, Exception ex = null)
AddLogEntry(object origin, SeverityEnum type, MethodBase m, Exception ex = null)
AddLogEntry(object origin, SeverityEnum type, MethodBase m, string message, Exception ex = null)
AddLogEntry(string origin, SeverityEnum type, MethodBase m, string message, Exception ex = null)
AddLogEntry(EZB origin, SeverityEnum type, MethodBase m, string message, Exception ex = null)

SeverityEnum: Info (0), Warning (1), Error (2).

Passing the worker as origin records the skill's type, the method name and the skill's window title, so the user can tell which instance logged it.

AddLogEntry() never blocks and never throws - entries are queued and written on a thread pool task, so they may appear slightly after the call. Do not log from a tight loop; every entry is written to disk.

UCLogBuffered - what the user reads

UCLogBuffered.AddEntry(this, "Connected to COM3 at 115200");

See the UI manual, section 8. A caught exception goes to ARCLogService; a status line the user wants to see goes to UCLogBuffered. Most user-visible actions should leave a line there.

21. Persistence and runtime state

Three stores, for three lifetimes.

CustomRegistryManagerService

Namespaced key/value, encrypted on disk with the signed-in account's email.

_customRegistry.SetRegistryEntry("MySkill", "ApiKey", key);
var key = _customRegistry.GetRegistryEntry("MySkill", "ApiKey", string.Empty);

SetRegistryEntry(ns, name, value) and GetRegistryEntry(ns, name, defaultValue) overload for bool, int, long, byte, decimal, double, DateTime and string (GetRegistryEntryAsLong for long). DeleteRegistryEntry(ns, name).

RuntimeStateService

_runtimeState.Set("mySkill.lastScan", scanPoints);

if (_runtimeState.ContainsKey("mySkill.lastScan"))
  var points = _runtimeState.Get<ScanPoint[]>("mySkill.lastScan");

Get<T> throws KeyNotFoundException when the key is missing and when it exists but is not a T. Check ContainsKey() first. Keys are case-sensitive. Remove(key) returns whether anything was removed.

ConfigRegistryManagerService

ARCx's own preferences (log intervals, font sizes, audio devices, computer-vision scale width, RobotSkillUIQueuedUpdateMS). A skill reads these to respect the user's preferences; it does not store its own settings here.

22. Custom UI and robot configuration services

CustomUIService

ARCx.Services.CustomUI. The screens the user lays out for their robot, which one is showing, and the breadcrumb behind it. The screens themselves live in the project properties, so they save and load with the project.

Navigation is one shared state, so a script, a control command and a phone all agree on which screen is up.

A skill contributes widgets through GetCustomUIWidgets() on the worker - see the UI manual, section 16.

RobotConfigService

ARCx.Services.RobotConfig. The picture of the robot and the ports marked on it.

It copies each marker's name into the project's port description dictionaries, so every port selector shows "V3 (Left elbow)" without knowing the picture exists, and it pushes the servo hard limits down to the controllers, which clamp every position in EZB.Servo.SetServoPosition. No skill, script or Auto Position frame can drive a servo past what the robot's body allows. The limits are re-pushed whenever a controller connects.

Limits set by a script or a robot skill directly are not tracked here and are left alone.

23. Utilities

ONNX Runtime

ARCx ships Microsoft ONNX Runtime (the CPU build) for skills that run neural networks - image embeddings, detection, speech, voices. ARCx itself does not use it; it is there so every skill shares one copy. Reference Microsoft.ML.OnnxRuntime.dll from the ARCx folder with Private=false (the line is in the template's References comment). Add no Microsoft.ML.OnnxRuntime* NuGet package and copy no onnxruntime native library: the one beside ARCx.exe is already the right one for the platform.

Create sessions through OnnxAccelerationService, not with new InferenceSession directly. The user chooses the hardware once, for every skill, in Options > Preferences > Acceleration, and the service puts the session there:

var acceleration = ServiceRequestor.GetRequiredService<OnnxAccelerationService>();

// Uses the user's choice (Auto picks the fastest that works), and falls back to the processor
// when the graphics card cannot start. configure runs once per attempt.
var created = acceleration.CreateSession(modelPath, (options, accelerator) => {

  options.IntraOpNumThreads = 4;
});

_session = created.Session;               // yours to dispose
log($"Running on {created.Description}"); // "WebGPU (NVIDIA GeForce RTX 4070)" or "CPU"

if (created.FallbackReason != null)
  log(created.FallbackReason);

Pass accelerator: OnnxAcceleratorEnum.Cpu to keep a model off the graphics card (small models, or a skill setting that forces the processor). For SessionOptions you build yourself, call ResolveAccelerator() and then AppendAccelerator(options, accelerator). Never call AppendExecutionProvider_DML or AppendExecutionProvider_CUDA: the runtime ARCx ships has neither built in, and graphics cards only work through the service's plugins. The choice is read when a session is created, so a skill picks up a change the next time it loads its model.

• Keep the ONNX Runtime types in a class of their own and create it inside a try. An older ARCx without ONNX Runtime then fails there with FileNotFoundException naming Microsoft.ML.OnnxRuntime, which the skill can report, rather than failing to load at all.
• Models are large. Download them on first use into a folder of your own under Constants.DEFAULT_FOLDER, which survives a skill update, and check a checksum before using one.
• Leave cores for the rest of ARCx: set SessionOptions.IntraOpNumThreads to a few, not all.
• Run inference on a thread of your own, never in a video event handler or a Blazor event. See Background threads.

Background threads

Work that loops for as long as the skill runs - reading a serial port or a HID device, polling a sensor, running a model - gets a dedicated thread with a name. Thread pool threads all show up as .NET TP Worker in a debugger's Threads window, a profiler and the OS task manager, so when ARCx is busy nobody can tell which skill is using the processor. A named thread says so.

Thread _readThread;

public override void Initialize() {

  _readThread = new Thread(readLoop) {
    IsBackground = true,
    Name = $"{Properties.WindowTitle} - Reader",
  };

  _readThread.Start();
}

void readLoop() {

  while (!IsDisposing && !IsDisposed) {

    // An exception that escapes a thread of your own ends ARCx, not just the skill
    try {

      if (AllWorkersPaused) {

        Thread.Sleep(100);

        continue;
      }

      // ... the work
    } catch (Exception ex) {

      ARCLogService.AddLogEntry(this, LogEntryCls.SeverityEnum.Error, MethodBase.GetCurrentMethod(), ex);

      Thread.Sleep(1000);
    }
  }
}

• Name it after the window title and the job, so two copies of the same skill are told apart.
• Always IsBackground = true, or the thread keeps ARCx running after the user closes it.
• Catch everything inside the loop. An unhandled exception on a Thread terminates the process.
• Task.Factory.StartNew(loop, TaskCreationOptions.LongRunning) also gets a thread of its own, named .NET Long Running Task. Set Thread.CurrentThread.Name as the loop's first line.
• Do not loop forever in Task.Run. It holds a thread pool thread that Blazor, timers and every other skill share, and it cannot be told apart from the rest.
• Do not rename a thread pool thread (inside Task.Run, a timer tick or after an await). The pool keeps the name after your work has finished, so it then points at the wrong skill. Short work belongs on the pool, unnamed; find it with a CPU profiler, which groups by call stack.

TimerSmart

ARCx.Core.TimerSmart.TimerSmart, a System.Timers.Timer that does not re-enter: a tick that arrives while the previous handler is still running is dropped, not overlapped.

public override void Initialize() {

  _timer = new TimerSmart();
  _timer.Elapsed += _timer_Elapsed;
  _timer.Interval = 1000;
}

protected override void DisposeOverride() {

  // Unsubscribe BEFORE disposing - an elapsed event can already be queued on a thread pool
  // thread and would otherwise run against a disposed worker
  if (_timer != null)
    _timer.Elapsed -= _timer_Elapsed;

  _timer?.Stop();
  _timer?.Dispose();
  _timer = null;
}

Elapsed hides the base event - subscribe to this one. IsEventRunning, IsDisposing, and the base Enabled. Start() is ignored while already running; both Start() and Stop() are ignored once disposing.

The handler's first two checks are always the same:

// The skill may have been removed while this event was queued
if (IsDisposing || IsDisposed)
  return;

// A configuration window is open somewhere, so hold off
if (AllWorkersPaused)
  return;

DisposableBase

ARCx.Core.Disposable.DisposableBase. IsDisposing is set before DisposeOverride() runs, IsDisposed after. Implement protected abstract void DisposeOverride().

ConcurrentList

ARCx.Core.ConcurrentList.ConcurrentList<T>. A thread-safe list, used throughout ARCx where a collection is written from skill threads and read from Blazor circuits.

Common

ARCx.Core.Common (static). Serialization (SerializeObjectString, SerializeObjectCompressed, DeserializeObjectFile, …), filename and path safety (GetSafeFilename, GetSafeSanatizedString, CombinePathSafe, CombinePath), string parsing (IsNumeric, IsInteger, GetNumbersFromString, GetDecimalFromString, TitleCase, SplitOnlyOnce, GetTextBetween), networking (GetIPAddresses, GetWebResponseV2, PostWebResponse, IsInternetConnected), and GetOperatingSystem().

ARCx.Core.Extensions adds DateTimeEx, NumericEx, StringEx, MemoryStreamEx, StreamToByteArray.

24. Gamepads, joysticks and HID devices

ARCx.Services.InputDevices.InputDeviceService. Game controllers (Xbox, PlayStation, Switch, 8BitDo, flight sticks, wheels - anything SDL recognizes, over USB or bluetooth) and raw HID devices (custom USB boards, sensors, button boxes), on Windows, Linux and macOS.

The service owns SDL3 and is the only thing in ARCx that touches it. Do not ship SDL, or any SDL binding, in a robot skill. Every skill loads into its own assembly load context, so a skill's SDL types would not match the host's, and a second SDL_Init in the process fights ARCx for the devices. The public API uses only ARCx types, so a skill needs no package reference at all and stays one AnyCPU build.

The user can turn the service off (Views, Gamepads And HID). Check IsRunning, and listen to OnRunningChanged if you show its state.

Controllers

// Initialize(): subscribe. DisposeOverride(): unsubscribe, or the plugin assembly is pinned.
_inputDeviceService.OnGamepadAdded += inputDevice_OnGamepadAdded;
_inputDeviceService.OnButtonChanged += inputDevice_OnButtonChanged;

// Save the DeviceKey in your configuration; the InstanceId changes every time it reconnects
var gamepad = _inputDeviceService.FindGamepad(_Configuration.DeviceKey);   // "" = first connected

// From a timer: a copy of the current state, never waits on the controller
var state = _inputDeviceService.GetState(gamepad.InstanceId);
float forward = -state.GetAxis(GamepadAxisEnum.LeftY);                     // up is negative
bool jump = state.IsPressed(GamepadButtonEnum.South);                      // A / Cross / B

Buttons and axes are named by position, not by the letter printed on them: South is A on Xbox, Cross on PlayStation and B on Nintendo. GamepadNames.GetButtonLabel(button, type) gives the printed label for display; GamepadNames.TryParseButton() accepts both kinds of name. Sticks read -1 to 1 with up and left negative, triggers 0 to 1.

GamepadInfo: InstanceId, Name, IsGamepad, Type (XboxOne, PS5, SwitchPro, ...), JoystickType, VendorId, ProductId, Serial, DeviceKey, Connection, RawAxisCount, RawButtonCount, HatCount, HasRumble, HasTriggerRumble, HasRgbLed, HasAccelerometer, HasGyro. GamepadState: Axes, Buttons, ButtonPressCounts, RawAxes, RawButtons, Hats, Accelerometer, Gyro, BatteryPercent, PowerState, LastInputUtc.

Events are raised on the service's own dispatch thread, in order. A slow handler delays the other handlers, not the controllers, but keep handlers short: start a script or set a flag. Use RefreshUI() from them as usual. OnAxisChanged fires hundreds of times a second while a stick moves and is dropped when handlers fall behind - to drive something from a stick, poll GetState() on a timer instead. A handler that throws is logged, rate limited, and does not stop the others.

HID devices

// Look the device up by VID:PID each time; the path is not stable across ports and reboots
using var device = _inputDeviceService.OpenHidDevice(0x1234, 0xABCD);

device.Write(new byte[] { 0x00, 0x01, 0x02 });        // byte 0 is the report ID, 0 if unnumbered
var report = device.Read(64, 100);                    // empty array on timeout

Nothing in ARCx lists HID devices until something asks, because listing every HID device is what can make macOS ask for Input Monitoring permission. Dispose every connection in DisposeOverride(). The service closes any still open when it stops, and they report IsOpen == false from then on.

A read and a write can run on different threads at once; two reads cannot. Do not do blocking reads on a UI or event thread - give them a named thread of their own (see Background threads), and end its loop on IsDisposing.

25. Browser windows: web pages, dashboards, video calls and kiosks

ARCx.Services.BrowserPresentation.BrowserPresentationService. Chromium windows on the monitors plugged into the computer ARCx runs on - a robot face, a dashboard, a WebRTC video call, a kiosk screen - on Windows x64, Linux x64 and Linux ARM64. Not to be confused with the browser showing the ARCx interface, which can be anywhere on the network.

On Linux no desktop environment is needed: a bare X server is enough, and the window is placed and made fullscreen without a window manager. INSTALL, Browser windows, has the setup.

Do not ship CEF, CefSharp, CefGlue, WebView2 or any other Chromium embedding in a robot skill. CEF can be initialized once per process, and ARCx runs it in a browser host process of its own anyway (below). The public API uses only ARCx types, so a skill needs no package reference and stays one AnyCPU build.

How it works

The service starts a browser host process (the ARCx executable again, with --arcx-browser-host) the first time a window is opened, and talks to it over a pipe. Chromium's crashes stay in that process: the service restarts it and reopens every window, raising OnWindowClosed (Reason = HostExited, WillReopen = true) and then OnWindowOpened again. The host stops two minutes after the last window closes.

Starting the host takes a second or two the first time (several on a Raspberry Pi). Never call OpenWindow from a Blazor render or event handler without Task.Run. Every other call blocks only for the few milliseconds the host takes to answer.

Opening a window

// Initialize(): nothing to create - windows open when you need them.
// Somewhere off the UI thread:
_window = _browserPresentationService.OpenWindow(new BrowserWindowOptions {
  Name = $"{Properties.WindowTitle} Face",          // global across ARCx - make it yours
  Url = "http://127.0.0.1:5000/_dynamic/" + Properties.InstanceGuid + "/face.html",
  DisplayMode = BrowserDisplayModeEnum.Kiosk,
  Monitor = _Configuration.Monitor,                 // -1 = primary
  AllowCamera = true,
  AllowMicrophone = true,
  OwnerTag = Properties.InstanceGuid,               // ARCx closes it if the skill is removed
});

_window.OnPageMessage += window_OnPageMessage;
_window.OnClosed += window_OnClosed;

// DisposeOverride(): unsubscribe, then close.
_window.OnPageMessage -= window_OnPageMessage;
_window.OnClosed -= window_OnClosed;
_window.Close();

OpenWindow with a name that is already open re-points that window (new Url, new options) and returns the same BrowserWindow, rather than opening a second one. Window names are shared by every skill and script in ARCx, so prefix yours with the skill's window title.

A page the skill serves itself fits well with OnDynamicResourceRequest (UI manual, UCDynamicImage): serve face.html from /_dynamic/[instance guid]/... and point the window at it. ARCx binds to 127.0.0.1 by default, which the browser host can always reach.

Kiosk is Fullscreen plus: always on top, no context menu, no keyboard shortcuts (Alt+F4 included), no developer tools, JavaScript alert logged and dismissed and confirm answered yes, the retry page when offline, the display kept awake, and the window reopened if closed on the screen. Close a kiosk with Close() - that one is final.

Controlling a window

BrowserWindowState: Url, Title, IsLoading, CanGoBack, CanGoForward, IsVisible, DisplayMode, Monitor, X/Y/Width/Height, ZoomPercent, IsMuted, IsCameraInUse, IsMicrophoneInUse, LastHttpStatus, LastLoadError, CrashCount, OpenedUtc, OwnerTag.

BrowserWindow events: OnStateChanged, OnLoadCompleted, OnLoadFailed, OnPageMessage, OnConsoleMessage, OnNavigationBlocked, OnCrashed, OnClosed.

Events are raised on the service's own dispatch thread, in order. A slow handler delays the other handlers, never the browser, but keep them short and use RefreshUI() from them as usual. A handler that throws is logged, rate limited, and does not stop the others. Handle OnClosed: a window closed with WillReopen == false is gone, and its BrowserWindow stays closed - open it again for a new one.

The page side: window.ARCx

Every document the window loads - reloads, navigations, iframes - gets a window.ARCx object before its own scripts run, so a page can use it at the top level:

<script>
  // Robot to page: BrowserWindow.SendMessage("mood", "happy") / Browser.sendMessage(...)
  ARCx.on('mood', function (data) { document.body.className = data; });

  // Any topic
  ARCx.on('*', function (data, topic) { console.log(topic, data); });

  // Page to robot: raises OnPageMessage, answers WaitForMessage / Browser.waitForMessage(...)
  document.getElementById('wave').onclick = function () { ARCx.send('command', { action: 'wave', speed: 5 }); };

  // Also delivered as a DOM event, for frameworks
  window.addEventListener('arcx-message', function (e) { console.log(e.detail.topic, e.detail.data); });
</script>

window.ARCx only exists in an ARCx browser window; guard with if (window.ARCx) so the same page still works in an ordinary browser. Data is anything JSON.stringify accepts. On the C# side, BrowserPageMessageEventArgs has Topic, DataJson, DataAsString (a JSON string unquoted) and GetData<T>(). Every frame in the window can call ARCx.send, including third-party iframes, so treat page messages as user input.

Scripts, Blockly and the Browser Windows page

Everything here is also the Browser script namespace (JavaScript Browser.open(...), Python Browser.Open(...)) and the Browser category in Blockly, built on the same service - a skill and a script can share a window by name. Views, Browser Windows lists the open windows with live controls, the monitors, and the service settings (GPU acceleration, one renderer process, low end device mode, the Linux X display, the remote debugging port). The Diagnostics page has a Browser Windows section, including the missing Linux libraries when Chromium cannot start.

26. Depth cameras

ARCx.Services.DepthSensing.DepthSensorService. The meeting point between robot skills that produce depth (an Intel RealSense, Orbbec, OAK-D, Kinect, ZED, a ROS depth topic) and everything that wants to use it, the way VideoRouteService is for pictures and KinematicsService is for arms. A depth camera skill registers once and hands over every depth frame; any skill can then ask how far away something is, follow the depth live, or build its own maps from it, without knowing which camera it is.

It also does one thing for every vision skill at no cost to them: a detection made on a video source that a depth camera is aligned with (its pixels line up with the depth) is measured from the depth before OnObjectsDetected is raised. Face, colour, glyph and object detections then carry their real distance and position, and the Object Detection Script robot skill can stop the robot at a distance or drive to the object.

Frames, units and frames of reference

Everything follows the NMS frame of section 14: metres and radians, +X forward, +Y left, angles counter-clockwise.

Providing depth

// Once configured (and again with UpdateSensorDetails when the mount or the aligned source changes)
_sensorGuid = _depthSensorService.AddSensor(new DepthSensorRegistration() {
  Name = Properties.WindowTitle,
  Description = "Orbbec Astra (serial 1234)",
  ProviderName = Properties.Title,
  AlignedVideoSourceName = $"{Properties.WindowTitle} Color",   // empty when the depth is not aligned
  Mount = new DepthSensorMount() { HeightM = 0.35f, PitchRad = 0.2f },
  MinRangeM = 0.3f,
  MaxRangeM = 6f,
});

// Every frame, on the capture thread
_depthSensorService.UpdateDepthFrame(_sensorGuid, new DepthFrame() {
  Width = w, Height = h, Data = copyOfTheDepth, DepthUnitM = 0.001f, Intrinsics = k, FrameNumber = n,
});

// DisposeOverride()
_depthSensorService.RemoveSensor(_sensorGuid);

Only register an aligned video source while the depth really is aligned to it pixel for pixel (same field of view; any resolution, the rectangle is scaled across). A detection on an unaligned source would be measured at the wrong place.

Turning depth into an NMS scan

DepthScanBuilder.Build(frame, settings, treatCliffsAsObstacles) is the shared way a depth camera feeds the navigation messaging system, so every camera behaves the same. It places every sampled pixel in the robot's frame with the mount, removes the floor by height (MinObstacleHeightM), keeps overhangs up to the robot's height that a lidar would miss (MaxObstacleHeightM, a table top), and with DetectCliffs finds the floor dropping away (stairs, ledges) and reports where the floor ends. The result holds the nearest obstacle and drop per slice of the circle (ObstacleRangeM, CliffRangeM, SliceWidthRad, SliceOf(), SliceAngle(), NearestBetween(start, end, includeCliffs)), the field of view edges, and Points, ready for NMSService.UpdateScan(). Without a mount height there is no floor to measure from and everything seen is reported, like a lidar. It is pure and thread safe; about 15 ms for 100,000 pixels on a desktop, and it samples to TargetSamples (30,000) by default.

A depth camera skill still publishes a placeholder pose when nothing else reports one, exactly as section 14 describes for a lidar.

Using depth

DepthFrame also has GetDepthM(x, y), Deproject(x, y, depthM), TryDeproject(), GetDepthInRectM(), GetValidFraction() and WritePly(stream, mount, step, colorBgr) for a point cloud MeshLab, CloudCompare, Blender and Open3D open. DepthVisualizer.Colorize(frame, minM, maxM) makes the near-red, far-blue picture to publish as a video source.

The attributes a measured detection gains (DetectedObject.ATTRIBUTE_*): DistanceM (from the lens, replacing a detector's estimate), ForwardM, LeftM, HeightM, RangeM, Angle and DepthSensor. A detection with no depth reading inside its box keeps the attributes its detector gave it.

27. Robot Bus: topics, services and frames

ARCx.Services.RobotBus.RobotBusService. A publish/subscribe message bus, named request/response services and a tree of frames, shared by robot skills, scripts (the Bus namespace), the MCP server and programs outside ARCx (WebSocket and REST at /api/bus, Python and JavaScript clients at /sdk/). The user manual is Manuals/Third-Party Integrations/robot-bus.md.

Use it when what your skill produces or offers should be reachable by things you cannot name in advance: another skill, a Python program on another computer, a ROS node through the ROS Connector, an AI assistant. The typed core services above (video, NMS, kinematics, depth) stay the way to plug into ARCx's own features; ARCx already publishes their data on /arcx/... topics, so do not publish it again.

Conventions

• Names look like ROS names: /my_skill/battery. Put your skill's topics under / + BusNames.Segment(Properties.WindowTitle) so two copies of the skill do not collide.
• Payloads are JSON. Publish a record or anonymous object; it is serialized once with camelCase names. Give a type (free text, such as my_skill/Battery) and advertise the topic with a description.
• Units are the NMS's: metres, radians, +X forward, +Y left, +Z up, angles counter-clockwise (ROS REP 103).
• The source of everything you publish or provide is your window title.

Publishing and subscribing

// Initialize()
_topic = $"/{BusNames.Segment(Properties.WindowTitle)}/battery";
_robotBusService.Advertise(_topic, "my_skill/Battery", "Pack voltage (V) and charge (0..1)", Properties.WindowTitle);

_subscription = _robotBusService.Subscribe("/arcx/speech/recognized", onHeard, new BusSubscribeOptions() {
  Owner = Properties.WindowTitle,
  QueueDepth = 16,       // oldest dropped when the callback falls behind
  MaxRateHz = 0,         // above 0: the newest message per topic at most this often
  DeliverLatest = false, // true: the current latest message straight away
});

// Whenever there is something new, on any thread
_robotBusService.Publish(_topic, new { voltage = 12.1, charge = 0.82 }, "my_skill/Battery", Properties.WindowTitle);

void onHeard(BusMessage message) {

  // On the subscription's own task, one message at a time. Never block for long.
  var text = message.GetField("text") as string;
}

// DisposeOverride()
_subscription?.Dispose();

BusMessage: Topic, Type, Sequence (per topic, from 1), StampUtc, Source, Data (a detached JsonElement), DataJson, GetData<T>(), GetPlainData(), GetField("pose.x"). Patterns: * one segment, ** any number. GetLatest(topic), GetTopic(topic) (rate, count, publishers), GetTopics(pattern) and WaitForMessageAsync(pattern, timeout, ct) read without subscribing.

Providing and calling services

_service = _robotBusService.RegisterService(
  $"/{BusNames.Segment(Properties.WindowTitle)}/calibrate",
  "Calibrate the sensor. Args: samples (int, default 50). Returns the offsets.",
  async (request, ct) => {

    var samples = request.GetInt("samples", 50);

    return await calibrateAsync(samples, ct);
  },
  Properties.WindowTitle,
  readOnly: false,
  argsSchema: "{\"type\":\"object\",\"properties\":{\"samples\":{\"type\":\"integer\"}}}");

// DisposeOverride()
_service?.Dispose();

// Calling one (never throws for a failed call)
var result = await _robotBusService.CallServiceAsync("/arcx/speech/say", new { text = "Calibrated" }, Properties.WindowTitle, TimeSpan.FromSeconds(5), ct);

if (!result.Ok)
  Log(result.Error);

The handler returns any value (serialized to JSON) and throws to fail the call; the exception message is what the caller sees. Honour the token: it is cancelled when the caller's timeout runs out. readOnly: true marks a service that changes nothing, which external programs may call even when Preferences forbid them to control the robot. A service name has one provider; registering a taken name throws.

Frames

// A camera bolted 10 cm forward, 30 cm up, tilted 0.2 rad down
_robotBusService.SetTransform("base_link", "my_camera", 0.10, 0, 0.30, 0, 0.2, 0, isStatic: true, source: Properties.WindowTitle);

// Where is it from the odometry frame?
if (_robotBusService.TryLookupTransform("odom", "my_camera", out var t, out var error)) {

  var (x, y, z) = t.Apply(1.5, 0, 0);   // a point 1.5 m in front of the camera, in odom
}

BusTransform is a translation and a unit quaternion: FromEuler, FromQuaternion, Then, Inverse, Apply, Rotate, Roll, Pitch, Yaw, Distance. ARCx keeps odom → base_link from the NMS pose and base_link → <depth sensor> from each depth camera's mount. Changes are published on /tf and /tf_static.

Recording

_robotBusService.Recorder records topics to ARCx Data\Recordings and plays them back (StartRecording(patterns, name), StopRecording(), ListRecordings(), StartPlayback(name, speed, loop, patterns, topicPrefix), StopPlayback()). A skill that consumes topics works on a replay exactly as on live data, which is the point: record once on the robot, debug at the desk.

28. Rules that will bite you

1. Create nothing in the constructor. Timers, executors, service registrations and event subscriptions all go in Initialize().
2. Unsubscribe from every event and dispose everything in DisposeOverride(). A leaked handler pins the plugin assembly; ARCx then cannot overwrite the installed files and the skill will not rebuild until ARCx is restarted.
3. Unsubscribe before disposing a timer, not after. A queued elapsed event runs against a disposed worker otherwise.
4. Check IsDisposing / IsDisposed and AllWorkersPaused at the top of every timer tick and background loop.
5. Look up other robot skills only in OnProjectLoadCompleted(), never in Initialize() or OnSetConfiguration().
6. Null-check TryGetEZB() and IsConnected before every hardware call. A project can reference a controller that is no longer there, and a disconnected board silently does nothing at best.
7. Save names, not guids. A video source guid and a kinematic chain guid change every session. Names and window titles survive a save and load.
8. Never hold a Mat past the end of a video event handler. Clone what you need.
9. Deregister what you registered: RemoveSource(), RemoveChain(), DeregisterInputSource(), RemoveAndDisposeExecutor() - all in DisposeOverride().
10. RefreshUI(), not StateHasChanged() on the worker. It is throttled, thread safe, and reaches the Custom UI widgets too.
11. Give every configuration property a /// summary. It is what Athena reads.
12. Build AnyCPU. A robot skill is platform-agnostic managed code and ARCx loads the same build on Windows, Linux and macOS. Never add a platform-specific code path or build: dotnet build "<Project>.csproj" -c Debug -p:Platform=AnyCPU. Name the .csproj: a solution file beside it (Visual Studio saves one) spells the platform "Any CPU", and dotnet build given only the folder picks the solution and fails with MSB4126.
13. Do not log from a tight loop. Every ARCLogService entry is written to disk.
14. Do not ship SDL. Controllers and HID devices come from InputDeviceService; a second copy of SDL in the process takes the devices away from ARCx and from every other skill.
15. Do not ship a Chromium embedding (CEF, CefSharp, CefGlue, WebView2). Browser windows come from BrowserPresentationService, which runs Chromium out of process for you. Open windows off the UI thread, set OwnerTag to your instance guid, and close them in DisposeOverride().
16. Reference the ARCx libraries you use, and only those. A skill compiles against the References item group in its .csproj: ARCx, MudBlazor and DeepCloner to start with. For another library ARCx ships - OpenCvSharp for camera frames, NAudio.Core for audio, Microsoft.ML.OnnxRuntime for neural networks, System.IO.Ports for serial ports, System.ServiceModel.Primitives for ARCx's web service clients and the rest listed in the template's comment - copy its line out of that comment. Each one is Private=false, so nothing is copied into the output; ARCx already has it loaded. CS0246 or CS0234 on one of its types, or CS0012 naming an assembly, means the reference is missing. A library ARCx does not ship is a NuGet PackageReference.
17. Do not ship ONNX Runtime. ARCx ships it, native library included, for every platform - see section 23. Only one onnxruntime native library can be loaded into a process, so a skill carrying its own fights ARCx and every other skill that runs a model.
18. Give a long-running loop its own named background thread, never a Task.Run that never returns. Name it after Properties.WindowTitle and the job so a profiler or debugger shows which skill is using the processor, and catch everything inside it - see Background threads.

Where to look next

• ARCx.xml - beside ARCx.dll, in the folder this copy of ARCx runs from. The full XML documentation for every public class, method, property and event, generated from the platform's own source. It is around a megabyte, so search it rather than reading it: every entry is a <member name="..."> element whose name is the fully qualified member, prefixed T: for a type, M: a method, P: a property, E: an event, F: a field. Grep T:ARCx to list the types, M:ARCx.Core.EZBManager for the manager's methods, VideoRoute for everything about cameras.
• RobotSkill-UI-Components.md - the visual side: the component base classes, layout, the UC* components, Custom UI widgets.