Camera Device

Main Window - Device

This camera device is either connected or built into your computer or your robot. It can be an integrated webcam, USB camera, serial camera, wireless camera, etc.

Additional camera devices may be selected or entered when the camera is stopped (not running). If the EZB supports video, the robot skill will display the IP Address and any other USB or local cameras. The camera robot skill will also display cameras for other supported robots, such as the AR Parrot Drone. Additionally, you may manually enter a JPEG Snapshot HTTP address.

USB Camera Device - Clicking on the video device list will display USB cameras and capture device. Select the USB device to use.

EZB Camera Device - If the EZB supports a video stream, the syntax for connecting to EZB cameras is EZB://192.168.0.1 or EZB://192.168.0.1:24 (if port is specified). The type of EZBs that support video for this syntax are ESP32, EZ-B v4, IoTiny, Raspberry Pi, etc..

COM (UART) Device - If a camera is connected via a USB COM UART adapter, the port will be displayed in the drop down. The syntax will be COMx (i.e. COM1, COM2, and so on). The default baudrate will be 3,333,333 if only the com port is selected. However, if the camera uses a different baudrate, it can be specified next to the com port with a : separator. Such as COM1:9600

Custom - This option allows the camera device robot skill to be in slave mode. It allows other robot skills to push video data to this robot skill. Some robot skills, such as the AR Parrot Drone, Intel Realsense, and Stream All The Video Sources will use this option.

JPEG Snapshot HTTP Video Device - a camera or video source which provides JPEG images may be selected as a camera device. Enter the complete HTTP URL of the video source. If the HTTP video source requires authentication, consult your device's documentation for correct syntax. You may test the JPEG URL in a web browser to verify that it is valid and that only the JPEG is displayed, not any HTML. Example: http://192.168.100.2/cgi-bin/snapshot.cgi or http://192.168.100.2/image.jpg, etc..

Main Window - Tracking

A tracking type is a function of what to recognize for tracking. For example, you would only select the Color tracking type to track the color RED. Each tracking type has configuration settings that are on its tabs. If you have Color checked as a tracking type, then the configuration for the Color tracking type is found in the tab labeled Color.

1. Tracking Type Checkboxes
These checkboxes will enable/disable the different tracking types. You can track more than one type at a time but it's not often used in this way.

Main Window - Color Tracking

This will track the specified predefined color from the Color tab. You can select the color between Red, Green, or Blue. The Color tab will also give you settings to adjust the brightness and size of the object. Hold the object in front of the camera while changing the settings. This is the basic color tracking method used in RoboScratch's "wait for color."

1. Detection Boxes
These boxes show where the selected color has been detected on the camera image.

2. Color Drop-down
This drop-down give you the selection of 3 colors (Red, Green, and Blue).

3. Minimum Object Size Slider
This slider allows you to select how large the detected colored object size must be before registering as a detection.

4. Example Minimum Size Display
This display gives you a visual reference of how large (in pixel size) the detected object will have to be in the camera image display.

5. Object Brightness Slider
This slider will adjust the brightness of the detected color. Adjust until only the desired object is detected.

Main Window - QR Code Tracking

You can use this tracking type to detect pre-encoded QR codes or the QR Code Generator (Add Skill -> Camera -> QR Code Encoder) to create your QR Codes and print them out. QR Code Tracking will detect QR codes, set the $CameraQRCode variable, or execute the code within the QR Code. Keep in mind that QR Code recognition is CPU intensive and therefore slow. We do not recommend using QR codes for tracking. It is best used for identification and triggering events, not movement tracking. Note that the QR Code tracking type has additional settings in the Camera Device Settings.

1. QR code Detection Output
This is where the detected output from the QR code will be displayed.

2. QR Code
This is the QR code that will be detected on the camera image display.

3. QR Code Checkbox
This checkbox will need to be enabled to track QR codes.

4. Camera Frame Rate
This displays the camera frame rate in frames per second (fps) and also shows the number of skipped frames.

Main Window - Multi Color Tracking

This is an advanced and more detailed tracking method over the generic Color tracking type. With Multi-Color, you can specify and fine-tune your color and add multiple colors. The camera robot skill will configure an ez-script variable to hold the name of the current detected color that has been detected. The multi-color definitions can be trained in the Multi-Color tab. If you name a custom color "Red," this does not override the basic predefined color tracking types. The basic color tracking (discussed above) has three predefined colors. This multicolor tracking type is separate and has no relationship to predefined colors.

1. Detection Boxes
These boxes show where the selected colors have been detected on the camera image.

2. Add Button
This button adds a color to the detection list.

3. Edit button
This button will open the Custom HSL Color window to edit/adjust the color you would like to detect. The main items that you will be adjusting are the color name field, the two color wheel wipers, the saturation min slider, and the luminance max slider. Here's how the Custom HSL Color window looks:

4. Enable Checkbox
This checkbox allows you to enable/disable the color that you have added to the detection list.

5. Color Name
This is a text display of the name that you have chosen in the Custom HSL Color window.

Main Window - Face Tracking

Face tracking will attempt to detect faces within the image. Face tracking uses calculations to detect eyes and a mouth. This does require additional processing and will slow the framerate. Also, detecting images within a video stream with many complicated objects will return false positives. Use Face Tracking against a white wall or non-complicated background. If you wish to detect specific faces, use the Object Tracking type, as you can train your face as an object.

1. Detection Box
This box shows where a face has been detected on the camera image.

2. Face Tracking Checkbox
This checkbox will need to be enabled to track faces.

3. Camera Frame Rate
This displays the camera frame rate in frames per second (fps) and also shows the number of skipped frames.

Main Window - Custom Haar Tracking

Unless you have experience with computer vision, ignore this tracking type. In brief, this tracking type is used for computer vision specialists who generate HAAR Cascades. This is very CPU intensive and will most likely bring a slower computer to a halt. If the input cascade is not optimized, it will greatly affect framerate processing performance. Leave this option to the experts :)

1. Detection Box
This box shows where a custom haar cascade has been detected on the camera image.

2. Custom Haar Tracking Checkbox
This checkbox will need to be enabled to track custom haar cascades.

3. Camera Frame Rate
This displays the camera frame rate in frames per second (fps) and shows the number of skipped frames.

Link to OpenCV Haar Cascades: XML Files

Main Window - Object Tracking

This is an advanced computer vision learning tracking type that gives you the ability to teach the robot an object (such as a face, logo, etc.) and have it detect it. Computer vision learning is very experimental and requires patience and consistent lighting.

*Note: For best results with object training, consider that you are teaching the robot specific details of the object, not the object itself. This means the entire object does not need to be in the training square. Only details unique to that object need to be in the training square. For example, you may not wish to teach the robot a can of soda/cola by putting the entire can in the square. Merely put the logo in the square or any other identifying features.

1. Detection Box
This box shows where a trained object has been detected on the camera image.

2. Train New Object Button
This button will open the Custom Object window to train new custom objects you would like to detect. To train a new object, first enter the name of the object, then click in the camera image area to create a pink box, place your object in front of the camera in such a way that a unique detail of your object fills the box, click the learn selected area button, and then tilt your object slowly at different angles until the learning is complete. Here's how the Custom Object window looks:

3. Clear Memory Button
This button removes all the trained objects from the trained images window.

4. Learn While Tracking Checkbox
This checkbox enables the object learning module to keep learning the object being tracked. Be aware that if this checkbox is left enabled, there is a chance that, over time, the module will start learning the objects around the tracked object.

5. Show Movement Tracer Checkbox
This checkbox enables the trajectory tracer to be displayed as the object is moved.

6. Trained Images Display
This displays the objects that have been trained in the train new object window.

Main Window - Motion Tracking

This observes changes within the camera image. The motion should not be confused with the "Movement" Setting from the configuration menu, as they are different things and should not be used together. The Motion Tracking Type will detect a change in the camera image and return the change area. For example, if your camera is stationary and you wave your hand in a small area of the camera image, you will see the motion display of your hand moving. If a robot moves during Motion Tracking Type, the entire image would have been considered changed, which is not helpful for tracking. So, the Motion Tracking Type is only beneficial for stationary cameras.

1. Detection Box
This box shows where motion has been detected on the camera image.

2. Motion Sensitivity Slider
This slider adjusts the amount of motion needed for motion to be detected.

3. Minimum Object Size Slider
This slider allows you to select how large the detected motion object size must be before registering as a detection.

4. Example Minimum Size Display
This display gives you a visual reference of how large (in pixel size) the detected object will have to be in the camera image display.

5. Check Every (frames) Drop-down
This drop-down allows you to select how many frames to wait between detection samples.

Main Window - Glyph Tracking

This tracking type will look for images of black and white squares. The camera robot skill will use a set of specific glyphs (1, 2, 3, and 4) for this tracking type. This tracking type will detect those glyphs if you download and print the glyph PDF in the link below. You may also visit the Camera configuration to set up augmented reality overlays on the glyphs. This means the camera will superimpose a selected image over the detected glyph. When tracking glyph images, the glyph will only execute its respective tracking script once until another glyph has been detected or the ClearGlyph control command has been called. To see all available controlcommand(), press the Cheat Sheet tab when editing scripts.

Glyph Downloads: Glyph PDF

1. Detection Box
This box shows where the glyph has been detected on the camera image.

2. Glyph Tracking Checkbox
This checkbox will need to be enabled to track glyphs.

3. Camera Frame Rate
This displays the camera frame rate in frames per second (fps) and also shows the number of skipped frames.

Main Window - YCbCr Tracking

This tracking type is much like multi-color. You can specify and fine-tune your color and add multiple colors. While multi-color uses classical RGB values, YCbCr uses blue-difference and red-difference chroma (Cb and Cr) and luma (Y) components.

1. Detection Box
This box shows where the glyph has been detected on the camera image.

2. Add Button
This button adds a color to the detection list.

3. Edit Button
This button will open the Custom YCbCr Color window to edit/adjust the color you would like to detect. The main items that you will be adjusting are the color name field, the Cb sliders, the Cr sliders, and the Y sliders. Here's how the Custom YCbCr Color window looks:

4. Enable Checkbox
This checkbox allows you to enable/disable the color that you have added to the detection list.

5. Color Name
This is a text display of the name that you have chosen in the Custom YCbCr Color window.

Main Window - Grid

These grid lines are used for movement and servo tracking. With tracking enabled, a robot can move itself to keep the detected object in the center of the grid lines. Move the sliders to adjust the position of the grid lines on the camera image. The field of detection can be narrowed or expanded with the grid lines.

1. Grid Line Overlay
These grid lines are overlaid on the camera image and are always there, even if you make them completely transparent.

2. Grid Line Sliders
These sliders are for moving the grid lines. There are 4 sliders for each of the 4 grid lines.

3. Defaults Button
This button resets the grid lines back to their default position.

4. Grid Line Transparency Slider
This slider controls the transparency of the grid lines, from bold red to invisible.

Configuration - Servo/Movement Tracking

These settings set up how a robot reacts when an object is detected by one of the enabled tacking types. Servo, Movement, or Scripts, are the three reaction types you can enable individually or simultaneously. If servo tracking is checked, the skill assumes the camera is mounted on the specified servos. The servos will be moved from left, right, up, and down to track the detected object based on the servo settings that you have provided. The Movement reaction type is physically moving the entire robot toward an object. If Movement is checked, the robot will follow the detected object when visible using the project's movement panel. The movement panel may be an Auto Position, H-Bridge, or more. The robot will move forward, left or right, to follow the desired object. The Scripts tab of the camera device configuration menu is a Tracking Start and Tracking End script. When an object is detected, the Tracking Start script will execute. The Tracking End script will run when the camera is no longer tracking an object.

1. Servo Tracking Checkbox
This checkbox will enable X and Y-axis Servos to move to track a detected object.

2. Relative Position/Grid Line Checkboxes
These checkboxes allow the servos to track the detected object via relative position (moving to the center of the camera image) or using the grid lines (moving to wherever the center of the grid lines is set). Using relative position assumes a stationary camera is being used, and using grid lines assumes a pan/tilt mechanism is being used.

3. Servo Setup Drop-downs
These drop-downs allows for setup of the X and Y axis Servos. You can set board index, port number, min and max position limits, and multiple servos. You can also invert the servo direction with the included checkbox.

4. Horizontal/Vertical Increment Steps Drop-downs
These drop downs adjust the number of steps the servos will move when moving toward the detected object. The smaller the number the more precise the movements will be.

Configuration - Scripts

This section of the Scripts tab includes a checkbox to enable the execution of tracking scripts. You can add tracking start and end scripts as well as use drop-downs for selecting how many frames to delay by before the tracking start/end scripts execute and sorting by detected object (blob) order.

This section includes all the variables that are used with the camera skill. You can change the variable names here if you'd like. You can use these variables throughout your scripts and applications. You can view their value or state in real-time with the Variable Watch skill. There is also a drop-down for selecting the maximum number of objects to detect. Each detected object will add an underscore and number to the end of the variables to distinguish one from another.

This section executes scripts when the corresponding glyph is detected. Find more information about glyphs in the Glyph Tracking section.

In this section you can generate a list of different QR Code texts and enter in a script to execute when that certain text is detected. You can also generate a script that executes when a QR Code is detected that is not in your list. There are also buttons included to manage your list of QR Codes.

Configuration - Augmented Reality

1. Select Image Buttons
These buttons assign an image to be displayed when the corresponding glyph is detected. Find more information about glyphs in the Glyph Tracking section.

Configuration - Detection

1. Load Button
This button loads custom haar cascades (in .XML format). You can download custom cascades from OpenCV on github.

2. Minimum Size Drop-down
This drop-down sets up the minimum size the detected face (in pixels) will have to be in order to register as a detection.

3. Maximum Size Drop-down
This drop-down sets up the maximum size the detected face (in pixels) will have to be in order to register as a detection.

4. Minimum Detection Count Drop-down
This drop-down sets the minimum of detected frames that the face will have to be present in order to register as a detection. There's more information on Face Tracking above.

Configuration - Media

1. Change Button
This button allows you to change the save location on your PC that your robot took photos with the camera device skill.

2. Snapshot Quality Drop-down
This drop-down sets up the percentage of JPEG quality you would like (10 to 100%). The larger the percentage, the larger the file size will be.

3. Video Capture Codec Drop-down
This drop-down sets up the video capture compression format (WMV1, WMV2, or H263P5)

4. Bit Rate Field
This value determines the codec resolution and size of each video frame.

5. Insert Video Title Checkbox
If checked this checkbox enables the Video Title text to be show at the beginning of your recorded video.

6. Title Length Drop-down
This drop-down allows you to set how long the title will be displayed (in seconds) at the beginning of your recorded video.

7. Title Text Field
This field allows you to enter in the text title you would like to be displayed at the beginning of your recorded video. The title can also be a script variable if you'd like.

Configuration - Advanced

1. Title Field
This field allows you to change the text title of the camera device skill. *Note: Changing the title here will also change the title in the controlCommand() associated with this skill.

2. Camera Image Orientation Drop-down
This drop-down sets the orientation of the camera image. If your camera is mounted in an odd orientation, you can use this drop-down to correct how it displays the camera image on the screen. There are 16 different orientation options.

3. Stop Camera on Error Checkbox
This checkbox is for plug-in troubleshooting. It allows the camera device to stop if an error occurs and gives you to show the error in the debug window to Synthiam.

4. Frame Rate Initialization Field
This field allows you to adjust the frame rate when the camera first initializes. It's best to leave this value at -1.

How to Use Camera Device

1) Add the Camera Device skill to your ARC project (Project -> Add Skill -> Camera -> Camera Device).

2) Select the Camera you would like to use from the Video Device drop-down.

3) Select the resolution. If the camera image doesn't show up, you may have selected an unsupported resolution for that camera.

4) Press the Start/Stop Button to begin transmitting camera images.

5) Select a tracking type in the tracking tab to begin using the camera image data to track objects.

6) Select the other settings tabs or the triple dots to configure the tracking type (if applicable).

Video Resolutions and Performance

Machine vision and computer recognition is a very high cpu intensive process. The cameras for computer vision are much less resolution than what you, as a human, would use for recording a birthday party. If you were to run computer vision to recognize objects and decode frames at HD quality, your computer response would grind to a halt.

160x120 = 57,600 Bytes per frame = 1,152,000 Bytes per second
320x240 = 230,400 Bytes per frame = 4,608,000 Bytes per second
640x480 = 921,600 Bytes per frame = 18,432,000 Bytes per second

So at 320x240, your CPU is processing complex algorithm on 4,608,000 Bytes per second. Soon as you move to a mere 640x480, it's 18,432,000 Bytes per second.

To expand on that, 4,608,000 Bytes per second is just the data, not including the number of CPU instructions per step of the algorithm(s). Do not let television shows, such as Person Of Interest, make you believe that computer vision and CPU processing is as accessible in real-time, but many of us are working on it! We can put 4,608,000 Bytes into perspective by relating that to a 2 minute MP3 file. Imagine your computer processing a 2 minute MP3 file in less than 1 second - that is what vision processing for recognition is doing at 320x240 resolution. Soon as you increase the resolution, the CPU has to process an exponentially larger amount of data. Computer vision recognition does not require as much resolution as your human eyes, as it looks for patterns, colors or shapes.

If you want to record a birthday party, do not use a robot camera - buy a real camera. Synthiam software algorithms are designed for vision recognition with robotics.

Use Camera with Code

You may also instruct the Camera Device Skill to change settings programmatically through code. The ControlCommand() can be called in EZ-Script to change the Camera Device settings. Learn more about the ControlCommand() here.

Requirements

You will need at least one of the following video devices connected to your computer:

JPEG Snapshot HTTP Camera

Integrated Camera

Wireless Camera

USB Camera

EZ-B Camera

Similar Skills

Upgrade to ARC Pro

Become a Synthiam ARC Pro subscriber to unleash the power of easy and powerful robot programming

Compare Pro Features View Subscription Plans

kazpet

PRO

Australia

#1 Dec 2018 — Edited Apr 30

It appears that the GLYPH pdf are no longer available. I followed the links and got

Quote:

This synthiam.com page can’t be found

Can I get the sample pdf please?

DJ Sures

PRO

Synthiam

#2 Dec 2018 — Edited Dec 2018

Link fixed

Jakob

Austria

#3 Mar 2019

Is there also a jpeg file for the glyphs.

mecizeldec

Canada

#4 Jul 2020

Hey Guys. Dusted off an old iotiny tank project with my son. Getting this error with the camera. Using 7.4V for a IOTiny source.
Anyone seen this?

Camera Initialized: EZB://10.0.1.231:24 @ 320x240
EZ-B v4 Camera Error: System.Exception: Client disconnected
at EZ_B.EZBv4Video.aenVYfUEGW(EZTaskScheduler , Int32 , Object )
Camera Disabled

Thanks!

TMesserschmidt

PRO

USA

#5 Oct 2020

Hello,

Color tracking works. Motion tracking works. Face tracking doesn't work. Version 2020.09.08.00.

1. I selected my device/camera.
2. I set the resolution to 640x408
3. I clicked the green Start button.
4. Camera went on.
5. My face was there.
6. I selected the Tracking tab
7. I selected "Face".

Expected: bounding boxes around my face
Actual: No bounding boxes.

Is this a SOFTWARE DEFECT? Or user error?

I also tried Color and Motion. Both of those worked just fine.

Thomas Messerschmidt

DJ Sures

PRO

Synthiam

#6 Oct 2020

Face tracking works. Perhaps check the settings of the face tracking in the camera config. Also ensure correct lighting. Based on your camera resolution, you may need to tweak the settings. Use the question marks for assistance on what different settings do

also note that this thread is the camera device manual. And in this manual page are the instructions for the face tracking settings

TMesserschmidt

PRO

USA

#7 Oct 2020 — Edited Oct 2020

DJ,

[u] I found the issue:
[/u]
1. The "face detection options"/ "Max Size" adjustment needs to be set at least as large as the height of the video resolution. For example, for a resolution of 650X360, the Max Size needs to be set at 360 or more. At 640X480, Max Size needs to be set at 480 or more. if I set the Max Size less than the height of the video's height, face tracking no longer works. Also, a resolution of 1280x720 doesn't work at all because Max Size has a preset maximum of 500.

2. Also, It might be the fault of my camera, but I cannot "Start" the camera with resolutions of less than 650X360.

3. Also, ARC crashes and shuts down when I change the camera resolution more than 6 or 8 times.

Thanks for getting back to me so quickly.

Thomas

robo rad

Canada

#8 Dec 2020

So I have always used EZ camera on projects but now I want to use another USB camera for another robot,to be clear I can do all the same stuff with a better USB camera if using on Latte Panda USB port right? Object,color tracking?

DJ Sures

PRO

Synthiam

#9 Dec 2020

1

Yes - you can use a USB camera

robo rad

Canada

#10 Dec 2020

Ok thanks great to know will try it out now!

thetechguru

PRO

USA

#11 Dec 2020 — Edited Dec 2020

3

I guess this is as good a place to say this since we don't have a general discussion area of the forum and this not a question or robot comment, but I have to say I am very impressed with the improvements to documentation lately. @Djsures did you hire someone to help with documentation or just determine that enough of us were complaining that videos as the primary documentation were difficult for long term reference?

The documentation of this skill in particular is absolutely fantastic, but I have also noticed it in others. Videos are great for demonstration of capabilities, but when you want to learn how to set up and configure something, nothing beats a good user manual you can print out and mark up as needed....

Alan

DJ Sures

PRO

Synthiam

#12 Dec 2020

2

That's great to hear - jeremie will appreciate the feedback. It's what he's been working on

Dave Schulpius

PRO

USA

#13 Dec 2020

1

+1 Alan. Right there with you.

ptp

PRO

USA

#14 Dec 2020

1

Thank you Alan for pointing that out.

(*) RTFM = Read the Friendly Manual

proteusy

Portugal

#15 Dec 2020

1

Good work jeremie. Had noticed that lately documentation is fast and spot on.*cool*

DJ Sures

PRO

Synthiam

#16 Dec 2020

2

He’s got me on the kick to make sure I keep updating me content as well.

robo rad

Canada

#17 Dec 2020

1

woohoo! Yay Jer!

Jeremie

PRO

Canada

#18 Dec 2020

1

Thank you, everyone! I definitely have put a lot of work into it, I've been taking a break from it for a little bit here but I'll be back at it over the Holiday break. It definitely encourages me to hear that you appreciate the effort!

robo rad

Canada

#19 Dec 2020

It's almost like Facebook here lately with all the "Like" going on, but that's a good thing! Even with FB falling apart in the courts lately.

Login to post a comment

Upgrade to ARC Pro

How to add the Camera Device robot skill

Don't have a robot yet?

How to use the Camera Device robot skill

Main Window - Device

Main Window - Tracking

Main Window - Color Tracking

Main Window - QR Code Tracking

Main Window - Multi Color Tracking

Main Window - Face Tracking

Main Window - Custom Haar Tracking

Main Window - Object Tracking

Main Window - Motion Tracking

Main Window - Glyph Tracking

Main Window - YCbCr Tracking

Main Window - Grid

Configuration - Servo/Movement Tracking

Configuration - Scripts

Configuration - Augmented Reality

Configuration - Detection

Configuration - Media

Configuration - Advanced

How to Use Camera Device

Video Resolutions and Performance

Use Camera with Code

Requirements

Similar Skills

Camera Overlay

Augmented Reality

Camera Click Servo

Related Tutorials

Control Robot With Virtual Reality Headset

Detect Multiple Colors

The Robot Program Episode 006: Introducing ARC

The Robot Program Episode 009: Getting Six To Move

The Robot Program Episode 012: Getting Adventurebot To Move

The Robot Program Episode 020: Detect Face And Wave -...

The Robot Program Episode 021: Detect Face And Wave -...

The Robot Program Episode 022: Detect Face And Wave -...

The Robot Program Episode 023: Mobile JD

Time Lapse With A Gopro

Use Camera As A Button

Vision Training: Object Recognition

Related Hack Events

"Robot Learn A New Object"

D-0 Droid Live Hack

The Lattepanda Robot Hack

Related Questions

"The Receptionist"

2 Iotinys With Camera Power Needs

Add Second Camera Video Into Synthiam Window

Camera Device Does Not Work And Arc Freezes When I Try To Connect To The Robot/

Camera Resolution And Face Detection

Can You Make A USB Camera Work With Ezb Controller

Can't Connect To Camera Device

Detect Multiple Face From EZ Blocky Logic

Extension From Ezrobot Camera Lens To The Module

Ezbpi Server Camera Error

Help! Camera Won't Start!

How To Create More Glyphs

Interacting With The JD Robot Via Flashcards And/Or Objects

Looking For Compatible EZB Camera Cable

Maximum Allowable Length Of EZ-Robot Camera Cable

Measure Distance From Camera

Movement Tracking Not Working

Robot Doesn Mention The Color

The Camera Suddenly Disconnected

Using Variables For TY2 And Camera

Upgrade to ARC Pro

Quote: