- Artificial Intelligence
- Augmented Reality
- Camera Click Servo
- Camera Device
- Camera Overlay
- Camera Snapshot
- Camera Variable Overlay
- Cognitive Emotion
- Cognitive Face
- Cognitive Vision
- Live Stream Broadcast
- Live Stream Receiver
- Omron HVC-P
- Omron HVC-P2
- Omron HVC-P2 (Second Camera)
- Programming With Pictures
- QR Code Generator
- Sighthound Cloud API
- Target Overlay
- Train Vision Object By Script
- Vuzix Augmented Reality
- Infrared Distance
- Third Party Robots
- Touch Tablet
- Ultrasonic Distance
- Virtual Reality
- What Is a Tracking Type?
- Different Kinds of Tracking Types
- Tracking With Servos and Movements
- Video Resolutions and Performance
- Video Overlay
- Choose Video Device
- Use Camera Programmatically in Code
IntroductionIf a camera is mounted on your robot, this control can be used to track color objects such as Red Balls, track motion, track human faces, watch television, or simply view the camera image. The brightness of the detected colored object can be adjusted manually to remove any false positives. Test in a bright room with the desired colored object and adjust the brightness until only that object is detected. Computer vision is an advanced technology which requires a bright room without shadows. Computer vision is a very experimental technology, and therefore requires a clean and bright environment to accurately detect objects and colors.
What is a Tracking Type?A tracking type is the function of what to track. For example, to track the color RED, you would only select the Color tracking type. Each tracking type has configuration settings which are on their own tabs. If you have COLOR checked as a tracking type, then the configuration for the COLOR tracking type is in the tab labelled COLOR.
On the Camera Control, select the TRACKING tab. This will display a selection of tracking type checkboxes. Select one or more of the types of tracking type to detect objects. If this is your first time, start with only one at a time to learn how they work. Rarely will you track more than one type at once.
Different Kinds of Tracking Types
ColorThis will track the specified predefined color from the COLOR tab. The color can be selected between Red, Green or Blue. The Color tab will also give you settings to adjust the brightness and size of the object. Hold the object in front of the camera while changing the settings. This is the basic color tracking method which is used in RoboScratch's "wait for color".
MotionThis observes changes within the camera image. Motion should not be confused with the "Movement" Setting from the configuration menu, as they are different things and should not be used together. The Motion Tracking Type will detect a change in the camera image and return the area of the change. For example, if your camera is stationary and you wave your hand in a small area of the camera image, you will see the motion display of your hand moving. If the entire robot moves during Motion Tracking Type, the entire image would have been considered changed and that is not useful for tracking. So, Motion Tracking Type is only used on stationary cameras. Meaning, your robot is not moving.
QR CodeThe QR Code has additional settings in the Configuration/Settings menu of the Camera Control (the little gear icon). You can use the QR Code Generator (Add Control->Camera) to create your own QR Codes and print out. Otherwise, the QR Code Tracking Type will detect QR Codes and set the variable, or execute the code within the QR Code. Visit the Camera's setting page (gear icon) to configure more options for QR Codes. QR Code recognition is CPU intensive and therefore slow. We do not recommend using QR Code for tracking. It is best used for identification and triggering events, not movement tracking such as demonstrated with Multi Color or Motion, etc...
Multi ColorThis is an advanced and more detailed tracking method over the generic Color tracking type. With Multi Color, you can specify and fine tune your own color, as well as add multiple colors. An ez-script variable will be configured to hold the name of the current detected color that has been detected. The multi color definitions can be trained in the Multi Color tab. If you name a custom color "Red", this does not override the basic predefined color tracking types. The basic color tracking (discussed above), has three predefined colors. This multicolor tracking type is separate and has no relationship to those predefined colors.
FaceThe face tracking type will attempt to detect faces within the image. The Face tracking uses calculations to detect eyes and a mouth. This does require additional processing and will slow the framerate. Also, detecting images within a video stream with many complicated objects will return false positives. Use the Face Tracking against a white wall or non-complicated background. If you wish to detect specific faces, use the Object Tracking type, as you can train your face as an object.
GlyphThis tracking type will look for images that consist of black and white squares. There are set Glyphs which will be used for this tracking type. The glyph images can be found on the Camera's Manual page on the website (accessed by pressing the ? next to the X on the control). Remember, every control has a ? (question mark) next to the X (close). Pressing the ? (question mark) will bring you to the help page for the control. If you download and print the Glyph PDF from the website, this tracking type will detect the glyph. You may also visit the Camera Settings (gear icon) to configure augmented reality overlays on the glyph. This means the camera will super-impose an image over the detected glyph. When tracking Glyph images, the glyph will only execute its respective tracking script once until another glyph has been detected or the ClearGlyph control command has been called. To see all available controlcommand(), press the Cheat Sheet tab when editing scripts.
Glyph Downloads - Glyph PDF
Custom HaarUnless you hare experienced with computer vision, ignore this tracking type. In brief, this tracking type is used for computer vision specialists who generate HAAR Cascades. This is very cpu intensive and will most likely bring a slower computer to a halt. If the input cascade is not optimized, it will also have great affects on framerate processing performance. Leave this option to the experts :)
ObjectThis is an advanced computer vision learning tracking type that gives you the ability to teach the robot an object (such as a face, logo, etc.) and have it detect it. Computer vision learning is very experimental and requires patience and consistent lighting. The object training is done in the Object tab, which is settings for the object. Remember, settings for specific tracking types are configured within their tab in the Camera Control. Some tracking types have additional setting/parameters within the Camera Settings (gear icon).
*Note: For best results with object training, consider that you are teaching the robot specific details of the object, not the object itself. This means the entire object does not need to be in the training square. Only details which are unique to that object need to be in the training square. For example, you may not wish to teach the robot a can of soda/cola by putting the entire can in the square. Merely put the logo in the square, or any other identifying features.
Tracking with Servo or MovementsThe above section describes tracking types, that is what the robot is looking for. This section describes how to react when the robot detects an object from the selected tracking type. Examples of reaction types are Servo, Movement, Script, or all three. The Config menu of the Camera Control has many options. On the first page of the config menu, you will find options for servo and movement tracking reaction types.
If servo tracking is checked, the control assumes the camera is mounted on the specified servo. The servo will be moved from left, right, up and down to track the colored object based on the servo settings that you have provided.
First, do not confuse Movement with Motion. The motion tracking type (as discussed earlier) is for detecting a moving object in the camera view. The Movement reaction type (as discussed here) is having your robot physically move toward an object. If Movement is checked, the robot will follow the object when it is visible using the project's Movement Panel. The robot will move forward, left or right to follow the desired object. A robot has only one way to "move", and that is defined in the project's movement panel. The movement panel may be an Auto Position, HBridge, or more. Selecting this Movement option will configure the robot to move toward an image by controlling the movement panel.
On the Script tab of the Camera control config menu is a Tracking Start and Tracking End script. When an object is detected from the tracking type, the Tracking Start script will execute. When the camera is no longer tracking an object, the Tracking End will execute.
Video Resolutions and PerformanceMachine vision and computer recognition is a very high cpu intensive process. The cameras for computer vision are much less resolution than what you, as a human, would use for recording a birthday party. If you were to run computer vision to recognize objects and decode frames at HD quality, your computer response would grind to a halt.
160x120 = 57,600 Bytes per frame = 1,152,000 Bytes per second
320x240 = 230,400 Bytes per frame = 4,608,000 Bytes per second
640x480 = 921,600 Bytes per frame = 18,432,000 Bytes per second
So at 320x240, your CPU is processing complex algorithm on 4,608,000 Bytes per second. Soon as you move to a mere 640x480, it's 18,432,000 Bytes per second.
To expand on that, 4,608,000 Bytes per second is just the data, not including the number of CPU instructions per step of the algorithm(s). Do not let television shows, such as Person Of Interest, make you believe that machine vision and CPU processing is as accessible in real-time, but many of us are working on it! We can put 4,608,000 Bytes into perspective by relating that to a 2 minute MP3 file. Imagine your computer processing a 2 minute MP3 file in less than 1 second - that is what vision processing for recognition is doing at 320x240 resolution. Soon as you increase the resolution, the CPU has to process an exponentially larger amount of data. Computer vision recognition does not require as much resolution as your human eyes, as it looks for patterns, colors or shapes.
If you want to record a birthday party, do not use a robot camera - buy a real camera. EZ-Robot cameras and software algorithms are designed for vision recognition for robotics only.
Variable OverlayRight-click on the video image of the camera control and you will be presented with a menu. The menu allows you to Add or Delete text and variables which are displayed with the video stream. To move or relocate the variable position, simply click and drag it around the video image.
Choose Video DevicesWhen the camera is stopped (not running), additional video devices may be selected or entered. Clicking on the video device input, a drop down will be displayed with selections. The current selected EZ-B IP Address will be displayed, as well as an USB or local cameras. There will also be video devices displayed for third party supported robots, such as the AR Parrot Drone. Additionally, you may manually enter a JPEG Snapshot HTTP address or EZB Camera address.
JPEG Snapshot HTTP Video Device - a third party camera or video source which provides JPEG images may be selected as a video source. Simply enter the complete HTTP URL of the video source. If the HTTP video source requires authentication, consult the documentation of your device for correct syntax. You may test the JPEG URL in a web browser to verify that it is valid and that only the JPEG is displayed, not any HTML. Example: http://192.168.100.2/cgi-bin/snapshot.cgi or http://192.168.100.2/image.jpg, etc..
EZB Camera Device - when the Camera Control is added to a project, the current EZ-B Index #0 ip address is displayed as the default device. If the EZ-B Index #0 ip address has been changed, to accommodate client mode for example, you will manually need to edit the IP Address of the camera video device. Here is the syntax, example: EZB://192.168.0.1 or EZB://192.168.0.1:24.
AR Parrot Drone Device - if using the AR Parrot Drone as a video device, the AR Parrot Drone movement panel control must be added, configured and connected to the drone. Please view the AR Parrot Drone Movement Panel documentation for more information.
Use Camera Programmatically In Code
You may also instruct the Camera Device Control to change settings programmatically through code. The ControlCommand() can be called in EZ-Script to change the Camera Device settings. Learn more about the ControlCommand() in the Activities section HERE.