Hello guys, Does anyone know how I can to get buffer array of the video frame inside js?(or inside something else script). If it is not exists then I'am not understand why ARC supported udp sending function.

The image buffer would be too large and incredibly slow to expose to per frame to the java compiler. For that, we encourage ppl to create a robot skill to access the camera image in a faster language, such as c# or c++.

There's a tutorial here: https://synthiam.com/Support/Create-Robot-Skill/Overview

DJ, Thanks. My idea is to send video frames to my own web server. As I understand it, the connection to the server will also need to be registered in a robot skill?

Hmmm how do you want the video transmitted? If there’s a common protocol for that, I can help whip something up. Can you expand a bit more on what your goal is?

Yes, of course. I have a EZ robot "six" model. Also I have a django backend, where I want to implement Object Segmentation task in real time. So, at the current time I already know how to sent string data to my backend by http protocol in js emulator inside arc. But I don't know how to create and send image buffer, then I started to learn c# tools and ARC API.

I moved this conversation into a new thread so it's better organized for your outcome. i think making a robot skill for this is the best option, and it's fun to learn something new. The only snag you will run into is you need Visual Studio Community 2019. You cannot use 2022 because of the bug microsoft has not fixed yet (https://developercommunity.visualstudio.com/t/WinForms-NET-Framework-Projects-cant-d/1601210)

So, what you will want to do is documented here: https://synthiam.com/Support/Create-Robot-Skill/Examples/Example-Camera-Control

That tutorial explains how to get the camera device, connect to the NewFrame events, and do something with the image. It'll be up to you to determine how to send the image to your webserver. You may wish to have it FTP or File Copy or something. I haven't looked in the Camera Robot Skills if there's anything already created to what you're looking for.

Here is a robot skill that is complete which uses the camera and demonstrates everything. You can reference it for your robot skill: Camera Overlay.zip

Hello DJ! Thanks for you response. I started plugin creating today, I still have a lot to figure out. Wanted to ask another question. Do I understand correctly that this function returns an image array? Just I have not yet figured out what type this variable is and have not yet figured out how to log it.

Yes and no. Here's exactly what you need...

using System;
using System.Collections.Generic;
using System.Drawing;
using System.IO;
using System.Windows.Forms;
using ARC;

namespace Camera_Overlay {

  public partial class FormMain : ARC.UCForms.FormPluginMaster {

    ARC.UCForms.FormCameraDevice _cameraControl;
    bool _isClosing = false;

    public FormMain() {

      InitializeComponent();

      detach();
    }

    private void FormMaster_FormClosing(object sender, FormClosingEventArgs e) {

      _isClosing = true;

      detach();
    }

    public override void SendCommand(string windowCommand, params string[] values) {

      if (windowCommand.Equals("attach", StringComparison.InvariantCultureIgnoreCase))
        attach();
      else if (windowCommand.Equals("detach", StringComparison.InvariantCultureIgnoreCase))
        detach();
      else
        base.SendCommand(windowCommand, values);
    }

    public override object[] GetSupportedControlCommands() {

      List cmds = new List();

      cmds.Add(Common.Quote("Attach"));
      cmds.Add(Common.Quote("Detach"));

      return cmds.ToArray();
    }

    private void button1_Click(object sender, EventArgs e) {

      if (_cameraControl == null)
        attach();
      else
        detach();
    }

    void detach() {

      if (_cameraControl != null) {

        if (!_isClosing)
          ARC.Invokers.SetAppendText(tbLog, true, "Detaching from {0}", _cameraControl.Text);

        _cameraControl.Camera.OnNewFrame -= Camera_OnNewFrame;

        _cameraControl = null;
      }

      if (!_isClosing)
        ARC.Invokers.SetText(btnAttach, "Attach");
    }

    void attach() {

      detach();

      Control [] cameras = ARC.EZBManager.FormMain.GetControlByType(typeof(ARC.UCForms.FormCameraDevice));

      if (cameras.Length == 0) {

        ARC.Invokers.SetAppendText(tbLog, true, "There are no camera controls in this project.");

        return;
      }

      _cameraControl = (ARC.UCForms.FormCameraDevice)cameras[0];

      _cameraControl.Camera.OnNewFrame += Camera_OnNewFrame;

      ARC.Invokers.SetAppendText(tbLog, true, "Attached to: {0}", _cameraControl.Text);

      ARC.Invokers.SetText(btnAttach, "Detach");
    }

    void Camera_OnNewFrame() {

      if (_cameraControl == null)
        return;

      try {

        using (Graphics g = Graphics.FromImage(_cameraControl.Camera.GetOutputBitmapManaged)) {

          using (MemoryStream ms = new MemoryStream()) {

            _cameraControl.Camera.GetOutputBitmapManaged.Save(ms, System.Drawing.Imaging.ImageFormat.Png);

            // here is the image array
            var IMAGE_ARRAY = ms.ToArray();
          }

        }
      } catch (Exception eX) {

        Invokers.SetAppendText(tbLog, true, eX.ToString());

        detach();
      }
    }
  }
}

Specifically, the part that you are focusing on is this...

 using (Graphics g = Graphics.FromImage(_cameraControl.Camera.GetOutputBitmapManaged)) {

          using (MemoryStream ms = new MemoryStream()) {

            _cameraControl.Camera.GetOutputBitmapManaged.Save(ms, System.Drawing.Imaging.ImageFormat.Png);

            // here is the image array
            var IMAGE_ARRAY = ms.ToArray();
          }

        }

Hello DJ! I want to thank you for your support! I was able to extract image frames from the camera in real time. I also managed to make a websocket client that sends a stream of frames to my django server.

This is your modified by me code:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;
using System.IO;
using System.Runtime.InteropServices;
using ARC;
using System.Threading;
using System.Net.WebSockets;

namespace CameraCapture {

  public partial class MainForm : ARC.UCForms.FormPluginMaster {

    Configuration _config;
    ARC.UCForms.FormCameraDevice _cameraControl;
    bool _isClosing = false;

    [DllImport("kernel32.dll", SetLastError = true)]
    [return: MarshalAs(UnmanagedType.Bool)]
    static extern bool AllocConsole();
    CancellationTokenSource source = new CancellationTokenSource();
    ClientWebSocket wsClient = new ClientWebSocket();




    public MainForm() {

        InitializeComponent();

      // show a config button in the title bar. Set this to false if you do not have a config form.
        ConfigButton = true;
        detach();
        
    }

    /// 
    /// Set the configuration from the project file when loaded.
    /// We'll extract the _config class that's from the project file.
    /// 
    public override void SetConfiguration(ARC.Config.Sub.PluginV1 cf) {

        _config = (Configuration)cf.GetCustomObjectV2(typeof(Configuration));

        base.SetConfiguration(cf);
    }

    /// 
    /// When the project is saving, give it a copy of our config
    /// 
    public override ARC.Config.Sub.PluginV1 GetConfiguration() {

        _cf.SetCustomObjectV2(_config);

        return base.GetConfiguration();
    }

    /// 
    /// The user pressed the config button in the title bar. Show the config menu and handle the changes to the config.
    /// 
    public override void ConfigPressed() {

      using (var form = new ConfigForm()) {

        form.SetConfiguration(_config);

        if (form.ShowDialog() != DialogResult.OK)
          return;

        _config = form.GetConfiguration();
      }
    }


    private void button1_Click(object sender, EventArgs e)
    {

        
        if (_cameraControl == null)
        attach();
        else
            detach();
    }
    void detach()
    {

        if (_cameraControl != null)
        {

            if (!_isClosing)
                ARC.Invokers.SetAppendText(tbLog, true, "Detaching from {0}", _cameraControl.Text);

            _cameraControl.Camera.OnNewFrame -= Camera_OnNewFrame;

            _cameraControl = null;
        }

        if (!_isClosing)
            ARC.Invokers.SetText(btnAttach, "Attach");
    }

    void attach()
    {

        detach();

        Control[] cameras = ARC.EZBManager.FormMain.GetControlByType(typeof(ARC.UCForms.FormCameraDevice));

        //Task returnedTask = OpenConnectionAsync();
        
        if (cameras.Length == 0)
        {

            ARC.Invokers.SetAppendText(tbLog, true, "There are no camera controls in this project.");

            return;
        }

        _cameraControl = (ARC.UCForms.FormCameraDevice)cameras[0];

        _cameraControl.Camera.OnNewFrame += Camera_OnNewFrame;

        ARC.Invokers.SetAppendText(tbLog, true, "Attached to: {0}", _cameraControl.Text);

        ARC.Invokers.SetText(btnAttach, "Detach");
    }

    void Camera_OnNewFrame()
    {

        if (_cameraControl == null)
            return;
    
        try
        {
            using (Graphics g = Graphics.FromImage(_cameraControl.Camera.GetOutputBitmapManaged))
            {

                using (MemoryStream ms = new MemoryStream())
                {

                    _cameraControl.Camera.GetOutputBitmapManaged.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
                    // here is the image array
                    byte[] byteArray = ms.ToArray();
                    SendAsync(byteArray, source.Token);
                }

            }
        }
        catch (Exception eX)
        {

            Invokers.SetAppendText(tbLog, true, eX.ToString());

            detach();
        }
    }
    public async Task OpenConnectionAsync(CancellationToken token) {
        Uri serverUri = new Uri("ws://127.0.0.1:8000/ws/socket-server/");
        await wsClient.ConnectAsync(serverUri, source.Token).ConfigureAwait(false);
    }
    public async Task SendAsync(byte[] byteImage, CancellationToken token)
    {
        try
        {

                await wsClient.SendAsync(new ArraySegment(byteImage), WebSocketMessageType.Binary, true, source.Token);
                Console.WriteLine(wsClient.State);
        }
        catch (Exception eX) {
            Console.WriteLine(wsClient.State);
            Console.WriteLine(eX);
        }

    }

    private void MainForm_Load(object sender, EventArgs e)
    {
        AllocConsole();
        OpenConnectionAsync(source.Token);
    }
  }
}

This websocket server part on django:

import json
import cv2
from channels.generic.websocket import WebsocketConsumer
import os

bytes_arr = 0

class WSConsumerTest(WebsocketConsumer):
    def connect(self):
        self.accept()
        self.send(text_data = json.dumps({
            'message': 'created_socket',
        }))
        print('success connection')
    def receive(self, text_data=None, bytes_data=None):
        global bytes_arr
        if (bytes_data != None):
            bytes_arr = bytes_data
        elif(text_data != None):
            self.send(bytes_data=bytes_arr)

Next, I output the frame stream to the browser in real time in the same way, but unfortunately here for some reason I can’t publish the html / js code, I don’t know why to be honest. The next step is server-side image segmentation and automatic motion control of the robot based on segmented image analysis. If you are interested in the further fate of the development, I can publish further results.

Best wishes!

That’s great! Nice work in such a short period of time. Please keep me updated. I enjoy watching progress of what ppl make