Uzbekistan
Asked
Resolved Resolved by DJ Sures!

Send Camera Image To Webserver

Hello guys, Does anyone know how I can to get buffer array of the video frame inside js?(or inside something else script). If it is not exists then I'am not understand why ARC supported udp sending function.


ARC Pro

Upgrade to ARC Pro

Elevate your robot's capabilities to the next level with Synthiam ARC Pro, unlocking a world of possibilities in robot programming.

Uzbekistan
#9  

Hello DJ! I want to thank you for your support! I was able to extract image frames from the camera in real time. I also managed to make a websocket client that sends a stream of frames to my django server.

This is your modified by me code:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;
using System.IO;
using System.Runtime.InteropServices;
using ARC;
using System.Threading;
using System.Net.WebSockets;

namespace CameraCapture {

  public partial class MainForm : ARC.UCForms.FormPluginMaster {

    Configuration _config;
    ARC.UCForms.FormCameraDevice _cameraControl;
    bool _isClosing = false;

    [DllImport("kernel32.dll", SetLastError = true)]
    [return: MarshalAs(UnmanagedType.Bool)]
    static extern bool AllocConsole();
    CancellationTokenSource source = new CancellationTokenSource();
    ClientWebSocket wsClient = new ClientWebSocket();




    public MainForm() {

        InitializeComponent();

      // show a config button in the title bar. Set this to false if you do not have a config form.
        ConfigButton = true;
        detach();
        
    }

    /// 
    /// Set the configuration from the project file when loaded.
    /// We'll extract the _config class that's from the project file.
    /// 
    public override void SetConfiguration(ARC.Config.Sub.PluginV1 cf) {

        _config = (Configuration)cf.GetCustomObjectV2(typeof(Configuration));

        base.SetConfiguration(cf);
    }

    /// 
    /// When the project is saving, give it a copy of our config
    /// 
    public override ARC.Config.Sub.PluginV1 GetConfiguration() {

        _cf.SetCustomObjectV2(_config);

        return base.GetConfiguration();
    }

    /// 
    /// The user pressed the config button in the title bar. Show the config menu and handle the changes to the config.
    /// 
    public override void ConfigPressed() {

      using (var form = new ConfigForm()) {

        form.SetConfiguration(_config);

        if (form.ShowDialog() != DialogResult.OK)
          return;

        _config = form.GetConfiguration();
      }
    }


    private void button1_Click(object sender, EventArgs e)
    {

        
        if (_cameraControl == null)
        attach();
        else
            detach();
    }
    void detach()
    {

        if (_cameraControl != null)
        {

            if (!_isClosing)
                ARC.Invokers.SetAppendText(tbLog, true, "Detaching from {0}", _cameraControl.Text);

            _cameraControl.Camera.OnNewFrame -= Camera_OnNewFrame;

            _cameraControl = null;
        }

        if (!_isClosing)
            ARC.Invokers.SetText(btnAttach, "Attach");
    }

    void attach()
    {

        detach();

        Control[] cameras = ARC.EZBManager.FormMain.GetControlByType(typeof(ARC.UCForms.FormCameraDevice));

        //Task returnedTask = OpenConnectionAsync();
        
        if (cameras.Length == 0)
        {

            ARC.Invokers.SetAppendText(tbLog, true, "There are no camera controls in this project.");

            return;
        }

        _cameraControl = (ARC.UCForms.FormCameraDevice)cameras[0];

        _cameraControl.Camera.OnNewFrame += Camera_OnNewFrame;

        ARC.Invokers.SetAppendText(tbLog, true, "Attached to: {0}", _cameraControl.Text);

        ARC.Invokers.SetText(btnAttach, "Detach");
    }

    void Camera_OnNewFrame()
    {

        if (_cameraControl == null)
            return;
    
        try
        {
            using (Graphics g = Graphics.FromImage(_cameraControl.Camera.GetOutputBitmapManaged))
            {

                using (MemoryStream ms = new MemoryStream())
                {

                    _cameraControl.Camera.GetOutputBitmapManaged.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
                    // here is the image array
                    byte[] byteArray = ms.ToArray();
                    SendAsync(byteArray, source.Token);
                }

            }
        }
        catch (Exception eX)
        {

            Invokers.SetAppendText(tbLog, true, eX.ToString());

            detach();
        }
    }
    public async Task OpenConnectionAsync(CancellationToken token) {
        Uri serverUri = new Uri("ws://127.0.0.1:8000/ws/socket-server/");
        await wsClient.ConnectAsync(serverUri, source.Token).ConfigureAwait(false);
    }
    public async Task SendAsync(byte[] byteImage, CancellationToken token)
    {
        try
        {

                await wsClient.SendAsync(new ArraySegment(byteImage), WebSocketMessageType.Binary, true, source.Token);
                Console.WriteLine(wsClient.State);
        }
        catch (Exception eX) {
            Console.WriteLine(wsClient.State);
            Console.WriteLine(eX);
        }

    }

    private void MainForm_Load(object sender, EventArgs e)
    {
        AllocConsole();
        OpenConnectionAsync(source.Token);
    }
  }
}

This websocket server part on django:

import json
import cv2
from channels.generic.websocket import WebsocketConsumer
import os

bytes_arr = 0

class WSConsumerTest(WebsocketConsumer):
    def connect(self):
        self.accept()
        self.send(text_data = json.dumps({
            'message': 'created_socket',
        }))
        print('success connection')
    def receive(self, text_data=None, bytes_data=None):
        global bytes_arr
        if (bytes_data != None):
            bytes_arr = bytes_data
        elif(text_data != None):
            self.send(bytes_data=bytes_arr)

Next, I output the frame stream to the browser in real time in the same way, but unfortunately here for some reason I can’t publish the html / js code, I don’t know why to be honest. The next step is server-side image segmentation and automatic motion control of the robot based on segmented image analysis. If you are interested in the further fate of the development, I can publish further results.

Best wishes!

PRO
Synthiam
#10  

That’s great! Nice work in such a short period of time. Please keep me updated. I enjoy watching progress of what ppl make