Enabling Real-Time Teleoperation of a Multi-Purpose Robotic Platform

The goal was to build a stable and reliable backend system capable of supporting multiple remote operations at once, real-time data exchange, and secure control of robotic platforms across various use cases. The potential use cases included warehouse operations, HoReCa customer service, promotional drone for exhibitions, etc.

We built the remote control system backend for a ground-based robotic platform entirely from scratch. The idea was to make it support real-time operation, direct peer-to-peer (P2P) connection, and full hardware management for multiple use cases.

Initially developed over 1.5 years and refined hereafter, the system possessed seamless teleoperation, secure data streaming, and smart integration with onboard sensors. Whether used in military or civil environments, we’ve delivered a core for any mobile robotics deployment.

The team included backend developers, an architect, an IoT specialist with expertise in computer vision engineering, all joined under the smooth guidance of an experienced PM.

We started this project a few years ago, and that’s why video streaming libraries became a major hurdle. At the time, available libraries lacked flexibility and cross-platform support. So, we had to develop a custom, low-latency solution.

Another challenge was diverse hardware components integration. As soon as the robot was custom-made with no standard parts, we needed to calibrate each module’s coordination by hand.

The video recognition feature surfaced another layer of complexity. It had to run in real-time while balancing limited computing resources and collecting data from components like gyroscope.

As a result, architectural decisions and tech explorations were made from scratch. Being an internal project with no direct client, we conducted experiments and delivered a flexible foundation for various future robotics use cases.

Backend – .NET Framework

Web interface – Razor

WebSockets were used for real-time communication

UDP for video streaming

PostgreSQL database for data storage. 

Xamarin for mobile support

YOLO (Neural Network) for real-time object detection and analytics.

We have developed a remote control system backend with real-time robot operation via P2P connections, including the following features:

• Command Management: Receive and perform commands from the operator, providing reports on performance with minimal latency.
• System Health Monitoring: Collect real-time data from the robot’s position, battery level, sensors, etc.) and output it to the operator’s device.
• Video Streaming and Recognition: Control the transmission of live video or sensor data for situational awareness and visual feedback.
• Device Authentication & Security: Manage permissions, verify users, and guarantee an encrypted connection.
• Real-Time Communication: For low-latency, two-way communication through WebSockets.