MYWAI™ VILMA™ is designed to bring human-like learning to robots via one-shot demonstration
Every day, hundreds of thousands of kits are prepared in warehouses before components ever reach an automotive production line. While robots have become commonplace in modern manufacturing, many upstream logistics activities still rely heavily on human operators performing repetitive pick-and-place and kitting tasks.
What if robots could learn these operations the same way humans do: by simply watching a demonstration?
That question was at the heart of I-GENIUS, a research project coordinated by MYWAI within the European ARISE initiative with Centro Ricerche FIAT (CRF) and the University of Genoa’s Department of Mechanical, Energy, Management and Transportation Engineering (DIME).
The project explored new approaches to Human-Robot Interaction, combining AI, computer vision, and robotics to enable machines to acquire manipulation skills from minimal human guidance.
One of the project’s key outcomes was VILMA (Visual Imitation Learning for Manipulation Activities), an AI-powered toolkit integrated into MYWAI’s EDGE AI middleware platform. VILMA helps enable robots and humanoids to learn complex manipulation tasks from one-shot human demonstrations, aiming to significantly reduce programming effort while improving flexibility in dynamic industrial environments.
The technology was evaluated in a large-scale automotive warehouse use case developed together with CRF and reproduced within the robotics laboratories at DIME.
Today, MYWAI is bringing this technology to a broader community of developers, makers, and robotics innovators by porting the VILMA Toolkit to new Arduino products powered by Qualcomm Dragonwing™ processors, including both the Arduino® UNO™ Q and VENTUNO™ Q boards.
This demonstration showcases the potential for edge-native robotics applications that use imitation learning techniques on hardware platforms built to support compact form factors and efficient power consumption.
At the iGenius final presentation, the founder and CEO of MYWAI, Fabrizio Cardinali, stated: “The dual-brain architecture of the UNO Q and VENTUNO Q platforms is an ideal foundation for MYWAI’s next generation of Edge AI robotics. After validating distributed intelligence concepts within the ARISE I-GENIUS project, we are now leveraging these platforms to bring World Action Models closer to the edge through the latest release of the MYWAI EdgeAI Management Platform and its mobile tracker, HEDGELOG™. By combining One-Shot Video Imitation Learning with edge-native AI execution, we aim to enable robots and intelligent industrial machines to acquire, distribute, adapt, and execute complex manipulation skills with unprecedented flexibility and scalability.”
Watch the full demonstration of the I-GENIUS project and see VILMA in action in this video.
The MYWAI VILMA agent
VILMA is a visual imitation learning toolkit that helps enable robots to learn manipulation tasks from human demonstrations. It is designed to process RGB-D recordings or MP4 videos to extract hand and object trajectories, generate reusable robot skills using Dynamic Movement Primitives (DMPs), and produce robot-ready trajectories for playback. It is constructed to serve as the demonstration learning module, supporting rapid robot programming, skill reuse, and deployment.
The AI pipeline
One-shot demonstration acquisition
The one-shot demonstration acquisition step is set to record a human performing the task or retrieve an existing demonstration from a selected MYWAI equipment event. It stages the video, RGB frames, depth data, and camera parameters, and allows the user to select the target object for tracking. This information provides the inputs required by the remaining pipeline stages.
Hand detection
Using MediaPipe, this stage is structured to detect 21 hand landmarks in each RGB frame and combine their 2D positions with depth data to calculate 3D camera coordinates. For demonstrations loaded from MYWAI, the staged RGB and depth data are processed through the same pipeline. Kalman smoothing and previous-position retention improve tracking robustness, and the resulting trajectories can be saved back to the MYWAI event.
Object detection
Using a YOLO model, this stage is designed to detect or track the object selected through the local or MYWAI interface. It combines the bounding-box centre with depth information to calculate the object’s 3D position, applies Kalman smoothing, and saves the trajectory and annotated frames. These results can then be included in the pipeline artifacts stored in MYWAI.
Trajectory and segmentation
This stage is constructed to load the smoothed hand and object trajectories, estimate the grasp point from the hand’s proximity to the object, and detect the release point from the object’s movement and stabilization. It uses the hand trajectory as the main motion path and divides it into reach, grasp, move, release, and post-release phases. The trajectories, event indices, and segmentation metadata can be packaged as MYWAI event data, a time and space data fusion format developed by MYWAI for its AI-IoT management platform particularly geared towards Multimodal AI and, next, towards World Action Models.
DMP generation
The DMP-generation stage is designed to learn separate Dynamic Movement Primitive models for the reach and move phases. It evaluates different regularization values, selects the model that provides the best accuracy and smoothness, validates the reproduced motion, and saves the trained models and trajectories. These DMP artifacts can be uploaded to MYWAI with the other pipeline results for later retrieval and reuse.
Demonstration
The demonstration stage is structured to convert the generated DMP trajectory into Cartesian robot positions using the configured scale, offset, and rotation, then apply inverse kinematics to calculate the joint trajectory. The robot model may be loaded from the selected MYWAI equipment, and the resulting motion is displayed through the MYWAI 3D Viewer, synchronized with the recorded video and its grasp and release events.
DMP adaptation with new goal and new object
The adaptation stage is set to load the learned skill – either from the current pipeline or a restored MYWAI event – and detect a new target object using RGB and depth data. It calculates the 3D offset between the original and new objects, redirects the reach and move trajectories toward the new pick and release positions, and preserves the demonstrated motion characteristics. The adapted trajectory can then be visualized with the MYWAI 3D Viewer or sent to the robot.
Live streaming adaptation and UNO Q and VENTUNO Q support
The Live stream phase represents the deployment and real-time inference stage of the VILMA Agent. While the initial learning phase is conducted on the MYWAI platform to generate Dynamic Movement Primitives (DMP), the Live stream phase focuses on shipping these DMPs along with a fine-tuned YOLOv8 model, supported today on UNO Q and VENTUNO Q.
Architecture and components
As illustrated in the system schematic below, the architecture is designed as a distributed setup divided into an Edge AI Layer for intelligence and a Communication Layer for hardware interfacing.
Let’s break down how the live stream pipeline is working considering the VENTUNO Q version.
1. Edge AI layer (VENTUNO Q)
Running on VENTUNO Q, this layer is structured to handle high-level decision-making.
- VILMA Control Loop: The primary application logic responsible for the overall control loop. It It is designed to orchestrate object detection and performs DMP Adaptation to translate learned human motions into the current physical environment.
- Video Object Detection Brick: This component runs on VENTUNO Q to manage the inference flow. It receives the incoming video feed and communicates with the inference service.
- Docker: YOLOv8 Inference Service is formed as a containerized service that runs the quantized YOLOv8 model. This model is designed to be fine-tuned and deployed via the Edge Impulse platform using the “Bring Your Own Model” feature.
2. ROS 2 communication layer (Workstation)
A separate workstation connected directly to the devices manages the high-bandwidth data streams and robotic control via ROS 2.
- ROS 2 Streaming Node: Interfaces with the ZED Camera/Depth Sensor to capture raw visual data, publishing it as a /camera_feed to VENTUNO Q.
- ROS 2 Command Node: This node acts as a wrapper around the Fairino Python SDK. It is engineered to serve as the receiver for the /learned_trajectory sent from the edge device, utilizing the SDK to directly control the robot and help ensure it accurately follows the planned trajectory.
3. Physical hardware
External hardware is connected to complete the runtime VILMA ecosystem, namely:
- ZED™ Camera: The stereo camera which is engineered to capture the image and depth data required for the vision system.
- Fairino™ FR10 Robot: The robotic arm that is constructed to execute the pick-and-place tasks based on the trajectories computed by VILMA.
Component communication and data flow
The communication between these components is designed for low-latency execution as shown in the schema above:
- Vision Input: The Workstation is engineered to stream the /camera_feed (image and depth) to VENTUNO Q.
- Edge Inference: The VILMA Control Loop is designed to utilize a WebSocket stream to send frames to the Docker YOLOv8 Inference Service. The service returns the detected object bounding box to the control loop.
- Motion Adaptation: The system is structured to take the detected object positions and adapts the human-learned DMP to calculate a precise pick-and-place trajectory.
- Robotic Execution: The resulting /learned_trajectory is published back to the Workstation’s ROS 2 Command Node, which drives the Fairino FR10 robot to complete the task.
This modular approach is designed to allow the heavy vision processing and motion adaptation to happen on the edge (VENTUNO Q) while leveraging the robust ROS 2 ecosystem for robot communication and sensor streaming.
To learn more about the project and MYWAI’s EDGEAI Platform and Middleware, visit myw.ai.
Qualcomm branded products are products of Qualcomm Technologies, Inc. and/or its subsidiaries.
Arduino, UNO, and VENTUNO are trademarks or registered trademarks of Arduino S.r.l.
Facts Only
* MYWAI VILMA is designed to bring human-like learning to robots via one-shot demonstration.
* The project was coordinated by MYWAI within the European ARISE initiative with CRF and the University of Genoa’s DIME.
* VILMA is an AI toolkit integrated into MYWAI’s EDGE AI middleware platform.
* VILMA enables robots and humanoids to learn complex manipulation tasks from one-shot human demonstrations.
* VILMA processes RGB-D recordings or MP4 videos to extract hand and object trajectories, generate reusable robot skills using Dynamic Movement Primitives (DMPs), and produce robot-ready trajectories.
* The pipeline includes one-shot demonstration acquisition, hand detection using MediaPipe, object detection using a YOLO model, trajectory and segmentation, DMP generation, demonstration playback, adaptation with new goals/objects, and live streaming.
* The technology is ported to Arduino products powered by Qualcomm Dragonwing™ processors, including Arduino® UNO™ Q and VENTUNO™ Q boards.
* The architecture features an Edge AI Layer (VENTUNO Q) for intelligence and a Communication Layer managed by ROS 2 on a workstation.
* The pipeline involves data flow from Vision Input (/camerafeed) to Edge Inference (YOLOv8), Motion Adaptation, and Robotic Execution via the ROS 2 Command Node.
Executive Summary
MYWAI utilizes VILMA, a visual imitation learning toolkit, to enable robots to learn complex manipulation tasks from single human demonstrations. This technology was developed as part of the I-GENIUS research project involving MYWAI, Centro Ricerche FIAT (CRF), and the University of Genoa’s DIME department, focusing on Human-Robot Interaction using AI, computer vision, and robotics. VILMA processes RGB-D data or MP4 videos to extract trajectories, generate reusable robot skills using Dynamic Movement Primitives (DMPs), and produce robot-ready motion sequences.
The system employs a multi-stage AI pipeline: one-shot demonstration acquisition, hand detection via MediaPipe for 3D coordinate calculation, object detection using YOLO for 3D positioning, trajectory segmentation, DMP generation, demonstration conversion to Cartesian positions, adaptation of learned skills to new goals, and live streaming execution. The architecture is distributed, featuring an Edge AI layer on hardware like the Qualcomm Dragonwing™-powered UNO Q and VENTUNO Q boards, communicating with a ROS 2 communication layer that interfaces with external sensors (ZED Camera) and physical hardware (Fairino FR10 Robot).
The process culminates in live streaming adaptation where learned skills are adapted to new objects, and the system executes the resulting trajectories. The final deployment leverages edge-native AI execution for real-time skill acquisition and execution on compact hardware.
Full Take
The narrative establishes a clear trajectory: moving complex human skills into robot execution through imitation learning, optimized for edge deployment using specific hardware platforms. The strength lies in bridging high-level AI concepts (imitation learning, DMPs) with concrete, distributed implementation details (MediaPipe, YOLOv8, ROS 2). The focus on the dual-brain architecture of UNO Q and VENTUNO Q positions the work not just as a software innovation but as an architectural shift for future Edge AI robotics.
A critical pattern observed is the framing of capability through sequential processing steps—from raw visual input to learned motion primitives and finally to physical execution. This sequence implies a deterministic, scalable path for skill transfer, which offers a powerful simplification for complex programming tasks. The context suggests a move away from purely programmed control toward embodied intelligence acquisition.
The implication for agency rests on where the 'learning' resides: if the adaptation phase successfully redirects demonstrated motion characteristics to new objects while preserving motion 'feel,' it suggests a high degree of adaptability, potentially granting robots dynamic situational awareness rather than just following pre-set paths. The missing element, which demands inquiry, is the robustness and generalization limits of the DMP adaptation across vastly different physical constraints or novel interaction styles. What assumptions are made about the transferability of learned DMPs when moving from demonstration space to execution space under real-world noise? Furthermore, who controls the 'reusable robot skills' generated in the initial learning phase, and how is that control maintained during the live streaming adaptation process?