Dual-Arm Shaping of Soft Objects in 3D Based on Visual Servoing and Online FEM Simulations

In this video, we use the robot plateform BAZAR to demonstrate different shaping tasks of soft objects in 3D. To estimate the the whole object's shape evolution, we relate visual feedback to online FEM simulations. We also show that the FEM simulations can be used to avoid damaging an object during manipulation.


Abstract of the paper:
In this work, we propose a vision-based and Finite Element Method (FEM) based controller to automate the 3D shaping of soft objects with dual-arm robots.
Our controller relies on a data-based approach to learn how the robot's actions result in object deformations, while also running FEM-based simulations to infer the shape of the whole body.
These model-based simulations are used to generate initial shape data, allowing to extract visual features through a Principal Component Analysis and thus estimate the interaction matrix of the object-robot system.
In contrast with most existing shape servoing controllers, our new model-based approach continuously predicts the object deformations produced by the robot, which are then compared to the visually observed deformation feedback. This iterative process enables to correct the deformed mesh model before updating the interaction matrix.
To validate this new control methodology, we present a detailed experimental study with a dual-arm robot and different soft objects, which showcases the performance of our automatic shaping framework.