22 August 2024 | Tübingen

Cyber Valley Research Fund advances robotic perception techniques

Integrating Vision and Touch

Image: Cyber Valley

Dr Jörg Stückler has successfully led a research project funded by the Cyber Valley Research Fund. His project titled “Learning of Physics-based Models for Visuo-Tactile Object Perception and Manipulation”, aimed to develop methods for learning physics-based models of object interactions using visual and tactile data. The Cyber Valley Research Fund provides resources for scientific research through the contributions of six Cyber Valley founding partners: Amazon, BMW, Bosch, IAV, Porsche, and ZF.

Current autonomous systems still lack the capabilities of humans in scene perception and object manipulation tasks. Jorg Stückler and his team developed a method that combines differentiable physics and deep learning to estimate an object’s position, orientation, and velocity, while also simultaneously determining the friction between an object and its surface. The team also explored how understanding an object's dynamics could improve control strategies, such as predicting how a robot's actions will impact the object and guiding it to a desired location.

Their findings inspire new methods and approaches that adapt robotics object manipulation skills to the physical properties of objects. Although the research is still in its scientific development stage, it has the potential to revolutionize applications by enabling robots to manipulate objects more flexibly and adaptively.

This project yielded the following publications:

Kandukuri, R. K., Strecke, M., Stueckler, J. Physics-Based Rigid Body Object Tracking and Friction Filtering From RGB-D Videos. Accepted for International Conference on 3D Vision (3DV), 2024. Preprint arXiv:2309.15703.
EV-SynPhys and EV-RealPhys datasets accompanying at https://keeper.mpdl.mpg.de/d/5ec213b655b44e40a382/
The 3DV 2024 paper was presented as poster presentation at the International Conference on 3D Vision 2024. Parts of the results reported in the paper have been also presented in a keynote given at the IEEE International Conference on Multi-Sensor Fusion and Integration / Symposium on Sensor and Data Fusion (MFI/SDF) in Bonn in November 2023. Parts of the project results have been presented at the Cyber Valley Research Fund symposium in November 2023.

This research project started in June 2020 and concluded in December 2023. Dr. Jörg Stückler is now a Professor of Computer Science at the University of Augsburg.