Overview
Robots still struggle to match human abilities in perceiving and manipulating objects, especially in complex, real-world environments. This project explored how robots can learn to understand objects using both visual data (camera images) and tactile feedback (touch).
Led by Jörg Stückler, the research used self-supervised learning and physics-based models to help robots understand how objects move and respond to forces. By combining vision and touch, the team aimed to improve both object perception and manipulation. The project ran from June 2020 to December 2023.
Key results
- Developed methods combining differentiable physics and deep learning.
- Enabled robots to estimate object position, orientation, velocity, and friction.
- Demonstrated how learned object dynamics can improve manipulation strategies.
- Introduced new datasets for visuo-tactile learning (EV-SynPhys, EV-RealPhys).
Why it matters
For robots to manipulate objects reliably, they must understand not only what objects look like but also how they move and respond to forces such as contact and friction. By combining visual information with physical models, this research supports more adaptive robotic interaction and could contribute to future applications in manufacturing, logistics, and service robotics.
Publications and outputs
- Kandukuri, R. K., Strecke, M., Stueckler, J. Physics-Based Rigid Body Object Tracking and Friction Filtering From RGB-D Videos. Accepted for International Conference on 3D Vision (3DV), 2024. Preprint arXiv:2309.15703.
- The accompanying EV-SynPhys and EV-RealPhys datasets are available through the Max Planck data repository.
- The 3DV 2024 paper was presented as poster presentation at the International Conference on 3D Vision 2024. Parts of the results reported in the paper have been also presented in a keynote given at the IEEE International Conference on Multi-Sensor Fusion and Integration / Symposium on Sensor and Data Fusion (MFI/SDF) in Bonn in November 2023. Parts of the project results have been presented at the Cyber Valley Research Fund symposium in November 2023.
