Research Fund

Support for basic research in machine learning, computer vision, and robotics

Today’s autonomous systems still lack behind the versatile capabilities of humans in scene perception and object manipulation tasks. This project addresses the research question of how robots can learn to perceive and manipulate objects from visual and tactile feedback in a self-supervised way. Jörg Stückler and his team will develop methods that will allow robots to learn models of their interaction with objects from camera images and tactile measurements. The scientists will investigate the use of learned models for perception and control in several robotic object manipulation tasks.

If the mobile service robots of the future, among them delivery robots and self-driving cars, were able to learn independently from their environments, they would also be capable of adapting to changes in their surroundings and thus move about more efficiently. In turn, this would eliminate the need for engineers to tune robots manually to their environments.

This research project addresses the question of how mobile robots can learn their driving capabilities in their environment (i.e. mobility affordances) in a self-supervised way. Jörg Stückler and his team will develop methods for learning motion models that will allow mobile robots to predict the effects of their actions. The scientists will develop a vision-based navigation approach that use learned models for motion planning. They will then evaluate this approach for the autonomous navigation of a mobile robot.

In various fields of research, such as the social sciences, biology, and computer science, network models are often applied to help describe complex systems with many individual elements interacting. In recent years, these models have often been used to draw new conclusions from observed data. The availability of large amounts of data has promoted this development.

Generative models are a popular network model. Here, latent variables are introduced which integrate the scientific findings in this field of knowledge (the "domain knowledge") and capture complex interactions. However, interactions among individuals are usually so complex that they are often approximated as independent. Conditioning upon these variables, the network edges are assumed to be independent and the distribution of probabilities within the network can be simplified. The disadvantage of these models is that in some real-world scenarios, the interactions within the network are not well captured. This means that the model's mathematical description does not correspond well to what is observed in real data. The coupling between variables, which are  too limited, are the main problem here. In comparison, network ensemble models do not use such latent variables, but rather network-specific variables (e.g. degree of distribution or clustering coefficient). However, these models also suffer from various problems that limit their practical application.

This project will combine certain features of the generative model and the network ensemble model with methods from statistical physics. The aim is to develop better principle-based models. In addition, the project aims to ensure that these models can be efficiently applied to concrete problems (e.g. repeatability or the simultaneous occurrence of different forms of relationships between two nodes). 

As technological development accelerates, millions of low-skilled workers are destined to lose their jobs to automation. To mitigate the resulting societal problems, this project aims to develop a scientific and technological foundation for rapidly and inexpensively teaching people the skills they will need to stay or become employable in the workplace of the future, which will be increasingly cognitively demanding.

Building on computational models of human learning and decision-making, Falk Lieder’s group proposes a general and scalable approach that leverages machine learning and artificial intelligence to teach workers the strategies they will need to meet the self-management challenges of the knowledge economy.

The researchers will test this approach by developing a series of intelligent tutors that develop and teach optimal decision strategies for increasingly realistic scenarios. They will illustrate the potential of this approach by developing and evaluating a simulation-based intelligent tutor that teaches high-level employees, freelancers, entrepreneurs, and academics far-sighted strategies for planning their projects, prioritizing their tasks, and managing themselves more effectively.

Many students struggle to stay focused long enough to learn effectively, and social media has exacerbated the problem. Constant distractions at work have led to losses in productivity, which cost the economy billions. These serious issues not only have a negative implications for the lives of individual people, but also for society as a whole.

In this project, Dr. Falk Lieder and Jun.-Prof. Dr. Maria Wirzberger will address the challenge of staying focused in the face of distractions by developing a brain training app called ACTrain together with their project team. ACTrain will be a personal assistant with a name and a customized appearance that will train people to stay focused on a task and effectively resume it after getting distracted. Unlike conventional brain training apps, ACTrain will allow people to train while they are working or studying, thereby turning their daily lives into a gym for the mind. ACTrain can thus be used in many different contexts, including education and the workplace.

The heart of ACTrain is an intelligent feedback mechanism based on computational models of how attention control skills are learned. Based on these models, the application gives people feedback when they get distracted. The feedback communicates the benefits of regaining focus for their productivity and success. In both online courses and the workplace, this software could improve the lives of millions of students and working professionals.

This project focuses on investigating new ways of transferring characteristics of the human visual system to artificial neural networks, with the aim of making them more robust against changes in image features. These can include, for example, changes in image style that do not alter the image's content. At present, no learning algorithm is capable of robustly generalize what it has learned to other untrained image features. Artificial neural networks quickly make mistakes when the image changes even slightly, for instance when noise is added or style changes are made. Humans have no problems recognizing the content of an image in such instances. Even if most of us grow up under the influence of a certain environment with specific visual characteristics (such as the Black Forest), our visual system easily generalizes to completely different environments (such as a desert environment or a painting).

Previous work has shown that deep artificial neural networks use very different image features for decision making than our visual system. For example, while we usually categorize objects by their shape, these networks rely mainly on local patterns in the images. It is still very difficult to incorporate the image features humans use to perceive into artificial systems, as we simply know too little about the exact properties of biological systems.

This is why we want to develop mechanisms that can transfer robust features directly from measurements of brain activity to artificial systems. Under controlled conditions, we will first investigate the mechanisms with which these features can be transferred between networks. In the final phase of the project, we will use publicly available measurements of neural activity from the visual system to test which of the neural properties can be transferred to artificial networks using the methods we have developed.

The Cyber Valley “Locomotion in Biorobotic and Somatic Systems” research group investigates the biomechanics of locomotion and underlying morphological adaptations, as evolved by nature. The researchers then apply their biological findings to develop life-like robots and functional materials that are similar to how they occur in nature. Their research is at the interface of engineering and biology – a relatively new and promising field. 

Dr. Ardian Jusufi and Hritwick Banerjee envision developing a flexible, stretchable, and biocompatible external sensor made from multi-functional smart materials that could one day be applied in healthcare, both for humans and in non-invasive veterinary care. The sheet-like sensor would adhere externally to the human or animal exterior as smoothly as a second layer of skin, and would stay in place no matter how a person or animal moves. The sensor could then detect a person’s health, sense blood pressure and other biometric values, or whether a person had an irregular heart beat that could indicate adverse health events such as heart attacks. In addition to a broad of range of biomedical applications, the soft and flexible sensor could also be built into smart clothes, wearable electronics, or soft robotics, to name just a few examples. They could also be used to improve human-machine interaction. For instance, self-driving cars could be equipped with such sensors. If a person touched the sensor while sitting in the vehicle, it could detect an imminent medical emergency and send a signal to the autopilot, which would immediately drive the car to the nearest hospital. 

There are substantial technological challenges on the path to developing soft interfaces of this kind, which would have to potentially gather a broad range of healthcare information while being wrapped around an arm or leg like silk. The fundamental features of such a sensor must be significantly improved to enable performance. This is why fundamental basic research is required to explore flexibility, sensitivity, repeatability, linearity, durability, and stimuli-responsive material, for instance. 

Top sum up: the key aims of the scientists’ research project are as follows: 

  • manufacturing pressure-sensitive tactile sensing which is strain invariant, and improving the interface between highly stretchable and biocompatible conducting materials which provide excellent adhesion
  • developing a sensing sleeve with a multi-stimuli response embedded into a single hybrid platform that could actively conform to the device or body without compromising efficacy, and
  • exploring innovative automobile, entertainment industry applications for cutting- edge soft sensors, including integration with mobile soft robots, rehabilitative systems, and possibly collision-aware surgical robotics

Airline pilots train many hundreds of hours in flight simulators before they take to the skies. In contrast, surgeons have very limited access to simulators, and those that are available do not offer sufficiently realistic conditions. Medical instruments used for robotic and minimally-invasive surgery are often tested on grapes or cans of meat and thus do not accurately reflect reality.

Dr. Tian Qiu, leader of the Cyber Valley “Biomedical Microsystems” research group, has set out to improve the situation. His research focuses on developing very realistic organ phantoms that optimize surgical training procedures and make them quantitatively measurable. Not only are these phantoms authentic physical replicas, they also have a cyber component. In other words. Qiu’s research program proposes to develop a “cyber-physical twin” of human organs.

Each 3D-printed organ twin is made of soft materials very similar to real organs in terms of anatomy and tissue properties. The cyber aspect is that the model can sense what it experiences and that this data is collected. Such data would be impossible to record if, for instance, a medical procedure were trained on a real human organ. With the data generated by a cyber-physical organ twin, the outcome of a surgery training session can now be clearly visualized, which is not even possible in a real surgical situation. The performance of a medical student who is training to become a surgeon could thus be evaluated automatically, and the feedback can be provided immediately after the training session to improve the training experience.

Such smart cyber-physical organ twins will one day transform surgical training. Tian Qiu and his team believe that they could gradually substitute medical training on human bodies and reduce animal experiments. The organ replicas not only offer the opportunity to develop and test new medical instruments, but also to develop better safety products such as helmets and airbags, for example when body part replicas are used in crash tests. Vital data on how they are affected in an accident can be collected and analyzed.

Learning to play a musical instrument is a long and difficult endeavor. Not everyone can afford the help of a professional teacher, and even with this help, feedback is limited in terms of latency and expressiveness. To tackle these problems, we will design a collection of new data-driven techniques and tools. The main idea is to systematically record musical practice data of students and feed it back through smart, visual interfaces. With a Visual Analytics web-tool, we will allow students, teachers, and professional musicians to detect errors and improve their style in a completely innovative way. By additionally recording motion data, we will also be able to convey fingering instructions or correct poses through augmented reality displays that visualize information directly attached to a physical music instrument.

As musical data can be complex, and notes or audio data signals recorded from instruments are usually noisy, AI is a useful, if not necessary vehicle for data processing and analysis. We will follow a human-centered design process that involves musicians and music teachers of different backgrounds and skill-levels in data acquisition, development, and evaluation of our techniques and tools. Our goal is to provide ready-to-use music education tools, re-usable data processing techniques, and datasets comprised of notes, audio, motion capturing, and other features that we record from instruments and players.

While most successful applications of machine learning to date have been in the realm of supervised learning, unsupervised learning is often seen as a more challenging, and possibly more important problem. Turing award winner Yann LeCun, one of the so-called “Godfathers of AI”, famously compared supervised learning with the thin “icing on the cake” of unsupervised learning. An approach called contrastive learning has recently emerged as a powerful method of unsupervised learning of image data, allowing, for example, to separate photos of cats from photos of dogs without using any labeled data for training. The key idea is that a neural network is trained to keep each image as close as possible to its slightly distorted copy and as far as possible from all other images. The balance between attractive and repulsive forces brings similar images together. In this project these ideas will be applied to single-cell transcriptomics, a very active field of biology where one experiment can measure gene activity of thousands of genes in millions of individual cells. The group will use contrastive learning to find structure in such datasets and to visualize them in two dimensions. They will then go back to the image data and use two-dimensional embeddings as a tool to gain intuition about how different modeling and optimization choices affect the final representation.

Animals generate movement in a fascinatingly efficient, dynamic, and precise way. They achieve this by a well-tuned dynamic interplay between nervous system and muscles where they exploit the visco-elastic properties of the muscles to reduce the neuronal load. This apparent computation performed by the body is termed morphological computation. Building on this idea, novel robotic systems, like muscle driven robots, soft robots, or soft wearable assistive devices are developed. However, the control of non-linear and elastic robotic systems is challenging.

In this project, we will employ machine learning approaches to learn a well-tuned dynamic interplay between controller and muscle(-like) actuator. The goal is to explicitly exploit the muscle properties and therefore rely on morphological computation. We will develop this approach with computer simulations of human arm movements which consider muscles and low-level neuronal control (like re exes). We will further add a model of a technical assistive device and learn a controller which helps to maximize the morphological computation in the human neuro-muscular arm model.

With our collaboration partners Syn Schmitt (Uni Stuttgart) and Dieter Büchler (MPIIS), we will also apply this approach to muscle-driven robotic systems. This will allow us to learn a control which also exploits morphological computation in such systems.

Learning to exploit morphological computation will provide a novel approach to controlling robotic systems with elastic actuators and soft structures with potential applications especially in human-robot interaction or assistance.

In recent decades, meteorologists have consistently improved weather forecasting systems, which have thus become increasingly complex. Sophisticated systems, such as the Consortium for Small Scale Modeling (COSMO) model, incorporate influences such as local topographical, soil, and vegetation properties. Despite these advances, the data remains  approximate because of spatio-temporal differences as well as interactions and influences that either cannot be observed or have not been taken into account. With their Distributed, Spatio-Temporal Graph Artificial Neural Network Architecture (DISTANA), Professor Martin Butz and his Neuro-Cognitive Modeling Group at the University of Tübingen’s Department of Computer Science have now developed a new approach, which may either enhance or serve as an alternative to traditional forecasting systems.

DISTANA applies inductive learning biases, which implement universal principles of weather dynamics, including the principle that system dynamics can be influenced by only partially observable or even fully unknown, but universally applicable local factors, and the principle that the propagation of weather dynamics over space is restricted to local neighborhoods in instances where temporal intervals are sufficiently small.

Over the course of this project, the researchers will be both developing combined weather prediction datasets as benchmarks and enhancing DISTANA. Ultimately, they expect DISTANA to outperform state-of-the-art weather forecasting systems, as it will be able to take unknown factors into account. Once successfully trained, DISTANA may be useful for weather forecasting on various spatio-temporal scales, potentially enabling better predictions of extreme weather events. This would in turn make it possible to take preventive measures accordingly. Additionally, the principle behind DISTANA may be applied in other areas, for instance water flow prediction, erosion modeling, or output prediction for wind park turbines. In the long term, this research could contribute to informing actions that aim to alleviate the negative impact of climate change.

Understanding how animals behave, in particular how animals move and interact with each other and their environment, is fundamental to addressing the most important ecological problems of our time. State of the art methods for tracking animal movement, which in turn allows us to understand their behavior, often disrupt their daily lives and are not scalable to vast environments. For this reason, Aamir Ahmad and his group will develop WildCap: a team of aerial robots that captures the movements of animals in a non-invasive way. Unlike existing methods, this novel approach allows the aerial robots to choose flight paths that provide optimal viewpoints for best estimating the movement of the animals and, in future, their behavior.
Thumb ticker sm allgower1
University of Stuttgart
Thumb ticker sm pi  053
University of Tübingen
Thumb ticker sm headshot2021
Max Planck Institute for Intelligent Systems
Thumb ticker sm kjkuchenbecker1
Max Planck Institute for Intelligent Systems
Thumb ticker sm bernhard sch%c3%b6lkopf  germany
Max Planck Institute for Intelligent Systems
Thumb ticker sm metin eth vertical
Max Planck Institute for Intelligent Systems
Square news event item blue
Porsche AG
Thumb ticker sm petergehler copy
Amazon
Square news event item green
ZF Friedrichshafen AG
Thumb ticker sm bosch research thomas kropf cr p 0578
Robert Bosch GmbH
Square news event item blue
BMW Group
Square news event item green
IAV GmbH