News Category: Press Release

Tübingen – Justus Thies will start a position as a Max Planck Research Group Leader on April 1, 2021. His “Neural Capture & Synthesis” group will be situated at the Tübingen site of the Max Planck Institute for Intelligent Systems (MPI-IS). Previously a postdoctoral scientist at the Technical University of Munich (TUM), Thies plans to push the boundaries in the research field of capturing and synthesizing the real world, which includes people, objects, rooms, and even entire scenes. His goal is to synthesize and model the human body and objects so realistically that a viewer will no longer notice the difference. With the rise of deep learning methods in general and neural rendering in particular, Thies’s facial reenactment method has reached a quality that raised a great deal of attention in academia, industry, and media. By combining novel findings in machine learning with classical computer graphics and computer vision approaches, his technology shows promising results.

Creating ever more realistic avatars

Thies’ research could be applied in a broad range of fields, from autonomous vehicles or household robots to medical applications and video post-production. However, his main goal is to revolutionize telecommunications. “I want to change the way people interact with each other at a distance,” he says. “In the future, if we want to speak to a person who is elsewhere, we will wear a virtual or augmented reality lens that projects a photorealistic 3D avatar of that person right in front of us. The person will be digitally reconstructed in a way that feels like they are in the same room. This will be an amazing experience, but only once the digital human and their appearance, expressions, and natural movements are indistinguishable from the real person. Hence, I aspire for everyone to be able to create a realistic 3D avatar of themselves whose movement is realistic.”

Thies’ goal is for users to capture themselves with cameras that are commonly available, without having to rely on a complex setup to create their own avatar. Placing people in a 4D body scanner would be much too complex and expensive. “One idea is to work with the cameras of game consoles, for example, whose Kinect sensors record color and depth. A built-in distance sensor measures how long it takes for light to travel from the camera to the scene and back. This gives me a three-dimensional point-cloud of a person from which I can then create an avatar.”

Syncing the movement of a face with the input audio

Revolutionizing the way people communicate is one potential area of application. Another is in video post-production, where dubbed films would show the actors' lips moving in the target language. “To learn the expressions and appearance of a target face, our algorithm only needs a short video clip of about two minutes,” Thies says.

Also known as real-time facial reenactment, audio-driven facial video synthesis, or neural voice puppetry, this technology is only just beginning to gain momentum. It is driven by a deep neural network that employs a latent 3D face model space. In this way, a video can be created showing a celebrity speaking in a different voice or even showing the person saying things in their own voice that they never said in real life.

“This is the negative side of facial reenactment, as the generation of photorealistic imagery can be misused. The algorithms applied when dubbing and lip-syncing a film can also be used to create a speech by a head of state, putting words in their mouth that are fake and untrue. If people don’t notice the hoax, this could potentially cause confusion, disruption, and even undermine a society. Deep fakes are increasingly popping up on the internet, and they can be difficult to recognize. Digital multi-media forensics is thus another pillar of my research. Here, I aim to develop algorithms that automatically detect whether a video sequence is synthetic or has been manipulated.”

Understanding deep fake technology in order to detect it

Multi-media forensics is about training an AI algorithm with a large dataset so it can identify fake videos based on the artifacts visible in a video sequence – often occurring along the seam between the synthesized face and the real background. The algorithm also examines the colors’ noise behavior and assesses whether it is realistic. In this way, artificial intelligence can detect relatively easily if there has been a manipulation. “This is exactly the kind of technique that can then be automated,” Thies says. “There are already attempts to offer this deep-fake detection technique as a browser plug-in for anyone surfing the internet, which automatically recognizes and flags fake images and videos. This would protect against fake news.”

It’s a game of cat and mouse, as deep fakes are continuously improving. Nowadays, it is relatively easy to recognize a deep fake. In the near future, however, it will become increasingly difficult. Creating ever-more photorealistic video content thus comes with a huge responsibility. “My work raises the bar so that not just anyone can fake a video that remains unnoticed. By gaining knowledge about the creation process, we can design the most advanced forgery detection algorithms. You have to be at the forefront of this technology in order to detect, flag, and reveal the fakes,” Thies concludes.

About

Justus Thies is formerly a postdoctoral researcher at the Technical University of Munich. In September 2017, he joined Prof. Matthias Nießner’s Visual Computing Lab. Previously, he completed his Ph.D. at the University of Erlangen-Nürnberg under the supervision of Gunther Greiner. Thies received his master of science from the University of Erlangen-Nürnberg. During the time as a doctoral student, he collaborated with other institutions, with internships at Stanford University and the Max Planck Institute for Informatics in Saarbrücken.

A medical project marked the beginning of Justus Thies’s research on capturing and synthesizing image and video content. The aim was to capture the faces of patients with cleft palates. For several decades, patients were photographed again and again, allowing scientists to see how their faces changed over the course of treatment and how their wounds healed. With the approach he developed, patients with a cleft palate can now see the results of particular treatment methods in ten years’ time.

His research can be watched on this Youtube channel.

Their work “Modeling and Control of a Soft Robotic Fish with Integrated Soft Sensing” was published in Wiley’s Advanced Intelligent Systems journal, in a special issue on “Energy Storage and Delivery in Robotic Systems”.

Each side of the robotic fish is lined with soft artificial muscles made of rubber-like small silicone chambers. Through these pockets, the researchers pumped air alternately on each side. While the air pockets on one side expand, they create a curvature, contracting the air pockets on the other side. This causes the robotic fish to bend right and left, swinging its tail, resembling the wave-like movement pattern seen by real fish.

To measure the bending, the scientists embedded state of the art soft strain sensors, consisting of a thin layer of windy microchannels filled with liquid metal and encapsulated in silicone. What was now like a wire, could stretch like a phone cord to a length of 900%. The more the air pressurized each silicone chamber of the actuator, the more the sensor placed atop bends, and its liquid metal ‘wire’ was elongated. The scientists then connected the sensor to an Ohmmeter to measure the electrical resistance. The greater the stretching elongation, the higher the electrical resistance within the ‘wire’. This number provided the scientists with data of how the curvature of their robotic fish changed given a certain change of air pressure.

The scientists then put this robotic fish platform under water in a flow tank. In several experiments they tested how the air pressure controller took the reading from the sensor to measure the swimming performance of the robot – an information loop that was feeding a self-learning algorithm inside the controller to push just the right amount of air through the pneumatic pockets so the swimming perfectly matched the different water flow velocities.

“We developed a hydrodynamic model that can predict how our soft robotic fish will behave. This builds upon previous findings where we measured peak thrust during swimming with co-contraction and body stiffening, and tested soft sensors in feedforward undulatory motion. Closing the loop with proprioceptive soft sensory feedback allows the robot to respond to different flow conditions”, Ardian Jusufi explains. He is the corresponding author of the publication and specializes on soft active materials, biomimetics, and robotics. Jusufi is the head of the Max Planck Cyber Valley research group for “Locomotion in Biorobotic and Somatic Systems” at the MPI-IS. “Our project builds upon several previous collaborative papers on soft robots and biorobotic modeling of swimming robots. To advance modeling and closed-loop control, our team has combined soft robotics expertise from three continents.”

“In this work, we have used a simple approach to build a data-driven model of a soft robotic fish, and extended it with the controller design. This model could also be easily extended without the need of completely rebuilding it, for example to study the scaling effects of robots or testing different types of sensing techniques”, describes Yu-Hsiang Lin of the “Locomotion in Biorobotic and Somatic Systems” group at the MPI-IS, and first author.

In their publication, the scientists describe the sensors as a completely new design technique, labelling them “hyper-elastic liquid metal strain sensors.” Developed by Prof. Yong-Lae Park and his team at Seoul National University, and Daniel Vogt from Harvard University, they are a true innovation. “In biology as well as in soft robotics, with an entirely soft robot, we have infinite degrees of freedom. That means any part of the body can deform. It is hard to estimate how the fish’s shape changes, because we cannot put an indefinite number of sensors onto its body, as it would be possible with rigid robots with a limited number of joints,” Park says.

“This robot will allow us to test and refine hypotheses regarding the neuromechanics of swimming animals as well as help us improve future underwater robots,” Jusufi continues. “In addition to characterizing the soft strain sensor under submerged dynamic conditions for the first time, we also developed a simple and flexible data-driven modelling approach in order to design our swimming feedback controller. The model will be of interest to the scientific community and will help accelerate future work in design and operation of soft robots. In forthcoming research, we will also utilize soft strain sensors in terrestrial robots that climb and can overcome complex obstacles.”

Understanding the underlying mechanisms how soft animals like fish swim helps in designing artificial soft structures and make it possible to use them to build robots. Such animal-inspired robots could one day be used to explore the depths of the sea and provide valuable data of marine life, avoiding noise and reducing the risk of entanglement that rigid, conventionally propelled submarines face. Swimming robots could also become a more energy-efficient option, which is why more and more scientists have shown a great deal of effort in developing soft actuators and sensors that constantly improve the capabilities of swimming robots by replicating the locomotion of aquatic vertebrates.

What could one-day serve as a more advanced under-water vehicle also serves another purpose: the fish replica with sensory feedback offers important insights into fish neuromechanics and their morphological intelligence, advancing the field of biology.

Since November 2019, the Stuttgart-based Fraunhofer Institute for Industrial Engineering IAO and Fraunhofer Institute for Manufacturing Engineering and Automation IPA have formed part of Cyber Valley, the largest research cooperation network in Europe for Artificial Intelligence (AI), with the AI Innovation Center “Learning Systems”. For companies located in Baden-Württemberg, the Center represents the main point of contact for application-oriented AI research, implementing research and transfer projects for production and services. It also acts as an interface between industry and basic research within the existing Cyber Valley consortium, which comprises several research institutes located in the Stuttgart-Tübingen area in addition to reputable industrial partners. In addition, the AI Innovation Center facilitates technology transfers to industrial settings by leveraging the strengths of existing partners.

Broad spectrum of AI and robotic technologies

As part of the virtual S-TEC summit held on 25 February 2021 under the motto “AI – Made in the Ländle”, Dr Nicole Hoffmeister-Kraut, Minister of Economic Affairs, Labour and Housing for the state of Baden-Württemberg, announced additional financial backing for the AI Innovation Center: “The combination of Artificial Intelligence and robotics opens up major opportunities for the economy of Baden-Württemberg. By expanding the Innovation Center, we are making a decisive contribution to strengthening the long-term development and successful commercialisation of these key technologies in our state. Our overall aim is to transform the latest research expertise into innovative products and business models even more effectively and rapidly. In so doing, our intention is to also contribute to ensuring that the economy is in a position to overcome the negative impacts of the coronavirus pandemic as quickly as possible, before ultimately emerging even stronger from the crisis.”

In the second funding phase, the Center will be expanded to include Fraunhofer IPA services in the area of “Cognitive Robotics”. Rebranded as the AI Innovation Center “Learning Systems and Cognitive Robotics” Center, the facility will now also support companies in their efforts to fully exploit the potential of service and industry robotics in addition to finding answers to megatrends such as demographic change, individualisation, sustainability and digitalisation. The Center will focus on generating technological developments from four key research areas:

•    Human-machine interactions: By enabling machines to perceive, understand, imitate and actively support humans, machines will become intuitive and easy to operate.
•    Cyber-physical robot systems: Services should help to put existing robot systems into operation more quickly, make them more precise and enable them to be programmed and commissioned more easily.
•    Perception and interaction: With the help of AI, robots should be able to carry out tasks autonomously even in unstructured and partially unknown environments.
•    Networked robotics: Networking serves the purposes of knowledge transfer, mutual learning and teamwork among robots, but also between humans and robots.

Helping companies to exploit technological benefits

All of these developments help to facilitate, simplify and render more economical an initial foray into the world of robotics in addition to the further development and quality improvement of existing applications. “With cognitive robotics, we are developing pioneering automation solutions for production processes as well as the retail, healthcare and agricultural sectors”, comments Dr Werner Kraus, Head of the Robot and Assistive Systems department at Fraunhofer IPA and Co-Director of the AI Innovation Center. The institute is therefore underlining its aim to become one of the leading institutes for cognitive robotics in the Fraunhofer-Gesellschaft.

As of now, any interested companies can submit applications to the AI Innovation Center for project funding in the area of AI for services and production as well as cognitive robotics. In so doing, they can determine the potential of the technologies for their individual application. The application deadline closes on 18 March 2021. All further information can be found on the website.

 
Practical applications of AI: 13 application projects completed

The topic of “Learning Systems” and the direct cooperation with industrial firms as well as networking and knowledge transfer within Cyber Valley remain important components of the AI Innovation Center. Since October 2019, Fraunhofer’s expert AI teams have already implemented 13 different AI applications in the form of prototypes as part of ‘exploring projects’. These cover a broad spectrum of application scenarios for AI: from text mining and processing for legal documentation and demand forecasts for an online bookseller or the combined automation solution for quality control and robot-supported packing all the way through to hazardous goods analyses. These application examples show that AI offers added value in a multitude of different areas. “For this, we are not only using the latest technology in the area of machine learning, but also prioritising efforts to ensure that these processes are robust and reliable and, in particular, we improve the ways in which these can be explained and understood”, comments Prof. Marco Huber, Head of the Center for Cyber Cognitive Intelligence at Fraunhofer IPA and Co-Director of the AI Innovation Center. He continues: “After all, this is often the primary basis for ensuring that the technology is actively embraced.” Dr Matthias Peissner, Head of the Human-Technology Interaction research unit at Fraunhofer IAO, also explains: “The Center offers companies a safe space to test ideas for AI applications without any financial risk. In this way, they learn to appreciate and exploit the potential of machine learning.”

Studies on human-centric AI at the interface of business and elite-level research

A total of ten studies developed at the AI Innovation Center provide an overview of which industries and topics are particularly well-suited to innovative AI applications for operational practice and how these can be effectively implemented. Over the next few weeks, these will gradually be made available on the website of the AI Innovation Center free of charge.

One of the university's strategic goals is to build the knowledge society needs to shape its future responsibly. With the IRIS network, researchers aim to cooperate with one another in a process of critical reflection on the foundations, mechanisms, implications, and effects of intelligent systems in research, teaching, and society as a whole. For example, at the interface between AI and ethics, the researchers are examining how discrimination can be prevented when artificial intelligence is used in automated decision-making, or how stereotyping can be avoided in technology development.

"The new IRIS research network illustrates what the University of Stuttgart does especially well: discussing and researching hot topics in an interdisciplinary manner, which includes a broad range of perspectives," said Professor Wolfram Ressel, Rector of the University of Stuttgart. “IRIS brings together scientists from all disciplines at the university, from the AI researcher and the philosopher to the linguist and software engineer. Together, they are working on designing intelligent systems in a manner that is responsible and sustainable. Ultimately, the expertise pooled within the IRIS network will enable us to be leaders in reflecting on intelligent systems.”

Among other things, IRIS spokesperson and Junior Professor Maria Wirzberger and her team are concerned with the question of how technology can discriminate and how this can be counteracted. "Put simply, algorithms are based on standards that are set by people and can thus carry the same stereotypes within them. These stereotypes are often unconscious and thus flow into technology development unnoticed." Co-speaker Professor Steffen Staab works with his team across disciplines to gain a comprehensive understanding of technical decision-making, for instance loan approval through artificial intelligence. "In recent years, we have learned how the naive use of AI for automated decisions can lead to discrimination. For this reason, we are developing new methods to avoid, detect, and explain unfairness," he says.

Other IRIS activities

In addition to research, IRIS creates opportunities for dialog on current ethical and societal challenges related to intelligent systems. Discussions between members of society, industry the university, and beyond focus on topics such as data ethics, informational self-determination, and trustworthy AI. Moreover, IRIS also focuses on teaching: the Reflecting on Intelligent Systems in the Next Generation (RISING) initiative, led by Wirzberger, teaches students across disciplines to critically reflect on intelligent systems.

For a responsible approach to intelligent systems

How does IRIS help address stereotypes in technology development? "The benefit is that students, when they enter the labor force, are sensitized to this issue. Then things like discriminatory technology don't happen to the same extent," says Wirzberger. Secondly, IRIS strengthens the university's international network. When people of different backgrounds come together, good ideas emerge and there is a lively exchange, the scientist says. Scientists from the clusters of excellence and all of the university's disciplines are involved in IRIS. The network is funded by the German Research Foundation (DFG) as part of the Excellence Strategy of the German federal and state governments, as well as by the university's research fund.

In the field of computer vision, a key challenge lies in teaching machines to “see” and draw conclusions about complex 3D scenes, just as human beings do. The ultimate aim is enabling technologies in realms such as autonomous driving and virtual reality to operate safely and reliably. While science is still a long way from achieving this, researchers at the University of Tübingen and the Max Planck Institute for Intelligent Systems (MPI-IS) have taken the field one step further with GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. They present their work this week at the Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS 2020). One of the world’s leading machine learning conferences, NeurIPS 2020 is being held virtually until December 12.

“Human beings are capable of looking at a 2D image and imagining exactly what it looks like from different angles and viewpoints in 3D,” said Katja Schwarz, the paper’s lead author.

“But they can do even more: they can also imagine completely new scenes because they understand the underlying concepts required to create them. This calls for a sophisticated ability to reason in 3D that machines simply don’t have yet. Machines can already generate new 2D images really well, but science is still working on enabling them to reason better in 3D and learn abstract three-dimensional concepts. With GRAF, we offer an approach that vastly improves 3D-aware image synthesis for single objects and takes an important step in this direction.”

Schwarz is currently a PhD student in the Autonomous Vision Group, which is split between the University of Tübingen and the MPI-IS. Her co-authors were fellow researchers Yiyi Liao and Michael Niemeyer, as well as Andreas Geiger, who is group leader.
While methods for 3D-aware image synthesis already exist, they often require 3D training data or multiple images from the same scene with known camera poses. This information can be difficult to collect in real-world situations. An autonomous vehicle, for instance, must be able to operate safely when pedestrians suddenly cross the road or other unexpected obstacles appear. For obvious reasons, such scenes cannot be pre-recorded in real-life, making simulation an important part of the training pipeline.

In response to this challenge, the scientists aimed to create a 3D-aware model that learns only from 2D images without viewpoint information. This is a challenging task, and existing approaches either produce low-image resolution or inconsistent object depictions when the viewpoint changes. With GRAF, the scientists set out to create a model that scales to high image resolution while maintaining consistent object depictions. In the long term, such models could generate realistic simulations for training robots or autonomous vehicles.

To find out how a generative model could be made aware of the world’s 3D nature solely on the basis of 2D images, the researchers incorporated a virtual camera into their model. They then controlled its position so it could view the objects in the images from different angles. The 3D objects are modeled in a memory-efficient way that uses specific representations known as radiance fields. This is key for scaling the method to high image resolution. For a given camera viewpoint, every 3D object can then be rendered to a 2D image. By comparing the generated renderings to real images, the model learns to generate 3D-consistent images across camera viewpoints.

“We analyzed our model on several synthetic and real-world datasets, and we found that our approach with radiance fields scales well to high-resolution while maintaining consistency across different camera viewpoints,” said Schwarz. The scientists not only tested GRAF on images of objects like cars and chairs, but also on natural images of people, birds, and cats, with very good results. In a next step, they plan to extend their approach beyond single objects to even more complex real-world scenes.

More information about the GRAF project can be found at https://autonomousvision.github.io/graf/.