Overview
Computer-vision systems can classify objects and segment images well, but they often fail when images are distorted or unfamiliar. Humans, by contrast, can interpret a wide range of visual input, from familiar landscapes to abstract paintings.
Led by Fabian Sinz at the University of Tübingen, this project investigated whether brain-activity data could help AI models process images more like humans. The project ran from 2019 to 2024.
Key results
- Developed a method that uses data from how the brain responds to images to guide an AI model’s behaviour, rather than trying to copy the brain’s internal mechanisms directly.
- Trained models on brain-activity data alongside standard computer-vision tasks.
- Found that these models showed an improved ability to process visual distortions, even though they had not been explicitly trained for that task.
- Observed that the models tended to focus on the parts of images that humans find important, indicating more human-like visual processing.
- Produced peer-reviewed publications at AISTATS and NeurIPS, including work on multi-task learning using monkey visual-cortex data and on generating images that excite neural responses.
Why it matters
The project addresses a central challenge in computer vision: making AI models more robust when images are distorted, degraded, or unfamiliar. By using brain responses as a guide rather than as a direct template, it offers a way to build systems that interpret visual information more like people. The findings could support applications in autonomous driving, robotics, and medical imaging, where reliable perception under varied and imperfect conditions is essential.
This project produced the following peer-reviewed publications:
- Nix, A., Shrinivasan, S., Walker, E. Y., and Sinz, F. H. (2022). Can Functional Transfer Methods Capture Simple Inductive Biases? In International Conference on Artificial Intelligence and Statistics (AISTATS).
- Pierzchlewicz, P. A., Willeke, K. F., Nix, A. F., Elumalai, P., Restivo, K., Shinn, T., Nealley, C., Rodriguez, G., Patel, S., Franke, K., Tolias, A. S., and Sinz, F. H. (2023). Energy Guided Diffusion for Generating Neurally Exciting Images. In Advances in Neural Information Processing Systems (NeurIPS).
- Safarani, S., Nix, A., Willeke, K., Cadena, S. A., Restivo, K., Denfield, G., Tolias, A. S., and Sinz, F. H. (2021). Towards robust vision by multi-task learning on monkey visual cortex. In Advances in Neural Information Processing Systems (NeurIPS).
- Nix, A., Burg, M., Sinz F., Leading by example: Guiding knowledge transfer with adversarial data augmentation. NeurIPS 2022 Workshop SyntheticData4ML.
