Exploring representation transfer and neural model innovations

From November 2019 to March 2024, the Cyber Valley Research Fund supported Prof. Dr. Fabian Sinz’s project “Mechanisms of representation transfer”. The project aimed to enhance how artificial neural networks learn and apply knowledge without limitations across different tasks. It also endeavored to develop methods for transferring inductive biases, or inherent preferences in learning patterns, from one neural network (teacher) to another (student). 

Sinz and his research team experimented with various augmentor networks to find efficient ways to modify training images, facilitating the transfer of knowledge. The team used techniques ranging from simple affine transformations to complex deep variational autoencoders (VAEs). This was particularly evident in tasks where robustness and pattern recognition abilities were successfully transferred from ResNet architectures to Vision Transformer Networks.  

In parallel to this project, research partner Arne Nix also contributed to two significant projects on neural models and visualization. In collaboration with Willeke & Restivo, Nix developed an attention-based readout head for neural system identification in macaque area V4. This work showcased similar features to deep neural networks and is currently under journal review.  Furthermore, Nix contributed to a method that guides diffusion models with an energy function to create Most Exciting Images (MEIs) and Most Exciting Natural Images (MENIs). This innovative approach provided potential applications for using diffusion models for knowledge transfer that could be used in the main project. 

This project yielded the following publications: 

1. Pawel A. Pierzchlewicz, Konstantin F. Willeke, Arne F. Nix, Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, Andreas S. Tolias, Fabian H. Sinz: Energy Guided Diffusion for Generating Neurally Exciting Images NeurIPS 2023 

2. Arne Nix, Max Burg, Fabian Sinz HARD: Hard Augmentations for Robust Distillation arXiv 2023

3. Konstantin F. Willeke1, Kelli Restivo1, Katrin Franke, Arne F. Nix, Santiago A. Cadena, Tori Shinn, Cate Nealley, Gabby Rodriguez, Saumil Patel, Alexander S. Ecker, Fabian H. Sinz2, Andreas S. Tolias2 Deep learning-driven characterization of single cell tuning in primate visual area V4 unveils topological organization bioRxiv, equal contribution: 1, 2  

Prof. Fabian Sinz holds the W3 chair of Machine Learning at the Georg August University of Göttingen, Germany. From 2018–2021 (and part-time until 2024), he was a member of the Cyber Valley faculty as an independent group leader at Eberhard Karls University Tübingen.