Frank Schneider and his research group have successfully completed their project \u201cResource-efficient autonomous training algorithms for deep learning\u201d at the University of T\u00fcbingen. The project developed more resource-efficient and autonomous training algorithms for neural networks that have the potential to reduce training costs and improve model performance. Their findings are publicly available to both academic and industry labs, offering the potential for this research to be transferred into real-world applications.<\/p>\n\n\n\n
The project was funded by the Cyber\u00a0Valley Research Fund and was carried out between 2022 and 2024. It aimed to design training methods that could automatically adapt its hyperparameters (such as the learning rate) during the training phase of deep neural networks, reducing the reliance on user-defined or well-tuned hyperparameters. This would bring about a decrease in computational demands and energy consumption, making deep learning more accessible to researchers outside of the few top academic or industry labs.<\/p>\n\n\n\n
An important outcome of the project was the development of the \u201cAlgoPerf: Training Algorithms\u201d benchmark: the first community-standard specifically designed to reliably measure the speeding up of neural network training resulting from algorithmic improvements. AlgoPerf was developed as an open collaboration within the MLCOmmons non-profit consortium, consisting of over 20 researchers from academia and industry, and led by two co-chairs: George Dahl (Google DeepMind) and Frank Schneider (University of T\u00fcbingen).<\/p>\n\n\n\n
The project\u2019s main achievement was to advance the evaluation standard for training methods, particularly those that do not require user-defined hyperparameters. All findings from this research project are publicly available, so that its algorithms and benchmark findings can be adopted by industry, potentially reducing training costs and improving model performance.<\/p>\n\n\n\n
Tatzel, L., Hennig, P., & Schneider, F. (2022). Late-Phase Second-Order Training. NeurIPS 2022 Workshop: Has it Trained Yet?. [open access]<\/p>\n\n\n\n
Eschenhagen, R., Immer, A., Turner, R., Schneider, F., & Hennig, P.<\/p>\n\n\n\n
(2023). Kronecker-Factored Approximate Curvature for Modern Neural Network<\/p>\n\n\n\n
Architectures. Advances in Neural Information Processing Systems 36 (NeurIPS). [open access]<\/p>\n\n\n\n
Kasimbeg, P., Schneider, F., Eschenhagen, R., Bae, J. et al. (2025). Accelerating neural network training: An analysis of the AlgoPerf competition. International<\/p>\n\n\n\n
Conference on Learning Representations (ICLR). [open access]<\/p>\n\n\n\n
Fernandez, A., Schneider, F., Mahsereci, M., & Hennig, P. (2025) Connecting Parameter Magnitudes and Hessian Eigenspaces at Scale using Sketched Methods. Transactions on Machine Learning Research (TMLR). [open access]<\/p>\n\n\n\n
The Cyber\u00a0Valley Research Fund was established to support Cyber\u00a0Valley research groups undertake basic research in the fields of artificial intelligence and robotics. The fund totaled five million euros, including contributions from six of Cyber\u00a0Valley\u2019s founding corporate partners: Amazon, BMW, Bosch, IAV, Mercedes-Benz, Porsche, and ZF. It supported 20 research projects, the first of which began in 2020, and the final of which will conclude in 2026.<\/p>\n","protected":false},"featured_media":11872,"parent":0,"template":"","meta":{"_acf_changed":false},"cv-tag":[],"location":[],"network-role":[],"news-category":[444],"user-group":[243,245,247],"class_list":["post-11871","news","type-news","status-publish","has-post-thumbnail","hentry","news-category-research-fund","user-group-investor","user-group-researcher","user-group-phd-student"],"acf":[],"yoast_head":"\n