Overview
Learning to play a musical instrument requires extensive independent practice, but learners cannot always receive regular feedback from a teacher. Led by Michael Sedlmair and Frank Heyen at the University of Stuttgart, the InstruData—Data-driven Musical Instrument Education project developed visual tools that help musicians identify mistakes and monitor their progress while practising.
The project ran from 2022 to 2025. Its Visual Analytics web tool uses data from practice sessions to generate visual feedback. The researchers designed it as a supplement to, rather than a replacement for, traditional music education.
Key results
- Developed methods that automatically colour parts of sheet music, making repetitions, changes, and the structure of a musical piece easier to identify.
- Created visualisations that show in real time how consistently a musician performs a particular rhythm across repeated attempts.
- Developed an interactive menu controlled directly through an electric guitar. The menu follows the grid-like structure of the guitar fretboard and requires neither modifications to the instrument nor additional hardware.
- Produced related tools for AI-assisted composition and augmented-reality guitar tutorials, including guitARhero.
- Developed GuitarPie, an audio-based interface that uses the guitar fretboard to control a computer menu.
- Produced six peer-reviewed publications between 2022 and 2026, alongside conference contributions and open-access research outputs.
Why it matters
The project demonstrates how practice data can be transformed into visual feedback that helps musicians notice errors, understand musical structure, and monitor their consistency. This could make independent practice more efficient and support learners who have limited access to teachers. With further refinement and user-experience development, the tools could be adapted into smartphone or web applications.
This project produced the following peer-reviewed publications:
- Frank Heyen, Michael Gleicher, Michael Sedlmair (2026). Make the Unhearable Visible: Exploring Visualization for Musical Instrument Practice. In IEEE Transactions on Visualization and Computer Graphics (TVCG). https://doi.org/10.1109/TVCG.2026.3658216
- Simeon Rau, Frank Heyen, Stefan Wagner, & Michael Sedlmair (2022). Visualization for AI-Assisted Composing. In Proceedings of the 23rd International Society for Music Information Retrieval Conference (ISMIR). https://doi.org/10.5281/zenodo.7316618 (open access)
- Lucchas R Skreinig, Denis Kalkofen, Ana Stanescu, Peter Mohr, Frank Heyen, Shohei Mori, Michael Sedlmair, Dieter Schmalstieg, & Alexander Plopski (2023). guitARhero: Interactive Augmented Reality Guitar Tutorials. In IEEE Transactions on Visualization and Computer Graphics (TVCG). https://doi.org/10.1109/TVCG.2023.3320266 (open access)
- Frank Heyen, Quynh Quang Ngo, & Michael Sedlmair (2023). Visual Overviews for Sheet Music Structure. In the 24th International Society for Music Information Retrieval Conference (ISMIR). https://doi.org/10.5281/zenodo.10265383 (open access)
- Simeon Rau, Frank Heyen, Benedikt Brachtel & Michael Sedlmair (2025). MAICO: A Visualization Design Study on AI-Assisted Music Composition. In IEEE Transactions on Visualization and Computer Graphics (TVCG). https://doi.org/10.1109/TVCG.2025.3539779
- Frank Heyen, Marius Labudda, Michael Sedlmair, & Andreas Fender (2025). GuitarPie: Using the Fretboard of an Electric Guitar for Audio-Based Pie Menu Interaction. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST). https://doi.org/10.1145/3746059.3747799 (open access)
