Overview

This two-week intensive workshop provides a hands-on dive into the theory and practice of deep learning applied to music and audio. Designed for practitioners, researchers, and engineers at all levels, the program moves from foundational signal processing and neural network concepts in Week 1, to a graduate-level survey of cutting-edge research and state-of-the-art models in Week 2. Participants will leave equipped to build, train, and adapt deep learning systems for real-world Music Information Retrieval (MIR) challenges using PyTorch.

Both weeks are available in-person at CCRMA, Stanford University, or online. All students receive the same teaching materials and access to the same tutorials and notebooks. In-person participants benefit from additional hands-on, 1:1 instructor discussion and direct feedback throughout the workshop.
 

Week 1 — Foundations of Deep Learning for MIR

A fast-paced introduction to audio and music processing with deep learning.

Week 1 brings you up to speed with the core theory and practice underpinning modern audio AI. Participants learn to analyze and manipulate digital audio signals programmatically with PyTorch and Librosa. Hands-on coding exercises accompany every major concept. Some familiarity with Python, linear algebra, and object-oriented programming is assumed.

What You'll Cover

• Digital Audio & Signal Processing — Analog-to-digital conversion, sampling rates, Nyquist theorem, quantization, the Short-Time Fourier Transform (STFT), and the mel-spectrogram.

• Spectral Audio Features — Zero crossing rate, spectral centroid, bandwidth, contrast, rolloff, chroma features, and MFCCs; understanding what each feature tells us about a sound.

• Linear Algebra & Calculus for ML — Vectors, matrices, dot products, gradients, partial derivatives, and the chain rule (the mathematical backbone of neural network training).

• Feedforward Neural Networks — Backpropagation, loss functions, the multilayer perceptron, and a hands-on XOR classification exercise.

• Autoencoders & Variational Autoencoders (VAEs) — Latent space representations and a hands-on lab building a pitch- and timbre-conditioned VAE for controllable guitar tone generation.

• Convolutional Neural Networks (CNNs) — Spectral feature-based instrument classification in PyTorch using mel-spectrogram inputs.

• Transformers & Self-Attention — Architecture walkthrough, multi-head attention, positional encoding, and implementation of a transformer-based pitch detector.

• Recurrent Neural Networks (RNNs) & LSTMs — Temporal modeling of sequences with a hands-on notebook generating ABC music notation.

• Fine-Tuning & Parameter-Efficient Adaptation — Introduction to LoRA (Low-Rank Adaptation) for adapting large pre-trained models.

• Research Methodology — How to conduct a literature review, identify research gaps, form hypotheses, design experiments, and navigate the academic publication process (conference papers, workshop papers, journal articles, and late-breaking results).

Week 2 — State-of-the-Art Research in MIR

A graduate-level survey of cutting-edge deep learning research for MIR, led by instructors and guest speakers from industry and academia.

Prerequisites: Completion of Week 1 (or equivalent experience in deep learning and audio signal processing).

Core Themes

• Generative Models — Diffusion-based synthesis, latent diffusion transformers (DiT), and VAEs for audio and music.

• Self-Supervised & Representation Learning — Learning from unlabeled audio, contrastive objectives, and music embeddings.

• Multi-Head Attention & Transformers at Scale — From BERT-style encoders to GPT-style autoregressive decoders applied to music.

• Music–Language Semantics — Joint audio–text models, cross-modal retrieval, and instruction-following in audio LLMs.

• Evaluation & Benchmarking — How to measure the capabilities and limitations of audio AI systems rigorously.

Models & Systems Covered

Students will study, implement, and adapt a curated set of architectures including: TCN, Transformer, RAVE, CREPE, GPT, Mustango, MusicGen, MusicLM, AudioLM, AudioCLIP, Jukebox, and more.

Research Papers & Benchmarks

Week 2 puts participants in direct dialogue with the literature that is actively shaping the field, including:

• Sketch2Sound (Flores García et al., ICASSP 2025) — A controllable generative audio model that synthesizes high-quality sounds from time-varying signals (loudness, brightness, pitch) and text prompts, implemented on top of a latent diffusion transformer with lightweight fine-tuning.

• MusicFlamingo (NVIDIA Research, 2025) — A large audio–language model for deep music understanding, combining theory-aware chain-of-thought reasoning with state-of-the-art performance on music QA, captioning, instrument recognition, and lyric transcription.

• The MUSE Benchmark (Carone, Roman & Ripollés, ICASSP 2026) — A diagnostic benchmark probing fundamental music perception and auditory relational reasoning in audio LLMs, revealing persistent gaps between current models and human musical experts.

• MMAR Benchmark (Ma et al., arXiv 2025) — A challenging benchmark for deep reasoning across speech, audio, music, and their combinations, testing the limits of multimodal audio AI systems.

• Suno — Examination of commercial text-to-music generation, its architecture insights, and open research questions it raises.

• Classic and foundational papers in MIR, including early neural approaches to beat tracking, music recommendation, automatic chord recognition, and source separation.

Week 2 culminates in group project presentations where students demo original work applying the models and research methods studied throughout the two weeks.
 

About the Instructors

Kitty Shi is an accordionist, pianist, bagpipes player, and music technologist. She received her PhD from CCRMA in 2021 and is currently a machine learning engineer at Pinterest. Her research focuses on computer-assisted expressive musical performance.

Iran R. Roman is Assistant Professor and Lecturer of Artificial Intelligence at Queen Mary University London (QMUL), leading research in theoretical neuroscience and multimodal machine perception. He holds a PhD from CCRMA and brings extensive experience teaching AI and signal processing at QMUL, Stanford University, New York University, and the National Autonomous University of Mexico. He has built and deployed AI models with Meta, Apple, Tesla, and Raytheon/BBN. iranroman.github.io