Count the Notes: Histogram-Based Supervision for Automatic Music Transcription

Abstract

Automatic Music Transcription (AMT) converts audio recordings into symbolic musical representations. Training deep neural networks (DNNs) for AMT typically requires strongly aligned training pairs with precise frame-level annotations. Since creating such datasets is costly and impractical for many musical contexts, weakly aligned approaches using segment-level annotations have gained traction. However, existing methods often rely on Dynamic Time Warping (DTW) or soft alignment loss functions, both of which still require local semantic correspondences, making them error-prone and computationally expensive. In this article, we introduce CountEM, a novel AMT framework that eliminates the need for explicit local alignment by leveraging note event histograms as supervision, enabling lighter computations and greater flexibility. Using an Expectation-Maximization (EM) approach, CountEM iteratively refines predictions based solely on note occurrence counts, significantly reducing annotation efforts while maintaining high transcription accuracy. Experiments on piano, guitar, and multi-instrument datasets demonstrate that CountEM matches or surpasses existing weakly supervised methods, improving AMT's robustness, scalability, and efficiency.

Qualitative Examples

Each video begins with the original audio and transitions into the model's transcription. The transcriptions are generated by models trained using our proposed counting-based alignment approach. Under each video, we also provide a link to the Original Performance for reference and comparison.

MAESTRO Histogram Supervision Results

Note-level transcription results for training with histogram-based supervision on the MAESTRO dataset. We report Precision (P), Recall (R), and F-score (F) for test and train sets across different histogram window sizes (or Full Track). For reference, results include a baseline trained on synthetic data only (Sy) and a supervised model (Sup).

Cross-Dataset Evaluation

Training was performed on MusicNet, with evaluation on MAESTRO, GuitarSet, and URMP. For URMP, we also report F-histogram, which does not enforce the 50ms onset threshold.