The Gradus Harmony Benchmark

Why this exists

Harmonic analysis produced from memory — by language models and by hurried humans — invents chords. The documented case that motivated this benchmark: bars 39–47 of Beethoven's Ninth, first movement, hold nothing but a bare open D–A fifth, yet a hand-authored published analysis narrated a full modulation through F major and C major across them (the receipts). There was no public, expert-labeled test set on which such claims could be checked measure by measure. Now there is one.

This is deliberately a hard set. Items entered human review precisely because independent readings of the same bar disagreed or a published claim looked suspect — and 75% of the original claims did not survive adjudication. Scores near 100% are not expected from anyone; the set measures behavior exactly where analysis is genuinely difficult.

What is in the data

384 candidate bars were adjudicated by a human reading the full score part by part. 361 are gradeable harmony items; 8 are texture-only items (bars whose content is a line or bare sonority, where no chord label applies); 15 were excluded (unresolvable span or a gap in the encoded score) and are listed by id for census transparency.

Each item carries the work, movement, measure, the accepted answer(s) as Roman numerals with their key-frame annotations, the adjudication class (confirmed, corrected, multiple-readings), and originalClaim — the published claim the review checked, preserved verbatim. The claim-vs-answer pairs make the set usable for studying how analysis goes wrong, not just scoring.

The music is 13 symphonies by Beethoven, Brahms, Bruckner, Dvořák, and Tchaikovsky across 33 movements. Every movement's MusicXML score is downloadable from the URLs in the data — predictions should be produced from the score, not from memory of the piece.

Grading

One rule: frame-aware chord identity. Every accepted answer resolves to a concert chord — root plus quality, inversions folded, a seventh on a major-quality numeral read as a dominant seventh — in its key frame: the answer's own in X annotation when present, otherwise the movement's stated homeKey. A prediction matches if any of its readings resolves to the same identity. Predictions are Roman numeral + key, or a concert chord directly, so tools with different vocabularies compete on what they actually claim the sonority is.

Two numbers are reported: primary-reading accuracy (your first reading only) and any-reading accuracy (alternates count). Texture-only items are scored separately and never enter the denominator. grade.mjs is the reference implementation — about two hundred lines, no dependencies.

curl -O https://gradusmusic.com/benchmark/gradus-harmony-benchmark-v1.json
curl -O https://gradusmusic.com/benchmark/grade.mjs
node grade.mjs gradus-harmony-benchmark-v1.json my-predictions.json

# my-predictions.json — first reading is your primary:
# { "beethoven9-mvt1-m41": [ { "rn": "i", "key": "D minor" } ],
#   "brahms2-mvt1-m10":    [ { "chord": "D maj" } ] }

Baseline

Reproduce it: the baseline predictions file above was generated from the analyzer's committed output and graded with the published grade.mjs — the number on this page is the number the harness prints. Run a model or tool of your own and send the result; verifiable submissions get listed.

How the labels were made

Candidates came from a 3,315-claim sweep of our own hand-authored, published measure-level analyses; the hard subset — bars where sources disagreed or a claim looked wrong — went to human adjudication in 2026-08. Every item was ruled by a reviewer reading the full orchestral score with per-part, per-beat extraction tooling; nothing was machine-labeled. The conventions: the bass is authoritative for inversions; pedal points are annotated (ped N) with the upper structure as the chord, never a slash chord; local frames are stated as in X; split bars carry per-beat tags; sustained bare sonorities (unisons, octaves, open fifths) are texture calls, not chords. Bars that could not be ruled inside a bounded deliberation were dropped rather than guessed.

The same review produced fixes to the source analyses and three analyzer features (per-onset texture classification, score-spelled key names, cadential sponsorship on detected key sections) — the benchmark, the corrected corpus, and the analyzer's hedging all come from one process.

License and citation

The labels are licensed CC BY 4.0 — reuse them, including in software and training data, with attribution. The scores are public-domain music, encoded editions provided as-is.

Cite it as: Gradus Harmony Benchmark v1.0.0 (2026), gradusmusic.com/harmony-benchmark. Item ids are permanent; future versions add items and never renumber.