CODE-test (827-record hold-out test set)

827 twelve-lead ECGs from 827 distinct patients of the CODE cohort, published as the evaluation set of Ribeiro et al. 2020. It is one of the smallest datasets in this catalogue and one of the most useful, because of how heavily it is annotated.

Seven independent readings of every record. Six binary abnormalities — 1dAVb, RBBB, LBBB, SB, AF, ST — were labelled independently by two cardiologists, with a third senior specialist adjudicating every disagreement to produce the gold standard. The same 827 tracings were then annotated separately by two 4th-year cardiology residents, two 3rd-year emergency residents and two 5th-year medical students, and scored by the paper’s own neural network. ECGBench exposes all seven side by side, so reader-agreement analysis needs no extra plumbing.

This is an evaluation set, and ECGBench still gives it ten folds. That is the framework applying its convention uniformly, not a recommendation. The release’s intended use is all 827 records as a single hold-out set — see the loading snippet below for the one-line way to get exactly that. Train on CODE-15%, not on eight tenths of a test set.

Nothing in the release has an identifier. ecg_tracings.hdf5 holds one (827, 4096, 12) array and nothing else; attributes.csv and the seven annotation CSVs each hold 827 unkeyed rows aligned to it by position alone — “the i-th line corresponds to the i-th tracing”, per the bundled README. So ECGBench’s record_id is the row index, 0–826, and every source file is refused unless it has exactly 827 rows. A positional join against a table of the wrong length does not partially match; it silently mislabels everything.

Its lead order is not standard. The augmented leads run aVL, aVF, aVR, so signal[3] is aVL. Its sibling CODE-15% — same cohort, same 400 Hz, same array layout — uses the standard aVR, aVL, aVF. Select leads by name if you touch both.

Every figure on this page was recomputed from the shipped files, not copied from the paper. The local copy’s data.zip matches Zenodo’s own md5 (cef8fe457abf4b8f66a34dc1269b4ced) exactly, so it is the v1.0.3 release unmodified. 827 records and 827 patients agree with the paper.

Four things the table above will mislead you about if left unqualified:

cardiologist1 and cardiologist2 are not independent of the gold standard. The gold standard is their agreement, with a third specialist breaking the ties. Scoring either against it measures how often that reader was overruled, not inter-rater reliability between strangers.

The three non-expert rows are each two people. Every one of the residents’ and students’ sets was produced by a pair who annotated half the records each, and the release does not say which record went to which annotator. A gap between medical_students and gold_standard is partly a difference in training level and partly a difference between two individuals; nothing in the data can separate the two.

It is multi-label, so the rows do not sum to the “≥1 finding” column. 12 records carry more than one gold-standard finding, which is why the six class counts total 158 against 146 records.

681 records (82.3%) carry none of the six — and that is not “normal”. This release publishes no normal flag at all, unlike CODE-15%, so the negative class here means strictly “none of these six findings” and nothing more.

Demographics: ages run 17–97 (mean 54.6) and the sample is 61.2% women (506 F / 321 M). That is not the parent CODE-15% balance, which is 40.3% men — worth knowing before treating this as a representative sample of the cohort.

The rarest classes are very small in absolute terms: AF appears in 13 records and SB in 16. Any per-class metric computed here has wide confidence intervals however the folds fall.

One Zenodo download, data.zip (218 MB), which extracts to a data/ directory. --data-path must be that directory, not its parent — it is the one holding ecg_tracings.hdf5, attributes.csv and annotations/. Getting this wrong is the commonest slip, and ECGBench’s error message says so.

download_url is null in the config even though a single URL exists, because the splitter generates the metadata CSV that the config names.

Two facts about the waveforms, both verified from the arrays rather than taken from the README:

The samples are already in millivolts. The bundled README says the signals are “at the scale 1e-4V: so it should be multiplied by 1000 in order to obtain the signals in V”, which cannot both be true — the first clause puts a median R wave at 0.15 mV and the second at 1,480 V. Measured: the median per-record peak is 3.51 mV and the median lead-II R amplitude 1.48 mV. signal_unit_scale is therefore 1.0.

The lead order really is the non-standard one the README states. Checked across all 827 records: taking channels 0 and 1 as I and II, channel 3 matches I − II/2 (aVL), channel 4 matches II − I/2 (aVF) and channel 5 matches −(I+II)/2 (aVR), each to a median relative error under 2.3%, while every other assignment is off by more than 150%.

Every record is stored as 4,096 samples at 400 Hz. The underlying acquisitions were 7 s (2,800 samples) or 10 s (4,000), zero-padded symmetrically to that length, so a window into the middle of the record is not guaranteed to be all signal.

14 records have a lead recorded as exactly zero throughout — a dead channel, caught by missing_leads. One lead each; no record has two.

5 records exceed the amplitude range, across 29 lead-level failures.

amplitude_range_mv here is [-20, 20], not ECGBench’s usual [-10, 10], and the same value is used for CODE-15% so the two releases are cleaned to one standard. The reason is the cohort: these are telehealth recordings made by non-specialist staff in primary-care units, so they carry far more electrode artefact and baseline wander than a hospital dataset. Measured over all 827 records, the exclusion rate would be 21.5% at ±5 mV, 6.7% at ±10, 0.6% at ±20 and 0.2% at ±30, against a median per-record peak of 3.51 mV. ±10 would discard 55 legitimate recordings from an 827-record evaluation set.

No record fails any other check: no NaN samples, no flat leads, no unreadable arrays.

Fold membership is identical between original/ and clean/ — clean/ is a row subset, not a re-split.

Ten folds, stratified on the rarest gold-standard abnormality each record carries (NONE for the 681 carrying none), with no patient grouping — the release states one record per patient and ships no patient identifier, so there is nothing to group on and nothing that needs it.

Rarest-wins matters more here than anywhere else in ECGBench. AF appears in 13 of 827 records, barely above the ten a ten-fold stratified split requires; any reduction that let a commoner class win would leave folds with no AF example at all. As built, AF lands 1–2 records per fold and SB 1–2 per fold, and fold sizes run 82–83 records.

Again: these folds exist so the tooling is uniform across the catalogue. For the use the release was published for, take all ten.