Accuracy QA
The tracker now advertises its open problems instead of hiding them.
If this is going to impress the hosts, the quality bar has to be visible. This page pulls integrity anomalies, historical coverage gaps, and maintenance priorities into one place.
6
open issues
Each issue is typed and severity-ranked.
3
high severity
These are the problems most likely to undermine credibility.
5%
reviewed rows
0 verified, 16 needs-review.
5%
evidence-backed rows
24 catalog episodes still have no explicit status record.
2
blocked QA gates
These prevent the tracker from being called verified.
Proof Chain
The answer to how we know is a gate, not a confidence score.
The tracker can look polished while still being wrong. These gates separate extracted data from evidence-backed rows.
Episode coverage
314 of 338 episodes have explicit status records.
Every catalog episode must be tracked as quiz, verified no-quiz, needs extraction, or blocked source.
Classify the remaining catalog episodes before treating coverage as complete.
Transcript receipts
Evidence currently leans on episode pages and manual notes, not transcript receipts.
A verified row needs a transcript excerpt, timestamp, reviewer, and source pointer.
Attach transcript snippets and timestamps during each rerun or manual audit.
Guest identity
15 of 21 non-standard or lineup-only speaker checks have attached evidence.
Every substitute host or non-standard panelist needs named speaker evidence, not just a host slot.
Audit lineup pages and transcript speaker turns for every guest-mapped row.
Outcome logic
294 of 294 rows pass derived-outcome and participant-shape checks.
Winner, tie, and stumper labels must be derived from response correctness, not copied from model output.
Keep this gate green by running the QA audit after every merge.
Human review
0 of 294 rows are verified.
Verified means a human checked the source evidence and the row can be defended.
Promote rows to verified only after source, speaker, answer, and outcome checks all pass.
Guest Identity
Guests need receipts, not slot assumptions.
A substitute host can still live in the Charlotte or Idrees scoring slot, but the row has to preserve the actual speaker name and the evidence that proves it.
Identity audit
15
non-standard rows
13
named guest rows
15
with evidence
5
lineup-only people
Open guest rows: 6. Start with these before trusting host-era stats.
Checks and Balance: Disruptor-in-chief
David Rennie: Person appears in lineup evidence but is not yet modeled in quiz response rows.
node scripts/podcast-series/economist/checks-and-balance/extract.mjs --catalog --include-tracked --from 2020-01-24 --limit 1 --title-contains "Checks and Balance: Disruptor-in-chief"Checks and Balance: Disruptor-in-chief
Shashank Joshi: Person appears in lineup evidence but is not yet modeled in quiz response rows.
node scripts/podcast-series/economist/checks-and-balance/extract.mjs --catalog --include-tracked --from 2020-01-24 --limit 1 --title-contains "Checks and Balance: Disruptor-in-chief"Checks and Balance: Getting a grip
Adam Roberts: Person appears in lineup evidence but is not yet modeled in quiz response rows.
node scripts/podcast-series/economist/checks-and-balance/extract.mjs --catalog --include-tracked --from 2020-03-13 --limit 1 --title-contains "Checks and Balance: Getting a grip"Checks and Balance: Getting a grip
David Rennie: Person appears in lineup evidence but is not yet modeled in quiz response rows.
node scripts/podcast-series/economist/checks-and-balance/extract.mjs --catalog --include-tracked --from 2020-03-13 --limit 1 --title-contains "Checks and Balance: Getting a grip"Checks and Balance: Welcome to New York
Rosemarie Ward: Person appears in lineup evidence but is not yet modeled in quiz response rows.
node scripts/podcast-series/economist/checks-and-balance/extract.mjs --catalog --include-tracked --from 2024-03-01 --limit 1 --title-contains "Checks and Balance: Welcome to New York"Checks and Balance: Welcome to New York
Nicole Gelinas: Person appears in lineup evidence but is not yet modeled in quiz response rows.
node scripts/podcast-series/economist/checks-and-balance/extract.mjs --catalog --include-tracked --from 2024-03-01 --limit 1 --title-contains "Checks and Balance: Welcome to New York"Checks and Balance: Disruptor-in-chief
Charlotte Howard, Jon Fasman: Guest or non-standard speaker is named, but the row still needs transcript-level proof before verification.
node scripts/podcast-series/economist/checks-and-balance/extract.mjs --catalog --include-tracked --from 2020-01-24 --limit 1 --title-contains "Checks and Balance: Disruptor-in-chief"Checks and Balance: Des Moines craft
Charlotte Howard, Jon Fasman: Guest or non-standard speaker is named, but the row still needs transcript-level proof before verification.
node scripts/podcast-series/economist/checks-and-balance/extract.mjs --catalog --include-tracked --from 2020-01-31 --limit 1 --title-contains "Checks and Balance: Des Moines craft"Review Standard
A row only becomes verified when every claim has a receipt.
This is the manual review path for each quiz row. It is intentionally stricter than model confidence.
Episode identity
Match by date and exact title, especially when same-day picks episodes exist.
Catalog row, episode URL, and title-specific rerun command.
Transcript receipt
Store enough proof to reproduce the extraction later.
Transcript excerpt, timestamp or tail window, audio source, model, batch file, and reviewer date.
Speaker identity
Name the person who spoke; use hostId only as the scoring slot.
Episode-page lineup plus transcript speaker turn for every guest or substitute host.
Question and answer
The question, answer, guesses, and correctness must all be recoverable from the transcript.
Short quoted excerpt or timestamped note covering the prompt, guesses, and revealed answer.
Outcome derivation
Winner is derived from exactly one correct response; ties and stumpers stay explicit.
Automated derived-winner check plus reviewer confirmation for ambiguous answers.
Coverage classification
No episode can be silently absent.
One of tracked-quiz, verified-no-quiz, needs-extraction, or blocked-source with notes.
Flagged Now
These are the current blockers to calling the tracker publication-grade.
The issue list is designed to become the operating system for backfills and review.
No quiz rows are verified yet
All 294 structured quiz rows are still unreviewed and 16 rows have evidence attached, so the dataset has no audit-complete layer yet.
Add transcript evidence and a reviewer pass before calling the tracker publication-grade.
Guest and substitute host names are only partially preserved
13 structured quiz rows now preserve named guests, which is real progress but still too sparse to treat substitute-host history as complete.
Backfill named guests through title-specific reruns using --include-tracked and merge them with --replace.
Jon Fasman is partially preserved but still not modeled as a first-class participant
Official Economist materials and targeted reruns now preserve Jon Fasman in 13 quiz rows, but he still lacks full person-level stats and appearance coverage across the archive.
Prioritize launch-era and 2021-2023 backfills so historical lineup changes are captured person-by-person.
The 2024-03-01 main-show quiz row no longer reproduces cleanly
Two title-specific reruns for 'Checks and Balance: Welcome to New York' on March 30, 2026 returned 'No quiz segment found', even after expanding the transcription tail from 7 minutes to 15.
Manually review the full episode audio or transcript before trusting this stored quiz row as publication-grade.
Structured quiz rows still trail the manual catalog count
The coverage ledger currently accounts for 314 episodes out of a working catalog count of 338, leaving 24 episodes to classify as verified no-quiz, blocked source, or still needing extraction.
Keep filling the manual coverage ledger so missing, no-quiz, and backfill-needed are separate states.
A sizable slice of rows still has lower extraction confidence
2 rows are below 0.80 confidence and should probably be first in line for human review.
Sort future QA work by confidence and participant anomaly instead of date alone.
Patterns
Response-shape audits make weird rows much easier to spot.
Once guest hosts and historical panelists are backfilled, this pattern table should get more interesting. Right now it is mainly useful for exposing flattening mistakes.
Coverage breakdown
tracked-quiz
294
verified-no-quiz
16
needs-extraction
4
blocked-source
0
unknown catalog episodes
24
Response patterns
Charlotte Howard + Idrees Kahloon
280
Charlotte Howard + Jon Fasman
9
Charlotte Howard + Jon Fasman + John Prideaux
3
Charlotte Howard + Idrees Kahloon + John Prideaux
1
Jon Fasman + Idrees Kahloon
1
Backfill Operating Order
Queue 1
Backfill launch-era and 2021-2023 episodes where Jon Fasman appears on Acast lineups.
Queue 2
Use title-specific reruns for 2023-2024 duplicate-date Fridays so picks episodes cannot collide with the main show.
Queue 3
Keep explicit episode coverage states current so no-quiz and extraction gaps are visible instead of implicit.
Queue 4
Add transcript evidence snippets and timestamps before sending the tracker externally.
Rerun Queue
These are the next concrete episodes to rerun or classify.
This queue turns the backfill plan into actionable episode targets with command hints, instead of leaving it as a vague to-do list.
Checks and Balance: Disruptor-in-chief
The row is already flagged for manual review.
Checks and Balance: Des Moines craft
The row is already flagged for manual review.
Checks and Balance: Left Bern
The row is already flagged for manual review.
Checks and Balance: The Trump pay bump
The row is already flagged for manual review.
Checks and Balance: Mike drop
The row is already flagged for manual review.
Checks and Balance: Joementum
The row is already flagged for manual review.
Checks and Balance: Getting a grip
The row is already flagged for manual review.
Checks and Balance: The invisible enemy
The row is already flagged for manual review.
Checks and Balance: Counting the cost
The row is already flagged for manual review.
Checks and Balance: How long?
The row is already flagged for manual review.
Checks and Balance: The covid campaign
The row is already flagged for manual review.
Checks and Balance: Oil be back
The row is already flagged for manual review.
Checks and Balance: Corona corralled?
The row is already flagged for manual review.
Checks and Balance: One year on
The row is already flagged for manual review.
Checks and Balance: A novel approach
The row is already flagged for manual review.
Checks and Balance: Welcome to New York
The row is already flagged for manual review.
Interventionist isolationist: Trump and Brazil
Episode is present in the site catalog but does not yet have a structured quiz record.
Disorganised chaos: why Democrats can't stand up to Trump
Episode is present in the site catalog but does not yet have a structured quiz record.
ICE show: how Trump is creating his own police force
Episode is present in the site catalog but does not yet have a structured quiz record.
Make babies great again
Episode is present in the site catalog but does not yet have a structured quiz record.
Command Hints
2020-01-24
node scripts/podcast-series/economist/checks-and-balance/extract.mjs --catalog --include-tracked --from 2020-01-24 --limit 1 --title-contains "Checks and Balance: Disruptor-in-chief"2020-01-31
node scripts/podcast-series/economist/checks-and-balance/extract.mjs --catalog --include-tracked --from 2020-01-31 --limit 1 --title-contains "Checks and Balance: Des Moines craft"2020-02-07
node scripts/podcast-series/economist/checks-and-balance/extract.mjs --catalog --include-tracked --from 2020-02-07 --limit 1 --title-contains "Checks and Balance: Left Bern"2020-02-14
node scripts/podcast-series/economist/checks-and-balance/extract.mjs --catalog --include-tracked --from 2020-02-14 --limit 1 --title-contains "Checks and Balance: The Trump pay bump"2020-02-21
node scripts/podcast-series/economist/checks-and-balance/extract.mjs --catalog --include-tracked --from 2020-02-21 --limit 1 --title-contains "Checks and Balance: Mike drop"2020-03-06
node scripts/podcast-series/economist/checks-and-balance/extract.mjs --catalog --include-tracked --from 2020-03-06 --limit 1 --title-contains "Checks and Balance: Joementum"