feat(seed): implement exam-corpus loader + filled 505-paper manifest

Implements the seed_exam_corpus.py skeleton TODOs against the real APIs and
fills the public exam corpus from official board sources.

Loader (run/initialization/seed_exam_corpus.py):
- _resolve_source_bytes: local path | url: fetch with on-disk cache + PDF validation
- upload_file: real StorageAdmin.upload_file, skip-if-exists+sha256 unless --force
- upsert_specification/upsert_paper: real upserts on spec_code/exam_code.
  Fix: QP/MS/INSERT/ER role -> eb_exams.type_code; doc_type set to 'pdf'
  (doc_type is CHECK-constrained to file formats; the skeleton wrote the role there).
- copy_user_test_subset: copy a QP subset into a test user's cc.users exam space + files rows
- first_sweep: auto_map + the /auto-map row mapper over seeded QPs -> system-owned
  exam_templates + questions/response_areas/boundaries/layout (idempotent)
- identity discovery via institute_memberships.profile_id

Manifest (run/initialization/manifests/):
- exam-corpus.yaml: 505 papers / 18 specs / AQA+Edexcel+OCR, every source URL HEAD-verified.
  AQA sciences GCSE 8461/8462/8463/8464 + AS/A-level 7401-7408, sessions JUN18-JUN24, QP+MS+ER, F+H.
- generate_corpus_manifest.py: regenerates + re-verifies all URLs from official hosts.

seed_curriculum.py: deprecation banner -> superseded by seed_exam_corpus.py; storage_loc
standardised on cc.examboards.

Verified on dev .94: full 505-paper seed (eb_specifications=18, eb_exams=505, QP=211),
idempotent re-runs, first-sweep + user-subset, 6/6 buckets provisioned.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
CC Worker
2026-06-07 22:58:03 +00:00
co-authored by Claude Opus 4.8
parent d8cf3bbc62
commit 5750413f43
4 changed files with 6960 additions and 83 deletions
+12 -7
View File
@@ -1,15 +1,20 @@
"""
seed_curriculum.py — Create curriculum data: exam board specifications and exams.
seed_curriculum.py — DEPRECATED hardcoded curriculum/exam seeder.
Seeds eb_specifications and eb_exams tables with realistic UK exam board data
(AQA, Edexcel, OCR) for Physics, Maths, and Computer Science across both schools.
⚠️ SUPERSEDED (2026-06-07) by the manifest-driven corpus loader:
run/initialization/seed_exam_corpus.py (+ manifests/exam-corpus.yaml)
Also seeds curriculum_topics in Neo4j for the school databases.
The exam-board parts of this file (eb_specifications / eb_exams) are now seeded from a
verified, provenance-bearing manifest with real uploaded PDFs — not the hardcoded rows
below. This module also had a storage_loc inconsistency the overhaul standardises away:
exam-board files belong in the `cc.examboards` bucket at the canonical path
`cc.examboards/{board}/{subject}/{award}/{paper}/{session}/{role}.pdf`, NOT under
`cc.public.snapshots/curriculum/...` (the placeholder rows below still show the old path).
Tables: eb_specifications, eb_exams
Neo4j: curriculum topic nodes in school databases
KEEP ONLY for the Neo4j `curriculum_topics` seed (step [3]) which has no replacement yet.
Do NOT use the eb_specifications/eb_exams blocks for new work — use seed_exam_corpus.py.
Run inside ccapi container:
Run (Neo4j curriculum topics only is the supported remaining use):
python3 -c "from run.initialization.seed_curriculum import seed; seed()"
"""
import os