batches.py upsert_mark previously never advanced a submission or batch to
'complete', and never validated an award against the question's max.
- Reject (422) an awarded_marks that exceeds the question's max_marks — only when
a max is actually set (0/None = not-yet-scored AI/unmapped question, unvalidatable).
- _advance_completion: a submission with a mark for every markable (leaf) question
→ 'complete'; a batch whose every non-absent submission is complete → 'complete'.
Container questions and absent students are excluded; the helper only promotes,
never regresses, so it is safe on every upsert.
Adds test_upsert_mark_rejects_over_max and test_upsert_mark_completes_submission_and_batch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The board grammar already parses per-part marks ([N marks] / "Total for Question
N is M marks"), but they were dropped building the first-pass template and the
row mapper hardcoded max_marks=0 — so every born-digital paper came back with
zero marks for a teacher to re-key by hand.
Thread the parsed mark through: bands.py derive_bands now carries each part's
`marks`, template.build passes it into part_bands, and _map_first_pass_to_rows
reads it via a defensive _safe_marks (non-negative int; unknown/None -> 0, so
image-only OCR — which has no marks yet — is unchanged). Containers still roll up
from parts. Adds test_auto_map_surfaces_born_digital_part_marks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Only AQA GCSE Physics 8463 (4.1-4.8) was seeded, so any other board/spec — or
any spec_ref that didn't match those 8 — projected zero (:Part)-[:ASSESSES]->
(:SpecPoint) edges, silently. Seed the full top-level topic catalogue for the 6
current test specs (GCSE + A-level Physics/Chemistry/Biology, 44 SpecPoints) and
loop the Specification/SpecPoint MERGE over all of them (board created once).
Idempotent; deterministic uuid5 keys unchanged. Sub-point granularity remains a
later data task. Run: python3 -c "from run.initialization.init_exam_graph import
init; import json; print(json.dumps(init()))" in the ccapi container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Only the born-digital fast path enqueued project_template_safe; the async OCR
job (_run_auto_map_job) — where image-only papers, R3's primary target, are
routed — never projected, so those papers never reached the graph after
auto-map. Project at the end of the async job too.
Adds test_auto_map_ocr_path_projects_to_neo4j.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_refresh_ai_rows deleted only unconfirmed AI rows, then bulk-inserted freshly
generated rows keyed by a deterministic uuid5 of (template_id, semantic key).
A confirmed ghost keeps that id, so a re-run re-emitted it → primary-key
conflict that failed the whole insert batch (reachable: auto-map is blocked only
when marks exist, not when ghosts are confirmed).
Fix: after the delete, read the ids that survived (confirmed-AI + manual) and
skip re-inserting any freshly generated row whose id matches — preserving the
teacher's curated row and making the insert PK-safe.
Adds test_auto_map_rerun_after_confirm_does_not_pk_collide, which confirms a
ghost at its REAL generated id (the prior test used an arbitrary id that never
collided) and asserts the row survives exactly once with confirmed=True.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The app sets canvas.height = Math.ceil(viewport.height) per page and stacks pages by those
heights; the backend page_top used the raw float, so it fell ~1px/page short, compounding to a
visible upward shape shift on later pages (~36px over 40 pages). Ceil rendered_h to match exactly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_pdf_page_geometry left rendered_w/h in PDF points (~595x842), but the app renders each PDF
page at PAGE_WIDTH=780 with proportional height and places shapes at the raw bounds. Result:
every detected region rendered shrunk (~0.76x) and shifted up-left. Set rendered_w=780 +
rendered_h=780*aspect (matches pdfLoader + pageGeometryFromImages), and scale px/point TOPLEFT
boxes into that space (was a hardcoded 0.5). Path-2 point boxes auto-correct via rendered_w/page_pt_w.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On papers where band detection yields few/no questions but opencv/gemma still emit response
regions, those regions referenced a synthetic default_qid that was never inserted -> FK violation
(exam_response_areas/exam_boundaries -> exam_questions). Ensure the fallback container question
exists and reattach orphan child rows to it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
B1-4 live-route validation: continuation bands re-emit the same stable AI id for
response_areas/boundaries/layout (not just questions), causing duplicate-pkey insert
failures. Add _dedupe_rows_by_id applied to all four tables in _refresh_ai_rows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Eval harness for AQA A-level + GCSE-science image-only papers: finalize.py --b1-only,
RapidOCR runner (rapid_pass.py via dsync), GT fixtures (make_b1_gt.py + b1_gt_labels.json),
and fetch_b1_corpus.py to pull the eval corpus from .94 cc.examboards at runtime.
Salvaged from t_15be12ed (which timed out on iteration budget re-running OCR): exam PDFs and
generated OCR caches/reports are NOT committed (third-party copyright + reproducible) — gitignored
and fetched/generated at runtime. Baseline coverage recorded in the task evidence file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
t_d1600327 added a standalone scope=user-subset, but a full reset (scope=all)
and scope=exam-corpus still left the --user-subset cc.users storage objects
orphaned (files rows are wiped by the table clear, but the Storage API objects
are not). Call the same _clear_user_subset_files() helper in both paths so the
finding-#2 gap is fully closed: storage removed before rows, idempotent.
Closes overwatch review finding #2 (user-subset not cleaned by reset).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
/admin/reset and reset_environment.reset() act on os.environ['SUPABASE_URL'].
A platform-admin call on a prod-deployed API would wipe prod data + exam
corpus + storage. Refuse when the target matches a known prod marker
(.156 / supabase.classroomcopilot) unless RESET_ALLOW_PROD=1 is set.
Addresses overwatch review finding #1 on feature/exam-seeding-overhaul.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements the seed_exam_corpus.py skeleton TODOs against the real APIs and
fills the public exam corpus from official board sources.
Loader (run/initialization/seed_exam_corpus.py):
- _resolve_source_bytes: local path | url: fetch with on-disk cache + PDF validation
- upload_file: real StorageAdmin.upload_file, skip-if-exists+sha256 unless --force
- upsert_specification/upsert_paper: real upserts on spec_code/exam_code.
Fix: QP/MS/INSERT/ER role -> eb_exams.type_code; doc_type set to 'pdf'
(doc_type is CHECK-constrained to file formats; the skeleton wrote the role there).
- copy_user_test_subset: copy a QP subset into a test user's cc.users exam space + files rows
- first_sweep: auto_map + the /auto-map row mapper over seeded QPs -> system-owned
exam_templates + questions/response_areas/boundaries/layout (idempotent)
- identity discovery via institute_memberships.profile_id
Manifest (run/initialization/manifests/):
- exam-corpus.yaml: 505 papers / 18 specs / AQA+Edexcel+OCR, every source URL HEAD-verified.
AQA sciences GCSE 8461/8462/8463/8464 + AS/A-level 7401-7408, sessions JUN18-JUN24, QP+MS+ER, F+H.
- generate_corpus_manifest.py: regenerates + re-verifies all URLs from official hosts.
seed_curriculum.py: deprecation banner -> superseded by seed_exam_corpus.py; storage_loc
standardised on cc.examboards.
Verified on dev .94: full 505-paper seed (eb_specifications=18, eb_exams=505, QP=211),
idempotent re-runs, first-sweep + user-subset, 6/6 buckets provisioned.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add 'exam-corpus' INIT_MODE: docker-entrypoint.sh case -> main.py --mode
exam-corpus -> run_exam_corpus_mode() -> seed_exam_corpus.load(). Driven by
EXAM_CORPUS_MANIFEST (+ DRY_RUN/FORCE/BOARD/SPEC/USER_SUBSET/FIRST_SWEEP env).
Skips gracefully (success) when no manifest is configured, so it is safe in a
comma list like INIT_MODE=infra,seed,exam-corpus before papers are gathered.
Bucket provisioning stays in infra mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
infra (buckets.py): add cc.public / cc.institutes / cc.admin to the bucket
provisioner alongside cc.examboards; make initialize_buckets idempotent
(already-exists treated as success). Bucket provisioning stays in infra init.
new (seed_exam_corpus.py): manifest-driven loader scaffold that USES the buckets
(does not create them) — validate -> upload to cc.examboards (canonical path) ->
upsert eb_specifications/eb_exams -> optional user test subset -> optional
--first-sweep auto-map pass. TODOs marked for the gathering task to complete.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
get_class selected a non-existent enrollment_requests.created_at column,
causing a PostgREST 42703 -> 500 on /database/timetable/classes/{id}
(class detail / ResultsWidget). The table column is requested_at.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The auto-map endpoint returns dict (sync 200) or JSONResponse (202 async OCR);
FastAPI cannot build a response model from that Union. Fixes import-time
FastAPIError introduced with the S5-2 endpoint.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add synthesize_part_box() as the single authoritative S5 part-box projection
(T3 swap point): content-margin x-extent x part-band y-extent, BOTTOMLEFT
coords; label_box retained as a separate anchor. build() attaches box per part.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Removed the teacher_id ownership check from _require_source_visibility_or_404.
RLS already ensures a teacher can only see templates in their institute;
the ownership gate was blocking shared templates (e.g. board-uploaded AQA papers)
for any teacher who didn't personally create them.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The download path now resolves the files row via SupabaseServiceRoleClient (to
sidestep the cabinet_memberships RLS recursion); the test must mock it like the
upload test does. Test-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-merge smoke caught a second issue: the source_file_id download path read `files`
as-the-user, tripping a PRE-EXISTING broken RLS policy on cabinet_memberships
(42P17 infinite recursion). Authz is already enforced (template fetch + source
visibility), and source_file_id is the template's own file, so resolve the row via
service role (documented exception, same as the catalogue lookup). Flagged the
cabinet_memberships RLS recursion separately as infra bug E8.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pre-merge live smoke on .94 caught 'Bucket not found': the upload wrote to a
per-institute bucket cc.institutes.<id>.private that isn't provisioned on dev.
Use the shared SOURCE_BUCKET_FALLBACK (cc.users); institute is namespaced in the
storage path + enforced by the files-row RLS. Per-institute buckets are a future
multi-tenant concern. Catalogue path + cross-institute 404 already verified green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Recovered from cc-worker WIP that was left uncommitted in the dev-centre clone
(card t_0055b89b). Multipart source_pdf upload at create -> source_file_id;
source-pdf download endpoint resolves from exam_id (catalogue) or source_file_id.
NOT yet human-reviewed/merged; preserving + verifying so it isn't clobbered.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Backend follow-on to migration 73:
- schemas: ResponseAreaPayload.kind extended to response|context|question_number|
mark_area|reference|furniture + context_type; QuestionPayload gains bounds+page.
- PUT serialization persists Part bounds/page and region context_type.
- Neo4j projection only emits Region nodes for response/context regions; the
metadata kinds (question_number/mark_area/reference/furniture) are physical-layer
only and stay out of cc.public.exams.
- Unit test: new kinds + Part geometry + context_type round-trip.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous commit added apikey to _create_base_client headers, but supabase-py
already sets apikey from the key arg → two apikey headers → Kong rejected every
as-user call with 401 'Duplicate API key found' (exam API 502'd on auth). Revert
to Authorization-only; fix the two header unit tests to assert the real contract
(apikey via the key arg; options.headers carries only the user Authorization).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
