Methodology

What the engine checks, what it models, and what it refuses to claim

Every figure on this page is read from the code that produces it. Where a claim is not yet backed by a measurement, it is listed below as pending rather than written into the marketing copy.

The 18 pedagogical gates, across 11 categories

Every generated lab is checked against this list, and each gate is written here as the question it asks. A gate that could not apply to your lab is reported as INAPPLICABLE. Applicable means an assertion actually executed, not that a candidate existed, so a gate that inspected nothing lands in the list of checks that did not run rather than in the list that passed.

assessment-alignment

ANSWER-LEAK Does any activity hand over its own answer before it is asked?
DISTRACTOR-QUALITY Is every wrong answer a real candidate rather than filler?
POSITION-BIAS Can a learner score by always choosing the same option index?
ASSESSMENT-COHERENCE Does the lab assess what it actually taught?

feedback

DISTRACTOR-QUALITY Is every wrong answer a real candidate rather than filler?
FEEDBACK-SPECIFICITY Does a wrong answer get diagnosed and corrected, not just marked wrong?

pedagogy

HINT-LADDER Do the hints climb from orientation to scaffold without leaking the answer?
DEPTH Does the prose teach judgement, or only state facts?

fidelity

ASSESSMENT-COHERENCE Does the lab assess what it actually taught?
OBJECTIVE-COVERAGE Does the lab deliver every objective it promised the educator?

cognitive-level

OBJECTIVE-PROGRESSION Do the objectives climb, or restate one demand four times?
BLOOM-SPREAD Is the spread of cognitive demand the one this level expects?
LEVEL-AUTHENTICITY Is this the level it claims, or a shallower lab with the verbs swapped?

interactivity

LEVEL-AUTHENTICITY Is this the level it claims, or a shallower lab with the verbs swapped?
VARIETY Does one activity format dominate the lab?

domain-authenticity

DOMAIN-AUTHENTICITY Does the lab make this discipline's reasoning moves and avoid its red flags?

redundancy

REDUNDANCY Does any chunk teach what an earlier chunk already taught?
TERM-REDEFINITION Is a term formally defined more times than a recap justifies?

volume

FILLER Does the lab spend a learner's minute without teaching anything?
BLOAT Is the lab the shape of the material it was made from?

accessibility

FIGURE-PROVENANCE Is every figure themed, labelled and free of executable content?

technical-portability

FIGURE-PROVENANCE Is every figure themed, labelled and free of executable content?

The safety overlay is counted separately

A separate safety overlay runs on top of these, from 14 discipline specific banks. It does not cover every discipline: a lab in a discipline with no bank has had no safety overlay applied, and the run reports that rather than implying a clean pass. Those banks hold 96 rules between them, and exactly one bank resolves for a lab, so the rules that ran on yours are that bank's and not all 96.

What is checked but holds no gate id

One further check, beyond the 18 gates above, runs as a limb of a gate rather than as a gate of its own and reports under that gate's id: misconception coverage reports under DISTRACTOR-QUALITY. So the engine performs 19 checks and publishes 18 ids, which is why a reader counting verdicts finds one more than the list above has ids for.

Why the list is longer than the count

A gate that speaks for two quality dimensions is listed under both of them, so the list above holds 22 rows for 18 distinct gate ids. Counting rows would overstate what runs; 18 is the number of gates, and 19 is the number of checks.

What this does not do

A short lab with no stated objectives switches several of these off, because a gate with nothing to measure cannot honestly report a verdict. That is the case where a lab passes having been checked for very little, and it is why the run reports the applicability denominator rather than a single pass or fail.

Teaching level is modelled, not decorated

All 8 RQF and FHEQ levels carry their own demand profile: how much scaffolding is given, how much ambiguity is tolerated, how many reasoning steps a task may take, and how much a learner is expected to justify. The level also weights which activity types a lab reaches for.

Level 1 - GCSE foundation (grades 3-1)

Remembering - recognising and recalling everyday facts, terms, and simple one-step procedures.

Level 2 - GCSE (grades 9-4)

Remembering and understanding - describing, summarising, and applying routine methods in familiar contexts.

Level 3 - A-level / BTEC National

Understanding and applying - explaining mechanisms and applying methods to structured, unfamiliar problems.

Level 4 - First-year undergraduate / HNC

Applying and analysing - using disciplinary techniques accurately and beginning to break problems into parts.

Level 5 - Second-year undergraduate / HND

Analysing and early evaluating - comparing methods, appraising evidence, and justifying choices between alternatives.

Level 6 - Bachelor's with honours (final year)

Evaluating and synthesising - critically weighing competing positions and integrating sources into a sustained argument.

Level 7 - Master's

Evaluating and creating - critically appraising research and designing or reframing approaches under uncertainty.

Level 8 - Doctoral

Creating - originating, defending, and extending knowledge at the discipline's frontier.

Discipline is modelled too

21 UK discipline packs, each with its own referencing standard and worked citation form, its own reasoning moves, and its own preferred activity mix. A law lab and a nursing lab do not differ only in their adjectives.

The six high-agency families

Beyond recognition work, 6 families ask for a decision and mark it against an authored key: scenario, experiment, errorRepair, investigation, prioritization, dialogue. The exported SCORM package imports the same evaluation modules the hosted player uses, so the two surfaces cannot disagree about whether an answer was right.

17 activity types, one display path

The Studio authoring canvas is the learner view, rendered from the same components. A byte exact snapshot over every type is what stops an editor and a learner seeing different markup.

Evidence is authored, not inferred

An assessed activity records whether its objective mapping was written at generation time or matched afterwards by lexical overlap. The two are not the same fact and the document no longer stores them as if they were.

The hint ladder has 3 named rungs

orientation, then structure, then scaffold. The scaffold rung declares what it must not reveal, so a hint that gives the answer away is a contract violation rather than a matter of taste. Enforcement is on assessed steps: a reflection has nothing to scaffold. Generation writes the rungs for each marked activity from that step's own answer key, and refuses a ladder it cannot write without a rung giving the answer away rather than padding one; the readiness report names every assessed step it refused, and that step is yours to ladder in Studio.

A further reading must state its provenance

A reading is tiered as foundation, core, extension, and carries one of 3 provenances: present-in-uploaded-source, suggested-search-topic, model-generated-unverified. A suggested search topic is never presented as a citation, and a reading that claims to come from your upload must name the unit it came from. Generation writes the reading list from your upload: every reading it writes is either tied to the passage it came from or marked a suggested search topic with no citation, so it never hands you an invented reference.

Source fidelity is measured in both directions

The coverage ledger answers which of your passages reached the lab. A separate precision measure answers which of the lab's sentences have nothing behind them. It is computed from the finished lab and the finished source alone: it takes no model verdict, so nothing can talk the figure up.

Small cohorts are withheld, not rounded

A figure below the disclosure threshold is replaced by the reason it was withheld, and nothing else on the page lets the withheld number be reconstructed. A withheld figure is not zero per cent, and the report says so in those words.

How long generation takes

Generation is bounded by a 300 second wall clock. A run that cannot finish inside it stops and says so; it does not keep spending and it does not hand back a partial lab dressed as a whole one.

What this does not do

We publish the deadline, which is 5 minutes, and not a typical time. A median would need a measured distribution over a real corpus, and a median measured against a stub provider would be measuring the stub. Until that measurement exists, no surface on this site states a typical generation time.

The public vocabulary

Every activity word this site uses names something the engine can produce. Words that had no referent were given one rather than deleted, because the thing they described was real. No activity type the engine can render is left without a public word.

challengeexercise, scenario, investigation, errorRepair, prioritization: An activity where the learner produces or decides something rather than recognising an option: a marked exercise, a branching scenario, an investigation, a fault to repair, or a set of competing priorities to resolve.
case studyscenario, investigation, dialogue: A situated activity built from one worked case: a decision path, an evidence hunt, or a conversation the learner steers.
practice taskexercise, numeric, fillGap: A task the learner works through and is marked on: a structured exercise, a numeric calculation, or a cloze passage.
quick checkmcq, matching, ordering, classify, fillGap, numeric: A short marked check with an authored answer key, marked by the machine rather than by the learner.
quizmcq, matching, ordering, classify, fillGap: The recognition family: choose, pair, sequence, sort, or complete against an authored key.
guided practiceexercise, branch, fillGap: A supported attempt where the structure is given and the learner supplies the reasoning.
branching decisionbranch, scenario: A choice that changes what the learner meets next, rather than one that is simply right or wrong.
experimentexperiment: A parameterised model the learner varies, with a transfer question marked against the model's own arithmetic.
error repairerrorRepair: A worked artefact containing exactly one authored defect, which the learner must locate and correct.
investigationinvestigation: A set of evidence items with authored roles, from which the learner must assemble a sufficient case.
prioritisationprioritization: A capacity-constrained choice between options with authored costs and impacts.
dialoguedialogue: A multi-turn exchange where each of the learner's replies carries an authored quality.
reflectionreflect: An unmarked written response the learner keeps, never scored as a correct answer.
diagramfigure: An application-drawn figure. The model supplies the relationships; this application owns every coordinate.
explanationtext: Taught prose, optionally opened by a prediction prompt before the passage is revealed.
smart hintChunk.hintLadder: One rung of the three-level hint ladder: orientation, then structure, then scaffold. The scaffold rung declares what it must not reveal.
readingLab.readings: A further-reading entry carrying a tier (foundation, core, extension) and a provenance that states whether it was found in your upload, is a suggested search topic, or is model generated and unverified.
objectiveLab.objectives: A stated learning objective with a Bloom level, which assessed activities cite by id.
misconceptionWrongAnswerDiagnosis: The authored reason a particular wrong answer is tempting, and what to do next about it.

Claims we corrected

An audit of this site classified 35 public claims against the code behind them. These 11 were changed, and here is what each one used to say.

SCORM 2004 3rd Edition is NOT offered.

The landing page rendered 'SCORM 2004 3rd Edition' as a selectable export tile. No third builder has ever existed: the route enum has two values, the package builder is a binary branch, and the manifest module builds 1.2 and 2004 4th Edition only. The page also contradicted itself, since the announcement bar correctly said 'SCORM 1.2 and SCORM 2004'. The tile is gone and the list is now derived from the enum. This is a capability the product genuinely does not have, so the claim was withdrawn rather than built: a 3rd Edition manifest builder is a change to src/lib/scorm/**, which this workstream does not own.

EngagedLab has 17 activity types, every one of them draws the same markup for an educator editing it and a learner meeting it, and an answer is marked by the same code in the browser and in an exported package.

The site said '11 activity types' against a seventeen-member union, while a provider comment said 'eleven' over a list of seventeen and a repair hint said 'fourteen'. The marketing UNDERSOLD by six, and the six omitted are the higher-order families that substantiate the site's weakest-supported claims. The count is now derived.

Every activity word the site uses names something the engine can produce and a function can count.

Marketing had invented a vocabulary layer (challenges, case studies, smart hints, practice tasks, readings) with no referent in the engine, so no claim about it was verifiable even in principle. The words are kept and given referents rather than deleted.

An ASSESSED step is not publishable without a three-rung hint ladder (orientation, structure, scaffold) whose scaffold rung declares what it must not reveal.

The copy said 'Every generated step carries a three-level hint ladder'. The schema was a CEILING with an empty default (zero hints validated), no quality gate read hints at all, and the tightening pass was instructed to 'shorten hints to a clause'. The universal quantifier could not survive any adversarial check. Enforcement is on ASSESSED chunks, so the copy now says assessed. A reflection step does not need a hint ladder and never did. CORRECTED A SECOND TIME, for the verb rather than the quantifier: 'carries' is present tense about something generation does not do. Measured against a real offline run of the pipeline, 0 of 7 chunks (both assessed ones included) came back with a tiered ladder; every one was flat, hintLadderOf reported legacy-flat, and no rung declared anything. This claim is now about what the contract REFUSES, which is what its own requires always said. The producer has since landed (hint-ladder-in-generated-labs, backed): the surfaces that said generation writes the hints flat now say what it writes and what it refuses, because a disclosure left standing after the capability ships understates the product in the same way the original sentence overstated it.

A further reading is not accepted unless it states its provenance: found in your upload, a suggested search topic, or model generated and unverified. A suggested search topic is not a citation.

The claim, and the feature card added for it, read 'A further reading states its provenance', in the present tense, about a field nothing in the pipeline writes. Measured against a real offline run this wave, readingsOf on a generated lab returns an empty array and publishabilityIssues raises PUB-READINGS. The provenance rules themselves are real and enforced, so the claim is kept and narrowed to what it always tested: what the contract refuses. The card and /methodology then said generation did not write the list yet. The producer has since landed (readings-in-generated-labs, backed), so both now say what generation writes and the one rule it never breaks: no invented reference.

Every lab is checked against 18 named pedagogical gates across 11 quality categories, and the report says which of them did not apply.

The homepage said 'Every lab is quality-checked' and named six categories, with no number and no way to check it. For a short, low-level, objective-free lab a majority of gates report inapplicable, so the sentence was true and uninformative at once. The count is now published and derived, and the methodology page states plainly that an inapplicable gate is not a passed gate.

A generation finishes inside 5 minutes or stops and says why.

The headline said 'in 2 Minutes'. Nothing measured generation duration in either direction, and the engine is budgeted to 300 seconds, two and a half times the advertised figure, with a single retried model call allowed to spend 120 seconds of it. The headline now states the deadline the engine actually enforces. It is deliberately not presented as a typical time.

Every destination the public navigation offers exists.

Five nav destinations did not resolve, two of them the highest commercial-intent links on the site. A rewrite table inside the nav component later hid them by sending 'For Educators' to the upload form and 'About' to a testimonial anchor, which is a 404 the visitor cannot see. 8 real pages now exist and the rewrites for them are gone.

Any figure on the site that was not measured says so.

The cohort mockup labelled its own figures 'Illustrative cohort preview' while the hero's animated output counters, the more prominent and more animated block, carried no label at all. The site was applying its own honesty standard to one surface and not the other. Both now use one constant. CORRECTED AGAIN for the third figure: the hero's '250 XP Earned' card is fabricated too, and it is a positioned sibling of the labelled panel rather than a child of it, so it was outside the label and outside the requirement, which said 'both panels' under a claim that says 'any figure'. The subtitle beneath it was made true in the previous wave; the number above it was not labelled until this one.

A complete example lab is reachable without an account.

'See Example Labs' appeared in three placements and every one linked to a single hand-authored biology lab, at one level, in one discipline, for a product whose stated differentiator is discipline and level awareness. The copy now matches the count, and the count is derived from the index.

The pricing section says whether paid plans are on offer, and when they are, shows the plans Stripe holds at the price Stripe charges.

The section told every visitor, in every billing state, that the application did not present a card checkout or promise a paid plan. That is false once STRIPE_MODE is test or live. It now says so only while billing is switched off.

Claims we are not making yet

These capabilities are being built. Nothing on this site claims them, and each one has a test that stays red until it is true.

A representative source generates in a measured median time.

PENDING ON THE MEASUREMENT, NOT ON THE CODE. report.ts computes a nearest-rank p50 and p90 and carries a `basis` of mock, cassette-replay or live, refusing to call anything but a live figure an educator's wait. A live corpus run exists (2026-09-23, docs/generation/live-benchmark.md), and it measured two engines that no longer generate labs here: one took p50 277.2 s and p90 289.3 s over 13 labs, slowest 326.1 s, and the one before it took p50 174.0 s over 20 labs. No later engine has been measured over the corpus: the only live observation of one is a single generation of the 2,705-character sample material, at 111.0 s on 2026-10-05, and one lab is not a median. A replay of the recordings takes under a second, so no figure can be taken from a replay either. Until a live corpus run of the current engine is committed, the headline states the deadline that the budget enforces and no surface states a typical time. The cited test asserts both halves of that, and asserts them behaviourally rather than by looking for the word `p50` in a source file, which is how this probe once went green having checked neither.

Example labs generated by the engine span at least three disciplines and three levels.

MEASURED ON 2026-10-06, not assumed. EXAMPLE_LABS holds three labs: the hand-authored Biology demo at Level 3, which counts for nothing here, and two GENERATED labs (Computing & Computer Science at Level 3, Engineering at Level 2). Each is the pipeline's own output with every model call answered by a live recording of gpt-5.6-terra and gpt-5.6-luna made on 2026-09-23, committed with its provenance and a sha256 under src/lib/claims/example-labs/ and served at /examples/[id]. Generated coverage is therefore 2 disciplines and 2 levels against a bar of 3 and 3. The recording this entry used to say was owed exists: three corpus recordings that would each add a third discipline and a third level (health-l8, humanities-l6 and law-l6) replay with zero misses. What is owed is building one of them into an example with scripts/bench/build-example-labs.mjs and committing it the same way. The two engine fallbacks the recording run advised fixing first, the fallback section stub and the dialogue constructor that spoke a heading as a turn, are both fixed. Both committed examples are an earlier engine's output: the computing recording still replays with zero misses, but the engine now builds a different lab from it, and the engineering recording no longer replays on the current prompts, so that example cannot be rebuilt until it is re-recorded. Committing output from the offline stub would be worse than no example: it emits byte-identical canned structures that pass every gate, so it would look like proof and be the opposite.

Generate a lab from your own material See the example labs