2.5.5 Testing Design Assumptions

Data-informed testing can be used by learning engineering teams to check theoretical assumptions, e.g. using learning curve analysis to check if a collection of performance tasks measure the same competencies or A/B testing to determine which design alternative is most effective.

From the Learning Engineering Toolkit

In learning engineering, testing is deliberately a means of checking the assumptions baked into a design. The companion tools chapter explains that testing each part of a design can involve confirming assumptions—about needs analyses, requirements, personas, scenarios and use cases, learning-design elements, and prototypes—with the very people the design is meant to serve [LET-12]. It presses teams not to hold testing back until a product is finished but to try out design components often and from the start, even paper sketches, as the Math Readiness development effort showed [LET-12].

Chapter 3 puts this assumption-checking on display. In one testing round for the Pattern Pathway game, the team supposed that two-year-olds would require more support than three-year-olds to pick up the activity, and set up a play-test with an adult moderator to find out which audio instructions and prompts the children genuinely needed [LET-03]. Design-based research supplies a methodological scaffold for this: designers record the choices they make, build a logic model of how a design's parts are supposed to feed into one another, and gather detailed data during use so they can weigh their predictions against what really occurs in practice [LET-03]. When outcomes surprise them, designers work out explanations and propose adjustments; when they cannot, they may expose gaps in the theories they have been relying on [LET-03].

The tools chapter provides further ways to make assumptions explicit and checkable. Logic models and conjecture maps let teams spell out how starting beliefs connect to desired results, with a conjecture map tracing a clear path from a conjecture grounded in learning theory, to how that conjecture is embodied in the experience, to the processes that mediate it, and on to the intended results [LET-12]. To assess whether a learning experience delivers the effects it aims for, the chapter turns to Kirkpatrick's four levels—reactions, learning, behavior, and results—as a frame for deciding what to measure [LET-12]. Throughout, the advice reflects Nielsen's warning that lightweight testing is acceptable, but running no users at all returns no insight [LET-12].

Further Reading

Source: wrgr/lecommons — curated by the learning engineering community. Confidence: medium — lecommons-curated; not yet independently expert-validated in this context. To validate or challenge any item: use the Edit button on this page. Upgrading confidence from medium → high requires expert sign-off.

Landmark Academic Papers

• Design principles for virtual humans in educational technology environments — Scotty D. Craig, Nicolaus L. Schroeder (2018). International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-017-0148-y · ~150 citations · tier: contemporary

  Synthesized human-factors engineering principles for designing pedagogical agents. Showed that voice, appearance, and persona of virtual instructors systematically affect learner cognitive load and perceived credibility — establishing that LE must integrate psychological design alongside algorithmic design. Source: lecommons/landscape/data/papers.json · ID: LE-LS-AP-009 · confidence: medium · expert-validated: false

• The voice quality of pedagogical agent impacts learning and agent perceptions — Ryan F. Siegle, Scotty D. Craig (2024). Journal of Computer Assisted Learning · doi:10.1111/jcal.12997 · ~20 citations · tier: contemporary

  Empirically demonstrated that voice quality (prosody, naturalness, warmth) of a pedagogical agent significantly affects both learning outcomes and learner perception. Illustrates the meticulous human factors engineering required in modern AI-driven learning systems. Source: lecommons/landscape/data/papers.json · ID: LE-LS-AP-010 · confidence: medium · expert-validated: false

Key People

• Scotty D. Craig (profile), Arizona State University (active 2000s–present) — Human factors researcher; virtual humans and pedagogical agents specialist

  Systematically investigated design principles for virtual humans and pedagogical agents in learning environments Source: lecommons/landscape/data/people.json · ID: LE-LS-PP-014 · confidence: medium · expert-validated: false

Organizations, Conferences & Journals

• International Journal of STEM Education (journal) · link

  Source: lecommons/landscape/data/organizations.json · ID: LE-LS-JO-007 · confidence: medium · expert-validated: false

Programs & Initiatives

• LENS @ JHU — Learning Engineering for Next-Generation Systems (PC) · link

  Concentration within JHU MEd in Learning Design & Technology. Targets practitioners in complex organizations: defense, healthcare, large-scale education. Grounded in human systems integration and learning engineering expertise. Capstone produces evidence dashboard, reproducible report, and governance/ethics plan. Unique JHU ecosystem: APL + Medicine + IEEE/ICICLE partnership. Source: lecommons/site/src/data/programs_people_registry.json · ID: LE-PP-001 · confidence: medium · expert-validated: false

• Learning Engineering Fellowship (CMU OLI) (PC) · link

  Nine-week intensive for educators and designers to apply learning science and data-informed methods to real educational products and contexts; part of OLI professional learning. Source: lecommons/site/src/data/programs_people_registry.json · ID: LE-PP-075 · confidence: medium · expert-validated: false

• ASU Learning Engineering Institute & Graduate Certificate (PC) · link

  Graduate certificate and research network fusing human systems engineering, design, and evidence to improve educational systems; connects students with Learning Engineering Research Network partners. Source: lecommons/site/src/data/programs_people_registry.json · ID: LE-PP-079 · confidence: medium · expert-validated: false

• Purdue School of Engineering Education (ENE) (PC) · link

  First-in-the-nation school of engineering education; graduate offerings include the online M.S. in Engineering Education, Ph.D. in engineering education research, and the stackable Teaching and Learning in Engineering graduate certificate—explicit “learning engineering” language appears in certificate and course titles. Source: lecommons/site/src/data/programs_people_registry.json · ID: LE-PP-089 · confidence: medium · expert-validated: false

• International Journal of STEM Education (CO) · link

  Discipline-based education research (DBER): problem-based learning, flipped classrooms, educational robotics, STEM learning outcomes at scale. Source: lecommons/site/src/data/programs_people_registry.json · ID: LE-PP-130 · confidence: medium · expert-validated: false

• Human Factors and Ergonomics Society (CO) · link

  Professional society for human factors and ergonomics. Publishes Human Factors journal and organizes the annual HFES conference. The Training Systems technical group directly addresses LE-relevant human factors in instructional systems. Source: lecommons/site/src/data/programs_people_registry.json · ID: LE-PP-133 · confidence: medium · expert-validated: false

Lecommons enrichment applied 2026-04-17. All items pending expert validation. See wrgr/lecommons for source data and curation methodology.