2.5.6 Testing for Learning Impact

User testing in learning engineering often includes measures of learning impact, such as retention and comprehension, to ensure that solutions support learning goals effectively.

From the Learning Engineering Toolkit

The Toolkit does not use the precise wording testing for learning impact, but it clearly separates whether a design is easy to use from whether it genuinely teaches. Among the traits that set learners apart from everyday software users, it observes that the computer interactions which are simplest to use are not always the ones that best help someone learn the intended concepts and skills. [LET-03] Testing therefore has to be steered in a human-centered fashion: the team decides which subgroups to include, what signals indicate progress toward the targeted goals, and how that progress will be measured. [LET-03]

For measuring learning impact in particular, the Toolkit puts forward Donald Kirkpatrick's four-level evaluation model, which runs from simpler-but-weaker to more-demanding-but-more-telling measures: reactions (whether learners enjoyed it), learning (what they took away, typically shown by a pre/post-test), behavior (whether the experience shifted their later conduct), and results (whether any real-world impact was noticeable). [LET-12] Usability tools like the System Usability Scale capture ease of use, satisfaction, and learnability rather than gains in learning, which highlights that usability testing and learning-impact testing address different questions. [LET-12]

Design-based research illustrates how impact can be judged on an ongoing basis instead of only at the finish: learning outcomes are typically built into the real-world data gathering, supplemented by snapshots of what learners are doing and saying, and designers set their predictions against what actually occurs so they can account for surprises and propose refinements. [LET-03] The Math Readiness story gives a concrete case. Together with research partners, the My Math Academy team assessed how well it worked and found that it sped up math learning for older pre-kindergarteners and kindergarteners but did less for the youngest children, who lacked prerequisite skills—an outcome that led to the creation of Math Readiness in the first place. [LET-03]

Further Reading

Source: wrgr/lecommons — curated by the learning engineering community. Confidence: medium — lecommons-curated; not yet independently expert-validated in this context. To validate or challenge any item: use the Edit button on this page. Upgrading confidence from medium → high requires expert sign-off.

Landmark Academic Papers

• Design principles for virtual humans in educational technology environments — Scotty D. Craig, Nicolaus L. Schroeder (2018). International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-017-0148-y · ~150 citations · tier: contemporary

  Synthesized human-factors engineering principles for designing pedagogical agents. Showed that voice, appearance, and persona of virtual instructors systematically affect learner cognitive load and perceived credibility — establishing that LE must integrate psychological design alongside algorithmic design. Source: lecommons/landscape/data/papers.json · ID: LE-LS-AP-009 · confidence: medium · expert-validated: false

• The voice quality of pedagogical agent impacts learning and agent perceptions — Ryan F. Siegle, Scotty D. Craig (2024). Journal of Computer Assisted Learning · doi:10.1111/jcal.12997 · ~20 citations · tier: contemporary

  Empirically demonstrated that voice quality (prosody, naturalness, warmth) of a pedagogical agent significantly affects both learning outcomes and learner perception. Illustrates the meticulous human factors engineering required in modern AI-driven learning systems. Source: lecommons/landscape/data/papers.json · ID: LE-LS-AP-010 · confidence: medium · expert-validated: false

Key People

• Scotty D. Craig (profile), Arizona State University (active 2000s–present) — Human factors researcher; virtual humans and pedagogical agents specialist

  Systematically investigated design principles for virtual humans and pedagogical agents in learning environments Source: lecommons/landscape/data/people.json · ID: LE-LS-PP-014 · confidence: medium · expert-validated: false

Organizations, Conferences & Journals

• International Journal of STEM Education (journal) · link

  Source: lecommons/landscape/data/organizations.json · ID: LE-LS-JO-007 · confidence: medium · expert-validated: false

Programs & Initiatives

• LENS @ JHU — Learning Engineering for Next-Generation Systems (PC) · link

  Concentration within JHU MEd in Learning Design & Technology. Targets practitioners in complex organizations: defense, healthcare, large-scale education. Grounded in human systems integration and learning engineering expertise. Capstone produces evidence dashboard, reproducible report, and governance/ethics plan. Unique JHU ecosystem: APL + Medicine + IEEE/ICICLE partnership. Source: lecommons/site/src/data/programs_people_registry.json · ID: LE-PP-001 · confidence: medium · expert-validated: false

• Learning Engineering Fellowship (CMU OLI) (PC) · link

  Nine-week intensive for educators and designers to apply learning science and data-informed methods to real educational products and contexts; part of OLI professional learning. Source: lecommons/site/src/data/programs_people_registry.json · ID: LE-PP-075 · confidence: medium · expert-validated: false

• ASU Learning Engineering Institute & Graduate Certificate (PC) · link

  Graduate certificate and research network fusing human systems engineering, design, and evidence to improve educational systems; connects students with Learning Engineering Research Network partners. Source: lecommons/site/src/data/programs_people_registry.json · ID: LE-PP-079 · confidence: medium · expert-validated: false

• Purdue School of Engineering Education (ENE) (PC) · link

  First-in-the-nation school of engineering education; graduate offerings include the online M.S. in Engineering Education, Ph.D. in engineering education research, and the stackable Teaching and Learning in Engineering graduate certificate—explicit “learning engineering” language appears in certificate and course titles. Source: lecommons/site/src/data/programs_people_registry.json · ID: LE-PP-089 · confidence: medium · expert-validated: false

• International Journal of STEM Education (CO) · link

  Discipline-based education research (DBER): problem-based learning, flipped classrooms, educational robotics, STEM learning outcomes at scale. Source: lecommons/site/src/data/programs_people_registry.json · ID: LE-PP-130 · confidence: medium · expert-validated: false

• Human Factors and Ergonomics Society (CO) · link

  Professional society for human factors and ergonomics. Publishes Human Factors journal and organizes the annual HFES conference. The Training Systems technical group directly addresses LE-relevant human factors in instructional systems. Source: lecommons/site/src/data/programs_people_registry.json · ID: LE-PP-133 · confidence: medium · expert-validated: false

Lecommons enrichment applied 2026-04-17. All items pending expert validation. See wrgr/lecommons for source data and curation methodology.