4.4.2 Prototyping and Testing Phases

• Rapid Prototyping: Quickly develops initial versions of learning interventions to gather early feedback, allowing for rapid assessment of design elements before full-scale implementation.

• Formative and Summative Evaluation: Collects data during development (formative) to inform adjustments, and evaluates the final solution’s impact on learning outcomes (summative).

  ---

From the Learning Engineering Toolkit

Engineering leans on prototypes before a solution is fully built out. [LET-04] The Toolkit lays out a common progression: work frequently begins with paper prototypes that sketch out an idea as wireframes, advances to rough models, and then to a minimum viable product (MVP) that a small group of users can try. [LET-04] Chapter 4 casts the MITx platform as precisely this sort of prototype. [LET-04]

Testing opened with pilot groups. Piotr Mitros was caught off guard when the first prototype performed well: the team drew strongly positive responses from a pilot group of on-campus students, and a month later equally positive responses from online students. [LET-04] The chapter reports that 98.6 percent of alumni judged the course to be as good as or better than an in-person offering. [LET-04] It also points out the limits of these early findings, cautioning that sample bias and experimenter bias colored them, even though they were enough to secure $60 million in funding for the edX project. [LET-04]

Prototyping brought constraints to light that reshaped the design. An early constructive-learning approach turned out to be too time-consuming for instructor professional development, so most learning sequences shifted to simpler understanding-check exercises woven through lecture-style material—still active learning, but less constructive. [LET-04] Testing choices were driven by pedagogy: multiple-choice questions were intentionally left out because they clashed with the mastery-learning model, in which students could retry open-ended questions as many times as they needed. [LET-04] The platform was then engineered for ongoing testing, with a pluggable component architecture (XModules, later renamed XBlocks) and support for A/B testing so that activities could be evaluated and refined using real learner data. [LET-04]

Further Reading

Source: wrgr/lecommons — curated by the learning engineering community. Confidence: medium — lecommons-curated; not yet independently expert-validated in this context. To validate or challenge any item: use the Edit button on this page. Upgrading confidence from medium → high requires expert sign-off.

Landmark Academic Papers

• Design principles for virtual humans in educational technology environments — Scotty D. Craig, Nicolaus L. Schroeder (2018). International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-017-0148-y · ~150 citations · tier: contemporary

  Synthesized human-factors engineering principles for designing pedagogical agents. Showed that voice, appearance, and persona of virtual instructors systematically affect learner cognitive load and perceived credibility — establishing that LE must integrate psychological design alongside algorithmic design. Source: lecommons/landscape/data/papers.json · ID: LE-LS-AP-009 · confidence: medium · expert-validated: false

• The voice quality of pedagogical agent impacts learning and agent perceptions — Ryan F. Siegle, Scotty D. Craig (2024). Journal of Computer Assisted Learning · doi:10.1111/jcal.12997 · ~20 citations · tier: contemporary

  Empirically demonstrated that voice quality (prosody, naturalness, warmth) of a pedagogical agent significantly affects both learning outcomes and learner perception. Illustrates the meticulous human factors engineering required in modern AI-driven learning systems. Source: lecommons/landscape/data/papers.json · ID: LE-LS-AP-010 · confidence: medium · expert-validated: false

Key People

• Scotty D. Craig (profile), Arizona State University (active 2000s–present) — Human factors researcher; virtual humans and pedagogical agents specialist

  Systematically investigated design principles for virtual humans and pedagogical agents in learning environments Source: lecommons/landscape/data/people.json · ID: LE-LS-PP-014 · confidence: medium · expert-validated: false

Programs & Initiatives

• Human Factors and Ergonomics Society (CO) · link

  Professional society for human factors and ergonomics. Publishes Human Factors journal and organizes the annual HFES conference. The Training Systems technical group directly addresses LE-relevant human factors in instructional systems. Source: lecommons/site/src/data/programs_people_registry.json · ID: LE-PP-133 · confidence: medium · expert-validated: false

• INCOSE Systems Engineering Body of Knowledge (SEBoK) (CO) · link

  Open wiki-based body of knowledge for systems engineering. Includes the Human Systems Integration knowledge area. The SE process framework (requirements, architecture, integration, verification) is the engineering process backbone that LE adapts for learning system development. Source: lecommons/site/src/data/programs_people_registry.json · ID: LE-PP-134 · confidence: medium · expert-validated: false

Lecommons enrichment applied 2026-04-17. All items pending expert validation. See wrgr/lecommons for source data and curation methodology.