Data-informed decision making is the practice of using systematically collected evidence about learner behavior and outcomes to guide specific, documented choices in learning solution design. It is distinguished from both data-driven decision making (where data mechanically determines outcomes) and from purely intuitive design (where decisions reflect designer experience without empirical grounding). The distinction matters because learning systems involve complex human factors — motivation, prior knowledge, context — that no data stream fully captures; learning engineers therefore use data as a primary but not sole input into judgment. The Learning Engineering Toolkit defines this balance explicitly, treating data as a tool for disciplining design intuition rather than replacing it. [LE-LS-GL-007]
From Analysis to Actionable Design Changes
The core challenge of data-informed decision making is closing the gap between observation and action. Collecting interaction data from a learning platform is technically straightforward; interpreting that data correctly to diagnose a design problem, generating a specific design hypothesis to address it, and implementing the change in a way that can be evaluated in the next cycle — this sequence requires both analytic skill and structured process. Ryan Baker's work on the Penn Center for Learning Analytics has produced practical frameworks for moving from educational data mining outputs to instructor-facing and designer-facing dashboards that present data in forms accessible to non-specialists. [LE-LS-CO-004]
Bayesian Knowledge Tracing exemplifies real-time data-informed decision making within deployed adaptive systems: the student model continuously updates its probability estimates of skill mastery with each learner response, and the system's instructional decisions — which problem to present next, whether to surface a hint, when to advance — are directly driven by these estimates. [LE-LS-AP-011] This architecture makes the data-decision loop operate at subsecond latency, far faster than the sprint-level cycles of agile development. At the other temporal extreme, large-scale analyses of platform telemetry — such as the doer effect study across seven courses — yield design principles that are then incorporated into platform architectures and authoring guidelines, influencing thousands of future learner experiences. [LE-LS-AP-012]
Decision Documentation and Iterative Feedback Loops
A critical but often underemphasized aspect of data-informed decision making is documentation: recording not only what design change was made but why, what data motivated it, and what outcome was expected. Without this documentation, iterative cycles lose their epistemic value — a design revision that improves performance cannot be distinguished from one that makes no difference, and the learning engineering team accumulates artifacts without accumulating knowledge. The Learning Engineering Adoption Maturity Model, developed by IEEE ICICLE, includes systematic decision documentation as a key capability indicator distinguishing mature LE practice from ad-hoc instructional design. [LE-LS-GL-009]
Ann Brown's foundational design-based research methodology established that educational interventions tested in authentic settings must include systematic tracking of what changed, when, and in response to what evidence — precisely because the complexity of classroom environments makes post-hoc reconstruction unreliable. [LE-LS-AP-003] Kenneth Koedinger's DataShop infrastructure extended this principle to digital scale by creating standardized formats for logging both interaction data and researcher annotations, enabling systematic comparison across studies and institutions. [LE-LS-PP-005] Together these precedents support a practice norm in which every design revision is an explicit, documented hypothesis test — not merely a response to perceived problems, but a prediction about how a specific change will affect a specific measured outcome, verifiable in the next iteration. [LE-LS-GL-007]