Learning analytics is the measurement arm of the learning engineering process. Learning engineering (LE) is defined as an iterative, evidence-based process of designing, implementing, and improving learning experiences — one that treats educational design as an empirical engineering discipline rather than a craft. [LE-LS-GL-007] Analytics provides the feedback signal that makes iteration possible: without data, "improve" is just guessing. This page explains precisely how LA functions within each phase of the LE cycle and what specific capabilities it must provide.
LA as Evidence for Iterative Design
Design-based research, established by Ann Brown as the canonical methodology for testing engineered interventions in authentic educational settings, frames research as iterative: design, test in real classrooms, analyze outcomes, revise, repeat. [LE-LS-AP-003] Learning analytics operationalizes the "analyze outcomes" step. For each design iteration, the LE team must answer: did learners achieve mastery? Where did they struggle? Did the sequence of activities produce the intended learning trajectory? These questions require performance data at the knowledge-component level, not just final test scores.
The Cognitive Tutor program at CMU exemplified this loop. Over a decade of development, logged performance data from thousands of students was used to refine knowledge-component models, adjust mastery thresholds, and redesign problem sequences. [LE-LS-AP-014] The resulting systems showed measurable learning gains in real school deployments — gains that could be attributed to specific design decisions traceable to data. Kenneth Koedinger's founding of DataShop (now LearnSphere) institutionalized this practice by providing a shared repository where such data could be accumulated, re-analyzed, and compared across studies. [LE-LS-PP-005] [LE-LS-GL-010]
Student Modeling as Real-Time LE Infrastructure
One of the most direct ways LA supports LE is through student modeling: computational representations of learner knowledge states that update continuously as learners interact with the system. Bayesian Knowledge Tracing (BKT), developed by Albert Corbett and John Anderson, is the foundational algorithm. [LE-LS-AP-011] BKT models the probability that a student has acquired a given knowledge component, updating this estimate after each practice attempt based on whether the response was correct. The system uses this estimate to decide whether to present the next problem, repeat practice, or provide remediation. [LE-LS-AP-002]
Student modeling is not merely a feature of the learning system; it is the analytical layer that allows the LE team to observe learning in near-real-time and make design decisions based on aggregated model outputs. If a knowledge component shows consistently low learning rates across students, that is a signal to redesign the instruction for that component, not just to give students more practice. [LE-LS-AP-014]
Supporting A/B Testing and Randomized Controlled Trials
A central LE practice is testing competing instructional designs against each other under controlled conditions. LA provides the infrastructure and outcome measures for this experimentation. When an LE team has two versions of an explanation — one text-based, one worked example — they need a mechanism to randomly assign learners to conditions and measure differential outcomes. At scale, this requires both a platform capable of running within-platform randomization and an analytics pipeline capable of attributing outcome differences to the treatment.
Neil Heffernan's E-TRIALS platform, built on top of the ASSISTments system serving over 500,000 students, was specifically designed for this purpose: running randomized controlled trials within live educational deployments without disrupting normal instruction. [LE-LS-PP-011] The doer effect research — demonstrating a causal advantage of active practice over passive reading across seven courses — was made possible by precisely this kind of LE-embedded experimental infrastructure. [LE-LS-AP-012]
The "High-Leverage Opportunities" report listed A/B testing infrastructure as one of the top strategic priorities for the field, noting that most LE teams lack the institutional access and shared platforms needed to run properly powered experiments. [LE-LS-GL-003]
Continuous Improvement Loops
In production learning systems — adaptive platforms, MOOCs, corporate training systems — LA enables continuous improvement without waiting for end-of-cycle evaluation. This means instrumenting the system to log all relevant learner interactions from the first deployment, establishing baseline performance metrics, and running ongoing monitoring to detect degradation or improvement as content changes. [LE-LS-AP-005]
The Learning Engineering Toolkit frames this as the "data investigation" phase of the LE process, following implementation. [LE-LS-GL-007] Analytics findings from one deployment cycle directly feed the design changes for the next. This is the feedback loop structure that distinguishes learning engineering from traditional instructional design, where evaluation is often summative and retrospective rather than formative and prospective.
Ryan Baker and colleagues at the Penn Center for Learning Analytics have argued that this continuous improvement model — analogous to quality management in manufacturing — requires dedicated analytics roles within LE teams, not just occasional analysis by researchers. [LE-LS-CO-004] The Generalizable Learning Engineering Adoption Maturity Model assesses organizational capability along this dimension: mature LE organizations have systematic analytics pipelines, not ad hoc data pulls. [LE-LS-AP-013]
Specific LA Capabilities That LE Teams Need
Translating these roles into practical team capabilities, an LE team conducting data-informed design needs:
- Log data access — the ability to query learner interaction logs at the knowledge-component level, with timestamps and system-response context. [LE-LS-AP-002]
- Student model outputs — probability estimates of knowledge state per student per KC, enabling identification of which KCs are systematically unlearned. [LE-LS-AP-011]
- Cohort-level aggregation — the ability to compute item-level and KC-level statistics across cohorts to identify systematic design problems. [LE-LS-AP-005]
- Experimental design capability — the infrastructure and statistical expertise to run within-platform comparisons with sufficient power to detect educationally meaningful effect sizes. [LE-LS-AP-012]
- Visualization and communication tools — dashboards and reports that translate analytic findings into actionable recommendations for instructors and designers. [LE-LS-GL-003]
The OpenSimon Toolkit — comprising DataShop, LearnSphere, CTAT, OLI, and DiscourseDB — provides open-source infrastructure for capabilities 1–4, making these tools available beyond CMU to the broader LE community. [LE-LS-GL-010]