8.1 Overview of Learning Analytics

Section 8.1 covers the foundational concepts of learning analytics (LA) as applied within the learning engineering (LE) process: what LA is, how it relates to adjacent fields like Educational Data Mining (EDM) and educational research more broadly, and how the field has evolved through its major institutional and publication venues.

Definition and Scope

Learning analytics is broadly defined as the measurement, collection, analysis, and reporting of data about learners and their contexts, with the purpose of understanding and optimizing learning and the environments in which it occurs. Within learning engineering, "learning analytics" in practice spans a wider scope than this definition implies, frequently incorporating methods from EDM and predictive modeling that originated in data science and statistics. [LE-LS-GL-007] The data instrumentation knowledge area covers how data is captured; this section focuses on what is done with that data once captured.

LA operates at multiple grain sizes simultaneously: the individual learner (is this student stuck on a specific knowledge component?), the cohort (which items are systematically harder than expected?), the course (where in the sequence do students disengage?), and the institution (which programs produce durable learning outcomes?). [LE-LS-AP-005] Each level of analysis serves different stakeholders and requires different methods.

Relationship to Educational Data Mining

EDM and LA are sister fields that emerged roughly simultaneously in the mid-2000s but carry different orientations. EDM is fundamentally discovery-oriented: it applies algorithmic techniques — Bayesian networks, clustering, sequence mining, text mining — to find previously unknown patterns in large educational datasets. [LE-LS-AP-005] The International Educational Data Mining Society organizes the annual EDM Conference and publishes the Journal of Educational Data Mining, which focuses on algorithmic and methodological advances. [LE-LS-CO-006] [LE-LS-JO-002]

LA is more application-oriented: it uses known methods to support actionable decisions in real educational contexts. The Society for Learning Analytics Research (SoLAR) organizes the annual LAK Conference and publishes the Journal of Learning Analytics, with emphasis on sociotechnical implications, dashboard design, ethical deployment, and policy. [LE-LS-CO-003] [LE-LS-CE-002] [LE-LS-JO-004]

In practice the boundary is porous. A learning engineer conducting an EDM-style clustering analysis of student errors is doing so in service of an LA goal: improving a specific course. The "High-Leverage Opportunities" report, produced by the Penn Center for Learning Analytics after convening 100+ academics and policymakers, explicitly called for tighter integration between EDM discovery methods and LE deployment infrastructure. [LE-LS-GL-003]

Relationship to Educational Research

Traditional educational research — including experimental psychology, educational psychology, and psychometrics — provides the theoretical foundations on which LA builds. Design-based research (DBR), as established by Ann Brown, supplies the methodological canon for testing learning interventions in authentic classroom settings. [LE-LS-AP-003] DBR's iterative design-test-revise cycle maps directly onto the LE process: LA provides the data that feeds each revision cycle. Where traditional educational research often operates on small samples and controlled conditions, LA operates on large-scale observational data from live deployments.

The Cognitive Tutor program at CMU, developed by Ken Koedinger and colleagues, pioneered the integration of fine-grained learning analytics with intelligent tutoring systems. [LE-LS-AP-014] The log data generated by Cognitive Tutors and stored in DataShop (now LearnSphere) became the empirical foundation for much early EDM and LA research. [LE-LS-PP-005]

Evolution of the Field

The emergence of learning analytics as a named discipline can be traced to three converging developments. First, the proliferation of digital learning platforms (LMSs, ITSs, MOOCs) produced unprecedented volumes of learner interaction data. Second, machine learning techniques matured to the point where they could be applied to educational log data by researchers without specialized ML backgrounds. Third, institutional and policy interest in "learning outcomes" drove demand for evidence-based improvement.

The 2009 paper by Baker and Yacef, "The State of Educational Data Mining in 2009," was the landmark codification of EDM as a distinct discipline and taxonomy of its methods. [LE-LS-AP-005] Ryan Baker at the Penn Center for Learning Analytics has since been the primary driver of EDM-to-policy translation, producing practical frameworks for how institutions should deploy analytics responsibly. [LE-LS-PP-008] [LE-LS-CO-004]

The Simon Initiative at CMU represents the institutional infrastructure side of this evolution: by consolidating DataShop, LearnSphere, OLI, and CTAT under one roof, it created the conditions for longitudinal, cross-study analytics research at scale. [LE-LS-CO-002]

¶ Subsections:

• 8.1.1 Definition and Scope of Learning Analytics
• 8.1.2 Learning Analytics in Support of Learning Engineering
• 8.1.2 Disambiguation of Terms Used in Other Domains (e.g. Statistics, Psychometrics)

From the Learning Engineering Toolkit

The Toolkit presents analytics as the second of the two ways learning engineers work with data — following instrumentation — describing it as the computer-driven, systematic examination of data that sits at the heart of the investigation phase of the learning engineering process. [LET-06] Its aim is pragmatic: to give feedback to learners and those who support them, and to guide successive improvements to whatever is being engineered. [LET-06]

The book argues that analytics is necessary precisely because intuition cannot be trusted — people's hunches about what aids learning are frequently mistaken, so evidence from even modest trials helps a team stay fixed on the design features that matter most. [LET-06] It offers a Kaplan LSAT logical-reasoning example, where a learning engineer judged that the video-and-workbook format would tax learners' working memory and proposed worked examples built to reduce split attention instead. [LET-06] Both worked-example conditions outperformed the roughly ninety-minute video-and-workbook version by a statistically significant margin, and the learners who reviewed just eight worked examples in about nine minutes did best of all. [LET-06]

The book is careful to separate learning engineering research from the broader, generalizable research of the learning sciences. The Kaplan team built an alternative, tried it out in its actual setting, and let the resulting data pick the winner — yet that particular result probably will not carry over to other situations, because learning engineering research yields highly context-bound findings tied to specific products and learner groups rather than universal laws of learning. [LET-06] It further warns that numbers do not always tell the complete story and that learners are embedded in a context, which means any experiment's outcome is context-bound as well. [LET-06]

Finally, the book describes an increasing sophistication in analysis — a progression from descriptive toward predictive and then prescriptive analytics — and its companion tools chapter supplies a learning analytics process model to help a team work out what sort of analysis a particular question calls for. [LET-06] [LET-18]