8.2.3 Identifying Areas for Enhancement

Learning analytics becomes actionable when it moves beyond description — summarizing what happened — into diagnosis and prescription: identifying what needs to change and why. Section 8.2.3 addresses this transition: the systematic use of data to locate knowledge components that are failing learners, flag individuals at risk, surface ineffective content, and trigger interventions before problems compound.

From Description to Diagnosis

Most learning platforms produce descriptive statistics by default: completion rates, average scores, time-on-task. These metrics are necessary but insufficient. A high average score can mask a bimodal distribution where half the class is excelling and half is failing. A completion rate tells nothing about whether the completed experience produced durable learning. The learning engineer's task is to move downstream from these surface statistics into structural diagnosis.

Bayesian Knowledge Tracing (BKT), formalized by Corbett and Anderson, provides a principled framework for this kind of diagnosis at the skill level.[LE-LS-AP-011] BKT estimates the probability that a given learner has acquired a given knowledge component, updated with each practice opportunity. Aggregating BKT estimates across a cohort reveals which knowledge components are systematically under-mastered — not because learners are unmotivated, but because the instructional sequence, worked examples, or feedback for that component may be inadequate. This is the grain size at which content improvement decisions are most productive.

The Learning Engineering Toolkit identifies knowledge-component analysis as a core LE practice: the learning engineer examines error patterns and mastery curves to pinpoint exactly which skills are bottlenecks, then redesigns the specific materials or practice sequences that address those bottlenecks.[LE-LS-GL-007] This is categorically different from the common instructional design practice of revising an entire course because aggregate satisfaction scores dropped.

Knowledge-Component Gap Analysis

A knowledge-component (KC) map links every instructional activity to one or more underlying skills. Once such a map exists, performance data from the learning system can be aggregated at the KC level to answer: Which KCs show the steepest error rates? Which KCs are taking the most practice opportunities to reach mastery? Which KCs show large variance across learners, suggesting that some instructional paths work and others do not?

Kenneth Koedinger's DataShop repository — the world's largest open archive of educational log data — enables exactly this kind of cross-course, cross-institution KC analysis.[LE-LS-PP-005] By comparing KC mastery curves from multiple instantiations of the same course, learning engineers can isolate whether a KC is inherently difficult (steep curve everywhere) or poorly taught in a particular version (steep curve in one section, shallow in another). The former calls for better instructional design; the latter calls for curriculum revision.

Early Warning Systems and At-Risk Identification

Gap analysis at the KC level identifies content that needs redesign. Early warning systems (EWS) identify learners who need human intervention before they disengage or fail. These two functions are complementary and require different analytical tools.

EWS models typically draw on a combination of engagement signals — login frequency, time-on-task, forum participation, assignment submission latency — and performance signals — quiz scores, BKT mastery estimates, error rates — to generate a composite risk score for each learner.[LE-LS-AP-005] Ryan Baker's work on Educational Data Mining codified the taxonomy of learner behaviors that EWS models must detect: not just poor performance, but gaming-the-system (exploiting hint loops without learning), off-task behavior, and boredom — all of which predict disengagement and eventual failure.[LE-LS-PP-008]

The High-Leverage Opportunities report identifies predictive advising — flagging at-risk learners and routing them to human advisors or automated scaffolds — as one of the ten strategic priorities for learning engineering infrastructure.[LE-LS-GL-003] The critical design challenge is ensuring that EWS alerts are actionable: the system must not only identify risk but specify what kind of support is indicated and make it easy for instructors or advisors to provide it.

Ineffective Content Identification

Beyond individual learners and individual KCs, analytics can surface content items — videos, readings, assessments — that are systematically failing to produce the intended learning. The doer effect research by Van Campenhout and colleagues established, across seven courses, that active practice produces substantially larger learning gains than passive reading of equivalent content.[LE-LS-AP-012] This has a direct implication for gap analysis: when a content item is identified as ineffective, the first diagnostic question should be whether it is primarily passive (reading, watching) and whether it could be replaced or supplemented by active practice.

Item-level analysis within assessments provides another lens. Items with discrimination indices near zero — items that do not differentiate high-performing from low-performing learners — are either too easy, too ambiguous, or assessing something other than what the surrounding instruction targeted. Psychometric flags of this kind, combined with qualitative review, guide item revision.

Closing the Loop: From Finding to Action

The analytical steps described above are valuable only if they feed back into the design process. The Learning Engineering Toolkit frames this as the "data investigation" phase of the iterative LE cycle: findings from gap analysis, EWS alerts, and content diagnostics become inputs to the next design iteration.[LE-LS-GL-007] This requires institutional infrastructure — shared data access, version-controlled content, and decision documentation protocols — that many organizations have not yet built.

The Penn Center for Learning Analytics has articulated the need for shared LE infrastructure that makes continuous improvement a default practice rather than a special project.[LE-LS-CO-004] Identifying areas for enhancement is not a one-time audit; it is an ongoing analytical posture that learning engineers must build into the operational rhythms of any learning system they are responsible for.

Further Reading

Source: wrgr/lecommons — curated by the learning engineering community. Confidence: medium — lecommons-curated; not yet independently expert-validated in this context. To validate or challenge any item: use the Edit button on this page. Upgrading confidence from medium → high requires expert sign-off.

Landmark Academic Papers

• SOAR: An Architecture for General Intelligence — John E. Laird, Allen Newell, Paul S. Rosenbloom (1987). Artificial Intelligence · doi:10.1016/0004-3702(87)90050-6 · ~5,000 citations · tier: foundational

  Established SOAR as a unified cognitive architecture. The chunking mechanism in SOAR provides a computational model of procedural learning from practice — directly informing how ITS should structure problem sequences and when to apply mastery criteria. Foundational for cognitive modeling in learning engineering. Source: lecommons/landscape/data/papers.json · ID: LE-LS-AP-015 · confidence: medium · expert-validated: false

• Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventions in Classroom Settings — Ann L. Brown (1992). Journal of the Learning Sciences · doi:10.1207/s15327809jls0202_2 · ~2,800 citations · tier: foundational

  The most-cited paper in JLS history. Established design-based research as the canonical methodology for testing engineered learning interventions in authentic classroom settings. Defined the epistemological standards that learning engineering uses when validating systems in the wild. Source: lecommons/landscape/data/papers.json · ID: LE-LS-AP-003 · confidence: medium · expert-validated: false

• Knowledge Tracing: Modeling the Acquisition of Procedural Knowledge — Albert T. Corbett, John R. Anderson (1994). User Modeling and User-Adapted Interaction · doi:10.1007/BF01099821 · ~2,500 citations · tier: foundational

  The formal presentation of Bayesian Knowledge Tracing (BKT) — the probabilistic model for tracking per-student, per-skill mastery. BKT remains the most widely deployed student model in production ITS and learning platforms worldwide, forming the core of personalization algorithms. Source: lecommons/landscape/data/papers.json · ID: LE-LS-AP-011 · confidence: medium · expert-validated: false

• Cognitive Tutors: Lessons Learned — John R. Anderson, Albert T. Corbett, Kenneth R. Koedinger et al. (1995). Journal of the Learning Sciences · doi:10.1207/s15327809jls0402_2 · ~1,800 citations · tier: foundational

  The comprehensive review of a decade of Cognitive Tutor development and deployment. Documented both the theory (ACT-R production rules, BKT) and the empirical learning gains in real schools. One of the most influential synthesis papers in ITS, directly shaping subsequent adaptive learning system design. Source: lecommons/landscape/data/papers.json · ID: LE-LS-AP-014 · confidence: medium · expert-validated: false

Policy, Reports & Grey Literature

• High-Leverage Opportunities for Learning Engineering — Ryan S. Baker, Ulrich Boser, Allison Shelley (2021). University of Pennsylvania Center for Learning Analytics · link

  Synthesized a 2020 convening of 100+ academics, policymakers, and practitioners to define ten strategic priorities for the field in three domains: Better LE Infrastructure (shared R&D architectures, A/B testing at scale, reusable algorithmic components), Supporting Human Processes (teacher dashboards, predictive advising), and Better Learning Technologies (algorithmic equity, complex-skills measurement). The field's most comprehensive contemporary roadmap. Source: lecommons/landscape/data/grey_literature.json · ID: LE-LS-GL-003 · confidence: low · expert-validated: false

• Learning Engineering Toolkit: Evidence-Based Practices from the Learning Sciences, Instructional Design, and Beyond — Jim Goodell, Janet Kolodner (2023). Taylor & Francis / Routledge · link

  The field's primary practitioner handbook. Frames LE as a verb — an iterative, evidence-based problem-solving process — not a technology set or a job title. Covers lean-agile methods, HCI design, data instrumentation, motivation modeling, and predictive analytics. Explicitly technology-agnostic: the principles apply equally to AI-driven software and low-tech interventions. Source: lecommons/landscape/data/grey_literature.json · ID: LE-LS-GL-007 · confidence: low · expert-validated: false

Key People

• John R. Anderson (profile), Carnegie Mellon University, HCII (1947–present) — Cognitive architect; creator of ACT-R; pioneer of Cognitive Tutors

  Developed ACT-R (Adaptive Control of Thought–Rational), the dominant cognitive architecture for modeling skill learning Source: lecommons/landscape/data/people.json · ID: LE-LS-PP-002 · confidence: medium · expert-validated: false

• Kenneth R. Koedinger (profile), Carnegie Mellon University, HCII (active 1988–present) — Co-originator of learning engineering as a named field; Cognitive Tutor pioneer; DataShop founder

  Led development of Cognitive Tutors deployed in thousands of schools; co-founded Carnegie Learning Inc. Source: lecommons/landscape/data/people.json · ID: LE-LS-PP-005 · confidence: medium · expert-validated: false

Organizations, Conferences & Journals

• Penn Center for Learning Analytics (research_center) · link

  Source: lecommons/landscape/data/organizations.json · ID: LE-LS-CO-004 · confidence: medium · expert-validated: false

• International Educational Data Mining Society (academic_society) · link

  Source: lecommons/landscape/data/organizations.json · ID: LE-LS-CO-006 · confidence: medium · expert-validated: false

Lecommons enrichment applied 2026-04-17. All items pending expert validation. See wrgr/lecommons for source data and curation methodology.

Community Recommended Reading

Items from the lecommons corpus identified as relevant to this topic. All text verbatim from lecommons sources. Confidence: medium. Expert validation required.

Data-Informed Course Improvement: The Application of Learning Engineering in the Classroom — Anne Fensie (2023, ICLS 2023)

Higher-ed classroom case study. Fensie walks through a specific course intervention, what data moved, and what didn't — an honest practitioner paper at a major learning sciences venue.

Designing a Student Progress Panel for Formative Practice: A Learning Engineering Process — Rachel Van Campenhout, Michelle Selinger, Bill Jerome (2023, ICLS 2023)

A compact case study of the LE process applied to a real adaptive-learning product (Acrobatiq). Worth reading back-to-back with Fensie's ICLS piece on Data-Informed Course Improvement.

Learning Engineering Enlightenment: Think Like an Engineer — Ellen Wagner (2024, New Learning Frontier)

Wagner's 2024 follow-on to her long thread of LE-vs-ID articles. The sharpest version of her argument that the shift isn't tools, it's disposition: LEs reason like engineers about uncertainty and evidence.

Why Did We Do That? A Systematic Approach to Tracking Decisions in the Design and Iteration of Learning Experiences — Lauren Totino, Aaron Kessler (2023, Journal of Applied Instructional Design)

Totino and Kessler's practical protocol for decision tracking across LE projects. Pairs well with our Field Note on Five Whys — this paper operationalizes the evidence-decision-tracker discipline ICICLE publishes.

Chapter 5: LE Toolkit — Learning Engineering is a Process (open access chapter) — Aaron Kessler, Scotty Craig, Jim Goodell, Dina Kurzweil, Scott Greenwald (2022)

Defines learning engineering as an iterative, evidence-based problem-solving process. Covers five-phase LE process: challenge identification, solution creation, implementation, data investigation, and continuous iteration. Technology-agnostic principles applicable to AI-driven and low-tech interventions.

Context enrichment applied 2026-04-17. See wrgr/lecommons for source data.