8.5.2 Fairness and Bias in Analytics

Algorithmic bias in learning analytics refers to the systematic production of predictions or recommendations that are differentially inaccurate, harmful, or unjust for learners from particular demographic groups. Bias in learning analytics systems is not a theoretical concern — it has been documented in deployed systems that affect real learners' academic trajectories. Understanding how bias enters these systems, how it propagates, and how it can be detected and mitigated is a core competency for learning engineers.

Sources of Bias in Training Data

The most fundamental source of algorithmic bias in learning analytics is biased training data. Models learn from historical data that reflects historical inequities in educational systems: differential access to technology, differential prior preparation, differential treatment by instructors, and structural inequities in the institutions whose data is being analyzed. A model trained on this data does not simply learn to predict outcomes — it learns to reproduce the patterns that generated those outcomes, including the inequitable ones.

Ryan Baker's research has documented specific mechanisms through which training-data bias enters learning analytics models: imbalanced class distributions (where historically underserved students are underrepresented in training data), feature selection that captures proxies for demographic characteristics (such as prior course completion rates that reflect institutional inequity rather than individual capability), and outcome variables that themselves reflect historical bias (such as GPA, which is affected by grading practices that have been shown to vary by instructor demographic and by course context).[LE-LS-PP-008]

The Baker-Yacef EDM taxonomy is relevant here: the "distillation for human judgment" category recognizes that the purpose of EDM is to support human decision-making — and that the quality of that support depends not just on technical accuracy but on whether the patterns being surfaced represent real learning phenomena or artifacts of historical inequity.[LE-LS-AP-005]

Documented Cases: E-Proctoring and Disparate Impact

The U.S. Department of Education's AI in education policy report provided extensive documentation of algorithmic bias in e-proctoring systems, where facial recognition algorithms used to verify student identity and detect cheating were shown to have significantly higher false-positive rates for darker-skinned students, transgender students, and students with certain physical disabilities.[LE-LS-GL-004] These false positives resulted in students being flagged for academic integrity violations, subjected to additional scrutiny, and in some cases receiving grade penalties — harms that accrued disproportionately to already-marginalized groups.

E-proctoring is an extreme case, but the same pattern appears in subtler forms across learning analytics applications. Early-warning systems that flag students for advising outreach, if systematically less accurate for first-generation college students, may either under-serve those students (by generating fewer accurate flags) or over-flag them (by generating more false positives that consume advising resources). Either outcome is an equity failure.

Fairness Metrics and Their Trade-offs

The learning analytics community has imported fairness metrics from the algorithmic fairness literature to evaluate deployed models. The primary metrics include:

Demographic parity — the requirement that a model's positive predictions be equally likely across demographic groups, regardless of the base rates in each group.

Equalized odds — the requirement that a model's true positive rate and false positive rate be equal across demographic groups.

Calibration — the requirement that predicted probabilities have the same interpretation across demographic groups (a 70% predicted probability of dropout should correspond to a 70% actual dropout rate in every demographic subgroup).

These metrics are mathematically incompatible in general when base rates differ across groups — satisfying one fairness criterion typically requires violating another.[LE-LS-CE-002] Learning engineers and institutional decision-makers must therefore make explicit value judgments about which fairness criterion is most important in their context, rather than assuming that any single metric captures what "fairness" means.

The Journal of Learning Analytics has published theoretical treatments of these trade-offs in educational contexts, arguing that the appropriate fairness criterion depends on the specific harm being avoided (under-prediction that withholds support, or over-prediction that stigmatizes) and on whether the intervention triggered by the prediction is beneficial or potentially burdensome.[LE-LS-JO-004]

Bias Auditing as Standard Practice

The High-Leverage Opportunities report identified equity auditing of LA systems as a high-priority infrastructure gap: most institutions that deploy LA systems do not have standard processes for measuring disparate impact across demographic groups, even when they have the data to do so.[LE-LS-GL-003]

The Penn Center for Learning Analytics has developed frameworks for institutional equity audits of LA systems, covering: how to disaggregate accuracy metrics by demographic group; how to identify which model features are functioning as demographic proxies; and how to evaluate whether any observed disparities reflect genuine underlying differences in educational risk versus artifacts of historical inequity.[LE-LS-CO-004]

The Generalizable Learning Engineering Adoption Maturity Model includes equity auditing as a maturity indicator: institutions at high maturity conduct pre-deployment and post-deployment equity audits as standard procedure for all LA systems that affect student academic decisions.[LE-LS-AP-013]

Mitigation Strategies

Bias mitigation in learning analytics can be applied at three stages:

Pre-processing — modifying training data to address imbalances, re-weighting samples from underrepresented groups, or removing features that function as demographic proxies.

In-processing — modifying the model training algorithm to include fairness constraints alongside accuracy objectives, explicitly penalizing predictions that produce disparate impact across demographic groups.

Post-processing — adjusting the model's decision threshold differently for different demographic groups to equalize a chosen fairness metric.

Each approach involves trade-offs between accuracy and fairness, and between different conceptions of fairness. Learning engineers have an obligation to document these trade-offs explicitly, communicate them to institutional stakeholders, and revisit them as deployment contexts change.

Community Recommended Reading

Learning Engineering Enlightenment: Think Like an Engineer — Ellen Wagner (2024, New Learning Frontier)

ID: lecommons-rl-learning-engineering-enlightenment-think-like-an-engineer · type: reading list item · confidence: medium · expert-validated: false

Chapter 6: LE Toolkit — Learning Engineering Applies the Learning Sciences — Jim Goodell, Janet Kolodner, Aaron Kessler (2022)

Source: goodell-ch6-le-applies-learning-sciences · confidence: medium · expert-validated: false

Context enrichment applied 2026-04-17. See wrgr/lecommons for source data.

8.5.2 Fairness and Bias in Analytics

Sources of Bias in Training Data

Documented Cases: E-Proctoring and Disparate Impact

Fairness Metrics and Their Trade-offs

Bias Auditing as Standard Practice

Mitigation Strategies

Further Reading

Policy, Reports & Grey Literature

Key People

Organizations, Conferences & Journals

Community Recommended Reading