Comparable Grading From Observational Data

Many-Facet Modeling with Soft Anchors

Håkon Otneim

NHH, 25 September 2026

My job is generally nice, but…

A procedure for grading in quantitative courses

  • We create an exam having a certain number of questions.
  • Each candidate’s answers are assigned to a grader.
  • Each question is scored from 0 to 10, giving a total percentage score from 0–100.
  • A grade on the A–F scale is assigned from cut-off points.

Two issues return every single time:

  • Should we adjust grading for question difficulty? If so, how?
  • Should we adjust grading for varying grader severity? If so, how?

The question

Can we recover a stable way to estimate student ability after correcting for difficulty and assessor severity, that is comparable across cohorts?

This is an interesting research question in its own right, but here presented as an example of doing research on my own teaching.

A measurement problem with a long history

Separating ability from difficulty and severity is exactly what Item Response Theory has studied for decades. The general problem is to measure latent traits: How well do individuals in a group like chocolate?, or in this case: To which degree do the students in a cohort achieve the learning outcomes in my course?.

Two subfields are directly relevant:

  • Linking / test equating — comparing results across different tests, usually through common (“anchor”) items.
  • Multi-facet Rasch models — modelling assessor severity, drift over time, and their interactions.

My twist: the data are fully observational. There is no cross-grading, only partial assessor overlap and repeated question types. A common scale has to be recovered from those two things alone.

My data: nine cohorts of MET4

  • 9 cohorts of MET4, a mandatory applied-statistics course.
  • 11–13 free-response questions per cohort, each scored 0–10 by a single assessor.
  • Scores recoded to an ordered 0–3 scale: \[\{0,1,2\}\to 0,\quad \{3,4,5\}\to 1,\quad \{6,7,8\}\to 2,\quad \{9,10\}\to 3\]
  • 8 assessors in total (2–4 per cohort); 13 recurring (“linked”) question types.
  • 2,075 candidates (≈230 per cohort); 25,555 item responses.
  • Cohort mean scores range from 1.67 to 2.17 on the 0–3 scale.

The model

  • Ordered score \(Y_{ijca}\in\{0,1,2,3\}\) for candidate \(i\), item \(j\), cohort \(c\), assessor \(a\).
  • Cumulative-logit (proportional-odds) model with shared thresholds \(\tau_1<\tau_2<\tau_3\): \[P(Y_{ijca}\le k\mid\eta_{ijca})=\operatorname{logit}^{-1}(\tau_k-\eta_{ijca}),\qquad k=1,2,3.\]
  • The linear predictor splits the score into facets: \[\eta_{ijca}=\underbrace{\mu_c}_{\text{cohort}}+\underbrace{u_i}_{\text{ability}}-\underbrace{\beta_j}_{\text{item}}-\underbrace{\alpha_a}_{\text{assessor}}-\underbrace{\delta_{ac}}_{\text{drift}}-\underbrace{\kappa_{a\ell(j)}}_{\text{assessor}\times\text{content}}-\underbrace{\gamma_{c\ell(j)}}_{\text{item drift}}\]
  • Student ability is \(\theta_i=\mu_{c(i)}+u_i\), with \(u_i\sim N(0,1)\) anchoring the scale.

Soft anchors — the key idea

The hard part is the recurring question types (the “linked” items).

  • Hard linking assumes a linked type is equally difficult every year. Posterior-predictive checks say this leaves far too much misfit — up to 42 of 78 assessor–content cells flagged.
  • Soft linking (\(\gamma_{c\ell}\)): linked items share a baseline difficulty but may drift cohort by cohort, with the drift regularised toward zero unless the data insist.

The model is built up sequentially, each step driven by residual checks and validated by leave-one-out cross-validation. Out-of-sample predictive checks strongly prefer the most complex model, so all interaction effects contribute to the model.

Item difficulty on a common scale

  • Estimated difficulties span roughly 6 logit units.
  • Open circles are a linked type’s shared baseline; the ticks are its cohort-specific difficulties.
  • Several linked types drift noticeably — the same task type was not equally hard every year. That is exactly what soft anchors absorb.

Assessor severity — modest in aggregate

  • Baseline severities span only about a third of a logit unit.
  • The 95% intervals are wide and mostly overlap zero.
  • No assessor is practically flagged at the aggregate level.

Keep this in mind for later.

Raw cohort means are confounded

  • Left: raw cohort mean vs estimated mean ability — a loose relationship.
  • Right: once we account for each cohort’s item easiness and assessor leniency, the alignment is tight.
  • Raw totals mix ability with the exam conditions a cohort happened to face.

A counterfactual experiment

  • Hold the historical A–F proportions in each cohort fixed, but re-rank students by estimated ability \(\widehat\theta_i\). How much would grades move?
Cohort Down (%) Same (%) Up (%)
C01 50.0 50.0 0.0
C02 0.0 56.5 43.5
C03 7.6 79.2 13.2
C04 17.3 77.2 5.5
C05 41.1 57.1 1.7
C06 17.1 77.9 5.1
C07 5.0 80.5 14.5
C08 7.5 69.0 23.5
C09 20.0 71.1 8.9

Some cohorts barely move; others move a lot — C01 and C05 (easy sets, lenient marking) send ~40–50% down, C02 and C08 send 24–44% up.

What drives the movement?

An ordered-logit model of the direction of movement (Down < Same < Up), with two standardised student-level predictors:

Term Estimate Std. Error OR OR 95% CI \(p\)
Ability 0.258 0.053 1.294 [1.17, 1.44] <0.001
Assessor severity deviation 0.473 0.051 1.605 [1.45, 1.77] <0.001

The assessor severity deviation is the largest effect — larger than ability itself. (Item difficulty is common to everyone in a cohort, so it is absorbed by the cohort intercept.)

Modest in aggregate, decisive individually

No single assessor’s estimated severity is particularly large, yet who graded you is the strongest predictor of whether your grade would move on the (estimated) level playing field.

  • Aggregate assessor effects are small and none is flagged…
  • …but within a cohort, being graded by a stricter-than-average assessor carries a systematic disadvantage that raw totals hide.
  • Small effects on the latent scale → potentially large consequences for individual grades.

Takeaways

  • With a many-facet model and soft anchors, we can put students from different cohorts on a comparable ability scale — from purely observational grading data.
  • Doing so exposes how much of a raw grade reflects the exam conditions a student happened to face.
  • For our own teaching: aggregate fairness checks can look reassuring while individual grades are still moved by who did the scoring.
  • Interesting parallel to the current debate on national school tests where the government proposes to remove the anchors in order to stop ranking schools and neighbourhoods.
    • Can researchers still recover a common scale via soft anchors?