2026 BOY premodelling audit: hierarchical subscores, Number Line policy, and accuracy-speed

Generated: 2026-06-14 08:57:14Z

This is a dependency-light, aggregate-only audit. It does not publish raw student identifiers or person-level score files.

Executive readout

• Hierarchical subscores are justified as a modelling direction: standalone subtest evidence is uneven, so teacher-facing profiles should be shrunken/coherent with the global score rather than independent standalone IRT scores.
• The first hierarchical Stan challenger should be H1_global_plus_subtest_deviations: global numeracy plus reportable subtest deviations, not the current nuisance testlet u residuals as subscores.
• Number Line cutoff changes should be screened from raw coordinate-derived accuracy distributions first. This audit writes item-by-target ECDF/category-count tables for .80/.90, .85/.95, .90/.97, binary >=.95, and a 4-category .80/.90/.95 option.
• Accuracy-speed remains shadow/QC-first. Timed D/trailing-zero already encodes reach/speed pressure, so RT must not be allowed to double-count speed in achievement bands without validation.

Hierarchical subscore readiness

Key implication: several subtests are not ideal standalone reporting scores, especially where reliability is weak/moderate or item counts are small. That is an argument *for* hierarchical shrinkage, not against subscores.

Current-policy subtest relationships

Profile-deviation spread

Number Line cutoff policy premodelling

Interpretation rule: a policy can be *modelable* from cell counts but still not promotable. Promotion requires validation, risk-band movement, fairness/subgroup checks, and interpretability. Current .85/.95 remains the benchmark.

Accuracy-speed / RT readiness

J2b-style rapid-row descriptive check

Person-level speed/reach correlations with current-policy scores

Reach/trailing correlations are partly mechanical under timed D/trailing-zero scoring. This is exactly why RT/tau should initially remain a shadow response-process layer rather than a direct achievement-band adjustment.

Recommended model ladders

Hierarchical global/subscore ladder

Number Line policy ladder

Accuracy-speed ladder

Decision gates / next actions

Written aggregate artifacts

• tables/premodeling/2026_boy_hierarchical_subscore_readiness.csv
• tables/premodeling/2026_boy_subtest_score_correlations.csv
• tables/premodeling/2026_boy_subtest_composite_correlations.csv
• tables/premodeling/2026_boy_subtest_profile_deviation_summary.csv
• tables/premodeling/2026_boy_nl_accuracy_distribution_by_item.csv
• tables/premodeling/2026_boy_nl_policy_item_cell_counts.csv
• tables/premodeling/2026_boy_nl_policy_overall_summary.csv
• tables/premodeling/2026_boy_rt_readiness_by_subtest.csv
• tables/premodeling/2026_boy_j2b_style_rapid_row_audit.csv
• tables/premodeling/2026_boy_speed_accuracy_correlations.csv
• model-ladder and decision-gate CSVs in the same folder
• TAM cutoff screen runner: analysis/modeling/v2_response_process_program/77_2026_boy_premodel_tam_cutoff_screens.R (requires TAM; intended for cisbox/AWS)

2026 BOY next-model specification notes

Status: pre-fit design note generated after aggregate premodelling audit. Do not treat as an operational scoring decision.

1. Hierarchical global + subscore model

Goal

Produce a coherent global score and teacher-facing subtest subscores, avoiding unrelated standalone subtest IRT scales.

First challenger: H1_global_plus_subtest_deviations

For student p and subtest/domain s:

g_p ~ broad numeracy level
z_ps ~ standard normal residual profile component
delta_ps = sigma_delta_s * z_ps, centered across subtests within student
theta_ps = g_p + delta_ps

Binary/timed or untimed non-NL item j in subtest s[j]:

y_pj ~ Bernoulli_logit(theta_p,s[j] - b_j)

Ordinal Number Line item j under a PCM-style policy:

eta_1 = 0
eta_k = eta_{k-1} + theta_p,s[j] - (b_j + step_j,k-1)
y_pj ~ categorical_logit(eta)

Identification/regularisation:

• Center each student's delta_ps across subtests so g_p remains the broad level.
• Center item difficulties within year/model as in current Stan practice.
• Start without a separate nuisance testlet u for every subtest; otherwise the reportable subtest deviation and nuisance residual compete for the same signal.
• Keep posterior intervals for subtest deviations; do not report profile differences smaller than measurement uncertainty.

Primary post-fit checks:

1. HMC: 0 divergences, no max-treedepth hits, Rhat/ESS acceptable for g, theta_ps, sigma_delta_s, item parameters. 2. Global movement vs hard-filtered H0: Spearman, median/p95 percentile shift, <15 and 15-35 risk-band movement. 3. Subscore quality: posterior SD by subtest, shrinkage size, profile-deviation stability. 4. Teacher-facing coherence: subscore intervals and relative-strength labels agree with observed subtest evidence without overclaiming. 5. Subgroup/admin movement: no adverse subgroup artefacts.

2. Year 1 BNL residual surgical sensitivity

Keep BNL0-100 items in the global/hierarchical score but do not give BNL an extra nuisance residual variance if the current sigma_u[BNL0-100] remains weak.

Data-side option:

active_testlet_idx[BNL0-100] = 0
active_testlet_idx[other_subtests] = 1..K_active

Likelihood option:

resid = 0 if active_testlet_idx == 0
resid = sigma_u[k] * u_z[p,k] otherwise
theta_eff = theta + resid

This tests whether the issue is the BNL residual component, not the BNL items themselves.

3. Number Line policy ladder

Premodelling audit outputs:

• tables/premodeling/2026_boy_nl_accuracy_distribution_by_item.csv
• tables/premodeling/2026_boy_nl_policy_item_cell_counts.csv
• tables/premodeling/2026_boy_nl_policy_overall_summary.csv

Frequentist screens before Stan:

nl_80_90_relaxed_3cat
nl_85_95_current_3cat
nl_90_97_strict_3cat
nl_binary_95
nl_80_90_95_4cat

Promotion burden:

• current .85/.95 remains the reference;
• challenger must have stable cells/thresholds;
• challenger must improve or match validation/risk classification;
• challenger must not cause unacceptable subgroup or risk-band movement;
• continuous NL requires Stan or another mixed continuous-response framework, not TAM/mirt alone.

Continuous challenger sketch:

accuracy = 1 - absolute_error / scale_range
accuracy_squeezed = clamp/Smithson-Verkuilen transform into (0,1)
logit(mu_pj) = alpha_j + theta_p,s[j]
accuracy_pj ~ Beta(mu_pj * phi_j, (1 - mu_pj) * phi_j)

Optional signed-error diagnostic, not first scoring model:

signed_error_scaled_pj ~ Normal(target_bias_j + method_bias_family + ability_slope_j * theta, sigma_j)

4. Accuracy-speed joint modelling ladder

Operational posture: RT is shadow/QC first. Timed D/trailing-zero already encodes reach/time-pressure, so response time can double-count speed if added naively.

Initial 2026 BOY data rule:

• Achievement accuracy model may continue to use D/trailing-zero for timed non-NL.
• RT likelihood should use observed/reached item rows only.
• Trailing unreached rows contribute to D accuracy/reach context, not item-level logRT.
• STPM remains shadow/non-math and is excluded from math achievement.
• Number Line RT is context-only initially.

Candidate shadow model:

y_pj ~ Bernoulli_logit(theta_p - b_j + gamma_family * rapid_pj)
logRT_pj ~ LogNormal(beta0 + beta_j - tau_p,family[j], sigma_rt_family)

Hierarchical pace extension:

tau_p,f = tau_overall_p + tau_residual_p,f

Pre-fit checks already written:

• tables/premodeling/2026_boy_rt_readiness_by_subtest.csv
• tables/premodeling/2026_boy_j2b_style_rapid_row_audit.csv
• tables/premodeling/2026_boy_speed_accuracy_correlations.csv

Do not use RT/tau to alter risk bands unless later evidence shows robust validation gain, no subgroup/admin artefact, and added information beyond D/reach.

2026 BOY operational accuracy + Number Line candidate — modelled job review

Review timestamp: 2026-06-14 UTC

Compute / sync status

All AWS model jobs are complete. There are no active EC2 instances matching the 2026 BOY operational Number Line model tags, no active cisbox rsync sessions, and the local sensitivity monitor was stopped after all six sensitivity .done markers were present.

Final outstanding run (year1_no_BNL0_100) is synced, checksum-verified, recovered from the known no-NL post-processing failure, and its EC2 instance was terminated.

Reviewed jobs

The review covers 10 Stan jobs:

1. Foundation inclusive baseline. 2. Year 1 inclusive baseline. 3. Foundation hard-item-filtered baseline. 4. Year 1 hard-item-filtered baseline. 5. Foundation sensitivity: no DMT10_2026. 6. Foundation sensitivity: no MQ1-20 and no DMT10_2026. 7. Foundation sensitivity: no BNL0-20. 8. Year 1 sensitivity: no MC0-100. 9. Year 1 sensitivity: no BNL0-100. 10. Year 1 sensitivity: core model with no MC and no NL.

Source output base:

/data/numeracy-screening-models/irt/2026_boy_operational_accuracy_nl_candidate

Local review artifacts:

outputs/runs/irt-2026-boy-subtest-audit/latest/reports/model_review/stan_review_summary.md
outputs/runs/irt-2026-boy-subtest-audit/latest/tables/model_review/stan_job_diagnostic_summary.csv
outputs/runs/irt-2026-boy-subtest-audit/latest/tables/model_review/stan_score_movement_comparisons.csv
outputs/runs/irt-2026-boy-subtest-audit/latest/tables/model_review/stan_testlet_sigma_summary_long.csv
outputs/runs/irt-2026-boy-subtest-audit/latest/tables/model_review/stan_item_difficulty_extreme_or_diagnostic_flags.csv
outputs/runs/irt-2026-boy-subtest-audit/latest/tables/model_review/stan_u_residual_diagnostic_summary.csv

Completion and sampler diagnostics

All 10 jobs have successful MCMC sampling evidence:

• 0 divergences in every job.
• 0 max-treedepth hits in every job.
• Minimum EBFMI across jobs: 0.568 (year1_core_no_MC_no_NL), acceptable.
• Student theta summaries are clean: max theta Rhat <= 1.006 and theta ESS_bulk is comfortably high in all jobs.
• Item difficulty summaries are clean by diagnostics: no item difficulty has Rhat > 1.01 or ESS_bulk < 400.

Three no-NL-style jobs exited with Stan runner exitcode 1 because of the known post-processing bug for empty/missing NL lookup files, not because of sampler failure:

• foundation_no_BNL0_20
• year1_no_BNL0_100
• year1_core_no_MC_no_NL

All three were recovered from QC summaries and now have final score, item, testlet, and fit-readout files.

Job-level diagnostic table

Main diagnostic finding

The global Year 1 baseline is usable from a sampler perspective, but the BNL0-100 testlet residual scale is weakly identified:

• Inclusive Year 1 BNL0-100 sigma: Rhat ~1.023, ESS_bulk ~109.
• Hard-filtered Year 1 BNL0-100 sigma: Rhat ~1.068, ESS_bulk ~78.

This issue is local to the BNL0-100 residual/testlet component. It does not show up as divergent transitions, treedepth failures, poor theta mixing, or item-difficulty non-convergence. It does show up in the latent residuals for the same component: in the hard-filtered Year 1 run, u[,5] corresponds to BNL0-100, and 1193/1221 residual terms had Rhat > 1.01, with max Rhat ~1.026. The likely interpretation is that the residual BNL0-100 testlet variance is near a boundary/small value and is hard for the sampler to estimate, while the BNL0-100 items themselves carry substantial global-theta information.

Auxiliary u residual diagnostic

No other job/testlet had u residual terms with Rhat > 1.01 or ESS_bulk < 400. This reinforces that the caveat is localized to Year 1 BNL0-100 dependence modelling, not to the global theta score or item difficulty estimates.

Hard-filtered vs inclusive baseline

The hard-item filter removes the 70 predeclared no-information items and has negligible impact on student ranking/risk classification.

Conclusion: hard-item-filtered should be the working operational baseline. The inclusive runs are useful historical evidence but should not be promoted over the filtered version.

Sensitivity findings vs hard-filtered baseline

Foundation

Foundation interpretation:

• The hard-filtered Foundation baseline is sampler-clean.
• BNL0-20 is important to the global score; dropping it causes large risk-band movement.
• DMT10_2026 also matters; despite being untimed, it contributes meaningfully to the Foundation global trait.
• Foundation supports retaining the full hard-filtered operational accuracy + NL candidate, subject to external validation and reporting review.

Year 1

Year 1 interpretation:

• MC0-100 is not a major concern; the no-MC sensitivity remains close to the hard-filtered baseline.
• BNL0-100 is the key decision point. It is highly influential for Year 1 risk classification and precision.
• The weak BNL0-100 sigma diagnostic should not be read as evidence to drop BNL. The no-BNL sensitivity shows the opposite: dropping it materially changes the construct coverage and low-achievement identification.
• The most defensible reading is: retain BNL0-100 as a strong candidate, but resolve/report the localized testlet-sigma issue before final operational promotion.

Frequentist model-rung context

Frequentist pre-screening remains consistent with the Stan review:

• TAM 1D PCM was the only clean frequentist baseline across both years.
• Foundation reliability ~0.914.
• Year 1 reliability ~0.952.
• mirt correlated subtest factors failed due quadrature burden (Greater than 20000 quadrature points).
• mirt flexible/bifactor screens did not provide stable enough evidence to justify multidimensional Stan challengers.
• Refined frequentist sensitivities also flagged Foundation no-BNL and Year 1 no-BNL/no-MC variants as the main score-movement cases.

Therefore, the current Stan evidence should be interpreted within a 1D+testlet operational-candidate frame, not as support for immediate multidimensional/bifactor escalation.

Recommendations

1. Promote the hard-item-filtered model frame as the working baseline for final reporting comparisons. The hard filter removes no-information items with near-zero impact on student scores/risk bands.

2. Foundation: keep BNL0-20 and DMT10_2026 in the operational candidate. Both materially affect risk identification; the Foundation hard-filtered Stan run is diagnostically clean.

3. Year 1: do not drop BNL0-100 based on the sigma diagnostic alone. Removing it causes major movement and loss of precision. Treat the issue as a localized residual-scale estimation problem, not a failed global score.

4. Run or design one surgical Year 1 sensitivity if final promotion requires clearing the sigma caveat: keep BNL0-100 items in the global score but omit/fix the BNL0-100 testlet residual scale. This directly tests whether the weak sigma parameter is harmless. This is more informative than a no-BNL model, which changes both construct coverage and precision.

5. Complete external validation and subgroup movement checks before final operational lock-in. Compare hard-filtered baseline and key sensitivities against PAT/teacher outcomes and demographic/school subgroup stability, with priority on the <15th and 15th–35th percentile bands.

6. Update the audit/report package. Add sections for item eligibility, hard-filtered vs inclusive comparison, frequentist model rungs, Stan sensitivity results, and the Year 1 BNL0-100 decision caveat.

Proposed immediate next steps

1. Add the generated model-review tables to the unified audit HTML/report. 2. Build a final score-movement table with student-level risk-band transitions for the hard baseline vs the three most important sensitivity contrasts:

• Foundation no BNL0-20.
• Year 1 no BNL0-100.
• Year 1 no MC0-100.

3. Run outcome validation comparisons for the hard baseline and sensitivity variants. 4. Review Year 1 BNL0-100 item-level diagnostics:

• coordinate audit;
• empirical item curves by theta bin;
• target-value distribution;
• ordinal threshold behavior;
• response category sparsity.

5. Decide whether to run the surgical Year 1 BNL-included/no-BNL-testlet-residual Stan sensitivity. 6. Draft the operational recommendation:

• hard-filtered baseline as default candidate;
• Foundation BNL retained;
• Year 1 BNL retained as candidate pending sigma caveat resolution and validation;
• MC0-100 not a major exclusion pressure.