Drift detection and the cost of recalibration

Current assessment (2026-09-15)

Powered session 01KWY1C0XCSCAXYNX24MQH07QJ used 168 runs across six seeds (84 reused, 84 newly attached). Trigger minus fixed-cadence error was +0.034 persons, 90% CI [+0.002, +0.066], outside the ±0.05 equivalence margin. Seed-cluster correlation was 0.68, 95% CI [0.10, 0.90]. The trigger used 1.17 versus 2.0 calibrations on the drifting arm but missed all six states on one harmful-drift placement.

The old 0.93 correlation and accuracy-parity-at-quarter-budget headline are pilot results and cannot represent the powered evidence. Fewer calibrations at some accuracy cost is the supported trade-off. Later e-process work also requires the documented three-way split; a nominal false-alarm bound is not inherited by a score fitted on its own calibration data.

Next evidence needed: Use recalibration-trigger-from-drift for the updated statistical contract. In EXP-F2, log reference age, trigger value and independent error together, and compare policies chronologically before refitting.

Basis: Thesis Narrative, the September 10 synthesis, and the September 15 register review. See the review record for evidence sources and outstanding checks.

Earlier protocol and dated results

The methods and observations below retain their original scope. Read their readiness statements and proposed next steps as historical. The current assessment above records what has changed; frozen thresholds have not been retrospectively revised.

The problem, in plain words

Imagine you calibrate a bathroom scale while standing on a soft carpet. Later someone swaps the carpet for tile. The scale now reads wrong — not because your weight changed, but because its environment did. Wi-Fi crowd counting has the same disease: the model learns "this pattern of signal strength ≈ 5 people," but the pattern depends on where the walls, furniture, and reflective surfaces are. Rearrange the room and the model silently drifts. This is a textbook case of concept drift — the input–output relationship changing over time — which is a whole research field of its own (gama2014 ↗).

The naive fix is to recalibrate on a timer ("every week, re-measure with a known number of people"). That works, but it is expensive and wasteful: most rooms are stable most of the time, so you pay for recalibrations you did not need, and you can still be caught out between timer ticks. The smarter idea — an event-driven trigger — only recalibrates when a monitor says the model has actually broken. The hard part is building that monitor without ground truth: at runtime you do not know the real headcount, so you cannot directly measure your own error.

What we are trying to prove

• Hypothesis (falsifiable): the cross-modal disagreement xmodal — computable online with no ground truth — ranks time-periods by true occupancy error (high Spearman ρ) and stays below threshold on periods where the environment shifted but the model did not break. If it fires on those error-irrelevant shifts, it is a naive change-detector, not a drift-relevant trigger, and the idea fails.
• Second, sceptical hypothesis: acting on the trigger lowers count error versus a fixed cadence at the same recalibration budget. We expected this to be the weak claim, and it is — see results.
• What a null means: if xmodal did not out-track single-modality detectors, it would mean the BLE channel adds nothing as a drift reference and the fusion premise weakens. It does out-track them — the value of BLE here is not as a people-counter (it is weak at that too) but as a second, differently-drifting physical witness.

How the experiment works (plain method)

1. Build a controlled drift corpus (csi-layout-drift): a synthetic corridor is ray-traced under ordered furniture displacements, with a deliberate split — a losin arm where displacement genuinely grows count error, and a losperp arm where the signal amplitude shifts ~27 dB but the count error stays flat (environment change without model breakage). Co-registered BLE is solved from the same scene.
2. Compute four candidate drift monitors online: csi-ks (Wi-Fi self-monitoring), ble-ks (BLE channel shift), xmodal (the cross-modal disagreement), and an idealised oracle-counter ceiling.
3. Detectability test: does each monitor's value rank-correlate with the true occupancy error, and does it stay quiet on the error-irrelevant arm?
4. Policy test: simulate deployment — over the drifted states, compare "never recalibrate" / "fixed cadence" / "trigger[xmodal]" / "oracle-error" on both error and recalibration spend, at matched budget.

What we've found so far (honest, across campaigns)

Policy at matched budget (signal arm): fixed cadence 0.399 error / 8.0 estate-wide recals; trigger[xmodal] 0.407 / 2.0 recals. Even the perfect-knowledge trigger (0.421) does not beat fixed cadence on error — because recalibrating on schedule also fixes below-threshold drift the trigger deliberately ignores. So: detectability confirmed, accuracy-policy refuted, budget-parity is the real result.

How to read the figures

• fig_drift_trigger_detect — four scatter panels (one per monitor), statistic on x, true occupancy error on y, with the per-panel Spearman ρ in the title. A good monitor makes the cloud rise left-to-right; note xmodal's tight rise vs csi-ks's scatter. n = 18 cells/arm, so read the rank trend, not individual points.
• fig_drift_trigger_policy — the small-multiples bar chart of error and recals per policy; the honest reading is overlapping error bars between trigger and fixed, with the recal-count bar far lower for the trigger.
• fig_drift_trigger_curves — per-arm displacement→error curves; the seed spread is the story (seed-1 placements happen to be drift-robust; the big excursion is seed-0).

Key references

• gama2014 ↗ — the drift-detection framing this whole experiment inherits.
• jung2025_3f2e ↗ — real multi-zone CSI occupancy; grounds the drift-on-layout premise.
• zou2018 ↗, zou2017_9087 ↗ — device-free CSI counting baselines.
• demrozi2021 ↗ — BLE occupancy, the second modality's real-world basis.
• zhang2026_ccac ↗ — the reproducibility bar the SWE voice invokes.
• huang2025_060d ↗ — a public real-CSI anchor for the red-team's cross-check demand.