Synthetic pretraining under a scarce real-data budget
Current assessment (2026-09-15)
The September 4 transfer comparison ran on occlusion-only and overlay corpora: gains over real-only were at most +0.006 R² at k=5/10/20 with overlapping reported intervals; zero-shot R² was −3.4 and −16.6. This is a negative within that setup, not a general equivalence result. The realistic-impairment, scarce-distinct-capture extension reopened September 9; the 939-row feature cache lacks the lineage needed to establish that budget.
The card's old 'transfer never executed' heading is obsolete. Overlapping arm intervals alone do not test the paired gain or establish equivalence. The overlay's monotone count response partly checks an imposed law, so it must not be described as independently derived physics. Scene diversity, measured receiver assumptions and the real-data budget remain separate factors.
Next evidence needed: Restore real capture IDs and group held-out evaluation by capture and environment. Compare clean, shipped-prior and measured-profile training under the same distinct-capture budget, including real-only augmentation. Record paired gain intervals and the declared practical margin.
Basis: Thesis Narrative, the September 10 synthesis, and the September 15 register review. See the review record for evidence sources and outstanding checks.
Earlier protocol and dated results
The methods and observations below retain their original scope. Read their readiness statements and proposed next steps as historical. The current assessment above records what has changed; frozen thresholds have not been retrospectively revised.
The problem, in plain words
Suppose you want to teach a computer to count people in a room from nothing but the Wi-Fi signal bouncing around it. To learn well it needs lots of labelled examples — "this wobble in the signal means 3 people, this one means 7" — across many different rooms, because the wobble depends heavily on where the walls, furniture and metal cabinets are. Collecting that real, labelled data is slow and expensive: someone has to stand in the room with a clipboard counting heads.
The tempting shortcut is a simulator: a physics engine that ray-traces radio waves through a 3-D model of a building with virtual people walking through it, spitting out synthetic Wi-Fi channel measurements (CSI) for free, in any quantity, perfectly labelled. If we pretrain on a mountain of that synthetic data and then fine-tune on just a few real measurements, maybe we get a great counter for almost no real-world effort. That is the classic sim-to-real promise, and it is exactly the promise domain-adaptation research keeps testing across wireless sensing (chen2023_5cbd ↗; wang2026_2758 ↗).
The catch — the reason this needs a hard test rather than a hopeful assumption — is the reality gap. A simulator is only as honest as its physics and its 3-D scenes. If its rooms are cleaner, more geometric, and more distinct than real rooms, a model trained on it learns to lean on cues that do not exist in the messy real world. It will look brilliant in simulation and fall over on the first real Pi. So the only trustworthy way to judge synthetic pretraining is to score it against real, public measurements — here OPERAnet (bocus2022_ce7f ↗) and WiMANS (huang2025_060d ↗).
What we are trying to prove
- Primary hypothesis (falsifiable): a crowd counter pretrained on synthetic-impaired CSI and fine-tuned on a small real budget (k = 5/10/20 examples per count) achieves lower held-out real count error than a real-only model trained on the same small budget — measured leave-one-environment-out on WiMANS. If it does not beat real-only, the ray-tracer's occupancy structure is not real-representative and the sim programme's licensing premise weakens.
- The honesty gate (non-negotiable): always report zero-shot (sim-only → real, no fine-tuning) and the clean-vs-impaired ablation, so any gain is attributed to genuine transfer rather than to the Doppler representation alone. That representation already reaches LOEO R² ≈ 0.47 on real WiMANS with no sim at all — so a naive "look, it works" is not evidence of transfer.
- What a null means: if synthetic pretraining does not help at small k, the finding is that our ray-traced occupancy structure differs from real occupancy structure in a way fine-tuning cannot repair — which is itself a publishable, defect-localising result (it points the fix at scene diversity, materials, or the body-scatterer prior, not at "more data").
How the experiment works (plain method)
The card spans two coupled campaigns — one that makes and audits the synthetic data, and one that tests the transfer:
- Fingerprint / calibration probe (
c-operanet-sim2real-fingerprint): simulate the same occupancy signal across several distinct real-building floor geometries (ResPlan floors), then ask three real-anchored questions. (a) Can a classifier tell the floors apart from synthetic CSI — and is that separability comparable to the real OPERAnet room-ID of 0.83, or does the sim over-code it? (b) Does a synthetic occupancy model trained on one floor still work on another (cross-floor retention), the way the real one keeps 0.73 across rooms? (c) Which knob — hardware-noise impairment or scene diversity — moves the synthetic fingerprint toward reality? This is framed as a calibration probe, not a hypothesis test: a match validates the sim as a sim-to-real substrate; a mismatch tells us exactly what to fix. - Transfer test (
c-sim2real-count-transfer): generate a class-balanced synthetic corpus (JuPedSim crowd → Sionna ray-traced CSI, 0–5 agents × 4 seeds, with a commodity-NIC impairment layer so it is not unrealistically clean), turn each run into a Doppler spectrogram, and then compare, on held-out real WiMANS: real-only(k) vs synthetic-pretrain-then-k-shot vs zero-shot vs real-full. The claim lands only if synthetic-pretrain beats real-only at small k, after the honesty gate rules out the representation-only explanation.
What we've found so far (honest, across campaigns)
The transfer test never executed (c-sim2real-count-transfer)
Both sessions of this campaign sealed with zero runs launched, zero CPU-hours, zero tokens spent on compute — blocked at the mandated preflight gate, correctly and cheaply, before orphaning an unsupervisable sweep:
| Session | Outcome | Why |
|---|---|---|
01KXBTWW…RD4M5 |
BLOCKED, 0 runs | Transfer reduction unwritten (blocks criteria 2 & 3 structurally); impairment channel undeclared in the brief; no cheap schema check exists for the CSI simulator family. |
01KXBWYQ…JP9Z8 |
PARTIAL, 0 runs | Unblocked the dependency chain and validated the corpus recipe on 1 of 24 probe cells (both csi.hdf5 + csi_impaired.hdf5 emit), then exhausted supervisor budget before the full corpus. |
So: the impairment contract is verified on a single cell, the reduction harness is authored and self-tests on planted synthetic data, and the working grid is recorded for a hand-off. None of the three success criteria (corpus, transfer effect, honesty gate) is resolved, and the central hypothesis has never touched real data. This is a wiring gap, not a negative result — but it must not be reported as anything more than "designed and unblocked, not run."
The fingerprint probe ran and warns us (c-operanet-sim2real-fingerprint)
Recorded from a local deterministic fan-out (31 static + coupled crowd runs), in the campaign note, without a sealed S3 session — treat as directional:
| Probe | Synthetic | Real target | Verdict |
|---|---|---|---|
| Environment fingerprint (2-floor room-ID, chance 0.5) | 0.98 | 0.83 | reproduced, but over-codes |
| Environment fingerprint (4-floor, chance 0.25) | 0.95 | — | strong |
| Occupancy — static snapshot (within-floor, chance 0.25) | 0.13 | — | at chance (static config too weak) |
| Occupancy — coupled temporal (within-floor) | 0.60 ± 0.39 | — | recovered by motion (≈2.4× chance) |
| Cross-floor occupancy retention (graded links both floors) | 0.57 | 0.73 | transfers worse than real |
Three things the probe actually taught us, each with a real anchor:
- The sim over-codes the environment. Synthetic floor-ID (0.98) beats the real room-ID (0.83): the ray-tracer's rooms are too distinguishable because they lack real nuisance (hardware drift, furniture change, temporal variation). The direction matches reality (environment is the hard generalisation axis) but the magnitude is optimistic — a model pretrained on this would over-fit the environment cue.
- Occupancy needs motion, and even then transfers worse than real. A static placement snapshot carries no count signal (0.13, at chance); the coupled JuPedSim→Sionna temporal chain recovers it (0.60), mirroring the real OPERAnet lesson that activity/occupancy lives in temporal dynamics. But cross-floor retention is 0.57 vs the real 0.73 — a steeper drop, exactly what over-coding predicts.
- Hardware noise is the wrong knob; scene diversity is the right one. Adding a commodity-NIC impairment layer (SNR 20 dB, CFO, STO, IQ, phase noise) left the fingerprint at ~1.0 and degraded occupancy retention (0.71 → 0.62) — moving away from real. Randomising furniture/material per run dropped synthetic floor-ID from ~1.0 toward 0.67. The over-coding is structural geometry, not a noise floor. This reframes the
exp-csi-calibrationrefit as a scene-diversity fit, not a noise-model fit — a concrete, useful outcome.
Net: the one arm that ran does not confirm the transfer dream — it produces a red flag for it (the sim is too clean/too distinct), plus a specific fix. That is a healthy, honest state for a sim-to-real card to be in.
Phase 2a corpus — generated 2026-09-04
Sweep 01M1P7DRCTFJN96E7Y8Q3G4YGQ, exp-csi-crowd, run locally on the native Metal lane at about
23 s per cell, 12:48–12:57 UTC. Grid n_agents 0-5 × seed 0-3 from
monad_knowledge/sim/exp_csi_crowd/sweeps/sim2real-corpus.grid.yaml; base as in the campaign
brief (5 GHz, 52 subcarriers, doppler.mode: cfr, IP-101 impairment profile at 30 dB SNR, 1 tx,
5 rx, floor resplan-12439-floor-0).
| Criterion | Result |
|---|---|
| C1 artefact contract | 20 of 24 cells pass the gate; every passing cell has csi.hdf5, csi_impaired.hdf5 and links.parquet; no NaN or Inf. The four n_agents = 0 cells fail: the walk stage writes no trajectory for zero agents, so nothing drives the coupled solve. The empty class belongs to exp-csi-static. |
| C2 occupancy signal survives impairment | Per-seed exact Kendall tau of log10 temporal amplitude variance over n_agents 1–5, one-sided increasing, declared floor tau ≥ 0.8 (the smallest reachable at p < 0.05 on five levels). Clean channel: 4 of 4 seeds monotone (tau 1.000, p 0.008). Impaired channel: 3 of 4 (tau 0.800, p 0.042); seed 3 at tau 0.600, p 0.117. |
| C3 coverage | 20 of 24 cells; the empty class is absent, so the corpus is class-balanced over 1–5 only. |
What the trend is made of. The clean channel is static (temporal CV about 1e-5) in every cell where no agent crosses a link: 3 of 4 seeds at one agent, 2 of 4 at two. Once any agent crosses, the CV jumps to 1.5–2.8 % and then rises only mildly from three to five agents (per seed 1.5→2.8, 2.1→2.4, 2.1→2.7, 1.7→1.7 %). The impaired channel sits at 3.4 % CV where the clean one is static (pure impairment noise) and at 3.8–4.4 % where it is not. The corpus therefore carries a discrete occlusion-event signal that saturates by three agents, the same mechanism c-csi-drift-real-floor found on the library floor, not a graded count. The runner's own per-run variance_monotonicity reads 0.0 in every cell because it is a within-run statistic over occluder levels, and RunSpec.metrics is empty by design, so sim_reduction_run cannot see this corpus; the check was run with notebooks/campaigns/python/csi_sim2real_corpus_check.py.
Reduction outputs: s3://monad-knowledge/sweeps/01M1P7DRCTFJN96E7Y8Q3G4YGQ/reduction/corpus_check.{json,csv,png}.
What this means for Phase 2b. The transfer test can run on this corpus as pre-registered, but a graded k-shot gain is unlikely from a corpus whose count signal saturates at three. The base config carries no scene_overrides.occupancy_diffuse (IP-118), the overlay the drift campaign named as the only channel through which deterministic ray tracing carries occupancy on these floors; a regenerated corpus with it on is the design change that would give the transfer test something graded to transfer.
Phase 2a regenerated with the IP-118 overlay — 2026-09-04
Decision after the occlusion-only result: keep sweep 01M1P7DRCTFJN96E7Y8Q3G4YGQ as the baseline and regenerate with
scene_overrides.occupancy_diffuse ON, values as c-cac-drift ran them (k0_db 8.0,
k_slope_db_per_person 2.0, r_max 2, coherence 10 packets × 4 subcarriers, tensor layer), grid
n_agents 1–5 × seeds 0–3, everything else byte-identical
(sim2real-corpus-diffuse.{base,grid}.yaml). Sweep 01M1P939H07989R73Z2X1DS8RC, 20 of 20 cells pass, 15:18–15:27 UTC.
| Criterion | Occlusion-only (01M1P7DRCTFJN96E7Y8Q3G4YGQ) |
Overlay (01M1P939H07989R73Z2X1DS8RC) |
|---|---|---|
| C1 contract | 20 of 24 pass, no NaN/Inf | 20 of 20 pass, no NaN/Inf |
| C2 clean channel monotone (exact tau, per seed) | 4 of 4 (tau 1.0, p 0.008) | 4 of 4 (tau 1.0, p 0.008) |
| C2 impaired channel monotone | 3 of 4 (tau 0.8, p 0.042) | 4 of 4 (tau 1.0, p 0.008) |
| Clean temporal CV, n_agents 1 → 5 (median over seeds) | 9e-6 → 2.7 % (a step at the first crossing) | 0.371 → 0.564 (a line) |
| Impaired minus clean, log10 variance, median | 0.76 (impairment noise dominates) | 0.018 (overlay fading dominates) |
Read the overlay result for what it is. The straight line and the near-zero seed spread (CV
0.371 on all four seeds at one agent) are the IP-119 law K_dB(N) = k0 − k_slope·N doing exactly
what it was written to do: diffuse power is a deterministic fraction of each link's ray-traced
specular power, so the count-to-variance response is designed in and the geometry adds almost
nothing. Criterion 2 passing here is a construction check, not evidence that people bend the
channel this way; the slope is a literature prior (IP-119) and c-csi-impairment-fit has not run.
What the regenerated corpus buys Phase 2b is a graded signal to attempt transfer with; whether that
designed slope transfers to WiMANS is the question Phase 2b now asks, and the occlusion-only corpus
is its ablation. Reduction outputs: s3://monad-knowledge/sweeps/01M1P939H07989R73Z2X1DS8RC/reduction/corpus_check.{json,csv,png}.
Phase 2b — the transfer test, run 2026-09-04 on both corpora
csi_sim2real_count_transfer.py --band 5 --per-cell 60 (n_win 3, 5 reps, 20 epochs), WiMANS 5 GHz
leave-one-environment-out, 20 synthetic runs → 60 spectrograms against 3240 real. Held-out real R²,
mean [95 % CI over 5 reps]:
| k real instances per count | real_only | sim_pretrain_ft, occlusion-only corpus | sim_pretrain_ft, IP-118 overlay corpus |
|---|---|---|---|
| 5 | 0.522 [0.503, 0.533] | 0.528 [0.519, 0.538] | 0.513 [0.491, 0.535] |
| 10 | 0.535 [0.524, 0.552] | 0.536 [0.532, 0.542] | 0.530 [0.520, 0.537] |
| 20 | 0.547 [0.534, 0.559] | 0.548 [0.538, 0.556] | 0.549 [0.544, 0.553] |
| real_full (ceiling) | 0.548 [0.514, 0.574] | ||
| zero_shot (sim only) | −3.42 [−5.38, −1.92] | −16.64 [−28.78, −2.75] |
Verdict: null, replicated. Synthetic-impaired pretraining plus k-shot fine-tuning moves held-out
R² by at most +0.006 over real-only at any k; every interval overlaps. The overlay corpus, built to
carry a graded count, is no better at k = 20 and slightly worse at k = 5, and its zero-shot floor is
five times deeper: the designed IP-119 law is not what WiMANS looks like. This is the third null on
this claim (2026-08-03: ≤ +0.007 R², 8 seeds, 2.6× corpus) and the first on sealed first-party runs.
The hybrid-fusion chain's link 5 moves to documented-negative. Caveat: 60 synthetic spectrograms is
small by construction of the 20-cell grid; the 2026-08-03 enlargement did not move the result.
Results: s3://monad-knowledge/sweeps/01M1P7DRCTFJN96E7Y8Q3G4YGQ/reduction/phase2b_transfer.json and
s3://monad-knowledge/sweeps/01M1P939H07989R73Z2X1DS8RC/reduction/phase2b_transfer.json.
How to read the figures
These figures are named and described in the c-operanet-sim2real-fingerprint campaign note; they are not attached to a sealed session, so read them as the operator's recorded outputs, not session-audited artefacts.
operanet_sim2real.png— two bars: (a) environment-ID accuracy, real OPERAnet room-ID (0.83) vs synthetic floor-ID, with the chance line; the synthetic bar rising above the real one is the "over-codes" story. (b) synthetic occupancy sitting at the chance line in the static config — the null that motivated the coupled re-run.operanet_crossfloor_graded.png— within-floor occupancy (0.60 and 0.72, both ≫ chance) vs cross-floor (0.37) and the derived retention 0.57, against the real 0.73 reference line. The honest reading is the gap below 0.73, not the point value — fold std ≈ 0.30-0.39 on 2 floors.operanet_scene_diversity.png— synthetic floor-ID under empty scene (~1.0), + impairment (~1.0, no change), and + metal-furniture diversity (0.67), against the real 0.83. The message is which bar moves: noise does not, scene diversity does.
There are no figures for the transfer test — it did not run.
Key references
- chen2023_5cbd ↗ — the domain-gap framing this whole card inherits; environment as the hard axis.
- wang2026_2758 ↗ — where and why sims flatter cross-environment generalisation.
- bocus2022_ce7f ↗ — the real anchor: room-ID 0.83, cross-room retention 0.73.
- huang2025_060d ↗ — the real count anchor for the (unrun) transfer test.
- hou2023_bf83 ↗, cha2025_cd6e ↗ — the few-shot real-only baselines the transfer must beat.
- vishwakarma2021_fe49 ↗ — sim-augments-real precedent (radar), and the warning that a learned bridge may be needed.
- jiang2018_77f6 ↗ — separating environment-invariant signal from nuisance (identifiability).
- brunello2025_d781 ↗ — how easily CSI metrics drift, inflating apparent effects (statistician).
- cominelli2023_e6ee ↗, xie2015_0389 ↗, gringoli2019_68e7 ↗ — real commodity-hardware CSI structure and extraction (field / deployment).
- zhang2026_ccac ↗, guarino2026_e72c ↗ — the reproducibility bar the SWE voice invokes against the broken artifact chain.