The estimator
from zero.
Nobody counts the people in the library. Nine radios hear Wi-Fi frames wobble and phones advertise, and the estimator has to turn those traces into a number it never sees. It keeps one probability for every headcount from 0 to 160, moves them forward a minute at a time, and corrects them with what was heard. This course builds that machine one equation at a time.
Companion labs in The estimator from zero
Frozen model 964a0a420ed0e427… · frozen 2026-10-17 · body sha256 checked on this request
Every default in the labs is a value of the frozen model, read from the file above and never copied into this page's code. You can move the constants to see what each one does; the frozen model does not move, and the scored predictions use only its values. Measured numbers in the text come from the thesis chapters on the instrument and the estimator and name their section.
The hidden count moves like a queue
Think of the library as a room with a door. Every minute, each person inside flips a biased coin and leaves with probability μ. New people walk in at random, more at noon than at seven in the morning. That is the whole model of how the count changes, and it is the only place time enters.
Try this: set Nt to 40 and move the hour from 07:00 to 13:00. Watch the bars shift right by the arrivals. Then make the stay short and watch the dashed survivors curve slide left.
| Departure probability μ per person per minute | 0.02222 (mean stay 45 min) |
|---|---|
| Capacity bound N̄ | 160 |
| Day classes | 0 Mon-Thu, 1 Fri, 2 Sat-Sun |
| Peak λ (Mon–Thu), arrivals per minute | 1.666 |
(thesis eq. est:dynamics)
| Symbol | Meaning | Frozen value |
|---|---|---|
| Nt | people in the whole library in minute t; never observed | 0 … 160 |
| μ | chance that one person present leaves before the next minute | 1/45, the largest value on the selection grid |
| 1 − μ | chance a person stays; survivors are a Binomial thinning of Nt | |
| λd,h | expected arrivals per minute in day class d (Mon–Thu, Fri, weekend) and hour h | 3 × 24 values, fitted without labels (lesson 7) |
| N̄ | capacity bound: 107 surveyed seats plus standing room; mass above it is put on N̄ | 160, a modelling choice |
Prove it: the step's mean and variance, the stay length, and where a Poisson crowd settles
- One step. Given Nt = n, survivors S ~ Binomial(n, 1 − μ) and arrivals A ~ Poisson(λ) are independent. So E[Nt+1] = n(1 − μ) + λ and Var[Nt+1] = nμ(1 − μ) + λ. These are the two rows in the lab's table.
- The stay is geometric. A person still present after k minutes has survived k independent coins, so P(stay > k) = (1 − μ)k. The stay T has P(T = k) = (1 − μ)k−1μ and E[T] = 1/μ. With μ = 1/45 the mean stay is 45 minutes.
- A Poisson crowd stays Poisson. If Nt ~ Poisson(m), thinning each person with probability 1 − μ gives a Poisson((1 − μ)m) survivor count: its generating function is exp(m((1 − μ)z + μ − 1)) = exp((1 − μ)m(z − 1)). Adding an independent Poisson(λ) gives Poisson((1 − μ)m + λ). The mean is unchanged exactly when m = (1 − μ)m + λ, that is m = λ/μ. That is the grey staircase in the lab: the level an hour would settle at if it lasted for ever.
- The cap keeps probabilities honest. Mass the kernel would put above N̄ is added to N̄, and each row is renormalised, so every row still sums to one. The bound breaks the Poisson argument near the top, which matters in lesson 4.
occupancy_fusion.immigration_death_kernel.
What the CSI level says about the count
A body in the room moves some of the echoes the receiver adds up, so the channel wobbles more. The instrument measures that wobble as the spread of the ratio between a receiver's two antenna chains, takes its logarithm, subtracts the receiver's empty-room floor, and keeps the median over the receivers that report: the panel level L. The model says how high L tends to be for each headcount, and how far one minute can stray.
Try this: compare how far the curve rises between 0 and 10 people with how far it rises between 80 and 120. Then drag the observed value far above the band, once with the outlier component and once without.
| Feature | a | c | s | σ | w |
|---|---|---|---|---|---|
| panel_level | 0.008 | 0.882 | 10.0 | 0.374 | 0.1 |
| log_amb | -1.935 | 2.898 | 10.0 | 1.348 | 0.1 |
| panel_disp | 0.908 | -0.040 | 160.0 | 0.298 | 0.1 |
(thesis eq. est:csi; the spread St and ambient At use the same form with their own a, c, s, σ)
| Symbol | Meaning |
|---|---|
| Lt | median over ≥ 4 receivers of (receiver level − receiver floor); the level is the median of ln(sd of the two-chain amplitude ratio, dB) over 2-s windows |
| a | the level at N = 0, after the floor is subtracted; near zero for L by construction |
| c | how much the level rises per unit of ln(1 + N/s) |
| s | the count at which the curve bends, at the reference scale; small s means early bending |
| σ | one minute's spread around the mean |
| w | weight of the four-times-wider component, a declared choice (0.1), not fitted |
Prove it: why the curve is concave, how to invert it, and what the outlier component does
- Concave, never flat. m′(n) = c/(s + n) > 0 and m″(n) = −c/(s + n)² < 0 for c > 0. The curve keeps rising, more slowly: it does not saturate. Each extra person moves the mean by c/(s + n), so at 100 people one more person moves it about (s + 1)/(s + 100) as much as the first one did.
- Inverting it. A change Δ above the empty level corresponds to n = s(eΔ/c − 1) people. Lesson 10 uses exactly this to say how many people one night of floor drift is worth.
- The outlier component is a softer ruler, not a wall. With a plain Gaussian, the log-likelihood ratio of counts n₁ and n₂ for an observation x is (m₁ − m₂)(2x − m₁ − m₂)/(2σ²): it grows linearly in x without limit. Far from the band the wide component dominates the mixture, and the same ratio becomes (m₁ − m₂)(2x − m₁ − m₂)/(32σ²). The slope falls by a factor of 16, so one wild minute moves the posterior sixteen times less than a plain Gaussian would let it.
occupancy_fusion.GaussEmission.
What the Bluetooth count says about the count
Phones, watches and laptops advertise over Bluetooth under addresses that change every few minutes. The fleet counts the distinct non-public addresses heard in a minute, keeping only those strong enough (−70 dBm at some receiver) to be near the floor: B. More people means more addresses, but nobody knows how many addresses one person adds. That number is α.
Try this: keep B at 224, the busiest minute of the development day in the thesis figure, and move α from 8 to 2 to 0.5. The same minute means 28, then 112, then more people than the room holds.
| Background b₀ (empty room) | 0.05 |
|---|---|
| Base shape rb | 50.0 |
| Shape growth k per person, at αref = 3.0 | 8.0 |
| Certified-empty ceiling on B | 0.086 |
(thesis eq. est:ble)
| Symbol | Meaning |
|---|---|
| Bt | distinct non-public BLE addresses in minute t, strongest signal ≥ −70 dBm at some receiver, instrument addresses excluded; missing when fewer than five receivers report |
| b₀ | addresses heard when the region is empty; from certified-empty minutes, where the count is so close to zero that b₀ sits at its lower clamp |
| α | addresses heard per person present: randomisation, several devices and people without one folded into one number; not fitted |
| rb, k | the shape: larger shape, closer to Poisson; growing with N makes the relative spread fall as the room fills |
Prove it: where the negative binomial's variance comes from, and why the relative spread falls
- A Poisson with a wobbling rate. Let the minute's rate Λ be Gamma-distributed with mean η and shape r, and B | Λ ~ Poisson(Λ). Then B is negative binomial with mean η. By the law of total variance, Var B = E[Var(B | Λ)] + Var(E[B | Λ]) = E[Λ] + Var Λ = η + η²/r.
- The relative spread. (sd/mean)² = 1/η + 1/(rb + kN). Both terms fall as N grows, so a busy room is read more precisely than a quiet one, in relative terms.
- Never below Poisson. The variance-to-mean ratio is 1 + η/r ≥ 1. Lesson 11 returns to this, because the measured short-scale count is often below 1.
occupancy_fusion.BleEmission.
The scale nobody can see
Imagine two libraries. In one, 30 people each carry devices worth 2 addresses. In the other, 60 people each carry devices worth 1 address. Both rooms produce the same Bluetooth count, and if the CSI curve bends at twice the count in the second, the same CSI level too. No radio can tell them apart. This is the central difficulty of the thesis, and it has a name in sensor networks: blind calibration of a global gain.
Try this: move κ anywhere. The two Bluetooth distributions stay on top of each other. Then read the table below the plot: the count process is where the two worlds differ, and the capacity bound is where one of them stops existing.
The lab needs JavaScript. The statement it shows: replacing N by κN, λ by κλ, α by α/κ, s by κs and k by k/κ leaves every observation law unchanged.
(thesis §est:scale; the same symmetry as the global gain of Balzano and Nowak 2007)
Prove it: the observations are invariant, the dynamics are not quite
- Bluetooth. The mean b₀ + (α/κ)(κN) = b₀ + αN and the shape rb + (k/κ)(κN) = rb + kN are unchanged, so the whole negative binomial is unchanged.
- CSI and ambient. m(κN) with s replaced by κs is a + c ln(1 + κN/κs) = m(N). The Gaussian mixture around it is unchanged.
- The queue's mean scales; its variance does not. From κN people with arrivals κλ, the next count has mean κ[(1 − μ)N + λ] — κ times world 1's mean, as the symmetry needs — but variance κNμ(1 − μ) + κλ, which is κ times world 1's variance. Multiplying world 1's next count by κ multiplies its variance by κ². The lab's variance ratio shows the gap.
- Two more breaks. κN must be an integer, and κN cannot exceed N̄ = 160 when the bound does not rescale.
- So what. The departures in 3 and 4 could carry some information about α. Whether they carry enough has not been tested, and the estimator does not rely on them. The scale comes from outside the data: the prior of lesson 5, the capacity bound and certified-empty minutes.
A prior instead of a label
If the data cannot choose α, someone has to state what is plausible, before any count is opened, and the estimator has to carry that uncertainty instead of hiding it. The thesis declares two priors. A is log-normal around 2 addresses per person; B spreads evenly on a log scale between 0.7 and 6. The estimator runs one filter per quadrature node, seven values of α, and mixes their answers with the prior weights.
Try this: leave the priors frozen and move the excess addresses. The dots spread over a factor of about ten under A and about seven under B, whatever the excess: that width belongs to the prior, and no amount of data narrows it. Then compare the browser's nodes with the server's in the table.
| node | α | weight |
|---|---|---|
| 1 | 0.1486 | 0.0005 |
| 2 | 0.3878 | 0.0308 |
| 3 | 0.8985 | 0.2401 |
| 4 | 2.0000 | 0.4571 |
| 5 | 4.4519 | 0.2401 |
| 6 | 10.3156 | 0.0308 |
| 7 | 26.9169 | 0.0005 |
(thesis §est:scale; at node αi the arrivals and CSI scale grow by αref/αi and the BLE shape slope shrinks by it)
Prove it: Gauss–Hermite nodes, the factor 9.8, and what the mixture is not
- Quadrature. For z ~ 𝒩(0, 1), Σi wi f(zi) equals E f(z) exactly when f is a polynomial of degree ≤ 2n − 1, here 13. The nodes are the roots of the Hermite polynomial He₇ (He₀ = 1, He₁ = z, Hek+1 = z Hek − k Hek−1) and the weights are proportional to 1/(n Hen−1(zi))². The lab finds the roots by bisection and checks itself against the Python class.
- Prior A's width. If ln α ~ 𝒩(ln 2, (ln 2)²), then for a fixed excess E, ln N = ln E − ln α has the same standard deviation ln 2. Its central 90 % spans ± 1.645 sd, a ratio of e3.29 ln 2 = 23.29 = 9.78 between the ends.
- Prior B's width. ln α uniform on [ln 0.7, ln 6] makes ln N uniform too; its central 90 % covers 0.9 of the range, a ratio of (6/0.7)0.9 = 6.91.
- A mixture with prior weights is not Bayesian learning of α. A full posterior over α would multiply each node's weight by the evidence p(y1:t | αi). The frozen estimator keeps the prior weights, so it never claims to have learned α from data that lesson 4 showed cannot supply it.
occupancy_fusion.ScalePrior, ScaleMixture.
The filter: predict, weigh, normalise
Here everything meets. At midnight the estimator knows nothing: every count from 0 to 160 is equally likely. Each minute it does three things. It predicts: pushes yesterday's belief through the queue of lesson 1. It weighs: multiplies each count's probability by how well that count explains what the radios heard (lessons 2 and 3). It normalises: rescales so the probabilities sum to one. That is all a hidden Markov model filter does.
Try this: scrub to a morning minute and watch the dashed prediction, the blue likelihood and the posterior bars. Switch the Bluetooth count off and see how much wider the answer gets. Then run the full mixture: your browser reproduces the posterior that the frozen Python code wrote for this day.
The filter lab needs JavaScript. It reads the exported day 2026-10-05 and runs 1,440 steps over 161 states for each of seven scale nodes.
Run the filter on another exported day: 09-30 · 10-01 · 10-02 · 10-03 · 10-04 · 10-05 · 10-06 · 10-07 · 10-08 · 10-09 · 10-10 · 10-11 · 10-12 · 10-13 · 10-14 · 10-15.
weigh and normalise: pt|t(m) = pt|t−1(m) ℓt(m) / ct, ct = Σm pt|t−1(m) ℓt(m)
ℓt(m) = p(Bt | m) p(Lt | m) p(St | m) p(At | m), a missing observation contributes 1
(thesis §est:inference)
Prove it: the recursion, the evidence, and the cost
- Bayes plus Markov. p(Nt | y1:t) ∝ p(yt | Nt, y1:t−1) p(Nt | y1:t−1). The observations are independent given the count, so the first factor is ℓt(Nt). The count is Markov, so p(Nt | y1:t−1) = Σn p(Nt | Nt−1 = n) p(Nt−1 = n | y1:t−1), which is the predict step.
- The normaliser is the evidence. ct = p(yt | y1:t−1), so ln p(y1:T) = Σt ln ct. EM in lesson 7 climbs exactly this number.
- Exact, and cheap. One step is a 161 × 161 vector–matrix product: 161² × 1,440 ≈ 37 million multiply-adds per day and node. No approximation is needed on the full count grid. The 3 × 24 kernels are built once per node.
- Filtered, never smoothed. Every published output uses p(Nt | y1:t): a minute is answered from the past only and never revised by later data, as a deployed estimator would have to.
occupancy_fusion.forward_backward; browser port lib/estimator.js, tested against predict.py's output for this day.
Fitting without labels
The arrival table and the CSI curves were never fitted to a people count, because the thesis has none to give them. They come from the radios alone, by expectation–maximisation: guess the parameters, ask the filter (run forwards and backwards) how many people were probably present each minute, refit the parameters to those soft guesses, and repeat. Each round makes the observed data at least as probable as before, for the exact EM steps.
Try this: read the evidence column: the first round gains thousands of nats, the eighth a fraction of one. Then click the heat table to compare a Monday noon with a weekend noon.
| 1 | -15028.657 |
| 2 | -8630.768 |
| 3 | -8619.800 |
| 4 | -8619.286 |
| 5 | -8619.247 |
| 6 | -8619.243 |
| 7 | -8619.242 |
| 8 | -8619.242 |
λd,h ← mean over minutes of class d, hour h of max(0, E[Nt+1] − (1 − μ) E[Nt]) (a moment update, floored at 0.001)
(a, c, σ) ← weighted least squares of xt on ln(1 + n/s) with weights γt(n), for each candidate s; keep the s with the smallest σ
(thesis §est:inference; b₀ and rb come from certified-empty minutes, not from EM)
Prove it: why exact EM never goes downhill, and where this fit departs from it
- The lower bound. For any distribution q over the hidden counts, ln p(y | θ) ≥ Eq[ln p(y, N | θ)] + H(q), by Jensen's inequality. With q the smoothed posterior at the current θold, the bound touches ln p(y | θold).
- The climb. Choosing θ to raise Eq[ln p(y, N | θ)] raises the bound, and the bound is below the evidence everywhere, so the evidence cannot fall. That is the monotonicity of Baum–Welch; it reaches a local maximum only.
- The arrival update is a moment match. Survivors have mean (1 − μ)E[Nt], so the expected arrivals are E[Nt+1] − (1 − μ)E[Nt]. Using smoothed means instead of pairwise marginals makes this a moment update, not the exact M-step, so the guarantee in 2 is not automatic for it. The frozen fit's trace climbed at every iteration all the same.
- The weighted least squares is exact for (a, c, σ) at a fixed s: the Gaussian log-likelihood weighted by γ is maximised by the weighted normal equations. The outlier weight w is held fixed, a declared choice.
occupancy_fusion.fit_em.
Levels, bands and a 90 % set
A posterior over 161 counts is honest but hard to read. The estimator reads it out in three ways, all from the same distribution, so they cannot contradict each other. Three levels answer "anyone here, a few, or busy?". Six bands answer "roughly how many?". And the 90 % band set says how wide the honest answer has to be.
Try this: with your own shape, widen the spread until the set grows from one band to four. Then switch to the frozen model's day and scrub from night to noon: the levels sharpen long before the bands do.
Band edges 0, 1, 11, 31, 61, 101; levels empty = 0, low = 1–30, busy = 31 and more, fixed on 9 October 2026 before any verification count existed.
90 % set: sort bands by probability, add them until the total reaches 0.9
(thesis §est:outputs)
Prove it: the greedy set is the smallest one, and why zero needs no scale
- Smallest by construction. Suppose the greedy set has k bands. Then the k − 1 most probable bands sum to less than 0.9, and any set of k − 1 bands sums to at most that. So no set with fewer bands reaches 0.9.
- Consistent outputs. The level edges are band edges, so each level probability is a sum of band probabilities; nothing is fitted separately per output.
- Zero is scale-free. Under the symmetry of lesson 4, κ × 0 = 0: whether anyone is present does not depend on α. That is why the empty level can be sharp while the bands stay wide.
- The mean is a footnote. The posterior mean minimises expected squared error, but no hypothesis uses it; it is printed for reference only.
viz/uncertainty.py holds the same rule).
How the estimator is judged
A forecast that says "70 % busy" is neither right nor wrong on one minute. It is judged by a score that rewards putting probability where the outcome lands, and punishes near misses less than far ones. The thesis uses the ranked probability score for the levels (H2) and coverage together with width for the band sets (H3), with the date as the unit, and a test that makes no large-sample assumption.
Try this: in the second panel, move the truth and watch the minimum of the expected score follow it: lying about your belief can only cost you. In the third, change the illustrative differences until the test stops holding.
The lab needs JavaScript. RPS = Σk (Pk − Ok)² over cumulative levels; the H2 test enumerates all 2n sign assignments of the n per-date differences.
H2 primary: dj = RPScalendar − RPSfused per date; holds if mean d > 0 and the 95 % sign-flip interval excludes 0
H3: coverage of the 90 % band sets in [0.85, 0.95], read together with their width
(thesis eq. hyp:rps, §hyp:h2, §hyp:h3)
Prove it: why the RPS is proper, and why twelve dates are enough for an exact test
- A sum of Brier scores. Each term (Pk − Ok)² is the Brier score of the event "level ≤ k". If the truth gives that event probability Qk, E[(Pk − Ok)²] = (Pk − Qk)² + Qk(1 − Qk), smallest at Pk = Qk. Every term is minimised by the true cumulative probability, so the sum is minimised by the true distribution: the score is proper.
- The sign-flip null. If the fused model and the calendar are equally good, each dj is symmetric about zero, so every one of the 2n sign assignments is equally likely. The exact p is the share of assignments whose |mean| is at least the observed |mean|.
- The floor of p. The observed assignment and its mirror image are always counted, so p ≥ 2/2n: 0.0625 for five dates, 0.031 for six. Below six dates the rule cannot hold; twelve dates give 4,096 assignments.
- The interval. The 95 % interval is every shift δ for which the test of d − δ does not reject, searched on a grid between min d and max d.
- Coverage alone can be gamed. A set of all six bands always covers and says nothing. Coverage above 0.95 also counts against H3: sets that cover too often are wider than they claim.
score.py (sign_flip_p, paired_rule), ported in lib/estimator.js.
The floor under everything (H1)
Every CSI level is measured above a floor: what that receiver reads in an empty room. The floor is not fixed. On this fleet single receivers jump between empty nights, by more than 0.8 on a natural-log scale on about one receiver-night in five, each on its own nights. The estimator uses the median over receivers, and a median ignores a minority. H1 asks how much that buys.
Try this: raise the jump chance until the panel row starts to flare too. Then press "Repeat 400 times" and compare the spread of simulated panel ratios with the measured 0.103.
| SD of the panel floor over nights | 0.179 |
|---|---|
| Median busy response | 1.741 |
| Panel ratio R | 0.103 (5–95 % of resamples 0.049–0.135) |
| Single-receiver ratios | 0.20 to 0.55 |
(thesis eq. hyp:h1; fr,j is receiver r's median level over the certified-empty minutes of night j minus its reference floor; Bd the median panel level of weekday d, 10:00–16:00)
Prove it: why the median holds, and what one night of drift is worth in people
- A median moves only with a majority. With nine receivers, the panel floor jumps upward only if at least five receivers jump upward on the same night. If each jumps up with probability 0.1 and down with 0.1, independently (the lab's default), P(≥ 5 of 9 up) = Σk≥5 C(9, k) 0.1k 0.99−k ≈ 0.00089, and either direction ≈ 0.0018 per night.
- Independence is enough. The thesis reports that a model with independent jumps reproduces the panel ratio (0.101 predicted, 0.103 measured): enough to explain it, not proof that it is the mechanism. The nightly deviations of different receivers have a mean pairwise correlation of −0.10.
- In people. Inverting the CSI curve of lesson 2, a floor offset Δ is worth s(eΔ/c − 1) people. With the frozen panel-level parameters (c = 0.882, s = 10.0), Δ = 0.179 is about 2.25 people at the reference scale. For a single receiver the thesis reports 4.6 to 18 people.
- The daily floor. The frozen estimator re-estimates each receiver's floor every morning at 06:00 from the certified-empty minutes of the previous 24 hours. H1 asks, on fixed floors, whether a panel could do without that; no score depends on the answer.
Where the model is knowingly wrong
Every model is a set of convenient lies, and a thesis has to say which ones it tells. Three matter here. Minutes are treated as independent given the count, and they are not. The Bluetooth count is modelled as at least as variable as a Poisson count, and it is often less. And the CSI floor is assumed stable between mornings, which lesson 10 showed holds for the panel and fails for single receivers.
Try this: set ρ to 0.35, the measured value for the Bluetooth count: an hour of minutes carries the evidence of about 29 independent ones. Then set the variance-to-mean ratio to 0.5 and compare the shapes.
| Lag-1 autocorrelation, BLE count / CSI panel level | 0.35 / 0.27 |
|---|---|
| Overstatement factor (1 + ρ)/(1 − ρ) | 2.1 / 1.75 |
| Short-scale BLE variance-to-mean ratio | below 1 in 85 % of 187 hours (median 0.40 to 0.64) |
negative binomial: Var/mean = 1 + η/r ≥ 1
(thesis tab. est:assumptions; Basseville and Nikiforov 1993)
Prove it: the AR(1) factor
- For a stationary AR(1) series with correlation ρ|i−j| between minutes i and j, Var(Σ xi) = σ² Σi,j ρ|i−j| ≈ nσ² (1 + 2ρ/(1 − ρ)) = nσ² (1 + ρ)/(1 − ρ) for large n.
- A likelihood that multiplies n minutes as if independent behaves as if it had n pieces of evidence, while the data carry about n(1 − ρ)/(1 + ρ). The posterior is therefore too sharp by roughly the square root of the factor in width: unaddressed correlation gives artificially high confidence (Khaleghi et al. 2013).
- The consequence is testable: if the band sets are too narrow because of it, H3's coverage falls below 0.85. The frozen model does not correct for it, and the scoring will show whether it should have.
See the frozen model's output on every exported day: the frozen estimator, day by day. For how one minute of CSI is made before any of this, how a level is made.
Sources and assumptions
The model, its symbols and every measured number come from the thesis chapters "Research Hypotheses and Evaluation Criteria", "A Commodity Multi-Link Sensing Instrument" and "A Fused Occupancy Estimator", and from the frozen model file (notebooks/studies/2026-10-09-fusion-estimator/results/frozen-model.json, copied into the vault by notebooks/studies/2026-10-09-fusion-estimator/export_site.py). Literature is cited by first author and year as in the thesis. Lab assumptions that are not the thesis's are labelled where they appear: the illustrative arrival rate of lesson 4, the wobble without a jump in lesson 10, and the twelve per-date differences of lesson 9, which are not results.