ML coding · Statistical inference
McNemar's test: is the new model really better?
Easy · Target 15 minutes · Python and NumPy · Type asked at
Squarepoint
Jump Trading
Apple
You get the true labels y and what two models a and b
predicted for the same examples. I'll call nab the number of examples where A is right and B is
wrong and nba the number where it's the other way round. If the two models are really as good as
each other then every one of those disagreements is as likely to go to A as to B, so it's like flipping
n = nab + nba fair coins. Your function should return the exact two-sided p-value,
which is the chance of getting a split that's at least as lopsided as the one you actually saw. When the
two models never disagree there's nothing to test and you should return 1.
Examples
Input y = [1, 0, 1, 1, 0, 1], a = [1, 0, 0, 1, 0, 1], b = [1, 1, 1, 0, 0, 1]
Output 1
B wins one disagreement and A wins two, and you can't split three flips any more evenly than that, so it's 1.
Input y = [0, 0, 0, 0, 0], a = [0, 0, 0, 0, 0], b = [1, 1, 1, 1, 1]
Output 0.0625
A wins all five, and the chance of five heads or five tails in five flips is 2/32, so you'd return 0.0625.
Input y = [2, 1, 0], a = [2, 1, 0], b = [2, 1, 0]
Output 1
They agree on every example so you've got nothing to go on, and that's a 1.
Constraints
- 1 ≤ len(y) = len(a) = len(b) ≤ 2000
- Labels are integers (any number of classes)
- Checked to an absolute tolerance of 10⁻¹²
Hints
Hint 1
If both models got an example right, or both got it wrong, it can't tell you which one is better, so I'd just skip it.
Hint 2
When the models are equally good each disagreement goes to A with probability a half, so n_ab is Binomial(n, 1/2) and that's all you need.
Hint 3
Because that distribution is symmetric you can work out the smaller tail and double it, as long as you don't let it go over 1.
Where candidates go wrong
- I see a lot of people put the two accuracies into an unpaired test, which throws the pairing away, and when the models agree on most examples it almost never finds anything.
- The chi-squared version is fine when you've got lots of disagreements, but under about 25 it's a poor approximation and the exact sum doesn't cost you anything anyway.
- With more than two classes both models can be wrong in different ways. That still counts as both wrong and not as a disagreement, because you compare each prediction with y and not with each other.
- If n_ab and n_ba are equal then doubling the tail gives you more than 1, so you have to cap it.
The idea
The only examples that matter are the n where the models disagree. If they were equally good then nab would just be the number of heads in n fair flips, so the p-value is the chance of a count that's at least as far from n/2 as yours. I take k = min(nab, nba) and add up C(n, j) for j from 0 to k, then double it and divide by 2n and cap it at 1. Python's integers are exact so I'd keep the sum in integers and only divide once at the end.
Say one model gets 80% and the other 85% on 200 examples. That looks like a clear gap but if you treat them as separate samples it's only about 1.3 standard errors. If the new model wins 14 disagreements and loses 4 the exact p-value comes out at about 0.03, but if it wins 18 and loses 8, which is the same net gain, it's about 0.08. That's why I'd always report the won and lost counts next to the accuracy.
Time O(len(y) + n) · Space O(1)
Reference solution
from math import comb
def mcnemar_p_value(y, a, b):
n_ab = n_ba = 0
for t, p, q in zip(y, a, b):
if p == t and q != t:
n_ab += 1
elif q == t and p != t:
n_ba += 1
n = n_ab + n_ba
if n == 0:
return 1.0
k = min(n_ab, n_ba)
tail = sum(comb(n, j) for j in range(k + 1)) # exact integer
return min(1.0, 2 * tail / 2 ** n)Follow-ups an interviewer asks
- I'd expect to be asked about the chi-squared version with the continuity correction next, and you should be able to say roughly when it gives the same answer as the exact one.
- You can get a confidence interval for the gain in accuracy out of the same two counts, so it's worth knowing how.
- If you've got five models and you run it on every pair you're doing ten tests, and then one of them coming in under 0.05 doesn't mean much.
- Sometimes somebody's changed the eval set between the two runs and then you can't pair them, so think about what you can still say.
Python runs in your browser with NumPy. The first run downloads the runtime, which takes a few seconds.
Why interviewers ask it
Anyone who has shipped a model has had to say whether a two point gain on the eval set was real. This checks that you know the two runs are paired and that only the questions where they disagree carry any information. Most people compare the two accuracies as if they came from separate samples and get a test that's much too weak.
The theory behind it is in the lesson Statistical inference in code: standard errors, permutation tests, the Kalman filter.
Keep going
The autocorrelation function of a series
Compute the sample autocorrelation of a time series at lags 0 to k, vectorised, with the standard estimator.
Open the problem → Medium20 minRegression coefficients with standard errors and t-statistics
Fit least squares with QR, then report each coefficient's standard error and t-statistic.
Open the problem → Medium20 minAn exact permutation test for a difference in means
Compute the exact two-sided p-value for a difference in means by trying every relabelling of the pooled data.
Open the problem → Easy15 minMcNemar's test: is the new model really better?
You've scored two models on the same eval set and want to know if the gain is real.
Open the problem → Medium20 minA one-dimensional Kalman filter
Track a drifting level through noisy measurements with the Kalman filter: predict, compute the gain, update.
Open the problem →