Evaluation Speed
Every strength result in this project compares engines at matched search depth. At matched time, GNU Backgammon is ahead — by a margin that decides what the engine can do.
Raccoon’s network is more accurate than GNU Backgammon’s, position for position. On the BGSage benchmark it selects moves better at 0-ply than GNU Backgammon does at 1-ply. That result is real and it is measured.
It is also bought with about three and a half thousand times the computation, and that has never been reported alongside it.
This page measures what one position evaluation costs each engine, checks whether the difference is the implementation or the network, and works out what it means for the searches both engines can afford. The conclusion is that the current architecture cannot be made practical by tuning, and that the choice recorded in goal.md’s first assumption is the thing to revisit.
What one position costs
Both figures below come from the same script on the same machine in the same run, so the ratio does not depend on comparing published numbers from different hardware.
| engine | architecture | MACs / position | positions / second |
|---|---|---|---|
| GNU Backgammon | 250-128-5 MLP (one net per position class) | 0.033 M | 1,476,488 |
| Raccoon ep22 | 10x256 ResNet, 11.9 M parameters | 285 M | 426 |
GNU Backgammon evaluates 3,470 times as many positions per second, while Raccoon does 8,718 times the multiply-accumulates per position.
It is the network, not the code
The two ratios differ, and the direction is worth noting: the arithmetic gap is larger than the measured gap. Raccoon extracts more work per multiply-accumulate than GNU Backgammon does, because a convolutional network evaluated in batches uses wide vector instructions and several cores, while a small network evaluated one position at a time does not.
Raccoon reaches 242 GFLOPS, against a theoretical peak near 410 for this processor — about 59% of it, which is good for convolutions on a 2x12 board. Its advantage in efficiency per operation is 2.5x; its arithmetic disadvantage is 8,718x.
So there is no implementation problem to fix. Quantisation, better kernels or a faster runtime might buy a factor of two or three. The gap is three to four orders of magnitude, and it is the shape of the network.
What it costs at matched time
Search cost is evaluation cost multiplied by tree size. Two engines searching to the same depth visit comparable numbers of positions, so a per-position gap of this size becomes a per-decision gap of the same size.
Putting the measured per-decision times next to the benchmark accuracy each configuration achieves:
| configuration | time / decision | benchmark PR |
|---|---|---|
| GNU Backgammon, 2-ply | 0.062 s | 0.588 |
| Raccoon, 0-ply | 0.049 s | 1.026 |
| Raccoon, depth-2 expectimax | 28 s | 0.426 |
The first two lines take about the same time, and GNU Backgammon wins. The third line is the configuration where Raccoon is clearly stronger, and it costs roughly 450 times as long as the opponent it beats.
Times are measured: GNU Backgammon’s from driving the binary (raccoon/eval/gnubg_cli.py), Raccoon’s 0-ply from the throughput above at about 21 evaluations per decision, and depth-2 from exp021’s recorded 12,016 evaluations per decision. PR figures are exp021’s, on a 2,000-decision sample.
Every comparison this project has published is at matched depth. exp021’s ladder, exp026’s head-to-head, the benchmark tables: all of them hold search depth fixed and vary the engine. None holds time fixed. At matched time the ordering reverses, and that is the comparison a real game imposes.
Everyone else uses a small network
| engine | network |
|---|---|
| TD-Gammon | one hidden layer, handcrafted inputs |
| GNU Backgammon | 250 → 128 → 5, one per position class |
| BGSage (measured close to eXtreme Gammon) | 244 → 200/400 → 5, five to nineteen networks by game plan |
| Raccoon | 26-channel input, ten residual blocks at 256 channels |
The convolutional trunk came from the AlphaZero framing in goal.md. Convolutions assume neighbouring squares are related in a way worth exploiting with shared filters. A backgammon board is 24 points where the relationships that matter are pip distances, contact and blot exposure — quantities Raccoon’s encoder already computes and hands the network directly as channels 17 to 25. It is possible that 285 M multiply-accumulates are being spent partly to rediscover structure the input already contains.
What a smaller network would cost
| architecture | MACs / position | projected positions / second | speed-up |
|---|---|---|---|
| ResNet 10x256 | 284.553 M | 426 | 1x |
| ResNet 6x128 | 43.186 M | 2,804 | 7x |
| ResNet 4x64 | 7.437 M | 16,280 | 38x |
| ResNet 2x64 | 3.898 M | 31,059 | 73x |
| MLP 250x128 | 0.033 M | 3,709,532 | 8,718x |
| MLP 250x400 | 0.102 M | 1,187,050 | 2,790x |
These rows scale the measured throughput by arithmetic cost. They say what an architecture would cost, not what it would score. A cheaper network is only useful if it stays accurate enough, and that requires training one and measuring it.
The measurement worth making is accuracy against speed, not accuracy alone. exp017 established that scaling from 8 M to 40 M training positions took PR from 2.16 to 0.93 at the current size, and the size itself was never varied against the same data. The 40 M-position cache is on disk and scripts/train_distill.py already does the work, so distilling it into progressively smaller networks — including a plain network over the existing handcrafted features — is a matter of compute rather than new machinery.
A cheaper network reaching, say, PR 1.2 at forty times the speed would beat ep22 outright at equal time, because it could afford the depth-2 search ep22 cannot.
exp028: a small network, measured
The page above argued that the experiment worth running was accuracy against speed, not accuracy alone. This is that experiment.
Protocol, fixed before the final arms and not varied: every score is the full BGSage benchmark, n = 14,693, with no subsampling anywhere; every comparison is paired on identical decisions and bootstrapped over the 500 generating games; the paired resolution is about 0.09 PR, and effects below it are reported as not resolved, which is not the same as a null. Training data is the same 43 M GNU Backgammon 2-ply labels ep22 was distilled from.
The architecture, and what the inputs cost
Distilling those labels into a network shaped after PureTD’s [512, 512, 256, 256] gives a curve that is monotone in capacity:
| configuration | PR |
|---|---|
| MLP [256], 624 inputs | 5.40 |
| MLP [256,256], 624 inputs | 2.99 |
| MLP [512,512], 624 inputs | 2.31 |
| MLP [512,512,256,256], 624 inputs | 1.93 |
| …dropping the handcrafted channels, 408 inputs | 1.74 |
| …de-broadcast to 201 inputs | 1.68 |
Two input results are worth separating. Dropping the nine handcrafted channels improved PR by 0.19 while removing compute — confirming PureTD’s claim that expert features are unnecessary. But the mechanism is duplication rather than the features themselves: channels 8 to 25 are all constant across the board, so a convolution receives them as planes and a flattened network receives 24 identical copies of each. De-broadcasting to one value per scalar channel gives 201 inputs, which is the size PureTD uses, and buys another 0.10.
More data bought nothing: 24 M to 43 M positions moved PR by 0.04, inside the noise. That measurement was made under the constant-LR schedule, so it is worth repeating rather than treating as settled.
Two hypotheses, tested
With capacity, inputs and data explored, two candidates remained. Four arms, identical but for one variable each, all at the same 20-epoch budget:
| arm | PR at epoch 20 | paired vs reference | |
|---|---|---|---|
| reference (constant LR, six-outcome head) | 1.632 | — | |
| scalar head, squared error on equity | 1.458 | +0.174 [+0.085, +0.264] | real |
| cosine learning-rate decay | 1.420 | +0.213 [+0.127, +0.302] | real |
| both | 1.368 | +0.264 [+0.174, +0.356] | real |
Both hypotheses are confirmed, each worth about 0.2 PR. The learning rate had been fixed at 1e-3 for the whole of every previous run, and the training target had been cross-entropy over six outcomes rather than squared error on the quantity move selection actually ranks.
They do not add.
Independent effects would have given 1.245; the combined arm reached 1.368. That prediction was written into the run script before the arm started. Against cosine alone the combination is +0.051 [-0.035, +0.136] — not resolved — while against the scalar head alone it is +0.090 [+0.009, +0.170], real. So the schedule does most of the work and the two were largely repairing the same deficiency.
Convergence had to be measured, repeatedly
The reference arm was still improving at epoch 20: each five-epoch step was individually below the resolution limit, but epochs 10 to 20 accumulate to a real gain.
| arm | epochs 10 → 20 | |
|---|---|---|
| reference (constant LR, six-outcome head) | +0.128 [+0.032, +0.225] | real |
| scalar head, squared error on equity | +0.055 [-0.017, +0.129] | not resolved |
| cosine learning-rate decay | +0.252 [+0.182, +0.322] | real |
| both | +0.057 [-0.003, +0.119] | not resolved |
This is why the protocol above insists convergence is measured rather than inferred. Training loss cannot see it — the reference network and a far weaker one produce indistinguishable loss curves. Nor can adjacent epoch comparisons, which read not resolved three times in a row on a curve that was genuinely still descending. Only the longer interval showed it.
Where it leaves the architecture
| engine | value MACs | PR |
|---|---|---|
| PureTD (see note) | 0.561 M | 0.94 |
| Raccoon ep22, ResNet 10x256 | 285 M | 0.950 |
| GNU Backgammon, 1-ply | — | 1.206 |
| This MLP, best configuration | 0.563 M | 1.368 |
| GNU Backgammon, 0-ply | 0.033 M | 2.14 |
A network doing 500 times less arithmetic than ep22 reaches PR 1.368, against ep22’s 0.950. It beats GNU Backgammon’s own evaluator comfortably and still loses to GNU Backgammon at 1-ply.
On the PureTD comparison, two qualifications. Their 0.94 is XG++ Performance Rating on cubeful money; ours is this benchmark’s PR on cubeless checker play. The two scales agree closely where they can be compared — GNU Backgammon at 0-ply is 2.18 on theirs and 2.14 on ours, at 1-ply 1.21 against 1.206 — which is why they are shown together, but they are not the same measurement. And while capacity and inference cost are matched (562 k parameters against 565 k, 0.561 M MACs against 0.563 M), training compute is not: 200 M self-play games over about 65 hours on a 64-core workstation with a GPU, against roughly 38 core-hours per arm here with no GPU. Two orders of magnitude.
That distinction decides what to try next. The remaining difference is either the method — on-policy TD with exact Bellman backups over all 21 rolls, rather than regressing a fixed teacher — or simply scale. The evidence against pure scale is that more data bought nothing, but that was measured under the broken schedule. Running the best configuration far longer separates the two for one arm’s cost, before committing to an RL pipeline that would take weeks. That run is exp029 below; it answers not scale.
exp029: five times the training, measured
exp028 ended by naming the cheapest experiment that could separate its two remaining explanations. This is that experiment.
The MLP is matched with PureTD on capacity and on inference cost — 562 k parameters against 565 k — but not on training compute, which differs by about two orders of magnitude. So the 0.42 PR between them is either the training method or simply scale. Running the same configuration five times longer answers that for one arm’s cost, and does not require building an RL pipeline first.
Protocol, unchanged from exp028 and fixed before the run: the full BGSage benchmark, n = 14,693, no subsampling; paired on identical decisions against exp028’s 20-epoch arm and bootstrapped over the 500 generating games; resolution about 0.09 PR. One arm, one variable — 100 epochs instead of 20, with the cosine schedule annealed over the longer horizon, since a schedule that reaches zero at epoch 20 cannot simply be continued. Everything else is byte-identical: the same architecture, the same 201 inputs, the same scalar head, the same 43 M labels.
One axis was available and one was not. data/distill/2ply holds all the labels there are, so this varies passes over fixed data, not fresh data. PureTD’s scale is 200 M self-play games, which is the other kind. That limit is why the supporting metric below matters.
The result
| arm | epochs | PR | paired vs 20-epoch arm | |
|---|---|---|---|---|
| exp028 combined | 20 | 1.368 | — | |
| exp029 combined | 100 | 1.296 | +0.072 [-0.013, +0.156] | not resolved |
Five times the training compute — 40.8 hours against 7.7 — moved PR by +0.072, which the instrument cannot separate from zero. The decision rule was written into the run script before the run started: a gain of 0.27 or more would have meant scale was the dominant explanation, 0.09 to 0.27 partial, and below the resolution limit exhausted.
The internal intervals say the same thing about where the curve stopped:
| interval | gain | |
|---|---|---|
| epochs 20 → 50 | +0.040 [-0.033, +0.112] | not resolved |
| epochs 50 → 100 | +0.038 [-0.012, +0.086] | not resolved |
| epochs 90 → 100 | +0.001 [-0.020, +0.021] | not resolved |
exp028 had to look across ten epochs to see a gain that three adjacent five-epoch steps had each missed. Here the same long intervals find nothing at any length: the last eighty epochs are flat within the instrument, and the last ten are flat to a thousandth.
It is not undertraining
The supporting metric separates “the network has not finished learning the labels” from “the labels are what they are”. Rollout-tier fit, on 149 k candidate positions never trained on:
| arm | rollout MSE | R² |
|---|---|---|
| exp028 combined, 20 epochs | 0.00219 | 0.99636 |
| exp029 combined, 100 epochs | 0.00215 | 0.99641 |
Five times the compute bought a 1% reduction in squared error against the teacher. The network had already fit these labels about as well as it is going to at epoch 20; the extra eighty epochs did not find more in them.
What this settles, and what it does not
Settled: training scale on this label set is exhausted. The remaining 0.35 PR to ep22 is not sitting behind more passes over data/distill/2ply. Anyone proposing to train this architecture longer now has a measurement to answer.
Nearly settled: it is not the number of labels either. The obvious reading of a converged network is that it needs more data, and convergence at a fixed dataset size is indeed the condition under which more data can lower the plateau. But ep22 was distilled from these same 86 shards and reached PR 0.950. So 43 M positions demonstrably carry enough signal for 0.950; a network 500 times larger got it out of them. “Not enough distinct positions” would have to explain why a smaller network needs more data than a larger one to reach the same error, which is backwards: small networks plateau early because of capacity, not sample size. Two caveats keep this from being airtight — ep22 reads 26 channels where this MLP reads 201 de-broadcast base-only inputs, though exp028 measured that dropping the handcrafted channels helped the MLP, which cuts against the caveat rather than for it.
Still open: capacity against method. Either 0.563 M multiply-accumulates cannot represent this function below about PR 1.3, or the training method is the limit — regressing a fixed teacher over GNU Backgammon’s self-play positions, rather than learning on-policy with exact Bellman backups over all 21 rolls. Note that the second absorbs the surviving form of the data argument: with capacity this tight, which positions the network spends it on matters more than how many there are, and that is a question about distribution, not quantity.
And capacity has never been measured under this recipe. exp028’s capacity curve — 5.40, 2.99, 2.31, 1.93 as the network grew — was still descending at the largest arm, and every one of those arms ran under the constant learning rate that the same experiment then showed leaves about 0.2 PR unclaimed. Widening the network under the fixed recipe is a far cheaper discriminator than an RL pipeline, and it is what exp030 does.
A note on the epoch curve. The best single epoch scored 1.268 at epoch 78, better than epoch 100. That is a minimum over 100 noisy scores and is quoted here as a selection, not a result; the pre-registered number is the endpoint. Picking it would be the winner’s curse this project has been caught by before.
exp029 conclusion. Does running exp028’s best configuration five times longer close the gap → no: combined100 = 1.296, paired +0.072 [-0.013, +0.156] over the 20-epoch arm, not resolved (BGSage PR, n = 14,693). Against ep22’s 0.950, the gap is 0.346 and training scale is not what closes it.
exp030: is it capacity?
exp029 ruled out training scale, and ep22’s 0.950 from the same labels ruled out their quantity. Two explanations were left: the network is too small, or the training method is the limit. This arm tests the first, which is the cheap one.
Protocol. One arm, one variable: the width of exp028’s combined configuration doubles to [1024, 1024, 512, 512], and depth, inputs, head, schedule, data and epoch budget do not move. Scoring is the full benchmark, n = 14,693, paired on identical decisions and bootstrapped over the 500 generating games, as everywhere else on this page.
Two baselines, both fixed before the run. The primary is the 1x network at the same 20-epoch budget — the clean one-variable comparison. The secondary is exp029’s 100-epoch network, the 1x architecture at its best under any budget, so a win cannot be dismissed as beating something unfinished.
What 3.6x the arithmetic bought
| comparison | PR | paired gain | |
|---|---|---|---|
| 1x, 20 epochs | 1.368 | — | |
| 1x, 100 epochs (exp029) | 1.296 | — | |
| 2x width, 20 epochs | 1.281 | +0.087 [+0.012, +0.162] vs 1x same budget | real |
| +0.015 [-0.061, +0.090] vs 1x at its best | not resolved |
Real, and immaterial. The interval against the matched-budget baseline excludes zero, so widening does help — but by 0.087 PR for 3.6x the arithmetic. The pre-registered rule asked for 0.27 before calling capacity the dominant explanation and 0.09 before calling it partial; this clears the resolution limit and misses the materiality threshold. Resolved and worth having are different questions, and this is the first case on this page where they separate.
The secondary comparison is the sharper one. Against the 1x network trained five times longer, the wider network is +0.015 [-0.061, +0.090] — not resolved. Spending 3.6x the arithmetic per position reaches the same place as spending 5x the training time, and both stop at about 1.28.
The exchange rate, and where it points
Two arms under one fixed recipe give a slope:
Accuracy costs about 0.068 PR per e-fold of arithmetic on this recipe. Extending that to ep22’s 0.950 asks for 274 M multiply-accumulates per position. ep22 reaches 0.950 at 285 M — within a few per cent of what the line predicts, from a different architecture family.
That agreement is a consistency check, not a third point, and the line itself rests on two arms. Taken at face value it says something unwelcome: under teacher-regression, this project buys accuracy with arithmetic at a roughly fixed rate, and ep22 is already on the efficient frontier of that trade rather than above it. The small network is not a cheaper way to ep22’s accuracy. It is the same trade, priced the same way, evaluated at a point further down the curve.
Which is exactly where PureTD’s claim does not fit:
It reports 0.94 at 0.561 M MACs — the arithmetic of our 1x arm, at accuracy this line places 488x further along it. If that number holds on a comparable scale, no amount of widening reaches it, and the method is what does.
What the capacity costs to evaluate
Accuracy without its price is half the question on this page.
| engine | MACs / position | positions / s | PR |
|---|---|---|---|
| GNU Backgammon, 0-ply | 0.033 M | 1,476,459 | 2.140 |
| MLP 1x [512,512,256,256] | 0.562 M | 128,029 | 1.368 |
| MLP 2x [1024,1024,512,512] | 2.041 M | 46,876 | 1.281 |
| Raccoon ep22, 10x256 ResNet | 285.000 M | 426 | 0.950 |
The wider network still evaluates 110 times faster than ep22 and is twice as accurate per position as GNU Backgammon’s evaluator, at 31x less throughput than it. That point is worth having. It is simply not a route to 0.950.
One tooling note
measure_eval_speed.py timed value_probs6, which raises on a scalar-head network — so it could not measure any of the arms exp028 and exp029 selected. It now times value_equity, the head-agnostic call 0-ply ranking actually makes. Re-measuring ep22 through the new path gives 428 positions/s against the 425.5 this page already quotes, so the change does not move any published figure.
exp030 conclusion. Is the small network’s plateau a capacity limit → no: 2x width = 1.281, paired +0.087 [+0.012, +0.162] over the same recipe at 1x (BGSage PR, n = 14,693) — real, but for 3.6x the arithmetic, and indistinguishable from simply training the 1x network longer. What remains is the training method, or the distribution of positions it trains on, and an RL pipeline is now worth its weeks rather than an assumption.
What this page does not say
- It does not say the network is bad. It is more accurate per position than GNU Backgammon’s by a wide margin. The question is the price.
- The projected rows above are arithmetic, not accuracy. exp028 measured the accuracy separately; see that section.
- exp028’s arms are compared at a fixed budget, not at their endpoints. The cosine arm is converged by construction, since its learning rate reaches zero; the constant-rate arm was still improving when it stopped. “Better at 20 epochs” is a weaker claim than “reaches a better endpoint”.
- It is one machine. A four-core desktop processor with no GPU. A graphics card changes Raccoon’s throughput far more than GNU Backgammon’s, because batched convolutions suit one and a per-position network does not. That changes the arithmetic for training and offline analysis; it does not change what a browser or a phone can run.
- GNU Backgammon’s figure is single-position and single-threaded. It has headroom of its own that this comparison does not give it.
- It retracts no published result. Raccoon does beat GNU Backgammon at matched depth on the benchmark. This page adds the axis those results were silent on.