Evaluation Speed
Every strength result in this project compares engines at matched search depth. At matched time, GNU Backgammon is ahead — by a margin that decides what the engine can do.
Raccoon’s network is more accurate than GNU Backgammon’s, position for position. On the BGSage benchmark it selects moves better at 0-ply than GNU Backgammon does at 1-ply. That result is real and it is measured.
It is also bought with about three and a half thousand times the computation, and that has never been reported alongside it.
This page measures what one position evaluation costs each engine, checks whether the difference is the implementation or the network, and works out what it means for the searches both engines can afford. The conclusion is that the current architecture cannot be made practical by tuning, and that the choice recorded in goal.md’s first assumption is the thing to revisit.
What one position costs
Both figures below come from the same script on the same machine in the same run, so the ratio does not depend on comparing published numbers from different hardware.
| engine | architecture | MACs / position | positions / second |
|---|---|---|---|
| GNU Backgammon | 250-128-5 MLP (one net per position class) | 0.033 M | 1,476,488 |
| Raccoon ep22 | 10x256 ResNet, 11.9 M parameters | 285 M | 426 |
GNU Backgammon evaluates 3,470 times as many positions per second, while Raccoon does 8,718 times the multiply-accumulates per position.
It is the network, not the code
The two ratios differ, and the direction is worth noting: the arithmetic gap is larger than the measured gap. Raccoon extracts more work per multiply-accumulate than GNU Backgammon does, because a convolutional network evaluated in batches uses wide vector instructions and several cores, while a small network evaluated one position at a time does not.
Raccoon reaches 242 GFLOPS, against a theoretical peak near 410 for this processor — about 59% of it, which is good for convolutions on a 2x12 board. Its advantage in efficiency per operation is 2.5x; its arithmetic disadvantage is 8,718x.
So there is no implementation problem to fix. Quantisation, better kernels or a faster runtime might buy a factor of two or three. The gap is three to four orders of magnitude, and it is the shape of the network.
What it costs at matched time
Search cost is evaluation cost multiplied by tree size. Two engines searching to the same depth visit comparable numbers of positions, so a per-position gap of this size becomes a per-decision gap of the same size.
Putting the measured per-decision times next to the benchmark accuracy each configuration achieves:
| configuration | time / decision | benchmark PR |
|---|---|---|
| GNU Backgammon, 2-ply | 0.062 s | 0.588 |
| Raccoon, 0-ply | 0.049 s | 1.026 |
| Raccoon, depth-2 expectimax | 28 s | 0.426 |
The first two lines take about the same time, and GNU Backgammon wins. The third line is the configuration where Raccoon is clearly stronger, and it costs roughly 450 times as long as the opponent it beats.
Times are measured: GNU Backgammon’s from driving the binary (raccoon/eval/gnubg_cli.py), Raccoon’s 0-ply from the throughput above at about 21 evaluations per decision, and depth-2 from exp021’s recorded 12,016 evaluations per decision. PR figures are exp021’s, on a 2,000-decision sample.
Every comparison this project has published is at matched depth. exp021’s ladder, exp026’s head-to-head, the benchmark tables: all of them hold search depth fixed and vary the engine. None holds time fixed. At matched time the ordering reverses, and that is the comparison a real game imposes.
Everyone else uses a small network
| engine | network |
|---|---|
| TD-Gammon | one hidden layer, handcrafted inputs |
| GNU Backgammon | 250 → 128 → 5, one per position class |
| BGSage (measured close to eXtreme Gammon) | 244 → 200/400 → 5, five to nineteen networks by game plan |
| Raccoon | 26-channel input, ten residual blocks at 256 channels |
The convolutional trunk came from the AlphaZero framing in goal.md. Convolutions assume neighbouring squares are related in a way worth exploiting with shared filters. A backgammon board is 24 points where the relationships that matter are pip distances, contact and blot exposure — quantities Raccoon’s encoder already computes and hands the network directly as channels 17 to 25. It is possible that 285 M multiply-accumulates are being spent partly to rediscover structure the input already contains.
What a smaller network would cost
| architecture | MACs / position | projected positions / second | speed-up |
|---|---|---|---|
| ResNet 10x256 | 284.553 M | 426 | 1x |
| ResNet 6x128 | 43.186 M | 2,804 | 7x |
| ResNet 4x64 | 7.437 M | 16,280 | 38x |
| ResNet 2x64 | 3.898 M | 31,059 | 73x |
| MLP 250x128 | 0.033 M | 3,709,532 | 8,718x |
| MLP 250x400 | 0.102 M | 1,187,050 | 2,790x |
These rows scale the measured throughput by arithmetic cost. They say what an architecture would cost, not what it would score. A cheaper network is only useful if it stays accurate enough, and that requires training one and measuring it.
The measurement worth making is accuracy against speed, not accuracy alone. exp017 established that scaling from 8 M to 40 M training positions took PR from 2.16 to 0.93 at the current size, and the size itself was never varied against the same data. The 40 M-position cache is on disk and scripts/train_distill.py already does the work, so distilling it into progressively smaller networks — including a plain network over the existing handcrafted features — is a matter of compute rather than new machinery.
A cheaper network reaching, say, PR 1.2 at forty times the speed would beat ep22 outright at equal time, because it could afford the depth-2 search ep22 cannot.
What this page does not say
- It does not say the network is bad. It is more accurate per position than GNU Backgammon’s by a wide margin. The question is the price.
- It does not measure a smaller network’s accuracy. Every projected row is arithmetic. Whether accuracy survives the reduction is unknown, and is the experiment this page argues for.
- It is one machine. A four-core desktop processor with no GPU. A graphics card changes Raccoon’s throughput far more than GNU Backgammon’s, because batched convolutions suit one and a per-position network does not. That changes the arithmetic for training and offline analysis; it does not change what a browser or a phone can run.
- GNU Backgammon’s figure is single-position and single-threaded. It has headroom of its own that this comparison does not give it.
- It retracts no published result. Raccoon does beat GNU Backgammon at matched depth on the benchmark. This page adds the axis those results were silent on.