The shogi AI on my blog was weak. Weak enough that I — an amateur 2-dan — beat it comfortably, and weak enough that it lost every single game to climbing silver, a plan beginners learn in their first month.

Over the next two months I ran 99 experiments. There are 400 run directories on disk to prove it.

Three of them were ever demonstrated, by playing games, to have made the engine stronger.

This post covers all three, the 42 that were measured and failed, and the 54 that were never measured at all. It's written so that by the end you know what to actually do if you want to make your own engine stronger.

---

How to read this

Top to bottom works. So does jumping. Every measurement lives in its chapter.

Chapter 1 is about measurement, deliberately. Without it the other eight chapters are unknowable. That's the single thing I most want to get across.

---

The setup this assumes

• A shogi AI that runs in the browser (Next.js + a Web Worker)
• A hand-written engine, ported from TypeScript to WebAssembly
• Evaluation moving from hand-written rules to NNUE, a small neural net
• Training data generated locally by YaneuraOu, an open-source engine well above professional strength
• Zero budget. The GPU is the one built into a Mac; the engine and the data are free

Where "it runs in the browser" changes the answer, I say so — a native engine would make different tradeoffs in a few places.

---

The conclusion, up front

Two months compressed into three lines:

1. Only structural changes worked. Raw speed (a 15x WASM port), a search algorithm fix, and which positions the net trains on.

2. Everything in the "make it bigger" family failed. More data, a wider network, a deeper teacher. All of it.

3. The most dangerous failure mode was believing something worked when it didn't. An engine playing itself scored 56.9% over 160 games (p=0.040). An 80-game verdict carries noise the same size as the effects it's being used to detect.

The third one is why this post exists.

---

Where it started

The site has two shogi pages. The engine was TypeScript: a 1-D board array, make/unmake, Zobrist hashing plus a transposition table, negamax with alpha-beta, PVS, iterative deepening. Textbook-correct.

Layered on top was a geological record of improvements, V2 through V18.

Baseline: three seconds of thinking reached ~4,000 interior nodes, ~51,000 quiescence nodes, and depth 5. That's one to two orders of magnitude slower than a decent JS engine — which, in hindsight, was the largest single lever available.

Two patterns stand out from that history:

• Features accumulated but stayed OFF, or "high difficulties only." Nobody had systematically A/B tested any of them
• The evaluation was still basically the V2 hand-written structure (material + piece-square tables + king safety + castles + file defense), with almost no notion of one piece attacking another

So: a pile of features nobody knew the value of, and seven months of hand-written rules.

That's the starting line.

<a id="0"></a>

0. How the work was run — AI agents in parallel

Before the technical chapters, a note on process. It has its own pitfalls, and it explains how the numbers later in this post were produced.

Partway through, the working mode changed:

"Do everything with subagents, in parallel."

Up to five Claude Code subagents ran concurrently.

• Each agent gets its own git worktree — a parallel checkout of the same repo — so they can't trample each other
• They write code and run it as real processes on the local machine, not in a cloud. There's no terminal window, but you can watch everything with ps aux, tail -f, top
• On completion each returns a commit. The orchestrator resolves conflicts, integrates, independently re-verifies, and sends it to human review

The verification protocol got corrected mid-flight too. Agents were initially told to beat the old V18. "Test against current V20, not V18" was the right call. The pre-change engine was frozen as a registered opponent (v20base), and "direct match against current" became the standard gate — more sensitive and more reproducible.

---

What five agents actually produced

This is the real adoption/rejection data.

Survival rate of plausible-sounding improvements: roughly 1 in 3–4. Stockfish folklore says 1 in 5–10. Without a rejection mechanism, every one of these would have shipped.

---

Living with review bots

PRs get automatic reviews (Gemini, Copilot). On the V19 PR, five of nine findings were false positives — they misread the parameter semantics of the attack-scan function, whose third argument is the defender (documented in the code).

Two findings were real (out-of-bounds ply access on typed arrays) and got fixed.

The right relationship with automated review is neither "accept everything" nor "ignore everything." It's verify every finding independently.

One of the nastier bugs in this post — where a fix introduced a 24-second UI freeze of its own (Chapter 2) — was caught by review, not by me.

---

An agent's report is not a fact until you verify it

This is the heaviest lesson of the chapter.

What actually happened:

• An agent reported "done" with a PR URL that did not exist. The branch had never been pushed, the test file it named wasn't there, and all that existed on disk was one "WIP" commit. I relayed its numbers before checking.
• Another agent built the WASM from the wrong source file — assembly/index.ts (36,787 bytes) instead of assembly/index-halfkp64-rki16.ts (38,288 bytes, the actual shipped binary). Three weeks of engine work silently reverted, measuring 38.5% against real production. The agent had seen the size and hash mismatch and dismissed it as toolchain drift. It had verified "my source rebuilds to my binary" when the question was "does the SHIPPED binary rebuild from this source"
• Stale-base branches injected unrelated deletions, twice. Both caught by git diff --stat origin/main HEAD

So the protocol hardened. Before merging or relaying anything:

gh pr view <n>                        # does the PR exist
git ls-remote origin <branch>         # was the branch actually pushed
git diff --stat origin/main HEAD      # is the diff scoped to what was intended

Every time.

It cut the other way too. An agent overrode my instruction and was right — it measured that the premise behind my instruction was false and told me so instead of quietly complying.

And once, an agent caught my own aggregation error. The candidate swaps sides between games, and I had looked at the first pair's header and assumed it held for all 384. Worse, I'd written that I "confirmed it two independent ways" — but both ways rested on the same wrong assumption, so it was never independent at all.

The thing this post keeps saying — don't extrapolate a whole from one observation — I did while writing it.

---

My own accidents, for fairness

• I ran git checkout in a worktree an agent was live in, pulling the branch out from under nine files of unfinished engine work. Nothing was lost, but a git commit -a would have swept unfinished engine changes into a documentation PR. Only the habit of reading the diff caught it
• I didn't account for the CPU demand of an agent I'd told not to run games. Another agent's 1,440-game gate held ten processes at 100% CPU and starved its test run for 40–50 minutes

---

Chapter summary

---

<a id="1"></a>

1. Measurement first — get this wrong and nothing else counts

Starting a post about making a shogi engine stronger with a chapter on measurement looks like a detour. It isn't.

If you can't measure, you can't know whether it got stronger. And the dangerous outcome isn't "I don't know." It's "I concluded it got stronger when it didn't."

That happens. Easily.

---

1.1 Play an engine against itself and see what number comes out

Do this experiment first.

Identical binary, identical weights, identical settings, on both sides. In principle it scores 50%.

Mine:

at 160 games:  56.9%   one-sided p = 0.040
at 320 games:  52.8%   [47.3, 58.2]

At 160 games it read 56.9% with a one-sided p of 0.040. Had that been "new candidate vs current engine," it would have been declared significant at the 5% level and shipped. It was the same engine on both sides.

Extended to 320 games it settled to 52.8%. An excursion.

This is an A/A control. Run it before you measure any candidate. A number you read without knowing what "no difference" scores can't be assigned meaning afterwards.

---

1.2 How many games — 80 is not enough

I assumed 80 games was plenty. It isn't.

The standard error of a win rate is √(0.25/n). At n=80 that's 5.6 points, so a 95% interval spans roughly ±11 points. A result of "61.9%" carries the interval [50.9%, 71.7%] — a lower bound that clears 50% by nine tenths of a point. That happens routinely.

Working backwards from effect size:

Make 500 games the floor. On my machine — a 14-core MacBook — 1,000 games takes about five hours. That's measured, not estimated. There's no longer an excuse for an 80-game verdict.

---

1.3 A staged gate separated the mirage from the real thing

The cleanest result in this whole post.

Two candidates went through the same three-stage gate, which increases the sample size at each stage.

Candidate A looked clearly strong at 56 and 96 games. In engine-vs-engine play, 59.8% is a wide margin. Taken to 768 games it became 51.2% — a coin flip — and missed the pre-registered bar (lower bound of the confidence interval above 50%).

Candidate B held. At 768 games: 69.7%, 95% CI [66.4%, 72.9%]. A lower bound of 66% is not something luck produces.

If I had stopped at 56 games, I'd have shipped Candidate A.

Staging buys something else too: obviously bad candidates die cheaply, so you don't spend 768 games on every idea.

---

1.4 Set the bar before you measure — then hold it

Decide the acceptance criterion before the first game, and don't move it once you've seen results.

Mine was typically "56 games, 62 or more points out of 112" (a win is 2, a draw is 1).

Four experiments came in one point short. Two of them had the identical scoreline:

29 wins, 3 draws, 24 losses = 61 points (needed 62)

That's a winning record. Twenty-nine wins against twenty-four losses, and it didn't ship. One point is half a game.

In all four cases the bar didn't move. There's no record of anyone dropping it because a result was close.

It can look pedantic. After §1.3 it doesn't. In a world where 59.8% becomes 51.2%, the moment you start waving through "close enough," everything built on top of it is sand.

---

1.5 Counterbalance the sides

Who plays first matters. So does the opening.

Split the candidate evenly between the two seats.

I learned this the hard way. A book-gate A/B produced p=0.033 — "significant." Re-run with the order swapped, the sign flipped. The difference wasn't the change. It was whichever engine moved second.

Practically: alternate which seat the candidate occupies per game, and at aggregation time resolve which side is the candidate from each log's own header. Summing raw A/B results straight through gives you garbage — I made exactly this mistake and had to be corrected.

---

1.6 Static metrics do not predict strength

This is the thing I confirmed most often, and always the hard way.

Example 1 — agreement with the teacher. A model improved agreement with the teacher engine's evaluations from 87.4% to 96.2%. As an offline number that's a large gain.

Its playing strength did not move at all.

Example 2 — validation loss. The model with the best validation loss in its cohort went 6 wins, 21 losses in direct play (p=0.006).

Example 3 — closeness to the teacher. An NNUE that passed a quality gate at "2.0–2.5x closer to the teacher than the hand-written eval" scored 19.6% in real games. A rout.

Why does this keep happening? Because what alpha-beta asks the eval for is the ranking of sibling positions, not accurate absolute scores.

A mean error of 405cp exceeds the typical gap between candidate moves (under 100cp), so rankings invert. The hand-written eval, by contrast, is self-consistent even where its absolute scale is off — and a monotone scale error is harmless to alpha-beta.

Static metrics are fine for narrowing candidates. They are not fine for deciding adoption.

---

1.7 The measurement rig itself will be the bug

Four times, in four different ways.

(a) Opening diversity collapsed. With only 2 forced opening plies, results suddenly read 2W–5L–7D. At 2 plies nearly every game starts from the same position, and deterministic engines replay "effectively 2–4 distinct games, duplicated." A no-book control losing under the same config proved the setup was the problem, not the change.

→ 6+ forced opening plies, varied seeds.

(b) Time-scale bias. 200ms blitz behaves differently from 1-second production games (mate-probe overhead overweighted, deep-search gains underweighted). One technique measured 3W–5L at medium (800ms) and flipped to 6W–2L–2D when re-gated to hard and up.

→ Final verdicts at production budgets.

(c) Mismatched defaults. The match script defaulted to a different eval mode than production. For a while I was comparing under non-production conditions.

→ Say "same as production?" out loud, every time.

(d) Null by construction. A book-gate A/B produced zero disagreements across ~1,500 book consultations. Both sides used the same book and the same threshold, so the games never reached a position where they could differ. Not "no difference" — no possible difference.

→ Harvest the positions where the gate actually fires, and start games from those.

---

1.8 Waits you can cut, and waits you must not

Once 500 games is the floor, the games dominate the wall clock. There are two different kinds of waiting here.

Do not cut: the thinking time of a single game.

What you're measuring is "strength when thinking for 1000ms." Speed up the thinking and you've changed the measurement itself. One game ≈ 150 moves × 1 second ≈ several minutes is the definition of the experiment, not waste.

Do cut: the serialization between games.

One match process effectively uses one core (both sides think alternately inside it). Spare cores can host several matches at once.

48 games through one process    2.7 hours
48 games across six in parallel  ~27 minutes

Nothing inside any game changed. Each still gets its full thinking time. Only the queue was rearranged, so the measurement is intact and only the clock shrinks.

The fairness argument is different from the data-generation case and worth spelling out: engine matches under a time limit are sensitive to machine load, but both sides of an A/B think alternately inside the same process, so load hits both equally. Keep parallelism comfortably below the core count (6 matches + training ≈ 8 of 14 cores) and "one side got unlucky with the scheduler" can't happen structurally.

---

1.9 When a reproduction fails, suspect your reproduction first

While independently re-verifying the mate solver, I got "both engines miss the mate." Panic.

The cause: the position I'd built had one gold in hand instead of two. The 5-ply mate needed two gold drops.

I was measuring "missed mates" in a position with no mate in it.

The first suspect in a failed reproduction is your own reproduction code.

This is §1.7 in personal form. Suspect the instrument in order: yourself first, then the subject.

---

1.10 The final judge is a human at the board

A self-play win rate is an instrument for driving development. It is not the judge of pass/fail.

An NNUE that won 77.1% in self-play shipped to production — and I beat it comfortably. With the report: "it keeps playing nonsense moves." Dropping a pawn somewhere irrelevant to the mate, walking its own king into danger — after a full two seconds of thought.

After that, the acceptance criterion itself changed.

Not a self-play win rate — the blunder rate on real game records, cross-checked against YaneuraOu at depth 18.

And that 81-ply game became a permanent regression test. Every change from then on replays it and checks the blunder rate hasn't risen. No test case is worth more than a failure a human actually punished.

---

Chapter summary

---

<a id="2"></a>

2. Is it even running? — the biggest wins were buried here

This chapter had the best return on effort of anything in the project.

Before you polish the search, before you train a neural net, check one thing:

Is the search actually running?

In my case the answer was no. Three separate times, in three different ways.

---

2.1 A move that comes back in one millisecond

The owner sent a real game. From a standard double-wing opening:

9.  ▲2四飛          rook takes on 2f
10. ☖4二飛 ??       ignores the rook-file exchange, shuffles its own rook
11. ▲2三歩打        pawn drop, threatening the bishop
12. ☖9四歩 ??       ignores the threat, pushes an edge pawn

Answering the rook-file exchange with △2三歩 is page one of double-wing theory. Missing it was bizarre.

The smoking gun was the thinking time.

Replaying the position through the production call path with a probe script:

move 10: AI(hard) plays 82->42 (23ms)   ← hard has a 2-second budget
move 12: AI(hard) plays 93->94 (1ms)    ← one millisecond

The engine had never searched at all.

The culprit was the book's "resync fallback" — a leftover from the slow-engine era. Whenever a position wasn't in the curated book, it answered instantly with a plausible-looking quiet developing move, validated only by a 1-ply static check.

Three flaws stacked:

1. Its only phase gate was "total pieces in hand ≤ 2," so it fired in the middle of a hot rook-file exchange
2. Its candidates were quiet board moves only, so the correct △2三歩 — a drop — was structurally impossible for it to choose
3. Self-play could never catch it, because both engines shared the same fallback. Only humans, who deviate from book, exposed it

Resync was removed entirely: out of book → always search. The same position now produces △2三歩打 after a real 2.0-second search.

Debugging heuristic: if a bad move comes back instantly at a difficulty with a multi-second budget, suspect a bypass path, not the search.

Every reproduction script written afterwards prints per-move thinking time.

---

2.2 The book's safety gate was rejecting verified book moves

Second one.

The book has a safety valve: if the engine's own static evaluation dislikes a book move, reject it. Insurance against corrupt book data. Sound idea.

But the threshold was so tight it was rejecting book moves that a superhuman engine had already verified.

Measured rejection rates:

master   9.4%
hard     6.4%
expert   6.4%

Nearly one book move in ten, thrown away. And after throwing it away the engine falls back to its own search, so the opening loses coherence.

Widening the threshold from 90cp to 150cp:

Piece losses more than halved.

§1.7(d) bites here. The first A/B I built produced zero disagreements across ~1,500 book consultations — same book, same threshold on both sides, so the games never reached a position where the gate could differ. Re-harvesting positions where the gate actually fires was what finally made it measurable.

---

2.3 The worker never started, so the neural net never ran

The third one is the worst.

I spent weeks improving the NNUE, gated it over hundreds of games, promoted the winners, deployed. None of it ever reached a user.

Playing in the browser, the AI was oddly weak and a badge read "低速互換モード" — compatibility mode. I'd added that badge days earlier. It appears when the engine fails to start in its Web Worker and falls back to a lightweight replacement on the main thread.

The design is right. A browser that can't run the worker should still play a game. The problem was that the degradation was too quiet.

The root cause: the browser was holding a corrupted copy of the worker script in its HTTP cache. And that file is served with cache-control: public, max-age=31536000, immutable.

immutable promises the content at this URL will never change, so don't revalidate while it's fresh. Standard and correct for content-hashed filenames.

The consequence is that once a bad copy lands in the cache, the browser keeps using it for a year. Reload, next day, redeploy (same URL) — no change.

Three loads of the same URL settled it:

And fetch(url, {cache:'no-store'}) returned 200, 818 bytes, correct content.

Server fine. Bytes fine. URL fine. The only broken thing was the copy inside the browser.

The repair was one line:

await fetch(url, { cache: 'reload' });   // overwrite the poisoned entry
new Worker(url);                          // → boots

The generalizable condition

immutable caching + a failure path that degrades silently = permanent, silent degradation

Either alone is safe. With immutable, visible breakage gets reloaded. With silent degradation, a recovering next request makes it a blip. Together, nobody notices.

Auditing my own codebase for assets meeting both conditions turned up four. Three were already defended:

// neural weights loader — existing code
const hit = await cache.match(key);
if (buf.byteLength === EXPECTED_BYTES) return buf;   // validate
await cache.delete(key);                             // evict if corrupt
// → refetch

Validate → evict → refetch. The one asset missing that pattern was the worker script — the one furthest upstream.

A diagnostic recipe

Two things worth memorizing:

• Empty error message + filename === null means "died before executing." A script that loads and throws gives a real message
• A successful fetch does not mean the resource is usable. fetch(url, {cache:'no-store'}) bypasses the HTTP cache; new Worker(url) does not. Same URL, different path to it

The fix — and two landmines on the way

I did not remove immutable. Removing it makes every visitor pay revalidation cost on every load — a permanent tax against a rare failure. Bad trade. The right layer to fix is elsewhere.

Instead the loader now self-heals:

1st boot failure  →  fetch(url, {cache:'reload'}) to overwrite the poisoned entry  →  respawn
2nd boot failure  →  respawn at ?__wcb=<n>, a different cache key
still failing     →  the existing storm guard stops cleanly

Landmine 1: never name the cache-busting parameter params.

The bundler's bootstrap reads:

var e = new URL(location.href).searchParams.get("params");
if (!e && o.hash.startsWith("#params=")) e = decodeURIComponent(o.hash.slice(8));

Query first, hash only as a fallback. Had I named my cache-busting parameter params, it would have hijacked the dependency-chunk list and broken the worker in a far more confusing way. Hence __wcb.

Landmine 2: don't round-trip the #params= fragment through URL's serializer.

It re-encodes %, and decodeURIComponent then returns a different string. Re-attach the fragment verbatim.

The fix introduced a bug of its own

Worth including, because it's the same class of mistake.

Cache repair means await fetch(...). In the cooldown-retry path, the "worker is available again" flag was cleared before that await.

1. Mark the worker as available
2. Wait for a network round trip ← the worker reference still points at a terminated instance
3. A move arriving in that window gets posted into a dead worker
4. Nothing answers until the hard deadline — up to 24 seconds of frozen UI

Precisely the stuck-UI failure the fast-fail path exists to prevent. Review caught it.

The fix was to make the flag structurally safe rather than patching each exit:

const rebuildWorker = async (...) => {
  respawnPending = true;              // set on entry
  try { /* repair, then rebuild */ }
  finally { respawnPending = false; } // released on every path
};

I got the root cause wrong four times

In order, because the order is the point:

1. "The weights download is failing" — there were real 503s in the log. But the weights fetch had never been attempted
2. "The worker starts too late" — weights readiness took 11 seconds. But the worker itself was dying
3. "The worker URL is malformed" — it ended in ],"",null,null]. That is the bundler's own bootstrap payload, working as designed
4. "The chunk is transiently 5xx-ing" — all six chunks returned 200, and the failures were deterministic, five of them inside 19 milliseconds

Four times I grabbed a plausible intermediate observation and never checked the final state. One look at "is it actually playing on the neural evaluation right now?" would have killed three of them.

---

2.4 Repetition history never reached the search

Fourth. The report was "it repeats the same sequence four times."

The cause: the positions already played in the real game were never handed to the search. It could only see repetitions inside its own reading, not in the actual game.

Priming the search with the game history, and removing the root TT cutoff:

loop reproduction rate 6/96 → 1/96

Not eradicated. Saying so plainly.

---

2.5 Blind spots in the eval come in symmetric pairs

Worth knowing as a pattern.

Example 1 — promoted pieces became invisible. A term granting +800 for a rook or bishop in the enemy camp stopped matching the moment the piece promoted, because its piece-type code changes. After being broken through, the engine's eval said "roughly equal."

Example 2 — the climbing silver vanished at the moment it succeeded. The term detecting a climbing-silver attack only scanned ranks 5–7 for the attacking silver. The penalty disappeared exactly when the silver entered the defender's half — that is, exactly when the attack worked.

Example 3 — the same shape existed on the other side. A review bot pointed out ranks 1–2 were also unscanned. It was right.

Bugs of the same shape nest in the same places: boundaries. Find one at a boundary and go look at the opposite one.

---

Chapter summary

Every fix in this chapter was a correctness fix, not a strength improvement. And they had the best return of anything I did. Chapters 8 and 9 come back to what that means.

<a id="3"></a>

3. Speed — the one lever you can guarantee mechanically

Most ideas for making an engine stronger die under measurement (Chapter 8 lists all of mine). Speed is different.

You can measure whether it got faster, and — this is the part that matters — you can make it faster while changing the strength by exactly zero. No other lever has that property.

---

3.1 Why speed becomes strength

Engines search by iterative deepening: solve to depth 1, then depth 2, then depth 3, until the clock runs out.

Each extra ply inflates the work by roughly 2–3x — it's a multiplication of branches, so exponential. Turn that around: make the engine twice as fast and you get roughly one more ply.

And a ply is worth a lot at the board.

---

3.2 TypeScript → WebAssembly: 15x, measured

The starting engine reached depth 5 in three seconds. Too slow.

Rather than port everything and hope, I ported one piece and measured.

Move generation alone, in AssemblyScript, benchmarked with perft (counting legal moves):

The perft counts matched JS exactly at every depth — proof the port was correct. On that evidence, the full port was green-lit.

It ran in four phases, and every phase demanded bit-identity.

Result: depth 11–12 at three seconds. 10–0 against the current engine.

Why demand bit-identity

This is the part of the chapter I'd most like to transplant into other projects.

Allow "close enough" and bugs slip the net. Demand exact equality and every bug lands in it. And the moment parity holds, the speed differential converts directly into strength.

If a fixed-depth search matches on node counts as well as move and score, that is proof the search trees are identical. Change one legality verdict anywhere and alpha-beta pruning shifts, moving the node count. It didn't move. So the behavior is the same.

A field test — the AI was right

After deployment, the owner (2-dan) played master and sent a move he suspected was a mistake: △3五角, attacking his own rook. "Wouldn't promoting the bishop and grabbing a pawn have been better?"

We asked the local YaneuraOu, depth 17.

The master-level AI matched a superhuman engine's first choice, and both human "improvements" dropped a whole bishop.

That's a different kind of evidence than a self-play number, and about as good as confirmation gets.

---

3.3 Lazy legality — one idea, several times faster

Profiling during the port found the real bottleneck.

Legal move generation was doing "make → test whether my own king is in check → unmake" for all ~80 generated moves. At every node.

Under alpha-beta most nodes cut off after one to three moves. So the legality checks for moves that are never searched are pure waste.

// After: generate pseudo-legal moves, verify only when actually making one
const moves = generatePseudoLegalMovesPooled(k, pool[ply]); // king safety NOT verified
for (const te of moves) {
  k.move(te);
  if (isKingInCheck(k, k.teban)) { k.back(te); continue; } // ← checked only here
  // ... search ...
}

Together with quiescence partial sorting (below), the same three seconds went from depth 5 to depth 7, and interior nodes from 4k to 22k.

---

3.4 SIMD — 6.2x on evaluation

SIMD means process multiple pieces of data with a single instruction. An ordinary CPU instruction adds a and b once. SIMD adds a1+b1, a2+b2, …, a8+b8 in one shot.

WebAssembly has a 128-bit register called v128; pack eight 16-bit integers into it and operate on all of them at once.

NNUE inference, unwrapped, is almost entirely multiply-accumulate — "weight × activation, add it in," thousands of times. SIMD's home turf.

6.2x on the microbenchmark, ~1.4x in real search. Search does more than evaluate — move generation, pruning checks — so a 6x faster eval doesn't stretch the whole thing that far. Amdahl's law, exactly.

1.4x is still a lot. And the important part is the correctness guarantee.

NNUE inference is built on integer arithmetic, and integer addition is associative — reordering doesn't change the answer. So adding in SIMD batches or one at a time produces a bit-for-bit identical result. Floating point wouldn't allow this; rounding error changes with order.

We searched the same positions both ways and confirmed the node counts match to the single node. Faster, and it doesn't play a single different move.

---

3.5 Multithreading (Lazy SMP)

In the visitor's browser, up to four threads search the same position simultaneously.

Lazy SMP is, as the name says, a lazy parallelization. Each thread searches independently, but they share results through a shared transposition table — a shared notepad holding results for positions already computed. If one thread finds "this position is a mate," the others read the note and skip the recompute.

Rather than strictly dividing the work, the synergy of varied search orders plus the shared notepad makes the whole thing faster.

nodes            ~3.0x  (four threads fall short of 4x; sharing/sync overhead)
strength         1000ms × 24 games: MT 14W–10L (58.3%, point estimate ~+58 Elo)

Being honest: n=24 is nowhere near significance. A 14–10 split over 24 games happens by chance about as often as 14 heads in 24 coin flips. By Chapter 1's standards this is a "seems to be working" signal, not a settled number. It should be pushed past 100 games.

A technical snag: cross-origin isolation

Sharing positions across threads requires SharedArrayBuffer, which browsers permit only under cross-origin isolation — a state achieved with two HTTP headers, COOP and COEP. Security reasons.

But applying those headers site-wide blocks externally loaded images and scripts across the board, and the home page breaks.

So the build config scopes COOP/COEP to the shogi page paths only. Only that page becomes an isolated environment; every other page is untouched.

---

3.6 The discipline: "read the same moves, just faster"

Working on the endgame slowdown (Chapter 7) made it clear there are two kinds of speed improvement, and they have completely different risk profiles.

Three examples from the "same moves" side.

(1) Skip the legality check on drops

A shogi rule so simple it's almost anticlimactic.

A drop doesn't move a single piece already on the board. So it can never expose your own king to a new attack.

Moving a piece can open a line — an enemy rook or bishop hidden behind it now attacks your king. That's why the check exists. But a drop only adds a friendly piece.

Therefore: if your king isn't already in check, every drop is automatically legal and the scan can be skipped entirely. Only when you are already in check do you still verify that the drop blocks or captures the checker.

const mover = k.teban;
k.move(te);
// A drop (te.from === 0) only adds a friendly piece, so it can't expose the king.
// If not already in check, it's always legal → skip the scan.
if ((te.from !== 0 || parentInCheck) && isKingInCheck(k, mover)) {
  k.back(te); continue;
}

Drops are 80–90% of moves in the endgame, so this one line made the legality phase ~27% faster. Generation order, move set, and node count unchanged to the bit.

(2) Scan the board once during drop generation

Drop generation is a triple loop of "piece type (7) × file (9) × rank (9)," and inside it, whether each square was empty got re-read from the board every time. With seven piece types in hand, the same 81 squares get scanned seven times.

But the set of empty squares and the double-pawn status don't depend on the piece type.

// One board pass before drop generation, indexed by file
for (let suji = 0x10; suji <= 0x90; suji += 0x10) {
  let bits = 0, nifu = false;
  for (let dan = 1; dan <= 9; dan++) {
    const c = ban[suji + dan];
    if (c === EMPTY) bits |= 1 << dan;        // record empty squares as bits
    else if (c === ownPawn) nifu = true;      // own pawn → no pawn drop on this file
  }
  emptyBits[suji >> 4] = bits;
  sujiHasOwnPawn[suji >> 4] = nifu;
}
// Inner loop: no board re-reads, just a bit test and a flag lookup
if (komashu === FU && sujiHasOwnPawn[s]) continue;
if ((bits & (1 << dan)) === 0) continue;

On positions with 5–21 pieces in hand, generation got 15–27% faster (median ~24%).

The WASM side had one trap. Sharing the scratch tables in a single global breaks when the pawn-drop-mate check re-enters the generator recursively, overwriting the tables mid-loop. JS was safe because that check goes through a separate generator; WASM comes back into the same function. The tables became per search ply to be recursion-safe.

A port must preserve not just "the same logic" but "the same safety under the same recursion."

(3) Borrow only the prep work of bitboards for check detection

Re-profiling showed the heavy fixed cost in the endgame was check detection, isKingInCheck — ~820ns per call, about half the per-node cost of an endgame position.

One reason serious engines are fast is bitboards: hold the board as bits of an integer and process "empty squares," "attack rays" and so on with AND/OR/shift as batched bit operations.

But I measured honestly first. JavaScript has two pitfalls: (1) JS bitwise ops are 32-bit, so 81 squares don't fit in one word, and (2) a real bitboard needs occupancy updated incrementally, and that update lands on move()/back() — the hottest path of all. The bookkeeping could eat the savings.

So before rewriting everything, I prototyped and profiled in isolation. A slider-list approach — keep sliding pieces in a separate list and test only their line-of-sight to the king — measured about 2x slower than the existing 8-direction ray walk.

The array-based ray walk was already well-optimized for the JS JIT.

So I took only the part of the bitboard toolkit that needs no incremental state: read ban[] directly instead of through a bounds-checked k.get(), and fold the per-cell friend/enemy branches into a single bit mask computed once from the side to move.

const ban = k.ban;
const enemyFlag = teban === SENTE ? GOTE : SENTE;
const selfFlag  = teban === SENTE ? SENTE : GOTE;
// Step attackers (12 dirs): enemy bit set AND the move table says it attacks inward
for (let d = 0; d < 12; d++) {
  const koma = ban[target - diff[d]];
  if ((koma & enemyFlag) !== 0 && canMove[d][koma]) return true;
}

The algorithm and attack tables are unchanged to the bit, so the result is exactly identical. Holding no incremental state, it adds zero cost to move()/back().

isKingInCheck        820ns → 557ns    (~32% off)
whole inner loop     300,700ns → 246,900ns  (~18% off)
WASM perft           +8.5% to +9%
nodes at fixed 2s    13,917 → 14,855  (+6.7%)

"Bitboards are faster" is true — as a statement about environments with 64-bit registers and cheap incremental updates. Drop it straight into JavaScript and the maintenance cost eats the win. Don't take the textbook structure on faith. Profile it in isolation in your own environment and pick up only the parts that pay.

---

Chapter summary

Speed is the most reliable lever there is. But as Chapter 4 shows, speed alone plateaus.

<a id="4"></a>

4. Search — the largest strength win came from here

Of 99 experiments, three were ever proven stronger by playing games. The biggest of the three is this chapter.

768 games 69.7% [66.4%, 72.9%] ≈ +145 Elo

The change was one thing: how quiescence search handles being in check.

---

4.1 The shape of the search

Alpha-beta plays a candidate move, reads a few plies ahead, and asks the evaluation function for a number. Recursively.

"Move quality" is a comparison of the scores of the positions a move leads to. The eval is the judge; the search is a tour guide parading each candidate's future past that judge.

Chapter 1's point — that search wants rankings, not absolute scores — falls straight out of this picture.

---

4.2 What quiescence search is, and why check evasion mattered

Quiescence search is an extra search run at the leaves of the main search.

Why it's needed: suppose you read to depth 4 and evaluate "+300." If, in that position, the opponent can capture your rook, the evaluation is a lie. Statically evaluating a position in the middle of an exchange is always wrong.

So at the leaves you keep reading captures and promotions until things settle. That's quiescence.

The problem is positions where you're in check. Ordinary quiescence only reads captures — but if you're in check, you have to get out of it first, and evasions include king moves and interposing drops, which are not captures.

That's what this fix addressed: when in check, generate and search evasions directly.

The measurement

Stage          Games   Candidate record         Score
Screen           56    35W  5D  16L             66.96%
Independent      96    56W  6D  34L             61.46%
Formal          768    520W 31D 217L            69.73%  [66.4, 72.9]

69.7% over 768 games, with a lower bound of 66.4%. It did not decay as the sample grew.

For contrast, the eval-side candidate that went through the same three stages read 59.8% → 58.3% → 51.2% and was rejected (Chapter 1). Same design, and it separated the mirage from the real thing.

---

4.3 The techniques that were already there, and worked

The pre-port engine had textbook pruning implemented but switched OFF. Systematic A/B testing turned on the ones that paid.

That last one is unglamorous and it worked. It had been scoring and sorting every move; now it sorts only the ones it will read.

---

4.4 The search changes that failed

Being direct: every "change which moves we read" idea I tried, lost.

(a) Check extensions — failed

The idea: if we can only read shallowly, at least read the forced single-file lines of consecutive checks deeper. Mates and finishing sequences resolve through consecutive checks, so extending only there should cut endgame oversights.

The logic looked right. I implemented it with a per-path budget on how often it could extend.

Result: couldn't win a self-play A/B. Not adopted.

The reason made sense afterwards. Check extension changes which moves get read. Reading one line deeper means another goes shallower. The endgame as a whole isn't faster, so the positions it helps and the positions it hurts cancel out.

(b) Late move pruning for drops — failed

The existing LMP stubbornly exempts drops — in shogi a drop is often the crux of a mating attack.

So I touched only the least valuable drops: at shallow depth, after enough moves had been read, cut only pawn and lance drops far from the enemy king (never gold/silver/knight/bishop/rook, never near the enemy king). Thresholds a notch stricter than the existing LMP.

Result: 16 wins, 24 losses, 8 draws across three seeds. It won no seed.

Same shape of reason. "A pawn or lance drop far from the enemy king" is not always meaningless in a shogi endgame — a dangling pawn setting up a later promotion, a sacrifice laying groundwork, sealing an escape square. A move that looks distant is often part of the mating net. Gains and losses didn't cancel; the losses won.

Not adopted. Production behavior stayed bit-identical, and the code remains as a "tried it, lost" record.

(c) Singular extensions — implemented, never gated

Read a move that is "clearly better than all the others" one ply deeper. Implementation and correctness verification (perft, bit-identity at fixed depth) both passed, CI green.

But at fixed depth 10 the node count roughly doubled (17.0M → 35.2M), and at the production 1000ms the engine reached lower depth (9.70 → 9.53).

Parameter sweeps give a monotone dose-response: the more it fires, the more depth it costs, and every depth-neutral setting is one where it effectively never fires. Both attempts to cheapen the exclusion search made it worse — a cheaper test concludes "singular" more often, and the extra extensions cost more than the test saves.

Shogi has drops, so the move list is enormous, and "verify that every other move is worse" is expensive. At a one-second budget it doesn't pay. Held, ungated.

(d) ProbCut and history gravity — failed

• ProbCut (two variants): faster on the bench, lost real games. At depth 7 the "shallow verification search" is depth 3 and prunes real tactics
• History gravity: at short time controls it just discards fresh information

---

4.5 Transposition table carry-over is worth more than it looks

The most interesting finding in this chapter came out of a failed experiment.

The attempt: the engine reads by iterative deepening, and the iteration it's in when the clock expires is thrown away entirely. Measured:

searches whose final started iteration was discarded   95.8%
time spent on that discarded iteration                 mean 713ms (of a 1000ms budget)

"If we're going to throw it away, don't start it." Skip iterations that can't finish, move sooner. Same strength, less waiting.

Measured:

thinking time   938.6ms → 633.4ms   (−32.5%)
strength        600 games, 44.2%  [40.3, 48.2]   ≈ −40 Elo

Cutting a third of the wait cost 40 Elo. The interval's upper bound is 48.2% — significantly worse, not neutral.

Why: the transposition table is not cleared between moves.

The deep entries that "wasted" iteration wrote get read by the next search. The engine was pondering on its own clock.

To confirm, the same match was re-run with the table cleared before every move on both sides:

normal (carry-over intact)    44.2%
cleared every move            54.7%   [47.0, 62.2]

The deficit vanished (z = 2.34, p ≈ 0.019). The loss wasn't shallower search on the move being played — it was the destroyed carry-over, roughly 73 Elo of swing living inside time that looked like pure waste.

For anyone reading this: "skip the iteration you'd throw away anyway" is wrong from its premise in an engine like this. There is no idle time to reclaim.

There's a forward-looking implication too. If cross-move carry-over is worth that much, a bigger transposition table might pay. Currently 2^20 ≈ 1.05M entries. If the cumulative unique positions in one game exceed that, the very entries worth 73 Elo are the first to be evicted. I haven't measured it yet.

---

4.6 Use the opponent's clock (pondering)

While your opponent thinks, the AI does nothing. That's free compute.

The moment the AI answers, it starts searching the position the human is now looking at, warming the transposition table that persists across the whole game. When the human finally moves, the real search probes a hot table and reaches deeper within the same budget.

The implementation question is "how do you make a synchronous search interruptible?" The WASM search is a synchronous call; run it naively for a long stretch and the worker goes deaf to messages.

The answer is a loop of short 200ms slices chained via setTimeout(0):

// Why slices:
// - The WASM search is synchronous; a single long call would make the worker
//   deaf to incoming messages (the next bestMove, clearTT, ...). Instead we run
//   one short slice (default 200ms), return to the event loop via setTimeout(0),
//   and queue the next slice. Any message that arrived during a slice is
//   dispatched *before* the queued slice callback, so calling stop() from
//   onmessage reliably cancels pondering with at most one slice of latency.

Four safety rails:

• A generation counter, bumped on every stop/start, so stale timers can't run an old session's search
• A 30-second cap per turn, so an abandoned tab doesn't burn CPU and battery forever
• Stop when the tab is hidden — a worker can't see page visibility, so the client relays visibilitychange
• Disabled on easy and when WASM isn't loaded — easy is intentionally weak, and without WASM there's no shared table worth warming

Result: mean search depth 9.00 → 9.35 (+0.35 plies). In the opening, where the table warms fastest, gains reached +2 plies.

---

Chapter summary

Every "change which moves we read" idea failed. What worked was making the search read something it had not been reading at all.

<a id="5"></a>

5. Evaluation — from hand-written rules to a neural net

The longest chapter. Also where I failed most spectacularly and succeeded most.

The map first:

---

5.1 What an evaluation function actually is

Start by clearing up a common assumption. An evaluation function does not judge "this move is good or bad." It does exactly one thing:

Show it a board, and it returns one number.

+250 means "Black is better by two and a half pawns." −1200 means "White is winning." 0 means equal.

Then who decides whether a move is good? The search (the diagram in Chapter 4).

The hand-written eval is a sum of human-readable rules:

score  = material balance;      // pawn=100, rook=1040, ... summed difference
score += piece-square bonuses;  // "this piece on this square is worth +N", phase-weighted
score += king-safety count;     // gold/silver defenders around the king
score += castle shapes;         // pattern-matching Yagura / Mino / Anaguma
score += rook-file defense;
score += climbing-silver pressure;
score += major-piece activity;
return score;                   // e.g. +250

The neural version is a completely different machine answering the same question. Open it up and you find 580,000 anonymous numbers, with no row labeled "king safety" anywhere.

Yet from the search's point of view the two are fully interchangeable. The search only demands "a box that returns a number when handed a position," and never asks what's inside.

The A/B matches in this chapter are literally "swap the judge, keep the same search, play the games."

If you can't read it, how can it be right?

Because "right" is measured by outcomes, not by explainability.

There's a perfect precedent close to home: a strong player's intuition. A 2-dan glances at a position and feels "Black is better," yet cannot fully verbalize it. The after-the-fact explanations — "material advantage," "thin king" — don't describe the computation actually happening in their head, which is invisible even to them. And still the judgment is usually correct.

A neural net implements exactly that kind of intuition that bypasses verbalization. Where the hand-written eval can only hold knowledge someone managed to put into words, the net absorbs patterns directly from a million teacher judgments.

You don't read it. You examine it.

1. Measure its deviation from the teacher on 4,000 unseen positions (holdout MAE)
2. Measure how often it agrees with the teacher on which of two positions is better (pair accuracy)
3. Finally, make it play

Chapter 1's refrain — "the final gate is always playing the games" — is a corollary of unreadable things can only be audited by their behavior.

The flip side lives in the same place. When the net is wrong, the reason is just as unreadable. With the hand-written eval we once pinpointed "the climbing-silver term cuts off at rank 4" and fixed that line. A net's mistakes can only be fixed by changing the data and retraining — which is why the seemingly roundabout journey from a 19.6% defeat to "scale up the teacher data" was, in fact, the only repair procedure a neural network offers.

---

5.2 What NNUE is

NNUE (Efficiently Updatable Neural Network) is, as it happens, a shogi invention. Devised in 2018 by shogi programmer Yu Nasu, it swept the shogi engine scene via the YaneuraOu family, and in 2020 Stockfish adopted it, making it the world standard.

The essence is in the name. A single move changes only 2–4 pieces' worth of board facts, so the first layer's activations can be updated differentially instead of recomputed — which lets a CPU keep up with alpha-beta's demand of hundreds of thousands of evaluations per second.

The network is small enough to quote in full:

class DistillNet(nn.Module):
    H1 = 256   # width of layer 1 — the traditional NNUE choice; dominates inference speed
    H2 = 32    # width of layer 2 — squeezing straight down to 1/8th, the NNUE signature

    def __init__(self):
        super().__init__()
        # (1) Board input layer. Each of 2,268 possible 'facts' — like "black pawn
        #     on 7f" — gets its own row of 256 numbers. Evaluating a position starts
        #     by summing the rows of every fact that holds.
        #     EmbeddingBag(mode="sum") does that lookup-and-sum in one op.
        self.board = nn.EmbeddingBag(BOARD_FEATS + 1, self.H1, mode="sum", padding_idx=PAD_IDX)
        # (2) Hand input layer: counts of the 14 droppable piece types -> same 256 dims
        self.hand  = nn.Linear(HAND_FEATS, self.H1)
        # (3)(4) Reduction layers: 256 -> 32 -> a single number (the evaluation)
        self.l2    = nn.Linear(self.H1, self.H2)
        self.l3    = nn.Linear(self.H2, 1)

    def forward(self, board_idx, hands):
        a1 = self.board(board_idx) + self.hand(hands)   # board + hand contributions
        h1 = torch.clamp(a1, 0.0, 1.0)                  # ClippedReLU: clip into [0,1]
        h2 = torch.clamp(self.l2(h1), 0.0, 1.0)
        return self.l3(h2).squeeze(-1)                  # one output ≈ cp / 600

Almost all the parameters live in table (1) — 2,268 rows × 256 columns ≈ 580,000 numbers — and nowhere in them is "king safety" written down.

ClippedReLU isn't a stylistic choice, it's a practical one. Because activations are pinned to [0,1], the trained float weights survive quantization to int16 for the WASM engine's integer arithmetic.

---

5.3 What actually happens inside loss.backward()

The heart of the training loop is this:

out = model(b, h)                          # score a minibatch of positions
loss = F.mse_loss(torch.sigmoid(out), t)   # deviation from the teacher's scores
opt.zero_grad()
loss.backward()   # backpropagation: for all 580k dials at once, compute which way
opt.step()        #                  reduces the loss — then nudge every dial

"Adjust the weights in the direction that reduces the loss" — but who knows that direction?

The principle is humble. For each of the 580,000 weights, ask: if I nudged this one up slightly, would the loss go up or down, and how steeply? That slope is the gradient, and then you turn each dial a small step downhill.

new_weight = current_weight - learning_rate * slope

The learning rate is how far to turn per step: too large and you overshoot into divergence, too small and you never arrive.

The non-obvious part is computing the slopes. Naively you'd nudge one weight, re-evaluate the whole net, and repeat — 580,000 times. Backpropagation uses the chain rule to push the blame for the error backward through the layers, computing every weight's slope simultaneously in a single backward pass:

output error: "the evaluation came out 0.3 too low"
  ↓ blame l3's weights     "the final judgment underweighted this feature"
  ↓ blame l2's weights     "that feature came out weak because of this reduction"
  ↓ blame the board table  "the numbers in the 'pawn on 7f' row were too small to begin with"

The picture to keep: someone descending a foggy mountain blindfolded. Nobody can see the whole map, but the slope underfoot is computable at every step. Take a small step downhill. One minibatch is one step, and 1M positions × 40 epochs is over a hundred thousand steps.

"val_mae dropped to 437cp" means this descent reached a valley whose altitude is a 437cp average error.

Training happens once, offline. On a Mac's GPU (MPS), 1M positions × 40 epochs takes 7–10 minutes. There is no training during play.

---

5.4 The distillation pipeline — zero budget

• YaneuraOu is open source. The public NNUE eval "Háo" is GPL-3.0 and free for local use
• PyTorch is free; Apple Silicon's MPS does the training
• Measured throughput: ~1,400 positions per second of labeling

Generation was slow — 11 hours became 45 minutes

Projecting toward 1M positions gave about 11 hours. "Can't you throw more GPU at it?" — the investigation landed somewhere unexpected.

node (generation driver)   : CPU 100%   ← one core out of 14, maxed out
YaneuraOu × 8 processes    : CPU ~0%    ← all idle
machine overall            : 81% idle

The pipeline alternates between creating positions via low-budget self-play and labeling them with the eight engines. Measured per chunk: generation 102s vs labeling 1.5s. The labeling side was massively underutilized; the bottleneck was creating, from start to finish. GPUs are irrelevant here.

Why was creating slow? Node.js is single-threaded, so one process = one core — and move selection was still using the JS version of our engine. The 15x faster WASM build was sitting right there.

Two steps:

1. Scale drivers from 1 to 3 (each writing its own file, concatenated later)
2. Swap move selection from JS to WASM (the position distribution is unchanged — same engine, same search)

chunk generation:  102s → 27s
total throughput:  ~8 positions/sec → ~110 (14x)
remaining time:    11 hours → 45 minutes

The core usage is the interesting part. There are 27 processes — 3 Node drivers, each commanding 8 YaneuraOu engines — yet on average only 3–4 cores are busy. The 24 engines are burst workers: idle until ~1,000 positions pile up, grade them in 1.5 seconds, sit back down. The only always-busy workers are the three Node cores producing positions.

Size the always-busy roles to your core count, and overprovision the burst roles so their bursts never stall the producers.

Two lessons. When a wait feels long, look at ps first to see which process is actually busy — a common-sense remedy like "use the GPU" whiffs entirely if the bottleneck lives elsewhere. And speed assets you build once get reused in unexpected places: the WASM engine built to make the browser opponent stronger made the ML teacher-data factory 14x faster.

Six walls hit while generating across two Macs

Running the 5.24M generation across two machines surfaced things invisible on paper. With the numbers intact, because anyone doing this will hit them.

(1) At depth 12, the bottleneck moved. At depth 8, generation was the bottleneck and labeling was instant. Raising it to depth 12 made labeling heavy, and the two became roughly fifty-fifty (~27s / 28s). Which stage is heavy shifts with your settings, so re-measure every time things slow down.

(2) Distributed duplication was auto-avoided by the seed. The naive worry is "won't both machines produce the same positions?" No. The generator's RNG seed is time ^ process ID, so different machines and processes don't collide. No coordination — no assigning a range per machine — was needed at all. This is the strength of position generation being embarrassingly parallel.

(3) The oversubscription trap. "More processes = faster" backfires. The second machine is a 12-core previous-generation Mac; starting it with --engines 8 (4 drivers × 8 engines = 32 engines) made 32 engines fight over 12 cores, and generation got 4x slower (117s).

Dropping to --engines 2 cleared the jam. Observing further, generation runs at 1 driver = 1 core, and 8 cores were idle — so drivers went from 4 to 8 to fill them.

It is not "the more processes, the faster." Look at the core count and each process's role — always-busy generation vs. bursty labeling — and allocate accordingly.

(4) Right tool: CPU vs GPU. Generation (search, scoring) runs on the CPU; training runs on the GPU. Search is branchy, unpredictable work that suits the CPU; training is "run the same computation over a huge batch at once," which suits the GPU. Even "heavy computation" wants different hardware depending on its shape.

(5) Concurrent writes corrupted a file. I accidentally launched the same generation command twice, and two processes appended to the same output file simultaneously. Lines interleaved and corrupted mid-line, to the point that git grep returned Binary file ... matches.

Concurrent appends to one file will almost certainly corrupt it. Only a small amount was ruined, so it got deleted and redone.

If you write in parallel, have each process write its own file and concatenate at the end.

(6) One GPU, 14 CPU cores — pipeline across different resources. This unpicks an assumption baked into the word "parallelize." Apple Silicon has one GPU built into the chip. "20 GPU cores" means 20 cores inside a single GPU — the same structure as "14 CPU cores = one CPU." So running two trainings at once just makes them fight over the one GPU.

But there's a way around it. An A/B match is CPU work; training is GPU work — different resources, so running them simultaneously doesn't contend.

So instead of a naive serial chain (base training → rank training → eval → quantization → A/B), it got pipelined: once base finished training, its A/B match ran on the CPU (about two hours) while the freed GPU trained the next variant in the background. Total went from about four hours to about three.

"Parallelize" isn't only "line up many copies of the same work." Pipelining that overlaps work on different resources is just as powerful. If (3) is "don't make the same resource fight itself," (6) is "don't let a different resource sit idle" — two sides of the same coin.

---

5.5 Round one: 100k positions → routed at 19.6%

The quality gate passed. On a 10,000-position validation split, "how close is each evaluator to the teacher?":

2.0–2.5x closer to the teacher than the hand-written eval.

The real games came back 19.6%.

Inference was verified bit-identical across torch/TS/WASM on 300 positions, and speed was fine. The implementation was correct. The model lost.

The diagnosis is the interesting part. "Closeness to the teacher" turned out to be a poor predictor of playing strength.

As Chapter 1 said, what alpha-beta needs is the relative ranking of sibling positions. The net's ~405cp error is larger than the typical eval difference between candidate moves (under 100cp), so it scrambles rankings.

On top of that, the search's margin constants were calibrated to the hand-written eval's scale — roughly 3.7x true centipawns — making them effectively ~3.7x too generous for a net that outputs true cp.

Killing hypotheses one at a time

There were several competing hypotheses. Fix them all at once and you learn nothing. So each was isolated.

To test "miscalibrated margins are the culprit" without moving any other variable, the engine got an output-scale setter:

export function nnueEvaluateCp(): i32 {
  const outQ = nnueEnabled && !nnueForceFull ? nnueEvaluateFast() : nnueEvaluate();
  // Fold the output rescale (numer/denom, default 1/1) into the same i64 division
  // so there is only ONE truncation — with 1/1 this is bit-identical to before.
  let cp = (<i64>outQ * <i64>nnueScaleK * <i64>nnueOutNumer) / (<i64>8128 * <i64>nnueOutDenom);
  // ...
}

The default 1/1 is bit-identical to the previous behavior — a guarantee of zero regression.

Same weights, same conditions, applying only the 37/10 calibration, 28 games:

Not only did it fail to recover — it got worse (−10.7pt). Hypothesis rejected.

The interpretation: the effectively looser margins had been quietly acting as insurance, re-verifying noisy evaluations with extra search. Calibrate the margins back to their intended strength, and the eval noise lands directly on the pruning decisions.

Even a well-reasoned mechanistic hypothesis dies in an A/B test. Had all three hypotheses been "fixed" at once, we'd never have learned which one mattered. The tedium of isolating one variable paid out right here.

---

5.6 Round two: 1M positions → 77.1%, shipped

With "teacher data quality × quantity" identified as the real battleground, that's where the investment went.

(1) 100k → 1M positions.

(2) The --balance option: positions with |cp| > 1200 — decided positions — are probabilistically thinned, raising the share of near-equal positions. Round one's data was over 60% |cp| > 1000: heavy on lopsided endgames, thin on exactly the subtle middlegame differences alpha-beta needs most.

(3) A ranking loss, encoding round one's lesson directly into the shape of the loss function:

if args.loss == "ranking":
    diff = c.unsqueeze(1) - c.unsqueeze(0)     # teacher cp difference for every in-batch pair
    mask = (diff >= args.rank_pair_min) & (diff <= args.rank_pair_max)
    #  keep only the *subtly different* pairs (default 50–600cp).
    #  Under 50cp: either is fine. Over 600cp: already obvious. In between is where search lives.
    ia, ib = mask.nonzero(as_tuple=True)
    if ia.numel() > 0:
        rank_loss = F.relu(rank_margin_logit - (out[ia] - out[ib])).mean()
        #  Penalize exactly the pairs where the teacher says A is better but the net
        #  does not rank A above B by the margin. Absolute accuracy is not demanded —
        #  only the ordering.
        loss = loss + args.rank_weight * rank_loss

In the foggy-mountain picture, the ranking loss reshapes the mountain itself. On a mountain that only punishes score error, places where siblings are ranked wrongly are shallow dips the descent ignores. The ranking term carves those into deep valleys, so the same descent walks toward lowlands where the ordering is right. Where you end up is decided by what you punish.

The 300k interim result

Even base gained +9pt with zero changes to the training method — 3x data plus balance thinning alone. Direct confirmation of the "data was the bottleneck" diagnosis.

A time-control asymmetry also appeared: rank reaches parity at 1000ms but sinks in 200ms blitz; base does the reverse. This fits the theory — the deeper the search, the more pair-ordering accuracy compounds; in shallow search, calibration of big scores feeds directly into pruning. Production budgets are 1–5 seconds. Which horse to back was obvious.

The reversal at 1M

At 1M, base overtook the ranking-loss runs even on pair ordering.

At 300k, "ranking loss directly optimizes ordering" had been the winning argument. With enough data, plain regression learns the ordering too.

Ranking loss was a crutch for data starvation. Scale the data before reaching for exotic loss functions.

The games

run1m-base 1000ms × 16 games × 3 seeds 37/48 (77.1%) every seed above 70%

Genealogy: 19.6% → 32.1% → 77.1%.

Shipping it

Design: aggressive switch, defensive depth.

• The weights (1,185,988 bytes) ship as a static asset, fetched asynchronously at worker startup and copied into WASM memory. Zero bundle-size increase
• Until the fetch resolves, the engine plays on the hand-written eval (the first moves come from the book anyway)
• Only medium and up (≥1s) use NNUE. Easy stays hand-written, following the measurement that it still wins at ~200ms budgets
• Fetch failure, size mismatch, or WASM trouble all fall back silently
• Switching NNUE ⇄ hand-written clears the TT, so 3.7x-scale scores never mix with true-cp scores inside the table

---

5.7 Round three: losing to a human — "sigmoid saturation"

This is the climax of the chapter, and probably of the whole post.

77.1% in self-play, quality gate cleared handily, shipped to production at medium and up. The genealogy climbed cleanly. Case closed.

Except the author — an amateur 2-dan — played production hard (2 seconds) and won.

Not by a hair, and with the report: "still way too weak. It keeps playing nonsense moves." Dropping a pawn somewhere irrelevant to the mate, walking its own king into the danger zone — chosen after a full two seconds of thought.

Round one's biggest lesson had returned in exact form: a self-play win rate does not guarantee strength against humans. The same trap, a second time.

How it was diagnosed

To hit it with facts instead of impressions, the entire 81-ply game was replayed through the reproduction harness — the exact production call path — comparing, for each move, what the engine picks with NNUE ON against NNUE OFF. Every move was then cross-checked against YaneuraOu at depth 18, quantifying where and how much was lost.

Fact (1): not an instant-move bug

The first suspect was a relapse of Chapter 2's bypass bug. No. The bad moves reproduced on the NNUE path (ON: 19 of 40 moves disagreed with YaneuraOu; OFF: only 7).

With NNUE ON, the engine searched the full two seconds and chose that bad move believing it was best. Not a bug — a flaw in the evaluation function itself. As diagnoses go, the worst kind.

Fact (2): the main culprit is sigmoid saturation

During training, the NNUE passes the teacher's cp through sigmoid(cp/600) before learning from it. Sigmoid is an S-curve that squeezes its input into 0–1, and as the input grows the output pins to 1 and stops moving.

A sigmoid on cp/600 flattens almost completely once the eval passes about ±2500cp.

During training this is fine — you don't need to finely distinguish already-decided positions.

In the real search it's a mortal wound.

At move 72 of that game — where its own king walks into mate, a loss of 31,934cp — the NNUE was asked to evaluate all 71 legal moves:

all 71 collapsed into a band of just 15cp

Whatever you play, it looks like roughly the same score. Shown the same position, the hand-written eval kept a spread of about 390cp. It can still rank moves in decided positions.

"In a position where the outcome is decided, every move looks like the same score" — this is the direct cause of the nonsense moves. If it can't tell moves apart, the engine effectively chooses at random. Walking into mate follows naturally.

Fact (3): a secondary example — a pawn dropped on its own back rank

At move 48, a meaningless pawn drop onto the deepest rank of its own camp (a loss of 1,626cp).

Here the NNUE misjudged the position as roughly even, so "pass-like moves" clustered at the top of the ranking. Dropping a pawn on your own back rank is, in shogi terms, nearly equivalent to killing a piece. But that cost is nearly invisible to the net.

Why? Because moves of this kind are essentially absent from the teacher data. YaneuraOu never plays them, so the net was never once taught how bad they are. It can't evaluate the cost of a folly it has never seen.

Fact (4): the real culprit was a well-intentioned thinning

Here the story connects back to round two.

Saturation itself is a property of the sigmoid. What worsened it into a mortal wound was how the teacher data was made.

Round two used --balance to discard 70% of decided positions (|cp| > 1200), keeping 30%, to raise the share of close positions. The aim was to make the net practice the middlegame fights it agonizes over most — like showing a medical resident plenty of subtle, hard-to-judge patients rather than only textbook-clear cases. Entirely sound on its own.

But over-thinning worsened the saturation. By discarding too many decided positions, the net graduated having barely practiced positions where the outcome is settled. So when it meets one in production, every move looks the same.

The region where the sigmoid mathematically flattens (±2500cp) and the region left thin by the thinning overlap in exactly the same place.

The resident analogy: show a resident only subtle cases and almost never obviously critical ones, then put them on the floor. When a patient in critical condition arrives, they have no sense of how bad is this, and everyone looks "medium."

Immediate response, and a changed standard

The NNUE's scope was narrowed to medium only.

Looking back soberly, 77.1% was measured at 1000ms only. Hard and up was an unverified extrapolation — "read deeper and it should be even stronger." And the deeper you read, the easier you reach saturation-region positions, so hard is if anything more prone to collapse.

The heaviest lesson: using a self-play win rate as the acceptance criterion was itself the mistake. 77.1% is a real number, but what it measured was "can it beat the old eval in self-play," not "does it avoid blunders against a human." We passed the proxy while the behavior that mattered was broken.

The gate became:

the blunder rate on real game records, cross-checked against YaneuraOu at depth 18

And that 81-ply game became a permanent regression test.

---

5.8 The cure: 5.24M positions

Change (1): balance-rate 0.3 → 0.5. Loosen the discard from 70% to 50%. Now the resident also sees the critical patients.

The measured distribution of the 5.24M positions:

Decided positions (|cp|>1200) recovered from round two's 36% to 49.7%. The region crushed by saturation now makes up nearly half the training set. This is the primary fix.

Change (2): labeling depth 8 → 12.

Why 12, and not 20 or 30? Three points.

1. Cost grows exponentially. Each extra ply inflates the work 2–3x. Depth 8→12 alone makes labeling 15–20x slower. Depth 20 is hundreds of times; depth 30 would be minutes per position — weeks to months for 5M
2. There's a ceiling on the precision the student can absorb. This is the essential reason. The student's approximation error is currently ~450cp. The labeling difference between the teacher's depth 12 and depth 20 averages only 20–50cp. The teacher's 20–50cp of precision is completely buried inside the student's 450cp of error. Even if the teacher grades to the millimeter, the student copies to the centimeter
3. Depth 12 is the standard "knee" of shogi NNUE distillation

Change (3): don't mix in the old 1M.

You might think "we already have 1M positions — why not add them for 5.5M?" We don't mix them.

The old data was made with depth-8 labeling and balance-rate 0.3 — that is, it is the very culprit carrying the two problems we're trying to fix. Mix it in and the hard-won saturation fix gets diluted.

Don't mix the cause of the disease into its cure.

The old 1M isn't used for training; it's kept as the baseline to beat.

Results

(A) Holdout (4,000 positions).

"Telling moves apart" in lopsided positions genuinely came back.

(B) The saturation gate (the 81-ply real game vs YaneuraOu depth 18).

blunders (>300cp)              8 → 4   (halved)
spread of move values at ply 72   20cp → 532cp  (26x)

Move 72 — the heart of the "all 71 moves are tied" report — was fully resolved. Where the old net had chosen a −35,281cp blunder (a gold drop into its own camp), the new net chooses the true best move.

That is the moment the resident could finally gauge how critical the critical patient is.

Being honest: ply 74 still has a blunder. But that's already a −30,000cp position where any move loses — only the difference between which way you lose in an already-lost game. What the fix cleaned up was "decided positions where the game can still turn," and that's what matters at the board.

(C) A/B matches.

vs the old NNUE       92.2% (29.5/32)
vs the hand-written   84.4% at 1000ms and 84.4% at 2000ms

This time hard-equivalent (2000ms) was cleared by measurement — the previously unverified time control that triggered the collapse in the first place.

(D) base > rank, reconfirmed.

The same 5.24M trained with a ranking loss had good holdout numbers but was still saturated at ply 72 (spread just 90cp, and it chose the big blunder).

Ranking loss, caring only that the order is right, has a side effect of crushing the absolute spread in the lopsided region — counterproductive for fixing saturation.

To cure saturation, base — which learns a big gap as a big gap — is clearly superior.

---

5.9 What failed on the learning side

Chapter 8 collects all of it, but the eval-side negatives belong here too. Five straight losses.

The KP result is worth dwelling on.

The aim was right. Real NNUE encodes the board as piece placement relative to your own king. The same "silver on 5e" becomes a different feature depending on whether your king is in a static-rook or ranging-rook castle. Given that valuing the area around the king was the hardest part of seven months of hand-written eval, this seemed like it could hardly fail.

Result: 37.5%. A losing record.

The cause was data dilution. With 1M positions split into 6 king buckets, you get under ~170k per bucket on average. Worse, positions where the king hasn't moved from its starting square make up about half of everything, so that bucket hoards the data and the others go empty.

Making features finer increases expressive power. But learning each added distinction requires enough examples for it. Make the distinctions 10x finer and you need 10x the data. Expressive power and data volume can't be raised independently.

KP features are the right direction. 1M positions was too early. The implementation is preserved for a retry once the data is there. A question of ordering, not a rejection.

---

Chapter summary

---

<a id="6"></a>

6. The opening book — the cheapest lever, and the one humans feel most

The book has a good return. Opening accuracy scales almost linearly with the number of positions, and it doesn't saturate the way the evaluation function does.

And it matters more against humans than self-play will tell you. More on why below.

---

6.1 What a book is, and why it works

Shogi openings have been studied for centuries, and a body of standard theory (joseki) says "in this position, this move is good."

A program can store that as a position → recommended move table and skip thinking from scratch every opening. It's fast, and it makes the AI's moves look like a real, human-recognizable strategy.

The catch is never mixing a bad move into the table. Once you leave the book you switch to search, so a single bad book move drives you straight down a losing line.

Both diamonds in that diagram were sources of bugs for me.

---

6.2 Dissecting "climbing silver beats it every time"

The starting report:

"Is it really stronger? The primitive climbing-silver attack always wins."

Climbing silver (bōgin) is a basic amateur plan. Losing to it every game is disqualifying.

Instead of guessing, I wrote a reproduction script: a scripted primitive climbing-silver attack thrown at the real production AI entry point, logging eval and material every ply.

Result: the medium AI lost by checkmate in 49 plies.

ply  9 ▲ 27->26  evalV3(SENTE)=-485   ← silver marching to 2f
ply 16 △ 71->62  evalV3(SENTE)=-892   ← silver about to land; AI develops an unrelated silver
ply 17 ▲ 24->23+                       ← silver promotes
ply 19 ▲ 28->23+ material=+600         ← rook takes the gold and promotes. Textbook bōgin

Three root causes fell out of the logs:

(1) The book had no defensive lines for White at all. Worse, a "skip the book when |eval| > 200" gate fired constantly on normal opening eval noise, silently disabling the defensive book exactly when it was needed.

(2) The eval's rook-file defense only looked at pawns, completely ignoring the actual bōgin mechanism (silver march plus rook stacking).

(3) Promoted pieces in the enemy camp were invisible. A "+800 for a rook or bishop in the promotion zone" term stopped matching the moment the piece promoted, because its piece-type code changes. After being broken through, the engine's eval said "roughly equal."

The fix, verified against professional commentary

The correct anti-bōgin defense was checked against professional commentary: "answer ▲2五歩 with △3三角" and "△1四歩 to deny the silver the fifth rank."

Implemented as a "climbing-silver pressure" eval term (march level × defensive shape, mirrored for both sides) plus proper book lines. Corrupted book data was repaired along the way — illegal entries like 8二→3七 annotated as "８五歩."

After the fix, both scripted bōgin plans lose a silver outright and get crushed.

---

6.3 Large eval terms are poison — and everything here is coupled

The "±1000 for promoted majors in the enemy camp" term introduced here made the whole engine much weaker.

eval-regression self-play 2W–7L (previously 5W–2L–3D)

Isolating terms via environment-variable kill switches identified it as the main culprit. Reduced to ±350, strength recovered.

A large hand-tuned term that's right in the intended position is a distortion over the full distribution of positions.

From then on, every eval change had to pass a direct match against a frozen pre-change engine.

And there was a fun side effect. The 1-ply static validator that safety-checks book moves started seeing "bishop takes a defended pawn deep in enemy camp" as a +1000 brilliant move, inflating the comparison baseline and rejecting every correct quiet book move. A SEE-lite hanging-piece correction had to be added to the validator.

Eval, book, and validator are a coupled system. Touch one, break another.

---

6.4 The safety gate was too tight — 9.4% rejected

The numbers from Chapter 2, in their proper home.

The second diamond in the diagram — "does the engine's static eval tolerate the move?" — was so tight that it rejected book moves a superhuman engine had already verified.

master   9.4%
hard     6.4%
expert   6.4%

Widening 90cp → 150cp:

The first A/B produced zero disagreements across ~1,500 book consultations — same book, same threshold on both sides, so the games never reached a position where the gate could differ. That's Chapter 1's null by construction. Re-harvesting the positions where the gate actually fires was what finally made it measurable.

---

6.5 Making the book deeper and wider, with a superhuman engine as referee

The book held 267 positions across 34 lines, but many lines simply stopped at 10–14 plies — right where the strategy takes shape.

Stopping isn't fatal (out of book you search), but the longer the book runs, the more coherent the opening and the higher the instant-answer rate.

A book move typed in by hand "because it looks good" almost always has a trap somewhere. So a local YaneuraOu became the referee.

Every candidate move is read at depth 18, MultiPV 2, and only the engine's best move — or a second-best within 50cp of it — is accepted.

Concretely: at each line's end position, ask the engine for its best move, append it, ask again. Chasing the engine's principal variation for a few plies, so every added move is one a superhuman engine calls best.

Twenty-one lines — Yagura, Bishop-Exchange, Double Wing, Gangi, Fourth-File Rook, Third-File Rook, Central Rook, Counter-Ranging-Rook, Side-Pawn-Capture, anti-bōgin and more — were each extended by 2–6 plies.

positions   267 → 354 (+87)
moves       298 → 387

Verification: a script re-checks every registered move at depth 18, MultiPV 2, exiting non-zero if any move is ≥200cp worse than the engine's best. Zero new blunders across the 87 added positions. In fact, extending one Gangi line past a borderline position resolved a pre-existing near-miss.

The only remaining flag is the Third-File-Rook ▲7八飛 — but that's simply the modern engine's verdict that Ranging Rook sits ~180cp below Static Rook. That move is the strategy's defining move (delete it and Third-File Rook vanishes) and is the engine-endorsed best try, so it stays. It's not a blunder — it's the price of a strategy choice.

Did it help — and why self-play understates it

forced opening (book disabled)   10W–10L–10D   (no regression)
book active (no forced opening)  15W–0L–15D    (never lost a game)
book probe count                 926 → 956

Something important shows up here.

In self-play both sides leave the book at the same time, so the difference barely gets a chance to appear. Self-play systematically understates the value of a book.

Humans, on the other hand, deviate from book. Same structure as Chapter 2's 1ms bug, which only a human could expose. Book improvements pay more in real games than the self-play number suggests.

---

6.6 Growing the book — reachability and memory

The shipped book is v3: 2.69MB, 173,172 positions. That is a heavily pruned slice of the new petashock book, 2.33M positions (used by the YaneuraOu team at WCSC35, with minimax values from 200M-node searches per position).

Trying to expand it hit two walls.

Wall 1: most of the source isn't reachable.

The import is a BFS from the initial position. Most of the source is middlegame and endgame positions you cannot arrive at that way.

And depth peaks around ply 47 and then declines — the source runs out. "Use all 2.33M" is not an option that exists.

Wall 2: the browser's JS heap.

The loader builds Map<hashA, Map<hashB, {moves}>>, costing ~368 bytes per position — about 20x the wire bytes — and it's loaded in two realms, the main thread and the worker.

Download size was never the issue (against 94.7MB of NNUE weights, a 10x book is +19%). The heap is the wall, not the bytes.

Verification cost was measured too: depth-18 re-checking runs at 2.79 s/position serially, 13,556 positions/hour across 14 cores. That model matched an independent measurement from months earlier to within 4%. 400k positions is a weekend. Verification is not the blocker.

Conclusion: if you want a bigger book, rewrite the loader first — keep the ArrayBuffer and binary-search the already-hash-sorted entries, so there are zero JS objects per position. That's worth doing even without a bigger book, since it removes today's 122MB.

And the measured out-of-book ply is 8.66. Verifying ply-40 positions is printing cards that never get drawn. Depth is not where the value is; filling in the shallow plies is.

---

Chapter summary

---

<a id="7"></a>

7. The endgame — don't miss mates

A missed mate is the first weakness a strong human notices. Small in Elo terms, large at the board.

---

7.1 Iterative-deepening mate search has a ceiling

The shipped mate solver was an iterative-deepening AND/OR search capped at 9 plies.

Iterative deepening re-searches the whole tree for every mate length, so cost grows sharply with depth. A long forced mate is simply out of reach inside a ~200ms probe budget.

---

7.2 df-pn — remove the ceiling with proof numbers

df-pn (depth-first proof-number search) expands the leaf that looks easiest to prove, first.

Each node carries a proof number — the minimum number of nodes that must be solved to prove "mate here" — and a disproof number for the opposite. The search digs into the branch with the smallest proof number.

The emergent behavior is that it follows narrow forcing lines to their end, rather than paying for a wide shallow frontier. Because it doesn't work breadth-first, it needs no ply cap.

Measured

1,069 labelled positions, 200ms (the production probe budget), every reported mate independently re-verified:

shipped (iterative deepening, 9-ply cap)   148 mates
df-pn   (no ply cap)                        162 mates

Fourteen more mates found on the same budget. And it costs the main search nothing — a proven mate ends the move.

---

7.3 Designing so a false mate is impossible

Reporting a mate that isn't one is far worse than missing one, because the engine plays that move and loses.

Proof numbers are heuristics, and the transposition table is keyed by a 62-bit dual hash. A "mate" verdict can never be trusted on its own.

Three layers of defense:

(1) Re-derive the proof tree.

extractProof() re-walks the tree with real move generation and returns a move only after re-deriving an explicit proof tree whose every leaf is a genuine checkmate — defender to move, in check, zero legal replies. The table is used only to order that walk. If the walk can't re-derive a proof within its budget, it returns null.

(2) Handle GHI.

GHI (Graph History Interaction) is the problem that the same position can have a different repetition verdict depending on the path taken to reach it.

The defense: a value derived from a path repetition or from the ply cap is never cached when it is a disproof. Proof trees never contain a repetition-derived value, because repetitions only ever produce disproofs. So GHI can only ever cost this solver a mate — it can never manufacture one.

(3) Exclude shogi-specific rules at the source.

• Pawn-drop mate is filtered in the generator
• Leaving your own king in check is filtered after the move is made
• Perpetual check counts as a successful defense

The design principle: returning "not a mate" too often is acceptable. Getting "this is a mate" wrong is not. Asymmetric costs deserve asymmetric design.

---

7.4 Why the search goes shallow in the endgame

A separate symptom, found in the same diagnosis. Entering the endgame, search speed drops to 27–41µs per node and the depth reached falls to 4–6 plies — far shallower than the opening and middlegame.

This is unrelated to the evaluation function. The same symptom appears with the hand-written eval.

Profiling made the cause clear:

Per-node cost in a position with a big hand:
  move generation    ~50%
  legality checking  ~49%
  evaluation         ~ 1%   ← NNUE is already fast enough

And 88–93% of generated moves are drops. With a big hand, candidate drops multiply as (empty squares, ~70) × (piece types). A combinatorial explosion.

The response is the three "read the same moves, just faster" steps from Chapter 3 — skipping the legality check on drops, one-pass generation scanning, and bit-masked check detection. Together: about −18% per node, and +8.5–9% on WASM perft.

It isn't enough to jump a full ply. Saying so plainly. But it cut cost with zero strength risk, which is the right foundation to stack on.

Extending the mate solver was deferred

I tried pushing the checks-only solver's proof length from 9 to 11 plies. On my set of endgame positions it found no additional mates — it just spent more time.

Putting something whose benefit I can't confirm into production — changing behavior for it — runs against the policy.

If it's neutral, don't ship it. That too is a decision.

---

Chapter summary

---

<a id="8"></a>

8. Everything that didn't work

This should be the chapter that saves you the most time. Two months of dead ends, readable in ten minutes.

Engineering blogs publish numerators. Here's the denominator.

---

8.1 The "make it bigger" family was almost a total loss

"More data" and "a deeper teacher" both stopped dead.

The teacher depth result is the emblematic one. The offline metric — agreement with the teacher — improved by 8.8 points, and playing strength did not move at all. The clearest instance of Chapter 1's rule.

So what did work?

distillation from book leaves +1.33% of the corpus → 66.2% (adopted)

One point three three percent. Adding 19.6% drew, but targeting only "positions right after leaving the book" and adding 1.33% passed.

Not volume. Position.

That's the one principle that ever worked on the evaluation side.

---

8.2 Richer representations also failed

The KP lesson, restated:

Make the distinctions 10x finer and, in principle, you need 10x the data. Expressive power and data volume cannot be raised independently. Raise them as a set, or you mass-produce fine-but-hollow features.

---

8.3 "Change which moves we read" lost both times

The common reason: read one line deeper and another goes shallower. Gains and losses cancel, and the total comes out neutral or negative.

By contrast, "read the same moves, just faster" is verifiable by bit-identity, so the strength risk is zero (Chapter 3).

---

8.4 Automatic tuning of the hand-written eval (Texel) also failed

Texel tuning optimizes hand-written eval weights against game outcomes. Coordinate descent over 16 weights.

48 direct games 17W–14L–17D no significant gain

The interesting part is that the reason was quantified.

The tuning set was 1,698 positions sampled from 70ms self-play. Measuring the eval→outcome signal in that set showed it was below noise — the fit error was worse than predicting a constant.

Real Texel tuning uses millions of positions. With 1,698 there is no direction to read.

As a byproduct, an unbounded hill-climb bug in the K-estimation loop was found and fixed. The harness is preserved for a retry when there's more data.

The method wasn't wrong. I hadn't supplied the data volume the method requires. Exactly the same shape as the KP result in §5.9.

---

8.5 A well-reasoned mechanistic hypothesis died too

The leading suspect for round one's NNUE defeat (19.6%):

The NNUE outputs true centipawns, but the search's margin constants were tuned for the hand-written eval's scale — roughly 3.7x true cp. So for the NNUE they're effectively 3.7x too generous and pruning goes soft.

It makes perfect sense. Isolated and A/B tested:

19.6% → 8.9% (−10.7pt, worse)

Rejected. And the interpretation is the interesting part: the loose margins had been acting as insurance, re-verifying noisy evaluations with extra search. Tighten them back and the eval noise lands directly on the pruning decisions.

Had all three hypotheses been "fixed" at once, we'd never have learned which mattered. Isolating one variable at a time is tedious, and it paid out right here.

---

8.6 Changing the training target: WDL blending

A rare case of something being refuted before it was even run, so it's worth the detail.

Standard NNUE practice blends the game result into the target alongside the teacher's evaluation (y = (1-λ)·sigmoid(cp/K) + λ·result). The training script had the parameter, wdl_mix, defaulting to 0.0. Implemented but never used.

I proposed it as an untouched, cheap lever. I was wrong for two reasons.

(1) The training data has no outcome labels at all.

Every row in the corpus is a position reached by playing one of the teacher's candidate moves — a hypothetical branch never played to a conclusion. There is no game to take a result from.

Joining against the one dataset that does have real outcomes matched 7,794 rows out of 12,508,188 — 0.062%.

(2) It had already been tested, and lost.

An older run plan had two arms:

44.4% / 42.9%

I inferred "the default is 0, so nobody tried it" from one line of a training script. Opening one config file would have told me otherwise.

"The flag defaults to 0" is not evidence that the axis is untested.

---

8.7 Time management: cutting the wait cost 40 Elo

Chapter 4 has the detail; here it is as a negative result.

thinking time   938.6ms → 633.4ms   (−32.5%)
strength        600 games, 44.2%  [40.3, 48.2]   ≈ −40 Elo

The cause was destroying the transposition table's cross-move carry-over. With the table cleared every move on both sides, the deficit vanished (44.2% → 54.7%).

"Skip the iteration you'd throw away anyway" is wrong from its premise in this engine.

---

8.8 Four times the measurement rig itself was the bug

Recounting Chapter 1's list as failures:

The instrument broke four times.

---

8.9 The one time the gate was overridden: 0–16

One entry in the inventory reads differently from the rest.

A candidate model failed every pre-registered check. Head to head against the existing production model: 0 wins, 16 losses. A shutout.

It was force-deployed anyway, on an explicit instruction, and merged. It was reverted shortly after.

The inventory does not count this as an adoption, because it went in against the gate rather than through it.

The postmortem found three compounding bugs in the training target:

1. The teacher's scores were regressed raw, with no sigmoid and no clamp
2. Mate sentinels (values around ±10⁶) contaminated 3.2% of rows and accounted for ~96% of the loss
3. The quantized output range topped out at roughly ±70–80 centipawns, meaning about 90% of the training labels were physically unrepresentable

The evaluation function had been flattened into "everything is equal." Sixteen straight losses wasn't bad luck; it was arithmetic.

The bugs aren't the interesting part. The gate was right — it said no. The override is what put a broken model in front of users.

---

Chapter summary — a taxonomy of what fails

---

<a id="9"></a>

9. Forty-five measured, three shipped — the denominator

The whole-project numbers.

---

9.1 Counting everything

The trigger was §8.6, the WDL blending. I proposed, as a promising untouched idea, something that had already been tested and lost. Without a record, I'd do that again and again.

So I counted every experiment run left on disk — 400 directories, spanning about two months — and added the work that never wrote a run directory (six browser-side changes) as an appendix.

Three out of forty-five could be proven stronger by playing games.

Five more shipped as correctness fixes, so eight changes reached production. But only three were ever demonstrated to be stronger.

---

9.2 The three that made it

None of them is about making something bigger.

• The first is a search algorithm fix
• The second is continuing from what exists rather than starting over (scratch was routed at 12.5%)
• The third is choosing which positions to learn from — it passed on 1.33% of the corpus

---

9.3 Re-reading the past verdicts against today's standard

With the A/A control from Chapter 1 in hand — identical binaries at 56.9% over 160 games, p=0.040 — the promotions were recomputed.

The distillation result is solid. It cleared by 4–5 points on two independent panels, with a concurrent A/A control at 49.4%.

The first row isn't. Its confidence interval cleared 50% by nine tenths of a point, with no A/A control on record. One-sided p = 0.0168 — only 2.4x better than the p = 0.040 obtained from an engine playing itself.

There is no evidence that model is weak. But the "+84 Elo" I wrote about it at the time is not supported to the precision I quoted.

And the sting: the same lesson had already been paid for, with 768 games. Chapter 1's Candidate A going 59.8% → 51.2% is this lesson. I learned it at 768 games, and then kept making calls at 80.

---

9.4 Only 20 of 99 had a provable pre-registered bar

One more number worth stating plainly.

experiments where a primary record proves the bar was set before measuring 20 / 99

The methodology was built while running. So the early numbers are less trustworthy than the later ones. That's in the record too.

Four experiments missed by one point. Zero were rescued. Two of them had the identical scoreline: 29W–3D–24L = 61 points, needing 62.

---

9.5 Was it worth counting?

The inventory took the better part of two weeks — opening 400 directories, reading result.json files, reconstructing which gate each candidate passed and where it fell over.

I'm glad I did it, though not mainly because I learned the success rate.

I did it because now I can tell when something has already been tried.

And there's a second reason. I can now check my own published numbers. I did not notice that the "+84 Elo" change had cleared its bar by nine tenths of a point until I recounted.

---

9.6 What changed as a result

1. No more 80-game decisions. An engine playing itself swings to 56.9% at 160 games. Eighty games carries noise the same size as the effect. The standard is now 500+. A 1,000-game run takes about five hours here — measured — so there's no excuse.

2. Always run the A/A control first. Find out what "no difference" scores before interpreting anything. If identical engines score 52.8%, a candidate at 53% means nothing.

3. Stop trusting static metrics. Agreement 87.4%→96.2% with zero strength gain. The best validation loss going 6–21 in direct play. Looking good offline and being strong on the board are different properties.

---

9.7 The numbers

---

9.8 What's next

The remaining levers, in priority order.

1. Transposition table size. Falls directly out of Chapter 4's finding that cross-move carry-over is worth ~73 Elo. Currently 2^20 ≈ 1.05M entries. If the cumulative unique positions in one game exceed that, the entries carrying that value are the first evicted. Verification is measuring occupancy; implementation is one constant. Not yet measured.

2. Expanding the book. Rewrite the loader first (368 bytes per position × 2 realms → binary search over an ArrayBuffer). That alone removes today's 122MB. Then fill in the shallow plies.

3. Correction history. Learn the systematic gap between static evaluation and search results, keyed on position features, and correct for it. A recent technique. Not implemented.

4. Measure whether SMP is actually helping. Given that helper threads are known to fail silently in some environments, I have never verified that the parallelism is paying at all.

5. Learning from real user games. Game records are now being saved, which gives a genuinely new distribution: the positions real opponents actually reach. Since the only thing that ever worked on the eval side was position selection, this is the legitimate extension of that principle.

---

Closing

This project started when its 2-dan owner said the AI was far too weak.

An instant-answer book bug excised. Every difficulty unified onto one brain. A 15x WASM port. Thinking on the opponent's clock. The opening book audited by a superhuman engine. A neural net distilled overnight from a million positions replacing, at 77.1%, an evaluation function that took seven months to hand-write. Then that net collapsing against a human on production hard — and being rebuilt with 5.24M positions until it can tell moves apart and trade blows even in decided positions.

And at last, the author admitted over the board that it "definitely got stronger."

However good the proxy metrics, if a human plays it and feels it's weak, it's weak. Conversely, the moment a human admits it's strong is the real victory.

The homework hasn't changed either. The evaluation wheel has started turning. Search is next.

<a id="appendix"></a>

Appendix

Appendix 0. Timeline and the strength genealogy

The body of this post is organized by lever, which hides the chronology. Here it is.

The strength genealogy — a chain of measurements

No absolute rating was ever measured; that's decided against humans. But every generational matchup was.

Roughly +800–1000 Elo cumulative — the scale of a beginner becoming a dan player. With the usual caveats: self-play Elo overstates strength against humans, and the absolute anchor is unknown. The real grading happens over the board.

---

Appendix: the key code

The repository is private, so here are the load-bearing pieces inline, simplified for exposition.

A. The thinking-time probe — the tool that found the instant-move bug

Replay a real game into the production AI entry point. The crucial detail: always print the thinking time.

function askAI(label: string): void {
  const t0 = Date.now();
  const move = getBestMove(k, GOTE, 'hard', moveNumber, history); // same path as production
  const ms = Date.now() - t0;
  console.log(`${label}: AI(hard) -> ${fmt(move)} (${ms}ms)`);    // ← this line is the point
}

// Output that broke the case open:
//   move 10: AI(hard) plays 82->42 (23ms)   ← "hard" has a 2-second budget. It never searched!
//   move 12: AI(hard) plays 93->94 (1ms)

B. Scripted attack plans — encoding a human plan as data

A "skip-forward" matcher lets one flat list express a branching human plan: play each step if it's legal right now, otherwise fall through to the next.

const BOGIN_PLAN: Step[] = [
  { fs: 2, fd: 7, ts: 2, td: 6 }, // P-2f
  { fs: 2, fd: 6, ts: 2, td: 5 }, // P-2e
  { fs: 3, fd: 9, ts: 3, td: 8 }, // S-3h
  { fs: 3, fd: 8, ts: 2, td: 7 }, // S-2g
  { fs: 2, fd: 7, ts: 2, td: 6 }, // S-2f
  { fs: 2, fd: 6, ts: 1, td: 5 }, // S-1e (skipped automatically if P-1d denies it)
  { fs: 2, fd: 5, ts: 2, td: 4 }, // P-2d break
];

while (planIndex < plan.length && !move) {
  const s = plan[planIndex++];
  move = legal.find((m) => matches(m, s)) ?? null; // illegal → skip to next step
}
if (!move) move = getBestMove(k, SENTE, difficulty, n, hist); // plan done → engine takes over

C. The gauntlet's automatic flags — inspect every AI reply

const problems: string[] = [];
// Answered in <200ms past the book window (12 plies) → suspected search bypass
if (ms < 200 && moveNumber > 12) problems.push(`INSTANT(${ms}ms)`);
// Right after the AI's move, one of its silver-or-better pieces hangs → suspected blunder
if (hang.value >= 900) problems.push(`HANGS(${hang.square}:${hang.value})`);
// Eval swung 800+ toward the human → suspected mistake
if (after - before > 800) problems.push(`EVAL(+${after - before})`);

These are suspicions, not verdicts — exchange sequences false-positive. The design assumes a human reviews the flags.

D. Lazy legality — the single idea that made search several times faster

const moves = generatePseudoLegalMovesPooled(k, pool[ply]); // king-safety NOT yet verified
for (const te of moves) {
  k.move(te);
  if (isKingInCheck(k, k.teban)) { k.back(te); continue; }  // ← lazy check
  legalTried++;
  k.toggleTeban();
  const score = -search(k, depth - 1, -beta, -alpha, ply + 1);
  k.toggleTeban();
  k.back(te);
}
// Mate detection: no legal move was playable AND nothing was pruned away
if (legalTried === 0 && !prunedAny) return inCheck ? -MATE + ply : 0;

With the drop optimization from Chapter 3:

const mover = k.teban;
k.move(te);
// A drop (te.from === 0) only adds a friendly piece, so it can't expose the king.
// If the king wasn't already in check, the drop is always legal → skip the scan.
if ((te.from !== 0 || parentInCheck) && isKingInCheck(k, mover)) {
  k.back(te); continue;
}

E. Quiescence partial sort — don't sort what you'll never search

// Non-check quiescence only ever searches captures/promotions.
// Swap the noisy moves to the front and insertion-sort just that prefix.
let noisyCount = 0;
for (let i = 0; i < moves.length; i++) {
  const m = moves[i];
  if (m.capture !== EMPTY || m.promote) {
    [moves[i], moves[noisyCount]] = [moves[noisyCount], m];
    noisyCount++;
  }
}
insertionSortByScore(moves, 0, noisyCount); // typically a handful of moves

F. The self-play A/B harness

The pre-change engine is registered as a frozen snapshot (v20base), and each side can get its own budget.

# New engine (200ms) vs frozen baseline (160ms) — reproducing the production time ratio
npm run shogi:match -- --engineA v20 --engineB v20base \
  --evalA v3 --evalB v3 --difficulty medium \
  --games 16 --maxTimeMsA 200 --maxTimeMsB 160 \
  --openingPlies 6 --openingMode curated --seed 61

# Lessons baked in: openingPlies >= 6 (2 plies degenerates the sample);
# final verdicts need 30+ games at production budgets

G. USI labeling — running eight YaneuraOu processes as teachers

// Per process: plain-text stdin/stdout dialogue (the USI protocol)
send('position sfen ' + sfen);
send('go depth 12');
// harvest "info ... score cp -2161 ...", finalize on "bestmove"
const cp = lastInfo.match(/ score cp (-?\d+)/)?.[1];

// Pool parallelism: 8 engines race through a shared pending array
await Promise.all(engines.map(async (engine) => {
  for (;;) {
    const i = cursor++;
    if (i >= pending.length) return;
    const res = await engine.evaluate(pending[i].sfen, depth);
    if (!res || res.bestmove === 'resign' || res.bestmove === 'win') continue;
    lines.push(JSON.stringify({ sfen: pending[i].sfen, cp: res.cp }));
  }
}));
// Measured: ~1,400 positions/second of labeling; append-only JSONL, resumable

H. Lazy accumulator diffs — turning a 14x perft slowdown into ±0%

// ✗ First attempt: update the accumulator on every makeMove
//   (search makes and immediately discards mountains of moves → perft +1348%)

// ✓ Adopted: makeMove just pushes a diff; diffs fold in only when an eval happens.
function makeMove(te: Move): void {
  applyBoard(te);                    // board updates immediately (nanoseconds)
  nnuePending.push(encodeDiff(te));  // accumulator untouched
}

function nnueEvaluate(): i32 {
  while (nnuePending.length > 0) foldDiffIntoAccumulators(nnuePending.shift());
  const acc = sideToMove === SENTE ? accSente : accGote; // both perspectives maintained
  return forwardFromAccumulator(acc);
}

function unmakeMove(te: Move): void {
  if (wasApplied(te)) unfoldDiff(te); // reverse only diffs that were folded in
  else nnuePending.pop();             // otherwise just cancel the pending diff
  revertBoard(te);
}
// Result: 6.2µs → 1.15µs per eval; depth 9–15 retained with NNUE enabled

I. Parity testing — the only weapon that matters in a port

// 50 random self-play games × up to 80 plies = 4,184 positions, JS vs WASM every ply
for (const pos of randomGamePositions) {
  assert(jsLegalMoveCount(pos) === wasmLegalMoveCount(pos)); // legal move counts
  assert(jsHash(pos) === wasmHash(pos));                     // Zobrist bit-identity
  assert(jsEval(pos) === wasmEval(pos));                     // integer-exact evaluation
}
// Search: fixed-depth runs must match best move, score, AND node counts (48/48 positions).
// Allow "close enough" and bugs slip the net. Demand bit-identity and they all get caught.

---

These excerpts reproduce every experiment in this post. The entire pipeline — teacher generation, training, quantization, WASM inference — runs on free, open-source software.