Deslop: A Clone Census Runs the Compiler Cleanup

The compiler is roughly 115,000 lines of Zig now, and none of it was written in one sitting. It accreted — feature by feature, pass by pass — and a codebase built that way doesn’t just contain duplication, it produces it. Every new emitter path inherits the last one’s prelude by copy-paste. Every new checker inherits the last one’s validation loop. The copies drift, and after enough drift nobody can tell you which copy is load-bearing.

That suspicion had been sitting in the air for months. On September 20th it became a tool.

The census

tools/deslop.zig is 253 lines. For every AST node spanning at least 48 tokens, it hashes the normalized token-tag sequence of the span:

all identifiers share a token tag already, string/char literals fold to STR, numbers to NUM, doc comments drop out — so two subtrees that differ only in names and literals hash identically.

Equal hashes form a cluster; weight = tokens × (members − 1) × log2(1 + bytes) floats the expensive ones to the top. The first run parsed 106 files into ~353,000 nodes and reported 698 clusters.

Two properties matter more than the count. The census doesn’t compare text — it fingerprints structure, so a clone renamed, re-indented, and re-literalized still hashes equal, and two fragments that share only a surface shape get read by a human before anything is merged. And it reruns: after every consolidation the census re-parses the tree and re-ranks what’s left. The board is not a report you finish; it’s a standing measurement of a living tree.

The ruling that makes the work safe

The first cluster the census surfaced was a trap wearing a safe costume. matchGlob existed in three copies across ast.zig, shape_checker.zig, and transform_pass_runner.zig — and a wired, tested, documented canonical sat right there in glob_pattern_matcher. Route the copies to the canonical, delete them, move on.

Reading the bodies first caught what the fingerprints couldn’t: the canonical was a subset. The stray copies supported *suffix and prefix.*.suffix; matchSegment returned false for both. The clean merge would have silently deleted two wildcard forms from the language’s transform matching.

So the discipline isn’t “pick a winner.” It’s: establish which member is the superset — not which sits in the nicest file — lift the canonical to the superset, then delete the copies. That ruling is written down in the repo (concepts/frag-dedup-is-lift-then-delete.md), and it kept earning extensions all day: drift hides in omitted struct fields (four clone sites dropped location and is_transformed_subtree, so the defaults quietly removed a checker exemption); the shared tail often already exists and the fix is routing, not extraction; and in emitter families the liftable unit is the analysis prelude, not the function.

A clone census without this ruling is a bug generator. With it, the loop is mechanical enough to run a hundred times.

The day of deletion

Five days after the tool landed, one session ran the loop 101 times: refresh census → take the highest non-test cluster → read the members semantically → extract the narrow helper → build → focused regression pins + the 425-test unit suite → commit → recensus.

The receipts:

• Sixteen errdefer cleanup sites in ast_functional.zig collapsed to two helpers — −66 lines.
• The omitted-?T argument rule, re-implemented in three emit paths, became one emitMissingOptionalArgs.
• A decimal-index scan handwritten in three files became codegen_utils.scanDecimalIndex.
• Two copies of an indent buffer that was built and never read — dead code the census made visible — deleted outright.
• A type-prefix strip loop existed in five places; the census flagged two. Reading the pattern surfaced the strip-all variant hiding from the fingerprint.

And the deliberate non-merges, which are the point as much as the merges: test fixtures top the board by weight and stay — repetition is how tests are written. The struct_literal/lexer twin is a declared intentional copy. The serializer dialects and a three-site emitter coincidence survived review because they share a skeleton, not a meaning. The fingerprint ranks candidates; a reader decides.

Repeatable, or a heroic afternoon?

The honest test of a cleanup is whether the next session can do it again without remembering anything. This one can, and that’s the part of the day worth keeping:

• The instrument is committed — tools/deslop.zig, 253 lines, no state, rerunnable against whatever the tree looks like next month.
• The judgment calls are committed — the concept doc carries the superset ruling and every extension earned, so the next run doesn’t re-learn the matchGlob trap.
• The verification is committed — every consolidation ran the same gate: build, focused regression pins, the unit suite, git-wall, and the invariant judge on the staged diff.

And the flywheel property: each pass makes the board more honest. With the real-code slop cleared off the top, today’s census is mostly test scaffolding and declared twins — which is itself a measurement. A month of accretion from now, the same command re-ranks whatever grew back. The cleanup didn’t close a project; it installed a gauge.

Does the judge belong in the loop?

The obvious next step was to ask whether the commit gate’s judge — Jev, the typesafe/jev-1.13 model behind odds:feels — could do the triage too: feed it a cluster, get back “mergeable” or “leave it.” So we ran it: the top 30 clusters, two yes/no questions each through the same binary the gate drives.

The scaffolding filter is genuinely good. “Is this test scaffolding?” came back confidently right almost everywhere — including test blocks buried inside non-test files, which means it was reading semantics, not filenames. As a cheap negative filter over a fresh board, it earns a seat.

The merge verdict is where it gets interesting. Its highest-confidence “yes, mergeable” — p=0.95 — landed on struct_literal.punnableName versus lexer.punnableName: the declared intentional twin. The two functions are identical, and the reason they must stay separate is written one line above the extracted span:

/// Mirrors `lexer.punnableName` (kept local so this module stays free of
/// a lexer import; the two must stay in lockstep).

The judge never saw it — the harness sent the function body, not the doc comment. But that’s the point, not a bug in the harness: the constraint that forbids the merge lives outside the fragment. Sometimes it’s a doc comment one line up; sometimes it’s a module boundary or a contract the code never spells out. The superset question — the ruling that makes this work safe — needs whole-codebase context no pair of fragments carries. Inference can nominate merges; it can’t ratify them. That stays with a reader, backed by tests.

The gate caught the one thing a diff can’t

The last commit of the day was the ceremony — publishing the board — and it failed in a way worth keeping. The staged diff regenerated the docs, and the regenerated docs carried prose using the word “event” — the language’s old name for the tor. The vocabulary had been swept by hand weeks earlier; the source test comments still said “event”, so regeneration faithfully reverted the sweep.

git-wall passed. The deterministic checks passed. What caught it was the commit gate’s judgment layer: the staged diff goes to a compiled gate binary that asks a model — typesafe/jev-1.13 via OpenRouter — whether the diff weakens each declared invariant. The the-construct-is-a-tor row came back VIOLATION, p=0.57. Commit blocked.

Sweep the comments, regenerate, re-judge: CLEAN, p=0.99. It shipped as 0a601ba28 and the board went live.

That’s the loop closing on itself: a tool found the candidates, an agent read semantics and did the lifting, and a model at the gate audited the prose no deterministic check could read. The instruments that did the work are the same ones that measured it — which is what makes this a flywheel and not a heroic afternoon.