How Symmetric Are the Insides of a Go Network?

Rotate a Go board ninety degrees and nothing about the game changes. The stones sit on different points of the grid, but every group has the same liberties, every fight has the same outcome, and the best move is the same move — just rotated along with everything else. Counting rotations, mirror flips, and their combinations, there are eight ways to view any position, and all eight are equally valid.

Strong Go programs like KataGo are built on neural networks, and those networks are not told about this symmetry. Nothing in their wiring makes rotated boards equivalent. The only nudge they get is indirect: during training, each example position is shown in a random one of the eight orientations, so the network can never profit from treating one orientation specially. And that nudge works, at least on the outside. Ask a trained network where it wants to play, rotate the board, and ask again: the map of move probabilities it produces is more than 99% identical, and its single favorite move is the same point roughly 90–95% of the time. Even the rare disagreements are near-ties — two moves the network rates almost equally, with the rotation tipping the balance.

The outside behaving well tells us very little about the inside, though. A network could learn genuinely orientation-free concepts. Or it could learn eight separately-tuned copies of its knowledge, one per orientation, that happen to agree on the final answer. Between those extremes lie many strange possibilities. Which one actually happens?

This article is a tour of the answer for four of KataGo’s networks: two convolutional networks (“convnets”) of the kind that has powered Go engines since AlphaGo, and two newer transformers, the architecture behind modern language models.

The short answer: all of them keep their main high-level picture of the board almost entirely symmetric, learning to do so on their own rather than being forced to, and the small part that is not symmetric is still highly structured. The two architectures differ significantly in their lower-level computations, though: the convolutional networks break symmetry constantly and re-form it afterward, while the transformers compute their directional information early on and then do the vast majority of their work symmetrically. And, to our surprise, in one corner of the transformers the networks do something that four-fold or eight-fold symmetry cannot describe at all.

The networks

All four networks follow the same overall plan, shown below. For every one of the 361 points on the board, the network maintains a large set of numbers that describe what it currently thinks about that point — 512 numbers per point in some of these models, 768 in others. You can picture this as a stack of feature maps: each of the 512 “channels” is one feature, with one value per board point. One channel might track something like “how safe is the group here,” another “is this area likely to become territory.” (Real learned features are rarely that clean, but that is the flavor.)

This shared stack of feature maps is called the trunk. It is the network’s working memory of the position, and it flows through the whole network from input to output. The computation itself happens in a series of blocks: each block reads the trunk, computes something, and adds its results back in. After the last block, output layers read the trunk one final time to produce the network’s judgments: a policy (how good each possible move looks) and a value (who is winning, by how much).

The two architectures differ in what a block does with the trunk, and the difference comes down to a single question: how does information move across the board?

A convolutional block is built from 3×3 convolutions. In these networks each block first shrinks the trunk’s channels down to a cheaper working width with a 1×1 convolution (a re-mix of each point’s own features, no movement across the board), runs two small skip-connected inner blocks of two 3×3 convolutions each, and expands back:

A transformer block has the same shrink-and-expand shell, but its three inner blocks each contain an attention layer followed by an MLP (a small per-point neural network). The attention layers move information across the board; the MLPs process what arrived:

Here are the specific networks in this study. All four are released KataGo networks that people play against and analyze with; the transformers are the newest, released this month. They were trained mostly on the same large pool of KataGo self-play games (full names and training notes are in the appendix):

network	architecture	blocks	trunk channels	parameters	approx Elo, equal evals/move	approx Elo, equal time/move*
b28	convnet	28	512	73M	0 (anchor)	0 (anchor)
b40	convnet	40	768	234M	+400	+70
b10	transformer	10	512	29M	+50	+120
b11	transformer	11	768	71M	+460	+300

Table 1. The four networks, named here by their block counts. *The equal-time column is especially approximate: it accounts for the larger networks being slower per evaluation, but was measured on one particular GPU and can vary greatly with hardware and with future optimization of each architecture’s code.

How do we measure the symmetry in a network?

More specifically: how do we tell when a concept the network is computing is symmetric, when the board position itself isn’t? The measurement is simple:

• Take a position, run it through the network, and record one channel’s feature map.
• Rotate the position and run it through again.
• Rotate the new feature map back, and compare it to the original.

If the two maps match, the feature doesn’t care which way the board was facing when it was computed. The technical word for this is equivariant: the feature turns with the board. Here is what that looks like for a real channel:

Why is this worth measuring? Because a feature can be genuinely useful and still fail this test completely. Take a simple hypothetical feature: “is there a black stone just left of me?” Run the same three steps on it:

This test is exactly what separates the two kinds of Go knowledge a feature can carry. Some concepts are the same from any direction: “this is an eye,” “this point is solid territory,” “this group is weak.” An equivariant feature can represent these once, in a form that turns with the board. Other concepts are inherently directional: “this territory is open from the left,” “this is a ladder running toward the upper right,” “this move makes a knight’s move toward the left side.” If the network wants to recognize a directional pattern wherever and however it appears, it has to spend capacity on each orientation.

Before looking inside, you could reasonably guess either way about what a network learns. Maybe training finds orientation-free encodings of almost everything. Or maybe the network stores every tactical shape separately in every orientation — eight copies of its Go knowledge that merely agree at the output. What we measure, feature by feature, is where the networks actually landed between those two extremes.

Finding 1: the trunk is almost entirely symmetric, in every network

Nothing forces any individual channel to be equivariant. Training only ever grades the network’s final answers. Yet both architectures converge on trunks that are overwhelmingly symmetric, one feature at a time. And here the two architectures can be compared directly: both have trunks of the same kind, measured with the same score:

The same picture holds at every depth we measured: the trunk stays 92–94% cleanly equivariant at every depth in the b28s and the transformers, and only slightly looser in b40, whose last dozen blocks dip to 87–92% — an early hint of a b40 oddity covered later. Just as striking is the shape of the histogram: sharply two-humped, with a gap in the middle. Features are either almost exactly equivariant or unmistakably not. (One reading note: since each channel’s score is its worst across seven views, a feature with no consistent relationship to its transformed self at all would land somewhat below zero — so the far-left bars mean “unrelated or sign-flipped in at least one view,” not necessarily anything more exotic.) The network does not do symmetry by halves, which means the un-symmetric minority is not noise or sloppiness. Classifying each channel by which of the eight views it does respect makes the structure plainer:

symmetry class of trunk channel	b28	b40	b10	b11
respects all 8 views (fully equivariant)	482	705	470	718
respects exactly one mirror (directional)	23	31	30	35
respects half the views (e.g. both axis flips)	5	5	9	10
respects none	2	27	3	5
total trunk channels	512	768	512	768
of the non-equivariant: have an orientation partner	26/30	21/63	40/42	48/50

Table 2. Trunk channels classified by the largest set of views they respect (score ≥ 0.9), measured two-thirds of the way through each network. Most non-equivariant channels respect exactly one mirror line — the signature of a feature aligned with one direction — and most have a “partner”: another channel that matches them under some rotation or flip, the subject of the next section. A few are one half of a plus/minus pair (2 in b28, 4 in b11). The odd column is b40, with 27 channels respecting no view and few partners; that is the “haze” covered in its own section below.

Finding 2: the exceptions come in matched sets

Here are four of the b28 convnet’s direction-sensitive trunk channels. Each one, on its own, fails the symmetry test badly. Together, they pass it perfectly:

This is exactly the situation Figure 7 anticipated. Whatever these four channels compute — and without more interpretability work we can’t say precisely what it is — we can see directly that the network computes it four times, once per direction, just as it would have to if the underlying concept is directional and it wants the concept everywhere. The four channels cycling into each other under rotation is the fingerprint of that arrangement, visible even without understanding the feature itself.

Mathematicians would call such a family an orbit of the symmetry group, and nearly all of the non-equivariant features in these networks organize this way — with one exception, b40’s unpartnered “haze,” covered in its own section below: quartets that cycle under rotation, mirror-image pairs, and a few plus/minus pairs where one channel is the negative of another. In each case the set as a whole is closed — rotating the board just permutes who plays which role — so the trunk’s directional vocabulary, about 6–8% of it in every model we measured, is symmetric as a set even though its members are not.

Finding 3: convnets break symmetry constantly, transformers don’t

So far the two architectures look alike: a mostly-symmetric trunk with a small, orbit-structured directional vocabulary. The difference appears when we stop looking at the trunk between blocks and look inside the blocks, at the outputs of the layers that move information: every 3×3 convolution in the convnet, and every attention and MLP layer in the transformer. (For the transformer we checked both the attention layers’ output channels and the attention patterns themselves; both give the same picture.)

Inside a typical convolutional block, symmetry is broken and then repaired, twice. At the deepest layers, around half of all channels are direction-specific — and these are not weak stragglers; they carry about half of the total signal. The block then recombines those oriented pieces and hands a symmetric result back to the trunk. Seen as a histogram, the deepest layers of the two architectures separate sharply:

These oriented inner features are the same matched sets we saw in the trunk, in much larger numbers: about 91% of the direction-specific first-convolution channels have a partner channel computing the same thing in another orientation, with rotation-quartets everywhere. The block pays two or four times to know each of these features in the orientations it needs.

Why would a convolutional network do this? Here is the intuition. A convnet’s only tool for moving information is the 3×3 convolution, and a convolution that moves information is inherently oriented: “copy the signal from the point to my left” is a direction-specific operation, and so is the feature it produces. When the specific shape of a pattern matters, the direction a message came from matters too:

Suppose the network wants a feature that says “playing here would form a one-point jump that helps a group in trouble.” The group is two points away, but a 3×3 convolution only sees one step, so the news must be relayed: one layer computes “the group just left of me is in trouble,” and the next layer, reading that feature from its left neighbor, concludes “trouble two to my left — a move here jumps to its rescue.”

Now imagine the relay instead carried only an orientation-free version of that intermediate feature — “a group next to me is in trouble,” with no direction attached. Then the escape point would know only that trouble is somewhere within two steps: it could not tell a one-point jump from a diagonal move or a move two spaces past the group. To route information across the board, when the exact shape matters, the convnet has to keep track of direction — so it builds directional features, in the orientations it needs, in nearly every block, and re-symmetrizes before writing back to the trunk. (This is an intuition, not a proof: a convnet could likely get by with less directional replication than this, just less efficiently. What the measurements say is that heavy replication is the solution these networks actually learned.)

The transformer mostly escapes this. Attention moves information by relative position and content in one step, and an attention rule like “look at the stones in my own group” is automatically equivariant — so a transformer can do most of its long-range routing without ever leaving symmetric form. Not all of it, though: sharp, short-range directional reads are still useful (the next section shows the transformer’s version of them). The interesting part is where that directional machinery lives:

This is the cleanest architectural contrast in the study. The convnet re-derives its oriented machinery locally, inside nearly every block: in its biggest layers, a third to a half of all channels go to features that exist in two or four rotated copies. The transformer builds direction once, in one dedicated block near the input, writes the results into the trunk’s small directional vocabulary, and lets every later block read the answer instead of recomputing it. If you have wondered how these transformers match convnets several times their size (Table 1), this is one concrete piece of the answer: they don’t need extra copies of everything. It is probably not the whole answer — just moving information across the board in one step, rather than relaying it through many convolution layers, is likely a saving by itself, and the architecture may help in other ways we haven’t measured, such as making some patterns easier to learn in the first place. The saving is not free, either: an attention layer takes considerably more computation than a 3×3 convolution, so some of what the transformer saves in parameters it pays back in compute per evaluation.

A tour of the direction block

The transformer’s one symmetry-breaking block deserves a visit, because its attention heads are unusually easy to read. First, the head-level version of Figure 8’s two-humped histogram — this one uses the attention-overlap score rather than the channel score:

One caution about reading the pictures that follow: an attention head is not a fixed stencil. Each head scores points by learned rules that usually depend on what is on the board — the sharp heads below, for instance, land on stones at about 1.2× their share of the board, and another head of the same layer attends almost exclusively to empty points. The panels show each head’s average attention, relative to the point doing the looking, across all queries and positions. For these sharp heads the average is honest: their per-query attention really is this concentrated. Here are eight of the twelve heads in the first attention layer of block 2:

Eight sharp heads, one per neighbor direction. These are the transformer’s answer to the 3×3 convolution, built in this one block rather than once per block. They form two clean orbits of four (the compass directions and the diagonals), exactly like the convnet’s channel quartets. Two more heads of the same layer are a subtler matched pair:

On a real position you can watch these heads work:

These are only the easiest heads to read. The appendix has a gallery of other heads with readable jobs — including a pair that traces a Go player’s notion of a “group” across the board, a head that marks the network’s own candidate moves, a head found to carry ladder information, and heads that watch an active ko.

Finding 4: the transformer learns a hexagonally symmetric feature!

Everything so far fits neatly inside the board’s eight-fold symmetry: features are either symmetric alone or symmetric in matched sets of two or four. The last finding does not fit, and it is our favorite.

The right panel of Figure 17 showed a “broad lobe” head: one that gathers information from a wide region lying toward one general direction. Block 4 of the b11 transformer contains six of these in a single attention layer. The lobes are genuinely blurry — each one softly covers a third of the board or so — so pinning down what they are doing takes some care. For each head we measured the average direction of its attention (the attention-weighted compass heading, excluding the point itself), separately on each of the 100 positions. Three facts came out:

• The directions are sharp even though the lobes are not. From position to position, each head’s measured direction wanders by only about 2°, so its mean direction is pinned down to a fraction of a degree.
• The six directions are spaced 60° apart — gaps of 60°, 61°, 59°, 62°, 59°, 60°. The centers of mass of the six lobes (marked × below) sit about 1.1 points from the query, and themselves form a near-regular hexagon (gaps 58–62°). This is the cleaner of block 4’s two hexagon layers; the other’s spacing is rougher, with gaps from 52° to 71°.
• The blur is real, not an averaging artifact. A typical single query’s attention from these heads is nearly as spread out as the averaged picture (appendix has the numbers), and the heads are close to indifferent about content, attending to stones and empty points near their base rates. They really do implement “softly gather everything lying that way.”

A hexagon cannot respect the symmetry of a square grid. Rotate the board 90° and each lobe lands where no lobe exists, 30° from the nearest one. In the measurements this produces a signature that at first looks impossible: under a 90° rotation each lobe’s best match is a different head, and following best matches takes three hops to return home. That cannot be the work of a genuine 90° symmetry: four 90° rotations bring the board back to where it started, so heads that truly turned with the board would return home in one, two, or four hops — never three. Those impossible three-hop cycles showed up in the numbers before we ever plotted a lobe; they were the first sign of something strange.

Why six? Here we can offer only an intuition, not an answer: six broad, overlapping lobes are an efficient way to cover all directions — 60° spacing is the honeycomb answer, not the checkerboard answer — and the lobes are so wide that being 30° off after a rotation costs almost nothing, so the network accepts a small, cheap violation of the symmetry in exchange for covering every direction with fewer heads. But we do not know what information these heads actually carry, why that information deserves this many heads in this spatial arrangement, or why six-fold coverage beats five-fold or some other structure. What we do know is that it is not a fluke of one training run. The smaller b10 transformer, trained separately, built the same six-direction system in a split form — three lobes per layer, 120° apart, with consecutive layers offset to cover six directions between them:

Scale, training, and one open puzzle

Across the four networks, the architectural signature is stable. One check remains: all four were trained at least partly with a newer optimizer (Muon), so in principle the training method, rather than the architecture, could be shaping some of what we see. As a control we added a fifth network: a released sibling of the b28 convnet trained purely with SGD, the classic optimizer used for most of KataGo’s history:

The SGD control settles one question cleanly: the optimizer makes essentially no difference (36.4% vs 36.7% direction-specific inner channels, and a trunk symmetry census that matches b28’s count for count). But it sharpens another. The b40 convnet — the largest network, and currently the main engine of KataGo’s self-play training — shows something none of the other four do: on top of the usual crisp directional features, a large diffuse population of mildly asymmetric channels, sitting in the 0.8–0.9 score band that is nearly empty in every other network:

This haze is not explained by anything we tested: not the optimizer alone (the SGD-trained b28 is exactly as clean as the Muon one, though the control is at b28’s size, not b40’s), and not the convolutional architecture itself (the b28s are clean). It travels with the model itself — b40’s final move choices are also a bit less consistent across orientations (the same top move 89% of the time, versus 93% and 95% for the b28 convnet and the b11 transformer, the two others we measured) — though we cannot directly tie that behavior to the haze. Model size, learning-rate schedule, and other details of b40’s training history all differ, and with training runs this large, clean controlled experiments are mostly out of reach. For now it stands as an open question.

What to take away

The networks learn the symmetry on their own, feature by feature. With no architectural constraint at all, only symmetry-augmented training data, all five networks converge on internals where each feature is either cleanly symmetric or purposefully directional, with almost nothing in between — and the directional features organize into matched sets that are symmetric collectively. The worry that a network might store every shape separately in every orientation turns out to be mostly wrong — but not entirely: the orbit families are exactly that, kept to a small, organized corner of the network.

The two architectures break symmetry in very different places. The convnet, whose only spatial operation is an oriented 3×3 stencil, breaks and repairs symmetry inside nearly every block, paying for rotated copies of the same feature throughout. The transformer factors nearly all of it into one small block near the input and shares the results through the trunk. Same game, same training signal, very different ways of spending capacity — and a concrete, measurable candidate for part of why these transformers get more strength per parameter.

A network can even learn a symmetry the board doesn’t have. The hexagon is a small, clear example of a network setting aside the geometry of its own board because a different geometry covers the problem more efficiently — found independently, in two forms, by two separately trained models. Whatever story we tell about what a network ought to learn, it is worth checking what it actually built.