Caro5: Data Pipeline - Day 3 🍄

Designing the Data Pipeline for a Custom Gomoku Variant with Self-Play Reinforced Learning

I'm building Caro5, a five-in-a-row game with a specific ruleset on top of standard Gomoku.

This is my first time doing this, so come along so as I go through the full data pipeline design for training an ML model via self-play reinforcement learning. This post documents the decisions, the traps I almost fell into, and the reasoning behind the final schema.

The game: what makes Caro5 different

Caro5 uses two rule layers on top of standard Gomoku:

Swap2 opening protocol. Instead of black simply placing the first stone, the game opens with a structured negotiation.

• Phase 0:The opener places 3 stones (2 black, 1 white).
• Phase 1: The chooser then decides: take black, take white, or place 2 more stones
• Phase 2: if the choice is to place 2 stones, the first player has the chance to pick a side.

This prevents first-mover advantage and is one of the fairest opening protocols in competitive Gomoku.

Caro5 win condition. You win by placing exactly 5 in a row, open on at least one end. Overlines (6 or more in a row) do not win. A sequence closed on both ends does not win either. This is stricter than both standard Gomoku (where overlines often count) and standard Caro (where overlines win). The result is a game that specifically rewards precise threat construction.

But, contrary to the vietnamese style, it's not yet an endless board.

These two rules interact in non-obvious ways when it comes to data design.

The state tensor

I had some notions regarding tensors. Been a big fan of 3Brown1Blue, and he explained it beautifully in his youtube videos.

State tensor: a representation of the game encoded as a tensor so a neural network can understand it.

Each board position fed to the network is encoded as a 13-channel tensor of shape (13, 15, 15). Think of this like a stack of 13 grids, each grid being 15 by 15 cells.

The channels

They are split into three groups:

Raw board state (Ch 0–2)

• Current player's stones
• Opponent's stones
• Last move indicator

Phase encoding (Ch 3–7)

• One-hot channels for Phase 0, Phase 1, Phase 2, and Normal play
• Current player indicator (all 1s = black to move)

Without the phase channels, the network sees identical board positions in Phase 1 and Phase 2 and has no way to know what the legal action set is.

Opening context (Ch 8–9)

• Opening stones mask: the initial 3 stones from Phase 0
• Offered stones mask: the 2 additional stones placed in Phase 2

Derived threat channels (Ch 10–12)

• Current player's open-ended threat map: cells that are part of a sequence of 3–4 with at least one open end
• Opponent's open-ended threat map
• Overline trap map: cells where placing would create a 6-in-a-row (not a win — a wasted move)

Notes taken: Channels 10–12 should be computed at training time from the board state, never stored in the game record. The network could learn them from Ch 0–1 alone, but encoding them should explicitly dramatically accelerate early training. AlphaGo encoded liberty counts ( the possible lines of threat a new piece has created ) rather than making the network discover them from scratch.

The overline trap channel is specific to Caro5. In standard Gomoku you can never "overshoot" a win. In Caro5, an overline may look like it was a lost opportunity to win, but it's a strategic liability. Opponents can deliberately extend your four into a six, neutralizing your winning threat. The network needs to see which cells are traps.

The action space

The action space is unified across all phases:

• Indices 0–224 board positions (row × 15 + col)
• Index 225 chooseBlack
• Index 226 chooseWhite
• Index 227 offerTwoMore

228 actions total.

Illegal actions are masked to -∞ before softmax. MCTS handles this naturally with masking. The alternative, separate policy heads per phase, adds complexity with marginal benefit for a network of this scale.

The game record format

After several iterations, the game record schema settled into this shape:

{
  schemaVersion: 1,
  id: string,
  seed: number,
  rules: { boardSize: 15, swap2: true, noOverlines: true },
  bots: {
    openerBotId: string,
    chooserBotId: string,
    blackBotId: string,
    whiteBotId: string
  },
  opening: {
    phase1Decision: "offer" | "black" | "white",
    phase2Decision: "black" | "white" | null,
    stones: [{
      moveNumber: number,
      x: number,
      y: number,
      player: 0 | 1,
      placedByBotId: string,
      mcts?: { policyTarget, mctsValue, temperature }
    }]
  },
  result: "win" | "draw",
  winner: 0 | 1 | null,
  moveCount: number,
  moves: [{
    moveNumber: number,
    x: number,
    y: number,
    player: 0 | 1,
    botId: string,
    mcts?: { policyTarget, mctsValue, temperature }
  }],
  durationMs: number,
  createdAt: string
}

A few decisions worth explaining, because swap2 can make the colors swap twice between the players, so it doesn’t set who’s black/white.

blackBotId / whiteBotId as ground truth color assignment. After a Swap2 opening, who is playing which color depends on Phase 1 and Phase 2 decisions. Rather than making consumers re-derive this from the decision chain, the record states it directly. player: 0 | 1 in moves is an index, never assuming index 0 means black. This took me a while to figure out.

opening is structurally separate from moves. The opening is not an alternating play. The training semantics are different from normal play and the opener's stone placement under Swap2. Keeping them in a separate block preserves the option to treat them differently without a schema change.

mcts is set and inline per move. This decision took a lot of reading. At first I was about to generate plays with some randomness and then feed it to a MCTS bot. But reading more deeply, I decided that a parallel mctsData array keyed by moveNumber would be more precise and earlier data wouldn’t be so trashy.

The MCTS policy target

The policyTarget inside each move's mcts block handles two structurally different action types:

policyTarget: {
  placements: [{ x: number, y: number, probability: number }],
  specialActions?: {
    chooseBlack?: number,
    chooseWhite?: number,
    offerTwo?: number
  }
}

Why self-play MCTS from game one

As mentioned above, is it tempting to generate 10k random games first to bootstrap the network, then introduce MCTS. This sounded like a trap. Let’s see if I took the right approach by improving mcts first.

I don’t have access to professional caro play games*, so I can only rely on self-play. Random Caro5 games have almost no meaningful threat patterns.

Starting with a randomly initialized network and MCTS from game one should be strictly better.

The cold start sequence will look like this

• Generation 0: Random network, 100–200 MCTS simulations, ~1–2k games. Just to get off the random baseline.
• Generations 1–5: Ramp games to 5k, keep simulations low (100–400). Network should start to recognize basic threats.
• Generation 6+: 800+ simulations, 5–10k games per generation. Full self-play loop.

Simulation count starts low because early networks give poor priors. Besides, extra simulations are paying for noise. Ramp up as the prior becomes meaningful.

• Yesterday I wrote about the gomocup that was happening this weekend, and since they share their games, we could be lucky to train on them!

What the MCTS data is actually for

The policy generates the dataset, and the dataset improves the policy: network policy

• → guides MCTS during self-play
  

• → MCTS plays games
  

• → games stored in dataset (with visit counts)
  

• → network trains on visit distributions
  

• → improved policy
  

• → repeat
  

The mcts.policyTarget.placements in each move record are the normalized visit counts from MCTS search at that position. This is what the policy head trains against, not the move that was played (one-hot), but the full distribution of how much search time MCTS spent on each candidate. This is substantially richer supervision: if MCTS ran 800 simulations and found 4 equally good moves, all four get meaningful probability in the target. Behavioral cloning from the single move played throws that information away.

The mctsValue is the root Q value estimate at that position, which is a better value training target than the binary game outcome alone — especially deep in a game where the outcome is many moves away.

Let's unpack.

leaf value. When the algorithm reaches a leaf node (a position not currently in the tree), it evaluates that position by either

• (1) calling the neural network's value head to get a predicted win probability, or
• (2) playing out a random rollout to a terminal state and using the actual game result (+1 for win, -1 for loss, 0 for draw) , or
• (3) a combination of both.

Once obtained, the algorithm immediately backs up this leaf value up the path to the root, updating each parent node's total value (W) and visit count (N).

Q value is the average outcome (usually win/loss/draw, scaled between -1 and +1) observed for a given node in the MCTS tree after multiple simulations. Q=W/N. At the root node, this Q value becomes the mctsValue in a network.

mctsValue it's the neural network's predicted value (win probability) for a given board position, as estimated by the MCTS algorithm after many simulations. A high Q value (close to +1) means that simulations from that position have consistently led to wins, while a low Q (close to -1) indicates likely losses.

The dataset is a snapshot of what the current policy discovered. Training on it makes the network approximate the search, which is always stronger than the raw policy. Each generation's network is better than the last, which makes MCTS better, which generates better data. The feedback loop is the algorithm.

A bug worth documenting

During schema validation, a medium-severity issue surfaced: the Swap2 Phase 1 and Phase 2 MCTS targets were being built from heuristic phase scores softmaxed into probabilities, not from actual MCTS visit counts. The field was labeled mcts.policyTarget but the data was heuristic.

The network trains on two different things under the same label: normal play positions get genuine search-derived supervision; opening decisions get heuristic supervision. The network can't distinguish them.

To be honest, I was getting crazy with the amount of detail I had to go through for swap2. I was eager to see some bots fighting and it was getting in the way. So, the general principle: if your training data has a field called mcts, it should contain MCTS data.

What's deferred

Full per-node search diagnostics — Q values, priors, candidate stats, timing — are useful for analyzing why the network makes specific bad moves and for tuning the UCB exploration constant. They'll be added as an optional debugData block, emitted only when a bot is configured with emitDebugData: true, written to separate files so they don't inflate production dataset storage.

Caro5 is still in early development. The ML pipeline described here hasn't run its first full self-play loop yet. But getting the data format right before generating thousands of games sounds better than having to redo them. Hopefully.