Hexagonal Chess Engine.
A chess engine for Gliński’s hexagonal chess. That article covers the shared concepts (alpha-beta search, negamax, Zobrist hashing, transposition tables, move ordering) covering only what is unique to the hex variant, I recommend you go read the standard chess engine article first.
Hex Chess Engine
This was the motivating project behind the chess engine article. As soon as I saw Grey’s video I wondered if the same core ideas of an engine could be adapted to the different board geometry and ruleset. The hex engine is a separate codebase (lives in this repo), but it shares the same overall architecture and many helper functions with the standard chess engine.
The Board
Gliński’s board is a regular hexagon with radius 5 — eleven files (a–l, skipping j) arranged symmetrically around the central f-file. The total number of cells is 91:
| Files from centre out | Cells per file |
|---|---|
| f (centre) | 11 |
| e, g | 10 |
| d, h | 9 |
| c, i | 8 |
| b, k | 7 |
| a, l (edges) | 6 |
f e __ g d __ / f1 \ __ h c __ / e \ __ / g \ __ i b __ / d \ __ / f2 \ __ / h \ __ k a __ / c \ __ / e \ __ / g \ __ / i \ __ l __ / b \ __ / d \ __ / f3 \ __ / h \ __ / k \ __ / a \ __ / c \ __ / e \ __ / g \ __ / i \ __ / l \ 1 \ __ / b \ __ / d \ __ / f4 \ __ / h \ __ / k \ __ / / a \ __ / c \ __ / e \ __ / g \ __ / i \ __ / l \ 2 \ __ / b \ __ / d \ __ / f5 \ __ / h \ __ / k \ __ / / a \ __ / c \ __ / e \ __ / g \ __ / i \ __ / l \ 3 \ __ / b \ __ / d \ __ / f6 \ __ / h \ __ / k \ __ / / a \ __ / c \ __ / e \ __ / g \ __ / i \ __ / l \ 4 \ __ / b \ __ / d \ __ / f7 \ __ / h \ __ / k \ __ / / a \ __ / c \ __ / e \ __ / g \ __ / i \ __ / l \ 5 \ __ / b \ __ / d \ __ / f8 \ __ / h \ __ / k \ __ / / a \ __ / c \ __ / e \ __ / g \ __ / i \ __ / l \ 6 \ __ / b \ __ / d \ __ / f9 \ __ / h \ __ / k \ __ / 7 \ __ / c \ __ / e \ __ / g \ __ / i \ __ / 8 \ __ / d \ __ /f10 \ __ / h \ __ / 9 \ __ / e \ __ / g \ __ / 10 \ __ /f11 \ __ / 11 \ __ /
Each cell is addressed by its file letter and its rank within that file (a1–a6, f1–f11, etc.).
Coordinate System
Standard chess maps squares to a flat integer array via rank * 8 + file. This breaks for a hex board because files have different lengths and the board is not rectangular. Instead the engine uses cube coordinates.
Every cell has three coordinates (q, r, s) satisfying q + r + s = 0, contained in the hexagon max(|q|, |r|, |s|) ≤ 5. The mapping between cube coords and algebraic names is:
q = file_index − 5 (a → −5, f → 0, l → +5) r = rank_index − minRank(q) s = −q − r
The centre cell f6 = (0, 0, 0). This approach has two concrete benefits:
- Direction arithmetic is uniform. Every direction is a constant
(dq, dr, ds)triple; adding it to any cell gives the neighbour in that direction regardless of file. - Lookup is a flat array.
CUBE_TO_INDEX[q+5][r+5][s+5]maps any valid coordinate to a linear index (or −1 if off-board) in O(1).
Direction Tables
Hex geometry has three axis pairs instead of two, giving more movement types than square chess:
| Piece type | Directions | Description |
|---|---|---|
| Rook | 6 (orthogonal) | Along cell edges: (±1, ∓1, 0) and permutations |
| Bishop | 6 (diagonal) | Through cell vertices: (±2, ∓1, ∓1) and permutations |
| Queen | 12 total | Rook + bishop |
| King | 12 total | One step in any rook or bishop direction |
| Knight | 12 leaps | All (q, r, s) permutations of (1, 2, −3) that sum to 0 |
Rook and bishop directions are orthogonal to each other in the topological sense — no rook ray overlaps a bishop ray. A queen therefore covers strictly more squares than in square chess.
Bitboard Representation
91 squares do not fit in a 64-bit integer. The engine uses a 128-bit struct:
typedef struct { uint64_t lo; // bits 0–63 (squares 0–63) uint64_t hi; // bits 64–90 (squares 64–90, 27 bits used) } Bitboard;
All piece bitboards in Board use this type. Inline helpers (bbGet, bbSet, bbClear, bbLsb, bbOr, etc.) in bitboards.h handle the two-word split transparently.
Zobrist hash stays 64 bits. The hash is a XOR accumulator, not a board representation, so it does not need 128 bits. board.hash is uint64_t.
Move Generation
The chess engine uses per-square magic bitboards: a 64-bit occupancy mask is multiplied by a magic constant to produce a table index in O(1). This breaks for a 128-bit hex board for a different reason — the centre square (f6) has 6 rook rays of 5 squares each, giving 24 relevant blocker bits. A per-square table would need 2^24 = 16 M entries × 16 bytes = 256 MB for that one square alone.
Instead the hex engine uses per-ray precomputed attack tables:
- For each (square, direction) pair there is a separate table indexed by the occupancy of that ray only.
- The longest possible ray has 10 squares; the endpoint is always attacked regardless of occupancy, so only 9 blocker bits are needed → 512 entries per ray.
- Total: 91 × 12 × 512 × 16 bytes ≈ 9 MB, built by
initMagics()in < 1 ms at startup.
Lookup per direction:
uint16_t pat = rayPattern(occ, &ROOK_RAYS[sq][d]); // extract 0..len-2 bits Bitboard atk = ROOK_RAY_ATK[sq][d][pat]; // O(1) table read
rayPattern walks up to 9 squares to pack the bits, then the table read gives the correct attack Bitboard in one memory access. The full rook attack is the OR of six such lookups; bishop and queen follow the same pattern.
The magic numbers of the standard chess engine were found offline by a random search and hardcoded in magics.c. For the hex engine the per-ray tables have no magic constants — the pattern index is extracted directly, so no search is needed. The tables are regenerated from scratch each run in under a millisecond.
Pawn Rules
Hex pawns differ from square pawns in two significant ways:
Movement direction. Each pawn moves along its own file axis: white pawns increase their rank within the file (r += 1 in cube coords), black pawns decrease it.
Capture squares. A pawn captures on the two rook-adjacent squares that flank its forward direction — not diagonally as in square chess. For white (forward = (0, +1, −1)), the capture squares are (+1, 0, −1) and (−1, +1, 0). These are the immediate neighbours that share a cell edge with the pawn and are “to the side and forward.”
Double push. Allowed only from the starting rank (rank 2 for white, second-from-top for black, detected via cube coordinates).
Promotion. When a pawn reaches the last rank of its file (the forward direction leaves the board), it promotes. All four standard promotions are offered.
En passant follows from the double push: if a pawn lands two steps ahead, the skipped square is recorded as the en passant target.
Search
The engine uses negamax alpha-beta with a transposition table — see the chess engine article for the general description. The hex-specific difference is in depth management.
Iterative Deepening
Instead of running a single alpha-beta pass at the requested depth, the engine searches depth 1, 2, 3, … up to the target:
for d = 1 .. maxDepth: eval = alphabeta(board, d, -∞, +∞) print "info depth d …"
Why the overhead is small. With effective branching factor b, the work at depth d is proportional to b^d. The total cost of all preliminary depths is the geometric sum b + b² + … + b^(N-1) = b^N × 1/(b-1) — for b = 6 this is only 20% more than the single depth-N search alone.
Why the payoff is large. At each depth d the TT stores the best move found for every visited position. At depth d+1 the move-ordering step reads the TT entry and tries that move first. A good first move almost always causes an alpha-beta cutoff on the first try, which prunes the vast majority of the subtree. In practice this halves the effective branching factor, turning hours into minutes.
The TT is a global table that persists across search() calls — no extra mechanism is needed; the warm-up is automatic.
Evaluation
The static evaluation function returns a score in centipawns, positive for white advantage.
Material
Standard values: pawn 100, knight 320, bishop 330, rook 500, queen 900, king 20000 (used as a terminal-node sentinel only).
Piece-Square Tables
initEvaluation() builds two tables, PST_W[6][91] and PST_B[6][91], from cube coordinates at startup. No hand-written arrays — values are computed geometrically so they are automatically correct for every hex square.
Key metric — hex distance from center:
dist = (|q| + |r| + |s|) / 2 ranges 0 (f6, centre) to 5 (corners/edges)
| Piece | Bonus formula |
|---|---|
| Pawn | r × 8 + (5 − |q|) × 3 — rewards advancement and central files |
| Knight | {30, 20, 10, 0, −15, −30}[dist] — strong centre bonus, rim penalty |
| Bishop | {20, 12, 6, 0, −5, −10}[dist] — moderate centre bonus |
| Rook | r × 2 + (5 − |q|) × 2 — small advancement and file bonus |
| Queen | {10, 5, 2, 0, 0, 0}[dist] — slight centre preference |
| King | {−40, −20, −5, 5, 15, 20}[dist] — edge safety (no phase detection yet) |
Black mirroring. White’s PST is built first. The black table is derived by a 180° point reflection through the board centre: (q, r, s) → (−q, −r, −s), then negating the value. This preserves the sign convention (white positive, black negative) and ensures black’s advancement and safety are evaluated symmetrically.
No Castling
Hex chess has no castling. The Board struct has no castling field, and the CASTLING table has been removed from Zobrist.
FEN Format
The engine uses a file-major hex FEN, different from standard FEN’s rank-major layout:
<file_a> / <file_b> / ... / <file_l> <side> <ep | -> <halfmoves> <fullmoves>
Each file section encodes squares from rank 1 (white’s side) upward. Piece letters and digit skip-counts work exactly as in standard FEN. Example empty board:
6/7/8/9/10/11/10/9/8/7/6 w - 0 1
The starting position constant START_HEX_FEN in fen.h defines the Gliński opening setup.
UCI Adaptation
The engine speaks a subset of UCI. Commands that differ from the chess engine article:
| Change | Detail |
|---|---|
| Square notation | Hex squares use a1–l11 instead of a1–h8. Move strings are still long algebraic: from-square + to-square [promotion], e.g. f5f6 or e1g3q. |
| No castling | O-O / O-O-O are not valid. |
| Position FEN | position fen <hex-fen> expects the file-major format above, not a standard FEN. |
display command |
Non-standard extension: prints the ASCII board to stdout (useful for debugging without a GUI). |
ASCII Display
The display command renders the board in a staggered hex layout. File letters appear above each column; the left and right margins show the square names of the outermost cells on each visual row:
a b c d e f g h i k l f1 / B \ e1 / Q \\ __ // K \ g1 d1 / N \\ __ // B \\ __ // N \ h1 ... a1 / . \\ __ // P \\ __ // . \\ __ // . \\ __ // P \\ __ // . \ l1