Building a Prediction Market From Scratch (3): One Deterministic Engine and a Write-Ahead Log

The engine has a name now: Zhulong (烛龙), the torch-dragon from the Classic of Mountains and Seas. The old text says it neither eats, sleeps, nor breathes; when it opens its eyes the world is day, when it closes them, night. A being whose output depends on nothing but its own state. I couldn’t have asked for a better mascot for what I’m building — because the heart of this design is exactly that: the same log replayed always produces the same state.

Before any code, I spent real time on the architecture, because this is the decision you don’t get to take back cheaply. I’ll tell you where I landed, walk through the alternatives I rejected and why, and lay out how it grows from a thing you run with cargo run on a laptop into something production-grade — by adding, never by rebuilding.

The one fact everything follows from

Here’s the thing that, once you really accept it, settles half the argument: matching is single-writer.

A single order book has to match in a strict order — price first, then time. Two buyers at the same price must fill in the order they arrived. That’s not an implementation detail you can relax; it’s the fairness guarantee that makes the market a market. And a strict global order means you cannot have two threads matching against the same book at once without locking it into a sequence anyway. Parallelism buys you nothing on the hot path — the code that runs for every single order — it only buys you bugs.

So the instinct a lot of us reach for first — “break it into microservices so it scales” — is solving a problem the matching engine doesn’t have. One CPU core, processing commands in a tight sequential loop, does tens of thousands of matches per second. That’s already more than almost any prediction market will ever need. The moment you internalize “matching is sequential anyway,” the whole architecture gets simpler, not poorer.

So: one process, in memory, fed through a queue

If matching is sequential, give it the cleanest possible home: a single process that holds all the state in memory — order books, balances, positions — and pulls commands off a queue one at a time. No locks. No two threads arguing over the same balance. No network hop in the middle of a fill.

But in-memory state has an obvious problem: pull the plug and it’s gone. That’s where the write-ahead log comes in, and it’s the spine of the whole design:

   HTTP gateway  ──(stamp seq_id + timestamp)──►  command queue
                                                       │
                                                       ▼
                                              WAL append + fsync   ← durable HERE
                                                       │
                                                       ▼
                                         deterministic in-memory matching
                                                       │
                                                       ▼
                                               events ──┬──► reply to caller
                                                        ├──► async: Postgres mirror
                                                        └──► async: WebSocket push

One rule the engine lives by: it never reads the clock and never does I/O in the matching step. Timestamps and IDs are stamped before the command enters the queue. That’s what makes it deterministic — replay the same commands and you land in the exact same state, every time. No SystemTime::now() hiding in the matching path to make today’s replay differ from yesterday’s.

The write-ahead log is the source of truth — not the database

This is the part that felt wrong to me for years, until it didn’t: the database is not the truth. The log is.

Every command is appended to the WAL and flushed to disk with fsync — the call that forces the bytes to actually land on the disk, not just sit in a buffer, and waits for the disk to confirm — before the engine applies it. Only then does the caller get a success reply. So “your order was accepted” means “your order is already on durable storage” — not “it’s somewhere in flight.” If the process dies one microsecond later, the order survives.

The database, then, is demoted to what it’s actually good at: an asynchronous mirror for queries and history. It can lag a few milliseconds. It can hiccup. It can fall over for a minute and come back — and matching never stops, because matching doesn’t read from it. The engine’s in-memory state, rebuilt from the WAL, is authoritative; Postgres is a convenient read replica of that truth.

And crash recovery is simple, which is exactly what you want it to be: on startup, open the WAL, replay every command through the same code path, and you’re back — bit for bit. Same log, same state. No reconciliation, no guessing whether the database is ahead of or behind the engine.

Why not microservices, why not Kafka

Two roads I seriously considered and turned down.

Microservices over a message bus (an HTTP tier, a separate engine, a DB worker, all talking through Redis streams). I’ve built this shape; it’s the architecture of my own earlier version. It looks scalable. But the engine is still single-writer, so the distribution buys no matching throughput — and it costs you correctness: now you’re reasoning about at-least-once delivery, idempotency, lost replies, network reordering, and a message broker whose durability is weaker than a local fsynced log. You take on every hard distributed-systems problem to solve a scaling problem you didn’t have.

A durable replicated log like Kafka or Redpanda as the source of truth. This is genuinely good engineering — it’s how a lot of serious fintech runs — and it gives you replication and replay for free. But it’s a cluster to operate. For an MVP, and for a project whose whole pitch is “one person plus an AI, clone it and run it,” standing up a Kafka cluster is far more than this needs. The day I need multi-node ingestion, I can put a log in front of the engine without changing the engine. Not before.

The throughline: don’t pay for high availability or horizontal scale with infrastructure before you actually need them. A local, fsynced write-ahead log takes very little effort and buys you the part that matters most — production-grade durability (no acknowledged order is ever lost) today — while leaving a clean seam to add the rest later.

The journey of one order

Make it concrete. Alice sends POST /orders — buy YES at 60.

The gateway authenticates her, then does the one thing it must do before anything else: stamps the command with a seq_id (a sequence number that only ever counts up) and a timestamp. (Those live in the command, not in the engine — that’s the determinism rule.) It drops the command on the queue and waits.

The engine pulls it off. First, it serializes the command and appends it to the WAL, and fsyncs. The instant that returns, the order is durable — and only now does anything else happen. The engine reserves Alice’s funds, runs it through the book (rest it, or match it and settle both sides), and produces a list of events: OrderPlaced, maybe TradeExecuted, BalanceUpdated. It replies to Alice’s waiting request with the outcome, and — completely off the hot path — it broadcasts those same events to whoever’s listening: the Postgres writer, the WebSocket pusher.

Notice what the matching step never touched: the network, and the database. That’s the whole trick. The expensive, flaky things live on the async side of a clean line; the fast, must-be-correct thing lives alone on the other side.

The MVP is already production-grade — it just isn’t big yet

The thing I want to be honest about: the first version is single-node. But single-node is not the same as a toy. On day one it is deterministic, durable, and crash-recoverable — the properties that actually decide whether you can put real money through it. What it lacks is scale and failover, and those are things you add, not things you rebuild for.

Here’s the road, and every step is “add,” never “rewrite”:

• M0 — MVP. One process: engine + WAL + recovery + an in-process HTTP gateway + an async Postgres mirror. cargo run. Correct and durable; single node.
• M1. Split the gateway into its own stateless tier so you can run several behind a load balancer; add the Postgres read models and WebSocket push.
• M2 — HA. Stream the WAL to a hot standby; on failure, the standby replays and takes over. Tune group commit (batch many commands into a single fsync); add snapshots so recovery doesn’t replay from the beginning of time.
• M3 — scale out. When one process finally saturates, shard by market group — many engines, each single-writer over its own books. Put a durable ingress log in front only if multi-node ingestion demands it.
• M4 — on-chain. The non-custodial settlement layer (funds stay in the user’s own control, never held by the platform), Polymarket-style: USDC collateral, conditional tokens, an oracle for resolution, tiered withdrawals. That’s a whole series of its own.

The WAL is what makes this a ladder instead of a series of rewrites. Replication, snapshots, failover — they all hang off that one seam.

What’s next

That’s the blueprint: one deterministic engine, a write-ahead log as the source of truth, a database demoted to an async mirror, and a ladder from a laptop to production where every rung is additive. It has a name now — Zhulong — and from the next chapter we stop talking and start building it, from an empty repository, one small piece at a time.

We’ll begin where every trading system should: the account model. A Balance type that splits available from frozen funds and can’t, by construction, make money appear or vanish. As promised — every line in the open repo, free, no strings.