More than one machine, and one history between them.

Three shapes

How it can be deployed

The same binary and the same statements produce all three. What separates them is what the catalog says, so moving between them is a declaration rather than a migration.

What you actually write

A cluster is declared, not configured

Peers are named in one transaction, because from the moment the first one commits this node is in a cluster and refuses local writes until a majority grants it a leadership — including the statements that would have described the rest of the cluster. One transaction is judged once, against a catalog that still names nobody.

BEGIN;
DEFINE REPLICA second AT 'db-2.internal:9000'
    NODE '9f2c4e1a70bb43d5a1c6e2f480937d55'
    ROLES serving, coordinating;
DEFINE REPLICA third AT 'db-3.internal:9000'
    NODE 'c81b0f37a4e94a6f8d2e5417b90c3f26'
    ROLES serving, coordinating;
COMMIT;
DEFINE NODE ROLES serving, writable, coordinating;

ok
ok
ok
ok
ok

The two halves are stored in two different places on purpose. What a machine is for is written to the machine; who its peers are is written to the catalog, and therefore travels in the backup. A node restored from another node’s backup comes up knowing every peer and nothing about itself, so it cannot inherit writable and start accepting writes the real writer never sees.

When a write counts

Acknowledged means a majority holds it

A write is readable on its leader the moment it commits. What the caller is told waits for more: on a namespace kept on more than one node, the answer comes once a majority of the range’s voters hold the write. Whichever majority elects the next leader includes one of them, so a leader lost a moment later takes nothing that was acknowledged with it.

When a leader is lost

How long the cluster waits is one statement

A leader holds a lease and stops writing before it runs out, on elapsed time rather than the clock. Peers hear each other on the link between them, so a leader that goes quiet is noticed before the slower awareness round would find it; a node with the coordinating role then stands, and a majority of the voters elects it for the next epoch. The periods that decide all of this are a fact every node must agree on, so they are a replicated row rather than a file on each machine.

DEFINE FAILOVER
    AWARENESS 10s
    COLLECTION 10s
    ROUND 1s
    CAMPAIGN 1s
    LEASE 30s;

ok

All five clauses are required, because they are checked against one another rather than one at a time. A set that would let a leader lose a leadership it could have kept, with no failure reported anywhere, is refused with the number to move named.

DEFINE FAILOVER AWARENESS 10s COLLECTION 10s ROUND 1s CAMPAIGN 5s LEASE 30s;

error: a failover policy's `campaign` is 5s against a `round` of 1s, and it
may not exceed 2s — twice the round time. The window between the moment a
leader should stand and the moment its fence shuts is exactly two round
times, so a slower cadence steps over it: the leader loses a leadership it
could have kept, and nothing reports a failure because no round was ever
attempted (at 0..75)

While nobody leads a range, that range takes no writes, and a write arriving there is told to wait rather than sent somewhere that is not yet true. The lease guard — the margin between a leader stopping and the cluster being allowed to replace it — is deliberately not settable.

One table, several machines

A table split by region, and changed while it serves

A table can name the field that decides where its records live. Each record is named after its region, so a region’s records sort together and SPLIT AT over region names gives each region a shard — a range with its own log, its own leader and its own followers. A read that fixes the region touches that one shard.

DEFINE TABLE customers (region string, name string)
    IDENTITY uuid
    PARTITION BY region
    SPLIT AT 'de', 'fr';
CREATE customers = { region: 'de', name: 'Ada' };
EXPLAIN SELECT * FROM customers WHERE region = 'de';

ok
'de:01a0f6a5-e55f-7540-9206-a2a2d8edaae9'
{ access: 'span', exact: true, shards: [2], table: 'customers' }

The map changes while the table takes writes. SPLIT AT retires the shard holding the point and mints two; MERGE SHARD joins two neighbours; ALTER REPLICA … LEADS hands a shard to another node. A new shard takes the next number the table has never used.

ALTER TABLE customers SPLIT AT 'it';
CREATE customers = { region: 'it', name: 'Grace' };
EXPLAIN SELECT * FROM customers WHERE region = 'it';

ok
'it:01a0f6a5-e560-7e5a-a3bc-6d895ccf5ebf'
{ access: 'span', exact: true, shards: [5], table: 'customers' }

No record moves when the map does: what a retired shard holds stays in its log and is still read, followed and backed up. And a table has to have been declared split for any of this to apply.

ALTER TABLE ledger SPLIT AT 'm';

error: table `ledger` is not split, so its records are filed in its database's log — declare a split table and move the records into it

A split table can keep its own shards within bounds. The store’s leader counts each shard every few seconds and splits one past the record count or write rate it was given, or merges two neighbours that fell under the floor — one change per table per pass, each the same statement an operator would have typed.

ALTER TABLE customers SPLIT AUTOMATICALLY ABOVE 100000 RECORDS
    OR 500 WRITES PER SECOND
    MERGE BELOW 20000 RECORDS;

ok

ALTER TABLE customers SPLIT AUTOMATICALLY ABOVE 1000 RECORDS MERGE BELOW 600 RECORDS;

error: table `customers` would split above 1000 records and merge the halves back below 600 — the merge bound must be under half the split bound

A table named by generated identities can spread its new records over every shard instead of the newest one: SPREAD begins each identity with a bucket taken from its random part, so these three records land in three shards.

DEFINE TABLE orders (total int) IDENTITY uuid SPREAD SPLIT AT '40', '80', 'c0';
CREATE orders = { total: 40 };
CREATE orders = { total: 41 };
CREATE orders = { total: 42 };

ok
'c5:01a1030c-1801-7eeb-8d9c-f14937c5d2c5'
'7e:01a1030c-1802-78c4-9dfc-779b52f1097e'
'37:01a1030c-1802-7b29-9f8a-d3ec4cbcca37'

One transaction, two leaders

Whole or not at all, even across machines

When two nodes lead two ranges, a transaction writing both is refused unless it ends COMMIT ACROSS LEADERS. Then the node it ran on writes the transaction’s record on the first range’s leader, has every leader prepare its part as intents held by a majority, and turns the record committed only once every part is prepared — held by a majority before the caller hears it. A failure before that aborts it, and nothing applies anywhere.

A reader sees such a transaction whole or not at all. That is atomic visibility rather than one order across the cluster: there is no clock every node shares, so two readers on two ranges are not ordered against each other. A driver that dies part-way leaves a record that lapses and is aborted, so no intent is held for ever. The cost is two majority rounds and a round trip to each leader — worth paying where records must change together, and avoided by keeping related records in one leader’s range.

Who leads what

Placed, preferred, given back and evened out

A LEADS row places a range on a node. Several rows make several candidates, and PREFERRED names the one that should win: a leader that is not preferred hands the range over once the preferred node is caught up and audible. LEADS NONE on the last placement gives the range back to the store line — carefully, because the range’s own leader still holds a lease.

Failovers leave leaderships wherever the elections put them. One clause on the failover policy asks the store’s leader to even them out: a node leading two lines more than an idle voter has its own placement moved there, at most once per two leases.

DEFINE FAILOVER
    AWARENESS 10s
    COLLECTION 10s
    ROUND 1s
    CAMPAIGN 1s
    LEASE 30s
    BALANCE LEADERSHIPS;

ok

Moving to a new release

One node at a time, writes throughout

Replace the followers first and the store’s leader last, each on its own store, waiting for each to answer before the next. One minor version at a time, forward only, and what the new release adds is turned on once the last node runs it.

The refusals

It would rather refuse than be wrong

Three of them, each quoted as the node actually writes it. They are the argument of this page: a distributed store earns trust by what it declines to do when it cannot be certain, and the wording of a refusal is the part an operator reads at three in the morning.

CREATE ledger:1 = { amount: 1 };

error: this node is in a cluster and holds no leadership: it does not
accept writes until a majority grants it one. A node that should be
leading needs the coordinating role (DEFINE NODE ROLES ... coordinating)
so that it stands for one

A cluster that has elected nobody takes no writes anywhere. That is a real availability change from a set of machines that each accepted writes the others would never see — and it is the point, because the second arrangement is faster and quietly wrong.

DEFINE NAMESPACE app;

error: `app` (at 17..20) does not say how many copies to keep, and this
store declares 1 peer(s) that could hold one: write `REPLICATION NONE`
to keep a single copy deliberately, or `REPLICATION FACTOR <n>` to keep
more

On a single machine that statement is accepted, because there is nowhere for a second copy to go. Once a peer exists the same statement is a question nobody answered, and the engine keeps never stated as a different stored value from deliberately none rather than collapsing them into a default.

SELECT * FROM ledger STALENESS 1s;

error: a staleness bound of 1s (at 31..33) is tighter than this cluster
can know about itself: the floor is 20s

The bound is a routing instruction, not a label applied afterwards: it decides which nodes are allowed to answer. A bound the cluster cannot honour is refused at the API with the floor named, rather than accepted and quietly widened — which is the version that would have made the number meaningless.

What it buys

23 things that are true today

Each of these runs in the released engine. None of them is a direction, and none is on this page because it sounded good in a sentence.

What it does not do

The limits, said first

• A table is split only if it was declared split. A table declared without SPLIT AT cannot be split later, and a split moves no record already written. The cluster splits and merges a split table’s shards only within bounds the table states, and it moves leaderships, never data: which nodes hold a range is what their subscriptions say.
• Two writers on one range give up single-copy semantics, and that is not a tuning setting. Two writes that saw neither each other compare as concurrent — a third answer beside before and after — and the second is refused with both versions named. A table may instead declare LAST WRITER WINS, and where it does, the discarded write is counted and the counter is readable, because a write that vanishes without a number is the worst of the three outcomes.
• A staleness bound has a floor, and the floor is how fast the cluster learns about itself. Asking for less is refused rather than rounded, so the number in your query is the number you get.
• A transaction across leaders is atomic, not ordered. Readers see it whole or not at all, but no clock orders two readers on two ranges against each other, and write skew is admitted as it is under snapshot isolation. It costs several times a one-leader commit.