One language over eleven engines.

Four ways of asking about a record

Four rather than one, because a store that answered only the first would send you elsewhere for the other three. They are not the whole set — shapes on the sphere, a key-value space, a file bucket and time bucketing are the rest.

Definitions are statements

There is no separate schema language and no migration tool between you and a table. DEFINE NAMESPACE, DEFINE TABLE, DEFINE INDEX, DEFINE ANALYZER and DEFINE USER are statements like any other, and they take IF NOT EXISTS — so a start-up script is re-runnable without a guard around it.

A DEFINE TABLE carries its columns and refuses a field nobody declared; DEFINE COLLECTION is the word for records whose shape is not fixed. The two are different statements rather than one statement and a flag, because they differ in what a caller may write — and a table with no columns at all is refused, since there is nothing there for it to be strict about.

DEFINE NAMESPACE prod;
USE NAMESPACE prod;
DEFINE DATABASE orders;
USE DATABASE orders;

DEFINE TABLE IF NOT EXISTS users (name string);
DEFINE FIELD email ON users TYPE string;
DEFINE INDEX IF NOT EXISTS by_email ON users FIELDS email;

Records go in one at a time at an identity you name, or several at a time at identities the store produces. The whole batch is one transaction, so a row that fails in the middle leaves none of it behind.

INSERT INTO users (name, email) VALUES
    ('grace', 'grace@example.com'),
    ('katherine', 'katherine@example.com');

SELECT name FROM users;

01a0532c115a7a108f566a5541bfa697: { name: 'grace' }
01a0532c115a7a379c8002fa776c8638: { name: 'katherine' }
(2 record(s), via scan)

USE selects where the statements after it apply, and stays in force for the rest of the connection. A connection is a session, not a single round trip.

Stream ingestion

One of those definitions runs.

A consumer is declared the same way a table is — one statement saying what to read, where the records land and under which consumer group. The node runs it, starts it again after a restart, and reports what it is doing. There is no connector to deploy beside the database and no process to remember to start.

DEFINE TOPIC CONSUMER events_in
    FROM order_events
    GROUP 'into-rows'
    INTO order_rows
    IDENTITY order_id
    MAP amount AS total
    ON FAILURE quarantine
    PARALLELISM 2;

Exactly once into the store. The node reads the topic as a member of the group, writes the records and acknowledges the messages in one transaction, so they commit together or not at all — a crash or a refused commit leaves the message unread and nothing written. ON FAILURE quarantine hands a message that cannot be applied back to the group, whose dead letter keeps it; stop halts at it and says why. It runs in every build, the published image included, and INFO FOR TOPIC CONSUMER events_in reports it.

From a Kafka broker

The same declaration reads a Kafka topic, with a broker, a format and one difference that follows from the second system — a guarantee of at least once.

DEFINE KAFKA CONSUMER orders_in
    FROM 'broker-1:9092', 'broker-2:9092'
    TOPIC 'orders'
    GROUP 'shop-orders'
    FORMAT json
    INTO orders
    IDENTITY order_id
    MAP amount AS total, placed.at AS placed_at
    ON FAILURE quarantine
    PARALLELISM 2;

Every clause but PARALLELISM is required, and ON FAILURE is required for a reason worth saying out loud: a default there would be a decision about data loss taken by whoever did not type the clause.

The guarantee, where you configure it

At-least-once, with idempotent application by record identity. Not exactly-once — and the difference is not a disclaimer, it is the thing that decides what belongs in the topic.

A consumer makes two commits into two different systems: this store’s transaction and the broker’s offset. No transaction spans both, so there is one gap where a crash is visible, and the order of those two commits decides what a crash there costs you.

The store commit goes first, which chooses duplicates over loss. A duplicate is something a record identity absorbs; a loss is not. The home page draws the loop; the table is the same thing in four rows.

The clauses, and what each one settles

Asking what it is doing, and stopping it

Two more statements, and no admin surface behind them — a consumer is inspected and dropped through the language, like everything else in the store.

INFO FOR KAFKA CONSUMERS;
INFO FOR KAFKA CONSUMER orders_in;

DROP KAFKA CONSUMER orders_in;

The report answers in three named groups. declared is what a backup carries and what every node agrees on. running is this machine only — whether it is running here, how many records it has applied, how many it quarantined, the last error it hit and where it had reached in each partition. guarantees repeats the delivery semantics in the place you are configuring them, because a guarantee documented away from its configuration is one that will be misread.

Nothing in running is stored. A persisted flag outlives the thread it describes: a node killed mid-batch would come back claiming to run a consumer that is not there. last_error survives recovery for the opposite reason — an error that cleared itself is the one nothing else will ever mention.

Three things it deliberately is not

• Not a transformation language. MAP renames and flattens, and that is the whole of it. A payload that needs reshaping is reshaped before the topic or after the table.
• Not a schema. There is no inference — not because it is hard, but because an inferred mapping is right in your test and wrong on the day a producer adds a field. A field nobody named does not land, and a producer adding one is a non-event.
• Not exactly-once. Stated on the statement, in INFO, and here.

An index changes the cost, not the answer

Adding an index does not change which records a condition returns. It changes what the question costs. That property holds for every index in the store with exactly one exception — the vector index, which cannot promise it, and therefore requires the statement to ask for approximation out loud.

SELECT * FROM points ORDER BY vector::euclidean(at, [0.1, 0.0]) LIMIT 2;
-- exact

SELECT * FROM points ORDER BY vector::euclidean(at, [0.1, 0.0]) LIMIT 2 APPROXIMATE;
-- the proximity graph; fast, and allowed to be wrong

A property worth stating plainly because it is nearly impossible to add later: once approximation is the default, nothing in the system can tell you what the right answer was.

A supplied value never becomes syntax

A term typed by a user reaches the store as a parameter, not as text spliced into a statement. That is what makes it safe to take a search term from a query string — and being remote does not give that back.

SELECT page FROM fragment WHERE text MATCHES $q;

The documentation site you can read next door is built on exactly this: its content is records in a store, and its search box is this statement.