For AI applications
Chunks are the easy part.
Storing text and a vector beside it is the first week of the problem. What an application asks for afterwards is words and neighbours and a filter and a traversal and what the record said before somebody edited it — and every one of those in a different system is a dual write nobody can make atomic.
What an application actually asks for
| The question | What answers it |
|---|---|
| Which records mention these words? | Full-text, analysed and scored |
| Which are nearest this embedding? | Vectors — exact unless the read says APPROXIMATE |
| Rank by the words and by the meaning at once | ORDER BY FUSE — by place in each order, never by adding a score to a distance |
| Only this tenant, this document, after this date | Ordinary fields, with ordinary indexes |
| What is this connected to, and how far? | The graph, traversed from a record |
| What did it say last week? | VERSION — the store as it stood |
| What changed since I last looked? | The change feed, addressed by log position |
| Where is the file this came from? | A bucket, written in the same transaction |
| What must never appear in an answer? | The vault — sealed before the record is encoded |
Nine questions, one store — which makes the two operations that matter ordinary ones. Writing is a single transaction that either lands whole or not at all, so there is no reconciliation job because there is nothing to reconcile. Reading is one statement that holds a nearest-neighbour ordering, a full-text match and a filter together, resolved by the store rather than intersected in your application.
Around the model
Retrieval is one part of an AI product’s data. The rest is what grows around it the week after the demo: state for the tools the model calls, a cache for answers that cost money to produce, a pipeline that embeds new documents, events other parts of the product react to, places, relations, files and credentials. Each of those is usually one more system.
| What the product needs | What it is here |
|---|---|
| The conversation and what the agent learned | Records, searched by words and meaning, read AS THEY STOOD with VERSION |
| The tools' state: a session, a rate limit, a lock | A space — keys that expire, INCR, SET … IF ABSENT EXPIRE |
| A cache of expensive model answers | A space with MAX n — the least recently written go first |
| The embedding pipeline's work | A queue — a claim lapses when a worker dies, and the work comes back |
| What happened, for everything that reacts to it | A topic — each reader's position commits with what it writes here |
| Where the user is, and what is near | Geometry — radius reads served by the index, cells for a map |
| Who knows whom, what cites what | The graph — edges as records, traversed |
| The uploaded files the answers cite | A bucket, in the same transaction as the chunks cut from them |
| The keys the agent uses to call other services | The vault — sealed, and read back only by an authorised act |
The stack around the model, in one store — one binary to run, one backup, one grant model, and one transaction across all of it. Not the fastest at any one of these jobs, and it does not claim to be; most AI products never push one of them to where the difference shows, and every one of them pays for running them all. The trade, stated in full →
One read, across four of them
Definitions are statements, so there is no separate schema step and nothing to migrate before an experiment. This runs against a store in memory, and the answer below is the node’s own.
DEFINE NAMESPACE demo;
USE NAMESPACE demo;
DEFINE DATABASE app;
USE DATABASE app;
DEFINE ANALYZER english FILTERS lowercase, ascii, stemmer;
DEFINE COLLECTION memories;
DEFINE FIELD body ON memories TYPE string ANALYZER english;
DEFINE INDEX by_body ON memories FIELDS body SEARCH;
DEFINE INDEX by_at ON memories FIELDS at VECTOR euclidean;
CREATE memories:1 = { body: 'the deploy script needs the staging key', agent: 'ada', at: [0.10, 0.02] };
CREATE memories:2 = { body: 'deploying on Friday was discouraged', agent: 'ada', at: [0.14, 0.05] };
CREATE memories:3 = { body: 'the compiler flag was reverted', agent: 'bob', at: [0.90, 0.30] };
SELECT body, search::score(body, 'deploy') AS score
FROM memories
WHERE body MATCHES 'deploy' AND agent = 'ada'
ORDER BY vector::euclidean(at, [0.10, 0.02])
LIMIT 2;1: { body: 'the deploy script needs the staging key', score: 0.42873501789732965 }
2: { body: 'deploying on Friday was discouraged', score: 0.4937678576907447 }
(2 record(s), via index)Two things in that answer are worth more than the fact that it worked. deploying matched deploy because the field was declared with a stemming analyzer, so the match is over terms rather than substrings. And the record with the higher score is second, because the statement asked for nearest-neighbour order and got exactly that — the score is a projection here, not the ordering. A store that quietly re-ranked would be easier to demo and harder to trust.
The trailer says via index, which is how the node reports that the read went through an index rather than across a scan.
Search is not one thing
Full text is analysed on the way in and on the way out by the same declared chain — lowercase, ASCII folding, stemming — so the question and the index agree about what a word is. search::score is an ordinary projection you can order by, sum across fields, or filter on. MATCHES PREFIX answers a partial word for a type-ahead, and MATCHES FUZZY reaches terms within two edits, which is what makes a typo find its record instead of finding nothing. The stemmer exists for English, Russian, German, French and Spanish; a word ending in * is a scored prefix inside an ordinary query; MATCHES INFIX finds a piece of a word without an n-gram in the index; and search::explain answers what a score was made of, word by word, adding up to the score exactly.
Several fields of several tables rank as one collection with DEFINE SEARCH: one analyzer, one set of statistics, a weight per field, synonyms and stop words read when the search runs, and each table searched only for a reader its grants allow.
DEFINE ANALYZER plain FILTERS lowercase, ascii;
DEFINE COLLECTION notes;
DEFINE COLLECTION runbooks;
CREATE notes:1 = { body: 'the deploy script needs the staging key' };
CREATE runbooks:1 = { title: 'Deploying', steps: 'rotate the staging key, then release' };
DEFINE SEARCH knowledge
ON notes FIELDS body
ON runbooks FIELDS title WEIGHT 3, steps
ANALYZER plain;
SELECT search::table_name() AS source, search::score() AS score
FROM SEARCH knowledge MATCHES 'staging dep*';1: { score: 1.2715528405310115, source: 'runbooks' }
1: { score: 0.8754687373538999, source: 'notes' }
(2 record(s), via index)The runbook ranks first because Deploying is its title and the title weighs three. A misspelling is measured against the stored terms, which under a stemming analyzer are stems, so a typo that changes the stem can stay out of reach — a limit, stated rather than discovered.
Vector reads are exact by default, and stay exact until a statement writes APPROXIMATE — declaring a vector store does not silently change that. When a read does ask for the index’s own answer it can be told to try harder for that read alone, with EFFORT, rather than by moving a global dial that changes every other query with it. The consequence is the one that matters when you are measuring: the ground truth you compare recall against comes out of the same store as the answer, not a second system you are also trusting.
The store remembers what it said
An agent that answered something last Tuesday was reading a store that has since changed. VERSION reads it as it stood, so what did it know at the time is a statement rather than an archaeology project across log files.
UPDATE memories:1 SET body = 'the deploy script reads the key from the vault';
SELECT body FROM memories:1;
SELECT body FROM memories:1 VERSION 8;1: { body: 'the deploy script reads the key from the vault' }
(1 record(s), via record)
1: { body: 'the deploy script needs the staging key' }
(1 record(s), via record)It reaches back as far as reclamation has left, and the store records the oldest sequence it can still answer exactly. Below that a read is refused rather than answered from whatever happens to remain — because “there was no such record then” and “the version that said so has been removed” are different facts, and an answer that cannot tell them apart is worse than a refusal.
Catch up, rather than poll
A client can follow a table over the wire and be sent each change as it lands. The subscription is addressed by position in the log and not by a moment in time, so a process that stops resumes exactly where it stopped, and starting from 0 replays everything the log still holds. That is the difference between an agent that knows what it has already seen and one that re-reads a table on a timer and hopes the window was wide enough.
Secrets that never reach the answer
An agent works with credentials — a token for a service, a key for a deploy — and the ordinary failure is that they end up in a record, then in a log line, then in a transcript. A field declared to the vault is sealed before the record is encoded, so the index, the change feed, the replication log and every backup carry ciphertext. Reading it back is a separate, authorised act rather than a column in a SELECT *.
The four shapes still hold
None of the above replaces the retrieval argument this page used to make; it is the floor rather than the ceiling. A chunk of text, the embedding it produced, the metadata you filter on and a link back to the document it came from are four different shapes, and keeping them in four systems is the dual write.
| What you need | What it is here |
|---|---|
| The chunk | A record |
| Its embedding | An array of numbers, a field of that record |
| What you filter on | Ordinary fields, with ordinary indexes |
| Where it came from | An edge, or a file in a bucket |
No dual write
The chunk, its embedding, its source document and its edges land in one transaction, or none of them do. Every failure between two writes in a stitched stack leaves an embedding with no text or a chunk with no vector; here there is no between.
Provenance is a traversal
A citation is an edge. Which document did this answer come from is a query rather than a column convention that half the pipeline forgets to fill in — and it survives a re-chunking, because the edge is data and not a naming scheme.
Hybrid is a clause, not a pipeline
The usual shape of hybrid search is two queries against two systems and a fusion step you write and tune. Here the keyword condition and the vector ordering are parts of one statement, so the filter applies to both at once — which is the part that is easy to get wrong when it lives in only one of the two.
The originals live here too
Source files go in a bucket, in the same transaction as the chunks derived from them. The thing you cite and the thing you chunked cannot drift apart, because they were never in two places.
We built the second product on it
The list at the top of this page is not a survey of what an AI application might want. It is what TessariDB Agent Memory turned out to need — a service that holds a fleet of agents’ goals, tasks, knowledge, sessions and activity, and which started as the pile of markdown notes and JSON registries that every agent framework starts as.
Files worked until the second agent started. What replaced them needed transactions rather than lock files, search and traversal over the same records, a history that says who changed what, and a feed an agent could resume from — which is exactly the row list above, arrived at from the other direction. What Agent Memory is, and honestly what it is not yet →