Sourceling

A retrieval agent over a customer’s own documents, built to answer the question the thesis above left open: what happens when the rules stop being instructions to a model and become part of the system around it. The tenant filter is a database policy. An answer that fails its citation check is not sent.

What it isMy own build. Multi-tenant retrieval over documents a customer uploads

My roleProduct, architecture, code

How it answersTen agent steps. BM25 and vector search fused in one database query, then a check that blocks the answer

IsolationA Postgres row-level policy, and an application role that cannot bypass it

StackPython · FastAPI · LangGraph · PostgreSQL with pgvector and pg_search · Next.js

StatusPrototype. Runs end to end on my machine · no users, nothing deployed

The constraint that mattered most

In a regulated file, a wrong answer with a confident citation is worse than no answer, because the citation is what stops the reader checking. The thesis found where that breaks: asked directly for a program, the system logged that it must not produce one and produced one. The rule was in the prompt, and the prompt is advice.

DecisionEvery rule that matters is held outside the model. The tenant filter is a database policy, the answer is scored for faithfulness after it is written, and a score under the threshold sends the question back to retrieval.

AlternativeAsk the model for grounded answers and check a sample. One service, no local classifier, no second database role, and what a normal RAG build does.

What it costA classifier loaded into the API process, a second model call on the critique step, and a retry that can double the latency of a single question. Four of the ten steps exist only to check the other six.

What it boughtA refusal is the failure mode. A step that raises degrades the turn to a refusal instead of aborting, and an answer under the citation threshold is not sent.

The threshold is 0.80 on citation accuracy, and it is a judgement rather than a measurement. Nobody has scored this system against labelled answers, so I know what the number does and not whether it is the right number. Saying which of the two it is costs nothing and is the first thing a reviewer would ask.

How it works

Fig. 01 — the answer is written before it is checked, and the check can send it back

The bug that decided the isolation model

Row-level security was on from the first migration and it was doing nothing. The application connected as the database owner, and an owner bypasses the policies it owns. Every tenant query ran against every tenant’s rows, and a test that connected the same way passed, because it was asking the database the same question the broken product was asking.

The fix is a second role. Migration 0007 creates a login role that is not a superuser and may not bypass row-level security, forces the policies on every tenant table so ownership stops being an exemption, and leaves migrations running as the owner. The application now connects as the role that cannot turn the rule off.

It is the same shape as the rest of the page. The isolation was correct as a configuration and absent as a guarantee, and the difference only showed once something was connecting that could not opt out.

The product

The Ask screen. The question, in a bubble at the top right, asks what a record of processing activities must contain and who has to keep one. Under it a monospaced list reports each phase of the run with its result and duration: searched documents, thirteen passages found, 405 milliseconds; ranked passages, eight kept, 306 milliseconds; checked relevance, three kept, 2.5 seconds; drafted the answer, 4.3 seconds; grounding check, 10.1 seconds. The answer begins below the list and its first bullets already carry numbered chips. — Fig. 02 — the run reporting itself as it goes: passages found, passages kept after ranking, passages kept after a relevance check, then the draft and the grounding check, each with the time it took

The end of the same answer. Bullets list what the record must contain, each closing with a small round numbered chip in gold, running four to seven. A heading reads Who must keep one, followed by two bullets carrying chips eight and nine, and a paragraph about the exemption for organisations under 250 people carrying chip ten. Beneath them a row labelled SOURCES repeats chips one to ten, above copy and regenerate controls. — Fig. 03 — the end of the same answer. Every claim closes with a numbered chip, and the row beneath repeats all ten

The same answer with an evidence panel open over the right half of the window. It is headed Article 30 paragraph 1, page 1, chunk 38a281b1. A section labelled CITED TEXT quotes the sentence the chip was attached to. Below it a section labelled SURROUNDING CHUNK gives the fuller passage, and below that the source PDF is rendered, with the cited sentence highlighted in a gold band. — Fig. 04 — one chip opened. The sentence that was cited, the chunk around it, and the page it came from with that passage highlighted in the PDF

The gap analysis screen. A banner reads eight of ten requirements assessed, zero covered, then in red that two requirements could not be assessed and that this is a finding about the run rather than about the documentation, above a line reading Degradations: ungrounded_requirements, gap_verifier_unavailable. Filter buttons read All 10, Gap 5, Partial 3, Covered 0, Not assessed 2. A legend gives each verdict a shape as well as a colour. The table lists gspr-1 to gspr-5 under a Chapter I heading, each with a title, a verdict and a passage count for mdr-clinical-evaluation.pdf. — Fig. 05 — ten MDR Annex I requirements swept against the workspace: five gaps, three partial, none covered. Two could not be assessed, and the run says so in red as a finding about itself rather than about the documents

The audit trail, twenty-eight events. Each row carries a timestamp, an actor id, an event name and a detail column. Answered turns read gate=passed with generator openai/gpt-5.6-luna, or gate=failed with anthropic/claude-haiku-4-5 and with openai/gpt-5.6-luna. Between them, tenant.model_config_changed rows name the two fields that changed, and a document.deleted row names one. A note above the table states that queries are stored as a one-way hash and never as text. — Fig. 06 — the trail the same workspace wrote. Answered turns record gate=passed or gate=failed beside the model that produced them, and every change of model configuration is its own row

The same question on the gateway path. The phase list runs twice over, each pass ending in a grounding check, the second taking 18.7 seconds. Below the list there is no answer at all, only the copy and regenerate controls and empty space. — Fig. 07 — the same question on the gateway path, where the citation data is stripped in transit. The pipeline ran twice, the check rejected the answer both times, and what reaches the reader is the footprints and nothing else

For the technical reader

How it is built, enforced and checked

Architecture

Ten agent steps, each owning one model role, so the cheap model routes and grades and the expensive one only writes. Both search arms run in one Postgres, which is why the two rankings can be fused by position instead of by score.

Enforcement

The tenant filter is a database policy under a role that cannot bypass it. The citation check blocks at 0.80 and sends a failure back to retrieval. A step that raises degrades the turn to a refusal rather than dropping the connection.

Checked

Fifty-six decision records, thirteen written specifications, and 974 backend tests across a hundred and fifty files, with a golden question set per vertical. Continuous integration has still never finished a run, for a reason given below.

What is broken, and what I would do differently

Both halves of the gate could pass without checking anything, and I found the second half by running the system to take the first set of pictures. The faithfulness scorer returned a passing score when its model could not be loaded. The citation score started at 1.0 and was recalculated only when there were citations to check, so an answer carrying none scored perfectly.

That second case was not theoretical. Routing generation through a gateway rather than calling the model directly returns the answer without the citation data, so the run stored an empty citation list, a citation score of 1.0 and a passing verdict. The answer was correct and the evidence was the right evidence. The gate that exists to establish that had nothing to measure and reported the best score available.

What this page said at the time was that an empty result is not a passing result, and that a missing citation set should be treated as a failure rather than as nothing to do. That is now what happens. The prefilter that could not load was deleted rather than repaired, because the skip it bought fell back to the audit that passes vacuously. The judge runs on every turn and every tier. The recorded verdict is three-valued, so a turn that nothing assessed is stored as nothing assessed instead of being rounded up to a pass.

Fig. 07 is what the gateway path does today. The answer ships, carries no citations, and the trail row for it reads gate=not run. An ungated answer may ship; it may never look gated. Fig. 06 has both kinds of row in one table.

The two smaller cases of code describing itself generously are closed as well. The rate limiter keeps its window in Redis now, shared by every worker, and the file records the period when its comment described Redis and its code did not. The daily token budget is read at the point every provider call passes through. Ingestion runs in its own worker process rather than inside the request that started it.

What is still open is the part that matters most. The three accuracy numbers this argument rests on have never been measured against a labelled set, so the threshold remains a choice rather than a result. Continuous integration has never finished a run either: the workflows exist and fired on this branch's own pull request, and every job was refused before it started because the account's minutes are spent. Until that is paid, every green in that repository is a claim made on my own machine. The repository is private and I walk through it on a screen, including the parts above.

Happy to walk through any of these in detail, including the parts that went wrong.

Email me LinkedIn