Hybrid search made my financial RAG worse. Here is what shipped instead.

Technical blog. The architecture write-up describes the system this page measures: the multi-layer financial data plane, structural chunking and hybrid retrieval, temporal resolution, the research orchestrator, filing change intelligence, and the evaluation and observability design. This page is about what was measured and what shipped.

TL;DR

I built an evidence-grounded research system over SEC 10-K and 10-Q filings, with SEC XBRL facts as the authority for numbers, and the evaluation harness that tells me whether a change helps: frozen corpus, source-audited benchmark, hash-pinned candidate pools, pre-registered decision rules, paired statistics and a certified LLM judge.
The first obvious upgrade, hybrid dense plus full-text retrieval with rank fusion, made answers worse (judged correctness 0.610 to 0.531). The lexical branch was issuer-blind: 82% of its top-20 chunks came from the wrong company.
Two cheap, deterministic changes cleared a pre-registered bar and were adopted: an issuer-aware selector (complete evidence 68 to 78 of 149, 10 gains and 0 regressions, p = 0.002, about 2 ms) and a deterministic XBRL numeric route in front of the generator (16/17 vs 9/17 on the rows it touches, p = 0.039).
Eleven techniques were tested and rejected with an inspectable mechanism each: BM25 and SPLADE fusion, a cross-encoder reranker, MMR, three deterministic selectors and a bounded agent. The whole programme cost about $40 in model calls.
Before replacing the served route I benchmarked it. It was indistinguishable from the challenger, so production stayed unchanged. The deployment rule allowed a change only if the challenger won, and it did not.

Every number on this page comes from a recorded run on a frozen 28-company corpus and is reported with its denominator. The detailed experiment record has every run, ledger row and identifier behind it.

The problem

An analyst question such as “Give the percentage change in operating income from FY2018 to FY2020” does not need a relevant passage. It needs the right company, consolidated operating income for both years, a consistent accounting basis, the right units and a valid calculation.

The corpus makes this hard in a specific way. 63% of the chunks are financial tables, and the same table recurs in every filing of an issuer with different numbers. A retriever that rewards passages that look relevant finds the right table and the wrong year, and the model then writes a numerically precise, well-cited, wrong answer.

The hard requirements were: every claim cited, every number traceable to a filing or an XBRL fact, say “insufficient evidence” rather than guess, and ingestion that is idempotent and safe under partial failure.

What I built

As-built architecture: Next.js workspace, HTTPS load balancer, FastAPI on Cloud Run with retrieve, research and compare-filings endpoints, one PostgreSQL plus pgvector database, Vertex AI, and an ingestion job from SEC EDGAR through content-addressed raw artifacts, parsing, chunking, embedding and atomic publish

Serving. Retrieval and answering are separate endpoints, so the workspace can show the evidence before an answer exists and the server validates every citation ID against the evidence it was given. A private session cookie is verified before any paid work starts. Filing comparison (latest versus previous 10-K or 10-Q, topic by topic) uses no model at all.

Ingestion. Raw filings are immutable, content-addressed objects written create-only. Parsing, chunking and embedding are prepared outside the transaction; publication commits sections, tables, chunks, vectors and metadata together, so a failed refresh leaves the last complete version searchable. A per-document advisory lock stops two workers publishing the same accession. Reruns skip unchanged work by content hash and by parser, chunker and embedding version.

Numbers. SEC CompanyFacts (XBRL) is the authority for reported values. A deterministic calculator does percentage change, margins, differences and CAGR. Text retrieval cites and explains; it does not compute.

Observability. OpenTelemetry behind a repository-owned facade so a missing collector can never fail a request: one server span per request with W3C propagation, model spans with GenAI semantic attributes, retrieval spans with bounded candidate diagnostics and no chunk text, Prometheus metrics with low-cardinality labels, tail-sampled traces into Phoenix, evaluation runs linked to traces, and a deterministic root-cause classifier for failed evaluation cases.

Scale. 110 companies, 3,783 filings, 283,982 chunks with 768-dimensional vectors, 167,493 XBRL facts, 1,058,558 rows and 33.2 GB in one PostgreSQL 18 database, plus 15.2 GB of raw and normalized filing artifacts.

The measurement system

Everything below depends on being able to tell a real gain from noise, label error, judge error or planner luck. This is the part of the project I would defend hardest.

Frozen inputs pinned by hash. Corpus (28 companies, 930 filings, 28,452 chunks), dataset release, dense candidate pool, selector code, judge rubric and fact store. The runner refuses to start if any pin drifts.
A benchmark that shrank on purpose. The legacy question sets had fiscal-year labels holding Q4 values and gold chunks that could not be reconstructed; an audit of 18 apparent retrieval failures found 17 annotation problems. I rebuilt from the frozen corpus, architecture-blind: 3,162 candidates, 2,171 tiered, 190 proposed, 131 after four audit rounds, because a number matching a labelled figure elsewhere does not prove which entity an unheaded table describes. Strict multi-hop went from 79 author-labelled questions to 8.
Evidence obligations, not chunk IDs. A question passes complete evidence@k only if every required fact, every period and every operand, is supported in the top k. Answer obligations (524) and evidence obligations (825) are tracked separately.
A retrieval-state contract. Found because a request for 20 candidates returned 16: on fresh connections PostgreSQL ran the approximate HNSW scan before the frozen-corpus filter, and the plan switched after five executions of the prepared statement. The evaluation retriever now applies eligibility before ranking, uses exact cosine distance and breaks ties by chunk UUID; 61 of 61 top-20 prefixes are stable across connections and plan modes.
Pre-registration with a bound approval. Hypothesis, arms, endpoint, decision rule and statistics are written and hashed before the first output exists. Exact McNemar on discordant rows, a cluster bootstrap, Holm correction, and rules that name minimum gains and maximum losses. An n = 14 result is reported as a direction, not an effect.
Gold never reaches the runtime. Retrieval, selection, routing and generation receive only the question. A 32-row reserve cannot be returned by the loader, and every run folder is scanned for it.
Cost caps on every live run, enforced from returned token counts. Unscored rows stay unscored; invalid runs are kept beside valid ones.

A confident hypothesis, a negative result, a root cause

The hypothesis was the one everybody holds: financial questions are full of exact terminology, so add PostgreSQL full-text search and fuse it with dense retrieval by reciprocal rank fusion.

Judged answer correctness fell from 0.610 to 0.531, complete evidence at rank 1 halved, and retrieval p50 went from 1.4 s to 18.9 s.

Before: dense and full-text branches fused with equal weight, 82% of the lexical top-20 from the wrong company, half of every context wrong. After: exact dense to depth 80 then an issuer-aware selector, wrong-issuer passages 76 to 17 of 400

The mechanism was not “lexical search is weak”. 82% of the lexical branch’s top-20 chunks came from the wrong company, against 19% for dense. In 39 of 61 queries the two branches shared no chunk at all, so equal-weight fusion simply interleaved them and wrong-company passages took half of every context. Fused ranks 1 and 2 tied exactly in 40 of 61 queries, and a chunk-ID string sort decided rank 1. An independent review reconstructed every fused ordering and overturned three apparent lexical wins.

What I did not do was drop lexical search and move on. On the larger benchmark I tested a real Okapi BM25 (fused 60 against 68 for dense; BM25 did recover 7 questions dense missed), then a learned sparse retriever (SPLADE fused: 38), then lexical as a candidate source feeding the selector rather than a rank competitor (2 extra rows in the pool, 0 in the context). The problem was never the strength of the lexical arm. In a corpus of near-duplicate tables that differ by fiscal period, no bag of terms can tell this year’s table from last year’s.

The fix that cleared the bar, and the one I did not ship

The change that worked is small. A deterministic selector over the dense top-80 resolves the issuer from the question text and a metadata catalog, prefers that issuer’s highest-ranked passages, and falls back conservatively on ambiguous names. Wrong-issuer passages in context fell from 76 to 17 of 400. On the 149-question development set, complete evidence@8 went from 68 to 78 with 10 gains and 0 regressions (p = 0.002), and under the certified judge the paired answer comparison was 8/14 against 5/14. It costs about 2 ms and makes no model call.

Then I benchmarked the thing I was about to replace. The served route already filters by the user’s selected ticker in SQL, so it had zero off-issuer passages. On 50 answer cases it and the new selector were indistinguishable (4 gains, 5 losses) and both beat unscoped dense. The frozen deployment rule allowed a change only if the challenger won. Production stayed as it was. The selector’s value is in the evaluated pipeline that does not depend on the UI’s selection, and wiring that pipeline into the API is the next step.

Where the loss actually is

Candidate depth showed the funnel, and the funnel decided where the next six experiments went. At dense depth 80, complete evidence is available for 119 of 149 questions. The selector keeps it for 78. The other 41 have it in the pool and lose it at selection.

Interactive · 149 development questions · frozen 28-company corpus · 8-passage budget

Where each question ends up at dense depth 80

784130

Complete in the final 8 · 78 of 149, 52.3%where E5A lands
Complete in the pool, lost at selection · 41 of 149, 27.5%39 of 41: the missing chunk is a sibling of a filing already chosen
Never complete in the 80-candidate pool · 30 of 149, 20.1%retrieval, not selection

Complete evidence@8: every retrieval arm against the same bar

Pre-registered rule for adoption: at least 5 gains, at most 1 regression, Holm-corrected p < 0.05, macro recall not lower. Select a row for the mechanism.

Reference

Lexical retrieval and rank fusion

Reranking

Selection over dense top-80

10 gains / 0 regressions vs densep = 0.002

Resolve the issuer from the question text and a metadata catalog, prefer that issuer’s highest-ranked passages, fall back conservatively on ambiguous names and short tickers. Wrong-issuer passages in context fell from 76 to 17 of 400. About 2 ms, no model call. Later confirmed under the certified judge: 8/14 against 5/14 on the paired rows.

Experiment A, 10 Oct · 0 USD in model calls

Agentic retrieval

Same data as a table
Arm	Complete evidence@8 of 149	Paired result	Verdict
Exact dense, top 8	68	—	Baseline
Okapi BM25 only	27	—	Reject
Dense + BM25, reciprocal rank fusion	60	14 gains / 22 regressions vs dense, p = 0.24	Reject
SPLADE learned sparse only	20	—	Reject
SPLADE + dense, reciprocal rank fusion	38	3 gains / 33 regressions vs dense, p = 2.3e-7	Reject
BGE cross-encoder reranker, then E5A	56	8 gains / 30 regressions vs E5A, p = 0.0005	Reject
Issuer-aware selector (E5A)	78	10 gains / 0 regressions vs dense, p = 0.002	Adopted
Coverage-aware scoring (D1)	77	2 gains / 3 regressions, p = 1.0	Reject
Period-structural promotion (G1)	78	1 gain / 1 regression, p = 1.0	Reject
Within-filing sibling by metric row (G1′)	77	2 gains / 3 regressions, p = 1.0	Reject
Statement-typed selection over a structure sidecar (H1)	79	2 gains / 1 regression, p = 1.0	Below bar
Obligation-planned sub-query retrieval (bounded agent)	65	4 gains / 17 regressions vs E5A, p = 0.007 against	Reject

The map that came out of this: topical relevance, whether from dense similarity, a cross-encoder, SPLADE expansion, BM25 terms or sub-query hits, finds the right company and the right table about three quarters of the time and cannot tell this year’s instance from last year’s. One post hoc query settled a line of three experiments: in 39 of the 41 selection losses the missing chunk is a sibling of a filing the selector had already chosen, and 36 of those chunks carry no section heading. The signal that is missing is “which statement is this chunk”, and the parser had dropped it. The next lever is representation, not ranking, and that is where the roadmap starts.

Numbers without a judge, then a judge that earns its verdicts

The judge failed first. Before the first judged experiment on the new benchmark, a 13-case conformance suite showed the LLM judge accepting a wrong liability measure, accepting “in thousands” for a USD-millions fact, crediting levels from a percentage change, and letting the answer contaminate evidence grading. The pre-registered stop rule halted every judged experiment. Retrieval work continued on a deterministic proxy.

So numbers were scored without one. A deterministic route parses the question into issuer, metric and periods, reads SEC XBRL facts, computes the shape (level, change, ratio, CAGR) and abstains on anything qualified, multi-part or unparseable. Scored by parsing the reference answers with tolerances, no model in the loop, it beat the text pipeline 7 to 1 on the rows both answered, then 8 to 0 after two named fixes. It was rejected for adoption both times because the paired cohort was nine rows against a floor of ten set before the first run. The deterministic scorer also caught the judge passing a sign error.

Then the judge was rebuilt. The model reports findings (measure, period, units, sign, value) and code makes the verdict. The evidence call never sees the answer. Oracle and adversarial providers test the rules offline; live suites rerun whole on any infrastructure failure.

Judge version	Live conformance suite	κ, answers (40 rows)	κ, evidence (111 obligations)
v4 (original)	6 of 13 failed	0.90	0.65
v5.2	13/13	0.95	0.62
v5.4 (certified)	19/19	0.95	0.68

The honest headline is the residual. Twelve evidence false accepts remain because the model reports the period and value the obligation asks for even when the passage does not contain them. A rule on a self-reported field inherits the honesty of the self-report; the next rule checks the quoted value against the passage text. Judge versions were only allowed to move verdicts the calibration packet had predicted, and v5.4 moved exactly those two on 77 rows.

And the route shipped under it. With a fixed budget for one pre-registered pipeline change judged on answer correctness, I read the route’s 19 answers against their questions before any model call and wrote four into the pre-registration as expected losses. The run rejected the route at 6 gains and 4 losses, and the four losses were those four. I fixed exactly those defects, two of them by making the route refuse, kept the rule, and ran again on fresh outputs.

	Required	Route v2 (K1)	Route v3 (K2)
Routed rows scored in both arms	at least 10	19	17
Gains	at least 5	6	8
Losses	at most 2	4	1
Exact McNemar p	under 0.05	0.75	0.039
Route correct / text pipeline correct		12 / 10	16 / 9

The text pipeline’s failures on these rows were reading and arithmetic errors on passages that contained the right figures. The pipeline configuration now carries the route as version 2 with a stable hash and a parity check that reproduces the evaluation contexts 160 of 160.

Operating it

Deployment. API and web on Cloud Run (1 vCPU, 512 MiB, scale to zero), ingestion as a Cloud Run job (2 vCPU, 4 GiB, one task, at most two workers), a global HTTPS load balancer with a managed certificate routing /api/* to the API and everything else to the web app, Cloud SQL PostgreSQL 18 with pgvector, GCS for content-addressed artifacts, secrets injected from Secret Manager, and a deploy script that refuses a dirty tree and builds through Cloud Build into Artifact Registry.
Operations. A rollout runbook with staged gates (1, 5, 20, then 100 issuers) and the measurements to capture at each, and a failure-mode table mapping SEC rate limits, fetch and parse failures, publication rollback and lost claims to the durable boundary and the operator action.
Cost policy. Cloud SQL was started only for evaluation runs and stopped after; standing burn was about $36 a month. Every live model run had a call and dollar cap enforced from returned token counts. Six of the thirteen retrieval experiments cost nothing in model calls, and the whole programme cost about $40.

What I learned

Build the measuring instrument before the system it measures. A retrieval failure report had turned into a label-quality report: 17 of 18 “failures” were annotations.
A reproducible benchmark is a systems artifact. A corpus hash is not enough if the query planner can change which candidates survive.
Read per-query diffs, not means. A mean hid that rank 1 was decided by a string sort. A net +2 can hide 20 gains and 18 losses.
In a multi-entity corpus, scoping is retrieval. Ranking only orders what scoping allows.
Measure which axis the loss is on before building a selector for it. One query would have saved an experiment.
Numbers belong to structured data, and abstention is the product. Two of the five fixes that got the route adopted were refusals.
A rubric is a wish; a rule is a guarantee; a reported field is a claim. Grade the evidence without the answer, then verify the finding against the source text.
Benchmark the thing you ship before you replace it.
A bigger candidate pool is an upper bound, not a gain. Depth 20 to 120 moved complete evidence in the pool from 97 to 124 of 149, and in the final 8 only from 75 to 78.
“I measured it and it did not earn its place” is a stronger answer than adding it.