Technical blog · 2026 · MarketLane Research
The architecture for programmable financial research
This is the architecture MarketLane Research is built toward: the data plane, the retrieval and reasoning layers, the ingestion state machine and the evaluation and observability design. The measured results, what shipped and what was rejected, are in the case study.
How to read this page. This is the design document: the system as it is meant to work end to end, including components that are specified but not yet wired to a public route. For the experimental programme, the numbers and the two changes that cleared a measured bar, read the case study; for every run, ledger row and decision behind it, the detailed experiment record.
MarketLane Research is an AI-native equity intelligence platform designed to turn fragmented financial information into structured, traceable research.
The system combines:
- SEC regulatory filings
- financial statements and XBRL facts
- investor materials
- market data
- multi-period financial analysis
- document-level retrieval
- deterministic financial calculations
- temporal reasoning
- grounded LLM synthesis
into a single research interface capable of answering questions that normally require an analyst to move repeatedly between filings, spreadsheets, financial databases, and company disclosures.
The core engineering problem is not simply document search.
It is building a system that can determine:
which company → which reporting period → which filing → which section → which financial fact → which calculation → which supporting evidence
before generating an answer.
The Problem
Traditional financial research is fragmented across multiple systems.
A single question such as:
How has Microsoft’s operating margin changed over the last four quarters, and what did management say drove the change?
can require:
- identifying the correct fiscal periods;
- retrieving revenue and operating income for each quarter;
- normalizing reported financial concepts;
- calculating margins consistently;
- locating the relevant MD&A commentary across several filings;
- separating current-period commentary from historical discussion;
- comparing changes across quarters;
- linking conclusions back to primary-source evidence.
Generic RAG architectures are poorly suited to this problem.
A vector database can retrieve semantically similar text, but it cannot reliably provide:
- exact financial numbers;
- fiscal-period normalization;
- deterministic arithmetic;
- latest/previous filing resolution;
- multi-document comparison;
- evidence lineage;
- structured table reasoning;
- financial metric consistency.
MarketLane Research therefore treats financial intelligence as a multi-source evidence orchestration problem, rather than a conventional chatbot problem.
System Architecture
USER QUERY
│
▼
Query Understanding
│
┌────────────────┼─────────────────┐
│ │ │
▼ ▼ ▼
Entity/Period Intent Metric Detection
Resolution Classification
│ │ │
└────────────────┼─────────────────┘
▼
Research Planner
│
┌─────────────────────┼─────────────────────┐
│ │ │
▼ ▼ ▼
Filing Intelligence Structured Financials Market Intelligence
│ │ │
▼ ▼ ▼
Hybrid Retrieval SEC XBRL Facts Market Data APIs
Dense + Lexical │ │
│ ▼ │
│ Financial Calculator │
│ │ │
└─────────────────────┼─────────────────────┘
▼
Evidence Assembly
│
▼
Reranking Layer
│
▼
Grounded Reasoning
│
▼
Citation Validation
│
▼
Research ResponseThe architecture deliberately separates retrieval, numerical truth, calculation and reasoning.
The language model is not treated as the database, calculator, or source of financial truth.
1. Multi-Layer Financial Data Plane
MarketLane Research uses different systems for different classes of information.
Narrative evidence
SEC filings and company disclosures provide qualitative context:
- management discussion and analysis;
- risk factors;
- operating commentary;
- liquidity discussion;
- business trends;
- capital expenditure commentary;
- segment performance;
- strategic priorities.
These are handled through the document retrieval layer.
Numerical truth
Structured financial values are sourced from XBRL and normalized financial facts.
Examples include:
- revenue;
- operating income;
- net income;
- EPS;
- cash flow from operations;
- capex;
- gross profit;
- assets;
- cash;
- debt.
These values bypass the document-generation path whenever structured data is available.
Market context
Price and market information can be independently queried through MarketLane’s market-data layer.
This allows research questions to combine fundamentals with market behavior without placing market time-series inside the RAG corpus.
2. Financial Document Intelligence Pipeline
MarketLane Research uses document-type-aware ingestion rather than routing every document through the same parser.
Incoming Document
│
┌───────────────┴───────────────┐
│ │
SEC HTML Complex PDF
│ │
▼ ▼
Structural HTML Parser Docling
│ │
│ Parse Quality Checks
│ │
│ ┌───────┴────────┐
│ │ │
│ PASS FALLBACK
│ │ │
└───────────────────────┼────────────────┘
▼
Normalized Document ModelSEC HTML is parsed structurally instead of first being converted into flat text.
The normalized document representation preserves:
- filing sections;
- heading hierarchy;
- paragraphs;
- tables;
- document order;
- fiscal metadata;
- company metadata;
- source locations;
- document lineage.
The same normalized representation can be produced from PDFs, allowing downstream retrieval to remain independent of the original document format.
3. Structural Financial Chunking
Financial reports contain strong document hierarchy.
For example:
10-Q
└── Item 2 — Management's Discussion and Analysis
└── Results of Operations
└── Intelligent Cloud
└── RevenueMarketLane preserves this hierarchy rather than treating the filing as an arbitrary token stream.
The retrieval architecture supports parent-child representations:
Parent section
~1,500–2,500 tokens
│
├── Child chunk
├── Child chunk
├── Child chunk
└── Child chunk
~400–700 tokensSmaller units improve search precision, while parent context allows the reasoning layer to recover surrounding financial context.
Tables are preserved as atomic evidence wherever possible rather than being split blindly across chunks.
4. Hybrid Retrieval
Semantic similarity alone is insufficient for financial documents.
Queries often contain exact language such as:
- ASC 606;
- deferred revenue;
- operating margin;
- fiscal Q3 2026;
- $14.2 billion;
- Intelligent Cloud;
- remaining performance obligations.
MarketLane combines two retrieval systems.
Query
│
┌──────────┴──────────┐
│ │
▼ ▼
Dense Search Lexical Search
pgvector PostgreSQL FTS
│ │
└──────────┬──────────┘
▼
Reciprocal Rank Fusion
│
▼
Candidate Evidence
│
▼
Reranker
│
▼
Final Evidence SetDense retrieval captures semantic similarity.
Lexical retrieval captures exact financial terminology, named concepts and numerical references.
Metadata filtering restricts the search space before ranking using dimensions such as:
- ticker;
- filing type;
- filing date;
- reporting period;
- document;
- filing section;
- fiscal year;
- fiscal quarter.
5. Temporal Financial Reasoning
Financial research is inherently temporal.
Queries such as:
latest 10-Q
previous annual filing
last four quarters
compared with the same period last year
cannot safely depend on model memory.
MarketLane resolves temporal expressions against filing metadata and structured reporting periods.
For example:
"Microsoft's latest 10-Q"
↓
ticker = MSFT
form = 10-Q
selector = LATEST
↓
database resolution
↓
specific SEC accessionThe resolved filing becomes a retrieval constraint.
This prevents semantic search from accidentally returning highly similar but outdated disclosures.
6. Structured Financial Reasoning
A core design principle is:
RAG retrieves explanations. Structured systems provide numbers.
For example:
What was Microsoft’s operating margin last quarter?
does not require the model to search paragraphs for two numbers and perform division.
Instead:
get_financial_facts(
ticker=MSFT,
metrics=[
revenue,
operating_income
],
period=latest_quarter
)
↓
deterministic calculator
↓
operating_marginSupported deterministic operations can include:
- percentage change;
- year-over-year growth;
- quarter-over-quarter growth;
- margins;
- basis-point movement;
- difference;
- aggregation;
- CAGR;
- return calculations.
The language model receives calculated results rather than being responsible for financial arithmetic.
7. Research Orchestration
MarketLane Research uses tool-based orchestration for questions requiring several evidence sources.
Core research tools include:
search_filings()
get_financial_facts()
calculate()
compare_filings()
get_market_data()Consider:
How has Microsoft’s operating margin changed over the last four quarters and why?
The research plan can decompose this into:
1. Resolve MSFT and the four relevant reporting periods
2. Retrieve revenue by period
3. Retrieve operating income by period
4. Calculate operating margin deterministically
5. Search corresponding filings for margin drivers
6. Rerank management commentary
7. Align quantitative changes with qualitative evidence
8. Generate a cited synthesisThis separation creates a much more controllable research system than allowing an LLM to reason over an unrestricted context window.
8. Filing Change Intelligence
One of the core workflows in MarketLane Research is longitudinal filing analysis.
Instead of asking the model to read two complete filings simultaneously, the system first aligns their structure.
Latest Filing Previous Filing
│ │
▼ ▼
Section Extraction Section Extraction
│ │
└──────────── Section Alignment ─┘
│
▼
Evidence Retrieval
│
▼
Change AnalysisThis supports questions such as:
- What risks were added?
- What risks disappeared?
- How did management’s AI capex language change?
- What changed in liquidity commentary?
- What shifted in demand expectations?
- Did management become more cautious?
- Which business areas received greater emphasis?
The output can be represented as a structured change set:
Topic
Previous disclosure
Current disclosure
Material change
Evidence from previous filing
Evidence from current filing9. Cross-Company Research
The same architecture supports research across multiple companies.
Example:
Compare Nvidia and AMD management commentary around data-center demand over the last four quarters.
The engine resolves:
NVDA
├── Q1
├── Q2
├── Q3
└── Q4
AMD
├── Q1
├── Q2
├── Q3
└── Q4Retrieval operates within each company’s reporting context before the evidence sets are merged.
This reduces the risk of cross-company evidence contamination and enables structured comparative research.
10. Evidence-First Generation
The reasoning model never receives anonymous text blobs.
Each piece of context is represented as explicit evidence.
E1
Company: Microsoft
Form: 10-Q
Period: FY2026 Q2
Section: MD&A
Source: <SEC filing>
Text: ...
E2
Company: Microsoft
Form: 10-Q
Period: FY2026 Q1
Section: Results of Operations
Source: <SEC filing>
Text: ...The model can cite only evidence IDs provided by the retrieval system.
The response layer subsequently maps those IDs back to:
- SEC filing;
- document;
- filing section;
- reporting period;
- source URL;
- document location.
This creates explicit lineage from answer → evidence → source document.
11. Citation Validation
Citation generation and citation validation are separate operations.
Before returning an answer, the system can verify:
Does the referenced evidence exist?
Did it belong to the retrieved context?
Does the cited source correspond to the correct company?
Does it correspond to the correct reporting period?
Does a numerical statement match structured financial data?
Are key research claims supported by evidence?Unsupported evidence references can therefore be rejected before reaching the final response.
12. Incremental SEC Ingestion
The research corpus is designed to update incrementally as companies publish new filings.
Cloud Scheduler
│
▼
SEC Filing Discovery
│
▼
New Filing Detection
│
▼
Raw Document Storage
│
▼
Parsing
│
▼
Normalization
│
▼
Chunking
│
▼
Embeddings
│
▼
Search IndexEach document moves through an explicit ingestion state machine:
DISCOVERED
↓
DOWNLOADED
↓
PARSED
↓
NORMALIZED
↓
CHUNKED
↓
EMBEDDED
↓
INDEXEDProcessing is designed to be idempotent.
A retry should not create duplicate documents, duplicate chunks or unnecessary embedding calls.
Version metadata can track:
- parser version;
- chunker version;
- embedding model;
- embedding dimension;
- document hash;
- chunk hash.
This makes selective reprocessing possible as the pipeline evolves.
13. Cloud Architecture
MarketLane’s existing product and the research platform remain independently deployable.
MarketLane domain
│
Cloudflare DNS
│
┌────────────┴────────────┐
│ │
▼ ▼
product app research app
│ │
AWS GCP HTTPS Layer
│
┌──────────┴───────────┐
│ │
Web API Research API
│ │
Cloud Run Cloud Run
│
┌─────────────────────────────┼──────────────┐
│ │ │
▼ ▼ ▼
Vertex AI Cloud SQL GCS
Gemini + Embeddings Postgres/pgvectorThe architecture intentionally minimizes cross-cloud coupling.
Research workloads remain primarily on GCP.
Existing MarketLane infrastructure is accessed only when the research query requires information already owned by the core platform, such as market-data functionality.
14. Storage Architecture
Different data classes are stored independently.
Object storage
GCS stores:
raw SEC filings
normalized documents
complex PDF sources
evaluation artifacts
processing artifactsPostgreSQL
Cloud SQL stores:
companies
document metadata
sections
chunks
financial facts
ingestion state
retrieval metadatapgvector
The same PostgreSQL layer provides vector retrieval for the initial corpus.
This avoids introducing an additional vector database before scale requires one.
The retrieval abstraction is intentionally storage-independent so the indexing layer can later move to a distributed retrieval system without changing research orchestration.
15. Evaluation Architecture
MarketLane Research treats evaluation as part of the system rather than a post-launch manual check.
The evaluation suite separates:
Retrieval quality
Recall@1
Recall@5
Recall@10
MRRGrounding
citation precision
citation coverage
evidence attribution
unsupported claim rateFinancial correctness
numeric accuracy
calculation accuracy
reporting-period accuracy
temporal correctnessSystem performance
retrieval latency
reranking latency
generation latency
end-to-end latency
token consumption
cost per research request16. Retrieval Ablation Framework
The system is structured so retrieval configurations can be benchmarked independently.
For example:
A
fixed token chunks
+
dense retrieval
B
structural chunks
+
dense retrieval
C
structural chunks
+
hybrid retrieval
D
structural chunks
+
hybrid retrieval
+
rerankingEach experiment produces an immutable evaluation artifact.
This makes architecture decisions measurable instead of subjective.
External datasets such as FinanceBench can also be used to measure financial retrieval against third-party evidence annotations.
17. Observability
Each research request can be traced across the entire inference path.
A request trace can include:
trace_id
original query
resolved companies
resolved reporting periods
research plan
tool calls
retrieved chunk IDs
dense ranks
lexical ranks
fusion ranks
reranker scores
financial facts retrieved
calculations executed
model invocation
token usage
retrieval latency
model latency
end-to-end latency
citations
final statusThis provides enough visibility to investigate whether a poor answer originated from:
query understanding
temporal resolution
retrieval
ranking
missing source data
financial normalization
calculation
generationrather than treating every failure as an LLM failure.
18. Reliability Model
AI systems fail differently from traditional APIs.
MarketLane Research therefore separates failure classes such as:
source acquisition failure
document parse failure
partial extraction
embedding failure
indexing failure
retrieval miss
reranker degradation
tool failure
insufficient evidence
citation validation failure
LLM generation failureIngestion failures can be retried independently from online research requests.
Unavailable evidence should result in an explicit insufficient-evidence response rather than unsupported generation.
19. Cost-Aware AI Infrastructure
The architecture is designed around selective use of expensive inference.
Examples:
Document parsing
SEC HTML
→ deterministic parser
Complex PDF
→ document AI parser
Poor extraction
→ expensive fallbackQuery execution
simple financial fact
→ structured lookup
retrieval question
→ hybrid search
complex research question
→ retrieval + reranking + LLM synthesisIndexing
unchanged chunk hash
→ skip embedding
new/changed chunk
→ embedThis prevents high-cost models from becoming the default implementation for operations that can be handled deterministically.
20. Enterprise Evolution
The current architecture is intentionally modular so individual components can evolve independently as corpus size, research complexity and query volume increase.
The design can extend toward:
Retrieval infrastructure
pgvector
↓
distributed vector retrieval
↓
partitioned / multi-stage retrievalData processing
scheduled ingestion
↓
event-driven ingestion
↓
distributed document-processing workersModel infrastructure
single-provider inference
↓
model routing
↓
task-specific model selection
↓
fallback and resilience policiesResearch engine
single-company analysis
↓
cross-company research
↓
sector intelligence
↓
portfolio-level researchThe important property is that these changes do not require redesigning the financial evidence model itself.
Representative Research Workflows
Fundamental Analysis
How has Amazon AWS operating income changed over the last three quarters and what drove the change?
Uses:
structured financial data
quarter resolution
deterministic calculations
multi-filing retrieval
management commentary
citationsManagement Commentary Analysis
Compare management’s AI infrastructure commentary across Microsoft’s last four reporting periods.
Uses:
temporal resolution
multi-document retrieval
semantic + lexical retrieval
reranking
longitudinal synthesisFiling Intelligence
What risk factors were materially added or removed from Nvidia’s latest annual filing?
Uses:
document resolution
section alignment
filing diff
evidence comparison
dual-source citationsCompetitive Intelligence
Compare Nvidia and AMD’s commentary about data-center demand.
Uses:
multi-company retrieval
company-level evidence isolation
temporal alignment
comparative synthesisMarket Reaction Research
What changed fundamentally after the latest earnings release and how did the stock trade over the following five sessions?
Uses:
SEC filings
structured fundamentals
MarketLane market data
financial calculations
multi-source synthesisWhat Makes MarketLane Research Different
The goal is not to build a better chatbot.
It is to build a financial evidence engine capable of transforming heterogeneous financial information into reproducible research.
The architecture is based on several principles:
LLMs reason; databases provide facts.
Retrieval finds evidence; it does not determine financial truth.
Arithmetic is deterministic.
Time is explicit.
Every meaningful claim should be traceable.
Document structure should survive ingestion.
Evaluation should determine architecture choices.
AI failures must be observable.
These constraints turn the system from a conventional RAG application into infrastructure for programmable financial research.
Technology
AI
Vertex AI
Gemini
Vertex embeddings
Retrieval
PostgreSQL
pgvector
full-text retrieval
hybrid search
reciprocal rank fusion
reranking
Document Intelligence
SEC EDGAR
XBRL
structural HTML parsing
Docling
table-aware normalization
Backend
Python
FastAPI
Pydantic
SQLAlchemy
Alembic
Infrastructure
Google Cloud Run
Cloud Run Jobs
Cloud Scheduler
Cloud SQL
Google Cloud Storage
Secret Manager
Cloud Logging
Cloud Monitoring
Platform Integration
Cloudflare
AWS-hosted MarketLane services
market-data APIsProduct Direction
MarketLane Research is intended to become the intelligence layer behind MarketLane.
The long-term opportunity is larger than SEC search.
The same evidence architecture can support:
public filings
company announcements
earnings materials
transcripts
macro data
market data
alternative datasets
internal research
portfolio datawhile keeping one consistent research interface above them.
The eventual system can continuously transform new information into:
structured facts
retrievable evidence
company timelines
change detection
cross-company comparisons
research signals
portfolio-level intelligenceThe objective is to move financial research from a workflow dominated by manual discovery and document navigation toward a system where complex questions can be expressed directly and answered with transparent evidence.
MarketLane Research is the architecture for that transition.