Ankit Jha · Staff AI Engineer · 7 years in production AI

AI systems built end to end, from the agent down to the inference stack.

I design, build and operate AI products across the whole stack: agents and retrieval, the evaluation that decides what ships, and the serving and infrastructure underneath. Recent work: a live market-intelligence product, a retrieval platform over SEC filings, and upstream fixes in vLLM, llm-d and NVIDIA Dynamo.

See the workOpen-source contributionsGet in touch

ExperiencePhysicsWallahPaytmMeeshoAmazon

Production AI systems shipped
10+
Users, products & transactions served by production AI systems
Millions
Upstream contributions
14
Merged upstream
8

Latest merged upstream · llm-d/llm-d-router #3931 · Nov 2026A one-line coordinator fix keeps every phase of a disaggregated request on one rollout revision

01 / Case studies

Selected work

All projects →
01MarketLane AIAI-native market intelligence for NSE equities: how the daily scan went from one cached LLM call to a verified multi-stage pipeline, and why the middle generation ships.Product2026PythonFastAPI02MarketLane ResearchEvidence-grounded retrieval over SEC filings, and the measurement system that decided what shipped: 13 pre-registered experiments, two adopted changes, eleven rejected with mechanisms, about $40 of model spend.Infrastructure2026Python 3.12FastAPI03AI Incident CommanderA pre-registered, equal-budget comparison of a multi-agent incident-diagnosis system against a single agent on 45 held-out microservice incidents, on two models, with four follow-up experiments.Research2026PythonLLM agents04Agent Context EngineA Python library that decides what a long-running AI agent puts in its model context on every call: three memory layers, hybrid retrieval, four compression methods and a rule-based adaptive planner, benchmarked against Claude on 2,204 graded answers.Infrastructure2026Python 3.11SQLite

02 / Open source

Upstream contributions

All 14 contributions across 8 repositories →

llm-d/llm-d-router· #3931 ·

Keeping a request on one rollout revision across a conditional-decode miss in llm-d-router

Made the coordinator's conditional-decode step capture the response headers of a 412 cache miss, so the rollout revision header reaches the encode, prefill and decode phases. Merged as part 1 of #3931; the EPP half, which stamps the revision on the 412, needs a new opt-in plugin hook and is proposed separately.

  • Cross-component header contracts
  • Splitting a fix along ownership lines
  • Fail-closed vs silent fallback
Merged

Medium · LLM request routing / disaggregated serving coordinator

Write-upPRIssue

BerriAI/litellm· #49931 ·

Stop caching empty replies in LiteLLM's response cache

Replaced the response cache's 'has output' check with one that reads what each content block or chat message actually holds, so an empty reply is no longer cached or replayed on /v1/messages or /v1/chat/completions, streamed or not. A reply holding only Gemini's thought signature counts as empty, stored empty entries count as misses, and whitespace replies and code-completion fills stay cached. Checked with unit tests, an 8-mutant run, and two-worker proxies against real Gemini.

  • Cache admission policy
  • Streaming protocol parsing
  • Over-tightening guards
Merged

Medium · LLM gateway / response caching

Write-upPRIssue

llm-d/llm-d-router· #3680 ·

Warning about data producers nobody reads in llm-d-router

Added a startup warning in CreateMissingDataProducers for a configured producer key that no other plugin reads while a consumer of the same data type reads an auto-created default. It catches the issue's config and the cross-type prefix producers, and stays silent on every shipped config.

  • Dependency resolution
  • Plugin data graphs
  • False-positive budgets
Merged

Medium · LLM request routing / EPP data layer

Write-upPRIssue

03 / Technical areas

Across the stack

  • Agent systems and evaluation

    Pre-registered, equal-budget comparisons of agent architectures, and context and memory management for long-running agents, measured on graded model answers with per-call cost accounting.

    AI Incident CommanderAgent Context Engine

  • Retrieval systems

    Evidence-grounded RAG over SEC filings, with a pre-registered evaluation programme that rejected eleven techniques and shipped two.

    MarketLane Research

  • AI products in production

    Three generations of a verified LLM scan pipeline, measured on accuracy, cost and latency, for a live market-intelligence product.

    MarketLane AI

  • LLM inference and serving

    Startup and teardown correctness, streaming lifecycles, and resource ownership inside production inference servers.

    vLLM #57650llm-d-inference-sim #801

  • Request routing and infrastructure

    Flow control, prefill/decode disaggregation, cache-aware routing, and admission feedback in LLM gateways.

    llm-d-router #2488vLLM Router #299

Get in touch

Hiring for a senior or staff AI engineer, or building something hard?

I’m happy to talk with teams working on agents, retrieval, or LLM inference at scale, and with founders who need someone who can take an AI product from prototype to production. Open to full-time roles and to consulting engagements.

ankitjha.email@gmail.comLinkedInGitHub