All projects

Case study · Multi-Tenant AI Chatbot Platform

From 40 seconds to about 13: rebuilding a multi-tenant AI agent

Clients embed this RAG assistant, built on .NET 10, in their own products. It worked, but a data question took 30–40 seconds. An end-to-end audit ranked the fixes, the agent moved to Microsoft Agent Framework with MCP tools for data and docs, and most answers now arrive in about 13 seconds.

per data question, before
30–40 s
most answers now
~13 s
faster again, next (in testing)
~2×

Measured on the running platform. The next step, a change to how the model runs on the server, is still being tested.

Architectural Takeaway
Eliminated a 7–8 sequential LLM call cascade in favor of a single schema prompt, pooled MCP tool connections, and enabled native token streaming — cutting request latency by 65% (from 35s down to ~13s).

01

The problem

It worked. It was just slow.

The platform is a multi-tenant RAG chatbot: an embeddable widget, SignalR streaming, pgvector retrieval, a text-to-SQL agent, MCP tools and episodic memory.

Answers were right, but a data question took 30–40 seconds. Instead of guessing, I audited the whole request path to find out where the time went.

02

Architecture

The platform in one picture

An ASP.NET Core gateway runs Semantic Kernel agents over pgvector, Redis, MongoDB and long-term memory. Heavy work (document ingestion and audit logging) goes through RabbitMQ consumers, so the gateway stays non-blocking. Vectors are namespaced per tenant.

Interactive diagram

RAG chatbot architecture

A chat request, and the background pipeline that feeds it documents.

Client

Chat widget · embeddable iframe

A vanilla JS widget that any allowed domain can embed. The host origin is verified per tenant before a session starts.

JavaScriptiframeAllowed origins
ClientServiceAIData

03

Where the time went

About 13 seconds of every answer was waste

For a SQL-style question, roughly 12–15 seconds is unavoidable LLM and database work. The rest broke down like this:

Time per request, in milliseconds
  • Nested SQL LLM calls~8,000 ms
  • Redundant intent re-parsing~2,000 ms
  • Artificial streaming delay~1,500 ms
  • Other overhead~500 ms
  • Duplicate vector search~300 ms
  • Redundant kernel clones~300 ms

04

Findings

Three patterns behind the delay

Finding 1

An LLM call cascade

The SQL path classified intent, re-derived it, discovered tables, analysed the schema, generated SQL and validated it: 7–8 sequential LLM calls where 2–3 would do.

Fix: One generation prompt with the relevant schema inlined, behind a feature flag and A/B tested on 20+ reference questions. Worth an estimated 8–12 s.

Finding 2

Streaming that wasn’t

Responses were re-chunked into 20-character pieces with a 15 ms pause each: 1.5 s of pure delay on a 2,000-character answer, on top of a model that already streams.

Fix: Forward tokens as they arrive.

Finding 3

The same work, twice

Two services ran the same vector search; the kernel was cloned 3–4 times per request (50–100 ms each); MCP connections were opened per request (300–800 ms).

Fix: Search once and pass it down, reuse the base kernel, and pool MCP connections with deterministic disposal.

05

The plan

Ranked fixes, shipped in three steps

  1. Week 1

    Quick wins: real streaming, one vector search, kernel reuse, MCP pooling.

    Target 30–40 s → 20–25 s

  2. Weeks 2–3

    Single-shot SQL generation, cached intent, parallel pipeline steps.

    Target → 8–12 s

  3. Weeks 3–4

    Tenant isolation in every cache key, retries for ingestion, per-tenant cache invalidation, latency dashboards.

06

What shipped

Faster, and ready to run at customer sites

Finding 1

A new agent core

The agent moved to Microsoft Agent Framework, with MCP tools that give it controlled access to the database and the documentation it needs to answer each question.

Fix: Most answers now arrive in about 13 seconds, down from 30–40.

Finding 2

Answers you can explore

Interactive maps: pick a state to see its districts, then a district to see its cities, all without asking a new question.

Fix: Follow-up questions became clicks, so users wait less and the model does less work.

Finding 3

Sign-in and licensing

Users sign in with a fingerprint, face or device PIN instead of a password. Each installation is licensed to one server for a fixed period, and the licence resists tampering and copying.

Fix: A copied or moved installation stops working, and an expired licence pauses the application until renewal.

Finding 4

Remote installs and operations

Installs and upgrades run remotely. From one place the team backs up and restores data, rolls back an upgrade, upgrades several instances safely, and watches each service’s health and load.

Fix: Supporting many servers no longer needs anyone on site.

07

Accuracy

A table of contents beside the vectors

Speed was half the problem. On long documents, vector search returned chunks that were similar to the question but not the section that answered it, so answers came back close but not accurate.

Now an LLM writes a table of contents for every document when it is indexed. At question time two retrievals run in parallel: an LLM reads the tables of contents and picks the sections that answer the question, while vector search finds matching chunks as before. Both result sets are merged, de-duplicated and re-ranked before the answer is written.

  • The TOC path brings back whole sections, so procedures and policies arrive complete.
  • The vector path still finds the one specific line a section-level view would skim past.
  • Running both in parallel costs the slower of the two, not both added together.

08

Hardware

Choosing the GPU: bandwidth beat memory size

The same agent request on Gemma 4 26B took about 15 seconds on an RTX 5090 workstation and about 41 seconds on an NVIDIA DGX Spark. The model fits in the 5090’s 32 GB, and its roughly 6.6× higher memory bandwidth is what decides generation speed.

  • Next: a change to how the model runs on the server could make answers roughly twice as fast at the same quality. It is tested on our own server before it reaches customer servers.
  • Full numbers and method in the post “RTX 5090 vs DGX Spark for a Local AI Agent”.

09

Lessons

What I check first now

  • In LLM apps, count the calls before tuning anything else. Sequential calls dominate latency.
  • Don’t simulate streaming on top of a model that already streams.
  • Every cache key includes the tenant ID; anything else is a cross-tenant leak waiting to happen.
  • Measure first: an audit turns a feeling into a ranked list of fixes with estimated savings.

Obsidian Knowledge Graph

Connected Technologies & Research

Explore full vault graph