Our multi-tenant RAG assistant indexed every document a client uploaded, split it into chunks, embedded them and searched them with pgvector. On paper it was a working pipeline. In practice the answers were often not accurate: close to the question, but not the part of the document that actually answered it. The fix wasn't a better embedding model. It was giving the system what a person uses first when they open a long document: the table of contents.
#1. How It Started: Chunks and Vectors Only
upload ──▶ split into chunks ──▶ embed ──▶ pgvector
│
question ──▶ embed ──▶ nearest chunks ◀───────┘
│
▼
LLM writes the answerThis is the textbook pipeline, and for short documents it was fine. With long documents (manuals, policies, product catalogues) the answers drifted.
#2. Why the Answers Missed
- Similar is not the same as relevant. Vector search returns the chunks whose wording is closest to the question. A chunk that mentions the right words in passing can outrank the section that actually answers it.
- Chunks lose their place. Once a document is cut into pieces, a chunk no longer knows which chapter it belongs to. "The limit is 30 days" means nothing without the heading above it.
- Answers span sections. When the answer is a whole procedure, the top few chunks bring back fragments of it, not the section.
None of this is a bug you can patch in the vector search. The index simply doesn't know how the document is organised.
#3. The Change: An LLM Writes a Table of Contents at Indexing Time
When a document is indexed, an LLM now reads it once and writes its table of contents: each section's title and what it covers, with a pointer to where that section's text lives. The TOC is stored next to the document's chunks. It is small, so it is cheap to read again later, and it describes the document the way its author organised it, not the way a chunker happened to cut it.
Generating the TOC with an LLM, rather than reading headings out of the file, matters because many uploaded documents don't have clean headings at all: scanned files, exported pages, long text with no structure. The LLM gives every document an outline either way.
#4. At Question Time: Two Paths in Parallel
question
│
┌──────────┴──────────┐ in parallel
▼ ▼
LLM reads TOCs, embed question
picks sections │
│ ▼
▼ pgvector top
fetch sections chunks
│ │
└──────────┬──────────┘
▼
merge, de-dupe, re-rank
│
▼
LLM writes the answer- The TOC path: an LLM reads the tables of contents and picks the sections that should answer the question, the way a person scans a contents page. Those sections are fetched whole.
- The vector path: the same nearest-chunk search as before, still good at finding a specific phrase or fact wherever it sits.
- In parallel: both start at once, so the wait is the slower of the two, not the two added together. This mattered to us: the same platform had just been through a latency audit, and retrieval was not allowed to give that time back.
#5. Merge, Then Re-rank
The two paths overlap: a vector chunk is often a piece of a section the TOC path already picked. So the results are merged, duplicates removed, and the whole set re-ranked against the question as one list. Only the top of that list goes to the model that writes the answer. Each path covers the other's blind spot: the TOC path brings the right section in full, the vector path brings the exact line that a section-level view would skim past.
#6. What Changed
The answers became accurate where they had been missing the point. Questions that needed a whole procedure got the whole section instead of fragments, and questions about one specific detail still found it through the vector path.
#7. The Costs
- Indexing does more work: one LLM pass per document to write the TOC.
- The TOC has to stay current: when a document is replaced, its TOC is written again with it.
- One more model call per question: the section picker. Its input is only the tables of contents, so it is small, and running it in parallel keeps it off the critical path.
#8. Takeaways
- When RAG answers are close but wrong, look at what the index knows before tuning the embeddings.
- A document's structure is information. A table of contents puts it back after chunking takes it away.
- Two retrieval paths that fail differently, run in parallel and re-ranked together, beat either one alone.