Every RAG debugging session starts the same way. The answers are wrong, so you check the logs. The query executed. The vector search returned results. The similarity scores look plausible - 0.78, 0.74, 0.71. The LLM received context. Everything "works."
And the answer is still garbage.
Here's the uncomfortable truth about that debugging session: the problem usually isn't anywhere your logs can see. Retrieval quality problems live in the geometry of your embedding space - and almost nobody ever looks at it.
Why logs can't show you this
A vector search never fails loudly. Ask for the 5 nearest neighbors and you get 5 nearest neighbors - even if all of them are boilerplate, duplicates, or chunks so similar to each other they add nothing. "Nearest" is relative. The scores only tell you these were the closest vectors available, not that they were good.
So the failure modes hide. Four patterns cause most of the damage:
1. Duplicates. An ingestion pipeline re-ran, a retry fired twice, a batch job overlapped. Now the same documents exist two or three times, and retrieval keeps returning the same content in every slot of your top-k - crowding out the context the LLM actually needed.
2. Boilerplate clusters. Headers, footers, cookie notices, legal disclaimers. If your chunking didn't strip them, they got embedded like real content - and they get retrieved like real content. Worse, boilerplate is textually similar to everything ("About us", "Terms of service"), so it surfaces for wildly unrelated queries.
3. Chunking blobs. A chunking strategy that produced hundreds of nearly identical vectors - overlapping windows over repetitive text, or over-aggressive splitting of structured documents. In vector space this is one dense blob, and retrieval inside a blob is close to random.
4. Outliers. Empty strings, encoding failures, OCR garbage. They all still produce vectors. Sometimes those vectors land somewhere unfortunate - and become someone's nearest neighbor.
None of these throw errors. All of them have a visual signature.
Project the space and look at it
Embeddings live in 384 to 3,072 dimensions, which no human can look at. But dimensionality reduction - UMAP or PCA - projects them to 2D while approximately preserving neighborhood structure. Clusters stay clusters. Duplicates stack on top of each other. Outliers sit alone in space.
A quick honesty note before you trust any 2D picture: projections distort. UMAP preserves local neighborhoods, but the global distances between clusters are not meaningful; PCA preserves global variance but can smear local structure. Use the projection to find candidates - then verify by reading the actual documents behind the suspicious points. (More on when each method lies in a companion article.)
The visual signatures map cleanly to the four failure patterns:
(illustrative - the shapes each problem makes when projected to 2D)
- A dense stack of points that looks like one fat point → duplicates
- A tight cluster far from your content, appearing for every query → boilerplate
- One giant undifferentiated blob instead of distinct topical clusters → chunking failure
- Lone points floating far from everything → encoding garbage
What it looks like in practice
Here's a real projection of a demo collection - a few thousand documentation chunks, colored by their position in the reduced space:
(UMAP 2D projection of a demo collection)
The diagnostic loop is short: project, spot the signature, then hover a suspicious point to read the document behind it - which confirms a bad cluster in seconds instead of writing a query loop. A tight cluster that turns out to be the same footer embedded a few hundred times is a duplicate/boilerplate problem you can now see and fix, instead of one you argue about in a standup.
The fixes
Once you can see the problem, the fixes are ordinary data engineering:
- Duplicates → deduplicate on a content hash at ingestion; make pipeline runs idempotent (upsert by stable ID, not insert).
- Boilerplate → strip repeated page furniture before chunking; a frequency check on identical chunks catches most of it.
- Chunking blobs → re-evaluate chunk size and overlap; if hundreds of chunks are near-identical, your source is repetitive and needs coarser chunking or section-level splitting.
- Outliers → validate text before embedding: minimum length, encoding checks, OCR confidence thresholds.
Then re-embed, re-project, and confirm the geometry looks healthy.
How to do this yourself
The notebook route, with any vector DB:
import umap
import matplotlib.pyplot as plt
# pull embeddings from your DB client into a matrix
# X.shape == (n_docs, dim)
proj = umap.UMAP(n_components=2, n_neighbors=15).fit_transform(X)
plt.figure(figsize=(10, 8))
plt.scatter(proj[:, 0], proj[:, 1], s=4, alpha=0.5)
plt.show()
Add hover inspection with plotly, wire up per-point document lookup, and re-run it every time your pipeline changes. It works - it's just enough friction that in practice, nobody does it twice.
Full disclosure: I work on VectorLens, a native desktop GUI for ChromaDB, Qdrant, Weaviate, and Milvus, and this workflow is the reason it exists. Connect to your database, pick a collection, and the UMAP/PCA projection with hover-to-inspect is a 30-second check instead of a notebook session. It runs locally, talks directly to your database, and the installer is under 5 MB.
(collection browser)
Either way - notebook or GUI - the habit is what matters: look at your embedding space before you tune your prompts. A surprising amount of "prompt engineering" is compensating for a vector store nobody has ever seen.
