RAG Wasn't the Magic Button I Thought It Was

I'd read that RAG solves hallucinations. It doesn't. It gives you a whole new set of things to get right.

Back in the first post, I said I knew RAG was meant to help with exactly the kind of mistake I was afraid of. And in the second post, when I drew those four boxes on a sticky note, "give content to AI" looked like a single step. Find the right content, hand it over, done.

After the rewards program answer, I started looking much more closely at that step. It wasn't one step. It was a whole system, and every part of it could quietly make the answer worse.

The model only knows what it's handed

This was the first thing that really sank in.

The model doesn't search the website. It doesn't look around for more information. It gets the question and a handful of pieces of content, and that's its whole world. If the right piece isn't in that handful, the model can't use it. If the wrong piece is in there, the model will happily use that instead.

So the quality of the answer depends almost entirely on what search hands over. The model is the part everyone talks about. Retrieval is the part that decides whether it has a chance.

Chunks matter

Before anything can be searched, it has to be broken into pieces. I'd done that back in April without thinking too hard about it.

It turns out the size of those pieces matters a lot.

If a chunk is too big, the one sentence that actually answers the question is buried inside a lot of text about something else. The chunk might still get found, but it's a weaker match, and the model has to dig the answer out of everything around it.

If a chunk is too small, it loses its context. A line that says "not covered under warranty" is pretty important. Without the sentence before it, you don't know what isn't covered.

There isn't a perfect size. There's just a trade-off you have to pick on purpose.

Ranking matters

Search doesn't just find pieces of content. It puts them in order.

The best piece isn't always first. Sometimes it's third, behind two things that are close but not quite right. If you only hand the model the top result, you miss it. If you hand over everything, you get the next problem.

How many pieces you hand over matters

Give the model too few pieces and you might leave out the one that matters.

Give it too many and you start inviting in the near misses from the embeddings post. The email sign-up page shows up next to the registration page. A policy about one thing sits next to a policy about something similar. The model doesn't know which ones are relevant and which ones just happen to be nearby. It treats them all as evidence.

Loosely related content isn't harmless background. It's material the model might build an answer from.

How close is close enough?

Every result from a search by meaning comes with a score for how close it is to the question. The higher the score, the closer the match.

Which raises a question I hadn't thought about at all: at what point is a result too far away to count? Somewhere there's a line where "related" turns into "not really about this." Draw it too high and you throw away good answers. Draw it too low and you let in things that only sound relevant.

I didn't have a good answer to that yet. I'd come back to it.

Meaning wasn't enough on its own

Embeddings had been the exciting discovery in April. Searching by meaning instead of matching words felt like a huge step forward.

But some questions aren't about meaning at all. If someone types a part number, like PRD-104, they don't want something that means roughly the same thing. They want that exact product. A model number, a specific term from a manual, an exact phrase. That's where the old kind of search I'd been doing for years is still better.

So in May, search started doing both. It looked for content that meant something close to the question, and it looked for content that actually contained the important words. Then it combined the two.

It turned out the boring kind of search I already knew wasn't obsolete. It just wasn't enough by itself, and neither was the new one.

RAG doesn't make correctness automatic. It gives you a new set of problems to solve.

Oh. This is the real work.

I'll be honest, this stretch was humbling. I'd thought of retrieval as plumbing, the part you set up once so you can get to the interesting AI part.

It's the other way around. The model is remarkably good at writing an answer from what it's given. Almost everything that goes wrong happens before it gets there.

That was oddly encouraging, though. Retrieval was something I could look at. I could see what search found, in what order, with what scores. When an answer was wrong, I could open up the database and see exactly what the model had been handed. That's the "one place to inspect" I wanted when I picked PostgreSQL, and it paid off here.

The next problem

Better retrieval meant fewer bad answers. But it couldn't mean zero.

Sometimes the right information just isn't there. The page was never crawled, like the rewards program. Or the website doesn't cover that question at all. No amount of tuning search fixes that, because there's nothing to find.

So what should the website agent do when that happens?