Then It Confidently Said Something Wrong

The worst answer it gave me that month was also one of the best-written.

After the first answers started coming back, I kept going. I asked about products, policies, shipping, returns. I tried different phrasing. I asked things I wasn't sure the website even covered. Most of the answers were good. Some were a little long. A few missed a detail.

Then I asked a simple one.

"Do you have a rewards program?"

The answer came back right away. It was polite. It was clear. It sounded exactly like a helpful person on a support team would sound.

It said no.

We do have a rewards program.

If you've been reading since the first post, you might recognize that question. It's the same one I put on the very first image in this series, back when the website agent was just an idea and a sticky note. I hadn't planned for it to be the question that tripped it up. It just was.

It didn't sound wrong

That's the part that bothered me most.

If the answer had been garbled or vague, I would have noticed something was off right away. A customer would too. But nothing about it looked broken. It was confident, friendly and completely wrong.

A customer reading that answer would have believed it. Why wouldn't they? It came from the company's own website. They'd leave thinking there was no rewards program, and they'd never go looking for it.

This was the exact thing I said I was afraid of in the very first post. A wrong answer, delivered confidently, is worse than no answer at all. I had written that sentence. Now I was looking at it.

It didn't invent a program. It invented a "no."

When I started thinking about what had actually happened, I realized it was a different kind of mistake than the one I'd been worried about.

I'd pictured the model making something up. Inventing a feature, a spec or a policy that didn't exist. That's what you hear about when people talk about AI getting things wrong.

This was the opposite. It didn't invent a rewards program. It decided there wasn't one.

Somewhere between the question and the answer, the information the model needed wasn't in front of it. And instead of saying so, it filled the gap with the most reasonable-sounding answer it could come up with. "I didn't see anything about a rewards program" quietly became "there is no rewards program."

Those are very different statements. One is about what the agent was given. The other is about the company. The model didn't know the difference, and it wasn't going to tell me.

Why it wasn't there

When I went looking for why, the answer was almost embarrassingly simple.

The rewards page had never been crawled. My crawler worked from the website's sitemap, and the rewards program wasn't on it. As far as my system was concerned, the page didn't exist. The model never saw a single word about it.

That somehow made it worse, not better. The crawler did exactly what I told it to. Search did exactly what it was supposed to. The model did exactly what models do. Every piece worked, and the answer was still wrong.

It wouldn't be the last time that rewards program taught me something.

Not finding something isn't the same as knowing it doesn't exist.

The model was doing its job

It took me a little while to accept this, but the model wasn't really malfunctioning.

It did what these models do. It took the question and whatever it was handed, and it wrote the most helpful answer it could. When there was a gap, it filled it with something that sounded right. That's what makes these models so good at writing, and it's exactly what makes them dangerous when the facts matter.

So the problem wasn't going to be fixed by finding a smarter model. It was going to be fixed by changing what the model was allowed to do.

A new rule

That day, the goal of the project changed.

Before, the goal was to get the website agent to answer questions. After, the goal was to get it to answer only when the evidence actually supported the answer.

That idea has a name: grounding. The answer has to come from the content the agent was given, and nothing else. If the evidence says something, the agent can say it. If the evidence doesn't say something, the agent doesn't get to fill in the blank. Not for a product spec, not for a price, not for a warranty detail, not for a policy, and not for a simple yes or no.

It sounds obvious written down. It was a lot less obvious when I'd spent the previous week being impressed by how well it could write.

A polished wrong answer is worse than no answer at all.

The next problem

I'd come into this thinking RAG would protect me from exactly this kind of mistake. That was the whole point of giving the model our content instead of letting it answer from whatever it already knew.

It turned out RAG wasn't the protection. It was just the beginning of the work.