The Day I Realized AI Could Mix Two Correct Answers Into One Wrong Answer

Every sentence in the answer was true. The answer was still wrong.

By the end of the last post, plans were in good shape. Steps stayed whole, in order and attached to the plan they came from. Follow-up questions stayed on the plan you were looking at. I was pretty happy with it.

Then I started testing with more plans, and I ran into a kind of mistake I hadn't seen before.

Two plans that look alike

There's a large variety of plans, and a lot of them are similar. Two shelves. Two storage projects. Two versions of the same idea in slightly different sizes. They share materials. They share steps. Sometimes they even share most of their wording.

To a person, they're obviously different projects. You'd never build half of one and half of the other.

To search, they're neighbors. Ask about one, and the other is right there, scoring almost as high. It's the near-miss problem from the embeddings post, except this time the near miss isn't wrong. It's a perfectly good plan. It's just not your plan.

A step from one, a measurement from the other

Here's the kind of thing that happened. (This is an illustrative example, but the pattern was real.)

Say there are two similar bookshelf plans, A and B. You're looking at plan A, and you ask:

What size screws does it use?

Search finds content from both plans, because they're so alike. And the agent answers with plan B's screw size. Or it walks through the directions and gives you plan A's first three steps, then plan B's fourth.

Each detail, on its own, is correct. It's really in one of the plans. But put together, the answer doesn't match either plan. If you followed it, you'd end up with something nobody designed.

Why grounding didn't catch it

This is the part that really got me.

Back in May, I'd built the whole system around grounding. The answer has to come from the evidence. Don't invent anything. And the agent hadn't invented anything. Every piece of that answer was in the evidence it was given.

That was the problem. The evidence was a mix of two plans, and the agent treated it all as one source of truth. Grounding protects you from making things up. It doesn't protect you from using real information in the wrong place.

Correct information used in the wrong context is still wrong.

Staying anchored to the right plan

The fix built on the work from the last two posts.

Once someone is looking at a plan, the conversation needed to stay anchored to that plan. Not plans that look like it. Not the closest match to the latest question. That one. When the follow-up comes in, the evidence for the answer should come from the plan we're actually talking about, and content from a similar plan shouldn't be allowed to sneak in just because it scored well.

And then I wrote tests specifically for this. Conversations that start on one plan, with a very similar plan sitting right next to it in the data, and checks that the answers never pick up a step or a measurement from the wrong one. That's the kind of bug that can come back quietly later, when you change something that seems completely unrelated. I wanted to know right away if it ever did.

The next problem

June was humbling in a good way. I started the month thinking a question was just a question. I ended it understanding that a question lives inside a conversation, that the conversation is about a specific thing, and that the specific thing matters as much as the facts.

The website agent was a lot better for it.

But I was also starting to feel something else. I was testing all of this by typing into a chat window, reading answers, and digging through the database when something looked off. That worked when the project was small. It wasn't small anymore.

At some point, you need more than a chat window.