The Problem With Conversations

The website agent was getting good at answering questions. Then I asked it a second one.

By the end of May, I was feeling pretty good. It answered from real content. It stuck to the evidence. When it didn't have enough information, it said so. I'd spent a lot of time making sure a single question got a good answer.

That's not how people talk, though.

The second question

I asked about a product. Something simple, like:

Tell me about PRD-101.

The answer was good. It pulled the right product, described what it does, and stayed grounded in what the website actually said.

So I asked the obvious next question. The kind anyone would ask:

Will it work with 1/2-inch material?

The answer had nothing to do with PRD-101.

It had lost the product completely. Instead, it pulled in a handful of other products and manuals that happened to mention material thickness, and answered from those. It wasn't even a bad answer, exactly. It just wasn't an answer to my question.

What is "it"?

It took me a minute to see what had happened, and then it was almost funny.

To me, "it" was obvious. We'd just been talking about PRD-101. Any person on the other end of that conversation would know exactly what I meant.

But search didn't see a conversation. It saw one question, all by itself: "Will it work with 1/2-inch material?" There's no product in that sentence. There's a word, "it," that means nothing on its own, and a measurement that shows up all over the website. So search did exactly what it was built to do. It found content that matched those words as well as it could, and the model answered from whatever came back.

Every question was being treated like the first question anyone had ever asked.

Even "I don't know" can go wrong here

This one bothered me in a different way, too.

I'd just taught the agent to say "I don't have enough information" when the evidence was weak. That was the right decision. But without the conversation, a perfectly answerable follow-up can look like a question with no good evidence behind it. The agent might honestly decline to answer something it could have answered easily, if it had only remembered what we were talking about.

Being honest isn't much help if it's honest about the wrong question.

People don't ask questions. They have conversations.

Looking back, I'm a little surprised it took me this long to run into this. I'd been testing the way a developer tests. One question, check the answer, next question. Clean and separate.

Real people don't do that. They ask about a product, then ask a follow-up, then another one. They say "it" and "that one" and "the other one." They expect you to keep up. On a real website, the follow-up is often the more important question. The first question is just how they got started.

People don't ask questions. They have conversations.

The next problem

So the website agent needed to remember the conversation. That part was obvious.

What wasn't obvious was how. Handing search the whole conversation would bring back a mess. Handing it only the last question was what got me here.

Somehow, before searching, it needed to figure out what "it" meant.