Teaching an Agent What We're Talking About

The fix was to stop searching for what the customer said, and start searching for what they meant.

At the end of the last post, the website agent had a very short memory. Every question was treated like the first one. Ask about a product, then ask "will it work with 1/2-inch material?", and "it" meant nothing.

The obvious idea was to give search the whole conversation. I thought about it for about five minutes. A conversation has greetings, earlier answers, side questions and things that don't matter anymore. Hand all of that to search and you'd get back a little bit of everything. That's the opposite of what I wanted.

Rewrite the question first

What actually worked was simpler.

Before searching, the agent looks at the conversation so far and rewrites the follow-up into a question that stands on its own. So this:

How thick of material can it handle?

becomes something like this, just for search:

How thick of material can PRD-101 handle?

That's it. The word "it" gets replaced with the thing we were actually talking about. Search gets a complete question with the product name in it, and suddenly it can find the right content again.

The first time I watched a rewritten question come back, it felt like one of those fixes that's so obvious you're a little embarrassed you didn't think of it sooner.

Two versions of the same question

There's an important detail here that took me a bit to get right.

The rewritten question is only for search. It is not what the agent answers.

The customer asked "How thick of material can it handle?" in their own words, in the middle of a conversation. The answer should feel like a reply to that, not to some cleaned-up version they never typed. So there are really two versions of every follow-up:

• what the customer said, which is what the agent answers;
• what search needs, which is what the agent uses to find the evidence.

Keeping those separate made a bigger difference than I expected. The answers still sounded like a natural part of the conversation, and search still got everything it needed to find the right content.

What someone says and what they mean aren't always the same words.

The test conversation

To know whether it actually worked, I needed something better than asking one follow-up and squinting at the answer.

So I wrote a short test conversation. Something like:

1. What does PRD-101 do?
2. How thick of material can it handle?
3. Does it come with screws?
4. Where's the manual?

Four questions. Only the first one names the product. Every answer after that has to stay on PRD-101 without being told again.

Watching it get all the way through, still talking about the right product by the last question, was one of those small, very satisfying moments. Nothing flashy. Just the agent keeping up with a conversation the way a person would.

Then I switched products

Of course, staying on one product is the easy version.

Real people change their minds. They ask about one product, then say "what about PRD-102?" and keep going. Now "it" means something different than it did two questions ago.

So I made the test harder. Start on one product, switch to another partway through, and keep asking follow-ups. The agent has to follow what we're talking about now, not just the first thing anyone mentioned.

That one took more work than the first test. But it mattered, because an agent that remembers the wrong thing is arguably worse than one that doesn't remember at all. At least the forgetful one is obviously confused.

The next problem

By the middle of June, the website agent could hold a conversation about products. It knew what "it" meant, and it could keep up when you changed your mind.

Then I added project plans. And "what we're talking about" got a lot more complicated.