I Started Saving the Conversations

Knowledge Gaps showed me where the website agent knew it was struggling. They didn't show me anything else.

By the end of July, I had a loop I was proud of. When the agent couldn't answer something, the question got saved, a person could review it, and the knowledge got better.

But that loop only ever caught one kind of problem. The kind where the agent was honest enough to say "I don't know."

The problems it didn't know about

Think back to the rewards program. The agent didn't say "I don't know." It said "no," politely and confidently. That kind of answer would never show up as a Knowledge Gap, because as far as the agent was concerned, it had answered the question.

The same was true for a lot of other things. An answer that was technically right but missed what the person was really asking. An answer that was way longer than it needed to be. A follow-up that drifted off topic. A conversation where someone asked one question and just left.

None of those were gaps. They were just conversations, and once they ended, they were gone.

Starting to keep them

So I started saving the conversations.

The questions people asked. The answers the agent gave. And, just as important, what the agent found along the way: what search brought back and what the answer was built from. A transcript alone tells you what happened. The evidence behind it tells you why.

Keeping conversations, not people

Before I saved a single conversation, I had to decide what not to save.

People type all kinds of things into a chat window on a website. Email addresses. Phone numbers. Order numbers. Sometimes things they really shouldn't be typing into a chat window at all. I wanted the conversations so I could learn from the questions. I didn't want to collect information about the people asking them.

So the saved conversations are anonymous. There's no IP address, no browser details, no cookie and no customer identity attached to any of them. I didn't just assume that, either. I went back through the code and searched for every place that could have slipped one in.

Then, before anything gets saved, the stored copy is cleaned. Email addresses, phone numbers, numbers shaped like Social Security numbers, order and account numbers and card numbers all get masked out. The website agent still sees the original text while it's answering, because it needs to. Only the copy that gets kept is cleaned.

I also built automatic deletion, so saved conversations are removed after a set number of days. It's built and tested, but it's turned off for now. How long conversations should be kept isn't my call to make on my own. That decision belongs to the company, and the feature is waiting on it.

None of this was the exciting part of the project. But if you're going to keep what people say to your website, it's the part you have to get right first.

Real questions are different

Reading through saved conversations taught me something I'd suspected since May.

Back when the agent first started answering, I noticed I was mostly asking questions I already knew it could answer. I knew the website. I knew the data. Without meaning to, I was testing it the easy way.

Real questions aren't like that. They're shorter, or longer, or misspelled. They mix two questions into one. They use words the website never uses. They ask things nobody would ever think to put in a test.

That was the whole point. I didn't need more questions I could come up with myself. I needed the ones I never would have thought of.

You can't improve what you can't see, and you can't see what you didn't keep.

The next problem

Saving conversations meant I started finding bad answers I'd never have caught otherwise. And I fixed them.

But I'd already learned that a fix isn't permanent just because it worked once. Change something else in the system, and an old problem can come right back without anyone noticing.

Every bad answer I found was something I'd fixed once and hoped would stay fixed. Hope wasn't going to be enough.