The Products Weren't the Hard Part

Once I had ten products crawled, I started writing down the questions I figured people would ask.

What does this product do? Which one should I buy? What are the specs? Does it come with a manual?

It was a good list. Every question on it lined up neatly with something the crawler had just pulled back. Description, details, specifications, key features, manuals. I had a field for almost all of it.

Then I wrote down the questions people actually ask when they come to a company's website, and the list looked completely different.

The other list

I started thinking about my own experiences shopping online. What do I do after buying something? Where do I go when I have a question or something doesn't work? I asked a few people how they shop online and where they turn when they need help. And honestly, I'd heard plenty of frustrations over the years from people who just wanted a simple answer and couldn't find one. It got me thinking that maybe I needed to look beyond just the products.

People don't only come to a website to learn about a product. A lot of the time, they've already bought it. Or they're about to, and they want to know what happens next.

So the real list looked more like this:

• Can I return this?
• How long will shipping take?
• What does the warranty cover?
• How do I register my product?
• Where can I find the manual?
• Who do I talk to when something's wrong?

Returns. Shipping. Warranties. Registration. Manuals. Policies. Support.

My ten crawled products could barely answer any of it.

Remember the rest of the sitemap?

In the last post, I mentioned that only about 249 of the 526 URLs looked like product pages. The rest were category pages, content pages, support and policy pages, and corners of the site most people never see.

I'd skipped all of that on purpose. I was building something that understood products, so I went after the products.

It turns out a lot of the answers people were actually looking for were sitting in the part of the sitemap I'd ignored.

That was a slightly uncomfortable realization. Not because anything was broken, but because my whole plan had been shaped around the data I found easiest to think about, not the questions people would actually bring.

These questions felt different

There was something else about the second list that took me a minute to put into words.

If a website agent gets a product description slightly wrong, that's bad. Somebody might be confused, or buy the wrong thing.

If it gets a return policy wrong, or makes up what a warranty covers, that's worse. Someone is going to act on that answer. They're going to keep a receipt, or throw it away. Ship something back, or not. Expect a replacement that isn't coming.

This is where the instinct from the very first post stopped being abstract. I'd said I didn't want an AI that made things up. Looking at this list, I could see exactly why. These were the questions where a confident, made-up answer would do the most damage, and also the questions where a model would be most tempted to fill in the blanks with something that sounded reasonable. Most companies have a return policy. Most products have some kind of warranty. A model can write a very believable version of either one without knowing anything about ours.

Every one of these answers had to come from the actual page that said it.

Manuals were on both lists

One thing showed up on both lists: manuals.

The crawler had already been pulling manual links from product pages. On the first list, a manual was just one more product detail. On the second list, it was a support question. Someone who asks "where's the manual?" usually isn't shopping. They're standing in front of something they're trying to put together or fix.

At the time, a manual felt like the easy one. It was already a link in the data. I'd find out later that getting a website agent to actually read a manual is its own story.

Redrawing the sticky note

So I went back to the sticky note.

The first box, "crawl website," had quietly assumed that "website" meant "products." It didn't. The plan needed to look more like this:

crawl website
  ├── product pages
  ├── support and policy pages
  └── manuals
        ↓
give content to AI
        ↓
customer asks a question
        ↓
AI answers

That's still a simple diagram. But it was the first time the plan changed because of something I learned, not something I'd planned. And it wouldn't be the last.

Build for the questions people actually ask, not the data you happen to have.

More than a catalog

It was still March. I'd crawled ten products. And the project had already shifted underneath me once.

I'd started out thinking I was building something that could understand a product catalog. What I actually needed was something that could understand a website, including all the pages that existed because customers had questions after they clicked "buy."

That meant the crawler I'd been so pleased with in a ten-product test now had to deal with the whole website. Product pages, support pages, policy pages, manuals, all of it, and all of it built a little differently.

I was about to find out how different those two things are.

The products weren't the hard part. They were just the first part.