Then I Met PDFs

Manuals felt like the easy win.

They were official. They were complete. They were written by people who knew the products inside and out, and they'd been reviewed before anyone printed them. If I was looking for a source of truth for a website agent, it was hard to imagine a better one.

They were also PDFs.

Manuals were on both lists

Back in March, when I wrote down the questions people actually ask, manuals showed up on both lists. On the product side, a manual was one more detail. On the support side, it was where people went when they were standing in front of something they were trying to put together or fix.

The crawler had already been collecting manual links from the product pages. So in my head, the manuals were half done. I had the files. I just needed to pull the text out.

How hard could that be? (I really need to stop asking that.)

A PDF is made for paper

The first time I pulled the text out of a manual and actually read it, I understood something I'd never had a reason to think about before.

A PDF isn't really a document the way a web page is. It's a set of instructions for where to put things on a printed page. It knows exactly where each word sits, but it doesn't really know what the words are for. Which part is a heading. Which part is a table. Which part is the same footer that appears on every single page.

So when you pull the text out, you get all of it, in roughly the order it was drawn on the page.

Something like this:

SAFETY INSTRUCTIONS                                   3
Read all instructions before use.
Model PRD-104     Rev. 2
Lea todas las instrucciones antes de usar.
1/2"   3/4"   1-1/2"
12.7   19.1   38.1
Model PRD-104     Rev. 2                              4

(That's a made-up example, but it's very close to what came back.)

The header and footer repeat on every page. Page numbers end up in the middle of sentences. A clean, readable table on paper turns into rows of numbers with no labels attached. And somewhere in the middle of all of it is the actual information a customer needs.

Then there were the other languages

I expected messy formatting. What I didn't expect was to scroll through the text of a manual and suddenly be reading something that wasn't English.

Some of the manuals were written for more than one market. The English instructions were in there, but so were whole sections in several other languages, translated and laid out right alongside them. On paper, that makes perfect sense. You flip to your language and ignore the rest.

A website agent can't flip to its language. It reads everything.

That means if someone asks a question, the agent might find the right step in the right manual, written in a language the customer can't read. Or it might find three versions of the same safety warning and treat them like three separate pieces of information. That was the moment I stopped thinking of manuals as the easy part.

You can't just delete what isn't English

My first thought was simple: keep the English, throw away everything else.

That's harder than it sounds. A lot of the most important content in a manual doesn't look like English either. Measurements. Dimensions. Part numbers. Rows from a table that are mostly numbers and fractions. If I'd told the system to throw away anything that didn't read like an English sentence, I'd have thrown away exactly the details people ask about most.

So I had a real problem, and no clean way to solve it yet.

And then I ran into another interesting problem. Some products shared the exact same manual. Different products, same PDF, word for word. I hadn't really thought about that. I couldn't just treat every manual as a separate document, so I had to figure out how to recognize duplicates while still keeping track of which products each manual belonged to.

Not solved yet

I'd like to say I figured this out in April. I didn't.

What I did was understand the problem. The manuals were valuable, and I wasn't going to leave them out. But I also knew that pulling text out of a PDF and calling it knowledge wasn't good enough. Some of what came back was noise, some of it was in the wrong language, and some of it was the most important information in the whole file.

The fix came later. When it did, it turned out to be a lot smaller than I'd feared. I didn't need to rebuild how PDFs were read. I needed to be smarter about what to keep. But that's a story for another post.

The most trustworthy-looking source can be the messiest one to use.

Somewhere to put it all

By the end of this stretch, I had a lot of text. Product details from the crawler. Support and policy pages. Manuals, flaws and all.

What I didn't have was somewhere to put it where a website agent could actually search it.

That was the next decision, and I already had a pretty good idea what I wanted to do. I wanted to keep it boring.