Your AI Didn't Get the Memo

You roll out an AI tool to your team. You spend a few weeks getting the documentation in shape, cleaning up Confluence pages nobody had touched in two years, making sure the right policies are in the right folders. (Maybe a previous issue of WIP even inspired the audit.) Things work.

A month later, someone in HR asks the AI about a policy that was updated last Tuesday. The AI confidently returns the version from before the update.

You check. The documentation is fine. The model is fine. The thing in between — the pipeline that is supposed to keep the AI's view of your data in sync with the actual data — is silently out of date. The policy went out, but the AI didn't get the memo.

So how does this work, and what do you do about it?

In this issue of Work in Progress, we asked Vardan, our backend engineer, to walk through what has to be true for an AI tool to give an answer that actually reflects your current data.

Here's the problem.

Enterprise data doesn't live in one place. It's in Confluence, Jira, Google Drive, HR tools, shared drives, and each one has its own format, structure, and update frequency. Getting all of that into a shape that produces quality embeddings isn't trivial. It's more of the work than people expect.

And that's just getting the data in. The harder part is keeping it current.

A document changes, the index hasn't caught up, and later someone asks a question that touches it and gets the version from before the change. The AI answers confidently, and it's wrong. Not because the model failed, but because the pipeline feeding it fell behind.

A lot of this is mechanical. Not every source system supports webhooks that reliably track every change in real time, so the index keeps answering from a snapshot that's slowly sliding out of date. Nobody notices until someone acts on the wrong answer.

So if you're evaluating an AI tool, the question isn't just "is it accurate." It's "how does it stay accurate, and what happens the moment a document changes."

Here's what "AI acts on the wrong answer" actually costs.

A Toronto man asked a BMW dealership to buy back his car, and an AI chatbot named "Quinn" quoted him about CA$27,000 — roughly CA$7,000 more than the dealership thought the car was worth.

The number was just wrong. The dealership tried to walk it back, then had to honor it once a reporter started asking questions. Quinn cost them five grand and never even had to file an expense report.

It's a small, almost funny example, but it's the exact failure we think about while building Leebry: the AI confidently produced a number that didn't match reality, and a real decision got made on it before anyone caught it.

The broader pattern is less funny. In a survey this year, half of consumers said they blame company leadership, not "the tech," when AI gives them a bad answer. So when the bot is wrong, it's not the bot's problem. It's yours.

What does all of this mean for you?

Aside from the obvious (apparently you can negotiate a car buyback through a chatbot now), the real takeaway is to find where in your own stack a confident wrong answer could quietly become a real decision, before anyone notices.

So here's a quick audit: pick a document you know changed recently — a policy, a price, an SOP — and ask your AI about it.

If it hands back the old version, the model didn't fail, it just never got the memo. Better you find that out than a customer does.

That's Issue #5.

— The Leebry Team