How should a model get your company's knowledge?

The problem, in plain words

Every company wants its AI to know how the company works: its policies, products, procedures and history. The usual answers are to paste everything into the prompt, to build retrieval (RAG), or to fine-tune a model. All three start from the knowledge. I start from the need.

The mistake is treating knowledge like a system prompt. A system prompt is always loaded, because it sets how the agent thinks, and that's right. Knowledge is different: no situation needs all of it, and every situation needs a different part. So the first step is to understand what the question, or the agent asking it, actually needs, and only then decide what to load.

That's progressive disclosure: give the model an entry point, let each link say what's behind it, and let it open only what the need calls for.

The picture

One question about workplace safety, answered three ways.

The options

Load it all up front. It works for a few pages, and it's the simplest thing to build. But most of what's loaded is irrelevant to the question, and it gets worse as the documents grow. Caching makes what's always loaded cheap. On Sonnet 5.5, at $2 per million input tokens with cache reads at 0.05 times that, 20,000 tokens cost about $0.04 a question uncached and about $0.002 cached. So cache what every question reads (the system prompt, the entry point), and don't take caching as a reason to load everything.

Retrieval as the strategy. Every question becomes a vector, and the passages closest in meaning are pasted in. It's very good at what it's for: finding meaning in a mass of text that has no structure, and doing it fast. I run it in production. On HappySkills, people describe the task they're trying to do, and search finds the skills that fit through Postgres and pgvector. As the strategy for a structured knowledge base, though, it chops procedures into passages and picks them by similarity, not by need. For exact words or every mention, it's the wrong tool altogether: that's full-text search, and Postgres has it built in. My verdict on OpenSearch is that it's overkill for most apps.

Progressive disclosure. The entry point is a README, a good one never more than 750 lines, with every other document cross-linked from it and each link saying what's behind it. On a website, the same thing is an llms.txt map with a Markdown copy of every page, leaner than HTML. The agent reads the entry point, decides which links the need calls for, ignores the rest, and loads them on demand. It doesn't stop at documents. Halfway through, the agent may decide it also needs some emails (a few specific ones, not all of them), some rows from the database, or some passages from an embedding index. Each is queried on the fly, where the agent thinks the answer is. It's progressive disclosure all the way, and retrieval becomes one of its sources.

This website is built that way. ask.cloudlesslabs.com is the map, with a note on every link saying what it answers, and every page has a Markdown copy. An assistant asked about a project reads the map, then only the pages the question needs. My /init-context skill does the same for a codebase: it reads the docs index first, then follows only the documents the task touches.

Fine-tuning. Training your own copy of a model changes how it writes more than what it knows, and knowledge changes faster than you'd retrain. I've never needed it in production. Keep it for a format or a style no prompt can hold.

A concrete scenario

A company keeps its workplace safety knowledge in three places: the safety manual and its procedures, an incident database, and years of written incident reports. Someone asks the assistant: "A colleague just cut their hand with a kitchen knife. What do we do?"

1. The need first: first aid, now, in the kitchen; then the paperwork.
2. The entry point: the README lists the manual's sections, one line each. The agent opens Workplace safety, then First aid.
3. Only what's needed: the first-aid procedure, and the incident form to fill in afterwards.
4. Other sources, on demand: today's first aiders on site, from the staff database; and, if it helps, the most similar past reports, found by meaning in the embedding index.

It read a handful of pages and a few rows. The assistant that loaded the whole manual paid for every page of it, and the one that only searched by meaning got passages from three different procedures.

The trade-offs

The rule of thumb

Knowledge isn't a system prompt. Understand the need first, then give the model an entry point and let it load only what the need calls for. Search by meaning is one of the places it can look, not the plan.

When the answer changes with scale

• A handful of documents: an entry page that links them is already progressive disclosure. Cache what's always read.
• A large knowledge base that changes every day: a README of at most 750 lines, every document cross-linked from it.
• Millions of tickets or emails, with nothing to navigate: add search by meaning with pgvector, as one more source.
• A reply needed within a second: a live chat has no time to follow links, so it's one search, then the answer.

Before you build on it

Check that it finds the right source. This part is verifiable, so make it a test. Write down real questions, each with the page or passage that holds the answer, and re-run the list after every change to the documents. Count how many reached the right source. If the safety assistant stops reaching the first-aid page, you find out before your staff do.

In Australia, the knowledge goes where the model does (Where may the data go?):

• Claude through Amazon Bedrock's Australian profile.
• The documents served from Sydney, behind a login if they're private. Bedrock has no server-side web search or code execution, so your own code fetches the pages for the model.
• When search by meaning is one of the sources, embeddings through Bedrock in Sydney (Titan Text Embeddings V2 is $0.026 per million tokens there), with the vectors in the client's own Postgres.