Training a language model
for your internal wiki.

Internal knowledge search was one of the six workloads in our roundup of private AI for internal tooling. It is also the one teams start with, because the pitch writes itself: you already have ten thousand pages of Confluence, Notion, or SharePoint, everyone complains that search is useless, and a model trained on all of it would finally answer the questions the wiki was supposed to answer.

The pitch is right about the problem and usually wrong about the method. Pointing a training run at a wiki export does teach the model something valuable. It also teaches the model every superseded policy, every runbook for a system that was decommissioned last spring, and every half-finished draft somebody left in a personal space, and it teaches all of that with exactly the same confidence as the pages that are still true.

This post is the build, in the order we would actually do it: what training buys you that retrieval cannot, why the corpus is the hard part rather than the model, how to construct the dataset from material you already have, and how to keep the thing current once the wiki changes underneath it.

The short version

Train the behavior, retrieve the facts. The weights should carry how your organization writes, names things, and reasons. A live index should carry what is true today. Teams that invert this get a model that is fluent, internally consistent, and eight months out of date.

Your wiki is not a dataset yet. The single highest leverage day in the project is the one spent sorting pages into current, superseded, duplicated, and abandoned. Everything downstream is capped by that.

Refusal is a trained behavior, not a guardrail you bolt on. A model that has never seen an unanswerable question in training will answer all of them.

The evaluation set comes before the training set. Fifty real questions from real employees, with the answers a knowledgeable colleague would give, is the artifact the whole project is measured against.

What "training on the wiki" actually means

Training a language model for an internal wiki means adapting an open-weight model so that it answers questions in your organization's own vocabulary, format, and citation habit, while the facts themselves are supplied at answer time by a retrieval index over the current corpus. The phrase covers three different techniques that get conflated constantly, and the confusion is expensive, because they fail in different ways.

Continued pretraining runs the base model further on raw wiki text. It is the one people picture when they say "train it on our docs", and it is the one that most reliably disappoints: it moves vocabulary and style, it does not reliably install retrievable facts, and it bakes in whatever was wrong in the corpus on the day of the export. Supervised fine-tuning trains on question and answer pairs, which is where the useful behavior comes from. Retrieval keeps a live index and hands the model passages at inference. We covered the general mechanics in what fine-tuning is; what follows is what changes when the corpus is a wiki.

Your wiki is not a dataset

Every internal corpus we have looked at has the same four-way split, and the ratio is worse than anyone on the team expects before they measure it. There are pages that are current and authoritative. There are pages that were superseded but never unpublished, which are the dangerous ones, because they are well written, they rank highly in the wiki's own search, and nothing about them announces that they are retired. There are duplicates, the same procedure written three times in three team spaces with small divergences. And there is abandonment: drafts, meeting notes, personal spaces, runbooks for systems that no longer exist.

Train on all four and you have taught the model that all four are equally true. Worse, you have destroyed the one signal an employee currently relies on: a wiki page carries a visible last-edited date and an author, and people discount it accordingly. A generated answer carries neither, so a stale fact that a human would have squinted at now arrives with no hedging at all.

The triage is unglamorous and it is the project. Pull the edit metadata you already have, which every wiki platform exposes: last edited date, editor count, view count over the last ninety days, inbound links from other pages, and whether the page has an owner who still works there. Those five signals sort most corpora quickly. A page nobody has viewed in a year, with one editor, no inbound links, and an owner who left, is not knowledge, it is sediment.

What fine-tuning actually fixes

Once the facts are in the index, the case for training gets much clearer, because the remaining problems are all behavioral. A well-configured retrieval pipeline over a clean corpus, driven by a general-purpose model, still fails in ways no amount of reranking repairs. It does not know that in your organization "the platform" means one specific service rather than the industry term. It cannot tell that a question about "Atlas" concerns the billing migration and not the observability stack that shares the codename. It answers a runbook question in three flowing paragraphs when the only useful shape is numbered steps that somebody can execute at two in the morning. And it answers when it should decline.

These are learnable from a few thousand examples, and they are the difference between a demo and something an employee reaches for twice a day.

Symptom, cause, and which layer fixes it

The most common mistake in this project is applying a training fix to a retrieval problem, which is slow and expensive and does not work. The reverse mistake, tuning the retriever to fix a behavior, is cheaper but equally futile. The diagnosis is usually straightforward once you ask whether the right passage reached the model.

Building the training set from material you already have

The good news about a wiki project is that the dataset is largely lying around in other systems. You do not have to invent questions; your colleagues have been asking them for years, in writing, with the answers attached.

The richest source is your helpdesk. Every ticket that was resolved by somebody pasting a wiki link is a question and answer pair with a verified source, written in the actual language employees use rather than the language the documentation uses. That gap between how a question is asked and how the answer is written is precisely what the model needs to learn, and it is invisible in the wiki itself.

Next, wiki search logs, specifically the failed ones: queries that returned results nobody clicked, or that were refined three times in a row. Those are questions the corpus answers badly, and they tell you where to write pages as much as where to train. Then chat channels where somebody asked, somebody answered authoritatively, and a page was linked. Then the page edit history itself, which is the only record that shows a fact changing, and is therefore where your conflict and recency examples come from.

Only then fill gaps synthetically. Generating questions from current pages works well for coverage, and badly as the primary source, because synthetic questions inherit the vocabulary of the document rather than the vocabulary of the person asking, which trains the exact mismatch you were trying to fix. We wrote up the technique and its failure modes in synthetic data for fine-tuning. The rule that has held up for us: real questions, synthetic coverage.

{
  "question": "who do i ping to get prod access for the
               billing box",          // AS ASKED, NOT AS DOCUMENTED

  "context": [
    {"page": "security/prod-access-policy",
     "as_of": "2026-08-19",            // 1. EVERY SOURCE DATED
     "span":  "Requests are raised in #access-requests and
               approved by the service owner of record..."},
    {"page": "eng/access-requests",
     "as_of": "2023-04-02",            // 2. THE SUPERSEDED PAGE
     "span":  "File an ACCESS-1 ticket with the platform
               team, who will grant within 2 business days."}
  ],

  "answer": "Raise it in #access-requests; the approver is the
             service owner of record for billing-api, currently
             R. Okafor. [security/prod-access-policy, 2026-08-19]
             Note the older ACCESS-1 ticket route in
             eng/access-requests is out of date as of 2026-08.",
                                       // 3. PREFERS NEWER, SAYS SO

  "refuse_if": "caller lacks read on security/*"
                                       // 4. THE EDGE IS EXPLICIT
}

// WHAT MAKES THIS EXAMPLE WORTH ITS PLACE IN THE SET IS THE
// SECOND CONTEXT ENTRY. A PAIR WITH ONLY THE CORRECT PAGE
// TEACHES THE MODEL TO SUMMARIZE. A PAIR WITH BOTH TEACHES IT
// TO CHOOSE, WHICH IS THE BEHAVIOR YOU ARE ACTUALLY BUYING.

Teaching it to say no

Refusal is the behavior teams add last and should add first. An internal assistant sits in front of a corpus with hard edges: questions about compensation, about an acquisition nobody has announced, about a team space the asker cannot read, about a system that was decommissioned and whose runbook is gone. A model that has only ever been trained on answerable questions will produce an answer to every one of them, and the answers will be good enough to act on.

Build the unanswerable set deliberately, and make it a meaningful fraction of the training data rather than a token handful. Three kinds earn their place. Out of scope: the corpus genuinely does not cover it, and the right answer names a human who would know. Out of bounds: the corpus covers it but this caller is not entitled to it, which must be enforced at the index and then also trained, so the model does not reason aloud about a document it was not given. And out of date: the only source is a page flagged retired during triage, where the right answer says so rather than reciting it.

Note the second one carefully. Permissions are enforced at query time, in the index, by filtering to what the caller can read, exactly as we argued for entitlements in the trading desk research agent. Training is not the control. Training is what stops the model from narrating the shape of the thing it could not retrieve, which is its own kind of leak.

Evaluating it before anyone trusts it

Write the evaluation set before the training set. Fifty real questions, collected from the helpdesk and the search logs rather than invented in a planning meeting, each with the answer a knowledgeable colleague would give and the page that answer comes from. This is the most useful artifact the project produces, and it keeps its value long after the first model is replaced.

Four numbers are worth tracking, and only one of them is the obvious one. Retrieval hit rate: how often the page that answers the question appears in the top twenty. Fix this before anything else, because it caps everything downstream and no amount of training recovers a passage that never arrived. Citation accuracy: how often the cited page actually supports the claim, checked by a grader rather than by eye. Staleness rate: how often an answer rests on a page flagged retired, which is the number that decides whether the desk keeps trusting the thing. And refusal precision, in both directions, since a model that refuses everything scores beautifully on the other three. The general argument for building your own task set instead of reading a leaderboard is in evaluating agents.

Keeping it current

A wiki changes every day, and this is where the discipline of the split pays for itself. Two clocks run at different speeds, and the whole design question is which facts are on which clock.

One operational habit is worth adding on top. Route every refusal and every low-confidence answer back to whoever owns that part of the wiki. The assistant then becomes the best documentation gap detector the organization has ever had, because it reports, continuously and without complaint, exactly which questions the corpus cannot answer. Most wiki programs have never had that signal.

A build order that survives contact

Roughly six weeks for a small team, ordered so that each step produces a number rather than a belief.

1. Collect fifty real questions. From the helpdesk and the search logs, with the answer a knowledgeable colleague would give and the page it comes from. No model work yet.
2. Triage the corpus. Sort into current, superseded, duplicated, and abandoned using edit metadata, inbound links, views, and whether the owner still works there. Expect this to be the week that surprises people.
3. Build retrieval only. Hybrid lexical and dense, chunked so tables and headings stay together, permissions filtered at query time. Measure hit rate against the fifty. Fix that number before touching a training script.
4. Ship retrieval to a small group. A general model over a clean index is already useful, and a month of real usage produces the corrections and the phrasing that make the training set good instead of adequate.
5. Then fine-tune on behavior. Pairs from the helpdesk, conflict examples from the superseded pages, unanswerable examples from the edges. Parameter-efficient methods are enough here, which keeps a training run cheap enough to repeat.
6. Re-run the evaluation and compare honestly. All four numbers, against the same fifty questions, against the retrieval-only baseline from step four. If citation accuracy and refusal precision did not move, the training set is the problem, not the method.

The step people want to skip is the fourth, and it is the one that makes everything after it work. Shipping retrieval alone buys a month of employees telling you, in their own words, what they actually ask and where the answers are wrong. There is no substitute for that, and it cannot be recovered later.

Where this fits at Numerata

An internal wiki is the densest single artifact of how a company works: its architecture, its incidents, its unannounced plans, its arguments. Sending that corpus to an external API, chunk by chunk, every time somebody asks a question, is a decision worth making deliberately rather than by default, which is the same argument we made about source code and AI assistants.

The whole pipeline described here runs inside your own environment. Fine-tune on your own tickets, search logs, and triaged corpus with P95, run the training and the spiky weekday inference load on Lupine compute that scales to zero overnight, and serve the assistant through NinetyFive inside your own trust boundary. Private cloud, on-premises, or fully air-gapped, with the index, the weights, and the query logs all somewhere your own security team can reach.

Related reading: the wider set of internal tooling use cases this post drills into, the basics in what fine-tuning is and what a small language model is, and the two posts this one leans on hardest, synthetic data for fine-tuning and evaluating agents.