Synthetic data for fine-tuning:
what to do when you have no labels.

The most common reason a team decides it cannot fine-tune a model is that it does not have a labelled dataset. No corpus of question and answer pairs, nobody with time to annotate ten thousand examples, no historical archive of correct behaviour sitting in a warehouse waiting to be exported. The conclusion follows quickly: fine-tuning is for companies with data, and we will get by on prompting.

The premise is usually right and the conclusion is usually wrong. For agentic work in particular, nobody has that dataset. The teams shipping fine-tuned agents did not find one either. They built it, mostly with a model and a script, and the human effort went somewhere other than labelling: into writing down what a correct outcome looks like precisely enough that a program can check it.

That is what synthetic data for fine-tuning means in practice. Not fabricated data standing in for the real thing, but generated attempts filtered by something that can tell a good one from a bad one. Get the filter right and the generator can be cheap. Get the filter wrong and no amount of generation helps.

What synthetic data for fine-tuning actually means

The term covers three different things, and conversations go sideways when people mean different ones. Generated inputs are the requests: plausible user questions derived from your own schemas, tickets, and documents, so you have something to train against without waiting for production traffic. Generated outputs are the responses and trajectories: what a model produced when it attempted a task with the real tools connected. Generated environments are the fixtures and staging replicas realistic enough that an attempt against them counts for something.

Most real pipelines use all three, and the important point about all of them is the same: synthetic describes where an example came from, not whether it is correct. Correctness is a separate question, answered afterwards, by a verifier. A generated example that survives a strict check is better training data than a hand-labelled one nobody re-read, because you know exactly what property it has.

The objection is aimed at the wrong bottleneck

"We don't have enough labelled data" assumes the scarce resource is labels. For a model that has to act, it is not. A label is one person's opinion about what the right output was. What you actually need is a way to decide, after the fact and without a person, whether an attempt worked. Once you have that, attempts are cheap: the model makes them, at whatever volume your GPUs allow.

So the question changes from "how many examples can we annotate" to "how many of our tasks have outcomes we can check". That is a much better question, because it is answerable in an afternoon, and because the answer is usually higher than people expect. A query either returns the same rows as the reference query or it does not. A reconciliation either balances or it does not. A generated report either cites figures that match the source system or it does not.

Three ingredients, then, and none of them is a labelled corpus.

Data generation for agent training, step by step

Data generation for agent training is a loop rather than an export. You assemble the task set, connect the tools, have the current model attempt each task several times, keep the attempts a checker accepts, train on the winners, and then run the improved model against the same tasks again. This is what expert iteration and rejection sampling refer to, and the reason it keeps paying is that every round produces better data than the round before.

Inside that loop there are five distinct places synthetic training data comes from, and they are not interchangeable. Most teams need three or four of them, because each one covers a gap the others leave.

Seeds from what you already have. Gateway logs, audit trails, support tickets, runbooks, and above all your API specifications and database schemas. None of these are training data yet. All of them constrain what a plausible request looks like, which is exactly what a generator needs to avoid producing generic nonsense.

Backwards generation from the spec. The counterintuitive one, and often the highest yield. You do not start from a question and try to produce the correct call. You start from calls, which you can sample correctly by construction from a JSON Schema or an OpenAPI document, execute them against a replica, and then work backwards to the request a person would have asked to motivate each one. Correctness comes free, because the call was generated valid.

Rejection sampling from the current model. Let the model under training attempt real tasks many times and keep only what the verifier accepts. This is the source that scales with compute rather than with headcount, and the only one that improves as the model does.

Distillation from a stronger model. A larger model generates the attempts, a smaller one trains on the verified subset. Effective and cheap, with two caveats that matter more than the technique: check the licence terms of whatever produced the data, because several major providers prohibit training competing models on their output, and note that sending your tasks to a hosted model means sending your schemas and your business logic to a third party.

Perturbation and adversarial variants. Take a verified example and deliberately break it: the malformed argument, the filter that returns zero rows, the timeout, the ambiguous request where the right move is to ask rather than guess. Then record the recovery. This is the category teams skip, and it is why so many fine-tuned agents are excellent until the first error and useless afterwards.

What one synthetic example carries

A generated example should record how it was made. This feels like bookkeeping until the first time a metric moves and you need to know which generator, seed, or verifier version was responsible. Provenance is also what makes a bad batch removable: when you discover that one checker was too loose, you want to delete exactly the examples it admitted, not retrain from scratch.

{
  /* what the trainer reads */
  "tools":    [ /* the schemas production will send */ ],
  "messages": [ /* goal, assistant turns, tool results */ ],
  "loss_mask": "assistant_only",          // never the tool results

  /* what the trainer ignores and you cannot do without */
  "provenance": {
    "source":      "rejection_sampling",  // or spec_backwards, log_seed
    "generator":   "ckpt-0412",           // which model produced it
    "seed_task":   "recon/fx-break-004",  // which task it came from
    "attempt":     3,                     // of 8 sampled
    "verifier":    "recon_balance@v2",    // and which version
    "verdict":     "pass",
    "task_hash":   "9f2c...",             // for dedup, not the wording
    "env":         "staging-replica-07"
  }
}

Two details in there are load-bearing. The loss mask, because training on tool result tokens teaches a model to write plausible API responses itself, which is the precise habit that turns an assistant into one that reports a position it never queried. And the task hash, because deduplication has to happen at the level of the underlying task, not the wording. Forty paraphrases of the same request are one example wearing forty hats, and a generator left unsupervised will produce them all day.

The verifier is the thing you are actually building

If there is one sentence to take from this: the quality of a synthetic dataset is set by its checker, not its generator. A weak checker with a strong generator gives you a large pile of confidently wrong examples. A strong checker with a mediocre generator gives you a smaller pile of correct ones, which is what you want.

Verification is not binary either, it is a ladder. The rungs at the top are mechanical, effectively free, and unlimited. Each step down judges something fuzzier and costs more per example, until the bottom rung is a person reading it. The right choice is the highest rung that genuinely settles whether the task succeeded, and being forced further down is a constraint on how much data you can make rather than a preference.

That last line is the real constraint, and it is worth saying plainly rather than discovering late. Tasks that only a person can judge are not trainable at volume, at least not affordably. If the workflow you want to automate is entirely of that kind, the honest answer is to reshape the task until part of it is checkable, or to spend the scarce human review on evaluation instead of training data.

Quality control lives in the funnel, not the generator

Once generation is running, the interesting number is not how much data you produced. It is how much survived, and where the losses were. Instrument each stage and the funnel becomes a dataset health metric you can watch across rounds.

Four checks are worth running on every batch before it goes near a training run. Deduplicate by task, not by string, using the hash of the underlying task and arguments rather than the phrasing. Track diversity of what survives: count distinct endpoints, argument shapes, and trajectory lengths, because filtering is itself a selection pressure and it tends to select for whatever the generator finds easy. Balance difficulty deliberately: a set where every task passes on the first attempt teaches nothing, so keep the tasks that pass sometimes. Check for contamination against your evaluation set, by task rather than by exact text, or your numbers will be measuring memorisation.

How much synthetic training data an LLM actually needs

Less than the objection assumes, and the amount depends on what kind of thing you are teaching rather than on how much you can generate. Format and convention, such as always emitting a particular call shape or always using your internal identifiers, are the cheapest thing to teach: a few thousand verified examples usually move it, and sometimes fewer. A catalogue of internal tools sits in a similar range, scaling with the number of endpoints and their argument complexity rather than with total volume.

Multi-step judgement is different. There, what buys you improvement is more attempts per task rather than more tasks, because the signal lives in the variation between attempts on the same problem. Eight rollouts on two hundred tasks teaches more about when to stop searching than one rollout on sixteen hundred.

Across all of it, one rule holds: coverage beats volume. A thousand examples spread across every endpoint, error path, and awkward edge case will outperform fifty thousand paraphrases of the same three requests, and it will do it while being cheaper to train and faster to debug. When a run disappoints, the fix is almost always a gap in the task set rather than a shortage of examples.

Model collapse, and the failure modes that arrive first

The published worry about synthetic data is model collapse: train on model output for enough generations and the distribution narrows until quality degrades. It is a real phenomenon, but it has a specific shape, namely repeatedly training on unfiltered generated text with no external signal. A verified loop does not have that shape. The verifier rejects the examples that drift, and the real tools supply observations no model invented. That said, the guards are cheap, so keep a real held-out evaluation set, deduplicate hard, watch the diversity of the surviving set, and mix in general instruction data so the model does not narrow to one task family.

In practice, four other failure modes show up long before collapse does.

Verifier gaming. Any checker that can be satisfied without doing the work eventually will be. A model rewarded for a passing test suite may learn to weaken the tests. Read a random sample of high-scoring attempts by hand every round, because a rising pass rate is not evidence that the check means what you think it means.

Distribution drift from production. Generated requests come out cleaner than real ones: better spelled, better formed, more complete. Train only on those and the first ambiguous message from an actual person lands somewhere the model has never been. Seed the generator from real phrasing, and keep the messy examples you do have.

Environment mismatch. If generation ran against a pristine replica while production has rate limits, stale caches, and partial outages, you have trained an agent for a world that does not exist. Put the real failure modes into the generation environment on purpose.

Teacher fingerprints. Distilled data carries the teacher's habits, including its hedging, its formatting tics, and occasionally its refusals. Those transfer along with the capability, and they are much harder to remove afterwards than to filter for at generation time.

The one thing that cannot be synthetic

Your evaluation set. It has to come from real traffic or real human-written tasks, be held out completely, and never be used as a seed for generation. Generate your evaluation set from the same pipeline that produced your training data and you will measure how well the model matches your generator, which is a number that goes up reliably and means nothing.

This is also where scarce human attention belongs. A few hundred carefully curated real tasks, reviewed by the people who understand the workflow, is worth more than ten thousand hand-labelled training examples. Synthetic for training, real for judging.

Why generation belongs inside your environment

Generating this data requires the real tools connected to real systems, which means the generation step is not a data problem you can hand off. To produce one verified trajectory, the pipeline touches internal endpoint names and argument shapes, database schemas, ticket contents, and the intermediate state of whatever process the task describes. The resulting dataset is a machine-readable description of how your firm operates, assembled in one place, which is a more sensitive artefact than any of the documents that went into it.

That constrains where the compute can sit. If the tools have to run, the generation has to happen where those systems already are, and the checkpoints trained on the result should not need to leave either. Which is a large part of why the teams with the strictest data requirements are often the ones fine-tuning most aggressively: once generation is inside the walls, the labelled-data objection stops being about data and starts being about infrastructure.

Where this fits at Numerata

P95 is Numerata's training layer, and a generation loop is exactly the shape of work it is built for: many runs rather than one, rollouts sampled and compared, each round grouped and tracked so you can tell which change to the task set or the checker moved the pass rate. Generation, verification, and training run as ordinary jobs on infrastructure you already control, drawing GPUs from the pool Lupine manages and handing finished checkpoints to NinetyFive to serve. The seeds, the generated trajectories, the tools they touched, and the weights they produced all stay inside your environment.

For the neighbouring questions: agentic fine-tuning covers what you do with trajectories once you have them, tool-use fine-tuning covers the single-call version of this problem and what one record looks like in detail, and SFT vs. RL covers how to choose an objective once you have a verifier worth trusting. For whether that verifier is measuring the job you actually care about, see evaluating agents. If you are earlier in the picture, start with what fine-tuning is, and for the constraint that shapes all of it, see air-gapped LLM training.