SFT or RL?
Choosing how to train an agent

Supervised fine-tuning teaches a model to imitate answers that worked. Reinforcement learning fine-tuning teaches an LLM to produce answers that score well, including ones no demonstration contained. For a single-turn task the gap between those two is mostly a question of polish. For an agent running twenty tool calls in a row, it is the difference between a model that looks right and a model that finishes.

This post is a decision guide, not an argument for one side. The honest answer is that nearly every team ships both, in a specific order, and the useful question is which one addresses the failure you are actually looking at. Here is the short version, before the detail.

The short answer

Use SFT when the failures are mechanical. Malformed tool arguments, wrong output schema, forgetting it already fetched something, narrating a plan instead of executing it. These are conventions, and imitation teaches conventions efficiently.

Use reinforcement learning when the failures are judgment. The agent calls valid tools in a sensible format and still picks the wrong next step, keeps searching after it has enough, or gives up one step before the answer. No amount of extra demonstrations reliably fixes this, because the problem is what the model does in states the demonstrations never visited.

Use preference-based methods when success is subjective. Tone, helpfulness, refusal behavior, and anything a program cannot grade. This is the RLHF lane, and it is a different problem from agent competence.

And the sequencing rule that follows from all three: SFT buys you reward variance, reinforcement learning buys you the states SFT never saw. You generally need the first before the second does anything at all.

What SFT actually optimizes

Supervised fine-tuning minimizes token-level cross-entropy against a recorded answer. Show it a prompt and a reference completion, and every token of that completion becomes a target. It is stable, cheap, and completely determined by the data you feed it, which is both its strength and its ceiling: a model trained this way is trying to be indistinguishable from your demonstrations, so it cannot be better than them.

The subtler problem is where the training data comes from. SFT is off-policy. Every state the model is trained on was produced by whoever or whatever generated the demonstrations, not by the model being trained. At inference, the model generates its own state, and the moment it makes a mistake it is somewhere the training set never described. This is the distribution shift that makes multi-step tasks hard, and it compounds: a model that takes the right action 95 percent of the time has roughly a 36 percent chance of getting twenty consecutive steps right. Per-step accuracy that reads as excellent produces a task success rate that reads as broken.

You can push back on this within SFT by making sure your data contains recoveries, which is the core idea behind agentic fine-tuning: keep the trajectories where the agent made a bad call, read the error, and corrected. That helps a great deal. It still only covers the failures somebody thought to record.

What reinforcement learning fine-tuning actually optimizes

Reinforcement learning fine-tuning inverts the data flow. The model attempts the task itself, a reward function scores the attempt, and the update raises the probability of the attempts that scored well and lowers it for the rest. Nothing is compared against a reference string. The objective is the outcome.

Two consequences matter for agents. First, the training data is on-policy, generated by the current model, so it is concentrated exactly on the states the model actually reaches, including the messy ones it reached by erring. That is a direct answer to the compounding failure above. Second, the model can discover strategies absent from every demonstration, because it is rewarded for the result rather than for the route.

The price is that you now need a reward, and every serious difficulty in reinforcement learning fine-tuning is really a question about where that reward comes from.

RLHF vs SFT, and what RLVR changed

Most people meet reinforcement learning fine-tuning through RLHF, which is worth separating from the agent case because it solves a different problem. In RLHF, humans compare pairs of model outputs, a reward model is trained to predict those comparisons, and the policy is optimized against that learned reward. It works well for exactly the thing it was built for: qualities no program can measure, such as whether a response is helpful, appropriately hedged, or pleasant to read.

For agents it is a poor fit, for two reasons. Human preference data is slow and expensive to collect, so you get thousands of comparisons rather than millions of rollouts. And a learned reward model is a proxy, which means it can be satisfied by outputs that look like what raters approved without being correct. Optimize hard enough against a proxy and you get a model that is persuasive about work it did not do.

RLVR, reinforcement learning with verifiable rewards, is the shift that made this practical. Throw away the learned reward model and let a program decide: the test suite passes or it does not, the number matches the ledger or it does not, the query returns the reference rows or it does not, the sandbox ends in the required state or it does not. The reward is cheap, deterministic, and available in unlimited quantity, which is the property that actually matters, because reinforcement learning is sample-hungry and a human-in-the-loop reward cannot feed it.

This is why the practical question for an agent team is rarely "SFT or RL". It is: can we write the checker? Teams that can have a path to reinforcement learning. Teams that cannot are, whether they frame it that way or not, restricted to supervised methods until they build one. Investment in the grader is investment in the ceiling of everything that follows.

GRPO, and why agent training converged on it

PPO, the algorithm behind classic RLHF, trains a second network alongside the policy: a value model that predicts the expected return from a given state, used as a baseline so the update knows whether an action was better or worse than typical. That critic is roughly the size of the policy, which doubles memory, and on long agent trajectories with a single reward at the very end it is being asked to estimate something it has almost no signal for.

GRPO, Group Relative Policy Optimization, deletes it. For each task, sample a group of rollouts, score them all, and use the group's own mean as the baseline. A rollout that beat its siblings gets a positive advantage, one that trailed them gets a negative one, and the normalization is done within the group.

# For one task, sample a group of G complete rollouts
rollouts = [policy.rollout(task) for _ in range(G)]
rewards  = [verifier.score(r) for r in rollouts]   # RLVR: a program, not a reward model

# The group is its own baseline. No value network anywhere.
advantages = [(r - mean(rewards)) / (std(rewards) + eps) for r in rewards]

# If every rollout scored the same, std is 0 and every advantage is 0.
# That group contributes nothing. Task difficulty is a training parameter.

That last comment is the part teams underestimate. A group whose rollouts all succeed teaches nothing, and a group whose rollouts all fail teaches nothing either. Learning happens only where the model is inconsistent, so your task mix has to sit near the model's current pass rate, and it has to keep moving as the model improves. Task curriculum stops being a nicety and becomes a thing you actively manage, alongside the usual KL penalty that keeps the policy from drifting too far from the reference model.

Where each approach wins

Four methods, laid out against the questions that actually decide between them.

Why multi-step tasks break the tie

Everything above sharpens when the task has a horizon. Consider what each method can even express about a twenty-turn trajectory that failed at turn fourteen.

SFT has nothing to say about it, because the trajectory is not in the training set. It only ever sees successes, or successes annotated with recoveries somebody curated. The failure that will actually sink the agent in production is, by construction, absent.

Reinforcement learning has a great deal to say, but it has to solve credit assignment: one reward arrives at the end, and thirteen good turns preceded one bad one. Outcome-only rewards handle this statistically, by averaging over many rollouts until the turns that correlate with success get reinforced. It works, and it is why sample-hungriness is not an implementation detail but the core cost.

The tempting shortcut is a process reward model that scores each step, giving a much denser signal. It genuinely speeds things up, and it reintroduces the RLHF problem in a new place: you are now optimizing against a learned judge of intermediate steps, which is easier to game than a final outcome and more expensive to label. Most teams do well to run outcome-only until they can prove it has stalled.

There is a related trap on the SFT side worth naming. Because trajectories are long, a single training example contains many tokens, and a dataset of a few thousand trajectories can look large by token count while covering very few distinct situations. Diversity of tasks, not volume of tokens, is what SFT is short of in agent training.

How to actually decide

Three questions, in order. They are ordered by cost, so the first yes is usually the right answer to act on.

What reinforcement learning needs that SFT does not

The methods differ less in their math than in what they demand from the infrastructure around them. Three requirements have no SFT equivalent, and each one is where projects stall.

The cost difference, concretely

SFT cost scales with tokens seen. You have a dataset, you pass over it, you are done, and you can estimate the bill in advance to within a rounding error.

Reinforcement learning cost scales with rollouts generated, and the dataset does not exist until the model makes it. Before a single gradient step, you need a group of complete trajectories per task, and for an agent each trajectory is a sequence of real tool calls with real latency. The gradient step itself is a small fraction of wall clock, often under a fifth of it. The rest is generation, waiting on tools, and grading.

That reshapes what the cluster has to be. An SFT run wants a block of GPUs saturated with backward passes. A reinforcement learning run wants a fast serving path co-resident with the trainer, weights syncing to it every step, and enough elasticity to absorb the fact that rollout demand spikes and collapses within each iteration. It is the shape of work that punishes a statically partitioned fleet, which is why GPU orchestration shows up as a real constraint on reinforcement learning fine-tuning long before the algorithm does.

Failure modes to expect

Reward hacking. Any grader that can be satisfied without doing the work eventually will be. An agent rewarded for a passing test suite may learn to weaken the tests. Audit high-scoring trajectories by hand throughout, because a rising reward curve is not evidence that the reward means what you intended.

Entropy collapse. As the policy sharpens it stops exploring, groups stop disagreeing, advantages go to zero, and learning quietly stops while the loss looks fine. Track output entropy and the fraction of groups with non-zero reward variance as first-class metrics, not afterthoughts.

A task set that stops teaching. Tasks the model now solves every time and tasks it never solves both contribute nothing. Without refreshing the mix toward the current pass rate, a run plateaus for reasons that look algorithmic and are not.

Narrowing. Heavy optimization against one reward costs general instruction-following. Keep a held-out general evaluation running beside the task metric, and consider parameter-efficient methods, which move the model less far from the base and are cheaper to roll back.

Tool drift. A policy trained against one set of tool schemas partly encodes those schemas. Rename an endpoint and performance degrades in a way that looks like general regression. Version tool definitions alongside checkpoints.

Where this fits at Numerata

P95 is Numerata's training layer, and the reinforcement learning loop is the shape of work it is built for: many runs rather than one, each producing groups of rollouts to compare, tracked so you can tell which round actually moved the pass rate rather than the reward number. It draws GPUs from the same pool Lupine manages, which is what lets a rollout-heavy run borrow serving capacity between gradient steps instead of reserving it, and it hands finished checkpoints to NinetyFive to serve.

The reason all of this runs inside your environment rather than somebody else's is the environment requirement above. Generating trajectories means connecting the real tools to the real systems, so the rollouts are a recording of your internal endpoints, schemas, and the order in which your people do the work. That is a more sensitive artifact than the training corpus, and it is why the compute doing the generating has to sit where those systems already are: private cloud, on-prem, or fully air-gapped.

If you are earlier in the picture, what fine-tuning is covers the single-turn case, and agentic fine-tuning covers the trajectory data both methods here depend on. For the verifier itself, which is the artifact everything here rests on, see evaluating agents. And for the inside-your-own-walls constraint specifically, see air-gapped LLM training.