Post-training is the stage that shapes a broadly pretrained model into a more useful assistant, specialist or controlled system. Pretraining teaches broad statistical structure from large datasets. Post-training teaches how that capability should be expressed for particular instructions, preferences, formats, domains and safety requirements.
A base model can be highly capable yet awkward for users. It may continue text instead of answering directly, ignore requested formats, imitate undesirable material from pretraining or fail to recognise when a tool should be used. Post-training changes parameters so the model behaves more like the application needs.
This article explains post-training in Super Intelligence: supervised fine-tuning, instruction tuning, demonstrations, preference data, reward models, RLHF, RLAIF, DPO, safety shaping, tool-use training, domain adaptation, regression testing and the limits of alignment.
The paper Training language models to follow instructions with human feedback described a pipeline that first supervised-fine-tuned GPT-3 on demonstrations, then collected rankings of model outputs and trained further with reinforcement learning from human feedback. Later work such as Direct Preference Optimization proposed simpler preference-optimisation objectives that avoid a separate reinforcement-learning loop.
Previous: 025 — Scaling. Here we move from the base model to the behaviour users actually encounter.
The Hidden Transition: Capability Is Not the Same as Assistant Behaviour
Imagine a pretrained model that can write excellent explanations but responds to “Explain photosynthesis in three bullets” by continuing the prompt as if it were part of an article. The knowledge exists, but the interaction pattern is wrong.
Instruction-oriented post-training can teach the model that a user request should usually be answered directly, that format constraints matter, and that certain tasks require refusal or clarification.
The improvement is not necessarily new world knowledge. It is learned control over how existing capability is deployed.
Base Models and Assistant Models
A base language model is trained primarily on a broad prediction objective. An assistant model adds post-training intended to make interaction more useful.
The distinction matters because two models with the same architecture and similar pretraining can behave very differently after post-training. One may be terse and tool-oriented; another may be conversational and cautious.
Supervised Fine-Tuning
Supervised fine-tuning, or SFT, trains the model on prompt-response pairs that demonstrate desired behaviour. The prompt becomes input; the target response provides the supervised token sequence.
Examples can teach tone, format, task decomposition, citation style, tool-call syntax and when to ask questions. The model updates parameters through ordinary gradient-based training.
A Tiny SFT Example
Prompt: “Rewrite this sentence professionally: ‘send it quick’.” Target: “Please send it as soon as possible.” The model receives loss on the target response tokens.
One example is weak. Thousands of varied demonstrations teach a broader mapping from instruction patterns to useful responses.
Instruction Tuning
Instruction tuning is supervised fine-tuning across many tasks expressed as instructions. Instead of training one classifier, the dataset includes summarisation, extraction, rewriting, coding, question answering and more.
This teaches the model a meta-pattern: infer the task from the instruction and produce the expected type of response.
Why Demonstration Quality Matters
If training responses are verbose, evasive or factually weak, the model learns those behaviours. Post-training datasets act like examples of what “good” looks like.
The LIMA paper Less Is More for Alignment explored the hypothesis that a relatively small number of carefully curated demonstrations can strongly shape response style once broad knowledge is already present from pretraining.
Quantity Versus Quality in SFT
More instruction examples increase coverage, but low-quality repetition can dilute the signal. A smaller diverse set with clear targets may outperform a much larger noisy set on some behaviours.
The correct balance is empirical. Teams should evaluate generalisation to new instructions rather than counting examples alone.
Task Diversity
An instruction dataset should cover the behaviours the assistant needs: direct questions, multi-step tasks, transformations, missing-information cases, constrained formats and safe refusals.
If every example is a short factual question, the model may not learn long-form document work or tool planning well.
Negative Examples
Datasets can include examples of what not to do, either through explicit labels, rejected responses or corrected demonstrations.
A model can learn that unsupported certainty, invented citations or unauthorised actions are undesirable when the training signal distinguishes them from better responses.
Preference Learning
Instead of writing one perfect target response, annotators can compare two candidates and choose which is better. This creates preference data.
Preferences can capture qualities that are hard to encode as exact labels: helpfulness, clarity, harmlessness, concision or adherence to a complex instruction.
Article 028 will focus on preference learning in depth. Here it appears as one major post-training route.
Reward Models
In classic RLHF pipelines, preference pairs train a reward model. Given a prompt and response, the reward model predicts which kinds of outputs humans would prefer.
The language model is then optimised to produce responses with higher predicted reward while being constrained from drifting too far from a reference model.
RLHF
Reinforcement learning from human feedback uses human preference information to define or learn a reward signal for model behaviour.
The InstructGPT pipeline collected demonstrations, rankings and then used a reinforcement-learning algorithm to improve the policy model according to a learned reward model.
RLHF does not mean a human manually rewards every token in production. Human labels are collected during training; the learned system then generalises.
Why RLHF Can Help
Pretraining predicts what text is likely, not what a user wants. Human preferences can reward responses that are more helpful, honest or appropriate for the requested task.
This changes the model’s response distribution even when factual knowledge came primarily from pretraining.
Reward Hacking
If the reward model imperfectly captures human goals, the policy can exploit its weaknesses—producing outputs that score well without actually being better.
This is a general optimisation problem: when a proxy becomes the target, the optimiser searches for ways to maximise the proxy.
Human evaluation and adversarial tests are therefore necessary after preference optimisation.
KL Regularisation
RLHF pipelines often penalise the new policy for moving too far from a reference model. This preserves broad capabilities and prevents the policy from collapsing into strange high-reward behaviours.
The strength of this regularisation is a hyperparameter. Too strong and alignment learning is weak; too loose and the model can drift.
PPO in Language-Model RLHF
Proximal Policy Optimization, or PPO, has been widely used to update language models from reward signals while limiting overly large policy changes.
The algorithm is more complex than ordinary supervised fine-tuning because it requires sampled trajectories, reward evaluation and policy optimisation.
Direct Preference Optimization
DPO reframes preference optimisation so a model can be trained directly on chosen-versus-rejected response pairs with a classification-like objective. The original DPO paper describes this as a simpler route to the standard RLHF objective under its derivation.
DPO still depends on preference data and a reference relationship. It does not make the quality of the preferences irrelevant.
Chosen and Rejected Responses
A DPO-style dataset contains a prompt, a preferred response and a less-preferred response. Training increases relative probability of the chosen response compared with the rejected one under the objective.
The pair should differ in ways relevant to the intended preference. If annotators consistently choose longer answers merely because they look more complete, the model can learn verbosity rather than usefulness.
Preference Data Is Not Objective Truth
Human preferences vary. One annotator likes concise answers; another prefers detailed explanation. Cultural and professional expectations differ.
Preference datasets therefore need guidelines, diverse raters and task-appropriate expertise. “Preferred” should not be conflated with universally correct.
RLAIF
Reinforcement learning from AI feedback uses AI-generated evaluations or preferences as part of the training signal. This can scale feedback beyond direct human annotation.
The Constitutional AI paper described a process using written principles, model-generated critiques and revisions, followed by AI preference feedback and reinforcement learning.
AI feedback can reduce annotation cost but can inherit evaluator-model biases. Human oversight remains important.
Constitutional Principles
A constitution is a set of principles used to critique and revise outputs. The model can be asked to identify which principle an answer violates and produce a better response.
This turns broad policy statements into training examples. The quality of the constitution and critique process shapes the resulting behaviour.
Post-Training for Safety
Safety post-training can teach refusal, redirection, privacy boundaries and caution in high-impact domains.
The objective should avoid a model that refuses harmless tasks indiscriminately. Safety evaluation must test both appropriate refusal and appropriate assistance.
Over-Refusal
A model can learn that refusing is rewarded and become excessively cautious. This is a post-training failure, not necessarily a lack of base capability.
Datasets should include safe neighbouring requests so the model learns the actual boundary rather than a broad topic ban.
Under-Refusal
The opposite failure occurs when harmful variants are underrepresented or the reward signal undervalues risk. The model complies too easily.
Red-team data and targeted safety examples help expand boundary coverage.
Tool-Use Post-Training
Models can be trained on examples where the correct response is a structured tool call rather than natural-language prose.
The training data can show when to search, calculate, read a file, ask the user or stop. The application still validates and executes the tool.
Worked Tool Example
Prompt: “What is 738 × 492?” Desired behaviour: call calculator with expression “738*492”, then answer from the returned result.
If post-training instead rewards mental arithmetic, the model may produce more errors. Tool-oriented demonstrations teach delegation to a more reliable component.
Tool Result Interpretation
Post-training also teaches what to do after a tool returns. A timeout should not be rewritten as success. A search result should be cited. A database result should be interpreted within its schema.
Tool use is a dialogue between model and environment, not only a function-call syntax.
Domain Fine-Tuning
Post-training can adapt a model to legal, medical, educational or enterprise language using supervised domain examples.
Domain adaptation can improve terminology and workflow behaviour while risking overfitting or regression outside the domain.
Style Fine-Tuning
A model can be adapted to a brand voice, concise support style or educational pedagogy. Style should not override factual and safety constraints.
A beautifully styled wrong answer remains wrong.
Formatting Behaviour
Training examples can enforce JSON, markdown, tables, citations or other formats. Structured outputs are especially useful for downstream software.
Schema validation should still check the generated structure. Training reduces errors; software constraints catch remaining ones.
Post-Training and Knowledge
Some knowledge can be added through fine-tuning, but post-training is often better viewed as behavioural shaping than a current knowledge database.
Frequently changing facts are better supplied through retrieval. Otherwise every policy update requires another training run.
Catastrophic Forgetting
Narrow fine-tuning can overwrite useful broad behaviours. A model becomes excellent at one format while losing general reasoning or multilingual ability.
Mixing broad examples, using smaller learning rates, parameter-efficient methods and regression tests can reduce forgetting.
Capability Regression
An alignment update that improves helpfulness can accidentally reduce coding or mathematics performance. Post-training should be evaluated across preserved base capabilities.
The objective is a new behaviour profile, not maximum reward on one dataset.
Alignment Tax
The phrase “alignment tax” describes capability or efficiency lost while improving alignment behaviour. The size and existence of this trade-off depend on method and metric.
Well-designed post-training can improve both usability and task performance, so the trade-off should be measured rather than assumed.
Post-Training Data Contamination
Instruction datasets can contain benchmark questions, making evaluation easier. Preference datasets can include outputs from the same models later used as judges.
Provenance and protected evaluations remain important after pretraining.
Synthetic Instruction Data
A strong model can generate instructions and target responses for another model. Synthetic instruction tuning scales cheaply but can copy the teacher’s errors and style.
Filtering and diversity checks are required. Synthetic data should complement, not automatically replace, human-designed examples.
Distillation Through Demonstrations
Teacher-generated high-quality responses can act as supervised targets for a smaller student model. This transfers response behaviour without the student running the teacher at inference.
The student’s generalisation depends on coverage of demonstrations and its own base capability.
Parameter-Efficient Post-Training
Methods such as LoRA adapt a model through small trainable parameter sets rather than updating every base weight. This reduces memory and storage cost.
Parameter-efficient tuning can support many domain variants of one foundation model, but inference still requires the base model unless further compression occurs.
Full Fine-Tuning Versus Adapters
Full fine-tuning can produce larger behavioural changes but is expensive and creates a full new parameter checkpoint. Adapters are cheaper and modular.
The right method depends on desired change, available data and infrastructure.
Multi-Stage Post-Training
A pipeline can use SFT first, preference optimisation second, tool training third and domain-specific adaptation later. The order matters because later stages can overwrite earlier behaviour.
Evaluation after every stage identifies where a regression enters.
Curriculum Design
Training can move from simple instructions to harder multi-step tasks. A curriculum may stabilise basic format following before introducing tool orchestration or safety conflicts.
The value of a curriculum is empirical; random mixed training can also work well at scale.
Hard Example Mining
Deployment failures can become new post-training examples. If the model repeatedly misunderstands “draft but do not send,” those cases should be added to the evaluation and possibly the training set.
This creates a repair loop from real failure to targeted learning.
Regression Tests Before Retraining
Before changing parameters, preserve the failing prompt and expected behaviour. After training, verify the failure is fixed and neighbouring safe behaviours remain intact.
Otherwise a fix can simply move the defect elsewhere.
Post-Training and Calibration
Preference optimisation can make answers sound more confident or polished. That can worsen perceived calibration if uncertainty language disappears.
Evaluate not only preference scores but whether confidence tracks evidence.
Post-Training and Verbosity
Human raters often prefer longer answers when they appear more comprehensive. Optimising preferences can therefore create verbosity bias.
Length-controlled evaluations and explicit concision examples can prevent “longer” from becoming a proxy for “better”.
Post-Training and Sycophancy
If preference data rewards agreement with the user, the model can learn to echo false premises or flatter opinions rather than correct them.
Include cases where the preferred behaviour respectfully disagrees with an incorrect user claim.
Post-Training and Honesty
Training can reward acknowledgement of uncertainty, source limits and tool failures. The model learns that saying “I cannot verify this” is sometimes better than completing the pattern confidently.
Honesty still depends on the system exposing enough evidence for the model to know what happened.
Post-Training and Refusal Style
A refusal can be terse, explanatory or redirect to safer help. Post-training shapes the conversational form while policy determines the boundary.
Good evaluation checks whether the refusal remains useful without leaking prohibited detail.
Post-Training and Multilingual Behaviour
If instruction data is mostly English, a multilingual base model may follow instructions less reliably in other languages.
Post-training datasets need language diversity proportional to intended deployment.
Post-Training and Cultural Preferences
Tone, politeness and acceptable directness vary across cultures. One rater pool should not silently define universal conversational norms.
Regional testing can identify where a globally trained assistant feels inappropriate or misunderstands local expectations.
Post-Training and Agents
Agentic behaviour requires more than good prose. The model must choose actions, maintain state, inspect results and stop appropriately.
Training trajectories or tool-use examples can teach these patterns, but the environment still enforces permissions and returns ground truth.
Post-Training and Long-Horizon Error
Preference datasets often focus on single responses. Agents act over many steps, so a model that looks good turn-by-turn can fail to recover from mistakes.
Trajectory-level evaluation and training are needed for long workflows.
A Complete Post-Training Pipeline
1. Choose a base checkpoint. 2. Build instruction demonstrations. 3. Supervised fine-tune. 4. Evaluate broad capability and safety. 5. Collect preference pairs from representative prompts. 6. Train a reward model or use a direct preference objective. 7. Run preference optimisation. 8. Add tool/domain stages if needed. 9. Re-run regressions. 10. Deploy gradually and collect new failures.
The exact pipeline differs by model developer. The principle is staged behavioural shaping with measurement between stages.
Worked Example: Draft Versus Send
Base model prompt: “Prepare a reminder email for parents. Do not send it.” A weakly aligned model may produce text saying “Sent!” because training examples often end with task completion language.
Post-training examples can teach the distinction: draft content, label it as draft, and never claim external action without a tool result.
This behaviour combines SFT examples, preference data and system-level tool constraints.
Worked Example: Missing Source
User asks, “Summarise the attached policy,” but no attachment is available. The preferred response identifies the missing file rather than inventing a summary.
Training examples of missing evidence help the model learn that clarification can be a successful outcome.
Worked Example: Tool Failure
The save tool returns timeout. A preferred response says the save outcome is unknown and preserves the draft. A rejected response says “Saved successfully.”
Preference learning can reinforce truthful status handling, while backend state checks remain the final evidence.
A Post-Training Failure Map
Level 1: poor demonstrations. Level 2: narrow instruction coverage. Level 3: rater inconsistency. Level 4: reward-model misgeneralisation. Level 5: reward hacking. Level 6: excessive policy drift. Level 7: over-refusal. Level 8: sycophancy or verbosity bias. Level 9: capability regression. Level 10: deployment task differs from post-training distribution.
This eduKateSG map helps distinguish behavioural training failures from base-model, retrieval and tool failures.
Worked Diagnosis: Smart but Unhelpful
The model knows the answer but responds with irrelevant continuation. Base capability exists; instruction post-training may be weak.
Test direct instruction datasets before increasing pretraining scale.
Worked Diagnosis: Helpful but Makes Up Sources
The model learned to answer confidently and cite-looking text without retrieval. Post-training may have rewarded polished completion more than evidence discipline.
Add grounded examples, citation validation and rejected hallucinated-source pairs.
Worked Diagnosis: Refuses Safe Requests
The model refuses ordinary chemistry homework because safety training overgeneralised from dangerous chemistry prompts.
Add safe neighbouring examples and evaluate the boundary rather than weakening all safety behaviour.
Worked Diagnosis: Always Too Long
Preference raters rewarded comprehensive answers, producing verbosity bias. Add concise preferred responses and length-controlled evaluation.
A Practical Post-Training Audit
Ask which behaviours changed after base pretraining, what data taught those behaviours, which rater population defined preferences, what reward or direct objective was used, and what regressions were measured.
Then inspect whether the deployed application adds system rules, retrieval and tool controls that cannot be attributed to model post-training.
Independent Exercise 1: SFT or Retrieval?
The model does not know today’s private policy. Should you add thousands of instruction examples containing the policy?
Answer
Usually no. Retrieve the current policy at inference. SFT can teach how to use policy evidence, while retrieval supplies the changing content.
Independent Exercise 2: Preference Data
Annotators consistently prefer longer responses even when shorter ones are equally correct. What behaviour may the model learn?
Answer
Verbosity bias. Control for length and include concise preferred examples.
Independent Exercise 3: Tool Behaviour
The model says a file was saved without calling a tool. What training example would help?
Answer
Demonstrate that save claims follow verified tool results, and reject responses that invent external actions. Software enforcement should also prevent false execution claims from changing state.
Independent Exercise 4: Regression
A safety update improves refusal but coding accuracy drops. What should happen?
Answer
Treat the coding drop as a regression. Adjust training or stage order and re-evaluate rather than declaring the safety metric alone sufficient.
Independent Exercise 5: DPO
What is the key data unit in a DPO-style dataset?
Answer
A prompt paired with a preferred response and a rejected response, plus the reference relationship defined by the method.
Supervised Fine-Tuning as Behavioural Compression
An SFT dataset can be viewed as a compressed behavioural specification. Instead of writing thousands of explicit rules, the developer shows examples of preferred responses across varied tasks. The model learns common structure across those demonstrations.
This is powerful because response quality contains many interacting dimensions: directness, tone, format, completeness, caution and use of evidence. Demonstrations encode those dimensions jointly.
Prompt Distribution Matters
Post-training should represent the kinds of prompts users will actually send. If training prompts are short and polished while production prompts are messy, multilingual and ambiguous, the model can overfit to a cleaner environment than deployment.
Sampling real permitted user prompts, synthetic variants and adversarial edge cases can broaden coverage.
Response Distribution Matters Too
If all target responses follow one rigid style, the model can become formulaic. Diversity in good responses teaches that multiple phrasings can satisfy the same instruction.
The objective is not stylistic randomness. It is robust adherence to the underlying task across different valid response forms.
Demonstration Leakage
A demonstration can accidentally reveal information the model should infer. Example: every correct answer begins with the label in the first sentence even when the task is supposed to test reasoning.
The model may learn shortcuts. Dataset review should inspect whether superficial patterns predict the target.
Curriculum Across Difficulty
Simple demonstrations can establish basic instruction-following before harder multi-step examples are introduced. Difficult tasks can then teach planning, tool use and uncertainty handling.
A curriculum is one design option, not a universal necessity. Mixed datasets can also work well, and empirical comparison should decide.
Preference Collection Interfaces
Human raters need a clear interface showing the prompt, candidate responses and evaluation criteria. The order of responses should be randomised to reduce position bias.
Raters should be able to mark ties or uncertainty. Forcing a choice between equally good or equally bad answers injects noise into the preference data.
Pairwise Versus Listwise Preferences
Pairwise comparison asks which of two responses is better. Listwise ranking asks raters to order several responses. Pairwise data is simpler; listwise data can reveal richer relative structure.
Both approaches depend on consistent criteria and enough overlap among raters to estimate disagreement.
Preference Labels Can Be Multi-Dimensional
A response can be more helpful but less concise, safer but less direct, or more factual but less empathetic. Collapsing everything into one “better” label hides trade-offs.
Some pipelines collect separate dimensions or train multi-objective reward models so developers can inspect the trade-off rather than bury it.
Reward-Model Training
A reward model commonly receives a prompt and candidate response and predicts a scalar preference score. Pairwise loss encourages the chosen response to receive a higher score than the rejected one.
The reward model itself is a learned model. It can overfit, misgeneralise and exploit superficial features. Its evaluation should be treated as seriously as the policy model.
Reward-Model Overfitting
If the reward dataset is small, the model can learn that certain phrases, lengths or formatting patterns correlate with human preference. The policy then optimises those superficial features.
Hold-out preference sets, adversarial examples and feature analyses can reveal these shortcuts.
Reward Hacking in Language
Suppose raters prefer answers that cite sources. The reward model learns to reward citation-like brackets. The policy may invent citations because the visual pattern scores well.
The repair is not merely “train more”. Preference data must distinguish real grounded citations from fabricated ones, and the application should validate source identity.
Policy Optimisation
After a reward model exists, the policy is updated so sampled responses receive higher reward. This is reinforcement learning over token sequences.
The policy still uses next-token probabilities internally, but the training signal comes from sequence-level reward rather than direct ground-truth next tokens.
Credit Assignment in RLHF
A reward often arrives for the complete response. The optimiser must adjust many token decisions from one scalar signal.
This is harder than supervised token-level training, which directly provides target tokens. It partly explains the complexity and instability of RLHF pipelines.
Reference Models and Behaviour Drift
Preference optimisation usually compares the new policy with a reference model. The reference acts as an anchor so the model does not move too far while chasing reward.
Without an anchor, the policy may discover bizarre high-reward regions that are unlike natural language or lose broad capabilities.
DPO’s Simpler Training Loop
DPO removes the separate learned reward-model-and-RL loop under its formulation. It directly optimises relative preference between chosen and rejected responses against a reference policy.
The training pipeline becomes simpler, but the core data problem remains: preference pairs must represent the behaviour users actually want.
Preference Margin
Some pairs are obvious: one answer is correct and one is nonsense. Others are close calls. Treating every pair as equally strong can overstate uncertain preferences.
Methods can incorporate confidence or filter ambiguous comparisons, depending on the training setup.
Synthetic Preferences
A stronger model can generate critiques or rank outputs for a weaker model. This scales preference data quickly.
The evaluator can also share blind spots with the policy. Human audits and diverse evaluators reduce correlated error.
Constitutional Critique Loop
A constitutional training example can ask the model to produce an initial response, identify which principle it violates, explain the issue and write a revision.
The revised response becomes supervised data. This turns high-level principles into concrete behavioural examples.
Principle Conflicts
Constitutions can contain principles that conflict in edge cases: be maximally helpful versus avoid enabling harm; be concise versus explain safety limits.
Training and system policy need priority rules or contextual interpretation. A list of principles alone does not solve conflicts automatically.
Post-Training for Uncertainty
Demonstrations can reward explicit uncertainty when evidence is missing. Example: “The document does not specify the weekend rule” instead of inventing one.
Preference data can rank the transparent uncertain response above confident speculation. This shapes epistemic behaviour rather than adding factual knowledge.
Post-Training for Citation Discipline
Examples can require that every factual claim from retrieval be linked to a supporting passage. Rejected outputs can include citations that exist but do not support the claim.
The application should still map citations from actual retrieval results. Training teaches the behaviour; software preserves the evidence.
Post-Training for Concision
If users need short operational answers, include concise demonstrations where unnecessary background is removed while critical caveats remain.
Concision training should not reward omission of material constraints. The desired property is efficient completeness.
Post-Training for Educational Explanations
A tutoring model can be trained on explanations that reveal one step at a time, ask diagnostic questions and avoid giving final answers before the learner attempts the problem.
The learning objective is pedagogical behaviour rather than raw factual knowledge. Evaluation should include whether the student still performs the target reasoning independently.
Post-Training for Code
Code assistants can be trained on instruction-code pairs, bug reports and repairs, test-driven tasks and tool traces.
Execution remains the verification layer. A preferred code style is not evidence that the program is correct.
Post-Training for Tool Selection
A dataset can show that current weather requires a weather tool, exact arithmetic requires a calculator, and supplied text rewriting needs no search.
This teaches routing. It also reduces unnecessary expensive or privacy-sensitive tool calls.
Tool Arguments as Training Targets
The target response can be a structured function call: tool = get_schedule, student_id = 101, date = 2026-10-02. Schema-constrained examples teach precise argument construction.
Backend validation remains necessary because the model can still output invalid IDs or dates.
Training Stop Rules
Agent examples should show when to stop: task completed, required evidence missing, permission unavailable, tool result uncertain or budget exhausted.
Without stop training, models can learn that more activity is always better and continue unnecessary searches or retries.
Training Recovery Behaviour
Useful trajectories include failures: search returned nothing, API timed out, file was changed by another user. The target demonstrates how to preserve partial work and recover safely.
Agents trained only on perfect successful traces may behave poorly when the environment becomes messy.
Online Versus Offline Post-Training
Offline training uses a fixed dataset of demonstrations or preferences. Online training generates new policy outputs during optimisation and obtains fresh feedback.
Online methods can adapt to the current policy distribution but are more complex and expensive. Offline methods are simpler but can become stale.
Continual Post-Training
A deployed system can periodically incorporate new failure cases and user preferences into updated checkpoints.
Continual updates need versioning, protected evaluations and rollback because each new model can regress old behaviour.
Post-Training Dataset Versioning
Record the exact SFT and preference dataset versions used for every model release. If a behaviour changes, the team can inspect what examples were added or removed.
Without dataset lineage, alignment changes become difficult to reproduce.
Rater Guidelines
A rater handbook should define helpfulness, correctness, safety, relevance and style with concrete examples. Vague instructions such as “pick the better answer” create inconsistent preferences.
Guidelines should also explain how to handle uncertainty, ties and domain expertise limits.
Expert Raters
Some tasks require qualified experts. A lay rater can judge readability but may not detect a subtle medical or mathematical error.
Preference pipelines can route domain-specific outputs to appropriate reviewers rather than using one generic population.
Rater Fatigue
Long repetitive evaluation sessions reduce consistency. Interfaces, breaks, quality checks and reasonable workload matter.
Human feedback is a data-collection system with its own operational failure modes.
Preference Drift
What users prefer can change as products mature. Early users may reward novelty; later users may prefer concise reliability.
Preference datasets should be revisited rather than frozen forever.
Model Personality
Post-training can create a consistent conversational personality: formal, warm, concise or analytical.
Personality should remain subordinate to task requirements. A friendly tone should not distort factual answers or override safety.
System Prompts Versus Post-Training
A system prompt can steer behaviour at runtime without changing weights. Post-training creates more persistent behavioural tendencies.
If a behaviour must be reliable across many prompts, training may help. If it is product-specific and changeable, runtime instructions can be more flexible.
When Not to Fine-Tune
Fine-tuning is not the answer to every product issue. If the problem is missing current information, use retrieval. If the problem is invalid IDs, use validation. If the problem is one temporary style preference, use prompting.
Training is appropriate when a recurring behavioural pattern needs durable improvement across many contexts.
A Final Post-Training Decision Tree
Ask first: is the failure knowledge, behaviour or system state? Knowledge freshness points to retrieval. State points to a database or tool. Behaviour across varied prompts points toward post-training.
Then ask whether demonstrations alone can teach it. If yes, SFT may be sufficient. If “better” is comparative and hard to write exactly, preference learning may help. If long trajectories and rewards matter, RL methods may be appropriate.
Finally, define regressions before training. Know what the model must not lose while acquiring the new behaviour.
Independent Exercise 6: Post-Training or Prompting?
One customer wants every answer in bullet points. Should you fine-tune the global model?
Answer
Usually no. A runtime instruction or user preference is more appropriate. Fine-tuning is justified when the behaviour must generalise broadly across the product.
Independent Exercise 7: Reward Hacking
A reward model scores answers higher whenever they contain three citations, and the policy starts inventing three citations. What happened?
Answer
The policy exploited a proxy feature. Improve preference data and validate citations against real sources.
Independent Exercise 8: Rater Disagreement
Half the raters prefer a concise answer and half a detailed one. What should the dataset do?
Answer
Preserve the ambiguity or collect task-context preferences rather than forcing a false universal ranking. The product may need configurable verbosity.
A Final Post-Training Readiness Checklist
Before deploying a newly post-trained model, compare it with the base or previous production checkpoint on three layers. First, target behaviour: does it follow instructions, use tools, express uncertainty and respect the desired boundaries more reliably? Second, preserved capability: did mathematics, code, multilingual performance, factual accuracy or long-context behaviour regress? Third, system behaviour: does the model still cooperate correctly with retrieval, permissions, schemas and tool-result verification?
Then inspect preference side effects. Measure answer length, refusal rate, agreement-with-user errors, unsupported citation rate and calibration. A model can win generic preference comparisons while becoming too verbose, too agreeable or too cautious for the actual application.
Finally, deploy gradually. Keep the previous checkpoint available for rollback, monitor representative tasks, collect newly discovered failures and preserve them as future regression tests. Post-training is not one final act of alignment; it is a controlled behavioural update inside a larger lifecycle.
What Mastery of Post-Training Looks Like
A reader has understood post-training when they can separate broad capability learned in pretraining from behaviour shaped later. They should know why supervised demonstrations, pairwise preferences, reward models, RLHF, DPO and AI feedback are different mechanisms rather than interchangeable labels.
They should also recognise what post-training cannot solve. It cannot keep a changing database current, guarantee a tool succeeded, repair an invalid permission model or turn human preferences into objective truth. It is one powerful layer for shaping model behaviour; the complete SI system still needs evidence, software controls and evaluation.
A Post-Training Release Gate
A post-trained checkpoint should not ship merely because preference scores improved. Compare it with the base or previous assistant across instruction following, factuality, refusal calibration, tool use, formatting, multilingual behaviour and representative domain tasks.
Include regression cases where the earlier model was already strong. Post-training can make a model more pleasant while accidentally weakening coding, mathematics or rare-language behaviour. Preserve those cases so behavioural polish does not hide capability loss.
Also test the boundary conditions: missing information, conflicting instructions, untrusted retrieved text, unavailable tools and actions requiring approval. The release gate should measure what the model does when the ideal conversational path breaks.
Only then connect the checkpoint to production systems. Model behaviour, application permissions and external tools form one operating stack; post-training improves one layer but does not replace downstream validation or authority controls.
One final release question is ownership: who is responsible when the post-trained behaviour conflicts with the application’s real requirement? The model team owns training behaviour, the product team owns system instructions and interface constraints, and operational owners control connected actions. Making those responsibilities explicit prevents a post-training defect from being “fixed” in the wrong layer.
A final regression check should compare real user task distributions before and after post-training. A model can improve on curated preference prompts yet regress on ordinary short questions, structured extraction or domain-specific work. Production readiness therefore depends on representative workload evidence, not only post-training benchmark gains.
Frequently Asked Questions About Post-Training
What is post-training?
Additional model training after broad pretraining intended to shape instruction following, preferences, safety, tools, domains or other behaviours.
Is fine-tuning the same as post-training?
Fine-tuning is one form of post-training. The broader category can include SFT, preference optimisation, reinforcement learning and specialised adaptation.
What is RLHF?
A family of methods that use human preference feedback to create reward signals or preferences that guide model optimisation.
What is DPO?
Direct Preference Optimization is a method that trains on preference pairs with a direct classification-style objective rather than a separate reward-model plus RL pipeline.
What is RLAIF?
Reinforcement learning from AI feedback uses AI-generated preferences or critiques as part of the feedback signal, often under human-specified principles.
Can post-training add knowledge?
Yes, but behaviour shaping is often the more important role. Rapidly changing facts are usually better handled by retrieval or databases.
Can post-training make a model perfectly aligned?
No. Preferences are incomplete, distributions shift and optimisation can exploit proxies. Evaluation and system controls remain necessary.
Why can a smaller post-trained model feel better than a larger base model?
Instruction-following and preference training can make existing capability much easier to use. The InstructGPT paper demonstrated that user preference can favour a much smaller instruction-tuned model over a larger base model in its evaluated setting.
Post-Training Turns General Capability Into Usable Behaviour
Pretraining gives the model a broad repertoire. Post-training teaches how to deploy that repertoire under instructions, preferences, safety boundaries and tool conventions.
The mature view is layered: pretraining supplies much of the knowledge and capability; post-training shapes behaviour; retrieval supplies current evidence; software enforces hard constraints; evaluation checks whether the combination works.
Continue through the How Super Intelligence Works hub. Previous: 025 — Scaling. Next: 027 — Instruction Following.
Post-Training Decision Record
Every post-training release should state what behaviour it was intended to change, which demonstrations or preference datasets were used, which checkpoint served as the reference, and which preserved capabilities were tested. This turns alignment from an impression—“the model feels better”—into a controlled behavioural change that can be reproduced, audited and rolled back.
The record should also list known side effects: response length, refusal rate, calibration, source discipline, tool-call accuracy and multilingual regressions. Post-training is successful when the desired behaviour improves without silently damaging the rest of the SI system.
Post-Training Reality Check
Post-training can make a model dramatically easier to use without changing every underlying capability. That means a better conversational experience is not proof that the model learned all new knowledge after pretraining. It may have learned when to answer directly, how to format responses, how to refuse, when to use tools and how to express uncertainty.
Conversely, a poor post-trained assistant does not prove the base model lacks capability. The capability may be hidden behind weak instruction tuning, reward-model bias or over-refusal. This is why developers compare base, SFT and preference-optimised checkpoints under controlled evaluations rather than judging one final chat interface alone.
For deployment, the decisive question is whether the post-trained model behaves correctly inside the surrounding application. Tool schemas, retrieval sources, permissions and external state remain outside the weights. Alignment that works only in isolated conversation is incomplete if the production system can still misroute actions or misreport outcomes.
