An action can look useful now and lead to a poor outcome later. Another can incur an immediate cost while opening a better route. Reinforcement learning provides a mathematical framework for learning decisions when actions, information and future consequences are connected.
The gold standard is an agent whose improvement can be explained and evaluated against its intended task. The reward, the learning update and the evidence of success must remain visible. This guide builds those connections from a small delayed-reward example to value learning, planning, policy optimisation and AI post-training.
This subject root uses original constructed teaching situations and links to the existing eduKate ecosystem. It introduces computational models; applications involving human learners require evidence about actual learning and appropriate oversight beyond a numerical reward.
EDUKATESG · GOLD STANDARD · Q–R ROOT
Find the part you need
- 1. Define the decision problem — Chapters 1–4: identify interaction, states, observations and return.
- 2. Connect decisions and information — Chapters 5–8: compare delayed rewards, values, bandits and exploration.
- 3. Read learning updates — Chapters 9–12: follow temporal differences, Q-learning, task endings and policy targets.
- 4. Develop and evaluate the method — Chapters 13–16: connect planning, function approximation, policy optimisation and reward design.
- 5. Test transfer and continue — Chapters 17–20: inspect evaluation, post-training and independent reasoning.
Full chapter index · Independent checks · Specialist reading directory
Chapter index
Chapters 1–4
Chapters 5–8
Chapters 9–12
Chapters 13–16
Chapters 17–20
Reinforcement learning studies how an agent can improve its decisions using experience and a reward signal. At each step it receives information, chooses an action and observes a consequence. Actions can change what happens next and what experience becomes available for learning.
This differs from simply predicting a supplied label for each independent example. In a sequential task, an apparently poor immediate result can enable a better later outcome. The learner must connect present decisions to later consequences.
The gold standard is an explanation that names the task, specifies the information and objective, shows how an update changes behaviour, and evaluates that behaviour on conditions beyond the training run. This root connects the eduKate branches on probability, planning, adaptive decisions and AI behaviour.
A Markov decision process describes states, available actions, transition and reward distributions, an initial-state distribution and a return objective. ‘Markov’ means the stated current state contains the information needed for the next-step distribution given the action.
For a small navigation model, state could include position and remaining battery; actions could be move left, move right or stop. If battery affects whether movement succeeds, position alone may omit information needed for the transition model.
Write the transition rule before discussing an algorithm. Are actions deterministic or noisy? Can a move leave the grid? What ends the task? These choices define the learning problem. An algorithm cannot repair an environment whose rules describe the wrong task.
The environment’s underlying state and the agent’s observation need not coincide. A camera view may hide an obstacle; a sensor reading may be noisy. Giving an observation the name ‘state’ does not make it sufficient for prediction.
An agent may use a history of observations and actions, or maintain a belief distribution over hidden states. Two identical current pictures can require different decisions when their histories imply different hidden conditions.
The hidden-state teaching-model branch provides a conceptual application. A learner’s knowledge is inferred from limited evidence rather than observed completely. That application requires educational judgement and appropriate evidence; a convenient numerical state representation does not fully describe a person.
A reward is a numerical signal received after an interaction. Return aggregates rewards according to the specified objective. One common discounted definition is G_t = r_(t+1) + γr_(t+2) + γ²r_(t+3) + …, with 0 ≤ γ < 1 for the usual bounded-reward infinite-horizon setting.
An episodic task can instead use a finite undiscounted sum under suitable conditions. Average-reward formulations ask another question. Specify which objective is used before comparing policies.
Discounting changes the relative contribution of later rewards. It can express a modelling preference or support a mathematical formulation; it is not automatically a moral measure of how much future outcomes matter. Units, time steps and horizon should remain consistent across comparisons.
Consider an invented deterministic task. From the start, action A gives reward 3 and ends the episode. Action B gives reward 0, reaches a second state, and then a forced final action gives reward 5.
With γ = 0.8, action A has return 3 while action B has return 0 + 0.8 × 5 = 4. The delayed route is preferred under this objective even though its first reward is smaller.
Change γ to 0.4 and B’s return becomes 2, so A is preferred. The environment’s rewards did not change; the objective did. This small example shows why ‘choose the biggest reward’ is incomplete without specifying immediate reward, future return and timing.
| Route | Reward sequence | Return at γ = 0.8 | Return at γ = 0.4 |
|---|---|---|---|
| A | 3, then terminal | 3 | 3 |
| B | 0, then 5 and terminal | 4 | 2 |
A policy specifies how actions are selected from the information available. It may choose one action deterministically or assign probabilities. A state value Vπ(s) is expected return from s when following policy π; an action value Qπ(s,a) conditions on taking action a first and then following π.
Bellman reasoning decomposes a return into the next reward plus the discounted continuation value. In the standard Markov setting, Vπ(s) = Eπ[r_(t+1) + γVπ(s_(t+1)) | s_t = s]. The expectation averages both policy choices and environmental uncertainty.
Evaluation asks how good a specified policy is. Improvement asks how to change that policy. Optimality introduces a comparison over policies or actions; it should not be silently substituted into a formula that evaluates a fixed policy.
In a basic multi-armed bandit, each choice yields a reward from an unknown distribution without the full action-dependent state-transition structure of a sequential decision problem. The agent must learn which choices perform well while still collecting useful information.
An observed reward is not the true expected reward. If one option has produced a single high outcome, that may be luck. A sample mean can improve with observations, but repeatedly choosing only the current favourite can leave other options poorly estimated.
A contextual bandit adds observed context to the decision. It still needs a justified boundary around delayed consequences. The contextual-bandit tutoring branch explains why a next-hint problem depends on learner context and why immediate response quality may differ from lasting learning.
An ε-greedy rule can choose a random available action with probability ε and otherwise choose the action with the largest current estimate. State how ties and the random branch are handled, because these details change the action probabilities.
Suppose there are four actions, a unique greedy action and ε = 0.20. If the random branch samples uniformly from all four, the greedy action is chosen with probability 0.80 + 0.20/4 = 0.85. Each other action has probability 0.05.
Sampling only non-greedy actions would define another rule. Exploration also has a cost: in an application involving people or equipment, unrestricted experimentation is not justified by an algorithm’s need for information. The safe-exploration branch examines the constraints of an educational setting.
Temporal-difference learning updates an estimate using an observed reward and an estimated continuation. For a non-terminal transition under a policy being evaluated, a basic error is δ = r + γV(s′) − V(s). An update is V(s) ← V(s) + αδ.
Let V(s) = 2, observed reward r = 1, V(s′) = 4, γ = 0.9 and α = 0.25. The target is 1 + 0.9 × 4 = 4.6. The error is 2.6, so the updated value is 2 + 0.25 × 2.6 = 2.65.
The estimate moves partway towards a target; one observed transition does not reveal the exact expected value. Bootstrapping means the target uses another estimate. The target can therefore move as learning proceeds.
Tabular Q-learning uses a greedy continuation target: Q(s,a) ← Q(s,a) + α[r + γ max_b Q(s′,b) − Q(s,a)] for a non-terminal transition. This describes one update, not a guarantee that any chosen learning setup will converge.
Take Q(s,a) = 1, r = 2, γ = 0.9, maximum next-state action value 4 and α = 0.5. The target is 5.6 and the new estimate is 1 + 0.5 × (5.6 − 1) = 3.3.
The next action value is maximised in the target even when the behaviour used for collecting experience sometimes explores. Keep the observed transition, the current estimate and the target distinct. Errors in those roles can survive a correct calculator calculation.
A terminal state belongs to the task definition: once reached, there is no further task return to bootstrap. In chapter 10’s update, if the transition truly terminates with reward 2, the target is 2 rather than 5.6. The new value is 1 + 0.5 × (2 − 1) = 1.5.
An external data-collection cutoff is different. If a continuing task is halted merely because the recorder reached a limit, the underlying state can still have future value. Treating that cutoff as terminal can bias the learning target.
A time limit built into a finite-horizon task is part of the task, and remaining time may need to be represented to preserve the Markov description. Gymnasium’s termination and truncation guide explains why these cases require different handling.
On-policy learning evaluates or improves the policy associated with the behaviour producing its data. Off-policy learning uses data from a behaviour policy to learn about a target policy that may differ. The distinction concerns the relationship between data and target, rather than whether learning happens online.
Sarsa uses the value of the next action actually selected in its basic target. With chapter 10’s old value, reward, discount and learning rate, suppose the selected next action has value 1 while another action has value 4. The Sarsa target is 2 + 0.9 × 1 = 2.9 and the update gives 1.95.
That differs from Q-learning’s 3.3 because the continuation target differs. Off-policy estimation also needs adequate coverage and an appropriate method for the mismatch. A historical dataset does not automatically support every policy an analyst wants to evaluate.
A model predicts transitions and rewards. Planning uses such predictions to compare possible futures. A model can be supplied by task rules or learned from experience; model-based reinforcement learning connects model learning, planning and policy improvement.
In the deterministic two-route example, a supplied transition model lets us compute both returns directly. We do not need repeated interaction merely to discover already-known rewards. Unknown or stochastic transitions introduce a different estimation problem.
A learned model can be wrong in regions the policy exploits. High predicted return inside that model may then fail in the actual environment. The dynamic-programming article, search branch and Monte Carlo tree-search guide connect recurrence, look-ahead and simulation.
A table stores separate values for a finite set of state–action entries. Function approximation uses shared parameters to estimate values or policies across many inputs, including large or continuous spaces.
Sharing can generalise from one observation to others. It can also change estimates for situations not present in the latest update. A neural network’s output is not an exact table entry merely because it is displayed as a Q-value.
Combining function approximation, bootstrapping and off-policy data can create instability. Experience replay and target networks are design techniques used in some methods, not universal proofs of convergence. Evaluate learning behaviour and sensitivity rather than assuming a larger network necessarily produces better decisions.
Policy-optimisation methods represent a policy through parameters and adjust them to improve an expected-return objective. Policy-gradient estimators connect observed trajectories to a direction for changing those parameters.
An actor represents action selection; a critic estimates quantities used to evaluate or guide it. An advantage compares an action’s expected return with a suitable state baseline. It asks whether the action is better or worse than the policy’s expected continuation from that state.
This comparison helps avoid interpreting a large raw reward as evidence that every preceding action deserves equal credit. Gradient estimates can be noisy, critics can be inaccurate and update size matters. Algorithm names are useful only when their objective, estimator and evaluation conditions remain visible.
Optimising a measurable reward can expose gaps between the score and the intended outcome. In an invented task, awarding +1 for every step without a suitable completion objective can make prolonged activity attractive even when a person wanted quick completion.
With discount γ = 0.9 and an indefinitely available reward of +1 per step, the return of continuing is 1/(1 − 0.9) = 10. A terminal action paying 4 may be rejected under that reward design. The arithmetic can be correct while the task specification misses the intended result.
Reward hacking concerns exploiting the scoring mechanism; goal misgeneralisation concerns behaviour pursuing an unintended objective in new settings despite training success. The reward-hacking and goal-misgeneralisation branch develops the difference. Inspect actual task outcomes alongside the training reward.
Report how evaluation episodes are generated, which initial conditions are used, whether exploration remains active and how many independent training runs are compared. A single best seed gives a selective picture of a variable learning process.
Keep evaluation conditions outside the tuning loop where the intended claim requires unseen performance. Compare with simple baselines under comparable interaction and computation budgets. Record reward, completion, failure modes and task-relevant costs separately.
An average return can hide different distributions of outcomes. A policy that succeeds very well most of the time but fails badly in some conditions may differ materially from a consistent policy with the same mean. Show enough distributional evidence for the intended use.
Reproducible seeds help inspect a run, while multiple independently managed runs help assess variability. The Randomness root supplies the probability and simulation distinctions behind those checks.
Language-model post-training can use demonstrations, preference judgements or outcome feedback to shape generated behaviour. Some approaches use reinforcement learning; post-training as a whole is a broader category.
In the published InstructGPT study, researchers first used supervised demonstrations and then human rankings in a reinforcement-learning-from-human-feedback procedure. That is one documented training design, not a statement that every current model follows the same pipeline.
A learned preference score is a model of the available feedback. It does not make every answer true, and improved judged helpfulness does not remove the need to verify factual or mathematical content. Keep training feedback and independent task evaluation distinct.
The reinforcement-learning and post-training branch and coding-agent verification article connect behaviour shaping with evidence about completed work.
Try four checks. A: rewards are 2 now and 6 one step later, with γ = 0.5; find the discounted return. B: in four-action ε-greedy with ε = 0.10 and uniform exploration over all actions, find the unique greedy action’s probability. C: old Q is 3, terminal reward is 1 and α = 0.25; calculate the update. D: explain why a high simulated return may fail in deployment.
Answers: A gives 2 + 0.5 × 6 = 5. B gives 0.90 + 0.10/4 = 0.925. C uses terminal target 1, giving 3 + 0.25 × (1 − 3) = 2.5. D may involve model error, changed observations, unseen conditions or a reward that does not capture the intended purpose.
Now turn C’s ending into an external cutoff in a continuing task. A continuation estimate may be required, so the previous terminal target cannot simply be reused. Ask what changed in the task description before choosing the equation.
This follows The Gold Standard of Education and the immutable Secondary 1 Mathematics foundation reference: define quantities, show each update, repair the exact misconception and test a changed situation independently.
Is reinforcement learning the same as giving immediate rewards? No. The objective concerns a specified return, and delayed consequences can change the preferred action.
Does an agent always need to explore randomly? Exploration has many forms. Its purpose is useful information gathering under the task’s constraints, not randomness for its own sake.
Does Q-learning always converge? Tabular convergence results require particular conditions. They should not be transferred automatically to arbitrary function approximation, data coverage or parameter choices.
Are a state and an observation identical? Only when the available observation contains the information the state model needs. Partial observation can require history or belief-state reasoning.
Does a rising reward curve prove that the intended task is solved? Check the reward definition, evaluation conditions, outcome distribution and failures. Training-score improvement alone cannot establish all of those.
Can an educational reward stand in for learning? A score or immediate response measures a particular signal. Independent understanding, retention and transfer need their own evidence.
Where should I go next? Use Markov chains and conditional probability for foundations, dynamic programming and search for planning, contextual bandits and hidden-state models for adaptive decisions, and post-training and verification branches for AI applications.
Continue through the reinforcement-learning ecosystem
Choose a branch by the task it explains. The mathematical foundations, adaptive-learning models and AI applications share concepts while retaining their own assumptions and evidence requirements.
Probability, states and dynamic decisions
- How Markov Chains Work | From States and Transition Probabilities to Stationary Distributions, Mixing, Hitting Times and Stochastic Systems
- The Core Aim of Mathematics Mastery | Markov Chains
- How Conditional Probability Works | From New Information and Restricted Sample Spaces to Dependence, Bayes, Diagnosis, Prediction and Better Decisions
- The Gold Standard of Randomness
- How Dynamic Programming Works in Computer Science | States, Recurrences, Memoization, Tabulation and Optimal Decisions
- How Monte Carlo Simulation Works | From Random Sampling and Expected Values to Numerical Integration, Uncertainty Propagation, Rare Events and Better Computational Decisions
Vocabulary and adaptive-learning models
- The Core Aim of Vocabulary Mastery | Reinforcement Learning Vocabulary
- How Contextual Bandit Tutoring Works | Personalise the Next Hint Without Pretending One Action Is Best for Every Learner
- How Safe Exploration in Adaptive Learning Works | Improve a Teaching Policy Without Turning Learners Into Unbounded Experiments
- How POMDP Teaching Works | Choose the Next Teaching Move When the Learner State Is Hidden
Planning and search
- How Super Intelligence Works | Search — How SI Explores Candidate Solutions, Branches, Trees and Paths
- Why Mathematics? | Monte Carlo Tree Search, UCB Scores, Simulations and Better Moves
AI behaviour, reward design and verification
- Super Intelligence | Reinforcement Learning and Post-Training | How AI Behaviour Is Shaped
- Super Intelligence | Reward Hacking and Goal Misgeneralisation | When the Score Replaces the Purpose
- Super Intelligence | Software Engineering | Coding Agents, Repositories, Verification and Long-Horizon Work
- How Super Intelligence Works | The Complete SI Guide to Models, Reasoning, Tools, Agents and Human Control
Sources and scope
Further background is available in Sutton and Barto’s Reinforcement Learning: An Introduction, second edition, Spinning Up’s key concepts, algorithm design choices and policy-optimisation explanation, and Gymnasium’s time-limit handling documentation. The historical post-training example comes from Ouyang and colleagues’ 2022 InstructGPT paper.
The routes, probabilities and update values in this guide were constructed for teaching. Convergence statements and evaluation claims depend on the particular environment, algorithm, data, representation and experimental conditions. Start with a transparent small model, then carry those conditions into each extension.
Back to chapter index · Return to The Gold Standard of Education