How do examples and rubrics improve Super Intelligence work? Examples show what acceptable work looks like in concrete form; rubrics explain which properties make that work good. Used together, they reduce ambiguity, make human review more consistent, improve reusable instructions and create a stronger evaluation layer for workplace SI.
This article follows How to Get Structured Outputs from Super Intelligence. Structure tells the system where information belongs. Examples and rubrics tell the system and reviewer what quality looks like inside that structure.
In this eduKateSG workplace series, Super Intelligence is the practical machine-intelligence layer commonly described as artificial intelligence, generative AI, assistants, copilots, agents and connected automation. Examples and rubrics are especially valuable because many workplace tasks cannot be specified completely through rules alone. Quality often includes judgment, relevance, clarity, evidence, tone or usefulness.
Examples and Rubrics Solve Different Problems
An example is one instance of work. It demonstrates form, tone, structure and level of detail. A rubric is an abstract standard. It defines the dimensions that should hold across many different cases.
Examples answer: “What does good look like here?” Rubrics answer: “Why is this good, and how do we judge another case that looks different?”
Why Examples Alone Are Not Enough
A model can imitate surface features of an example without understanding which features matter. If an example report is three pages long, the model may reproduce the length even when the real standard is decision usefulness and evidence quality.
Examples become safer when paired with a rubric that explains which characteristics are essential and which are incidental.
Why Rubrics Alone Are Not Enough
Rubrics can be abstract. “Clear, concise, evidence-based” may be interpreted differently by different users. Examples anchor those concepts in real artefacts.
A strong evaluation system therefore uses both: rubric for principle, examples for calibration.
The Example-Rubric Loop
- Define the task. What output and receiver matter?
- Collect real examples. Include strong, weak and edge cases.
- Extract quality dimensions. What distinguishes the strong examples?
- Build the rubric. Define each dimension and severity or score.
- Test the rubric. Can different reviewers use it consistently?
- Use examples with the rubric. Show how criteria apply in practice.
- Evaluate SI output. Record recurring failure patterns.
- Revise the examples or rubric. Improve when the task or standards change.
The Five Example Types
1. Strong positive example
Shows an output that meets the intended standard. It should be representative, not a rare masterpiece impossible to reproduce under normal conditions.
2. Weak example
Shows common failure without being absurd. Weak examples are valuable because they teach boundaries, not just aspiration.
3. Edge example
Shows a difficult but valid case: incomplete information, unusual formatting, competing priorities or an exception that still belongs in scope.
4. Should-stop example
Shows a case where the correct behaviour is to abstain, escalate or request more information. This is especially important in high-consequence workflows.
5. Boundary example
Shows two cases close to the line between categories, severity levels or approval thresholds. Boundary examples clarify definitions better than obvious cases.
Examples Should Be Representative
Do not curate only polished outputs from expert users. Include the kinds of cases the workflow actually sees: messy inputs, conflicting sources, ambiguous language and ordinary routine work.
Representative examples teach the system to operate inside real conditions rather than demo conditions.
Examples Should Be Current
A strong example can become wrong when policy, product, market or organisational process changes. Examples should be reviewed alongside the sources and workflow they illustrate.
Examples Should Preserve Source Boundaries
If an example contains a policy claim, number or approval, make clear whether that element is part of the current standard or merely historical context.
Examples should not become frozen sources of truth accidentally.
Examples Should Not Leak Sensitive Data
Training and evaluation examples may contain customer, employee or confidential information. Use approved environments, minimise sensitive fields and anonymise or synthesise examples where appropriate.
The Rubric Anatomy
- Dimension: what aspect of quality is judged?
- Definition: what does the dimension mean?
- Evidence: what should the reviewer look for?
- Levels: what distinguishes strong, acceptable and weak performance?
- Severity: which failures block acceptance?
- Weight: if used, how important is this dimension relative to others?
- Exception: when does the criterion not apply?
Rubric Dimension: Factual Accuracy
Does the output match authoritative sources and avoid unsupported claims? For high-impact work, a material factual error should often block acceptance regardless of strength elsewhere.
Rubric Dimension: Completeness
Does the output include the required information for the receiver? Completeness should be defined by the workflow, not by word count.
Rubric Dimension: Relevance
Does the output focus on information that changes the receiver’s decision or action? A long answer can be complete but irrelevant.
Rubric Dimension: Evidence
Are important claims supported by traceable evidence? Does the evidence actually support the claim?
Rubric Dimension: Reasoning
Are inferences consistent with the evidence, and are assumptions visible? Reasoning criteria are especially important for decision support and analysis.
Rubric Dimension: Clarity
Can the intended receiver understand the output without unnecessary reconstruction? Clarity depends on audience, terminology and structure.
Rubric Dimension: Concision
Does the output contain the necessary information without avoidable repetition? Concision should not reward omission of important qualifiers.
Rubric Dimension: Tone
Is the style appropriate to the relationship and purpose? Tone should remain secondary to factual and authority constraints.
Rubric Dimension: Actionability
Does the output make next action, owner or decision clear where the workflow requires it?
Rubric Dimension: Risk
Does the output preserve important uncertainty, avoid unsupported commitments and flag cases requiring escalation?
Rubric Dimension: Format
Does the output follow the required schema, sections or fields? Format can often be validated deterministically.
Rubric Dimension: Source Currentness
Does the output rely on sufficiently current information for the action? This criterion matters in customer, financial, project and policy workflows.
Rubric Dimension: Authority
Does the output stay within the system’s or user’s legitimate role? A draft should not masquerade as approval; a recommendation should not become an executed action.
Scoring vs Pass/Fail
Some tasks benefit from numeric or multi-level scoring; others are better handled as pass/fail on mandatory criteria plus qualitative feedback.
Do not average away a critical failure. A report with excellent style and one fabricated figure should not receive a reassuring overall score.
Critical, Major, Minor and Suggestion
Severity levels can be more useful than numeric scores. Critical blocks release. Major changes meaning or compliance. Minor affects clarity or completeness but remains recoverable. Suggestion is optional improvement.
Severity helps reviewers focus on what actually matters.
The Mandatory-Criteria Pattern
For high-impact tasks, define a small set of non-negotiable criteria: facts grounded, required approval present, no prohibited data, required source current, no unsupported commitment.
The output must pass these before optional quality scoring matters.
The Weighted-Rubric Trap
Weighting can create false precision. A 9/10 in style should not compensate for a 2/10 in factual accuracy if factual accuracy is mandatory.
Use gates for critical criteria and weights only for genuinely compensatory dimensions.
The Rubric Calibration Session
Give several reviewers the same outputs and ask them to score independently. Discuss disagreements. The purpose is to clarify the rubric, not to force identical opinions where judgment is legitimately different.
Large disagreement often reveals ambiguous definitions.
Inter-Reviewer Consistency
For repeated review workflows, track whether reviewers apply the rubric similarly. Inconsistency can make SI evaluation noisy and confuse users about what standard matters.
The Model-as-Judge Trap
A model can help apply a rubric, but it should not automatically be treated as an independent objective evaluator. It may share biases or failure modes with the generator.
For consequential tasks, combine model-assisted review with deterministic checks and human judgment where appropriate.
Rubrics for Drafting
A drafting rubric might include factual grounding, audience fit, clarity, commitment safety, tone and call to action.
Examples should show strong and weak versions of each dimension.
Rubrics for Research
Research rubrics can include source quality, citation accuracy, coverage, methodology awareness, disagreement handling, uncertainty and synthesis.
Rubrics for Summaries
Summary rubrics should define what must survive compression: decision, risk, date, owner, figure, uncertainty or source.
Rubrics for Classification
Classification rubrics can define category boundaries, evidence required, acceptable ambiguity and when to use needs_review.
Rubrics for Handoffs
Handoff rubrics can include current state, evidence, completed work, unresolved questions, next action, owner, deadline and acceptance.
Rubrics for Meetings
Meeting output can be judged on decision accuracy, action ownership, deadline capture, unresolved questions and source traceability.
Rubrics for Email
Email rubrics can include factual support, purpose, recipient fit, commitment safety, tone and next action.
Rubrics for Structured Outputs
Structured-output rubrics can include schema validity, evidence coverage, field accuracy, missing-state correctness and downstream usefulness.
Rubrics for Agent Work
Agent rubrics can include objective completion, tool-use appropriateness, permission compliance, state accuracy, exception handling and world-return evidence.
Rubrics for Code
Code rubrics can include functional correctness, tests, security, maintainability, scope discipline and adherence to repository conventions.
Rubrics for Executive Briefs
Executive rubrics can include decision relevance, evidence quality, risk visibility, options, uncertainty and concision.
Rubrics for Learning Content
Education rubrics can include correctness, level appropriateness, learning objective alignment, explanation quality, practice quality and transfer.
The Example Library
A workplace can maintain a small example library tied to recurring workflows. Each example should identify why it is strong or weak, which rubric criteria it illustrates and whether it remains current.
A small high-quality library is better than a large uncurated archive.
Example Ownership
Examples should have an owner or owning team. This is especially important when they encode policy, professional judgment or customer commitments.
Example Versioning
If the workflow changes materially, mark old examples as historical or retire them. A model should not learn current policy from obsolete examples.
Example Retrieval
When context is limited, retrieve the most relevant examples by task, edge case or output type rather than loading the entire library.
Too many examples can create noise or contradictory patterns.
Few-Shot vs Many-Shot
A few representative examples can clarify form and boundaries. Many examples may help when categories are subtle, but they increase context cost and the chance of inconsistency.
Use enough examples to teach the distinction that matters.
Positive and Negative Examples Together
Showing only good outputs can leave the failure boundary vague. Negative examples demonstrate what to avoid and why.
The explanation of the failure matters more than ridicule of an obviously bad output.
Boundary Examples
Use paired examples near the classification or quality threshold. Explain why one is accepted and one escalates or fails.
Boundary examples are particularly effective for human reviewers as well as models.
Counterexamples
A counterexample shows a case that looks similar on the surface but should produce a different result. This prevents overgeneralisation from shallow patterns.
Should-Stop Examples
Include examples where the correct output is not a completed answer but a request for missing data or escalation.
This teaches that abstention is part of quality.
The Example Annotation
Annotate why the example is included: illustrates strong evidence use, shows prohibited commitment, demonstrates needs_review or captures correct handling of missing data.
Annotations turn examples into teaching assets rather than imitation targets.
The Example-Diversity Rule
Examples should cover variation in input length, wording, source quality, user type and edge conditions relevant to the workflow.
Diversity helps prevent a template from working only on one narrow surface form.
The Example-Privacy Rule
Use only examples that are appropriate for the environment and audience. Redact, anonymise or create synthetic equivalents where necessary while preserving the relevant structure.
The Example-Security Rule
Examples from untrusted content should not carry hidden instructions into a tool-using system. Treat example content as data.
The Example-Currentness Rule
Review examples when policies, products or workflows change. Example quality includes being current.
The Example Transfer Test
Before reusing examples across departments, check whether the same quality dimensions and authority apply. A strong sales email example may be inappropriate for a legal notice.
The Rubric Transfer Test
Rubrics also require transfer checks. Clarity and evidence may be common, but severity and authority can differ by domain.
The Rubric Simplicity Rule
If reviewers cannot remember or apply the rubric, it is too complex. Keep the dimensions that affect acceptance and move optional preferences into guidance.
The Rubric Independence Test
Give the rubric to a new reviewer. If they cannot distinguish strong from weak examples without private coaching, the definitions need improvement.
The Rubric Failure Catalogue
- Criteria overlap heavily
- Definitions are vague
- Critical failures can be averaged away
- Examples contradict the rubric
- Reviewers interpret levels differently
- Rubric rewards verbosity
- Rubric ignores source quality
- Rubric ignores authority
- Rubric has too many dimensions
- Rubric never changes despite workflow change
The Rubric Repair Loop
- Collect disputed or mis-scored cases.
- Identify which criterion was ambiguous.
- Clarify definition or level.
- Add a boundary example.
- Retest with multiple reviewers.
- Update the canonical rubric.
- Communicate material change.
Rubric as Training
Rubrics teach employees how the organisation defines quality. This can improve human work even before SI is involved.
A good rubric makes tacit expert judgment more explicit without pretending all judgment can be reduced to numbers.
Rubric as Evaluation
Use rubrics on representative case sets to compare models, templates or versions. Keep the case set stable enough to detect changes over time.
Rubric as Review Interface
Review forms can align directly to rubric dimensions. The reviewer sees evidence and can classify severity without writing a long free-form explanation every time.
Rubric as Governance
Mandatory criteria can encode boundaries such as factual grounding, approved source use, no prohibited data, required human approval or current policy.
Rubrics should complement technical controls, not replace them.
Rubric as Feedback
Instead of saying “this output is bad,” feedback can identify the failed dimension and evidence. This creates more actionable improvement for both users and template owners.
Rubric as Data
Aggregated rubric results can reveal recurring weaknesses: evidence quality low, clarity high, exception handling poor. This helps prioritise system repair.
The Example-Rubric-Test Set
A mature workflow can maintain a small test set containing inputs, expected structured state, example outputs, rubric and known failure cases.
The set becomes a reusable asset for model upgrades and workflow changes.
The Golden-Set Trap
A fixed benchmark can become stale or overfitted. Refresh the evaluation set with new real cases while preserving enough continuity to compare performance.
The Perfection Trap
Not every output needs top scores on every dimension. Define the acceptance threshold appropriate to the task. Internal brainstorming and customer commitments deserve different standards.
The Rubric-Gaming Trap
If a model or user optimises visibly for rubric wording, it may produce surface compliance without real usefulness. Receiver feedback and outcome metrics provide a counterbalance.
The Style-Dominance Trap
Reviewers often notice tone and grammar before evidence. Rubric ordering can place factual and authority criteria first so cosmetic quality does not dominate.
The Length Bias Trap
Longer outputs can appear more complete. The rubric should reward necessary information, not sheer volume.
The Familiarity Bias Trap
An output resembling a known example may feel better even if it misses the current task. Use rubric criteria to prevent imitation from substituting for fit.
The Reviewer-Fatigue Trap
Complex rubrics slow review and encourage superficial scoring. Use a mandatory quick gate plus deeper criteria only where needed.
The Model-Agreement Trap
Two models agreeing does not prove correctness. Agreement can be another signal but not a substitute for sources and domain review.
Worked Example: Customer Email
Rubric: facts grounded, customer state correct, no unsupported promise, tone appropriate, next action clear. Positive example shows a concise source-grounded reply. Negative example promises delivery based on stale information. Boundary example shows when compensation needs human approval.
Worked Example: Executive Brief
Rubric: decision relevance, evidence, risk, uncertainty, options, concision. Strong example surfaces the decision in the first paragraph and links evidence. Weak example summarises every project without indicating what leadership must decide.
Worked Example: Contract Comparison
Rubric: clause accuracy, source reference, deviation classification, uncertainty, legal-review flag. Examples should show both standard and ambiguous clauses.
Worked Example: Research Synthesis
Rubric: source quality, coverage, citation accuracy, disagreement, limitations and synthesis. Counterexamples show why a large number of weak sources does not equal strong evidence.
Worked Example: Support Triage
Rubric: correct category, evidence, exception detection, routing and missing-context handling. Boundary examples clarify routine versus specialist escalation.
Worked Example: Meeting Closure
Rubric: confirmed decisions, actions, owners, deadlines, unresolved questions and source traceability. A negative example incorrectly turns discussion into decision.
Worked Example: Code Review
Rubric: functional correctness, tests, security, maintainability, scope discipline and repository conventions. Examples can show acceptable refactor versus over-broad change.
Worked Example: Learning Feedback
Rubric: correctness, alignment to objective, diagnosis, actionability, learner level and transfer. Strong feedback points to evidence in the learner’s work rather than generic praise.
The Rubric Pilot
- Collect five to ten real outputs.
- Have experts identify what matters.
- Draft four to seven dimensions.
- Define mandatory failures.
- Create strong and weak examples.
- Score independently.
- Discuss disagreements.
- Revise definitions.
- Add edge cases.
- Release canonical rubric.
The Example Library Pilot
Start with a small set: two strong examples, two common failures, one boundary case and one should-stop case. Expand only when repeated confusion justifies it.
The Example-Rubric Metrics
- First-pass acceptance
- Reviewer agreement
- Correction severity
- Time to review
- Failure by rubric dimension
- Boundary-case escalation accuracy
- Receiver satisfaction
- Outcome quality
Examples and Rubrics in Prompt Templates
A reusable instruction can reference the rubric and retrieve only relevant examples. This keeps the main prompt concise and the quality layer maintainable.
Examples and Rubrics in Structured Outputs
Rubric fields can align with structured outputs, allowing reviewers to score evidence, completeness or authority at field level.
Examples and Rubrics in Agents
Agents can use rubrics as checkpoints before external action, but high-impact release gates should include deterministic or human controls where necessary.
Examples and Rubrics in Training
The same examples and rubrics can train employees, reducing the gap between human and SI standards.
Examples and Rubrics in Procurement
Procurement can use examples of acceptable evidence packets and rubrics for comparability, completeness and source support without automating the award decision.
Examples and Rubrics in HR
HR can use rubrics for administrative quality and documentation. High-stakes employment decisions require careful fairness, privacy and human accountability beyond a generic model rubric.
Examples and Rubrics in Finance
Finance can use rubrics for explanation quality, source fidelity and materiality while exact figures remain authoritative in financial systems.
Examples and Rubrics in Legal
Legal teams can use examples and rubrics for issue spotting, clause comparison and drafting support while qualified professionals retain interpretation.
Examples and Rubrics in Marketing
Marketing rubrics can include claim support, audience fit, clarity, brand alignment and regulatory constraints. Factual claim accuracy should remain a hard gate.
Examples and Rubrics in Operations
Operations can use examples of strong incident packets and rubrics for state accuracy, evidence, owner, next check and escalation quality.
Examples and Rubrics in Education
Education already relies on rubrics. SI workflows can benefit from the same logic: criteria, evidence, levels and feedback, with educators retaining judgment over learning context.
What This Article Owns
This page owns examples and rubrics as the quality layer for workplace Super Intelligence: example types, rubric design, calibration, evaluation, training, failure modes and maintenance.
The previous article owns structured outputs. The next article owns constraints—the boundaries that define what the system may not do even when an output looks high quality.
Frequently Asked Questions
Should I use examples or a rubric?
Use both when the task matters. Examples show concrete quality; rubrics explain the transferable dimensions.
How many examples are enough?
Start with a small representative set including strong, weak, edge and should-stop cases. Add examples only when they clarify a recurring distinction.
Should rubrics use numbers?
Not necessarily. Pass/fail gates and severity levels are often clearer for workplace tasks.
Can SI grade its own work?
It can assist with rubric-based review, but should not be assumed independent or objective. Combine model review with sources, deterministic checks and humans according to risk.
How do I calibrate reviewers?
Have multiple reviewers score the same examples independently, discuss disagreements and refine the rubric definitions.
What if the examples become outdated?
Version or retire them. Examples are part of workplace knowledge and need maintenance.
What comes next?
Continue to How to Set Constraints for Workplace Super Intelligence, which defines the hard boundaries around data, action, authority, scope and risk.
The Core Example-and-Rubric Rule
Examples show the system what good looks like; rubrics tell the organisation why it is good.
Used together, they make quality more explicit, review more consistent and workflow improvement more measurable without pretending all professional judgment can be reduced to one score.
The Quality Stack
Examples and rubrics sit inside a wider quality stack. The source establishes facts. The instruction defines the task. The schema defines the output shape. The example shows a concrete standard. The rubric explains quality. Verification determines whether the result can be accepted.
Weak SI workflows often ask one layer to do the job of all the others. A long prompt tries to encode policy, examples, format, evaluation and safety simultaneously. Separating the stack makes maintenance much easier.
Layer 1 — Source Quality
No rubric can rescue a workflow built on stale or incorrect sources. Start by ensuring the system has access to the evidence required for the task.
Layer 2 — Task Definition
A clear instruction tells the system what to do, what inputs to use and where authority stops.
Layer 3 — Output Structure
Structure makes required elements visible and allows validation. It does not determine whether the content is good.
Layer 4 — Examples
Examples demonstrate how the task and structure look in real cases, including edge conditions.
Layer 5 — Rubric
The rubric abstracts the quality dimensions so new cases can be judged even when they do not resemble the examples closely.
Layer 6 — Verification
Verification uses sources, tests, rules and human judgment to determine whether the output meets the standard and can proceed.
The Example Selection Process
- Collect real accepted and rejected outputs.
- Remove cases that are obsolete or depend on hidden context.
- Identify which examples represent common work.
- Add edge and should-stop cases.
- Annotate the relevant rubric criteria.
- Check privacy and permissions.
- Version the accepted example set.
The goal is not to build a museum of good work. The set should teach distinctions that matter to the live workflow.
The Strong-Example Problem
A perfect example created by the best expert can be too distant from normal operating conditions. The model may imitate a standard that ordinary users cannot supply enough context to reproduce.
Use examples that are both high quality and representative.
The Weak-Example Problem
Weak examples should illustrate realistic failure, not strawmen. A near-miss teaches more than an obviously broken output because it clarifies the actual acceptance boundary.
The Edge-Case Problem
Edge examples should be selected because they reveal a meaningful boundary: missing evidence, unusual language, conflicting sources, a high threshold or a category ambiguity.
Do not overload the example set with rare curiosities that do not change the workflow design.
The Should-Stop Example
A should-stop example teaches the system and reviewers that completion is not always the goal. The correct output can be needs_review, missing_information or escalation.
These examples are especially important for agents and high-consequence workflows because they normalise abstention as quality.
The Counterexample Pattern
A counterexample looks superficially similar to a strong case but should produce a different result. It teaches the model not to rely on shallow cues.
For example, two customer requests may mention refunds, but one is inside policy and the other is a dispute requiring specialist review.
The Paired-Example Pattern
Show two outputs side by side and annotate the difference. Pairing is particularly effective for tone, evidence, concision and boundary decisions.
The Progressive-Example Pattern
Show a weak first draft, improved revision and accepted final version. This demonstrates how rubric feedback changes work rather than only showing the endpoint.
The Role-Specific Example Pattern
The same state can be expressed differently for executives, engineers, customers or finance. Role-specific examples teach audience adaptation without changing facts.
The Rubric Design Process
- Identify the receiver and decision.
- Collect accepted and rejected examples.
- Ask experts what they notice.
- Group repeated quality concepts.
- Reduce overlapping dimensions.
- Define mandatory gates.
- Define levels or severity.
- Test with multiple reviewers.
- Add boundary examples.
- Version and release.
The process begins from real work rather than generic words such as good, professional or accurate.
Rubric Design: One Dimension, One Meaning
Avoid dimensions that mix several concepts. “Clear and accurate” makes it hard to know why an output failed. Separate factual accuracy from clarity.
Rubric Design: Observable Evidence
Each dimension should tell the reviewer what to look for. “Useful” is vague. “States the decision required, owner and deadline” is observable.
Rubric Design: Distinct Levels
Adjacent levels should differ meaningfully. If reviewers cannot explain the difference between 3 and 4, the scale may be too fine.
Rubric Design: Mandatory Gates
Critical criteria should gate acceptance. Examples include unsupported factual claims, prohibited data exposure, missing required approval or unsafe action.
Rubric Design: Contextual Criteria
Some dimensions apply only in certain cases. Tone may be important for external email but irrelevant to structured extraction. Mark conditional criteria instead of forcing every output through every dimension.
Rubric Design: Receiver Fit
Quality depends on who uses the result. An executive brief that is too detailed may be weak even when every fact is correct. A technical incident packet that omits logs may be weak even if concise.
Rubric Design: Currentness
For time-sensitive work, include whether the evidence is current enough for the action. A correct old value can still make the output unusable.
Rubric Design: Uncertainty
Reward outputs that preserve uncertainty appropriately rather than pretending confidence. The rubric can check whether assumptions, missing data and source conflicts remain visible.
Rubric Design: Authority
Check whether the output stays within the role’s legitimate authority. A draft recommendation should not be written as a final approval.
Rubric Design: Actionability
When the workflow requires next steps, quality includes a clear action, owner and trigger. A beautiful summary without actionable state may be poor operational work.
The Four-Level Rubric
A simple four-level scale can work well: Unacceptable, Needs Revision, Acceptable, Strong. The definitions should focus on observable differences.
Four levels reduce false precision and are easier to calibrate than a ten-point scale.
The Pass/Revise/Escalate Rubric
For some operational workflows, three outcomes are enough: pass, revise, escalate. Pass meets the standard, revise is recoverable within the normal workflow, and escalate requires specialist or authorised review.
The Severity Rubric
Severity can be independent of overall quality. A minor grammar issue does not equal a major unsupported claim. Store severity alongside the criterion.
The Binary Gate Plus Quality Score
A practical pattern is mandatory pass/fail gates for safety and correctness, followed by a smaller quality score for clarity, concision or style.
This prevents cosmetic strengths from compensating for critical failure.
Rubrics for Human Review
A human reviewer benefits from seeing the rubric criteria beside the output and source evidence. The interface should emphasise mandatory gates first.
Review time should be measured because a complex rubric can erase the productivity gain.
Rubrics for Automated Review
SI can apply a rubric to candidate output and flag likely issues. This can reduce human search but should not be assumed independent verification where the model shares the same failure mode as the generator.
Rubrics for Hybrid Review
Use deterministic checks for types, calculations and required fields; SI review for semantic dimensions such as relevance or completeness; human review for judgment and authority.
Hybrid review aligns each check with the mechanism best suited to it.
Rubrics for Model Comparison
Run the same representative cases through different models and score with the same rubric. Compare failure patterns as well as averages.
A model with slightly lower average quality may be better if it makes fewer critical errors on the task.
Rubrics for Prompt Comparison
When testing prompt versions, hold the case set and rubric stable. This reduces subjective preference and makes changes more interpretable.
Rubrics for Workflow Comparison
Compare manual, assistive and automated variants using the same acceptance standard. This shows whether speed improvements change quality or review burden.
Rubrics for Agent Evaluation
Agent quality includes not only final output but behaviour: tool choice, permission compliance, stopping, recovery, state accuracy and evidence of action.
A strong final answer cannot excuse an unauthorised intermediate action.
Rubrics for Retrieval
Retrieval rubrics can include relevance, authority, currentness, coverage and whether the cited source supports the answer.
Rubrics for Knowledge Answers
Knowledge-answer quality can include source grounding, completeness, currentness, uncertainty and appropriate abstention when the knowledge base lacks an answer.
Rubrics for Planning
Planning quality can include objective alignment, dependencies, assumptions, feasibility, risk, ownership and whether proposed dates are supported by real constraints.
Rubrics for Decision Support
Decision-support rubrics can include evidence quality, options, trade-offs, uncertainty, bias toward one recommendation and whether the authorised decision-maker remains clear.
Rubrics for Data Analysis
Data-analysis rubrics can include correct calculations, transparent assumptions, source quality, appropriate method, uncertainty and distinction between correlation and causal interpretation.
Rubrics for Documentation
Documentation rubrics can include correctness, currentness, completeness, usability, ownership, exception handling and alignment with the real workflow.
Rubrics for SOPs
SOP quality can include trigger, prerequisites, ordered steps, decision points, exceptions, owner, verification and recovery.
Rubrics for Customer Service
Support quality can include account-state accuracy, policy fit, empathy, commitment safety, resolution and appropriate escalation.
Rubrics for Sales
Sales support output can be judged on account accuracy, customer relevance, claim support, commercial boundary, follow-up clarity and CRM state.
Rubrics for Marketing
Marketing quality can include audience fit, claim support, differentiation, brand alignment, clarity and regulatory constraints.
Rubrics for Finance
Finance output can be judged on figure fidelity, calculation source, materiality, explanation, evidence and approval boundary.
Rubrics for HR
HR administrative outputs can be judged on completeness, policy fidelity, privacy, clarity and next action. Employment judgments require additional domain-specific fairness and legal controls.
Rubrics for Legal
Legal support can be judged on source accuracy, clause fidelity, issue spotting, uncertainty and whether professional review is appropriately triggered.
Rubrics for Engineering
Engineering output can be judged on functional correctness, tests, security, maintainability, scope, documentation and operational impact.
Rubrics for Operations
Operations quality can include current state, severity, evidence, actions already attempted, owner, next check and escalation.
Rubrics for Education
Education rubrics can include correctness, alignment to learning objective, level, feedback evidence, practice quality and transfer.
Examples as Organisational Memory
A curated example library captures more than style. It shows how the organisation handles normal, edge and exception cases.
This makes examples a form of organisational memory that must be maintained alongside policies and SOPs.
Rubrics as Organisational Memory
A rubric captures tacit standards that experts may otherwise apply inconsistently. It turns implicit quality judgment into a shared language.
The rubric should remain open to revision when the work or evidence changes.
Examples as Training Data for Humans
New employees can learn from annotated examples faster than from abstract guidance alone. Examples show how standards appear in real outputs.
Rubrics as Training Data for Humans
Rubrics teach employees what to notice. They can improve human work and review independently of SI.
The Reviewer Calibration Workshop
- Select five representative outputs.
- Review independently.
- Compare scores or severity.
- Discuss disagreements.
- Clarify definitions.
- Add one boundary example.
- Repeat on a new set.
- Release the revised rubric.
A short calibration workshop can reveal more about organisational standards than many pages of guidance.
The Reviewer Drift Check
Over time, reviewers can become stricter, looser or adapt to common model errors. Periodically rescore a small stable set to detect drift.
The Rubric Change Ledger
Record material changes to criteria or severity. If the evaluation standard changes, performance trends need to be interpreted accordingly.
The Example Change Ledger
Record why examples were added, replaced or retired. This prevents the library from accumulating contradictory cases.
The Quality Dashboard
- Acceptance rate
- Critical failure rate
- Major correction rate
- Review time
- Reviewer disagreement
- Failure by rubric dimension
- Boundary-case escalation
- Receiver rework
- Example retrieval frequency
Choose only the metrics that support the workflow. Quality dashboards should guide repair rather than become reporting overhead.
Quality Improvement From Failure Data
If outputs repeatedly fail evidence quality, improve source retrieval. If they fail actionability, redesign the schema. If they fail tone only, improve style guidance or examples. Rubric data points to the correct layer.
Quality Improvement From Reviewer Disagreement
Disagreement may indicate ambiguous rubric language, legitimate judgment variation or insufficient evidence. Diagnose before forcing consensus.
Quality Improvement From Receiver Rework
If receivers consistently rewrite outputs that reviewers accepted, the rubric may be optimising the wrong standard. Add receiver usefulness to the quality model.
Quality Improvement From Should-Stop Cases
If the system completes cases it should escalate, add stronger boundary examples and mandatory rubric criteria for evidence and scope.
The Example Overfitting Trap
A model can learn to imitate phrasing rather than underlying quality. Vary examples while keeping the rubric stable so the important dimensions remain visible.
The Rubric Overfitting Trap
A team can optimise to a rubric while missing real-world value. Keep outcome metrics and receiver feedback alongside rubric scores.
The Benchmark Staleness Trap
A static case set can stop representing new work. Refresh part of the set periodically while preserving a stable core for comparison.
The Synthetic-Example Trap
Synthetic examples are useful for privacy and rare edge cases, but should be validated by domain experts. Unrealistic examples can teach distinctions that do not exist in real work.
The Prestige-Example Trap
Do not select examples merely because senior leaders produced them. Select them because they clearly demonstrate the standard.
The Rubric Complexity Trap
A 30-dimension rubric may be theoretically complete and operationally unusable. Keep the quality model small enough that reviewers can apply it consistently.
The Rubric Precision Trap
Decimal scores imply measurement precision that subjective judgments may not support. Use broad levels where appropriate.
The Rubric Authority Trap
A rubric can describe quality but does not grant decision rights. An output can score highly and still require legal, managerial or professional approval.
The Example Authority Trap
A previous approved example may not create precedent for every future case. Preserve current policy and context.
The Quality Gate for High-Impact Work
- Authoritative sources present.
- Material claims supported.
- Required approval state correct.
- No prohibited data exposed.
- Uncertainty visible.
- Stop conditions respected.
- External action still permission-gated.
Only after these gates pass should softer quality dimensions matter.
The Quality Gate for Low-Risk Work
Low-risk brainstorming or internal drafting can use lighter review. The objective is proportionality, not maximal governance.
The Example-Rubric Release Pack
- Task and receiver
- Rubric dimensions
- Mandatory gates
- Level definitions
- Positive examples
- Negative examples
- Boundary examples
- Should-stop examples
- Owner
- Version
- Last calibration
The Example-Rubric Update Triggers
- Workflow or policy change
- New material failure type
- Reviewer disagreement increases
- Model update
- Receiver needs change
- New edge case becomes common
- Legal or professional standard changes
The Example-Rubric Retirement Rule
Retire examples and criteria that no longer represent current work. Keep historical material separate if it remains useful for audit or learning.
The Quality Layer and Structured Output
Structure answers where the information belongs. The rubric answers whether the information is good enough. Examples show what the standard looks like. This three-part combination is especially powerful for repeatable workflows.
The Quality Layer and Constraints
Rubrics describe quality, but constraints define hard boundaries. A system can produce high-quality prose and still violate an action limit or data restriction. The next article focuses on those non-negotiable constraints.
The Final Calibration Checklist
- Examples reflect real work.
- Positive and negative cases exist.
- Boundary cases are clear.
- Should-stop cases exist.
- Rubric dimensions are distinct.
- Mandatory gates cannot be averaged away.
- Reviewers can apply the rubric consistently.
- Receiver usefulness is represented.
- Sources and authority remain visible.
- Version and owner are known.
The Final Quality Principle
Quality becomes scalable when the organisation can show it, describe it and review it consistently. Examples show it. Rubrics describe it. Evaluation tests whether the workflow achieves it.
That is the deeper role of examples and rubrics in workplace Super Intelligence: they turn tacit expectations into a shared quality interface without pretending professional judgment is reducible to one number.
The Rubric Should Follow the Receiver
A rubric is strongest when it reflects what the receiver needs to do next. A manager may care whether the brief exposes the decision, risk and evidence. A customer may care about correctness, clarity and next step. An engineer may care about reproducibility, tests and scope.
Quality therefore depends partly on use. A generic rubric can miss the thing that makes an output operationally valuable.
Receiver Rubric: Executive
- Decision visible in first section
- Evidence sufficient for the decision
- Material risk and uncertainty visible
- Options comparable
- Recommendation clearly separated from fact
- Required owner or action stated
- Excess background removed
Receiver Rubric: Manager
- Current state accurate
- Blockers and dependencies visible
- Actions have owners
- Deadlines distinguish planned from confirmed
- Exceptions surfaced
- Evidence available for material claims
Receiver Rubric: Customer
- Facts correct
- Language understandable
- Commitment within authority
- Next step clear
- Tone appropriate
- No unnecessary internal detail
Receiver Rubric: Engineer
- Problem reproducible
- Environment and conditions clear
- Evidence or logs linked
- Scope bounded
- Tests included where relevant
- Unresolved technical questions explicit
Receiver Rubric: Finance
- Authoritative figures used
- Calculations traceable
- Variance materiality clear
- Explanation supported
- Approval boundary visible
- Period and units correct
Receiver Rubric: Legal
- Source text accurate
- Jurisdiction or version explicit
- Deviation correctly identified
- Uncertainty preserved
- Qualified review triggered
- Commitment authority not assumed
Receiver Rubric: Educator
- Learning objective aligned
- Evidence from learner work
- Diagnosis plausible
- Feedback actionable
- Level appropriate
- Next practice supports transfer
Examples by Workflow Phase
Examples can be organised by where they sit in the workflow. Intake examples show good and bad source packages. Production examples show strong outputs. Review examples show correctly identified failures. Exception examples show when escalation is required.
This phase-based library helps teams teach the entire workflow rather than only the final artefact.
Intake Examples
Show what sufficient input looks like and what should be rejected as incomplete. This can be more valuable than another output example because many SI failures originate before generation begins.
Context Examples
Show a good context packet: current sources, relevant history, explicit constraints and no unnecessary sensitive material.
Output Examples
Show accepted outputs and annotate the rubric dimensions they satisfy. Keep examples tied to current workflow versions.
Review Examples
Show an output with one factual error, one authority error and one style issue. Reviewers learn to prioritise severity rather than treating all edits equally.
Exception Examples
Show cases that cannot proceed: missing approval, conflicting policy, unknown identity, unsupported field, security concern or novel case. The example should show the correct escalation packet.
Recovery Examples
For tool-connected systems, show what good recovery looks like after a timeout, failed action or uncertain state. This teaches that failure handling is part of quality.
The Rubric Should Follow Consequence
High-consequence workflows need stronger mandatory criteria and tighter evidence. Low-risk creative work can tolerate weaker outputs and broader variation.
A single corporate rubric for every SI task is therefore a poor design.
Low-Risk Rubric
- Relevant
- Useful
- Clear
- Within basic scope
Brainstorming and internal drafting may not need heavy factual gates if no external claim or decision is made.
Medium-Risk Rubric
- Facts supported
- Required fields complete
- Receiver fit
- Uncertainty visible
- Human review practical
- Next action clear
High-Risk Rubric
- Authoritative source present
- Material claims verified
- Permission and approval state correct
- Sensitive data controlled
- Critical uncertainty surfaced
- Stop conditions respected
- External action separately authorised
- Recovery available
The Rubric Acceptance Threshold
Define what level counts as acceptable. Some dimensions may need Strong for external release, while Acceptable is sufficient for an internal first draft.
Acceptance thresholds should reflect the receiver and consequence.
The Rubric Promotion Threshold
A workflow should gain more autonomy only after outputs repeatedly meet the relevant rubric and the human review process remains effective.
Quality evidence is one part of promotion; permissions, exceptions and recovery also matter.
The Rubric Demotion Threshold
If critical failures rise, reviewer disagreement increases or source currentness degrades, move the workflow back toward more review while the cause is repaired.
The Rubric Retirement Threshold
Retire criteria that no longer affect acceptance. A bloated rubric becomes less reliable because reviewers stop applying it carefully.
The Example Acceptance Threshold
An example should be included only if it clearly illustrates a current standard or boundary. If experts cannot explain why it belongs, remove it.
The Example Replacement Threshold
Replace examples when a newer case better illustrates the same distinction, when the old example depends on obsolete policy or when surface form is causing overfitting.
The Example Archive
Historical examples can remain in an archive for audit or training context but should not compete with current examples during live prompting.
The Rubric Library
An organisation can maintain reusable rubric patterns: factual grounding, executive brief, customer communication, handoff, review, research, code, agent action. Local workflows add domain-specific criteria.
This reduces duplication while preserving local meaning.
The Example Library Structure
- Example ID
- Workflow
- Case type
- Input version
- Output
- Accepted or rejected
- Rubric dimensions illustrated
- Why included
- Owner
- Currentness status
The Rubric Registry
- Rubric name
- Workflow
- Owner
- Version
- Mandatory gates
- Dimensions
- Levels
- Examples
- Last calibration
- Next review
The Rubric and Template Connection
The template can say: produce output using the current task instruction, structured schema and rubric version X. The model need not receive the entire organisational library every time.
Modularity keeps prompts smaller and quality standards maintainable.
The Rubric and Schema Connection
Some rubric criteria map directly to fields: evidence_present, owner_present, current_source, exception_flag. Others such as clarity or reasoning require semantic review.
This lets the system automate simple quality gates while reserving human judgment for the rest.
The Rubric and Workflow Connection
A rubric should sit at the correct workflow gate. A draft may be reviewed for factual support before external send. A contract issue list may be reviewed before legal interpretation. A code patch may be reviewed before merge.
The Rubric and Agent Connection
Agents can use a rubric as an internal quality checkpoint before returning work, but the workflow should not let an agent self-certify high-impact external action without independent controls.
The Rubric and Monitoring Connection
Recurring rubric failures can become monitoring signals. If source-currentness failures rise, alert the knowledge owner. If exception handling declines, review the eligibility boundary.
The Rubric and Incident Connection
After an incident, map the failure to rubric and control layers. Was the criterion missing, ignored, mis-scored or insufficient to prevent the action?
This turns incident review into quality-system improvement.
The Rubric and Cost Connection
Every criterion has review cost. High-use workflows should automate deterministic checks and focus human review on dimensions that genuinely require judgment.
A rubric should improve review efficiency, not multiply it.
The Rubric and Latency Connection
Real-time workflows may need a small fast gate and a deeper audit later. Match evaluation depth to timing and consequence.
The Rubric and Scale Connection
At scale, reviewer disagreement and exception rates create large workload. Test the rubric under realistic volume before using it as a production gate.
The Example and Model Change Test
After a major model update, rerun the canonical examples. Check whether the model now imitates them differently, handles edge cases better or fails in new ways.
The Rubric and Model Change Test
Use the same rubric to compare old and new model behaviour. Preserve enough stable cases for longitudinal comparison.
The Example and Prompt Change Test
When the instruction changes, rerun examples to confirm the change did not improve one criterion while degrading another.
The Rubric and Prompt Change Test
Score before and after on the same cases. Record whether correction time, not only rubric score, improved.
The Example and Source Change Test
If authoritative sources change, review examples that relied on the old content. Current examples should not teach obsolete facts or policy.
The Rubric and Receiver Change Test
If the downstream user changes, the rubric may need to change. A new receiver can require different information, terminology or actionability.
The Rubric and Workflow Change Test
When automation changes who does what, update the criteria. Human reviewers may now need to focus on exceptions rather than every field.
The Quality Operating Cycle
- Generate or perform task.
- Validate deterministic requirements.
- Apply rubric.
- Human reviews material judgment.
- Accept, revise or escalate.
- Record failure dimension.
- Aggregate recurring patterns.
- Repair source, instruction, schema or constraint.
- Recalibrate examples and rubric when needed.
The cycle turns quality from final inspection into continuous operating feedback.
The Quality Error Map
- Evidence failure: repair sources or retrieval.
- Completeness failure: repair schema or intake.
- Reasoning failure: repair task decomposition or model role.
- Tone failure: repair examples or style guidance.
- Authority failure: repair constraints and permissions.
- Currentness failure: repair knowledge lifecycle.
- Review inconsistency: repair rubric definitions and calibration.
- Receiver failure: repair output design.
The Quality System Should Identify the Right Repair
A rubric is most useful when it points to the layer that should change. Rewriting the prompt repeatedly is wasteful when the source is stale or the schema omits the field reviewers need.
The Rubric Owner Triangle
Process owner defines outcome; domain expert defines quality; technical owner integrates evaluation. In small teams, one person can hold several roles.
The Example Owner
The example owner ensures the case remains current and properly annotated. A strong example can become harmful when detached from the context that made it correct.
The Calibration Owner
Someone should convene periodic review when disagreement or workflow change justifies recalibration. This can be lightweight for low-risk tasks.
The Reviewer Training Pack
- Rubric definitions
- Mandatory gates
- Strong examples
- Common failures
- Boundary examples
- Should-stop examples
- Escalation path
This pack helps reviewers apply the standard without relying on private expert memory.
The Model Evaluation Pack
- Representative inputs
- Expected structured state
- Reference examples
- Rubric
- Known critical failures
- Should-stop cases
- Scoring instructions
The Receiver Feedback Pack
Collect brief structured feedback: missing information, unnecessary detail, clarification needed, actionability and downstream correction. Receiver evidence keeps the quality system connected to real work.
The Quality Review Cadence
High-volume or high-impact workflows may need frequent quality review. Low-risk templates can be checked only after material change or recurring error.
Cadence should match how quickly the workflow can drift.
The Quality Escalation Rule
If mandatory gates fail, the output should not proceed merely because the overall score is high. Escalate or return for repair.
The Quality Exception Rule
If the rubric does not fit a novel case, mark the case as exception rather than forcing a score. The exception may teach the organisation to extend the rubric later.
The Quality Uncertainty Rule
Reviewers should be able to mark uncertain rather than pretending precision. Disagreement can be useful evidence that the task definition is ambiguous.
The Quality Human-Agency Rule
Rubrics should support judgment, not imprison it. Authorised professionals may override a rubric when context justifies it, but the reason should be visible in high-impact workflows.
The Quality Transparency Rule
Users should know what standards their work or SI output is being judged against. Hidden rubrics make feedback difficult to interpret and improve.
The Quality Simplicity Rule
Keep the quality system only as complex as the consequence demands. Over-engineering evaluation can prevent useful low-risk experimentation.
The Quality Transfer Rule
Transfer generic criteria such as evidence or clarity only when their meaning remains stable. Domain-specific standards should stay domain-specific.
The Quality Release Gate
Before a rubric controls production release, confirm reviewer calibration, mandatory criteria, exception behaviour and receiver alignment.
The Quality Final Audit
- Rubric reflects the real receiver.
- Critical criteria are hard gates.
- Examples include strong and weak cases.
- Boundary and should-stop examples exist.
- Reviewers are calibrated.
- Sources remain authoritative.
- Currentness is monitored.
- Rubric changes are versioned.
- Receiver feedback is included.
- Outcome metrics sit beside rubric scores.
The Example-and-Rubric Final Threshold
The quality layer is mature when different authorised people can look at different cases, use the same standard, and reach decisions that are consistent enough for the workflow without hiding legitimate uncertainty.
That threshold turns examples and rubrics into shared organisational judgment infrastructure.
The Final Quality Rule
Use examples to make standards concrete. Use rubrics to make standards transferable. Use real outcomes to make sure the standards remain relevant.
When those three layers reinforce one another, Super Intelligence quality becomes easier to teach, test and improve across time.
The Rubric Production Threshold
A rubric should not become a production gate merely because it exists. Before it controls release, routing or approval, the organisation should know that reviewers understand it, mandatory criteria reflect real risk and edge cases have a route.
The production threshold is higher than the training threshold because bad evaluation can now block good work or release bad work at scale.
The Rubric Review Capacity Test
Estimate how long meaningful review takes under the rubric and compare that with expected volume. A detailed quality system can become a bottleneck if every case needs deep human scoring.
Use deterministic checks and SI-assisted pre-review to narrow human attention to the dimensions that genuinely require judgment.
The Rubric Exception Capacity Test
If the rubric sends many cases to needs_review, make sure the exception owner has enough capacity. High exception volume may indicate the rubric is too strict, the workflow is too broad or the source quality is weak.
The Rubric Boundary-Test Set
Maintain a small set of outputs that sit near acceptance boundaries. These cases are more useful for calibration than obvious good and bad examples because they reveal where reviewer definitions differ.
Boundary cases should be revisited when the task, policy or model changes.
The Rubric Counterfactual Test
Ask whether changing one dimension would alter the acceptance decision. If every criterion can change without affecting the outcome, the rubric may contain decorative factors.
This test helps remove dimensions that do not actually matter.
The Rubric Receiver-Cost Test
A high-scoring output should reduce downstream effort. If receivers still reconstruct context or rewrite the artefact, the rubric may reward the wrong properties.
Receiver correction is strong evidence for quality-system repair.
The Rubric Outcome Test
Where possible, compare rubric scores with later real outcomes. Do highly rated support replies reduce repeat contacts? Do highly rated briefs help decisions? Do highly rated educational explanations improve learner transfer?
Rubrics should remain connected to the real purpose of the work.
The Rubric Gaming Test
Watch for outputs that satisfy visible rubric wording while becoming less useful. A system may overproduce citations, repeat keywords or force sections merely to score well.
Outcome metrics and receiver feedback help detect this kind of gaming.
The Rubric Model-Bias Test
If the same model family generates and judges the work, shared blind spots may remain invisible. Use source verification, deterministic checks or independent human review for material dimensions.
The Rubric Human-Bias Test
Human reviewers can also apply standards unevenly. Calibration, explicit evidence and structured severity help reduce avoidable variation.
The objective is not perfect objectivity. It is sufficiently consistent, transparent judgment for the workflow.
The Rubric Fairness Test
For workflows affecting people, ask whether criteria are legitimate, job- or task-related and free from irrelevant proxies. A generic writing-quality rubric should not silently become a hiring-quality rubric.
The Rubric Privacy Test
Review whether evaluation examples or reviewer notes contain personal or confidential information unnecessary for the quality decision. Minimise and control access appropriately.
The Rubric Security Test
If examples include external or untrusted content, ensure they cannot alter the workflow’s trusted instructions. Example text is evidence, not authority.
The Rubric Currentness Test
A quality standard can become stale when the receiver, product, policy or regulatory environment changes. Review criteria and examples together after material change.
The Rubric Version Test
When a rubric changes, record whether historical scores remain comparable. If a critical criterion changes meaning, performance trends may need a new baseline.
The Example Set as Regression Test
Canonical examples can double as regression cases. After prompt, model or workflow changes, rerun them and compare whether previously accepted distinctions still hold.
Add new cases when real failures expose a missing boundary.
The Example Set as Onboarding
New reviewers can learn the standard by scoring examples before reviewing live work. Discuss differences and clarify the rubric.
This is often more effective than reading a long quality manual.
The Example Set as Model Evaluation
Run multiple models on the same cases and inspect not only average scores but critical failure distribution, escalation behaviour and review cost.
The Example Set as Prompt Evaluation
Use the same cases to compare prompt versions. If a new instruction improves style but worsens boundary handling, the trade-off becomes visible.
The Example Set as Workflow Evaluation
Compare manual, SI-assisted and automated versions using the same quality standard. This shows whether automation preserves accepted quality at lower total cost.
The Example Set as Incident Learning
After a material incident, add a representative anonymised case to the regression set when appropriate. The workflow should prove it can recognise or prevent the failure before regaining previous autonomy.
The Quality Release Ladder
- Draft: examples and rubric still being developed.
- Calibrated: multiple reviewers can apply the standard consistently enough.
- Pilot: used on representative live or shadow cases.
- Production: controls real workflow acceptance.
- Scaled: used across teams or high volume with monitoring.
- Maintained: versioned, reviewed and updated from failure data.
Not every rubric needs to reach the scaled stage. The release level should match workflow consequence and repetition.
The Quality Demotion Rule
If reviewer disagreement rises, source quality falls or critical failures escape, move the workflow back to deeper human review while the quality system is repaired.
The Quality Retirement Rule
Retire rubrics and examples when the task disappears, the receiver changes fundamentally or a better canonical quality system replaces them.
The Quality Portfolio
Large organisations may have many rubrics. Group them by reusable pattern—evidence, communication, review, decision support, code, knowledge, agent action—then layer domain-specific criteria on top.
This balances reuse with local meaning.
The Quality Registry
- Workflow
- Rubric owner
- Current version
- Mandatory gates
- Examples
- Last calibration
- Production status
- Critical failure types
- Next review
A simple registry keeps quality standards visible and prevents competing versions.
The Quality Owner Triangle
Process owner defines what outcome matters. Domain expert defines quality. Technical owner integrates evaluation. A receiver representative can validate usefulness.
This shared ownership prevents the rubric from becoming either purely technical or purely subjective.
The Quality Change Request
A change request should cite real cases, explain which criterion failed to capture the distinction and describe expected impact on review. This creates disciplined improvement rather than preference-driven editing.
The Quality Maintenance Cadence
High-impact or fast-changing workflows may need regular calibration. Stable low-risk workflows can update only when material triggers occur.
Cadence should follow drift risk rather than a universal calendar.
The Quality Review Meeting
A short periodic review can examine top failure dimensions, reviewer disagreement, receiver complaints, new edge cases and whether any criteria should be removed.
The objective is to make the quality system smaller and sharper over time.
The Quality Evidence Pack
- Current rubric
- Definitions and severity
- Canonical examples
- Boundary examples
- Should-stop cases
- Reviewer calibration results
- Representative live failures
- Receiver feedback
- Change history
- Outcome metrics
The pack gives future reviewers and model evaluators a stable reference point.
The Quality Transfer Pack
When another department wants to reuse the system, provide the mechanism: task purpose, rubric definitions, example rationale, failure boundaries and which criteria are domain-specific.
Do not assume the same score means the same thing across different consequence levels.
The Quality Human-Override Rule
Authorised humans may override a rubric result when context justifies it, especially in novel cases. High-impact workflows should record the reason so the exception can teach the system.
Human override is not evidence that the rubric failed; repeated similar overrides are.
The Quality No-Override Rule
Some deterministic or policy gates should not be overridden casually. A missing required approval, invalid financial control or prohibited data exposure may require a formal exception process.
The Quality Stop Rule
If a case does not fit the rubric or examples, stop and escalate rather than manufacturing a score. Novelty is an operational state.
The Quality Learning Rule
Repeated corrections should change one of four layers: source, instruction, structure or constraint. The rubric should help identify which layer needs repair.
The Quality Simplicity Rule
A quality system should reduce ambiguity faster than it adds review burden. If reviewers need a training course simply to use a low-risk rubric, simplify it.
The Quality Final Checklist
- Task and receiver are clear.
- Examples reflect current real work.
- Strong, weak, boundary and should-stop cases exist.
- Rubric dimensions are distinct.
- Mandatory failures cannot be averaged away.
- Evidence expectations are explicit.
- Reviewers are calibrated.
- Receiver usefulness is measured.
- Rubric changes are versioned.
- Outcome metrics remain connected to quality.
The Final Example-and-Rubric Standard
A mature quality system makes good work recognisable before release, makes bad work diagnosable after failure and makes the standard teachable to people who did not invent it.
That is why examples and rubrics matter for workplace Super Intelligence: they turn tacit expectations into shared quality infrastructure while leaving room for legitimate human judgment.
The Quality Production Gate
Before examples and rubrics become the formal release gate for a workplace SI workflow, confirm that the standard has been tested on real cases, reviewers can apply it consistently enough, and mandatory failures align with the actual risk of the process. A rubric should not become powerful simply because it looks systematic.
The production gate should also include receiver evidence. If downstream users routinely correct outputs that reviewers rated highly, the quality model is incomplete. Quality exists in the workflow, not only inside the rubric.
The Quality Maintenance Threshold
Once a rubric governs many cases, maintenance becomes an operating responsibility. Review failure patterns, disagreement, source changes, new edge cases and model updates. Remove criteria that no longer matter and add boundaries only when real evidence justifies them.
The Quality Transfer Threshold
A rubric can be transferred to another team only when the same quality concept, receiver and consequence still apply. Evidence and clarity may transfer broadly; approval, fairness, legal interpretation and professional standards often require local redesign.
The Final Quality Threshold
Examples and rubrics have become organisational infrastructure when people can explain what good looks like, why it is good, where the boundary lies and what should happen when the case falls outside the standard.
At that point, the quality layer can support model evaluation, human review, training, workflow promotion and incident learning without pretending that every professional judgment is reducible to one score.
The Quality Floor
The minimum workplace standard is that reviewers can distinguish acceptable output from unsafe, unsupported or unusable output without relying on the original prompt author. The standard should remain understandable after a change of user and should survive ordinary variation in cases.
A small number of well-chosen examples and clearly defined rubric gates usually outperform a sprawling quality manual. Precision comes from meaningful distinctions, not from the number of criteria.
The Quality Operating Rule
Show the standard, explain the standard, test the standard, and keep the standard connected to real outcomes. Examples make quality concrete. Rubrics make it transferable. Receiver and outcome evidence make sure it remains useful.
When these layers stay aligned, Super Intelligence can be reviewed and improved systematically without turning professional judgment into false mathematical certainty.
The Quality Acceptance Threshold
Before the workflow accepts an SI output, mandatory gates should be satisfied and the remaining rubric dimensions should meet the level appropriate to the task. Internal brainstorming can tolerate wider variation; customer commitments, professional advice and consequential actions require a much stronger threshold.
The acceptance threshold should be written before review begins so the standard is not quietly changed to fit a preferred output.
The Quality Final Rule
Examples and rubrics work when they reduce ambiguity without hiding uncertainty. They should help people and Super Intelligence recognise the same operating standard, while leaving room for escalation when a case does not fit.
The final workplace test is whether a new reviewer can use the current rubric and examples to reach an acceptable decision without private coaching from the system’s original designer. If that independence is missing, the quality standard still lives partly in tacit memory rather than in the organisation.
Once the standard survives that test, examples and rubrics can support scaling, training, model changes and workflow automation with a much stronger shared definition of quality.
The practical floor is therefore simple: quality must be visible enough to teach, consistent enough to review, and connected closely enough to real outcomes that the organisation can tell whether Super Intelligence is genuinely improving the work.