The Gold Standard of Universities | Reading Graduate Outcomes When Responses Are Missing

Chapter index

Imagine a graduating class as a set of individual records. A survey invitation reaches some or all of those people. Some answer. Some provide the particular outcome needed for the calculation. Some meet the stated criterion. Each step can produce a different denominator, and the final percentage is meaningful only when its denominator remains attached.

Start with the target population: the people about whom you want to make a statement. Perhaps it is all eligible graduates of one programme in a particular year. “Everyone who studied here” is wider. “Everyone who answered this question” is narrower. Neither should quietly replace the target halfway through a report.

Next define the outcome and reference time. Working at one observation point, ever having worked after graduation and being in a specified category of work are different outcomes. A person can satisfy one and not another. Our exercise uses a single yes-or-no criterion at one stated time so that the missing-data problem is visible without importing a real survey’s detailed classifications.

Separate eligibility from contactability. A graduate with an outdated email address might still belong to the target population. Removing that person from the denominator because contacting them was difficult would change the population being described. If an official survey applies exclusions, read its definitions and use them consistently rather than inventing a convenient local rule.

Write a plain-language measurement sentence: “Among these eligible graduates, we want the proportion meeting this criterion at this reference time.” Then record how the observed data connect to it. That sentence is the anchor for every later calculation. It prevents a correct division from acquiring a claim about people whose outcomes were never established.

For a family comparing courses, this first step is useful even without statistical training. Ask who is included and when their outcome was observed. If those answers are missing, pause the numerical comparison. A precise question to the provider is more useful than assuming two attractive percentages were constructed in the same way.

Contents · Next chapter

Fictional Harbour Programme has 200 eligible graduates. There are valid outcome answers for 120. Of those respondents, 108 meet the criterion and twelve do not. The respondent outcome rate is 108 divided by 120, or 90%. The valid-outcome response fraction is 120 divided by 200, or 60%.

Those numbers answer different questions. Ninety per cent describes outcomes within the observed group. Sixty per cent describes how much of the eligible group has a valid outcome answer. Multiplying them gives 54%, which is 108 divided by 200: the fraction of all eligible graduates known through this exercise to meet the criterion.

That 54% is not automatically the whole-cohort rate. It is a confirmed fraction under our assumptions that the eligible list and recorded answers are accurate. Eighty people have unknown outcomes. Some, all or none of them could meet the criterion. The observed records alone do not tell us which possibility is correct.

Notice how the word “known” changes the final statement. “At least 54% are known to meet the criterion” is different from “only 54% meet it”. The first preserves uncertainty about the eighty missing outcomes. The second turns every missing answer into a negative outcome without evidence.

Similarly, the twelve observed negative outcomes are real information in this fictional dataset. You cannot obtain a 100% whole-cohort rate by assuming all missing people succeeded; the twelve observed negatives remain. Keeping both kinds of observation visible will give the upper bound in the next chapter.

A useful checking habit is to reconstruct the counts in both directions. A reported 90% among 120 corresponds to 108 people. With rounded percentages in real publications, reconstruction may not be exact. Use published counts and rounding notes when available. Do not manufacture precise headcounts from a rounded display and then present them as original records.

Contents · Previous chapter · Next chapter

Let x be the number of the eighty nonrespondents who meet the criterion. The whole-cohort rate is then (108 + x) / 200. The observed records constrain x to lie between zero and eighty. They do not choose one value within that range.

If none of the missing graduates meets the criterion, the rate is 108 / 200 = 54%. If all eighty meet it, the rate is 188 / 200 = 94%. The data-compatible range is therefore 54% to 94%, given the assumptions of this exercise. These are logical bounds, not a confidence interval and not a forecast of the most likely rate.

The range is wide because 40% of the target group has an unknown outcome. Its width is exactly eighty divided by 200, or forty percentage points. Every missing record can change the numerator by at most one while the target denominator remains fixed. That is the complete reason for the width; no elaborate statistical machinery is needed.

This exercise does not assert that either extreme is realistic. It shows what the observed data alone rule out. If someone proposes a narrower range, ask which additional information or assumption justifies it. They might have follow-up evidence, administrative records with a compatible definition or an explicit sensitivity assumption. Those can be useful, but they are additions to the original respondent count.

An assumption that nonrespondents have exactly the same 90% outcome rate would imply seventy-two positives among eighty missing people. The whole rate would then be 180 / 200 = 90%. The arithmetic is correct. The equality of the two groups is the unverified step in this fictional case.

This is a valuable distinction for students: uncertainty can concern a missing input rather than a faulty formula. Once x is supplied, the calculation is straightforward. The intellectual work is deciding what supports x. A polished graph cannot make an unsupported value more informative, and additional decimal places cannot compensate for eighty unknown outcomes.

Contents · Previous chapter · Next chapter

Response rate concerns how many eligible people supplied an answer under the stated definition. Nonresponse bias concerns whether the resulting estimate systematically differs from the target quantity because of the response process. Coverage and representativeness are connected questions, but they are not identical.

Consider two hypothetical response mechanisms. In the first, respondents and nonrespondents happen to have the same outcome proportion. The respondent rate matches the whole cohort despite incomplete coverage. In the second, graduates meeting the criterion are much more likely to respond. The respondent rate then overstates the whole-cohort rate. The same response fraction could accompany either pattern.

The direction could also reverse. People experiencing a difficulty might be particularly motivated to answer a survey, while others ignore it. You cannot infer the sign of bias from a story about who “probably” answers. Treat such stories as hypotheses requiring evidence, rather than explanations already established by the missingness itself.

High response can narrow the maximum possible effect of missing outcomes in a bounded binary exercise. Suppose 190 of 200 graduates answer. Only ten unknown records remain, so the logical whole-cohort range has a width of five percentage points. That is a useful mathematical property. It still does not prove that the remaining ten have the same outcome distribution as respondents.

Nor does a low response fraction automatically make every published statistic unusable. Assess the actual survey design, validation, available auxiliary information and the decision being made. A measure may be informative at one aggregation level while too uncertain for a small programme comparison.

HESA’s methodological discussion of Graduate Outcomes examines evidence about nonresponse rather than assuming that a response percentage alone establishes bias. Its current data-analysis page reports no evidence of substantial nonresponse bias in the survey data examined there. That is a finding about its analysis, not a guarantee transferable to our invented programmes or every future survey.

The practical habit is balanced: retain the statistic, identify its scope and ask what the producer has done to assess the missing part. Neither automatic trust nor automatic rejection helps you understand the evidence.

Contents · Previous chapter · Next chapter

Now introduce fictional Ridge Programme. It also has 200 eligible graduates. It has valid answers from 160, of whom 136 meet the criterion. Its respondent rate is 85%, below Harbour’s 90%, while its valid-outcome response fraction is 80%, above Harbour’s 60%.

Ridge has forty unknown outcomes. Its lower bound is 136 / 200 = 68%. Its upper bound is 176 / 200 = 88%. Harbour’s range remains 54% to 94%. These intervals overlap substantially. The respondent ordering alone does not establish the ordering of the two whole cohorts.

Construct two possible completions of the missing records. In Completion A, all missing Harbour graduates meet the criterion and none of Ridge’s missing graduates does. Harbour reaches 94% and Ridge remains at 68%. In Completion B, none of Harbour’s missing graduates meets it and all of Ridge’s do. Harbour is at 54% and Ridge reaches 88%.

Both completions preserve every observed answer. They produce opposite whole-cohort rankings. Their purpose is to demonstrate that the observed data do not identify the ranking; they are not claims about plausible behaviour in an actual survey.

You can make less extreme examples too. If 40% of Harbour’s nonrespondents meet the criterion, its total becomes (108 + 32) / 200 = 70%. If 70% of Ridge’s nonrespondents meet it, its total becomes (136 + 28) / 200 = 82%. The five-point respondent advantage now accompanies a twelve-point whole-cohort disadvantage.

Do not report those assumed whole-cohort rates as discoveries. Label them as scenarios and keep the assumptions beside the values. A reader should be able to distinguish an observed count, a calculated bound and a sensitivity scenario without looking through a footnote. That separation is the central communication standard of this example.

Contents · Previous chapter · Next chapter

Sensitivity analysis becomes more useful when it answers a precise question: how different would the missing outcomes need to be for the comparison to change? We can derive that condition directly instead of selecting a few convenient scenarios.

Let a be Harbour’s unknown-group outcome proportion and b be Ridge’s, each between zero and one. Harbour’s whole rate is 0.54 + 0.40a. Ridge’s is 0.68 + 0.20b. Subtracting Ridge from Harbour gives a difference of -0.14 + 0.40a – 0.20b.

Harbour leads when 0.40a – 0.20b is greater than 0.14. Rearranging gives a > 0.35 + 0.5b. The equality is the tipping boundary, under the exact fictional counts and common outcome definition. It is not an empirical law of university surveys.

If Ridge’s missing-group proportion is 60%, Harbour needs more than 65% among its missing group to lead. At exactly 65%, Harbour has 108 + 52 = 160 positives and Ridge has 136 + 24 = 160, so both are at 80%. This count check makes the algebra inspectable.

Suppose, instead, someone assumes the same missing-group rate q for both programmes. Then the difference becomes -0.14 + 0.20q. The programmes tie when q = 0.70. Harbour leads if that common assumed rate exceeds 70%; Ridge leads if it is below 70%. Even the assumption “both missing groups are alike” does not automatically preserve the respondent ordering.

For an accessible classroom version, replace the formula with a table of q values: zero, one-half, seven-tenths and one. Calculate the two totals each time and notice where they cross. Introduce the algebra only after the learner can explain which people each term represents.

The useful decision is whether credible additional evidence places the missing-group rates on one side of the boundary. If it does not, retain the uncertainty. A sensitivity calculation clarifies which assumption matters; it cannot certify the assumption merely because the equation is neat.

Contents · Previous chapter · Next chapter

Survey producers may have information about all eligible graduates, such as programme, study mode or other appropriately available characteristics. Those records can help compare respondent composition with the target population. They may also support an adjustment under a stated method.

Consider a separate fictional cohort with two groups of 100 graduates each. In Group A, eighty respond and sixty-four meet the criterion: 80%. In Group B, twenty respond and ten meet it: 50%. The pooled respondent rate is seventy-four out of 100, or 74%. Respondents disproportionately come from the higher-rate group.

Giving the groups their actual equal population weight produces 0.5 times 80% plus 0.5 times 50%, or 65%. This is an adjusted estimate under the assumption that each group’s respondents adequately represent its missing members for the outcome. It is not the confirmed whole-cohort rate.

Why does the assumption matter? Within Group A, perhaps the twenty missing people have a much lower outcome rate than its eighty respondents. Weighting the observed 80% up to represent all 100 would not reveal that hidden difference. Matching the known group totals repairs composition on those group labels; it does not establish equality on every unobserved outcome relationship.

Weights can also make individual responses influential when a target group has few respondents. If a group has no observed outcomes, a simple within-group weighting scheme has no observed rate to expand. A real producer needs an appropriate documented method and may restrict publication. Do not create a value by borrowing another group’s result without stating the modelling assumption.

The existing university article on cohort-mix reversal owns the broader problem of comparing mixtures. This article adds the missing-outcome boundary: even correct composition weights do not make unknown outcomes observed. Read the two together when a university comparison involves both different programme mixes and different response patterns.

For users of published data, the next question is straightforward: are these figures weighted, and what does the methodology say the weights address? Avoid assuming that every official survey uses weighting or that applying weights always improves a result. Follow the producer’s actual evidence and method.

Contents · Previous chapter · Next chapter

A person can complete a survey while leaving one question unanswered. That is different from not responding to the survey at all. For a particular outcome calculation, both processes may reduce the number of usable answers, but the reporting definitions can separate them.

Imagine 150 of the 200 eligible graduates return something. Thirty omit the outcome item. The outcome table still contains 120 valid answers, but the overall survey participation fraction is 75%, while valid-outcome coverage is 60%. Reporting only the 75% figure beside the outcome percentage can give a reader a mistaken impression of how many outcome records exist.

Now add eligibility rules for a particular question. Perhaps an earnings item is asked only of people in a defined employment category. The denominator is then narrower by design. An unanswered earnings item should not be confused with a person correctly excluded from that item under its rules. Read the questionnaire and publication notes before calling all blanks “missing salaries”.

Keep categories mutually clear: eligible for the survey, responded to the survey, eligible for the item, supplied a valid item answer and met the outcome definition. A simple flow of counts can expose an inconsistency that a polished percentage table hides. The count should never grow between a subset and its parent unless the categories are defined differently.

Outcomes collected at different dates create another issue. A late respondent describing a later situation is not automatically comparable with an early respondent describing the intended reference period. The producer’s collection and reference-time rules matter. A classroom exercise should hold timing fixed before adding that complication.

For an actual request to a university, ask for the denominator used for the particular published measure. You need not demand personal records. An aggregate count and a clear methodology may answer the question. Respect confidentiality and small-group disclosure restrictions while seeking enough information to interpret the published result.

This distinction also prevents unfair criticism. A missing value can be a deliberate suppression, an inapplicable item or an unknown response. Those have different meanings. Find the legend before diagnosing the data problem.

Contents · Previous chapter · Next chapter

Suppose Harbour follows up with twenty of its eighty original nonrespondents and obtains outcomes for all twenty. Ten meet the criterion and ten do not. The cohort now contains 118 known positives, twenty-two known negatives and sixty unknown outcomes.

The updated logical range is 118 / 200 = 59% to 178 / 200 = 89%. Compared with the original 54% to 94%, the range narrows by ten percentage points because twenty additional outcomes became known. Both ends change because the new evidence includes positives and negatives.

Can we use the follow-up group’s 50% rate for the remaining sixty? That depends on how the twenty were selected and how they came to answer. If they were simply the first people reached through an easy channel, they may differ from those still unreachable. A follow-up adds direct information about the people reached; its representativeness for the rest needs separate assessment.

Administrative evidence can also help, provided it measures the intended outcome for the appropriate people and time. A record showing one type of activity may leave other categories unseen. Absence from a particular administrative source is not automatically proof of no employment, no study or no useful activity anywhere.

Check linkage and classification too. Were records matched to the correct people? Does the administrative period align with the survey reference time? Were all eligible categories covered? A second source is valuable when its scope is understood, rather than treated as a universal replacement for missing answers.

In a classroom, keep the exercise simple: add known records first, recalculate the bounds, then discuss any extrapolation. This sequencing separates evidence gained from assumptions added. It also gives students a visible way to appreciate improvement without claiming complete certainty.

For a decision-maker, the next step may be requesting a published quality assessment rather than constructing a private correction. Official producers often investigate these issues systematically. Use their current documentation and explain its limits. Our fictional follow-up is a demonstration of reasoning, not a proposed substitute for a properly designed graduate survey.

Contents · Previous chapter · Next chapter

Try this original exercise. Elm Programme has 300 eligible graduates, 180 valid outcome answers and 144 positives. Pine Programme also has 300 eligible graduates, 240 answers and 180 positives. Use the same fictional binary criterion and reference time for both.

Find each respondent rate, each known-positive fraction and each whole-cohort logical range. Then decide whether the observed records establish which whole cohort has the higher rate. Finally assume a common outcome proportion q among the missing members of both programmes and find the value at which the programmes tie.

Start by naming quantities. Elm’s respondent rate is 144 / 180 = 80%. Pine’s is 180 / 240 = 75%. Their known-positive fractions are 48% and 60% respectively. Elm has 120 unknown outcomes, while Pine has sixty.

The bounds are therefore 48% to 88% for Elm and 60% to 80% for Pine. The intervals overlap, so the observed records alone do not establish the whole-cohort ordering. This conclusion concerns the defined fictional outcome, not institutional quality or an individual student’s likely result.

With a common missing-group rate q, Elm’s whole rate is 0.48 + 0.40q. Pine’s is 0.60 + 0.20q. Equality requires 0.20q = 0.12, giving q = 0.60. At that value, Elm has 144 + 72 = 216 positives and Pine has 180 + 36 = 216. Both are at 72%.

Now change only Elm’s response count to 270 and its positives to 216. Its respondent rate remains 80%, but its bounds become 72% to 82%. Notice what changed: response coverage and the extent of uncertainty, even though the headline respondent rate did not.

Explain that change aloud. If you can calculate all six percentages but cannot identify which people each denominator represents, return to Chapters 1 and 2. If the counts are clear but the tipping equation is difficult, use a scenario table first. Repair the unstable connection rather than repeat the entire exercise without a purpose.

Contents · Previous chapter · Next chapter

Two uncertainties can appear together. The first concerns the fact that some outcomes are not observed. The second concerns what a statistical procedure estimates from a dataset under a model or sampling design. A result can have a small reported standard error and still depend on an important assumption about missing people.

Imagine a survey invites every eligible graduate rather than drawing a sample of the list. It is tempting to say that there can be no uncertainty because everyone was invited. But invitations are not outcomes. If only part of the cohort supplies usable answers, the missing-outcome question remains even though there was no initial sampling of invitees.

In the opposite direction, suppose every member of a genuinely selected sample responds. There is no nonresponse within that sample, while estimating a larger population can still involve sampling uncertainty. These are different sources of uncertainty and should not be merged into one reassuring label such as “large study”.

Our logical bounds do not require estimating a response probability. They use only the recorded outcomes, target count and binary nature of the criterion. That makes their meaning unusually transparent. Their weakness is also clear: they can be too wide to support a useful comparison without additional information.

A model-based estimate can be narrower because it introduces assumptions connecting respondents and nonrespondents. A confidence interval around that estimate typically reflects the uncertainties represented by the specified procedure. It should not be read as covering every possible error in eligibility, definitions, linkage, reporting or the response model unless the method explicitly addresses those issues.

For a public report, read the quality documentation rather than attempting to infer the whole method from the interval’s width. Ask which sources of uncertainty are quantified, which are assessed through separate investigations and which remain limitations. A responsible producer may provide several forms of evidence because one interval cannot carry the complete quality account.

There is also a distinction between descriptive inference and causal inference. Even if every graduate’s outcome were known without error, the comparison would not prove that one programme caused a higher outcome rate. Entry profile, prior experience, preferences, location and other factors could differ. Resolving nonresponse does not remove every other inference problem.

Use the fictional example to practise a careful handover. One analyst says, “The respondent rate is calculated correctly, but the whole-cohort comparison remains sensitive to missing outcomes.” Another says, “The published procedure reports a narrow interval under its model, and the producer has separately investigated nonresponse.” These are compatible kinds of evidence, not opposing declarations that statistics are either reliable or worthless.

The useful next action depends on the decision. For choosing tomorrow’s classroom example, the observed counts may be enough. For a consequential programme comparison, current methodological expertise and producer documentation may be necessary. Match the strength of the conclusion to the evidence and the consequence of being wrong.

Contents · Previous chapter · Next chapter

A defensible short account of Harbour’s original data might say: “There were 120 valid outcome responses among 200 eligible graduates. Of respondents, 108, or 90%, met the criterion. Eighty outcomes were unknown. Without additional assumptions, the whole-cohort proportion is bounded between 54% and 94%.” It describes the observation and the uncertainty in one place.

Would you put that full range in every public headline? Communication depends on the measure and official methodology. The lesson is to make the evidence available and avoid silently expanding a respondent statement into a claim about everyone. A reader should be able to find definitions, counts and quality notes without reconstructing them from promotional language.

Does a 90% respondent rate mean 90% of graduates succeeded?

It means 90% of the defined respondent group met the stated criterion. Extending that result requires evidence or assumptions about the target population. Also check what “succeeded” means; an outcome category is not a complete judgement of a graduate’s life.

Are logical bounds the same as a confidence interval?

No. Our bounds enumerate the extremes compatible with the observed binary outcomes and unknown records. A confidence interval arises from a statistical procedure and its assumptions. A narrow interval for respondent sampling variability does not automatically account for nonresponse bias.

Should we count everyone missing as a negative outcome?

Only if the defined measure explicitly treats them that way, in which case describe that measure accurately. In this exercise it gives a lower bound, not an established outcome for each missing person. Do not turn unknown status into an invented personal result.

Can a higher response rate prove the better university?

No. It concerns survey coverage under a definition. University choice also involves curriculum, assessment, support, practical access and the learner’s goals. Even a complete outcome census would not isolate the causal contribution of the institution.

What is a useful first question for a provider?

Ask: “What are the eligible count, valid-outcome count, reference time and quality notes for this percentage?” That is precise enough to answer and does not require disclosure of individual graduates’ information. Then use the connected university and survey guides for the next question the evidence raises.

Contents · Previous chapter · Next chapter

Use the routes below for the specific question you want to pursue. The subject root keeps the wider map; the specialist guides develop different evidence jobs. A connected reading sequence is most useful when those responsibilities remain clear.

Understand university outcomes

Why English? | Reading a Graduate Employment Survey. Begin here for the language and definitions of published graduate employment findings. It helps keep an employment category separate from an informal claim about success.

Graduate Tracer Studies and Labour-Market Feedback. Follow the wider institutional purpose of learning what happens after students leave. Our missing-response calculation addresses one evidence problem within that larger feedback system.

Compare populations

When Cohort Mix Reverses an Outcome Comparison. Use this worked companion when group composition changes a pooled result. Read it alongside the distinction here between observed composition and unknown outcomes.

Choose a course

International Courses and University Fit Hub. Return to curriculum, learning tasks and practical fit when a survey number has become the whole conversation. A statistical comparison is one input to a decision.

Courses for Statistics. Explore the study route if investigating assumptions, uncertainty and data quality interests you. Check current programme conditions directly with institutions before making an application decision.

Authoritative sources and scope

The following primary sources support the identified method or institutional context. Source routes were reviewed on 21 October 2026. All fictional counts, probability cases, timelines and learner exercises in this guide are original teaching constructions. They are not findings about an actual institution, service or wildlife site.

HESA: Graduate Outcomes data analysis. Primary producer documentation about its analysis and nonresponse investigations. Its findings apply to the assessed data; our invented cohorts do not demonstrate bias in that survey.

HESA: Sampling error and non-response error. Definitions and quality discussion, including unit and item nonresponse. Consult the relevant cohort’s current documentation for an actual published result.

HESA: Graduate Outcomes user guide. Start here for definitions, data quality and supporting methodology. The logical bounds and tipping examples in this article are original arithmetic constructions, not HESA estimates.

For the wider learning purpose, continue with Learning That Transfers into Life. For related systems and subjects, use How X Works. Choose one new example and explain which relationship carries over, which assumptions change and what evidence you would need before applying the conclusion.

Contents · Previous chapter · Continue into Universities