EDUKATE SINGAPORE · GOLD STANDARD PRACTICAL CASE STUDIES · CASE 138
One hundred and fifty survey replies do not necessarily represent one hundred and fifty households.
A fictional community study-room survey receives 150 forms from only 100 distinct households. A headline about majority household support changes when the data are examined using the actual unit of analysis. This Gold Standard case explains duplication, survey design, representation and honest arithmetic.
All households, opinions, survey conditions and numbers are fictional. AAPOR and US Census guidance provide established survey-research principles, not statistical validation of this invented study.
Chapter index
- 1. A majority headline built from repeated households
- 2. The unit of the question is part of the promise
- 3. The invented arithmetic makes the distortion visible
- 4. AAPOR explains why respondents and units need careful design
- 5. Repeated responses do not automatically mean dishonesty
- 6. A household may contain genuinely different opinions
- 7. The first public correction belongs before the planning vote
- 8. Improve the invitation and response controls together
- 9. Report what the results actually support
- 10. Practice: 150 forms or 100 households?
1. A majority headline built from repeated households
A fictional neighbourhood learning association surveys residents about a proposed shared study room. Its questionnaire asks whether households support the proposal, and volunteers distribute invitations online and at community events. The response system collects 150 completed forms. The committee’s publicity slide says 53% of responding households support the plan and recommends that the association announce a community majority.
A data reviewer notices that some homes completed the form more than once through different invitations. Under the fictional verification method, the 150 records came from only 100 distinct responding households. Some homes supplied two or four forms, while others submitted one. The headline calculated a percentage of replies but named the resulting figure as a percentage of households.
The Gold Standard question is not whether enthusiastic residents should be ignored. Their opinions matter. It is whether the same household was given an unintended greater influence in a claim that purported to represent one response per household, and whether the association can responsibly infer anything about all households in the neighbourhood from its collection.
2. The unit of the question is part of the promise
The organisation’s stated research question is household support for a study-room proposal. That makes a household the intended unit for the reported household-level statistic. A different project asking what individual adults think might legitimately collect several responses from one household, but it would need a design and analysis appropriate to individuals rather than casually switching denominators.
The records include S1, the survey invitation and eligibility definition; S2, the response logs; S3, an authorised household linkage method; S4, the published yes/no question; S5, the analysis rules; and S6, the final reported count. Each needs to be preserved and reviewed with proportionate privacy safeguards.
Counting completed form submissions establishes how many forms were received. It does not automatically establish the number of households represented, the number of unique people or the distribution of opinions in the wider neighbourhood. Those quantities require distinct identities, definitions and suitable study design.
GS-R01 specifies the population and purpose. GS-R02 defines the eligible response unit, and GS-R08 ensures that the label on the percentage matches the unit the data actually represent.
3. The invented arithmetic makes the distortion visible
For this fictional exercise, suppose seventy households submit one form each, twenty households submit two forms each, and ten households submit four forms each. That gives 70 + 40 + 40 = 150 form responses from 70 + 20 + 10 = 100 distinct responding households. The different submission counts are intentionally exaggerated for teaching clarity.
Suppose all twenty double-responding homes and ten quadruple-responding homes support the study room, while the seventy single-responding homes oppose it. Under these artificial assumptions, there are 40 + 40 = 80 supportive forms, and 80/150 = 53.3% of recorded form responses. But there are only thirty supportive households among the 100 verified responding households, or 30%.
These two percentages measure different things. The arithmetic supports the 53.3% figure among forms and the 30% figure among these responding households under the stated consistency assumptions. Neither establishes that 30% or 53% of every household in the neighbourhood supports the proposal. The recruited households may not be representative, and some eligible homes may not have responded.
The Gold Standard prevents a form-completion number from being silently promoted into an unbiased household opinion estimate.
4. AAPOR explains why respondents and units need careful design
The American Association for Public Opinion Research’s Best Practices for Survey Research explains the importance of defining the target population, choosing a suitable sample and reporting survey design, weighting and limitations rather than relying on attractive percentages alone.
AAPOR’s Transparency Initiative identifies information such as the sampling frame, response counts, how weighting was performed and what assumptions influence reported results. Its Disclosure Standards describe the importance of explaining duplicate-response and data-quality procedures where relevant.
The US Census Bureau’s survey response-rate standards likewise differentiate the selected population unit, completed response and sampling weights under specific designs. None of these resources certifies this hypothetical neighbourhood study or supplies a universal rule that one individual always speaks for every other resident of a household.
The management principle is that the study must choose what it is measuring and keep that definition stable from invitation through collection, analysis and communication.
5. Repeated responses do not automatically mean dishonesty
The committee initially suggests that the multi-form households attempted to manipulate the result. The available records do not justify that accusation. Some received separate online and paper invitations; others had several adults who each thought the survey asked for their personal view. The invitation may have failed to explain the intended response rule.
An enthusiastic participant could submit once through the website and again while speaking to a volunteer without recognising they were duplicating a household response. The organisation must examine its distribution and wording before treating the outcome as intentional misconduct. A flawed survey design can invite repeated entries without anybody acting deceptively.
A responsible review uses a permitted and proportionate method to reconcile records. It should not publicly identify households or reveal addresses and opinions to other residents. Nor should it arbitrarily keep whichever duplicate response best supports management’s preferred conclusion.
GS-R03 requires ownership of the analysis decision; GS-R09 requires a transparent correction without unsupported allegations about participants.
6. A household may contain genuinely different opinions
The simplified numerical example assumes each household’s multiple forms express the same yes-or-no preference. Real homes may contain people who disagree. If the research question is about individual preferences, those responses can be valuable and should not be discarded merely because they share an address. If the research question is one household decision, the collection method needs a defined, legitimate approach to obtaining that information.
An analyst cannot retrospectively turn four conflicting individual responses into one household yes or no without choosing a rule whose justification may be weak. Possible designs might use one selected eligible respondent, an agreed household representative or another explicit research approach. The relevant method must be decided for the actual question and protected from post-result manipulation.
The organisation should not assume that every postal address is one household or that every household has one contact address. Shared housing and other living arrangements can complicate an apparently simple deduplication operation. A reliable household definition is part of the study design, not something inferred from the neatness of a spreadsheet column.
The Gold Standard’s purpose is methodological fairness: give respondents a survey that says what is expected and analyse their answers according to an appropriate, stable rule.
7. The first public correction belongs before the planning vote
The association pauses the claim that a majority of neighbourhood households support the proposed study room. It may accurately state that it received 150 forms from 100 distinct responding households under its current verification method and that the first 53% figure was a form-level calculation, not the intended household-level measure.
The corrected summary should explain the analysis limitation without claiming that all residents opposed the idea. A response pattern among 100 participating households may still be affected by who heard about the survey, who could respond and who was motivated. The association needs an appropriate design before treating the respondents as a statistical representation of the wider neighbourhood.
The committee can proceed with legitimate programme planning under its actual governance arrangements while acknowledging that the specific majority claim is unsupported. It should not use the corrected 30% from the artificial example as if it described the real community without appropriate evidence.
GS-R09 requires the uncertainty to be presented honestly. A flawed survey is not repaired merely by picking a lower number and calling it accurate.
8. Improve the invitation and response controls together
The survey owner defines the target population and response unit before distributing the next form. The invitation explains whether it seeks one response per household or separate views from individual residents, what eligibility means and how participants may ask questions. The response method can support the chosen design without collecting unnecessary identifying details.
A controlled duplicate-management procedure is specified before results are inspected. Where the collection mode includes both paper and online forms, the system must reconcile them according to the published method. Duplicate or uncertain states become owned exceptions rather than silently contributing multiple weights to a household-level statistic.
The association also considers accessibility: the response route should not exclude people merely because they do not use an online form, but allowing paper responses requires a workable intake and reconciliation plan. The solution is not to disable every alternate channel for administrative convenience.
GS-R07 links collection channels; GS-R11 asks whether the process is sustainable; GS-R12 governs adjustments when the method or eligibility definition changes.
9. Report what the results actually support
The committee’s revised dashboard distinguishes invited units, eligible contacted units where known, unique responding units, form submissions, incomplete records and appropriate analysis weights or exclusions. It explains when percentages refer to people, forms or households.
Data quality is not an excuse to manufacture a false margin of error. A convenience collection cannot automatically use probability-sampling precision claims without a defensible design. AAPOR’s transparency resources call for explaining the method and any limitations relevant to the inference.
An analysis could legitimately present several questions: how many individuals support the proposal, how many participating households support it, and whether that pattern might reflect the neighbourhood population. Those are different questions with different data needs. A survey should not treat an answer to one as a shortcut to all three.
The Gold Standard values a report that helps the association make a careful decision rather than one that provides a politically comfortable number without a clear denominator.
10. Practice: 150 forms or 100 households?
Exercise. Seventy households submit one negative reply each, twenty submit two positive replies each and ten submit four positive replies each. The organiser says 53% of households support a community study room. Calculate the form-level and distinct-household-level figures under the stated simplified assumptions, and describe what remains unknown about the whole neighbourhood.
One reasoned answer is: ‘Eighty of 150 forms are positive, approximately 53.3% of form submissions. Thirty of the 100 distinct responding households are positive, or 30% of those households, under the artificial consistency assumptions. Neither figure can be generalised automatically to all neighbourhood households without an appropriate sampling and analysis design.’
The organisation should correct the unit of analysis, preserve the original records and test the next mixed-channel survey with repeated household invitations and conflicting responses. The Gold Standard lesson is that a correct percentage becomes misleading when the people or units behind its denominator are not the ones the headline promises.
Survey methodology and related Gold Standard studies
- AAPOR · Best Practices for Survey Research
- AAPOR · Transparency Initiative
- AAPOR · Disclosure Standards
- US Census Bureau · Survey Response and Sampling Standards
- Gold Standard · Evidence, Claims and Verification
- Gold Standard · Measurement and Performance Review
- Gold Standard · Core Requirements and Evidence
- Case 06 · Misleading Survey Response Rate
- Case 87 · Reading Programme Missing Outcomes
- Case 133 · Random Scholarship Draw Excluded Paper Applicants
- Gold Standard Casebook and Decision Practice
A result from a convenience or otherwise unverified sample must not be represented as a probability estimate of the population without proper support. GS-R01–GS-R12 is eduKate’s educational management reference, not external survey accreditation.