eduKate Singapore

VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Next useful routeCanonical hubHELP

How AI Works | Missing Data — Observation Gaps, Imputation and Robust Evaluation

—

by

eduKate Asia

A machine-learning model does not automatically know what an empty cell means. It may represent an unasked question, a faulty sensor, a delayed report, a deliberate refusal or a measurement that never existed. A responsible AI workflow first identifies why a value is absent, distinguishes missing features from missing outcomes, and checks whether the method for filling or representing gaps uses only information genuinely available at prediction time. Dropping every incomplete row or replacing every blank with zero can bias the population and give a persuasive but misleading performance score.

Start with an observation contract: what should have been recorded, when, by whom and for what decision? Then map missingness by group and time, establish a simple baseline, choose an imputation or native-missing-value method compatible with the model, and fit preprocessing only on the training fold. Test whether predictions and error rates change for the people whose data is incomplete. This guide offers a fully invented 120-record education-support case, checked arithmetic, a changed-context transfer problem, practical repair methods and honest limits. It is about contemporary AI systems, not a claim that speculative artificial superintelligence is already operating.

The wider How AI Works introduction explains the machine-learning pipeline. The data leakage chapter addresses the integrity of the evaluation split. Here the distinct question is how to reason about the absence of observations even when the split is correct. For useful statistical cautions, start with scikit-learn’s Imputation of Missing Values documentation and its common pitfalls guide.

Choose your learning route

  • New to machine learning? Read the distinction between missing, zero and not applicable, then practise the 120-record calculation.
  • Preparing a model in Python? Focus on train-only fitting, indicators, estimators that handle missing values and tests on fresh data.
  • Evaluating educational or public decisions? Follow subgroup coverage, missing labels and the risk of selective observation.
  • Teaching statistics? Try the changed-context transfer before examining the solution.

Contents and navigation

1. A blank is not a measurement

Suppose a study-support table records hours of independent revision for each learner. One pupil enters 0 because no revision took place. Another leaves the question blank because a device crashed. A third selects “not applicable” because the programme has no assigned revision time. Converting all three to zero collapses distinct states into a single numerical fact. The model may interpret technology access, programme enrolment and genuine effort as the same behaviour. Before selecting an algorithm, design a data dictionary that gives each status its own meaning.

Missing values can occur in input features, outcomes, identifiers or timestamps. Their consequences differ. A missing input measurement may be handled through imputation or a model capable of representing absence. A missing outcome can prevent supervised training or reliable evaluation. A missing identifier may make a join unsafe, while a missing timestamp can make it impossible to decide whether a feature was available when a prediction should have been made. Treat the table as a record of an observation process, not just a rectangle of floating-point numbers.

The consequence of a missing value depends on the intended decision. When estimating mean study hours, inserting a guess changes the descriptive quantity being estimated. When predicting which learners may need additional support, it may be preferable to model a missingness flag alongside available features. When sending an individual warning or allocating scarce tutoring support, uncertainty and fairness matter because an imputed number is not a personal fact. A model should not produce a false claim that a pupil actually studied for the value used internally.

Consider privacy-related absence. A person may decline to answer an optional question or may have a lawful reason for not disclosing it. That absence does not automatically justify collecting the information from unrelated sources. Feature engineering and imputation must respect the data-collection purpose and permissions. A system’s technical ability to predict a protected or private attribute is not permission to use it for an outcome. Keep the distinction between model prediction and user-authorized data processing explicit.

The first design move is to identify a trustworthy semantic layer: missing because uncollected, missing because unavailable, withheld by the user, unavailable at the prediction time, not applicable, corrupted and censored. Some conditions may require specialized modelling rather than ordinary imputation. Recording these distinctions at ingestion helps engineers detect failures, reviewers ask better questions and affected users challenge conclusions drawn from incomplete information.

2. Three mechanisms of missingness

Statistical discussions often distinguish missing completely at random (MCAR), missing at random conditional on observed information (MAR) and missing not at random (MNAR). These names describe assumptions about the mechanism generating the observed pattern, not three settings a software function can determine automatically from an empty cell. In MCAR, missingness is unrelated to the values under the specified model. In MAR, after conditioning on relevant observed variables, missingness is assumed not to depend on the unseen value itself. In MNAR, the chance that a value is absent depends on unobserved information even after the considered conditioning.

Imagine a fictional device losing a small random set of form submissions independent of learner behaviour. MCAR might be a useful modelling hypothesis, but it would need evidence about device failures and collection design. Now suppose one campus has older devices, and campus membership is recorded. Missingness may be partly explained by this observed feature; a MAR assumption may be more plausible conditional on campus and other measured factors. If a learner is less likely to report revision hours specifically when those hours are low, the missingness depends on an unobserved value and a simple MAR model may mislead.

It is important not to turn the terminology into certainty. The full missingness mechanism cannot always be identified from observed data alone, especially when MNAR alternatives fit the same observed records. Sensitivity analysis asks how conclusions would change under plausible assumptions about the absent values. This is more intellectually honest than announcing that a missing-data test has established a universal mechanism. Observed patterns can falsify some simple assumptions, but they do not disclose every unrecorded cause.

Missingness can also be structural. A school club may record a rehearsal duration only for pupils who enrolled in the theatre activity. Learners in other clubs are not “missing” the same event; the field is not applicable. Imputing an imaginary theatre rehearsal duration for them invents a measurement. Encode eligibility or structural absence separately. Otherwise a classifier may infer participation from the model’s synthetic numbers rather than from actual enrolment.

Across time, the mechanism can change. An updated reporting app may reduce missingness among people with strong internet access but leave offline participants behind. A policy change may make a question optional. An observation that is available eventually may be absent at the prediction deadline. Track collection version, effective date and prediction-time availability. A dataset may look complete months later while a genuine real-time prediction system had only partial information at the moment of use.

3. Inventory observation routes and provenance

Before imputation, list every field and its definition: unit, allowed range, collection point, source authority, time granularity, known reasons for absence and the population expected to supply it. For a sensor, distinguish a genuinely zero reading from disconnection. For forms, distinguish unanswered, refused and not applicable. For timestamps, record source time separately from ingestion time where needed. Without these definitions, an imputation routine may clean the spreadsheet while destroying its meaning.

Create missingness summaries by subgroup, time period and collection channel. One overall missing-data percentage can hide that almost all incomplete records come from a particular device, school, language group or hour. The appropriate subgroup follows a legitimate purpose and available authorization, not curiosity about a person’s identity. Aggregate counts may be suitable for oversight, while raw sensitive attributes should be protected from unnecessary use in models and reports.

Inspect whether data joins create artificial gaps. A learner record might use one ID format while a revision app uses another. The join then returns missing study hours even when the hours were recorded somewhere else. Changing those blanks to the mean conceals a data-integration defect. The right repair is to reconcile identity mappings within authorized rules, confirm whether the value exists and only then decide how to model remaining absence. Do not invent a match based solely on similar names.

Missingness should be checked against the relevant prediction horizon. If the decision is made at 09:00, a laboratory result reported at 11:00 cannot be treated as an available feature even if the completed archival dataset contains it. Preserve “as-of” snapshots and feature release times. This is related to leakage but conceptually distinct: missingness is the fact that an observation was unavailable, whereas leakage occurs when future or held-out information improperly influences a model. The two can interact if the engineer fills a gap using information obtained later.

Plan for an observation route to fail. A data collector may stop, a form field may be renamed, a schema revision may mark a number as a string, or one API may quietly omit optional fields. Record collection-health signals and compare them with historical baselines. A sharp change in the fraction of missing values often deserves investigation before any statistical treatment. A model should not silently continue producing confident decisions when the underlying observation process has broken.

4. Establish baselines before imputation

The simplest baseline is not always a mean. It can be a rule that withholds a decision when too much critical information is unavailable. For remaining numeric features, compare a training-fold median, mean, known sentinel plus a missingness indicator, and an estimator with built-in missing-value handling when supported. For categorical features, a distinct “missing” category or training-fold most-frequent imputation may be evaluated. Choice depends on the feature semantics, distribution and consequence of a wrong prediction.

A zero-filled baseline has an easily checked failure mode: zero may have genuine mathematical meaning. If a learner reported no revision, zero is observed behaviour; if the form was never sent, zero is an assertion the data cannot justify. A mean-filled baseline also has limits: it reduces observed variance in that column and may obscure systematic differences between groups. An imputed feature is a modelling device, not a recovered historical truth.

When the purpose is prediction, a simple imputer plus an explicit missingness indicator can be a sensible starting point. The official scikit-learn imputation documentation describes SimpleImputer, MissingIndicator, more elaborate approaches and estimators that natively accept missing values. Its documented guidance cautions that sophisticated reconstruction often gives diminishing predictive returns. These are general software and modelling considerations, not evidence that one algorithm always wins across all tasks.

When the purpose is estimating a population mean or causal effect, the assumptions are different. A model that predicts acceptably despite missing inputs may still produce biased descriptive estimates if records are selectively absent. Do not use predictive accuracy as a substitute for representativeness. Analysts may need weighted adjustments, multiple imputation, design-based methods or sensitivity bounds under explicit assumptions. An educational guide should say what quantity it is estimating before choosing the machinery.

Construct an honest baseline before adding complexity: count distinct cases, measure missingness, note observed denominators, fit a training-only imputer, and compare a conservative model against a no-imputation alternative where feasible. Record metrics by missingness pattern and by relevant group. If a complex method improves a single validation number while producing implausible values for a sensitive group, investigate rather than immediately ship.

5. Fit transformations only inside the training boundary

Imputation statistics are learned parameters, even when the algorithm looks simple. A mean fitted on all available data, including future validation or test rows, reveals information about the evaluation population to the training process. This can bias the reported result. Split according to the real deployment unit and time horizon before fitting preprocessing. On each training fold, fit imputation, scaling, feature selection and the estimator using only eligible training information; then apply the fitted transformation to held-out records.

A clean software pattern is an appropriate scikit-learn Pipeline or ColumnTransformer with preprocessing inside cross-validation. A preprocessing step belongs inside the fold because even unsupervised statistics such as column means can leak characteristics of test rows. Group-aware and time-aware splitting may also be necessary, as explained in the existing data leakage guide. Missing-data strategy and evaluation integrity must be designed together.

Consider repeated learners appearing in several terms. If records from the same learner cross training and evaluation, the model may exploit stable personal signals and appear stronger than it will on new learners. If the deployment target concerns future terms, use a split that reflects time as well. A good imputer cannot repair an invalid split. Conversely, a perfect split cannot make an invented measurement truthful. These are independent quality gates.

At inference time, apply the same feature schema and fitted imputation policy to incoming records. An unseen category, new absent field or changed data type should trigger a known handling route rather than a different ad hoc cleaning rule. Keep the training-time feature list, missing indicators and imputation statistics versioned with the model. Test that the runtime does not silently drop a column and shift the meaning of every later feature.

A common misconception is that using a test-set mean is harmless because it does not include labels. The test set still contains information the deployed system would not have been entitled to use when fitting its pipeline, especially under grouped or temporal evaluation. Split first, fit using training data and reserve held-out data for the intended validation purpose. This makes the claim being tested match the process that would actually run for new users.

6. Indicators and models that accept missing values

A missingness indicator is a binary feature identifying that a value was absent according to a stated rule. It can help a model distinguish a genuinely observed number from a substitute. For example, two records may both carry an imputed revision-hour value of five, but one has an indicator of one because the hours were absent while the other has an indicator of zero because five was observed. This is not permission to infer the reason for the absence or to discriminate against people whose records are incomplete.

scikit-learn offers MissingIndicator and an add_indicator option in several imputers. The documentation describes how to combine indicators with imputed values and how features are selected. If new patterns of missingness arise after fitting, the feature representation needs careful design; a missing-indicator column that exists only for fields absent during training may omit new failure patterns unless the pipeline was configured to handle them.

Some estimators can use missing-value patterns natively. This avoids a separate imputation stage for those models, but it does not solve absent outcome labels, reporting bias, incorrect joins or legal obligations. Test the actual estimator and version rather than assuming that every tree or neural network handles NaN safely. Native support means the algorithm accepts a representation, not that it knows why data is missing or what a learner would have reported.

For explanatory analyses, an imputation can increase apparent precision without increasing actual observation. Distinguish model-filled values from observed ones in reports, especially if people may challenge an individual prediction. A reviewer should be able to see the original absence and the modelling approach without receiving unrelated private records. Use data minimization and purpose limits throughout the feature pipeline.

An even better intervention may be to repair collection. If one campus app fails forty percent of the time, developing a clever imputer may be less valuable than fixing the device path and helping users submit information reliably. Analyse technical defects, language barriers, opt-out choices and delayed outcomes before treating absence as an unavoidable property of the world. Statistical methods should complement good observation design, not excuse indefinitely broken systems.

7. Missing outcomes are a different statistical problem

A supervised model learns a relationship between features and a known target. If the target is unknown, inserting a convenient negative class is not ordinary feature imputation. A delayed examination result, an unresolved support case and an unobserved future event are not proof that the outcome was negative. A learner who has not yet completed a programme may be censored in time, not a failure. The data team needs a definition of label availability, event time and observation window before deciding which records can take part in training or evaluation.

When labels are available selectively, checking predictive accuracy only among labelled cases can create false reassurance. Suppose the hardest cases require longer investigation; their outcomes may arrive late. The easy cases are labelled quickly and dominate the dashboard. The model appears accurate on the subset that happens to be visible, but that subset does not represent all decisions. Report label coverage, timing and relevant subgroup differences alongside any score. Do not silently treat unknown outcome as zero, success or failure.

An original missing outcome may later be resolved by a trusted source. Maintain a versioned label record and document its effective date. For time-respecting validation, use only labels available at the specified evaluation cutoff. A retrospective study can explicitly use later labels for a well-defined historical prediction exercise, but it should not pretend those labels were known to a live agent at the original decision time.

Missing outcome labels may also demand different methods from ordinary feature imputation: follow-up collection, survival-analysis techniques for censored events, sensitivity analysis, sample-selection modelling or a qualified decision to withhold an estimate. Choosing among these requires a genuine statistical design and domain expertise. An imputer that fills a missing revision-hour input is not automatically suitable for reconstructing a student’s future examination outcome.

8. A 120-record education-support laboratory

Take a fictional dataset of 120 distinct learners, sixty from Campus A and sixty from Campus B. The example records independent-revision hours during a named week. Campus A has fifty-four observed records and six missing values. Campus B has thirty-six observed records and twenty-four missing values. Every number in this case is hypothetical. We do not know the true missing values, and the observed means are summaries of the visible records, not guarantees about all learners.

Suppose the fifty-four observed Campus A records have an average of six hours, while the thirty-six observed Campus B records have an average of four hours. Their observed sums are fifty-four multiplied by six = 324 hours and thirty-six multiplied by four = 144 hours, for a combined observed total of 468 hours. Ninety learners have recorded values; thirty do not. Dividing 468 by ninety gives an observed-only mean of 5.2 hours per recorded learner. It does not establish the population mean for all 120.

The overall missingness rate is thirty divided by 120 = 25%. Campus A’s rate is six divided by sixty = 10%, while Campus B’s is twenty-four divided by sixty = 40%. That fourfold rate difference is a practical diagnostic signal. It might reflect collection devices, language accessibility, scheduling or a behavioural difference; the counts alone cannot establish the cause. Investigate the observation process and avoid assigning deficiency to learners who did not supply a value.

Consider three deliberately simple data-handling strategies. If every missing value is incorrectly treated as zero, the overall average is 468 divided by 120 = 3.9 hours. If missing values are imputed with the observed-only global mean 5.2, all 120 records average 5.2 by construction. If Campus A missing records are imputed with six and Campus B missing records with four, the resulting totals are sixty times six = 360 and sixty times four = 240, giving 600 divided by 120 = 5.0 hours. These are properties of three calculations under their assumptions, not established true revision habits.

The pooled mean and campus-specific means differ because Campus B, with a lower observed average, has more missing values. The observed-only pool weights Campus A fifty-four to thirty-six, or 60% to 40%, while the full population contains sixty from each campus, 50% to 50%. Neither estimate is unbiased merely because its arithmetic checks. The correct estimator depends on the missingness mechanism and the quantity being estimated, not on which number looks tidier.

To see how much is unknown, conduct a transparent sensitivity exercise. Suppose, only for illustration, that the thirty missing values average two hours. Then the full sum would be 468 + 30 × 2 = 528 hours, or 4.4 hours per learner. If the missing values instead average eight hours, the sum would be 468 + 30 × 8 = 708 hours, or 5.9 hours per learner. These hypothetical scenarios are not confidence intervals or empirical bounds; they illustrate that the observed 5.2-hour mean cannot pin down the unseen records.

Now imagine the organisation wants a model that predicts which learners might request additional support next month. The input features may include revision hours, but it must not label an imputed value as actual effort. Prepare a legitimate training and holdout split, for example ninety distinct learners for training and thirty for validation, before fitting any imputation statistic. Use only the training portion to fit the imputer, and evaluate against real observed outcomes in the holdout. The descriptive means calculated above from the complete teaching packet are for explanation and must not be reused as training-only fitted parameters.

A practical report would present the data dictionary, original observed-vs-missing masks, counts by campus, training and validation eligibility, imputation method and comparison to simpler baselines. It would include a caution that Campus B had many more absent records and explain whether technical collection repairs are planned. A stakeholder should be able to reproduce each calculation and distinguish observed data from model-supplied substitutes.

9. Checked denominators and arithmetic

Start by reconciling the complete count. Campus A: 54 observed + 6 missing = 60. Campus B: 36 observed + 24 missing = 60. Therefore 54 + 36 = 90 observed, 6 + 24 = 30 missing, and 90 + 30 = 120 total. The observed fraction is ninety divided by 120 = 75%. A dashboard that says “75% complete” should specify whether it means field coverage, labelled outcomes, complete individual cases or some other measure. Those are not interchangeable.

The campus comparison requires care: 40% missing at Campus B minus 10% at Campus A is a difference of thirty percentage points. The rate at Campus B is four times the rate at Campus A, because 40 ÷ 10 = 4. It would be incorrect to say Campus B has “30% more” missing data without specifying the denominator and interpretation. Neither comparison proves that campus membership causes the missingness.

To quantify the choice of filling rule, compare zero fill at 3.9 hours, campus-mean fill at 5.0 and pooled-mean fill at 5.2. The pooled versus campus difference is 0.2 hours per learner under those constructed substitutions; 0.2 hours is twelve minutes, because 0.2 × 60 = 12. A tiny numerical difference is not evidence that the approaches are interchangeable for individual cases, rare outcomes, fairness or prediction performance.

Suppose an independent validation set contains thirty learners, with twenty-six confirmed outcome labels and four labels still pending. If a model gets twenty-two of the twenty-six observed labels correct, accuracy among observed labels is 22 ÷ 26 ≈ 84.62%. The denominator is twenty-six, not thirty. Reporting 22 ÷ 30 ≈ 73.33% would implicitly treat pending labels as errors, which is unsupported. Reporting the 84.62% alone could also mislead if the four pending outcomes come from the hardest cases. The team must report both observed-label performance and four-out-of-thirty pending coverage.

There is no reason to regard the different examples as independent measurements of one real model. The 120-record hours example describes feature absence; the thirty-record validation example separately illustrates target availability. They are fictional exercises under stated assumptions. Combining their percentages into a provider quality score would be invented evidence. This distinction is as important as the arithmetic itself.

10. Diagnose failures before replacing values

Failure A: zero represents both a real observation and an absent field. The model falsely treats a device crash as no learner activity. Repair by restoring a typed missing state, checking raw ingestion records and using an appropriate model representation, then retest with real zeros and genuine missing values side by side.

Failure B: a global imputation figure is fitted before the train-test split. The held-out population influences training. Repair by fitting preprocessing separately within the eligible training folds using a Pipeline or equivalent controlled process, then apply transformations to each holdout. Recalculate evaluation results and record the change without claiming a specific improvement unless it was measured.

Failure C: a school record join fails because one system changed its learner identifier format. Filling the resulting blanks with means hides a broken integration. Repair the authorized identity mapping, reconcile record counts and review false matches before deciding which values remain genuinely missing. Do not merge learner records based only on similar names.

Failure D: only twenty-six of thirty outcomes are known, but unknown outcomes are coded as negative. The confusion matrix includes labels that have not occurred or have not been observed. Repair by distinguishing pending, censored, disputed and confirmed labels. Evaluate using stated eligibility and follow-up rules and report label-coverage limitations.

Failure E: missingness indicators are added for training fields, but a new source field becomes completely absent at deployment. The transformation silently drops or reorders columns, changing feature meanings. Repair the schema contract, test unseen missingness patterns, preserve consistent feature order and stop or route for review when the model input no longer matches supported configurations.

Failure F: one subgroup has much higher missingness but the overall imputation score looks acceptable. Inspect source collection, subgroup errors and whether the missingness pattern changes across time. Repair accessible collection paths if possible; compare conservative baselines and review high-impact decisions. A numerical fill is not a substitute for equal opportunity to provide accurate information.

11. Transfer exercise and explained solution

Change the teaching setting to a fictional youth-science programme with eighty participants. Fifty attend Centre A and thirty attend Centre B. The programme records hours spent practising a laboratory skill. Centre A has forty-five observed values averaging five hours and five missing. Centre B has fifteen observed values averaging three hours and fifteen missing. Compute the missingness rate for each centre, the pooled observed mean, and the full-population mean produced by centre-specific mean imputation.

The correct counts are twenty missing of eighty, or 25% overall. Centre A has 5 ÷ 50 = 10% missing; Centre B has 15 ÷ 30 = 50% missing. The observed total is forty-five times five + fifteen times three = 225 + 45 = 270 hours. Sixty observed cases therefore have a pooled mean of 270 ÷ 60 = 4.5 hours. Under centre-specific mean filling, Centre A contributes fifty times five = 250 hours and Centre B thirty times three = 90 hours. The constructed full-population mean is 340 ÷ 80 = 4.25 hours.

If the analyst instead inserts zero for missing values, the resulting average is 270 ÷ 80 = 3.375 hours. This arithmetic says nothing about true unseen practice. Ask whether absence might depend on an observation path, whether centre membership is an adequate proxy for missingness and whether an educational decision would be appropriate based on imputed effort. The worked solution should finish by recognising that neither 4.25 nor 3.375 can be promoted to a factual description of individual unseen cases.

Change the context again: the missing values are not practice hours but final pass/fail outcomes not yet published by an examiner. The centre-mean strategy no longer answers the question. Keep outcomes pending and define valid follow-up or censored-observation methods. If an assessment decision must be made now, it should rely on legitimate available evidence and clear uncertainty, not a manufactured pass/fail label. The transferable skill is to distinguish feature substitution from inventing a target.

12. Fairness, deployment and practical validation

Evaluation should separate missingness groups instead of treating the dataset as one homogeneous population. Compare performance for records with complete features against records requiring a substitute, with appropriate statistical uncertainty. If training and validation populations differ in collection method, investigate whether the deployed feature pattern is outside the assessed distribution. No model should automatically be declared fair because a single aggregate accuracy number looks good.

A better upstream action may be a more accessible form, offline submission route, repaired sensor or clear option to decline a question. Record the practical cost of obtaining better data as well as model performance. Do not pressure individuals into revealing unnecessary sensitive information just to reduce missingness. Consent and purpose limitations remain separate from predictive usefulness.

Model monitoring should track changes in missingness rates and label delays by meaningful time windows. Re-evaluate after new forms, school policies, devices or data suppliers are introduced. A model may accept missing values natively but still suffer if the distribution of absence changes. Set escalation thresholds from risk and evidence; do not assume that a sudden surge in NaNs is business as usual because the prediction endpoint still returns numbers.

For a learner, understanding this is a valuable numeracy skill. When presented with a neat percentage, ask how many observations were actually present, whose records were not observed, and what assumptions converted blanks into results. For a data engineer, the same questions become schema validation and reproducible pipelines. For a manager, they become decisions about whether the evidence justifies affecting real people’s opportunities.

13. Limits and ethical boundaries

Observed data generally cannot identify every plausible missingness mechanism without additional assumptions. An imputer may preserve a useful predictive relationship yet distort uncertainty or distributions. Even a strong test score may reflect only the labelled subset. A missingness indicator may act as a proxy for an unequal observation system. Sensitivity analysis, subgroup evaluation and collection improvement are therefore complementary, not optional decoration.

Be careful about what an individual-facing explanation claims. “We substituted the training median for a missing feature under a named model” can be accurate. “You studied exactly five hours” is not. Sensitive decisions should include a route for correction, human review and relevant legal or institutional requirements. These are current data-science and governance considerations, not evidence that AI can divine absent facts.

The central habit is to treat absence as information about an observation process before treating it as a number. That habit transfers to scientific measurements, survey design, financial records, school assessment and everyday reasoning. When data is incomplete, a responsible analyst makes the assumption visible and tests how the answer depends on it.

Frequently asked questions

Is a blank the same as zero?

No. Zero may be a valid observed quantity. A blank can represent not recorded, unknown, withheld, not applicable or delayed. Specify its meaning before preprocessing.

Does mean imputation recover the missing historical value?

Not necessarily. It supplies a modelling substitute under a chosen rule. It can reduce variance and obscure group differences; it should never be described as proof of what an individual actually reported.

Are MCAR, MAR and MNAR directly visible from the data?

No. They express assumptions about missingness. Observed patterns can inform analysis, but some mechanisms and counterfactual missing values cannot be identified without additional assumptions or data.

Is training-only imputation important if no labels are used?

Yes. Statistics fitted on the evaluation set can leak information about held-out feature distributions. Fit transformations inside training folds according to the real intended evaluation design.

Can some models handle NaN directly?

Some can, depending on estimator and version. That is not proof that they understand why an input is absent or that missing outcomes and selection bias are solved.

What if the target is missing rather than an input feature?

Keep the outcome pending, unknown or censored as appropriate. Define a legitimate observation and evaluation scheme instead of inserting an invented positive or negative label.

Original sources and references

Reader pathways

For practical modelling, connect this lesson to data leakage and honest validation, then revisit the How AI Works root guide. For critical thinking, the How Intelligence Works library explores evidence, limits and learning more broadly. For education and careers, use the How X Works hub and Gold Standard of Education. The useful question to carry into work or school is not just “what number did the program return?” but “which observations support that number, and what did we assume about the missing ones?”

Discover more from eduKate Singapore

Subscribe now to keep reading and get access to the full archive.

Continue reading