Did you know? A large data table may contain dozens of measurements that overlap in what they tell us. Two sensors may respond to the same underlying machine condition, while a student’s Mathematics and Additional Mathematics results might vary together. Sometimes a smaller set of new numerical coordinates can summarise much of the original variation.
One method for doing this is principal component analysis (PCA). Rather than selecting only a few existing columns, PCA creates new directions that combine information from the original numerical features.
Quick answer: prepare numerical features → centre the training data → identify directions of greatest variance → project onto a chosen number of components → measure what was lost → evaluate the downstream task. PCA can help AI workflows, but it does not automatically discover the most important real-world causes.
PCA in 30 seconds
- Feature: a numerical measurement describing an example.
- Dimension: one coordinate or feature direction in the numerical data.
- Variance: a measure of how much values spread around their mean.
- Principal component: a new axis chosen to capture substantial remaining variance.
- Projection: expressing each example using positions along selected axes.
- Explained variance ratio: the fraction of total variance represented by certain components.
- Dimensionality reduction: representing information with fewer numerical coordinates.
A principal component is not necessarily one familiar category such as “ability” or “interest.” It may be a weighted combination of many features, so its physical or educational meaning needs separate interpretation.
Imagine a diagonal cloud of points
Draw a scatterplot containing two hypothetical measurements for several observations. Suppose the points form a long, thin cloud from bottom-left to top-right. Most of the visible spread is along that diagonal.
Instead of measuring horizontal and vertical position separately, we could describe each point by how far it lies along a new diagonal axis. If deviations away from that line are small, the one-dimensional description may preserve much of the cloud’s variation.
That is the geometric intuition of PCA. It selects new axes based on the shape of the data cloud, not on a human-written classification such as “good” or “bad.”
Step 1: Choose meaningful, reliable input features
PCA requires numerical inputs. A factory might analyse several vibration measurements; a research dataset might contain dimensions of flowers; a school exercise could use two invented columns of marks to practise visualising relationships.
Missing values, mismatched units and inaccurate data require care before applying the method. Turning arbitrary categories into numbers without a suitable encoding can create fake distances between categories.
Also decide whether dimension reduction is actually needed. A small table with three straightforward variables may be easier to understand without an added mathematical transformation.
Step 2: Subtract the mean of each column
In ordinary PCA, features are centred so the data cloud has its mean at the origin. Consider three fictional points: (1,1), (2,2), (3,3). The mean is (2,2).
Subtracting the mean produces the centred points (−1,−1), (0,0), (1,1). Those points lie exactly on one diagonal line.
The official scikit-learn PCA reference confirms that its PCA implementation centres the input before decomposition, but does not automatically scale each original feature to unit variance.
Step 3: Find the direction of greatest spread
The first principal component is the direction in which the centred observations vary most. In our three-point example, that direction is the diagonal line y = x.
The next component, where one exists, is perpendicular to the first and captures as much of the remaining variation as possible. Additional components follow the same idea, subject to the number of features and observations.
The axes are learned from the training data, not chosen by the author simply because a graph looks tidy. In higher-dimensional datasets, the directions involve weights for numerous original features.
Step 4: Why one component can sometimes be enough
With (−1,−1), (0,0), (1,1), all variation lies along the first diagonal direction. For this exact fictional construction, the first principal component accounts for 100% of the variance, and the perpendicular second direction contains none.
This perfect result is unusual outside teaching examples. Real observations have noise, measurement errors and relationships that do not fall exactly on a straight line.
The idea is not that PCA miraculously removes every unnecessary feature. It is that the principal axes give a principled way to describe a data cloud with fewer numerical coordinates if the trade-off is acceptable.
Step 5: Understand explained variance with arithmetic
Imagine a separate made-up dataset whose four components account for variance quantities of 5, 3, 1 and 1. Total variance is 5 + 3 + 1 + 1 = 10.
Keeping the first two components accounts for (5 + 3) ÷ 10 = 80% of the total variance. The omitted components account for the remaining 20% in this illustrative example.
That does not mean that predictions will be 80% accurate or that the representation preserves 80% of every meaningful fact. Explained variance is a specific statistical quantity.
The essential distinction: PCA tries to preserve variance in numerical data, not necessarily the features most important for truth, prediction or human decisions.
Step 6: Project records into the new coordinate system
After fitting PCA on the training data, a new record can be transformed into coordinates along the learned principal axes. When fewer axes are retained, the record has a shorter representation.
The transformation is linear in ordinary PCA. Its projection can be useful for plotting, comparing broad patterns or preparing input for some later models. The discarded directions are generally not recoverable exactly from the reduced representation.
When displaying a two-dimensional PCA scatterplot, remember that it shows an approximation of the original feature relationships. Distances and apparent clusters in that plot need careful interpretation.
Step 7: Why feature scaling can change everything
Suppose one input feature is a salary measured in dollars, while another is age in years. Dollar values can have much larger numerical spread simply because of their units.
PCA is sensitive to scale: the large-variance feature may dominate the selected directions. For certain tasks, developers standardise variables using training-set means and standard deviations before PCA.
But scaling is a choice, not a compulsory universal rule. When absolute physical variation has a special meaning, equalising all features may change the question. Apply the preparation that matches the task and document it.
Step 8: Where do eigenvectors and SVD appear?
Technical descriptions of PCA often mention eigenvectors, eigenvalues or singular value decomposition (SVD). These are mathematical tools for finding suitable component directions and quantifying associated variation.
For a beginner, the essential idea is enough: PCA finds directions through data that account for large amounts of spread. The algorithm can then express examples along those directions.
Scikit-learn provides different solvers for computing PCA efficiently according to data size and other requirements. Those engineering details do not change the meaning of the first principal component.
Step 9: Do not accidentally leak test information
Imagine using an entire dataset to calculate feature means and PCA directions, then dividing it into training and test records. The supposedly unseen test examples have already influenced preprocessing.
That can make downstream evaluation misleading. A sound workflow splits appropriately first, fits scaling and PCA on training data, then applies the trained transformations to validation and test data.
Keep the final test separate from choices about how many components to retain. Otherwise, repeated tuning on the same test can make the reported result optimistic.
Step 10: Variance is not the same as importance
Imagine a machine-safety dataset in which one sensor changes only rarely, but those rare changes strongly indicate an important failure. Its variance might be small relative to a frequently changing temperature signal.
PCA could place the rare signal in a low-variance direction and discard it if the designer retained too few components. That would be harmful for the failure-detection task even if most total variance were preserved.
Therefore, a PCA-based preprocessing step should be evaluated against the actual task and error costs. A percentage of preserved variance is not an automatic safety guarantee.
Step 11: PCA versus clustering
Clustering groups examples according to a chosen similarity definition. PCA changes the axes or dimensions used to represent examples. These methods can work together, but they are not synonyms.
A scatterplot after PCA may appear to contain several groups. That visual pattern needs separate checks; projecting data into fewer dimensions can obscure relationships or make some separations appear stronger than they are.
Read How AI Finds Patterns Without Labels for the distinction.
Step 12: PCA versus an autoencoder
An autoencoder can learn a latent representation by trying to reconstruct its inputs through a trained encoder and decoder. PCA is ordinarily a linear transformation chosen to preserve variance.
Some autoencoder setups have mathematical links to PCA under special assumptions, but nonlinear architectures can learn different transformations and objectives.
See How Autoencoders Compress and Reconstruct Information for the reconstruction-based approach.
Step 13: When PCA can be useful
- Visual exploration: inspect broad patterns in datasets with many numerical columns.
- Representation reduction: provide fewer features to an appropriate downstream algorithm.
- Noise investigation: examine whether some low-variance directions are less useful under tested assumptions.
- Scientific analysis: study variation among related measured signals.
- Educational mathematics: show why new coordinates can describe the same cloud of points.
These are possible applications, not promises. The right number of retained components and their practical interpretation depend on data quality and the intended analysis.
Try it: find the long direction of your own data
Plot (1,1), (2,2), (3,3), (4,4) on graph paper. Draw the line y = x through them. Now add (3,4), which lies slightly above the original line.
If you describe each point only by its location along the diagonal, what happens to the difference between (3,3) and (3,4)? Some information about displacement away from the diagonal is lost.
That small experiment shows what a lower-dimensional projection can preserve and omit—without requiring any programming or advanced matrix calculations.
Five misconceptions to avoid
- “The first component causes the outcome.” PCA examines variance, not causal influence.
- “Explained variance equals accuracy.” These are different quantities.
- “PCA automatically scales each feature.” Centering and scaling are separate operations.
- “Low-variance information is always unimportant.” Rare signals can be consequential.
- “Test data can guide PCA before model evaluation.” That can undermine independence.
Frequently asked questions
What does PCA stand for?
Principal component analysis, a numerical method that finds new directions ordered by how much variance they explain.
Is PCA itself a neural network?
No. Ordinary PCA is a linear mathematical transformation, although it is widely used in machine-learning workflows.
Why keep only the first two components?
Two dimensions can be convenient for plotting, but the right number depends on the variance retained and downstream performance.
Does 90% explained variance imply 90% correct predictions?
No. It describes preserved data variation, not model accuracy.
Should I standardise every feature before PCA?
It depends on units, task and the meaning of the measurements. Ordinary PCA centres values; standardisation must be considered separately.
Your one-minute takeaway
- PCA turns original numerical features into new directions of variance.
- Keeping fewer components reduces dimension at the cost of discarding some information.
- Use task-specific evaluation and fit transformations on training data to avoid misleading results.
Trusted reading and connected AI guides
- scikit-learn: Principal Component Analysis — definition, centring and explained variance.
- scikit-learn: Decomposing Signals Into Components — methods and interpretation.
- scikit-learn: StandardScaler — how feature scaling differs from centring.
- How Autoencoders Compress and Reconstruct Information — a different reduction approach.
- How AI Finds Patterns Without Labels — another type of unsupervised analysis.
- How AI Learns From Examples — training and independent validation.
- Secondary 1 Mathematics Tutor Clementi — numerical reasoning and mathematical concepts.
- How AI Works: Complete Overview — further explainers.