Speech recognition begins with pressure waves and ends with words, timestamps or actions. Between them lies a chain of mathematics: sampling turns air-pressure variation into numbers, short-time Fourier analysis separates frequencies, mel filters compress spectral detail, logarithms reshape dynamic range, and edit distance measures transcription errors.
This article explains **why mathematics is important in speech recognition and AI** without pretending that a few formulas reproduce a production system. Modern recognisers may learn features directly, use log-mel inputs, attention, transducers or large language models. The mathematical pipeline remains valuable because it shows what is measured, transformed, lost and evaluated.
Speech is a time-varying signal
A microphone produces an electrical signal related to changing acoustic pressure. Digitisation samples its amplitude at regular times. If sample rate is fs samples per second, the interval is 1/fs seconds.
At 16,000 samples per second, adjacent samples are 62.5 microseconds apart. A one-second mono recording contains 16,000 amplitude values before compression or metadata.
Sampling rate is not the number of bits per sample. Rate controls time sampling; bit depth controls amplitude quantisation. File size depends on both, plus channels and duration.
Did You Know? The recogniser does not “see” a continuous sound
It receives a finite array of numbers. Every later feature depends on how that array was sampled, scaled, channelled and segmented.
The Nyquist boundary
For a band-limited signal, a sample rate must exceed twice the highest frequency to avoid idealised aliasing. Frequencies above half the sample rate can fold into lower apparent frequencies if not filtered.
At fs=16 kHz, the Nyquist frequency is 8 kHz. This does not mean every component below 8 kHz is represented perfectly. Anti-alias filtering, microphone response, noise and finite windows matter.
Students can generate a sine wave, sample it at several rates and plot the apparent samples. Aliasing becomes visible when two different continuous frequencies produce the same discrete sequence.
Why speech is divided into frames
Speech changes over time, but over a short interval its spectral structure is often treated as approximately stationary. A common feature pipeline divides the waveform into overlapping frames.
For a 25 ms frame at 16 kHz, the frame contains 0.025×16,000=400 samples. A 10 ms hop advances 160 samples, so adjacent frames overlap.
Short frames improve time localisation but reduce raw frequency-bin spacing for a fixed transform length. Long frames improve frequency detail but can blur rapid changes. There is no universally best window; the task and model determine the trade-off.
Window functions reduce edge discontinuities
Taking a finite frame is equivalent to multiplying the signal by a window. A rectangular window stops abruptly at the edges, which can create strong spectral leakage when the frame does not contain an integer number of cycles.
A Hann window tapers toward zero. It reduces sidelobes but widens the main lobe. A cleaner-looking spectrum therefore trades one kind of resolution for another.
The window changes the data. Always record its type, length and normalisation. Comparing spectra from different windows without those details can misattribute a processing choice to the speaker.
The discrete Fourier transform
For N samples xn, the discrete Fourier transform computes complex coefficients Xk. Each coefficient represents correlation with a complex sinusoid at a discrete frequency.
The fast Fourier transform is an efficient algorithm for calculating the DFT; it is not a different transform. Complexity drops dramatically compared with direct evaluation for large N.
For sample rate fs and transform length N, bin spacing is fs/N. With fs=16,000 and N=512, spacing is 31.25 Hz. Zero-padding can create more plotted bins but does not invent information beyond the original window’s resolving power.
Magnitude, phase and power
Each Fourier coefficient has magnitude and phase. A power spectrogram commonly uses squared magnitude. Depending on normalisation, values may represent power-like quantities rather than calibrated physical sound power.
Many speech features discard or deemphasise phase. That can work for recognition, but it loses information needed for faithful waveform reconstruction.
Students should avoid calling every colour plot a “frequency.” A spectrogram has time on one axis, frequency on another and a colour scale representing magnitude or power under declared processing.
Building a spectrogram
Apply the window and FFT to each frame, retain non-redundant positive-frequency bins for a real signal, and stack spectra over time. The result shows how energy is distributed across frequency as speech evolves.
Vowels often show bands related to vocal-tract resonances; consonants can contain bursts, noise-like energy or silence. These observations are descriptive, not enough to infer identity or meaning reliably.
PyTorch’s torchaudio documentation explicitly describes spectrogram, mel-scale and MFCC transforms, with parameters such as FFT size, window length, hop length and number of mel bands. Reproducibility requires all of them.
The mel frequency scale
Human pitch perception is not linear in hertz. Mel scales map frequency into a perceptually motivated coordinate. One common formula is m=2595 log10(1+f/700), though conventions differ.
A mel filterbank places overlapping triangular filters across frequency, often denser at lower frequencies and broader at higher frequencies after mapping. Each filter sums weighted spectral energy.
The result is a lower-dimensional vector per frame. It smooths fine spectral detail and emphasises broad patterns. Changing the number of filters, frequency limits or scale convention changes the representation.
A small mel mapping example
Using m=2595 log10(1+f/700), 700 Hz maps to 2595 log10(2)≈781 mels. Inverting requires f=700(10^(m/2595)−1).
This nonlinear map is monotonic: higher hertz always gives higher mels. But equal mel steps do not equal equal hertz steps.
Students can place 20 equally spaced mel points, invert them to hertz and observe increasing gaps. That picture explains filter placement better than memorising the formula.
Why logarithms are used
Spectral energies can span a large range. Taking a logarithm compresses multiplicative differences and turns products into sums. Perceptual loudness relationships also motivate log-like representations.
Zero cannot be logged, so implementations use a floor or small offset. That constant must be recorded because it affects silent or low-energy bins.
A 10-fold power ratio corresponds to 10 decibels under the usual power definition. A 10-fold amplitude ratio corresponds to 20 decibels when power is proportional to amplitude squared. Mixing the two formulas creates a factor-of-two error.
From log-mel energy to MFCCs
Mel-frequency cepstral coefficients commonly apply a discrete cosine transform to log-mel filterbank energies. The transform concentrates smooth spectral-envelope information into earlier coefficients.
Keeping a subset reduces dimension and can decorrelate features for older statistical models. The coefficient index is not a frequency in hertz; it describes variation across the mel-filter sequence.
MFCC conventions differ in DCT type, normalisation, whether the zeroth coefficient is kept, energy replacement and liftering. Two arrays both called MFCC may not be directly comparable.
Delta features describe change
Static features describe one frame. Delta features estimate change across nearby frames, and delta-delta features estimate change of change.
A simple central difference is sensitive to context width and boundary handling. Wider windows smooth noise but can blur rapid transitions.
Modern neural models may learn temporal patterns directly, yet derivative features remain an excellent teaching example of finite differences and local regression.
Normalisation choices
Cepstral mean normalisation subtracts an average, often reducing channel effects. Variance normalisation rescales spread. The scope may be an utterance, speaker, sliding window or dataset.
Normalising with future test data can leak information. A reproducible experiment states which statistics were computed from training data and how they were applied at evaluation time.
Normalisation improves comparability only relative to its assumptions. It can also remove meaningful level information if the task needs it.
From features to probabilities
A recognition model maps feature sequences to units such as phonemes, characters, subwords or words. Older systems often combined hidden Markov models with acoustic likelihoods. Modern systems may use connectionist temporal classification, attention or recurrent neural transducers.
The output is not usually one certain sentence. It is a distribution or score over many possible sequences. Decoding searches this space using acoustic evidence and perhaps a language model.
The highest-scoring path depends on model probabilities, search approximations and penalties. Changing beam width or language-model weight can change the transcript without changing the audio.
Dynamic programming and alignment
Speech and transcript lengths differ. Dynamic programming finds an alignment without checking every possible path independently.
The same idea appears in edit distance. Build a grid where rows represent reference words and columns represent hypothesised words. Each cell stores the lowest cost after insertion, deletion or substitution.
This turns a global comparison into local recurrence. The final cost gives the minimum number of edits under the chosen costs, while backtracking reveals the alignment.
Word error rate
NIST’s traditional automatic speech recognition metric is
**WER=(S+D+I)/N**,
where S is substitutions, D deletions, I insertions and N reference words.
Reference: “we learn maths today.” Hypothesis: “we learn math very today.” Depending on tokenisation, “maths” to “math” is one substitution and “very” is one insertion, so WER=2/4=50%.
WER can exceed 100% if insertions are numerous. It is not an accuracy percentage constrained between zero and one.
Transcript conventions change WER
Should punctuation count? Are numbers written as digits or words? Are fillers such as “um” included? How are hyphenated words and contractions tokenised?
Two teams can score the same audio differently if their normalisation rules differ. A benchmark must publish reference preparation and scoring rules.
NIST evaluations emphasise common data and common metrics so systems can be compared under one protocol. That does not make a single WER universal across languages, accents, domains or consequences.
Average error can hide group differences
An overall WER can conceal worse performance for particular acoustic conditions, dialects, languages, microphones or speaking styles. A test set also may not represent actual users.
Report subgroup and condition performance when ethically and statistically appropriate. Include sample counts and uncertainty rather than ranking tiny groups by unstable percentages.
Fairness is not solved merely by equal average WER. The consequences of an error, access to correction and deployment context matter.
Confidence and calibration
A recogniser may output confidence scores. A calibrated score of 0.8 would mean that among comparable outputs assigned 0.8, about 80% are correct under the defined event and dataset.
High softmax probability is not automatically calibrated confidence. Reliability diagrams and proper scoring rules can test calibration.
Confidence can support review workflows, but thresholds trade false acceptance against unnecessary rejection. A threshold should follow the application’s costs and safety needs.
Noise, reverberation and mismatch
Additive noise changes spectral energy. Reverberation convolves speech with a room impulse response, smearing energy across time. A microphone or codec adds its own transfer characteristics.
Training on clean close-microphone speech and testing in a reverberant classroom creates distribution shift. Data augmentation can simulate some conditions, but simulated diversity is not proof of real-world robustness.
Students can add noise at controlled signal-to-noise ratios and compare log-mel features, recognition outputs or a simpler classifier. Keep original clean data and random seeds.
Sampling and feature parameters interact
An FFT size of 512 means different time duration and frequency spacing at 8 kHz and 16 kHz. A mel filterbank’s upper frequency should not exceed Nyquist.
Resampling requires filtering. Simply dropping every other sample can alias high-frequency content.
Store sample rate with the waveform. A raw array without rate has no correct time or frequency axis.
Privacy and responsible data use
A voice recording can contain identity, health, location or private conversation. Classroom work should use consented, public or synthetic audio and minimise retained data.
Do not scrape voices merely because they are audible online. Record purpose, permission, retention and access. Delete raw audio when it is no longer needed under the agreed plan.
Mathematical skill includes deciding what should not be collected.
Misconceptions to avoid
- A spectrogram is not the same as a transcript.
- More FFT bins do not automatically create more true resolution.
- Mel frequency is not measured in hertz.
- MFCC coefficient index is not a physical frequency.
- WER is not always between 0% and 100%.
- Low average WER does not guarantee reliable performance for every user or situation.
Beam search in decoding
The number of possible output sequences grows rapidly. Beam search keeps only a limited set of promising partial hypotheses at each step. A wider beam explores more alternatives but costs more computation.
Because it is approximate, a narrow beam can discard the path that would later become best. Increasing width should not worsen the best model score under a correctly implemented search, yet the final text and runtime may change.
Students can decode a tiny probability tree exactly and then with beams of width one, two and three. The first width that recovers the exact optimum makes search error visible.
Language-model weighting and insertion penalties
Decoders often combine acoustic or sequence-model scores with a language-model score. Since these may be on different scales, a weight balances them. An insertion or length penalty can control a tendency toward too many or too few words.
Tuning on the test set leaks evaluation information. Parameters should be selected on development data and then frozen before final scoring.
A lower WER after tuning does not prove better acoustic understanding; it may reflect stronger linguistic prior or a domain-specific vocabulary.
Character and subword error rates
Word error rate depends on tokenisation and language structure. Character error rate applies edit distance to characters, while subword metrics use learned or defined units.
No metric is universally superior. Character scoring may treat a long misspelled word as several errors, while word scoring treats it as one substitution. Languages without whitespace word boundaries require careful conventions.
Report the metric and normalisation rules together. A bare “5% error” is incomplete.
Statistical uncertainty in evaluation
A finite test set produces an estimate. Two systems whose WER differs by 0.1 percentage point may not be meaningfully different if errors cluster by speaker or recording.
Resampling entire utterances or speakers can estimate uncertainty while respecting grouping better than pretending every word is independent. Matched-pair tests can compare systems on the same utterances.
Always report test-set size and composition. Large word counts from a few speakers do not create broad population coverage.
Real-time factor and latency
Accuracy is not the only operational measure. Real-time factor is processing time divided by audio duration. A factor below one means faster-than-real-time processing under the tested hardware and settings.
Streaming latency also depends on frame look-ahead, buffering, network delay and endpoint detection. A model can have fast average throughput yet delay the first useful transcript.
Students can time repeated runs, separate warm-up from steady state and report a distribution rather than one fastest result. Performance claims require hardware and software versions.
Endpoint detection
A streaming recogniser must decide when speech has ended. Ending too early deletes words; waiting too long increases latency. Thresholds therefore create a measurable trade-off between truncation and responsiveness.
Test endpoint rules on pauses of different lengths and report both error and delay. Silence is not merely zero amplitude: background noise and breathing require a defined speech-activity model.
How students can build transferable skill
Begin with one synthetic sine wave, then two. Sample, window and transform them. Verify bin frequencies by hand before using a library.
Next, use a short public speech clip. Vary window length, hop and number of mel bands one at a time. Save plots with identical colour scales.
Finally, score several toy transcripts with a handwritten edit-distance grid, then confirm with code. This connects signal processing, logarithms, matrices and algorithms in one reproducible chain.
Guidance for parents and teachers
Speech technology is an inviting path into AI because students can hear the input and inspect each transformation. Keep expectations realistic: feature extraction is only one part of recognition.
Ask “what information was lost?” after every step. Sampling loses content above the supported band; magnitude spectra may discard phase; mel filters smooth detail; truncated cepstra compress the envelope.
Use accents and languages respectfully. An error demonstrates a system limitation, not a speaker deficiency.
Careers and pathways
Speech technology draws on signal processing, machine learning, linguistics, software engineering, accessibility design, data curation and evaluation science. Work appears in transcription, captioning, voice interfaces, language learning and assistive tools.
Mathematics does not guarantee a fair or useful product. Dataset design, privacy, language knowledge, user research and human review remain essential.
Frequently asked questions
What is a mel spectrogram?
It is a time-frequency representation produced by applying a mel-spaced filterbank to a spectrogram, with declared processing choices.
Why overlap frames?
Overlap improves time coverage and reduces the chance that a rapid event lies only at a hard frame boundary.
Are MFCCs still used?
They remain important in many systems and in education, while modern systems also commonly use log-mel features or learned front ends.
What does WER measure?
It counts the minimum substitutions, deletions and insertions relative to reference word count under a scoring convention.
Can WER compare any two systems?
Only when they use the same evaluation data and transcript/scoring rules. Domain and subgroup performance still matter.
Useful next reading
- PyTorch Audio: spectrogram, mel-scale and MFCC transforms
- NIST: speech-processing measurement and evaluation programmes
- NIST: traditional ASR word error rate definition
- Why Mathematics? | Digital Audio Sampling, Nyquist Rate and Aliasing
- Why Mathematics? | Microphones, Polar Patterns, Decibels and Distance
- Mathematics Learning Hub
A final perspective
Speech recognition makes the journey from continuous experience to discrete computation visible. A waveform becomes frames, frames become spectra, spectra become mel energies, features become probability scores and words become an edit-distance alignment. Mathematics does not erase language’s richness. It gives us a way to see exactly where a machine representation begins, what it preserves and where careful evaluation must take over.
A Practical Investigation Studio
Use synthetic or openly released teaching data. These investigations expose assumptions and error signals; they do not replace clinical prescription, accelerator engineering, personalised dietetics or production speech-system evaluation.
Investigation 1: Sampling check
Record or generate a short synthetic vowel-like signal. Verify sample rate and Nyquist limit. Use synthetic or openly released teaching data. Begin by naming the response variable, input variables, units and prediction before calculating. Preserve raw values and unrounded intermediates in a table, then make one graph whose axes communicate the model clearly. Add an independent hand estimate and a dimensional or limiting-case check. Deliberately introduce one unit error and describe the numerical signature rather than merely correcting it. Label every quantity observed, assumed, fitted or derived. Record the model domain and one condition that would require a richer model. Ask a peer to reproduce the result from the written procedure without seeing the answer. If the reproduction differs, locate whether the ambiguity arose in definitions, units, rounding or software defaults. End by explaining the result to a younger student in three sentences. This remains a classroom calculation, not professional validation or a safety, production or security decision.
Investigation 2: Framing
Divide a waveform into overlapping short windows. Compare time resolution at two window lengths. Create a data dictionary before entering a spreadsheet or script. State how each row was obtained, which values are exact by definition and which are measurements or simulated observations. Calculate once with full precision and once with premature rounding, then compare the final difference. Vary the principal input across at least five sensible values and look for linearity, curvature or instability instead of reporting only two endpoints. Keep signed residuals if a model is fitted, because absolute errors hide direction. Include a graph and a small results table that another person can audit. Write a one-paragraph uncertainty note distinguishing input uncertainty from model-form limitations. A peer should be able to reconstruct one row from the formula and raw data alone. State what evidence would falsify the interpretation. Do not present the exercise as a certified engineering, manufacturing, metrology or cryptographic assessment.
Investigation 3: Window function
Apply rectangular and Hann windows. Inspect leakage without declaring one universally best. Treat the investigation as a comparison of hypotheses, not a hunt for an attractive number. Write a baseline model and at least one plausible alternative, then predict where their outputs should diverge. Hold all unrelated inputs fixed while varying the chosen factor. Preserve coordinate, sign, phase, size-weighting or bit conventions beside the data. Use a ratio, residual or normalised error that has a declared denominator. Repeat the calculation with that denominator changed and explain why the percentage changes. Check one extreme case where the answer should approach zero, unity or another known limit. Make the graph before writing the conclusion and note any outlier without deleting it. Separate repeatability of one prepared sample from coverage across positions, times or populations. The final claim should match the observed range and must not be extrapolated into real operational approval.
Investigation 4: FFT bins
Map discrete frequency-bin indices into hertz. Keep bin spacing and true resolving ability distinct. Plan the computation so that errors leave visible fingerprints. Save the raw input, transformed input, intermediate result and final result in separate columns or variables. Insert one deliberate sign reversal, one missing observation and one hidden reset, each in a separate copy, and document how the plots or checks respond. Compare an analytical shortcut with the more complete calculation over a range where the shortcut is expected to fail. Report the largest absolute difference and where it occurs. Include conservation, balance, boundedness or monotonicity checks appropriate to the topic. If a fitted curve is used, inspect residual clusters and changing spread instead of relying on one fit statistic. Have a peer review only the assumptions first, then the arithmetic. Keep the conclusion educational and conditional on the fictional data.
Investigation 5: Mel filters
Build triangular filters on a mel-spaced axis. Plot their overlap and normalisation. Build a sensitivity map rather than changing several settings at once. Choose a baseline, vary one input downward and upward, and express output change in both original units and a dimensionless ratio. Then select a second input and repeat. If interactions seem likely, test a small two-dimensional grid and identify where the one-factor interpretation breaks down. State why the chosen ranges are plausible for the teaching model. Preserve failed or surprising runs and annotate them. Compare computational resolution or sample length at three levels so numerical artefacts are not confused with physical behaviour. Provide a hand estimate for the order of magnitude. Finish with a decision table using cautious language—stable, sensitive or unresolved—rather than safe or unsafe. Classroom evidence cannot authorise a product, structure, process or security control.
Investigation 6: Log energy
Compare linear power with decibel-like log features. Handle zeros with an explicit floor. Make uncertainty visible at every stage. Assign each input a source and a plausible interval, then identify which inputs are correlated rather than automatically treating them as independent. Propagate uncertainty with a simple high-and-low calculation or transparent simulation, and retain the seed or full synthetic samples for reproduction. Compare the spread caused by measurement variation with the shift caused by changing the model assumption. Plot both if possible. Explain why more decimal places do not narrow an interval supported by weak inputs. If a result lies near a fictional decision boundary, rewrite the conclusion to reflect the chance of crossing it. Ask a peer to challenge the largest assumed uncertainty and rerun the analysis. Report the conclusion as evidence about the model, never as professional certification.
Investigation 7: Cepstral transform
Apply a cosine transform to log-mel energies. Explain compression as decorrelation, not magic. Audit representation choices. Recalculate after changing the coordinate origin, phase wrapping convention, histogram bins, mesh labels or binary mapping while preserving the underlying situation. A correct physical or probabilistic conclusion should change only where the definition genuinely changes. Show one example where a visual choice exaggerates or hides a pattern. Use identical axes for fair comparison and include sample count or resolution in captions. Test an invariant such as total mass, probability, force, energy sign or average ratio. Where values are averaged, identify whether the mean is number-, mass-, volume-, time- or state-weighted. Ask another student to explain the plot without reading the conclusion; revise labels if their interpretation differs. The display supports reasoning but cannot replace domain review.
Investigation 8: Delta features
Estimate first differences across frames. Show sensitivity to context width and boundaries. Separate verification from validation. First verify arithmetic or code against a case with a known answer, including units and at least one manually calculated row. Next ask whether the teaching model represents the intended phenomenon and list the omitted mechanisms. Refine numerical resolution or increase synthetic sample length and record whether the chosen quantity stabilises. A stable answer can still arise from an unrealistic model, so compare with an alternative assumption or openly documented benchmark. Record software version, solver or library settings and stopping rules. Avoid tuning settings after seeing the desired result; predeclare the comparison instead. Summarise numerical error, input uncertainty and model-form uncertainty separately. No classroom agreement with a benchmark should be described as product or system validation.
Investigation 9: Edit distance
Align a reference sentence and hypothesis. Count substitutions, deletions and insertions explicitly. Design the investigation to reveal dependence over position, time or sequence order. Keep the original ordering, then compare with a deliberately shuffled copy and explain which statistics change. Examine at least two lags, locations or neighbourhood scales rather than only the overall average. Plot local values alongside the aggregate so compensating errors are visible. State whether successive observations are plausibly independent and why that matters to the calculation. If repeated measurements share one specimen, source or initial state, do not count them as independent coverage. Add a block or subgroup analysis and note any drift. Preserve the random seed for simulated data but use more than one seed before generalising. The conclusion should describe detected structure without claiming causation from the pattern alone.
Investigation 10: Noise stress test
Add controlled noise and reverberation to a teaching clip. Compare features and errors without using private recordings. Create an adversarial edge-case test. Identify the smallest, largest, most symmetric and most irregular inputs allowed by the classroom model, predict the expected behaviour, and then compute it. Include zero or a limiting value only when mathematically meaningful. Watch for division by a small number, logarithms of invalid values, wrapped angles, negative geometry, impossible probabilities or solver singularities. Replace silent software errors with explicit checks and explanatory messages. Compare the edge cases with the ordinary baseline on the same scale and describe why a method that works in the middle may fail near a boundary. Record the first assumption that breaks. The exercise is about model literacy; it is not permission to explore hazardous physical extremes.
Investigation 11: Evaluation card
Archive preprocessing, transcript rules and WER counts. Have a peer reproduce the alignment. Prepare a compact reproducibility package: raw synthetic data, formulas or code, version information, one chart, a results table and a short read-me file. Give every file and column a meaningful name and avoid values copied manually between tools. Re-run from a clean state and confirm that outputs are regenerated rather than cached. Ask a peer to alter one declared input and predict the direction of change before execution. Compare the prediction with the result and investigate any disagreement. Add a limitations section naming data coverage, numerical resolution, model assumptions and the next useful measurement. Archive corrections instead of erasing them so the reasoning trail remains visible. Conclude with what the calculation demonstrates, what it does not demonstrate and which qualified professional or authoritative standard would govern a real application.