GPUs and other AI accelerators speed up particular kinds of numerical work by executing many suitable operations in parallel. But a faster GPU kernel does not necessarily make an application faster: moving data, allocating memory, waiting for other components and verifying the result all take time. The correct comparison follows the complete task from input to checked output.
This chapter uses a fictional Number Tiles laboratory to compare one operation with repeated work that can reuse GPU-resident data. You will calculate why a four-millisecond kernel can lose to a twelve-millisecond CPU job, then see the comparison reverse under different data-flow assumptions. A second exercise tests memory capacity, bandwidth and transfer costs. It fills topic 081 in the How Super Intelligence Works curriculum.
Super Intelligence is eduKateSG’s editorial name for examining current AI mechanisms, not an assertion that contemporary computing hardware produces artificial superintelligence. This guide teaches first-principles engineering: identify the bottleneck, state the units and verify the complete measured outcome.
Chapter contents and reader routes
Understand acceleration · Read Number Tiles · Compute timing · Inspect memory · Independent practice · Sources
- 1. Why accelerators exist
- 2. Think in terms of workload before choosing hardware
- 3. Parallelism means doing suitable work together
- 4. The kernel is not the entire task
- 5. The fictional Number Tiles laboratory
- 6. Work out a single GPU operation honestly
- 7. Now keep the data on the GPU
- 8. The transfer exercise changes one condition
- 9. Data movement is real computation-system work
- 10. Memory capacity and memory bandwidth are different
- 11. A complete invented memory-budget packet
- 12. Change the activation requirement and retest
- 13. Precision changes storage but also numerical behaviour
- 14. An idealised bandwidth bound
- 15. Latency and throughput answer different reader questions
- 16. The CPU still matters in an accelerator system
- 17. Different accelerators and different workloads
- 18. Training and inference load hardware differently
- 19. Why peak theoretical rates need caution
- 20. Energy efficiency requires a whole-system denominator
- 21. How a failed experiment should be reported
- 22. Independent practice: two-stage processing
- 23. Answers to the independent exercise
- 24. A procurement and engineering checklist
- 25. Sources and the next learning route
1. Why accelerators exist
An accelerator is specialised computing hardware intended to perform selected workloads more efficiently than a general-purpose processor under suitable conditions. Graphics processing units, or GPUs, are widely used for modern machine-learning workloads because many operations can be expressed as large groups of related numerical calculations. This does not mean every task becomes faster on a GPU.
A CPU is flexible and efficient at many kinds of control and sequential work. A GPU devotes substantial resources to processing parallel work, often with its own high-bandwidth memory and execution scheduling. A useful system combines them: the CPU orchestrates parts of the work, while the GPU handles operations suited to its architecture. The answer depends on computation, data movement and the constraints of the complete application.
2. Think in terms of workload before choosing hardware
A learning assistant can generate a sentence, retrieve documents, classify images, compute a spreadsheet total or update a record. Those jobs do not share identical hardware requirements. Retrieval may depend heavily on storage and network access; a small exact calculation may finish on a CPU before accelerator setup pays for itself. A large matrix calculation may benefit from the GPU’s parallel design.
Begin with the task, its input sizes, accepted accuracy, deadline and how often similar computations repeat. Then ask which stage dominates actual processing time. A device’s advertised peak computation rate cannot explain a workload whose bottleneck is transferring inputs over a slow connection.
3. Parallelism means doing suitable work together
Parallelism allows multiple units of work to proceed simultaneously. For a vector addition, separate elements can often be calculated independently. For a large matrix operation, many partial products can be organised into parallel work, subject to data dependencies and memory access requirements. GPUs provide specialised execution resources and programming models for such patterns.
Not every task decomposes cleanly. A decision tree may require choosing a branch before later work is known. A serial tool call may wait for an external service. The maximum useful parallelism is limited by the structure of the task and the system’s capacity to supply data. Increasing the number of workers cannot make an inherently dependent chain entirely simultaneous.
4. The kernel is not the entire task
In GPU computing a kernel is a function executed on the device, generally over many threads. Real applications must also prepare inputs, allocate resources, transfer data where necessary and interpret or deliver results. These steps can be larger than the kernel’s computation time.
A trustworthy benchmark measures the operation the user actually needs, not merely the fastest interior step. GPU kernel time may be an important diagnostic, but end-to-end time includes overhead that cannot be wished away. The result must name exactly where the timer started and stopped.
5. The fictional Number Tiles laboratory
Our teaching case is an invented Number Tiles exercise. A CPU completes one small tile transformation in twelve milliseconds, including its input and output handling. A hypothetical GPU computes the same transformation in four milliseconds after its inputs are available. Getting the initial data to the GPU takes eight milliseconds; returning the required output takes five milliseconds.
These numbers are deliberately fictional. They are not product specifications, model performance claims or instructions to purchase equipment. The task is to compare complete workflows under stated conditions and see when a faster kernel fails to produce a faster answer.
6. Work out a single GPU operation honestly
The GPU path takes eight milliseconds for input transfer, four for computation and five for output transfer. Its total is seventeen milliseconds. The CPU finishes the same complete job in twelve milliseconds. Under this simplified one-shot setup, the CPU is faster by five milliseconds even though its computation stage is slower.
An advertisement saying the GPU ‘computes the operation three times faster’ would compare twelve with four and ignore the transfer costs. If the CPU’s twelve-millisecond time is end-to-end, that comparison is not like-for-like. The full input-to-output user job is seventeen versus twelve. This is a classic case where a highly capable component loses within the surrounding system.
7. Now keep the data on the GPU
Suppose four different tile operations use the same transferred input, and only one combined output is needed at the end. Keep the data resident on the GPU between operations. Under the invented constant times, the GPU uses eight milliseconds to transfer the input, four times four milliseconds to compute, and five milliseconds to retrieve the combined result.
The total is eight plus sixteen plus five, or twenty-nine milliseconds. The CPU takes four times twelve, or forty-eight milliseconds, if the tasks run one after another without an equivalent optimisation. The GPU now has a nineteen-millisecond advantage. The result depends on the assumption that the input is reusable and intermediate results need not leave the device.
8. The transfer exercise changes one condition
Change the Number Tiles task: each of the four operations now requires a new independent input and a separate output returned immediately. Under the same fictional costs, each GPU operation costs seventeen milliseconds end to end. Four sequential operations require sixty-eight milliseconds; the CPU path remains forty-eight milliseconds.
The apparent accelerator advantage disappears because the reuse condition vanished. Students should write the data movement plan before comparing computation speeds. The transferable lesson is not ‘GPU always wins with four batches’ but ‘GPU may win when enough relevant work remains on the device to amortise movement and setup’.
9. Data movement is real computation-system work
GPUs access different memory spaces with differing scope and latency. NVIDIA’s current CUDA programming documentation distinguishes device global memory, thread-block shared memory and registers. The architecture offers opportunities to reuse data close to the execution units, but moving data and coordinating access still have costs.
A programming approach that repeatedly copies the same data through slow paths can limit throughput even when arithmetic units are underused. A good implementation therefore asks which data should remain resident, which can be shared and how to avoid unnecessary transactions. The exact best pattern depends on hardware, memory alignment, access sequence and software.
10. Memory capacity and memory bandwidth are different
Capacity is the amount of data a memory system can hold, while bandwidth measures how much data can move per unit time under specified conditions. A model that cannot fit into available device memory raises one kind of problem; a model that fits but repeatedly streams large quantities of data may face another.
Do not treat memory bandwidth as synonymous with compute rate. Faster arithmetic does not rescue a workload that waits for data. Nor does a large memory capacity guarantee rapid access. Hardware comparisons must separate size, throughput, latency and the overhead of moving data among memory levels.
11. A complete invented memory-budget packet
Consider a fictional accelerator with twelve gibibytes of usable memory capacity under a simplified teaching assumption. The intended program needs six gibibytes for model weights, three for activations during a particular batch, two for temporary workspace and half a gibibyte for outputs. Sum the parts before deciding whether the program fits.
The paper allocation is 6 + 3 + 2 + 0.5 = 11.5 GiB, leaving 0.5 GiB of the stated twelve. This is a narrowly defined arithmetic budget. It does not prove that a real runtime will fit, because framework allocations, memory fragmentation, caching, additional buffers and concurrent processes can consume space.
12. Change the activation requirement and retest
If a larger batch raises the invented activation requirement from three to four gibibytes, the new total becomes 6 + 4 + 2 + 0.5 = 12.5 GiB. That exceeds the twelve-gibibyte simplification by 0.5 GiB, before other overhead. The revised batch does not satisfy the original memory budget.
Possible engineering responses include reducing batch size, changing the computation strategy, offloading suitable data or selecting different hardware. Each option changes performance or complexity and needs testing. A system that simply reports a larger model ‘will run because the weights fit’ has confused one allocation with the whole workload.
13. Precision changes storage but also numerical behaviour
Numerical values can be represented with different numbers of bits. Lower-precision formats often reduce storage and data movement and may permit more efficient operations on supported hardware. But precision is not merely a compression switch; it can affect rounding, overflow behaviour and the accuracy of the model’s computations.
Evaluate the effect on the actual task. An exact financial calculation or a sensitive scientific simulation may require stronger numerical guarantees than a tolerant approximate feature transformation. Developers can test quality changes and use mixed-precision designs where appropriate. Do not infer that halving the bits always halves end-to-end latency.
14. An idealised bandwidth bound
Suppose an invented workload must read eight gibibytes of data from a device memory system with a hypothetical effective transfer rate of eight hundred gibibytes per second, assuming a perfectly sustained rate and no useful caching. The ideal lower bound from these two numbers is 8/800 seconds, or 0.01 second: ten milliseconds.
Real execution may take longer because of access patterns, arithmetic, synchronisation and other overhead. The bound is not a prediction of observed speed or a benchmark for any actual GPU. It teaches how units connect and why even very fast arithmetic can wait on movement of large data.
15. Latency and throughput answer different reader questions
Latency is time to complete one request under stated conditions. Throughput is the amount of work completed per unit time. A service might handle many requests concurrently and show strong throughput while the wait experienced by one user remains unacceptable. Batching can improve device utilisation while adding queueing delay.
A student waiting for one checked explanation and a service processing thousands of independent queries do not necessarily prefer the same settings. Evaluate queue wait, computation, evidence retrieval and delivery separately. A single ‘tokens per second’ or ‘operations per second’ number should not be promoted into a guarantee of user-perceived responsiveness.
16. The CPU still matters in an accelerator system
CPU tasks include aspects of application logic, data preparation, runtime coordination and interaction with external tools. The GPU may execute intense numerical kernels while the CPU oversees work and handles operations less suitable for parallel acceleration. Performance depends on their coordination rather than a contest in which one component replaces the other.
Poor sequencing or excessive transfers can leave expensive hardware idle. A systems diagram that shows only the model and the GPU omits network, storage, CPU orchestration, permission checks and result validation. The broader How Super Intelligence Works series asks readers to understand that complete chain.
17. Different accelerators and different workloads
GPUs are important but not the only hardware specialised for AI. Other devices may be designed for matrix computation, inference, low-power edge operation or particular numerical formats. The exact software and hardware ecosystem changes, so a generic article should not assume one manufacturer or product is universally preferred.
Compare supported operations, model compatibility, capacity, available memory bandwidth, toolchain maturity, measured workload cost and deployment constraints. A device suitable for a small sensor inference job can be inappropriate for distributed model training, and the reverse. ‘Accelerator’ describes a role in a system, not a single guaranteed speed.
18. Training and inference load hardware differently
Training typically performs forward computation, computes gradients and updates parameters. It needs storage for parameters, gradients, optimiser state and activations under the chosen method. Inference applies an already trained model to new inputs and may have different demands such as cached context, request batching and low-latency serving.
A chip’s performance on one stage should not automatically be used to estimate the other. Compare complete experiments under the same model architecture, numerical format, input size and operating constraints. Later curriculum chapters on distributed training and inference infrastructure examine those systems in more depth.
19. Why peak theoretical rates need caution
Hardware specifications may list peak operations under a particular numeric format and ideal workload. Real applications rarely sustain those peaks across every input shape and operation. Memory movement, synchronisation, software overhead and unsupported operations can reduce realised performance.
A useful benchmark measures the real task with actual dimensions and reports both the tested setup and limits. Repeating a vendor’s maximum number is not independent evidence that a specific AI application will achieve it. Treat the peak as a capacity descriptor, not a universal forecast.
20. Energy efficiency requires a whole-system denominator
An accelerator may complete a computation rapidly while drawing substantial power. Whether it is more energy-efficient depends on measured energy for an equivalent completed task, including relevant memory, CPU, networking and facility overhead within the chosen boundary. Watts measure instantaneous power, while watt-hours measure energy over time.
An invented illustration: a device draws 200 watts during a five-second operation, giving 200×5/3600 ≈ 0.278 watt-hours under a constant-power assumption. Another device draws 100 watts over twelve seconds, giving 0.333 watt-hours. Faster and higher-power can still use less task energy in this example, but actual measurements need appropriate baselines and idle-power accounting.
21. How a failed experiment should be reported
If the GPU program runs out of memory, report the error and the attempted allocation rather than presenting a hypothetical result as observed. If the runtime cannot access a device, do not pretend a CPU emulation measured GPU speed. If timing varies, record the conditions and repeat appropriately.
A comparison that omits failed runs or changes the benchmark after viewing results can create a misleading success narrative. Honest evaluation includes unsuccessful attempts, tool limitations, hardware details and the exact steps necessary to reproduce the claim where permitted.
22. Independent practice: two-stage processing
A fictional image-processing pipeline transfers data to a GPU in seven milliseconds, performs one operation in five milliseconds and returns the result in three milliseconds. The CPU completes the same full operation in thirteen milliseconds. Which path is faster for one operation? Calculate using all stages, not only the kernel.
Now assume three operations can reuse the uploaded image and a single final output is returned. Compute the GPU total and compare with three sequential CPU jobs. Finally change the input on every operation and explain why reusing the old transfer calculation would be invalid.
23. Answers to the independent exercise
One GPU operation takes 7+5+3 = 15 milliseconds, which is two milliseconds slower than the stated CPU thirteen. With three operations on one uploaded input, GPU time is 7+3×5+3 = 25 milliseconds. Three CPU jobs take 3×13 = 39 milliseconds, so the GPU wins by fourteen in this invented reused-input case.
If every operation needs its own input and output, three sequential GPU jobs take 3×15 = 45 milliseconds, which exceeds the CPU’s thirty-nine. The result changes because the movement assumptions changed. The correct reasoning is to draw the data flow and recompute the full route.
24. A procurement and engineering checklist
Before selecting hardware, state the task and success criteria. Measure the complete application, identify which component limits performance, and estimate memory demand including weights, activations, temporary buffers and other runtime allocations. Examine how often data must cross memory or device boundaries.
Test a representative workload, document numerical formats and tool versions, compare latency, throughput, energy and total cost where relevant, and keep the validity boundary visible. A GPU’s reputation is not evidence that this exact workload benefits. Decisions should be based on observed performance and practical constraints.
Applied accelerator decisions: when the faster kernel loses
The difference between throughput and response time
A system may complete many requests per second while an individual request waits in a queue. Throughput and latency are different measurements; improving one may worsen the other under a new batch size. For an educational chatbot that must reply during a lesson, long delays matter even when aggregate batch processing is efficient.
Report both using clearly bounded measurements. A hardware headline about processed tokens per second cannot by itself tell a parent how quickly a particular explanation will appear after retrieval, scheduling, inference and output checking.
Batching has a limit determined by memory
Batching multiple requests may improve hardware utilisation by sharing suitable work and reducing some fixed overhead. But a larger batch can require more activation memory and intermediate buffers. At some point the available memory or latency requirements constrain further growth.
Compare the total job with realistic input lengths and output limits. The best batch size depends on the model, application and device; the exercise cannot determine it from a GPU model name alone.
A serial stage can cap the benefit of acceleration
Imagine a fictional workload in which twenty percent of time belongs to a serial stage that cannot be accelerated, while eighty percent belongs to a parallel stage. Even if the parallel stage became infinitely fast, the full workflow would still require at least the original twenty-percent serial portion. That is the intuition behind Amdahl’s law.
Under this simplified assumption the maximum theoretical speed-up is 1/0.2 = five times, not infinite. Real performance is further constrained by transfers, synchronisation and other stages. The calculation encourages readers to measure the full pipeline before purchasing faster arithmetic hardware.
A small workload can run better on an ordinary CPU
Very small datasets may not contain enough parallel arithmetic to compensate for launching a GPU kernel and preparing its inputs. A CPU that already holds the information may complete the job without the extra movement. The outcome varies with architecture, software and how much work is reused.
Rather than claiming one platform always wins, benchmark the complete task under the actual intended conditions. A familiar Python loop and a highly tuned GPU library are not equivalent implementations by default, so tests should state the method and its limitations.
Numerical precision needs an acceptance test
Lower-precision arithmetic can reduce storage and accelerate supported operations, but rounding and quantisation may change outputs. Sometimes small numerical differences are irrelevant to the user’s task; sometimes they change a classification threshold or accumulation over many steps.
Test representative and challenging inputs against an appropriate reference. A faster result is not automatically a better result if its numerical error alters a consequential decision. Accuracy and performance should be measured separately and reported together.
Memory locality can matter more than arithmetic count
A program that repeatedly reloads data from a slower memory level may spend much of its time waiting rather than multiplying values. Reusing data in suitable caches or shared memory can change performance, depending on access patterns and capacity.
NVIDIA’s CUDA programming documentation describes global, shared and other memory spaces. Their existence does not mean a developer can ignore correctness, access bounds or synchronisation. A successful optimisation must preserve numerical results while improving a measured bottleneck.
Multiple accelerators introduce communication costs
Splitting a large model across several devices may solve a memory-capacity problem or enable additional parallelism. It also creates work moving information or synchronising updates between devices. Some tasks scale well; others are dominated by communication or serial stages.
A useful performance report specifies the number of devices, interconnect, workload division and total elapsed time. Dividing one device’s time by the device count is not a valid general speed-up prediction.
Training and inference are not interchangeable benchmarks
Training repeatedly computes predictions, gradients and parameter updates, usually maintaining more state than a simple forward inference run. Inference produces outputs from existing model parameters but may need memory for context, caches and concurrent requests. Their preferred hardware and performance measures can differ.
A device advertised as excellent for one training operation may not have the best end-user latency for a specific inference application. The evidence must match the job, data type, input size and configuration.
What a measured watt figure does not reveal
Power is a rate of energy use, while energy equals power integrated over time. If a fictional computation draws 200 watts for two seconds, its simplified energy use is 400 joules. Another configuration drawing 300 watts for one second uses 300 joules, despite its higher instantaneous power.
Those examples are arithmetic exercises, not measured accelerator behaviour. Real energy analysis should consider the whole system, idle periods, cooling and data-centre overhead where relevant. Lower instantaneous power is not automatically lower energy per completed task.
Why a data-centre view changes the question
A GPU does not operate in isolation. It needs power delivery, cooling, networking, host machines, storage and operational support. A hardware choice that improves one workload’s arithmetic speed may increase cost or complexity elsewhere.
For educational readers, separate what the accelerator does from what the service must provide. A website’s answer quality depends on sources and verification, while availability depends partly on systems that keep computation running. The CPU, GPU and network are parts of one operational chain.
Explain a benchmark result without inventing causation
If a new implementation becomes faster, possible causes include better parallel work distribution, less data movement, changed precision, improved software or different input conditions. The result alone does not prove which cause dominated. A careful benchmark isolates changes where practical.
Record model and library versions, timing scope, warm-up, batch size and input shapes. A percentage improvement presented without those details may be correct for the experiment while remaining misleading for a different user’s job.
The distinction between advertised and achieved rates
Peak theoretical compute and memory-bandwidth specifications describe idealised capabilities under defined assumptions. Applications usually achieve a different rate because of workload patterns, dependencies, utilisation and the surrounding software. Comparing two headline specifications is not an application benchmark.
Use a measured end-to-end task and analyse the difference between theory and observation. The gap can reveal a genuine optimisation opportunity, but it can also expose an unhelpful task for that hardware.
An independent diagnostic exercise
Suppose the fictional Number Tiles task is repeated ten times with the initial data uploaded once and the output returned after the final operation. Under the article’s original teaching costs, the GPU route is 8 + 10×4 + 5 = 53 milliseconds. Ten separate CPU operations at twelve milliseconds each take 120 milliseconds.
Change one rule: every GPU operation now needs a new upload and immediate returned result. The sequential GPU route becomes ten times 17 milliseconds, or 170 milliseconds. The result reverses because the data-flow assumption changed, not because the device suddenly became slower.
The transfer lesson for choosing hardware
The two calculations illustrate why an accelerator can be valuable for reused large work but disappointing for tiny, independent tasks. The right questions are how much work exists, what data remain resident, which stage limits elapsed time and what accuracy the application needs.
No amount of parallelism grants authority to publish, proves a source claim or guarantees model reliability. Those responsibilities remain in the wider application, where evidence and human control must be evaluated separately.
Applied analysis: end-to-end limits, failed runs and independent transfer
Serial work caps speedup
A quarter of a workflow may be inherently sequential. Even if the other three-quarters could be made infinitely fast, the complete job would be at most four times faster. This is a simplified expression of Amdahl’s law: end-to-end improvement is limited by the portion that cannot be accelerated.
With four workers under an ideal parallel model, the normalised time is 0.25 + 0.75/4 = 0.4375, for a speedup of about 2.29. With eight, the time is 0.25 + 0.75/8 = 0.34375, giving roughly 2.91. Real communication overhead may reduce both gains.
Cold starts and warmed devices
A first application request may compile code, allocate buffers and load model parameters. Later calls can reuse some of that work. Measuring only the first cold request or only the warmed kernel describes different user experiences, neither of which is universally superior.
A classroom demo used once might care greatly about startup. An always-on processing service may care more about sustained throughput. A defensible benchmark reports when the timer starts, whether caches and allocations were warm, and whether both compared systems were tested under equivalent conditions.
Asynchronous calls and honest timing
A GPU computation may be queued asynchronously while the CPU proceeds. A timer that stops after the command is submitted can report a short duration even though the device has not completed the operation. The measured time then reflects submission rather than output availability.
Use suitable completion or synchronisation mechanisms and inspect the result before calling a task finished. This mirrors a key lesson of reliable AI tools: an accepted operation, a completed computation and a verified answer are not the same state.
Fragmentation and memory headroom
The invented 11.5-GiB allocation leaves just 0.5 GiB under the toy device budget. That narrow margin does not promise a successful real run. Runtime allocations, caches, fragmented memory, driver needs and other processes may reduce usable room.
Measure actual peak usage with the intended framework and input. If the device fails to allocate a buffer, do not claim success on the strength of a spreadsheet estimate. Adjust batch size or allocation strategy and retest the complete job.
The batch-size trade-off
A larger batch can improve accelerator utilisation, but assembling it may require waiting for enough requests. That could improve throughput while increasing the time until the first user gets a response. The correct batch size therefore depends on the service objective.
Test latency across realistic arrival patterns, not only batches already prepared in memory. Report queue time, execution time and output time separately. The same hardware can behave differently in a private study application and in a high-volume public service.
Numerical precision versus correctness
Fewer bits per value may reduce storage and data movement and enable faster supported operations. Yet rounding and numerical range can change computed values. Near a decision threshold, a small change may alter which category a model selects.
Benchmark answer quality alongside speed. Some tasks tolerate approximate calculations, while others need exact arithmetic or greater numerical precision. A memory saving alone is not evidence that a transformed model remains reliable for every user request.
Data location makes or breaks reuse
The four-operation GPU win in Number Tiles assumes the same data remain available on the accelerator. If the inputs live on disk or another machine, or each step requires new files, that assumption no longer holds. A memory-resident benchmark can conceal the cost of fetching the data.
Draw the full path from storage and network to CPU memory, device memory, computation and user output. Count only transfers that the workload actually performs. Then change one data-location assumption and recalculate to check whether the result transfers.
Queues can outweigh a faster kernel
Suppose a fast kernel requires four milliseconds but must wait behind a thirty-millisecond queue. The request already spends at least thirty-four milliseconds before output handling. A slower device with no wait could complete the whole job sooner.
A service benchmark must examine delays and variability, not just arithmetic throughput. Rate limits, synchronisation, network calls and shared-resource contention all affect the user’s observed experience. Component speed cannot replace an end-to-end measurement.
Energy per completed task
Power measured in watts is not the same as energy measured in watt-hours. A fictional device drawing 200 watts for five seconds consumes about 0.278 watt-hours under a constant-power assumption. Another drawing 100 watts for twelve seconds consumes about 0.333 watt-hours.
The high-power device can use less energy for that one invented job. Real measurements require power profiles, idle baselines, completed outcomes and a defined system boundary. Do not infer energy superiority from nominal chip wattage alone.
Correct output before performance claims
A GPU kernel can run quickly and still produce an incorrect answer due to a programming bug, data race, numerical issue or mistaken input. A benchmark must check correctness before rewarding speed. A failed or incomplete task is not a successful acceleration.
Use small hand-verifiable cases, compare against a trustworthy reference implementation and define suitable tolerances. Include timeouts and out-of-memory errors in the recorded results. Silently excluding failures can make a system appear more efficient than it is.
A second memory practice problem
A fictional program needs five gibibytes of weights, three of activations and two of workspace: ten gibibytes before unspecified overhead. It fits within a simplified twelve-gibibyte budget. If a new input doubles activations to six, the total becomes thirteen gibibytes and no longer fits.
This changed-context test shows why a previous result cannot be copied into a new setting. Recompute the allocations, check peak simultaneous use and verify the actual runtime. The correct conclusion is conditional, not a promise of real deployment.
Selecting hardware for a user’s goal
A decision may involve latency, throughput, output quality, cost, accessibility, maintainability and risk. The fastest fictional accelerator can be the wrong choice if a simpler processor completes the actual task within the deadline at lower cost and with clearer error handling.
Keep the benchmark scope precise and use measured workloads rather than broad hardware reputations. In a teaching context, the most valuable result is the habit of identifying assumptions, bottlenecks and evidence before making a claim about what a system can do.
25. Sources and the next learning route
NVIDIA’s CUDA Programming Guide and GPU memory and kernel documentation describe the memory hierarchy and programming model for CUDA hardware. These official materials explain mechanisms; they do not validate any fictional timing or memory figure in Number Tiles.
Continue with eduKate’s Chips, Memory and the Geography of Computing, Code Execution, and the How Super Intelligence Works curriculum. The How X Works hub and Gold Standard of Education extend the mechanism into learning and real decisions.
All transfer times, power figures, model allocations and device characteristics in this chapter are invented numerical teaching fixtures. No real GPU, CPU or vendor product was benchmarked or purchased. Source review: 22 October 2026.
Fast execution does not certify the surrounding workflow. Connect this hardware explanation to Alignment, Oversight and Human Control for legitimate tool actions and Evaluations for test conditions and user outcomes. Return to How Super Intelligence Works for the complete reading route.