GPUs and other accelerators make AI computation practical by handling large numbers of mathematical operations with considerable parallelism. Their usefulness comes from matching the workload to the hardware and moving data through memory efficiently. A headline measure of arithmetic throughput alone cannot predict how quickly an entire AI application will answer a question.
This chapter follows an invented Array Lab computation with exact matrix dimensions, storage requirements and a bandwidth estimate. You will see why an array can occupy 16 MiB, why an elementary addition may move 48 MiB, and how compute, memory, precision and software affect performance. It fills topic 081 of How Super Intelligence Works, beginning the series’ infrastructure section.
Super Intelligence is the site’s editorial umbrella for analysing AI systems, not a claim that current processors create artificial superintelligence. The goal is to understand a mechanism and carry the reasoning to a changed data shape, not to make unsupported claims about a particular manufacturer’s product.
Contents and reader routes
Start with the processor · Open Array Lab · Verify byte counts · Try the independent case · Sources
- 1. Why an AI workload uses an accelerator
- 2. CPU, GPU and specialised accelerators have different jobs
- 3. Parallelism begins with independent work
- 4. The fictional Array Lab data packet
- 5. Calculate how many bytes one array occupies
- 6. Work out a minimal elementwise data path
- 7. Arithmetic throughput is not memory bandwidth
- 8. The bottleneck depends on the operation
- 9. Registers, shared memory and global memory
- 10. Coalesced access and data locality
- 11. Matrix operations and dedicated units
- 12. Training and inference have different resource needs
- 13. Weights are only one part of model memory
- 14. Precision choices affect memory and error
- 15. Batch size and latency trade-offs
- 16. Transfers between host and device
- 17. The role of kernels and launch overhead
- 18. Utilisation, concurrency and scheduling
- 19. Distributed computation adds communication
- 20. Energy and thermal limits
- 21. Independent transfer on a changed array
- 22. Full answers and repair checks
- 23. FAQ for students and builders
- 24. Sources and continuation routes
1. Why an AI workload uses an accelerator
A GPU is a processor designed to run many similar operations concurrently, which makes it useful for parts of machine learning that involve large arrays, matrices and repeated numerical transformations. An accelerator is a broader term for hardware specialised to execute particular operations efficiently. Neither is a thinking machine on its own; models and applications supply the mathematical workload.
The user-visible answer from an AI system rests on several layers: encoded inputs, model parameters, computation, memory, software orchestration and sometimes external tools. Hardware influences how quickly and at what resource cost those operations can run. The result’s factual correctness still depends on model capability, data and verification.
To understand performance, do not start with a marketing claim that a chip has more of something. Start with the operation: how many values must be processed, how much data must move and what operations can run independently? Those quantities let us reason from first principles.
2. CPU, GPU and specialised accelerators have different jobs
A CPU typically handles varied control tasks, branches and operating-system work efficiently. GPUs organise many threads to execute data-parallel work. Other accelerator designs may emphasise particular matrix operations, numerical formats or inference characteristics. The best choice depends on the workload, its memory and the surrounding system.
A system may use both CPU and GPU. The CPU prepares input and coordinates work; the GPU performs a collection of numerical kernels; other services retrieve data or verify external results. A successful inference therefore cannot automatically be attributed to the accelerator alone.
An accelerator’s advertised operations-per-second figure may describe a specific precision or kernel rather than an entire application. Ask which numeric format, matrix shape, batch size, software stack and power condition produced the number before comparing devices.
3. Parallelism begins with independent work
Consider adding a million pairs of numbers. Each pair can often be processed independently, so the work can be split into many threads. This is a natural example of data parallelism. A different task with a chain of steps where each depends on the previous result offers fewer opportunities for the same kind of simultaneous computation.
AI networks contain both parallel and sequential components. Multiplications over many matrix entries can be highly parallel, while successive dependent model layers constrain how the complete computation is scheduled. Input preparation, memory movement and synchronisation also take time.
A useful learning question is therefore not ‘How many cores does the GPU have?’ but ‘How much independent work is available at once, and can the system feed it with data?’ Very large theoretical capacity can sit idle if the workload or memory path does not support it.
4. The fictional Array Lab data packet
Our fictional Array Lab contains two arrays X and Y with 2,048 rows and 4,096 columns each. Every element is assumed to occupy exactly two bytes, corresponding to a simplified 16-bit storage representation. The planned operation is Z = X + Y, an elementwise addition producing an output with the same shape and element size.
Nothing here describes a commercial chip or a measured benchmark. The data are fabricated for arithmetic practice. We assume dense arrays without headers, padding or compression; ignore caches and launch overhead when calculating an initial traffic lower bound; and treat one output value as the sum of the corresponding pair of inputs.
The exercise asks how much storage the arrays require, how many bytes an ideal one-pass computation would read and write, and why a nominal bandwidth figure cannot certify observed runtime.
5. Calculate how many bytes one array occupies
The array contains 2,048 × 4,096 = 8,388,608 elements. At two bytes per element, its storage is 16,777,216 bytes. A mebibyte is 1,048,576 bytes, so the array occupies exactly 16 MiB under these assumptions. Two input arrays occupy 32 MiB, and a third equally sized output occupies 16 MiB.
Thus the three dense arrays occupy 48 MiB of payload in aggregate. A real programme may allocate additional buffers, alignment padding, temporary tensors or runtime bookkeeping. This simple calculation is a capacity starting point, not a measurement of peak device memory.
If a programmer inadvertently stored each element using four rather than two bytes, each array would instead occupy 32 MiB and the same three-array arrangement would use 96 MiB of payload. Representation choices have real consequences even before we discuss mathematical accuracy.
6. Work out a minimal elementwise data path
For Z = X + Y, an ideal streaming pass that loads each value of X and Y once and writes each value of Z once transfers two input arrays and one output array: 16 + 16 + 16 = 48 MiB. Each output element requires a numerical addition, so there are 8,388,608 conceptual additions.
This ’48 MiB’ describes a simplified amount of data moved between the relevant storage and compute path. Caches and reuse can change actual memory transactions; host-to-device transfer can add data movement; device allocation and kernel startup can add overhead. The arithmetic is not a measurement of total bus traffic.
One safe conclusion is that the task includes a substantial data-movement requirement relative to a single arithmetic operation per element. Whether it is memory-bound depends on hardware characteristics and how the kernel is implemented.
7. Arithmetic throughput is not memory bandwidth
Compute throughput describes the rate of arithmetic operations under particular conditions. Memory bandwidth describes a rate of data movement between specified levels of a memory hierarchy. A kernel can be limited by either, or by another bottleneck such as synchronisation, instruction dependency or launch overhead.
For a deliberately hypothetical device bandwidth of 12 GiB per second, 48 MiB divided by 12 × 1,024 MiB per second gives 0.00390625 second, or about 3.9 milliseconds. That is only an idealised traffic-time estimate at the assumed rate. It ignores kernel launch, transfer direction, contention and achieved throughput.
If an answer calls 3.9 milliseconds the actual runtime, it has mistaken a simplified lower-bound calculation for an observation. The distinction between theoretical constraints and measured outcomes should remain explicit in every hardware discussion.
8. The bottleneck depends on the operation
Arithmetic intensity compares numerical operations with bytes transferred. For our elementwise addition, the simplified count is one addition per output value and six bytes associated with reading two two-byte inputs and writing one two-byte output. The notional intensity is therefore one addition per six bytes, before cache or other effects.
A matrix multiplication can reuse values and perform many operations per data element when appropriately tiled. That changes the relationship between arithmetic and memory movement. Such an operation may take better advantage of a GPU or specialised matrix accelerator, subject to shape, precision, scheduling and implementation.
Performance analysis begins with recognising which resource the algorithm stresses. More arithmetic units may do little for a workload stalled on data movement. More memory bandwidth may do little for a kernel that cannot expose enough parallel work.
9. Registers, shared memory and global memory
GPU programming exposes a memory hierarchy. Registers are private resources associated with executing threads, while shared memory can be used by threads cooperating within a thread block. Global device memory is accessible more broadly and typically has greater capacity but different latency and bandwidth characteristics. The exact details depend on architecture.
A programme may stage frequently reused values into a faster local memory space rather than repeatedly fetching them from global memory. The benefit depends on access pattern, available capacity, synchronisation and the cost of staging. More fast-memory allocation is not automatically better if it prevents enough work from running concurrently.
NVIDIA’s CUDA programming guide describes these memory spaces and their scopes. The relevant educational point is that ‘memory’ is not one uniform box: where data live and how threads access them changes performance.
10. Coalesced access and data locality
Neighbouring GPU threads often run more efficiently when their global-memory requests can be combined into well-aligned transactions. Scattered accesses may move extra bytes or generate additional transactions. A mathematically identical operation can therefore perform differently when its data are laid out differently.
Consider a matrix stored row by row. Adjacent threads traversing adjacent entries in that layout can sometimes use the memory system more effectively than threads jumping between distant columns. The exact result depends on kernel mapping, alignment, caches and hardware, so the pattern must be measured.
A useful independent test changes only the access pattern while preserving the calculation and inputs. Confirm the result is numerically correct in both variants before attributing any timing difference to memory behaviour.
11. Matrix operations and dedicated units
Matrix multiplication contains opportunities for extensive data reuse and many arithmetic operations. Specialised hardware units may accelerate particular matrix shapes and numeric formats. That does not mean every model workload spends all of its time in those units or that every shape reaches a peak advertised rate.
Preparation, normalisation, attention, movement of activations and communication can influence total execution time. Model architecture and implementation determine the actual mixture. Some kernels may be limited by capacity, some by bandwidth and others by arithmetic or synchronisation.
An accelerator comparison should therefore examine representative end-to-end tasks alongside component kernels. A chip’s theoretical peak can establish a limit for a particular operation; it cannot by itself report the observed speed of an entire user request.
12. Training and inference use different resources
Training repeatedly applies a model to examples, computes a loss and adjusts parameters through gradients. It commonly needs memory for weights, activations, gradients and optimiser state, depending on the method. Inference uses trained parameters to produce outputs and may have different activation, cache and batching needs.
A model fitting in memory for inference may not fit with the same settings for training. A numerical format that saves inference memory may require additional care during optimisation. Consequently, the phrase ‘the model fits on this device’ needs a workload qualifier.
An application also carries tokenizer, retrieval, context, output buffering and tool costs that may sit outside the accelerator. Hardware sizing should match the full use case rather than only a model’s nominal parameter count.
13. Weights are only one part of model memory
Suppose a fictional model contains one billion parameters, and each parameter is stored using two bytes. Its raw parameter payload is two billion bytes, approximately 1.86 GiB under binary units. That does not include runtime buffers, intermediate activations, attention caches, framework allocations or replication overhead.
For inference, sequence length and batch size can increase activation and cache requirements. For training, gradients and optimisation structures can make the memory demand substantially larger than raw weights. The exact factor cannot be asserted without specifying architecture, optimizer, numerical formats and implementation.
This illustrates why a device’s advertised memory capacity cannot be compared blindly with a model’s weight-file size. Capacity, achievable speed and numerical quality must each be assessed for the intended workload.
14. Precision choices affect memory and numerical error
Our Array Lab uses two-byte elements. A four-byte representation would double the payload per array from 16 MiB to 32 MiB. Some numerical formats support faster operations on suitable hardware, but reduced precision can also change rounding and error behaviour.
Model developers choose formats with attention to training stability, inference accuracy and hardware support. Quantisation may reduce storage using fewer bits per parameter, but scales, metadata, intermediate precision and application quality complicate the actual total. A nominal four-bit weight does not imply exactly a quarter of end-to-end memory or identical outputs.
Students should distinguish numeric representation from mathematical meaning. Repeating a calculation in two formats and comparing results is more informative than assuming that every smaller type is always safe.
15. Batch size and latency trade-offs
Batching several requests may let hardware process more work efficiently, raising throughput. It can also introduce waiting time, require more memory and change how quickly an individual request completes. Low latency for one pupil’s question is not identical to maximum throughput for a large queue.
An interactive tutor may benefit from prompt response times, while an overnight analysis could favour throughput. Even within one service, preprocessing, context length and response length alter the work per request. A performance result without a batch and queueing description is incomplete.
Benchmark both requests per second and the time a user actually experiences, with the same quality checks and stopping criteria. Faster computation is not helpful if the application returns the wrong answer or waits unnecessarily before beginning.
16. Transfers between host and device
Computing on a GPU sometimes requires data to move between CPU-side memory and device-side memory. These transfers can take meaningful time relative to a very short kernel. Where supported and appropriate, reusing device-resident data can reduce unnecessary traffic.
Take the fictional array example. Moving X and Y onto the device and retrieving Z adds transport beyond the 48 MiB modelled for a simple device-side addition. Exact transport volume and timing depend on allocation, data residency, interconnect, overlap and caching; they should not be guessed from the kernel’s byte count alone.
A benchmark that times only the inner kernel answers a narrower question than a user waiting from upload to final download. State which boundaries are inside the timer before claiming a speedup.
17. Kernels and launch overhead
A GPU kernel is a unit of work launched on the device. A small calculation can spend a material fraction of elapsed time preparing and launching the kernel rather than doing arithmetic. That helps explain why a highly parallel processor may not be the best solution for a tiny dataset.
Combining operations into a suitable kernel can sometimes avoid intermediate transfers or extra launches. The change must preserve the intended arithmetic and be tested against independent expected outputs; optimising a wrong computation only produces incorrect results faster.
When comparing devices, record whether the benchmark includes data allocation, preparation, kernel launches, synchronisation and result collection. A timing labelled ‘GPU computation’ can describe different boundaries in different reports.
18. Utilisation, occupancy and the meaning of peak
An accelerator has finite resources for registers, shared memory and concurrent execution. A kernel that uses many resources per thread block may allow fewer blocks to run simultaneously. Another kernel may expose too little independent work to occupy the device. High occupancy can help hide some latency, but it is not a universal predictor of high performance.
Nominal throughput figures are measured under stated test conditions. Real tasks include varying array shapes, branching, memory access patterns, stalls and orchestration. A system might use its accelerator intensively for one stage while remaining idle during data loading or network waits.
An honest performance report includes achieved throughput under the actual workload, latency from the user’s perspective and relevant memory and power conditions. Do not equate device utilisation alone with educational usefulness or answer correctness.
19. Distributed computation adds communication
When one device cannot hold or process the workload effectively, systems may divide work across accelerators or machines. Data parallelism divides examples; model parallelism divides pieces of a model; other strategies distribute computation and communication differently. Every design introduces coordination costs.
Moving tensors between devices can reduce the benefit of extra arithmetic units. A faster local kernel may not speed up the complete training job if the network or synchronisation stage dominates. The right partition depends on memory, interconnect, model structure and workload size.
An engineer should calculate which information must cross a boundary and how often, then check actual transfer performance. More accelerators do not automatically imply proportional speedup, just as more pupils working on one answer do not help if everyone is waiting for a single shared book.
20. Energy, cooling and reliability
Accelerators consume electrical power and create heat that must be removed. Device workload, utilisation, hardware configuration and cooling influence energy demand. A power rating is not the same as a measured energy total. Energy equals power integrated over time under the relevant operating conditions.
Suppose a hypothetical device draws 250 watts steadily for two hours. Under that explicit assumption, the energy consumed is 500 watt-hours, or 0.5 kilowatt-hour, for that device. The calculation does not include host computers, networking, cooling or conversion losses. Nor does it claim that any real model produces 250 watts constantly.
The next curriculum chapter examines the data centre as a complete facility. That is where IT equipment power and supporting infrastructure must be combined and distinguished from energy cost, emissions and water use.
21. Independent transfer with a different array
Consider a second fictional laboratory with arrays containing 1,024 rows and 2,048 columns, stored using four bytes per element. Two inputs A and B are added elementwise to produce C of the same shape. Assume dense arrays, no padding and one load of each input plus one store of the output for an idealised traffic calculation.
Compute the number of elements in one array, bytes per array, total payload storage for A, B and C, and the idealised three-array traffic. Convert bytes to MiB using 1 MiB = 1,048,576 bytes. Then estimate a best-case transfer-only time at an invented bandwidth of 24 GiB per second.
Explain at least three reasons why this transfer-only time must not be published as actual end-to-end runtime. Finally decide whether increasing the arithmetic unit count necessarily accelerates this simple workload, given its low amount of arithmetic per element.
22. Transfer answers and validation
One array contains 1,024 × 2,048 = 2,097,152 values. Four bytes each gives 8,388,608 bytes, exactly 8 MiB. Three arrays require 24 MiB of payload, and the idealised one-pass read-read-write traffic is likewise 24 MiB. The arithmetic follows directly from the printed shape and element size.
Twenty-four GiB per second equals 24 × 1,024 = 24,576 MiB per second. Therefore 24 MiB / 24,576 MiB per second equals 0.0009765625 second, about 0.977 millisecond. This is a lower-bound-style estimate under an explicitly hypothetical sustained bandwidth; it is not a reported measurement.
Actual elapsed time includes kernel launch, synchronisation, cache effects, achieved bandwidth, host transfers, allocation and possible contention. More arithmetic capacity may have little effect if data movement dominates. The correct transfer lesson is to identify both the numerical work and the resource that limits it.
23. Frequently asked questions
Does a faster GPU always create a better AI answer? No. Speed and answer quality are different properties. A model, its evidence and verification steps still determine whether an answer is supported.
Is model size equal to GPU memory required? No. Weights, activations, caches, temporary data and runtime allocations add to the total. Training and inference can require substantially different memory configurations.
Does half precision always halve total memory? No. It halves raw storage per value compared with a four-byte format under the simplified arithmetic, but not every buffer uses the same representation, and other memory costs remain.
Do all AI tasks need a GPU? No. The correct hardware depends on computation, data movement, cost, latency and how much work can run in parallel. Simple exact calculations can be better served by a CPU or other appropriate tool.
From component benchmarks to a useful application decision
A hardware design decision should begin with the actual task mix. Imagine one fictional afternoon in which an education publisher runs five thousand short text classifications, prepares one long report and renders a handful of teaching diagrams. The workloads may differ more in memory, token length and latency needs than in their apparent subject. Buying or allocating a device based on just the largest task could leave the shorter tasks inefficient.
Next inspect the shape of each calculation. A matrix multiplication may reuse data many times, making arithmetic capability important; a simple elementwise operation may spend most of its time moving bytes. A programme that repeatedly transfers a small tensor to and from a device may be slowed by orchestration even when the device executes each individual kernel quickly. Count what happens across the full application.
Memory capacity and bandwidth must be kept separate. A device can have enough memory to store the arrays and still be slow moving them. Another may have excellent transfer rates but not enough capacity for the required model state. Paging, swapping or partitioning the workload can change the operating conditions, so a simple benchmark under one configuration should not be generalized without checking.
Different model stages also change the optimal policy. Training may require repeated gradient calculations and additional optimiser state; inference may depend on prompt length, generated-token sequence and request concurrency. A model that fits for a single request might exceed memory limits when many requests run together. Measure the relevant stage rather than extrapolating from file size.
Always label units. MB and MiB differ, and gigabytes are not gibibytes. A decimal gigabyte represents one billion bytes, while a gibibyte represents 1,073,741,824 bytes. When sizing a buffer near a memory limit, this distinction can matter. The exact byte count is often more reliable than a rounded marketing label.
Numerical speed and numerical validity are separate. Half-precision arithmetic may support useful speed or memory savings for a particular workload, but rounding, underflow, overflow and accumulation behaviour can affect results. A suitable test should compare the output with an appropriate reference and tolerance, not merely confirm that the kernel finished.
Finally, a measured throughput improvement is not an educational outcome. Faster generation of worksheets might help a teacher, but only if explanations, exercises and answer keys remain correct and can be reviewed. The system should report the speed under its measurement conditions, verify the produced content and preserve human judgement about what the pupils actually need.
24. Official sources and the wider learning route
The NVIDIA CUDA Programming Guide documents the GPU kernel execution model and memory spaces, including global, shared and local memory. Its current programming-guide index is the appropriate starting point for supported implementation details. Vendor documentation describes a particular software and hardware ecosystem; other accelerators have different interfaces.
Continue with Inference, Scaling and Code Execution for numerical work and its checking chain. Use the Super Intelligence curriculum, How X Works master hub and Gold Standard of Education for the larger learning route.
All Array Lab shapes, memory and speed values, device bandwidth figures and examples here are original fictional fixtures. The calculations are verified from stated assumptions. No accelerator benchmark, product test or energy measurement was conducted. Sources reviewed 22 October 2026.