Did you know? A moving ball can look almost identical in two photographs even though it changed position. If we compare two nearby video frames carefully, we can estimate the direction and apparent distance that parts of the picture have moved. That estimated field of movement is called optical flow.
Quick answer: take two video frames → locate comparable visual patterns → estimate horizontal and vertical displacement → assemble motion vectors → check errors caused by camera movement, blur or occlusion. Modern machine learning can help, but optical flow is not always produced by AI; classical computer-vision algorithms also exist.
Imagine filming a small red ball rolling across a desk. In Frame A, the ball appears near the left edge. In Frame B, it appears farther right. The visual change is obvious to you. Optical-flow methods try to assign numbers to that apparent motion.
The 30-second guide
- Frame: one image from a video sequence.
- Optical flow: a two-dimensional field of estimated apparent motion between frames.
- Motion vector: a horizontal and vertical displacement, commonly represented as (u, v).
- Sparse flow: estimate displacement for selected keypoints, such as trackable corners.
- Dense flow: estimate a displacement for many or all pixels under a given method.
- Camera motion: movement of the recording device that can affect apparent motion across the entire image.
- Occlusion: when a surface becomes hidden and a matching visible point may disappear.
A vector in an optical-flow field does not automatically measure physical metres per second. Its units often begin as pixels per frame or another image-coordinate displacement.
A tiny worked example
Suppose an identifiable dot is at image coordinates (5, 3) in one frame. In the next frame, its matching location is estimated at (8, 7).
The estimated horizontal displacement is 8 − 5 = 3 pixels; vertical displacement is 7 − 3 = 4 pixels. The displacement magnitude is √(3² + 4²) = 5 pixels.
These numbers are fictional and use one simple image-coordinate convention. They describe apparent movement in the video. They do not tell us the dot’s true speed through a room without timing, geometry and additional assumptions.
Step 1: Choose frames close enough in time
Optical flow is frequently estimated between consecutive or nearby frames. A large gap may allow an object to move so far that matching the correct feature becomes difficult.
If a ball suddenly disappears behind a book, the later frame may not contain the same visible point. A model can still estimate movement in some cases, but the task becomes more uncertain.
Frame rate and motion blur matter as well. Even a powerful learned method cannot guarantee a correct path when the relevant evidence was never captured.
Step 2: Identify visual features that can be tracked
Some classical methods begin with distinctive points, such as corners or textured regions. A plain white wall contains little local detail, making it difficult to tell which part of one frame corresponds to which part of another.
A checkerboard corner is easier to recognise across nearby frames because its neighbourhood contains distinctive contrast.
The OpenCV optical-flow tutorial demonstrates using corner detection to choose points for tracking and then applying Lucas–Kanade optical flow.
Step 3: Estimate local motion
The classical Lucas–Kanade approach makes assumptions about nearby image structure and motion. One important approximation is that corresponding visual patterns retain similar brightness over a small interval.
The method uses a local region rather than relying on a single ambiguous pixel. A system can then estimate how the region moved between the two frames under the model’s assumptions.
Those assumptions can fail when lighting changes sharply, a surface is hidden or the displacement is too large for the chosen setup. This is why an optical-flow estimate is not a photographic measurement of physical movement.
Step 4: Why a moving pyramid can help
A large displacement may be difficult to estimate when working only at full resolution. Some methods build an image pyramid: several versions of the same frame at progressively coarser scales.
At a lower resolution, a large apparent movement may become a smaller motion in pixel units, making an initial estimate easier. The method can then refine it at finer scales.
OpenCV’s pyramidal Lucas–Kanade function provides an implementation for tracking selected points in video. Its success still depends on scene conditions, parameter choices and data quality.
Step 5: Sparse optical flow versus dense optical flow
Sparse flow tracks selected locations. For example, it may estimate motion at ten clearly visible corners of a moving book.
Dense flow aims to assign motion throughout much more of the image. In OpenCV, the Farnebäck method provides a classic dense-flow implementation, generating horizontal and vertical estimates across the frame.
Dense flow can support visualising the overall movement pattern, but there is more computation and potentially more unreliable estimation in low-texture regions. “Dense” does not mean “perfect.”
Step 6: What do the arrows in an optical-flow display mean?
A flow visualisation might draw arrows over the image. Their direction represents estimated image motion, and their length may represent displacement magnitude under the display’s chosen scale.
If a ball rolls to the right while the camera stays still, vectors on the ball may point generally rightward. But if the camera itself pans to the right, many apparently stationary background features can move leftward in image coordinates.
The display is about apparent image motion, not automatically about which real-world object initiated the movement.
A moving camera can make a stationary scene appear to move. Optical flow measures the visual displacement first; the cause needs separate interpretation.
Step 7: Why a moving camera complicates everything
Imagine recording a stationary bookshelf while walking sideways. The books move across the video even though none of them moved relative to the shelf.
Optical flow will show apparent movement because the camera’s position changed. A more sophisticated system can try to estimate camera motion and distinguish it from independently moving objects, using suitable geometric assumptions and observations.
That distinction is useful in robotics, video stabilisation and navigation research. But optical flow alone is not a full three-dimensional reconstruction or proof of what moved in the physical world.
Step 8: The aperture problem
Suppose a long, straight edge moves in a video. Looking only at a tiny section of that edge may reveal motion perpendicular to the edge but not enough evidence about movement along the edge.
This is known as the aperture problem. A local view can make different full movements appear similar.
Texture, multiple surrounding features, temporal information and modelling assumptions can help. Still, where visual evidence is ambiguous, there may be no uniquely correct estimate from the available frames.
Step 9: How neural networks learn optical flow
AI-based methods can learn to estimate motion from pairs of images using appropriately constructed training and evaluation data.
One important learned architecture is RAFT—Recurrent All-Pairs Field Transforms. A model uses visual feature information, correlations between locations and iterative updates to produce a flow estimate.
The official Torchvision RAFT optical-flow example demonstrates that a flow model takes two images and predicts a two-channel field containing horizontal and vertical displacement values.
The method can perform strongly under tested conditions, yet still make mistakes with occlusion, unusual images or motion that differs from its training distribution.
Step 10: How is a learned optical-flow model trained?
Some supervised datasets provide a reference flow field generated from suitable sources, including controlled synthetic scenes. A network can compare its estimated vectors with reference motion and adjust its parameters.
Other learning approaches use photometric consistency and additional assumptions rather than full reference flow labels for every training example.
Training objective and labels matter. If a scene contains inconsistent lighting or surfaces disappear, a simple brightness-based loss can be misleading.
Step 11: How do engineers test flow accuracy?
When a suitable reference flow is available, one common error measure is the endpoint error, which compares predicted displacement vectors with reference vectors.
For example, suppose the verified reference displacement is (3, 4) pixels, but a model estimates (2, 4). The Euclidean difference is √[(3 − 2)² + (4 − 4)²] = 1 pixel.
This fictional calculation shows how a vector can be wrong even when its general direction looks sensible. A good evaluation also examines difficult motion boundaries, rare scenes and missing correspondence.
Step 12: Optical flow is not object detection
Object detection identifies the location of selected items, often by bounding boxes. Optical flow describes how image information appears to move from one frame to another.
A system can use both. For example, one component detects a bicycle while another estimates motion around it. An object tracker may then attempt to associate the same bicycle through a longer sequence.
But these methods have different failure modes. A detected object might be correctly classified while the optical flow around it is unreliable, or vice versa.
Step 13: Optical flow and human pose estimation
A pose model estimates landmarks such as wrists and elbows. Optical flow estimates changes in visual positions. They can support one another when an application needs to interpret motion.
A hand can move without changing its finger configuration, and a static gesture can be visible without motion. Recognising a gesture’s meaning requires additional modelling beyond estimating pixel movement.
For the landmark task, read How AI Recognises Hand Gestures and Body Poses.
Step 14: What about video compression?
Motion estimation is also important in video encoding. A compression system can use motion-related predictions to represent changes between frames instead of storing each frame entirely independently.
Optical-flow and motion-compensation concepts are related, although an actual video codec may use different block-matching or optimisation procedures. They should not be treated as identical algorithms.
This shows why computer vision and ordinary video engineering share mathematical ideas even when a task does not use a modern neural network.
Step 15: Practical uses of optical flow
- Video stabilisation research: analyse apparent camera or scene motion.
- Visual tracking: estimate how selected features move across nearby frames.
- Robotics: support motion estimation as one input to a wider navigation system.
- Computer vision experiments: visualise direction and magnitude of movement.
- Video processing: help understand temporal relationships between images.
These applications require separate safeguards. Optical flow cannot by itself establish identity, motivation or whether a scene is safe.
Try it with two drawings
Draw a small ball at coordinates (2, 2) on one paper grid. On a second grid, draw it at (6, 5). Calculate the displacement: (4, 3) and magnitude 5 grid units.
Now draw an entirely stationary table behind the ball. For a second exercise, pretend you moved the camera instead: shift the whole table and ball in the same direction. The observed image movement changes even though the physical story could be different.
This illustrates why optical flow must be interpreted in context.
Five mistakes worth avoiding
- “Optical flow is always AI.” Classical algorithms exist.
- “A flow arrow proves an object moved in the real world.” Camera motion can create apparent movement.
- “Dense flow means perfectly correct pixels.” Texture, occlusion and lighting still cause errors.
- “Flow gives metres per second directly.” Physical speed needs scale, timing and geometry.
- “One video comparison explains every action.” High-level activity and intention are separate tasks.
Frequently asked questions
What is optical flow in simple words?
It is an estimate of how visual patterns move between images, usually nearby video frames.
What is the difference between sparse and dense optical flow?
Sparse methods estimate motion at selected points; dense methods estimate displacement more broadly across an image.
Does optical flow measure actual physical speed?
Not by itself. It typically estimates image displacement, which must be interpreted using timing and geometry.
Can AI estimate optical flow?
Yes. Learned approaches such as RAFT estimate movement from image pairs, alongside classical methods such as Lucas–Kanade and Farnebäck.
Why can an optical-flow model fail?
Fast motion, occlusion, camera changes, weak texture and unfamiliar scenes can make correspondences unreliable.
Your one-minute takeaway
Optical flow measures estimated apparent movement between video frames. It helps connect pictures through time, but camera motion and uncertainty complicate interpretation. Classical algorithms and learned AI methods both contribute to the field, and trustworthy use requires careful evaluation.
Trusted reading and connected articles
- OpenCV: Optical Flow Tutorial — sparse Lucas–Kanade and dense Farnebäck methods.
- OpenCV: Video Tracking Functions — practical optical-flow functions.
- Torchvision: Optical Flow With RAFT — learned per-pixel motion.
- How AI Recognises Objects and Actions in Video — the wider video-understanding task.
- How AI Recognises Hand Gestures and Body Poses — movement and keypoints.
- How Robots Navigate Around Obstacles — combining perception with navigation.
- How AI Works: Complete Overview — more short educational guides.