How artificial intelligence works when tracking an object through a video becomes interesting at the precise moment the object disappears. Imagine a red football rolling across a playground. A student walks in front of it for one second. When the ball emerges on the other side, how could a computer decide whether it is the same ball instead of a different red ball?
Did you know that recognising a ball in each separate frame is not the same as tracking one ball through time? Object detection asks what is visible now. Tracking asks whether a newly detected object belongs to an earlier sequence of detections. When the object is hidden, the system has less evidence, so a reliable tracker uses motion predictions, appearance clues and cautious association rules rather than claiming to see through the student.
The useful sequence is video frames → object detections → assign temporary track identifiers → predict short-term movement → match new detections to existing tracks → handle gaps or occlusions → flag uncertainty or identity changes. This is called multi-object tracking when several objects are followed. Its aim is continuity, not a guarantee of permanent identity.
Five questions that reveal how tracking works
- What does the camera observe, and what is hidden?
- How does object detection differ from persistent tracking?
- What movement does the tracker predict between frames?
- How does it match a reappearing object to an old track?
- When should it stop claiming the track is continuous?
1. The everyday problem: objects do not stay visible
Watch people walking through a busy school corridor. Two students may pass behind a pillar, cross in front of one another or leave the camera’s field of view. A camera receives one image at a time. Between frames, the scene can change: positions, apparent sizes, lighting and visible surfaces are different.
A simple object detector might correctly find two students in one frame and two in the next but make no statement about which new box corresponds to which earlier box. Tracking adds that relationship. In controlled demonstrations, a tracker often draws coloured boxes and temporary numbers to show its estimated continuity.
Those numbers are not verified legal identities. They are working labels for objects in a video sequence. A tracked person can be incorrectly assigned a different number, or two individuals can exchange identifiers. When an object is fully hidden, the system’s location estimate is a prediction rather than a measurement of where it visibly is.
2. What does occlusion mean?
Occlusion occurs when one visible object blocks the camera’s view of another object. In partial occlusion, only some of the target is hidden. In complete occlusion, the target may disappear from view entirely. This is different from an object physically vanishing or leaving the scene.
Imagine a toy car moving behind a cardboard box. The car keeps moving even while the camera cannot see it. People can often predict where it will emerge because they remember its speed and direction and understand that the cardboard box has not destroyed the car.
An AI tracker can make a similar limited prediction by extrapolating movement, but it does not have perfect knowledge. The car might stop, turn, reverse or be picked up behind the box. As time passes without observation, uncertainty generally grows. A trustworthy system should not render a predicted location as though it had just been directly observed.
3. Detection and tracking answer different questions
A detector may identify a region as a bicycle, car, ball or person in each frame and produce a bounding box and confidence score. A tracker assigns an identifier and attempts to associate that object’s new detection with its earlier track. Many practical systems use tracking-by-detection: run a detector, then solve the association problem.
An object can be detected correctly but assigned to the wrong existing track. Conversely, a detector may miss an object for a few frames, while the tracking system maintains a tentative track based on a motion model. These are separate outcomes and should be measured separately.
Optical flow is another related technique that estimates how image patterns move between nearby frames. It can support tracking, but flow vectors and object identity are not equivalent. A moving shadow may create flow; a static-looking object could be tracked as the camera moves. Always ask which job a model is performing.
4. How a track starts, updates and ends
At the beginning, a tracker sees detections without history. It can start tentative tracks for possible objects. When detections remain consistent across subsequent frames, a track may become confirmed. The system stores a state such as position, approximate velocity, bounding-box dimensions, an appearance representation and the time of the last observation.
As each new frame arrives, the tracker predicts where a known object might appear. It then compares candidate detections against the predicted track states. A compatible detection updates the track. A missing detection may cause the tracker to retain the track temporarily or mark it as lost.
The track should eventually expire when evidence is insufficient. Keeping every track alive indefinitely produces wrong matches: a red ball that entered minutes later could inherit the earlier ball’s identifier. Ending a track too quickly can also fragment one continuous movement into many identifiers. The lifetime rule is an engineering choice to validate.
5. Motion prediction: what might happen next?
A basic tracking model can estimate position and velocity and predict a short distance forward. For instance, a toy ball moving steadily to the right may be expected slightly farther right in the next frame. Kalman filters and related estimators combine a predicted state with new, noisy observations in suitable models.
This calculation assumes something about movement. A constant-velocity approximation may work over a short interval for a rolling ball, but a bouncing basketball or a person changing direction can violate it quickly. The error depends on time gaps and on how accurately the model represents the real motion.
A predictive position is useful because it restricts the candidate search area. Yet it is not proof that the ball went there. When an object is obscured, the correct description is ‘predicted track position with uncertainty’. If the application must make a consequential decision, it needs safeguards beyond that prediction.
6. Appearance clues and data association
Two objects can pass near one another, making position alone ambiguous. A system may therefore compare visual features such as colour distributions, shape, texture or learned embeddings. If one student wears a bright green bag and another carries a yellow folder, appearance may help maintain separate tracks.
The matching problem is called data association. Each incoming detection is compared with candidate existing tracks. The system balances evidence such as predicted position, box overlap, time since last detection and appearance similarity. An assignment algorithm or learned approach can select plausible pairings.
Appearance clues are fallible. People can wear the same uniform. Lighting changes colour. The front of an object may look unlike its back. A scene can contain several nearly identical balls. A responsible tracker combines clues and allows uncertainty rather than treating a single similar colour as definitive identity.
| Task | Typical output | What it does not prove |
|---|---|---|
| Object detection | Box and class for a current frame | That the next frame shows the same object |
| Optical flow | Apparent movement of image patterns | Persistent object identity |
| Object tracking | Estimated continuity and track identifier | Verified personal identity |
| Physical speed measurement | Distance per unit time under calibration | That a predicted hidden trajectory was observed |
7. A complete fictional school-playground example
Imagine a fixed camera watching two footballs, one red and one blue, rolling across a playground. The tracker detects both and assigns temporary tracks T1 and T2. The red ball rolls behind a student for three frames. The blue ball remains visible.
During the hidden interval, the tracker predicts where T1 might be based on recent movement. When a red circular object emerges nearby, it considers whether that detection is compatible with the old track. It may compare the object’s size, appearance, position and timing before reassigning T1.
Now alter the experiment: another red football enters from behind the student at almost the same moment. The available evidence may not uniquely distinguish the balls. A sensible system can mark the association uncertain or start a new track. Reusing T1 confidently because a red circle emerged is not a justified conclusion.
This demonstration is invented. It shows the information boundary, not a claim about a particular product’s ability to identify sports equipment.
8. Why occlusion changes confidence
When a detector directly sees an object, it has a fresh visual observation. When the object is hidden, the tracker depends on earlier observations and motion assumptions. The longer the gap, the more plausible alternatives can accumulate, especially in a dynamic environment.
One method is to increase the uncertainty around the predicted location over time. The software may consider a wider area for reappearance, but that also brings more possible wrong matches. Tightening the search too much can miss a genuine return after a turn. Loosening it too much can create identity swaps.
Confidence should also reflect how many similar objects share the scene. One visible red ball entering an empty frame is easier to associate than five identical red balls crossing simultaneously. A display that uses the same solid box for an observed detection and a tentative prediction can mislead users about what is actually known.
9. How ByteTrack illustrates a useful design choice
An influential multi-object tracking approach named ByteTrack explores a practical problem: detectors can assign low scores to real objects when they are partly hidden. A tracker that discards all low-confidence detections may fragment tracks unnecessarily and lose continuity.
ByteTrack’s published work describes associating detections across score groups so that compatible low-score detections can sometimes help maintain existing tracks while background detections are filtered. This is not equivalent to accepting every low-quality detection. Association still needs constraints and testing.
The broader lesson is subtle: information that is individually weak may be useful in the context of a well-supported sequence. A low-score glimpse of an object can fit an established track. But the same glimpse must not be promoted into certainty when several alternative explanations remain.
10. Object identity can switch during a crossing
Suppose two nearly identical cyclists cross in front of a camera. Before crossing, one travels left and the other right. Their paths overlap temporarily. After the overlap, a tracker could accidentally attach the left cyclist’s original identifier to the right cyclist. The system continues drawing smooth boxes, but the labels have switched.
This is an identity switch. It matters because a user might count how many separate objects passed through a region or estimate each object’s journey. A tracker can seem visually convincing while making a basic continuity mistake.
Performance therefore needs metrics beyond the number of correctly detected boxes. Tracking benchmarks assess factors such as detection performance, identity association and continuity. Evaluating only accuracy on isolated frames cannot tell whether the track identifiers remain meaningful through crowded passages.
11. The camera itself may be moving
In a fixed security-style camera, apparent motion often reflects moving objects. When the camera pans, shakes or rides on a vehicle, the whole background moves. A tracker that assumes stationary camera coordinates can misinterpret the motion.
Camera-motion compensation, scene geometry and suitable transformations may help align frames before matching objects. However, the scene can also contain independently moving subjects, and visual features may change under perspective or exposure. A good system distinguishes changes caused by the camera from those caused by the objects.
A learner should be wary of simplistic claims such as ‘the bike moved ten pixels, so its speed was ten metres per second’. Pixel displacement needs camera calibration, time information and appropriate geometry to become a real-world speed. Tracking identities and measuring physical velocity are separate tasks.
| Situation | Likely risk | Useful response |
|---|---|---|
| Object hidden briefly | Temporary missed detection | Short-lived predicted track |
| Long hidden interval | Uncertainty grows | Expire or flag the track |
| Two identical objects crossing | Identifier switches | Use multiple features and mark ambiguity |
| Moving camera | False apparent motion | Camera-motion estimation |
| Reflective surface | False target or duplicated appearance | Check scene geometry and alternative hypotheses |
| Low-scoring partial detection | Track fragmentation | Evaluate association with care |
12. How to evaluate a tracker fairly
A suitable dataset includes videos with trustworthy annotated object boxes and identity sequences. Include crowded moments, temporary disappearances, similar-looking targets, low light, camera movement and fast changes of direction. Test on sequences that were not used to tune the model.
Evaluate false detections, missed detections, track fragmentation, identity switches and the ability to recover after occlusion. Metrics such as IDF1, MOTA and HOTA capture different aspects of tracking and should be interpreted with their definitions and limitations. A single score cannot explain why a particular crossing failed.
Human inspection remains useful. Show annotated video frames around the difficult moment, not just an average benchmark number. If the model is used for a real operational purpose, measure the consequences of mis-association and whether the software appropriately declines to make an unsupported continuity claim.
13. Privacy and safety limits
Following object trajectories can be harmless for sports analytics or science demonstrations, but tracking people over time creates privacy concerns. A temporary on-screen track number should not be repurposed as a personal identifier without a separate justified and authorised process.
Use the least identifying data necessary for the purpose. If counting how many toy balls cross a line is sufficient, do not collect faces, names or prolonged personal movement histories. Explain retention, access and human review where real people are involved.
A tracking output should also not drive unsafe physical motion without a broader validated control process. Robots and automated equipment need limits, collision avoidance, uncertainty handling and appropriate independent safety checks. A plausible track is only one source of information.
14. Try a paper experiment with three moving objects
Draw six simple video frames on a sheet of paper. Put a red ball, a blue ball and a cardboard barrier in each frame. Move the red ball progressively behind the barrier, then show it reappearing. Ask a classmate to assign track numbers without being told what moved behind the barrier.
Now add a second red ball at the point of reappearance. Does the same track number remain certain? Ask what evidence would be needed to distinguish the objects. The answer might be a distinctive mark, another camera view or continuous observation before the gap.
Finally draw a dashed box where the object is predicted while hidden. Use a solid box only where an object is actually visible. This simple drawing makes the difference between prediction and direct observation clear to readers of any age.
15. The deeper AI lesson: continuity is a hypothesis
AI tracking teaches a transferable idea: a system does not automatically know that two observations refer to the same thing. It constructs that relationship from timing, motion, appearance and context. The relationship is strongest when supported by consistent evidence and weakest after long gaps or confusing crossings.
That principle applies beyond video. It appears in matching repeated records, tracing a package’s updates, combining sensor measurements and linking references in text. The label that ties observations together is an inference that can be tested and revised.
The best explanation does not pretend that AI sees objects through walls. It says what the camera observed, what movement was predicted, why a new detection was associated with an old track and when the evidence became too weak for a confident match.
Frequently asked questions
How can AI track an object if it is hidden?
It can predict a temporary location from past observations and later try to associate a new detection with the earlier track; it does not directly see the hidden object.
What is occlusion in computer vision?
One object blocks the camera’s view of another object, either partly or completely.
What is the difference between detection and tracking?
Detection finds objects in frames, while tracking estimates which detections correspond to the same continuing objects across frames.
Does optical flow solve object tracking by itself?
Not generally. Optical flow estimates image-pattern movement; maintaining object identities through occlusions requires additional reasoning or association.
Why can track IDs change after two people cross?
The software may associate a new detection with the wrong earlier trajectory when positions and appearances are ambiguous.
What does a Kalman filter contribute?
Under a suitable motion model, it estimates a state and updates that estimate using noisy observations.
What is ByteTrack known for?
It uses data association that can recover useful lower-score detections for existing tracks instead of simply discarding all of them.
Can a temporary tracking number identify a person?
No. A track identifier is a software label for continuity, not proof of personal identity.
How should tracking be tested?
Use independently annotated videos with occlusion, crossing, camera motion and changing appearance, and measure identity errors as well as detection quality.
Research and connected explanations
- ByteTrack: Multi-Object Tracking by Associating Every Detection Box (ECCV 2022)
- ByteTrack published manuscript (arXiv)
- NIST AI Risk Management Framework
- How Artificial Intelligence AI Works | How AI Tracks Motion Using Optical Flow
- How Artificial Intelligence Works | How AI Counts Overlapping Objects With Density Maps
- How Artificial Intelligence Works | How AI Estimates Depth From One Photograph
- eduKateSG | How X Works Hub
Editorial note: The football, toy-car and corridor scenes are invented educational examples. This article explains algorithmic continuity, not surveillance or identification of actual students. Track labels and short-term predictions are not independently verified identities or locations.