What can this image actually establish?
Follow visual inputs into task-specific answers, then inspect two complete symbolic scenes, exact crops and worked repairs.

An image can support several different answers. It can tell us that a board contains shapes, locate two red squares, supply the visible cells of a green shape, or show a readable label. Those are different jobs. A system that succeeds at one has not automatically completed the others. Understanding machine vision begins by connecting the requested answer to the information that survives the journey from the source image to the model.
This guide follows that journey: pixels and colour channels, resizing and cropping, learned representations, and task-conditioned interpretation. It then supplies two complete symbolic visual exercises with worked answers. You can inspect every visible cell, check every coordinate, and see exactly why some questions remain unanswered. The exercises are original teaching fixtures, not photographs, measured model outputs, or demonstrations of a commercial service. No live model performance test is claimed.
Super Intelligence is eduKateSG’s practical-AI editorial umbrella. Artificial superintelligence, or ASI, is a hypothetical broadly superhuman capability. Explaining present computer-vision mechanisms does not establish that existing tools are ASI. The classroom header is illustrative; it is not evidence for the fictional boards used below. This article concerns interpreting a still image. For the separate process of observing an interface, taking actions and checking the changed state, continue to How Super Intelligence Works: Computer Use.
Choose your reading route
Understand the mechanism · Work through Scene A · Try independent Scene B · Check a visual workflow
From image to representation
- 1. Start with the answer you need
- 2. Pixels are measurements with a coordinate system
- 3. Preparation can preserve the task or remove its evidence
- 4. Learned features are useful representations, not miniature descriptions
- 5. A Vision Transformer is one representative patch-based mechanism
- 6. CLIP explains an image–language connection without promising exhaustive reading
- 7. Five visual tasks need five kinds of answer
Scene A, transforms and repair
- 8. Scene A: inspect the complete board before answering
- 9. Scene A answers: classification, detection and segmentation
- 10. Scene A answers: text and relationships
- 11. Crops change the evidence window
- 12. An exact resize experiment shows irreversible information loss
- 13. A defective answer needs several different repairs
- 14. Coordinate transforms need their own check
Independent transfer with Scene B
- 15. Scene B: independent transfer with occlusion
- 16. Scene B answers: complete shapes and visible fragments
- 17. Scene B answers: text, hidden content and targeted repair
- 18. A second crop exercise tests what remains unknown
Using and checking visual answers
- 19. Counting needs an inclusion rule and an instance ledger
- 20. Confidence, correctness and answerability are separate questions
- 21. Write requests that expose the reasoning to inspection
- 22. Evaluate the task rather than the impressiveness of a caption
- 23. Transfer to screenshots, diagrams and photographs carefully
- 24. People and consequential decisions require stronger boundaries
- 25. Common questions about interpreting images
- 26. Your final checkpoint: follow one claim all the way back
1. Start with the answer you need
Imagine asking an assistant to examine a photograph of a school display. “What is this?” might require only a broad description. “How many complete red squares are visible?” requires a counting rule and sufficiently clear boundaries. “Read the code beneath the left square” requires access to particular characters. “Which shape touches the blue rectangle?” requires positions and a definition of touching. The source file may be identical while the required evidence changes substantially.
This distinction is useful before any technical vocabulary appears. A broad description can tolerate omissions that would make an inventory wrong. A caption may say that books are on a table without listing every book. An inventory must decide whether partly hidden volumes count. A transcription must preserve an unusual spelling rather than quietly replacing it with a familiar word. Different jobs create different success conditions, so a single impression that the system “understands the picture” is too coarse to guide checking.
A practical visual request therefore has three parts. First, identify the input: the whole image, a named crop, or several views. Second, identify the target: visible objects, boundaries, printed text, relationships, or a supported answer. Third, identify the evidence rule: count only fully visible shapes, preserve uncertain characters, or report a location in a specified coordinate frame. These choices reduce ambiguity without pretending that clearer wording restores missing pixels.
Suppose an answer says, “There are three red squares.” You should be able to ask which three regions it refers to. If the answer cannot connect its count to identifiable instances, the number has not been checked. If one alleged square is hidden behind a panel, the answer must distinguish a visible red fragment from a confirmed complete square. The important habit is to make the requested output inspectable. A claim becomes easier to repair when the reader can point to the exact region and operation that produced it.
Back to contents · Next: 2. Pixels are measurements with a coordinate system
2. Pixels are measurements with a coordinate system
A digital raster image is an arrangement of samples. Width and height describe the number of sample positions along two axes. Channels carry quantities at each position, such as red, green and blue values in a familiar colour representation. The file can also contain metadata, compression information and other structure. A model does not receive a miniature physical room. It receives a representation made from a particular capture, encoding and preparation process.
Consider a deliberately small teaching image with width 120 and height 80. It contains 9,600 pixel positions. With three channels, that is 28,800 channel values before considering a file format or compression. This arithmetic does not tell us the file size: storage depends on bit depth, compression and overhead. Nor does it tell us how many objects exist. A single object can occupy many pixels, and several physical objects can overlap at the same image position along the camera’s line of sight.
We will use an origin at the top left. Horizontal coordinates increase to the right; vertical coordinates increase downward. A box written [10,10,30,30) includes positions starting at x=10 and y=10, stopping before x=30 and y=30. Its width and height are both 20. The final bracket is a reminder that the upper limits are excluded. Other tools use other conventions. An answer can be numerically tidy and still wrong if one person uses inclusive endpoints while another uses half-open intervals.
Colour also needs context. An RGB triplet is meaningful within a defined representation and processing chain. A photographed red card under unusual lighting may not have the same values as a red patch created in an editor. For our symbolic boards, the letter R explicitly means a red occupied cell, so no colour estimation is required. That simplification lets us concentrate on evidence and geometry. When transferring the method to photographs, replace that convenient legend with a real colour, shape or object-recognition problem, and keep the resulting uncertainty visible.
Back to contents · Next: 3. Preparation can preserve the task or remove its evidence
3. Preparation can preserve the task or remove its evidence
Images are often resized, cropped, padded, converted between formats or normalised before a vision model processes them. These operations are not interchangeable. Resizing changes the sampling grid. Cropping removes an area outside a selected window. Padding adds a border. Normalisation changes the numerical scale or distribution presented to a model. Official Torchvision transform documentation illustrates these as separate operations, including transforms that update images and associated boxes or masks together.
Suppose a label occupies a small part of a wide photograph. Reducing the whole photograph to a small square can give the label very little representation. A targeted crop from the original can allocate more of the available input to that label. However, enlarging a crop taken from an already degraded thumbnail does not supply the original missing detail. The interface may display a larger rectangle while the underlying evidence remains the same. Size on screen and information in the source must be checked separately.
A crop can also harm a task by removing context. A close view may reveal a character clearly but omit the arrow that links it to the correct object. A close view of one red square cannot establish how many red squares occur elsewhere on the board. For a mixed task, retain the full view for location and use targeted crops for detail. Give each crop a name and record its origin, so a later answer can say both what was read and where it belongs.
The right preparation depends on the question. For a broad scene label, an overview may be sufficient. For a small serial number, a higher-quality local view may be essential. For a segmentation boundary, a resize that smooths thin protrusions could change the result that matters. Before blaming reasoning, ask whether the relevant evidence was still present at the point of interpretation. A careful pipeline keeps the source, the transformed input and the output distinguishable, so a failure can be located rather than described vaguely as poor vision.
Back to contents · Next: 4. Learned features are useful representations, not miniature descriptions
4. Learned features are useful representations, not miniature descriptions
A model needs a way to turn many measurements into quantities that help it perform a task. A learned feature is one such useful internal quantity or pattern. It need not correspond to a word that a person would choose. A representation can carry information that helps separate categories without containing an explicit sentence saying what every region depicts. Describing these quantities as a hidden photograph or a complete inventory is therefore misleading.
One familiar family uses convolutional operations, which apply learned filters across local regions. Other systems use attention-based processing, and some combine approaches. The Torchvision model catalogue separates classification, detection and segmentation models and includes different architecture families. That variety matters: the fact that one documented model uses a certain component does not establish the undisclosed internals of every image-capable assistant.
A simple hand-built analogy helps without pretending to reproduce training. Take two small binary patches. One has occupied cells mainly on its left side; the other has occupied cells mainly on its right. A quantity that subtracts right-side occupancy from left-side occupancy can distinguish them. A quantity that merely totals occupied cells may not. Both are representations of the same input, but they preserve different information. Learned systems discover more complex useful transformations through training rather than relying only on these manually chosen measurements.
The analogy also shows why an answer can lose a distinction before language generation begins. If a later stage receives only a total of eight occupied cells, it cannot determine whether those cells formed two squares or one rectangle unless additional structure survived. Real representations are richer than a single total, but the logical point remains: the requested answer depends on the information available in the representation. A fluent final sentence cannot serve as evidence that every earlier distinction was preserved. We should evaluate the whole input-to-answer path on the particular task we care about.
Back to contents · Next: 5. A Vision Transformer is one representative patch-based mechanism
5. A Vision Transformer is one representative patch-based mechanism
The original Vision Transformer, or ViT, divides an image into fixed-size patches, maps flattened patch values into learned embeddings, adds positional embeddings and processes the sequence with a Transformer encoder. Its classification setup uses an additional learned classification token and a prediction head. This is a documented representative architecture, including a discussion of hybrid inputs from convolutional feature maps. It is not a universal description of all machine vision. See the method section of An Image Is Worth 16×16 Words.
We can make the bookkeeping concrete without running a trained model. If a hypothetical 120-by-80 RGB input is divided into non-overlapping 10-by-10 patches with no padding, it yields 12 patches across and eight down: 96 patches. Each flattened patch contains 10×10×3=300 channel values before projection. These are counts of input groups, not counts of objects. A red square can cross patch boundaries; a patch can contain parts of several things. The chosen numbers here are a teaching configuration, not a claim about a particular service’s settings.
Now double both spatial dimensions while keeping the hypothetical patch size unchanged. A 240-by-160 input would produce 24×16=384 patches, four times as many. This does not by itself prove four times better recognition or four times the end-to-end cost. It establishes only the number of patches under the stated geometry. Architecture, preprocessing, implementation and task all affect the consequences. Arithmetic is most useful when we keep its conclusion as narrow as its assumptions.
Position deserves special attention. If we keep the same patch contents but change where they occur, the answer to “Which square is left of the blue rectangle?” may change. A bag containing the right pieces is insufficient for that question unless their arrangement is also represented. Conversely, rotating a board may preserve the number of squares while changing “above” and “below.” These examples explain why image interpretation needs both appearance and arrangement, and why a successful category label is a weak test of detailed spatial understanding.
Back to contents · Next: 6. CLIP explains an image–language connection without promising exhaustive reading
6. CLIP explains an image–language connection without promising exhaustive reading
CLIP jointly trains image and text encoders so that paired images and texts have more compatible representations than mismatched pairs within the training objective. At use time, candidate text labels can be compared with an image representation. The paper studies both modified ResNet and Vision Transformer image encoders. This illustrates learning a connection between visual and linguistic representations; it does not make every image-text system identical to CLIP. See Learning Transferable Visual Models From Natural Language Supervision, particularly Sections 2.2–2.5.
Consider an original classroom analogy. You are shown the symbolic board in this article and offered the labels “shape board,” “restaurant receipt” and “mountain landscape.” Selecting “shape board” would be useful. It would not show that you had counted the red components, transcribed a label, or identified every occupied cell. A broad image-label match and a complete source audit answer different questions. The stronger the requested claim, the more directly its particular evidence must be checked.
Candidate labels also shape the task. If the choices are “one red square” and “two red squares,” a selection concerns those alternatives. If all offered labels are unsuitable, choosing the least unsuitable one would still produce a bad description. A practical system therefore needs a way to express mismatch, uncertainty or the need for another task. The existence of a highest score among candidates does not establish that the selected statement is a verified account of the scene.
For a reader using an image-capable assistant, the useful question is not whether language has been attached to vision in some general sense. Ask what operation the interface is actually supporting: retrieval, classification, transcription, grounded answering, or another task. A tool may combine several components, and an assistant may call specialised tools. Keep the distinction between a documented research mechanism and a product’s observed behaviour. Our worked answers below are manual solutions to supplied inputs; they are not CLIP outputs and do not establish performance for any commercial model.
A different representative connection appears in the original LLaVA architecture: a vision encoder supplies visual features, a learned projection maps them into the language model’s embedding space, and the language model produces responses conditioned on visual and instruction inputs. Its training procedure distinguishes feature alignment from visual instruction tuning. This gives a concrete route from image representations to question-dependent language, beyond ranking candidate labels. It remains one documented design, not a claim about every assistant. See Section 4 of Visual Instruction Tuning.
In a teaching example, “count the complete squares” and “read the label” supply different linguistic conditions alongside the same board. A suitable system must use those conditions to select the relevant visual evidence and form the requested kind of output. The words change what should be returned; they do not alter what is actually visible. If the text is absent, changing the instruction cannot manufacture evidence. This separation between conditioning and information is central to understanding both the flexibility and the limits of image-based conversation.
Back to contents · Next: 7. Five visual tasks need five kinds of answer
7. Five visual tasks need five kinds of answer
Classification assigns one or more category labels to an image or region. Detection identifies instances and their locations, commonly through bounding boxes. Segmentation identifies which image locations belong to a region or instance. OCR transcribes visible text. Visual question answering combines a question with image evidence to produce a relevant answer. The VQA project frames questions about images as a research task in its own right. These categories overlap in applications, but their outputs remain distinguishable.
Two primary papers make the location distinction concrete. YOLO describes an object-detection approach that predicts boxes and class probabilities. Mask R-CNN adds an object-mask prediction branch alongside box recognition in its instance-segmentation framework. A rectangle around a shape includes background that lies inside the rectangle. A mask can represent an irregular outline more precisely. Neither output automatically supplies the text printed nearby or explains the relationship between that text and the object.
On our first board, classification can return “symbolic shape board.” Detection can return two red-square boxes. Segmentation can return the exact five occupied cells of the green shape. Transcription can return the visible large label T7. A question-answering task can return that the blue rectangle is below the red squares in image coordinates. Each answer is checked against a different part of the supplied evidence. Asking only for a caption and then treating it as a completed extraction of all these outputs would hide missing work.
A useful exercise is to specify the output before inspecting a model answer. If you need detection, decide whether boxes should be in source pixels, normalised coordinates or named grid cells. If you need segmentation, decide whether you require visible surfaces or an estimated complete object behind an obstruction. If you need OCR, decide how to represent an unreadable character. These choices make a result easier to compare, correct and use. They also prevent a plausible paragraph from quietly substituting for the structured answer that the next step actually needs.
Back to contents · Next: 8. Scene A: inspect the complete board before answering
Vision lab: supplied inputs and worked answers
8. Scene A: inspect the complete board before answering
The following is a fictional symbolic visual exercise. It is deliberately printed as a grid so that readers can inspect the source directly. Columns A–L run left to right; rows 1–8 run top to bottom. A dot means visible empty background. Each R cell belongs to red material, B to blue material, and G to green material. Adjacent same-letter cells form the shapes shown. T and 7 are two literal large printed characters. The question mark is an annotation for an unreadable text region whose underlying characters are not supplied; it is not a literal character to transcribe.
For coordinate calculations only, each grid cell occupies a 10-by-10 square in an idealised 120-by-80 coordinate canvas. This is a stated geometry model, not a claim that the browser renders each character as ten pixels. The grid contains the complete visible scene for this exercise. Nothing outside it may be assumed. Shape colour is provided by the legend rather than inferred from the displayed ink, so the exercise works without relying on colour vision.
Scene A — full symbolic view
A B C D E F G H I J K L
1 . . . . . . . . . . . .
2 . R R . . . . R R . . .
3 . R R . . . . R R . . .
4 . . . . . . . . . . . .
5 . . . B B B . . . G G .
6 . . . B B B . . . G . .
7 . T 7 . . . . . . G G .
8 . . . . . . ? . . . . .
Use these exact questions. A1: choose the best category among symbolic shape board, receipt and landscape. A2: count fully visible red squares and locate each with a half-open coordinate box. A3: give the blue rectangle’s box and occupied-cell count. A4: give the green shape’s mask and compare its occupied area with its bounding-box area. A5: transcribe the large two-character label and state whether the unreadable region can be transcribed. A6: is the blue rectangle below both red squares, and is the green shape entirely right of the blue rectangle? A7: does the label establish which shape it names?
Before reading the answers, notice what the board does not contain. It gives no physical unit such as centimetres, no camera distance, no temporal history and no rule linking the label to an object. It contains no arrow from T7 to a shape. The lack of those details is intentional. A complete answer should finish the supported tasks while leaving unsupported questions open, rather than turning uncertainty into an excuse to avoid the clear parts.
Back to contents · Next: 9. Scene A answers: classification, detection and segmentation
9. Scene A answers: classification, detection and segmentation
A1 is “symbolic shape board.” The supplied legend and visible arrangement support that category. This is an elementary classification exercise rather than evidence of a model’s classification accuracy. A2 is two fully visible red squares. The left square occupies B2, C2, B3 and C3; the right square occupies H2, I2, H3 and I3. Their half-open boxes are [10,10,30,30) and [70,10,90,30). Each covers four grid cells, or 400 square coordinate units under the stated scale.
A3 is a blue rectangle occupying D5, E5, F5, D6, E6 and F6. Its box is [30,40,60,60). Width is 30 and height is 20, so its area is 600 square coordinate units, matching six cells of area 100 each. This agreement is a useful consistency check. If a proposed box had width 20, it would not cover the third blue column. If a proposed count were four, it would omit two visible occupied cells. Location, extent and area should agree with the same source.
A4 requires a mask: J5, K5, J6, J7 and K7. The bounding box is [90,40,110,70), which covers six cells, but only five are green. K6 is background. The green occupied area is therefore 500 square coordinate units, while the box area is 600. A box-based answer that reports six green cells has confused an enclosing rectangle with the shape itself. This is exactly the kind of error that can remain invisible in an otherwise convincing caption.
There is also a useful terminology boundary. A semantic mask might identify all green material as one class. An instance mask identifies a particular object or region separately from other instances. Here there is only one connected green region, so the distinction does not change the numerical answer. It would matter if the board contained two separated green pieces. The fixture makes the mask easy to inspect; in a real image, deciding the boundary can itself be uncertain. Do not transfer the exactness of the symbolic answer to a photograph with blur, transparency or poorly defined edges.
Back to contents · Next: 10. Scene A answers: text and relationships
10. Scene A answers: text and relationships
A5 returns T7 for the large visible label. The region marked ? remains unreadable because its underlying characters have not been supplied. “Unknown” is the complete answer for that region. It would be wrong to return a likely code from context, and equally wrong to transcribe the annotation itself as a literal question mark printed on the original board. The exercise separates a known label from a deliberate gap in the source. Real OCR systems likewise need to distinguish recognised text from missing or uncertain text; Google’s OCR documentation describes text detection as an operation with text and location outputs.
A6 has two supported answers. The blue rectangle is below both red squares because its top boundary is y=40 while each red square ends at y=30. There is a ten-unit vertical gap. The green shape is entirely right of the blue rectangle because green material begins at x=90 while blue material ends at x=60. There is a thirty-unit horizontal gap. These are statements in the declared image coordinate system. They do not establish a physical distance or depth relationship in a room.
A7 is unanswered: the label’s ownership is not specified. T7 lies near the lower left, but proximity alone does not establish that it names the left red square, the whole board, a row or something outside the drawing’s purpose. A sensible answer can describe its location while withholding its meaning. To establish the binding, we would need an arrow, an explicit layout rule, a legend or another source. Asking the same question more insistently does not create that missing relationship.
This split is a model for useful reporting. State the exact transcription that is supported. State the geometric relation that is supported. Then identify the specific unresolved relationship. Avoid a blanket statement that the whole scene is uncertain when most of it is clear. Equally, avoid allowing success on T7 to create confidence about the unavailable small text. Each claim has its own evidence requirements, and a reliable answer keeps those requirements visible all the way to the final wording.
Back to contents · Next: 11. Crops change the evidence window
11. Crops change the evidence window
Here are three exact crops from Scene A. The first includes columns A–D and rows 1–4. Its origin in the full coordinate canvas is (0,0), and its extent is [0,0,40,40). The second includes columns A–D and rows 6–8, with origin (0,50) and extent [0,50,40,80). The third includes columns F–H and rows 7–8, with origin (50,60) and extent [50,60,80,80). The drawings retain their original row and column labels so that you can verify the correspondence yourself. No new source information has been added.
A-left crop
A–D; rows 1–4
A B C D
1 . . . .
2 . R R .
3 . R R .
4 . . . .
A-label crop
A–D; rows 6–8
A B C D
6 . . . B
7 . T 7 .
8 . . . .
A-unreadable crop
F–H; rows 7–8
F G H
7 . . .
8 . ? .
The left crop supports a count of one red square within that crop. It does not support “the full board has one red square.” That mistake changes the scope of the claim after the evidence window has narrowed. The label crop supports T7, but it omits the red squares and most other shapes. It is suitable for checking the characters and poor for deciding which distant shape the label refers to. The unreadable crop preserves the explicit unknown region; it provides no secret text to recover.
If a real high-resolution original existed, a targeted crop from that original might reveal details missing from a thumbnail. In this teaching packet, no such higher-resolution source is supplied for ?. The correct repair request is therefore to obtain a better source of that particular region, not to invent a crop result. Keeping this distinction explicit prevents a common demonstration error: printing an answer in surrounding prose and then treating that prose as proof that unreadable image content was recognised.
Crop provenance is especially important when several similar objects appear. Name the source and crop, retain the origin, and specify whether coordinates are local or global. A useful answer might say, “One red square is visible in A-left; the full Scene A contains two.” Both statements can be true. A useful transcription might say, “T7 is visible in A-label; no binding to a shape is established.” Detail work becomes safer when its narrower field of view remains attached to every conclusion drawn from it.
Back to contents · Next: 12. An exact resize experiment shows irreversible information loss
12. An exact resize experiment shows irreversible information loss
We now leave the shape board for a separate numeric fixture. X and Y are two different 4-by-4 binary inputs. A zero and a one represent two distinct sample values, not printed characters requiring OCR. Define the resize operation exactly: replace every non-overlapping 2-by-2 block with its arithmetic mean. This produces a 2-by-2 output. Such an averaging experiment is a teaching calculation; it is not a claim about the preprocessing of an unnamed product. Image libraries provide several interpolation methods, as documented in OpenCV’s geometric transformations reference.
Input X Input Y 0 1 0 1 0 0 0 0 0 1 0 1 1 1 1 1 0 1 0 1 0 0 0 0 0 1 0 1 1 1 1 1 Block-mean resize of X and of Y 0.5 0.5 0.5 0.5
Each 2-by-2 block in X contains two zeros and two ones, so its mean is 0.5. Each corresponding block in Y also contains two zeros and two ones. The outputs are identical even though X has vertical alternating structure and Y has horizontal alternating structure. The mean intensity survives. The distinction between these two arrangements does not. If a later stage receives only the reduced array, it cannot determine which of these two originals produced it from that array alone.
Enlarging the reduced array cannot solve this inverse problem uniquely. Many larger inputs could produce the same block means. A system might generate a plausible pattern using learned expectations, but that pattern would not establish the lost original. This is why an enhancement and a recovery of evidence are different claims. The correct repair for deciding between X and Y is to access the original or another observation that preserves the distinction. No additional eloquence in the final answer changes the information available.
This experiment also identifies what we can still answer. The mean of the original sixteen values is 0.5 for both inputs, and the reduced array supports that result under the stated transform. We should not discard a valid aggregate merely because another property was lost. Good visual reasoning is property-specific: a transformation can preserve one quantity while destroying another. Ask which property your task needs, then test whether the chosen preparation preserves it. That is more informative than labelling an entire image either readable or unreadable.
Back to contents · Next: 13. A defective answer needs several different repairs
13. A defective answer needs several different repairs
Consider this deliberately invented bad answer to Scene A: “The board has one red square, a six-cell green rectangle, and a blue square. The label T7 belongs to the red square. The small code is T8.” This is not an output from a tested model. It is a teaching specimen designed to expose several failure types in one short paragraph. Repairing it well means separating the errors rather than replacing it with a generic warning about hallucinations.
“One red square” could come from looking only at A-left and silently extending a crop-local count to the full scene. The repair is to restore the full evidence window and enumerate the two red instances. “Six-cell green rectangle” confuses a bounding box with its occupied mask. The repair is to inspect K6 and use the five-cell mask. “Blue square” ignores the three-column by two-row extent. The repair is to compare width and height under the declared cell scale. Each intervention targets the information the particular answer lost.
“The label T7 belongs to the red square” adds an unsupported relation. Better resolution alone would not necessarily fix that: the problem is the absence of a binding rule. We need another source that establishes the relationship. “The small code is T8” fabricates unavailable text. A tighter crop of the same unknown marker adds no information; the repair is a better original or a direct transcription supplied by someone who can inspect it. These two unknowns need different evidence, even though both appeared in the same sentence.
A corrected answer is therefore precise: Scene A contains two complete red squares, a six-cell blue rectangle, and a five-cell green shape whose enclosing box covers six cells. The visible large label is T7. The small text is unavailable, and the label-to-shape relationship is unspecified. The blue rectangle is below both red squares, and the green shape is entirely to its right. This answer completes the supported work, preserves the two unresolved questions, and gives a reviewer enough detail to verify the correction without trusting the tone of the response.
Back to contents · Next: 14. Coordinate transforms need their own check
14. Coordinate transforms need their own check
A crop introduces a new origin. In A-label, the full-scene character T begins at x=10, y=60. Relative to the crop whose origin is (0,50), it begins at local x=10, y=10. To map a local point back to the full view, add the crop origin. To map a full-view point into the crop, subtract it. These elementary operations are easy to forget when a tool returns convincing-looking numbers without naming its coordinate frame.
Now suppose A-label is displayed at twice its coordinate width and height. In that displayed representation, the same point begins at (20,20) relative to the displayed crop. Undo the display scale first to obtain local source coordinates (10,10), then add the full-scene origin to obtain (10,60). Reversing those operations incorrectly would produce a different result. The transformation history matters. A box or point is not fully specified by its four or two numbers alone; it also needs the image, frame, scale and endpoint convention.
We can check this with a second point. The character 7 occupies C7, whose full-scene cell box is [20,60,30,70). In A-label coordinates, that becomes [20,10,30,20). At twice the display scale, it becomes [40,20,60,40). Mapping back should recover the exact original box. This round trip provides a simple verification: if the recovered location does not match C7, the coordinate transform is wrong even if the character itself was transcribed correctly.
For this article, coordinates describe observations only. Turning a location into a mouse click, robotic motion or other action adds a different system problem: current state, authority, side effects and verification. That broader loop belongs in the linked computer-use guide. Here the narrower lesson is already valuable. Before reusing a detected location, ask where its origin is, whether the image was resized, whether borders were added, and whether coordinates refer to visible content or a larger canvas. Correct visual recognition cannot compensate for a mismatched coordinate frame.
Back to contents · Next: 15. Scene B: independent transfer with occlusion
15. Scene B: independent transfer with occlusion
Scene B changes the arrangement, the number of complete red squares, the large text and the visibility conditions. Use the same A–L columns, eight rows, top-left origin and ten-unit cell scale. R, B, dots and literal characters retain their meanings. The # cells form an opaque panel. A lowercase r marks a visible red fragment at the panel’s edge; its complete shape and object identity are unspecified. The ? annotation again denotes an unreadable text region with no underlying characters supplied.
Scene B — full symbolic view
A B C D E F G H I J K L
1 . . . . . . . . . . . .
2 . R R . . R R . . R R .
3 . R R . . R R . . R R .
4 . . . . . . . . . . . .
5 . B B B . . . . . . r .
6 . B B B . . . . . # # #
7 . . . . . . Q 4 . # # #
8 . . . . . . . . ? . . .
Use these exact transfer questions. B1: how many fully visible red squares are there, and what are their boxes? B2: what red evidence appears near the panel, and does it establish another complete square? B3: give the visible blue mask, box and area. B4: transcribe the large label and handle the unreadable region. B5: is the blue rectangle below all complete red squares? B6: can the image establish the total number of physical red objects, the hidden red fragment’s full outline, or what happened before the panel was placed? B7: select the smallest evidence request needed to settle each unresolved question.
Work from the full scene before examining the answers. A successful transfer changes the numerical result when the source changes, while keeping the evidence rules stable. Do not copy Scene A’s count of two or its label T7. Do not automatically count the red fragment as a fourth square simply because a square would be a plausible continuation. The point of the second scene is to distinguish a reusable method from a memorised answer. Its apparent simplicity lets you check that distinction directly.
Back to contents · Next: 16. Scene B answers: complete shapes and visible fragments
16. Scene B answers: complete shapes and visible fragments
B1 has three fully visible red squares. Their occupied cells are B2, C2, B3, C3; F2, G2, F3, G3; and J2, K2, J3, K3. The corresponding boxes are [10,10,30,30), [50,10,70,30), and [90,10,110,30). Each is twenty units wide and twenty high. The changed count is a direct consequence of the changed source, not a reinterpretation of what “complete square” means. All three have their whole symbolic two-by-two outline visible.
B2 identifies the lowercase r at K5. Its visible region lies in [100,40,110,50). It is red evidence at the panel’s edge, but the fixture does not specify the complete shape or identity to which it belongs. A fourth red square is one possible hypothesis; another shape is also compatible with the visible fragment. We therefore report three confirmed complete red squares plus one unresolved red fragment. We do not report four confirmed squares, and we do not erase the fragment from the observation record.
B3 gives blue cells B5, C5, D5, B6, C6 and D6. The blue box is [10,40,40,60), with area 600 square coordinate units and a six-cell visible mask. B5 is yes: the blue region begins at y=40, below the y=30 lower boundaries of all three complete red squares. The panel does not obstruct those squares or the blue region. An obstruction in one part of an image does not make every claim about the image uncertain; uncertainty should stay attached to the affected region and question.
Notice how the output separates an instance count from a material observation. Three complete red squares is an object-level conclusion under the fixture rules. A red fragment at K5 is a location-level observation with unresolved identity. Combining them into one confident object count would discard that distinction. In photographs, similar mistakes occur when a visible sleeve becomes a counted person or a visible corner becomes a counted box without enough evidence. The proper response is to describe the visible part and identify what additional view would establish the whole.
Back to contents · Next: 17. Scene B answers: text, hidden content and targeted repair
17. Scene B answers: text, hidden content and targeted repair
B4 returns Q4 for the large label in G7 and H7. The unreadable annotation is at I8, and its underlying text cannot be transcribed from this packet. As in Scene A, position does not establish ownership. The label is near the panel, but no arrow or explicit association rule links Q4 to the panel, the fragment or a complete red square. A reliable extraction preserves the visible characters while keeping their intended reference unresolved.
B6 cannot establish the total number of physical red objects. It establishes three complete red squares and a red fragment, but the panel can conceal information and the symbolic scene does not define a hidden inventory. The fragment’s full outline is likewise unknown. Nor can a still image establish what happened before the panel was placed. A claim that someone moved a red square behind it would add a history that the source does not contain. Plausible stories should be labelled hypotheses, if they are useful at all, rather than reported as observations.
B7 calls for different repairs. To identify the fragment’s full shape or hidden inventory, obtain an unobstructed view or a trusted inventory that explicitly covers the concealed region. To read the unavailable text, obtain a sharper original or a legible close-up of that region. To establish label ownership, obtain an annotated diagram or the board’s labelling rule. To establish earlier events, obtain temporal evidence such as a relevant recording or a reliable event record. A single generic instruction to “zoom in” is inadequate for all four problems.
Here is the complete bounded summary: Scene B visibly contains three complete red squares and a six-cell blue rectangle. Q4 is legible. A red fragment appears at K5 beside an opaque panel covering J6–L7. Its complete object identity is unknown; text represented by ? at I8 is unavailable; hidden inventory, label ownership and prior events are not established. This answer is useful because it finishes the inspectable work and names exactly what would change the unresolved parts. It does not confuse an honest limit with failure to answer the questions that are answerable.
Back to contents · Next: 18. A second crop exercise tests what remains unknown
18. A second crop exercise tests what remains unknown
The following crops isolate two parts of Scene B. B-panel covers columns I–L and rows 4–8, with full-scene origin (80,30) and extent [80,30,120,80). B-label covers columns F–I and rows 6–8, with origin (50,50) and extent [50,50,90,80). Compare each cell with the full board. These crops are exact excerpts, so they cannot reveal material concealed by # or characters absent from the source packet.
B-panel crop
I–L; rows 4–8
I J K L
4 . . . .
5 . . r .
6 . # # #
7 . # # #
8 ? . . .
B-label crop
F–I; rows 6–8
F G H I
6 . . . .
7 . Q 4 .
8 . . . ?
Suppose a second invented answer says, “The close-up confirms a hidden square with code Q4, so the full board contains four squares.” The crop does not confirm any of those added relationships. It shows a fragment, a panel and a separately visible label. It does not show the complete hidden outline, establish that Q4 labels it, or rule out other concealed objects. The answer’s confidence increased because the display became more focused, while its evidential basis did not acquire the missing facts.
A corrected crop-level answer is narrower: B-panel shows a red fragment at local cell position corresponding to K5 and an opaque panel at J6–L7. B-label shows Q4 and the unavailable-text annotation. To map the fragment’s upper-left source point (100,40) into B-panel, subtract (80,30), giving local (20,10). That location is checkable, even though the fragment’s complete identity is unknown. Precise coordinates and uncertain semantics can coexist without contradiction.
This exercise offers a strong independent checkpoint. Ask a learner to name one question the crops help answer and one they cannot settle. A good response identifies the label’s characters or the fragment’s location as supported, and hidden shape or label ownership as unresolved. A weaker response merely repeats that vision models sometimes make mistakes. The stronger response demonstrates transferable diagnosis: it connects a particular uncertainty to the kind of evidence needed, instead of treating every visual problem as one undifferentiated limitation.
Back to contents · Next: 19. Counting needs an inclusion rule and an instance ledger
19. Counting needs an inclusion rule and an instance ledger
Counting sounds simple because the final answer is a number. The difficult work often lies in deciding what counts as one instance. Are partly visible items included? Do reflections count? Are printed pictures of objects included? Are nested objects counted separately? Does the task concern visible components, complete objects, or a physical inventory? A useful visual count starts with these definitions and then makes each included instance locatable.
For Scene A, the rule “fully visible red squares” produces two entries: left at B2–C3 and right at H2–I3. For Scene B, the same rule produces three entries and excludes the fragment at K5 from the confirmed-square count. The exclusion is not a claim that no object exists there. It says the evidence does not establish membership in the requested category. If the question instead asks for visible red regions, the output needs a different definition and a separate treatment of the fragment.
An instance ledger can remain very small. Record an identifier, a location, the reason for inclusion and any uncertainty. A-left-red, B2–C3, complete two-by-two red square, included. B-fragment, K5, red material with incomplete outline, excluded from confirmed complete squares. Such a ledger makes double counting easier to spot. It also prevents a reviewer from mistaking a lower count of confirmed instances for an assertion about the total physical inventory behind obstructions.
The repair for a counting error should follow its cause. If an item was missed at the edge, inspect the full field or an edge crop. If one object was counted twice across overlapping crops, reconcile locations. If two touching objects were merged, seek boundaries or another view. If the inclusion rule was ambiguous, settle the rule before recounting. More detailed prose is not the primary remedy. The aim is a count whose members and exclusions can be inspected, so that another person can reproduce the number under the same stated conditions.
Back to contents · Next: 20. Confidence, correctness and answerability are separate questions
20. Confidence, correctness and answerability are separate questions
An answer can be correct for the wrong reason, confidently wrong, or appropriately uncertain. It can also be correct within a crop and misleading when presented as a claim about the whole image. These possibilities explain why a confidence-sounding phrase is a poor substitute for an evidence check. “Clearly” adds no visible cell, and “probably” does not identify which observation supports the probability. Useful uncertainty points to a particular unresolved property.
For Scene B, “three complete red squares are visible” is well supported under the fixture rules. “The fragment is probably a fourth square” is a hypothesis with no supplied numerical probability. “The small text is Q5” is unsupported. These claims should not share a single confidence label simply because they occur in the same answer. A response-level score would conceal that some parts are exact, some are tentative and some should not be asserted at all.
Answerability is prior to performance. If an exercise deliberately omits the characters behind ?, no recognition system can prove their exact identity from this packet alone. That is an input limitation, not a measured failure rate. By contrast, miscounting the three fully visible squares is an error on available evidence. Keeping these categories separate makes evaluation fairer and repair more effective. We should not reward guessing unavailable information, and we should not excuse mistakes on clear evidence merely by invoking general uncertainty.
A practical review asks three questions for every consequential claim. What source region supports it? What operation connects that region to the answer? What additional assumption, if any, is being used? For a box, the operation is locating boundaries. For an area, it includes counting occupied cells and applying the scale. For hidden history, there is no supporting observation in the still image. This review method is simple enough for a classroom and precise enough to prevent several different forms of overclaiming in ordinary image-assisted work.
Back to contents · Next: 21. Write requests that expose the reasoning to inspection
21. Write requests that expose the reasoning to inspection
A good request does not need elaborate wording. It needs a named input, a clear task and an output that can be checked. For Scene B, one useful request is: “Using the full Scene B grid and its legend, list each fully visible red square with its occupied cells and half-open box. Keep the red fragment separate. Transcribe only supplied literal characters, and mark unavailable text as unknown.” This asks for the evidence needed to verify the result without demanding an invented account of internal model processing.
A second request can handle relations: “State whether the blue rectangle is below every complete red square in image coordinates. Compare their vertical extents. Do not infer physical distance, label ownership or prior events.” The qualification matters because it names the frame and the exact relation being tested. It does not merely ask the assistant to be careful. If the answer gives the boxes and the relevant boundary comparison, another reader can inspect the conclusion directly.
When a first result is wrong, ask for a targeted correction. “Recheck K6; distinguish the green occupied mask from its enclosing box” is more useful than “try harder.” “The last count used A-left; recount the full Scene A” identifies a scope error. “What evidence links Q4 to the fragment?” tests a binding claim. These prompts are examples of task repair, not a guarantee that any particular model will respond correctly. The source still has to be suitable for the requested operation.
Avoid asking for unavailable certainty. “Read the small code exactly; do not say you cannot” encourages an answer that the packet cannot support. A better instruction is to preserve uncertainty and identify the smallest additional source needed. Also avoid confusing a model’s explanation of its answer with proof of its actual internal computation. We can ask for an externally checkable justification: cited regions, arithmetic and assumptions. That is enough to review the result without claiming access to hidden mechanisms or treating a fluent explanation as a faithful trace of every model operation.
Back to contents · Next: 22. Evaluate the task rather than the impressiveness of a caption
22. Evaluate the task rather than the impressiveness of a caption
To evaluate a visual workflow, define the input and expected output before seeing the result. For the symbolic boards, our exact source is the printed grid plus its legend and coordinate convention. The answer key specifies counts, boxes, masks, literal text, relationships and unknowns. This lets us distinguish a correct label from a correct detection list, or an accurate transcription from an unsupported label-to-object association. A single overall impression would lose those differences.
A small evaluation record can use separate fields. Did the answer include all confirmed instances? Did it add an unsupported one? Were coordinates in the correct frame? Did the green mask exclude K6? Did text preserve T7 or Q4 exactly? Did unknown text remain unknown? Did the answer incorrectly claim hidden history? These checks are meaningful even without running a model, because they define what a later test would measure. The exercises in this article are not benchmark results; they are fully specified teaching tasks with manual solutions.
If you do run a tool later, preserve the exact file or fixture, crop selection, question, relevant settings, tool identity and response. Changing the crop while keeping the same question can change answerability. Changing the inclusion rule can change the correct count. Repeated attempts on the same tiny set may help debugging, but they do not by themselves establish broad reliability on new images. Use fresh cases with changed layouts and failure conditions to test whether the method transfers.
Measure the error that matters to the next step. A one-character transcription mistake may be more consequential than a slightly loose decorative-object box. An unsupported fourth square may invalidate an inventory even if the general scene description is excellent. For a classroom exercise, the useful evidence of progress is that a learner can locate the defect, choose the right repair and state the corrected bounded answer. The goal is a reliable connection between source and conclusion, not a competition to produce the longest or most confident description.
Back to contents · Next: 23. Transfer to screenshots, diagrams and photographs carefully
23. Transfer to screenshots, diagrams and photographs carefully
The symbolic boards make boundaries explicit. Real inputs may contain shadows, texture, perspective, transparency, glare, compression and overlapping objects. The transfer is therefore methodological rather than a promise of equal difficulty. Begin with the same questions: what is the input, what task is requested, what information survives preparation, and what output can be checked? Then add the uncertainty introduced by the actual image instead of borrowing the grid’s convenient certainty.
For a screenshot, text size and display scaling matter. A label can be legible while a nearby icon’s meaning remains ambiguous. If the task is to read a status, record the source region and literal text before interpreting it. If the task becomes clicking a control, the workflow must also check current interface state and permission; that action problem belongs to the computer-use guide. A screenshot from an earlier state is evidence about that earlier view, not proof that the same target remains available now.
For a diagram, the legend and arrows are part of the evidence. Colour alone may not encode the intended category, especially when printing or accessibility changes the appearance. A line crossing another line may or may not indicate a connection depending on the notation. Transcribing labels without reconstructing their relationships can yield a technically accurate list of words and an incorrect interpretation. The Scene A label problem is a small version of this larger issue: visible proximity does not establish the intended binding.
For a photograph, remember that image coordinates are not physical measurements without additional calibration and assumptions. An object appearing above another in the picture need not be vertically above it in the world. An apparently small object may simply be farther away. If a task needs physical size, depth or precise distance, request the required scale or geometric evidence. The grid’s ten-unit cells were explicitly defined; a photograph does not inherit that convenience. A useful assistant should explain what the image supports and what additional measurement would be needed for the stronger claim.
Back to contents · Next: 24. People and consequential decisions require stronger boundaries
24. People and consequential decisions require stronger boundaries
Visual systems can describe visible clothing, posture or the presence of objects without establishing private identity, motives, ability or character. A classroom picture can show people seated near books. It does not establish their examination results, family circumstances or the quality of the lesson. The same evidence discipline that prevents a hidden square from being invented should prevent unsupported claims about people from being attached to an attractive image.
When an image includes a person, separate observation from interpretation carefully. “A hand is raised” describes a visible configuration. “The student understands the lesson” adds a mental-state conclusion that the image alone does not establish. “This person is suitable for an opportunity” is a consequential judgement requiring relevant, appropriate evidence beyond appearance. Do not use a vision output as a shortcut for decisions about education, employment, housing or other high-impact outcomes based on unsupported personal inferences.
Similarly, an image-assisted workflow should not turn a visual impression into a medical, legal or safety verdict. A photograph may help someone communicate an observation to a qualified professional, but that does not make a general-purpose interpretation a reliable diagnosis or inspection. The repair is not simply a more confident model or a sharper crop. It may require professional examination, a validated instrument, relevant records or an authorised procedure. The appropriate evidence standard follows the consequence of the decision.
Privacy also changes the practical task. Use only images you are entitled to process and avoid sending unnecessary personal detail when a crop or redacted copy would answer the question. Preserve the relevant context while limiting unrelated information. A crop is therefore sometimes a privacy measure as well as an attention aid, though it must not remove evidence needed for the conclusion. The guiding principle is proportionate use: identify the legitimate visual question, supply the minimum suitable evidence, and keep the resulting answer within what that evidence can fairly support.
Back to contents · Next: 25. Common questions about interpreting images
25. Common questions about interpreting images
Does a patch equal an object? No. A patch is an input region under a chosen partition. It can contain part of a shape, several shapes or background. In our bookkeeping example, there are 96 patches regardless of whether the symbolic board contains two or three red squares. Object identity is a task-level interpretation. Counting patches cannot substitute for counting objects, and a patch boundary need not coincide with a meaningful boundary in the scene.
Does a readable caption prove that all text was read? No. A caption can correctly describe a board while omitting every label. OCR requires a different output and check. Our Scene A answer can classify the board and count its squares while leaving the small text unknown. That combination is coherent. The right next step is to inspect the relevant region or obtain a better source, rather than assuming that success on a general description proves detailed transcription.
Can a crop remove hallucinations? A suitable crop can reduce distraction or preserve detail from a better original, but it cannot guarantee correctness. It can also remove crucial context and create a new scope error. A-left contains one red square, while the full Scene A contains two. The crop becomes useful when its name, origin and purpose remain attached to the answer. A crop without provenance can make a wrong conclusion look easier to justify.
Should every uncertain answer be withheld? No. Separate the supported parts from the unresolved ones. Scene B supports three complete red squares, a six-cell blue rectangle and Q4. It does not establish the fragment’s complete shape, unreadable text or hidden history. Returning those distinctions is more useful than either guessing everything or refusing everything. The aim is to answer each question at the strongest level its evidence permits.
Does a vision-enabled language model always use ViT and CLIP? No universal architecture follows from the interface. These papers are representative explanations of important ideas. Specific products can use different encoders, preprocessing, specialised tools or combinations of components. A responsible account distinguishes documented mechanisms from unknown internals and evaluates the actual workflow on the task being performed. This article’s symbolic solutions do not identify any product’s architecture or measure its performance.
Back to contents · Next: 26. Your final checkpoint: follow one claim all the way back
26. Your final checkpoint: follow one claim all the way back
Choose the claim “the green shape in Scene A occupies five cells.” Locate the source: J5, K5, J6, J7 and K7. Identify the operation: enumerate occupied green cells, excluding the background at K6. Verify the number: five entries. Check the geometry: each cell has area 100 square coordinate units, so the occupied area is 500. Compare the enclosing box: it covers six cells and area 600. The difference is explained by a visible background cell, not by an unexplained confidence score.
Now choose “Scene B contains four complete red squares.” Locate the proposed fourth instance: the fragment at K5. Ask which cells establish its full square outline. The packet supplies none beyond the fragment. The claim therefore fails before any elaborate reasoning is needed. Repair it to “three complete red squares plus an unresolved red fragment,” and request an unobstructed view if the complete hidden shape matters. This is a finished diagnosis because it identifies the missing evidence and the smallest useful next step.
Finally, choose “the small code is known after zooming.” Compare the exact full view and crop. The unknown annotation remains unknown, and no additional underlying characters were supplied. The claim fails because display focus was mistaken for new information. The numeric resize fixture provides a related lesson: different originals can lead to the same reduced representation. Recovering the lost distinction requires a source that preserves it. A plausible completion is not enough to establish which original was present.
You have understood the central mechanism when you can perform these checks on a new example: identify the representation, state the task, locate the supporting evidence, distinguish the available output from the stronger unsupported claim, and choose a repair matched to the missing information. Continue with How Super Intelligence Works for the wider system, or the computer-use mechanism when a visual observation is about to become an action. Good image interpretation ends with a claim another person can inspect, including a clear account of what the image cannot settle.
