Skip a row group only when it cannot contain an answer.
A report asks for every delivery that weighed more than 80 kilograms. One block of stored records has a trustworthy upper bound of 70. A reader can skip that block without opening its rows. Another block spans 60 to 95. That range is not an answer: it only tells you not to rule the block out using this information alone.
This guide works through numeric bounds, missing null counts and a compound query. It then shows how to test that a faster reader still returns the same records. All datasets are fictional, and the logic is scoped to the stated predicates and reliable metadata.
Remember the distinction: a summary can sometimes prove that no row qualifies. A group that survives that check still needs ordinary filtering unless separate evidence proves exactly what can be returned.
Information subject guide · Learning that transfers into life · How X Works hub
Choose your reading route
Route 1 · Chapters 1–3: Learn the exclusion rule
Start with an unopened box of rows
Read bounds as one-sided evidence and distinguish a stored null from a missing statistic.
Route 2 · Chapters 4–6: Work harder predicates
Combine conditions without turning possibility into proof
Check Boolean expressions, type ordering and bounds that are not actual observed values.
Route 3 · Chapters 7–9: Verify and transfer
Build a test that can reveal a lost row
Compare full scans with pruned reads and transfer the proof to a different search.
Full chapter index · Independent worked case · Questions and practical checks
Chapter index
Learn the exclusion rule
Start with an unopened box of rows
Work the boundaries before trusting a shortcut
Keep three meanings of missing separate
CHAPTER 1 OF 9 · Learn the exclusion rule
1. Start with an unopened box of rows
A row group is a set of rows stored together in a Parquet file, with the values for each stored column organised in its own column chunk within that group. A reader does not necessarily have to read every chunk to answer a query. The file metadata may supply statistics for a column chunk. Those statistics can sometimes rule out the group before the relevant values are fetched or decoded.
Imagine a paper box containing delivery slips. A trusted label says that every non-null weight in the box lies between 42 and 70 kilograms. You are looking for weights greater than 80. No slip in that box can satisfy the comparison under ordinary SQL filtering. You can leave it closed. If the label says between 60 and 95, you cannot conclude that every slip qualifies. You also cannot conclude how many qualify.
A predicate is a condition applied to rows, such as weight greater than 80. A query engine may push a condition closer to storage so the reader can do less work. Predicate pushdown covers several possible optimisations; this article focuses on deciding whether an entire row group can be skipped using trustworthy evidence. The exact work saved depends on the reader and the available metadata.
Use one question to guide every example: could any row in this group make the condition true? If the evidence proves no, skipping can be sound. If the answer is maybe, the summary has done all it can for that condition. Read more evidence or apply the filter to rows. Confusing maybe with no is how a performance improvement starts dropping valid results.
CHAPTER 2 OF 9 · Learn the exclusion rule
2. Work the boundaries before trusting a shortcut
Use three fictional groups with integer weights. Group A has a trustworthy upper bound of 70. Group B has bounds 60 and 95. Group C has an upper bound of exactly 80. For the condition weight greater than 80, A and C can be excluded: 80 does not pass a strict greater-than comparison. B remains a candidate because its bounds do not rule out values above 80.
Change the condition to weight greater than or equal to 80. A is still excluded. C can no longer be ruled out by its upper bound alone because 80 would now qualify. Notice how a tiny change in the query changes a correct optimisation. A reviewer should see the original comparison operator, not a paraphrase such as “high weights,” which hides the boundary case.
For equality, a requested value below the lower bound or above the upper bound cannot appear among the non-null values. A value between the bounds need not exist. If a group contains only 60 and 95, then 81 lies inside the interval but does not occur. A dictionary or another suitable index might answer a narrower question, but a pair of bounds alone cannot certify membership.
For a bounded interval, use the same logic at both ends. If the request is from 81 through 90 inclusive, a group with upper bound 80 or lower bound 91 cannot qualify. A group spanning 60 through 95 may still contain no values in the requested interval. Keep the distinction between overlap of possible ranges and existence of actual matching records.
CHAPTER 3 OF 9 · Learn the exclusion rule
3. Keep three meanings of missing separate
A null is a possible state of a value under a data schema. A missing null count is an absent statistic. A missing column is a separate schema or projection issue. All three can lead to blanks in a debugging display, but they do not justify the same decisions. Before using a null count, make sure the statistic describes the correct stored column and the relevant group.
In a flat optional column, a verified null count of zero can rule out a predicate asking for nulls in that group. If the count is absent, the Parquet format explicitly tells readers not to assume it is zero. The correct inference is unknown. There might be no nulls or many. Read more evidence if the difference matters to the query.
For ordinary SQL comparisons, null does not make a condition such as weight greater than 80 true. In a WHERE filter, rows pass only when the predicate is true. But IS NULL explicitly selects nulls. A reliable maximum of 70 is enough to exclude the group for weight greater than 80, even with an unknown null count, because neither values within bounds nor nulls can make that comparison true. The same maximum tells you nothing decisive about IS NULL.
If you have verified metadata that every row in a flat scalar column is null, ordinary comparisons on that column cannot pass. Do not extend that shortcut mechanically to repeated or nested fields by comparing a column-value count with a row count. The unit of the statistic and the semantics of the query must match. Check the reader’s treatment of absent structures, null parents and repeated entries.
CHAPTER 4 OF 9 · Work harder predicates
4. Combine conditions without turning possibility into proof
Consider a flat weight column with upper bound 70 and unknown null count. The filter is weight greater than 80 OR weight IS NULL. The first branch is impossible, but the second is unresolved. You cannot skip the group. If a reliable null count of zero is also available, both branches are impossible and the group can be excluded. The whole condition, not the strongest-looking part, controls the decision.
Now change OR to AND between weight greater than 80 and a condition about a delivery route. If no weight can exceed 80, no row can satisfy both conditions, regardless of what the route metadata says. For an AND of two ordinary requirements, an impossible branch is enough to rule out the whole condition. For OR, each branch must be ruled out before that reasoning applies.
Separate statistics for two columns do not show which values appear together on the same row. Imagine a group with one 90-kilogram parcel on route North and one 50-kilogram parcel on route South. Each condition, weight greater than 80 and route South, is possible somewhere in the group. The conjunction is false for both rows. Without evidence about co-occurrence, retain the group and filter the rows.
Be careful with NOT. Under SQL’s treatment of nulls, NOT of an unknown comparison is still unknown, not true. Also, knowing that some row might satisfy a condition does not establish that some row might satisfy its negation in the way a particular optimisation assumes. Work from the actual predicate and truth rules. If the reader cannot safely express the condition as an exclusion test, it can still apply it after reading.
CHAPTER 5 OF 9 · Work harder predicates
5. Compare values in the right order and representation
Bounds are useful only under a comparison order compatible with the query. For integers with one well-defined representation, the intended order is usually easy to see. Strings, decimals, floating-point values and logical types require more care. Do not compare display strings to numeric bounds or assume a bytewise order means the same thing as a language-aware text comparison.
Suppose a text field stores “100” and “20.” Under a simple lexicographic order, “100” can come before “20,” while 100 is greater than 20 in numeric order. If the query casts the field to an integer, raw text bounds cannot automatically rule out numeric matches. A reader would need a justified translation of the predicate or a compatible statistic. Otherwise the safe choice is to read and evaluate the cast.
The Parquet specification distinguishes older min and max fields from the newer min_value and max_value fields, and ties interpretation to the applicable ordering rules. A reader must know which fields it received and whether it can interpret them for the column type. Do not silently substitute one sort convention for another because the numbers print the same way in a tool.
Floating-point edge cases deserve deliberate fixtures. The format has specific guidance for values such as signed zero and NaN. A summary that does not list NaN as a bound does not by itself prove that NaN cannot appear in a group. Use the exact type and predicate semantics supported by the engine, and test the affected queries with pruning on and off rather than guessing from a printed minimum.
CHAPTER 6 OF 9 · Work harder predicates
6. Treat conservative bounds as fences, not necessarily values
Some writers can store bounds that are not actual values in the group, provided the bounds remain suitable for exclusion in the specified order. This can matter for long strings, where storing an entire extreme value may be costly. An indicator of whether a bound is exact is not the same thing as permission to ignore the ordering or trust an arbitrary truncated string.
Picture a fictional set with actual values 62, 65 and 68. A reliable enclosing lower bound of 60 and upper bound of 70 is weaker than the true extrema but still safe for exclusion. It rules out a query for 72. It cannot rule out 69 even though 69 is absent. That loss of pruning power is acceptable: it costs extra reading, not lost answers.
A bound on the wrong side is different. If someone reports an upper bound of 66 when a value of 68 exists, a query for values greater than 67 can wrongly skip the entire group. The problem is not merely that the bound is inexact. It is not an upper bound at all. For any proposed truncation rule, ask which direction it can move the boundary and whether the resulting value still encloses every relevant value.
Do not use row-group bounds as substitutes for actual minima in a public report unless exactness and the calculation’s other assumptions are verified. A conservative upper bound may help a reader avoid work without being a measurement that occurred. The same metadata can be fit for one job and unfit for another. State which job you are using it for.
CHAPTER 7 OF 9 · Verify and transfer
7. Build a test that can reveal a lost row
Make a fixture with stable row IDs and several groups: a group wholly below a threshold, one ending exactly at it, one straddling it, one with a known zero null count and one without a recorded null count. Include at least one genuine null. Write expected matching row IDs for both strict and inclusive comparisons. Then add an OR query that can keep a group because of a null even when all non-null values are below the threshold.
Use an ordinary row-level scan as a comparison, taking care that both paths use the same input files, schema mapping and query semantics. Compare results as multisets when duplicate rows matter, or use stable row IDs to compare like with like. A matching row count can miss a wrong substitution. A different output order need not be wrong unless the query promises an order.
Check optimisation separately from correctness. Record which groups were read or pruned and which bytes or chunks were needed, using whatever trustworthy metrics the engine exposes. A reader may legitimately read more than the theoretical minimum because an optimisation is unsupported or because metadata are missing. That is a performance result, not evidence that the filter returned incorrect rows.
Do not validate solely with modern files written by the same engine that will read them. Include representative writer versions and any historical files relevant to the archive. Verify behaviour when a statistic is absent, marked inexact or unsuitable for the selected type. A test that can fail on the plausible wrong case is more valuable than a dashboard showing only faster execution.
CHAPTER 8 OF 9 · Verify and transfer
8. Diagnose a discrepancy without deleting evidence
Suppose a faster run drops a known matching row. First identify its file, row group and stored column. Then inspect the metadata and compare the query actually used by the reader with the one intended by the application. Was a comparison changed at an inclusive boundary? Was a missing null count treated as zero? Was a cast or field mapping assumed safe when it was not?
Check the same file with a trusted alternative path and preserve the original metadata. If you disable a pruning rule for diagnosis, record that choice and the version used. Avoid rewriting the file solely to make the test pass before you understand the mismatch. A rewrite can remove the evidence needed to distinguish an old writer issue from a reader interpretation issue.
A useful report distinguishes three outcomes: which rows were verified, which groups were read and which exclusion decision was unsupported. If the correctness result is unresolved, do not let the improved performance figure stand in for it. Report the verified scope, the discrepancy and which configuration is suitable for the intended use while the investigation continues.
Ask a second person to reproduce one exclusion from the predicate, type and statistic alone. Then ask them to identify a group that must remain unresolved with the evidence provided. If they cannot do either without knowing the expected final count, the evidence chain is not clear enough. Preserve the reason for each decision, not just the number of skipped groups.
CHAPTER 9 OF 9 · Verify and transfer
9. Transfer: use a summary from a different system
A fictional archive keeps attendance sheets in envelopes. Each envelope is labelled with the earliest and latest date found on its sheets. To search for 12 May, you can rule out an envelope whose verified latest date is 11 May. An envelope spanning 10 to 14 May might contain nothing from 12 May. The safe exclusion logic transfers even though the storage format does not.
Now ask for every sheet without a date. A label giving earliest and latest dates does not tell you whether some sheets were undated. If the archive also keeps a trustworthy count of undated sheets, the decision can change. The new condition needs different evidence. The labels were not wrong; they were answering a narrower question.
Change the setting again: the labels represent dates as entered, while the search uses a revised calendar field after a conversion. A comparison is only safe if the label’s order and scope can support the revised query. The equivalent of a cast or schema mismatch has appeared outside software. You still need to connect the summary to the condition before excluding anything.
Finish by writing two sentences. The first should justify a safe exclusion. The second should name the evidence that is missing for a case you cannot exclude. If you can make both statements without claiming that an unopened envelope contains a particular sheet, you have understood what the summary can actually do.
Independent worked case and evidence checks
Use an independent fictional group of three rows with values 12, null and 18. Suppose its footer has trustworthy bounds 10 and 20, but no null count. A filter for values above 20 cannot pass on these bounds and ordinary SQL comparison semantics. A filter for values at least 20 cannot be ruled out by the bounds, even though the actual rows do not contain 20. A filter for null must also retain the group because the missing count cannot settle it.
Now run the hypothetical filter value above 20 OR value IS NULL. It must retain the group and ultimately return the null row. If a faulty implementation treated the absent count as zero, it could incorrectly skip the group. This fixture catches the bug because the query returns a known stable row ID on the comparison path. It does not rely on an explanation inferred from speed.
Add a second group with an exact maximum of 20 and a known zero null count. Under the strict OR filter it can be skipped; under an inclusive filter for values at least 20 it cannot. This deliberately pairs a missing-statistic case with an exact-boundary case. A reader can get one right and the other wrong, so keep both expected decisions in the test.
For a final check, imagine a different query also requests another column only after a match. Avoid claiming that all readers do the same physical work even when the results agree. One might prune a whole group; another might read only selected chunks or pages; another might scan more. Separate the result contract from the implementation metrics and record which was actually observed.
Try a three-valued check with actual rows rather than a rule of thumb. A group contains the values 18, 18 and null. Under value NOT EQUAL TO 18, neither 18 qualifies and the null does not make the comparison true. The result is empty. If a reliable pair of numeric bounds both equal 18, those bounds support excluding the group for this ordinary comparison. Now replace the filter with value IS DISTINCT FROM 18 in SQL. This condition treats null as distinct from 18, so the null row qualifies. Identical bounds cannot support the same exclusion unless additional evidence rules out nulls.
A changed expression may be different even when the query text looks almost the same. Consider COALESCE(value, 100) greater than 80: for a null input, the expression uses 100 and the row passes. A bound below 80 on the stored non-null values is not enough to skip the group. To apply an exclusion, the reader would need evidence about nulls as well as the non-null bounds and would need to implement the expression correctly. A tool that only recognises simple comparisons can leave this predicate for normal evaluation without affecting correctness.
There is another useful distinction between a logically sound decision and an engine feature. For a known zero null count and upper bound below 80, a person can reason that the COALESCE filter cannot pass in this example. An engine might nevertheless read the group because it does not push down that function or because the metadata is not trusted for a particular writer. Record what the engine did and why it is safe, not what you think every engine ought to do. This keeps a missing optimisation from being mistaken for a missing row.
Finally, change only the column mapping. The query now asks for the renamed weight field, but the inspected bound came from an old field with a reused name that represented a shipping limit. Even a mathematically perfect bound is irrelevant to the predicate if it describes the wrong values. Verify the stored field identity, then the comparison order, then the exclusion logic. Each step can fail independently. A robust test fixture should make it possible to detect which one failed.
Worked comparison
| Evidence | Question | Safe decision |
|---|---|---|
| Upper bound 70 | Can weight exceed 80? | Exclude under the stated numeric ordering |
| Bounds 60 to 95 | Does weight 81 occur? | Do not assume either presence or absence |
| Null count missing | Can a null match IS NULL? | Unresolved; do not substitute zero |
| Upper bound 70, null count missing | Can weight exceed 80 OR be null? | Retain because the null branch remains possible |
Questions and practical checks
Does predicate pushdown change the result?
A correct optimisation preserves the query result. Check it against an equivalent row-level scan.
Do min and max prove a value between them exists?
No. They bound the values under an applicable order, but do not establish membership.
Is an omitted null count zero?
No. The Parquet format explicitly rejects that inference.
Can a conservative bound still be useful?
Yes, if it safely encloses all relevant values in the applicable ordering. It may exclude fewer groups than an exact bound.
Why keep a group when both columns might match separately?
Column summaries alone do not show whether their values co-occur in the same row.
Why did my engine read a group that looked skippable?
A required statistic, supported predicate or trusted ordering may be missing. Extra reading is preferable to excluding a valid result.
Continue with the connected subjects
Follow field identity and evidence
Parquet field IDs: verify that the predicate is looking at the intended stored field
Schema evolution: keep a compatible representation from silently changing meaning
Counting Bloom filters: another case where an unsafe negative can lose an answer
Understand storage and regression checks
Parquet row groups, chunks and statistics: read the underlying file vocabulary
Result-set comparison: preserve multiplicity when checking a changed query path
Sources and scope
Apache Parquet format: statistics fields, ordering and null-count rules
Apache Arrow documentation: Parquet metadata and predicate pushdown
PostgreSQL documentation: SQL comparisons and null predicates
The records and statistics in the worked examples are fictional. Test the actual reader, writer, data types and query semantics before relying on a pruning decision. Sources were checked on 5 November 2026.