THE GOLD STANDARD OF INFORMATION
A high bit is not automatically a minus sign.
A counter rises from 2147483647 to 2147483648. An export suddenly shows a negative number. The source did not necessarily run backwards: the reader may have interpreted an unsigned 32-bit value through a signed 32-bit type.
This guide follows that boundary from bits to Parquet annotations, decoded values, query comparisons and downstream exports. It also tests a second trap: a large integer can remain positive yet lose its exact identity when converted to a floating-point number.
The examples are synthetic counters and identifiers. They are not instructions to reinterpret every negative value as unsigned. First establish the field’s contract, then preserve its numerical meaning through each boundary. A clever repair applied to the wrong type can damage valid data.
Information subject guide · Learning that transfers into life · How X Works hub
Choose your reading route
Route 1 · Chapters 1–3: Decode the boundary
Start with bits and their interpretation
Connect width, signedness, physical storage and expected values.
Route 2 · Chapters 4–6: Keep comparisons exact
Test filtering and downstream types
Separate correct numeric order, overflow, floating-point precision and missing values.
Route 3 · Chapters 7–9: Verify and transfer
Build a boundary-focused fixture
Repair only established mismatches and apply the method to wrapping device counters.
Full chapter index · Independent worked case · Questions and practical checks
Chapter index
Decode the boundary
1. Give the bit pattern a declared interpretation
2. Read the logical width as well as the physical carrier
3. Work the 32-bit signed boundary explicitly
CHAPTER 1 OF 9 · Decode the boundary
1. Give the bit pattern a declared interpretation
An eight-bit pattern of 11111111 can describe unsigned 255 or signed two’s-complement −1. The pattern alone does not choose. Width and interpretation supply the meaning. This small example is the same conceptual problem as a much larger counter crossing the signed boundary in a data pipeline.
For an unsigned n-bit quantity, the mathematical range is 0 through 2ⁿ−1. A signed two’s-complement n-bit quantity ranges from −2ⁿ⁻¹ through 2ⁿ⁻¹−1. These ranges overlap for small non-negative values, which is why an incorrect reader can appear reliable for months before the first high value arrives.
Take the unsigned eight-bit values 126, 127, 128 and 255. A mistaken signed interpretation agrees on 126 and 127, then reports −128 and −1. A fixture containing only 0, 1 and 10 cannot distinguish the interpretations. Test where the meanings diverge, not only where they happen to agree.
Write the intended field definition before opening a hex viewer. “Unsigned device count with values from 0 through 255” gives a testable contract. “A number returned by the sensor” leaves the decisive questions unanswered. The field name, a positive sample and a familiar chart are not substitutes for that contract.
The existing Computing data-representation guide develops binary, hexadecimal and signed-byte reasoning. Here we carry that reasoning into a file and a consumer: a correct bit-level explanation is useful only if the saved annotation, reader type and downstream comparison continue to honour it.
CHAPTER 2 OF 9 · Decode the boundary
2. Read the logical width as well as the physical carrier
Parquet’s INT annotation specifies a bit width and whether the value is signed. Unsigned widths 8, 16 and 32 use the INT32 physical carrier; unsigned width 64 uses INT64. A logical eight-bit column therefore does not mean every value occupies one byte in PLAIN encoding.
For a simple PLAIN-encoded UINT8 value of 255 in its INT32 carrier, the payload is FF 00 00 00. This is four-byte little-endian integer storage. Reading only the first byte as a signed eight-bit value produces −1, but that is not a valid way to decode the declared column.
The logical width still imposes a range. A physical INT32 can hold values outside the UINT8 range, but a writer must not use that extra capacity to smuggle in 256 under an eight-bit annotation. A saved carrier that has enough bits is not automatically a valid logical value.
For an inspection record, retain physical type, logical bit width, signedness and value encoding. Add the reader’s exposed in-memory type. Those entries can explain why two consumers show different numbers without guessing that the underlying file changed between reads.
Keep our byte examples scoped to PLAIN value payloads. Other encodings and compression introduce additional structure. A file’s visible sequence of bytes is not a row-by-row list of four-byte integers unless the surrounding format evidence establishes that layer. Decode the enclosing page before interpreting a selected payload.
CHAPTER 3 OF 9 · Decode the boundary
3. Work the 32-bit signed boundary explicitly
The largest signed 32-bit value is 2147483647. The next unsigned 32-bit value, 2147483648, has hexadecimal bit pattern 80000000. Under the PLAIN little-endian payload representation, its bytes are 00 00 00 80. The highest-order bit is set, but the UINT32 logical meaning remains positive.
If that same 32-bit pattern is interpreted as signed two’s complement, its value is 2147483648 − 4294967296 = −2147483648. This subtraction explains the apparently dramatic sign change. The value did not merely lose a display symbol; it crossed into a different interpretation of the same width.
At the top unsigned boundary, FFFFFFFF represents 4294967295. A signed 32-bit interpretation gives −1. Do not generalise this into “−1 always means the maximum counter.” In another field, −1 may be a genuine signed quantity or an explicitly documented status code. The annotation and source contract decide whether the reinterpretation is relevant.
An exact signed 64-bit destination can represent every UINT32 value. Widening into that type can preserve the numerical value, provided the converter interprets the source correctly before widening. Widening an already misread −1 yields signed 64-bit −1, not 4294967295. More destination bits cannot undo a mistaken source interpretation by themselves.
Use the sequence 2147483646, 2147483647, 2147483648 and 4294967295 as an expected-answer fixture. Ask the consumer to display values, order them and compare them with a threshold. This reveals more than a success message from a write operation: it tests the arithmetic decisions a real reader will make.
CHAPTER 4 OF 9 · Keep comparisons exact
4. Compare and filter under the right numeric order
Suppose a fictional archive contains three unsigned 32-bit counters: 10, 2147483648 and 4294967295. A query asks for values greater than 3000000000. The correct result contains only 4294967295. Calculate that answer independently before running the query.
A reader that treats the high values as signed sees 10, −2147483648 and −1. None exceeds the positive threshold. The result is empty, which can look like a legitimate business outcome. This is a silent semantic failure: the query completes without an obvious decoding exception.
A metadata shortcut can make the same mistake earlier. If bounds or literals are compared under the wrong signedness, a reader may exclude a data region that actually contains a match. The existing statistics guide explains the logic of safe exclusion. That logic depends on a correct ordering of the represented values.
Keep a full decoded scan under a known-correct interpretation as the fixture baseline, then compare the filtered path where the application supports such a test. Do not assume that a setting called “disable statistics” exists in every tool; inspect the actual consumer. The objective is to separate value decoding from an optimisation that may avoid reading values.
Sorting also needs its own expected answer. Unsigned order for our three values is 10, 2147483648, 4294967295. Signed order of the reinterpreted patterns would place the two negative results before 10. A small ordered fixture makes the wrong rule visible without a performance benchmark or a large dataset.
CHAPTER 5 OF 9 · Keep comparisons exact
5. Treat 64-bit capacity and floating-point exactness as separate limits
An unsigned 64-bit quantity extends to 18446744073709551615. That is larger than the maximum signed 64-bit quantity, 9223372036854775807. There is no lossless conversion of the entire UINT64 range into signed INT64. A source sample that happens to stay small cannot prove that the declared full range will fit.
A floating-point destination introduces a different issue. Binary64 can exactly represent many integers, but it cannot represent every consecutive integer above 2⁵³. The values 9007199254740992 and 9007199254740993 are adjacent integers; a binary64 conversion can map both to the same value. Positivity and a large apparent numeric range do not guarantee exactness.
For an identifier, that collision is especially serious. Two distinct keys can become one displayed number and produce a false join. For a counter, a one-unit change can disappear. Decide whether exact integer preservation is required before choosing a destination type; do not infer suitability from the number of digits a spreadsheet happens to show.
Possible exact handovers include a genuinely supported unsigned type, an adequately sized exact decimal integer type or a documented decimal-string representation. Each has a contract to verify. A string must not later be parsed through a lossy number, and its lexical sort order is not automatically numeric order.
RFC 8259 discusses JSON number interoperability and the range of consecutive integers that common binary64-based implementations agree on exactly. JSON syntax does not itself force every parser to use binary64, so test the actual endpoints. A JSON payload containing a long integer token may be valid while a particular consumer silently approximates its value.
CHAPTER 6 OF 9 · Keep comparisons exact
6. Keep null, reserved values and counter semantics distinct
A counter at zero is a present value. A null counter is absent. An all-ones bit pattern is a large unsigned value unless the application assigns it a special role. These distinctions should survive export instead of being compressed into one generic “no data” label.
Imagine a protocol reserves 4294967295 as “reading unavailable.” That is an application convention layered over a representable UINT32 value. Preserve the source token or transform it under an explicit rule with a corresponding missing-state field. Do not decide from the magnitude alone that the reading must be an error.
Conversely, if the protocol uses the full range as a genuine count, converting its maximum to null throws away valid information. A generic import rule inherited from another device can be wrong even when both devices use four-byte counters. The source protocol, not a habitual sentinel convention, defines the meaning.
A useful fixture includes zero, null, the signed-boundary values and any reserved application value. State expected presence counts and expected numeric results separately. A total can remain plausible after a missing reading becomes zero, while the average’s denominator changes. Correct arithmetic on an incorrect population still gives the wrong report.
Do not infer counter behaviour from storage type alone. A count may reset on reboot, wrap at a fixed width or start a new epoch after maintenance. Unsigned storage only tells you how individual values are represented. Interpreting differences between readings requires the device’s reset, rollover and sampling contract as additional evidence.
CHAPTER 7 OF 9 · Verify and transfer
7. Build a fixture around the boundaries a sample would miss
For UINT32, include 0, 1, 2147483647, 2147483648, 4294967295 and null. Add 4294967296 as an intentionally out-of-range input that the proposed contract should reject. Do not silently reduce it modulo 2³² just because a low-level cast can do so.
For UINT64, include values on both sides of 2⁶³ and the maximum. Add 9007199254740992 and 9007199254740993 to detect a binary64 detour. Give every row a stable text key so two approximated values cannot disappear into one dictionary entry before you compare the results.
Predict the results of equality, ordering and one threshold filter. Also predict accepted and rejected inputs. Then inspect the actual saved annotation and the reader’s exposed type. A writer can choose another compatible representation under its options; the observed schema is evidence you should record, not something to assume from the source language type.
Apache Arrow’s writer documentation describes format-version choices and the availability of particular logical types. That is implementation-specific information, not a guarantee about all readers. Record the installed writer and consumer versions and retain their settings with the fixture receipt.
Use a deliberate wrong-path control in a safe local test: reinterpret the high UINT32 pattern as signed, or pass the adjacent large integers through binary64. Confirm that the expected-value check fails. A test that cannot distinguish the feared wrong result from the right one is not ready to justify a migration.
CHAPTER 8 OF 9 · Verify and transfer
8. Repair established interpretation errors without guessing from signs
Suppose the source contract and saved annotation establish UINT32, but one export contains −2147483648 where the expected value is 2147483648. The likely boundary is a signed 32-bit interpretation. Confirm it with the source bytes or an authoritative decoded value before applying a correction.
For a proven 32-bit signed reinterpretation, a negative result can be related to the original unsigned value by adding 2³². This is a mathematical explanation of that specific representation error. It is not a general cleaning rule for negative numbers. An already widened, rounded or otherwise transformed export may require different evidence.
If the discrepancy first appears after conversion to binary64, the original integer may no longer be recoverable uniquely from the approximate number. Return to the exact source rather than choosing the nearest attractive identifier. Two source integers can share the same approximation; reversing that mapping without source evidence invents a fact.
Preserve the original and failing outputs, repair the conversion boundary, and regenerate a controlled destination. Recheck joins, thresholds and derived aggregates that consumed the bad values. If a report used an empty threshold result as evidence that nothing happened, correcting the displayed counters is only part of the repair.
Close with a clear population statement: which source type and records were checked, which conversion was corrected, which consumers passed the boundary fixture and which records remain unresolved. A sentence such as “all negatives fixed” conceals the very distinction the repair depends on.
CHAPTER 9 OF 9 · Verify and transfer
9. Transfer from stored values to a wrapping device counter
A fictional eight-bit device counter reads 250 and then 5. If the device is known to wrap modulo 256, did not reset, and advanced less than 256 counts between readings, the increment is (5 − 250) modulo 256 = 11. That is a changed-context problem: individual unsigned values are correct, but their difference needs a process model.
The same readings could also follow a reset, or more than one wrap. Without the stated conditions, 11 is not a uniquely established total. A longer interval could contain 267, 523 or other increments compatible with repeated wraps. The storage type cannot decide which history happened.
A second field such as a reboot marker, counter epoch or independent accumulated total may narrow the possibilities. Its reliability needs checking too. Do not manufacture certainty by assuming the most convenient wrap count whenever a reading decreases.
Ask a learner to compare three cases: a signedness bug, a genuine modular rollover and a device reset. Which evidence distinguishes them? The first concerns interpretation of one stored pattern, the second concerns a known counter process, and the third concerns a discontinuity in that process. Applying one correction to all three is not transferable understanding.
Carry that habit into any numerical handover. Establish the type, unit, range and process meaning; identify the boundary where the question changes; then design a test that can reject the plausible wrong story. Exact bytes, exact integers and correct real-world conclusions are connected, but they are not interchangeable achievements.
Independent worked case: an identifier survives Parquet but not its dashboard export
A fictional catalogue has two rows, K1 with unsigned integer key 9007199254740992 and K2 with key 9007199254740993. Both are valid UINT64 values. The Parquet reader returns them exactly. A dashboard connector then converts both through binary64 before serialising a report. Predict which test should expose the damage.
The original keys differ by one. In the common binary64 conversion used for this exercise, both become 9007199254740992. Row count remains two. The values stay positive and large. A screen that shows abbreviated scientific notation may look unchanged. An exact keyed comparison reveals that K2 no longer carries its assigned identity.
A join to an exact catalogue can now associate both report rows with K1 or fail according to the surrounding type conversions. The exact outcome is implementation-dependent, so do not invent one universal query result. The invariant already failed before the join: the connector collapsed two distinct integer values.
A proposed repair exports the keys as decimal strings with a documented unsigned-integer interpretation. Test that the endpoint preserves the full digits and that no later consumer converts them back through binary64. Also test numeric ordering deliberately: plain lexical order would put “10” before “2,” so exact text preservation does not by itself solve every analytical operation.
The final acceptance receipt should state two exact source keys, two distinct destination keys, the agreed downstream parsing rule and the expected mapping from K1 and K2 to their catalogue rows. Do not say the file was corrupt when the first lossy boundary was the connector. The correct diagnosis makes the repair smaller and the conclusion more honest.
Add a third key, 18446744073709551615, the largest UINT64 value. A destination signed INT64 cannot hold it. Under an exact-only migration, an overflow exception is an appropriate result, not evidence that the source value should be clamped. A connector that replaces it with the largest signed value has changed the key while concealing the range failure.
A decimal-string destination can preserve all twenty digits. Validate that the string contains the exact expected decimal representation and fits the declared application range. Do not accept an arbitrary long string merely because the export type is text. Type validation moves to an explicit parser; it does not disappear.
For a numeric threshold test, choose 9223372036854775807. K1 and K2 are below it; the maximum UINT64 key is above it. A signed reinterpretation of that maximum produces −1 and gives the wrong answer. This exercise distinguishes signedness failure from the earlier floating-point collision: they can affect the same pipeline at different stages.
For a lexical-order test, add small keys 2 and 10. An ascending string sort puts 10 before 2, whereas numeric order puts 2 first. If the interface requires numeric ordering, specify a correct numeric parser or an explicitly designed order-preserving representation. Padding strings is a separate design choice with its own width and validation rules, not an automatic property of decimal text.
Now separate record identity from numerical magnitude. An application may use integer-shaped keys purely as labels and never add them. It still needs exact equality. Saying that a one-unit difference is negligible because the numbers are large misunderstands their role: the difference can name another record rather than a small measurement error.
For a final changed-context check, replace the keys with millisecond-free sequence counters. Losing a unit now hides an event increment instead of merging identities. The same representation defect has a different operational consequence. Explain both from the same invariant: the downstream value must equal the exact source integer unless a separately declared transformation intentionally changes it.
Reconcile every source row, including rejected overflow and missing values. A result containing fewer rows can be acceptable only if the exceptions are explicit and the task permits them. A claim of lossless completion requires more than correct values among the surviving rows; it also requires an account of the rows that did not survive.
Keep these distinctions visible
| Boundary | Useful fixture | Failure it exposes |
|---|---|---|
| Signed versus unsigned 32-bit | 2147483647 and 2147483648 | High values become negative |
| Logical width versus carrier | UINT8 value 256 as rejected input | Oversized physical values accepted |
| UINT64 versus signed INT64 | 2⁶³−1 and 2⁶³ | Destination range is too small |
| Integer versus binary64 | 2⁵³ and 2⁵³+1 | Adjacent identities collapse |
| Counter process | 250 then 5 with stated conditions | Wrap inferred without evidence |
Questions and practical checks
Does INT32 always mean a signed business value?
The physical carrier alone is not the whole contract. An unsigned logical annotation can refine its interpretation. Inspect physical type, logical width and signedness, then compare the expected decoded value.
Can every UINT32 fit exactly into signed INT64?
Mathematically yes: the full UINT32 range is below the signed 64-bit maximum. The converter must still interpret the source as unsigned before widening. Widening an already misread negative number preserves the mistake.
Can every UINT64 fit into a floating-point number exactly?
No. Binary64 cannot preserve every consecutive integer in the UINT64 range. Test adjacent values around 2⁵³ and use an exact destination contract when identity or unit increments matter. A large numeric range is not the same as exact precision.
Should I add 2³² to every negative export?
No. That relation applies to a proven 32-bit signed reinterpretation of unsigned data. Genuine signed values, status codes, resets and rounded exports need different treatment. Establish the source contract and first changed boundary before repairing.
Does a decreasing counter prove rollover?
No. It may be a rollover, reset, source switch or interpretation error. A modular increment is justified only under stated process conditions, including limits on wraps between observations. Preserve uncertainty when the evidence cannot distinguish histories.
What is the most useful practical check?
Predict exact values, sort order and a threshold result for a tiny boundary fixture. Then verify the saved annotation and actual consumer output. Include a wrong-path control so you know the check can detect the error you are worried about.
Continue by the question you need to answer
Build the representation foundation
Read these for file layers, field identity and historical contracts before changing a numerical representation.
Parquet structure and documentation overview
Schema evolution and semantic drift
Field IDs, column renames and historical reads
Make query shortcuts defensible
Bounds, page selection, dictionaries and membership filters depend on correct type interpretation. Use these alongside the exact expected values in this guide.
Row-group statistics, predicate skipping and nulls
Page indexes, row ranges and selective reads
Dictionary fallback and page encoding checks
Bloom filters and physical value encodings
Keep identity, absence and integrity separate
These routes explain why a valid key, a missing value, intact bytes and a faithful decoded quantity require different evidence.
Natural keys, surrogate keys and identity drift
Null, blank, zero and the meaning of missing
Page checksums and corruption boundaries
Canonical serialization and stable verification
Sources and scope
The guide separates unsigned value representation from application-level sentinel rules and counter processes. Arithmetic vectors are original synthetic tests. Exact downstream behaviour must be established for the actual writer, reader, query engine and export path; no production compatibility result is claimed.
Apache Parquet integer logical types — Logical widths, signedness, carriers and comparison order.
Apache Parquet PLAIN encoding — Integer payload widths and byte order.
Apache Arrow writer reference — Check version-specific logical-type support and options.
RFC 8259: JSON — Number interoperability and common exact-integer limits.
Python integer and byte conversion reference — Explicit byte order and signedness for independent fixture calculations.
The scenarios and fixture values in this guide are original teaching examples, not reports about a live organisation. Format and implementation references were checked on 6 November 2026. The checking sequences are practical teaching suggestions; they do not certify a particular reader or production dataset.
Return to the Gold Standard of Education · Explore How X Works
Related representation checks
Parquet Decimals, Scale Changes and Exact Value Preservation — Preserve exact quantities when decimal scales or physical carriers change.
Parquet Timestamps, UTC Instants and Local Clock Meaning — Distinguish resolved instants from local calendar and clock fields.
Parquet UUIDs, Byte Order and Identity-Preserving Joins — Keep established identifiers unchanged across UUID and GUID byte layouts.