A renamed column is not necessarily a new column.
You rename a column from workshop to session. New files read correctly, but last year’s values turn into blanks in a report. Another reader returns them just fine. Neither result tells you the old files were rewritten. The readers may simply be using different ways to decide which stored field belongs in the result.
This guide follows a field through a rename, a new column that reuses its old name, and a mixed archive containing files without IDs. You will work through the expected results by hand, then design a check that distinguishes a genuine null from a lookup failure. The examples are fictional and the reader policies are named explicitly.
Keep three questions in view: what identity was assigned, what the files actually carry, and what the reader is configured to match. The field name alone cannot answer all three.
Information subject guide · Learning that transfers into life · How X Works hub
Choose your reading route
Route 1 · Chapters 1–3: Find the identity
Begin with the field that the report is asking for
Read a file schema, follow a rename and see why reusing an old name is a harder test.
Route 2 · Chapters 4–6: Explain differing read results
Read an old file that has no field IDs
Compare missing IDs, nulls and nested fields across real reader policies.
Route 3 · Chapters 7–9: Verify a mixed archive
Build a fixture that catches the plausible wrong answers
Build a fixture, preserve the evidence and transfer the method to a different change.
Full chapter index · Independent worked case · Questions and practical checks
Chapter index
Find the identity
Begin with the field that the report is asking for
Work a rename with two files and one requested field
Reuse an old name and watch the wrong values look plausible
CHAPTER 1 OF 9 · Find the identity
1. Begin with the field that the report is asking for
A Parquet file stores data in columns and carries a schema describing the stored structure. A reader may request a particular set of fields instead of loading every field in the file. That request is sometimes called a projection. The important step is matching each requested field to a stored field before interpreting the values. A table name or a spreadsheet heading does not perform that match by itself.
A name is a label used to identify a field in one schema. A field ID, when supplied, is a numeric identifier associated with a schema field. If a table’s rules keep that identifier stable across a rename, two files can store different names for the same intended field. The identifier does not mean the same thing across unrelated tables just because the number happens to match. Keep the table or schema scope alongside it.
A Parquet schema can carry field IDs, but not every file has them. Nor does their presence guarantee that a particular consumer uses them. A file reader, table-format reader, export process and reporting application can each do something different. The job of an audit is to trace the actual path from the stored file to the report, not to infer every layer’s behaviour from the file extension.
Try a paper model. Put a card marked 17 behind a folder labelled workshop. Later replace the label with session but keep card 17. A person instructed to retrieve card 17 can follow the change; a person instructed to retrieve the folder named session may miss the old version. Now imagine someone removed the cards when making a copy. The system has lost a piece of evidence, even if all the written values are still there.
CHAPTER 2 OF 9 · Find the identity
2. Work a rename with two files and one requested field
In a fictional workshop archive, file A stores a field named workshop with ID 17. Its two rows contain “Robotics” and “Printmaking.” File B, produced after a rename, stores session with ID 17. Its two rows contain “Ceramics” and “Robotics.” The table contract says these are the same field and that this rename does not change the meaning of the values. That last statement is a design premise, not something the ID can prove.
A reader requesting session with ID 17 and matching by ID will retrieve all four strings in this model. A name-only reader asking for session finds the two values in B. In A it has no exact name match. What happens next depends on the implementation: the reader might supply null for a missing column, raise an error or use a separately defined mapping. Do not call one of those outcomes a universal Parquet rule.
Suppose the report counts Robotics. Under the ID-based read it counts two occurrences across four rows. Under a name-only read that supplies null for unmatched columns, it counts one occurrence among the two matched values, or one out of four rows if the report uses all rows as a denominator. A count can change even though neither file was rewritten. Check both the matches and the report’s treatment of missing values.
Now ask a colleague what would establish a safe rename. The answer should include the same field identity across the file schemas, the reader’s effective policy, and an independent check on expected values from both files. A current table schema with the new label is useful, but it cannot demonstrate on its own what happened when old files were read.
CHAPTER 3 OF 9 · Find the identity
3. Reuse an old name and watch the wrong values look plausible
A later file C adds a new field called workshop with ID 23. It means “workshop venue” rather than “session title.” The same file also has session with ID 17. Both fields are strings. For one row, session is “Robotics” and workshop is “North Annex.” The older file A still uses workshop for the session title under ID 17. Reusing a name has created a case where exact name matching can return values from the wrong meaning, not just return blanks.
Consider a historical export that asks for workshop by name across A and C. It can put “Robotics” from A and “North Annex” from C into one column, with a valid string in every row. A schema or data-type check might pass. A semantic check should fail: the resulting column mixes a title with a venue. The export’s own label may make the result look more consistent than the evidence underneath it.
Under the model’s ID-based read, a request for 17 follows the title across the rename. A request for 23 finds the venue in C but has no such field in A. The correct outcome for an old file is not to fetch 17 merely because its old name matches 23’s current name. The two identities must remain separate. An ID-based reader still needs a defined policy for what to do when the requested field is absent.
Make the same-name test part of your review before and after a rename. A fixture containing only an old name and a new name catches missing matches, but may not catch a reader that falls back to the wrong name later. Use deliberately different values for the old and new meanings. If you use “Studio” in both fields, a bad match can pass the test by coincidence.
CHAPTER 4 OF 9 · Explain differing read results
4. Read an old file that has no field IDs
Now add file D from an early export. It has a column called workshop and the value “Drawing,” but the file schema has no ID for that field. Can an ID-based reader retrieve it by identity? Not directly from the stored ID, because that evidence is absent. An external table contract or mapping can sometimes connect the legacy name to the intended ID. Without such evidence the word workshop alone may be ambiguous.
A policy that returns null for unavailable IDs can keep a batch read running. It does not establish that the producer supplied a null. A strict policy that fails the read is useful when completeness matters more than returning a table at any cost. A name-based fallback can help when there is a trusted and unambiguous history of names. It can misattach fields when names were reused. State the policy and its consequences before choosing it.
A mapping can be credible when it is tied to the right files, original schema and known writer era. For example, evidence that D came from the original workshop-only schema may justify a mapping to 17. A general mapping that sends all stored fields named workshop to 17 would be wrong for file C. A historical mapping should not erase the identity already present in a more recent file.
If the necessary history was not preserved, mark the field unresolved for those files. Keep the rows if the report can still use their other columns. Do not turn the unresolved rows into confirmed zeros, empty strings or ordinary nulls without showing what happened. The aim is not to punish an older format. It is to avoid claiming that the archive contains evidence you can no longer connect safely.
CHAPTER 5 OF 9 · Explain differing read results
5. Distinguish a stored null from a null produced by the read
Add a row in file B whose session field is present in the file schema but whose value is null. Under the fictional producer contract, it means that the session was not yet assigned. Compare it with a null returned for file A by the name-only reader. The first is a stored missing value with a documented interpretation. The second may be the result of projecting a field name that was not present. They look identical in the final table but are evidence of different things.
Consider a report measuring unassigned sessions. Counting every displayed null would misclassify the old rows as unassigned if the reader could not find the renamed column. Use a verification column or a diagnostic report that records how a field was resolved. It need not appear in the public table, but a reviewer should be able to distinguish a known null from a value unavailable because of schema matching.
A small ledger is enough for a worked example: file A, field ID 17, matched by ID, value Robotics; file B, field ID 17, matched by ID, stored null; file D, no field ID, identity unresolved. If the reader produced the same visible null for the last two cases, the ledger preserves the difference. Do not attempt to reconstruct that difference later from the value column alone.
A blank can also reflect a legitimate field addition, a table default, a filter or an upstream conversion. The exact possibilities depend on the system. When someone says “the null rate increased,” ask whether they counted stored nulls, fields missing from some file schemas, or rows affected by a particular read policy. You may need a separate statistic for each, especially across a schema change.
CHAPTER 6 OF 9 · Explain differing read results
6. Check the reader and table format rather than guessing the default
Apache Spark’s Parquet documentation distinguishes options for writing field IDs, reading by IDs and handling files that do not contain IDs when a read schema uses them. Check the exact version and configuration used in your environment, as well as whether the requested schema actually supplies the IDs. Enabling an option is not proof that each request carried the metadata the option needs.
Apache Iceberg is a table format with its own schema-evolution rules. Its specification requires column IDs to be stored as Parquet field IDs when using Parquet and describes name mapping for files that lack IDs. A read through the table’s metadata can therefore have more context than a read pointed straight at a pile of file paths. Do not assume a generic file read is equivalent to reading the same data through its owning table.
Nested fields deserve a separate test. Suppose a record contains booking.session and followup.session. The repeated leaf name does not establish that they represent the same thing. Confirm the relevant nested paths, field IDs and parent structures. Test the nested rename using the actual reader rather than manually flattening names and assuming the result will match the table’s rules.
Record whether the consumer read a table snapshot, a file set, or an exported copy. Record the engine version, field-ID settings and any name mapping it actually used. Then test one known old record and one known new record. If those results disagree across readers, identify which stage applied a different mapping. Differences in behaviour are more useful when you can trace them to a concrete input and setting.
CHAPTER 7 OF 9 · Verify a mixed archive
7. Build a fixture that catches the plausible wrong answers
Use a small test with an old file, a renamed file, a file where the old name was reused, and a legacy file with no IDs. Give them distinct values and known row keys. Add one genuine stored null. Write expected results for a strict identity read, a name-only read and any approved legacy mapping. Keep the row-level expectations separate from totals so that matching counts cannot hide a misattached field.
Add a reordered-column case. Field identity is not the same as ordinal position. If one file stores session first and another stores venue first, a reader that chooses the first string column can return a plausible but wrong result. Test the renamed field and the reused name together in that reordered file. A good test checks the failure you actually worry about, not only a file where every possible strategy would give the same answer.
Use at least one real historical file when allowed, not just synthetic files written by a modern library. A modern fixture can be valuable and still miss an older writer’s metadata or schema quirks. Record which claims came from the controlled fixture and which were confirmed on the historical sample. A handful of files cannot establish that every file in the archive was written the same way.
Decide in advance what a failed case blocks. If a missing venue is tolerable for a display, show it as unavailable and record coverage. If a wrongly matched venue would send participants to the wrong place, do not publish that field until the mapping is verified. Different consequences can justify different release rules, but they should not change how the evidence is described.
CHAPTER 8 OF 9 · Verify a mixed archive
8. Make a reproducible change without rewriting the history by accident
Before changing a table, preserve a snapshot or another authorised record of the input files and their schemas. Keep the old name and stable identity in the change record. If the meaning itself changes, decide whether a distinct identity is required rather than stretching a rename to cover a new field. “Same ID” is only as trustworthy as the process that assigned and retained it.
After the change, compare row keys and values from before and after the rename, including old files that were not rewritten. A matching count of non-null rows is helpful but insufficient: a reused name could replace every old value with a different non-null value. Compare a sample against an independent source or known expected results. Document which readers and snapshots were exercised.
If a report changes, ask whether you changed the data, the projection, the table metadata or the report’s null policy. Change one part at a time in a diagnostic comparison where possible. A report can legitimately change when it starts recognising old values that were previously unmatched. That is a correction to interpretation, not evidence that historical source files changed.
Finally, keep a repair path. If legacy files lack IDs and no safe mapping exists yet, preserve the original files and the unresolved file list. A future recovered schema history can help without overwriting the original evidence. Do not bulk rewrite every file merely to get a consistent display until the mapping itself has passed the checks.
CHAPTER 9 OF 9 · Verify a mixed archive
9. Transfer: a rename becomes a new measurement
A fictional transit table stores waiting_minutes under field ID 7. The team renames it to queue_minutes and keeps the measurement definition. An old file says waiting_minutes = 4, and a new file says queue_minutes = 4. With suitable metadata these can be the same field. Now change one fact: the new measure excludes time spent at the ticket machine, whereas the old measure included it. Should the team keep using the old ID automatically? The label is not the deciding issue. The measurement definition changed.
If the two measures should not be compared as the same quantity, keep their identities and definitions separate or document a valid conversion where one is available. A compatible type and a stable field ID do not make different measurements equivalent. A reader can carry the right bytes into the wrong comparison if the table’s semantic contract is wrong.
Make a second change: an export strips field IDs but includes an independently verified schema reference for each file. The old name might be recoverable through that reference, depending on the evidence and reader design. What transfers is the rule to preserve identity and verify how it was resolved. What does not transfer is the idea that one setting always repairs every historical file.
Test yourself by describing what you would publish: the definition of the measure, the period over which it is comparable, which rows were unresolved, and which reader policy was verified. If the report says only “we changed the column name,” it has left out the evidence that makes the comparison safe.
Independent worked case and evidence checks
Take four fictional records with stable row keys. A contains workshop, ID 17, value “Robotics.” B contains session, ID 17, value null. C contains session, ID 17, value “Ceramics,” plus workshop, ID 23, value “East Annex.” D contains workshop with no ID and value “Drawing.” Write the result of requesting 17 under a strict reader that is required to fail on the ambiguous legacy file. It can resolve A, B and C in principle, but the combined read should not be reported as complete if the chosen strict policy fails on D.
Now test a permissive policy that marks D unresolved. The usable output contains two observed titles, one genuine stored null, and one unresolved value. If the report prints only Robotics, null, Ceramics and null, it has lost a distinction that matters. A separate diagnostic record can carry the origin of each null without changing the public display.
Suppose a verified name mapping for the legacy writer links D’s workshop to 17. Now the report gains “Drawing” without changing any original file. That mapping must not also use C’s workshop for 17. The strongest fixture checks all four records after the mapping is applied, not only the one row that was missing before.
For an independent test, ask someone to predict both a count of confirmed titles and a count of genuinely unassigned sessions. Before the mapping, these are two and one. After the mapping, they are three and one. If both figures move together, ask where a lookup failure or recovery was counted as a producer’s decision. The record-level evidence will tell you more than a new summary total.
Add one more check: file C contains the correct field IDs, but the exported read schema used for a direct-file test was generated from a list of column names. It has no requested IDs. A comparison that merely checks that the files contain IDs would miss this. Inspect both sides of the match. If a table read succeeds and a direct read fails, preserve the different requested schemas alongside the result instead of writing that one engine understands Parquet and the other does not.
Finally, suppose C and D both return the expected values in one engine, but you cannot tell whether it applied the approved mapping or simply guessed from names. Use the reused-name case to make those two behaviours diverge. Ask for ID 23 on the legacy file: it must not become Drawing. A test that forces the wrong method to produce a different answer is much stronger than another test where every method happens to agree.
Worked comparison
| What is observed | What it supports | What to check |
|---|---|---|
| Same stable field ID with a new name | Potentially the same field under the table contract | Reader uses IDs and the meaning did not change |
| Same name with a different field ID | The label was reused | Do not merge two meanings through name fallback |
| No field ID in an older file | Identity cannot be read from the file ID | Trusted legacy mapping or unresolved state |
| Null in the query result | The projection returned null | Stored null, missing field or configured fallback |
Questions and practical checks
Are field IDs mandatory in every Parquet file?
No. The general Parquet schema supports them; a table format can impose stronger rules for files it writes.
Will a Parquet reader always match by ID?
No. Check the reader, its version and configuration, and whether the read schema contains the requested IDs.
Can I reuse a deleted field’s ID?
Do not reuse identities where the governing table format forbids it or where reuse would make historical data ambiguous. Check the table contract.
Does a successful read prove the mapping was right?
No. A name-based match can return a plausible wrong string. Use known historical values and deliberately different fixture cases.
What if only some files have IDs?
Test the missing-ID policy explicitly. A verified historical name mapping may be appropriate; otherwise keep the unresolved cases visible.
Is a field ID a row identifier?
No. It identifies a field in a schema scope, not an individual record. Keep row keys and field identities separate.
Continue with the connected subjects
Compare schema identity and meaning
Schema evolution: distinguish a compatible read from a preserved meaning
Avro fingerprints and registry identity: another example of scoped identity
Natural and surrogate keys: distinguish a label from stable record identity
Safe replay: preserve the evidence that makes a historical read reproducible
Explore the underlying file format
Parquet row groups, column chunks and logical types: read the file vocabulary precisely
Sources and scope
Apache Parquet format schema: optional field ID in SchemaElement
Apache Spark documentation: Parquet field-ID read, write and missing-ID options
Apache Iceberg specification: schema evolution, Parquet field IDs and name mapping
The worked records are fictional. Verify the exact file schemas, reader versions and table-format rules used by your own archive. Sources were checked on 5 November 2026.