The Gold Standard of Information | Parquet Modular Encryption, Column Access and File-Identity Checks

THE GOLD STANDARD OF INFORMATION

Encrypted columns still need a clear access and identity contract

An analyst is allowed to read totals but not names. A file contains both. Encrypting the storage disk does not by itself create that distinction for someone who can download the file. Parquet modular encryption can protect different parts with different keys, but the surrounding access and verification rules still matter.

This guide follows a fictional research export from protection choices to selective reads, failed access and recovery. It explains the information boundaries rather than offering a production cryptographic configuration. Use supported libraries and your organisation’s security review for real sensitive data; the worked fixtures should contain no real personal information.

Information subject guide · Learning that transfers into life · How X Works hub

Choose your reading route

Full chapter index · Independent worked case · Questions and practical checks

CHAPTER 2 OF 9 · Choose the protection boundary

2. Understand why the file is encrypted in modules

Back to contents

Parquet is organised into separately serialised components, including pages, headers, indexes and a footer. Modular encryption protects components separately so that a reader can locate and access selected columns without treating the file as one indivisible encrypted blob. Compression and encoding still have their own roles.

For protected columns, the design covers more than the main value bytes. Their pages, page headers, column indexes, offset indexes and Bloom-filter structures are protected under the applicable column key. Otherwise a supposedly hidden column could leak through an unprotected optimisation structure. Check actual writer support rather than assuming every implementation emits every optional feature.

Data pages are compressed before encryption. A reader must reverse the appropriate operations through the supported library. Do not attempt to modify compressed bytes or copy encrypted modules into another file as if they were independent harmless chunks. Their identity and authentication context are part of the format.

In the study example, score may use a key distinct from contact_name. The footer can have another key. A reader may need footer access to discover the structures required for a selective read. Possessing a column key is therefore not always enough to complete a read through a particular footer mode.

This organisation is a mechanism, not an access policy by itself. It provides places to enforce separation. Your key service, reader options, deployment credentials and output handling determine who can actually use those places. Test the complete read route under restricted credentials, not just the file produced by an all-powerful writer account.

Contents · Previous chapter · Next chapter

CHAPTER 4 OF 9 · Verify the evidence

4. Bind authentication to the expected file

Back to contents

Authenticated encryption checks that protected data and its authenticated context have not been altered in the ways covered by the scheme. It does not mean the plaintext is factually true, nor that the file is the latest authorised dataset. A correctly encrypted old snapshot can still be the wrong input for today’s report.

Parquet builds module authentication context using internal file and module identity. An optional application-supplied additional-authenticated-data prefix can bind further context, such as dataset, version and partition identity. Full-file replacement protection depends on how that optional mechanism is used and how the reader checks the expected identity.

Suppose the report expects study-R/version-8/part-2. A file claiming version 7 may be internally authentic and fully decryptable. The application must compare it with an independently expected version-8 identity, not accept the file’s own label as the answer. An attacker need not forge a tag to present a genuine but stale file.

Likewise, checking each file independently does not establish that every required partition arrived. Retain an authorised expected manifest or other trusted dataset boundary. Three valid partitions out of four can produce a plausible incomplete total. Authentication and completeness are separate checks.

Do not put secret participant names into an authentication prefix merely to make it distinctive. A prefix can be visible depending on the chosen storage mode and options. Use an appropriate non-sensitive identity convention and verify its disclosure properties. Context should distinguish the intended object without becoming another unreviewed leak.

Contents · Previous chapter · Next chapter

CHAPTER 6 OF 9 · Verify the evidence

6. Follow keys and plaintext beyond the file

Back to contents

Key metadata helps a reader locate or recover a key through the configured system. It is not automatically the plaintext secret key itself. Different deployments can use wrapped data keys, external key material or caller-managed arrangements. Record the chosen scheme without exposing credentials in logs or examples.

For the study, imagine separate data-key paths for score and contact_name, protected through the organisation’s key service. The analyst’s service identity can obtain score access and the required footer capability but not the contact key. Test that identity directly. Running the test with an administrator’s credentials proves another capability entirely.

Revoking future key retrieval does not erase plaintext already read, screenshots, exported CSV files or cached key material. State the actual boundary. A revocation can block later retrieval through a controlled route while leaving previously authorised derivatives governed by separate retention and access rules.

A derived aggregate may still disclose sensitive information when groups are very small or queries can be combined. Column encryption does not evaluate statistical disclosure risk. Decide whether the proposed output is authorised in its own right. ‘No names in the output’ is not a complete privacy argument.

Keep operational logs useful but restrained. Record key identifiers, policy decisions and safe correlation IDs where allowed; do not dump raw keys, access tokens or private payloads. A troubleshooting routine that copies everything into an unsecured log can undo the protection the file format was chosen to provide.

Contents · Previous chapter · Next chapter

CHAPTER 8 OF 9 · Operate the handover

8. Rehearse rotation, recovery and failure honestly

Back to contents

Distinguish changing a wrapping key from changing a data-encryption key. Rewrapping may preserve the encrypted data while updating how its key is protected. Re-encrypting changes the protected data representation. The surrounding key-management arrangement determines what must be updated and where key material resides.

Do not delete old key paths immediately after a successful new write. Inventory historical files, backups and external key-material references. Rehearse recovery of representative old and new files under the intended policy. A current file opening correctly does not prove that an archive can still be recovered.

For a test, retain two synthetic file generations and an explicitly versioned key-material arrangement. Demonstrate the planned permitted recovery before and after rotation. Then test a deliberately denied old route if retirement is intended. Keep the rollback period and authority clear; this is not permission to weaken actual access controls.

Key loss can make encrypted data unrecoverable. Conversely, keeping every historical key indefinitely may conflict with the intended retirement boundary. These are governance decisions that require a documented balance, not a single technical trick. Use the existing envelope-encryption and secure-deletion guides for the wider lifecycle.

When the key service times out, reconcile the read’s state rather than declaring data missing. Retry only under the service’s normal policy and without bypassing authentication. Report an unavailable or unresolved result when necessary. A report that quietly proceeds with a subset can be more damaging than an honest temporary failure.

Contents · Previous chapter · Next chapter

Independent worked case: an authentic but incomplete study export

A fictional release contains four expected partitions, each with ten rows. Every partition is encrypted using an approved all-GCM configuration. The analyst is allowed site and score access but not contact names. The expected manifest identifies release 8 and partitions 0 through 3. These numbers are teaching inputs, not production measurements.

The delivery includes three correct release-8 partitions and one genuine release-7 partition renamed to resemble the fourth. All four decrypt successfully with the available key paths. A test that checks only decryption and a total of forty rows passes. It has not established that the input is the expected release.

The independent identity check rejects the release-7 partition against the expected release-8 identity. Merely reading the file’s own version label and accepting whatever it says would not solve the problem. The recipient needs a trusted expectation from the release process, and the application must actually enforce it.

Now remove the stale partition rather than replacing it. The remaining thirty rows may all be authentic and correctly decoded. The report is still incomplete for the forty-row release. A required-partition check detects a different failure from the version check. Preserve both outcomes in the receipt.

Next, ask the analyst to calculate a mean only over rows satisfying a protected contact-category predicate. The visible output would contain no names, but evaluating the predicate crosses a protected boundary. The query should use an explicitly authorised route or fail honestly; selecting fewer output columns does not eliminate that access requirement.

Finally, revoke the analyst’s future key-service access after an authorised export has been downloaded. New controlled reads should follow the revised policy, but the already exported plaintext does not disappear. The handover therefore needs derivative handling and retention rules as well as key access. The fixture demonstrates four independent boundaries: authenticity, expected identity, completeness and post-read custody.

A safe projection test uses deliberately distinctive synthetic values. Give each fake participant a contact label that cannot be mistaken for a real person and scores with independently calculated totals. Verify that the analyst’s permitted output matches those totals while protected contact reads fail as designed. Do not search production logs for actual private names to prove that encryption worked. The test must not create the disclosure it is trying to prevent, and successful denial should be recorded without copying sensitive request details into a broad-access dashboard.

Test a protected filter with an unprotected output. Imagine site is readable but eligibility_flag is protected. SELECT site WHERE eligibility_flag = true still needs an authorised way to evaluate the flag. If the system substitutes an unknown value, it may exclude every row and return an empty but apparently successful result. The expected outcome should therefore cover predicate access, not just output-column access. Query planning, remote filtering and key retrieval must preserve the same policy boundary throughout execution.

A full-validation receipt must also identify what was not opened. Reading only the footer can establish some metadata properties without checking every page. Reading a single projection can authenticate the modules involved in that read while leaving other modules unexamined. Name the inspected scope instead of writing ‘file verified’ after any successful operation. When a release requires a complete validation pass, schedule and authorise that operation explicitly, and retain a distinct completion status for missing or unreadable parts.

Operational availability can affect the interpretation of a security test. A denied request and a key-service outage may both prevent a read, but only the first directly demonstrates a policy denial under the tested identity. If the service is unavailable, repeat the test through its normal recovery process before claiming that access separation passed. Avoid weakening policy to obtain a green result. The honest outcome is unresolved until the dependency recovers; preserve the test identity and expected capability so the retry measures the same boundary.

Before final handover, ask the custodian to explain how a future authorised reader will obtain the correct key material for an old synthetic file, how the reader recognises its intended version, and how a missing partition is reported. Ask the analyst what happens to an already downloaded derivative when access is revoked. If these answers depend on an undocumented personal workaround, the handover is unfinished. The goal is not merely a file that opens today, but a supported, bounded access route whose privacy and recovery limits remain understandable after the original writer leaves.

Keep the synthetic fixture and the expected access matrix available to the receiving team. They can rerun the same bounded checks after a reader upgrade without needing real sensitive records or administrator credentials.