A private document never appears in the results.
Good.
But autocomplete reveals its project name.
An aggregate count reveals that one secret document contains a rare term.
A vector search says the user’s query is extremely similar to something they cannot open.
Search has leaked without returning the document.
Permission-aware indexing means access policy travels with every searchable representation strongly enough that users cannot recover protected information through results, snippets, suggestions, statistics, vectors or stale permission state.
This is the fourth pillar beneath How Indexing Works. The master already states that visibility is not cosmetic. This article owns the security consequences: how authorisation intersects with lexical postings, fields, embeddings, caches, autocomplete, aggregations, permission changes and deletion.
Quick Read
A search system can enforce access at several layers: exclude restricted resources from an index entirely, partition them into separate indexes, store permission metadata and filter candidates at query time, or use engine features such as document- and field-level security. Each strategy has trade-offs for scale, freshness and leakage. The important rule is that authorisation must cover every derived output, not merely the final document fetch. Elasticsearch’s current security documentation supports document- and field-level restrictions, while also documenting important limitations: some global index statistics and aggregate information can reveal facts about inaccessible documents. Permission-aware architecture therefore combines search-engine controls with index design, role/tenant separation, safe snippets/suggestions, ACL freshness, auditing and canonical revalidation before consequential access.
canonical resource + ACL/visibility → indexing eligibility/partition → searchable features → authenticated query → permission-aware candidate generation → safe ranking/snippet/aggregation → authorised source fetch → audit → permission-change propagation
Search Has More Outputs Than Results
Documents are only one output.
- result titles;
- snippets;
- highlighted phrases;
- autocomplete suggestions;
- spelling suggestions;
- facets;
- term counts;
- aggregations;
- “related document” recommendations;
- vector similarity scores;
- ranking behaviour;
- timing and existence signals.
A security review that checks only “can the user click the private PDF?” is incomplete.
Permission Is Part of Index Eligibility
Some content should never enter a particular search index.
Public web index.
Internal staff index.
Research-confidential index.
Student-access index.
Physical separation can simplify authorisation because the wrong query path has no searchable representation of the protected object.
But One Index Can Serve Many Permission States
Large enterprise systems often need one logical collection containing documents visible to different users and groups.
Then each indexed document can carry permission metadata such as:
- tenant ID;
- owner ID;
- group IDs;
- role requirements;
- classification;
- allow/deny principals;
- visibility state.
The query path must enforce those conditions before unauthorised content becomes observable.
Document-Level Security Restricts Which Documents a User Can Read
Elastic’s current document- and field-level security documentation describes document-level security as restricting which documents can be accessed, commonly through a role-associated query.
Conceptually:
user attributes + role policy + document ACL → authorised document set
Search should rank and return within the authorised set rather than retrieve globally and hide forbidden documents afterward.
Post-Filtering Is Often Too Late
Search all documents.
Take top 10.
Remove private ones.
The user receives only three results.
Problems:
- relevance is distorted because private docs consumed top slots;
- result-count patterns can leak hidden matches;
- snippets/highlights may already have been generated;
- downstream rerankers may have seen restricted content;
- latency behaviour can reveal hidden work.
Permission filtering should happen as early as the search architecture can enforce it safely.
Field-Level Security Protects Sensitive Parts of Otherwise Visible Documents
A document may be visible while some fields are not.
Employee directory:
- name: public internally;
- department: public internally;
- salary: restricted;
- medical note: highly restricted.
Field-level controls need to cover:
- stored field retrieval;
- highlighting;
- sorting;
- aggregations;
- script access;
- embedding generation if restricted fields were used.
A hidden field can still leak if it influences a derived representation that remains visible.
Embedding a Secret Field Can Leak the Secret’s Influence
Document body is public.
Confidential notes are private.
One embedding is generated from both.
Even if the notes are never displayed, vector similarity can be influenced by them.
A user may retrieve or infer relationships that exist only because of hidden content.
field-level privacy must propagate into every derived feature built from that field.
Separate Embeddings by Visibility Boundary
Possible pattern:
- public embedding from public fields;
- internal embedding from internal fields;
- restricted embedding in a restricted index.
Then a public query cannot be influenced by a representation trained on protected text it should not know exists.
Snippets Are Content Disclosure
A snippet contains text.
Therefore its security level cannot be weaker than the content used to generate it.
Generate snippets only after permission filtering and from fields authorised for the current user.
Cached snippets must include the permission context or be generated from universally visible content.
Highlighting Has the Same Rule
“We hid the field but highlighted its matching text” is still disclosure.
Highlight generation must use only authorised source fields and authorised documents.
Autocomplete Can Leak Vocabulary
User types the first letters of a confidential project.
Autocomplete completes the project name because that term exists only in private documents.
No private document was returned.
Confidential vocabulary was still disclosed.
Suggestions should be built from permission-appropriate corpora or generated under the same access filtering as search.
Spell Correction Can Leak Too
A spelling model trained on restricted names may suggest those names to users without access.
Dictionary/model training data is part of the security boundary when outputs expose learned tokens or associations.
Aggregations Can Reveal Hidden Population Structure
User cannot see restricted documents.
But an aggregation says:
classified_project = 1 document
The existence of one hidden document is now visible.
Elastic explicitly documents a related limitation: document-level security can prevent restricted documents from being returned while some global index statistics or aggregate information may still reveal facts about inaccessible documents.
This is why permission-aware indexing is larger than enabling one DLS flag.
Global Term Statistics Can Cross the Security Boundary
A ranking model may use document frequency computed across the entire index.
If inaccessible documents influence that statistic, a user might not see the documents but can potentially observe scoring effects or statistical APIs that reveal hidden corpus structure.
Whether this matters depends on threat model and consequence.
High-secrecy tenants often justify stronger index partitioning rather than sharing one statistical universe.
Tenant Separation Can Be the Cleanest Boundary
Organisation A and Organisation B must never infer each other’s content.
Separate indexes, shards or clusters can reduce cross-tenant statistical and operational leakage.
This costs more infrastructure.
Security architecture is allowed to spend resources to simplify the trust boundary.
Filtered Aliases Are Not Automatically Security Controls
Convenience filters can narrow a search view.
But Elastic’s current security limitations explicitly warn that filtered aliases are not a secure substitute for document-level security.
The principle generalises:
query convenience ≠ authorisation enforcement.
Roles Can Combine in Surprising Ways
User has Role A with restricted documents.
User also has Role B that grants broad access to the same index.
The combined effective access may be broader than an engineer expects.
Elastic’s documentation warns that document/field security across multiple roles can combine in permissive ways.
Test effective permissions, not individual role definitions in isolation.
ACL Freshness Is a Security SLO
At 10:00 a document becomes private.
The index still treats it as public until 10:05.
Those five minutes are not ordinary relevance staleness.
They are an exposure window.
The third pillar, Incremental Indexing & Freshness, owns the delivery machinery. Permission-Aware Indexing sets a stricter requirement for access changes.
Revocation Often Needs Faster Propagation Than Content Edits
Typo correction can wait a minute.
Permission revocation may need immediate enough enforcement to meet security policy.
Use separate priority channels, direct policy lookup, short ACL cache lifetimes or query-time authoritative checks where consequence justifies them.
Query-Time ACL Filtering Stays Current More Easily
If every search consults current ACL attributes, permission changes can affect retrieval without reindexing all content features.
Trade-offs:
- query complexity;
- filter cost;
- permission-store availability;
- cache correctness;
- supported query features.
There is no universal best boundary.
Index-Time Permission Materialisation Can Be Fast
Store allowed group IDs on each document.
Query filters by the user’s groups.
This can be efficient for stable group-based access.
But if group memberships or ACLs change frequently, the derived permission state can become stale.
The more permission truth is materialised into the index, the more seriously permission-index freshness must be monitored.
Precomputed “Can See” Lists Can Explode
Document visible to 50,000 users.
Storing every user ID per document may be huge.
Roles/groups/attributes can compress permission representation.
But group nesting and policy logic create their own correctness burden.
ABAC Can Express Richer Policy
Attribute-based access might depend on:
- department;
- region;
- clearance;
- project membership;
- employment status;
- resource classification.
This is more expressive than static role names.
It also means search authorisation depends on current identity attributes and policy evaluation, not only document metadata.
The Search Cache Must Include Security Context
User A searches “Project Falcon.”
Result cache stores the authorised response.
User B asks the same query.
If the cache key ignores identity/permission context, User B can receive User A’s privileged result set.
Caches must be permission-aware or restricted to public/universal responses.
Rerankers and LLMs Are Downstream Search Consumers
Search retrieves 50 candidates.
An LLM reranks them.
If permission filtering happens only after reranking, the model has already received restricted text.
Apply authorisation before candidates cross into external models or lower-trust processing boundaries.
Tool Logs Can Leak Restricted Content Too
Search service correctly filters final results.
Debug logs record the top 100 raw candidates before filtering.
Now operators or downstream observability tools see protected content.
Security boundary must include telemetry, traces and evaluation datasets.
Indexing Pipelines Need Least Privilege Too
The indexing worker may require permission to read protected source content so it can build a restricted index.
That does not mean every search service, analytics worker or developer needs the same access.
Separate service identities and narrowly scoped credentials reduce the blast radius of an indexing compromise.
Encryption Protects Storage, Not Query Authorisation
Encrypted disk.
Encrypted network.
Important.
But once the search engine is authorised to decrypt and index the content, application-level authorisation still decides which user may retrieve which derived information.
Encryption and permission-aware retrieval solve different layers.
Deletion Has Two Security Clocks
Clock 1: retrieval revocation. When does the user stop being able to search/see the document?
Clock 2: physical reclamation. When are obsolete segment bytes actually removed?
The second pillar, Index Segments & Merges, owns physical cleanup.
Security policy usually cares first that retrieval stops immediately enough; regulatory retention/deletion rules may also care about physical lifecycle.
Snapshots and Backups Extend the Deletion Boundary
Document removed from live index.
Old snapshots still contain segment files.
A complete data-lifecycle policy must define backup retention and restoration behaviour.
Otherwise “deleted from search” can be mistaken for “eliminated from every copy.”
Reindexing Is a Permission Migration Too
Build Green index from Blue.
If the reindex source does not preserve current ACLs or if the migration accidentally defaults missing permissions to public, the new generation can be semantically correct and catastrophically insecure.
Blue/green validation must compare permission state as well as document counts and relevance.
Default-Deny Is Safer for Missing Permission Metadata
Document has no ACL field because ingestion failed halfway.
Should that mean public?
For sensitive systems, safer default is usually:
permission unknown → not searchable until resolved.
Missing security metadata is not evidence of universal access.
Owner Resolution Is the Final Authority Check
The index says User A can see document X.
Before a high-consequence download, the application can revalidate against the canonical permission owner if freshness or policy requires it.
This follows the site’s broader rule:
index accelerates discovery; canonical owner settles authority.
Permission Checks Need Auditable Receipts
For a sensitive retrieval, a receipt can record:
- user/service identity;
- effective roles/groups/attributes;
- policy version;
- document ACL/version;
- index generation;
- permission-filter decision;
- canonical recheck if performed;
- timestamp;
- result/action boundary.
This makes “the system checked permissions” testable after an incident.
Security Evaluation Needs Adversarial Search Cases
Do not test only:
Can User B open User A’s document?
Also test:
- autocomplete with secret prefixes;
- facets containing secret-only values;
- count queries;
- rare-term score shifts;
- semantic nearest neighbours;
- snippets;
- empty-result patterns;
- permission changes during active sessions;
- cached results after revocation;
- reindexed documents with missing ACLs.
A Better Permission-Aware Indexing Model
canonical content + canonical permission state → eligibility/partition → permission-tagged searchable representations → authenticated receiver context → pre-retrieval authorisation → safe lexical/vector candidate set → permission-safe snippets/aggregations/reranking → authorised source action → audit + rapid ACL revocation propagation
A 30-Lens Permission-Aware Indexing Audit
- Content owner: who owns canonical resource state?
- Permission owner: who owns authoritative access state?
- Tenant: should data be physically separated?
- Eligibility: should this resource enter this index at all?
- Document ACL: which principals/groups may access it?
- Field ACL: which fields are restricted?
- Derived features: which embeddings/statistics depend on hidden fields?
- Authentication: who is the receiver?
- Authorisation: what policy applies now?
- Role composition: can multiple roles widen access unexpectedly?
- Pre-filter: are private candidates removed before ranking?
- Snippet: is generated text permission-safe?
- Highlight: can hidden fields leak?
- Autocomplete: is suggestion corpus permission-aware?
- Spelling: can learned vocabulary reveal restricted terms?
- Facet: can category values reveal hidden docs?
- Aggregate: can counts expose protected existence?
- Statistics: do global term stats cross tenant boundaries?
- Vector: can similarity reveal hidden content influence?
- Cache: does key include permission context?
- Reranker: are restricted candidates sent downstream?
- Logs: do traces capture protected raw candidates?
- Indexer credentials: are they least privilege?
- ACL freshness: how fast do revocations propagate?
- Default: what happens when permission metadata is missing?
- Delete: when does retrieval stop?
- Physical retention: when do segments/snapshots release bytes?
- Reindex: are ACLs verified in new index generation?
- Audit: can a sensitive retrieval be reconstructed?
- World return: does the canonical permission owner agree that this user may access the result now?
Laboratory 1: Result Hidden, Suggestion Leaked
Create one confidential project name that occurs only in private documents. Make the final result filter correct but leave autocomplete global. Show how the user learns the name without opening any document, then redesign the suggestion corpus.
Laboratory 2: ACL Revocation Timeline
At 10:00 revoke access. Track identity cache, permission store, index ACL state, result cache and active session. Identify the longest remaining exposure path and shorten it.
Laboratory 3: Public Body, Private Notes
Design a document with public fields and one restricted note field. Decide which fields feed public lexical indexing, public embedding, internal embedding, snippets and facets.
For Primary Readers
A teacher has one class list for everyone and one private note for teachers only. The search box should not reveal words from the private note just because the student’s name is public.
For Secondary Readers
Explain why filtering a private document out of final results is insufficient if autocomplete, snippets or aggregates were built from that document first.
For Advanced Readers
Model permission-aware indexing as information-flow control over a family of derived search representations. Authorisation must constrain not only document retrieval but every observable function of protected content; security correctness depends on ACL freshness, receiver context, role composition, derivation lineage and the absence of unauthorised side channels through statistics or learned representations.
Common Misconceptions
- “If the private document is not in results, search is secure.” snippets, suggestions, counts and vectors can still leak information.
- “A filtered alias is an access-control system.” convenience filters are not automatically security enforcement.
- “Field-level hiding is enough if the field fed the embedding.” derived representations can preserve hidden influence.
- “Permission changes can use ordinary indexing freshness.” revocation often needs a tighter exposure-window SLO.
- “Encryption solves search permissions.” encryption protects data at rest/in transit; authorised query-time disclosure remains a separate policy problem.
Research Corridor
- Elastic — Document and Field Level Security — current role-based restrictions over documents and fields.
- Elastic — Security Limitations — documented limits including aggregate/global-statistic exposure considerations.
- Elastic — Connector Document-Level Security — ACL-aware ingestion for supported content connectors.
- eduKateSG — How Indexing Works.
Frequently Asked Questions
What is permission-aware indexing?
It is designing searchable representations and query paths so document, field and tenant access rules constrain every user-observable search output, not only final document retrieval.
Should private documents be stored in the same search index as public documents?
Sometimes, with strong document/field-level controls and careful leakage analysis. Higher-isolation threat models may justify separate indexes, tenants or clusters to simplify the security boundary.
Why are permission changes an indexing concern?
Because searchable documents, ACL fields, caches, snippets and derived representations may contain materialised permission assumptions that must change when access is granted or revoked.
Final Thought: Search Security Is About What the User Can Learn
A search system does not protect a secret merely by refusing to hand over the source file.
Permission-aware indexing succeeds when every searchable shadow of protected content obeys the same authority boundary closely enough that the receiver cannot learn what they were never entitled to know.
INDEXING · FOUR PILLAR LEGS
Return to How Indexing Works, or continue through Inverted Index & Postings, Index Segments & Merges and Incremental Indexing & Freshness. Return to the Information & Representation Hub.