A RAG pilot can work perfectly in internal demos and still get stopped during security review. The generated answers may look correct, but security focuses on specific verification: which documents the retriever actually searched, and proof that it stayed strictly inside the caller’s permissions. Those questions center on retrieval, making it the core component when building a robust enterprise RAG system.

This article covers what the retrieval layer must enforce before text reaches the prompt, then prices three deployment classes — managed public APIs, private cloud enclaves, and sovereign infrastructure — while evaluating key risks like data leakage.

Do You Need RAG at All?

RAG earns its complexity when you need to search across enterprise-scale knowledge. For simpler tasks, such as comparing twelve contracts, passing documents directly into a long-context prompt is often more efficient and avoids overkill.

Beyond context size, ownership and risk management play a major role in deciding whether RAG is necessary. Security teams can measure retrieval accuracy, and engineering can test permission enforcement, but the business must ultimately decide whether the use case justifies RAG’s complexity.

Retrieval Can Break Before the Model Ever Sees the Prompt

The model in a RAG system answers only from the chunks the retriever places into context. When a relevant chunk falls outside the top k, the model must proceed without it, resulting in a confident answer that omits existing documents.

On sensitive data, retrieval has two jobs:

1. It has to find the right material.
2. It also has to keep the search inside the caller’s permissions.

RAG blurs structural boundaries: a vector database holds both standard content and behavioral instructions in the same text format. In traditional applications, code and data arrive through separate pathways. In a RAG system, standard text can directly influence model behavior, so mandatory access enforcement is required before retrieved text reaches the prompt.

Three Deployment Classes for Sensitive RAG

Public APIs include providers like OpenAI, Anthropic, and Google. Private cloud enclaves run in a VPC deployment or use services such as Azure OpenAI and AWS Bedrock.

The main difference between these three classes is who owns the infrastructure and operations, which directly impacts compliance frameworks like SOC 2:

• Class 1 (Public APIs): The provider manages the serving layer, infrastructure, and maintenance. Your team focuses entirely on usage.
• Class 2 (Private Cloud Enclaves): You use private network paths such as a VPC or Private Link, making this setup ideal for regulated environments requiring HIPAA compliant RAG. Both parties share responsibility for configuration and compliance.
• Class 3 (Sovereign Infrastructure): Your team assumes full ownership. Staff must monitor, patch, plan capacity, and respond to outages.

At low usage, managed endpoints are usually the cheaper and simpler option. Strict mandates such as GDPR data residency narrow Class 1 to providers with in-region processing and often move the workload to Class 2.

Keep Permissions Intact Through Retrieval

The common failure is that permissions work in the source system but get lost in translation on the way to the index. SharePoint knows who can open the document. The file server knows. The ERP knows. But once those documents are chunked and embedded into a new index, the retriever only knows the permissions copied into the retrieval layer.

Why Pre-Filtering Is Essential for Recall and Security

Enforcing access controls as a post-filter introduces recall and safety risks. In a post-filtering setup, vector search retrieves top candidate chunks based on semantic similarity before applying permission filtering. If unauthorized documents dominate the initial retrieval pool, post-filtering strips them out, leaving too few context chunks for the model and weakening answer quality.

To prevent this degradation, enforce ACL checks as a pre-filter during retrieval, before candidate ranking and generation. By pruning unauthorized chunks early in the pipeline, the retrieval pool contains only authorized context. This approach helps manage context window utilization, supports recall stability, and prevents unauthorized chunks from reaching downstream reranking or generation stages.

Fixing post-filtering by increasing the candidate retrieval count only scales compute costs and latency without eliminating RAG security risks. Pre-filtering ensures access-boundary logic runs at the earliest stage of the RAG pipeline.

ACL-aware retrieval applies this filter inside the search on a shared index. However, on traditional HNSW indexes, highly restrictive pre-filters can fragment the search graph and reduce recall—which is why filter-aware indexes such as ACORN, or partitioning approaches such as HoneyBee, are needed.

This pre-filtering strategy keeps the audit logging process straightforward: every candidate chunk evaluated in the candidate set has already passed explicit authorization bounds before reranking models score it.

The Cost of Role-Based Indexes

One option is to isolate retrieval by role so only authorized documents enter the search.

The design is straightforward: the index includes only the documents that role can access. The benefit is predictable query-time behavior, because query-time filtering disappears. Authorized candidates are already isolated before search.

The cost shows up when the same document belongs to several roles. For the system to work, it must replicate those vectors across multiple indexes, increasing memory usage. Depending on the permission structure, partitioning by role rather than using a single shared index raised storage overhead to between 3.5× and 7× that of a single index. That is where the infrastructure cost starts to follow the permission model instead of the corpus size. Add more roles or legal entities, and the index footprint can grow even if the underlying document set barely changes.

Dynamic Partitioning as a Middle Ground

You can keep permission checks inside retrieval without building a completely separate index for every role. In practice, the system copies enough vectors to give each permission group a useful search space.

You pay for some duplication, while avoiding a full copy of the index every time the business creates another role. The relevant pattern is dynamic partitioning of the vector space, implemented in the HoneyBee framework. For example, most companies have far fewer roles than users. That gives the system a practical basis for routing queries and partitioning the index. Instead of full isolation, the system strategically duplicates vectors across partitions to reduce memory and latency while keeping recall above target.

Compared with fully dedicated indexes for each role, dynamic partitioning implemented in the HoneyBee framework reduces memory overhead by 90.4%. At that point, the trade-off becomes much more practical. You keep much of the predictable behavior of separate indexes without paying for a full copy of the corpus for every role.

Measure Permission-Aware Retrieval

I would want to see four numbers before approving this design:

• Authorized recall: how often the system still finds the right document after enforcing permissions.
• Filter selectivity: the share of the corpus a caller can access, and authorized recall at the lowest selectivity you support.
• Index duplication: how many times the corpus is effectively replicated across roles or partitions.
• Permission-sync lag: how long it takes a source-system role change to affect search results.

A long permission-sync lag may look like a small operational detail, but if permission changes take six hours to reach the index, that creates a six-hour access-control gap. Source systems (such as SharePoint and Entra ID) update well before the RAG index catches up, so revocation is only as fast as the index itself.

Security Has to Start Before Retrieval Reaches the Prompt

For sensitive-data RAG, the main exposure points are ingestion and context retrieval. So ingestion and retrieval both need to enforce that policy and leave enough of an audit trail to reconstruct what happened later.

Decide What Is Allowed Into the Index

Ingestion begins by establishing clear eligibility rules for the index. Some document categories — legal holds, investigation materials, restricted archives — stay out of the corpus entirely.

Sanitization works alongside structure preservation. Tables, captions, and paragraphs require their original context to remain useful for retrieval. Maintaining document structure ensures accurate ranking, as flattening documents into plain text removes essential search cues. High-quality retrieval quality depends on structural integrity (e.g. keeping procurement tables searchable with their column headers), ensuring data like procurement tables remain fully searchable.

A design can pass security review and still give users poor search results on day one if the strategy lacks effective chunking.

From there, apply PII redaction before generating embeddings in the following step and keep each chunk tied to a resolvable source. Eventually, someone will ask where a particular answer came from. Security may require the original file, while Legal needs the specific version active at the time, so the retrieval layer must preserve enough metadata to resolve both.

Enforce Access Before Retrieval

This is where authority can get mixed up if retrieval is not scoped correctly. In plain English, the user may have high privileges, but the instruction that influences the model may come from a document written by someone with much lower privileges. The model cannot safely infer that distinction from prose alone. That makes retrieval scope one of the main containment controls. When ACLs are applied before retrieval, unauthorized documents never enter the candidate set or model context.

For every lookup, the audit log should answer four questions:

• Who asked?
• Which policy was applied?
• Which exact chunks reached the model?
• Can we still resolve those chunks back to the source document later?

SIEM, synchronized clocks, and tamper-evident storage preserve the operational evidence needed to satisfy compliance requirements such as the EU AI Act. Attestation specifically verifies environment trust, while detailed logs record the exact discovery output provided to the model.

What Each Deployment Class Costs

• Class 1 is mostly token usage.
• Class 2 adds private cloud infrastructure and staffing.
• Class 3 adds owned hardware and operations.

What Drives Cost in Each Class

In Class 1, most direct model cost comes from token usage. The provider maintains the inference infrastructure. When you submit simple queries to fast models (Fast Tier), you pay only between $0.05 and $1.50 per million tokens. However, if deep reasoning is required (Power Tier), the price rises to $30 per million input tokens and $180 per million output tokens. Semantic caching can reduce repeated model calls and lower token spend. Still, ultimately your bill will be based on the amount of text that passes through the model across the whole RAG stack.

Class 2 adds private cloud infrastructure and the people required to keep it compliant. You still pay for tokens, but you also pay for isolated cloud infrastructure and compliance. For an enterprise RAG implementation, initial setup, implementation, and compliance deployment costs typically range from $80,000 to $180,000. In contrast, ongoing operational and maintenance expenses for mid-market systems range from $3,200 to $13,000+ per month. And over time, the infrastructure itself may not even be the biggest recurring expense. Staffing usually becomes the larger expense. A dedicated security or compliance role can run $125,000–$200,000 per year and stays necessary long after the environment is provisioned.

Class 3 puts the hardware and day-to-day operations on your team to run a self-hosted LLM, up to a fully air-gapped deployment. While hardware is the easy line item, ownership quickly becomes expensive. A dual-RTX 5090 workstation would have cost roughly $6,900 at 2025 card prices; at 2026 street prices, the entire dual-card workstation build now runs past $13,000. The hardware also changes the power and cooling requirements. An RTX 5090 has a 575W TDP, so the surrounding system needs to be sized for that load. Once you’ve paid for the hardware, electricity can be one of the smaller recurring costs. Assuming continuous 24/7 operation at full TDP (575W per card), you’ll pay about $65 per month for a single card at $0.16/kWh.

When Self-Hosting Breaks Even

The break-even point depends on sustained utilization. For a small team of 10 employees running 40 queries per day (around 400 total queries per day) with an average context of 2,500 input and 500 output tokens per request, monthly API costs on Fast Tier models ($0.50/1M tokens) amount to roughly $18, or ~$216 per year. At this low volume, relying on public APIs is vastly more cost-effective than running dedicated infrastructure.

As sustained usage rises—for instance, to 5,000 queries per day (around 150,000 queries per month)—monthly API costs on Fast Tier models ($0.50/1M tokens) can reach $150 to $300+. In this scenario, a single-GPU build costing ~$2,000 (utilizing a mid-range card with ~250W TDP and moderate power consumption of ~$28/month) achieves hardware payback within 7 to 16 months under high daily throughput. This 7 to 16 month hardware payback period is calculated strictly based on infrastructure and energy expenses, excluding staff costs.

Break-even should include time as an economic variable: a managed service may cost more per month and still offer better economics if it reaches production months earlier.

I would model utilization month by month, especially when evaluating a private LLM infrastructure. One busy week can make owned hardware look much better than it really is.

The same logic extends to data storage. For example, consider an organization storing 50 million 1536-dimensional vectors. Managed SaaS providers (such as Qdrant Cloud or Pinecone) typically charge around $1,800 to $2,700 per month for quantized configurations handling this volume. By contrast, a self-hosted setup (such as Qdrant using scalar/product quantization or disk-backed storage on a dedicated host like a Hetzner AX102 with 128 GB RAM) costs roughly $150 to $300 per month. Storage can also hit its own break-even point separately from inference. Here’s the part teams miss: staff time rarely shows up in the first cost model.

Engineering and security costs form a core part of your ongoing run rate. When you commit to Class 3 hardware, I would budget for dedicated personnel and operational processes from day one, rather than treating ongoing support as an afterthought.

Test Permission Failures Before Cutover

Before launching into production, be sure to validate access control and system behavior against these test scenarios:

• an authorized query that should find the document;
• the same query under a role that lacks access;
• a recently revoked permission;
• an prompt injection attempt or instruction inside an authorized document to test against retrieval poisoning;
• a stale ACL synchronization event.

I prefer to slip the cutover rather than launch with a revoked user who can still find yesterday’s document. Production will eventually test that edge case for you anyway.

Five Questions Before You Choose a Class

Before choosing a deployment class, I would want five answers. You can also evaluate these choices using our AI tool evaluation framework:

• Can contract controls such as ZDR satisfy the data-handling requirement, or does the workload truly require physical isolation?
• Can source permissions be expressed as stable roles, or are they effectively user-specific?
• Is sustained utilization high enough for owned infrastructure to pay for itself within a realistic timeframe?
• Who will operate the serving layer, vector store, ingestion pipeline, and permission sync?
• What evidence does the audit team need for each retrieval operation: standard logs, tamper evidence, or cryptographic attestation?

Where Each Class Breaks Down

Each deployment class has specific architectural limitations and failure modes that can undermine security or performance if unaddressed.

Class 1 Application Limits: Zero Data Retention (ZDR) limits what the provider retains, but the retriever still selects documents on your side. If your retrieval layer lacks proper ACL pre-filtering, ZDR contracts cannot prevent unauthorized context from being retrieved and exposed within the prompt.

Class 2 Application Limits: Class 2 breaks down when granular role hierarchies force extensive index duplication or complex pre-filtering logic, leading to substantial memory overhead and latency penalties on shared vector infrastructure.

Class 3 Application Limits: Class 3 fails under low or highly variable query volumes, where capital expenditure and operational maintenance burdens outweigh the benefits of physical isolation.

Sovereign infrastructure shifts the entire operational burden to the internal team. That team must sustain patching, maintain permission synchronization, recover from system failures, and preserve audit evidence.

Include Exit Costs From the Start

One last thing: price the exit before version one ships.

Keep Embeddings Replaceable

The API bill for embeddings is usually the small part. Rebuilding the index is where migration gets expensive. How painful that rebuild is depends a lot on which deployment class you started with. For Class 1, model changes become managed re-embedding jobs while production stays live. In Class 3, the same GPUs may already be serving production traffic.

Re-Test After Re-Embedding

In practice, I’d expect validation to cost more than the tokens used for re-embedding, so I would make sure that number is in the migration budget. Re-testing is critical because updating embedding models can alter distance metrics and semantic representations across the vector space. Comprehensive quality checks are necessary to ensure that authorized recall remains stable and that existing permission filters continue to perform as expected without exposing unintended documents.

Keep the Vector Store Behind an Adapter

Same story with the vector store. Eighteen months in, managed storage may cross your cost target. The tighter the coupling, the bigger the migration bill, especially when implementing complex search mechanics like hybrid search.

If you start on Class 1, put retrieval behind an adapter from day one. That gives you a clean path to another vector store later. In Class 3, keep the embedding interface replaceable so model changes don’t force you to redesign the rest of the system.

That is also what we look at first in a Devox feasibility pass: whether usage is sustained enough to justify owned infrastructure, and whether the source permissions map cleanly to roles.

What the Architecture Review Should Prove

Before I approve a sensitive-data RAG design, I would want the architecture review to show me:

• one query that succeeds under the right identity;
• the same query failing under the wrong identity;
• the source document and ACL behind every retrieved chunk;
• the permission-sync latency from source system to index;
• the authorized recall at the lowest filter selectivity you support;
• the rollback plan for a new embedding model;
• and the operational owner for each stack component.

If the team can show those seven things, the trade-offs are usually clear enough that choosing the deployment class becomes a much more concrete decision.

What the Choice Actually Rests On

That’s the split: the deployment class decides who carries the operational load. The permission model decides whether the system holds up at all. That means treating retrieval scope as a containment control, keeping permission-sync lag within a defined latency budget, and logging which chunks reached the model under which policy.

Price each class against month-by-month utilization rather than one busy week, add engineering and compliance staffing to the model, and budget re-embedding and vector-store migration before version one ships. If those numbers hold and the ACL tests pass for users whose access has been revoked, the deployment class stops being a debate and becomes a budget line.

Send us your corpus profile and permission model, and we’ll run a Devox feasibility pass that measures authorized recall and permission-sync lag on your own data and tells you which deployment class the numbers support.