On-premises LLMs for Enterprise: Hardware, TCO, and Strategy

TL;DR: In 2026, break-even sits around $3,900/month of API spend for a single-GPU deployment and scales roughly linearly with cluster size — an 8-GPU production cluster needs closer to $46,000/month to justify itself. Street prices below are as of 29 August 2026. Recalculate the single-GPU row before a CFO signs it: an RTX PRO 6000 Blackwell 96 GB now lists near $14,000, so the $3,900 line only holds if CapEx still sits at the old $7,000 assumption.

Most teams reach this point the same way. A pilot works, the monthly API bill starts to hurt, and someone asks whether buying GPUs would be cheaper.

The answer is usually no. Buying hardware looks like a procurement decision. It behaves like a hiring decision. In the 8× H200 model below, engineering salaries account for roughly 65% of annual TCO, while GPU hardware is the smaller line item. A cluster without a platform team is an idle asset, and a platform team serving a single application almost never pencils out.

This guide covers what three years actually cost.

The Full-Stack TCO Framework

Formula: TCO = (Hardware CapEx / 3 yrs) + (Power/Cooling/Rack) + (OpEx: MLOps/Eng Salaries) + (Lifecycle/Maintenance).

Example: A three-year model for an 8× H200 cluster with $320,000 in CapEx, plus 30% for power/cooling, and 1.5 FTE puts annual TCO at about $552,000.

Key TCO Assumptions

Cost Driver	Assumption Value
CapEx (8× H200 Cluster)	Eight H200s at ~$30,000 is about $240,000 of silicon.
Colocation & Power	Varies by density (~$2,500/mo for air-cooled to ~$6,000/mo for DLC)
Engineering/MLOps Support	~$240,000/year per FTE (1.5 FTE ≈ $360,000/year)
Depreciation Period	3 years

Use the Enterprise TCO Calculator to estimate costs for your stack and find your break-even point.

We use these assumptions to calculate the three deployment scenarios and break-even points below.

Choosing an Accelerator: VRAM and Bandwidth

Deployment selection is governed by these hardware strategies:

• NVIDIA: The established standard and easiest production path, offering seamless software integration and predictable performance.
• AMD: An increasingly popular option that offers more VRAM for the money, though its ROCm software stack requires more hands-on management.
• CPUs and lower-end GPUs: viable for smaller models and lighter workloads.

Engineering teams must calculate hardware requirements based on the rule of thumb: 2 GB VRAM per 1B parameters for FP16 precision; Q8 quantization halves this (1 GB); Q4 reduces this to 0.5 GB per 1B parameters.

Here is the breakdown of what companies are using today.

Feature	NVIDIA H200 SXM	NVIDIA B200 SXM	NVIDIA RTX PRO 6000	AMD MI300X OAM	Intel Gaudi 3 OAM
VRAM Capacity	141 GB HBM3e	192 GB HBM3e	96 GB GDDR7	192 GB HBM3	128 GB HBM2e
Memory Bandwidth	4.8 TB/s	8.0 TB/s	1.6 TB/s	5.3 TB/s	3.7 TB/s
Concurrency Math	Example: 70B model at Q4 ≈ 35 GB weights. A single H200 (141 GB) leaves ≈100 GB for KV cache, enough for ≈35 streams at 8k context. Sustained throughput (~1,000 tokens/s per GPU) is the tighter constraint, capping practical concurrency at ≈20 streams @ 50 tokens/s.
Estimated TDP	700 W	1000 W+	600 W max	750 W	900 W
Estimated Unit Price (USD)	$30,000	$60,000	$14,000	$15,000	$15,000

Street pricing, 29 August 2026. H200 NVL median about $30,300 (reseller prints $27,760–$32,000). RTX PRO 6000 Blackwell, 96 GB, median is about $14,000 (Nvidia store lists $16,000). MI300X and Gaudi 3 still swing 20–30% by volume.

The remaining reasons to pick PRO 6000 are a single-socket workstation rack and a lighter colo bill.. At $14,000 versus $30,000, the per-GB gap narrows. The remaining reasons to pick PRO 6000 are a single-socket workstation rack and a lighter colo bill. H200 still wins when you need HBM bandwidth and NVLink for a 70B-class KV cache.

Picking the GPU only gets you halfway there. VRAM is where the math starts to bite.. Quantization and memory management determine whether the economics hold up — or fall apart

Quantization and Memory Math

The most common mistake is simple. The model needs more VRAM than its weights alone suggest. Selecting the most powerful accelerators in 2026 is no longer enough — you need to manage memory efficiently.

Total VRAM = model weights + KV-cache + temporary activations + framework overhead (usually +30–50%)

Skip this math and you’re flying blind.

Quantization has become mandatory. FP8 was widely used because it reduced memory usage by roughly half with little loss in quality. With Blackwell, the market shifted rapidly toward 4-bit formats, especially NVFP4. 4-bit quantization can significantly reduce memory requirements, but more aggressive quantization can reduce quality in reasoning-heavy workloads.

NVFP4 can make inference far more efficient. Four-bit quantization can reduce the number of GPUs required, but teams should first validate quality on their own reasoning and retrieval workloads.

Teams also tend to underestimate the KV cache.

KV Cache (bytes) ≈ 2 × n_layers × n_kv_heads × d_head × sequence_length × precision_bytes × batch_size.

Use n_kv_heads, not n_heads. Grouped-query attention models like Llama-70B have 8 KV heads against 64 attention heads, an 8× difference in cache size.

The KV cache creates a separate memory challenge. Modern serving frameworks and model architectures can significantly reduce KV-cache overhead, but teams still need to validate the results under realistic traffic and context lengths.

A model can fit in VRAM and still hit a wall somewhere else.

Before you sign the PO confirm that the data center can support it.

Check Your Rack Before You Commit

Most midsize companies need to know whether their current infrastructure can support an eight-GPU server.

Rather than analyzing global energy metrics, enterprises should focus on local infrastructure constraints that directly affect AI implementation:

• Available power (kW) per rack;
• Liquid cooling infrastructure;
• Timelines for capacity expansion (including UPS and generators).

Run this check before buying anything. A cheaper GPU can open a much bigger facilities bill.

Cooling Technology	Maximum Rack Density	Typical PUE (Power Usage Effectiveness)	Scope of Application
Traditional Air Cooling	Up to 20 kW/rack	1.3–1.6	Legacy systems, previous-generation servers
Rear Door Heat Exchangers (RDHx)	20–40 kW/rack	1.2–1.4	Medium-density GPU clusters
Direct Liquid Cooling (DLC)	40–150 kW/rack	1.03–1.1	Blackwell, ultra-high compute density

Full immersion cooling is for experimental campuses. Most medium-sized companies simply do not need it. If the cooling and power can handle the peak, you can safely plan for years ahead. your scaling plan exists only on paper

Ready to assess if your infrastructure is ready for enterprise LLMs? Contact our team for a hardware and readiness assessment.

Power and cooling capacity set the physical ceiling. The next layer is financial: once the hardware can actually run, the decision turns on whether owning it is cheaper than continuing to pay for cloud inference.

On-Prem Only Pays Off at High Utilization

The key distinction is the workload pattern.

Training is usually bursty. Inference runs continuously. That difference matters because continuous utilization is what can make owned hardware cheaper than the cloud.

On-prem only pencils out when utilization stays high.

While the break-even against API spend sits around $46k/month, achieving full CapEx payback within the first year requires higher sustained demand, closer to $65k/month (approximately 40% above the break-even point). Owned hardware earns its keep only when you can keep it busy.. The total cost of ownership (TCO) breaks down as follows:

Deployment scale	Configuration	Annual TCO	Break-even vs API
Single GPU	1× RTX PRO 6000 (96 GB) at ~$14,000 street, in-house rack, 0.17 FTE (~7 hrs/week)	≈ $47,000	≈ $3,900/mo
Small node	4× RTX PRO 6000, air-cooled colo (~$2,500/mo), 0.5 FTE	≈ $165,000	≈ $13,750/mo
Production cluster	8× H200 SXM, DLC colo, 1.5 FTE	≈ $552,000	≈ $46,000/mo

Assumptions: 3-year depreciation; single node, no redundancy or failover capacity; open-weight model, no per-token licensing. Note: Production clusters typically require N+1 redundancy, which nearly doubles CapEx.

Power tariffs, rack costs, and salary bands vary widely; the FTE line can swing by 2–3× across the US, EU, and Eastern Europe.

Sanity Check on Token Cost

Use this table as a gut check on token economics.. Assumptions: 70B model at Q4, vLLM, high-batch serving, ~1,000 sustained output tokens/s per GPU (~8,000 tokens/s aggregate).

Scenario	Output tokens/yr	Effective cost
Cluster at 100% utilization	252 B	$2.19 / M tokens
Cluster at 70% utilization	177 B	$3.12 / M tokens
Cluster at 40% utilization	101 B	$5.47 / M tokens
At 70%, platform team shared across 5+ workloads (0.3 FTE allocated)	177 B	$1.49 / M tokens
Comparator: hosted open-weight 70B API	–	$0.60–0.90 / M tokens

As of 29 August 2026, that band still fits a hosted Llama-class 70B (the Together list on Llama 3.3 70B sits near $0.88–$ 1.04/M). Cheaper open-weight APIs now sit under that: DeepSeek V4 Flash prints about $0.14 / $0.28. An owned cluster at $ 2.19/M, even at full batch, still loses to a well-batched open-weight host. Owned silicon wins against frontier APIs at $3–$ 30/M output, or when tokens cannot leave the building. While an owned cluster is generally more expensive at the unit level ($2.19/M tokens vs $0.60–0.90/M for hosted alternatives), it becomes competitive in three scenarios: when the platform team is amortized across 5+ workloads, when comparing against frontier models ($3–15/M), or when price is not the primary driver.

For most mid-market teams, the sweet spot is hybrid. Maintain a stable baseline on owned on-premises hardware, while utilizing the cloud for demand spikes.

Hardware is only one line on the bill. The software layer that serves the model has its own cost structure and long-term trade-offs, especially when choosing between a proprietary stack and an open-source alternative.

vLLM vs NVIDIA NIM

The inference stack determines whether that hardware earns its keep.

NVIDIA NIM provides rapid turnkey deployment in containers and works well on Hopper/Blackwell. However, the license costs about $4,500 per GPU per year and ties you strictly to NVIDIA, limiting flexibility in hardware platform selection. However, for the H100 and H200 NVL models, the license is included in the package, so the argument that NIM is more expensive does not apply to this hardware. That is the NVIDIA AI Enterprise list price: $4,500 per GPU per year on a one-year subscription (NVIDIA licensing guide, 2026). The entitlement ships in the box with H100, H200 NVL, and A800 PCIe. An 8× H200 NVL node therefore adds $0 of NIM tax; an 8× RTX PRO 6000 node adds $36,000/year if you want the supported NIM path.

vLLM is an open framework with no licensing fees. Its real advantage is that it keeps your hardware options open. Although vLLM requires a more experienced MLOps team for fine-tuning, it keeps you from painting yourself into one vendor’s corner

Your choice of infrastructure determines how much flexibility you retain. An open stack lets you switch hardware without being locked into a single vendor.

Cost and stack are not the only constraints. For enterprises in regulated industries, the question of where data is processed comes before the question of what it costs.

Data Sovereignty and Compliance

Enterprises in regulated industries cannot afford compromises here.. ITAR, HIPAA, and CMMC still decide where tokens may live. CMMC Phase 1 self-assessment against NIST SP 800-171 remains in force. On 13 July 2026, the Department of War paused Phase 2’s mandatory third-party certification pending a Reform Task Force review due mid-September 2026. DFARS 252.204-7012 is unchanged. An air-gap or a GovCloud region is still the clean evidence path for CUI and ITAR technical data.

Enterprises now understand the difference between simple network isolation and a true air-gapped deployment. The right choice depends on the compliance requirements and threat model. A true air-gapped system runs on local infrastructure that is physically disconnected from the internet.

Air-gapped systems need an operating model designed for offline maintenance.

For clients and partners, data security is becoming a key factor in trust. Companies demonstrating full control over information and compliance with international standards gain an advantage in regulated markets. Strong data protection can become a competitive advantage and open new markets. Companies that implement clear risk and security management processes during development gain access to regulated markets as reliable partners.

Why Most Pilots Fail in Production

Plenty of pilots look great in a sandbox and fall apart in production.. The model usually isn’t what kills them. The surrounding stack is.. To avoid this disconnect, we use the Context-Grounded Discovery approach. We start coding only after we understand the system well enough to control the risk. This approach provides the team with enough context to transition from a pilot to a repeatable production process.

The pilot often performs much worse once it is connected to fragmented enterprise data and legacy systems. Building agents on fragmented data and legacy integrations increases the risk of hallucinations. Security requirements added late can also force expensive changes to the architecture and access model.

Before approving the investment, the company must demonstrate that the workload justifies it.

Hardware alone does not make the system production-ready. The team also needs to integrate the model into real business workflows. Engineers must configure a management layer (the harness) that provides context memory, permission control, and a full audit trail. Without that control layer and a team that can operate it, even powerful hardware remains an expensive demo.

The decisive layer is the control system that makes the model safe to operate. We call this layer the harness.

The Harness Layer

What happens without the harness?

Without a harness, the model lacks the controls needed to protect data, limit actions, and support reliable operations.

Four components of the harness:

• Memory and context: Models do not remember past actions on their own. The harness maintains a persistent state that includes decision history, code context, and business process context. This layer allows the agent to “remember” what was done earlier, why a certain decision was made, and what constraints apply in the system. Such memory ensures consistency across long projects where one task affects another weeks later.
• Boundaries and permissions: Agents need guardrails with teeth. The harness allows only actions permitted by business policy and security rules. This prevents unauthorized code execution.
• Tool integration: The agent interacts with external systems through controlled tools. The harness provides access to external systems.
• Observability and audit: Regulated environments need a complete audit trail. Teams should be able to reconstruct what happened. That record supports audits under the EU AI Act, CMMC, and other frameworks.

Together, these controls let agents handle routine work while people retain final authority.

The team matters as much as the technology. Experienced operators turn the stack into a system the business can actually use.

Conclusions

Add a fourth gate before CapEx: today’s rental print. If a neocloud H200 at $2.60–$4.50/hour already clears the SLO, own silicon waits until utilization, the platform team, and the data-residency rule all sit on the same workload. On-premises LLMs succeed only when three conditions hold together: the workload is steady enough to sustain high utilization, the data cannot leave your control for compliance or security reasons, and you have — or can hire — a dedicated platform team. If any one is missing, reserved cloud capacity is the better option, and unlike a hardware purchase, that decision is reversible.