On-premises LLMs for Enterprise: Hardware, TCO, and Strategy

TL;DR: In 2026, break-even sits around $3,900/month of API spend for a single-GPU deployment and scales roughly linearly with cluster size — an 8-GPU production cluster needs closer to $46,000/month to justify itself.

Most teams reach this point the same way. A pilot works, the monthly API bill starts to hurt, and someone asks whether buying GPUs would be cheaper.

The answer is usually no. Buying hardware looks like a procurement decision. It behaves like a hiring decision. In the 8× H200 model below, engineering salaries account for roughly 65% of annual TCO, while GPU hardware is the smaller line item. A cluster without a platform team is an idle asset, and a platform team serving a single application almost never pencils out.

This guide covers how to size the hardware, what three years actually cost, where break-even sits against both API and reserved capacity, and what has to exist between the model and your systems before any of it survives production.

The Full-Stack TCO Framework

Formula: TCO = (Hardware CapEx / 3 yrs) + (Power/Cooling/Rack) + (OpEx: MLOps/Eng Salaries) + (Lifecycle/Maintenance).

Example: A three-year model for an 8× H200 cluster with $320,000 in CapEx, plus 30% for power/cooling, and 1.5 FTE puts annual TCO at about $552,000.

Key TCO Assumptions

Cost Driver	Assumption Value
CapEx (8× H200 Cluster)	~$320,000
Colocation & Power	Varies by density (~$2,500/mo for air-cooled to ~$6,000/mo for DLC)
Engineering/MLOps Support	~$240,000/year per FTE (1.5 FTE ≈ $360,000/year)
Depreciation Period	3 years

Use the Enterprise TCO Calculator to estimate costs for your stack and find your break-even point.

We use these assumptions to calculate the three deployment scenarios and break-even points below.

Choosing an Accelerator: VRAM and Bandwidth

Deployment selection is governed by these hardware strategies:

• NVIDIA: The established standard and easiest production path, offering seamless software integration and predictable performance.
• AMD: An increasingly popular option that offers more VRAM for the money, though its ROCm software stack requires more hands-on management.
• CPUs and lower-end GPUs: viable for smaller models and lighter workloads.

VRAM capacity and memory bandwidth are the primary constraints for high-concurrency inference SLOs.

Inference efficiency depends on VRAM capacity and memory bandwidth. Engineering teams must calculate hardware requirements based on the rule of thumb: 2 GB VRAM per 1B parameters for FP16 precision; Q8 quantization halves this (1 GB); Q4 reduces this to 0.5 GB per 1B parameters.

Here is the breakdown companies are using today.

Feature	NVIDIA H200 SXM	NVIDIA B200 SXM	NVIDIA RTX PRO 6000	AMD MI300X OAM	Intel Gaudi 3 OAM
VRAM Capacity	141 GB HBM3e	192 GB HBM3e	96 GB GDDR7	192 GB HBM3	128 GB HBM2e
Memory Bandwidth	4.8 TB/s	8.0 TB/s	1.6 TB/s	5.3 TB/s	3.7 TB/s
Concurrency Math	Example: 70B model at Q4 ≈ 35 GB weights. A single H200 (141 GB) leaves ≈100 GB for KV cache enough for ≈35 streams at 8k context. Sustained throughput (~1,000 tokens/s per GPU) is the tighter constraint, capping practical concurrency at ≈20 streams @ 50 tokens/s.
Estimated TDP	700 W	1000 W+	600 W max	750 W	900 W
Estimated Unit Price (USD)	$30,000	$60,000	$7,000	$15,000	$15,000

Note: SXM = Server Accelerator Module; OAM = Open Accelerator Module.

*Street pricing, mid-2026; varies 20–30% by volume and reseller.

On a dollar-per-GB basis, RTX PRO 6000 is roughly 3× cheaper than H200 — the trade-offs are bandwidth, NVLink, and the ceiling on model size.

Choosing an accelerator solves only part of the problem. Once the hardware is selected, the real constraint is how efficiently the model uses available VRAM. Quantization and memory management determine whether the system remains practical or becomes prohibitively expensive.

Quantization and Memory Math

The most common mistake is simple. The model needs more VRAM than its weights alone suggest. Selecting the most powerful accelerators in 2026 is no longer enough — you need to manage memory efficiently.

Total VRAM = model weights + KV-cache + temporary activations + framework overhead (usually +30–50%)

Without this calculation, the rest of the design is guesswork.

Quantization has become mandatory. FP8 was widely used because it cut memory use roughly in half with little loss in quality. With Blackwell, the market shifted rapidly toward 4-bit formats, especially NVFP4. 4-bit quantization can significantly reduce memory requirements, but more aggressive quantization can reduce quality in reasoning-heavy workloads.

NVFP4 can make inference far more efficient. Four-bit quantization can reduce the number of GPUs required, but teams should first validate quality on their own reasoning and retrieval workloads.

Teams also tend to underestimate the KV cache.

KV Cache (bytes) ≈ 2 × n_layers × n_kv_heads × d_head × sequence_length × precision_bytes × batch_size.

Use n_kv_heads, not n_heads. Grouped-query attention models like Llama-70B have 8 KV heads against 64 attention heads, an 8× difference in cache size.

The KV cache creates a separate memory challenge. Modern serving frameworks and model architectures can significantly reduce KV-cache overhead, but teams still need to validate the results under realistic traffic and context lengths.

The model may fit in memory and still hit another infrastructure limit.

Before committing to the hardware, confirm that the data center can support it.

Check Your Rack Before You Commit

Most midsize companies need to know whether their current infrastructure can support an eight-GPU server.

Rather than analyzing global energy metrics, enterprises should focus on local infrastructure constraints that directly affect AI implementation:

• Available power (kW) per rack;
• Liquid cooling infrastructure;
• Timelines for capacity expansion (including UPS and generators).

Run this check before buying anything. Lower-cost hardware can trigger expensive infrastructure upgrades.

Cooling Technology	Maximum Rack Density	Typical PUE (Power Usage Effectiveness)	Scope of Application
Traditional Air Cooling	Up to 20 kW/rack	1.3–1.6	Legacy systems, previous generation servers
Rear Door Heat Exchangers (RDHx)	20–40 kW/rack	1.2–1.4	Medium-density GPU clusters
Direct Liquid Cooling (DLC)	40–150 kW/rack	1.03–1.1	Blackwell, ultra-high compute density

Full immersion cooling is for experimental campuses. Most medium-sized companies simply do not need it. If the cooling and power can handle the peak, you can safely plan for years ahead. If not, future scaling plans will remain theoretical.

Ready to assess if your infrastructure is ready for enterprise LLMs? Contact our team for a hardware and readiness assessment.

Power and cooling capacity set the physical ceiling. The next layer is financial: once the hardware can actually run, the decision turns on whether owning it is cheaper than continuing to pay for cloud inference.

On-Prem Only Pays Off at High Utilization

The key distinction is the workload pattern.

Training is usually bursty. Inference runs continuously. That difference matters because continuous utilization is what can make owned hardware cheaper than the cloud.

The economics of on-premises deployment work only under one condition: sustained utilization.

While the break-even against API spend sits around $46k/month, achieving full CapEx payback within the first year requires higher sustained demand, closer to $65k/month (approximately 40% above the break-even point). On-premises starts to make sense when the workload is steady enough to keep the hardware busy. The total cost of ownership (TCO) breaks down as follows:

Deployment scale	Configuration	Annual TCO	Break-even vs API
Single GPU	1× RTX PRO 6000 (96 GB), in-house rack, 0.17 FTE (~7 hrs/week)	≈ $47,000	≈ $3,900/mo
Small node	4× RTX PRO 6000, air-cooled colo (~$2,500/mo), 0.5 FTE	≈ $165,000	≈ $13,750/mo
Production cluster	8× H200 SXM, DLC colo, 1.5 FTE	≈ $552,000	≈ $46,000/mo

Assumptions: 3-year depreciation; single node, no redundancy or failover capacity; open-weight model, no per-token licensing. Note: Production clusters typically require N+1 redundancy, which nearly doubles CapEx.

Power tariffs, rack costs and salary bands vary widely; the FTE line alone moves by 2–3× between the US, EU and Eastern Europe.

Sanity Check on Token Cost

This table provides a sanity check on token costs, serving as an example of the method rather than a strict benchmark, as numbers depend heavily on real-world throughput. Assumptions: 70B model at Q4, vLLM, high-batch serving, ~1,000 sustained output tokens/s per GPU (~8,000 tokens/s aggregate).

Scenario	Output tokens/yr	Effective cost
Cluster at 100% utilization	252 B	$2.19 / M tokens
Cluster at 70% utilization	177 B	$3.12 / M tokens
Cluster at 40% utilization	101 B	$5.47 / M tokens
At 70%, platform team shared across 5+ workloads (0.3 FTE allocated)	177 B	$1.49 / M tokens
Comparator: hosted open-weight 70B API	–	$0.60–0.90 / M tokens

While an owned cluster is generally more expensive at the unit level ($2.19/M tokens vs $0.60–0.90/M for hosted alternatives), it becomes competitive in three scenarios: when the platform team is amortized across 5+ workloads, when comparing against frontier models ($3–15/M), or when price is not the primary driver.

For most mid-market teams, the optimal approach is a hybrid model: maintain a stable baseline on owned on-premises hardware, while utilizing the cloud for peaks, spikes, and resource-intensive training runs.

Hardware and facility costs are only part of the overall cost picture. The software layer that serves the model has its own cost structure and long-term trade-offs, especially when choosing between a proprietary stack and an open-source alternative.

vLLM vs NVIDIA NIM

Choosing the right inference stack is critical to getting full value from on-premises hardware. In 2026, teams typically choose between NVIDIA NIM and the open-source vLLM framework.

NVIDIA NIM provides rapid turnkey deployment in containers and works well on Hopper/Blackwell. However, the license costs about $4,500 per GPU per year and ties you strictly to NVIDIA, limiting flexibility in hardware platform selection. However, for H100 and H200 NVL models, the license is included in the package, which means the argument that NIM is more expensive does not apply to this hardware.

vLLM is an open framework with no licensing fees. The main advantage is that it works with AMD, Intel, and NVIDIA, which is important for cost control. Although vLLM requires a more experienced MLOps team for fine-tuning, it helps avoid vendor lock-in.

Your infrastructure choice determines how much flexibility you retain. An open stack lets you switch hardware without being locked into a single vendor.

Cost and stack are not the only constraints. For enterprises in regulated industries, the question of where data is processed comes before the question of what it costs.

Data Sovereignty and Compliance

Enterprises in defense, healthcare and financial services cannot afford compromises here. Regulations such as the updated CMMC 2.0 (Cybersecurity Maturity Model Certification) for Department of Defense contractors, ITAR, and HIPAA impose strict controls on where data is processed, stored, logged, and accessed. For some workloads — ITAR-controlled technical data in particular — those controls are far easier to evidence on-premises or in an air-gapped environment than on shared infrastructure.

Enterprises now understand the difference between simple network isolation and a true air-gapped deployment. The right choice depends on the compliance requirements and threat model. A true air-gapped system runs on local infrastructure that is physically disconnected from the internet.

Air-gapped systems need an operating model designed for offline maintenance.

For clients and partners, data security is becoming a key trust factor. Companies demonstrating full control over information and compliance with international standards gain an advantage in regulated markets. Strong data protection can become a competitive advantage and open new markets. Adherence to data security and sovereignty requirements turns technical limitations into a strategic advantage. Companies that implement clear risk and security management processes during development gain access to regulated markets as reliable partners. This approach turns compliance from an obstacle into proof of the organization’s operational maturity and ability to create reliable, sustainable systems.

Why Most Pilots Fail in Production

Most projects that look promising in isolation never reach production. They fail because of weak data infrastructure and poor architectural decisions, not because of the models themselves. To avoid this disconnect, we use the Context-Grounded Discovery approach. We start coding only after we understand the system well enough to control the risk. This approach gives the team enough context to move from a pilot to a repeatable production process.

The pilot often performs much worse once it is connected to fragmented enterprise data and legacy systems. Building agents on fragmented data and legacy integrations increases hallucination risk. Security requirements added late can also force expensive changes to the architecture and access model.

Before approving the investment, the company must prove that the workload can justify it.

Hardware alone does not make the system production-ready. The team also needs to integrate the model into real business workflows. Engineers must configure a management layer (the harness) that provides context memory, permission control, and a full audit trail. Without that control layer and a team that can operate it, even powerful hardware remains an expensive demo. That integration work separates a production system from a proof of concept.

The pattern is consistent: successful production systems treat the model as only one component. The decisive layer is the control system that makes the model safe to operate. We call this layer the harness.

The Harness Layer

What happens without the harness?

Without a harness, the model lacks the controls needed to protect data, limit actions, and support reliable operations.

Four components of the harness

The harness turns a language model into a system the business can operate safely and predictably.

• Memory and context: Models do not remember past actions on their own. The harness maintains a persistent state that includes decision history, code context, and business process context. This layer allows the agent to “remember” what was done earlier, why a certain decision was made, and what constraints apply in the system. Such memory ensures consistency across long projects where one task affects another weeks later.
• Boundaries and permissions: Safe operation requires clearly defined boundaries. The harness allows only actions permitted by business policy and security rules. This prevents data loss, credential leaks, and unauthorized code execution. Every request is verified before execution, ensuring compliance with rules even in complex scenarios.
• Tool integration: The agent interacts with external systems through controlled tools. The harness provides access to APIs, databases, build systems, and code execution environments. The harness runs external actions in isolated sandboxes or ephemeral containers, then resets the environment when the task ends. It logs the results for review.
• Observability and audit: Regulated environments need a complete audit trail. Teams should be able to reconstruct what happened, why the system acted, and where an anomaly began. That record supports audits under the EU AI Act, CMMC, and other frameworks.

Together, these controls let agents handle routine work while people retain final authority.

The team matters as much as the technology. Experienced operators turn the stack into a system the business can actually use.

Conclusions

On-premises LLMs succeed only when three conditions hold together: the workload is steady enough to sustain high utilization, the data cannot leave your control for compliance or security reasons, and you have — or can hire — a dedicated platform team. If any one is missing, reserved cloud capacity is the better option, and unlike a hardware purchase, that decision is reversible.