AI agents in production: how to choose the right GPU

Introduction

The spread of AI agents in business is redefining IT infrastructure needs. As organisations multiply generative AI projects, internal chatbots, analytical assistants, decision agents, the question of GPU sizing becomes critical.

In a context where language models are becoming commonplace, the choice between an entry-level datacenter GPU and a high-performance configuration can mean a threefold difference in cost. Faced with that variability, technical teams have to arbitrate between raw power, versatility and budget.

This article details the technical characteristics of the main NVIDIA GPUs for AI agents, L40S, A100 and H100, sets out their optimal use cases and proposes a decision framework built on measurable criteria: model size, request volume, fine-tuning needs and economic constraints.

Understanding your needs before the hardware

Originally designed for graphics rendering, GPUs rest on thousands of cores able to process identical operations in parallel. That contrasts sharply with CPUs, which are optimised for sequential processing. The difference lets GPUs accelerate repetitive matrix operations such as multiplications and convolutions, which are essential to modern machine learning. One accelerator can therefore handle a very large number of neurons or vectors in parallel, which explains the gains observed during intensive training.

The four dimensions of GPU sizing

Before identifying the right GPU, the use case must be qualified precisely along four technical axes.

First dimension: the size of the language model.

A model with 7 billion parameters (7B) requires roughly 14 to 16 GB of VRAM in FP16 precision, or 7 to 8 GB quantized to INT8. A 13B model needs 26 to 28 GB in FP16. Beyond 34B parameters the required VRAM passes 68 GB, making GPUs of 80 GB or more mandatory, or multi-GPU configurations for 70B+ models that exceed 140 GB.

Second dimension: the type of operation planned.

Inference, generating responses in production, imposes strict VRAM constraints but prioritises throughput. Fine-tuning, by contrast, multiplies memory needs three to fourfold. That increase comes from having to store backpropagation gradients and optimiser states (momentum, variance for Adam). Fine-tuning also demands high memory bandwidth to speed up iterative computation.

Third dimension: request volume and target latency.

A simple formula estimates the GPU capacity needed:

GPU capacity = (requests per second × inference time) / target utilisation rate

For example, handling 50 requests per second with 0.8 second latency and 70% utilisation needs capacity for about 57 simultaneous inferences, that is two to three GPUs depending on the model deployed.

Fourth dimension, operational constraints.

The available budget, whether capex (purchase) or opex (rental), directly influences the choice. Power consumption and cooling needs range from 350 W for an L40S to 700 W for an H100. Software compatibility with the frameworks (PyTorch, TensorFlow, vLLM) and compliance requirements, notably hosting data in Switzerland for certain categories of sensitive data under the nFADP, complete the selection criteria.

Qualification method

It is worth stating the need in this form: "My AI agent uses a [size] model, handles [volume] requests a day, needs [fine-tuning, yes/no], with [sensitive/non-sensitive] data." That qualification then maps the use case precisely onto the available GPU specifications.

Technical focus on datacenter GPUs

NVIDIA A100, the versatile reference GPU

NVIDIA has sold the A100 since 2020. Built on the Ampere architecture, this datacenter GPU comes in two versions: 40 GB and 80 GB of HBM2e memory. The 40 GB version offers 1,555 GB/s of memory bandwidth, the 80 GB version reaches 2,039 GB/s. TDP is 400 W for both variants.

Detailed technical characteristics.

The A100 80 GB has 6,912 CUDA cores and 432 third-generation Tensor Cores. It supports FP64, FP32, FP16, BF16, TF32 and INT8 precision. NVLink support allows up to eight GPUs to be connected in a multi-node configuration, with a total bandwidth of 600 GB/s per GPU.

Optimal use cases.

The A100 positions itself as the reference for models from 7 to 34 billion parameters. It suits architectures needing a mix of inference and occasional fine-tuning particularly well. Finance, research and digital services use it to deploy RAG (Retrieval-Augmented Generation) agents, document analysis assistants or high-availability internal chatbots. Rental rates for the A100 80GB generally sit between CHF 1,200 and 1,800 a month depending on the provider and commitment length.

---

NVIDIA H100: raw power for intensive loads

The H100, launched in 2022, is built on the Hopper architecture. NVIDIA states that this GPU carries 80 GB of HBM3 memory with 3,350 GB/s of bandwidth, 2.15 times that of the A100 80GB. TDP reaches 700 W, requiring reinforced cooling infrastructure.

Advanced technical characteristics.

The H100 includes 16,896 CUDA cores and 528 fourth-generation Tensor Cores. The differentiator is the Transformer Engine, which executes operations in FP8 precision while maintaining result accuracy through dynamic scaling management. That capability improves throughput on transformer models by 30 to 60% according to NVIDIA benchmarks, although the real gains observed in production generally vary between 20 and 50% depending on the workload and the optimisations applied.

Measured performance gains.

On fine-tuning 70B models, the H100 shows a 2.8× speed gain over the A100 according to the manufacturer's benchmarks. In inference with INT8 quantization, the gain varies between 40 and 60% depending on model size and the optimisations applied.

Use cases justifying the investment.

The H100 imposes itself in three scenarios: deploying models beyond 40 billion parameters, regular fine-tuning (monthly or more frequent) of complex models, and applications requiring latency below 500 milliseconds. H100 rental rates generally sit between CHF 1,800 and 2,400 a month depending on the provider, a premium of 30 to 50% over the A100 80GB. That extra cost is justified when the performance gains translate into saved compute time or into additional revenue from lower latency.

---

NVIDIA L40S: the inference-optimised alternative

The L40S, based on the Ada Lovelace architecture and introduced in 2023, carries 48 GB of GDDR6 memory with 864 GB/s of bandwidth. Its 350 W TDP makes it the most energy-efficient datacenter GPU of the three options analysed.

Specific technical characteristics.

The L40S has 18,176 CUDA cores and 568 fourth-generation Tensor Cores optimised for inference. Unlike the A100 and H100, which use HBM (High Bandwidth Memory), the L40S relies on GDDR6, which explains its lower memory bandwidth but also its lower unit cost. That memory difference particularly affects operations needing intensive memory access, such as fine-tuning large models, where HBM bandwidth brings a significant advantage.

Technical positioning.

NVIDIA positions the L40S as a versatile GPU able to handle mixed workloads: generative AI, graphics rendering and scientific computing. For AI agents it excels at pure inference on models from 7 to 20 billion parameters. Its GDDR6 memory configuration makes it less suited to intensive fine-tuning of models beyond 20B, where the higher bandwidth of HBM GPUs becomes decisive.

Energy advantage.

At 350 W against 400 W (A100) and 700 W (H100), the L40S holds a significant advantage in multi-GPU architectures. A 4×L40S configuration draws 1,400 W, the equivalent of 2×H100. Over 36 months of continuous operation, the electricity saving can amount to several thousand francs compared with an equivalent H100 configuration. L40S rental rates generally sit between CHF 700 and 1,000 a month.

Technical comparison table

*These are not market averages: they are the monthly prices published in the Hikube catalogue for L40S, A100-80 and H100, checkable on the pricing page on the day you read this. The ranges quoted earlier in the article are orders of magnitude observed across providers.

An overview of the GPU alternatives

NVIDIA professional GPUs.

The RTX A6000 (48GB GDDR6) and A5000 (24GB) range targets workstations and prototyping environments. The RTX A6000, with a 300 W TDP, suits small-scale deployments or development phases. However, the absence of NVLink support and performance below datacenter GPUs limit their relevance in intensive production.

RTX 40 consumer GPUs.

The RTX 4090, with 24 GB of VRAM and a 450 W TDP, offers excellent raw performance for a low purchase cost (around CHF 1,800). But the absence of datacenter-optimised drivers, the lack of ECC memory support and a limited warranty make it unsuitable for critical production environments. It remains relevant for local development or non-critical micro-production.

AMD and Intel alternatives.

AMD offers the MI300X with 192 GB of HBM3 memory, targeting very large models. The ROCm software ecosystem is progressing but remains less mature than CUDA. Intel is developing Gaudi 2 and 3, optimised for training, but their adoption stays marginal for general inference. NVIDIA holds around 90% of the AI GPU market, with the CUDA ecosystem a competitive advantage that is hard to contest in the short term.

Matching use cases and economic trade-offs

Conversational RAG agent (7-13B model)

This type of agent typically handles 100 to 500 requests a minute with target latency below 2 seconds. A 7B model quantized to INT8 requires 8 GB of VRAM, a 13B model around 14 GB.

Recommended configurations:

Analysis

For pure inference without fine-tuning, the L40S offers excellent TCO with performance amply sufficient for this segment. The A100 is justified if the model has to be regularly retrained on specific business data.

---

Analytical agent (13-34B model)

Document and data analysis agents deploy models of 13 to 34 billion parameters, with quarterly or monthly fine-tuning on business data.

Recommended configuration by size:

Economic analysis

The cost difference between the A100 80GB and the H100 is justified when fine-tuning time divided by 2.8 represents a team productivity gain greater than that monthly premium. For a technical resource billed at CHF 150 an hour, saving 20 hours of compute a month amply covers the investment.

---

Complex multimodal agent (70B+ model)

Models beyond 70 billion parameters necessarily require multi-GPU configurations.

Typical configurations:

Recommendation.

For an organisation retraining a 70B model monthly, the H100 cuts compute time from 15 days to 5, freeing team resources and accelerating time to market. For pure inference under budget constraint, multi-A100 offers the best compromise.

---

Multi-agent architecture

Organisations deploying several distinct agents on shared infrastructure favour homogeneous GPU pools to simplify orchestration.

Strategies observed:

The mistakes to avoid

Mistake 1, preventive over-sizing

Symptom observed.

Technical teams systematically choose the most powerful GPU "to be safe" without analysing real needs. That approach leads to deploying H100s for agents running 7-13B models in pure inference.

Measured consequence.

Production analyses show that a significant proportion of the high-performance GPUs deployed for conversational agents run below 40% of capacity. The extra cost amounts to several thousand francs a month with no perceptible performance gain.

Recommended fix.

Start with a GPU matched to the current need (L40S or A100 depending on the case), measure real utilisation for 2 to 4 weeks, then adjust if necessary. Cloud infrastructure allows that flexibility without penalty.

---

Mistake 2: underestimating fine-tuning needs

Symptom observed.

Selecting an L40S for a project needing monthly fine-tuning of 13B+ models, on the argument of lower cost.

Measured consequence.

Fine-tuning a 13B model on an L40S takes three to four times longer than on an A100, because of the lower memory bandwidth. For a monthly retraining needing 48 hours on an A100, that means 6 days on an L40S. The cost in team time quickly exceeds the GPU savings.

Recommended fix.

For any project with fine-tuning planned more than once a quarter, choose an A100 at minimum. Calculate the total cost including human time: (compute hours × team hourly cost) plus GPU cost.

---

Mistake 3, ignoring non-GPU bottlenecks

Symptom observed.

Investing in a high-performance GPU without correctly sizing the rest of the infrastructure, CPU, RAM, storage, network.

Consequence.

The GPU spends a significant share of its time waiting for data. Observed performance falls short of expectations, at a higher infrastructure cost.

Recommended minimum specifications:

• CPU, 32 cores minimum (64 for multi-GPU)
• RAM, a 1:4 ratio with GPU VRAM (for example A100 80GB → 320GB of system RAM)
• Storage, NVMe mandatory (3,000 MB/s read minimum)
• Network, 25 Gbps for multi-GPU, 100 Gbps for configurations of 4 GPUs and more

Fix.

Before investing in a premium GPU, audit the complete infrastructure and identify the existing bottlenecks.

---

Mistake 4, neglecting data sovereignty

Symptom observed.

Selecting a GPU provider on price alone, without checking the physical location of the servers and the compliance certifications.

Regulatory consequence.

The nFADP imposes specific hosting requirements for certain categories of sensitive data. Deploying on GPUs hosted outside the territory can constitute non-compliance for certain data types and sectors.

Recommended fix.

For regulated sectors (finance, healthcare, public administration) or processing of sensitive personal data, systematically verify:

• The physical location of the GPU servers
• The provider's certifications (ISO 27001) and Swiss residency (nFADP); Hikube is not a French HDS host and is not HDS-certified
• The contractual data protection clauses

---

Mistake 5, forgetting that models evolve

Symptom observed.

Sizing the GPU infrastructure strictly for the current model without anticipating changes.

Consequence.

Language models progress quickly. An agent deployed on a 7B model today may need to move to 13B or 20B within 12 to 18 months to stay competitive. A GPU sized to the minimum then becomes limiting, forcing an expensive migration.

Recommended fix.

Plan a 30-50% margin on VRAM. If the current need is a 7B model (14GB), choose a GPU with 24GB minimum rather than 16GB, allowing a move to 13B with no infrastructure change.

Enterprise deployment: a practical guide

Phase 1, audit and qualification (2-4 weeks)

Map the use cases.

Identify all the AI agents planned over 18 to 24 months, conversational, analytical, decision agents. For each, document the model size envisaged, the anticipated request volume and the retraining frequency.

Assess the regulatory constraints.

For regulated sectors or sensitive data processing, check the location and certification requirements. That analysis determines whether hosting in Switzerland is mandatory, which directly affects the choice of provider.

Audit the existing infrastructure.

Measure current CPU, RAM, storage and network capacity. Identify the potential bottlenecks that would constrain a high-performance GPU.

Expected deliverables.

A technical specification document listing, for each agent: model, VRAM required, target throughput, latency and compliance constraints.

---

Phase 2, proof of concept and technical validation (2-4 weeks)

Method.

Rent the candidate GPUs (L40S, A100, H100 as pre-selected) for 1 to 2 weeks. Deploy the agent under a realistic simulated load: representative request volume, usage patterns (peaks, troughs), prompt types.

Metrics to measure:

• GPU utilisation, average rate and peaks (target 60-80%)
• Latency, P50, P95, P99 (identify the outliers)
• Throughput, requests handled per second at nominal load and at peak
• Cost per request, divide the hourly GPU cost by the number of requests handled

Decision.

Compare the tested configurations on a cost-performance table. A GPU with 30% unused capacity signals over-sizing. A P95 latency above the objective indicates under-sizing.

Good practice.

Also test the software optimisations (vLLM, INT8 quantization), which can double performance without changing GPU.

---

Phase 3: deployment and architecture (4-8 weeks)

Single-GPU architecture.

For a single agent under moderate load, a single-GPU configuration is enough. Recommendations:

• Install the GPU on a dedicated server with the minimum specs identified in phase 1
• Configure GPU monitoring (utilisation, temperature, memory errors)
• Plan a backup strategy, regular backups of fine-tuned models and configurations

Homogeneous multi-GPU architecture.

For several agents or a high-availability agent, deploy 2 to 4 identical GPUs with load balancing. This approach allows:

• Automatic load distribution between GPUs
• Fault tolerance (one failing GPU does not stop the service)
• Simple horizontal scaling (adding further GPUs)

Hybrid architecture.

For mixed needs (inference plus fine-tuning), combine inference-optimised GPUs (L40S) with versatile ones (A100):

• L40S dedicated to continuous inference (agents in production)
• A100 reserved for monthly or quarterly fine-tuning
• A 20-30% saving against a full A100 configuration

Network recommendation.

For multi-GPU, a 25 Gbps interconnect is the minimum. Beyond 4 GPUs, choose 100 Gbps to avoid congestion during synchronisation.

---

Phase 4: governance and continuous optimisation

Continuous monitoring.

Put dashboards in place tracking:

• GPU utilisation rate by hour and by day (identify the quiet periods)
• P95 latency over time (detect degradation)
• Monthly cost per request (optimise TCO)
• GPU errors (ECC errors, timeouts, OOM)

Quarterly review.

Analyse three months of metrics and identify the optimisations:

• Chronic under-use (below 50%) → the GPU could be downgraded
• Regular saturation (above 85%) → scaling needed
• Strong day/night variation → optimise capacity by time of day

Model evolution.

Plan migrations to newer or larger models. Anticipate additional VRAM needs six months ahead to avoid expensive emergency migrations.

Cost management.

Compare provider rates regularly and renegotiate the contracts. The GPU market moves quickly, and differences of 15-20% can appear between providers for identical configurations.

TCO analysis and economic considerations

Total cost of ownership over 36 months

Calculation assumptions:

• 24/7 use in production
• Electricity rate, CHF 0.18/kWh (average Swiss industrial rate)
• Cooling, 1.5× GPU consumption (standard PUE, Power Usage Effectiveness)
• Network and storage infrastructure, +15% of the monthly GPU cost

Single-GPU configuration over 36 months:

*Rates of the referenced provider. **Including cooling (TDP × 1.5 × 24/7 × 36 months × CHF 0.18/kWh). ***15% of the rental cost for network, storage and maintenance.

Analysis. The L40S shows a TCO 48% below the A100 and 61% below the H100 over 36 months for pure inference. That advantage narrows as soon as monthly fine-tuning enters the picture, the time saved on an A100 or H100 partly offsetting the higher cost.

Multi-GPU configuration: cost strategies

Scenario, 4 GPUs for a multi-agent architecture

*Including electricity and infrastructure

Hybrid strategy in detail.

For an organisation deploying 5 agents (3 in pure inference on 7-13B models, 2 needing monthly fine-tuning on 13-20B):

• 2× L40S dedicated to the 3 pure-inference agents
• 2× A100 80GB for the 2 agents with fine-tuning

Saving, CHF 57,572 over 36 months against a full A100 configuration. This approach needs more complex orchestration but maximises ROI by allocating each GPU to its optimal use.

Break-even: rent against buy

For organisations considering buying GPUs rather than renting, calculating the break-even point becomes relevant.

Estimated purchase cost (hardware only):

• L40S, ~CHF 8,000
• A100 80GB, ~CHF 15,000
• H100, ~CHF 30,000

Approximate break-even (excluding infrastructure, electricity and maintenance costs):

• L40S, 11 months of rental
• A100 80GB, 10 months of rental
• H100, 15 months of rental

Analysis. Buying becomes worthwhile for deployments of 24 months minimum with intensive use (above 80% of the time). Below 18 months, or for variable usage, renting remains preferable for its flexibility.

Factors to consider before buying:

• Obsolescence, AI GPUs evolve quickly (an 18-24 month cycle)
• Maintenance, manufacturer warranty, spare parts
• Resale, residual value after three years (30-40% for datacenter models)

Implications and outlook

What these trade-offs change for organisations

The diversification of the datacenter GPU offering now lets technical teams tune their infrastructure finely along three axes, raw performance, versatility and economic efficiency. The L40S makes inference at scale accessible for 7-13B models, the majority segment in SME and mid-cap deployments. The A100 80GB keeps its position as the versatile reference GPU, suited to evolving architectures where needs swing between inference and fine-tuning. The H100 stays reserved for use cases where its higher capability is economically justified: 70B+ models, ultra-low latency, or intensive fine-tuning with demonstrable ROI.

Trends observed and expected developments

First trend, software optimisation is catching up with hardware.

Frameworks such as vLLM, TensorRT-LLM and SGLang improve throughput by 2 to 3× on identical hardware. A properly optimised A100 can rival an unoptimised H100 on some inference loads. Testing the software optimisations before investing in premium hardware is therefore advisable.

Second trend: quantization is becoming standard.

INT8 precision is becoming general in production with a quality loss generally below 3% on most benchmarks. INT4 quantization, more aggressive, can show losses of 5 to 8% depending on the task, requiring case-by-case validation. These techniques divide VRAM needs by two to four, allowing 13B models on 24GB GPUs or 34B models on 48GB GPUs.

Third trend: data sovereignty is asserting itself.

Regulatory constraints and sector requirements (finance, healthcare, public administration) favour hosting GPU infrastructure in Switzerland. Cloud providers offering L40S, A100 and H100 in the Swiss region answer that growing demand, with certifications suited to regulated sectors.

Conclusion: an accelerator that has become indispensable

Important note, the recommendations in this article are based on generic use cases. Every infrastructure has its own specifics. Consulting a cloud architect or an infrastructure specialist is advisable to validate the sizing in your particular context.