Tuning a GPU inference engine: batching, latency, metrics

This article’s angle: the inference engine itself, batching, quantisation, latency, and the metrics that say whether the card is really working. Nothing on scheduling or sharing: those are other articles.

Introduction

Metrics, frameworks and strategies to maximise the ROI of your GPUs

The cost of GPUs in production puts growing economic pressure on organisations. An H100 GPU costs between CHF 2 and CHF 10 per hour depending on the provider, that is CHF 17,500 to CHF 87,600 a year for a single instance running continuously. Yet according to a 2024 AI Infrastructure Alliance study, only 7% of organisations reach more than 85% GPU utilisation at peak load. The remaining 93% waste between 15% and 60% of their GPU capacity, tens of thousands of francs a year per GPU.

This inefficiency comes from three technical factors: sub-optimal data pipelines that leave GPUs waiting, unsuitable batching configurations that cap throughput, and the absence of granular monitoring, which makes bottlenecks impossible to identify. This article sets out the four levers of GPU optimisation in production, details the critical metrics to watch, and proposes a monitoring framework based on DCGM and Prometheus.

The goal: turning under-used GPUs into performing assets with measurable ROI.

Diagnosing inefficiency in four metrics

GPU optimisation starts with measurement. Four key metrics identify the sources of inefficiency precisely and quantify the potential gains.

GPU utilisation: the fundamental metric

GPU utilisation measures the percentage of time during which at least one kernel is executing on the GPU. This metric, available through nvidia-smi or DCGM, is the first-level indicator. Utilisation below 60% signals an optimisation problem, while utilisation above 85% can indicate saturation calling for a scale-up.

Target GPU utilisation = 60-80%

Interpretation:
- < 40%, severe under-use, data pipeline or batching unsuitable
- 40-60%, optimisation needed, significant gains available
- 60-80%, optimal zone, good performance/cost balance
- > 85%, potential saturation, check P95/P99 latency

A caution, a GPU showing 100% utilisation is not necessarily optimal. This metric does not distinguish between compute-intensive and memory-bound kernels. A GPU saturated by memory accesses will show 100% utilisation while delivering sub-optimal performance.

Memory bandwidth: detecting bottlenecks

Memory bandwidth utilisation measures the percentage of theoretical bandwidth actually used. For an A100 80GB with 2,039 GB/s of HBM2e bandwidth, utilisation at 30% means only 612 GB/s are being exploited. That gap reveals either a data pipeline problem or poorly optimised kernels requiring too much computation per byte transferred.

LLM inference workloads are typically memory-bound: the GPU spends more time waiting for data than computing. On these workloads, optimising memory bandwidth through quantization (FP16, INT8) or through optimised kernels (FlashAttention) yields 2 to 3× gains in throughput.

Throughput against theoretical: measuring the gap

Real throughput is measured in tokens per second for language models, or inferences per second for vision models. This metric is compared to the theoretical throughput calculated from the manufacturer's specifications. A gap greater than 40% between theoretical and real indicates missing optimisations.

Example of a theoretical throughput calculation (LLM inference):

Theoretical throughput = (GPU TFLOPS × precision) / (model parameters × 2)

A100 80GB, 13B model in FP16:
= (312 TFLOPS × 0.5) / (13B × 2)
= 156 / 26 = ~6 tokens/second (order of magnitude)

If measured throughput < 4 tokens/s → optimisation required

P50/P95/P99 latency, guaranteeing quality of service

Latency is measured in percentiles, not averages. P50 latency (the median) indicates the typical user experience, while P95 and P99 reveal the degraded cases affecting 5% and 1% of requests. For interactive applications (chatbots, assistants), P99 latency above one second significantly degrades the user experience.

The four technical optimisation levers

Once the baseline metrics are in place, four technical levers substantially improve GPU performance in production.

1. Dynamic batch size: maximising throughput

Batch size determines how many requests are processed simultaneously. A batch size that is too small under-uses the GPU (30-40% utilisation), while one that is too large saturates VRAM and causes out-of-memory errors. The optimal calculation takes into account available VRAM, model size and sequence length.

Optimal batch size = (available VRAM - model size) / memory per sequence

Example: A100 80GB, 13B model in FP16, 512-token sequences

Model size in FP16, 13B × 2 bytes = 26 GB
Memory per sequence, 512 tokens × 13B × 2 bytes / 1e9 ≈ 13 GB (KV cache approximation*)
Available VRAM, 80 - 26 = 54 GB

Optimal batch size ≈ 54 / 13 ≈ 4 concurrent requests

Modern frameworks such as vLLM implement continuous batching (or in-flight batching), which dynamically merges new requests into a batch already generating. This technique improves throughput by 2.2 to 3.5× over static batching, according to the vLLM benchmarks on LLaMA.

*This estimate gives an order of magnitude but does not faithfully model real memory consumption. In practice that consumption is dominated by the KV cache, the context length and the inference engine's optimisations, and needs empirical validation to be sized correctly.

2. Precision and quantization: FP32 → FP16 → INT8

Reducing arithmetic precision lowers VRAM requirements and speeds up computation. FP32 (32-bit floating point) is the training standard but is oversized for inference. FP16 halves memory use with a quality loss generally under 1%. INT8 quantization divides VRAM use by four, with degradation of 2 to 5% depending on the model.

*INT4 requires case-by-case validation; quality losses vary by task.

Quantization is implemented in practice through specialised frameworks: AWQ (Activation-aware Weight Quantization), GPTQ, or the FP8 kernels of Hopper GPUs (H100). The H100 includes a Transformer Engine that executes natively in FP8, improving throughput by 20 to 50% on transformer architectures compared with FP16.

3. Optimised inference frameworks: vLLM, TensorRT-LLM, SGLang

Specialised inference frameworks implement optimisations that cannot be reproduced with standard PyTorch or TensorFlow. Three frameworks dominate the market in 2025, each with distinct performance-complexity trade-offs.

vLLM positions itself as the reference framework for production. Its key innovation, PagedAttention, treats KV cache memory as virtual memory, removing fragmentation and allowing more concurrent requests to be served. According to the Cerebrium 2024 benchmarks, vLLM shows the best Time To First Token (TTFT) at 123 ms on LLaMA 3.1 70B with an H100, and reaches 4,741 tokens/second at 100 concurrent requests according to Clarifai. Its native Hugging Face integration and OpenAI-compatible API make adoption straightforward.

TensorRT-LLM from NVIDIA targets deployments demanding maximum performance. Built on TensorRT, it compiles models into optimised CUDA kernels and exploits Tensor Cores to the full. The LMSYS 2024 benchmarks show TensorRT-LLM matching or beating vLLM on latency at low concurrency (35-50 ms TTFT), but degrading under heavy load. Its setup complexity (per-model compilation, mandatory Docker container) reserves it for teams with CUDA expertise.

SGLang introduces RadixAttention, a prefix-sharing structure built on a radix tree. This approach excels on workloads with heavy context reuse: multi-turn chat, few-shot prompting, agents. The LMSYS benchmarks show SGLang reaching up to 3.1× the throughput of vLLM on LLaMA-70B with intensive prefix reuse. For RAG applications or agents with repetitive prompts, SGLang delivers substantial gains.

Framework selection guide:

• vLLM, general production, high concurrency (>50 req/s), Hugging Face integration required
• TensorRT-LLM, maximum performance, ultra-low latency (<50 ms), teams with NVIDIA expertise
• SGLang, RAG, agents, multi-turn chat with heavy context reuse
• Ollama, fast prototyping, local development, not suited to production at scale

4. Optimising the data pipeline

Data pipeline inefficiency is the main cause of GPU under-use. According to Clarifai, poorly optimised pipelines waste up to 40% of GPU cycles. Three critical optimisations remove those losses.

First, data locality. Storing datasets in the same region as the GPUs cuts transfer latency from 80-200 ms to 5-15 ms. For cold data that is rarely accessed, using archive storage (S3 Glacier, Azure Archive) rather than high-performance storage reduces cost by 70 to 90% with no impact on hot data.

Second, compressed formats. Storing data in Parquet or ORC rather than CSV reduces size by 60 to 80% and speeds up reads thanks to columnar compression. Decompression time is negligible against the network and disk transfer gains.

Third, prefetching and caching. Loading batch N+1 while the GPU processes batch N removes waiting time. The PyTorch DataLoader implements this through the num_workers parameter. A rule of thumb: num_workers = 2 × number_of_gpus to balance parallelism and overhead.

Recommended monitoring stack

Effective GPU monitoring rests on three layers, low-level metric collection, time-series aggregation and storage, visualisation and alerting. The recommended architecture combines DCGM, Prometheus and Grafana.

DCGM: collecting NVIDIA metrics

NVIDIA Data Center GPU Manager (DCGM) is the reference collection layer for datacenter GPUs. Unlike nvidia-smi, designed for one-off checks, DCGM exposes a continuous stream of metrics through an API. The DCGM Exporter turns those metrics into Prometheus format, available on an HTTP endpoint at port 9400.

DCGM Exporter installation (Docker):

docker run -d
--gpus all
--cap-add SYS_ADMIN
--network host
--name dcgm-exporter
--restart unless-stopped
nvcr.io/nvidia/k8s/dcgm-exporter:3.3.8-3.6.0-ubuntu22.04

# Verification
curl localhost:9400/metrics | grep DCGM_FI_DEV_GPU_UTIL

DCGM exposes more than 50 metrics, including, GPU utilisation (DCGM_FI_DEV_GPU_UTIL), VRAM use (DCGM_FI_DEV_FB_USED), GPU and memory temperature (DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_MEMORY_TEMP), power draw (DCGM_FI_DEV_POWER_USAGE), SM and memory clocks (DCGM_FI_DEV_SM_CLOCK, DCGM_FI_DEV_MEM_CLOCK), and PCIe error counters (DCGM_FI_DEV_PCIE_REPLAY_COUNTER).

Prometheus, time-series storage

Prometheus scrapes DCGM Exporter metrics at regular intervals (typically 15 seconds) and stores them in its time-series database. The Prometheus configuration defines the targets to scrape and data retention.

Prometheus configuration (prometheus.yml):

scrape_configs:
- job_name, 'dcgm-exporter'
static_configs:
- targets, ['localhost:9400']
scrape_interval, 15s
scrape_timeout, 10s

For Kubernetes deployments, DCGM Exporter is deployed through Helm or as part of the NVIDIA GPU Operator, with automatic service discovery. Prometheus detects DCGM Exporter pods automatically through Kubernetes labels.

Grafana: visualisation and alerting

Grafana consumes Prometheus metrics and generates interactive dashboards. NVIDIA provides a pre-configured dashboard (ID 12239) showing the essential GPU metrics. The critical panels include: GPU utilisation over time (cluster average plus per-GPU), VRAM use with threshold lines at 70% and 90%, GPU and memory temperature, instantaneous and cumulative power draw, and P50/P95/P99 latency of application requests.

Grafana alerts trigger notifications on Slack, PagerDuty or email according to configurable thresholds. Recommended alerts: GPU utilisation below 40% for 30 minutes (under-use), VRAM above 90% for 5 minutes (OOM risk), GPU temperature above 80°C for 10 minutes (throttling risk), PCIe errors above 0 (hardware problem).

The five mistakes that degrade performance

1. Neglecting continuous monitoring

Symptom: deploying GPUs without granular monitoring, relying only on occasional nvidia-smi checks or aggregated cloud-provider metrics.

Impact: without visibility, inefficiency cannot be identified. A Spheron 2025 study shows that without monitoring, organisations discover GPUs running below 30% utilisation for weeks, wasting tens of thousands of francs. Bottlenecks (data pipeline, unsuitable batch size) stay invisible until users complain about high latency.

Fix: implement DCGM Exporter, Prometheus and Grafana from the first GPU deployed. Overhead cost: around 5% of GPU performance, amply offset by the optimisations it reveals. Define dashboards with alerts on utilisation below 40%, VRAM above 90%, P95 latency above 500 ms. Review metrics weekly with the engineering team to identify optimisation opportunities.

2. Using standard PyTorch or TensorFlow in production

Symptom: serving models directly with PyTorch or TensorFlow without a specialised inference framework.

Impact: PyTorch and TensorFlow are optimised for training, not production inference. The absence of continuous batching, optimised KV cache handling and specialised kernels cuts throughput by 2 to 4× compared with vLLM or TensorRT-LLM. For a service receiving 1,000 requests a day, that means serving 250 to 500 instead of 1,000 on the same GPU, requiring 2 to 4 GPUs instead of one. Additional annual cost: CHF 35,000 to 105,000 for A100s.

Fix: migrate to vLLM for general production (setup in 1-2 days), TensorRT-LLM for critical latency (setup 1-2 weeks), or SGLang for RAG and agents. Benchmark before and after migration to quantify the real gains. In 90% of cases vLLM offers the best performance-to-complexity ratio.

3. Ignoring quantization

Symptom: serving every model in FP32 or FP16 without evaluating INT8 or advanced quantization techniques.

Impact: INT8 quantization halves VRAM requirements against FP16 with a quality loss of 2 to 5%. For a 13B model going from 26 GB (FP16) to 13 GB (INT8), you can either double the batch size (2× throughput) or serve two models on the same GPU. No quantization means wasting 40 to 50% of GPU capacity.

Fix: evaluate INT8 systematically for production inference. Test on representative datasets, measure quality degradation (accuracy, F1, BLEU depending on the task). If degradation is below 5% and acceptable for the use case, deploy in INT8. Use AWQ or GPTQ for optimal quantization. On Hopper GPUs (H100), exploit native FP8 through supported frameworks (vLLM, TensorRT-LLM).

4. Under-sizing the batch size

Symptom: using batch size 1, or a static batch size left too small by default, with no optimisation.

Impact: a batch size of 1 massively under-uses the GPU (20-40% utilisation). The GPU spends more time waiting between requests than computing. An optimal batch size (calculated from available VRAM and sequence length) improves throughput by 3 to 5× without raising median latency. An unsuitable batch size wastes 60 to 80% of GPU capacity.

Fix: calculate the optimal batch size with the formula (available VRAM - model size) / memory per sequence. For production applications, use continuous batching (vLLM, TGI), which adjusts dynamically. Monitor GPU utilisation: if below 60%, raise the batch size until 70-80%. Measure the impact on P95/P99 latency: if the increase is acceptable (below 20%), validate the new batch size.

5. Neglecting the energy cost

Symptom: optimising only for performance without considering power consumption.

Impact: an H100 can draw up to 700 W at maximum load, that is 16.8 kWh a day or 6,132 kWh a year. In Switzerland (electricity around CHF 0.20/kWh including cooling), that comes to CHF 1,226 a year per GPU. A sub-optimal GPU running at 100% utilisation needlessly (instead of 70% with optimisations) wastes 30% of that energy, CHF 368 a year. Across a 10-GPU cluster, CHF 3,680 a year of avoidable electricity cost.

Fix: monitor power draw through DCGM (DCGM_FI_DEV_POWER_USAGE). Optimise for performance per watt, not raw performance alone. Techniques: quantization (cuts consumption by 20-30%), optimal batch size (avoids under-use), automatic shutdown schedules (dev and test environments off outside working hours), choosing a GPU matched to the workload (L40S at 350 W against H100 at 700 W for pure inference). Calculate a TCO including electricity over 36 months when arbitrating between GPUs.

Worked example: optimising a RAG chatbot

Take a concrete case, an internal RAG chatbot for a Swiss company, 500 employees, around 2,000 requests a day, LLaMA-2 13B model, initially deployed on an A100 40GB.

Starting point: model served with standard PyTorch in FP16, batch size 1, no detailed monitoring. Observed metrics: GPU utilisation 35%, throughput 1.2 tokens/second, P95 latency 800 ms, GPU cost CHF 3,200/month (cloud provider).

Optimisations applied:

1. Migration to vLLM with continuous batching → GPU utilisation 72%, throughput 3.8 tokens/second (+217%)
2. INT8 quantization through AWQ → VRAM reduced from 26 GB to 13 GB, allowing an optimal batch size of 8 instead of 3
3. DCGM, Prometheus and Grafana implemented → full visibility, alerts on anomalies
4. Data pipeline optimised → prefetching enabled, datasets in Parquet, storage in the same region

Measured results:

• GPU utilisation, 35% → 68% (+94%)
• Throughput, 1.2 → 4.1 tokens/second (+242%)
• P95 latency, 800 ms → 320 ms (-60%)
• Possible downgrade from A100 40GB to L40S 48GB, cost CHF 3,200 → 1,800/month (-44%)
• Annual saving, CHF 16,800

This case illustrates the ROI of GPU optimisation, an investment of about 40 engineering hours (framework migration, testing, monitoring), with payback in under three months through infrastructure savings.

In short, three key points

Monitoring turns GPU cost into an optimisable asset

Implementing DCGM, Prometheus and Grafana immediately reveals inefficiency that is invisible without instrumentation. The four critical metrics: GPU utilisation (target 60-80%), memory bandwidth (above 50% for memory-bound workloads), throughput (above 60% of theoretical) and P95/P99 latency, identify precisely where to optimise. Monitoring overhead (around 5% of GPU performance) is negligible against the gains: according to the AI Infrastructure Alliance 2024 study, only 7% of organisations reach above 85% utilisation without structured monitoring. The remaining 93% waste 15 to 60% of capacity, tens of thousands of francs a year per GPU. Monitoring is not a cost but an investment with immediate ROI.

Specialised inference frameworks multiply performance by 2 to 4

Standard PyTorch and TensorFlow are not designed for production inference. Specialised frameworks (vLLM, TensorRT-LLM, SGLang) implement continuous batching, optimised KV cache handling and specialised CUDA kernels, delivering 2 to 4× gains in throughput. vLLM reaches 4,741 tokens/second at 100 concurrent requests against around 1,500 with standard PyTorch, according to the Clarifai 2024 benchmarks. For general production, vLLM offers the best compromise: setup in 1-2 days, OpenAI-compatible API, native Hugging Face integration. TensorRT-LLM targets ultra-low latency (below 50 ms TTFT) but requires CUDA expertise. SGLang excels on RAG and agents with intensive context reuse (3× gains through RadixAttention). The choice of framework depends on the use case, not on an absolute hierarchy.

Quantization frees 40 to 50% of capacity without significant degradation

INT8 quantization halves VRAM requirements against FP16, with quality losses of 2 to 5% that are generally acceptable in production. A 13B model goes from 26 GB (FP16) to 13 GB (INT8), allowing either a doubled batch size (2× throughput) or two models served on the same GPU. Advanced techniques (AWQ, GPTQ) optimise the performance-quality trade-off. On Hopper GPUs (H100), native FP8 precision through the Transformer Engine improves throughput by 20 to 50% on transformer architectures. Systematically evaluating quantization on representative datasets frequently shows that INT8 delivers 95 to 98% of FP16 quality for a fraction of the cost. Not using quantization in production means wasting 40 to 50% of the GPU capacity you paid for.