A language model that answers one prompt on a development machine can behave very differently when a team starts using it throughout the working day. Responses queue, long documents consume more memory, and a server that looked sufficient during testing becomes slow at peak demand.
Good AI inference infrastructure starts with the application: the model it needs, the number of simultaneous requests, and the response time users will accept. Those requirements determine the GPU, memory, storage and network configuration worth paying for.
This guide focuses on serving text-based large language models. Image generation, multimodal applications and model training need their own measurements.
Start with a workload specification
Before comparing GPU server quotations, write down the following information. If a value is unknown, identify it as something your pilot must measure.
| Requirement | What to record | Why it matters |
|---|---|---|
| Model | Exact model, revision and licence | Establishes compatibility and deployment constraints |
| Precision | BF16, FP16 or a supported quantized format | Changes memory use and can affect quality |
| Input size | Typical and long prompts, including retrieved documents | Defines realistic request sizes |
| Output size | Typical response length and maximum generation limit | Helps explain request duration |
| Demand | Requests per minute, peak bursts and active requests | Distinguishes daily usage from peak capacity |
| User experience | Acceptable first-response and completion times | Defines whether a benchmark actually passes |
| Availability | Acceptable downtime and recovery time | Determines whether one server is sufficient |
For a document assistant, include the system prompt, chat history and retrieved passages when measuring input length. Testing with a short question alone can substantially understate the real workload.
Estimate GPU memory without confusing weights with total VRAM
An initial calculation is straightforward:
Raw weight memory = parameter count × bits per parameter ÷ 8.
The following values are arithmetic estimates in decimal gigabytes, assuming all parameters use the stated precision. They are not measured deployment requirements.
| Model parameter count | 16-bit weights | 8-bit weights | 4-bit weights |
|---|---|---|---|
| 8 billion | 16 GB | 8 GB | 4 GB |
| 14 billion | 28 GB | 14 GB | 7 GB |
| 32 billion | 64 GB | 32 GB | 16 GB |
| 70 billion | 140 GB | 70 GB | 35 GB |
Actual deployment needs additional memory for the KV cache, intermediate calculations and serving runtime. Quantized checkpoints also carry metadata and may retain some components at higher precision. A GPU with capacity close to the raw weight figure is therefore not a confirmed fit.
NVIDIA explains how model weights and the KV cache contribute to inference memory use in its LLM inference optimization guide.
Treat quantization as a configuration to validate. It can reduce the weight-memory footprint, but the model, GPU and inference engine must support the chosen format. Check answer quality on representative tasks and benchmark speed separately; smaller weights do not establish a universal speed improvement. Hugging Face’s quantization documentation provides implementation and compatibility details for bitsandbytes.
Account for context length and concurrent requests
During generation, many transformer models retain attention keys and values in a KV cache so earlier computations can be reused. Its size depends on the architecture, cache precision and amount of retained context across active sequences. Longer contexts and more simultaneous requests can therefore increase memory demand even though the model weights stay unchanged. See NVIDIA’s explanation of KV caching.
There is no dependable conversion from “registered users” to “required GPUs.” Fifty people with occasional questions create a different workload from fifty requests arriving together.
For your pilot, test several concurrency levels with the same realistic prompt mix. Record where response times exceed your target or requests fail. Set production limits below that failure point, then repeat the test with your longest allowed inputs. Choose headroom from the measured variability rather than an arbitrary percentage.
Measure response quality and speed together
A useful benchmark should answer two questions: does the model produce acceptable answers, and can the service deliver them quickly enough under load?
| Measurement | What it tells you |
|---|---|
| Time to first token | How long a user waits before a response starts |
| Inter-token latency | How smoothly the response streams after it starts |
| End-to-end latency | How long the complete answer takes |
| Aggregate output throughput | How many output tokens the deployment produces across requests |
| Successful requests within your latency target | How much work the service completes acceptably |
Report median and tail latency, such as p95, alongside throughput. A p95 of two seconds means roughly 95% of the measured observations were at or below two seconds. It is a description of that test, not a guarantee for every future request. Anyscale’s metrics guide explains these measures and their trade-offs.
Do not compare providers using headline tokens-per-second figures unless the model, precision, input lengths, output lengths and traffic conditions match. Ask whether the number describes one request or the combined output of many requests.
Keep the test reproducible: save the model revision, container version, serving configuration and benchmark inputs with the results. Separate cold-start tests from tests of a fully loaded, warmed-up service.
Decide between a single GPU, model splitting and replicas
Begin by testing a single GPU when the complete serving workload fits. If the model is too large, a supported parallel configuration can distribute it across GPUs. vLLM’s parallelism and scaling documentation describes single-GPU, multi-GPU and multi-node deployment options.
Splitting one model across multiple GPUs and running multiple independent copies solve different problems. Model splitting can help fit a larger model; replicas can distribute separate requests when each copy fits. Benchmark the option that addresses your actual limitation.
GPU memory does not automatically become one shared pool simply because several cards are installed. The serving engine, model support and interconnect topology matter. For distributed inference, request the exact GPU arrangement and test communication performance on the proposed server.
As a purchasing rule, make complexity earn its place: ask what a second GPU or second node improves in the measured workload before adding it to the deployment.
Size the rest of the server
Use the following questions when reviewing a quotation. These are planning checks, not universal minimum specifications.
CPU and system RAM: What else runs on the host? Include tokenization, request handling, model loading and any retrieval services. Test peak host-memory use during startup as well as steady operation. Do not assume spare GPU memory compensates for insufficient system RAM.
Storage: Budget for model versions, downloaded checkpoints, application data and logs. Measure startup and model-loading time from the actual storage configuration. Decide which data needs backup and how it would be restored onto a replacement host.
Networking: Separate public API traffic from storage access and inter-node communication. A high advertised internet speed does not describe GPU-to-GPU communication performance. Test each path the application depends on.
Power and cooling: Ask whether the quoted configuration can sustain its expected load and how throttling or hardware faults are handled. Include a sustained load test, not just a short successful request.
Recovery: Decide who restores the application after a host failure and what happens to requests during maintenance. Multiple GPUs inside one machine do not remove that machine as a failure point.
Compare hosting costs using completed work
A low hourly GPU price is only useful when the configuration meets your service target. Compare the complete deployment cost: compute, storage, data transfer, software, operational effort and any capacity reserved for availability.
One useful internal measure is:
Cost per successful request = total cost for a period ÷ successful requests completed in that period.
Keep the definition of success consistent. For a production assistant, it might require an error-free response that meets both the latency target and your quality checks.
For illustration only, assume two deployments each cost €900 per month, with all relevant costs already included. One completes 300,000 acceptable requests; the other completes 150,000. Their costs are €0.003 and €0.006 per successful request respectively. These figures are a calculation example, not HelloServer pricing or benchmark results.
Evaluate hourly capacity when demand is intermittent or uncertain. Evaluate a dedicated server when sustained usage and control over the environment are valuable. Compare both over the same expected month, including idle periods, peak demand and the operating work each option requires.
Prepare the service for production
Treat the inference engine as one component behind a controlled application entry point. Use authentication, encrypted connections and request limits. Keep management, monitoring and internal communication interfaces private.
For vLLM specifically, do not assume setting an API key protects every endpoint. Its security documentation recommends additional controls and restricting exposed interfaces. Review the guidance for the exact version you deploy.
Monitor application behaviour as well as hardware. vLLM exposes Prometheus-compatible metrics for observing request and engine behaviour through its metrics system. Pair those measurements with GPU memory, temperature, hardware error and host-health monitoring.
As an operational acceptance check, confirm that the team can identify a failed request, detect a growing queue and restore a known-working release. Define prompt-log retention and access before collecting production conversations.
A practical GPU-server acceptance test
Before committing to a configuration, run this sequence:
- Load the exact model and precision you intend to serve.
- Evaluate answers against a small, representative quality test set.
- Test normal inputs, long inputs and maximum permitted output lengths.
- Increase request arrival rate and concurrency while measuring latency and failures.
- Run sustained traffic and observe memory, thermal behaviour and queue growth.
- Restart the service and record the time until it can answer requests again.
- Calculate costs using the capacity that actually meets your targets.
Document the passing configuration and its limits. That gives the application team a defensible capacity plan and gives the hosting provider a concrete workload to support.
Frequently asked questions
How much VRAM does an LLM need?
Start with the raw weight calculation, then measure the complete deployment. Precision, model architecture, context length, concurrency and runtime overhead all affect whether a configuration fits. The table above estimates weights only.
Will an 80 GB GPU run every 70B model?
No. Seventy billion parameters at 16 bits require about 140 GB for weights alone. A compatible quantized version may fit on a smaller memory configuration, but its runtime overhead and serving workload still need testing.
Do I need a GPU cluster for a private chatbot?
Not necessarily. Test whether one GPU can meet the model, capacity and latency requirements first. Availability requirements may still justify additional servers even when one server can handle the normal workload.
Is a dedicated GPU server always cheaper than an API?
No. The comparison depends on traffic, model quality, utilisation and operating costs. Make the decision using measured demand and equivalent service requirements.
Plan your deployment with HelloServer
Bring a workload specification to the infrastructure discussion: model and precision, prompt lengths, expected traffic, response-time targets, storage needs and recovery requirements.
Explore HelloServer’s GPU dedicated server options and ask which currently available configuration matches that specification. For a broader introduction to hardware selection, read our practical guide to choosing a GPU for AI.
Choose the configuration that proves it can serve your application reliably at an acceptable cost—and keep the benchmark results so you know when it is time to scale.