VRAM for AI Models
What VRAM Is
VRAM for AI models is the memory attached to a graphics card, where the model's weights and working data sit while the GPU does its calculations. VRAM stands for video random access memory. If the model and its working data do not fit into VRAM, the model cannot run on that GPU at full speed, or at all.
VRAM is separate from the main system memory (RAM) of the computer. Moving data between the two is much slower than working inside VRAM, which is why AI software tries to keep everything it needs on the GPU.
What Uses Up VRAM
Three things consume VRAM when you run a model:
- Model weights: the largest part. A model with 7 billion parameters stored at 16-bit precision needs about 14 GB for its weights alone.
- Context: the text the model reads and generates. Longer prompts and longer answers need a larger working cache, so memory use grows with context length.
- Overhead: the inference software, internal buffers and any memory used by the operating system's display.
Concurrent use also matters. Serving several users at once multiplies the working memory, even though the weights themselves are shared.
What Happens When a Model Does Not Fit
When a model is larger than the available VRAM, the outcome depends on the software. It may refuse to load, show an out-of-memory error, or spill part of the model into system RAM, which slows generation considerably. The operator may then choose a smaller model or a compressed version instead.
Common fixes are to use a smaller model, shorten the context, reduce precision through model quantization, or move to a GPU with more memory. Buying a larger GPU is not always the cheapest option, so compression is often a practical first step. For the bigger picture, see what you need to host an LLM.
Frequently asked questions
How Much VRAM Do I Need to Run an LLM?
It depends on the model size, its numeric precision and how much context you use. A common rule of thumb is about 2 GB per billion parameters for 16-bit weights, plus extra room for the working cache. Compressed versions need considerably less, so check the model's requirements and test with a typical prompt length.
Can System RAM Replace VRAM?
Partly. Some software can offload parts of a model to system RAM, so a model may run on a GPU with less VRAM. It will be noticeably slower, because data crosses a slower connection. For interactive use, keeping the model fully in VRAM is much better. System RAM adds flexibility, but it does not replace VRAM for speed.