Docs

What is a GGUF model?

GGUF is the file format that makes it practical to run large language models on your own GPU or CPU. This guide explains what the format is, why quantization matters, and how to pick a GGUF that fits your hardware.

GGUF in one sentence

GGUF (GPT-Generated Unified Format) is a single-file container for a quantized language model, its tokenizer, and the metadata needed to load it. It was created by Georgi Gerganov and the llama.cpp project to replace the older GGML format.

Why GGUF matters for local LLMs

A full-precision 70 B model can take 140 GB of VRAM. Most desktops do not have that. GGUF files solve the problem by compressing the weights through quantization — reducing precision from 16-bit floats to 8-bit, 4-bit, or even lower representations.

Because everything the runtime needs is in one file, GGUF models are easy to download, move between machines, and load without a complex Python environment.

What is quantization?

Quantization maps a model’s weights to a smaller set of numbers. Fewer bits per weight means a smaller file and lower memory use, but it can also reduce output quality. The trick is finding the right balance for your hardware.

Common GGUF quantization levels

  • Q4_0 / Q4_K_M — Smallest files, fast inference, good for GPUs with limited VRAM. A solid default for 7 B models on 8 GB cards.
  • Q5_K_M / Q6_K — Mid-range trade-off. Noticeably better quality than Q4 with a modest size increase.
  • Q8_0 / F16 — Largest and highest quality. Use when you have the VRAM and want output as close to the original model as possible.

How OpenLLM Studio helps

OpenLLM Studio scans your GPU, RAM, and CPU, then recommends a GGUF quantization level that fits your machine. It downloads the model from Hugging Face, caches it locally, and loads it with the right backend — CUDA, ROCm, Vulkan, or Metal — so you do not have to hand-pick quantizations or inference engines.

Where to find GGUF models

Hugging Face hosts thousands of GGUF checkpoints. Search for “GGUF” in a model’s files, or browse repositories like TheBloke for pre-quantized versions of popular open models.

Quick rules of thumb

  • 8 GB VRAM → 7 B Q4_K_M
  • 12–16 GB VRAM → 7 B Q5/Q6 or 13 B Q4
  • 24 GB VRAM → 13 B Q6 or 70 B Q4_K_M
  • CPU-only → smaller Q4 files load and run best

VRAM is not the only limit: context length, batch size, and your chosen backend also affect whether a model runs smoothly.

Download OpenLLM StudioBack to docs