Aibytec

logo
AI BY TEC

How to Run a 7B LLM for Free on Google Colab (4-Bit Quantization, Step-by-Step)

Leave a Comment / Blogs / By
.aib-post{--aib-ink:#0f172a;--aib-muted:#475569;--aib-accent:#2563eb;--aib-accent-2:#0d9488;--aib-soft:#f1f5f9;--aib-line:#e2e8f0;--aib-code-bg:#0b1220;--aib-code-ink:#e2e8f0; color:var(--aib-ink);font-size:17px;line-height:1.75;max-width:820px;margin:0 auto;background:#fff;word-wrap:break-word} .aib-post *{box-sizing:border-box} .aib-post h1{font-size:clamp(28px,4.2vw,40px);line-height:1.2;margin:0 0 14px;font-weight:800;letter-spacing:-.02em} .aib-post h2{font-size:clamp(22px,3vw,28px);line-height:1.3;margin:48px 0 14px;font-weight:750;padding-top:8px;border-top:1px solid var(--aib-line)} .aib-post h3{font-size:20px;margin:30px 0 10px;font-weight:700} .aib-post p{margin:0 0 16px} .aib-post a{color:var(--aib-accent);text-decoration:underline;text-underline-offset:3px} .aib-post ul,.aib-post ol{margin:0 0 18px;padding-left:24px} .aib-post li{margin-bottom:6px} .aib-post .aib-meta{color:var(--aib-muted);font-size:14px;margin-bottom:22px} .aib-post .aib-lead{font-size:19px;color:#1e293b} .aib-post .aib-box{border-radius:12px;padding:18px 20px;margin:24px 0;border:1px solid var(--aib-line);background:var(--aib-soft)} .aib-post .aib-box strong:first-child{display:block;margin-bottom:6px} .aib-post .aib-answer{background:#eff6ff;border-color:#bfdbfe} .aib-post .aib-tip{background:#f0fdfa;border-color:#99f6e4} .aib-post .aib-warn{background:#fffbeb;border-color:#fde68a} .aib-post .aib-toc ol{margin:0;padding-left:20px} .aib-post .aib-toc a{text-decoration:none} .aib-post .aib-table-wrap{overflow-x:auto;margin:20px 0} .aib-post table{width:100%;border-collapse:collapse;font-size:15.5px;min-width:480px} .aib-post th,.aib-post td{border:1px solid var(--aib-line);padding:10px 12px;text-align:left;vertical-align:top} .aib-post th{background:var(--aib-soft);font-weight:700} .aib-post code{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-size:.9em;background:var(--aib-soft);padding:2px 6px;border-radius:5px} .aib-post .aib-code{position:relative;margin:18px 0 24px} .aib-post .aib-code pre{background:var(--aib-code-bg);color:var(--aib-code-ink);padding:18px 18px 16px;border-radius:12px;overflow-x:auto;font-size:14px;line-height:1.6;margin:0} .aib-post .aib-code pre code{background:none;padding:0;color:inherit;font-size:inherit} .aib-post .aib-code .aib-label{position:absolute;top:0;left:16px;transform:translateY(-50%);background:var(--aib-accent-2);color:#fff;font-size:12px;font-weight:700;padding:2px 10px;border-radius:999px} .aib-post .aib-copy{position:absolute;top:10px;right:10px;background:#1e293b;color:#e2e8f0;border:1px solid #334155;border-radius:7px;font-size:12.5px;padding:5px 10px;cursor:pointer} .aib-post .aib-copy:hover{background:#334155} .aib-post .aib-step{display:inline-block;background:var(--aib-accent);color:#fff;font-weight:700;font-size:13px;border-radius:999px;padding:2px 10px;margin-right:8px;vertical-align:middle} .aib-post .aib-faq details{border:1px solid var(--aib-line);border-radius:10px;padding:12px 16px;margin-bottom:10px;background:#fff} .aib-post .aib-faq summary{font-weight:700;cursor:pointer} .aib-post .aib-faq details p{margin:10px 0 0} .aib-post .aib-cta{background:linear-gradient(135deg,#1d4ed8,#0d9488);color:#fff;border-radius:16px;padding:28px 24px;margin:44px 0 10px;text-align:center} .aib-post .aib-cta h2{border:0;margin:0 0 8px;padding:0;color:#fff} .aib-post .aib-cta p{margin:0 0 18px;color:#e0f2fe} .aib-post .aib-cta a{display:inline-block;background:#fff;color:#1d4ed8;font-weight:800;text-decoration:none;padding:12px 22px;border-radius:10px} @media (max-width:600px){.aib-post{font-size:16px}.aib-post .aib-code pre{font-size:13px}}

How to Run a 7B LLM for Free on Google Colab (4-Bit Quantization, Step-by-Step)

By Muhammad Rustam · AIBYTEC · Python, Hugging Face & Google Colab

Want to run a real 7-billion-parameter language model without paying for a GPU? In this hands-on guide you'll load Qwen2.5-7B-Instruct on Google Colab's free T4 GPU using 4-bit quantization. You'll also learn how temperature, top_p and top_k control what the model writes.

Quick answer A 7B model in FP16 needs about 15 GB of VRAM just for its weights, which is too much for Colab's free T4 GPU. Loading it in 4-bit NF4 with bitsandbytes cuts that to roughly 5–5.5 GB, so it fits comfortably and runs at no cost.
What's in this guide
  1. Why a 7B model doesn't fit on a free GPU
  2. What 4-bit quantization (NF4) actually does
  3. Step-by-step: run the model on Colab
  4. Temperature, Top-P and Top-K explained
  5. Common errors and how to fix them
  6. FAQ

Why a 7B Model Doesn't Fit on a Free GPU

A model's memory footprint comes from a simple formula:

VRAM (weights) ≈ number of parameters × bytes per parameter Qwen2.5-7B actually has about 7.6 billion parameters. A Colab T4 has 16 GB of memory, with about 15 GB usable.
PrecisionBytes per parameterWeights onlyFits on a free T4?
FP32430.4 GB❌ No
FP16 / BF16215.2 GB❌ No
INT817.6 GB✅ Yes
4-bit (NF4)0.53.8 GB✅ Yes, with lots of room

Remember that these numbers cover weights only. The KV cache (the model's short-term memory of your conversation), activations and CUDA overhead all add to the total. That's why FP16 crashes on a T4 even though 15.2 GB looks close to 16 GB.

What Is 4-Bit Quantization (NF4)?

Quantization compresses a model's weights from 16-bit decimals down to small integers, much like compressing a high-resolution photo. The basic idea uses a scale (S) and a zero-point (Z):

Formula
X_quantized = round(X / S) + Z
X_restored  = (X_quantized - Z) × S

NF4 (NormalFloat 4-bit), the format used by bitsandbytes, improves on this in two ways:

  • It splits the weights into small blocks and scales each block separately, so one extreme value can't ruin the precision of everything else.
  • It places its 16 levels where trained weights actually cluster, following a bell-shaped (normal) distribution. That means less rounding error than evenly spaced levels.
The trade-off 4-bit loading uses about 4× less memory than FP16. In return you get a small drop in quality on hard reasoning tasks. For learning, demos and most chatbot work, that's an excellent deal. In practice, expect around 5–5.5 GB rather than 3.8 GB, because the large embedding and output layers stay in higher precision.

Step-by-Step: Run a 7B LLM on Google Colab

Step 1Switch to the free T4 GPU

In Colab, open Runtime → Change runtime type, choose T4 GPU, and click Save. 4-bit loading needs a CUDA GPU, so it won't work on the CPU runtime.

Step 2Install the libraries

Colab cell
# bitsandbytes = 4-bit quantization, accelerate = automatic GPU placement
!pip install -q -U transformers accelerate bitsandbytes

Step 3Load the model in 4-bit

Python
import torch
from transformers import pipeline, BitsAndBytesConfig

if not torch.cuda.is_available():
    raise RuntimeError("No GPU found. Runtime → Change runtime type → T4 GPU, then re-run.")

model_id = "Qwen/Qwen2.5-7B-Instruct"

# 4-bit NF4 quantization settings
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",             # NormalFloat 4-bit
    bnb_4bit_compute_dtype=torch.float16,  # T4 GPUs don't support bfloat16
    bnb_4bit_use_double_quant=True,        # also compresses the scale factors
)

llm = pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"quantization_config": bnb_config},
    device_map="auto",                     # place layers on the GPU automatically
)

print(f"Model memory: {llm.model.get_memory_footprint() / 1e9:.2f} GB")
Heads-up The first run downloads about 15 GB of model files, which takes a few minutes. Later runs in the same session start much faster.

Step 4Write a reusable generation function

Python
def ask_llm(system_prompt: str, user_prompt: str,
            temperature: float = 0.7, max_new_tokens: int = 500) -> str:
    """Send a system + user message and return the model's reply."""
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_prompt},
    ]
    output = llm(
        messages,
        max_new_tokens=max_new_tokens,  # maximum reply length
        temperature=temperature,        # creativity control (see next section)
        top_p=0.90,                     # nucleus sampling cutoff
        do_sample=True,                 # sample instead of always picking the top token
    )
    # The pipeline returns the whole chat; the last message is the reply
    return output[0]["generated_text"][-1]["content"]

Step 5Try it: factual vs creative

Python
# Low temperature → focused, factual answer
print(ask_llm(
    "You are an AI teacher. Give clear, accurate answers.",
    "Explain the difference between self-attention and cross-attention.",
    temperature=0.2,
))

# High temperature → creative, different every run
print(ask_llm(
    "You are an imaginative storyteller.",
    "Write a 5-sentence story about a robot who learns to cook biryani in Karachi.",
    temperature=1.0,
))

Run the creative prompt twice and you'll get two different stories. Run the factual one twice and the answers will be nearly the same. The next section explains why.

Want a newer or different model? Change model_id to another 7–8B instruct model on Hugging Face. Gated models such as Llama also require you to accept the license and log in with a Hugging Face token first.

Temperature, Top-P and Top-K Explained

At every step, the model scores each word in its vocabulary. These raw scores are called logits. A function called softmax turns them into probabilities, and the decoding settings decide how a word is picked from those probabilities.

Temperature

  • Low temperature (0.1–0.3) sharpens the probabilities so the top word almost always wins. Use it for code, maths, facts and JSON.
  • High temperature (1.0 and above) flattens the probabilities so less likely words get a real chance. Use it for stories, marketing ideas and brainstorming.

Top-K vs Top-P (nucleus sampling)

  • Top-K always keeps exactly the K most likely words (for example K = 50), however confident the model is.
  • Top-P keeps the smallest group of words whose combined probability reaches P (for example 0.9). Its advantage is that it adapts: when the model is confident it keeps just one or two words, and when the model is unsure it keeps more options.
Use caseTemperatureTop-P
Code, JSON, data extraction0.1–0.30.9
Factual Q&A, RAG chatbots0.3–0.50.9
General chat assistants0.70.9
Stories, ads, brainstorming0.9–1.20.95
Myth buster Temperature does not change generation speed. Each new token costs one forward pass through the model, whatever the temperature. Temperature only changes which token gets picked.

Common Errors and How to Fix Them

Error or symptomFix
"No GPU found" or a bitsandbytes CUDA errorSwitch the runtime to T4 GPU and run all cells again.
CUDA out of memoryMake sure you loaded in 4-bit, lower max_new_tokens, and restart the runtime to clear old models from memory.
Error about passing load_in_4bit directlyThat style is deprecated. Use BitsAndBytesConfig as shown in Step 3.
bfloat16 errors or very slow runs on a T4T4 GPUs don't support bfloat16. Set bnb_4bit_compute_dtype=torch.float16.
Error when setting temperature=0Sampling needs a temperature above 0. For fully deterministic output, use do_sample=False instead.

Know the Limits

  • Hallucinations: LLMs can state false facts with full confidence. For anything factual, ground the model with RAG (retrieval-augmented generation) and validate its output.
  • Lost in the middle: even with long context windows, models recall information buried in the middle of a long prompt less reliably. Put key instructions at the start or end.

Frequently Asked Questions

Can I run a 7B LLM on Google Colab for free?

Yes. Load the model in 4-bit with bitsandbytes on Colab's free T4 GPU. A 7B model then needs roughly 5–5.5 GB of VRAM instead of about 15 GB.

How much VRAM does a 7B model need?

For the weights alone: about 30 GB in FP32, 15 GB in FP16, 7.6 GB in 8-bit and 3.8 GB in 4-bit (for a 7.6B-parameter model). Add extra headroom for the KV cache and runtime overhead.

Does 4-bit quantization reduce quality?

Slightly, mainly on complex multi-step reasoning. For chatbots, summaries, learning and demos, the difference is usually small, and the memory savings are huge.

What temperature should I use for an LLM?

Use 0.1–0.3 for code, JSON and facts, around 0.7 for general chat, and 0.9–1.2 for creative writing. Keep top_p around 0.9.

What is the difference between top-k and top-p?

Top-k always keeps a fixed number of candidate words. Top-p keeps however many words are needed to reach a probability threshold, so it narrows down when the model is confident and widens when it's unsure.

Build Real AI Apps With AIBYTEC

Learn LLMs, RAG chatbots and AI agents hands-on, with runnable code and real projects, at AIBYTEC in Karachi.

Explore AIBYTEC Courses
{ "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ {"@type": "Question", "name": "Can I run a 7B LLM on Google Colab for free?", "acceptedAnswer": {"@type": "Answer", "text": "Yes. Load the model in 4-bit with bitsandbytes on Colab's free T4 GPU. A 7B model then needs roughly 5–5.5 GB of VRAM instead of about 15 GB."}}, {"@type": "Question", "name": "How much VRAM does a 7B model need?", "acceptedAnswer": {"@type": "Answer", "text": "For the weights alone: about 30 GB in FP32, 15 GB in FP16, 7.6 GB in 8-bit and 3.8 GB in 4-bit (for a 7.6B-parameter model). Add extra headroom for the KV cache and runtime overhead."}}, {"@type": "Question", "name": "Does 4-bit quantization reduce quality?", "acceptedAnswer": {"@type": "Answer", "text": "Slightly, mainly on complex multi-step reasoning. For chatbots, summaries, learning and demos, the difference is usually small, and the memory savings are huge."}}, {"@type": "Question", "name": "What temperature should I use for an LLM?", "acceptedAnswer": {"@type": "Answer", "text": "Use 0.1–0.3 for code, JSON and facts, around 0.7 for general chat, and 0.9–1.2 for creative writing. Keep top_p around 0.9."}}, {"@type": "Question", "name": "What is the difference between top-k and top-p?", "acceptedAnswer": {"@type": "Answer", "text": "Top-k always keeps a fixed number of candidate words. Top-p keeps however many words are needed to reach a probability threshold, so it narrows down when the model is confident and widens when it's unsure."}} ] } (function () { document.querySelectorAll(".aib-post .aib-copy").forEach(function (btn) { btn.addEventListener("click", function () { var code = btn.parentElement.querySelector("code").innerText; var done = function () { btn.textContent = "Copied!"; setTimeout(function () { btn.textContent = "Copy"; }, 1500); }; if (navigator.clipboard && window.isSecureContext) { navigator.clipboard.writeText(code).then(done).catch(function () { btn.textContent = "Select & copy"; }); } else { // Fallback for non-HTTPS pages var ta = document.createElement("textarea"); ta.value = code; document.body.appendChild(ta); ta.select(); try { document.execCommand("copy"); done(); } catch (e) { btn.textContent = "Select & copy"; } document.body.removeChild(ta); } }); }); })();

Leave a Comment

Your email address will not be published. Required fields are marked *

Verified by MonsterInsights