Model Weight Quantisation

Quantisation is how you make a 109B model fit on hardware that should not be able to run it.

At a simple level, it means storing numbers with less precision so the model takes up less memory.

Instead of keeping everything in BF16, you move to formats like:

• FP8
• INT8
• INT4

The trade-off is tiny precision for a huge drop in size.

That matters because model sizes are growing faster than GPU memory.

So quantisation helps you:

• run models on fewer GPUs
• cut latency
• support longer conversations
• lower cloud costs

There are two main versions:

• 𝗪𝗲𝗶𝗴𝗵𝘁-𝗼𝗻𝗹𝘆 𝗾𝘂𝗮𝗻𝘁𝗶𝘀𝗮𝘁𝗶𝗼𝗻: smaller model, but compute stays mostly the same
• 𝗪𝗲𝗶𝗴𝗵𝘁 + 𝗮𝗰𝘁𝗶𝘃𝗮𝘁𝗶𝗼𝗻 𝗾𝘂𝗮𝗻𝘁𝗶𝘀𝗮𝘁𝗶𝗼𝗻: smaller model and faster matrix maths

The second one is where you get the biggest speed gains, because the model can use lower-precision Tensor Cores.

And the best part?

Good quantisation usually does 𝗻𝗼𝘁 destroy accuracy.

With calibrated methods like GPTQ, AWQ, and SmoothQuant, many models stay very close to their original performance even at 4-bit.

So the real lesson is this:

Quantisation is not just a compression trick.
It is a deployment strategy.

If you are running LLMs on limited hardware, it is one of the first levers worth pulling.