SmoothQuant: Weight-Only vs W8A8

Large Language Models(LLMs) are the pinnacle of technology in computer science, the architecture behind these models is capable of translating, conversing and writing code on a level comparable to humans in some cases. But as with all things, there is a catch, they require massive amounts of compute power to do so. Taking GPT-3 as an example, it is a 175 billion parameter model, in simple words that means it takes 175 billion multiplication operations and 175 billion addition operations to process each token. So if there's a sentence with say 10 tokens, it would take a total of 3.5 trillion calculations just to process the input. To do all these calculations with a 16 bit floating point number(What we call FP16) makes it even more challenging.

This is where quantization comes in, the concept is to convert these higher precision numbers into a lower precision so the calculations become faster and less computationally expensive . By compressing high precision numbers such as FP16 or FP32 down to INT8 for example, the latency and memory requirements to run the model can be decreased significantly.

Not just compute, a model like GPT-3 having 175 billion parameters is also a nightmare to fit inside the RAM of servers. The memory required to store model weights depends on the number of parameters and the bit precision of those parameters.

Memory Footprint (Bytes) = (Number of Parameters × Bits per Parameter) / 8 Bits per Byte

• FP32 (32-bit): (175 × 10⁹ × 32) / 8 = 700 Billion Bytes ≈ 700 GB
• FP16 (16-bit): (175 × 10⁹ × 16) / 8 = 350 Billion Bytes ≈ 350 GB
• INT8 (8-bit): (175 × 10⁹ × 8) / 8 = 175 Billion Bytes ≈ 175 GB

Serving a 350GB model in production means holding 350+ GB in the memory, plus KV caches. This requires multiple enterprise GPUs across multiple server nodes, and serving to a large number of users means horizontal scaling of this entire setup. Downscaling to INT8 slashes the footprint in half, allowing the model to fit inside smaller GPUs or GPU nodes.

Understanding Weights and Activations

To understand why quantizing a large language model is not as simple as discarding the extra decimal digits , we first need to distinguish between the two components of a model: weights and activations.

• Weights: Weights are the core parameters of the model,representing the "knowledge" it learned during the training process. They are entirely static, once training is complete, the weight matrix (W) is fixed. Because they do not change, their numerical values are highly predictable, typically falling into a smooth, symmetric, bell-shaped distribution like the gaussian distribution.
• Activations (The Computations): Activations is the collective term for the input embedding vector of the first layer and the output vector of each layer. Since each layer involves matrix multiplication Y = X.W + B, multiplication of 8-bit numbers can result in overflow, so the results need to be stored in a higher precision number like FP16 or FP32 and then re-quantized back to INT8. Since the weights are frozen after training, the activations depend only on the input vector embeddings, hence they are susceptible to fluctuations.

Anisotropic Activation Geometry

Why can't weights and activations be treated the same? Because activations develop something called Anisotropic distribution, their vector space clusters into a narrow directional cone with extreme channel-wise outliers.

Step Size (Delta) and Representation Collapse

Quantization maps continuous floats into integer buckets using a step size, Delta (Δ):

When no outliers are present and max(|X|) = 1.0, the step size is fine: Δ = 1.0 / 127 ≈ 0.00787. A float like 0.27 maps to round(0.27 / 0.00787) = 34, retaining high resolution. But, if a single outlier channel reaches 10.0, the required step size goes up  to Δ = 10.0 / 127 ≈ 0.0787. Dividing normal values by this coarse step size results in 0.25, rounding straight to zero. This phenomenon, known as representation collapse, affects the model’s accuracy.

The SmoothQuant Equivalence Transformation

SmoothQuant solves this problem through a mathematically equivalent transformation. Because weights and activations are multiplied together in linear layers (Y = X · W), we can insert a diagonal scaling matrix S between them:

• X̂ = X · S⁻¹ : Divides activation channel j by scaling factor sⱼ (flattening outlier spikes)
• Ŵ = S · W : Multiplies weight matrix row j by sⱼ (absorbed offline into static weights)
• Equivalence: Since S⁻¹ · S = I (Identity), the mathematical output Y remains completely invariant.

Consider an activation vector across six channels: [0.3, 0.5, 0.4, 8.7, 0.6, 0.5]. Channel 4 has an outlier spike of 8.7. Setting s₄ = 7.25 scales channel 4 down to 1.2, smoothing the vector to approximately [0.7, 0.8, 0.6, 1.2, 0.9, 0.8]. Row 4 of the weight matrix absorbs the 7.25 multiplier offline. Because weights follow smooth Gaussian distributions, they absorb this scaling factor easily without precision loss.

Migration Strength (α)

SmoothQuant introduces a hyperparameter α ∈ [0, 1], to control sharing of quantizing difficulty between the weights and the activations.

• max(|Xⱼ|): The tallest spike in the activations
• max(|Wⱼ|): The tallest spike in the weights

• α = 1: Sets sⱼ = max(|Xⱼ|). This transfers the anisotropic spikes in the activations completely to the weights through the diagonal matrix making the activations easy to quantize but the weights are left with spikes that are hard to quantize.
• α = 0: Sets sⱼ = 1 / max(|Wⱼ|). Doesn’t transfer anything to the weights, meaning that the activations retain their anisotropic spikes and remain hard to quantize.
• α = 0.5 (Balance): Splits difficulty evenly, allowing both tensors to quantize cleanly to INT8.

Integration with Transformers

Dividing activations by S at runtime does not introduce extra latency. In Transformer blocks, every linear projection (Query, Key, Value, Output, and MLP Feed-Forward) is preceded by LayerNorm or a preceding linear layer.

• Offline Weight Scaling: Weight rows are multiplied by S offline (Ŵ = S · W) and stored as INT8 parameters.
• Combining Scale and LayerNorm: The scaling matrix S⁻¹ is folded directly into the scale (γ) and bias (β) parameters of the preceding LayerNorm. Activations emerge already smoothed.
• Pure INT8 GEMM: Matrix multiplications execute directly on high-speed hardware INT8 Tensor Cores without mixed-precision branches.

Hardware Efficiency and Production Impact

SmoothQuant delivers major operational improvements for large-scale model serving:

• Memory requirement reduction: cuts the amount of VRAM required in half, so a 350GB FP16 model will be able to fit in 175GBs of VRAM
• Speeding up the Inference time: Accelerates the inference time, since less computation is required, hence inference is faster.
• Downscaling Reasoning: A quantized model has similar reasoning capabilities as the original model, the loss in reasoning capabilities is not significant.
• Local Feasibility: Allows running better more capable models locally on your laptops, reducing dependence on cloud hosted APIs.

The W8A8 Advantage: 

W8A8 stands for Weight 8-bit, Activation 8-bit, this is the differentiating feature of SmoothQuant, unlike other quantization methods like ZeroQuant which have W8A16 format of quantization, SmoothQuant quantizes the Activations along with the Weights down to INT8. Since multiplication of vectors of different formats is not possible, methods that use the W8A16 format have to dequantize the weights at runtime, in order to multiply them to the activations in each layer. SmoothQuant smoothly avoids this problem by using W8A8 format and hence it’s more memory and compute efficient than other quantization methods.