• ModelQuantization: Concepts, Methods, and Why It Matters

Introduction

The goal of quantization a model is apparent, just for deploying alarge model in a resource-constrained environment which maintainingapproximate high performances. To this end, quantization infrastructuresare designed. For example, the commonly used data types includesBF16,FP16,FP8 and NVFP4 for floating numbers.They differ from the the number of bits to represent sign, exponent andmantissa (or significand).

• BF16: 1 bit for sign, 8 bits for exponent and 7 bitsfor mantissa. Share an approximate same value range withFP32 by sacrificing the precision. Number range-3.39e-38 to 3.39e38.
• FP16: 1 bit for sign, 5 bits for exponent and 10 bitsfor mantissa. More precise than BF16 by sacrificing therepresentation range. Number range -65504 to65504.
• FP8: Two types of FP8 are commonly used,one is E4M3 and the other is E5M2.
• NVFP4: Only 1 bit for mantissa and 2 bits for exponent,additional scaling factors are used for different blocks of tensors toextend the representation range.

In total, the representation range can be calculated via(normalized):

where stands for sign, is the exponent value, is the bias for exponent, is the binary of -th mantissa, is the total number of bits formantissa.

• Normalized/Standard number range: the mantissa will add , forming a value like . The minimum value of exponent is . For example, for BF16,the maximum value is . And the minimum value is .
• Denormalized number range: when the exponent is , the mantissa will add but the exponent keeps to be . Take BF16 as example, thevalues around zero are to force thevalues approach the zero slowly.
• Zero: When both and , the number is exactly .
• NaN/Inf: (inBF16) is a specitial value for NaN/Inf.

Quantization are tipically used in three scenes: the model weightquantization, the activation quantization and quantization on KV cachefor decoder-only transformers.

Quantization Algorithms

Quantization maps a float number to a low-precision value .

Affine/Asymmetric Quantization: Defined by scalefactor and the zero-point . Scale factor defines the step size ofthe quantizer. And the zero-point is the same type with , mapping the real value zeroexactly during quantization. The quantization process canwritten as:

where:

• round converts the scaled value to the nearestquantized representation.
• clip ensures the scaled value stay within the range ofquantized representation.

To recover the full-precision from a quantized value:

As we can see that round and clip bringerrors naturally, which are inherent to the quantization process.

Symmetric Quantization: Fixing the zero-point , symmetric quantization reducesadditional computation overhead by eliminating the addition operators.The process of quantization and de-quantization can be written as: > NOTE: The mostly used quantization algrithm is symmetricquantization as the affine quantization does not offer a significantboost on model performances. NVIDIA TensorRT and Model Optimizer usesymmetric quantization.

AbsMax algorithm: A widely used algorithm tocalculate scale factor isAbsMax due to its simplicity and effectiveness. Forsymmetric quantization, it can be written as:

For affine quantization:

Other Advanced algorithms: The main goal ofquantization algorithms is to find a balance between the computationaloverhead and the model performances, and some advanced algorithms aredesigned to reduce the errors brought by quantization process.

• Activation-aware WeightQuantization (AWQ)
• Generative Pre-trainedTransformer Quantization (GPTQ)
• SmoothQuant

Quantization Granularity

To define which elements of a tensor share the same quantizationparameters, the quantization process can be conducted on differentgranularities. That is, we use which elements of a tensor to get our and .

• Per-Tensor/Per-Layer: All values of a tensor sharethe same quantization parameters, which is the simplest and mostmemory-efficient approach, but leading to higher quantization errorsespecially when the data distribution varies across dimension.
• Per-Channel: Different set of quantizationparameters are shared inner the same channel. This method reducesquantization errors by isolating outlier values to their own dimensions,rather than affecting entire values.
• Per-Block/Per-Group: This method divides a tensorinto smaller blocks and each has its own set of quantization parameters,which is useful when different regions of the tensor have varyingdistributions.

Quantization Approaches

In most cases, quantizing the pre-trained model weights is straightforward as they are static values and input data independent. However,quantizing the activation values is more troublesome for theirdependency of input values and important ouliers, which willsignificantly affect the quantization scaling factor.

Post-Training Quantization (PTQ): During PTQ, we addobservers to activation we want to quantize and inference themodel with representative calibration dataset. The observers return theactivation values and we use the same quantization algrithms todetermine a scaling factor.

• Weight-only quantization: Given a pre-trained model, themodel weights are static and data independent. As the weights are fixed,no additional data is required. We just calculate the scaling factorwith/without the zero-point and map the weights to low-precisionvalues.
• Weight and Activation quantization: We use representativedata for both weight and activation quantization as the activationvalues are calculated runtime. The process of collecting statistics ofactivation values and determine suitable scale factor and zero-point iscalled calibration. During calibration, the model weights arefrozen, and the input representative data is used to calculatequantization parameters.
  • Static quantization: Input all calibration data at once andcalculate the scale factor and zero-point. And then, these parametersare static after deploying the model.
  • Dynamic quantization: No calibration data is required, calculate thescale factor and zero-point at runtime, which are vary for each inputdata.

Quantization Aware Training (QAT): Designed tooffset the accuracy degradation accompanied the quantization process, aquantization aware module is integrated into the model during trainingto conduct a fake-quantization. To be specific, the fake quantizationmodule mimics the quantized model behaviour with quantization followedby a de-quantization step to introduce the quantization errorsexplicitly during training. As the model parameters are optimizedconditioned on quantization errors, it remains high-performance evenafter quantization.

NOTE: The non-differentiability problem raises from QAT as thequantization function such as round is not differentiable.QAT uses straight throughestimator (STE) to approximate the gradients for these functions asidentity during backpropogation.

References

Step-by-Step Diffusion:An Elementary Tutorial

Overview of Diffusions

The goal of generative models is constructing asampler for an unknown distribution . As long as the sampler iscontructed, then we can generate new samples from thedistribution.

As the distribution is unknown, we should get a set of representativesamples from the unknown distribution to estimate it.

We can directly learn a transformation between the unknowndistribution and a simple-to-sample distribution such as Gaussiandistribution. However this may be untractable as the target unknowndistribution can be very large and complicated. Diffusion models providea general framework for learning such transformations, reducing theproblem of sampling from distribution to a sequence of easier samplingproblems.

Gaussian Diffusions

Let random variable , then construct a sequence of random variables by successively adding independent Gaussian noise withsome small scale : which is called the forward process. Let be themarginal distribution of each .We have . In practice, is sufficiently large and we can assumethat , where is the standard deviation. Whatwe want to do is to learn a transformation such that given themarginal distribution , we canproduce . And if the finalmarginal distribution is given,we can produce iteratively. This process is called reverse process, and themethod to implemente this process is called reversesampler.

The reverse process can be represented by conditional probability. Attime step , given the input sampled from , the output of the reverse samplergenerate a sample from the conditional distribution However, though this implies that a generative model shouldlearn the conditional distribution for every which could be complicated. But wehave the following insight, which will be proved in DDPM samplersection:

Fact 1 (Diffusion Reverse Process). For small and the Gaussian diffusion process, where , thenthe conditional distribution is itself close to Gaussian. For all time step and condition , there exists somemean parameter such that .

Given this fact, we find that if s are provided, then the onlything the model should learn is the mean of the distribution , which is noted as . Fortunately, learningthe mean is much simpler than learning as this can be seen as aregression problem: Given the joint distribution , from which we can easilyto sample, and the definition of the mean of the distribution : we have where the expectation is taken over sample from the target distribution .

Now, the problem of learning to sample from an arbitrary distributionis converted into optimizing the regression problem.

Abstract of Diffusions

We abstract the Gaussian settings of diffusions. Given a set ofsamples extracted from target distribution , and the easy-to-sample basedistribution (e.g. Gaussian ormultinomial), we try to construct a sequence of distributions which interpolatebetween and , such that and the adjacentdistribution aremarginally close. The aim is to learn a reverse sampler to transformdistribution to .

The reverse sampler at step is a potentially stochastic function such that if , then the marginaldistribution of is exactly, which means the reversesampler is used to transform the distribution to :

Dicretization

What is exactly mean about close between and ?

Assuming we have a time-evolving function and the constructed sequence , then thesequence is discretization of the time-evolving function such that and . If the sequence is sampleduniformly, then where controls thefineness of the discretization.

We want the terminal variance of the final distribution to be independent of , the incremental variance can bedefined as as and .

Following description of DDPM, DDIM and Flow-matching will use so the diffusion process canbe written as or

Now, let’s start the journey of DDPM, DDIM, and Flow-matching 😃.