Quantized Matrix Multiplication - theory and practice

Matrix multiplication is the main computational building block of large language models, and is often bottlenecked by moving data from memory. Quantization reduces this cost by representing matrices with fewer bits, but introduces errors that can degrade model accuracy. This talk explores how information theory can guide the design of accurate and efficient quantization schemes. We begin with the rate-distortion function for Gaussian matrix multiplication and a nested-lattice construction that asymptotically achieves this limit. We then discuss NestQuant, which brings these ideas to LLM quantization, and hierarchical lattice schemes that enable matrix multiplication using small lookup tables. Finally, we turn to activation statistics and hardware constraints: WaterSIC combines rate allocation with successive interference cancellation to approach information-theoretic limits for weight quantization, while OSFP4 jointly optimizes diagonal smoothing and block scales for the NVFP4 format. These results connect fundamental limits with practical methods for preserving LLM accuracy at low precision.