Matt Kappes’ Post

"1-bit LLM variant in which every single parameter (or weight) of the LLM is ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance, while being significantly more cost-effective in terms of latency, memory, throughput, and energy consumption." Whitepaper: https://lnkd.in/dTAaqvjG Github: https://lnkd.in/dKFNV-DX Twitter: https://lnkd.in/djMjGYYF

The era of 1-bit LLMs. Key takeaways: - Every single parameter (or weight) of the LLM is ternary {-1, 0, 1} - Matches the full-precision Transformer LLM with the same model size - More cost-effective in terms of latency, memory, throughput, and energy consumption #artificialintelligence #largelanguagemodel

Extreme quantization in Q2 2024: BitNet b1.58 matches LLaMA's accuracy using 3x less memory and running 2.5x faster. Researchers converted every Float16 into a ternary value (-1, 0, or 1) requiring far less memory, while keeping the number of total model parameters constant

Neat-o https://lnkd.in/g8PMDjwR It shows that BitNet b1.58 starts to match full precision LLaMA LLM at 3B model size in terms of perplexity, while being 2.71 times faster and using 3.55 times less GPU memory. In particular, BitNet b1.58 with a 3.9B model size is 2.4 times faster, consumes 3.32 times less memory, but performs significantly better than LLaMA LLM 3B.

Very excited to share some of the research I've been doing on optimizing LLM inference using Block Floating Point 16 (BFP16), an effective 9-bit format and a viable alternative to FP8. Activation quantization is normally quite difficult, but we've shown that BFP16 is effective for quantizing both weights and activations without sacrificing model quality, providing a significant inference speed-up. Check out the blog post for more details! #llm #llama3 #quantization

A really nice result for LLama3 quantization using Block Float formats. Substantial improvements in compute performance with a simple numerical conversion.

LLM in a flash... Our method involves constructing an inference cost model that takes into account the characteristics of flash memory, guiding us to optimize in two critical areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks

Can we reduce the cost and latency of an LLM without impacting quality?  This study into BFP16 quantization for both weights and activations draws some very interesting conclusions: https://lnkd.in/dQwcPi_9 #machinelearning #llama3

I HAVE completed algorithm like searching (binary,linear, interpolation) sorting traversal (DFS,BFS, INORDER PRE-ORDER, POST ORDER)

Good presentation on transformers encoders vs decoders https://lnkd.in/eXf7q6s4