Скачать или смотреть Quantization in LLM to Trinary State

We take a 16-bit floating point number, convert it to a one-bit integer by rounding to the nearest whole number. This makes computation faster, requires less hardware, uses less energy, and speeds up the process. This process is called quantization, which is basically rounding to the nearest data point.

When they say 1.58 bits, it means using three states: negative one, zero, and one. Using just one bit gives us only zero and one. With two bits, we can include a sign, allowing a negative number, but we ignore the fourth state.

So, in hardware, we use two bits, but in software, the algorithm uses three states. This lets us convert floating-point numbers into a matrix of, 1, 0, and 1, making it easier to add matrices instead of multiplying them. This keeps the quality of the output close to the full precision model while reducing cost, latency, speed, and energy.

It works, but they mention needing new hardware to better handle the trinary state, which does sound like a bit of a cheat, though it makes sense. The math is still correct, and the benefits are real as we get a net gain by using this method.