DeepSeek-R1 support

[loretoparisi]

Add support or a recipe to quantize DeepSeek-R1 and related distilled versions DeepSeek-R1-Distill-Llama-70B, DeepSeek-R1-Distill-Qwen-32B, eventually including 7B, 14B Qwen, and Llama 8B

[YangWang92]

We are almost there, and I have prepared the hessian collection codes for Deepseek R1/V3. Please wait for us for one week.

[wejoncy]

Here is a way to quantize those related distilled versions

1. install vptq-algorithm by following https://github.com/microsoft/VPTQ/blob/algorithm/algorithm.md#environment-setting
2. install qllm by pip install qllm
3. create a example quant-config.json python -m qllm --model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B --quant_method=vptq --quant_config=help
4. copy and save the config, edit those values with your thought
5. do quatization python -m qllm --model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B --quant_method=vptq --quant_config=xx.json --save DeepSeek-R1-Distill-Qwen-32B-vptq

[YangWang92]

Some updates: we have already collected all hessian matrices and are trying to adapt the quantization algorithm.

[YangWang92]

Some updates: I have adapted the algorithm to deepseek models and stay tuned.

[YangWang92]

Now, we have an early ~2 bits version of Deepseek R1, and it works well on 4xA100 80G.

[loretoparisi]

Now, we have an early ~2 bits version of Deepseek R1, and it works well on 4xA100 80G.

💯 This is truly amazing.
Running a test on our 4 x H100 today!

[YangWang92]

Now, we have an early ~2 bits version of Deepseek R1, and it works well on 4xA100 80G.

💯 This is truly amazing. Running a test on our 4 x H100 today!

The current inference speed is still a bit slow (based on deepseek repo's bare torch design). Please wait a moment while I prepare it—you can give it a try soon.