Issue:1506 Add the min, max to embedding 4bit. #1507
Conversation
pytorch#1506 Please refer it for more context information. Case: Failed to export the embedding 4bit model with the following command: python torchchat.py export stories110m --quantize torchchat/quant_config/mobile.json --output-pte-path stories110me4.pte Solution: Add the min, max to embedding 4bit. Test: The 4bit,8bit of embedding are exported successfully by the commands: Change torchchat/quant_config/mobile.json to: { "embedding": {"bitwidth": 8, "groupsize" : 32}, "linear:a8w4dq": {"groupsize" : 256} } python torchchat.py export stories110m --quantize torchchat/quant_config/mobile.json --output-pte-path stories110me8.pte { "embedding": {"bitwidth": 4, "groupsize" : 32}, "linear:a8w4dq": {"groupsize" : 256} } python torchchat.py export stories110m --quantize torchchat/quant_config/mobile.json --output-pte-path stories110me4.pte Signed-off-by: jijie <[email protected]>
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1507
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 3c1404c with merge base 4251a54 ( BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at [email protected]. Thanks! |
facebook-github-bot
added
the
CLA Signed
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
|
Amazing! Thanks for the find and fix |
#1506
Please refer it for more context information.
Case:
Failed to export the embedding 4bit model with the following command: python torchchat.py export stories110m --quantize torchchat/quant_config/mobile.json --output-pte-path stories110me4.pte
Solution:
Add the min, max to embedding 4bit.
Test:
The 4bit,8bit of embedding are exported successfully by the commands: Change torchchat/quant_config/mobile.json to:
{
"embedding": {"bitwidth": 8, "groupsize" : 32},
"linear:a8w4dq": {"groupsize" : 256}
}
python torchchat.py export stories110m --quantize torchchat/quant_config/mobile.json --output-pte-path stories110me8.pte
{
"embedding": {"bitwidth": 4, "groupsize" : 32},
"linear:a8w4dq": {"groupsize" : 256}
}
python torchchat.py export stories110m --quantize torchchat/quant_config/mobile.json --output-pte-path stories110me4.pte