Enable quant model support #1074
Conversation
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
|
Hi @sywangyi . Please review this PR, thanks. |
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
|
Hi @IlyasMoutawwakil @echarlaix . Do you have time to review this PR? Thanks! |
| convert_functions(model, GPT2Block, "forward", _gpt2_block_forward) | ||
| convert_class(model, GPT2Attention, _IPEXGPT2Attention, model.config) | ||
| convert_class(model, GPT2MLP, _IPEXGPT2MLP, model.config) | ||
| convert_class(model, GPT2Block, _IPEXGPT2Block, model.device, model.config) |
There was a problem hiding this comment.
why no longer patching the mlp and attention here ?
There was a problem hiding this comment.
Because they are in the _IPEXGPT2Block
| - name: Install bitsandbytes | ||
| run: | | ||
| git clone --branch multi-backend-refactor https://github.com/bitsandbytes-foundation/bitsandbytes.git | ||
| cd bitsandbytes | ||
| pip install . |
There was a problem hiding this comment.
no luck with autoawq ?
There was a problem hiding this comment.
Yes, the autoawq installation have some issues, I will figure out why this env cannot install autoawq. But the tests passed in my local env which have the autoawq installed.
I guess it's because of the python version. I updated python from 3.9 to 3.10, but git the error in CI: Do you know why python 3.10 was detected as 3.1 ? |
that's just github parsing it as a number (3.10=3.1), you need to add "3.10" |
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
|
Hi @IlyasMoutawwakil . I have removed autoawq and only support bitsandbytes in this PR. The autoawq will be supported once we fixed the installation issue. |
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
Signed-off-by: jiqing-feng <[email protected]>
This PR could enable BitsandBytes model's support. Even we cannot use fused linear in a quant model, there is still a 5-10% speed-up on llama2-7b when batch_size=1, the speed-ratio will increase by batch size.