Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve qwen vl impl #2943

Merged
merged 18 commits into from
Feb 4, 2025
Merged

Improve qwen vl impl #2943

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
77ef543
feat: refactor model, improve startup and re enable tests
drbh Jan 21, 2025
d12e075
fix: improve multimodal rotary embed caching
drbh Jan 22, 2025
a0ab962
fix: limit vision flop calc to qwen2 vl models and update config typing
drbh Jan 22, 2025
cf5c660
fix: include clippy lint
drbh Jan 22, 2025
7ab99bc
feat: refactor position ids in warmup and bump tests
drbh Jan 22, 2025
eef3c7b
fix: prefer default dtype
drbh Jan 23, 2025
5f416f6
fix: enable all cuda graphs and bump snapshots
drbh Jan 23, 2025
6893eb3
fix: adjust rotaty init path
drbh Jan 27, 2025
68e3ee8
fix: simplify get position ids and remove usused vision config
drbh Jan 28, 2025
c75c01e
fix: update position ids so first dim is batch, simplify rotary and b…
drbh Jan 28, 2025
79a2c95
fix: improve position id init during cuda warmup for mrope and simplf…
drbh Jan 28, 2025
d0e2332
fix: check existance before accessing rope type in cuda warmup
drbh Jan 28, 2025
585e270
fix: check key before access
drbh Jan 29, 2025
cb7ec9c
fix: improve mrope check in cuda graph warmup
drbh Jan 29, 2025
79550f8
fix: remove check for default rope type
drbh Jan 29, 2025
9eaa163
fix: add more test and improve model generation
drbh Jan 31, 2025
6cb0cb6
fix: improve and simplify get_cos_sin, refactors and cleanup get_pos…
drbh Feb 4, 2025
58f5f2e
fix: adjust signatures with types
drbh Feb 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
422 changes: 409 additions & 13 deletions ...ssed_tensors_w8a8_int_dynamic_weight/test_compressed_tensors_w8a8_int_dynamic_weight.json
View file
Open in desktop

Large diffs are not rendered by default.

50 changes: 25 additions & 25 deletions ...s_w8a8_int_dynamic_weight/test_compressed_tensors_w8a8_int_dynamic_weight_all_params.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,67 +7,67 @@
"seed": 0,
"tokens": [
{
"id": 1939,
"logprob": -2.2460938,
"id": 5267,
"logprob": -1.1464844,
"special": false,
"text": "?\n\n"
"text": "?\n"
},
{
"id": 33464,
"logprob": 0.0,
"logprob": -0.83203125,
"special": false,
"text": "Deep"
},
{
"id": 20909,
"logprob": -0.48608398,
"logprob": -0.5625,
"special": false,
"text": " Learning"
},
{
"id": 4102,
"logprob": -2.265625,
"id": 320,
"logprob": -2.1464844,
"special": false,
"text": " "
"text": " ("
},
{
"id": 285,
"id": 16524,
"logprob": 0.0,
"special": false,
"text": "is"
"text": "DL"
},
{
"id": 458,
"logprob": -0.6328125,
"id": 701,
"logprob": -2.2089844,
"special": false,
"text": " an"
"text": "),"
},
{
"id": 20443,
"logprob": -0.1796875,
"id": 476,
"logprob": -0.27368164,
"special": false,
"text": " artificial"
"text": " or"
},
{
"id": 11229,
"logprob": 0.0,
"id": 20443,
"logprob": -0.09442139,
"special": false,
"text": " intelligence"
"text": " artificial"
},
{
"id": 320,
"logprob": -0.37695312,
"id": 29728,
"logprob": 0.0,
"special": false,
"text": " ("
"text": " neural"
},
{
"id": 15469,
"id": 14155,
"logprob": 0.0,
"special": false,
"text": "AI"
"text": " networks"
}
],
"top_tokens": null
},
"generated_text": "What is deep learning?\n\nDeep Learning is an artificial intelligence (AI"
"generated_text": "What is deep learning?\nDeep Learning (DL), or artificial neural networks"
}
80 changes: 40 additions & 40 deletions ...tensors_w8a8_int_dynamic_weight/test_compressed_tensors_w8a8_int_dynamic_weight_load.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,61 +9,61 @@
"tokens": [
{
"id": 18183,
"logprob": -1.4912109,
"logprob": -1.5195312,
"special": false,
"text": " Deep"
},
{
"id": 6832,
"logprob": -0.075683594,
"logprob": -0.06817627,
"special": false,
"text": " learning"
},
{
"id": 374,
"logprob": -0.12408447,
"logprob": -0.13122559,
"special": false,
"text": " is"
},
{
"id": 264,
"logprob": -0.12768555,
"logprob": -0.13415527,
"special": false,
"text": " a"
},
{
"id": 25993,
"logprob": -0.82128906,
"logprob": -0.87353516,
"special": false,
"text": " subset"
},
{
"id": 315,
"logprob": -0.0012636185,
"logprob": -0.0011396408,
"special": false,
"text": " of"
},
{
"id": 5662,
"logprob": -0.12878418,
"logprob": -0.16442871,
"special": false,
"text": " machine"
},
{
"id": 6832,
"logprob": -0.0015888214,
"logprob": -0.0026416779,
"special": false,
"text": " learning"
},
{
"id": 429,
"logprob": -0.49194336,
"logprob": -0.48754883,
"special": false,
"text": " that"
},
{
"id": 5711,
"logprob": -1.2626953,
"logprob": -1.2294922,
"special": false,
"text": " uses"
}
Expand All @@ -82,61 +82,61 @@
"tokens": [
{
"id": 18183,
"logprob": -1.4912109,
"logprob": -1.5195312,
"special": false,
"text": " Deep"
},
{
"id": 6832,
"logprob": -0.075683594,
"logprob": -0.06817627,
"special": false,
"text": " learning"
},
{
"id": 374,
"logprob": -0.12408447,
"logprob": -0.13122559,
"special": false,
"text": " is"
},
{
"id": 264,
"logprob": -0.12768555,
"logprob": -0.13415527,
"special": false,
"text": " a"
},
{
"id": 25993,
"logprob": -0.82128906,
"logprob": -0.87353516,
"special": false,
"text": " subset"
},
{
"id": 315,
"logprob": -0.0012636185,
"logprob": -0.0011396408,
"special": false,
"text": " of"
},
{
"id": 5662,
"logprob": -0.12878418,
"logprob": -0.16442871,
"special": false,
"text": " machine"
},
{
"id": 6832,
"logprob": -0.0015888214,
"logprob": -0.0026416779,
"special": false,
"text": " learning"
},
{
"id": 429,
"logprob": -0.49194336,
"logprob": -0.48754883,
"special": false,
"text": " that"
},
{
"id": 5711,
"logprob": -1.2626953,
"logprob": -1.2294922,
"special": false,
"text": " uses"
}
Expand All @@ -155,61 +155,61 @@
"tokens": [
{
"id": 18183,
"logprob": -1.4912109,
"logprob": -1.5195312,
"special": false,
"text": " Deep"
},
{
"id": 6832,
"logprob": -0.075683594,
"logprob": -0.06817627,
"special": false,
"text": " learning"
},
{
"id": 374,
"logprob": -0.12408447,
"logprob": -0.13122559,
"special": false,
"text": " is"
},
{
"id": 264,
"logprob": -0.12768555,
"logprob": -0.13415527,
"special": false,
"text": " a"
},
{
"id": 25993,
"logprob": -0.82128906,
"logprob": -0.87353516,
"special": false,
"text": " subset"
},
{
"id": 315,
"logprob": -0.0012636185,
"logprob": -0.0011396408,
"special": false,
"text": " of"
},
{
"id": 5662,
"logprob": -0.12878418,
"logprob": -0.16442871,
"special": false,
"text": " machine"
},
{
"id": 6832,
"logprob": -0.0015888214,
"logprob": -0.0026416779,
"special": false,
"text": " learning"
},
{
"id": 429,
"logprob": -0.49194336,
"logprob": -0.48754883,
"special": false,
"text": " that"
},
{
"id": 5711,
"logprob": -1.2626953,
"logprob": -1.2294922,
"special": false,
"text": " uses"
}
Expand All @@ -228,61 +228,61 @@
"tokens": [
{
"id": 18183,
"logprob": -1.4912109,
"logprob": -1.5195312,
"special": false,
"text": " Deep"
},
{
"id": 6832,
"logprob": -0.075683594,
"logprob": -0.06817627,
"special": false,
"text": " learning"
},
{
"id": 374,
"logprob": -0.12408447,
"logprob": -0.13122559,
"special": false,
"text": " is"
},
{
"id": 264,
"logprob": -0.12768555,
"logprob": -0.13415527,
"special": false,
"text": " a"
},
{
"id": 25993,
"logprob": -0.82128906,
"logprob": -0.87353516,
"special": false,
"text": " subset"
},
{
"id": 315,
"logprob": -0.0012636185,
"logprob": -0.0011396408,
"special": false,
"text": " of"
},
{
"id": 5662,
"logprob": -0.12878418,
"logprob": -0.16442871,
"special": false,
"text": " machine"
},
{
"id": 6832,
"logprob": -0.0015888214,
"logprob": -0.0026416779,
"special": false,
"text": " learning"
},
{
"id": 429,
"logprob": -0.49194336,
"logprob": -0.48754883,
"special": false,
"text": " that"
},
{
"id": 5711,
"logprob": -1.2626953,
"logprob": -1.2294922,
"special": false,
"text": " uses"
}
Expand Down
Loading
Loading