Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How much RAM needed to convert gpt2 13b model to ggml using your Manual convert function? #33

Open
JohnClaw opened this issue Sep 12, 2023 · 1 comment
Open

How much RAM needed to convert gpt2 13b model to ggml using your Manual convert function? #33

JohnClaw opened this issue Sep 12, 2023 · 1 comment

Comments

@JohnClaw
Copy link

I'm trying to convert it on 16gb RAM but converting process seems to last forever.
@LLukas22
Copy link
Owner

Well you can calculate it via: 13b times 16 Bit (f16) = 26 GB.
Accelerate will probably try to page some of the layers, if you exceed your 16 GB and get stuck there. Theoretically it's possible to stream the layers in, but i think neither GGML or this project has implemented that yet for GPT 2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@LLukas22 @JohnClaw