prompt generation very slow

[sipie800]

For 4096 token(which is forced by omost), use llama-3 model at 4090, it take 120s to complete prompt. And it take only 7s for SD. It's a big gap.
How can we accelerate the local GPT?

[zhaijunxiao]

The official LLM's method runs slow.
You can speed up by using TGI or llama.cpp.