Just getting repeating chars as response from the model

[chriszero]

Hi,
I've running a docker container of text-generation-webui on a Nvidia Jetson Orion NX 8GB.
I've installed the HACS addon and configured the addon to connect to the the webui running on the Jetson device.
Using the "Home-3B-v2.q4_k_m.gguf" model
But all promts replies are just a bunch of the same chars. eg "3333333333333333333333..." or "!!!!!!!!!!!!!..." or "GGGGGG..." capped at "max_tokens"
Also interaction directly through the webui reproduce the same output.
The repeating chars changing when I select a other "Present" under "Parameters -> Generation"

If I run a other model, like TheBloke/Llama-2-7B-Chat-GGUF - llama-2-7b-chat.Q4_K_M.gguf, I can "chat" with the AI totally normal.

[acon96]

Make sure you are running the latest version of text-generation-webui. That sounds like a known bug with GPU generation in llama-cpp-python v0.2.29 to v0.2.32

[colaborat0r]

Maybe check: Is TextGen actually using your GPU in the Docker container?
Are you using the llama.cpp model loader in TextGen?