sd.next in WSL2: server dies when hashing the model #2563

Tillerz · 2023-11-27T10:01:49Z

Tillerz
Nov 27, 2023

Hi,

any suggestions where to look? I installed all the things, UI works, I can add extentions, but the moment when I select a model and it calculates its hash, it takes some time and then the server dies with a "Killed" message

What I installed was:

sudo apt-get update 
sudo apt install python3-pip
sudo pip3 install virtualenv
sudo apt -y install python3-venv
sudo apt install python-is-python3
sudo apt install libgl1
sudo apt install nvidia-cudnn
add export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH to .bashrc
git clone https://github.com/vladmandic/automatic.git automatic.latest

This is the server log from start to crash:

10:52:50-465093 DEBUG    Logger: file=/home/tillerz/automatic.latest/sdnext.log level=10 size=0 mode=create
10:52:50-467475 INFO     Starting SD.Next
10:52:50-468133 INFO     Python 3.10.12 on Linux
10:52:50-496393 INFO     Version: app=sd.next updated=2023-11-27 hash=8136723f url=https://github.com/vladmandic/automatic.git/tree/master
10:52:51-060193 INFO     Platform: arch=x86_64 cpu=x86_64 system=Linux release=5.15.133.1-microsoft-standard-WSL2 python=3.10.12
10:52:51-061511 DEBUG    Setting environment tuning
10:52:51-062295 DEBUG    Cache folder: /home/tillerz/.cache/huggingface/hub
10:52:51-063244 DEBUG    Torch overrides: cuda=False rocm=False ipex=False diml=False openvino=False
10:52:51-064389 DEBUG    Torch allowed: cuda=True rocm=True ipex=True diml=True openvino=True
10:52:51-067960 INFO     nVidia CUDA toolkit detected: nvidia-smi present
10:52:51-120462 DEBUG    Repository update time: Mon Nov 27 09:19:15 2023
10:52:51-121763 INFO     Startup: standard
10:52:51-122753 INFO     Verifying requirements
10:52:51-135359 INFO     Verifying packages
10:52:51-136999 INFO     Verifying submodules
10:52:51-375414 DEBUG    Submodule: extensions-builtin/sd-extension-chainner / main
10:52:51-389235 DEBUG    Submodule: extensions-builtin/sd-extension-system-info / main
10:52:51-396134 DEBUG    Submodule: extensions-builtin/sd-webui-agent-scheduler / main
10:52:51-404789 DEBUG    Submodule: extensions-builtin/sd-webui-controlnet / main
10:52:51-439818 DEBUG    Submodule: extensions-builtin/stable-diffusion-webui-images-browser / main
10:52:51-446405 DEBUG    Submodule: extensions-builtin/stable-diffusion-webui-rembg / master
10:52:51-452309 DEBUG    Submodule: modules/k-diffusion / master
10:52:51-459273 DEBUG    Submodule: modules/lora / main
10:52:51-467686 DEBUG    Submodule: wiki / master
10:52:51-471907 DEBUG    Register paths
10:52:51-517727 DEBUG    Installed packages: 221
10:52:51-518736 DEBUG    Extensions all: ['sd-extension-system-info', 'sd-webui-agent-scheduler', 'sd-extension-chainner', 'sd-webui-controlnet',
                         'stable-diffusion-webui-rembg', 'stable-diffusion-webui-images-browser', 'Lora']
10:52:51-519757 DEBUG    Running extension installer: /home/tillerz/automatic.latest/extensions-builtin/sd-extension-system-info/install.py
10:52:51-654739 DEBUG    Running extension installer: /home/tillerz/automatic.latest/extensions-builtin/sd-webui-agent-scheduler/install.py
10:52:51-783664 DEBUG    Running extension installer: /home/tillerz/automatic.latest/extensions-builtin/sd-webui-controlnet/install.py
10:52:51-894683 DEBUG    Running extension installer: /home/tillerz/automatic.latest/extensions-builtin/stable-diffusion-webui-rembg/install.py
10:52:52-004630 DEBUG    Running extension installer: /home/tillerz/automatic.latest/extensions-builtin/stable-diffusion-webui-images-browser/install.py
10:52:52-112070 DEBUG    Extensions all: ['StyleSelectorXL', 'sd-dynamic-prompts', 'adetailer']
10:52:52-133425 DEBUG    Running extension installer: /home/tillerz/automatic.latest/extensions/sd-dynamic-prompts/install.py
10:52:52-246484 DEBUG    Running extension installer: /home/tillerz/automatic.latest/extensions/adetailer/install.py
10:52:52-341784 INFO     Extensions enabled: ['sd-extension-system-info', 'sd-webui-agent-scheduler', 'sd-extension-chainner', 'sd-webui-controlnet',
                         'stable-diffusion-webui-rembg', 'stable-diffusion-webui-images-browser', 'Lora', 'StyleSelectorXL', 'sd-dynamic-prompts', 'adetailer']
10:52:52-342622 INFO     Verifying requirements
10:52:52-349685 DEBUG    Setup complete without errors: 1701078772
10:52:52-356041 INFO     Extension preload: {'extensions-builtin': 0.0, 'extensions': 0.0}
10:52:52-356976 DEBUG    Starting module: <module 'webui' from '/home/tillerz/automatic.latest/webui.py'>
10:52:52-357701 INFO     Command line args: ['--debug'] debug=True
10:52:56-795016 INFO     Load packages: torch=2.1.1+cu121 diffusers=0.23.1 gradio=3.43.2
10:52:57-355339 DEBUG    Read: file="config.json" json=34 bytes=1678
10:52:57-357061 DEBUG    Unknown settings: ['queue_keyboard_shortcut']
10:52:57-357718 INFO     Engine: backend=Backend.ORIGINAL compute=cuda mode=no_grad device=cuda cross-optimization="Scaled-Dot-Product"
10:52:57-391678 INFO     Device: device=NVIDIA GeForce RTX 4090 n=1 arch=sm_90 cap=(8, 9) cuda=12.1 cudnn=8902 driver=546.17
10:53:01-273682 DEBUG    Entering start sequence
10:53:01-274887 DEBUG    Initializing
10:53:01-279865 INFO     Available VAEs: path="/home/tillerz/models/VAE" items=3
10:53:01-280543 INFO     Disabling uncompatible extensions: backend=Backend.ORIGINAL []
10:53:01-283313 DEBUG    Read: file="metadata.json" json=88 bytes=177531
10:53:01-284252 INFO     Available models: path="/home/tillerz/models/Stable-diffusion" items=34 time=0.00
10:53:01-487035 DEBUG    Load extensions
10:53:02-263399 INFO     Extension: script='extensions-builtin/sd-webui-agent-scheduler/scripts/task_scheduler.py' Using sqlite file:
                         extensions-builtin/sd-webui-agent-scheduler/task_scheduler.sqlite3
10:53:02-493251 INFO     Extension: script='extensions-builtin/sd-webui-controlnet/scripts/controlnet.py' Warning: ControlNet failed to load SGM - will use LDM instead.
10:53:02-494008 INFO     Extension: script='extensions-builtin/sd-webui-controlnet/scripts/controlnet.py' ControlNet preprocessor location:
                         /home/tillerz/automatic.latest/extensions-builtin/sd-webui-controlnet/annotator/downloads
10:53:02-497232 INFO     Extension: script='extensions-builtin/sd-webui-controlnet/scripts/hook.py' Warning: ControlNet failed to load SGM - will use LDM instead.
/home/tillerz/automatic.latest/venv/lib/python3.10/site-packages/numba/np/ufunc/parallel.py:371: NumbaWarning: The TBB threading layer requires TBB version 2021 update 6 or later i.e., TBB_INTERFACE_VERSION >= 12060. Found TBB_INTERFACE_VERSION = 12050. The TBB threading layer is disabled.
  warnings.warn(problem)
10:53:04-964828 INFO     Extension: script='extensions/adetailer/scripts/!adetailer.py' [-] ADetailer initialized. version: 23.11.1, num models: 9
10:53:05-020847 INFO     Extensions time: 3.53 { Lora=0.26 sd-webui-agent-scheduler=0.44 sd-webui-controlnet=0.24 stable-diffusion-webui-rembg=0.84 adetailer=1.58 }
10:53:05-075107 DEBUG    Read: file="html/upscalers.json" json=4 bytes=2640
10:53:05-076216 DEBUG    Read: file="extensions-builtin/sd-extension-chainner/models.json" json=24 bytes=2693
10:53:05-077111 DEBUG    chaiNNer models: path="/home/tillerz/models/chaiNNer" defined=24 discovered=0 downloaded=0
10:53:05-078665 DEBUG    Upscaler type=ESRGAN folder="/home/tillerz/models/ESRGAN" model="8x_NMKD-Superscale_150000_G"
                         path="/home/tillerz/models/ESRGAN/8x_NMKD-Superscale_150000_G.pth"
10:53:05-079543 DEBUG    Upscaler type=SwinIR folder="/home/tillerz/models/SwinIR" model="SwinIR_4x" path="/home/tillerz/models/SwinIR/SwinIR_4x.pth"
10:53:05-080083 DEBUG    Load upscalers: total=52 downloaded=7 user=2 ['None', 'Lanczos', 'Nearest', 'ChaiNNer', 'RealESRGAN', 'SCUNet', 'ESRGAN', 'LDSR', 'SwinIR', 'SD']
10:53:05-128057 DEBUG    Load styles: folder="/home/tillerz/models/styles" items=355
10:53:05-131591 DEBUG    Creating UI
10:53:05-213124 INFO     Load UI theme: name="black-teal" style=Dark base=sdnext.css
10:53:05-303534 DEBUG    Extra networks: page='model' items=34 subfolders=4 tab=txt2img folders=['/home/tillerz/models/Stable-diffusion', '/home/tillerz/models/Diffusers',
                         'models/Reference', '/home/tillerz/models/Stable-diffusion'] list=0.01 desc=0.00 info=0.00
10:53:05-311140 DEBUG    Extra networks: page='style' items=355 subfolders=3 tab=txt2img folders=['/home/tillerz/models/styles', 'html'] list=0.02 desc=0.00 info=0.00
10:53:05-312908 DEBUG    Extra networks: page='embedding' items=18 subfolders=1 tab=txt2img folders=['/home/tillerz/models/embeddings'] list=0.00 desc=0.00 info=0.00
10:53:05-314910 DEBUG    Extra networks: page='hypernetwork' items=0 subfolders=1 tab=txt2img folders=['/home/tillerz/models/hypernetworks'] list=0.00 desc=0.00 info=0.00
10:53:05-315992 DEBUG    Extra networks: page='vae' items=3 subfolders=1 tab=txt2img folders=['/home/tillerz/models/VAE'] list=0.00 desc=0.00 info=0.00
10:53:05-317575 DEBUG    Extra networks: page='lora' items=52 subfolders=2 tab=txt2img folders=['/home/tillerz/models/Lora', '/home/tillerz/models/LyCORIS'] list=0.01
                         desc=0.00 info=0.00
10:53:05-492591 DEBUG    Read: file="ui-config.json" json=9 bytes=398
10:53:05-511957 DEBUG    Themes: builtin=6 default=5 external=55
10:53:05-918559 DEBUG    Script: 0.14 ui_tabs /home/tillerz/automatic.latest/extensions-builtin/stable-diffusion-webui-images-browser/scripts/image_browser.py
10:53:05-929510 DEBUG    Read: file="/home/tillerz/automatic.latest/html/extensions.json" json=328 bytes=185644
10:53:06-001380 DEBUG    Extension list: processed=312 installed=10 enabled=10 disabled=0 visible=312 hidden=0
10:53:06-321947 INFO     Local URL: http://127.0.0.1:7860/
10:53:06-322729 DEBUG    Gradio functions: registered=2164
10:53:06-323159 INFO     Initializing middleware
10:53:06-329152 DEBUG    Creating API
10:53:06-440037 INFO     [AgentScheduler] Task queue is empty
10:53:06-441718 INFO     [AgentScheduler] Registering APIs
10:53:06-777794 DEBUG    Script: 0.33 app_started /home/tillerz/automatic.latest/extensions-builtin/sd-webui-agent-scheduler/scripts/task_scheduler.py
10:53:06-817694 DEBUG    Scripts setup: ['ControlNet:0.045', 'ADetailer:0.016', 'Dynamic Prompts v2.17.1:0.009']
10:53:06-818488 DEBUG    Model metadata: file="metadata.json" no changes
10:53:06-818917 DEBUG    Model auto load disabled
10:53:06-819786 DEBUG    Save: file="config.json" json=34 bytes=1678
10:53:06-820453 INFO     Startup time: 14.45 { torch=3.46 gradio=0.93 libraries=4.48 extensions=3.53 face-restore=0.20 upscalers=0.06 extra-networks=0.05
                         ui-extra-networks=0.19 ui-txt2img=0.07 ui-img2img=0.06 ui-settings=0.29 ui-extensions=0.23 ui-defaults=0.15 launch=0.17 app-started=0.45 }
10:53:16-994785 DEBUG    Themes: builtin=6 default=5 external=55
10:53:17-207310 INFO     Browser session: user=None client=127.0.0.1 agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
                         Chrome/119.0.0.0 Safari/537.36
10:53:23-620393 INFO     Select: model="favourites/dreamshaper_8"
10:53:23-621834 DEBUG    Load model weights: existing=False target=/home/tillerz/models/Stable-diffusion/favourites/dreamshaper_8.safetensors info=None
Loading model: /home/tillerz/models/Stable-diffusion/favourites/dreamshaper_8.safetensors ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/2.1 GB -:--:--
10:53:23-915460 DEBUG    Load model: name=/home/tillerz/models/Stable-diffusion/favourites/dreamshaper_8.safetensors dict=True
10:53:23-984592 DEBUG    Desired Torch parameters: dtype=FP16 no-half=False no-half-vae=False upscast=False
10:53:23-985956 INFO     Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16 unet=torch.float16 context=no_grad fp16=True bf16=False
10:53:23-987292 DEBUG    Model dict loaded: {'ram': {'used': 1.62, 'total': 3.83}, 'gpu': {'used': 1.58, 'total': 23.99}, 'retries': 0, 'oom': 0}
10:53:23-998799 DEBUG    Model config loaded: {'ram': {'used': 1.62, 'total': 3.83}, 'gpu': {'used': 1.58, 'total': 23.99}, 'retries': 0, 'oom': 0}

Killed

Tillerz · 2023-11-27T10:35:44Z

Tillerz
Nov 27, 2023
Author

Ok, looks like 4GB memory was the problem. It's hashing the file now. But that takes looooooong (2 minutes 37 seconds), that was way faster on plain Windows 11. I guess it's working in memory instead of in VRAM? Do I have to set a toggle for that somewhere?

11:32:24-324103 INFO     Select: model="dream-zovya-revanim"
Loading model: /home/tillerz/models/Stable-diffusion/dream-zovya-revanim.safetensors ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/2.1 GB -:--:--
11:32:24-722354 INFO     Setting Torch parameters: device=cuda dtype=torch.float16 vae=torch.float16 unet=torch.float16 context=no_grad
                         fp16=True bf16=False
config.json: 100%|█████████████████████████████████████████████████████████████████████████████████████| 4.52k/4.52k [00:00<00:00, 29.6MB/s]
model.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████| 1.71G/1.71G [02:37<00:00, 10.8MB/s]

But it looks like rendering an image is way faster now. As the hashing is only done once for a model, I can live with that. :P

But I get a warning in the log:

/home/tillerz/automatic.latest/venv/lib/python3.10/site-packages/numba/np/ufunc/parallel.py:371: NumbaWarning: The TBB threading layer requires TBB version 2021 update 6 or later i.e., TBB_INTERFACE_VERSION >= 12060. Found TBB_INTERFACE_VERSION = 12050. The TBB threading layer is disabled.
  warnings.warn(problem)

0 replies

vladmandic · 2023-11-27T12:22:01Z

vladmandic
Nov 27, 2023
Maintainer

Difference in hashing performance is not due to ram VS Vram, it's purely due to wsl horrible way of handling external filesystem. Model load is also slower.
Placing model inside wsl filesystem is 3x faster.

Regarding warning, havent seen that before, but looking at it now and not much i can do about it. Its a numba warning that is outside of sdnext control.
Basically, when numba loads, it checks if its threading library is available or not. sdnext does not use numba with threading, but numba checks blindly. in my case, threading library is not available and numba doesn't complain. in your case, threading library is available, but wrong version - which sdnext doesn't care about since it doesn't use it (nor it installs it). i don't know what installed tbb (that's the threading library in question), but i don't want to force uninstall or upgrade it, since i didn't install it.

1 reply

Tillerz Nov 27, 2023
Author

I moved the models into WSL, so that isn't it. Gonna check with other models, maybe it was a one-time thing. :)

Will check if I can get rid of that numba warning, if not I'll just ignore it.

Tillerz · 2023-11-27T14:37:41Z

Tillerz
Nov 27, 2023
Author

Ok, I checked with another model and generating the hash was as fast as it usually is. So it may have been a "load a model for the first time in this environment" thing. :)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sd.next in WSL2: server dies when hashing the model #2563

{{title}}

Replies: 3 comments 1 reply

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

sd.next in WSL2: server dies when hashing the model #2563

Tillerz Nov 27, 2023

Replies: 3 comments · 1 reply

Tillerz Nov 27, 2023 Author

vladmandic Nov 27, 2023 Maintainer

Tillerz Nov 27, 2023 Author

Tillerz Nov 27, 2023 Author

Tillerz
Nov 27, 2023

Replies: 3 comments 1 reply

Tillerz
Nov 27, 2023
Author

vladmandic
Nov 27, 2023
Maintainer

Tillerz Nov 27, 2023
Author

Tillerz
Nov 27, 2023
Author