Docker, Intel Arc performance tanked from ~11 It/s to ~5 It/s after updating docker container

[cutterjohn]

`System:
  Kernel: 6.6.7-1-default arch: x86_64 bits: 64 compiler: gcc v: 13.2.1
    Desktop: KDE Plasma v: 5.27.10 tk: Qt v: 5.15.11 wm: kwin_wayland dm: SDDM
    Distro: openSUSE Tumbleweed 20231226
Machine:
  Type: Desktop Mobo: ASRock model: X670E PG Lightning
    serial: <superuser required> UEFI: American Megatrends LLC. v: 1.24.AS02
    date: 05/02/2023
Memory:
  System RAM: total: 64 GiB available: 62.51 GiB used: 9.11 GiB (14.6%)
  RAM Report: permissions: Unable to run dmidecode. Root privileges
    required.
CPU:
  Info: 12-core model: AMD Ryzen 9 7900X bits: 64 type: MT MCP arch: Zen 4
    rev: 2 cache: L1: 768 KiB L2: 12 MiB L3: 64 MiB
  Speed (MHz): avg: 533 high: 3593 min/max: 400/5733 cores: 1: 400 2: 400
    3: 400 4: 400 5: 400 6: 400 7: 400 8: 400 9: 400 10: 400 11: 400 12: 3593
    13: 400 14: 400 15: 400 16: 400 17: 400 18: 400 19: 400 20: 400 21: 400
    22: 400 23: 400 24: 400 bogomips: 225243
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
Graphics:
  Device-1: Intel DG2 [Arc A770] driver: i915 v: kernel arch: Gen-12.7 pcie:
    speed: 2.5 GT/s lanes: 1 ports: active: DP-3,DP-4 empty: DP-1, DP-2,
    HDMI-A-1, HDMI-A-2, HDMI-A-3 bus-ID: 03:00.0 chip-ID: 8086:56a0
  Display: wayland server: X.org v: 1.21.1.9 with: Xwayland v: 23.2.2
    compositor: kwin_wayland driver: X: loaded: modesetting unloaded: fbdev,vesa
    alternate: intel dri: iris gpu: i915 d-rect: 3128x1920 display-ID: 0
  Monitor-1: DP-3 pos: primary,left res: 2048x1152 size: N/A
  Monitor-2: DP-4 pos: right res: 1080x1920 size: N/A
  API: EGL v: 1.5 platforms: device: 0 drv: iris device: 1 drv: swrast
    surfaceless: drv: iris wayland: drv: iris x11: drv: iris inactive: gbm
  API: OpenGL v: 4.6 compat-v: 4.5 vendor: intel mesa v: 23.2.1 glx-v: 1.4
    direct-render: yes renderer: Mesa Intel Arc A770 Graphics (DG2)
    device-ID: 8086:56a0 display-ID: :1.0
  API: Vulkan v: 1.3.268 surfaces: xcb,xlib,wayland device: 0
    type: discrete-gpu driver: mesa intel device-ID: 8086:56a0
`

Looking at btop CPU is NOT being used.

Looking at Mission Control it also looks like GPU is not being used either, but I have my doubts wrt Mission Control Intel Arc support(as in I dont think that it really works well, or maybe at all).

See nothing really odd in the docker output... or at least nothing that I recall being different from my earlier original docker container installation from roughly 2.5w ago.

I would try native, but I ran into some dependency problems which I didn't feel like resolving(SUSE quirks and idiosyncrasies) so I decided to give the container version a shot which originally worked fairly well... ~11 It/s.

`03:10:38-438563 INFO     Device: device=Intel(R) Arc(TM) A770 Graphics n=1 ipex=2.0.110+xpu                                        
2024-01-01 03:10:39.848924: I itex/core/wrapper/itex_gpu_wrapper.cc:35] Intel Extension for Tensorflow* GPU backend is loaded.
2024-01-01 03:10:39.868257: W itex/core/ops/op_init.cc:58] Op: _QuantizedMaxPool3D is already registered in Tensorflow
2024-01-01 03:10:39.870986: I itex/core/devices/gpu/itex_gpu_runtime.cc:129] Selected platform: Intel(R) Level-Zero
2024-01-01 03:10:39.870997: I itex/core/devices/gpu/itex_gpu_runtime.cc:154] number of sub-devices is zero, expose root device.
`

Progress 5.81it/s ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:01:08

versus earlier (not the best run but, similar prompt, same model)

Progress 10.09it/s ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:39

EDIT: Also Im brand new to even trying docker, so be somewhat specific if you want me to try something beyond the basics, as Ive been at it for a whole 2w and haven't had a great deal of need to troubleshoot, delve into its inner plumbing etc. It was just a shortcut for now since I was too lazy to work around SUSE quirks... (which worked for me for a while as I had fun messing around with SD last week, try models, prompts, etc. was going to try a LORA but w/this perf no thanks...)...

EDIT2: I did also see these for the docker container ...

01:30:28-182862 WARNING Package version mismatch: accelerate 0.24.1 required 0.25.0
01:30:28-183341 INFO Installing package: accelerate==0.25.0
01:30:35-218004 WARNING Package version mismatch: diffusers 0.24.0 required 0.25.0
01:30:35-218784 INFO Installing package: diffusers==0.25.0
01:30:37-248497 WARNING Package version mismatch: huggingface_hub 0.19.4 required 0.20.1
01:30:37-249171 INFO Installing package: huggingface_hub==0.20.1
01:30:38-840998 WARNING Package version mismatch: transformers 4.35.2 required 4.36.2
01:30:38-841668 INFO Installing package: transformers==4.36.2
01:30:42-812799 WARNING Package version mismatch: urllib3 1.26.15 required 1.26.18
01:30:42-813463 INFO Installing package: urllib3==1.26.18
01:30:44-144155 WARNING Package version mismatch: timm 0.9.7 required 0.9.12
01:30:44-144809 INFO Installing package: timm==0.9.12
01:30:45-919006 WARNING Package version mismatch: typing-extensions 4.8.0 required 4.9.0
01:30:45-919884 INFO Installing package: typing-extensions==4.9.0

but it looks like it updates all of those....

[Disty0]

Update IPEX with --reinstall. New release is tuned for IPEX 2.1 but you are using IPEX 2.0.
Main change is MKL and DPCPP is handled by SDNext now. No need to install OneAPI BaseKit
(We assume that you use webui.sh).

But the new version of MKL and DPCPP are not compatible with IPEX 2.0.
(Older MKL and DPCPP are also not compatible with the new IPEX 2.1 and that docker image is pretty old at this point.)

Also i dropped support for IPEX 2.0 in dev branch so update before the next release cycle.

Changelog: https://github.com/vladmandic/automatic/blob/master/CHANGELOG.md

• IPEX, thanks @Disty0
  • update to Torch 2.1
    if you get file not found errors, set DISABLE_IPEXRUN=1 and run the webui with --reinstall
  • built-in MKL and DPCPP for IPEX, no need to install OneAPI anymore
  • StableVideoDiffusion is now supported with IPEX
  • 8 bit support with NNCF on Diffusers backend
  • fix IPEX Optimize not applying with Diffusers backend
  • disable 32bit workarounds if the GPU supports 64bit
  • add DISABLE_IPEXRUN and DISABLE_IPEX_1024_WA environment variables
  • performance and compatibility improvements

[Disty0]

Wrote a new Dockerfile:

Build it:

docker build -t sdnext-ipex -f Dockerfile .

Run it:

docker run -it --device /dev/dri -v ~/sdnext:/sdnext -v sdnext-venv:/python -v huggingface:/root/.cache/huggingface -p 7860:7860 --name sdnext sdnext-ipex

Dockerfile:

FROM ubuntu:jammy

RUN apt-get update && \
    apt-get install -y --no-install-recommends --fix-missing \
    ca-certificates \
    wget \
    gpg \
    git

RUN wget -qO - https://repositories.intel.com/gpu/intel-graphics.key | \
    gpg --dearmor --output /usr/share/keyrings/intel-graphics.gpg
RUN echo "deb [arch=amd64,i386 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu jammy client" | \
    tee /etc/apt/sources.list.d/intel-gpu-jammy.list
RUN apt-get update

RUN apt-get install -y --no-install-recommends --fix-missing \
    intel-opencl-icd=23.35.27191.42-775~22.04 \
    intel-level-zero-gpu=1.3.27191.42-775~22.04 \
    level-zero=1.14.0-744~22.04

RUN apt-get update && apt-get install -y --no-install-recommends --fix-missing \
    python3.10 \
    python3-pip \
    python3-venv

RUN pip --no-cache-dir install --upgrade \
    pip \
    setuptools

RUN apt-get install -y --no-install-recommends --fix-missing \
    libgl1 \
    libglib2.0-0 \
    libgomp1 \
    libjemalloc-dev

RUN apt-get clean && rm -rf /var/lib/apt/lists/*

RUN echo '#!/bin/bash\ngit status || git clone https://github.com/vladmandic/automatic.git .\n./webui.sh "$@"' | tee /bin/startup.sh
RUN chmod 755 /bin/startup.sh

VOLUME [ "/deps" ]
VOLUME [ "/sdnext" ]
VOLUME [ "/root/.cache/huggingface" ]

ENV venv_dir=/python/venv
WORKDIR /sdnext
 
ENTRYPOINT [ "startup.sh", "-f", "--use-ipex", "--listen" ]

[cutterjohn]

Thanks for the reply and work, but since I am pretty new to docker and learning as I go along, and need to do something, which appears to be now unless I make another attempt at a native install today.

I'm not sure if you're suggesting to do a 'native' install using the script method? Or to modify the docker image?

If it's the docker image, Ill need to do some reading to figure out how to access the image contents and modify them, as Im doing this all CLI ATM. I had briefly considered using podman but decided best not to layer complications of podman on top of docker when I haven't used either, the former not at all, and docker about 2w...

either way, w/no 'latent' (I guess thats what you call a stopped image?), I fresh start vladmandic via:

docker run -it --device /dev/dri -v ~/docker-mount/sd-webui:/sd-webui -v deps:/deps -v huggingface:/root/.cache/huggingface -p 7860:7860 --name sd-server nuullll/ipex-arc-sd:latest

I think that all I might need to do is replace starting up the webui/etc. w/bash, and go from there.

Building it natively, on OpenSUSE Tumbleweed also requires a modification for the Intel Arc card, there is, Im trying to remember either a boot variable or some other variable needs to be set. I remember reading about it several weeks ago, but since I ran into some dependency problems, I decided on the docker container image shortcut, since I was able to get it up and running fairly easily/quickly.

Thanks again for your reply.

Happy New Year.

[Disty0]

Nuullll's docker image is pretty old. If you can install intel-compute-runtime or intel-level-zero-gpu on OpenSUSE, go with the native install. No need to install intel-oneapi stuff, SDNext will use it's own from pypi.

If you can't, use the docker image in the message above: #2673 (comment)

Copy the Dockerfile into a text file named Dockerfile and follow the Build and Run steps.

[cutterjohn]

Thanks again!

I'll give it a shot, and I think that you're teaching me something about docker, by giving me the above file, which I figured was probably how to try it.

So docker containers are NOT entirely reliant on what is contained in their own 'images'? This gives me some avenues for what I probably should look at next wrt docker, might move on to podman if I feel like Im comfortable enough w/docker itself at that point as I get the feeling that SUSE prefer podman over docker, as presumably does Fedora since that's where it originated, or so I gather.

[Disty0]

Docker images you install with docker are prebuilt ones. Dockerfile is the source code for them.
My method creates a new image form the source.

You can use Ubuntu 22.04 image and manually follow the commands in the Dockerfile too.
This is basically what docker is.

[cutterjohn]

Thanks again!

Gotcha, that's more what I expected but wasn't entirely certain.

There is a SLES repo from Intel for SuSE that I added as IIRC the standard OpenSuSE repos were rather lacking in Intel software, for whatever reason. Also have the pacman repo for native codecs, primarily. And the MS VS Code repo... I probably should make the Intel repo a bit higher priority than the OpenSuSE repos, e.g. pacman suggests prio 99 IIRC whereas standard repos are prio 90, such that pacman packages take precedence over the opensuse repos.

[cutterjohn]

Thanks again, but ran into a new problem where I can't seem to see where it went wrong... I's pretty familiar w/ubuntu systems and from I can infer about how dockerfiles work, the one that you provided appears to be fine although I could be easily missing fine details since Im a complete docker newb, well almost...

Building the image goes w/o errors, however running ends up with this oddity:

docker run -it --device /dev/dri -v ~/sdnext:/sdnext -v deps:/deps -v ~/.cache/huggingface:/root/.cache/huggingface -p 7860:7860 --name sdnext sdnext-ipex
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Cloning into '.'...
remote: Enumerating objects: 37873, done.
remote: Counting objects: 100% (931/931), done.
remote: Compressing objects: 100% (418/418), done.
remote: Total 37873 (delta 537), reused 865 (delta 506), pack-reused 36942
Receiving objects: 100% (37873/37873), 49.25 MiB | 61.43 MiB/s, done.
Resolving deltas: 100% (27082/27082), done.
Create and activate python venv
Launching ipexrun launch.py...
./webui.sh: /deps/venv/bin/ipexrun: /deps/venv/bin/python3: bad interpreter: No such file or directory
./webui.sh: line 95: /deps/venv/bin/ipexrun: Success

where when I look at the docker file, puthon3 is explicitly installed to the image, or so I am doing the dangerous thing of assuming it is.

and the image status ends up being,

b31da61df84d sdnext-ipex "startup.sh -f --use…" About a minute ago Exited (126) About a minute ago sdnext

as I would expect since it cannot run the startup python program.

I cloned the github repo and the webui.sh looks OK to me.

So now Im at a loss until I do some research, maybe something got skipped in the dockerfile specification? Im just running on by the seat of my pants, as it were ATM, and need to go look at docker file specs I guess...

Maybe while Im at it, Ill take another stab at building 'natively'... just need to relook up whatever that environment variable or boot variable was that needed to be set and double check installed Intel APIs... as I think that my python3 setup is fine although IIRC I went w/latest 311 available in repos as there shouldn't be any breaking changes...

[Disty0]

Try adding -e DISABLE_IPEXRUN=1 to the run command. Ipexrun fixes a lot of memory leaks with IPEX but it can cause random issues like this in some systems (even if it's the same docker image).

Took another look and it seems like your issue isn't from ipexrun. Try deleting deps volume: docker volume rm deps

[cutterjohn]

Ill retry the docker in a few minutes(maybe hours depending).

Took a stab at 'native' again and seem to be running into the same problem as #2627 , however Ive only gotten it to run w/CPU, so right now Im trying:

DISABLE_IPEXRUN=1 ./webui.sh --use-ipex --debug --reinstall

looks like it keeps trying to run CUDA for some reason:

13:49:10-910593 INFO Extension preload: {'extensions-builtin': 0.01, 'extensions': 0.0}
13:49:10-911381 DEBUG Starting module: <module 'webui' from '/home/xxx/dev/sdnext/automatic/webui.py'>
13:49:10-911898 INFO Command line args: ['--use-ipex', '--debug', '--reinstall'] debug=True use_ipex=True
reinstall=True
13:49:10-912427 DEBUG Env flags: []
There was a problem when trying to write in your cache folder (/home/xxx/.cache/huggingface/hub). You should set the environment variable TRANSFORMERS_CACHE to a writable directory.
13:49:51-411079 DEBUG Load IPEX==2.1.10+xpu
There was a problem when trying to write in your cache folder (/home/xxx/.cache/huggingface/hub). Please, ensure the directory exists and can be written to.
13:50:05-831038 INFO Load packages: torch=2.1.0a0+cxx11.abi diffusers=0.25.0 gradio=3.43.2
13:50:14-194069 DEBUG Read: file="config.json" json=8 bytes=354 time=3.177
13:50:14-195303 INFO Engine: backend=Backend.ORIGINAL compute=ipex mode=no_grad device=xpu
cross-optimization="Scaled-Dot-Product"
13:50:14-208338 INFO Device: device=Intel(R) Arc(TM) A770 Graphics n=1 ipex=2.1.10+xpu
13:50:14-209187 DEBUG Migrated styles: file=styles.csv folder=models/styles
13:50:14-590727 DEBUG Load styles: folder="models/styles" items=288 time=0.38
2024-01-01 13:50:28.022751: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:9342] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-01-01 13:50:28.022792: E tensorflow/compiler/xla/stream_executor/cuda/cuda_fft.cc:609] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-01-01 13:50:28.022813: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:1518] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-01-01 13:50:37.651859: I itex/core/wrapper/itex_cpu_wrapper.cc:60] Intel Extension for Tensorflow* AVX512 CPU backend is loaded.
2024-01-01 13:50:46.465634: I itex/core/wrapper/itex_gpu_wrapper.cc:35] Intel Extension for Tensorflow* GPU backend is loaded.
2024-01-01 13:50:46.489715: I itex/core/graph/xpu_graph.cc:130] Please enable TF_ENABLE_ONEDNN_OPTS.So can benefit from the optimization of the MklCPUAllocator memory allocation.
2024-01-01 13:50:46.597115: I itex/core/devices/gpu/itex_gpu_runtime.cc:129] Selected platform: Intel(R) Level-Zero
2024-01-01 13:50:46.597144: I itex/core/devices/gpu/itex_gpu_runtime.cc:154] number of sub-devices is zero, expose root device.
2024-01-01 13:50:46.597628: F itex/core/utils/op_kernel.cc:54] Check failed: false Multiple KernelCreateFunc registration
If you need help, create an issue at https://github.com/intel/intel-extension-for-tensorflow/issues

LIBXSMM_VERSION: main_stable-1.17-3651 (25693763)
LIBXSMM_TARGET: cpx [AMD Ryzen 9 7900X 12-Core Processor]
Registry and code: 13 MB
Command: python3 launch.py --use-ipex --debug --reinstall
Uptime: 58.739190 s
Aborted (core dumped)

guess Ill give the docker a retry

EDIT: whoops just noticed that that huggingface directory had root ownership from somewhere, changed that and rerunning native, but that CUDA stuf... eh...

EDIT2: decided to clean re-install, trying this as initial run:

DISABLE_IPEXRUN=1 ./webui.sh --use-ipex --debug

EDIT3: well that resulted in the same

EDIT4: did rm deps, and yep docker version runs now, still same 5 It/s though...

I think that the 'native' install may be dependencies, Ill look into it tomorrow.

[cutterjohn]

OK, did a little more testing managed to hit 6 It/s so it's a slight improvement. also I noticed:

─────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────╮
│ /sdnext/modules/call_queue.py:54 in f │
│ │
│ 53 │ │ │ │ pr.enable() │
│ ❱ 54 │ │ │ res = func(*args, **kwargs) │
│ 55 │ │ │ if res is None: │
│ │
│ /sdnext/modules/ui_common.py:131 in save_files │
│ │
│ 130 │ │ │ destination = shared.opts.outdir_save │
│ ❱ 131 │ │ │ namegen = modules.images.FilenameGenerator(p, seed=p.all_seeds[i], prompt=p. │
│ 132 │ │ │ dirname = namegen.apply(shared.opts.directories_filename_pattern or "[prompt │
│ │
│ /sdnext/modules/images.py:328 in init │
│ │
│ 327 │ │ self.p = p │
│ ❱ 328 │ │ if seed is not None and seed > 0: │
│ 329 │ │ │ self.seed = seed │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: '>' not supported between instances of 'str' and 'int'

after doing an upscale (Realesrgan 4x+ or whatever its called) then clicking on the save button. It also happened to me once on the txt2img 'front' page when clicking on save, however it did NOT happen every time... very strange

Take these with a grain of salt as my damned iGPU managed to re-enable itself, so I need to reboot again and check out what happened in BIOS... known problem at least on ASROCK boards where this happens occasionally...

EDIT: crashing in launch.py

LIBXSMM_VERSION: main_stable-1.17-3651 (25693763)
LIBXSMM_TARGET: cpx [AMD Ryzen 9 7900X 12-Core Processor]
Registry and code: 13 MB
Command: /home/xxx/dev/sdnext/automatic/venv/bin/python3 -u launch.py --use-ipex
Uptime: 2.574399 s
Traceback (most recent call last):
File "/home/xxx/dev/sdnext/automatic/venv/bin/ipexrun", line 8, in
sys.exit(main())
^^^^^^
File "/home/xxx/dev/sdnext/automatic/venv/lib64/python3.11/site-packages/intel_extension_for_pytorch/launcher.py", line 109, in main
cpu_run_main_with_args(args)
File "/home/xxx/dev/sdnext/automatic/venv/lib64/python3.11/site-packages/intel_extension_for_pytorch/cpu/launch/launch.py", line 447, in run_main_with_args
launcher.launch(args)
File "/home/xxx/dev/sdnext/automatic/venv/lib64/python3.11/site-packages/intel_extension_for_pytorch/cpu/launch/launcher_multi_instances.py", line 322, in launch
raise subprocess.CalledProcessError(
subprocess.CalledProcessError: Command 'taskset -c 0-11 /home/xxx/dev/sdnext/automatic/venv/bin/python3 -u launch.py --use-ipex' died with <Signals.SIGABRT: 6>.