Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New ComfyUI Dockerfile features #739

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

lainedfles
Copy link

@lainedfles lainedfles commented Sep 29, 2024

These changes allow Comfy to leverage the powerful image generation and editing capabilities provided by the Krita AI Diffusion project, GGUF, x-flux, ControlNet, and IPAdapter. Users can now access advanced features like outpainting, inpainting, upscaling, etc. within the Comfy UI.

The commit also improves the build process by using a cache for pip installs and specifying types for mounted volumes in Docker for better performance and reproducibility.

  • Non-root default
  • Use git describe to automatically use latest ComfyUI tag (or USE_EDGE=true variable for latest commit)
  • Support custom_nodes:
    • ComfyUI-GGUF (USE_GGUF=true)
    • x-flux-comfyui (USE_XFLUX=true)
    • comfyui_controlnet_aux (USE_CNAUX=false)
    • ComfyUI_IPAdapter_plus (USE_IPAPLUS=true)
    • comfyui-inpaint-nodes (USE_INPAINT=true)
    • comfyui-tooling-nodes (USE_TOOLING=true)
  • Support krita-ai-diffusion (USE_KRITA=true)
    This automatically enables the required dependency nodes.
  • Downloads recommended models for Krita AI Diffusion if enabled with KRITA_DOWNLOAD_MODELS=true
  • Update base image to pytorch 2.3.1

My preference:

podman-compose --verbose --podman-build-args='--build-arg="USE_EDGE=true" --build-arg="USE_GGUF=true" --build-arg="USE_XFLUX=true" --build-arg="USE_KRITA=true"' --profile comfy build

Closes issue #728

Bleeding edge or latest ComfyUI commit adds flux1-dev support.

Update versions

Edit: update to reflect latest commits

- Non-root default
- Use `git describe` to automatically use latest ComfyUI tag (or `USE_EDGE=true` variable for latest commit)
- Support custom_nodes:
  * **ComfyUI-GGUF support** (`USE_GGUF=true`)
  * **x-flux-comfyui support** (`USE_XFLUX=true`)
  * **comfyui_controlnet_aux support** (`USE_CNAUX=false`)
    - Update base image to pytorch 2.3.1
    - Fix missing comment line escape
    - Support [krita-ai-diffusion](https://github.com/Acly/krita-ai-diffusion/wiki/ComfyUI-Setup) (`USE_KRITA=true`)
      This enables the following custom nodes (including comfyui_controlnet_aux):
      * **ComfyUI_IPAdapter_plus** (`USE_IPAPLUS=true`)
      * **comfyui-inpaint-nodes** (`USE_INPAINT=true`)
      * **comfyui-tooling-nodes** (`USE_TOOLING=true`)
This commit adds support for integrating Krita AI Diffusion models and nodes into the Comfy image editing application. It includes the following changes:

- Downloads recommended models for Krita AI Diffusion if enabled
- Symlinks the downloaded upscale models into the models directory
- Adds environment variables to enable/disable dependancy node packs
- Clones and sets up the required Git repositories for those node packs if enabled
- Installs additional dependencies like aiohttp and tqdm if Krita AI Diffusion is used

These changes allow Comfy to leverage the powerful image generation and editing capabilities provided by the Krita AI Diffusion project. Users can now access advanced features like outpainting, inpainting, upscaling, etc. within the Comfy UI.

The commit also improves the build process by using a cache for pip installs and specifying types for mounted volumes in Docker for better performance and reproducibility.
@SummitCollie
Copy link

SummitCollie commented Nov 16, 2024

Sorry to post asking for help here, but I can't get this to run due to some issue with your UID/GID changes. Any idea what's going on here? Do I have to configure my host system in a particular way?

# ...
 => [comfy stage-0  9/10] COPY --chown=131:131 . /docker/                                                          0.1s 
 => [comfy stage-0 10/10] RUN chmod u+x /docker/entrypoint.sh && cp /docker/extra_model_paths.yaml /stable-diffus  0.2s 
 => [comfy] exporting to image                                                                                     3.0s 
 => => exporting layers                                                                                            3.0s 
 => => writing image sha256:ec115f42b580bc2ea9cb9c50466b17760444bf5d64198279d31ac9766f7f5992                       0.0s 
 => => naming to docker.io/library/sd-comfy:7                                                                      0.0s
 => [comfy] resolving provenance for metadata file                                                                 0.0s
[+] Running 1/1
 ✔ Container webui-docker-comfy-1  Recreated                                                                       0.1s 
Attaching to comfy-1
comfy-1  | Mounted comfy
comfy-1  | Mounted input
comfy-1  | Mounted .cache
comfy-1  | created directory '/stable-diffusion/models/upscale_models.stock'
comfy-1  | copied '/stable-diffusion/models/upscale_models/put_esrgan_and_other_upscale_models_here' -> '/stable-diffusion/models/upscale_models.stock/put_esrgan_and_other_upscale_models_here'
comfy-1  | removed '/stable-diffusion/models/upscale_models/put_esrgan_and_other_upscale_models_here'
comfy-1  | removed directory '/stable-diffusion/models/upscale_models'
comfy-1  | mv: cannot create directory '/data/config/comfy/custom_nodes/comfyui_controlnet_aux': Permission denied
comfy-1 exited with code 1

ComfyUI and A1111 run fine using the base repo FWIW.

@lainedfles
Copy link
Author

lainedfles commented Nov 16, 2024

@SummitCollie

Sorry to post asking for help here, but I can't get this to run due to some issue with your UID/GID changes. Any idea what's going on here? Do I have to configure my host system in a particular way?

Hi, this seems to be a great place for such issues. Testing is helpful. Based on your comment:

ComfyUI and A1111 run fine using the base repo FWIW.

I presume that your volumes have been created using the base repo default of UID/GID 0:0 or the root user. This is expected behavior since the UID has changed. The most "secure" option is to change the ownership of the existing files. I ran a command like this (only a single execution should be required):

podman run --rm -u root -v sd-comfy-data:/data -v sd-comfy-output:/output -it localhost/sd-comfy:7 bash -c 'chown -R app:app /data /output'

Perhaps something like:

docker run --rm -u root -v <data volume name or path>:/data -v <output volume name or path>:/output -it localhost/sd-comfy:7 bash -c 'chown -R app:app /data /output'

Alternatively, you could re-build the image to use the root UID by passing build arguments: USE_UID=0 & USE_GID=0.

While I'd explicitly avoided adding an automatic scripted action to enforce the recursive directory ownership, perhaps we can re-consider it if enough users require it.

@SummitCollie
Copy link

@lainedfles thanks for the quick response! As you can tell, I don't know what I'm doing with Docker, but I think I see what was going on. This is the command that worked for me (no "actual" docker volumes, just directories on host):

docker run --rm -u root -v ./data:/data -v ./output:/output -it sd-comfy:7 bash -c 'chown -R app:app /data /output'

It's running and generating images now, but I do get this CNAUX error upon server startup:

 ✔ Container webui-docker-comfy-1  Created                                                                         0.0s 
Attaching to comfy-1
comfy-1  | Mounted comfy
comfy-1  | Mounted input
comfy-1  | Mounted .cache
comfy-1  | Total VRAM 24086 MB, total RAM 63878 MB
comfy-1  | pytorch version: 2.3.1
comfy-1  | Set vram state to: NORMAL_VRAM
comfy-1  | Device: cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
comfy-1  | Using pytorch cross attention
comfy-1  | ****** User settings have been changed to be stored on the server instead of browser storage. ******
comfy-1  | ****** For multi-user setups add the --multi-user CLI argument to enable multiple user profiles. ******
comfy-1  | [Prompt Server] web root: /stable-diffusion/web
comfy-1  | Adding extra search path checkpoints /data/models/Stable-diffusion
comfy-1  | Adding extra search path configs /data/models/Stable-diffusion
comfy-1  | Adding extra search path vae /data/models/VAE
comfy-1  | Adding extra search path loras /data/models/Lora
comfy-1  | Adding extra search path upscale_models /data/models/RealESRGAN
comfy-1  | Adding extra search path upscale_models /data/models/ESRGAN
comfy-1  | Adding extra search path upscale_models /data/models/SwinIR
comfy-1  | Adding extra search path upscale_models /data/models/GFPGAN
comfy-1  | Adding extra search path hypernetworks /data/models/hypernetworks
comfy-1  | Adding extra search path controlnet /data/models/ControlNet
comfy-1  | Adding extra search path gligen /data/models/GLIGEN
comfy-1  | Adding extra search path clip /data/models/CLIPEncoder
comfy-1  | Adding extra search path embeddings /data/embeddings
comfy-1  | Adding extra search path unet /data/models/unet
comfy-1  | Adding extra search path clip_vision /data/models/clip_vision
comfy-1  | Adding extra search path xlabs /data/models/xlabs
comfy-1  | Adding extra search path inpaint /data/models/inpaint
comfy-1  | Adding extra search path ipadapter /data/models/ipadapter
comfy-1  | Adding extra search path custom_nodes /data/config/comfy/custom_nodes
comfy-1  | Traceback (most recent call last):
comfy-1  |   File "/stable-diffusion/nodes.py", line 2012, in load_custom_node
comfy-1  |     module_spec.loader.exec_module(module)
comfy-1  |   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
comfy-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
comfy-1  |   File "/data/config/comfy/custom_nodes/comfyui_controlnet_aux/__init__.py", line 2, in <module>
comfy-1  |     from .utils import here, define_preprocessor_inputs, INPUT
comfy-1  |   File "/data/config/comfy/custom_nodes/comfyui_controlnet_aux/utils.py", line 4, in <module>
comfy-1  |     import cv2
comfy-1  |   File "/home/app/.local/lib/python3.10/site-packages/cv2/__init__.py", line 181, in <module>
comfy-1  |     bootstrap()
comfy-1  |   File "/home/app/.local/lib/python3.10/site-packages/cv2/__init__.py", line 153, in bootstrap
comfy-1  |     native_module = importlib.import_module("cv2")
comfy-1  |   File "/opt/conda/lib/python3.10/importlib/__init__.py", line 126, in import_module
comfy-1  |     return _bootstrap._gcd_import(name[level:], package, level)
comfy-1  | ImportError: libGL.so.1: cannot open shared object file: No such file or directory
comfy-1  | 
comfy-1  | Cannot import /data/config/comfy/custom_nodes/comfyui_controlnet_aux module for custom nodes: libGL.so.1: cannot open shared object file: No such file or directory
comfy-1  | 
comfy-1  | Import times for custom nodes:
comfy-1  |    0.0 seconds (IMPORT FAILED): /data/config/comfy/custom_nodes/comfyui_controlnet_aux
comfy-1  |    0.0 seconds: /stable-diffusion/custom_nodes/websocket_image_save.py
comfy-1  |    0.0 seconds: /data/config/comfy/custom_nodes/comfyui-inpaint-nodes
comfy-1  |    0.0 seconds: /data/config/comfy/custom_nodes/ComfyUI_IPAdapter_plus
comfy-1  |    0.1 seconds: /data/config/comfy/custom_nodes/comfyui-tooling-nodes
comfy-1  | 
comfy-1  | Starting server
comfy-1  | 
comfy-1  | To see the GUI go to: http://0.0.0.0:7860
comfy-1  | To see the GUI go to: http://[::]:7860

Here's my config, which I specified directly in the Dockerfile (hope that's not terribly bad practice). Intended use is with Krita + some SDXL checkpoints:

ARG USE_UID=131
ARG USE_GID=131
ARG USE_EDGE=false
ARG USE_GGUF=false
ARG USE_XFLUX=false
ARG USE_CNAUX=true
ARG USE_KRITA=true
ARG USE_IPAPLUS=true
ARG USE_INPAINT=true
ARG USE_TOOLING=true

@lainedfles
Copy link
Author

lainedfles commented Nov 16, 2024

@lainedfles thanks for the quick response!

Edit: this contains good troubleshooting information so I'll leave it but I think I've identified the issue. I'll add a new commit to the PR soon.

@SummitCollie No prob! I've been playing around with Aider.

comfy-1 | Cannot import /data/config/comfy/custom_nodes/comfyui_controlnet_aux module for custom nodes: libGL.so.1: cannot open shared object file: No such file or directory

This is strange. Have you made any modifications aside from env arg defaults? I'd guess a few possible causes:

  • Corrupt image, you might try re-building the image with --no-cache
  • FROM image set to source without libGL support
  • Incorrect LD_LIBRARY_PATH

On my system, /usr/lib/x86_64-linux-gnu/libGL.so.1 is a symbolic link to libGL.so.1.7.0. I'd suggest the following to gain more information:

$ docker exec -it comfy-1 bash -c 'find / -iname libGL.so.1'
$ docker exec -it comfy-1 bash -c 'ls -lh /usr/lib/x86_64-linux-gnu/libGL.so.1'

Here's my config, which I specified directly in the Dockerfile (hope that's not terribly bad practice). Intended use is with Krita + some SDXL checkpoints:

You do you! It looks OK to me but FYI, enabling the single variable USE_KRITA automatically enables USE_CNAUX, USE_IPAPLUS, USE_INPAINT, and USE_TOOLING.

@SummitCollie
Copy link

SummitCollie commented Nov 16, 2024

@lainedfles Ah that was my mistake, I ended up solving it about 1 minute before you responded 😿

I had commented out the second part of this line because the USE_XFLUX thing was preventing build from succeeding (with no useful error). I had inadvertently also commented out the part that installs libgl1-mesa-glx which was the source of my problem.

Ultimately I removed the USE_XFLUX check like below and now it's all working correctly. Thanks for all your help and your work on this PR! I hope it can get merged someday if the base project is un-abandoned!

-RUN apt-get update && apt-get install -y git && ([ "${USE_XFLUX}" = "true" ] && apt-get install -y libgl1-mesa-glx python3-opencv) && apt-get clean
+RUN apt-get update && apt-get install -y git libgl1-mesa-glx python3-opencv && apt-get clean

@lainedfles
Copy link
Author

@SummitCollie good work. I think that the new commit should provide a fix. Just tested Flux.1 Schnell (Have you tried the Flux.1 models? They're incredible!). Here's the result, cheers!

image

@SummitCollie
Copy link

SummitCollie commented Nov 16, 2024

@lainedfles Thanks again! Also btw, KRITA_DOWNLOAD_MODELS didn't work until I changed the script path to remove duplicate scripts/scripts/:

-  if [ "${KRITA_DOWNLOAD_MODELS:-false}" = "true" ]; then
-    cd "${ROOT}/krita-ai-diffusion/scripts" && python scripts/download_models.py --recommended /data && cd ..
+  if [ "${KRITA_DOWNLOAD_MODELS:-true}" = "true" ]; then
+    cd "${ROOT}/krita-ai-diffusion/scripts" && python download_models.py --recommended /data && cd ..

I also changed the default from false to true because I couldn't figure out how to supply build args using docker compose syntax (I am a noob and the example uses podman so... yeah)

@lainedfles
Copy link
Author

@SummitCollie Good catch! Thanks for the help testing this. I'll add a new commit tomorrow to fix that (unless you want to submit a PR to my branch). I'll also fire up my dusty Ubuntu VM tomorrow and try building this with docker compose. I suspect that you may be able to set the environment variables prepended to the command, something like USE_KRITA=true KRITA_DOWNLOAD_MODELS=true docker compose --profile comfyui up --build or possibly docker compose --profile comfyui build --build-arg USE_KRITA=true --build-arg KRITA_DOWNLOAD_MODELS=true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants