ModuleNotFoundError: import of xformers halted; None in sys.modules #1154

SamSilmarilData · 2023-05-24T21:34:23Z

SamSilmarilData
May 24, 2023

i get this error when i start dreambooth training. i'm running on ubuntu 22.04, cuda version 11.8

what i get when launching automatic:

$ automatic/webui.sh --share --insecure --upgrade
Create and activate python venv
Launching launch.py...
21:01:55-805540 INFO     Running extension preloading                                                          
21:01:55-911536 INFO     Starting SD.Next                                                                      
21:01:55-912733 INFO     Python 3.10.6 on Linux                                                                
21:01:55-920205 INFO     Version: 9e66d88e Wed May 24 15:21:49 2023 -0400                                      
21:01:56-101116 INFO     Setting environment tuning                                                            
21:01:56-102380 INFO     nVidia CUDA toolkit detected                                                          
21:01:57-510536 INFO     Torch 2.0.1+cu118                                                                     
21:01:57-534355 INFO     Torch backend: nVidia CUDA 11.8 cuDNN 8700                                            
21:01:57-551275 INFO     Torch detected GPU: Tesla V100-SXM2-16GB VRAM 16161 Arch (7, 0) Cores 80              
21:01:57-553348 INFO     Verifying requirements                                                                
21:01:57-569527 WARNING  Package wrong version: tqdm 4.64.1 required 4.65.0                                    
21:01:57-570546 INFO     Installing package: tqdm==4.65.0                                                      
21:01:59-910126 WARNING  Package wrong version: diffusers 0.14.0 required 0.16.1                               
21:01:59-911357 INFO     Installing package: diffusers==0.16.1                                                 
21:02:02-798498 INFO     Installing packages                                                                   
21:02:02-803979 INFO     Running setup                                                                         
21:02:02-805192 INFO     Installing repositories                                                               
21:02:03-037145 INFO     Installing submodules                                                                 
21:02:03-406516 INFO     Updating submodules                                                                   
21:02:37-524851 INFO     Extension installed packages: sd_dreambooth_extension ['tqdm==4.64.1',                
                         'diffusers==0.14.0']                                                                  
21:02:37-526349 INFO     Extensions enabled: ['stable-diffusion-webui-images-browser', 'SwinIR',               
                         'sd-webui-controlnet', 'a1111-sd-webui-lycoris', 'sd-extension-system-info',          
                         'prompt-bracket-checker', 'sd-dynamic-thresholding', 'sd-webui-model-converter',      
                         'clip-interrogator-ext', 'ScuNET', 'Lora', 'stable-diffusion-webui-rembg', 'LDSR',    
                         'multidiffusion-upscaler-for-automatic1111', 'sd_dreambooth_extension']               
21:02:37-527907 INFO     Updating Wiki                                                                         
21:02:37-768149 INFO     Running extension preloading                                                          
21:02:37-769379 INFO     Server arguments: ['--share', '--insecure', '--upgrade']                              
21:02:42-970696 INFO     Unloading xFormers                                                                   
21:02:43-494630 INFO     Libraries loaded                                                                      
21:02:43-496150 INFO     Using data path: /home/jolin18211/automatic                                           
21:02:43-498506 INFO     Available VAEs: /home/jolin18211/automatic/models/VAE 1                               
21:02:43-505274 INFO     Available models: /home/jolin18211/automatic/models/Stable-diffusion 2                
ControlNet v1.1.190
ControlNet v1.1.190
Image Browser: ImageReward is not installed, cannot be used.
21:02:45-530769 INFO     Setting Torch parameters: dtype=torch.float16 vae=torch.float16 unet=torch.float16    
Loading weights: /home/jolin18211/automatic/models/Stable-diffusion/v1-5-pruned-emaonly.safetensors ━━━ 0… -:-…
                                                                                                        GB     
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
21:02:46-668005 INFO     Model created from config: /home/jolin18211/automatic/configs/v1-inference.yaml       
21:02:48-561842 INFO     Applying scaled dot product cross attention optimization                              
21:02:48-592265 INFO     Embeddings loaded: 0 []                                                               
21:02:48-599591 INFO     Model loaded in 3.1s (load=0.6s create=0.6s apply=0.8s vae=0.5s move=0.6s)            
21:02:48-878688 INFO     Model load finished: {'ram': {'used': 7.63, 'total': 29.38}, 'gpu': {'used': 2.39,    
                         'total': 15.78}, 'retries': 0, 'oom': 0}                                              
21:02:49-465117 INFO     Loading UI theme: name=black-orange style=Auto                                        
Create LRU cache (max_size=16) for preprocessor results.
Create LRU cache (max_size=16) for preprocessor results.
21:02:50-104321 INFO     Available models: /home/jolin18211/automatic/models/Stable-diffusion 2                
Running on local URL:  http://127.0.0.1:7860
Running on public URL: https://c9d09a915b4cb28133.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces
21:02:52-199067 INFO     Local URL: http://127.0.0.1:7860/                                                     
21:02:52-200371 INFO     API Docs: http://127.0.0.1:7860/docs                                                  
21:02:52-201359 INFO     Share URL: https://c9d09a915b4cb28133.gradio.live                                     
21:02:52-202286 INFO     Initializing middleware                                                               
Create LRU cache (max_size=16) for preprocessor results.
21:02:52-952923 INFO     Startup time: 15.2s (torch=2.8s gradio=2.4s libraries=0.5s scripts=5.9s ui=0.9s       
                         launch=1.9s scripts app_started_callback=0.1s checkpoint=0.7s)                        
21:02:52-955021 INFO     Memory used: 3.73 total: 29.38

these are my settings:

i've experimented with different attention options e.g. defualt, xformers,this.

i've experimented with xformers, sdp

this is the output i'm getting:

Total images: 12
Largest prime: 3
Best factors: (3, 1)
Total VRAM: 16
Total images: 12
Largest prime: 3
Best factors: (3, 1)
Total VRAM: 16
Custom model name is ohwx_test
Replacing flash attention in config to xformers
Initializing dreambooth training...
The version of diffusers is greater than 0.14.0, hopefully they merged the PR by now
Pre-processing images: classifiers_0: : 0it [00:00, ?it/s]                                                     
Nothing to generate.
Traceback (most recent call last):
  File "/home/jolin18211/automatic/extensions/sd_dreambooth_extension/dreambooth/ui_functions.py", line 727, in start_training
    result = main(class_gen_method=class_gen_method)
  File "/home/jolin18211/automatic/extensions/sd_dreambooth_extension/dreambooth/train_dreambooth.py", line 1371, in main
    return inner_loop()
  File "/home/jolin18211/automatic/extensions/sd_dreambooth_extension/dreambooth/memory.py", line 119, in decorator
    return function(batch_size, grad_size, prof, *args, **kwargs)
  File "/home/jolin18211/automatic/extensions/sd_dreambooth_extension/dreambooth/train_dreambooth.py", line 297, in inner_loop
    import xformers
ModuleNotFoundError: import of xformers halted; None in sys.modules
Restored system models.
Duration: 00:00:06

i have checked if xformers is actually installed:

jolin18211@instance-1:~/automatic$ source venv/bin/activate
(venv) jolin18211@instance-1:~/automatic$ pip show xformers
Name: xformers
Version: 0.0.20
Summary: XFormers: A collection of composable Transformer building blocks.
Home-page: https://facebookresearch.github.io/xformers/
Author: Facebook AI Research
Author-email: [email protected]
License: 
Location: /home/jolin18211/automatic/venv/lib/python3.10/site-packages
Requires: numpy, pyre-extensions, torch
Required-by: 
(venv) jolin18211@instance-1:~/automatic$ deactivate

vladmandic · 2023-05-24T22:16:52Z

vladmandic
May 24, 2023
Maintainer

Unless you specify xformers as desired cross attention method they will be force unloaded even if you have them installed. So just select then!

7 replies

vladmandic May 24, 2023
Maintainer

Just select then as active it UI!
Settings - stable diffusion - cross attention - xformers.

SamSilmarilData May 25, 2023
Author

sorry! but now i'm getting different kind of errors:
this is normally:

 Custom model name is ohwx_test
Initializing dreambooth training...
The version of diffusers is less than or equal to 0.14.0. Performing monkey-patch...
Pre-processing images: classifiers_0: : 0it [00:00, ?it/s]                                                     
Nothing to generate.
Found 60 reg images.
Preparing dataset...
Init dataset!
Preparing Dataset (With Caching)
Caching latents...:   0%|                                                               | 0/72 [00:00<?, ?it/s]Loading cached latents...
Bucket 0 (768, 768, 0) - Instance Images: 12 | Class Images: 60 | Max Examples/batch:  24                      
Total Buckets 1 - Instance Images: 12 | Class Images: 60 | Max Examples/batch: 24                              
Caching latents...: 100%|███████████████████████████████████████████████████| 72/72 [00:00<00:00, 18553.17it/s]
Total images / batch: 24, total examples: 24
Caching latents...: 100%|███████████████████████████████████████████████████| 72/72 [00:00<00:00, 17829.13it/s]
Total dataset length (steps): 24
Initializing bucket counter!
  ***** Running training *****
  Num batches each epoch = 24
  Num Epochs = 100
  Batch Size Per Device = 1
  Gradient Accumulation steps = 1
  Total train batch size (w. parallel, distributed & accumulation) = 1
  Text Encoder Epochs: 100
  Total optimization steps = 1200
  Total training steps = 2400
  Resuming from checkpoint: False
  First resume epoch: 0
  First resume step: 0
  Lora: False, Optimizer: 8bit AdamW, Prec: fp16
  Gradient Checkpointing: True
  EMA: False
  UNET: False
  Freeze CLIP Normalization Layers: False
  LR: 2e-06
  V2: False
Steps:   0%|                                                                          | 0/2400 [00:00<?, ?it/s][2023-05-25 05:23:36,212] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-05-25 05:23:37,290] torch._dynamo.convert_frame: [INFO] converting frame raised unsupported, leaving it unconverted
[2023-05-25 05:23:37,291] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing config
[2023-05-25 05:23:37,293] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-05-25 05:23:37,320] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-05-25 05:23:37,322] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function debug_wrapper
[2023-05-25 05:23:38,683] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 0
[2023-05-25 05:23:39,210] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 0
[2023-05-25 05:23:39,211] torch._dynamo.output_graph: [INFO] Step 2: done compiler function debug_wrapper
[2023-05-25 05:23:39,831] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing dtype
[2023-05-25 05:23:40,707] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing get_parameter_dtype
[2023-05-25 05:23:41,566] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-05-25 05:23:41,596] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-05-25 05:23:41,597] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function debug_wrapper
[2023-05-25 05:23:41,684] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 1
python3: /root/.triton/llvm/llvm+mlir-14.0.6-x86_64-linux-gnu-centos-7-release/include/llvm/ADT/SmallVector.h:273: T& llvm::SmallVectorTemplateCommon<T, <template-parameter-1-2> >::operator[](llvm::SmallVectorTemplateCommon<T, <template-parameter-1-2> >::size_type) [with T = mlir::Value; <template-parameter-1-2> = void; llvm::SmallVectorTemplateCommon<T, <template-parameter-1-2> >::reference = mlir::Value&; llvm::SmallVectorTemplateCommon<T, <template-parameter-1-2> >::size_type = long unsigned int]: Assertion `idx < size()' failed.
Aborted (core dumped)

this is when i use lora training with lora extended:

Custom model name is ohwx_test
Initializing dreambooth training...
The version of diffusers is less than or equal to 0.14.0. Performing monkey-patch...
Pre-processing images: classifiers_0: : 0it [00:00, ?it/s]                                                     
Nothing to generate.
Found 60 reg images.
Preparing dataset...
Init dataset!
Preparing Dataset (With Caching)
Caching latents...:   0%|                                                               | 0/72 [00:00<?, ?it/s]Loading cached latents...
Bucket 0 (768, 768, 0) - Instance Images: 12 | Class Images: 60 | Max Examples/batch:  24                      
Total Buckets 1 - Instance Images: 12 | Class Images: 60 | Max Examples/batch: 24                              
Caching latents...: 100%|███████████████████████████████████████████████████| 72/72 [00:00<00:00, 17156.57it/s]
Total images / batch: 24, total examples: 24
Caching latents...: 100%|███████████████████████████████████████████████████| 72/72 [00:00<00:00, 16419.63it/s]
Total dataset length (steps): 24
Initializing bucket counter!
  ***** Running training *****
  Num batches each epoch = 24
  Num Epochs = 100
  Batch Size Per Device = 1
  Gradient Accumulation steps = 1
  Total train batch size (w. parallel, distributed & accumulation) = 1
  Text Encoder Epochs: 100
  Total optimization steps = 1200
  Total training steps = 2400
  Resuming from checkpoint: False
  First resume epoch: 0
  First resume step: 0
  Lora: True, Optimizer: 8bit AdamW, Prec: fp16
  Gradient Checkpointing: True
  EMA: False
  UNET: False
  Freeze CLIP Normalization Layers: False
  LR: 0.0001
  LoRA Extended: True
  LoRA Text Encoder LR: 5e-05
  V2: False
Steps:   0%|                                                                          | 0/2400 [00:00<?, ?it/s][2023-05-25 05:41:48,684] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-05-25 05:41:50,452] torch._dynamo.convert_frame: [INFO] converting frame raised unsupported, leaving it unconverted
[2023-05-25 05:41:50,453] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing config
[2023-05-25 05:41:50,455] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-05-25 05:41:50,480] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-05-25 05:41:50,482] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function debug_wrapper
[2023-05-25 05:41:51,869] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 0
[2023-05-25 05:41:52,030] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 0
[2023-05-25 05:41:52,031] torch._dynamo.output_graph: [INFO] Step 2: done compiler function debug_wrapper
[2023-05-25 05:41:52,687] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing dtype
[2023-05-25 05:41:54,235] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing get_parameter_dtype
[2023-05-25 05:41:55,743] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-05-25 05:41:55,773] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-05-25 05:41:55,774] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function debug_wrapper
[2023-05-25 05:41:55,863] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 1
loc("/tmp/torchinductor_jolin18211/triton/0/f7d65e5f4214fa0268fc8ca0eeb87c2d/triton_mm.ttgir":134:22): error: use of value '%54' expects different type than prior uses: 'tensor<64x64xf32, #triton_gpu.mma<{versionMajor = 1, versionMinor = 0, warpsPerCTA = [4, 1]}>>' vs 'tensor<64x64xf32, #triton_gpu.mma<{versionMajor = 1, versionMinor = 1, warpsPerCTA = [2, 2]}>>'
Segmentation fault (core dumped)

also, is the output belowSteps: 0%| | 0/2400 [00:00<?, ?it/s][2023-05-25 05:41:48,684] suppose to show?

vladmandic May 25, 2023
Maintainer

first check if by accident you turned on this setting: UI -> Settings -> Compute Settings -> Enable model compile (experimental)
it should be disabled.

if its already disabled, then its dreambooth extension itself that is trying to do stuff on its own and you should report to extension github.

SamSilmarilData May 26, 2023
Author

i was able to debug this part:

[2023-05-25 05:41:50,452] torch._dynamo.convert_frame: [INFO] converting frame raised unsupported, leaving it unconverted
[2023-05-25 05:41:50,453] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing config
[2023-05-25 05:41:50,455] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-05-25 05:41:50,480] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-05-25 05:41:50,482] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function debug_wrapper
[2023-05-25 05:41:51,869] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 0
[2023-05-25 05:41:52,030] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 0
[2023-05-25 05:41:52,031] torch._dynamo.output_graph: [INFO] Step 2: done compiler function debug_wrapper
[2023-05-25 05:41:52,687] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing dtype
[2023-05-25 05:41:54,235] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing get_parameter_dtype
[2023-05-25 05:41:55,743] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-05-25 05:41:55,773] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-05-25 05:41:55,774] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function debug_wrapper
[2023-05-25 05:41:55,863] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 1

i had to install cudnn for that. but this part remains:

loc("/tmp/torchinductor_jolin18211/triton/0/f7d65e5f4214fa0268fc8ca0eeb87c2d/triton_mm.ttgir":134:22): error: use of value '%54' expects different type than prior uses: 'tensor<64x64xf32, #triton_gpu.mma<{versionMajor = 1, versionMinor = 0, warpsPerCTA = [4, 1]}>>' vs 'tensor<64x64xf32, #triton_gpu.mma<{versionMajor = 1, versionMinor = 1, warpsPerCTA = [2, 2]}>>'
Segmentation fault (core dumped)

first check if by accident you turned on this setting: UI -> Settings -> Compute Settings -> Enable model compile (experimental)
it should be disabled.

its disabled. i checked.

if its already disabled, then its dreambooth extension itself that is trying to do stuff on its own and you should report to extension github.

probably. i found a similar existing issue on there, but its stale. i put up a discussion there. it'd be awesome if anyone could help me out here. the sd_dreambooth_extension is having a ton of problems relating to memory and this.

SamSilmarilData Jun 6, 2023
Author

alright i found a fix.
turns out it was a fault with the tensorrt libraries. in particular, if you activated python and tried to import tensorrt, it would show errors, that libcudnn.so files could not be found.
so i copied the whole tensorrt_libs folder and nvidia folder from my local dist-packages and pasted into my venv site packages and the the error persists no more.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ModuleNotFoundError: import of xformers halted; None in sys.modules #1154

{{title}}

Replies: 1 comment 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

ModuleNotFoundError: import of xformers halted; None in sys.modules #1154

SamSilmarilData May 24, 2023

Replies: 1 comment · 7 replies

vladmandic May 24, 2023 Maintainer

vladmandic May 24, 2023 Maintainer

SamSilmarilData May 25, 2023 Author

vladmandic May 25, 2023 Maintainer

SamSilmarilData May 26, 2023 Author

SamSilmarilData Jun 6, 2023 Author

SamSilmarilData
May 24, 2023

Replies: 1 comment 7 replies

vladmandic
May 24, 2023
Maintainer

vladmandic May 24, 2023
Maintainer

SamSilmarilData May 25, 2023
Author

vladmandic May 25, 2023
Maintainer

SamSilmarilData May 26, 2023
Author

SamSilmarilData Jun 6, 2023
Author