although folder preparation button is resumed, image folder still does not work

[jjiikkkk]

see, I do not know what it is wrong

Traceback (most recent call last):
File "/kaggle/working/kohya_ss/sd-scripts/train_network.py", line 1115, in
trainer.train(args)
File "/kaggle/working/kohya_ss/sd-scripts/train_network.py", line 234, in train
model_version, text_encoder, vae, unet = self.load_target_model(args, weight_dtype, accelerator)
File "/kaggle/working/kohya_ss/sd-scripts/train_network.py", line 101, in load_target_model
text_encoder, vae, unet, _ = train_util.load_target_model(args, weight_dtype, accelerator)
File "/kaggle/working/kohya_ss/sd-scripts/library/train_util.py", line 4387, in load_target_model
text_encoder, vae, unet, load_stable_diffusion_format = _load_target_model(
File "/kaggle/working/kohya_ss/sd-scripts/library/train_util.py", line 4363, in _load_target_model
original_unet = UNet2DConditionModel(
File "/kaggle/working/kohya_ss/sd-scripts/library/original_unet.py", line 1427, in init
attn_num_head_channels=attention_head_dim[i],
IndexError: list index out of range
[2024-04-09 05:08:36,532] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 1114) of binary: /opt/conda/bin/python
Traceback (most recent call last):
File "/opt/conda/bin/accelerate", line 8, in
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
args.func(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1008, in launch_command
multi_gpu_launcher(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 666, in multi_gpu_launcher
distrib_run.run(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

/kaggle/working/kohya_ss/sd-scripts/train_network.py FAILED

Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time : 2024-04-09_05:08:36
host : c9fb5313e204
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 1114)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

[jjiikkkk]

seeming the same errors like #2244

[bmaltais]

Not sure what cause this error… it is caused by the training script and I don’t think I can do anything about it. You might want to open an issue directly on the as-scripts repo.

[jjiikkkk]

okay

[jjiikkkk]

File "/kaggle/working/kohya_ss/sd-scripts/sdxl_train_network.py", line 185, in
trainer.train(args)
File "/kaggle/working/kohya_ss/sd-scripts/train_network.py", line 272, in train
train_dataset_group.cache_latents(vae, args.vae_batch_size, args.cache_latents_to_disk, accelerator.is_main_process)
File "/kaggle/working/kohya_ss/sd-scripts/library/train_util.py", line 2080, in cache_latents
dataset.cache_latents(vae, vae_batch_size, cache_to_disk, is_main_process)
File "/kaggle/working/kohya_ss/sd-scripts/library/train_util.py", line 1023, in cache_latents
cache_batch_latents(vae, cache_to_disk, batch, subset.flip_aug, subset.random_crop)
File "/kaggle/working/kohya_ss/sd-scripts/library/train_util.py", line 2428, in cache_batch_latents
raise RuntimeError(f"NaN detected in latents: {info.absolute_path}")
RuntimeError: NaN detected in latents: /kaggle/working/results/img/25_ohwx tanglaoya/1 (1)_resized.png
[2024-04-11 07:13:14,425] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 1115) of binary: /opt/conda/bin/python
Traceback (most recent call last):
File "/opt/conda/bin/accelerate", line 8, in
sys.exit(main())
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
args.func(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1008, in launch_command
multi_gpu_launcher(args)
File "/opt/conda/lib/python3.10/site-packages/accelerate/commands/launch.py", line 666, in multi_gpu_launcher
distrib_run.run(args)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

/kaggle/working/kohya_ss/sd-scripts/sdxl_train_network.py FAILED， I checked several threads here and found out some people come across the same problem with me, so really puzzled

[attashe]

Same error, launched train_network.py directly without guid

[attashe]

Same error, launched train_network.py directly without guid

It was a stupid mistake, I forgot to change the script name to sdxl_ version