SDXL model generate pitch black images.

[walkingclark]

Hi, people,
I love this colab project and benefit great from it. The SD1.5 version works fine for me, but I fail over and over again trying to make SDXL Loras. The result is that it produces pitch black images. Reasons unknown. Could anyone give me suggestions?

This is my training data:

[[datasets]]

[[datasets.subsets]]
num_repeats = 10
image_dir = "/content/drive/MyDrive/Loras/XiaoBu/dataset"

[general]
resolution = 1024
shuffle_caption = true
keep_tokens = 2
flip_aug = false
caption_extension = ".txt"
enable_bucket = true
bucket_no_upscale = false
bucket_reso_steps = 64
min_bucket_reso = 256
max_bucket_reso = 4096
[network_arguments]
unet_lr = 0.0002
text_encoder_lr = 0.0001
network_dim = 32
network_alpha = 16
network_module = "networks.lora"
network_train_unet_only = false

[optimizer_arguments]
learning_rate = 0.0002
lr_scheduler = "cosine_with_restarts"
lr_scheduler_num_cycles = 3
lr_warmup_steps = 150
optimizer_type = "AdamW8bit"
optimizer_args = [ "weight_decay=0.1", "betas=[0.9,0.99]",]

[training_arguments]
pretrained_model_name_or_path = "stabilityai/stable-diffusion-xl-base-1.0"
vae = "stabilityai/sdxl-vae"
max_train_epochs = 10
train_batch_size = 1
seed = 42
max_token_length = 225
xformers = false
sdpa = true
min_snr_gamma = 6
lowram = false
no_half_vae = true
gradient_checkpointing = true
gradient_accumulation_steps = 1
max_data_loader_n_workers = 8
persistent_data_loader_workers = true
mixed_precision = "bf16"
full_bf16 = true
cache_latents = true
cache_latents_to_disk = true
cache_text_encoder_outputs = false
min_timestep = 0
max_timestep = 1000
prior_loss_weight = 1.0
multires_noise_iterations = 6
multires_noise_discount = 0.3

[saving_arguments]
save_precision = "fp16"
save_model_as = "safetensors"
save_every_n_epochs = 1
save_last_n_epochs = 10
output_name = "XiaoBu"
output_dir = "/content/drive/MyDrive/Loras/XiaoBu/output"
log_prefix = "XiaoBu"
logging_dir = "/content/drive/MyDrive/Loras/_logs"
wandb_api_key = ""
log_with = "wandb"

[hollowstrawberry]

SDXL training is more sensitive than SD1 or even Pony. Pay attention to the loss value in the messages while training. If it becomes loss = nan, you trained too long or too hard (so you have to reduce the learning rate or repeats or epochs)

[walkingclark]

This is my training diagram from wandb:

[hollowstrawberry]

Try 1e-4 unet and 1e-5 text encoder. You're essentially burning your lora. This is simply because SDXL training is more sensitive.

[walkingclark]

Humm... I wouldn't know, because it works fine on SD1.5. I guess it's a totally different game. And I know nothing about. So I guess I can't quite change the settings. The freedom is very limited.