Is there a Paperspace version?

[CHJ85]

Hi. So Paperspace is the new kid on the block and a pretty good Colab alternative. In fact, much better than Colab in my opinion.
Could you please consider it?
Thanks.

[CHJ85]

I've put together a rough and dirty Paperspace version. But I need some help.
I'm getting this error related to Hugging Face's accelerate library:

FileNotFoundError: [Errno 2] No such file or directory: 
'/root/.cache/huggingface/accelerate/default_config.yaml'

Here's what I got so far. Idk how to obtain this file:

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "rmCPmqFL6hCQ"
   },
   "source": [
    "# ⭐ Lora Trainer by Hollowstrawberry\n",
    "\n",
    "This is based on the work of [Kohya-ss](https://github.com/kohya-ss/sd-scripts) and [Linaqruf](https://colab.research.google.com/github/Linaqruf/kohya-trainer/blob/main/kohya-LoRA-dreambooth.ipynb). Thank you!\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "vJ8clWTZEu-g"
   },
   "source": [
    "### ⭕ Disclaimer\n",
    "The purpose of this document is to research bleeding-edge technologies in the field of machine learning inference.  \n",
    "Please read and follow the [Google Colab guidelines](https://research.google.com/colaboratory/faq.html) and its [Terms of Service](https://research.google.com/colaboratory/tos_v3.html)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "dPQlB4djNm3C"
   },
   "source": [
    "| |GitHub|🇬🇧 English|🇪🇸 Spanish|\n",
    "|:--|:-:|:-:|:-:|\n",
    "| 🏠 **Homepage** | [![GitHub](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/github.svg)](https://github.com/hollowstrawberry/kohya-colab) | | |\n",
    "| 📊 **Dataset Maker** | [![GitHub](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/github.svg)](https://github.com/hollowstrawberry/kohya-colab/blob/main/Dataset_Maker.ipynb) | [![Open in Colab](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/colab-badge.svg)](https://colab.research.google.com/github/hollowstrawberry/kohya-colab/blob/main/Dataset_Maker.ipynb) | [![Abrir en Colab](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/colab-badge-spanish.svg)](https://colab.research.google.com/github/hollowstrawberry/kohya-colab/blob/main/Spanish_Dataset_Maker.ipynb) |\n",
    "| ⭐ **Lora Trainer** | [![GitHub](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/github.svg)](https://github.com/hollowstrawberry/kohya-colab/blob/main/Lora_Trainer.ipynb) | [![Open in Colab](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/colab-badge.svg)](https://colab.research.google.com/github/hollowstrawberry/kohya-colab/blob/main/Lora_Trainer.ipynb) | [![Abrir en Colab](https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/assets/colab-badge-spanish.svg)](https://colab.research.google.com/github/hollowstrawberry/kohya-colab/blob/main/Spanish_Lora_Trainer.ipynb) |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "id": "OglZzI_ujZq-"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "import toml\n",
    "import shutil\n",
    "import zipfile\n",
    "from time import time\n",
    "from IPython.display import Markdown, display\n",
    "\n",
    "# These carry information from past executions\n",
    "if \"model_url\" in globals():\n",
    "  old_model_url = model_url\n",
    "else:\n",
    "  old_model_url = None\n",
    "if \"dependencies_installed\" not in globals():\n",
    "  dependencies_installed = False\n",
    "if \"model_file\" not in globals():\n",
    "  model_file = None\n",
    "\n",
    "# These may be set by other cells, some are legacy\n",
    "if \"custom_dataset\" not in globals():\n",
    "  custom_dataset = None\n",
    "if \"override_dataset_config_file\" not in globals():\n",
    "  override_dataset_config_file = None\n",
    "if \"override_config_file\" not in globals():\n",
    "  override_config_file = None\n",
    "if \"optimizer\" not in globals():\n",
    "  optimizer = \"AdamW8bit\"\n",
    "if \"optimizer_args\" not in globals():\n",
    "  optimizer_args = None\n",
    "if \"continue_from_lora\" not in globals():\n",
    "  continue_from_lora = \"\"\n",
    "if \"weighted_captions\" not in globals():\n",
    "  weighted_captions = False\n",
    "if \"adjust_tags\" not in globals():\n",
    "  adjust_tags = False\n",
    "if \"keep_tokens_weight\" not in globals():\n",
    "  keep_tokens_weight = 1.0\n",
    "\n",
    "COLAB = False # low ram\n",
    "COMMIT = \"e6ad3cbc66130fdc3bf9ecd1e0272969b1d613f7\"\n",
    "BETTER_EPOCH_NAMES = True\n",
    "LOAD_TRUNCATED_IMAGES = True\n",
    "\n",
    "\n",
    "project_name = \"\" #@param {type:\"string\"}\n",
    "folder_structure = \"Organize by project (/Loras/project_name/dataset)\" #@param [\"Organize by category (/lora_training/datasets/project_name)\", \"Organize by project (/Loras/project_name/dataset)\"]\n",
    "training_model = \"Stable Diffusion (sd-v1-5-pruned-noema-fp16.safetensors)\" #@param [\"Anime (animefull-final-pruned-fp16.safetensors)\", \"AnyLora (AnyLoRA_noVae_fp16-pruned.ckpt)\", \"Stable Diffusion (sd-v1-5-pruned-noema-fp16.safetensors)\"]\n",
    "optional_custom_training_model_url = \"\" #@param {type:\"string\"}\n",
    "custom_model_is_based_on_sd2 = False #@param {type:\"boolean\"}\n",
    "\n",
    "if optional_custom_training_model_url:\n",
    "  model_url = optional_custom_training_model_url\n",
    "elif \"AnyLora\" in training_model:\n",
    "  model_url = \"https://huggingface.co/Lykon/AnyLoRA/resolve/main/AnyLoRA_noVae_fp16-pruned.ckpt\"\n",
    "elif \"Anime\" in training_model:\n",
    "  model_url = \"https://huggingface.co/hollowstrawberry/stable-diffusion-guide/resolve/main/models/animefull-final-pruned-fp16.safetensors\"\n",
    "else:\n",
    "  model_url = \"https://huggingface.co/hollowstrawberry/stable-diffusion-guide/resolve/main/models/sd-v1-5-pruned-noema-fp16.safetensors\"\n",
    "\n",
    "resolution = 768 #@param {type:\"slider\", min:512, max:1024, step:128}\n",
    "flip_aug = False #@param {type:\"boolean\"}\n",
    "#markdown Leave empty for no captions.\n",
    "caption_extension = \".txt\" #param {type:\"string\"}\n",
    "shuffle_tags = True #@param {type:\"boolean\"}\n",
    "shuffle_caption = shuffle_tags\n",
    "activation_tags = \"1\" #@param [0,1,2,3]\n",
    "keep_tokens = int(activation_tags)\n",
    "\n",
    "num_repeats = 10 #@param {type:\"number\"}\n",
    "preferred_unit = \"Steps\" #@param [\"Epochs\", \"Steps\"]\n",
    "how_many = 6000 #@param {type:\"number\"}\n",
    "max_train_epochs = how_many if preferred_unit == \"Epochs\" else None\n",
    "max_train_steps = how_many if preferred_unit == \"Steps\" else None\n",
    "save_every_n_epochs = 1 #@param {type:\"number\"}\n",
    "keep_only_last_n_epochs = 10 #@param {type:\"number\"}\n",
    "if not save_every_n_epochs:\n",
    "  save_every_n_epochs = max_train_epochs\n",
    "if not keep_only_last_n_epochs:\n",
    "  keep_only_last_n_epochs = max_train_epochs\n",
    "train_batch_size = 2 #@param {type:\"slider\", min:1, max:8, step:1}\n",
    "\n",
    "unet_lr = 5e-4 #@param {type:\"number\"}\n",
    "text_encoder_lr = 1e-4 #@param {type:\"number\"}\n",
    "lr_scheduler = \"cosine_with_restarts\" #@param [\"constant\", \"cosine\", \"cosine_with_restarts\", \"constant_with_warmup\", \"linear\", \"polynomial\"]\n",
    "lr_scheduler_number = 3 #@param {type:\"number\"}\n",
    "lr_scheduler_num_cycles = lr_scheduler_number if lr_scheduler == \"cosine_with_restarts\" else 0\n",
    "lr_scheduler_power = lr_scheduler_number if lr_scheduler == \"polynomial\" else 0\n",
    "lr_warmup_ratio = 0.05 #@param {type:\"slider\", min:0.0, max:0.5, step:0.01}\n",
    "lr_warmup_steps = 0\n",
    "min_snr_gamma = True #@param {type:\"boolean\"}\n",
    "min_snr_gamma_value = 5.0 if min_snr_gamma else None\n",
    "\n",
    "lora_type = \"LoRA\" #@param [\"LoRA\", \"LoCon Lycoris\", \"LoHa Lycoris\"]\n",
    "\n",
    "\n",
    "\n",
    "network_dim = 16 #@param {type:\"slider\", min:1, max:128, step:1}\n",
    "network_alpha = 8 #@param {type:\"slider\", min:1, max:128, step:1}\n",
    "conv_dim = 8 #@param {type:\"slider\", min:1, max:64, step:1}\n",
    "conv_alpha = 1 #@param {type:\"slider\", min:1, max:64, step:1}\n",
    "conv_compression = False #@param {type:\"boolean\"}\n",
    "\n",
    "network_module = \"lycoris.kohya\" if \"Lycoris\" in lora_type else \"networks.lora\"\n",
    "network_args = None if lora_type == \"LoRA\" else [\n",
    "  f\"conv_dim={conv_dim}\",\n",
    "  f\"conv_alpha={conv_alpha}\",\n",
    "]\n",
    "if \"Lycoris\" in lora_type:\n",
    "  network_args.append(f\"algo={'loha' if 'LoHa' in lora_type else 'lora'}\")\n",
    "  network_args.append(f\"disable_conv_cp={str(not conv_compression)}\")\n",
    "\n",
    "#markdown ### ▶️ Experimental\n",
    "#markdown Save additional data equaling ~1 GB allowing you to resume training later.\n",
    "save_state = False #param {type:\"boolean\"}\n",
    "#markdown Resume training if a save state is found.\n",
    "resume = False #param {type:\"boolean\"}\n",
    "\n",
    "\n",
    "\n",
    "# 👩‍💻 Cool code goes here\n",
    "\n",
    "if optimizer == \"DAdaptation\":\n",
    "  optimizer_args = [\"decouple=True\",\"weight_decay=0.02\",\"betas=[0.9,0.99]\"]\n",
    "  unet_lr = 0.5\n",
    "  text_encoder_lr = 0.5\n",
    "  lr_scheduler = \"constant_with_warmup\"\n",
    "  network_alpha = network_dim\n",
    "\n",
    "root_dir = \"/notebook\" if COLAB else \"/notebooks/Loras\"\n",
    "deps_dir = os.path.join(root_dir, \"deps\")\n",
    "repo_dir = os.path.join(root_dir, \"kohya-trainer\")\n",
    "\n",
    "if \"/Loras\" in folder_structure:\n",
    "  main_dir      = os.path.join(root_dir, \"/notebooks/Loras\") if COLAB else root_dir\n",
    "  log_folder    = os.path.join(main_dir, \"_logs\")\n",
    "  config_folder = os.path.join(main_dir, project_name)\n",
    "  images_folder = os.path.join(main_dir, project_name, \"dataset\")\n",
    "  output_folder = os.path.join(main_dir, project_name, \"output\")\n",
    "else:\n",
    "  main_dir      = os.path.join(root_dir, \"/notebooks/lora_training\") if COLAB else root_dir\n",
    "  images_folder = os.path.join(main_dir, \"datasets\", project_name)\n",
    "  output_folder = os.path.join(main_dir, \"output\", project_name)\n",
    "  config_folder = os.path.join(main_dir, \"config\", project_name)\n",
    "  log_folder    = os.path.join(main_dir, \"log\")\n",
    "\n",
    "config_file = os.path.join(config_folder, \"training_config.toml\")\n",
    "dataset_config_file = os.path.join(config_folder, \"dataset_config.toml\")\n",
    "accelerate_config_file = os.path.join(repo_dir, \"accelerate_config/config.yaml\")\n",
    "\n",
    "def clone_repo():\n",
    "  os.chdir(root_dir)\n",
    "  !git clone https://github.com/kohya-ss/sd-scripts {repo_dir}\n",
    "  os.chdir(repo_dir)\n",
    "  if COMMIT:\n",
    "    !git reset --hard {COMMIT}\n",
    "  !wget https://raw.githubusercontent.com/hollowstrawberry/kohya-colab/main/requirements.txt -q -O requirements.txt\n",
    "\n",
    "def install_dependencies():\n",
    "  clone_repo()\n",
    "  !apt -y update -qq\n",
    "  !apt -y install aria2 -qq\n",
    "  !pip -q install toml\n",
    "  !pip -q install --upgrade -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu118\n",
    "\n",
    "  # patch kohya for minor stuff\n",
    "  if COLAB:\n",
    "    !sed -i \"s@cpu@cuda@\" library/model_util.py # low ram\n",
    "  if LOAD_TRUNCATED_IMAGES:\n",
    "    !sed -i 's/from PIL import Image/from PIL import Image, ImageFile\\nImageFile.LOAD_TRUNCATED_IMAGES=True/g' library/train_util.py # fix truncated jpegs error\n",
    "  if BETTER_EPOCH_NAMES:\n",
    "    !sed -i 's/{:06d}/{:02d}/g' library/train_util.py # make epoch names shorter\n",
    "    !sed -i 's/\".\" + args.save_model_as)/\"-{:02d}.\".format(num_train_epochs) + args.save_model_as)/g' train_network.py # name of the last epoch will match the rest\n",
    "\n",
    "  from accelerate.utils import write_basic_config\n",
    "  if not os.path.exists(accelerate_config_file):\n",
    "    write_basic_config(save_location=accelerate_config_file)\n",
    "\n",
    "  os.environ[\"TF_CPP_MIN_LOG_LEVEL\"] = \"3\"\n",
    "  os.environ[\"BITSANDBYTES_NOWELCOME\"] = \"1\"  \n",
    "  os.environ[\"SAFETENSORS_FAST_GPU\"] = \"1\"\n",
    "\n",
    "def validate_dataset():\n",
    "  global lr_warmup_steps, lr_warmup_ratio, caption_extension, keep_tokens, keep_tokens_weight, weighted_captions, adjust_tags\n",
    "  supported_types = (\".png\", \".jpg\", \".jpeg\", \".webp\", \".bmp\")\n",
    "\n",
    "  print(\"\\n💿 Checking dataset...\")\n",
    "  if not project_name.strip() or any(c in project_name for c in \" .()\\\"'\\\\/\"):\n",
    "    print(\"💥 Error: Please choose a valid project name.\")\n",
    "    return\n",
    "\n",
    "  if custom_dataset:\n",
    "    try:\n",
    "      datconf = toml.loads(custom_dataset)\n",
    "      datasets = [d for d in datconf[\"datasets\"][0][\"subsets\"]]\n",
    "    except:\n",
    "      print(f\"💥 Error: Your custom dataset is invalid or contains an error! Please check the original template.\")\n",
    "      return\n",
    "    reg = [d for d in datasets if d.get(\"is_reg\", False)]\n",
    "    for r in reg:\n",
    "      print(\"📁\"+r[\"image_dir\"].replace(\"/notebooks/\", \"\") + \" (Regularization)\")\n",
    "    datasets = [d for d in datasets if d not in reg]\n",
    "    datasets_dict = {d[\"image_dir\"]: d[\"num_repeats\"] for d in datasets}\n",
    "    folders = datasets_dict.keys()\n",
    "    files = [f for folder in folders for f in os.listdir(folder)]\n",
    "    images_repeats = {folder: (len([f for f in os.listdir(folder) if f.lower().endswith(supported_types)]), datasets_dict[folder]) for folder in folders}\n",
    "  else:\n",
    "    folders = [images_folder]\n",
    "    files = os.listdir(images_folder)\n",
    "    images_repeats = {images_folder: (len([f for f in files if f.lower().endswith(supported_types)]), num_repeats)}\n",
    "\n",
    "  for folder in folders:\n",
    "    if not os.path.exists(folder):\n",
    "      print(f\"💥 Error: The folder {folder.replace('/notebooks/', '')} doesn't exist.\")\n",
    "      return\n",
    "  for folder, (img, rep) in images_repeats.items():\n",
    "    if not img:\n",
    "      print(f\"💥 Error: Your {folder.replace('/notebooks/', '')} folder is empty.\")\n",
    "      return\n",
    "  for f in files:\n",
    "    if not f.lower().endswith(\".txt\") and not f.lower().endswith(supported_types):\n",
    "      print(f\"💥 Error: Invalid file in dataset: \\\"{f}\\\". Aborting.\")\n",
    "      return\n",
    "    \n",
    "  if not [txt for txt in files if txt.lower().endswith(\".txt\")]:\n",
    "    caption_extension = \"\"\n",
    "  if continue_from_lora and not (continue_from_lora.endswith(\".safetensors\") and os.path.exists(continue_from_lora)):\n",
    "    print(f\"💥 Error: Invalid path to existing Lora. Example: Loras/example.safetensors\")\n",
    "    return\n",
    "\n",
    "  pre_steps_per_epoch = sum(img*rep for (img, rep) in images_repeats.values())\n",
    "  steps_per_epoch = pre_steps_per_epoch/train_batch_size\n",
    "  total_steps = max_train_steps or int(max_train_epochs*steps_per_epoch)\n",
    "  estimated_epochs = int(total_steps/steps_per_epoch)\n",
    "  lr_warmup_steps = int(total_steps*lr_warmup_ratio)\n",
    "\n",
    "  for folder, (img, rep) in images_repeats.items():\n",
    "    print(\"📁\"+folder.replace(\"/notebooks/\", \"\"))\n",
    "    print(f\"📈 Found {img} images with {rep} repeats, equaling {img*rep} steps.\")\n",
    "  print(f\"📉 Divide {pre_steps_per_epoch} steps by {train_batch_size} batch size to get {steps_per_epoch} steps per epoch.\")\n",
    "  if max_train_epochs:\n",
    "    print(f\"🔮 There will be {max_train_epochs} epochs, for around {total_steps} total training steps.\")\n",
    "  else:\n",
    "    print(f\"🔮 There will be {total_steps} steps, divided into {estimated_epochs} epochs and then some.\")\n",
    "\n",
    "  if total_steps > 10000:\n",
    "    print(\"💥 Error: Your total steps are too high. You probably made a mistake. Aborting...\") \n",
    "    return\n",
    "\n",
    "  if adjust_tags:\n",
    "    print(f\"\\n📎 Weighted tags: {'ON' if weighted_captions else 'OFF'}\")\n",
    "    if weighted_captions:\n",
    "      print(f\"📎 Will use {keep_tokens_weight} weight on {keep_tokens} activation tag(s)\")\n",
    "    print(\"📎 Adjusting tags...\")\n",
    "    adjust_weighted_tags(folders, keep_tokens, keep_tokens_weight, weighted_captions)\n",
    "  \n",
    "  return True\n",
    "\n",
    "def adjust_weighted_tags(folders, keep_tokens: int, keep_tokens_weight: float, weighted_captions: bool):\n",
    "  weighted_tag = re.compile(r\"\\((.+?):[.\\d]+\\)(,|$)\")\n",
    "  for folder in folders:\n",
    "    for txt in [f for f in os.listdir(folder) if f.lower().endswith(\".txt\")]:\n",
    "      with open(os.path.join(folder, txt), 'r') as f:\n",
    "        content = f.read()\n",
    "      # reset previous changes\n",
    "      content = content.replace('\\\\', '')\n",
    "      content = weighted_tag.sub(r'\\1\\2', content)\n",
    "      if weighted_captions:\n",
    "        # re-apply changes\n",
    "        content = content.replace(r'(', r'\\(').replace(r')', r'\\)').replace(r':', r'\\:')\n",
    "        if keep_tokens_weight > 1:\n",
    "          tags = [s.strip() for s in content.split(\",\")]\n",
    "          for i in range(min(keep_tokens, len(tags))):\n",
    "            tags[i] = f'({tags[i]}:{keep_tokens_weight})'\n",
    "          content = \", \".join(tags)\n",
    "      with open(os.path.join(folder, txt), 'w') as f:\n",
    "        f.write(content)\n",
    "\n",
    "def create_config():\n",
    "  global dataset_config_file, config_file, model_file\n",
    "\n",
    "  if resume:\n",
    "    resume_points = [f.path for f in os.scandir(output_folder) if f.is_dir()]\n",
    "    resume_points.sort()\n",
    "    last_resume_point = resume_points[-1] if resume_points else None\n",
    "  else:\n",
    "    last_resume_point = None\n",
    "\n",
    "  if override_config_file:\n",
    "    config_file = override_config_file\n",
    "    print(f\"\\n⭕ Using custom config file {config_file}\")\n",
    "  else:\n",
    "    config_dict = {\n",
    "      \"additional_network_arguments\": {\n",
    "        \"unet_lr\": unet_lr,\n",
    "        \"text_encoder_lr\": text_encoder_lr,\n",
    "        \"network_dim\": network_dim,\n",
    "        \"network_alpha\": network_alpha,\n",
    "        \"network_module\": network_module,\n",
    "        \"network_args\": network_args,\n",
    "        \"network_train_unet_only\": True if text_encoder_lr == 0 else None,\n",
    "        \"network_weights\": continue_from_lora if continue_from_lora else None\n",
    "      },\n",
    "      \"optimizer_arguments\": {\n",
    "        \"learning_rate\": unet_lr,\n",
    "        \"lr_scheduler\": lr_scheduler,\n",
    "        \"lr_scheduler_num_cycles\": lr_scheduler_num_cycles if lr_scheduler == \"cosine_with_restarts\" else None,\n",
    "        \"lr_scheduler_power\": lr_scheduler_power if lr_scheduler == \"polynomial\" else None,\n",
    "        \"lr_warmup_steps\": lr_warmup_steps if lr_scheduler != \"constant\" else None,\n",
    "        \"optimizer_type\": optimizer,\n",
    "        \"optimizer_args\": optimizer_args if optimizer_args else None,\n",
    "      },\n",
    "      \"training_arguments\": {\n",
    "        \"max_train_steps\": max_train_steps,\n",
    "        \"max_train_epochs\": max_train_epochs,\n",
    "        \"save_every_n_epochs\": save_every_n_epochs,\n",
    "        \"save_last_n_epochs\": keep_only_last_n_epochs,\n",
    "        \"train_batch_size\": train_batch_size,\n",
    "        \"noise_offset\": None,\n",
    "        \"clip_skip\": 2,\n",
    "        \"min_snr_gamma\": min_snr_gamma_value,\n",
    "        \"weighted_captions\": weighted_captions,\n",
    "        \"seed\": 42,\n",
    "        \"max_token_length\": 225,\n",
    "        \"xformers\": True,\n",
    "        \"lowram\": COLAB,\n",
    "        \"max_data_loader_n_workers\": 8,\n",
    "        \"persistent_data_loader_workers\": True,\n",
    "        \"save_precision\": \"fp16\",\n",
    "        \"mixed_precision\": \"fp16\",\n",
    "        \"output_dir\": output_folder,\n",
    "        \"logging_dir\": log_folder,\n",
    "        \"output_name\": project_name,\n",
    "        \"log_prefix\": project_name,\n",
    "        \"save_state\": save_state,\n",
    "        \"save_last_n_epochs_state\": 1 if save_state else None,\n",
    "        \"resume\": last_resume_point\n",
    "      },\n",
    "      \"model_arguments\": {\n",
    "        \"pretrained_model_name_or_path\": model_file,\n",
    "        \"v2\": custom_model_is_based_on_sd2,\n",
    "        \"v_parameterization\": True if custom_model_is_based_on_sd2 else None,\n",
    "      },\n",
    "      \"saving_arguments\": {\n",
    "        \"save_model_as\": \"safetensors\",\n",
    "      },\n",
    "      \"dreambooth_arguments\": {\n",
    "        \"prior_loss_weight\": 1.0,\n",
    "      },\n",
    "      \"dataset_arguments\": {\n",
    "        \"cache_latents\": True,\n",
    "      },\n",
    "    }\n",
    "\n",
    "    for key in config_dict:\n",
    "      if isinstance(config_dict[key], dict):\n",
    "        config_dict[key] = {k: v for k, v in config_dict[key].items() if v is not None}\n",
    "\n",
    "    with open(config_file, \"w\") as f:\n",
    "      f.write(toml.dumps(config_dict))\n",
    "    print(f\"\\n📄 Config saved to {config_file}\")\n",
    "\n",
    "  if override_dataset_config_file:\n",
    "    dataset_config_file = override_dataset_config_file\n",
    "    print(f\"⭕ Using custom dataset config file {dataset_config_file}\")\n",
    "  else:\n",
    "    dataset_config_dict = {\n",
    "      \"general\": {\n",
    "        \"resolution\": resolution,\n",
    "        \"shuffle_caption\": shuffle_caption,\n",
    "        \"keep_tokens\": keep_tokens,\n",
    "        \"flip_aug\": flip_aug,\n",
    "        \"caption_extension\": caption_extension,\n",
    "        \"enable_bucket\": True,\n",
    "        \"bucket_reso_steps\": 64,\n",
    "        \"bucket_no_upscale\": False,\n",
    "        \"min_bucket_reso\": 320 if resolution > 640 else 256,\n",
    "        \"max_bucket_reso\": 1280 if resolution > 640 else 1024,\n",
    "      },\n",
    "      \"datasets\": toml.loads(custom_dataset)[\"datasets\"] if custom_dataset else [\n",
    "        {\n",
    "          \"subsets\": [\n",
    "            {\n",
    "              \"num_repeats\": num_repeats,\n",
    "              \"image_dir\": images_folder,\n",
    "              \"class_tokens\": None if caption_extension else project_name\n",
    "            }\n",
    "          ]\n",
    "        }\n",
    "      ]\n",
    "    }\n",
    "\n",
    "    for key in dataset_config_dict:\n",
    "      if isinstance(dataset_config_dict[key], dict):\n",
    "        dataset_config_dict[key] = {k: v for k, v in dataset_config_dict[key].items() if v is not None}\n",
    "\n",
    "    with open(dataset_config_file, \"w\") as f:\n",
    "      f.write(toml.dumps(dataset_config_dict))\n",
    "    print(f\"📄 Dataset config saved to {dataset_config_file}\")\n",
    "\n",
    "def download_model():\n",
    "  global old_model_url, model_url, model_file\n",
    "  real_model_url = model_url.strip()\n",
    "  \n",
    "  if real_model_url.lower().endswith((\".ckpt\", \".safetensors\")):\n",
    "    model_file = f\"/notebook{real_model_url[real_model_url.rfind('/'):]}\"\n",
    "  else:\n",
    "    model_file = \"/notebooks/downloaded_model.safetensors\"\n",
    "    if os.path.exists(model_file):\n",
    "      !rm \"{model_file}\"\n",
    "\n",
    "  if m := re.search(r\"(?:https?://)?(?:www\\.)?huggingface\\.co/[^/]+/[^/]+/blob\", model_url):\n",
    "    real_model_url = real_model_url.replace(\"blob\", \"resolve\")\n",
    "  elif m := re.search(r\"(?:https?://)?(?:www\\.)?civitai\\.com/models/([0-9]+)\", model_url):\n",
    "    real_model_url = f\"https://civitai.com/api/download/models/{m.group(1)}\"\n",
    "\n",
    "  !aria2c \"{real_model_url}\" --console-log-level=warn -c -s 16 -x 16 -k 10M -d / -o \"{model_file}\"\n",
    "\n",
    "  if model_file.lower().endswith(\".safetensors\"):\n",
    "    from safetensors.torch import load_file as load_safetensors\n",
    "    try:\n",
    "      test = load_safetensors(model_file)\n",
    "      del test\n",
    "    except Exception as e:\n",
    "      #if \"HeaderTooLarge\" in str(e):\n",
    "      new_model_file = os.path.splitext(model_file)[0]+\".ckpt\"\n",
    "      !mv \"{model_file}\" \"{new_model_file}\"\n",
    "      model_file = new_model_file\n",
    "      print(f\"Renamed model to {os.path.splitext(model_file)[0]}.ckpt\")\n",
    "\n",
    "  if model_file.lower().endswith(\".ckpt\"):\n",
    "    from torch import load as load_ckpt\n",
    "    try:\n",
    "      test = load_ckpt(model_file)\n",
    "      del test\n",
    "    except Exception as e:\n",
    "      return False\n",
    "      \n",
    "  return True\n",
    "\n",
    "def main():\n",
    "  global dependencies_installed\n",
    "\n",
    "  if COLAB and not os.path.exists('/notebook'):\n",
    "    print(\"📂 Connecting to Google Drive...\")\n",
    "  \n",
    "  for dir in (main_dir, deps_dir, repo_dir, log_folder, images_folder, output_folder, config_folder):\n",
    "    os.makedirs(dir, exist_ok=True)\n",
    "\n",
    "  if not validate_dataset():\n",
    "    return\n",
    "  \n",
    "  if not dependencies_installed:\n",
    "    print(\"\\n🏭 Installing dependencies...\\n\")\n",
    "    t0 = time()\n",
    "    install_dependencies()\n",
    "    t1 = time()\n",
    "    dependencies_installed = True\n",
    "    print(f\"\\n✅ Installation finished in {int(t1-t0)} seconds.\")\n",
    "  else:\n",
    "    print(\"\\n✅ Dependencies already installed.\")\n",
    "\n",
    "  if old_model_url != model_url or not model_file or not os.path.exists(model_file):\n",
    "    print(\"\\n🔄 Downloading model...\")\n",
    "    if not download_model():\n",
    "      print(\"\\n💥 Error: The model you selected is invalid or corrupted, or couldn't be downloaded. You can use a civitai or huggingface link, or any direct download link.\")\n",
    "      return\n",
    "    print()\n",
    "  else:\n",
    "    print(\"\\n🔄 Model already downloaded.\\n\")\n",
    "\n",
    "  create_config()\n",
    "  \n",
    "  print(\"\\n⭐ Starting trainer...\\n\")\n",
    "  os.chdir(repo_dir)\n",
    "  \n",
    "  !accelerate launch --config_file={accelerate_config_file} --num_cpu_threads_per_process=1 train_network.py --dataset_config={dataset_config_file} --config_file={config_file}\n",
    "\n",
    "  if not get_ipython().__dict__['user_ns']['_exit_code']:\n",
    "    display(Markdown(\"### ✅ Done! [Go download your Lora(s) from Google Drive](https://drive.google.com/drive/my-drive)\"))\n",
    "\n",
    "main()\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "mBMUJ7BuvNcn"
   },
   "source": [
    "## *️⃣ Extras\n",
    "\n",
    "You can run these before starting the training."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "Wd4916Eu1tb9"
   },
   "source": [
    "### 📚 Multiple folders in dataset\n",
    "Below is a template allowing you to define multiple folders in your dataset. You must include the location of each folder and you can set different number of repeats for each one. To add more folders simply copy and paste the sections starting with `[[datasets.subsets]]`.\n",
    "\n",
    "When enabling this, the number of repeats set in the main cell will be ignored, and the main folder set by the project name will also be ignored.\n",
    "\n",
    "You can make one of them a regularization folder by adding `is_reg = true`  \n",
    "You can also set different `keep_tokens`, `flip_aug`, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Y037lagnJWmn"
   },
   "outputs": [],
   "source": [
    "custom_dataset = \"\"\"\n",
    "[[datasets]]\n",
    "\n",
    "[[datasets.subsets]]\n",
    "image_dir = \"/notebooks/Loras/example/dataset/good_images\"\n",
    "num_repeats = 3\n",
    "\n",
    "[[datasets.subsets]]\n",
    "image_dir = \"/notebooks/Loras/example/dataset/normal_images\"\n",
    "num_repeats = 1\n",
    "\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "W84Jxf-U2TIU"
   },
   "outputs": [],
   "source": [
    "custom_dataset = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "id": "sy9jU2yrdYar"
   },
   "outputs": [],
   "source": [
    "optimizer = \"AdamW8bit\" #@param [\"AdamW8bit\", \"AdamW\", \"Lion\", \"DAdaptation\", \"SGDNesterov\", \"SGDNesterov8bit\", \"AdaFactor\"]\n",
    "optimizer_args = \"weight_decay=0.1\" #@param {type:\"string\"}\n",
    "optimizer_args = [a.strip() for a in optimizer_args.split(\",\") if a]\n",
    "\n",
    "weighted_captions = False #@param {type:\"boolean\"}\n",
    "\n",
    "#markdown By enabling `adjust_tags`, you will let this colab modify your tags before running to automatically adjust to `weighted_captions` being on or off. <p>\n",
    "#markdown Then, you may increase `activation_tag_weight` to improve how effective your activation tag is.\n",
    "adjust_tags = False #param {type:\"boolean\"}\n",
    "activation_tag_weight = \"1.0\" #param [\"1.0\",\"1.1\",\"1.2\"]\n",
    "keep_tokens_weight = float(activation_tag_weight)\n",
    "\n",
    "continue_from_lora = \"\" #@param {type:\"string\"}\n",
    "if continue_from_lora and not continue_from_lora.startswith(\"/notebooks//notebook\"):\n",
    "  import os\n",
    "  continue_from_lora = os.path.join(\"/notebooks//notebook\", continue_from_lora)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "id": "WDjkp4scvPgE"
   },
   "outputs": [],
   "source": [
    "zip = \"/notebooks/my_dataset.zip\" #@param {type:\"string\"}\n",
    "extract_to = \"/notebooks/Loras/example/dataset\" #@param {type:\"string\"}\n",
    "\n",
    "import os, zipfile\n",
    "\n",
    "if not os.path.exists('/notebook'):\n",
    "  print(\"📂 Connecting to Google Drive...\")\n",
    "\n",
    "os.makedirs(extract_to, exist_ok=True)\n",
    "\n",
    "with zipfile.ZipFile(zip, 'r') as f:\n",
    "  f.extractall(extract_to)\n",
    "\n",
    "print(\"✅ Done\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "id": "aKWlpsG0jrX3"
   },
   "outputs": [],
   "source": [
    "folder = \"/notebooks/Loras\" #@param {type:\"string\"}\n",
    "\n",
    "import os\n",
    "\n",
    "if not os.path.exists('/notebook'):\n",
    "    print(\"📂 Connecting to Google Drive...\\n\")\n",
    "\n",
    "tree = {}\n",
    "exclude = (\"_logs\", \"/output\")\n",
    "for i, (root, dirs, files) in enumerate(os.walk(folder, topdown=True)):\n",
    "  dirs[:] = [d for d in dirs if all(ex not in d for ex in exclude)]\n",
    "  images = len([f for f in files if f.lower().endswith((\".png\", \".jpg\", \".jpeg\"))])\n",
    "  captions = len([f for f in files if f.lower().endswith(\".txt\")])\n",
    "  others = len(files) - images - captions\n",
    "  path = root[folder.rfind(\"/\")+1:]\n",
    "  tree[path] = None if not images else f\"{images:>4} images | {captions:>4} captions |\"\n",
    "  if tree[path] and others:\n",
    "    tree[path] += f\" {others:>4} other files\"\n",
    "\n",
    "pad = max(len(k) for k in tree)\n",
    "print(\"\\n\".join(f\"📁{k.ljust(pad)} | {v}\" for k, v in tree.items() if v))\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "provenance": []
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}

[clbustos]

I add at the beggining of the file a cell with:

!python -m venv vlora !source vlora/bin/activate !pip install toml

I set COLAB = False # low ram

and adjust

folder_structure = "lora_training/datasets/<YOUR_DATASET>"

root_dir = "/content" if COLAB else "/notebooks/Loras"

[duskfallcrew]

If Holo hasn't done a paperspace version or a generic jupyter version yet - If it's OK with him i'm gonna ATTEMPT to port his to jupyter, I know a TINY bit of where code goes, i won't yoink his notebooks i'll literally just find him on discord and give him the files whne i'm done :3

[hollowstrawberry]

Oh hey duskfallcrew. Good luck with that!

[duskfallcrew]

Yea, the dataset version would be extremely helpful because while YES Civitai does a great job in auto tagging, it-- doesn't have the "omg you have far too many of the same images" - IN THEORY uncoupling it from google shouldn't be extreme it should just be like my other ones, just taking out google code, making sure no login references are there.

@CHJ85 Holo knows i don't program, I just patch. xD i've been inching to convert his stuff to jupyter when I have the hyperfocus, which is clearly sooner than later because while some of us have advacned access to things (ie my peers) - I prefer sometimes things that are just touch a button and go for the most part, and holo's notebook taught me what i know about lora training XD

[CHJ85]

@duskfallcrew Thank you. I appreciate that.

[theWitchR]

@duskfallcrew Hi! Any luck?