Script for manual database backups on unRAID for when borgbackup doesn't work

[DesertCookie]

Background: Borgbackup has been regularly failing on me to the point where I just gave up. I back up my data directory separately anyway, so there was only missing a way to backup the database and configs too.

Compatibility: If you are using unRAID, this script should work without any configuration. Simply copy the code into a file, for example /mnt/Backup/scripts/backup-ncaio-volumes.sh and make it executable with chmod +X /mnt/Backup/scripts/backup-ncaio-volumes.sh. Either run the script manually or add it on a schedule, for example via the Appdata Backup plugin. It will store backups in /mnt/user/Backup/nextcloud-aio/... and keep them for 30 days.

Configurability: There are a few dials you can tweak in the script to make it fit your environment:

1. VOLUME_PATH - The path where Docker volumes are mounted on your system.
2. BACKUP_TARGET_PATH - Where the archives and log is stored.
3. DAYS_TO_KEEP - Older than X days archives will be deleted.
4. LOG_FILE - Location and name of the log file.
5. Change the compression algorithm from ZSTD to XZ in lines 59-62.
6. Add more volumes to back up in line 63+.
7. Potentially up the timeout in line 55 (see comment).

What the script does: The script sets your nextcloud_aio_nextcloud container into maintenance mode, then taking nextcloud_aio_database offline which triggers a database dump in the nextcloud_aio_database_dump volume. This volume, alongside some others is compressed into an archive and stored in the specified backup location. Archives older than the specified amount in days automatically get deleted. Finally, the script restarts the database container, waiting for a few seconds before turning off maintenance mode again. Your config and database backup should be finished within 60-90 seconds.

#!/bin/bash

set -e  # Exit immediately if a command exits with a non-zero status

# Path to the Docker volumes (source path)
VOLUME_PATH="/var/lib/docker/volumes"
# Path where backups will be stored (target path)
BACKUP_TARGET_PATH="/mnt/user/Backup/nextcloud-aio"
# Number of days to keep backups
DAYS_TO_KEEP=30
# Log file
LOG_FILE="$BACKUP_TARGET_PATH/aiobackup.log"

# Function for logging messages
log() {
  echo "$(date +"%Y-%m-%d %H:%M:%S") - $1" | tee -a "$LOG_FILE"
}

# Function to handle errors
on_error() {
  log "Error: Backup process failed."
  echo "" >> "$LOG_FILE"  # Add an empty line before exiting
  exit 1
}

# Trap errors and call the on_error function
trap on_error ERR

# Remove backups older than X days and log which files are removed
log "Checking for backups older than $DAYS_TO_KEEP days..."
OLD_BACKUPS=$(find "$BACKUP_TARGET_PATH" -name "*.tar.zst" -type f -mtime +$DAYS_TO_KEEP)

if [[ -z "$OLD_BACKUPS" ]]; then
  log "No old backups found."
else
  log "The following old backups will be removed:"
  echo "$OLD_BACKUPS" | tee -a "$LOG_FILE"
  find "$BACKUP_TARGET_PATH" -name "*.tar.zst" -type f -mtime +$DAYS_TO_KEEP -exec rm -v {} \; | tee -a "$LOG_FILE"
fi

# Start timer
START_TIME=$(date +%s)

# Create a timestamped backup file path
BACKUP_FILE="$BACKUP_TARGET_PATH/$(date +%Y%m%d_%H%M%S).tar.zst"

log "Starting backup process..."

# Enable maintenance mode
log "Enabling maintenance mode..."
docker exec -u 33 nextcloud-aio-nextcloud ./occ maintenance:mode --on

# Stop the database container to trigger the dump
log "Stopping the database container to trigger the dump..."
docker stop -t 60 nextcloud-aio-database

# Create the archive and add all relevant volumes
log "Creating backup archive with Zstandard compression..."
# uncomment the following line and comment the ZSTD line out to use XZ compression which is about 30% smaller than ZSTD but takes up to 1>
# tar -I "xz -T4 -9" -cf "$BACKUP_FILE" \
# ZSTD provides a nice balance between speed and compression; increase T4 to the number of threads you want to use for compression
tar --use-compress-program="zstd -T4 -9" -cf "$BACKUP_FILE" \
  "$VOLUME_PATH/nextcloud_aio_mastercontainer/" \
  "$VOLUME_PATH/nextcloud_aio_database_dump/" \
  "$VOLUME_PATH/nextcloud_aio_nextcloud/"

# Restart the database container
log "Restarting the database container..."
docker start nextcloud-aio-database

# Wait for 10 seconds to ensure the database reconnects
log "Waiting for 10 seconds to ensure database connection..."
sleep 10

# Disable maintenance mode
log "Disabling maintenance mode..."
docker exec -u 33 nextcloud-aio-nextcloud ./occ maintenance:mode --off

# End timer and calculate duration
END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))

# Completion message
log "Backup completed successfully and stored in $BACKUP_FILE"
log "Backup duration: $DURATION seconds."

# Add an empty line after each execution
echo "" >> "$LOG_FILE"

# Exit with code 0 on success
exit 0

Example run:

The following run backed up my AIO volumes for a 40-user Nextcloud-AIO instance with about 4TB of data (to give you an idea of the amount of files referenced in the database) with less than a minute in maintenance mode, resulting in a backup archive of about 530MB (down from 900MB uncompressed data).

root@STOWER: /mnt/user/Backup/scripts/backup-ncaio-volumes.sh
2025-01-15 15:18:29 - Checking for backups older than 30 days...
2025-01-15 15:18:32 - No old backups found.
2025-01-15 15:18:32 - Starting backup process...
2025-01-15 15:18:32 - Enabling maintenance mode...
Maintenance mode enabled
2025-01-15 15:18:33 - Stopping the database container to trigger the dump...
nextcloud-aio-database
2025-01-15 15:18:44 - Creating backup archive with Zstandard compression...
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
2025-01-15 15:19:06 - Restarting the database container...
nextcloud-aio-database
2025-01-15 15:19:06 - Waiting for 10 seconds to ensure database connection...
2025-01-15 15:19:16 - Disabling maintenance mode...
Maintenance mode disabled
2025-01-15 15:19:17 - Backup completed successfully and stored in /mnt/user/Backup/nextcloud-aio/20250115_151832.tar.zst
2025-01-15 15:19:17 - Backup duration: 45 seconds.

Restoring from this backup:

It's up to your particular use case to restore from this backup. In the past, I was successful with these steps:

1. Reset the AIO instance, making sure my data directory is empty (I simply moved my data to a different directory).
2. Start and set up the fresh AIO instance.
3. Turn off all containers but the database (the nextcloud_aio_nextcloud container is probably enough, though).
4. Drop the database and import the dump from your backup. For this purpose I simply placed the dump in the nextcloud_aio_database volume and then imported it via the container's console.
5. Copy installed apps and configurations for the nextcloud_aio_nextcloud container from its backup into its volume (I'm unsure which exactly I could safely copy so I actually reinstalled my apps via the app store and copied over my configuration line for line into the new config.php to make sure nothing breaks).
6. Restart all other AIO containers and you should be good.

[szaimen]

Hi, thanks for this guide.

However the better approach would have been to follow https://github.com/nextcloud/all-in-one?tab=readme-ov-file#how-to-enable-automatic-updates-without-creating-a-backup-beforehand more closely.

---

Also I wonder what were the exact issues that you were experiencing?

Borgbackup has been regularly failing on me

[DesertCookie]

I was unable to really pin down the issue, but suspect it's something with the automatic backup schedule.

The backup would regularly get stuck on creating the archive. Sitting there for hours upon hours, with the container using about 6% CPU - so obviously doing at least something. I would have to reset the whole backup directory about once every two weeks to get it to back up properly for a couple of nightly backups. But then there would inevitably come a day, when the backup would hang (it does about 2TB in 24 hours, meaning the initial sync always takes two days - I can't have my NC offline for two days once/twice a month). My initial statement above is due to me noticing, the initial backup would often fail, when there was an automatic backup in that same timeframe which causes the backup to seemingly start again (checking in the next day, I see the previous thousands of log lines gone and the dozen or so initial lines that end with "creating the archive"). I only reliably got it to do the initial sync, when I previously disabled the schedule and only reenabled it afterwards. Since, however, even the nightly backups regularly get stuck, I can't help but abandon this for now, as I've been unable to find any solutions in the posts about borgbackup getting stuck in this situation so far.

Had to recreate from database dump earlier this week and it's already been more pleasant than the last half year of using borg backup. Even when it worked, it would take hours to recreate the instance, whereas now I can get it up in less than 30 minutes.

[szaimen]

docker stop nextcloud-aio-database

This is missing overwriting the timeout as the database might take longer than the default of 10s.

[DesertCookie]

Thanks for pointing this out!

Testing with my database, it takes on average 16 seconds to actually stop. That's on an M.2 NVMe SSD that sees its writes spike to 85MB/s from <1MB/s during that time. I feel comfortable setting the default timeout to 60 seconds but advise anyone to test how long it takes on their system:

#!/bin/bash

# Container name
CONTAINER_NAME="nextcloud-aio-database"
# Long timeout to prevent force stop (e.g., 600 seconds)
TIMEOUT=600

echo "Stopping container '$CONTAINER_NAME' with a timeout of $TIMEOUT seconds..."

# Record the start time
START_TIME=$(date +%s)

# Stop the container with the specified timeout
docker stop -t $TIMEOUT "$CONTAINER_NAME" &

# Wait until the container is stopped
while docker ps | grep -q "$CONTAINER_NAME"; do
  echo "Container is still running... checking again in 1 seconds."
  sleep 1
done

# Record the end time
END_TIME=$(date +%s)

# Calculate the duration in seconds
DURATION=$((END_TIME - START_TIME))

echo "The container '$CONTAINER_NAME' took $DURATION seconds to shut down."

# Restart the container
docker start nextcloud-aio-database