Backups never finish because filesystem scan starts

[jamesdeanreeves]

I've been running the Docker-ized Crashplan fine since it was put into docker, but here of late my backups never complete.

They run for about 3 hours then stop. THe history says "Reason for stopping backup: Full filesystem scan started".

I checked the web console and the verification scan is set every day at 3AM, so I'm clueless why the scans are starting and stopping the backups.

Anyone have any ideas? I'm pretty sure Crashplan won't offer support for this.

[TheBekker]

I have the exact same issue.

I have tried setting verification scans to every 8 days, to avoid this via the web console.
But it stil does this.

[jlesage]

That seems to be an issue on CrashPlan site. Why do you say they won't offer support ?

[TheBekker]

Just looked a bit more into it.
Looks like there is no way to change the verification scan period on a an old/active device.

So the change set in web console only affect devices added after the change.

I ended up having to create a completely new device, which is now running backup.
Still have to see if it will work in the long run, cause it's gonna be doing backup for many days now.
8tb+ of data that has to be re-uploaded.

But i can see that this "new" device now has a 8 day full scan verification period.

Not a great work around, but hopefully a working one.
And i agree, this could seem to be a crashplan issue, and not a docker issue.

It might just be that large arrays just take too long, and full scans keep happening. (mine is 12TB)
But i will be asking Crashplan support.

[jamesdeanreeves]

@jlesage : Crashplan does not support a headless configuration. It's been this way as far back as I can recall.

I opened a case with Crashplan anyway in the hopes that possibly they've changed the policy.

This is the initial response from CP Support:

"We will need to begin by learning about your computing environment. What I see within our monitoring for the device is very strange to me, and I am unfamiliar with a lot of what I am seeing.
What type of setup are you running, from a hardware perspective? How does CrashPlan connect from the host device to the drives? What distro of Linux is this running, and what version number is this distro?
The behavior you are reporting sounds strange, and we'll need to know what type of environment CrashPlan is running in, in order to be able to know how this issue can be approached."

So at this point, they were "on to me". I fessed up, and:

"the problem is that CrashPlan was never designed to run in a container, and that type of setup has been fully unsupported since day one. We don't design the software for that use case, but we of course don't enforce preventing users from creating whichever unsupported setup they feel confident in managing themselves."

@TheBekker : mine is 12TB as well, and here's what they recommended:

"You have 5 different backup sets set to send data all to the same destinations. Each of them are set to run for once per day, and each of them are scheduled to run a scan every 3 hours. I would generally recommend that you condense each of these backup sets back down to one backup set, as you're not actually using the backup sets feature in any meaningful way, unless there's something unique about how they would function in an unsupported environment which I am not aware of. Backup sets are intended to be used in order to send data to different destinations. When you configure them to all send to the same destinations, it all gets dumped into the exact same archive(s), so they just function as scheduling brackets. 5 schedules can get in the way during the best of times, but is going to be especially problematic in this scenario.
My recommendation is that you condense all of your backup sets into the "Homes" backup set, by adding their file selections to the "Homes" backup set", and then deleting each of those now-redundant extra 4 backup sets. Once you have done so, a verification scan schedule of once every 2 or 3 days should reduce the amount of time CrashPlan is stuck in scans, and greatly improve your uptime for the backup. "

So I actually ended up deleting all archives and starting over with the above recommendations with the exception that the verification scans were set to 7 days (reduce the # of times that occurred until it completed). After about a month, I had a completed 12TB backup to CP and I set my scan time to 2 days and it seems to be fine now.

[jlesage]

It's true that running CP in Docker container is not officially supported. But if you don't tell the support team about it, or if your setup doesn't provide clear indication that CP is running in container (e.g. by naming your backup set Synology), the support team would help most of the time.

[Darknight016]

@TheBekker

By web console do you mean using the web interface connected to your Synology instance? If so, that's correct but there's a way to change it. You need to login to your Crashplan account on Crashplan's website to do it. At the top go to Device Backup and then under "Preferred time for verification scan" set it to how many days you want it to be. After you save that setting, here's the important part, you need to hit the button on the right side that says "Push settings to device". There's no text on that icon until you hover on it, but the changes won't take effect on the Synology instance until you do that.

I was having the same problem where since it was defaulted for every day, it could never finish the backup because it would hit the scan. Once I changed it to like 5 days through Crashplan's site, it gave it enough breathing room and I was able to finish the backup before the next scan. I even went to the restore section to verify if the files were showing up there now and they were.

The one problem I have now though is it seems like when it does a full scan, it has to do some extra validating on anything over 2TB. While it doesn't have to upload that data, it does seem to be doing something that takes way longer than the first 2TB. The only hope I have right now is that because this is the first time I've been able to complete my backup after this change, that this additional part of the scan is just happening for this one instance and it'll be fine after the next but my guess is that won't be the case.

[TheBekker]

@Darknight016

With Web Console i mean the Crashplan's own website, that what they call it.
Didn't notice the "Push settings to device", i might look into that.

I'm currently testing a Borg/Borgmatic setup instead combined with a Hetzner storragebox.
Requires a bit more setup and has no fancy UI, but is extremely fast.

[theRealKLH]

@TheBekker : mine is 12TB as well, and here's what they recommended:

"You have 5 different backup sets set to send data all to the same destinations. Each of them are set to run for once per day, and each of them are scheduled to run a scan every 3 hours. I would generally recommend that you condense each of these backup sets back down to one backup set, as you're not actually using the backup sets feature in any meaningful way, unless there's something unique about how they would function in an unsupported environment which I am not aware of. Backup sets are intended to be used in order to send data to different destinations. When you configure them to all send to the same destinations, it all gets dumped into the exact same archive(s), so they just function as scheduling brackets. 5 schedules can get in the way during the best of times, but is going to be especially problematic in this scenario. My recommendation is that you condense all of your backup sets into the "Homes" backup set, by adding their file selections to the "Homes" backup set", and then deleting each of those now-redundant extra 4 backup sets. Once you have done so, a verification scan schedule of once every 2 or 3 days should reduce the amount of time CrashPlan is stuck in scans, and greatly improve your uptime for the backup. "

This is funny cuz it's EXACTLY the same thing they told me. Word for word except my "Homes backup set" is named different. I was actually using multiple backup sets to organize what I was backing up. I never remember reading anything that said "the purpose of multiple backup sets is to backup to different destinations".

So I actually ended up deleting all archives and starting over with the above recommendations with the exception that the verification scans were set to 7 days (reduce the # of times that occurred until it completed). After about a month, I had a completed 12TB backup to CP and I set my scan time to 2 days and it seems to be fine now.

So far I've "deleted" one of the additional sets by FIRST adding the files to the original backup (in essence duping things) and then once that backup was complete, THEN removing the set. That way not to lose anything/MAYBE quicker backup.

However, I still have issues with my setup since November. It still tells me after a full scan 3TB to backup out of 7TB EVEN THOUGH it has already backed up the 3TB. it runs through (sometimes) all data, gets to 0 to backup and then the next scan resets it right back to 3TB. All the backup sets I have do the same thing. to make it so bad one keeps restarting at "247 bytes to back up". COME ON GUYS!

Also have issue where I can only change settings of the original backup set via the web. None of the other backup sets allow changes. Can't "save" only "cancel".

I just want to know what they broke (on purpose?) beginning after the announcement of "no more forever backups". I do have a LARGE backup (I admit it) but NEVER had a single problem in the years since I first began sending them my data. Now I can't get correct unless I "delete and start over"? How can this "not backed up" pointer get reset? They release "fixes" for everything but this. Can't just be a "headless" issue (or is it?)... Somebody reverse engineer their bogus java app (memory eater!) and tell me what bomb they put in there (grin).

[dalelotts]

Personally, I suspect the "Scanning backup selection" step is there to throttle traffic/disk space usage for Code42. Intentionally showing down the backup benefits them when it comes to new, large backup sets.