Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make migration able to resume after erroring out #22

Open
Javier-Rotelli opened this issue Nov 17, 2020 · 4 comments
Open

Make migration able to resume after erroring out #22

Javier-Rotelli opened this issue Nov 17, 2020 · 4 comments

Comments

@Javier-Rotelli
Copy link

so that, one big way to add reliability to a tool like this one is to save the progress somewhere, and give the option of resuming from there.
that way even if you have trouble making it to the finish line, you don't lose all you progress.

for example, I have a big forum, and half way it breaks due to a lack of memory, that could be easily worked around by just starting from the last thing migrated.

@MichaelBelgium
Copy link
Owner

MichaelBelgium commented Nov 17, 2020

Good idea

EDIT: how the hell is it possible to detect where u left off? And to detect "the breaking"?

@Javier-Rotelli
Copy link
Author

well, Vanilla forums migrator registered in a text file its current progress.
if there's no file, we start a new migration.
otherwise, we take from last know good state.
this requires making all the queries able to accept an offset. and to make it more resilient, I'm adding a "step" to run it in batches.

@Javier-Rotelli
Copy link
Author

I'm taking a stab on this. as it turns out, it's harder than I initially thought.
you can take a look here: https://github.com/Javier-Rotelli/mybb_to_flarum/commits/make-migration-resumable
so far, I got all the mybb queries extracted and added a offset and limit params to make them run in batches, and to have the option to fetch from a certain offset.
I need to code the second part, that is:
soemhow register the current progress somewhere (i'm thinking a json file in the storage folder) and make the migrator functions able to resume from the last known good state (IE, the count on the file)

I'd love to sya a date to finish this but time and concentration are two things I´m finding hard to get lately =/

@Javier-Rotelli
Copy link
Author

Javier-Rotelli commented Jul 6, 2022

so, a year and a half later, but I have a working proof of concept in #31
now, it is nowhere near production ready, as it has a lot of other changes I'm making in order to be able to migrate a big forum I admin. and it will probably benefit a lot from someone who knows a little bit more about flarum than I do

But if you like it, I can try to clean it up and make it ready to merge.

I tried to comment the PR in order to explain what I did. summarized, I made all the queries able to take an offset, and saved the count object into a json file so we can read it, and restart where we stopped the last time

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants