Table ordering during dump / restore

[janmeier]

We have some large table, which take considerably longer time to dump / restore than others. When we do a dump / restore, we often end up waiting for those tables to finish, while all other tables are already done. That means we don't use the full number of parallel processes.

If we could somehow specify which tables to start with, this would not be an issue. Consider the chart below - The top section (slow) shows how it works now, while the bottom section shows how it would work if we could start with the biggest tables

gantt
    dateFormat  mm
    axisFormat %M
    section Slow
    t1           :t1, 0, 3m
    t2           :t2, 0, 4m
    t3     :after t1  , 6m
    
    section Fast
    t1           :t4, 0, 3m
    t2           :after t4,  4m
    t3                  :t3 ,0 , 6m

Loading

[wwoytenko]

@janmeier that's a good point. Looks like we could score the tables according to their size and plan workers properly for doing so.

Thank you. I took it into account

[wwoytenko]

This is going to be fixed in #53. Now greenmask introspects the table size, adds the transformation scoring using the formula below and uses strategy "Largest First":

score = tableSizeInBytes + (tableSizeInBytes * 0.03 * tableTransformationsCount)

[janmeier]

Looks like exactly what I need 🚀

[wwoytenko]

The v0.1.9 version has been released. Verify that it fixes the problem you described

[wwoytenko]

@janmeier I've added the toast table size in v0.1.10

[wwoytenko]

Looks like I've just found that if the table has a transformation config the table size is not assigned. Will be fixed in the next release

[wwoytenko]

@janmeier
It was fixed in v0.1.11. Let me know if everything is okay

[janmeier]

@wwoytenko Sorry for being unresponsive here!

I can confirm that everything works as expected in v0.1.11, and greenmask has been working perfectly for us for the last three months!