Orchestrator-agent. Redesign and support for new seed methods

[MaxFedotov]

Hi @shlomi-noach,
We spoke some time ago about adding support for different seed methods for orchestrator and orchestrator-agent, and here it is.
This PR adds following:

• Redesign base struct and methods for agent
• New logging
• New command execution with pipes support
• New API commands
• New configuration file parameters and format
• 5 supported seed methods: mysqldump, mydumper, xtrabackup, lvm and clone plugin
• Packaging using goreleaser
• Integration tests for all new seed methods (see /tests/integration/README.MD)

We define seed as an operation, which consists of a set of predefined stages, where 2 orchestrator agents participate - one agent is called source agent(so it is on a source side of seed), another - target agent (so it is on a target side of seed), and the goal of seed is to transfer all data from source agent to target agent and add target agent as a slave to source agent.

Seed can be executed using different seed methods - special ways or programs used to transfer data from source agent to target agent

In order to simplify developing new seed methods for orchestrator-agent, each of seed methods should support following interface:

type Plugin interface {
	Prepare(side Side)
	Backup(seedHost string, mysqlPort int)
	Restore()
	GetMetadata() (*SeedMetadata, error)
	Cleanup(side Side)
	isAvailable() bool
	getSupportedEngines() []mysql.Engine
	backupToDatadir() bool
}

By design, each seed operation consists of 5 stages, which are executed one after another by orchestrator (and each stage is a function, which seed method should support and had a corresponding orchestrator-agent API method which is used by 'orchestrator' to start it) :

• Prepare. On this stage different preparations for seed are done on both sides, target and source. These preparations could be creating or cleanup of different directories, starting socat for data transfer
• Backup. On this stage backup is executed - transfer of data from source to target. The side of the execution (target or source) depends on the seed method (some of them, like mydumper\mysqldump\clone plugin, are executed on target side, while xtrabackup\lvm are executed on source side). Each of the registered seed methods should provide information about the side, on which backup process is executed
• Restore. On this stage restore is executed. This stage is always executed on target side. Operations on this stage depend on seed method - for mysqldump it's just uploading of .sql backup file to database, for xtrabackup - it's more complex process of uncompressing\preparing data in backup.
• ConnectSlave. This stage will be executed completely by orchestrator. During this stage target agent is connected as a slave for source agent. But in order to implement this, orchestrator should know position\gtids from backup. So it asks target agent to provide backup metadata by calling GetMetadata method.
• Cleanup. This stage is executed on both sides, target and source. Different cleanup activities will be executed on this stage - cleaning backup directories, unmounting snapshots

Other functions in this Interface are helper methods:

• isAvailable() - is a function, which tests if seed method is available on a current host (for example, for xtrabackup it calls xtrabackup --version command to check availability of xtrabackup binary)
• getSupportedEngines() - returns an array of supported engines for seed method. We need this because different seed methods supports different engines, and we can't, for example, use xtrabackup on MySQL 5.7 if there are tables with Myrocks engine, as they are not supported
• backupToDatadir() - is a helper function, which returns true in case of this seed method is capable of backing up data directly to mysql data directory (like xtrabackup`cloneplugin`, 'lvm' - the do not create intermediate backup files with data)

Stages are exposed through orchestrator-agent API and orchestrator calls them in order.
As each stage can take quite a big amount of time to execute, they are executed asynchronously in goroutines. In order to be able to track progress of each stage, 'seed method' sends to 'orchestrator-agent' information about seed stage progress via channel and this information is also exposed through API, so orchestrator can call this API to be able to track stage progress.

That is the basic overview of the process from the orchestrator-agent side. All the philosophy is still the same as was - it's just exposes API calls to process seed and all logic is handled on the orchestrator side.

If you want to test it - there are integration tests available for all seed methods for different versions of mysql (5.7 and 8) and different binlog settings (gtid and positional). They are located in /tests/integration/ folders and in README.MD there is information about how to run them.

We started testing new agent and orchestrator functionality on our staging environments and I think we will find different issues related to this PR, so I think this functionality should be mostly classified as beta right now.

But I will be very grateful for your feedback and comments :)
Thanks,
Max

[aderumier]

@MaxFedotov

Hi,

Thanks for your works !

I'm interested to test it, do you have any builds somewhere for both agent && orchestrator ?