Tracking issue for speed up PiTR by pre-compacting logs

[YuJuncen]

Enhancement

For now PiTR is slow. Because we need:

• Download the logs to the leader.
• Leader duplicating the mutations to all peers by raft (Requires a sync of Disk).
• Sort this run in the memtable.
• Flush and compact the mutations in the LSM tree.

This routine includes many overheads:

• Write amplification.
• Sorting the mutations.
• One extra write in the raft engine.

We may "pre-compact" the backup data to SSTs, and use the download/ ingest API to put them into the cluster.

Progress Tracking

Generate SSTs from Logs Offline

• tracking issue for implementing compact-log-backup in tikv-ctl tikv/tikv#17283

  • Add interfaces needed by compact-log-backup. tikv/tikv#17284
  • Add CRC64 for logs generated by log backup. tikv/tikv#17406
  • compact_log_backup: implement the basic library for compacting tikv/tikv#17631
  • Add compacting log backup to tikv-ctl. tikv/tikv#17844

This Issue tracks the subcommand compact-log-backup of tikv-ctl.

Manage the Metadata of Compactions

• Implement the toolkit for managing Migrations. #55661
• Implement the "put-and-verify" lock for external storage. #56523
• Allow br log truncate to delete outdated compactions #56758
• br/operator: add some debug commands to operate migrations / external storages. #56760
• BR: Add Verision Checking for Migrations #57540

We will introduce the migration system for log backup storages. Generally, we now modify the "stabilized" logs by putting an atomic "migration".

Point in Time Restore with SSTs

The whole changes is in #54813

• Restore: refine split code for both compacted sst files and logs #56917
• Read compacted meta from migrations. #57210
• Refine restorer to support both logs and compacted sst files. #57209
• Implement different split strategy for logs and compacted sst files. #57212
• Implement checkpoint for compacted log files restoration #57535
• log_client: should use input_min_ts and input_max_ts to filter compactions #58733

Implement a More Friendly User Interface.

• Implement the TiDB Operator interface for compacting
• logs br: add a new CRD for compact backup tidb-operator#5822
• br: make schdule support compact tidb-operator#6033)

Testing

General Workload

This is an internal workload, with:

• Cluster Size: 23 TB
• QPS: 6k inserts/18k updates

(Short Duration Compaction) Restore Performance:

• Log Restore: Completed in 2 hours(24 hours data Retention)
• Log Restore with Log Compaction: Completed in 40 minutes (There's still room for improvement：As the data in the log backup retention period grows, the effectiveness of compaction improves, demonstrating an optimization effect that exceeds linear growth, because of less write amplification and less Raft I/O)

(Long Duration Compaction) Restore Performance:

• Log Restore: There is data skew during restoring hence the disk space isn't enough. We scaled out to 10 TiKV stores. The restore takes 10 hours to finish with several breaks(break due to disk almost full, because compact SST consumes too many space during log restore, so it will take much more space when restore with a longer interval(7 days)).
• Log Restore with Log Compaction: Without split, we finished the restoration in about 3hrs.