HDDS-11658. Skip known tombstones when scanning rocksdb deletedtable in KeyDeletingService

[guohao-rosicky]

What changes were proposed in this pull request?

KeyDeletingService, iterate several records from scratch each time, send deletion requests, and then delete these records.

If each iteration starts from scratch, Rocksdb will also include the recently deleted tombstone data in the scan.

The optimization is to optimize the iteration strategy, record breakpoints for each iteration, skip known tombstones, and reset breakpoints if errors occur during the iteration or if the endpoint is reached.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11658

How was this patch tested?

TestKeyDeletingService set OZONE_KEY_DELETING_LIMIT_PER_TASK is 10

[adoroszlai]

Thanks @guohao-rosicky for the patch.

Please wait for clean CI run in fork before opening PR.

There are some failures that seem related:

Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 31.757 s <<< FAILURE! - in org.apache.hadoop.ozone.om.service.TestKeyDeletingService$Normal

https://github.com/guohao-rosicky/ozone/actions/runs/11851572260/job/33028663632#step:6:2342

Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 37.414 s <<< FAILURE! - in org.apache.hadoop.ozone.om.snapshot.TestSnapshotDeletingServiceIntegrationTest

https://github.com/guohao-rosicky/ozone/actions/runs/11851572260/job/33028664194#step:6:2895

[sadanand48]

If each iteration starts from scratch, Rocksdb will also include the recently deleted tombstone data in the scan.

Are we sure about this? Doesn't rocksdb abstract out the tombstone info? I think it does. The Java API for iterating the table should not get the tombstones.

[sadanand48]

Also recording the last scan key will cause it to skip the newer entries which alphabetically (sorted order) appear before the last scan key causing many keys to remain undeleted.

[errose28]

Also recording the last scan key will cause it to skip the newer entries which alphabetically (sorted order) appear before the last scan key causing many keys to remain undeleted.

+1. This also adds coupling to internal RocksDB behavior that may be optimized or irrelevant in later releases. Since the change is not without drawbacks it would be good to run a micro benchmark to quantify the speedup. A simple junit test that generates large amounts of delete entries and times how long the new vs old implementation takes to clear them out would probably suffice.