HDDS-11475: Verify EC reconstruction correctness

[devabhishekpal]

What changes were proposed in this pull request?

HDDS-11475: Verify EC reconstruction correctness

Please describe your PR in detail:

• In current implementation the stripe checksum is formed in ECKeyOutputStream in private StripeWriteStatus commitStripeWrite(ECChunkBuffers stripe)
• To verify the recreated data we can use the stripe checksum - which is a concatenation of all the chunk checksums in the stripe and compare the recreated chunk checksum with the stripe checksum at recreated index to verify the correct data was recreated

Example, for EC 3-2 we will have chunks c1, c2, c3, c4, c5
(stripe checksum) s1 = add checksum of(c1 to c5)

Let recreated chunk be c2
Then:
checksum(c2) == 2nd checksum in s1 checksum sequence

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11475

How was this patch tested?

Unit tests

[adoroszlai]

Thanks @devabhishekpal for working on this.

Please wait for clean CI run in fork before opening PR (or marking as "ready for review").

Unit tests

Since this is controlled by a new config, which defaults to the old behavior, I don't think it is validated by any unit tests.

[sodonnel]

The approach used here, is to take the chunk buffer, which holds the real data just written to the block, and calculate the checksum on it.

However that is duplicating work, as the act of writing the data through the ECBlockOutput stream already performs that checksum and persists it in the block metadata as part of the put block.

I have had to look at this for some time to try to understand the current flow. Its been a long time since this EC code was written, and the checksum stuff was not written by me. @aswinshakil might be a good person for a second look.

Starting in the ECReconstructionCoordinator, there is code where it calls executePutBlock(...) on the reconstructed streams. Here, I think, is where we can validate the checks match the stripe checksum:

        for (ECBlockOutputStream targetStream : allStreams) {

         // You can get the current chunkList and its checksums calculated while writing. These are what will be written
         // as part of the putBlock call. However if we get them here, each chunk has its checksums.
         // Using blockDataGroup, which is all the blockData that existed on the containers prior to any reconstruction, we can
         // search it for one which contains the stripChecksum. We know it lives in replicaIndex=1 or any parity, however you
         // many not have index 1 (it could be getting reconstructed) or all the parities, but you must have at least 1 of them
         // to make the thing reconstructable. There you must search until it can be found.
         // 
         // targetStream.getContainerBlockData().getChunksList().get(0).getChecksumData();
         // blockDataGroup[0].getChunks().get(0).getStripeChecksum();
         //
         // From above, if you have the chunkList and hence its checksums for the current stream, and you can locate
         // the existing stripe checksum in the blockDataGroup, then you can "simply" iterate the chunkList:
         //  
         //  List<Chunk> chunks =  targetStream.getContainerBlockData().getChunksList();
         //  List<Chunk> existingChunks = blockDataGroup[0].getChunks();
         // for (int i = 0; i < chunks.length; i++ ) {
         //      validateChecksum(chunks.get(i).getChecksumData(), existingChunks.get(i).getStripChecksum());
         // }
         //

          targetStream.executePutBlock(true, true, blockLocationInfo.getLength(), blockDataGroup);
          checkFailures(targetStream, targetStream.getCurrentPutBlkResponseFuture());
        }

Inside validateChecksum() you need to figure out how to index into the strip checksum to find the relevant part of it to compare against the chunkchecksum.

I think that approach will work, and it avoids calculating the checksum from the data a second time.