Implement disk-based DataCache with no eviction

[dannycjones]

Description of change

Initial version of the data cache on disk.

We now write data to disk including some metadata to check we haven't mixed up blocks.

Related: #255

Does this change impact existing behavior?

No. This code is not yet adopted in main releases.

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

[passaro]

Few comments from a quick pass.

[passaro]

Besides/instead of the version (which is not really needed for a temporary cache), should we include the process id, or something equivalent, to ensure the path is unique?

Maybe for this type it could just be a token to be passed at construction.

[dannycjones]

I'll push an update to this PR now, but not address this today. Let me think about it.

I left in the versioning while I'm not super happy about a) not purging the cache on mount and b) not having things like etag, s3key in the blocks themselves on disk.

[jamesbornholt]

Versioning here seems good. We definitely need some safety around (a) not using a non-empty cache directory where we might overwrite existing files, and (b) coordinating enough to avoid multiple Mountpoints trying to use the same directory. But happy for that to come in a followup.

For (a), we'll need to think about whether we should allow non-empty directories that appear to be Mountpoint caches from a previous execution (it's perhaps very cumbersome for customers to have to manually rm -rf on every restart), and if so, whether to clean them up or not.

[dannycjones]

Keeping the versioning for now, including that into the block data persisted to disk also.

Deferring the safety around non-empty cache and concurrent access to cache from different MP processes to a later discussion.

[dannycjones]

Just pushed a few changes based on some feedback, but still need to address tomorrow:

• What to do about any versioning in path (v1 or PID or something else)
• Look into comments on if we need base64 encoding, and to consider splitting on the first part of the directory
• Identify path forward with macOS since it has a full path limit of 1024 (or maybe smaller), which is why the macOS unit tests are failing

[jamesbornholt]

Versioning here seems good. We definitely need some safety around (a) not using a non-empty cache directory where we might overwrite existing files, and (b) coordinating enough to avoid multiple Mountpoints trying to use the same directory. But happy for that to come in a followup.

For (a), we'll need to think about whether we should allow non-empty directories that appear to be Mountpoint caches from a previous execution (it's perhaps very cumbersome for customers to have to manually rm -rf on every restart), and if so, whether to clean them up or not.

[dannycjones]

Planning to update this PR as follows to have it in a state we're happy to merge and iterate later on:

• Pull in the block data struct that we serialize to disk rather than just the bytes (w/ TODO for checksumming this struct). WIP: Add checksums to on-disk cache dannycjones/mountpoint-s3#15
• Replace Base64 encoding of S3 key only with SHA256. Consider what we want to do about ETag and block numbers later.
• Remove body of cached_block_indices and replace with todo!("implement or deprecate"). Less to review, since we are now considering removing this outright.

[dannycjones]

I'll merge this now. If there's any more feedback, please still add and I can follow up in another PR.