Skip to content

Commit

Permalink
Update documentation based on PR feedback
Browse files Browse the repository at this point in the history
Signed-off-by: Daniel Carl Jones <[email protected]>
  • Loading branch information
dannycjones committed Nov 3, 2023
1 parent 6f835cd commit 5c423e5
Show file tree
Hide file tree
Showing 2 changed files with 26 additions and 21 deletions.
35 changes: 17 additions & 18 deletions doc/CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ In most scenarios, you can use Mountpoint by running the following command, wher

We've tried hard to make this simple command adopt good defaults for most scenarios. However, some scenarios may need additional configuration. This document shows how to configure these elements of Mountpoint:
* [AWS credentials](#aws-credentials)
* [Caching configuration](#caching-configuration)
* [S3 bucket configuration](#s3-bucket-configuration), including mounting a bucket prefix or changing the endpoint to which Mountpoint sends S3 requests
* [File system configuration](#file-system-configuration), including making a bucket read-only or allowing file deletion
* [Caching configuration](#caching-configuration), where metadata and object data can be served from a cache
* [Logging](#logging) for troubleshooting Mountpoint

## AWS credentials
Expand Down Expand Up @@ -81,23 +81,6 @@ Here is an example least-privilege policy document to add to an IAM user or role

Mountpoint also respects access control lists (ACLs) applied to objects in your S3 bucket, but does not allow you to automatically attach ACLs to objects created with Mountpoint. A majority of modern use cases in Amazon S3 no longer require the use of ACLs. We recommend that you keep ACLs disabled for your S3 bucket, and instead use bucket policies to control access to your objects.

## Caching configuration

Mountpoint now offers caching of metadata and object content allowing for reduced requests when reading files.
This is particularly useful when reading the same files many times for the same Mountpoint filesystem.

To enable caching, use the `--caching` command-line flag.
This alone will enable caching of metadata using a default time-to-live (TTL) of 60 minutes.
Caching of object/file content on disk can also be enabled
by providing a caching location using the `--data-cache-directory <DIR>` command-line flag.

Enabling caching relaxes the strong read-after-write consistency offered by Mountpoint with default configuration.
Please read more in the [consistency and concurrency section of the semantics documentaton](./SEMANTICS.md#consistency-and-concurrency).

Mountpoint caching can be further configured,
such as adjusting the metadata time-to-live (TTL) or the maximum space allowed to be used by the data cache.
Review the caching options available using `mount-s3 --help`.

## S3 bucket configuration

By default, Mountpoint will automatically mount your S3 bucket given only the bucket name, and will automatically select the appropriate S3 HTTPS endpoint. However, you can override this automation if you need finer control over how Mountpoint connects to your bucket.
Expand Down Expand Up @@ -219,6 +202,22 @@ In its default configuration, there is no maximum on the size of objects Mountpo

To increase the maximum object size for writes, use the `--part-size` command-line argument to specify a maximum number of bytes per part, which defaults to 8 MiB. The maximum object size will be 10,000 multiplied by the value you provide for this argument. Even with multipart upload, S3 allows a maximum object size of 5 TiB, and so setting this argument higher than 524.3 MiB will not further increase the object size limit.

## Caching configuration

Mountpoint can optionally cache object metadata and content to reduce cost and improve performance for repeated reads to the same file.

To enable caching, use the `--cache <CACHE-DIR>` command-line flag.
This flag will enable caching of metadata using a default time-to-live (TTL) of 60 minutes.
Object/file content will also be cached within the cache directory specified.

Mountpoint caching can be further configured,
such as adjusting the metadata time-to-live (TTL) or the maximum space allowed to be used by the data cache.
Review the caching options available using `mount-s3 --help`.

> [!WARNING]
> Caching relaxes the strong read-after-write consistency offered by Amazon S3 and Mountpoint in its default configuration.
> See the [consistency and concurrency section of the semantics documentaton](./SEMANTICS.md#consistency-and-concurrency) for more details.
## Logging

By default, Mountpoint emits high-severity log information to [syslog](https://datatracker.ietf.org/doc/html/rfc5424) if available on your system. You can change what level of information is logged, and to where it is logged. See [LOGGING.md](LOGGING.md) for more details on configuring logging.
12 changes: 9 additions & 3 deletions doc/SEMANTICS.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,12 +66,18 @@ Mountpoint has limited support for other file and directory metadata, including
Amazon S3 provides [strong read-after-write consistency](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html#ConsistencyModel) for PUT and DELETE requests of objects in your S3 bucket.
By default, Mountpoint provides strong read-after-write consistency for file writes, directory listing operations, and new object creation. For example, if you create a new object using another S3 client, it will be immediately accessible with Mountpoint. Mountpoint also ensures that new file uploads to a single key are atomic. If you modify an existing object in your bucket with another client while also reading that object through Mountpoint, the reads will return either the old data or the new data, but never partial or corrupt data. To guarantee your reads see the newest object data, you can re-open the file after modifying the object.

Mountpoint also offers metadata and object content caching which can be enabled using CLI flags: see the [caching section of the configuration document](./CONFIGURATION.md#caching) for more information.
However, Mountpoint may return stale metadata for an existing object within 1 second of the object being modified or deleted in your S3 bucket by another client.
This occurs only if the object was accessed through Mountpoint immediately before being modified or deleted in your S3 bucket.
The stale metadata will only be visible through metadata operations such as `stat` on individual files.
Directory listings will never be stale and always reflect the current metadata.
These cases do not apply to newly created objects, which are always immediately visible through Mountpoint.
Stale metadata can be refreshed by either opening the file or listing its parent directory.

Mountpoint also offers optional metadata and object content caching which can be enabled using CLI flags:
see the [caching section of the configuration documentation](./CONFIGURATION.md#caching) for more information.
When opting into caching, the consistency model is relaxed and you may see stale entries until they have expired.
Stale entries may live as long as the cache's metadata time-to-live (TTL).

However even with caching off, Mountpoint may return stale metadata for an existing object within 1 second of the object being modified or deleted in your S3 bucket by another client. This occurs only if the object was accessed through Mountpoint immediately before being modified or deleted in your S3 bucket. The stale metadata will only be visible through metadata operations such as `stat` on individual files. With caching mode off, directory listings will never be stale and always reflect the current metadata. These cases do not apply to newly created objects, which are always immediately visible through Mountpoint. Stale metadata can be refreshed by either opening the file or listing its parent directory.

Mountpoint allows multiple readers to access the same object at the same time. However, a new file can only be written to sequentially and by one writer at a time. New files that are being written are not available for reading until the writing application closes the file and Mountpoint finishes uploading it to S3. If you have multiple Mountpoint mounts for the same bucket, on the same or different hosts, there is no coordination between writes to the same object. We recommend that your application does not write to the same object from multiple instances at the same time.

## Durability
Expand Down

0 comments on commit 5c423e5

Please sign in to comment.