Move O_DIRECT to semantics doc

Signed-off-by: Daniel Carl Jones <[email protected]>

doc/CONFIGURATION.md

@@ -245,9 +245,6 @@ Mountpoint caching can be further configured,
 such as adjusting the metadata time-to-live (TTL) or the maximum space allowed to be used by the data cache.
 Review the caching options available using `mount-s3 --help`.
 
-To avoid using the cache, an application can use the `O_DIRECT` option (or language equivalent) when opening a file for reading.
-When this option is provided, Mountpoint will check S3 when opening the file to ensure the latest object content is returned to the application.
-
 > [!WARNING]
 > Caching relaxes the strong read-after-write consistency offered by Amazon S3 and Mountpoint in its default configuration.
 > See the [consistency and concurrency section of the semantics documentaton](./SEMANTICS.md#consistency-and-concurrency) for more details.

doc/SEMANTICS.md

@@ -73,16 +73,22 @@ Directory listings will never be stale and always reflect the current metadata.
 These cases do not apply to newly created objects, which are always immediately visible through Mountpoint.
 Stale metadata can be refreshed by either opening the file or listing its parent directory.
 
+Mountpoint allows multiple readers to access the same object at the same time. However, a new file can only be written to sequentially and by one writer at a time. New files that are being written are not available for reading until the writing application closes the file and Mountpoint finishes uploading it to S3. If you have multiple Mountpoint mounts for the same bucket, on the same or different hosts, there is no coordination between writes to the same object. We recommend that your application does not write to the same object from multiple instances at the same time.
+
+### Using optional metadata and object content caching
+
 Mountpoint also offers optional metadata and object content caching which can be enabled using CLI flags:
 see the [caching section of the configuration documentation](./CONFIGURATION.md#caching) for more information.
 When opting into caching, the consistency model is relaxed and you may see stale entries until they have expired.
 Stale entries may live as long as the cache's metadata time-to-live (TTL).
+
 For example, without caching it was not possible to read stale data if the file was opened after it was updated or deleted.
 With caching, it is now possible to open a file for a stale object.
 Reads may either succeed returning any cached object content or return I/O errors for accesses to uncached object content.
 However, it is not possible for object content from two different objects to be returned for the same file handle.
 
-Mountpoint allows multiple readers to access the same object at the same time. However, a new file can only be written to sequentially and by one writer at a time. New files that are being written are not available for reading until the writing application closes the file and Mountpoint finishes uploading it to S3. If you have multiple Mountpoint mounts for the same bucket, on the same or different hosts, there is no coordination between writes to the same object. We recommend that your application does not write to the same object from multiple instances at the same time.
+To avoid reading data from an old object, an application can use the `O_DIRECT` option (or language equivalent) when opening a file for reading.
+When this option is provided, Mountpoint will check S3 when opening the file to ensure the latest object content is returned to the application.
 
 ## Durability