Add new cases of expected behaviours in troubleshooting doc #789

sauraank · 2024-02-27T09:40:38Z

Description of change

Updated troubleshooting documentation with 4 new cases. It includes the expected behaviour in case of all file deletion, credential expiry while bucket is still mounted, throttling error and invalid hostname for storage provider not supporting virtual style.
NOTE: I am not adding the the cases of invalid credentials at the time of mounting as the error message in those cases are super clear and user don't need to go to logs to understand the issue.

Relevant issues:
#722

Does this change impact existing behavior?

No. It is just documentation change.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

doc/TROUBLESHOOTING.md

vladem · 2024-02-28T18:56:57Z

doc/TROUBLESHOOTING.md

+
+## Throttling Errors
+
+When looking at the logs, these will appear as failed requests with `http_status=503` or `http_status=429` .


Log examples, please! Also, as a developer apart from realising that S3 is throttling my app, what should I do? Will MP retry those automatically?

From mountpoint POV it does not perform anything special on throttling like retries. I added it as it was there in support runbook. I can provide ways to mitigate 503 like restructuring the bucket with more number of prefix which are there in public aws troubleshooting pages.

I see, so the main bit here is that Mountpoint does not do throttling itself. Thanks for describing a way of mitigating S3 throttling errors, but where does this come from? You mention there 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD, these values may become outdated, I'd add a link to the source of that information.

Otherwise LGTM!

jamesbornholt · 2024-03-07T00:53:42Z

doc/TROUBLESHOOTING.md

+
+In this case, try using `--force-path-style` CLI option when you are mounting the bucket using Mountpoint.
+
+NOTE - Third party storage provider are not officially supported by Mountpoint for Amazon S3.


nit: delete

jamesbornholt · 2024-03-07T00:53:57Z

doc/TROUBLESHOOTING.md

+    1: Client error
+    2: Unknown CRT error
+    3: CRT error 1059: aws-c-io: AWS_IO_DNS_INVALID_NAME, Host name was invalid for dns resolution.
+Error: Failed to create mount process


nit: delete this line

jamesbornholt · 2024-03-07T00:54:10Z

doc/TROUBLESHOOTING.md

+
+For more details on how Mountpoint handles endpoint, please see our [configuration documentation](https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#endpoints-and-aws-privatelink).
+
+## Directory disappear after deleting all the files within it


Suggested change

## Directory disappear after deleting all the files within it

## Directory disappears after deleting all the files within it

jamesbornholt · 2024-03-07T00:54:51Z

doc/TROUBLESHOOTING.md

+
+## Directory disappear after deleting all the files within it
+
+Amazon S3 does not support directories and objects are just grouped using prefix.


Just echo the existing doc:

Suggested change

Amazon S3 does not support directories and objects are just grouped using prefix.

The Amazon S3 data model is a flat structure, with no hierarchy of subdirectories.

jamesbornholt · 2024-03-07T00:57:15Z

doc/TROUBLESHOOTING.md

+So, if all the files within a prefix are deleted, the prefix itself and the corresponding directory cease to exist.
+In this case, it is expected that mountpoint will not be able to show the directory for listing or other file system operation.
+
+Workaround to persist a directory could be creating an empty file (for example, `.keep`). These files can be hidden as `ls` filters out files with prefix `.` without `-a` option.


Suggested change

So, if all the files within a prefix are deleted, the prefix itself and the corresponding directory cease to exist.

In this case, it is expected that mountpoint will not be able to show the directory for listing or other file system operation.

Workaround to persist a directory could be creating an empty file (for example, `.keep`). These files can be hidden as `ls` filters out files with prefix `.` without `-a` option.

If all the files within a prefix are deleted, the prefix itself and the corresponding directory cease to exist.

In this case, it is expected that Mountpoint will no longer show the directory or be able to create new files within it. You can recreate the directory with `mkdir` and then continue creating new files within it. Alternatively, you can prevent a directory from disappearing by creating an empty, hidden file (for example, `.keep`) inside it.

jamesbornholt · 2024-03-07T00:57:54Z

doc/TROUBLESHOOTING.md

+## Input/output error after running workload for some time
+
+It is possible that your AWS credentials may get expired while your bucket is still mounted using Mountpoint.
+In that case you will get following errors for filesystem operations:
+
+```
+ls: reading directory '.': Input/output error
+```
+
+```
+cat: new_file.txt: Input/output error
+```
+
+Please check the Mountpoint logs. If you see the following errors, then your AWS credentials have expired.
+
+```
+[WARN] lookup{req=104 ino=1 name="Input"}:
+list_objects{id=20 bucket=plutodemo continued=false delimiter=/ max_keys=1 prefix=Input/}: mountpoint_s3_client::s3_crt_client:
+meta request failed duration=10.023623ms error=ClientError(Forbidden("The provided token has expired."))
+
+[WARN] lookup{req=106 ino=1 name="Input"}:
+head_object{id=21 bucket=plutodemo key=Input}: mountpoint_s3_client::s3_crt_client:
+meta request failed duration=11.087781ms error=ClientError(Forbidden("<no message>"))
+```
+
+You might need to have a fresh mount of your bucket with valid credentials or increase your credential session duration if that suits your use case.
+For more details to configure AWS credentials, see the [configuration documentation](https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#aws-credentials).


Delete this for the moment. This sounds like a bug — we should be refreshing temporary credentials automatically.

jamesbornholt · 2024-03-07T00:58:35Z

doc/TROUBLESHOOTING.md

+
+## Throttling Errors
+
+When looking at the logs, these errors will appear as failed requests with `http_status=503` or `http_status=429` . For example:


Suggested change

When looking at the logs, these errors will appear as failed requests with `http_status=503` or `http_status=429` . For example:

When looking at the logs, throttling errors will appear as failed requests with `http_status=503` or `http_status=429`. For example:

jamesbornholt · 2024-03-07T00:59:53Z

doc/TROUBLESHOOTING.md

+```
+
+The 503 or 429 status codes means the request limits have been exceeded.
+Mountpoint itself does not do any throttling, so any throttling will be from S3 or possibly dependent services, like STS which is used to provide credentials.


Suggested change

Mountpoint itself does not do any throttling, so any throttling will be from S3 or possibly dependent services, like STS which is used to provide credentials.

Mountpoint itself does not do any throttling. These errors are returned from S3 or from dependent services, like STS which is used to provide credentials.

jamesbornholt · 2024-03-07T01:01:18Z

doc/TROUBLESHOOTING.md

+You can try to mitigate  throttling (503 Slow Down) by distributing objects across multiple prefixes if your use case allows it.
+Since, you can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket, increasing the prefix in the bucket would allow more requests to be processed by S3.
+Amazon S3 gradually scales up to handle requests for each of the prefixes separately.


Suggested change

You can try to mitigate throttling (503 Slow Down) by distributing objects across multiple prefixes if your use case allows it.

Since, you can send 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix in an S3 bucket, increasing the prefix in the bucket would allow more requests to be processed by S3.

Amazon S3 gradually scales up to handle requests for each of the prefixes separately.

Amazon S3 automatically scales to high request rates.

Your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 prefix.

You can reduce the impact of throttling errors by distributing objects across multiple prefixes in your bucket.

Signed-off-by: Ankit Saurabh <[email protected]>

sauraank had a problem deploying to PR integration tests February 27, 2024 09:40 — with GitHub Actions Failure

sauraank force-pushed the troubleshooting_update branch from fe7afc1 to 82a5fe6 Compare February 27, 2024 09:41

sauraank temporarily deployed to PR integration tests February 27, 2024 09:41 — with GitHub Actions Inactive

sauraank requested review from passaro, vladem, ahmarsuhail and arsh February 27, 2024 10:16

vladem reviewed Feb 28, 2024

View reviewed changes

sauraank had a problem deploying to PR integration tests February 29, 2024 11:54 — with GitHub Actions Failure

sauraank force-pushed the troubleshooting_update branch from 216c995 to 96c580e Compare February 29, 2024 11:57

sauraank temporarily deployed to PR integration tests February 29, 2024 11:57 — with GitHub Actions Inactive

sauraank temporarily deployed to PR integration tests February 29, 2024 13:21 — with GitHub Actions Inactive

sauraank force-pushed the troubleshooting_update branch from 96c580e to 6431806 Compare February 29, 2024 13:34

sauraank temporarily deployed to PR integration tests February 29, 2024 13:34 — with GitHub Actions Inactive

jamesbornholt reviewed Mar 7, 2024

View reviewed changes

Ankit Saurabh added 3 commits March 8, 2024 13:41

Add new cases of expected behaviours in troubleshooting doc

1cb5e17

Signed-off-by: Ankit Saurabh <[email protected]>

Added a few more logs and mitigations

2eb9a99

Signed-off-by: Ankit Saurabh <[email protected]>

add recommended changes in troubleshooting doc

ec07f83

Signed-off-by: Ankit Saurabh <[email protected]>

sauraank force-pushed the troubleshooting_update branch from 6431806 to ec07f83 Compare March 8, 2024 16:10

sauraank temporarily deployed to PR integration tests March 8, 2024 16:10 — with GitHub Actions Inactive

jamesbornholt approved these changes Mar 8, 2024

View reviewed changes

jamesbornholt added this pull request to the merge queue Mar 8, 2024

Merged via the queue into awslabs:main with commit 81ae0da Mar 8, 2024
23 checks passed

dannycjones mentioned this pull request Jun 4, 2024

Add the expected behaviours of Mountpoint in the troubleshooting page that customers face #722

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new cases of expected behaviours in troubleshooting doc #789

Add new cases of expected behaviours in troubleshooting doc #789

sauraank commented Feb 27, 2024

vladem Feb 28, 2024

sauraank Feb 29, 2024 •

edited

Loading

vladem Mar 6, 2024

jamesbornholt Mar 7, 2024

jamesbornholt Mar 7, 2024

jamesbornholt Mar 7, 2024

jamesbornholt Mar 7, 2024

jamesbornholt Mar 7, 2024

jamesbornholt Mar 7, 2024

jamesbornholt Mar 7, 2024

jamesbornholt Mar 7, 2024

jamesbornholt Mar 7, 2024


		## Throttling Errors

		When looking at the logs, these will appear as failed requests with `http_status=503` or `http_status=429` .


		In this case, try using `--force-path-style` CLI option when you are mounting the bucket using Mountpoint.

		NOTE - Third party storage provider are not officially supported by Mountpoint for Amazon S3.


		For more details on how Mountpoint handles endpoint, please see our [configuration documentation](https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#endpoints-and-aws-privatelink).

		## Directory disappear after deleting all the files within it

	## Directory disappear after deleting all the files within it
	## Directory disappears after deleting all the files within it


		## Directory disappear after deleting all the files within it

		Amazon S3 does not support directories and objects are just grouped using prefix.

	Amazon S3 does not support directories and objects are just grouped using prefix.
	The Amazon S3 data model is a flat structure, with no hierarchy of subdirectories.


		## Throttling Errors

		When looking at the logs, these errors will appear as failed requests with `http_status=503` or `http_status=429` . For example:

	When looking at the logs, these errors will appear as failed requests with `http_status=503` or `http_status=429` . For example:
	When looking at the logs, throttling errors will appear as failed requests with `http_status=503` or `http_status=429`. For example:

	Mountpoint itself does not do any throttling, so any throttling will be from S3 or possibly dependent services, like STS which is used to provide credentials.
	Mountpoint itself does not do any throttling. These errors are returned from S3 or from dependent services, like STS which is used to provide credentials.

Add new cases of expected behaviours in troubleshooting doc #789

Add new cases of expected behaviours in troubleshooting doc #789

Conversation

sauraank commented Feb 27, 2024

Description of change

Does this change impact existing behavior?

Choose a reason for hiding this comment

sauraank Feb 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sauraank Feb 29, 2024 •

edited

Loading