Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add sqs source retry for connection reset errors #215

Open
colmsnowplow opened this issue Sep 26, 2022 · 1 comment
Open

Add sqs source retry for connection reset errors #215

colmsnowplow opened this issue Sep 26, 2022 · 1 comment
Labels
bug Something isn't working

Comments

@colmsnowplow
Copy link
Collaborator

colmsnowplow commented Sep 26, 2022

Currently, we return an error and shut down if the sqs client returns any error.

In the case of a connection reset error, the behaviour of the sdks will be to treat it as non-retryable: aws/aws-sdk-go#3027, aws/aws-sdk-go#3971.

As I understand it, this behaviour is general to all AWS sdks, and this error is considered non-retryable because the requests in question may not be idempotent.

In our case, for the sqs source the request will always be idempotent, and we should retry. We can achieve this (as the AWS sdk maintainers advise on the above isssues) by adding a retryer.

As far as I can see, this bug shouldn't have much of an impact on the app in production as we currently use it - when we hit it we will simply return an error, crash, and reboot without data loss.

@colmsnowplow colmsnowplow added the bug Something isn't working label Sep 26, 2022
@colmsnowplow
Copy link
Collaborator Author

Came across this while building the e2e tests, which use localstack - SQS on localstack seems relatively flaky so we seem to hit it consistently (at least on 0.11 localstack, which is fairly old by now).

A quick check into sentry shows that we do indeed hit this every now and again in prod, for a couple of different deployments - looks consistent with what I'd expect for the scenario I describe above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant