Add support for S3 Express One Zone

[AndrewBarba]

S3 Express One Zone Buckets are currently unsupported. Recent AWS SDKs for other languages added dedicated support, detecting the s3 express style bucket names and updating the singing algorithm and endpoints accordingly.

It would be great to get full support, including S3 session management, directly in Soto

[adam-fowler]

As far as I can tell all the operations are available to do this already.

I don't see anything about a custom signing process in the docs. As far as I can tell you extract the credentials out of the s3.createSession response and use those. Have you tried this? If so how far did you get and what errors did you see?

If you want to ensure you have valid credentials all the time you could use a RotatingCredentialProvider to request new credentials when your current ones are about to expire.

[AndrewBarba]

Yes I have a bucket setup that I'm testing against and actually createSession using Soto just times out against the directory bucket, works fine using awslabs/aws-sdk-swift.

But even then we must support the new sigv4-s3express signing scheme. If you do some grepping on GitHub you'll see it in their other sdks. Unfortunately the docs are so bad around this, but it's effectively just one extra header that gets passed a long with the value of the session token. I'll see what I can help dig up here

[AndrewBarba]

I opened a similar issue on aws-sdk-swift: awslabs/aws-sdk-swift#1844

[AndrewBarba]

@adam-fowler Here's a full project to help test this: https://github.com/AndrewBarba/swift-s3-express-demo

Assuming you have docker installed, and AWS credentials locally, you can deploy the whole stack with:

swift run Infra deploy --stage demo

You'll see the lambda using aws-sdk works fine and is able to create a session, the Soto one does not.

[adam-fowler]

I had a very quick look at this and found the Soto version fails because the server doesn't return credentials. I know aws-sdk-swift sets everything to optional so it doesn't matter if the credentials are returned. Does the aws-sdk-client example have credentials set in the response?

[AndrewBarba]

Yes it does. Has values for all 4 properties. Here's my deployed url: https://ba3qux3gvwwyc5ndtsbutlfmhu0yjsgy.lambda-url.us-east-1.on.aws

[adam-fowler]

Looks like I'm gonna have to set up my own EC2 instance to work out what is going on here. S3 express seems to only be available in the same zone as it is setup.

As far as I can tell to use S3 express you need to use custom endpoints but the createInstance operation still uses the standard endpoints. I haven't got anywhere near to considering the signing process yet.

[AndrewBarba]

You do not need to be in the same zone that the bucket is deployed. It's recommended to get those single digit TTFBs but it is not a requirement. The project I sent you would be a sufficient setup to test everything you need.

The main hiccup is if you deploy into a VPC, you need to configure a VPC endpoint. But the project I sent does not deploy the lambda into a VPC so it works just fine.

[adam-fowler]

Running on a lambda is no use. I can't debug it. There is very little visibility on what is happening outside of print statements.

Ideally I would be running code locally on my Mac, but if that doesn't work I can use an EC2 instance and VSCode and ssh. I tend to use awscli to find out what is required, if docs are sparse. But calling createSession with it on my Mac fails.

[AndrewBarba]

Understood. Calling create session from your Mac should be fine assuming your credentials have access to s3express:CreateSession

[adam-fowler]

Yes of course. I've got admin rights on everything and it is saying I don't have access. I'll have another look see if I can work out what is going on.

[AndrewBarba]

Did you also try calling create session with aws swift sdk? When I initially tested that did work, and the Soto create session did not. And I did this from my Mac as well as the deployed lambda (not in a VPC). I didn't have a chance to dig further into the Soto error. My main suspicion though is that Soto is not using the regional endpoint when making the call, where as the aws sdk is. The endpoints for express buckets are a bit different. You use the regional endpoint to create a session and then zonal for object calls.

EDIT: my mistake create session does use the zonal endpoint. So that is probably the issue

[adam-fowler]

Haha worked out why awscli was failing. I was running create-session from the wrong region 🤦

[adam-fowler]

soto-project/soto-core#629

[adam-fowler]

So I think I know what is required. Basically the session id needs to be in a different header. It just makes me wonder why does S3 have to do it different to everyone else. The number of special cases required by S3 keeps growing.

[adam-fowler]

#749

[AndrewBarba]

Looking good! I can successfully put/get objects now. I am seeing one issue on listObjectsV2 returning a decoding error:

The operation could not be completed. The data isn't in the correct format.

[adam-fowler]

Fixed. S3 added a new ChecksumAlgorithm type without updating the models

[adam-fowler]

Released

[AndrewBarba]

Really appreciate the work on this! Thanks a ton