Future creates fail for files that are deleted remotely

[sauraank]

Mountpoint for Amazon S3 version

mountpoint-s3 v1.1.0

AWS Region

eu-west-1

Describe the running environment

Running in EC2 on OS Amazon Linux2 with S3 bucket in the same account as EC2 instance.

What happened?

When we are creating a new object and then deleting the object using CLI/Console it gets deleted successfully. However when we try re-creating the same object name it gives error No such file or directory.

Relevant log output

Sequence of Fuse requests made: 

2023-10-25T14:31:28.613010Z DEBUG fuser::request: FUSE(  2) ino 0x0000000000000000 INIT kernel ABI 7.37, capabilities 0x73fffffb, max readahead 131072    
2023-10-25T14:31:28.613358Z DEBUG fuser::request: INIT response: ABI 7.28, flags 0x402021, max readahead 131072, max write 16777216    
2023-10-25T14:31:32.648763Z DEBUG fuser::request: FUSE(  4) ino 0x0000000000000001 GETATTR    
2023-10-25T14:31:57.381988Z DEBUG fuser::request: FUSE(  6) ino 0x0000000000000001 LOOKUP name "ini.txt"    
2023-10-25T14:31:57.425159Z DEBUG fuser::request: FUSE(  8) ino 0x0000000000000001 CREATE name "ini.txt", mode 0o100664, flags 0x8241    
2023-10-25T14:31:57.425492Z DEBUG fuser::request: FUSE( 10) ino 0x0000000000000001 MKNOD name "ini.txt", mode 0o100664, rdev 0    
2023-10-25T14:31:57.471106Z DEBUG fuser::request: FUSE( 12) ino 0x0000000000000002 OPEN flags 0x8001    
2023-10-25T14:31:57.511767Z DEBUG fuser::request: FUSE( 14) ino 0x0000000000000002 FLUSH fh FileHandle(1), lock owner LockOwner(10151073290680498340)    
2023-10-25T14:31:57.512014Z DEBUG fuser::request: FUSE( 16) ino 0x0000000000000002 GETXATTR name "security.capability", size GetSize(GetXAttrSize)    
2023-10-25T14:31:57.512157Z DEBUG fuser::request: FUSE( 18) ino 0x0000000000000002 WRITE fh FileHandle(1), offset 0, size 6, write flags 0x0    
2023-10-25T14:31:57.547845Z DEBUG fuser::request: FUSE( 20) ino 0x0000000000000002 FLUSH fh FileHandle(1), lock owner LockOwner(10151073290680498340)    
2023-10-25T14:31:57.719598Z DEBUG fuser::request: FUSE( 22) ino 0x0000000000000002 RELEASE fh FileHandle(1), flags 0x8001, flush false, lock owner None    
2023-10-25T14:32:48.217609Z DEBUG fuser::request: FUSE( 24) ino 0x0000000000000001 GETATTR    
2023-10-25T14:32:48.221167Z DEBUG fuser::request: FUSE( 26) ino 0x0000000000000002 OPEN flags 0x8001 

Note when we are trying to recreate there is no fuse request to lookup and mknod.

[inevity]

I think the mount-s3 is not designed for this usage scene: multiprotocol and distributed. Any fops shoulde be go through from the only mount point Or console/cli fop should serialize with the mount point fops, not concurren/parallel.
But let the lib developer to give precise answer.

[ddennedy-gpsw]

I ran into this problem as well, and it is not a concurrency issue. All you need is a system with more than one component (or I suppose more than one instance of a component). In our case, worker A creates object at key using mountpoint-s3. A minute later, service B deletes object at key using API. A couple of minutes later, worker A tries to create object at same key again using mountpoint-s3 and gets this error No such file or directory. One may ask why the object is being recreated like this. Because the user changed their mind, or they saved a change to a document that is used to create said object.

[dannycjones]

Hey both! While we don't recommend mutating the same file/object when other processes are reading/writing to it, this isn't the behavior we expect. The use case described in the issue should work - it should be possible for the file to be created.

This issue is actually self-reported by the mountpoint-s3 team, @sauraank has opened pull request #584 that we believe addresses the cause of the issue. It still requires review from our team.

[dannycjones]

Hey folks, this should be fixed in the latest Mountpoint release: https://github.com/awslabs/mountpoint-s3/releases/tag/mountpoint-s3-1.1.1