Unable to upload file to bucket: getting 404

[mswami1]

Error:

Writer.Close: googleapi: Error 404: Not Found, Not Found

docker service:

storage:
image: fsouza/fake-gcs-server
environment:
STORAGE_EMULATOR_HOST: localhost:4443
ports:
- '4443:4443'
command: -scheme http -port 4443 -public-host localhost:4443
volumes:
- .:/gcs_storage

client, err1 := storage.NewClient(context.TODO(), option.WithEndpoint("http://127.0.0.1:4443/storage/v1/"))

[KeithScheiwiller]

Did you create a directory within /storage with the matching bucket name? I faced the same issue but doing so solved it. I didn't see it mentioned in the docs, but it seems it does not auto-create buckets.

[fsouza]

Yeah upload will fail if the bucket doesn't exist. I'll keep this open to improve the error message or at least log that the bucket doesn't exist.

[daaain]

Does the bucket need to be created with an API call after startup?

I tried mounting a volume with a directory matching the bucket name, but that didn't help.

Also, is /storage a separate thing from the -data parameter?

services:
  fake-gcs:
    container_name: fake-gcs
    image: fsouza/fake-gcs-server:latest
    entrypoint: /bin/fake-gcs-server
    command: -data /storage -scheme both -cors-headers x-goog-content-length-range -public-host fake-gcs
    ports:
      - "127.0.0.1:9023:4443"
      - "127.0.0.1:8000:8000"
    volumes:
      - ${PWD}/.docker/gcs-emulator:/storage

[daaain]

To answer my own question, /data is what supposed to be a volume and /storage is the internal backend where *.bucketMetadata and *.metadata files get created, so if you set -data /storage and have a volume attached to /storage you'll get one level deeper *.metadata.metadata on every restart 😅

Also, attaching a subdirectory to /data does make the emulator treat it as an existing bucket. Also, files uploaded via the emulator only get written to /storage, but not to /data, so it's just an initial seed.

Would be really helpful if this was documented!

So I can upload to the bucket using the gsutil example, but I still get 404 when trying to use a signed URL generated by the Node SDK 🤷 the container logs it so the request gets through, it just returns 404

This is how the logs look like, the first POST is the successful gsutil upload, then a successful OPTION for CORS and then the 404 signed upload PUT:

2025-02-03 16:51:44 time=2025-02-03T16:51:44.277Z level=INFO msg="server started at http://0.0.0.0:8000"
2025-02-03 16:51:44 time=2025-02-03T16:51:44.277Z level=INFO msg="server started at https://0.0.0.0:4443"
2025-02-03 16:51:55 time=2025-02-03T16:51:55.580Z level=INFO msg="172.20.0.1 - - [03/Feb/2025:16:51:55 +0000] \"POST /upload/storage/v1/b/platform-onboarding-uploads/o?alt=json&fields=crc32c%2Cgeneration%2Cmd5Hash%2CcustomerEncryption%2Csize%2Cetag&uploadType=multipart HTTP/1.1\" 200 902\n"
2025-02-03 17:02:13 time=2025-02-03T17:02:13.744Z level=INFO msg="172.20.0.1 - - [03/Feb/2025:17:02:13 +0000] \"OPTIONS /platform-onboarding-uploads/bec97b0702b47525ada988ca6af9705b/cm6p0vpgj0002qzs7ticlxhdp/DeepSeek_R1.pdf?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=... HTTP/1.1\" 200 0\n"
2025-02-03 17:02:13 time=2025-02-03T17:02:13.747Z level=INFO msg="172.20.0.1 - - [03/Feb/2025:17:02:13 +0000] \"PUT /platform-onboarding-uploads/bec97b0702b47525ada988ca6af9705b/cm6p0vpgj0002qzs7ticlxhdp/DeepSeek_R1.pdf?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=... HTTP/1.1\" 404 119\n"

[daaain]

Got it working! Apparently it was a public-host mismatch, with this configuration the PUT signed URL did upload:

services:
  fake-gcs:
    container_name: fake-gcs
    image: fsouza/fake-gcs-server:latest
    command: -scheme both -cors-headers x-goog-content-length-range,content-length -public-host 127.0.0.1:8000
    ports:
      - "127.0.0.1:9023:4443"
      - "127.0.0.1:8000:8000"
    volumes:
      - ${PWD}/.docker/gcs-emulator:/data
  app:
    ...
    environment:
      STORAGE_EMULATOR_HOST: http://127.0.0.1:8000