AWS Managed Opensearch Sink IAM Auth from ECS trying to auth to dashboards uri

[tfmm]

We have an aws-managed Opensearch cluster which is setup to auth via IAM and SSO (Okta) for humans. When setting up the elasticsearch sink, it receives an auth error from opensearch that makes it seem like it is trying to auth to the dashboards endpoint.

Sink config:

sinks:
  es_cluster:
    inputs:
      - alb_xform
    type: elasticsearch
    endpoints:
      - https://SUB_OS_URL:443
    bulk:
      index: "SUB_INDEX-%Y-%m-%d" # daily indices
    aws:
      region: SUB_REGION
    api_version: auto
    opensearch_service_type: managed
    healthcheck:
      enabled: false

Error:

2025-02-11T17:24:44.166755Z ERROR sink{component_kind="sink" component_id=es_cluster component_type=elasticsearch}:request{request_id=2}: vector::sinks::elasticsearch::service: Response contained errors. error_code="http_response_401" response=Response { status: 401, version: HTTP/1.1, headers: {"date": "Tue, 11 Feb 2025 17:24:44 GMT", "content-type": "text/plain; charset=UTF-8", "content-length": "0", "connection": "keep-alive", "access-control-allow-origin": "*", "www-authenticate": "X-Security-IdP realm=\"OpenSearch Security\" location=\"https://OKTA_URL\" requestId=\"REDACTED\""}, body: b"" }

The Endpoint does not have the /_dashboards uri appended to it, and is used successfully by other IAM-based tools to interface with opensearch.

[tfmm]

I'm not sure if it's something in my configuration or what, but this behavior tells me that Vector is not using the rest API URLs internally. I do not know nearly enough Rust to even begin to look into why, so if anyone has any ideas, it would be extremely helpful.

[pront]

You can increase log verbosity with VECTOR_LOG=debug. Also, if you know the API version you can hardcode it instead of using auto.

[tfmm]

I have tried auto as well as v7 with no change to behavior. I'm trying to sift through debug logs, but don't initially see more relevant information than the log entry I provided above.

[tfmm]

If I add an auth:aws section like this to the sink, I get a 403 instead:

sinks:
  es_cluster:
    inputs:
      - alb_xform
    type: elasticsearch
    endpoints:
      - https://redacted.region.es.amazonaws.com:443
    bulk:
      index: "logs-%Y-%m-%d" # daily indices
    auth:
      strategy: aws
      region: us-east-2
      imds:
        connect_timeout_seconds: 10
        max_attempts: 5
        read_timeout_seconds: 10
    aws:
      region: us-east-2
    api_version: v6
    opensearch_service_type: managed
    healthcheck:
      enabled: false

error:

2025-02-13T15:40:10.008637Z ERROR sink{component_kind="sink" component_id=es_cluster component_type=elasticsearch}:request{request_id=1}: vector::sinks::elasticsearch::service: Response contained errors. error_code="http_response_403" response=Response { status: 403, version: HTTP/1.1, headers: {"date": "Thu, 13 Feb 2025 15:40:10 GMT", "content-type": "application/json", "access-control-allow-origin": "*", "connection": "keep-alive"}, body: b"{\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\\n\\nThe Canonical String for this request should have been\\n'POST\\n/_bulk\\ntimeout=60s\\ncontent-type:application/x-ndjson\\nhost:redacted'\\n\"}" }

[tfmm]

So this now looks like a signing issue. Is there some undocumented option or something to setup proper signing?

2025-02-14T13:45:12.656517Z ERROR sink{component_kind="sink" component_id=es_cluster component_type=elasticsearch}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the request. reason="client-side error, 403 Forbidden: {\"message\":\"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\\n\\nThe Canonical String for this request should have been\\n'POST\\n/_bulk\\ntimeout=60s\\ncontent-type:application/x-ndjson\\nhost:redacted.us-east-2.es.amazonaws.com\\nx-amz-date:20250214T134512Z\\nx-amz-security-token:TOKEN_REDACTED\\n\\ncontent-type;host;x-amz-date;x-amz-security-token\\nREDACTED'\\n\\nThe String-to-Sign should have been\\n'AWS4-HMAC-SHA256\\n20250214T134512Z\\n20250214/us-east-2/es/aws4_request\\nREDACTED'\\n\"}" internal_log_rate_limit=true

[pront]

Is this the root cause?
#22442 (comment)
If so, we can close this discussion.

[tfmm]

I believe so, yes.

…

On Fri, Feb 14, 2025, 16:34 Pavlos Rontidis ***@***.***> wrote: Is this the root cause? #22442 (comment) <#22442 (comment)> If so, we can close this discussion. — Reply to this email directly, view it on GitHub <#22422 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABUP2GRXHRGVKRZGT3P3UO32PZOOBAVCNFSM6AAAAABW5UNHBKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMRQGU3TCMQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>