Splunk log sink http error messages

[matt-simons]

We infrequently see some of our requests fail with a 400 status code e.g.:

2022-03-23T16:20:20.895212Z ERROR sink{component_kind="sink" component_id=splunk component_type=splunk_hec_logs component_name=splunk}:request{request_id=21}: vector::sinks::util::retries: Not retriable; dropping the request. reason="response status: 400 Bad Request"

but unlike the generic http sink the response body containing the reason why it's a Bad Request isn't visible.

e.g.

2022-03-23T16:45:20.809535Z ERROR sink{component_kind="sink" component_id=scalyr component_type=http component_name=scalyr}:request{request_id=17}: vector::sinks::util::retries: Not retriable; dropping the request. reason="response status: 400 Bad Request"
2022-03-23T16:45:20.809652Z ERROR sink{component_kind="sink" component_id=scalyr component_type=http component_name=scalyr}:request{request_id=17}: vector::sinks::util::sink: Response failed. response=Response { status: 400, version: HTTP/1.1, headers: {"server": "nginx", "date": "Wed, 23 Mar 2022 16:45:20 GMT", "content-type": "application/json;charset=UTF-8", "content-length": "83", "connection": "keep-alive", "access-control-allow-credentials": "true", "access-control-allow-methods": "GET, POST, PUT, DELETE, OPTIONS", "access-control-allow-headers": "Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With"}, body: b"{\n  \"message\": \"request is not valid JSON\",\n  \"status\": \"error/client/badRequest\"\n}" }

Is there any way of configuring just the Splunk sink to be more verbose or so that the error message is visible?

setting the global verbose flag emits more details however the body is still omitted

2022-03-24T13:06:59.181727Z DEBUG sink{component_kind="sink" component_id=splunk component_type=splunk_hec_logs component_name=splunk}:request{request_id=2630}:http: vector::internal_events::http_client: HTTP response. status=400 Bad Request version=HTTP/1.1 headers={"date": "Thu, 24 Mar 2022 13:06:57 GMT", "content-type": "application/json; charset=UTF-8", "x-content-type-options": "nosniff", "content-length": "60", "vary": "Authorization", "connection": "Keep-Alive", "x-frame-options": "SAMEORIGIN", "server": "Splunkd"} body=[60 bytes]

[jszwedko]

Hi @matt-simons !

This seems to be a similar issue to #11397

Are you able to grab the Splunk error logs? We are very curious to see what the failed request is due to.

Unfortunately, Vector doesn't log the response body, even when the log level is debug. We have an issue tracking that: #5024

[matt-simons]

I was able to setup mitmproxy to find out what's going on. It looks to be a configuration issue on our side where some of the logs don't have a correct Splunk index set. We use templating to get the index name from a Kubernetes label dynamically.

AFAIK Splunk HEC process all events in a batch prior to an invalid event and all events after that are discarded.

Quick follow up question: Is Vector aware of this behaviour and will it retry sending the other events in the batch or will it discard the entire batch if a single event is invalid?

Thanks

[matt-simons]

My question appears to be related to #11434 and Vector does not retry. I'm not sure if the docs are up to date but it seems to suggest partitioning the batches may be possible and could solve this by grouping on index or sourcetype etc. https://vector.dev/docs/reference/configuration/sinks/splunk_hec_logs/#partitioning If partitioning is possible that would be great for ensuring independent events are not impacted by other misconfigured and unrelated events.

[matt-simons]

Thanks for the info!

I don't have access to the Splunk error logs unfortunately, I'll try to find out the failed request another way.