in_splunk: add ability to map records from specific tokens to specific tags

[nuclearpidgeon]

Splunk HEC logging is usually set up in a manner where data sent in using particular HEC tokens is indexed in particular spots or ways - see https://docs.splunk.com/Documentation/SplunkCloud/9.3.2408/Data/UsetheHTTPEventCollector#How_the_Splunk_platform_uses_HTTP_Event_Collector_tokens_to_get_data_in . This often means using many HEC tokens for many different sources of data.

To support logging sources following this pattern better I have updated the in_splunk plugin to allow specifying tag mapping rules that will set the fluent tag for incoming records sent with a particular HEC token value to a particular configured tag value.

You can kind of already do this in a bootleg manner by using the rewrite_tag plugin combined with putting the HEC token in the log record, but it involves introducing a new input emitter and buffer, as well as storing the potentially sensitive tag values in the record. This can be cut out all together if the right tag can be set on the way in through the splunk input plugin.

---

Testing

• Example configuration file for the change

[INPUT]
    name splunk
    splunk_token abc,def,ghi
    map_token_to_tag abc tagged_abc
    map_token_to_tag def tagged_def

[OUTPUT]
    name stdout
    match *

• Debug log output from testing the change

cURL test commands:

curl -v -X POST -H "Authorization: Splunk abc" -H "Content-Type: application/json" -d '{}' http://localhost:8088/services/collector/event
curl -v -X POST -H "Authorization: Splunk def" -H "Content-Type: application/json" -d '{}' http://localhost:8088/services/collector/event
curl -v -X POST -H "Authorization: Splunk ghi" -H "Content-Type: application/json" -d '{}' http://localhost:8088/services/collector/event

Fluentbit output:

Fluent Bit v4.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _             ___  _____ 
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/ 


[2025/01/14 17:08:23] [ info] Configuration:
[2025/01/14 17:08:23] [ info]  flush time     | 1.000000 seconds
[2025/01/14 17:08:23] [ info]  grace          | 5 seconds
[2025/01/14 17:08:23] [ info]  daemon         | 0
[2025/01/14 17:08:23] [ info] ___________
[2025/01/14 17:08:23] [ info]  inputs:
[2025/01/14 17:08:23] [ info]      splunk
[2025/01/14 17:08:23] [ info] ___________
[2025/01/14 17:08:23] [ info]  filters:
[2025/01/14 17:08:23] [ info] ___________
[2025/01/14 17:08:23] [ info]  outputs:
[2025/01/14 17:08:23] [ info]      stdout.0
[2025/01/14 17:08:23] [ info] ___________
[2025/01/14 17:08:23] [ info]  collectors:
[2025/01/14 17:08:23] [ info] [fluent bit] version=4.0.0, commit=14ca011ff8, pid=18759
[2025/01/14 17:08:23] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2025/01/14 17:08:23] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/01/14 17:08:23] [ info] [simd    ] disabled
[2025/01/14 17:08:23] [ info] [cmetrics] version=0.9.9
[2025/01/14 17:08:23] [ info] [ctraces ] version=0.5.7
[2025/01/14 17:08:23] [ info] [input:splunk:splunk.0] initializing
[2025/01/14 17:08:23] [ info] [input:splunk:splunk.0] storage_strategy='memory' (memory only)
[2025/01/14 17:08:23] [debug] [splunk:splunk.0] created event channels: read=25 write=26
[2025/01/14 17:08:23] [debug] [input:splunk:splunk.0] token #1 will map to tag tagged_abc
[2025/01/14 17:08:23] [debug] [input:splunk:splunk.0] token #2 will map to tag tagged_def
[2025/01/14 17:08:23] [ warn] [input:splunk:splunk.0] token #3 has no tag mapping, records from this token will not re-map to specific tag
[2025/01/14 17:08:23] [debug] [downstream] listening on 0.0.0.0:8088
[2025/01/14 17:08:23] [debug] [stdout:stdout.0] created event channels: read=28 write=29
[2025/01/14 17:08:23] [ info] [sp] stream processor started
[2025/01/14 17:08:23] [ info] [output:stdout:stdout.0] worker #0 started
[2025/01/14 17:08:46] [debug] [task] created task=0x7c3df0024030 id=0 OK
[0] tagged_abc: [[1736834926.396183503, {"hec_token"=>"Splunk abc"}], {}]
[2025/01/14 17:08:46] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2025/01/14 17:08:46] [debug] [out flush] cb_destroy coro_id=0
[2025/01/14 17:08:46] [debug] [task] destroy task=0x7c3df0024030 (task_id=0)
[2025/01/14 17:08:49] [debug] [task] created task=0x7c3df00242d0 id=0 OK
[2025/01/14 17:08:49] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] tagged_def: [[1736834929.731088564, {"hec_token"=>"Splunk def"}], {}]
[2025/01/14 17:08:49] [debug] [out flush] cb_destroy coro_id=1
[2025/01/14 17:08:49] [debug] [task] destroy task=0x7c3df00242d0 (task_id=0)
[2025/01/14 17:08:51] [debug] [task] created task=0x7c3df0024570 id=0 OK
[2025/01/14 17:08:51] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] splunk.0: [[1736834931.124135388, {"hec_token"=>"Splunk ghi"}], {}]
[2025/01/14 17:08:51] [debug] [out flush] cb_destroy coro_id=2
[2025/01/14 17:08:51] [debug] [task] destroy task=0x7c3df0024570 (task_id=0)

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Happy to add docs for this but would like to get approval on the PR and general approach first.

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.