OTLP Logs drop Resource and Scope attributes

[braydonk]

Bug Report

Describe the bug
When you send OTLP logs into Fluent Bit and then send them back out, any Resource or Scope attributes are lost. This is because out_opentelemetry sending logs just initializes and empty resource and scope, thus meaning any data sent in are ignored.

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:
  When this OTLP goes in:

{
  "resourceLogs": [
    {
      "resource": {
        "attributes": [
          {
            "key": "host.name",
            "value": {
              "stringValue": "debian-dev-machine"
            }
          },
          {
            "key": "os.type",
            "value": {
              "stringValue": "linux"
            }
          }
        ]
      },
      "scopeLogs": [
        {
          "scope": {},
          "logRecords": [
            {
              "observedTimeUnixNano": "1700497457844514864",
              "body": {
                "stringValue": "here"
              },
              "attributes": [
                {
                  "key": "log.file.name",
                  "value": {
                    "stringValue": "a.txt"
                  }
                }
              ],
              "traceId": "",
              "spanId": ""
            },
            ...
          ]
        }
      ],
      "schemaUrl": "https://opentelemetry.io/schemas/1.6.1"
    }
  ]
}

I passed it into this Fluent Bit config:

[SERVICE]
    Log_level debug
    Flush 1

[INPUT]
    Name opentelemetry
    Listen 127.0.0.1
    Port 6969

[OUTPUT]
    Name     opentelemetry
    Match    *
    Host     127.0.0.1
    Port     4318
    Logs_uri /v1/logs

Then sent it to another local OTel collector and dumped the result to a file.
This is the OTLP that came out:

{
  "resourceLogs": [
    {
      "resource": {},
      "scopeLogs": [
        {
          "scope": {},
          "logRecords": [
            {
              "timeUnixNano": "1700497457945114315",
              "body": {
                "kvlistValue": {
                  "values": [
                    {
                      "key": "message",
                      "value": {
                        "stringValue": "here"
                      }
                    }
                  ]
                }
              },
              "attributes": [
                {
                  "key": "log.file.name",
                  "value": {
                    "stringValue": "a.txt"
                  }
                }
              ],
              "traceId": "",
              "spanId": ""
            },
            ...
          ]
        }
      ]
    }
  ]
}

• Steps to reproduce the problem:
  The config I mentioned above, and what I did to reproduce was used an otel collector to read logs from a file and export otlphttp to Fluent Bit, then Fluent Bit output to another otel collector that receives the OTLP and dumps the result to a file.

Expected behavior
The data I passed in with OTLP should, at minimum, be the same when it comes out if there is no processing done.

Screenshots

Your Environment

• Version used: built from master
• Configuration: See above
• Environment name and version (e.g. Kubernetes? What version?): normal Debian VM
• Server type and version: Debian bullseye VM
• Operating System and version: Debian 11
• Filters and plugins: See above config

Additional context

This makes Fluent Bit basically unusable for any OTLP-focused pipelines. Losing resource attributes is a particularly big problem, especially because Fluent Bit doesn't appear to provide any way to decorate Resource with attributes to begin with. (That is worth a separate issue probably). I haven't tried traces, but when I tried metrics this wasn't a problem; the data I passed in was what I got out (although all my metric names were prefixed with underscore, which is also definitely a bug, see issue #8083

[braydonk]

This is the problematic section:

fluent-bit/plugins/out_opentelemetry/opentelemetry.c

Lines 764 to 770 in 5626757

	scope_log.log_records = logs;
	scope_log.n_log_records = log_count;
	scope_logs[0] = &scope_log;
	resource_log.scope_logs = scope_logs;
	resource_log.n_scope_logs = 1;
	resource_logs[0] = &resource_log;

[nokute78]

There are two issues to fix this issue.

1. in_opentelemetry ignores resource and scope
2. out_opentelemetry creates blank resource and scope

I sent a patch to fix issue 1 storing resouce and scope as metadata.
#8294

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[braydonk]

Unstale

…

On Sat, Apr 20, 2024, 6:50 PM github-actions[bot] ***@***.***> wrote: This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label. — Reply to this email directly, view it on GitHub <#8205 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWJXJSHVA3PLY6VBIVF2HJDY6MLPPAVCNFSM6AAAAAA7TTWOIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRXHA2DOMRTGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[yoyoraso]

Hi, any updates on this, work arounds , anything. I have same problem, I use k6 to generate logs and sent it to fluentbit incase of http it works as expected and I see full logs but in case of opentelemetry some logs info are missing.

[edsiper]

OTel logs compat issues are solved with this:

• core: add log groups support and full OTLP <--> OTLP  #8898
• https://github.com/fluent/fluent-bit/tree/otlp-metadata-end-to-end

This will be part of v3.1 release (~June 24)

[scila1996]

+1. Same issue

@edsiper Expecting for 3.1 soon

[edsiper]

Closing this ticket since #8898 and other enhancements are already merged.