distro: Add opt-in logs support

[ysolomchenko]

Why

Fixes #3514

What

Add opt-in logs support. The user has to set OTEL_LOGS_EXPORTER env var to otlp to enable the OTLP gRPC logs exporter (default is none).

In a separate PR I will expand example with logging support.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 71.92982% with 16 lines in your changes missing coverage. Please review.

Project coverage is 79.43%. Comparing base (ce88a64) to head (4d94cdd).

Files with missing lines	Patch %	Lines
distro/exporter.go	68.96%	7 Missing and 2 partials ⚠️
distro/otel.go	74.07%	5 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3673      +/-   ##
==========================================
- Coverage   79.56%   79.43%   -0.13%     
==========================================
  Files          88       88              
  Lines        3503     3560      +57     
==========================================
+ Hits         2787     2828      +41     
- Misses        643      655      +12     
- Partials       73       77       +4     

Flag	Coverage Δ
Linux	79.15% <71.92%> (-0.12%)	⬇️
Windows	75.24% <71.92%> (-0.07%)	⬇️
macOS	75.25% <71.92%> (-0.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[MrAlias]

What about HTTP support?

[pellared]

This is OK.
It is only needed for our trace and metrics ingest endpoints. We do not have logs ingest (sic!).

[pellared]

I think that it is worth adding a comment that SPLUNK_REALM is not supported as our Splunk Observability ingest does not support OTLP.

[MrAlias]

What about sending logs to a local collector over HTTP?

[pellared]

@MrAlias

We are always sending to collector using gRPC for all signals.
See:

• splunk-otel-go/distro/exporter.go

  Lines 60 to 92 in 46aa0ab

  	func newOTLPTracesExporter(_ logr.Logger, c *exporterConfig) (trace.SpanExporter, error) {
  	ctx := context.Background()
  	splunkEndpoint := otlpRealmTracesEndpoint()
  	if splunkEndpoint != "" {
  	// Direct ingest to Splunk Observabilty Cloud using HTTP/protobuf.
  	return otlptracehttp.New(ctx,
  	otlptracehttp.WithEndpoint(splunkEndpoint),
  	otlptracehttp.WithURLPath(otlpRealmTracesEndpointPath),
  	otlptracehttp.WithHeaders(map[string]string{
  	"X-Sf-Token": c.AccessToken,
  	}),
  	)
  	}
  	var opts []otlptracegrpc.Option
  	if c.AccessToken != "" {
  	opts = append(opts, otlptracegrpc.WithHeaders(map[string]string{
  	"X-Sf-Token": c.AccessToken,
  	}))
  	}
  	if c.TLSConfig != nil {
  	tlsCreds := credentials.NewTLS(c.TLSConfig)
  	opts = append(opts, otlptracegrpc.WithTLSCredentials(tlsCreds))
  	} else if noneEnvVarSet(otelExporterOTLPEndpointKey, otelExporterOTLPTracesEndpointKey, splunkRealmKey) {
  	// Assume that the default endpoint (local collector) is non-TLS.
  	opts = append(opts, otlptracegrpc.WithTLSCredentials(insecure.NewCredentials()))
  	}
  	return otlptracegrpc.New(ctx, opts...)
  	}

• splunk-otel-go/distro/exporter.go

  Lines 205 to 237 in 46aa0ab

  	func newOTLPMetricsExporter(c *exporterConfig) (metric.Exporter, error) {
  	ctx := context.Background()
  	splunkEndpoint := otlpRealmMetricsEndpoint()
  	if splunkEndpoint != "" {
  	// Direct ingest to Splunk Observabilty Cloud using HTTP/protobuf.
  	return otlpmetrichttp.New(ctx,
  	otlpmetrichttp.WithEndpoint(splunkEndpoint),
  	otlpmetrichttp.WithURLPath(otlpRealmMetricsEndpointPath),
  	otlpmetrichttp.WithHeaders(map[string]string{
  	"X-Sf-Token": c.AccessToken,
  	}),
  	)
  	}
  	var opts []otlpmetricgrpc.Option
  	if c.AccessToken != "" {
  	opts = append(opts, otlpmetricgrpc.WithHeaders(map[string]string{
  	"X-Sf-Token": c.AccessToken,
  	}))
  	}
  	if c.TLSConfig != nil {
  	tlsCreds := credentials.NewTLS(c.TLSConfig)
  	opts = append(opts, otlpmetricgrpc.WithTLSCredentials(tlsCreds))
  	} else if noneEnvVarSet(otelExporterOTLPEndpointKey, otelExporterOTLPMetricsEndpointKey, splunkRealmKey) {
  	// Assume that the default endpoint (local collector) is non-TLS.
  	opts = append(opts, otlpmetricgrpc.WithTLSCredentials(insecure.NewCredentials()))
  	}
  	return otlpmetricgrpc.New(ctx, opts...)
  	}

There was never an ask to add support for OTEL_EXPORTER_OTLP_PROTOCOL.

Maybe we can consider reusing autoexport module as a separate PR. What do you think?

[MrAlias]

I'm confused. Why do we support HTTP for the other signals?

[pellared]

// Direct ingest to Splunk Observabilty Cloud using HTTP/protobuf. 

Because back in the days ingest was not supporting gRPC.

https://github.com/signalfx/gdi-specification/blob/6981bf3249dd1466dd89fca1a73428d0c14bd451/specification/configuration.md?plain=1#L311

Even if we do support gRPC as well I heard that it is better to use http/protobuf with our backend. This is why AFAIK all our distros are using http/protobuf when SPLUNK_REALM is used. We use grpc in other cases for backwards compatibility.

[MrAlias]

Even if we do support gRPC as well I heard that it is better to use http/protobuf with our backend. This is why AFAIK all our distros are using http/protobuf when SPLUNK_REALM is used. We use grpc in other cases for backwards compatibility.

Right ... so shouldn't we support HTTP? This is only adding gRPC exporters but you are saying that HTTP is preferred, right?

[pellared]

Right ... so shouldn't we support HTTP? This is only adding gRPC exporters but you are saying that HTTP is preferred, right?

Ship has sailed.
It would break users that use OTEL_EXPORTER_OTLP_ENDPOINT which targets gRPC endpoints.

Reference: https://docs.splunk.com/observability/en/gdi/get-data-in/application/go/configuration/advanced-go-otel-configuration.html

Back in the days we required to default to grpc in the GDI spec...