How do I ship Vector's internal metrics to Honeycomb or an OpenTelemetry collector?

[britton-from-notion]

Problem

Honeycomb as a sink does not support the metrics data type, rendering the shipment of vector's internal metrics to honeycomb incompatible out of the box. However, this telemetry is extremely valuable and I'd love to still make use of it.

Honeycomb expects to receive telemetry shipped in OTLP format, which vector supports via the OpenTelemetry sink but does not provide a pre-built way to translate a metric event into the OTLP proto.

Vector's documentation recommends using VRL to transform data into OTLP which if I'm not mistaken could be a somewhat cumbersome transform. It would be far preferred to have the ability to either send a metric event directly to the honeycomb or OTLP sink (having the transformation occur in the background) or provide a simple means of mapping a metric to OTLP via vector such as a pre-built function.

Example

x Data type mismatch between vector_internal_metrics (["Metric"]) and honeycomb (["Log"])

References

https://vector.dev/docs/reference/configuration/sinks/opentelemetry/
https://vector.dev/docs/reference/configuration/sinks/honeycomb/
https://vector.dev/docs/about/under-the-hood/architecture/data-model/metric/

[britton-from-notion]

Here's a rough idea of what I believe would be the VRL work required to transform one metric type into OTLP

[pront]

Problem

Honeycomb as a sink does not support the metrics data type, rendering the shipment of vector's internal metrics to honeycomb incompatible out of the box. However, this telemetry is extremely valuable and I'd love to still make use of it.

You can open a feature request for enhancing the honeycomb sink.

Honeycomb expects to receive telemetry shipped in OTLP format, which vector supports via the OpenTelemetry sink but does not provide a pre-built way to translate a metric event into the OTLP proto.

The opentelemetry sink is still in beta phase and is a thin wrapper over the http sink. Adding more OOTB support is on our radar. For now, you will have to transform your data and configure the endpoint. Please share your setup if you can get it work, it might take some trial and error. Also, let me know if you find any blockers, we might be able to help.

[britton-from-notion]

I could open a feature request for improvement of the honeycomb sink. But also it seems like the best solution is to add support within vector to translate between the metric data model and the OpenTelemetry metric proto.

I see the metrics proto is imported in this directory and logs are defined in the Rust module.

And I notice you're doing a similar sort of encoding from the internal metrics model for sinks like the AWS cloudwatch metrics sink. So my thought is that adding the metrics types via the Protos and then adding an encoder which transforms from the internal model into those types would both help the honeycomb sink and the OpenTelemetry sink in one shot.

Thoughts?

[pront]

I could open a feature request for improvement of the honeycomb sink.
For this, I meant just a feature request for supporting the metrics in addition to logs. For the OTLP encoding specifically, please read below.

it seems like the best solution is to add support within vector to translate between the metric data model and the OpenTelemetry metric proto.

This is a certainly a desirable feature. We just don't have time to develop it right now. But it would be awesome if the community contributed this. It would also enable batching for OTLP sink.

And I notice you're doing a similar sort of encoding from the internal metrics model for sinks like the AWS cloudwatch metrics sink. So my thought is that adding the metrics types via the Protos and then adding an encoder which transforms from the internal model into those types would both help the honeycomb sink and the OpenTelemetry sink in one shot.

Thoughts?

This is sounds like a good path forward IMO.

[britton-from-notion]

Following up on initial metrics support attempt in #17310