k6 to OpenTelemetry Collector

[JCzz]

Hi

Can you document how to use k6 with OpenTelemetry Collector(or Alloy), I am having a hard time to run example like:

K6_OTEL_GRPC_EXPORTER_INSECURE=true K6_OTEL_GRPC_EXPORTER_ENDPOINT=htp://localhost:4317 K6_OTEL_METRIC_PREFIX=k6_ k6 run --out experimental-opentelemetry ./demo.js

or with http: K6_OTEL_HTTP_EXPORTER_ENDPOINT

And using Mimir & Tempo would be very cool.

Thanks in advance

[heitortsergent]

Hi @JCzz! Would you be able to share the output you're seeing after you run that command?

cc'ing @olegbespalov if you might be able to help as well. 🙇

[olegbespalov]

Sure, happy to help and based on the need we could certainly improve our documentation!

@JCzz could you please provide more context about your set-up, like you have up & running Alloy? Or do you want to know how to set up alloy, and how to configure k6 to send metrics to it?

Maybe helpful could be that at the xk6-output-opentelemetry repository we also have a docker-compose with some simple dashboard which implements set up Alloy + Prometheus.

One thing, just to be clear, the current k6's open telemetry output is only about metrics, so it can't integrate with Tempo.

[JCzz]

Hi there

Thanks for replying, I am on vacation until friday, but then I will get back with more. But basically I just need to collect and show metrics, traces and logs in Grafann - from custom instrumentation and k6 metrics from tests.

It would really help with an example using OpenTelemetry Collector og Alloy.

Thanks

[olegbespalov]

It would really help with an example using OpenTelemetry Collector og Alloy.

Sure, there is exactly example of using k6 with allow, the only thing to modify from there is output name, instead of -o xk6-opentelemetry it should be -o experimental-opentelemetry, precisely something like:

 K6_OTEL_GRPC_EXPORTER_INSECURE=true K6_OTEL_METRIC_PREFIX=k6_ k6 run --tag testid=1 -o experimental-opentelemetry examples/script.js

Which starts to export metrics using some default configuration, where it uses localhost:4317. And Alloy, in example, configured to forward metrics to the Prometheus.

Once you return and check, please let me know if that helps, or I could provide more context.

[JCzz]

Very good Alloy example in:
https://github.com/grafana/xk6-output-opentelemetry/blob/main/grafana/alloy/config.alloy

Is there anyway I can use Mimir instead of Prometheus for storing Metrics - maybe it is just to add Mimir instead of Prometheus in Docker Compose?

[olegbespalov]

Sure! Grafana Alloy supports forwarding to many metrics BE including Mimir.

Is there anyway I can use Mimir instead of Prometheus for storing Metrics - maybe it is just to add Mimir instead of Prometheus in Docker Compose?

We keep Prometheus there for simplicity, and just for showcasing.

If you look into Alloy to Mimir I'd recommend checking out alloy's examples. Please note that all examples still just examples to showcase the working integrations and configuration for production set up could be different, depending on your case.

[JCzz]

I have tried running the following with the beneath changed configurations to https://github.com/grafana/xk6-output-opentelemetry/tree/main examples:

K6_OTEL_GRPC_EXPORTER_INSECURE=true K6_OTEL_METRIC_PREFIX=k6_ k6 run --tag testid=1 -o experimental-opentelemetry examples/script.js

But I just get "Failed to call resource"... in Grafana?

docker-compose

version: '3.8'

networks:
  k6:

services:
  # prometheus:
  #   image: prom/prometheus:v2.52.0
  #   command:
  #     - --web.enable-remote-write-receiver
  #     - --enable-feature=native-histograms
  #     - --config.file=/etc/prometheus/prometheus.yml
  #   networks:
  #     - k6
  #   ports:
  #     - "9090:9090"

  mimir:
    image: grafana/mimir:latest
    ports:
      - "9009:9009"
    networks:
      - k6

  alloy:
    image: grafana/alloy:latest # v1.0.0
    networks:
      - k6
    volumes:
      - ./grafana/alloy:/etc/alloy
    environment:
      REMOTE_WRITE_HOST: mimir:9009 # prometheus:9090
    depends_on:
      - mimir # prometheus
    command:
      - run
      - /etc/alloy/config.alloy
      - --storage.path=/var/lib/alloy/data
      - --server.http.listen-addr=0.0.0.0:12345
      - --stability.level=experimental # Enable all functionality
    ports:
      - "12345:12345" # Alloy UI
      - "4317:4317" # gRPC receiver
      - "4318:4318" # HTTP receiver

  grafana:
    image: grafana/grafana:10.1.2
    networks:
      - k6
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_BASIC_ENABLED=false
    volumes:
      - ./grafana:/etc/grafana/provisioning/`](https://github.com/grafana/xk6-output-opentelemetry/tree/main)

Alloy config

// expose OTEL receivers
// https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.otlp/
otelcol.receiver.otlp "default" {
	// configures the default gRPC receiver endpoint "0.0.0.0:4317"
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.otlp/#grpc-block
	grpc {}
	
	// configures the default HTTP receiver endpoint "0.0.0.0:4318"
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.receiver.otlp/#http-block
	http {}

	output {
		metrics = [otelcol.processor.batch.default.input]
	}
}

otelcol.processor.batch "default" {
	// https://grafana.com/docs/alloy/latest/reference/components/otelcol.processor.batch/
	output {
		metrics = [otelcol.exporter.prometheus.default.input]		
	}
}

otelcol.exporter.prometheus "default" {
	add_metric_suffixes = false
	forward_to          = [prometheus.remote_write.mimir.receiver]
}

// Collect metrics from the local running Alloy instance and forward to
// Prometheus.
prometheus.exporter.self "alloy" {}
prometheus.scrape "alloy" {
	targets    = prometheus.exporter.self.alloy.targets
	forward_to = [prometheus.remote_write.mimir.receiver]
}

prometheus.remote_write "mimir" {
	endpoint {
		url = string.format(
			"http://%s/api/v1/push",
			coalesce(sys.env("REMOTE_WRITE_HOST"), "localhost:9009"),
		)
	}
}

// prometheus.remote_write "default" {
// 	endpoint {
// 		url = format(
// 			"http://%s/api/v1/write",
// 			coalesce(env("REMOTE_WRITE_HOST"), "localhost:9090"),
// 		)
// 	}
// }

Grafana Data source

# For configuration options, see
#   https://grafana.com/docs/grafana/latest/administration/provisioning/#example-data-source-config-file

apiVersion: 1

datasources:
  - name: prometheus
    type: prometheus
    access: proxy
    url: http://mimir:9009/prometheus
    basicAuth: false
    isDefault: true
    jsonData:
      tlsAuth: false
      tlsAuthWithCACert: false
    editable: false

[olegbespalov]

Hi @JCzz

let me to double-check 🤔

But I just get "Failed to call resource"... in Grafana?

Do you get this error in Grafana or k6?

[JCzz]

Thanks @olegbespalov

The error shows up in Grafana... note that I am using Mimir directly from Alloy - using Prometheus remote write. I have tried both "push" and "remote write". The config above is currently set to "push", but should be "write" if possible. Hope that is a valid way using the Prometheus included in Alloy. As I understand both Opentelemetry Collector and Prometheus is wrapped in Alloy?

[JCzz]

Anyone that have this working: k6 >> Alloy >> Mimir >> Grafana

[olegbespalov]

Hi @JCzz

Sorry, I'm pretty sure that is a configuration error somewhere, but I didn't have an opportunity to look at it, and unfortunately won't have time to look at it next week.

Try to have a look to the example alloy's examples https://github.com/grafana/alloy/tree/main/example, they should have a working Alloy >> Mimir >> Grafana path and after that you just need to ensure that the alloy's gRPC or HTTP OTEL receivers ports are available and you could use the k6's OTEL output.

Hope that helps.

[heitortsergent]

@JCzz you could also try posting in the Grafana Labs community forum: https://community.grafana.com/c/grafana-k6/70, I think that gets a little more visibility, and maybe someone else from the community has the same configuration setup that you're looking for.