Grafana Alloy unable to send traces to Grafana Tempo on same network - unknown service opentelemetry.proto.collector.trace.v1.TraceService.

[henryobiaraije]

What's wrong?

From the docker alloy container, I get this error

2025-02-20 16:09:29 ts=2025-02-20T15:09:29.962010795Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlp.tempo error="not retryable error: Permanent error: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.trace.v1.TraceService" dropped_items=18

And I am successfully sending traces from my nextjs app to alloy. It looks like alloy is having issues sending the traces to tempo.

Steps to reproduce

docker-compose.yaml

networks:
  loki:

services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - ./loki/local-config.yaml:/etc/loki/local-config.yaml
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  tempo:
    image: grafana/tempo:latest
    command: [ "-config.file=/etc/tempo.yaml" ]
    volumes:
      - ./tempo/tempo.yaml:/etc/tempo.yaml 
    networks:
      - loki
    # Add these resource limits
    deploy:
      resources:
        limits:
          memory: 8G
        reservations:
          memory: 5G

  alloy:
    image: grafana/alloy:latest
    volumes:
      - ./alloy/config.alloy:/etc/alloy/config.alloy
    ports:
      - "12345:12345"
      - "4317:4317"
      - "4318:4318"
    networks:
      - loki
    command:
      - run
      - --server.http.listen-addr=0.0.0.0:12345
      - --storage.path=/var/lib/alloy/data
      - /etc/alloy/config.alloy
    depends_on:
      - tempo

  promtail:
    environment:
      - GRAFANA_TOKEN={{GRAFANA_TOKEN}}
    image: grafana/promtail:latest
    volumes:
      - ./promtail/promtail-config.yml:/etc/promtail/promtail-config.yml
      - ${LOGS_LOCATION}:/var/log/
    command: -config.file=/etc/promtail/promtail-config.yml
    networks:
      - loki

  grafana:
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_SERVER_HTTP_TIMEOUT=1200
      # Add these environment variables
      - GF_DATAPROXY_TIMEOUT=1200
      - GF_DATAPROXY_KEEP_ALIVE_SECONDS=120
      # Add these new environment variables
      - GF_SERVER_MAX_REQUEST_SIZE=512
      - GF_SERVER_MAX_BODY_SIZE=512
      # Add these environment variables to increase message size limits
      - GF_DATAPROXY_MAX_RESPONSE_BYTES=499919430400
      - GF_DATAPROXY_RESPONSE_LIMIT=499919430400
    volumes:
      - ./grafana/grafana.ini:/etc/grafana/grafana.ini
    command:
      - -config=/etc/grafana/grafana.ini
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy 
          orgId: 1
          url: http://loki:3100
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
          jsonData:
            timeout: 120
            maxLines: 1000
            maxDataPoints: 1000
            tlsSkipVerify: true
        - name: Tempo
          type: tempo
          access: proxy
          orgId: 1
          url: http://tempo:3200
          basicAuth: false
          version: 1
          editable: false
        EOF
        /run.sh
    image: grafana/grafana:latest
    ports:
      - "3101:3000"
    networks:
      - loki

Then allloy config

livedebugging {
  enabled = true
}

otelcol.receiver.otlp "default" {
  http {
    endpoint = "0.0.0.0:4318"
  }
  grpc {
    endpoint = "0.0.0.0:4317"
  }

  output {
    // logs = [otelcol.exporter.loki.default.input]
    traces = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    // metrics = [otelcol.exporter.otlp.production.input]
    // logs    = [otelcol.exporter.otlp.production.input]
    traces  = [otelcol.exporter.otlp.tempo.input]
  }
}

otelcol.exporter.loki "default" {
  forward_to = [loki.write.local.receiver]
}

otelcol.exporter.otlp "tempo" {
    client {
        endpoint = "tempo:9095"
        tls {
            insecure             = true
            insecure_skip_verify = true
        }
    }
}

loki.write "local" {
  endpoint {
    url = "loki:3100"
  }
}

Then tempo config

server:
  http_listen_port: 3200
  grpc_listen_port: 9095
  grpc_server_max_recv_msg_size: 104857600  # 100MB
  grpc_server_max_send_msg_size: 104857600  # 100MB

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
        http:

storage:
  trace:
    backend: local
    local:
      path: /tmp/tempo/blocks

System information

On windows docker

Software version

No response

Configuration

Logs

2025-02-20 16:09:29 ts=2025-02-20T15:09:29.962010795Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlp.tempo error="not retryable error: Permanent error: rpc error: code = Unimplemented desc = unknown service opentelemetry.proto.collector.trace.v1.TraceService" dropped_items=18

[UserProgrammer]

Dealing with the same issue. My configuration is nearly identical to OP's

[henryobiaraije]

I am still looking for a solution. I can't tell if its a bug from alloy or from tempo.

[Joren-vanGoethem]

same problem here but using yaml as config for alloy as well
also i'm on linux, not windows but similar issue
ts=2025-02-24T18:09:44.587837381Z level=info msg="Exporting failed. Will retry the request after interval." component_path=/ component_id=otelcol.exporter.otlp.tempo error="rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp [::1]:4317: connect: connection refused\"" interval=42.505203049s

[henryobiaraije]

For now, I've changed to otel collector and it successfully receiving and sending traces to tempo. I am still looking for a solution to this as I'll like to stick with the grafan stack if I can.