Metrics can't be send due to gRPC error ResourceExhausted

[kuzaxak]

time="2024-09-19T11:58:13Z" level=info msg="2024/09/19 11:58:13 failed to upload metrics: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4755553 vs. 4194304)"
time="2024-09-19T12:03:03Z" level=info msg="2024/09/19 12:03:03 failed to upload metrics: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (7880311 vs. 4194304)"

Seems if message wasn't delivered once it tries to redeliver and keep increasing the message size.

[olegbespalov]

Hi @kuzaxak

Thanks for reporting the bug, yes it seems like we're hitting the limits 😢

Briefly looking I can see that there is an open issue open-telemetry/opentelemetry-go#4266 which should help here, but as the temporary alternative we could introduce the ability to set the parameters for gRPC

[FilipeGoncalvesBasecone]

As Kuzaxak mentioned I am having the same issues using grpc as well so I had to switch to using http instead to send the metrics without an issue.

After a couple of minutes the gRPC error ResourceExhausted is shown in the console and it stops sending metrics.

[tpodom]

Switching to http sort of worked for me. I eventually started hitting errors reporting 400 Bad Request from the otel collector receiving the HTTP message. I ended up having to modify the otel collector to allow larger body sizes and then drop the url & name attributes because our URLs resulted in really high cardinality and then caused problems with the upstream collection.

I think having some way in k6 to reduce the cardinality of the url & name attributes may be helpful but also sending the metrics in smaller batches would get around the max size issues.

[FilipeGoncalvesBasecone]

Yeah I have the same issue when I tested with otel collector @tpodom.
Using gRPC I get the ResourceExhausted error after a while
Using http it sends metrics but at some point I start getting 400 error's.
In the company I work on we use DataDog so I am sending the metrics to the DataDog agent as it supports open telemetry and using http it sends the metrics from start to finish of the test but it seems the agent has a massive memory leak as it keeps consuming RAM like there is no tomorrow 😅