[BUG]:Traces Dropped After 5 Retries When Sending to Local Agent Proxy

[ryuichi1208]

Tracer Version(s)

2.19.2

Python Version(s)

Python 3.11

Pip Version(s)

pip 22.04

Bug Report

When using version 2.19.2, the following error message occurs:

failed to send, dropping 1 traces to intake at http://localhost:8126/evp_proxy/v2/api/v2/llmobs

This issue arises in our application, which is a WebSocket-based service built with gunicorn and uses uvicorn as its worker. The traces fail to be sent to the local agent proxy (http://localhost:8126) after 5 retry attempts, leading to the traces being dropped. However, this issue is limited to traces sent to LLMObs, as traces to other destinations were successfully sent without any issues.

This issue does not occur in versions 2.18.2 or 2.18.0. If you need any additional information to investigate the problem, please let us know. We are happy to provide further details or logs as required.

Operating System

AWS Fargate for Amazon ECS

[adilnaimi]

I have the same issue as well:
Flask socketio application (flask-socketio)
gunicorn + eventlet
Deployed to ECS Fargate
ddtrace==2.19.0

failed to send, dropping 2 traces to intake at http://localhost:8126/evp_proxy/v2/api/v2/llmobs after 5 retries

[Yun-Kim]

Hi @ryuichi1208, thanks for reaching out and apologies for the delayed response.
Can you try upgrading to the latest ddtrace==2.20.0 or ddtrace==2.19.3 and see if this issue persists? I believe this may be related/fixed by #11961 which was merged after ddtrace==2.19.2.
If the issue persists, can you send over your tracer debug logs (DD_TRACE_DEBUG=1) to help us debug the underlying issue? Thanks!

[blimmer]

We are getting this on ddtrace==2.20.0 as well. We hadn't seen this on 2.18.0, our previous version.