Rate Limiting

[samschiff19]

Rate Limiting Issues with Multi-threaded Training

Issue

When training with num_threads > 1, operations are consistently throttled by rate limits, even when configured with high API limits.

Current Behavior

• Training attempts with multiple threads (num_threads > 1) consistently hit rate limits
• This occurs even when API limits are set high
• Currently no automatic fallback mechanism exists when rate limits are hit

Proposed Solution

Implementation of an automatic fallback system that would:

1. Accept multiple LLM configurations
2. Automatically switch to alternate configurations when rate limits are encountered

Related Issues

While there are existing issues discussing litellm router configurations, this proposal suggests directly implementing fallback functionality within the core system.