use max_tokens to do better rate limit handling

[evan-danswer]

Description

Fixes https://linear.app/danswer/issue/DAN-1518/use-max-tokens-in-agent-search-where-possible

• Some timeout bumps to make o1 happy
• use max_tokens to make (especially the smaller) Claude models not give rate limit errors.

From https://docs.anthropic.com/en/api/rate-limits

"OTPM rate limits are estimated based on max_tokens at the beginning of each request, and the estimate is adjusted at the end of the request to reflect the actual number of output tokens used. If you’re hitting OTPM limits earlier than expected, try reducing max_tokens to better approximate the size of your completions."

How Has This Been Tested?

Tested in UI

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

• This PR should be backported (make sure to check that the backport attempt succeeds)
• [Optional] Override Linear Check

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
internal-search	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 6, 2025 11:25pm

[greptile-apps]

PR Summary

This PR adds max_tokens parameters across LLM calls to better handle rate limits, particularly for smaller Claude models, following Anthropic's guidance on OTPM rate limit management.

• Added _should_restrict_tokens utility in utils.py to conditionally apply token restrictions based on LLM provider/model
• Increased timeout values in agent_configs.py to accommodate slower models like o1, though duplicated code blocks should be cleaned up
• Fixed role handling in chat_llm.py where None role was incorrectly defaulting to None instead of 'unknown'
• Renamed run_with_timeout to run_with_timeout_multiproc in timeout.py for better clarity on multiprocessing behavior
• Inconsistency in custom_llm.py where max_tokens parameter is added but not utilized in _execute method, still using fixed _max_output_tokens

_{18 file(s) reviewed, 1 comment(s)
Edit PR Review Bot Settings | Greptile}

[greptile-apps]

logic: Environment variable name AGENT_MAX_TOKENS_INITIAL_ANSWER_GENERATION doesn't match the config variable name AGENT_MAX_TOKENS_ANSWER_GENERATION

Suggested change

	os.environ.get("AGENT_MAX_TOKENS_INITIAL_ANSWER_GENERATION")
	os.environ.get("AGENT_MAX_TOKENS_ANSWER_GENERATION")

[joachim-danswer]

LGTM