improved basic search latency

[evan-danswer]

Description

Before this PR, if a user did basic search with

• old "Agentic" search (LLM-based document filtering) enabled
• a reranking model enabled
  The time-to-first-docs was pretty high because we were running filtering and reranking before sending docs.

We also decided to look for other opportunities to reduce latency. In particular, the first LLM call we make to decide what tool and args to use (if any) often takes as long as a second. In that time, we now run

• our intent model (classifying the query as keyword or non-keyword)
• query embedding (for use in vector db retrieval)
  in parallel with tool choice.

All told, there used to be 1-2 seconds of latency after choosing a tool that is down to 0.1-0.2s with these changes, with the caveat that at worst the newly introduced parallelism makes the tool choice node take worst case 0.5 seconds longer.

NOTE: Technically we could pass the TimeoutThread objects around until their value is absolutely necessary, but there just isn't enough going on between the time I decided to join() and the place where those values are needed, so I opted for code cleanliness.

How Has This Been Tested?

Tested + Benchmarked in UI

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

• This PR should be backported (make sure to check that the backport attempt succeeds)
• [Optional] Override Linear Check

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
internal-search	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 5, 2025 1:31am