Chat: unify keyword rewrite and extraction #6767
Conversation
|
Usually we run the context evals without query rewriting. I enabled rewriting, then ran the baseline vs. the changes in this PR. The results are extremely similar:
Moreover, when examining the rewritten queries, they tended to be les noisy, and to better preserve symbols from the query. Aside: I think we should run evals with query rewriting enabled by default, to better match the chat experience. |
| @@ -10,7 +10,7 @@ import { outputChannelLogger } from '../output-channel-logger' | |||
|
|
|||
| import { francAll } from 'franc-min' | |||
|
|
|||
| const containsMultipleSentences = /[.!?][\s\r\n]+\w/ | |||
| const containsMultipleSentences = /[\n\r]|[.!?]\s*\w/ | |||
There was a problem hiding this comment.
In a follow-up, I'll look into removing this heuristic altogether. Previously in evals, we saw that rewriting tends to make performance much worse for simple queries. However, the new rewrite strategy is "lighter touch", so I'm hopeful it will work well even for short queries. It will be a nice behavior simplification to always rewrite, instead of having a heuristic here.
| @@ -118,7 +60,7 @@ export async function extractKeywords( | |||
| ...preamble, | |||
| { | |||
| speaker: 'human', | |||
| text: ps`You are helping the user search over a codebase. List terms that could be found literally in code snippets or file names relevant to answering the user's query. Limit your results to terms that are in the user's query. Present your results in a *single* XML list in the following format: <keywords><keyword>a single keyword</keyword></keywords>. Here is the user query: <userQuery>${query}</userQuery>`, | |||
| text: ps`You are helping the user search over a codebase. List English terms that could be found literally in code snippets or file names relevant to answering the user's query. Limit your results to terms that are in the user's query. Present your results in a *single* XML list in the following format: <keywords><keyword>a single keyword</keyword></keywords>. Here is the user query: <userQuery>${query}</userQuery>`, | |||
There was a problem hiding this comment.
I had to add 'English' here to keep the nice behavior where we translate foreign languages to common English code terms. I was worried that the LLM might start excluding symbols or acronyms because of this, but I see no evidence of that... many rewritten queries still contain terms like symf, zoekt, etc.
In #6655 we introduced a method for extracting keywords from chat questions that look like simple searches. Looking into the prompt, it's actually an improvement over our current prompt in
rewriteKeywordQuery. It also fixes a bug where we failed to parse the response when it contained a single keyword. This PR switches to the newextractKeywordsmethod to unify our approach.Other changes:
Test plan
Adapted unit tests. Reran context evals -- see next comment for results.