Integrate Tavily search API into web search functionality #860
Conversation
|
The changes introduce a new search provider, Tavily, alongside the existing Bing search functionality. Both search functions now return a list of Improvements & Suggestions:
Overall, the changes add new functionality while maintaining a consistent design pattern. LGTM 🚀
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| ## Tavily Configuration <a href="" id="tavily" /> | ||
|
|
||
| The [Tavily API](https://docs.tavily.com/docs/rest-api/api-reference#endpoint-post-search) | ||
| provides access to a powerfull search engine for LLM agents. |
There was a problem hiding this comment.
The word "powerfull" is misspelled. It should be "powerful".
generated by pr-docs-review-commit
spelling
packages/core/src/promptcontext.ts
Outdated
| try { | ||
| files = await f(q, { trace }) | ||
| break | ||
| } catch (e) {} |
There was a problem hiding this comment.
The loop that attempts to use multiple search providers does not handle errors effectively. If all providers fail, the variable files will remain undefined, leading to a potential runtime error when trying to access it later.
generated by pr-review-commit
error_handling
packages/core/src/websearch.ts
Outdated
|
|
||
| // Create a fetch function for making the HTTP request. | ||
| const fetch = await createFetch({ trace }) | ||
| console.log({ url }) |
There was a problem hiding this comment.
The console.log statement on line 168 is left in the production code. This can lead to unnecessary console output and potential exposure of sensitive information.
generated by pr-review-commit
debug_logging
| const fetch = await createFetch({ trace }) | ||
| console.log({ url }) | ||
| const res = await fetch(url, { | ||
| method: "POST", |
There was a problem hiding this comment.
The Tavily search function uses the HTTP POST method for a search request, which is unconventional. Typically, GET is used for search operations unless there is a specific reason to use POST.
generated by pr-review-commit
http_method
| if (!files) | ||
| throw new Error( | ||
| `No search provider configured. See ${DOCS_WEB_SEARCH_URL}.` | ||
| ) | ||
| trace.files(files, { model, secrets: env.secrets }) | ||
| return files |
There was a problem hiding this comment.
The webSearch function throws an error if no search provider is configured, but this error is not caught or handled within the function. Consider adding error handling to manage this scenario gracefully. 🛠️
generated by pr-review-commit
unhandled_error
| const err = await res.text() | ||
| trace?.detailsFenced("error response", err) | ||
| logVerbose(err) | ||
| throw new Error(`Tavily search failed: ${res.status} ${res.statusText}`) |
There was a problem hiding this comment.
In the tavilySearch function, an error is thrown if the response is not OK, but this error is not caught or handled. Consider implementing error handling to manage failed requests more effectively. 🚨
generated by pr-review-commit
unhandled_error
|
|
||
| // Throw an error if the response is not OK, and log details for debugging. | ||
| if (!res.ok) { | ||
| trace?.detailsFenced("error response", await res.text()) | ||
| throw new Error(`Bing search failed: ${res.statusText}`) | ||
| throw new Error(`Bing search failed: ${res.status} ${res.statusText}`) |
There was a problem hiding this comment.
In the bingSearch function, an error is thrown if the response is not OK, but this error is not caught or handled. Consider implementing error handling to manage failed requests more effectively. 🚨
generated by pr-review-commit
unhandled_error
| if (!files) | ||
| throw new Error( | ||
| `No search provider configured. See ${DOCS_WEB_SEARCH_URL}.` | ||
| ) | ||
| trace.files(files, { model, secrets: env.secrets }) | ||
| return files |
There was a problem hiding this comment.
The webSearch function throws an error if no search provider is configured, but it doesn't handle the case where both bingSearch and tavilySearch return undefined due to missing API keys. Consider adding error handling for this scenario to provide a more informative message to the user. 🛠️
generated by pr-review-commit
error_handling
| const err = await res.text() | ||
| trace?.detailsFenced("error response", err) | ||
| logVerbose(err) | ||
| throw new Error(`Tavily search failed: ${res.status} ${res.statusText}`) |
There was a problem hiding this comment.
In the tavilySearch function, when the response is not OK, the error message is logged but not returned to the caller. Consider including the error details in the thrown error to aid in debugging. 🛠️
generated by pr-review-commit
error_handling
| throw new Error( | ||
| `TAVILY_API_KEY secret is required to use Tavily search. See ${DOCS_WEB_SEARCH_TAVILY_URL}.`, | ||
| { cause: "missing key" } | ||
| ) |
There was a problem hiding this comment.
In the tavilySearch function, if the TAVILY_API_KEY is missing, the function returns undefined when ignoreMissingApiKey is true. This can lead to unexpected behavior. Consider returning an empty array or a more explicit result to indicate the absence of results. 🛠️
generated by pr-review-commit
error_handling
| [Web search](/genaiscript/reference/scripts/web-search) using Bing or Tavily. | ||
|
|
||
| ```js wrap | ||
| const pages = await retreival.webSearch("what are the latest news about AI?") |
There was a problem hiding this comment.
The term "retreival" is likely a typo. Consider changing it to "retrieval".
generated by pr-docs-review-commit
typo
| ## Tavily Configuration <a href="" id="tavily" /> | ||
|
|
||
| The [Tavily API](https://docs.tavily.com/docs/rest-api/api-reference#endpoint-post-search) | ||
| provides access to a powerfull search engine for LLM agents. |
There was a problem hiding this comment.
The word "powerfull" is misspelled. It should be "powerful".
generated by pr-docs-review-commit
typo
| @@ -196,6 +199,11 @@ export const DOCS_CONFIGURATION_CONTENT_SAFETY_URL = | |||
| "https://microsoft.github.io/genaiscript/reference/scripts/content-safety" | |||
| export const DOCS_DEF_FILES_IS_EMPTY_URL = | |||
| "https://microsoft.github.io/genaiscript/reference/scripts/context/#empty-files" | |||
| export const DOCS_WEB_SEARCH_URL = | |||
| "https://microsoft.github.io/genaiscript/reference/scripts/web-search/" | |||
| export const DOCS_WEB_SEARCH_BING_SEARCH_URL = | |||
There was a problem hiding this comment.
The URL in DOCS_WEB_SEARCH_BING_SEARCH_URL seems to have a typo: '#bingn' should likely be '#bing'. Please verify the URL.
generated by pr-review-commit
incorrect_url
|
|
||
| // Retrieve the API key from the runtime host. | ||
| const apiKey = await runtimeHost.readSecret("BING_SEARCH_API_KEY") | ||
| if (!apiKey) | ||
| if (!apiKey) { | ||
| if (ignoreMissingApiKey) return undefined |
There was a problem hiding this comment.
The function bingSearch returns undefined when ignoreMissingApiKey is true, but the return type is Promise<WorkspaceFile[]>. Consider returning an empty array instead.
generated by pr-review-commit
return_type_mismatch
| // Retrieve the API key from the runtime host. | ||
| const apiKey = await runtimeHost.readSecret("TAVILY_API_KEY") | ||
| if (!apiKey) { | ||
| if (ignoreMissingApiKey) return undefined |
There was a problem hiding this comment.
The function tavilySearch returns undefined when ignoreMissingApiKey is true, but the return type is Promise<WorkspaceFile[]>. Consider returning an empty array instead.
generated by pr-review-commit
return_type_mismatch
|
|
||
| // Retrieve the API key from the runtime host. | ||
| const apiKey = await runtimeHost.readSecret("BING_SEARCH_API_KEY") | ||
| if (!apiKey) | ||
| if (!apiKey) { | ||
| if (ignoreMissingApiKey) return undefined | ||
| throw new Error( |
There was a problem hiding this comment.
Returning undefined when the API key is missing and ignoreMissingApiKey is true can lead to unexpected behavior. Consider returning an empty array or a specific error.
generated by pr-review-commit
undefined_return
| if (ignoreMissingApiKey) return undefined | ||
| throw new Error( | ||
| `TAVILY_API_KEY secret is required to use Tavily search. See ${DOCS_WEB_SEARCH_TAVILY_URL}.`, | ||
| { cause: "missing key" } |
There was a problem hiding this comment.
Inconsistent error messages for missing API keys. Ensure uniformity in error messaging for clarity.
generated by pr-review-commit
inconsistent_error_message
| else { | ||
| for (const f of [bingSearch, tavilySearch]) { | ||
| files = await f(q, { ignoreMissingApiKey: true, trace, count }) | ||
| if (files) break |
There was a problem hiding this comment.
Potential infinite loop in webSearch function if neither bingSearch nor tavilySearch returns a result. Ensure a termination condition or maximum retries.
generated by pr-review-commit
infinite_loop
| throw new Error( | ||
| "BING_SEARCH_API_KEY secret is required to use bing search. See https://microsoft.github.io/genaiscript/reference/scripts/web-search/#bing-web-search-configuration." | ||
| `BING_SEARCH_API_KEY secret is required to use bing search. See ${DOCS_WEB_SEARCH_BING_SEARCH_URL}.`, | ||
| { cause: "missing key" } |
There was a problem hiding this comment.
The error message for a missing BING_SEARCH_API_KEY is being constructed with an object as the second argument, which is not standard for the Error constructor. Consider removing the object or using a custom error class.
generated by pr-review-commit
error_handling
Add support for the Tavily search API alongside the existing Bing search functionality, enhancing the web search capabilities. Update documentation to reflect the new integration and provide configuration details.
🔄 File Format Update: Changed the file extension for
web-searchdocumentation from.mdto.mdxto potentially support richer content with React components.🌐 Search Engine Integration:
📜 Documentation and Configuration:
webSearchfunction documentation to describe the new Tavily option.🗺️ Sample Scripts:
dataanalyst.genai.mjs, in samples for working with environment files and variables.mermaid.genai.mjsscript for better context.🛠️ API Changes:
webSearchmethod within theRetrievalinterface now accepts an optionalproviderargument to specify the search service (tavilyorbing).🚀 Enhancements: