refactor: Add other bedrock models #432
Conversation
…to add_other_bedrock_models
…d rag tools to agent config
11zhouxuan
changed the title
| from langchain.llms.sagemaker_endpoint import LLMContentHandler, SagemakerEndpoint | ||
| from langchain.embeddings import SagemakerEndpointEmbeddings, BedrockEmbeddings | ||
| from langchain.embeddings.sagemaker_endpoint import EmbeddingsContentHandler | ||
| from langchain_community.llms import SagemakerEndpoint |
There was a problem hiding this comment.
Does glue job's requirements.txt need to be update as well?
langchain -> langchain_community
There was a problem hiding this comment.
Yes, need to add "langchain_community" to it's requirements.txt
| @@ -22,8 +22,11 @@ export const LLM_BOT_MODEL_LIST = [ | |||
| ]; | |||
|
|
|||
| export const LLM_BOT_COMMON_MODEL_LIST = [ | |||
| 'anthropic.claude-3-sonnet-20240229-v1:0', | |||
| 'anthropic.claude-3-haiku-20240307-v1:0', | |||
| 'anthropic.claude-3-5-sonnet-20240620-v1:0', | |||
There was a problem hiding this comment.
Some user are still using claude 3,
we can provide them both claude 3 and claude 3.5
would you add claude 3 sonnet and haiku back?
| @@ -2,6 +2,7 @@ | |||
| import traceback | |||
| import uuid | |||
| from datetime import datetime, timezone | |||
| import traceback | |||
There was a problem hiding this comment.
Duplicate with L2, two traceback imported
|
|
||
| from common_logic.common_utils.chatbot_utils import ChatbotManager | ||
| from common_logic.common_utils.constant import ( | ||
| IndexType, | ||
| LLMTaskType, | ||
| SceneType, | ||
| ToolRuningMode, | ||
| GUIDE_INTENTION_NOT_FOUND, |
There was a problem hiding this comment.
Duplicate with L13, keep only one
| @@ -179,4 +188,5 @@ class KBType(Enum): | |||
|
|
|||
| class Threshold(ConstantBase): | |||
| QQ_IN_RAG_CONTEXT = 0.5 | |||
| INTENTION_ALL_KNOWLEDGAE_RETRIEVE = 0.4 | |||
There was a problem hiding this comment.
Typo issue,
KNOWLEDGE_RETRIEVAL
There was a problem hiding this comment.
Is it a good practice to add __ or _ in a folder name? never see this before in popular repos
There was a problem hiding this comment.
It means the files should be retired, i have removed those files.
|
|
||
|
|
||
| AGENT_SYSTEM_PROMPT_COHERE = """\ | ||
| ## Task & Context |
There was a problem hiding this comment.
Better to have in the furture: move the prompt templates into a config or constant file, separate templates with codes
There was a problem hiding this comment.
ok, good suggestion.
| Claude2ChatChain, | ||
| Claude21ChatChain, | ||
| ClaudeInstanceChatChain, | ||
| Iternlm2Chat7BChatChain, |
refactor: add other bedrock models
🤖 AI-Generated PR Description (Powered by Amazon Bedrock)
Description
This pull request includes several changes to the codebase, primarily focused on reorganizing and renaming files within the
source/lambda/online/__functions/directory. The changes are summarized as follows:__functions/__init__.pyfile has been added, potentially indicating the creation of a new Python package._tool_base.py,retriever.py, andwebsearch_retrievers.py.functions_utils/retriever/utils/have been marked with an error status, suggesting potential issues or conflicts.lambda_aws_qa_tools/,lambda_common_tools/, andlambda_retail_tools/have also been marked with an error status.rag.pyfile underlambda_common_tools/has been renamed.Type of change
Motivation and Context
The motivation behind these changes is not explicitly stated, but it appears to be a reorganization and restructuring of the codebase, potentially to improve code organization, modularity, or to introduce new features or functionality.
Dependencies
Due to the nature of the changes, there may be dependencies or conflicts that need to be resolved, especially for the files and directories marked with an error status. It would be advisable to investigate these errors and ensure that any necessary dependencies or compatibility issues are addressed before merging this pull request.
File Stats Summary
File number involved in this PR: 7, unfold to see the details:
The file changes summary is as follows:
lambdaOnlineFunctionsLambda function, as well as the granting of permissions for other Lambda functions to invoke it.AmazonKnowledgeBasesRetrieverimport from the core LangChain package to the community package.langchain_communitypackage instead oflangchain, updating type annotations forcontent_typeandacceptsvariables, and modifying theExtraconfiguration for theConfigclass to use itsvalueattribute.🤖 AI-Generated PR Description (Powered by Amazon Bedrock)
Description
This pull request includes a refactoring of the project's source code structure and file organization. The primary changes are:
__init__.pyunder thesource/lambda/online/__functions/directory._tool_base.pyto clarify its purpose.chat-stack.tsfile in the infrastructure directory, likely to accommodate the changes in the source code.sm_utils.pyfile, which is a dependency for the language model bot.Additionally, there are several files marked with an "error" status, which may indicate issues or conflicts that need to be resolved before merging the changes.
Type of change
Motivation and Context
The motivation behind this refactoring is likely to improve the overall code organization, maintainability, and potentially separate concerns or responsibilities within the codebase. However, since there are files marked with errors, it's essential to ensure that all dependencies and conflicts are resolved before merging the changes to avoid breaking existing functionality.
Dependencies
This change may require updates to the project's documentation to reflect the new file structure and any changes in the usage or import statements related to the refactored files.
File Stats Summary
File number involved in this PR: 7, unfold to see the details:
The file changes summary is as follows:
lambdaOnlineFunctionsLambda function by commenting out the relevant lines.langchain_communitypackage instead oflangchain, updating type annotations forcontent_typeandacceptsvariables, and modifying theExtraconfiguration in theConfigclass to use the.valueattribute.🤖 AI-Generated PR Description (Powered by Amazon Bedrock)
Description
This pull request includes several changes to the codebase, primarily focused on reorganizing and renaming files and directories within the
source/lambda/online/__functions/directory. The changes are as follows:__functions/__init__.pyfile has been added, likely to initialize the package._tool_base.pyfile has been renamed.functions_utils/retriever/utils/directory have been renamed or marked with an error, suggesting potential issues or conflicts.lambda_aws_qa_tools/,lambda_common_tools/, andlambda_retail_tools/directories and their respective files have been marked with errors, indicating potential issues or conflicts.rag.pyfile within thelambda_common_tools/directory has been renamed.These changes appear to be primarily organizational and may involve refactoring or restructuring the codebase. However, some of the errors marked suggest potential conflicts or issues that need to be addressed.
Type of change
File Stats Summary
File number involved in this PR: 7, unfold to see the details:
The file changes summary is as follows:
lambdaOnlineFunctionsLambda function by commenting out the related lines of code.langchaintolangchain_community, adding type annotations forcontent_typeandacceptsvariables, and modifying the value assignment for theextraconfiguration in theConfigclass.🤖 AI-Generated PR Description (Powered by Amazon Bedrock)
Description
This pull request includes various changes to the project's source code. Several files have been modified, added, or renamed across different directories. The changes seem to be related to the chat functionality, AWS tools, common tools, and retail tools.
The modifications include updates to the chat stack infrastructure, changes to the language model bot dependencies, and the addition of a new file for online functions. Several files related to retrievers, AWS QA tools, common tools, and retail tools have been renamed or marked with errors, indicating potential issues or incomplete changes.
Type of change
Motivation and Context
Please provide more context and motivation behind these changes, as well as any relevant dependencies that might be required. Additionally, it would be helpful to clarify the errors or issues associated with the files marked with errors, and whether these changes are intended to address specific bugs or introduce new features.
File Stats Summary
File number involved in this PR: 7, unfold to see the details:
The file changes summary is as follows:
lambdaOnlineFunctionsby commenting out the relevant lines of code that create and configure the function.langchain_communitypackage instead oflangchain, updating type annotations forcontent_typeandacceptsvariables, and modifying the value assignment for theextraconfiguration in theConfigclass.🤖 AI-Generated PR Description (Powered by Amazon Bedrock)
Description
This pull request includes several changes to the codebase. The main modifications are:
Updates to the
.viperlightignorefile andchat-stack.tsfile in the infrastructure and chat components.Addition of a new directory
__functionsundersource/lambda/online/with various Python files added, renamed, or marked as errors. These files appear to be related to different tools and functionalities, such as AWS QA tools, common tools, and retail tools.Modifications to the
sm_utils.pyfile in thellm_bot_depdirectory.Type of change
Motivation and Context
The motivation behind these changes is not entirely clear from the provided information. However, it seems that new functionalities and tools are being added or updated, potentially related to AWS services, common utilities, and retail-specific features.
Dependencies
Based on the file changes, it is likely that this pull request may introduce new dependencies or require updates to existing dependencies. However, without access to the actual code changes and project structure, it is difficult to determine the specific dependencies required.
File Stats Summary
File number involved in this PR: 8, unfold to see the details:
The file changes summary is as follows:
🤖 AI-Generated PR Description (Powered by Amazon Bedrock)
Description
This pull request includes several changes to the source code, primarily focused on refactoring and reorganizing the
langchain_integrationmodule and its submodules. The main changes are:chainssubmodule to improve code organization and clarity.tool_calling_chain_api.pyto handle API-related functionality.The motivation behind these changes is to enhance code maintainability, improve the structure of the project, and potentially introduce new functionality related to API integration.
No external dependencies are introduced with this change.
Type of change
File Stats Summary
File number involved in this PR: 27, unfold to see the details:
The file changes summary is as follows:
LLMChainclass from a relative import to an absolute import within the same package.LLMChainhas been updated to use relative imports from the__llm_chain_basemodule.lambdaOnlineFunctionsLambda function by commenting out the relevant lines of code.langchain_communitypackage instead oflangchain, updating thetransform_outputmethod to explicitly define the types ofcontent_typeandacceptsvariables as strings, and modifying theConfigclass to use the.valueattribute when setting theextrafield.Claude2ToolCallingChainand several subclasses for different language models to create chains that can call external tools using LangChain's tool integration. It includes methods for creating chat histories, binding tools with examples, and setting up system prompts with contextual information.TOOL_CALLING_XML,TOOL_CALLING_API,MTK_RAG, andINTENTION_ALL_KNOWLEDGE_RETRIEVALthreshold.CLAUDE_3_5_HAIKU,CLAUDE_3_5_SONNET_V2,LLAMA3_1_70B_INSTRUCT,LLAMA3_2_90B_INSTRUCT,MISTRAL_LARGE_2407, andCOHERE_COMMAND_R_PLUS.CHATGPT_35_TURBOtoCHATGPT_35_TURBO_0125. || source/lambda/online/common_logic/langchain_integration/chains/conversation_summary_chain.py | 50 added, 13 removed | The code changes involve importing RunnableLambda instead of RunnablePassthrough, updating import paths for Model and LLMChain, renaming classes Iternlm2Chat7BChatChain and Iternlm2Chat20BConversationSummaryChain to Internlm2Chat7BChatChain and Internlm2Chat20BConversationSummaryChain, adding support for enabling/disabling prefill in create_messages_chain, and introducing several new conversation summary chain classes for different models like Claude, Mixtral, Llama, and Cohere. |
| source/lambda/online/common_logic/langchain_integration/chains/marketing_chains/mkt_conversation_summary.py | 4 added, 4 removed | The code changes involve renaming the class
Iternlm2Chat7BChatChaintoInternlm2Chat7BChatChainand updating the corresponding class namesIternlm2Chat7BMKTConversationSummaryChainandIternlm2Chat20BMKTConversationSummaryChain. || source/lambda/online/common_logic/langchain_integration/chains/marketing_chains/mkt_rag_chain.py | 5 added, 5 removed | The code changes involve fixing a typo in the import statement, changing the task type from RAG to MTK_RAG, renaming the class names from "Iternlm2Chat7BKnowledgeQaChain" to "Internlm2Chat7BKnowledgeQaChain" and "Iternlm2Chat20BKnowledgeQaChain" to "Internlm2Chat20BKnowledgeQaChain". |
| source/lambda/online/common_logic/langchain_integration/chains/chat_chain.py | 39 added, 17 removed | The code changes involve updates to the LLMChain and Claude2ChatChain classes, including adding new model types like Claude 3.5 Sonnet, Llama 3.1/3.2 Instruct, Mistral Large, Cohere Command R+, and updating the ChatGPT-3.5 Turbo model ID. It also refactors the chain creation logic for streaming and non-streaming responses. |
| source/lambda/online/common_logic/langchain_integration/chains/init.py | 206 added, 0 removed | This code defines a metaclass
LLMChainMetaand a base classLLMChainfor creating different types of language model chains. It provides a centralized way to register and retrieve specific chain implementations based on model ID and intent type. The code also includes functions to dynamically import various chain modules, such as chat chains, conversation summary chains, intention recognition chains, and others, based on the provided intent type. The mapping between intent types and module import functions is stored in theCHAIN_MODULE_LOAD_FN_MAPdictionary. || source/lambda/online/common_logic/langchain_integration/chains/query_rewrite_chain.py | 15 added, 7 removed | This code change introduces new classes for query rewrite chains using different Anthropic models like Claude-3-Sonnet, Claude-3.5-Sonnet, Claude-3.5-Sonnet-V2, Claude-3.5-Haiku, and Internlm2Chat7B and Internlm2Chat20B models, along with some import and naming corrections. |
| source/lambda/online/common_logic/langchain_integration/chains/translate_chain.py | 3 added, 3 removed | This code change renames the class
Iternlm2Chat7BChatChaintoInternlm2Chat7BChatChainand creates a new classInternlm2Chat20BTranslateChainthat inherits fromInternlm2Chat7BTranslateChain. || source/lambda/online/common_logic/common_utils/prompt_utils.py | 176 added, 39 removed | The code changes involve updating the list of LLM model types to include additional models like CLAUDE_3_5_HAIKU, CLAUDE_3_5_SONNET_V2, LLAMA3_1_70B_INSTRUCT, MISTRAL_LARGE_2407, and COHERE_COMMAND_R_PLUS. It also registers new prompt templates for conversation summary, XML agent prompts, and API agent prompts to support these models. |
🤖 AI-Generated PR Description (Powered by Amazon Bedrock)
Description
This pull request includes several changes related to the integration of the LangChain library and the refactoring of the project structure. The main changes are:
langchain_integrationmodule and its submodules, including renaming and moving files.These changes aim to improve the overall organization and maintainability of the codebase, as well as introduce new features and functionalities through the integration of LangChain.
Type of change
File Stats Summary
File number involved in this PR: 25, unfold to see the details:
The file changes summary is as follows:
langchain_community==0.3.5package.LLMChainmodule from a relative import to an absolute import within the same package.Iternlm2Chat7BChatChaintoInternlm2Chat7BChatChainand updates related class names accordingly.lambdaOnlineFunctionsLambda function, which was previously responsible for various tasks related to online functions and utilities.Added import statements for os and threading modules.
Introduced a StateContext class to manage the current state in a thread-safe manner.
Modified the LambdaInvoker class to use model_validator instead of root_validator, and handle region_name from environment variable.
Updated invoke_with_lambda function to handle response body parsing.
Added StateContext context manager in the wrapper function to set and clear the current state. |
| source/lambda/online/common_logic/common_utils/response_utils.py | 0 added, 1 removed | The code change removes an empty line and modifies the logic to append the chunk to the answer_str variable instead of assigning it. |
| source/lambda/online/common_logic/langchain_integration/chains/chat_chain.py | 41 added, 19 removed | This code change updates the implementation of various chat chains for different language models, including Claude, Mixtral, LLaMa, Mistral, Cohere, Baichuan, InternLM, Qwen, and ChatGPT. It introduces new chat chain classes for models like Claude 3.5 Sonnet V2, Claude 3.5 Haiku, LLaMa 3.1 70B Instruct, LLaMa 3.2 90B Instruct, Mistral Large 2407, and Cohere Command R Plus. It also refactors the
create_chainmethod to useStrOutputParserand streamline the streaming and non-streaming behavior. || source/lambda/online/common_logic/langchain_integration/chains/init.py | 206 added, 0 removed | This code defines a metaclass
LLMChainMetaand a base classLLMChainfor managing different types of language model chains. It provides a centralized way to register and retrieve specific chain implementations based on the model ID and intent type. The code also includes functions to dynamically import various chain modules for different tasks, such as chat, conversation summary, intent recognition, retrieval-augmented generation (RAG), translation, marketing-related chains, step-back prompting, query rewriting, tool calling, and auto-evaluation chains. || source/lambda/online/common_logic/common_utils/constant.py | 14 added, 5 removed | The code changes include:
Added new LLM task types:
TOOL_CALLING_XML,TOOL_CALLING_API,MTK_RAG.Added new LLM model types:
CLAUDE_3_5_HAIKU,CLAUDE_3_5_SONNET_V2,LLAMA3_1_70B_INSTRUCT,LLAMA3_2_90B_INSTRUCT,MISTRAL_LARGE_2407,COHERE_COMMAND_R_PLUS.Added new model type:
CHATGPT_35_TURBO_0125.Added new threshold:
INTENTION_ALL_KNOWLEDGE_RETRIEVAL. || source/lambda/online/common_logic/langchain_integration/chains/rag_chain.py | 36 added, 9 removed | The code changes include:
Importing
StrOutputParserfromlangchain_core.output_parsers.Updating import paths for
ModelandLLMChain.Adding a
StrOutputParserto the chain and handling streaming/non-streaming cases.Introducing new LLMChain classes for various models like Claude 3.5 (Sonnet and Haiku), LLaMa 3.1/3.2, Mistral Large, Cohere Command R+, and Mixtral 8x7b. |
| source/lambda/online/common_logic/common_utils/prompt_utils.py | 176 added, 39 removed | The code changes: