chore: merge from dev

[NingLu]

Fixes #

🤖 AI-Generated PR Description (Powered by Amazon Bedrock)

Description

This pull request includes several changes aimed at refactoring and improving the project's infrastructure and API management. The key changes are:

1. Added new API management files for handling chat history, intention management, model management, and prompt management.
2. Removed the deprecated aos and embedding Lambda functions and their associated files.
3. Modified various infrastructure files to accommodate the new API management functionality.
4. Updated the chat_history_management.py file to align with the changes.

These changes are part of an ongoing effort to enhance the project's architecture, improve maintainability, and introduce new features for better API management and functionality.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

File Stats Summary

File number involved in this PR: 30, unfold to see the details:

The file changes summary is as follows:

Files	Changes	Change Summary
source/lambda/aos/main.py	0 added, 94 removed	This file is removed in this PR
source/lambda/aos/utils/aos_utils.py	0 added, 611 removed	This file is removed in this PR
source/lambda/aos/utils/sm_utils.py	0 added, 94 removed	This file is removed in this PR
source/lambda/embedding/main.py	0 added, 155 removed	This file is removed in this PR
source/lambda/embedding/requirements.txt	0 added, 13 removed	This file is removed in this PR
source/lambda/embedding/utils/auto.py	0 added, 39 removed	This file is removed in this PR
source/lambda/embedding/utils/cleaning.py	0 added, 229 removed	This file is removed in this PR
source/lambda/embedding/utils/constant.py	0 added, 12 removed	This file is removed in this PR
source/lambda/embedding/utils/csvx.py	0 added, 179 removed	This file is removed in this PR
source/lambda/embedding/utils/docx.py	0 added, 97 removed	This file is removed in this PR
source/lambda/embedding/utils/htmlx.py	0 added, 81 removed	This file is removed in this PR
source/lambda/embedding/utils/image.py	0 added, 3 removed	This file is removed in this PR
source/lambda/embedding/utils/json.py	0 added, 77 removed	This file is removed in this PR
source/lambda/embedding/utils/jsonl.py	0 added, 81 removed	This file is removed in this PR
source/infrastructure/lib/shared/types.ts	1 added, 0 removed	This change introduces a new boolean property 'useOpenSourceLLM' to the SystemConfig interface, potentially enabling the use of open-source language models.
README_zh-cn.md	2 added, 0 removed	The code changes enable the chat functionality, specify the AWS region for Bedrock, and enable the use of an open-source language model.
source/infrastructure/lib/knowledge-base/knowledge-base-stack.ts	2 added, 1 removed	The code changes update the version of the nltk Python library to 3.9.1 and add a new environment variable "--BEDROCK_REGION" that gets its value from the configuration object props.config.chat.bedrockRegion.
source/infrastructure/cli/magic-config.ts	12 added, 1 removed	The code changes introduce an option to use open-source large language models (LLMs) like Qwen, ChatGLM, and IntelMLM for the chat feature, and enable the knowledge base by default during the solution creation process.
source/infrastructure/lib/api/prompt-management.ts	82 added, 0 removed	This code adds an API Gateway with Lambda integration for managing prompts, models, and scenes, including CRUD operations with authorization and environment variables.
source/infrastructure/lib/api/model-management.ts	84 added, 0 removed	This code defines a ModelApi construct that sets up an API Gateway with Lambda integration for model management operations like deploy, destroy, and status, with appropriate IAM permissions and environment variables.
README.md	2 added, 0 removed	The code adds two new configuration options: 'bedrockRegion' to specify the AWS region for the chat service, and 'useOpenSourceLLM' to enable the use of an open-source language model.
source/lambda/chat_history/chat_history_management.py	2 added, 2 removed	The code changes add the 'chatbotId' field to the list of fields returned when listing chat sessions and messages, allowing tracking of which chatbot was used in each conversation.
source/infrastructure/lib/shared/lambda-layers.ts	38 added, 33 removed	The code changes involve creating and configuring Lambda layers for different components of an AI-based customer service application, including a shared layer, a model deployment layer, an ETL (Extract, Transform, Load) layer, and an authorizer layer.
source/infrastructure/lib/api/chat-history.ts	68 added, 0 removed	This code defines a ChatHistoryApi construct that sets up API Gateway resources and Lambda function for managing chat sessions and messages, including retrieving sessions, messages, and providing feedback on messages.
source/infrastructure/bin/config.ts	1 added, 0 removed	The code change introduces a new configuration option called 'useOpenSourceLLM' and sets it to true, while also enabling the 'amazonConnect' feature.
source/infrastructure/lib/chat/chat-stack.ts	27 added, 23 removed	This code change introduces the following modifications:

1. Created a new Lambda layer for model deployment.
2. Modified the online Lambda function deployment to include a dependency file from the job Lambda.
3. Added CloudFormation permissions to the online Lambda function role.
4. Removed the unused "lambdaOnlineFunctions" Lambda function. |
   | source/infrastructure/lib/api/intention-management.ts | 245 added, 0 removed | This code change adds an API Gateway and Lambda function for managing intentions (likely for a chatbot or conversational AI system). It defines resources for scanning indexes, getting presigned URLs for file uploads, downloading templates, executing intentions, and retrieving intention execution details by ID. The Lambda function interacts with DynamoDB tables, S3 buckets, and other AWS services. |
   | source/infrastructure/lib/api/api-stack.ts | 63 added, 342 removed | The code changes involve importing additional modules and adding new functionality for managing chat history, prompts, intentions, and models in the API stack. Specifically, it creates new APIs and Lambda functions for handling these operations, with appropriate permissions and configurations. |
   | source/infrastructure/lib/model/model-construct.ts | 89 added, 56 removed | The code changes include setting up an EventBridge rule to trigger a Lambda function when CodePipeline execution state changes, handling open-source LLM configuration, refactoring SageMaker configuration setup, and updating Lambda runtime version. |
   | source/infrastructure/lib/shared/iam-helper.ts | 88 added, 2 removed | The code changes add new PolicyStatement objects for CodePipeline, CloudFormation, SageMaker model management, and service quota management permissions, as well as additional permissions for IAM role management. |