-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fixes for concurrent document indexing and querying #684
Closed
danielaskdd
wants to merge
38
commits into
HKUDS:main
from
danielaskdd:fix-extract-entity-concurrent-problem
Closed
Fixes for concurrent document indexing and querying #684
danielaskdd
wants to merge
38
commits into
HKUDS:main
from
danielaskdd:fix-extract-entity-concurrent-problem
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Abandon the approach of temporarily replacing the global llm_model_func configuration - Introduce custom_llm function with new_config for handle_cache while extracting entities - Update handle_cache to accept custom_llm
- Separate insert/query embedding funcs - Add query-specific async limit - Update storage classes to use new funcs - Protect vector DB save with lock - Improve config handling for thresholds
@ParisNeo This update involves some core functionalities in LightRAG, and I hope you could help with a preliminary review of the PR. |
… very long context is provided.
…lready control by limit_async_func_call)
- Convert list to numpy array if needed - Maintain existing functionality
This reverts commit 21481db.
- Introduce asyncio.Lock for save operations - Ensure thread-safe file writes
…ormance. - Replace custom counter with asyncio.Semaphore - The existing implementation cannot follow the FIFO order
- Removed custom LLM function in entity extraction - Simplified cache handling logic - Added `force_llm_cache` parameter - Updated cache handling conditions
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problems Addressed
enable_llm_cache_for_entity_extract
is True andenable_llm_cache
is False, there is race condition when doing document indexing and user query at the same time. Funcextract_entities
temporary modify global configenable_llm_cache
to True, causing user query also using cache temporary.limit_async_func_call
decorator implementation using sleep to wait for another thread to release resource is so bad, that user query will not have a chance to run until document index is over.Solutions
force_llm_cache
parm tohandle_cache
, forcing the function handle cache no mater what global config settings are.extract_entities
to use this new parm to control the behavior of handle_cache.Above changes improve concurrent document indexing and querying reliability while maintaining system performance and improve thread safety.
Bug fixes
quantize_embedding
for not support embedding vector of list typecache_type
missing in llm_response_cache, and add new cache_typeextract
for entities extraction llm respond cache.is_embedding_cache_enabled
is true.llm_model_func
retrieval error inhandle_cache
, this function is for LLM similarity verification.