Handling of context overflow in Local LLMs #33295
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…naming for config params
… LLMs. Added tests that verifies outputs of a small LLM.
bjorncs
approved these changes
Feb 11, 2025
| maxPromptTokens int default = 0 | ||
|
|
||
| # Specifies what to do when the number of tokens in a prompt + maxTokens > contextSize / parallelRequests. | ||
| # NONE - default option, run inference no matter context and prompt sizes. |
There was a problem hiding this comment.
We should look into how users can detect when overflow occurs in combination NONE, for instance logging or metric. If do logging we do need a mechanism for silencing the warnings if degraded execution is fine.
| # It can be used for better reproducibility of LLM completions. | ||
| # Still, it doesn't guarantee reproducibility, especially on different hardware. | ||
| # -1 means no random seed will be set. | ||
| randomSeed int default = -1 |
There was a problem hiding this comment.
Consider renaming to seed. I assume the seed is only random if value is set to -1 and non-random for all other?
|
|
||
| @Test | ||
| public void testContextOverflowPolicySkip() { | ||
| downloadFileIfMissing(SMALL_LLM_URL, SMALL_LLM_PATH); |
There was a problem hiding this comment.
Consider using @BeforeClass for shared initialization.
| } | ||
|
|
||
| var numRequestTokens = promptTokens.length + maxTokens; | ||
| // logger.info("Prompt tokens: " + promptTokens.length + ", max tokens: " + maxTokens + ", request tokens: " + numRequestTokens); |
There was a problem hiding this comment.
Consider replacing with logger.log(Level.DEBUG, () -> "Prompt ... instead.
| // Small LLM tests use a quantized Mistral model ca. 4.3 GB. | ||
| // It produces sensible completions which can be verified as part of the test. | ||
| private static final String SMALL_LLM_URL = "https://huggingface.co/lmstudio-community/Mistral-7B-Instruct-v0.3-GGUF/blob/main/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf"; | ||
| private static final String SMALL_LLM_PATH = "src/test/models/llm/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf"; |
There was a problem hiding this comment.
Will this file be ignored by git?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I confirm that this contribution is made under the terms of the license found in the root directory of this repository's source tree and that I have the authority necessary to make this contribution on behalf of its copyright owner.
This is necessary for
generatedocument augmentation to make local LLM outputs more robust.Add constraints and logic to avoid context overflow.