Refactor KV Storage and Setup Models data to be stored #17
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sroussey
changed the title
- Introduced `validateTableAndSchema` function in `common_sql_helpers.ts` to enforce naming conventions and prevent schema conflicts. - Integrated validation in `SqliteKVRepository` and `PostgresKVRepository` constructors to ensure proper table and schema setup during initialization.
…t handling - Added a `search` method to `KVRepository` and its implementations, allowing for partial key lookups. - Updated constructors to accept a `searchable` parameter, enabling dynamic configuration of searchable fields. - Enhanced documentation across various repository classes to clarify usage and parameters. - Refactored event emission for put, get, delete, and clearall operations to improve consistency and usability. - Removed the `common_sql_helpers.ts` file and integrated its validation logic into a new `BaseSqlKVRepository` class for better code organization.
✅ Deploy Preview for loquacious-valkyrie-d5baf1 ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
WIP -- using a mock registery that is sync only, so not using the storage system even though files are here now, as i need to change the sync part of the task system to either be async and named something else, or pre-download model info. - Changed model identifiers from "Xenova/LaMini-Flan-T5-783M" to "ONNX Xenova/LaMini-Flan-T5-783M q8" across multiple files to standardize model naming. - Introduced a new `ModelRegistry` to manage model instances and their associations with tasks, enhancing the model management system. - Updated task registration functions to utilize the new model registry, ensuring tasks are correctly linked to their respective models. - Refactored various components to improve integration with the new model registry, including adjustments in task execution and model retrieval logic. - Enhanced documentation and type definitions for better clarity and usability.
The reactiveness is key, not the fact that there is no async code, and we will need async code in reactive stuff soon.
There was a problem hiding this comment.
Copilot reviewed 82 out of 97 changed files in this pull request and generated 1 comment.
Files not reviewed (15)
- examples/web/package.json: Language not supported
- examples/web/tsconfig.json: Language not supported
- package.json: Language not supported
- packages/ai-provider/src/hf-transformers/model/ONNXModelSamples.ts: Evaluated as low risk
- packages/ai-provider/src/hf-transformers/bindings/all_sqlite.ts: Evaluated as low risk
- packages/ai-provider/src/hf-transformers/bindings/all_inmemory.ts: Evaluated as low risk
- examples/web/src/main.tsx: Evaluated as low risk
- packages/ai-provider/src/ggml/model/GgmlLocalModel.ts: Evaluated as low risk
- examples/web/src/RunGraphFlow.tsx: Evaluated as low risk
- docs/developers/03_extending.md: Evaluated as low risk
- examples/web/src/QueueSatus.tsx: Evaluated as low risk
- packages/ai-provider/src/hf-transformers/bindings/registerTasks.ts: Evaluated as low risk
- packages/ai-provider/src/hf-transformers/model/ONNXTransformerJsModel.ts: Evaluated as low risk
- docs/developers/01_getting_started.md: Evaluated as low risk
- packages/ai-provider/src/hf-transformers/browser.ts: Evaluated as low risk
Comments suppressed due to low confidence (2)
packages/ai-provider/src/hf-transformers/provider/HuggingFaceLocal_TaskRun.ts:5
- The variable name 'progess' is misspelled. It should be 'progress'.
const progess = status.status === "progress" ? Math.round(status.progress) : 0;
packages/ai-provider/src/hf-transformers/provider/HuggingFaceLocal_TaskRun.ts:173
- The fallback to 'true' for 'model.normalize' might change the behavior if 'model.normalize' is 'undefined'. Confirm if this is intended.
const vector = new ElVector(hfVector.data, model.normalize ?? true);
|
|
||
| program.version("1.0.0").description("A CLI to run Ellmers."); | ||
|
|
||
| AddBaseCommands(program); | ||
|
|
||
| await registerHuggingfaceLocalModels(); |
There was a problem hiding this comment.
The 'await' keyword is used outside of an async function, which will cause a syntax error. Wrap the code in an async function or use '.then()' for the promises.
Suggested change
| await registerHuggingfaceLocalModels(); | |
| (async () => { await registerHuggingfaceLocalModels(); await registerMediaPipeTfJsLocalModels(); })(); |
Copilot is powered by AI, so mistakes are possible. Review output carefully before use.
|
Failed tests are a bug in current version of bun |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Very much work in process. Stuff is broken. But time to have the list of models not (always) hard coded.