Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi Datasource Support #578

Conversation

shreyas-damle
Copy link
Collaborator

Changes in this PR:

  • Multi-datasource support added.
  • Fixed size empty issue on Dashboard => Safe Loader => Data Source
  • Fixed data source name missing on Dashboard => Safe Loader => Documents With Findings

srics and others added 30 commits March 15, 2024 11:29
* Add timezone to local UI dates
* Remove /totalCount from Snippet tab in local UI
* Add ut for reports module

* Fix ruff/pylint errors
* Switch sample app to use Qdrant(in memory) with collection name (daxa-ai#278)

* Added folder id (daxa-ai#279)

Added folder id and requirements

* Update README.md for googledriveloader prereq (daxa-ai#280)

* RAG App Fixes (daxa-ai#281)

* Separate data loading and retrieval (daxa-ai#286)

* Separate data loading and retrieval

* Documented GoogleDriveLoader prerequisites for authorization (daxa-ai#282)

Signed-off-by: Rahul Tripathi <[email protected]>
Co-authored-by: Rahul Tripathi <[email protected]>

* Updated the sample app (daxa-ai#288)

* Updated the sample app.
* Ruff fixes.

* Added .gitignore files for identity rag apps. Deleted extra file. Fixed documentation.

- Added .gitignore files for identity rag apps. 
- Deleted extra file. 
- Fixed documentation.

* Fix formatting & lint errors. Code cleanup.

Fix formatting & lint errors. Code cleanup.

---------

Signed-off-by: Rahul Tripathi <[email protected]>
Co-authored-by: Rahul Triptahi <[email protected]>
Co-authored-by: Shreyas Damle <[email protected]>
Co-authored-by: Rahul Tripathi <[email protected]>
* Added Empty state for Reports
* Added readme and requirements file for pebblo cloud sample.
* Create Dockerfile.base

* Create docker-bake.hcl

* Create docker_release.yml

* Update docker_release.yml
* Added docker installation readme

* Update docker install cmds in README

* Update README and installation guide for docker

---------

Co-authored-by: Sridhar Ramaswamy <[email protected]>
* loader doc response model added

* Ruff fixes

* Fixed test case
* added pebblo config.yaml for docker image
* Update docker_release.yml
* Added changes for file download
* Update changes for whl file download

---------

Co-authored-by: Sridhar Ramaswamy <[email protected]>
…ts (daxa-ai#303)

* Adding pebblo client as well as server verison for local UI and reports
* Added pebblo version on reports and local UI

* Modify xhtml2pdf template for version

* Remove unused svg
* Add bubble chart for findings

* Update empty states for findings panel

* Add snippet count inside chart
…ai#307)

* Sample app for Identity and Semantic Enforcement

* handle error on missing authorized_identities

* Changes after testing.

* Added semantic filtering sample in pebblo_saferetrieval

* Added api_key and load_semantic flags.
Print unique identities and semantic topics.

* PR Comments, Added readme, Code cleanup

* Updated PR links

---------

Co-authored-by: Shreyas Damle <[email protected]>
…axa-ai#308)

* Authorized Entities added

* LocalUI changes for authorized-entities

* Ruff fixes.

* Changes after testing and fixed unit tests.
* Identity UI added for local ui and reports

* Fixed linting issue
* Update bubble chart data

* Change tooltip label
rajburnwal07 and others added 27 commits September 14, 2024 11:56
This reverts commit 15857e8.
This reverts commit 17a2bf7.
This reverts commit 89a57c1.
This reverts commit 717bc4b.
This reverts commit b6b4f7f.
* added changes for documentation 0.19

* made changes in docs for 0.19
* added multi-arch build for pebblo docker.


---------

Co-authored-by: Raj Burnwal <[email protected]>
* New field added azure-client-secret for Azure client secret ID.


* Updated test cases
* Samples: text loader sample using PebbloTextLoader
* Fix lint
…#558)

* Adding Anonymize_snippet as a input filed in /loader/doc API

* Added type checking, and some private functions

* Added UTs
…a-ai#559)

* Adding deprecation message for db storage type as file in config

* Used Enum values in message
* added doc version for 0.1.19

* Remove Azure Client secret from 0.1.19 docs

---------

Co-authored-by: Sridhar Ramaswamy <[email protected]>
* added sonarscan workflow.

* added sonarscan workflow with html output format.

* added sonar properties with html output format.

* renamed sonar-project.properties.

* Reverted renaming sonar-project.properties.

---------

Co-authored-by: Raj Burnwal <[email protected]>
* Adding anonymizeSnippets in reports config

* Updated config processing logic, it is now using pydantic models and validation.

* Updated docs
* Returning default value if no config path is passed

* Adding anonymizeSnippets in reports in default config
* Add owner and size to text metadata

* Add anonymize_snippets flag

* Added readme for textloader sample. Update sample env file.

* Fix formatting

* Set default value for Pebblo classifier url

* Added double quotes for consistency
* added sonarscan workflow.

* added sonarscan workflow with html output format.

* added sonar properties with html output format.

* renamed sonar-project.properties.

* Reverted renaming sonar-project.properties.

* Update sonarscan_run.yml

---------

Co-authored-by: Raj Burnwal <[email protected]>
Co-authored-by: sridhar-daxa <[email protected]>
Co-authored-by: Sridhar Ramaswamy <[email protected]>
* Used 'in' construct and reduced number of queries performed while getting snippet details.

* Implemented pagination.

* Added UTs.
…ces.*.findingsDetails, so that it can be consumed on UI easily.

- topFindings.*.dataSource key added, so that it can be shown on Documents With Findings tab.
@shreyas-damle shreyas-damle changed the base branch from main to feature_multi_data_sources October 4, 2024 08:42
@shreyas-damle shreyas-damle deleted the branch daxa-ai:feature_multi_data_sources October 4, 2024 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.