Gml 1666 ecc as a microservice (#167)

* GML-1666 - Refactored into microservices. Common, Copilot and Eventual Consistency Checker * Moved storage classes and gsql scripts to common as well * A bit more refactoring - of the pyschemas, and made it possible test the services * GML-1666 - Eventual Consistency Checker More cleanup, this time of the common/gsql queries. Improved eventual consistency check. Refactored connection logic in the ECC. Added a status method * Fixed the docker compose files for the separate services * Fixed everything, and with multiprocessing * Replaced processing with threading * Revert the docker compose to include milvus again * Removed logs from being tracked in ECC * Remove the test logs as well * Fixed some imports and a dangling conflict * Merged from dev * dockerized * fix typo * move tests out one at a time to resolve import issues * update reqs path * incremental changes * incremental changes * incremental changes * incremental changes * incremental changes * incremental changes * incremental changes * incremental changes * incremental changes * incremental changes * removed unused stuff * Testing the onprem nightly with a pull request github action * More testing * More testing * Fixed the dockerfile configs again * More testing * More testing * Running from the top level again * Changed context back to top level * Removed config symlinks too. minimized the build script * Readded the configs * Readded the symlinks and removals from the build script * Using build instead of nightly for latest * rm testing run script * fixing llm tests * fixing llm tests * fixing llm tests * fixing llm tests * Updated the other workflows --------- Co-authored-by: Rob Rossmiller <[email protected]> Co-authored-by: RobRossmiller-TG <[email protected]>
tigergraph · Jun 6, 2024 · 58c0cc7 · 58c0cc7
1 parent 674850c
commit 58c0cc7
Show file tree

Hide file tree

Showing 234 changed files with 1,235 additions and 733 deletions.
diff --git a/.github/workflows/build-test-ci.yaml b/.github/workflows/build-test-ci.yaml
@@ -21,11 +21,11 @@ jobs:
     - name: Set up Docker Buildx
       uses: docker/setup-buildx-action@v3
 
-    - name: Build and push Docker image
+    - name: Build Docker.tests image
       uses: docker/build-push-action@v5
       with:
-        context: .
-        file: ./Dockerfile.tests
+        context: . 
+        file: copilot/Dockerfile.tests
         push: false
         load: true
         tags: nlqs/tests:0.1
@@ -66,13 +66,13 @@ jobs:
       if: ${{ github.event_name }} == 'pull_request'
       run: |
         docker rm -f nlqs-tests || true
-        docker run -it -v $(pwd)/configs/:/code/configs -e GOOGLE_APPLICATION_CREDENTIALS=/code/configs/GCP_CREDS.json -e WANDB_API_KEY=${{ secrets.WANDB_API_KEY }} -e PR_NUMBER=${{ github.event.number }} --name nlqs-tests -d nlqs/tests:0.1
+        docker run -it -v $(pwd)/configs/:/code/app/configs -e GOOGLE_APPLICATION_CREDENTIALS=/code/app/configs/GCP_CREDS.json -e WANDB_API_KEY=${{ secrets.WANDB_API_KEY }} -e PR_NUMBER=${{ github.event.number }} --name nlqs-tests -d nlqs/tests:0.1
 
     - name: Run Docker Container for Regress
       if: ${{ github.event_name == 'schedule' }}
       run: |
         docker rm -f nlqs-tests || true
-        docker run -it -v $(pwd)/configs/:/code/configs -e GOOGLE_APPLICATION_CREDENTIALS=/code/configs/GCP_CREDS.json -e WANDB_API_KEY=${{ secrets.WANDB_API_KEY }} -e PR_NUMBER="DailyRegression" --name nlqs-tests -d nlqs/tests:0.1
+        docker run -it -v $(pwd)/configs/:/code/app/configs -e GOOGLE_APPLICATION_CREDENTIALS=/code/app/configs/GCP_CREDS.json -e WANDB_API_KEY=${{ secrets.WANDB_API_KEY }} -e PR_NUMBER="DailyRegression" --name nlqs-tests -d nlqs/tests:0.1
 
     - name: Execute PR Tests
       if: github.event_name == 'pull_request'

diff --git a/.github/workflows/cloud-build-deploy-ci.yaml b/.github/workflows/cloud-build-deploy-ci.yaml
@@ -33,12 +33,22 @@ jobs:
         echo "IMAGE=$IMAGE" >> $GITHUB_ENV
         echo "IMAGE=$IMAGE" >> $GITHUB_OUTPUT${{ needs.setup.outputs.image }} 
  
-    - name: Build and push Docker image
+    - name: Build and push Docker image CoPilot
       uses: docker/build-push-action@v5
       with:
         context: .
-        file: ./Dockerfile
+        file: ./copilot/Dockerfile
         push: true
         tags: |
            ${{ env.IMAGE }}
-           tginternal/copilot:cloud-latest
+           tginternal/copilot:cloud-latest
+ 
+    - name: Build and push Docker image ECC
+      uses: docker/build-push-action@v5
+      with:
+        context: .
+        file: ./eventual-consistency-service/Dockerfile
+        push: true
+        tags: |
+           ${{ env.IMAGE }}
+           tginternal/ecc:cloud-latest
diff --git a/.github/workflows/cloud-build-nightly.yaml b/.github/workflows/cloud-build-nightly.yaml
@@ -33,12 +33,22 @@ jobs:
         echo "IMAGE=$IMAGE" >> $GITHUB_ENV
         echo "IMAGE=$IMAGE" >> $GITHUB_OUTPUT${{ needs.setup.outputs.image }} 
  
-    - name: Build and push Docker image
+    - name: Build and push Docker image CoPilot
       uses: docker/build-push-action@v5
       with:
         context: .
-        file: ./Dockerfile
+        file: ./copilot/Dockerfile
         push: true
         tags: |
            ${{ env.IMAGE }}
-           tginternal/copilot:cloud-dev
+           tginternal/copilot:cloud-dev
+ 
+    - name: Build and push Docker image ECC
+      uses: docker/build-push-action@v5
+      with:
+        context: .
+        file: ./eventual-consistency-service/Dockerfile
+        push: true
+        tags: |
+           ${{ env.IMAGE }}
+           tginternal/ecc:cloud-dev
diff --git a/.github/workflows/onprem-build-nightly.yaml b/.github/workflows/onprem-build-nightly.yaml
@@ -33,12 +33,22 @@ jobs:
         echo "IMAGE=$IMAGE" >> $GITHUB_ENV
         echo "IMAGE=$IMAGE" >> $GITHUB_OUTPUT${{ needs.setup.outputs.image }} 
  
-    - name: Build and push Docker image
+    - name: Build and push Docker image CoPilot
       uses: docker/build-push-action@v5
       with:
         context: .
-        file: ./Dockerfile
+        file: ./copilot/Dockerfile
         push: true
         tags: |
            ${{ env.IMAGE }}
            tigergraphml/copilot:dev
+ 
+    - name: Build and push Docker image ECC
+      uses: docker/build-push-action@v5
+      with:
+        context: .
+        file: ./eventual-consistency-service/Dockerfile
+        push: true
+        tags: |
+           ${{ env.IMAGE }}
+           tigergraphml/ecc:dev
diff --git a/.github/workflows/onprem-build.yaml b/.github/workflows/onprem-build.yaml
@@ -33,12 +33,22 @@ jobs:
         echo "IMAGE=$IMAGE" >> $GITHUB_ENV
         echo "IMAGE=$IMAGE" >> $GITHUB_OUTPUT${{ needs.setup.outputs.image }} 
  
-    - name: Build and push Docker image
+    - name: Build and push Docker image CoPilot
       uses: docker/build-push-action@v5
       with:
         context: .
-        file: ./Dockerfile
+        file: ./copilot/Dockerfile
         push: true
         tags: |
            ${{ env.IMAGE }}
-           tigergraphml/copilot:latest
+           tigergraphml/copilot:latest
+ 
+    - name: Build and push Docker image ECC
+      uses: docker/build-push-action@v5
+      with:
+        context: .
+        file: ./eventual-consistency-service/Dockerfile
+        push: true
+        tags: |
+           ${{ env.IMAGE }}
+           tigergraphml/ecc:latest
diff --git a/.github/workflows/pull-test-merge.yaml b/.github/workflows/pull-test-merge.yaml
@@ -33,12 +33,11 @@ jobs:
           python -m venv venv
           source venv/bin/activate
           python -m pip install --upgrade pip
-          pip install -r requirements.txt
+          pip install -r copilot/requirements.txt
           pip install pytest
 
       - name: Create db config
         run: |
-          source venv/bin/activate
           mkdir configs
           echo "$DB_CONFIG" > configs/db_config.json
           echo "$LLM_CONFIG_OPENAI_GPT4" > configs/llm_config.json
@@ -65,7 +64,9 @@ jobs:
       - name: Run pytest
         run: |
           source venv/bin/activate
-          ./venv/bin/python -m pytest --disable-warnings
+          cp -r copilot/tests/*test* copilot/tests/create_wandb_report.py copilot/app/ 
+          cd copilot/app
+          python -m pytest --disable-warnings
         env:
           DB_CONFIG: ${{ secrets.DB_CONFIG }}
           LLM_CONFIG:  ${{ secrets.LLM_CONFIG_OPENAI_GPT4 }}
@@ -78,4 +79,4 @@ jobs:
           LLM_TEST_EVALUATOR: ${{ secrets.LLM_TEST_EVALUATOR }}
           MILVUS_CONFIG: ${{ secrets.MILVUS_CONFIG }}
           PYTHONPATH: /opt/actions-runner/_work/CoPilot/CoPilot:/opt/actions-runner/_work/CoPilot/CoPilot/tests:/opt/actions-runner/_work/CoPilot/CoPilot/tests/app:/opt/actions-runner/_work/_tool/Python/3.11.8/x64/lib/python3.11/site-packages
-
+
diff --git a/.gitignore b/.gitignore
@@ -18,4 +18,11 @@ log.WARNING
 logs/*
 tmp
 .idea
-volumes
+eventual-consistency-service/app/logs/*
+eventual-consistency-service/logs/*
+eventual-consistency-service/tests/logs/*
+copilot/app/logs/*
+copilot/logs/*
+copilot/tests/logs/*
+volumes
+build/
diff --git a/Dockerfile b/Dockerfile
diff --git a/README.md b/README.md
@@ -350,11 +350,21 @@ TigerGraph CoPilot is designed to be easily extensible. The service can be confi
 ## Testing
 A family of tests are included under the `tests` directory. If you would like to add more tests please refer to the [guide here](./docs/DeveloperGuide.md#adding-a-new-test-suite). A shell script `run_tests.sh` is also included in the folder which is the driver for running the tests. The easiest way to use this script is to execute it in the Docker Container for testing.
 
+### Testing with Pytest
+You can run testing for each service by going to the top level of the service's directory and running `python -m pytest`
+
+e.g. (from the top level)
+```sh
+cd copilot
+python -m pytest
+cd ..
+```
+
 ### Test in Docker Container
 
 First, make sure that all your LLM service provider configuration files are working properly. The configs will be mounted for the container to access. Also make sure that all the dependencies such as database and Milvus are ready. If not, you can run the included docker compose file to create those services.
 ```sh
-docker compose docker-compose.yml up -d --build
+docker compose up -d --build
 ```
 
 If you want to use Weights And Biases for logging the test results, your WandB API key needs to be set in an environment variable on the host machine.

diff --git a/app/supportai/extractors/__init__.py b/app/supportai/extractors/__init__.py