Merge pull request #744 from Digital-Engineering/development

1.6.5

.docker-env

@@ -20,6 +20,12 @@ ENCRYPTION_KEY_PATH=
 # plaintext secret used to generate secure session markers
 SESSION_SECRET=CHANGEME
 
+# controls which file storage method to use, possible values are azure_blob, minio, and filesystem
+# note that minio can also be used to communicate with AWS S3 services
+FILE_STORAGE_METHOD=filesystem
+# must end with a directory character (e.g. "/")
+FILESYSTEM_STORAGE_DIRECTORY=./../storage/
+
 # controls whether or not DeepLynx should emit data events
 EMIT_EVENTS=true
 

.gitignore

@@ -72,4 +72,11 @@ devenv.local.nix
 
 /NodeLibraries/deeplynx/Cargo.lock
 /NodeLibraries/deeplynx/target
-/NodeLibraries/deeplynx/*.node
+/NodeLibraries/deeplynx/*.node
+
+# directory for filesystem storage
+storage/*
+!storage/.gitkeep
+
+# helm
+*.tgz

chart/.helmignore

@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/

chart/CONTRIBUTING.md

@@ -0,0 +1,78 @@
+## Contributing
+
+The DeepLynx Helm package relies on several key resources:
+- `Chart.yaml`: The Helm chart which importantly specifies the name, helm chart version, app version (must
+  match a docker image release version), and dependencies.
+- `values.yaml`: The customizable config supplied to the Helm install that determines which dependencies
+  are deployed, critical DeepLynx environment variables, and can also overwrite values in subcharts (the
+  charts of dependencies) and determine other settings related to the deployment.
+- `deployment.yaml`: A yaml template file that passes in various values from both the Chart and values files.
+  Of note is the environment variables that are passed to the DeepLynx image.
+  - Note that boolean environment variables passed to a templated yaml file like this one should be "quoted" 
+  via the [quote](https://helm.sh/docs/chart_template_guide/functions_and_pipelines/) function.
+
+### Local Development
+
+Commands for refreshing the Helm chart (described below and viewable within `helm-refresh.sh`) have been consolidated 
+into a single executable bash script, `helm-refresh.sh`. Therefore, it is possible to iterate in local development by 
+simply executing `./helm-refresh.sh` to apply new changes to a fresh set of pods and PVCs. The following describes the 
+general set of commands used to install and uninstall a release.  
+
+After making changes, it's best practice to lint the chart. This can be done by running `helm lint` within the
+`chart` directory.
+
+Once you are ready to test your changes, you can package the chart by running `helm package .` **within the
+`chart` directory**.
+
+This will create a `.tgz` file that contains the Helm chart and can be installed. Run the command `helm install
+dl1 ./chart/deeplynx-0.1.0.tgz --namespace deeplynx --create-namespace` **from the root project directory** to install 
+the chart (where in this example `dl1` is the name of the release, `deeplynx` is the name of the namespace to create, 
+and `deeplynx-0.1.0.tgz` is the name of the package to install).
+
+`helm uninstall` allows you to remove the pods (leaving PVC in place, see below commands). Iterate with a
+new package and install.
+
+### Other useful commands
+- `kubectl get pods`: See status of pods
+- `kubectl describe pod [podName]`: See details around a pod where `[podName]` is the name of the pod
+- `kubectl logs [podName]`: See the logs for the specified pod
+- `kubectl get svc`: See information on services within the cluster
+- `kubectl get ingress`: See ingress rules for the cluster
+- `kubectl exec [podName] -- [command]`: Execute a command on a pod where `[command]` is the command to
+  execute (e.g. `ls`)
+- `kubectl get pvc`: See persistent volume claims (PVC, persistent storage used by a pod upon a new install
+  or upgrade if available)
+- `kubectl delete pvc [pvcName]`: Delete a PVC where `[pvcName]` is the name of the PVC to delete. Useful
+  when values that may be stored in an older PVC need to be overwritten
+
+### Kubernetes Tips
+
+If working locally with [K3d](https://k3d.io/), a custom certificate may need to be
+[provided](https://k3d.io/v5.3.0/faq/faq/#pods-fail-to-start-x509-certificate-signed-by-unknown-authority).
+You can forward the included certificate to your local custom cert with the command
+`k3d cluster create local --agents 3 --volume /Users/Shared/myCert.crt:/etc/ssl/certs/myCert.crt -p "8081:80@loadbalancer"`
+where `/Users/Shared/myCert.crt` is the local path to your cert.
+
+As can be seen in the above command, a load balancer and ingress controller is desirable for local development. The load 
+balancer port(s) must be defined at the cluster level. In the above command, we are mapping the localhost's port 8081 
+to port 80 of the cluster's load balancer. 
+
+**To use DeepLynx and the other cluster resources in a local environment with a load balancer enabled**, follow these 
+steps. Note that the ingress-nginx controller needs to be created before the DeepLynx Helm release is created in order 
+to acquire external IPs. Additionally, for local development and use the `ingress-nginx.enabled` value in `values.yaml` 
+should be set to `false` as attempts to stand up the ingress-nginx controller within the same release and namespace as 
+the rest of the DeepLynx release have been unsuccessful thus far.  
+1. Create the cluster (a certificate example is shown here) and expose port 8081 to the load balancer: 
+`k3d cluster create local --agents 3 --volume /Users/Shared/CAINLROOT_B64.crt:/etc/ssl/certs/CAINLROOT_B64.crt -p "8081:80@loadbalancer"`
+2. Stand up an ingress-nginx ingress controller in its own namespace: 
+`helm upgrade --install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace`
+3. Ensure a pod has been created for the controller: `kubectl get pods --namespace ingress-nginx`
+4. Ensure the service for the controller has been created and that there are one or more IPs exposed via `EXTERNAL-IP`:
+`kubectl get service ingress-nginx-controller --namespace=ingress-nginx` 
+5. Create a release of DeepLynx (can be done by using the `helm-refresh.sh` script via `./helm-refresh.sh`)
+6. Once all pods are running, you can access DeepLynx at `localhost:8081`
+
+
+### Pushing to Artifact Hub
+
+TBD

chart/Chart.lock

@@ -0,0 +1,27 @@
+dependencies:
+- name: postgresql
+  repository: https://charts.bitnami.com/bitnami
+  version: 15.5.16
+- name: redis
+  repository: https://charts.bitnami.com/bitnami
+  version: 19.6.2
+- name: ingress-nginx
+  repository: https://kubernetes.github.io/ingress-nginx
+  version: 4.11.1
+- name: prometheus
+  repository: https://prometheus-community.github.io/helm-charts
+  version: 25.24.1
+- name: kubernetes-dashboard
+  repository: https://kubernetes.github.io/dashboard
+  version: 7.5.0
+- name: airflow
+  repository: https://airflow.apache.org/
+  version: 1.14.0
+- name: datahub
+  repository: https://helm.datahubproject.io
+  version: 0.4.19
+- name: mlflow
+  repository: https://charts.bitnami.com/bitnami
+  version: 1.4.16
+digest: sha256:6ce3b5cc3d2fdd19e247e70adf8b7938a34f095cda2aaa3b99f395ff24298f0c
+generated: "2024-08-02T11:23:31.972755-06:00"

chart/Chart.yaml

@@ -0,0 +1,73 @@
+apiVersion: v2
+name: deeplynx
+description: A Helm chart for DeepLynx
+
+type: application
+
+# This is the chart version. This version number should be incremented each time you make changes
+# to the chart and its templates, including the app version.
+# Versions are expected to follow Semantic Versioning (https://semver.org/)
+version: 0.1.0
+
+# This is the version number of the application being deployed. This version number should be
+# incremented each time you make changes to the application. Versions are not expected to
+# follow Semantic Versioning. They should reflect the version the application is using.
+# It is recommended to use it with quotes.
+appVersion: "1.6.5"
+home: https://github.com/idaholab/Deep-Lynx
+icon: https://github.com/idaholab/Deep-Lynx/blob/master/lynx_blue.png
+keywords:
+  - deeplynx
+  - "data warehouse"
+  - "data lake"
+dependencies:
+  - name: postgresql
+    version: 15.5.16
+    repository: https://charts.bitnami.com/bitnami
+    condition: postgresql.enabled
+    tags:
+      - core
+  - name: redis
+    version: 19.6.2
+    repository: https://charts.bitnami.com/bitnami
+    condition: redis.enabled
+    tags:
+      - core
+  - name: ingress-nginx
+    version: 4.11.1
+    repository: https://kubernetes.github.io/ingress-nginx
+    condition: ingress-nginx.enabled
+    tags:
+      - core
+  - name: prometheus
+    version: 25.24.1
+    repository: https://prometheus-community.github.io/helm-charts
+    condition: prometheus.enabled
+  - name: kubernetes-dashboard
+    version: 7.5.0
+    repository: https://kubernetes.github.io/dashboard
+    condition: k8s-dashboard.enabled
+  - name: airflow
+    version: 1.14.0
+    repository: https://airflow.apache.org/
+    condition: airflow.enabled
+    tags:
+      - digitalTwin
+  - name: datahub
+    version: 0.4.19
+    repository: https://helm.datahubproject.io
+    condition: datahub.enabled
+    tags:
+      - digitalTwin
+  - name: mlflow
+    version: 1.4.16
+    repository: https://charts.bitnami.com/bitnami
+    condition: mlflow.enabled
+    tags:
+      - digitalTwin
+
+maintainers:
+  - name: "Jeren Browning"
+    email: [email protected]
+  - name: "John Darrington"
+    email: [email protected]

chart/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Idaho National Laboratory
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

chart/README.md

@@ -0,0 +1,104 @@
+# DeepLynx Kubernetes Helm Charts
+
+## Introduction
+This chart uses [Helm](https://helm.sh/) to bootstrap a [DeepLynx](https://github.com/idaholab/Deep-Lynx) 
+deployment to a [Kubernetes](http://kubernetes.io) cluster along with its dependencies and specified 
+companion apps (Airflow, DataHub, etc.)
+
+## Setup
+1. Set up a kubernetes cluster
+    - In a cloud platform like [Amazon EKS](https://aws.amazon.com/eks),
+      [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine),
+      and [Azure Kubernetes Service](https://azure.microsoft.com/en-us/services/kubernetes-service/) **OR**
+    - In local environment using a local kubernetes like [K3d](https://k3d.io/) 
+      or [Minikube](https://minikube.sigs.k8s.io/docs/).
+    - *Note:* certain companion apps may have hardware requirements
+      (for example, DataHub requires more than 7GB of RAM)
+2. Install the following tools:
+    - [kubectl](https://kubernetes.io/docs/tasks/tools/) to manage kubernetes resources
+    - [helm](https://helm.sh/docs/intro/install/) to deploy the resources based on helm charts
+
+## Configuration
+Additional apps that may be deployed to kubernetes by Helm are listed under `dependencies` 
+in `Chart.yaml`. Each listed dependency may be conditionally stood up as part of a Helm install 
+by setting the appropriate items in `values.yaml`. These items include the `enabled` key
+(a boolean included for each dependency) and optional tags that indicate a type of deployment
+(e.g. `core` for regularly used dependencies, `digitalTwin` for dependencies needed to 
+support digital twin deployments, etc.)
+
+In general, conditions (the `enabled` key) take precedence over tags. See the 
+[documentation](https://helm.sh/docs/topics/charts/#tags-and-condition-fields-in-dependencies)
+for more details.
+
+Besides determining which dependencies will be created, `values.yaml` also contains all other settings 
+that a user may need to configure for a successful deployment. DeepLynx environment variables can be 
+found under `deeplynx.env`. PostgreSQL auth, storage volume size, and other relevant settings can be 
+found under `postgresql`. The same is true for Redis (`redis`). Please refer to the provided comments as well as the 
+[documentation](https://artifacthub.io/) provided by each dependency's helm chart for further details. 
+Additional settings regarding the management and policies of the DeepLynx pod that will be deployed to 
+kubernetes may also be set here.  
+
+Note that for the PostgreSQL and Redis connection strings, these values are stored as secrets in the 
+`secret.yaml` template file. The credentials supplied in these connections strings must match the 
+credentials stored under the `postgresql.auth` and `redis.auth` settings in `values.yaml`.
+
+## Quickstart
+
+Add the INL Helm repo by running the following:
+
+```(shell)
+helm repo add TBD https://helm.TBD.io/
+```
+
+Configure the dependencies to enable and other settings by either updating a copy of the 
+[values.yaml](https://github.com/idaholab/Deep-Lynx) file 
+and supplying it to the `helm install` command with the `-f` option **OR** providing them directly with the 
+`--set` flag. See the [documentation](https://helm.sh/docs/chart_template_guide/values_files/) for more details.  
+
+For an installation with a custom `values.yaml`:
+
+```(shell)
+helm install dl1 -f myvals.yaml --namespace deeplynx TBD/deeplynx
+```
+
+Note that the above command is in the form of: **helm install \[_release name_\] -f 
+\[_path and name of custom values.yaml file_\] --namespace \[_namespace name_\] \[_repository name_\]/deeplynx**
+
+A successful install will show some values defining the deployment and release of the DeepLynx pod to 
+kubernetes as well as some commands that can be run to perform port forwarding to allow for access to 
+DeepLynx outside the kubernetes cluster (a simpler port forwarding method is provided below).
+
+Run `kubectl get pods` to check whether all the pods for the dependencies are running.
+You should see a DeepLynx pod in addition to one or more pods for the dependencies specified to be installed.
+
+You can run the following to expose the frontend locally. Note, you can find the pod name using the command above.  
+
+```(shell)
+kubectl port-forward <DeepLynx pod name> 8090:8090
+```
+
+You should be able to access DeepLynx via http://localhost:8090.
+
+### Running for a Local Environment
+
+Please see the steps in **Kubernetes Tips** of the [Contributing Guidelines](CONTRIBUTING.md) for instructions on how 
+to release this Helm chart in a local environment using ingress-nginx as a load balancer and reverse proxy.
+
+### Kubernetes Dashboard
+
+The [kubernetes dashboard](https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/) 
+can be enabled and included with this Helm chart release. It provides insights into status, metrics, and 
+configuration of kubernetes clusters. After ensuring that the chart is enabled and included in the release, 
+please follow these steps to access it:
+1. Run `kubectl port-forward svc/[releaseName]-kong-proxy 8443:443` where \[releaseName\] is the name of 
+the release (`dl1` by default in `helm-refresh.sh`). Note that you can find the name of the kong-proxy 
+service under services (`kubectl get svc`)
+2. Note that a user for accessing the dashboard is created as part of the Helm chart release via the 
+dashboard-user.yaml and dashboard-clusterrolebinding.yaml template files
+3. Run `kubectl create token dashboard-user` to generate a token (add namespace with the `-n` option 
+if not in the default namespace) 
+4. Navigate to `localhost:8443` and supply the generated token to login 
+
+## Contributing
+
+Please refer to our [Contributing Guidelines](CONTRIBUTING.md).

chart/charts/.gitignore

@@ -0,0 +1,4 @@
+# Ignore everything in this directory
+*
+# Except this file
+!.gitignore

chart/helm-refresh.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+printf "Refreshing helm release...\n\n"
+
+# uninstall current release
+helm uninstall dl1 --namespace deeplynx
+
+printf "\n"
+
+# delete persistnce volume claims
+kubectl delete pvc --all --namespace deeplynx
+
+printf "\n"
+
+# delete any old deeplynx helm packages
+find . -name '*.tgz' -delete -maxdepth 1
+
+# create a new helm package
+helm package .
+
+printf "\n"
+
+# install the new package
+helm install dl1 deeplynx-0.1.0.tgz --namespace deeplynx --create-namespace
+
+printf "\nSuccessful refresh!\n\n"
+
+kubectl get pods -A -o wide
+