Finalise work done for Seldon Metrics Discovery during Obeservability Workshop

[i-chvets]

Finalise work done for Seldon Metrics Discovery during Obeservability Workshop

Work items are tracked in https://warthogs.atlassian.net/browse/KF-829
Branch: https://github.com/canonical/seldon-core-operator/tree/kf-829-gh68-feat-metrics-discovery
Prometheus deployment https://github.com/canonical/prometheus-k8s-operator

Design

Failure alerts are implemented through integration with Prometheus Charm from Canonical Observability Stack. For metrics provided by models targets can change from model to model and from deployment to deployment. Metrics Endpoint Observer provided by COS is integrated. Updates to targets are handled by Mertics Endpoint Observer and relayed to Prometheus by Seldon Core Operator Charm.

Testing

• Setup MicroK8S cluster and Juju controller:

microk8s enable dns storage metallb:"10.64.140.43-10.64.140.49,192.168.0.105-192.168.0.111"
juju bootstrap microk8s uk8s
juju add-model test

• Deploy Istio, Prometheus and Seldon Core Operator and relate them.

juju deploy prometheus-k8s --trust
juju deploy ./seldon-core_ubuntu-20.04-amd64.charm seldon-controller-manager --trust --resource oci-image="docker.io/seldonio/seldon-core-operator:1.14.0"
juju relate prometheus-k8s seldon-controller-manager

Resulting deployment should like this:

Model  Controller  Cloud/Region        Version  SLA          Timestamp
test   uk8s        microk8s/localhost  2.9.34   unsupported  15:28:26-05:00

App                        Version  Status  Scale  Charm           Channel      Rev  Address         Exposed  Message
prometheus-k8s             2.33.5   active      1  prometheus-k8s  stable        79  10.152.183.239  no       
seldon-controller-manager           active      1  seldon-core                    0  10.152.183.182  no       

Unit                          Workload  Agent  Address     Ports  Message
prometheus-k8s/0*             active    idle   10.1.59.80         
seldon-controller-manager/0*  active    idle   10.1.59.79         

Deploy model with custom metrics:

microk8s.kubectl -n test apply -f examples/echo-metrics-v1.yaml

Get IP address of model classifier and use it for prediction request:

microk8s.kubectl -n test get svc | grep echo-metrics-default-classifier
echo-metrics-default-classifier       ClusterIP      10.152.183.34    <none>         9000/TCP,9500/TCP                       25m

Request prediction using IP address of model classifier:

for i in `seq 1 10`; do sleep 0.1 && \
   curl -v -s -H "Content-Type: application/json"    \
  -d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}'    \
  http://<echo-metrics-default-classifier-IP>:9000/predict > /dev/null ; \
done

Metrics are available at pod's IP address and Prometheus port:

 microk8s.kubectl -n test describe pod  echo-metrics-default-0-classifier-5bf6cf86cd-r7c8l
 IP:           10.1.59.82
      PREDICTIVE_UNIT_METRICS_SERVICE_PORT:  6000
      PREDICTIVE_UNIT_METRICS_ENDPOINT:      /prometheus

curl http://10.1.59.82:6000/prometheus

• Navigate to Prometheus dashboard https://<Prometheus-unit-IP>:9090, select Status->Targets

[i-chvets]

Currently blocked by COS: require some changes to Prometheus.

[i-chvets]

Dec 9, 2022 debug session notes:
I am hitting juju tools issue: juju-run is not installed in my pod, logs from /var/log/discovery.log:

FileNotFoundError: [Errno 2] No such file or directory: '/var/lib/juju/tools/unit-seldon-controller-manager-0/juju-run'

I guess, my observer is not even starting because of this.
And this is on the pod:

# ls -la /var/lib/juju/tools/unit-seldon-controller-manager-0/juju-run
ls: cannot access '/var/lib/juju/tools/unit-seldon-controller-manager-0/juju-run': No such file or directory

It does live on pod though, but in different place:

# ls -la /usr/bin/juju-run 
lrwxrwxrwx 1 root root 25 Dec 13 19:44 /usr/bin/juju-run -> /charm/bin/containeragent

[DnPlas]

@i-chvets I believe this is done, please confirm.

[i-chvets]

Merged #94

[i-chvets]

Testing work is still required. Not completed yet.
By following steps in description, no metrics could be retrieved. Needs more debugging.
Currently, the above setup fills in 32GB of RAM and swap space and the whole setup becomes unresponsive.