ApmServer set up

[Jose-Matsuda]

Follow up to #1858
Continue work done in pr https://github.com/StatCan/aaw-argocd-manifests/pull/354.

I must determine WHY the ApmServer is erroring out with messages likes;

"http: TLS handshake error from some IP: remote error: tls: bad certificate","service.name":"apm-server","ecs.version":"1.6.0"
http: TLS handshake error from some IP: tls: first record does not look like a TLS handshake","service.name":"apm-server","ecs.version":"1.6.0"}

I think I need to verify the contents of the /usr/share/apm-server/apm-server.yml

The thing we want to see is with our elastic demo deployed in the monitoring system via helm install -n monitoring-system -f values.yaml my-otel-demo open-telemetry/opentelemetry-demo and then eventually when we see data in APM we can remove with helm uninstall -n monitoring-system my-otel-demo

Try to compare and contrast what we have in apm.yaml vs kind-apm.yaml where the apm-server.yml config is set more for the apm.yaml manual deployment.

List of resources in monitoring-system that this applies to
apmserver named apm-server-op
pods with my-otel-demo used to generate logs and data in APM
pod of otelcollector-collector for the collector

[Jose-Matsuda]

Just bringing and using the raw yaml configmap that i used for the standalone apm deployment (not using operator) I was able to see the no longer blocking but before that I get a

The odd part here that I didn't consider either is that my standalone deployment was not on the service mesh, which I think it should have been.
The one deployed by the operator is on the mesh which is correct, but also the one container doesnt come up correctly (probably due to that error above).

Actually now it doesnt stay in one spot after changing to use the correct config i guess via the base yaml config instead of trying to do it (copy the config file like I did for the manual deployment) via the container spec.

[Jose-Matsuda]

Status 05/12/2023

apm-server logs:

remote error: tls: bad certificate

collector logs:

tls: failed to connect to "kibana-monitoring-kb-http.monitoring-system.svc.cluster.local:5601" first record does not look like a TLS handshake
-->adding tls: insecure option now gives
failed to connect to apm-server-op-apm-http.monitoring-system.svc.cluster.local:8200 Err: connection error: desc = "error reading server preface: EOF