Support multiple alert manager replicas

[BryanFauble]

In this helm chart it looks like 1 url is ever used/considered:

charts/charts/signoz/templates/_helpers.tpl

Line 298 in bb53857

{{- define "alertmanager.url" -}}

Ask:
When running multiple replicas of alert mamager (https://github.com/SigNoz/alertmanager) allow for any alerts to be sent to 1 or more configured replicas.

Taken from the Readme of the signoz fork of alertmanager it supports the overall ask of this issue, it shows how it expects prometheus to point at multiple alert manager instances:

To point your Prometheus 1.4, or later, instance to multiple Alertmanagers, configure them in your prometheus.yml configuration file, for example:

alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - alertmanager1:9093
      - alertmanager2:9093
      - alertmanager3:9093

Important: Do not load balance traffic between Prometheus and its Alertmanagers, but instead point Prometheus to a list of all Alertmanagers. The Alertmanager implementation expects all alerts to be sent to all Alertmanagers to ensure high availability.

Here are more links to code where changes are likely needed:

1. charts/charts/signoz/templates/query-service/statefulset.yaml

   Line 134 in bb53857

   value: http://{{ include "alertmanager.url" . }}/api/

2. charts/charts/signoz/templates/query-service/configmap.yaml

   Line 18 in bb53857

   - {{ include "alertmanager.url" . }}

[grandwizard28]

Hi @BryanFauble,
Thank you for opening this issue!

Would you be willing to contribute a PR for this?

[BryanFauble]

Thanks @grandwizard28 ,

This is something that gives a trivial benefit so I don't want to spend time on it for now.

Something of a larger benefit that I started to look at was starting to consider how k8s secrets might be used to configure things like the clickhouse password. However, there was some more work to consider how stuff like the traces url could also be fed in as it includes the username/password as well:

1. main...BryanFauble:charts-signoz:main
2. https://github.com/BryanFauble/charts-signoz/blob/7a29318aabc8bccab36a4db3aaf3bf7e71fb27df/charts/signoz/templates/_clickhouse.tpl#L234

The secretFrom stanza was something that the underlying clickhouse implementation is doing: https://altinity.com/blog/clickhouse-confidential-using-kubernetes-secrets-with-the-altinity-operator

[grandwizard28]

Let's discuss this on a separate issue: #525.

I'd like to keep this open for supporting multiple alert managers :)