Skip to content
This repository has been archived by the owner on Oct 23, 2024. It is now read-only.

A basic monitoring guide. #35

Merged
merged 2 commits into from
Jan 16, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added docs/images/cassandra-dashboard.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/images/upload.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
98 changes: 85 additions & 13 deletions docs/monitoring.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,93 @@
# Monitoring
# Monitoring KUDO Cassandra

The KUDO Cassandra operator will by default export metrics to Prometheus via a
Prometheus exporter based on the
[criteo/cassandra_exporter](https://github.com/criteo/cassandra_exporter).
This guide explains how to set up monitoring for KUDO Cassandra.

## Description

A running Prometheus service is required for the Prometheus exporter to work.
The KUDO Cassandra operator will export metrics to Prometheus by default. It
achieves this using a Prometheus exporter based on the
[criteo/cassandra_exporter](https://github.com/criteo/cassandra_exporter).

When the KUDO Cassandra operator is deployed with the
`PROMETHEUS_EXPORTER_ENABLED` parameter set to `true` (the default):
When the `PROMETHEUS_EXPORTER_ENABLED` parameter is at its default value of
`true`:

- A `prometheus-exporter` container will run in the same pod as every Cassandra
`node` container and listen for connections at `PROMETHEUS_EXPORTER_PORT`,
which is set to `7200` by default.
`node` container. It will listen for connections on
`PROMETHEUS_EXPORTER_PORT`, which is set to `7200` by default.
- A `prometheus-exporter-port` will be added to the KUDO Cassandra operator
[Service](https://kubernetes.io/docs/concepts/services-networking/service/).
- A
[ServiceMonitor](https://coreos.com/operators/prometheus/docs/latest/user-guides/cluster-monitoring.html)
will periodically fetch the
[Service](https://kubernetes.io/docs/concepts/services-networking/service/)'s
`prometheus-exporter-port` for metrics.
[ServiceMonitor](https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md#related-resources)
will be created to make Prometheus poll that port for metrics.

## Pre-conditions

- KUDO Cassandra instance running
- [Prometheus operator](https://github.com/coreos/prometheus-operator) and
[Grafana](https://grafana.com/) set up in the cluster. The
[kube-prometheus](https://github.com/coreos/kube-prometheus) project provides
both of them.
- KUDO CLI installed (only necessary if you _had_ disabled this feature before).

The examples below assume that the instance and namespace names are stored in
the following shell variables. With this assumptions met, you should be able to
copy-paste the commands easily.

```bash
instance_name=cassandra
namespace_name=default
```

## Steps

### 1. Make sure that Prometheus Exporter is enabled on the KUDO Cassandra instance

This parameter is `true` by default, so you only need to worry about this if you
explicitly disabled it.

If you do not remember, you can check the value of the parameter on a running
instance with a command like:

```bash
kubectl get instance --template '{{.spec.parameters.PROMETHEUS_EXPORTER_ENABLED}}{{"\n"}} $instance_name -n $namespace_name'
```

An output of `<no value>` or `true` means that the exporter is enabled.

_Any other output_ means that the exporter is _disabled_. In that case you need
to enable it with a command such as the following. If you need customization,
see other [parameters](parameters.md) that start with `PROMETHEUS_EXPORTER_`.

```bash
kubectl kudo update -p PROMETHEUS_EXPORTER_ENABLED=true --instance $instance_name -n $namespace_name
```

Expected output:

```
Instance cassandra was updated.
```

### 2. Install the Grafana dashboard

A sample grafana dashboard is provided in
[the monitoring directory](../monitoring/grafana).

How you access the Grafana UI depends on how it was installed. Upon accessing
the `/dashboard/import` URI you will be able to upload or copy-paste the
`cassandra.json` file:

![uploading the dashboard](images/upload.png)

Once done, you will be able to see various Cassandra metrics in the dashboard:

![Cassandra dashboard](images/cassandra-dashboard.png)

### Notes

:warning: Depending on how your prometheus operator was deployed, you may need
to check the `Prometheus` resource. The `serviceMonitorNamespaceSelector` and
`serviceMonitorSelector` attributes on that resource need to be configured to
match the
[labels on the `ServiceMonitor` resource](../operator/templates/service-monitor.yaml#L7)
created by the KUDO Cassandra operator.
2 changes: 1 addition & 1 deletion docs/upgrading.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
This guide explains how to upgrade a running KUDO Cassandra instance to a newer
version of KUDO Cassandra.

## Requirements
## Pre-conditions

- KUDO Cassandra instance running
- KUDO CLI installed
Expand Down