Skip to content

Commit

Permalink
Update docs on severity (#1686)
Browse files Browse the repository at this point in the history
* Update additional-settings.rst
  • Loading branch information
aantn authored Jan 13, 2025
1 parent 382e3d5 commit 60c307f
Showing 1 changed file with 45 additions and 9 deletions.
54 changes: 45 additions & 9 deletions docs/setup-robusta/additional-settings.rst
Original file line number Diff line number Diff line change
Expand Up @@ -52,15 +52,15 @@ The following labels determine which Kubernetes resource relates to an alert:
* - HorizontalPodAutoscaler
- horizontalpodautoscaler, namespace
* - Node
- node
- node or instance (used as a fallback if node doesn't exist)

If your alerts have different labels, you can change the mapping with the ``alertRelabel`` helm value.

A relabeling has 3 attributes:

* ``source``: Use the value from this label
* ``target``: This label will contain the value from ``source``
* ``operation``: Operation can be ``add`` (default) or ``replace``.
* ``source``: The label's name on your alerts (which differs from the expected value in the above table)
* ``target``: The standard label name that Robusta expects (a value from the table above)
* ``operation``: Either ``add`` (default) or ``replace``. If ``add``, your custom mapping will be recognized *in addition* to Robusta's default mapping.

For example:

Expand All @@ -79,18 +79,54 @@ For example:
Mapping Custom Alert Severity
------------------------------------

To correctly map your custom alert severity, you need to add ``custom_severity_map``. The values for each alert should be: high, medium, low, info, and debug.
To help you prioritize alerts from different sources, Robusta maps alert severity to five standard levels:

For example:
* HIGH - requires your immediate attention - may indicate a service outage
* MEDIUM - likely not a current outage, but could be a warning sign beforehand - should be investigated within a reasonable timeframe (hours to days)
* LOW - minor problems and areas for improvement (e.g. performance) - to be reviewed periodically on a weekly or bi-weekly cadence
* INFO - you probably want to be aware of these, but do not necessarily need to take action
* DEBUG - debug only - can be ignored unless you're actively debugging an issue

You are free to interpret these levels differently, but the above is a good starting point for most companies.

Prometheus alerts are normalized to the above levels as follows:

.. list-table::
:header-rows: 1

* - Prometheus Severity
- Robusta Severity
* - critical
- HIGH
* - high
- HIGH
* - medium
- MEDIUM
* - error
- MEDIUM
* - warning
- LOW
* - low
- LOW
* - info
- INFO
* - debug
- DEBUG

Prometheus alerts with a severity **not in the above list** are mapped to Robusta's INFO level.

You can map your own Prometheus severities, using the ``custom_severity_map`` Helm value. For example:

.. code-block:: yaml
globalConfig:
custom_severity_map:
devs_high: high
ops_high: high
devs_low: low
# maps a p1 value on your own alerts to Robusta's HIGH value
p1: high
# maps a p2 value on your own alerts to Robusta's HIGH value
p2: medium
The mapped values must be one of: high, medium, low, info, and debug.

Two-way Interactivity
------------------------
Expand Down

0 comments on commit 60c307f

Please sign in to comment.