Skip to content

Commit

Permalink
Adding sinks configurations
Browse files Browse the repository at this point in the history
Changing playbooks output to sinks
Removed slack mentions
Add prometheus_url override global param
Fix node cpu analysis prometheus query
  • Loading branch information
arikalon1 committed Jul 22, 2021
1 parent cbef6bd commit 74a5283
Show file tree
Hide file tree
Showing 67 changed files with 1,833 additions and 539 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
.idea/
.vscode/
src/build/
playground/
src/dist/
*.egg/
*.egg-info/
Expand Down
44 changes: 44 additions & 0 deletions docs/developer-guide/reference.rst
Original file line number Diff line number Diff line change
Expand Up @@ -95,3 +95,47 @@ For example:
@on_recurring_trigger(seconds_delay=10, repeat=3)
def my_scheduled_playbook(event: RecurringTriggerEvent):
logging.info(f"My scheduled playbook is running for the {event.recurrence} time")
Sinks
-------------
| Playbooks results can be forwarded to one or more sinks. See :ref:`Playbooks sinks` for details.
| For that to happen, we have to create ``Finding`` and ``Enrichments`` during the playbooks processing.
| The Robusta platform will automatically forward it to the configured sinks
Finding
^^^^^^^^^^^^^^^^^
| The ``Finding`` contains the base information of the playbooks result.
| Creating a ``Finding`` is easy:
.. code-block:: python
@on_recurring_trigger(seconds_delay=10, repeat=3)
def my_scheduled_playbook(event: RecurringTriggerEvent):
event.processing_context.create_finding(
title=f"My scheduled playbook is running for the {event.recurrence} time",
severity=FindingSeverity.INFO
)
Enrichments
^^^^^^^^^^^^^^^^^
| Each ``Finding`` can contain any number of ``Enrichments``.
| Each ``Enrichment`` has a list of ``blocks`` describing it:
* **MarkdownBlock:** - A text block
* **DividerBlock:** - Dividing section between ``Enrichment`` parts. (If the sink supports that)
* **HeaderBlock:** - A header block
* **ListBlock:** - A block containing list of items
* **TableBlock:** - A block containing table of items
* **KubernetesFieldsBlock:** - A block containing information describing kubernetes fields
* **DiffsBlock:** - A block containing information describing yaml diff attributes
* **JsonBlock:** - A block containing any json data
* **FileBlock:** - A block containing any file with the file data
* **CallbackBlock:** - A block containing callback information, that can be invoked back from the sink. (a Slack button for example, running some command)

| **Note:** - Not all block types supported by all sinks. If an unsupported block arrives to a sink, it will be ignored
| Adding an ``Enrichment``:
.. code-block:: python
my_log_file_data = "..."
event.processing_context.finding.add_enrichment([FileBlock("log.txt", my_log_file_data)])
2 changes: 1 addition & 1 deletion docs/getting-started/manual-triggers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,4 @@ Manually triggering the python_profiler playbook

.. code-block:: bash
robusta trigger python_profiler pod_name=<POD_NAME> namespace=<NAMESPACE> slack_channel=<SLACK_CHANNEL>
robusta trigger python_profiler pod_name=<POD_NAME> namespace=<NAMESPACE>
10 changes: 4 additions & 6 deletions docs/user-guide/alerts.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,6 @@ Lets look at the simplest possible ``active_playbooks.yaml`` which instructs Rob
active_playbooks:
- name: "alerts_integration"
action_params:
slack_channel: "robusta-alerts"
The above configuration isn't very useful because we haven't enriched any alerts yet.
However, we do get a minor aesthetic benefit because Robusta adds pretty formatting to alerts as you can see below:
Expand All @@ -56,7 +54,6 @@ Now lets add an enricher to ``active_playbooks.yaml`` which enriches the ``HostH
active_playbooks:
- name: "alerts_integration"
action_params:
slack_channel: "robusta-alerts"
alerts_config:
- alert_name: "HostHighCpuLoad"
enrichers:
Expand Down Expand Up @@ -90,7 +87,6 @@ You can change the default enricher(s) for all alerts using the ``default_enrich
active_playbooks:
- name: "alerts_integration"
action_params:
slack_channel: "robusta-alerts"
default_enrichers:
- name: "AlertDefaults"
Expand All @@ -105,7 +101,6 @@ Now lets look at an example ``active_playbooks.yaml`` which silences KubePodCras
active_playbooks:
- name: "alerts_integration"
action_params:
slack_channel: "robusta-alerts"
alerts_config:
- alert_name: "KubePodCrashLooping"
silencers:
Expand All @@ -122,7 +117,6 @@ Here is an example which shows all the features discussed above working together
active_playbooks:
- name: "alerts_integration"
action_params:
slack_channel: "robusta-alerts"
default_enrichers:
- name: "AlertDefaults"
alerts_config:
Expand All @@ -145,10 +139,14 @@ Available enrichers
**AlertDefaults:** send the alert message and labels to Slack

**NodeCPUAnalysis:** provide deep analysis of node cpu usage
This enricher use ``prometheus``. The ``prometheus`` url can be overriden in the ``global_config`` section.
(For example - ``prometheus_url: "http://prometheus-k8s.monitoring.svc.cluster.local:9090"``)

**OOMKillerEnricher:** shows which pods were recently OOM Killed on a node

**GraphEnricher:** display a graph of the Prometheus query which triggered the alert
This enricher use ``prometheus``. The ``prometheus`` url can be overriden in the ``global_config`` section.
(For example - ``prometheus_url: "http://prometheus-k8s.monitoring.svc.cluster.local:9090"``)

**StackOverflowEnricher:** add a button in Slack to search for the alert name on StackOverflow

Expand Down
4 changes: 2 additions & 2 deletions docs/user-guide/builtin-playbooks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ python_profiler
.. code-block:: bash
robusta trigger python_profiler pod_name=your-pod namespace=you-ns process_name=your-process slack_channel=general seconds=5
robusta trigger python_profiler pod_name=your-pod namespace=you-ns process_name=your-process seconds=5
| **Parameters:** see below. All parameters are optional except for ``pod_name`` and ``namespace``. ``pod_name`` can be the prefix of the pod name and doesn't need to be a full match.
Expand Down Expand Up @@ -127,7 +127,7 @@ http_stress_test
.. code-block:: bash
robusta trigger http_stress_test url=http://grafana.default.svc:3000 slack_channel=test1 n=1000
robusta trigger http_stress_test url=http://grafana.default.svc:3000 n=1000
| **Enabling it:** add to active_playbooks.yaml before manually trigger using the Robusta CLI (as described above):
Expand Down
59 changes: 59 additions & 0 deletions docs/user-guide/playbook-configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,63 @@ Here is a sample ``active_playbooks.yaml`` which enables two playbooks:
- name: "python_profiler"
- name: "restart_loop_reporter"
Playbooks sinks
^^^^^^^^^^^^^^^^^^
| With Robusta, playbooks results can be send to one or more sinks.
| Currently 3 sink types are supported:
* *slack:* - Send playbooks results to Slack channel
* *robusta:* - Send playbooks results Robusta's dedicated UI
* *kafka:* - Send playbooks results Robusta's dedicated UI

| To use sinks, first define the available named sinks.
| For example: (in the ``active_playbooks.yaml`` file)
.. code-block:: yaml
sinks_config:
- sink_name: "robusta ui"
sink_type: "robusta"
params:
account_id: "MY ROBUSTA ACCOUNT ID"
store_url: "MY ROBUSTA STORE URL"
api_key: "MY ROBUSTA ACCOUNT API KEY"
- sink_name: "alerts slack"
sink_type: "slack"
params:
slack_channel: "robusta alerts channel"
- sink_name: "my kafka sink"
sink_type: "kafka"
params:
kafka_url: "localhost:9092"
topic: "robusta-playbooks"
| **Note:** Create your Robusta account, to get the parameters for the Robusta sink `here <https://robusta.dev>`_ .
| By default, all playbooks will forward the results to the default sinks.
| The default sinks are defined in the ``global_config`` section.
| For example:
.. code-block:: yaml
global_config:
sinks:
- "robusta ui"
- "alerts slack"
| The default sinks list can be overridden, per playbook:
.. code-block:: yaml
active_playbooks
- name: "add_deployment_lines_to_grafana"
sinks:
- "my kafka sink"
action_params:
grafana_dashboard_uid: "uid_from_url"
grafana_api_key: "grafana_api_key_with_editor_role"
grafana_service_name: "grafana.namespace.svc.cluster.local:3000"
Playbook parameters
^^^^^^^^^^^^^^^^^^^
Expand Down Expand Up @@ -98,6 +155,7 @@ To avoid repeating yourself you can define trigger_params and parameters globall
.. code-block:: yaml
global_config:
cluster_name: "my-staging-cluster"
grafana_api_key: "grafana_api_key_with_editor_role"
grafana_service_name: "grafana.namespace.svc.cluster.local:3000"
Expand All @@ -114,3 +172,4 @@ To avoid repeating yourself you can define trigger_params and parameters globall
trigger_params:
name_prefix: "App2"
| **Note:** The ``cluster_name`` is a required parameter, since it's used for sinks as the cluster identifier. (``cluster_name`` should be unique among different clusters)
17 changes: 15 additions & 2 deletions playbooks/active_playbooks.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,19 @@
sinks_config:
- sink_name: "robusta platform"
sink_type: "robusta"
params:
account_id: "<ROBUSTA_ACCOUNT_ID>"
store_url: "https://xvyhpoxfmtpuqqeyqkso.supabase.co"
api_key: "<ROBUSTA_API_KEY>"
- sink_name: "alerts slack"
sink_type: "slack"
params:
slack_channel: "<DEFAULT_SLACK_CHANNEL>"
global_config:
slack_channel: "<DEFAULT_SLACK_CHANNEL>"

cluster_name: "<CLUSTER_NAME>"
sinks: # These are the default sinks. The named sinks list can be overridden, per playbook
- "alerts slack"
#<ENABLE_ROBUSTA_SINK>
active_playbooks:
- name: "restart_loop_reporter"
action_params:
Expand Down
Loading

0 comments on commit 74a5283

Please sign in to comment.