Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a notification script if monit restarts chia #388

Merged
merged 11 commits into from
Oct 21, 2024
1 change: 1 addition & 0 deletions chia-blockchain/defaults/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,7 @@ chia_add_healthcheck_service: true
chia_healthcheck_port: 9950
add_monit_config: true
monit_start_now: true
chia_alerts_receiver_token: "{{ lookup('env', 'CHIA_ALERTS_RECEIVER_TOKEN') }}"
cmmarslender marked this conversation as resolved.
Show resolved Hide resolved
# How many times the port check needs to fail before monit will restart the service
chia_monit_failure_threshold: 3

Expand Down
9 changes: 9 additions & 0 deletions chia-blockchain/tasks/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,15 @@
when: add_monit_config
notify: restart-monit

- name: Add a notify script that send a webhook to Keybase if monit restarts the chia service
become: true
ansible.builtin.template:
src: monit_notify.sh.j2
dest: /etc/monit/monit-notify.sh
mode: "0644"
when: add_monit_config
notify: restart-monit

- name: Start Monit Service
become: true
ansible.builtin.service:
Expand Down
1 change: 1 addition & 0 deletions chia-blockchain/templates/monit_config.j2
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ check host localhost with address 127.0.0.1
{% endif %}
{% endif %}
if 5 restarts within {{ 5 * chia_monit_failure_threshold }} cycles then timeout
if 1 restarts within {{ chia_monit_failure_threshold }} cycles then exec "/etc/monit/monit-notify.sh"
cmmarslender marked this conversation as resolved.
Show resolved Hide resolved
38 changes: 38 additions & 0 deletions chia-blockchain/templates/monit_notify.sh.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#!/bin/bash

# Webhook URL where the alert will be sent
WEBHOOK_URL="https://alert-receiver.chiaops.com/ips-info"
cmmarslender marked this conversation as resolved.
Show resolved Hide resolved
HOSTNAME=$(/bin/hostname)
BEARER_TOKEN="{{ CHIA_ALERTS_RECEIVER_TOKEN | quote }}"

# Get the Tailscale IPv4 address
TAILSCALE_IPV4=$(tailscale ip | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}' | head -n 1)
cmmarslender marked this conversation as resolved.
Show resolved Hide resolved
PAYLOAD=$(
cat <<EOF
{
"alerts": [
{
"status": "warning",
"annotations": {
"title": "Monit Alert",
"description": "Monit has restarted the Chia services on host $HOSTNAME with Tailscale IP $TAILSCALE_IPV4"
}
}
]
}
EOF
)

# Send curl request with JSON payload and authentication
HTTP_RESPONSE=$(curl -v -s -o /dev/null -w "%{http_code}" -X POST -H "Content-Type: application/json" \
-H "Authorization: Bearer $BEARER_TOKEN" \
-d "$PAYLOAD" "$WEBHOOK_URL")

# Check the HTTP response code
if [ "$HTTP_RESPONSE" -eq 200 ]; then
echo "Webhook alert sent successfully."
elif [ "$HTTP_RESPONSE" -eq 401 ]; then
echo "Failed to send webhook alert. Unauthorized (401)."
else
echo "Failed to send webhook alert. HTTP Status: $HTTP_RESPONSE"
fi
Loading