Some structural issues to having this happen consistently. You can only report failures when you Rescue an error from a block and all our playbooks don't use this pattern consistently. See if we can group roles into a task block, where we can use built-in rescue handling at the playbook level.

Tower has the concept of alerts for jobs - we might be able to use that instead of changing the playbooks. See https://docs.ansible.com/ansible-tower/latest/html/userguide/notifications.html. I'm happy to pair with someone to try and get this done.

@acozine and I looked into adding notification in tower when a job fails. The email notification that @acozine had already tested is working. The slack notification needs some more investigation. For now I’m adding our team’s emails in the DACS email notifications.

notifications are sent through email
DACS email notification template

@acozine setup successfully the integration in slack so we can receive alerts in slack.
I've edited the scheduled jobs to use Notify #dacs-monitors template