[SPIKE] [Monitors] Determine what to convert Sentry-based monitors to in light of upcoming Sentry deprecation

[jilladams]

Status

[2024-07-23] [Fran] Turned this into a Spike. Will create an Epic, and then will turn the AC into individual tickets. Moving to Next refinement to refine the spike.

Description

Some monitors for Sitewide products in Datadog rely on Sentry to push up errors, that are then aggregated in Datadog.

The Platform team intends to deprecate use of Sentry, timing TBD. (Asked for date here)

We need to convert any monitors that use Sentry for alerts to use another mechanism.

Monitors that use sentry

Forms

• Monitoring Documentation: https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/find-a-va-form/engineering/monitoring.md

• Monitors:
  • Sentry Forms: Find Forms - Form Detail - onDownloadLink Click Error Monitor: https://vagov.ddog-gov.com/monitors/165098?view=spans
  • Sentry Forms: Find Forms - Form Detail - invalid PDF link Error Monitor: https://vagov.ddog-gov.com/monitors/167982?view=spans
  • Sentry Forms: Find Forms - Form Detail - invalid PDF accessed Error Monitor: https://vagov.ddog-gov.com/monitors/167983?view=spans
  • Sentry find forms errors report: https://sentry.vfs.va.gov/organizations/vsp/discover/results/?id=19&project=4&statsPeriod=7d

Facilities

• Facilities Notification of error on VAMC pages:
  • Monitor: https://vagov.ddog-gov.com/monitors/170096/edit
  • Documentation: https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/facilities/medical-centers/engineering/monitoring.md#facilities-notification-of-error-on-vamc-pages
• Facilities Notification of error on Vet Center pages
  • Monitor: https://vagov.ddog-gov.com/monitors/168205/edit
  • Docs: https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/facilities/vet-centers/engineering/monitoring.md#facilities-notification-of-error-on-vet-center-pages

ACs

• Monitors using Sentry are reviewed to understand what options we have for monitoring in Datadog:
  • Forms
  • VAMC
  • Vet Centers
• Findings are documented in ticket comments and reviewed with PO / PM
• PM (Fran) updates the existing tickets (linked as being blocked by this ticket)

[jilladams]

For forms: As I'm thinking about this ticket: I stopped logging into Sentry ages ago, and the Datadog rollups for forms never go off. That's been the case for ages. I logged in today just to see, and what used to be a pretty busy report has 0 results for the last 90 days: https://sentry.vfs.va.gov/organizations/vsp/discover/results/?environment=production&id=19&project=4&statsPeriod=90d. I'm not even sure Sentry is actively monitoring anymore? Or this report may just be busted? that may or may not matter to the heart of the ticket.

[jilladams]

Whoever works this ticket will need Sentry and Datadog access. If VA won't provide a new seat to new hires, this might require a longer-term team member to work it. If a seat can be gotten, this might be a good ticket for @SnowboardTechie since he's got devops background.

[Agile6MSkinner]

Hey team! Please add your planning poker estimate with Zenhub @dsasser @omahane @SnowboardTechie @chriskim2311

[jilladams]

@SnowboardTechie context: asking for your input on the assumption that you have Datadog background. If not, ignore.
@chriskim2311 pretty sure you helped set up the original Sentry > Datadog monitors for Forms.

If we aren't sure how we could replace the functionality we theoretically were / are getting from Sentry, we can get some help from the Datadog channel and folks like Adrian Rollett / Steve Albers to think it through. Feel free to factor that kind of coordination into points estimates.

[SnowboardTechie]

could replace the functionality we theoretically were / are getting from Sentry, we can get some help from the Datadog channel and folks like Adrian Rollett / Steve Albers to think it through. Feel free to factor that kind of coordination into points estimates.

I don't have Datadog access here yet, but if I'm following correctl, I believe these Sentry monitors for Forms can be implemented using Datadog Synthetic Monitoring. Not super familiar with Sentry but will put out a poker estimate for how long I think it would take to come up with details to confirm.

[chriskim2311]

@SnowboardTechie Context on usage of Sentry. Initially we had these loggers catching form related events and pushing to Sentry. Sentry however did not have an effective way of alerting the team through slack, so we used webhooks and exported the events to Datadog and created monitors there to alert the team when a threshold was hit.
https://github.com/department-of-veterans-affairs/vets-website/blob/main/src/applications/find-forms/widgets/createFindVaFormsPDFDownloadHelper/DownloadPDFGuidance.js.
https://github.com/department-of-veterans-affairs/vets-website/blob/main/src/applications/find-forms/widgets/createFindVaFormsPDFDownloadHelper/index.js

@jilladams I haven't logged into Sentry either and I can't see any events for the monitors we have set up. We can take a look at this from Datadog and see if we can catch the same errors from the FE: https://docs.datadoghq.com/api/latest/events/?code-lang=typescript#post-an-event. But I do think we could check in with Adrian/Steve though to make sure this is the best/cost effective way to do this.

[omahane]

This appears to be FE, so I'm removing myself from planning poker.

[jilladams]

@SnowboardTechie we are contemplating this as your first ticket, if we can get you into Datadog once you have a PIV. Let's discuss if that feels appropriate.

[SnowboardTechie]

@SnowboardTechie we are contemplating this as your first ticket, if we can get you into Datadog once you have a PIV. Let's discuss if that feels appropriate.

@jilladams this seems very appropriate for a first ticket. I'll work on getting Datadog access today now that I have my PIV so I can get my bearings on the existing monitors and start exploring our options for Datadog alternatives 👍🏻

[SnowboardTechie]

@FranECross @Agile6MSkinner having dug through this spike here is what I've found. TLDR; most of these should be fairly straightforward to bring into Datadog directly with the exception of Vet Center monitors due to it's lack of activity. If you have any questions or need more details please don't hesitate to tag me.

Form Monitors:

All Sentry Form monitors are written in such a way that they can be adjusted via code to send events directly to Datadog instead of Sentry, then the monitors will need to be adjusted to look for the error being sent into the logs. I was able to find an example of the alerts being sent from vet website today in this slack thread. Since we will be moving events that are already being sent, this should not result in any additional Datadog costs.

Facilities:

Facilities Notification of error on VAMC pages: This sentry alert is implemented today via a pattern that should be easily adaptable to a Datadog monitor. We should be able to simply adjust the monitor in Datadog to look for the pattern Sentry uses today.

Facilities Notification of error on Vet Center pages: This one has not occurred in the past 3 months and was implemented using very specific searches on the Sentry side. This being the case, I believe we should deprecate this monitor entirely as we will not have examples to point to if trying to implement this more unique one in Datadog.

[FranECross]

@SnowboardTechie Thanks so much for digging into this and the information! I'll update the individual tickets using your info provided. cc @Agile6MSkinner

[FranECross]

Closing as complete. Thanks again!