Escalate all QA Standards to launch-blocking

[shiragoodman]

OCTO OKR

1.3: 100% of transactions received via our digital experiences are either processed correctly or we have notified the user of an error.

Key result: All OCTO led projects log silent failures to Datadog by March 31, 2025

Assignee: @it-harrison
Peer Reviewer: @erinrwhite

Description

Governance launched the Logging Silent Failures QA standard in October 2024. The new standard was not launch-blocking, however OCTO-DE has decided to enforce it as a launch-blocking standard. In addition, because it is not currently a launch-blocking standard, VFS teams have not adhered to it. Per QA specialist @it-harrison on 1/7/2025 , no team, "has attempted to satisfy the requirement in any way."

1/10/25 UPDATE: Changing the Logging Silent Failures QA Standard to launch-blocking would make 6 out of 7 of the QA standards launch-blocking, and only 1 non-launch-blocking issue would remain. In order to make things simpler and easier to follow for both Governance and VFS teams, all QA Standards will be escalated to launch-blocking, meaning the Traceability Reports standard will also be escalated to launch-blocking within the scope of this ticket.

Impacted Artifacts

Live pages

• QA Standards page on Platform Website
• Collaboration Cycle Change History page.
• How to complete a QA review for the Staging review touchpoint
• QA Standards feedback template

Drafted pages

Tasks

• Change the severity of the Traceability Reports and Logging Silent Failures QA standards from warning to launch-blocking on the QA Standards page on Platform Website
• Update the QA Standards feedback template to add a note at the top specifying that all QA Standards are launch-blocking
• Update the How to complete a QA review for the Staging review touchpoint to reflect this change
• Draft a change history post for the Collaboration Cycle Change History page. Follow up with Matt for an enforcement date via this comment.
• Draft a Slack change announcement following the Slack change announcement format
• Once Platform Website pages, internal guidance, and the feedback template are up to date, post announcement on the #vfs-all-teams Slack channel. Notify Shira so she can communicate at Team of Teams.

Peer Review

To be completed by peer reviewer

• User story and acceptance criteria are met

Acceptance Criteria

• We have escalated the severity of the Traceability Reports and Logging Silent Failures QA Standards to launch-blocking
• We have communicate this change to VFS teams

Team Notification

• Slack post on #platform-governance-team
• Presentation at Weekly Initiative Meeting
• No notification needed

How to prepare this issue

Refinement

• Ticket has user story, description, tasks, and acceptance criteria
• Ticket has been sized
• Once all above items are checked, add the 'Ready' label.

Planning

If this ticket is picked up from the Backlog mid-sprint, connect with Shira to ensure the below items are completed correctly

• Ticket has been discussed as a team at Planning
• Ticket has been assigned to appropriate initiative or quarterly epic
• Ticket has been assigned to a Sprint
• Ticket has been moved to the Planned pipeline in the Governance team board
• Ticket as an assignee and peer reviewer

[humancompanion-usds]

To clarify, while we may not have seen the QA artifacts requested, a number of teams have in fact started logging to Datadog:

https://vagov.ddog-gov.com/dashboard/n6c-twn-swr/silent-failure-tracker?fromUser=false&refresh_mode=sliding&from_ts=1728334457110&to_ts=1736286857110&live=true

• Silent Failures Avoided - no confirmation of email notification available - 18 products logged
• Silent Failures - 10 products logged
• Silent Failure Avoided (email verified) - 13 products logged

That's just the products who have logged. 28 products experienced silent failures and all are claiming that they are logging. There are only 4 products left that are potentially not yet logging that should be. So it is happening and we want to be sure it keeps happening.

As part of this initiative, we should announce to VFS teams that they should be using the call back VA Notify has available so that they can update their logging to go from "no confirmation of email notification available" to "(email verified)" as emails can now be verified. Thus a result of the largerer initiative would be that 31 products are logging under "email verified".

[shiragoodman]

@humancompanion-usds we have 2 different QA standards that were recently released:

Endpoint monitoring, which is launch-blocking, was launched in September, and logging silent failures, not launch-blocking, was launched in October. Your comment above I think is in reference to the endpoint monitoring standard, which teams have absolutely adhered to, however this ticket was written to change the logging silent failures standard to launch-blocking.

@it-harrison can correct me if I'm wrong, but I think what we're seeing is teams monitoring endpoints in Datadog, but not linking their lines of code to StatsD for reporting purposes.

[shiragoodman]

per call with Matt on 1/10, we will also escalate the traceability reports to launch-blocking.

[shiragoodman]

@humancompanion-usds do you want to enforce this change ASAP or should we wait to enforce with VFS teams? Possibly a 2 week notice?

[shiragoodman]

Per Matt on 1/16, 2 week notice is ok.

[it-harrison]

• PR to update QA Standards template
• Change history draft
• Draft update of How to complete a QA Review for the Staging Review touchpoint
• Draft update QA Standards page on public website

[it-harrison]

Draft of slack post for #vfs-all-teams

:megaphone Attention: VFS Teams

All QA Standards are now launch-blocking.

On [xx/xx/xx], all QA Standards will be enforced as launch-blocking at the Staging Review touchpoint.

Impact: No product moving through the Collaboration Cycle can launch on VA.gov without satisfying all QA Standards.

Documentation: QA Standards.

Questions: Please post any questions in thread or reach out to #vfs-platform-support.

[it-harrison]

@erinrwhite this is ready for review

[erinrwhite]

@it-harrison All these are looking good. Only one super-small change to the Slack change announcement title: "All QA Standards are now launch-blocking at Staging Review"

[it-harrison]

All documentation updated; will post slack once publication to public website complete.

[shiragoodman]

@it-harrison what will be our enforcement date for this?

[it-harrison]

@it-harrison what will be our enforcement date for this?

@shiragoodman Feb 10

[shiragoodman]

@it-harrison I haven't seen the change announcement for this get published to #vfs-change-announcments. Please adjust the Feb 10 date to 2 weeks after the announcement is published. For example, if we publish today, the enforcement date should be Feb 11. If we publish tomorrow, enforcement should be Feb 12.

Please lmk if you have any questions. ty!

[erinrwhite]

@it-harrison @shiragoodman this just got published to platform website! If the change announcement goes out this afternoon we should be good, yeah? 2/10 is 2 weeks from today...

[erinrwhite]

@it-harrison these are all looking good! Don't forget to squash and merge the PR. AC box checked