The VA Platform Product team is undergoing an EKS upgrade. As part of the process, it is necessary to conduct comprehensive testing on various components of Vets-API to ensure seamless functionality and performance during the upgrade.

From the EKS Testing Proposal:

As part of the current EKS upgrade process, comprehensive testing on various components of Vets API is essential in order to ensure seamless functionality and performance. Upgrading EKS can introduce changes that affect the behavior of deployed applications. Testing verifies that deployed applications remain compatible with any EKS changes. It ensures that all aspects of Vets API continue to operate as expected, without introducing regressions or unexpected behavior that could disrupt services for users.

Because of future goals to conduct upgrades on a 3-6 month cadence, it is the Product Platform team’s goal to maximize the automation of these upgrade testing processes. By automating these tests, verification of successful upgrades is potentially expedited and the right amount of time for necessary adjustments can be allocated. The Vets API components that require testing are below, along with the potential automation strategies. The objective is to standardize testing methodologies across the platform, thereby eliminating the inconsistency of testing solutions across EKS applications.

Part Ten of Testing plan - Load Testing

Tasks

• Discovery: Explore loading testing options
• Document Load Testing Procedure
• Outline load testing process
• Document the steps that will need to be followed by the person doing the upgrade testing in this Confluence document

Things to consider:

• This is the load testing repo: vets-api-loadtest
• @Kshitiz Shrestha Please fill out the details here that were discussed
• Use script from GitHub - rothgar/k8s-random-deploys: Deploy example workloads with random requests and limits to randomly scale and roll pods to check how it behaves.
• Use locust to mock traffic to application and we can create custom script which can watch the targeted HPA on behavior we want so we don't have to surf different page to see what is happening in system the script can notify us on slack or other means
• Another idea is to use GitHub - asobti/kube-monkey: An implementation of Netflix's Chaos Monkey for Kubernetes clusters to randomly kill pod and check how the system will handle it.
• Multiple resources may be required to do this load testing

How could this be automated?

• Could continue the use of Locust, as it is a perfectly fine tool with an existing set of tests (per Eric, this required several testing resources but perhaps an automated solution could be figured out)
• Peter also sees the recommendation of Jmeter in the new load testing repo. That’s also a great option. His personal preference would be for Postman, but we can adopt whatever requires the least amount of lead up time for now.

Things to consider:

• Review our current load testing documentation. Can we use this to load test vets-next?
• HPA Sidekiq Testing: Ensure Sidekiq Datadog integration works in the new EKS clusters: Sidekiq jobs will be enqueued in dev to promote scaling. The Sidekiq metrics should be the same for the old and new clusters.
• We might need a new domain for the VetsNext Cluster

Acceptability Criteria

• Comprehensive Load Testing Strategy
• Detailed Load Testing Documentation
• Readiness for Release
• The documentation is updated with the necessary steps and instructions for this step.