WNMGDS-3037 Adds script to compute percentage of nightly tests that pass #3320

tamara-corbalt · 2024-11-15T19:10:23Z

Summary

This script analyzes jenkins nightly build data to compute a general pass rate and outputs a list of failed builds for manual inspection.
Note: It utilizes the Puppeteer library to simulate logging into Jenkins as a workaround, pending resolution of issues with setting up a proper API token.

// Sample output for a range of builds
% npx ts-node get-nightly-build-pass-rate.ts 750 3
Logging in to Jenkins...
Login successful.
Analyzing builds from 748 to 750...
Fetching data for build #748...
Fetching data for build #749...
Fetching data for build #750...
Analysis complete.
Pass Count: 3
Fail Count: 0
Pass-to-Fail Rate: 100.00%
Failed Builds:

Jira ticket

How to test

Pull down the branch
Ensure you are connected to the Zscaler VPN.
Navigate to scripts directory:cd scripts
Create a .env file in the scripts directory with the following variables:

JENKINS_ID=<your-eua-id>
JENKINS_PASSWORD=<your-password>
JENKINS_LOGIN_URL=<Jenkins login URL>
JENKINS_BASE_API_URL=<Jenkins base API URL>

Run yarn install
Run npx ts-node get-nightly-build-pass-rate.ts <build number> <numberOfDays>
- Replace with the latest jenkins build number you want to inspect.
- Replace with the range of days to analyze (optional; defaults to 1 if omitted).

Checklist

Prefixed the PR title with the Jira ticket number as [WNMGDS-####] Title or [NO-TICKET] if this is unticketed work.
Selected appropriate Type (only one) label for this PR, if it is a breaking change, label should only be Type: Breaking
Selected appropriate Impacts, multiple can be selected.
Selected appropriate release milestone

tamara-corbalt · 2024-11-18T19:12:37Z

scripts/get-nightly-build-pass-rate.ts

+        }
+
+        let data: { result?: string } | null = null;
+        try {


Planning to move the block that handles parsing and processing build data into a separate function in the future. This will make the code more modular and easier to maintain.

Nit: This seems pretty straightforward and not super long, I think you could leave it inlined but a separate function is fine too!

pwolfert · 2024-11-18T19:21:45Z

Would the private toolkit repo be a more appropriate place for this?

If you're using Puppeteer instead of Playwright, there's no gain in shared dependencies by putting it in this repo anyway. That is, you'd have to install a browser-testing tool in the toolkit, but you're installing a new one anyway.

tamara-corbalt · 2024-11-18T21:09:40Z

Would the private toolkit repo be a more appropriate place for this?

If you're using Puppeteer instead of Playwright, there's no gain in shared dependencies by putting it in this repo anyway. That is, you'd have to install a browser-testing tool in the toolkit, but you're installing a new one anyway.

Hey @pwolfert, that's a great call-out, and I completely agree. I think this script would be a better fit for our private toolkit repo. I'll work on moving it over. Thanks for pointing that out! 😊

jack-ryan-nava-pbc · 2024-11-19T00:28:33Z

scripts/get-nightly-build-pass-rate.ts

+
+const loginAndAnalyzeBuilds = async (
+  latestBuildNumber: number,
+  numberOfDays: number,


I think we're more likely to have the start date of a PI on hand rather than the number of days back we want to go. Do you think you could change this to accept a PI start date?

I think accepting a date as input might introduce some challenges. Are you suggesting that the user manually identify the build number for the start of the PI based on that date? I feel that could make things more difficult, especially since we'll likely be running this script after a PI concludes.

It seems easier to locate the build number for the end of the PI—jenkins and our slack channel make it straightforward to find the most recent build. Backtracking to determine the exact build corresponding to a PI's start date would likely require more effort on the user's part. That said, if there’s an automated way to map dates to build numbers, I’d be happy to explore that!

Oh I'm just suggesting that instead of saying the number of days back you want to go back to have a user specify a date to go back to. You could do the calculation by replacing const startBuildNumber = latestBuildNumber - numberOfDays + 1; with something more complicated but a little more natural for users:

const daysInAMillisecond = .000000011574 const now = Date.now() const then = new Date(process.agv[3]) <- or whatever the date input argument is const diff = Math.abs(now - then) const numberOfDays = Math.floor(daysInAMillisecond * diff) <- or Math.ceil(), depends on how we need to round const startBuildNumber = latestBuildNumber - numberOfDays + 1

I'm also asking you to work with Dates and Time in Javascript, so it's ok if that's just a categorical 'no' 😆

@jack-ryan-nava-pbc, Ohhh I misunderstood! I think I see the appeal in using a specific date to calculate the starting build number—it definitely aligns more naturally with how we think about time. However, since our PIs are typically a fixed duration (I should double check this assumption), sticking to a set number of days might keep things simpler and more predictable for our use case?

Well, a fixed number of days wouldn't work if the PI isn't complete yet right?

Well, a fixed number of days wouldn't work if the PI isn't complete yet right?

@jack-ryan-nava-pbc Ah, I see your point now! I think I might not have been super clear earlier. When I mentioned a set number of days, I was referring to the fact that PIs are typically a fixed duration (e.g., 54 days for PI4). Since this script is mostly being used in retrospect, I thought it might be easier to find the final build number and work backward a set number of builds, rather than trying to pinpoint the exact date for the start of the PI.

If you feel strongly about using dates, let’s move this discussion over to pr-75 since I’m planning to close this one out.

jack-ryan-nava-pbc · 2024-11-19T00:29:12Z

scripts/get-nightly-build-pass-rate.ts

+      page.click('button[type="submit"]'),
+      page.waitForNavigation({ waitUntil: 'networkidle2' }),
+    ]);
+    console.log('Login successful.');


Just checking but pretty much everything before here will be removed once you get access to the Jenkins REST API?

Yes, that's the plan. Once we sort out the issue with API credentials, we can probably dispense with Puppeteer, which feels like a heavy-handed tool for what this script could be. At that point, we’d likely switch to something simpler, like node-fetch, to handle the requests.

jack-ryan-nava-pbc · 2024-11-19T00:31:26Z

scripts/get-nightly-build-pass-rate.ts

+          continue;
+        }
+
+        const responseBody = await response.text();


Could you call response.json() here and remove the later call to JSON.parse? Maybe not since Puppeteer is returning HTML?

So, I didn't think it was guaranteed that the response would always be JSON. Since we're retrieving page content, it might return HTML or plain text, so I felt using response.text() and handling JSON parsing in a try/catch block was a safer approach for now. We can revisit and potentially refactor this once we move away from Puppeteer.

jack-ryan-nava-pbc · 2024-11-19T00:34:54Z

scripts/get-nightly-build-pass-rate.ts

+        }
+
+        let data: { result?: string } | null = null;
+        try {


Nit: This seems pretty straightforward and not super long, I think you could leave it inlined but a separate function is fine too!

jack-ryan-nava-pbc · 2024-11-19T00:40:34Z

scripts/get-nightly-build-pass-rate.ts

+    console.log(`Pass Count: ${passCount}`);
+    console.log(`Fail Count: ${failCount}`);
+    console.log(`Pass-to-Fail Rate: ${passToFailRate.toFixed(2)}%`);
+    console.log(`Failed Builds: ${failedBuilds.join(', ')}`);


This is looking great! I would love to have these results written to a file. Do you think you could write these out to a CSV with 4 columns? PI number, pass count, fail count, ratio and maybe a list of the builds that were not accounted for for whatever reason. Does that sound like a good structure/reasonable?

Oh yes, good call! I'd like to see better formatting for this output.

@jack-ryan-nava-pbc, After giving it some more thought, I think it might be best to hold off on finalizing the output format. I discussed some possible new features with @kim-cmsds earlier, and I foresee that we might be adding some elements down the road that could impact how we structure our data output. So to avoid revisions, I'd like to delay this for now. What do you think?

Sounds good! I'm totally onboard.

jack-ryan-nava-pbc · 2024-11-19T00:41:31Z

scripts/get-nightly-build-pass-rate.ts

+        }
+
+        if (!data || !data.result) {
+          console.log(`Build #${buildNumber}: Data unavailable or missing result.`);


Can we track all these weird data unavailable cases? Like if the server returns a 500 or if there are missing data on a build for some other reason?

I'll add more descriptive logging for unavailable cases

tamara-corbalt · 2024-11-20T20:50:34Z

Moving this script into experience-toolkit repo.

tamara-corbalt added 2 commits November 15, 2024 13:39

Adds logic to compute pass rate

7675b93

Adds todo comments.

5a29155

tamara-corbalt added Type: Internal This item relates to internal tooling/maintenance Type: Added Indicates a new feature. labels Nov 15, 2024

tamara-corbalt added this to the 12.1.0 milestone Nov 15, 2024

Adds logic to handle the single build case

3d25604

tamara-corbalt marked this pull request as draft November 15, 2024 19:27

tamara-corbalt added 5 commits November 18, 2024 12:46

Adds a puppeteer script

67888d8

Integrates puppeteer logic with original script

cf1bac0

Removes jenkins-login.ts

8369e9c

Removes unnecessary comment

214b0d5

Removes two additional unncessary comments.

c0f460a

tamara-corbalt marked this pull request as ready for review November 18, 2024 19:05

tamara-corbalt requested review from kim-cmsds, jack-ryan-nava-pbc and pwolfert November 18, 2024 19:05

tamara-corbalt commented Nov 18, 2024

View reviewed changes

jack-ryan-nava-pbc reviewed Nov 19, 2024

View reviewed changes

tamara-corbalt closed this Nov 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WNMGDS-3037 Adds script to compute percentage of nightly tests that pass #3320

WNMGDS-3037 Adds script to compute percentage of nightly tests that pass #3320

tamara-corbalt commented Nov 15, 2024 •

edited

Loading

tamara-corbalt Nov 18, 2024

jack-ryan-nava-pbc Nov 19, 2024

pwolfert commented Nov 18, 2024 •

edited

Loading

tamara-corbalt commented Nov 18, 2024

jack-ryan-nava-pbc Nov 19, 2024

tamara-corbalt Nov 19, 2024

jack-ryan-nava-pbc Nov 19, 2024

jack-ryan-nava-pbc Nov 19, 2024

tamara-corbalt Nov 19, 2024

jack-ryan-nava-pbc Nov 19, 2024 •

edited

Loading

tamara-corbalt Nov 20, 2024 •

edited

Loading

jack-ryan-nava-pbc Nov 19, 2024

tamara-corbalt Nov 19, 2024

jack-ryan-nava-pbc Nov 19, 2024

tamara-corbalt Nov 19, 2024

jack-ryan-nava-pbc Nov 19, 2024

jack-ryan-nava-pbc Nov 19, 2024

tamara-corbalt Nov 19, 2024

tamara-corbalt Nov 19, 2024

jack-ryan-nava-pbc Nov 19, 2024

jack-ryan-nava-pbc Nov 19, 2024

tamara-corbalt Nov 19, 2024

tamara-corbalt commented Nov 20, 2024

WNMGDS-3037 Adds script to compute percentage of nightly tests that pass #3320

WNMGDS-3037 Adds script to compute percentage of nightly tests that pass #3320

Conversation

tamara-corbalt commented Nov 15, 2024 • edited Loading

Summary

How to test

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pwolfert commented Nov 18, 2024 • edited Loading

tamara-corbalt commented Nov 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jack-ryan-nava-pbc Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

tamara-corbalt Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tamara-corbalt commented Nov 20, 2024

tamara-corbalt commented Nov 15, 2024 •

edited

Loading

pwolfert commented Nov 18, 2024 •

edited

Loading

jack-ryan-nava-pbc Nov 19, 2024 •

edited

Loading

tamara-corbalt Nov 20, 2024 •

edited

Loading