Move performance testing YAML from dotnet/runtime to dotnet/performance #4639

caaavik-msft · 2025-01-15T11:06:46Z

This PR implements the dotnet/performance portion of the work to move the performance testing YAML to do the dotnet/performance repository. You can find the corresponding change for the dotnet/runtime repository here: dotnet/runtime#111454.

Summary

The goal of this work is to move as much of the CI logic as possible out of the dotnet/runtime repository and into the dotnet/performance repository. The reason we want to make this change is to decouple what is being tested from the code that does the testing.

We have had many occurrences over the last few years where there would be a bug in the performance testing logic that caused invalid/skipped data or we have had to make a breaking change. Having this logic live in the dotnet/runtime repository meant we were unable to re-run performance tests when the bug was present or before a breaking change was made. With the changes from this PR, it means that if we ever have a bug that causes invalid/skipped data, we can make the change in the dotnet/performance repo and then re-run the performance tests with those fixes and correct any data issues.

Implementation

It is not practical to move everything out of the dotnet/runtime repository as there are some steps of our performance tests that may be strongly coupled to the version of the runtime. In particular, these would be any jobs that involve building the runtime or building test applications. To handle this, we have made extensive use of cross-repository template references.

If you look at the dotnet/runtime portion of this PR, you will see that eng/pipelines/coreclr/perf.yml will include the following lines:

resources:
  repositories:
    - repository: performance
      type: git
      name: internal/dotnet-performance

- template: /eng/pipelines/runtime-perf-jobs.yml@performance

There is a repository resource named performance which describes where the performance repository is located. For more information about repository resources, see the documentation here. Then we are able to reference a template from the performance repository by putting @performance at the end. When manually running the pipeline in Azure DevOps, under "Advanced options" there is a section called Resources which allows you to customise a branch/commit of the performance repository you want to run against, which is useful when testing changes that need to be made against both the runtime and performance repositories.

If you look at /eng/pipelines/runtime-perf-jobs.yml in this PR, you can see the following line:

- template: /eng/pipelines/coreclr/templates/perf-build-jobs.yml@${{ parameters.runtimeRepoAlias }}

This line is how the performance repository is able to reference back into the runtime repository and call the perf-build-jobs.yml template inside it. In the dotnet/runtime repository PR, it sets runtimeRepoAlias to self which means that it will reference the exact version of the runtime repository that called it. In a future PR, we will be able to add an additional pipeline to the dotnet/performance repository which will has a runtime repository resource defined so that we can verify that a PR isn't going to cause the runtime performance tests to break.

YAML Convergence

Before this PR, there was a lot of duplicated but slightly differing YAML files that did similar things. As part of this PR we are able to unify all this duplicate logic so that everything is only defined once and doesn't require making changes in lots of files. In particular I want to point out the following two files which are worth paying close attention to (both in the /eng/pipelines/templates/ directory):

run-performance-job.yml: Defines a job that clones the runtime and performance repositories, runs the performance job python script, and sends the helix job. Every performance test will go through this YAML file including those that run against the SDK or a build of the runtime.
runtime-perf-job.yml: A wrapper around run-performance-job.yml which sets up all the necessary parameters and additional steps for running performance tests against a build of the runtime. Most of this is just downloading the build artifacts and arranging them into the correct directories so that they can be used by the python scripts.

Other Changes

Job Names and Display Names
- Job names are now standardised and should be more human readable. As an example:
  - Before: Performance osx x64 release iOSMono JIT ios_scenarios perfiphone12mini NoJS True False True net10.0
  - After: Performance ios_scenarios iOSMono JIT iOSLlvmBuild osx x64 perfiphone12mini net10.0
- The ordering of the parts of the job name were changed so it is easier to scan through to find a particular job
- An additionalJobIdentifier parameter can be specified as extra information to include in the job name if needed to disambiguate two jobs.
Applying a parameter to every job
- If you look at eng/pipelines/coreclr/perf.yml you will see I added a parameter called onlySanityCheck which uses the new jobParameters parameter to set the onlySanityCheck parameter on every job that gets run.
- I have been using this in my testing and it has greatly sped up testing time because I don't have to block the helix queues up with a large amount of jobs when I don't care about performance test results.

Validation

Since this change is large, it is likely to introduce bugs. To validate that things have been ported correctly, I looked at the Run performance job script step of the job to see the command line arguments that were passed to the python script to ensure that they are identical. There are still potentially other classes of bugs due to variables being different which are harder to detect, but from my testing I expect any bugs to be minimal.

We should also address in a future PR ensuring that onlySanityCheck is properly implemented for all scenarios and test cases as it is only be active for our microbenchmarks. This should be fine for now though as our microbenchmarks are the ones that take the longest.

…sed in

DrewScoggins · 2025-01-15T17:56:10Z

eng/performance/gc_jobs.yml

Was this file dead code? I don't see us doing the work that was done by this file anywhere else.

Yes, this code hasn't been in use for a long time, the pipeline that used to reference this got deleted but this yaml file was not cleaned up with it.

DrewScoggins · 2025-01-15T17:58:16Z

eng/pipelines/sdk-perf-jobs.yml

Should we follow the naming convention we seem to have setup for other pipeline yaml files, and have performance- at the beginning of this?

This is following a convention, although we could change them all to a new convention if needed. There are 5 fives in this directory:

runtime-ios-scenarios-perf-jobs.yml

runtime-perf-jobs.yml

runtime-perf-slow-jobs.yml

runtime-wasm-perf-jobs.yml

sdk-perf-jobs.yml

I realised the slow one should be runtime-slow-perf-jobs to be consistent which I can fix. Essentially they all end in perf-jobs and the first section is either runtime or sdk depending on if it runs against a build of the runtime or against the sdk. Did you have any preference for how we should name these yaml files?

e-kharion · 2025-01-16T13:32:53Z

eng/pipelines/runtime-ios-scenarios-perf-jobs.yml

+    parameters:
+      hybridGlobalization: ${{ parameters.hybridGlobalization }}
+
+  # run mono iOS scenarios scenarios HybridGlobalization


nit: double "scenarios"

e-kharion · 2025-01-16T13:58:51Z

eng/pipelines/runtime-perf-jobs.yml

+        - android_arm64
+
+jobs:
+  - template: /eng/pipelines/coreclr/templates/perf-build-jobs.yml@${{ parameters.runtimeRepoAlias }}


I'm wondering whether it would make sense to convert those jobs to job strategy with matrix of parameters https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema/jobs-job-strategy?view=azure-pipelines
as majority of the "code" seems to be duplicated and only the params seems to vary

I'd hope this might result in much shorter code with easy to compare parameters

There is a lot of room to reduce duplication in these jobs using either matrix strategy or potentially using YAML loops. For now I'm keeping it the same as before, but would be open to such a change down the line.

e-kharion · 2025-01-16T14:08:31Z

eng/pipelines/templates/run-performance-job.yml

+- template: ${{ parameters.jobTemplate }}
+  parameters: 
+    enableTelemetry: ${{ parameters.enableTelemetry }}
+    name: run_performance_test_${{ replace(format('{0}_{1}_{2}_{3}_{4}{5}_{6}_{7}_{8}', parameters.runKind, coalesce(parameters.runtimeType, 'NULL'), coalesce(parameters.codeGenType, 'NULL'), coalesce(replace(parameters.additionalJobIdentifier, ' ', '_'), 'NULL'), parameters.osGroup, parameters.osSubGroup, coalesce(parameters.osVersion, 'NULL'), parameters.archType, coalesce(parameters.machinePool, parameters.logicalMachine)), '_NULL', '') }}


Would it make a sense to extract this logic into some dedicated piece of code? I suspect this might evolve into even more complicated code

I spent a lot of time trying to do this, but Azure Pipelines YAML does not have any way to really extract this logic out except by maybe more and more nested templates which I don't think is a scalable approach. If we can find any way to extract it into a reusable bit of code I would be open for it.

e-kharion · 2025-01-16T14:12:17Z

eng/pipelines/templates/run-performance-job.yml

+      - ${{ if eq(parameters.osGroup, 'windows') }}:
+        - name: Python
+          value: 'py -3'
+      - ${{ if ne(parameters.osGroup, 'windows') }}:


nit: using if-else pattern would make reading the code easier

e-kharion · 2025-01-16T14:14:36Z

eng/pipelines/templates/run-performance-job.yml

+        - ${{ step }}
+      - ${{ if and(ne(variables['System.TeamProject'], 'public'), notin(variables['Build.Reason'], 'PullRequest'), eq(parameters.downloadPdn, true)) }}:
+        - task: AzureCLI@2
+          displayName: 'Download PDN'


just curious: what do we need Paint.NET for???

and why do we have a personal "mirror" for it?

We maintain an internal mirror as we are not wanting to provide a way for people to download PDN without going through their official website/downloads. And we have some scenarios that do some performance testing against Paint.NET (startup time, size on disk etc.).

LoopedBard3

General high level functionality check seems to look good to me, will take a closer look at changes individually soon.

caaavik-msft added 30 commits January 3, 2025 07:38

Move runtime yaml into performance repository

45f3fe1

Fix missing end parenthesis

83f7e39

Fix string quoting issues

594cd74

Fix elseif

3ed6a84

Fix variable propagation

cb80bad

Fix variables to be an array

5daf236

Another attempt at fixing variables

ac24b7b

Fix invalid variables

24d4b60

Fix template paths

0b3ccc1

Fix more template paths

5734396

Fix indentation

0b0965b

Try fix the job name

b8d8cf1

Use variables correctly

f3a610f

Use only compile time parameters for job names

104d46b

Try remove new lines

25e2521

Use single quotes

4230f8c

Ensure spaces are converted to underscore

ed6c507

Change runtime flavor to be additional job identifier

b18dc9d

Fix conditions on arguments for python script

092fb6c

Add missing parenthesis

7e959aa

Ensure buildConfig is properly capitalised for artifacts

e1938e6

Ensure args are properly quoted and ensure custom targetCsproj is pas…

f2b0f78

…sed in

Fix dotnet version link argument

5db23b3

Fix duplicated logs

f50d21c

Merge branch 'main' into caaavik/move-runtime-yaml-to-performance

3b0ace9

Fix logs name

999bba6

Fix attempt# replace

13983ff

Another attempt at getting the log name to work

3be0ce3

Remove runtime expression from logs name

e37c149

Ensure sys.exit is called in run_performance_job.py

7a161ca

caaavik-msft added 14 commits January 15, 2025 13:15

Move all runtime jobs to performance repository

ee6b3bd

Fix cross repository template reference

774035b

Ensure runtime-perf-job.yml uses repository reference

e13a345

Ensure runtime is cloned first as it cleans the directory

3e9d633

Merge branch 'main' into caaavik/move-runtime-yaml-to-performance

4811589

Port changes from dotnet/runtime

c7cb12d

Fix template conditions

f1b3f25

Fix template condition

49bff19

Set correct path to performance repository

67209e1

Move pipelines to separate directory

6be2b25

Fix incorrect template for download artifact step

c2289fc

Fix unbalanced parentheses

c9f69e1

Fix more unbalanced parentheses

ecc7def

Fix indentation

7a191f8

caaavik-msft mentioned this pull request Jan 15, 2025

[PERF] Move performance testing YAML from dotnet/runtime to dotnet/performance dotnet/runtime#111454

Open

caaavik-msft requested review from DrewScoggins, LoopedBard3 and e-kharion January 15, 2025 11:11

DrewScoggins reviewed Jan 15, 2025

View reviewed changes

e-kharion reviewed Jan 16, 2025

View reviewed changes

LoopedBard3 reviewed Jan 17, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move performance testing YAML from dotnet/runtime to dotnet/performance #4639

Move performance testing YAML from dotnet/runtime to dotnet/performance #4639

caaavik-msft commented Jan 15, 2025 •

edited

Loading

DrewScoggins Jan 15, 2025

caaavik-msft Jan 15, 2025

DrewScoggins Jan 15, 2025

caaavik-msft Jan 15, 2025 •

edited

Loading

e-kharion Jan 16, 2025

e-kharion Jan 16, 2025

caaavik-msft Jan 16, 2025

e-kharion Jan 16, 2025

caaavik-msft Jan 16, 2025

e-kharion Jan 16, 2025

e-kharion Jan 16, 2025

e-kharion Jan 16, 2025

caaavik-msft Jan 16, 2025

LoopedBard3 left a comment

Move performance testing YAML from dotnet/runtime to dotnet/performance #4639

Are you sure you want to change the base?

Move performance testing YAML from dotnet/runtime to dotnet/performance #4639

Conversation

caaavik-msft commented Jan 15, 2025 • edited Loading

Summary

Implementation

YAML Convergence

Other Changes

Validation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caaavik-msft Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LoopedBard3 left a comment

Choose a reason for hiding this comment

caaavik-msft commented Jan 15, 2025 •

edited

Loading

caaavik-msft Jan 15, 2025 •

edited

Loading