bench: add utilities for profiling and tracing benchmarks #4041
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #4041 +/- ##
==========================================
+ Coverage 90.23% 90.24% +0.01%
==========================================
Files 458 457 -1
Lines 58369 58401 +32
Branches 5687 5691 +4
==========================================
+ Hits 52671 52706 +35
+ Misses 4317 4312 -5
- Partials 1381 1383 +2 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| setup() | ||
|
|
||
| offset_start = time.perf_counter() | ||
| benchmark_class_empty() |
There was a problem hiding this comment.
Can you leave a comment somewhere in the code to say why this is useful?
There was a problem hiding this comment.
Could be here, or on the class, or in both places
| return ( | ||
| mean(times) - mean(offset), | ||
| median(times) - mean(offset), | ||
| stdev(times) + stdev(offset), | ||
| ) |
benchmarks/bench_utils.py
Outdated
| def get_benchmark_runs( | ||
| args: Namespace, | ||
| benchmarks: dict[ | ||
| str, Callable[[], Any] | tuple[Callable[[], Any], Callable[[], Any]] |
There was a problem hiding this comment.
Added type alias -- this also makes the invocation in benchmark files neater which is nice!
Co-authored-by: Sasha Lopoukhine <[email protected]>
Co-authored-by: Sasha Lopoukhine <[email protected]>
|
Small note: our convention is to let the original commenter resolve the comments, makes it easier to track whether it's been addressed in the way that was expected. |
| from statistics import mean, median, stdev | ||
| from typing import Any, NamedTuple, cast | ||
|
|
||
| DEFAULT_OUTPUT_DIRECTORY = Path(__file__).parent / "profiles" |
There was a problem hiding this comment.
Isn't more natural to use the current working directory (i.e., os.get_cwd())? it might be convenient for the CI runs ATM though.
There was a problem hiding this comment.
I did this because it avoids making multiple profiles directories if you run the script in different places, as they are often large and we want to ensure they are correctly gitignored.
If it is confusing behaviour, am happy to change though.
Add the
bench_utilsfile to provide a simple command-line interface for developers to manually run and profile benchmarks outside of ASV.Peeled off from tracking PR #4008.