Skip to content

Commit

Permalink
#0: Update pgm_dispatch_golden.json
Browse files Browse the repository at this point in the history
Since this golden file was created, the performance of
one_processors_all_cores_1_rta and all_processors_all_cores_1_rta seems to have
improved, so update the baseline file from a newer CI run.

The kernel preloading change slightly increased the size of kernel binaries,
since it added a loop that spun waiting for the go message. This slighlty hurt
the performance of some tests, since the increased size pushed them over some
thresholds (that were near powers of two in size).

Also increase the threshold to 5, since all the test flakes have been less than
5% off, and update the comparison script to show the percent difference.
  • Loading branch information
jbaumanTT committed Jan 19, 2025
1 parent eef7b86 commit a3b8577
Show file tree
Hide file tree
Showing 2 changed files with 459 additions and 456 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
)
)

THRESHOLD = 4
THRESHOLD_PCT = 5

parser = argparse.ArgumentParser(description="Compare benchmark JSON to golden")
parser.add_argument("json", help="JSON file to compare", type=argparse.FileType("r"))
Expand Down Expand Up @@ -59,12 +59,15 @@

golden_time = benchmark["IterationTime"] * 1000000
result_time = result["IterationTime"] * 1000000
if result_time / golden_time > (1 + THRESHOLD / 100):
print(f"Error:Test {name} expected value {golden_time:.2f}us but got {result_time:.2f}us")
result_diff_pct = result_time / golden_time * 100 - 100
if result_diff_pct > THRESHOLD_PCT:
print(
f"Error:Test {name} expected value {golden_time:.2f}us but got {result_time:.2f}us ({result_diff_pct:.2f}% worse)"
)
exit_code = 1
if golden_time / result_time > (1 + THRESHOLD / 100):
if result_diff_pct < -THRESHOLD_PCT:
print(
f"Consider adjusting baselines. Test {name} got value {result_time:.2f}us but expected {golden_time:.2f}us."
f"Consider adjusting baselines. Test {name} got value {result_time:.2f}us but expected {golden_time:.2f}us ({-result_diff_pct:.2f}% better)."
)

for name in result_benchmarks:
Expand Down
Loading

0 comments on commit a3b8577

Please sign in to comment.