Skip to content

Commit

Permalink
Update 2-1 Noise In Modern Systems.md
Browse files Browse the repository at this point in the history
  • Loading branch information
dendibakh authored Sep 13, 2024
1 parent 27a1e0b commit 4a85ad8
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Consider a situation when you implemented a code change and want to know the rel

However, it is not possible to replicate the exact same environment and eliminate bias completely: there could be different temperature conditions, power delivery spikes, unexpected system interrupts, etc. Chasing all potential sources of noise and variation in a system can be a never-ending story. Sometimes it cannot be achieved, for example, when you're benchmarking a large distributed cloud service.

You should not eliminate system nondeterministic behavior when you want to measure real-world performance impact of your change. Users of your application are likely to have all the features enabled since these features provide better performance. Yes, these features may contribute to performance instabilities, but they are designed to improve the overall performance of the system. In fact, your customers probably do not care about nondeterministic performance as long as it helps to run as fast as possible. So, when you analyze the performance of a production application, you should try to replicate the target system configuration, which you are optimizing for. Introducing any artificial tuning to the system will change results from what users of your service will see in practice.[^3]
You should not eliminate system nondeterministic behavior when you want to measure the real-world performance impact of your change. Users of your application are likely to have all the features enabled since these features provide better performance. Yes, these features may contribute to performance instabilities, but they are designed to improve the overall performance of the system. In fact, your customers probably do not care about nondeterministic performance as long as it helps to run as fast as possible. So, when you analyze the performance of a production application, you should try to replicate the target system configuration, which you are optimizing for. Introducing any artificial tuning to the system will change results from what users of your service will see in practice.[^3]

[^1]: By cold processor, we mean the CPU that stayed in idle mode for a while, allowing it to cool down its temperature.
[^2]: One approach to enable statistically sound performance analysis was presented in [@Curtsinger13]. This work showed that it's possible to eliminate measurement bias that comes from memory layout by repeatedly randomizing the placement of code, stack, and heap objects at runtime. Sadly, these ideas didn't go much further, and right now, this project is almost abandoned.
Expand Down

0 comments on commit 4a85ad8

Please sign in to comment.