Profiling

This page contains resources and things to be aware of when profiling.

Do not make conclusions from profiling in debug mode

There is approximately a 3-6 speedup difference between debug and release mode.

Performance optimizations in release mode does not improve the performance uniformly across the program, so the bottleneck in debug mode might not be as in release mode.

Confirmation bias

A typical programmer mistake is basing decisions on hypothesis formed prior to testing.

In general, there are more ways a hypothesis can be wrong than there are ways it can be right.

When beliefs are formed prior to testing, it is easy to design tests to confirm the hypothesis, while lacking tests to check for cases that indicate the hypothesis is false.

Example:

Alice believes that drawing triangles clockwise is faster than drawing them counter clockwise. She thinks of a specific use case when clockwise might be faster, and then writes a test to check it. However, she forgot to think of other use cases where counter clockwise might be faster.

Performance is often not the only goal

Whenever there is a finite amount of multiple resources, a single goal might lead to waste of resources.

Because performance is easy to measure, it is also harder to communicate the importance of other goals.

Examples:

• Ergonomics
• Conceptually simple design
• Finishing within 2 years? 5 years?
• Cost of brain cycles when maintaining project
• Portability