Benchmarking

[JoshuaKimsey]

Figured I would go ahead and start the discussion for the Benchmarking page of the wiki.
(If a true template get's added, then I'll edit my results to fit with that. :) )

I chose not to include the images because, in theory, using the same seed and setup should produce the exact same images. And also I wanted to keep this benchmarking page from being overly bloated.

System Specs

Hardware:
    - CPU: Intel i9-9900k
    - GPU: ZotacGaming Nvidia RTX 2080 (Factory Overclocked) - 8GB VRAM
    - RAM: 64GB DDR4

Software:
    - System OS: Pop!_OS 22.04 Linux (Debian/Ubuntu-based)

Benchmark Results

k_lms: Took 8.86s total (8.86s per image) Peak memory usage: 7897 MiB / 8192 MiB / 96.393%
k_heun: Took 16.07s total (16.07s per image) Peak memory usage: 7810 MiB / 8192 MiB / 95.33%
k_euler: Took 8.37s total (8.37s per image) Peak memory usage: 7804 MiB / 8192 MiB / 95.254%
k_euler_a: Took 8.22s total (8.22s per image) Peak memory usage: 7821 MiB / 8192 MiB / 95.468%
k_dpm_2: Took 16.15s total (16.15s per image) Peak memory usage: 7794 MiB / 8192 MiB / 95.14%
k_dpm_2_a: Took 16.16s total (16.16s per image) Peak memory usage: 7808 MiB / 8192 MiB / 95.301%
PLMS: Took 8.43s total (8.43s per image) Peak memory usage: 7832 MiB / 8192 MiB / 95.597%
DDIM: Took 8.26s total (8.26s per image) Peak memory usage: 7863 MiB / 8192 MiB / 95.976%

If you'd like to do your own benchmarks, just check out the wiki page I linked at the top.
(Sorry @hlky if this isn't the final format you wanted to settle on for doing benchmarking. I'll happily adjust this accordingly as desired.)

[kalkal11]

Settings other than sampler not listed.

Assuming the defaults, however defaults chan be changed.

[JoshuaKimsey]

The Benchmark wiki page has the settings to use. I just posted the results of my benchmarking here to keep things cleaner looking.

[DenShlk]

Wrote small script in collab to make graphs for results in this format

[JoshuaKimsey]

Ok, this is awesome! Hey @hlky, think there is any chance this fairly simply bit of code from the collab could be implemented into a simple Benchmark function for people to use? Perhaps even just as a side-script users can call when they wanna benchmark Samplers?

[DenShlk]

Also, i frequently see that later batches are faster then first ones, so you might want to use multiple batches