Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More parallel tests #1255

Open
Sjors opened this issue Mar 30, 2023 · 5 comments
Open

More parallel tests #1255

Sjors opened this issue Mar 30, 2023 · 5 comments

Comments

@Sjors
Copy link
Member

Sjors commented Mar 30, 2023

When running make -j… check I noticed it doesn't use more than 3 CPU's at any one time, one for tests, exhaustive_tests and noverify_tests. At least when I tried it on an Intel macOS 13.2 machine.

This is a bottleneck when running the full Bitcoin Core test suite on something slow like with msan (tried on a Ubuntu 22.10 x86_64 machine using Docker).

That said, it's not the end of the world either, since I could also just run more test suite (Docker containers) in parallel.

@real-or-random
Copy link
Contributor

When running make -j… check I noticed it doesn't use more than 3 CPU's at any one time, one for tests, exhaustive_tests and noverify_tests. At least when I tried it on an Intel macOS 13.2 machine.

Yep, the only parallelism we have is that one from different binaries.

As you say, it's not the end of the world, but it would be good to improve this if it's a bottleneck in Core. I thought a few times about this issue, but my pain was never big enough that I actually spend time working on this.

  • Pragmatically speaking, we could just run 4x tests 16 and 4x noverify_tests 16 instead of 1x tests 64 and 1x noverify_tests 64. This means we'll run the deterministic part of the test suite 4x... Not elegant, but also not the end of the world. It errs on the other side of the spectrum.
  • Some "proper" unit testing framework would give us a way to run different test cases in different processes/threads. tests: allow user to select tests via command line args #1211 is a bit of a poor man's approach version of this, and it is ready right now. This will make it possible to run different subsets in different processes. One disadvantage is that we'll need to run to split the test suite manually and find the right command line args. It would perhaps make sense to add more generic arguments like "--split 4 --part 3", which would split the test cases into four groups and run only the third of it.

@apoelstra
Copy link
Contributor

This is also a pain-point for me -- with a 32-core machine, the bottlenecks in building Core are ./configure and the secp tests.

I like the idea of running ./tests 16 four times (or maybe we could be a little bit smart and divide 64 by the -j value). Yes, it's an inelegant hack, but it can also be done extremely quickly and backed out when we have a better solution.

@Sjors
Copy link
Member Author

Sjors commented Mar 31, 2023

Supporting -j is probably the easiest way to hook into Bitcoin Core's existing CI system. This value is passed to make but also to the functional tests. It's probably also the most intuitive: make -j33 check

@apoelstra
Copy link
Contributor

@Sjors FYI if you don't actually care to run the tests in full, you can set the environment SECP256K1_TEST_ITERS=1 (I think you can even set it to 0 ... we had a bug in the past from somebody doing that, so I guess we fixed it ... though even 0 will still run some tests).

@real-or-random
Copy link
Contributor

(or maybe we could be a little bit smart and divide 64 by the -j value

I checked, and I don't think there's no way to access the -j value from within the Makefile.

But yeah, we could still just use 4x16.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants