Ensure that CMake and Python frontend script generate the same build system

[robertodr]

As discussed in #219 we need to make sure that using bare CMake and the frontend script gives the same result. i admit that using the frontend script should be preferred: it can be more concise and actually saves the setup command for later use. However, some users might be already very familiar with bare CMake and we need to make sure not to confuse them.

The strategy here is to make sure that options defined in modules always have a default value. The Python frontend script defines all such options with sensible defaults, the mechanism for this is the define section on top of the module files. This has to be replicated for bare CMake and I've added three macros (thanks @loriab and @ryanmrichard 🎉) to the top of the autogenerated root CMakeLists.txt that can be used for the purpose:

• print_option(<option to print> <was specified>). Pretty-prints an option.
• option_with_print(<option name> <description> <default value>). Wraps an ON/OFF option and pretty-prints it too.
• option_with_default(<option name> <description> <default value>). Wraps any other type of option and pretty-prints it too.

I have fixed some bugs and "fortified" math_libs.cmake to illustrate how this works. Basically, any -DTHIS and -DTHAT that appear in the define sections need to be also listed in the CMake script using option_with_print or option_with_default. This is tedious, but saves a lot of WTF moments and imposing one way of doing on users. As a byproduct of hande-qmc/hande#3, detection and linkage of Intel MKL ScaLAPACK works slightly better. ScaLAPACK from other vendors can be set via SCALAPACK_LIBRARIES.

[bast]

Thank you! I like it :-) The one thing I would change is to not build the text in gen_cmake_options_wrappers line by line, but with one long string between triple-quotes. But that is a detail just to make it easier to read and can be patched later and should not block this one.

[robertodr]

I've implemented your suggested change. The string now also does include(CMakeDependentOption). I have no idea why the tests are failing...

[bast]

So what separates us from WIP to "done" is the failing Boost MPI test? We could either try to figure out why the warning is there or adapt the test script to not choke on the warning.

[robertodr]

Yeah, definitely the failing test should be fixed (though I can't reproduce it on my laptop...) I was also hoping to go through the modules and check whether any of them needs "fortification" with the option wrappers.

[robertodr]

I can't find the time to work on "consolidating" the standard modules as I had intended to do. Do you think we could get this merged and fix the rest piecemeal in other PRs?

[bast]

I agree. I anyway prefer smaller PRs. I have restarted the build.

[bast]

If it fails, I shall prioritize the debugging so that we get this change in.

[bast]

I am working on this today in-between things.

[bast]

test_boost_libs passes on my laptop and fails on Travis - argh! What to do? Merge, let it fail and open an issue that documents that we know about the failing test?

[bast]

Alternative is that I go for some fun-times Travis debugging :-)

[robertodr]

I say merge-and-issue. Those modules should be rewritten as a superbuild anyway...

[bast]

Boost issue followed up in #227. Merging.

[miroi]

Hi Rado and Roberto,

....could not fetch //sourceforge.net/projects/boost/files/boost/1.59.0/boost_1_59_0.zip/download

some "download demanding" tests must be improved. For instance, I was recommended to build openblas from source, or, how about to make safe web space for downloading all important libraries (openblas, boost) ?

We can not always rely on external providers....

[bast]

I agree, these downloads from SourceForge et al. can be a pain. But do you know a place where one can store tarballs where the place is not a Git repo and not yet another server we need to maintain?

[miroi]

Somewhere

yourwebserver/~bast
mywebserver/~miro
robertowebserver/~roberto

or http://diracprogram.org/coderefinery_storage

[bast]

I am really looking for a solution that we do not maintain ourselves.

[robertodr]

I have no idea. It is a "vulnerability" of the current scheme of things that we have to live with, I think. I don't want to maintain an FTP server, full stop. The Boost thing was created as a workaround, I don't like it and I would never recommend its usage to anybody. If a library is needed for code A, install it by hand before building by code A.

On Travis you can have build caches, but that's not the answer you guys were looking for.

[bast]

Same here, I am running too many servers already. I only want to reduce the number, never increase.

[miroi]

Folks, Rado,

no new server for maintenance, only our old, good DIRAC server.

see
http://diracprogram.org/doc/coderefinery_storage/

and try online command

 wget http://diracprogram.org/doc/coderefinery_storage/hello

[bast]

But then we entangle two unrelated projects (well they are linked through the people). If "DIRAC" decides to change things, Autocmake will break. So thank you for the suggestion and efforts but for me this is still a self-maintained solution which has own risks.