Instructions for building using spack

[sampotter]

It would be helpful to add some instructions for how to compile and install this library using spack. @steo85it pinging you since you just had some luck with this. I'm going to use spack on the clusters at work once I start running some larger tests for the paper (most likely this week).

[steo85it]

Sure, I can give it a try: creating the main package is easy, what will be a bit "challenging" (but helpful to learn) is how to include instructions to include and install, e.g., embree or cgal dependencies.
Also, for specific libraries (not too widely used by different users and projects) even on a cluster one would probably still use a pip environment (based, e.g., on a system-wide spack installation of python, matplotlib, numpy, gcc, etc).

[steo85it]

Ok, here is a start of a spack package which should go in ${SPACK_REPO}/var/spack/repos/tutorial/packages/python-flux/package.py and be called as spack install python-flux (following spack's standard, it would rather be py-python-flux, but well). I need to figure out how to include the headers... that shouldn't be too hard.

python-flux.tar.gz

[steo85it]

Ok, it was really easy (just tell spack that cgal should be both ('build','link') and it will include it automatically in the paths). Here is the functioning (in the sense of building+installing) package
python-flux.tar.gz

On the other hand, one would also need to update the cgal package, to be able to require [email protected]: in the dependencies, as we have seen that [email protected] gives conflicts (for some reason, the source path in github changed for the latest versions, which forces to set it individually for this version).
cgal.tar.gz

Have a look if it works for you (then we can "decorate it" with infos, etc), else we can iterate and correct what's missing! ;)

[steo85it]

(clearly if we want to use embree, we would also need to add depends_on('embree', type=('build','link')) and the dependency on python-embree - which I'm not sure how to do, except for creating an additional spack package for it ... but maybe there is another way )

[sampotter]

Thanks for looking into this.

How do I install your package? Feel free to just point me to relevant part of the documentation if you prefer...

For getting the right version of CGAL installed, it seems like spack edit cgal followed by messing around with the contents of the Python file that opens up is the right approach. Not sure how to commit any edits I make to spack later, but can at least attempt to get the right version of CGAL installed to start.

[steo85it]

How do I install your package? Feel free to just point me to relevant part of the documentation if you prefer...

Do you have spack installed on your system? If yes, just copy the content of that tar.gz to ${SPACK_REPO}/var/spack/repos/tutorial/packages/. and type spack install python-flux (after doing the same with the modified cgal, this time in ${SPACK_REPO}/var/spack/repos/builtin/packages/.). If not, first install it by following https://spack.readthedocs.io/en/latest/getting_started.html#installation.

Not sure how to commit any edits I make to spack later, but can at least attempt to get the right version of CGAL installed to start.

The installation works with my modified package (which I edited the way you point out - that Python file IS the "package" - which is the same as tracking down the file and opening it with whatever ${EDITOR} you like). To make it part of spack, one needs to make a PR in their github repo: they are usually quite responsive. We can have a look at it later, after testing if the loaded module really works as expected. ;)

[sampotter]

OK, before using your package, I wanted to get this working on my own so I could understand how it works a bit better.

I had to do something like this:

# get spack
git clone -c feature.manyFiles=true https://github.com/spack/spack.git

# make spack active
. spack/share/spack/setup-env.sh 

# make an "environment" for the radiosity project
spack env create radiosity
spack env activate radiosity

spack edit cgal
# ... make modifications so that CGAL 5.3.1 is installed. Just need to:
#     - replace the link to latest version with the one for 5.3.1 from GitHub
#     - update the version string
#     - update the SHA256 string ...

# install and load everything
spack install gcc cgal boost python py-pip py-cython
spack load gcc cgal boost python py-pip py-cython

# install python-flux
git clone https://github.com/sampotter/python-flux
cd python-flux
python setup.py install --user

However, in order to get the final line to work, I had to update setup.py and manually pass the include directories for spack's cgal and boost. To do this, I had to find where they were using e.g. spack location -i cgal. One thing I've noticed is that spack is SLOOOW. Observe:

[sfp8544@snappy1 python-flux]$ time spack location -i cgal
/home/sfp8544/spack/opt/spack/linux-centos7-ivybridge/gcc-9.1.0/cgal-5.3.1-negdj6dxxmey2o2rb4fkxugge77fxbsj

real    0m1.864s
user    0m1.001s
sys     0m0.235s

I have no idea what spack is doing, but nearly two seconds to find a directory??? Anyway, because I didn't want python setup.py install --user to take an additional four seconds (!) to run, I added:

`spack location -i cgal`/include
`spack location -i boost`/boost

to the include_dirs argument of the CGAL AABB extension in setup.py.

Some questions:

• Is what I'm doing here correct?
• Should spack pick up the location of CGAL and boost automatically?

[steo85it]

What you are doing looks correct to me (at least it's the same I did). I just manually added the include paths to the setup.py (which is faster), but your solution is more flexible. So no, I still don't know how to modify the module files created by spack so that the "include" dir gets automatically added to the path (only works for the lib dirs, somehow).  Else, is that command equally slow on your laptop? Sometimes our cluster is slow when running any command for the first time (i/o issues in the structure, I guess). Ah, and I didn't use my own package to install python-flux, just python+tk, cgal and boost, the same way you did: it should work.