Calling another build system from dune with a job limit

[rlepigre]

I have a situation where I need to call make to build a large set of files, so currently I'm doing something like the following (simplified):

(rule
 (targets %{read-lines:targets.txt})
 (deps ...)
 (action (bash "make -j4 ...")))

The problem is that if I do not specify a -j option for make, then the build takes ages. On the other hand, one might want to tweak the given number of jobs to the number of available CPUs. Another issue is that if dune runs with N processes, then at some point during the build there may be up to N - 1 + 4 jobs, which is not very nice. In my case, I actually have several such rules, so the situation is even worst.

I thought I might be able to use the DUNEJOBS variable here, but I don't think that's true.

If there is no solution, it could be nice to extend actions DSL with a (with-jobs N ...) stanza, that specifies that N jobs need to be allocated for the action. Any thoughts on that?

[ejgallego]

Hi @rlepigre , this has been discussed quite a few times (don't have time now to dig old discussions issues, but I think it was titled "jobserver protocol")

As far as I know, there is not easy solution yet, but indeed, it would be great to be able to implement something in the lines you propose.

[SkySkimmer]

#4331