parallel::makeCluster after setwd() to project directory takes forever

[Arthfael]

I am just adapting an existing workflow which makes heavy usage of parallel to renv. First user here, so might be doing something wrong. I noticed that if my R work directory is the project folder (as created using renv::init() or reloaded using renv::load()), and I then attempt to create my usual parallel cluster using parallel::makeCluster(), then it takes forever (a least 5-10 min, I have always ended up cancelling). Normally it might take 10-20 seconds but not this long. My current solution is to create a subfolder within the project folder and setwd() to it, it does work normally, but I was wondering whether this is expected behaviour.
And yes, I have read that setting work directory can be bad practice. I need to do it for 2 reasons:

• I used to and this is a complex workflow, and while I have probably replaced everything with explicit paths now I may still have old bits here and there which assume I am in wd.
• I do not have complete control over what some of the packages I use do and in some specific cases I definitely need to setwd().

Below abbreviated output from sessionInfo():

R version 4.4.2 (2024-10-31 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows Server 2022 x64 (build 20348)
...
renv_1.0.11

[KKamel67]

I had a renv project which run with version 0.17.3. I updated renv to install the version 1.0.11 and fix one problem. Since then, the command Parallel::makeCluster() inside the renv project takes between 5 and 10 minutes. Instead of the previous few seconds.

R version 4.1.0 (2021-05-18)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows Server x64 (build 14393) 
...
renv_1.0.11

[kevinushey]

Since then, the command doParallel::makeCluster() inside the renv project takes between 5 and 10 minutes.

Do you mean parallel::makeCluster()?

---

First, can you try running:

Sys.setenv(RENV_WATCHDOG_ENABLED = FALSE)
library(parallel)
cl <- makeCluster(2L)
stopCluster(cl)

andd see if the issue still reproduces?

If it does still, then can you try replacing your project's .Rprofile with something like:

if (!any(grepl("parallel:::", commandArgs())))
   source("renv/activate.R")

and see if that helps?

[KKamel67]

Sorry, I mean parallel::makeCluster(). I tried your first solution and the issue still reproduced. With the second one, parallel::makeCluster() works in a few seconds. Thank you for your answer.

[kevinushey]

Thanks, this is helpful! Can I ask you to try one more thing, so I can get some more information to fix this on the renv side?

Can you try using the following .Rprofile:

library(utils)

local({
  Rprof("trace.Rprof")
  on.exit(Rprof(NULL))
  source("renv/activate.R")
})

Then, try running:

library(parallel)
cl <- makeCluster(1L, type = "PSOCK")
stopCluster(cl)
summaryRprof("trace.Rprof")

and let me know what the output there is?

[KKamel67]

Here are the 2 tables in the output object.

byself.txt
bytotal.txt

And the 2 other elements :

tt$sample.interval
[1] 0.02
tt$sampling.time
[1] 130.04

N.B. : The same renv project is slow when I open it by a double click on the .Rproj file (~ 3 minutes).

[kevinushey]

Thanks! Can you confirm whether you're using Bioconductor packages within this project? It looks like the slowness is coming from a call (through the BiocManager package) to check whether the version of R + the version of Bioconductor being used are compatible.

[KKamel67]

Yes, I 'm using Bioconductor inside the project. I confirm that.

"Bioconductor": {
"Version": "3.14"
}

[Arthfael]

I haven't found time yet to run the tests but I am running Bioconductor too.

…

On Tue, Jan 28, 2025, 1:51 AM Kevin Ushey ***@***.***> wrote: Thanks! Can you confirm whether you're using Bioconductor packages within this project? It looks like the slowness is coming from a call (through the BiocManager package) to check whether the version of R + the version of Bioconductor being used are compatible. — Reply to this email directly, view it on GitHub <#2080 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSHV4LF5MZHHZIVQFG5MED2M3IB5AVCNFSM6AAAAABV4SOEP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMJXGM2DCOJZHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[kevinushey]

One other (simpler) workaround would be to set:

options(renv.bioconductor.validate = FALSE)

in your project's .Rprofile, before source("renv/activate.R"). I also have a fix for this in the development version of renv; hopefully to be released to CRAN soon-ish.

[KKamel67]

Thank you for your quick solution. Yes it is faster for sure.

[kevinushey]

This should now be fixed in the latest 1.1.0 release; please let me know if you're still having trouble.

[KKamel67]

Sorry for my late answer. I just tried the new version renv 1.1.1 and the project opens quickly without adding the lines that you mentioned in the .Rprofile.

I made a renv::snapshot() on the project and now when I open it i get the following message :

Error in (function (err) : Failed to parse 'renv.lock':
input string 1 is invalid UTF-8

Traceback (most recent calls last):
20: source("renv/activate.R")
19: withVisible(eval(ei, envir))
18: eval(ei, envir)
17: eval(ei, envir)
16: local(...) at activate.R#2
15: eval.parent(substitute(eval(quote(expr), envir)))
14: eval(expr, p)
13: eval(expr, p)
12: eval(quote(...), new.env())
11: eval(quote(...), new.env())
10: renv_bootstrap_exec(project, libpath, version) at activate.R#1309
 9: if (!renv_bootstrap_load(project, libpath, version))
      renv_bootstrap_run(project, libpath, version) at activate.R#1139
 8: renv::load(project) at activate.R#994
 7: renv_lockfile_load(project)
 6: renv_lockfile_read(path)
 5: withCallingHandlers(json <- renv_json_read(text = contents), 
        error = function(err) {
            stop("Failed to parse 'renv.lock':\n", conditionMessage(err))
        })
 4: renv_json_read(text = contents)
 3: stop(json)
 2: <condition-handler>(...)
 1: stop("Failed to parse 'renv.lock':\n", conditionMessage(err))

R version 4.1.0 (2021-05-18)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows Server x64 (build 14393)

Matrix products: default

locale:
[1] LC_COLLATE=French_France.1252  LC_CTYPE=French_France.1252    LC_MONETARY=French_France.1252 LC_NUMERIC=C                  
[5] LC_TIME=French_France.1252    

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.1.0 tools_4.1.0    renv_1.1.1 

I just saw after this message that there is an issue about this (#2089 )

[kevinushey]

Any chance you can share a reproducible example, or the lock file you have in that project?

[KKamel67]

Sure. Here is the lock file that I have.

renv_lock.txt

Should I open a new issue for that?