MultisessionFuture () failed to call gassign() on cluster ......

[elgabbas]

Hello,

I am trying to use the furrr package, but I think I am having a problem related to the use of future.

First, when I prepare to run the script in parallel using the following command, I receive these warnings. I am unsure if I should ignore them or there is an issue I need to solve.

plan(multisession, workers = 20)

During startup - Warning messages:
1: package ‘utils’ in options("defaultPackages") was not found
2: package ‘stats’ in options("defaultPackages") was not found
During startup - Warning messages:
1: package ‘utils’ in options("defaultPackages") was not found
2: package ‘stats’ in options("defaultPackages") was not found
During startup - Warning messages:
1: package ‘utils’ in options("defaultPackages") was not found
During startup - 2: package ‘stats’ in options("defaultPackages") was not found
Warning messages:
1: package ‘utils’ in options("defaultPackages") was not found
2: package ‘stats’ in options("defaultPackages") was not found
During startup - Warning messages:
1: package ‘utils’ in options("defaultPackages") was not found
2: package ‘stats’ in options("defaultPackages") was not found
During startup - Warning messages:
1: package ‘utils’ in options("defaultPackages") was not found
2: package ‘stats’ in options("defaultPackages") was not found
During startup - Warning message:
package ‘utils’ in options("defaultPackages") was not found
During startup - Warning messages:
1: package ‘utils’ in options("defaultPackages") was not found
2: package ‘stats’ in options("defaultPackages") was not found
During startup - Warning message:
package ‘utils’ in options("defaultPackages") was not found
During startup - Warning message:
package ‘utils’ in options("defaultPackages") was not found
During startup - Warning message:
package ‘utils’ in options("defaultPackages") was not found
During startup - Warning message:
package ‘utils’ in options("defaultPackages") was not found
Error: Initialization of plan() failed, because the test future used for validation failed. The reason was: Package 'future' is not installed on worker (r_version: 4.1.2 (2021-11-01); platform: x86_64-conda-linux-gnu (64-bit); os: Linux 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018; hostname: prod-0316)

Then, when I use map function from the furrr package, I receive the following error:

Error: Problem with mutate() column Preds.
Preds = future_map(...).
MultisessionFuture () failed to call gassign() on cluster RichSOCKnode #2 (PID 114424 on ‘localhost’). The reason reported was ‘error reading from connection’. Post-mortem diagnostic: The total size of the 15 globals exported is 14.99 MiB. The three largest globals are ‘...furrr_chunk_args’ (14.35 MiB of class ‘list’), ‘Models_WS’ (374.44 KiB of class ‘list’) and ‘predictMaxEnt’ (185.81 KiB of class ‘function’)
Backtrace:

1. ├─global::ExtractPreds(Data = CurrD)
2. │ └─%>%(...)
3. ├─tidyr::unnest(., cols = c(data))
4. ├─dplyr::rename_at(...)
5. │ └─dplyr:::tbl_at_vars(.tbl, .vars, .include_group_vars = TRUE)
6. │ └─dplyr::tbl_vars(tbl)
7. │ ├─dplyr:::new_sel_vars(tbl_vars_dispatch(x), group_vars(x))
8. │ │ └─base::structure(...)
9. │ └─dplyr:::tbl_vars_dispatch(x)
10. ├─tidyr::unnest(., cols = c(Preds))
11. ├─dplyr::mutate(...)
12. ├─dplyr:::mutate.data.frame(...)
13. │ └─dplyr:::mutate_cols(.data, ..., caller_env = caller_env())
14. │ ├─base::withCallingHandlers(...)
15. │ └─mask$eval_all_mutate(quo)
16. └─furrr::future_map(...)
17. └─furrr:::furrr_map_template(...)

18. └─furrr:::furrr_template(...)
    

19.   └─future::future(...)
    

20.     ├─future::run(future)
    

21.     └─future:::run.Future(future)
    

22.       ├─future::run(future)
    

23.       └─future:::run.ClusterFuture(future)
    

24.         ├─base::suppressWarnings(...)
    

25.         │ └─base::withCallingHandlers(...)
    

26.         └─future:::cluster_call(...)
    

Execution halted
Warning message:
system call failed: Cannot allocate memory

Please note that I am running this code on an HPC computer (CentOS Linux 7 ), with a total of 110 GB ram and 36 nodes. I used conda to install R (v 4.1.2). The same code worked nicely on different R installations but failed on this computer. Here is the results of the session info:

sessionInfo()

R version 4.1.2 (2021-11-01)
Platform: x86_64-conda-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
Matrix products: default
BLAS/LAPACK: /home/----/-----/.conda/envs/REnv2/lib/libopenblasp-r0.3.18.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods base

other attached packages:
[1] tibble_3.1.6 readr_2.1.2 furrr_0.2.3 future_1.24.0 purrr_0.3.4 dismo_1.3-5 raster_3.5-15 sp_1.4-6 sf_1.0-6 stringr_1.4.0
[11] tidyr_1.2.0 dplyr_1.0.7 snow_0.4-4

loaded via a namespace (and not attached):
[1] Rcpp_1.0.8 pillar_1.7.0 compiler_4.1.2 class_7.3-20 tools_4.1.2 digest_0.6.29 lifecycle_1.0.1 lattice_0.20-45
[9] pkgconfig_2.0.3 rlang_0.4.12 cli_3.2.0 DBI_1.1.2 e1071_1.7-9 terra_1.5-21 hms_1.1.1 globals_0.14.0
[17] generics_0.1.2 vctrs_0.3.8 classInt_0.4-3 grid_4.1.2 tidyselect_1.1.1 glue_1.6.2 listenv_0.8.0 R6_2.5.1
[25] parallelly_1.30.0 fansi_1.0.2 tzdb_0.2.0 magrittr_2.0.2 codetools_0.2-18 ellipsis_0.3.2 units_0.8-0 assertthat_0.2.1
[33] utf8_1.2.2 KernSmooth_2.23-20 stringi_1.7.6 proxy_0.4-26 crayon_1.5.0

The same issue occurred when using other strategies (multisession, multicore, cluster). I do not think this is due to a memory issue because the same code runs well on the same node specifications, but using an older version of R (not installed via conda). Is there something related to using conda?

Any suggestions?

Thanks in advance,
Ahmed

[orozcoae89]

Can you test this code? Is posible that you need the cl object, with follow:

library(future)
workers <-rep("localhost",4)
cl <- makeClusterPSOCK(workers, revtunnel=TRUE, outfile="")
# starting worker pid=134xx on localhost:11xxx at 13:46:34.292
# starting worker pid=134xx on localhost:11xxx at 13:46:34.580
# starting worker pid=134xx on localhost:11xxx at 13:46:34.817
# starting worker pid=134xx on localhost:11xxx at 13:46:35.050
plan <- plan(list(future::tweak(cluster, workers=workers), multisession))
plan

If you dont't solve send us the short example for replicate and help you.

Erick

[HenrikBengtsson]

A few comments and FYIs here:

When your workers run on the local machine (localhost), there is no need to use revtunnel=TRUE. That only applies for SSH:ing to an external machine. Also, makeClusterPSOCK() is quite clever and tries to automatically figure out when revtunnel needs to be enable or not, so it's rare that you need to specify it.

There is no need to set up nested parallelization for troubleshooting this problem. When you use plan(list(future::tweak(cluster, workers=workers), multisession)), you set up a nested set of workers. In this case you'll have four localhost workers at the first layer, and at the second layer, you'll end up with a set of multisession workers. This is explained in https://future.futureverse.org/articles/future-3-topologies.html

If one wish to use cluster futures running on localhost, I would set it up as:

library(future)
cl <- parallelly::makeClusterPSOCK(4, outfile="")
plan(cluster, workers = cl)

which is the same as:

library(future)
plan(cluster, workers = 4)

which is basically the same as:

library(future)
plan(multisession, workers = 4)

Finally, to display the new plan set, you need to use:

plan()

When you use:

plan <- plan(list(future::tweak(cluster, workers=workers), multisession))
plan

you'll end up displaying the old plan set.

[elgabbas]

Hello,

Sometimes, I also receive this error

Fatal error: couldn't allocate memory for pointer stack
Fatal error: couldn't allocate memory for pointer stack
Error: cannot allocate vector of size 9 Kb

*** caught segfault ***
address (nil), cause 'memory not mapped'
An irrecoverable exception occurred. R is aborting now ...

*** caught segfault ***
address (nil), cause 'memory not mapped'
An irrecoverable exception occurred. R is aborting now ...

*** caught segfault ***
address (nil), cause 'memory not mapped'
An irrecoverable exception occurred. R is aborting now ...

*** caught segfault ***
address (nil), cause 'memory not mapped'
An irrecoverable exception occurred. R is aborting now ...

*** caught segfault ***
address (nil), cause 'memory not mapped'
Error: memory exhausted (limit reached?)
Execution halted
Error: memory exhausted (limit reached?)
Execution halted
Warning message:
system call failed: Cannot allocate memory

*** caught segfault ***
address 0x5555559b9568, cause 'invalid permissions'
An irrecoverable exception occurred. R is aborting now ...

Thanks,
Ahmed

[elgabbas]

Thanks @orozcoae89 for your reply. Here is the results of the script you provided.

workers <-rep("localhost",4)
cl <- makeClusterPSOCK(workers, revtunnel=TRUE, outfile="")

starting worker pid=175624 on localhost:11798 at 13:59:27.898
starting worker pid=175627 on localhost:11798 at 13:59:27.898
starting worker pid=175625 on localhost:11798 at 13:59:27.899
starting worker pid=175626 on localhost:11798 at 13:59:27.907

starting worker pid=134xx on localhost:11xxx at 13:46:34.292

starting worker pid=134xx on localhost:11xxx at 13:46:34.580

starting worker pid=134xx on localhost:11xxx at 13:46:34.817

starting worker pid=134xx on localhost:11xxx at 13:46:35.050

plan <- plan(list(future::tweak(cluster, workers=workers), multisession))
plan

List of future strategies:

1. sequential:
   • args: function (..., envir = parent.frame())
   • tweaked: FALSE
   • call: NULL

I will implement my code then and see if it works.

Cheers, Ahmed

[orozcoae89]

This error is common when you don't have set the connections with remote machines. Can you give us more details about your code for help you?

Erick

[elgabbas]

After running my script, I receive similar error:

Error: Problem with mutate() column Preds.
ℹ Preds = future_map(...).
✖ ClusterFuture () failed to call gassign() on cluster RichSOCKnode #3 (PID 175879 on ‘localhost’). The reason reported was ‘error writing to connection’. Post-mortem diagnostic: The total size of the 15 globals exported is 72.55 MiB. The three largest globals are ‘...furrr_chunk_args’ (71.76 MiB of class ‘list’), ‘Models_WS’ (374.44 KiB of class ‘list’) and ‘predictMaxEnt’ (331.60 KiB of class ‘function’)

My code is simple. I am using future_map function from the furrr package to mutate a new column. However, the data is huge and consists of ~1M observations. Nevertheless, the same script worked nicely using the RStudio server on the HPC (same specifications), but now I need to use the same script over different datasets and therefore need to batch the jobs using different R runs .

Cheers,
Ahmed

[orozcoae89]

Well, I think that you could improve your code, because this error is that you have a excedence in the globals fixed by default. More details: https://www.rdocumentation.org/packages/future/versions/1.24.0/topics/future.options. Prove this solution:
options(future.globals.maxSize = ??)

[HenrikBengtsson]

this error is that you have a excedence in the globals fixed by default. ... Prove this solution: options(future.globals.maxSize = ??)

This is actually not the case. The reason is:

The reason reported was ‘error writing to connection’.

The latter part of the error message:

Post-mortem diagnostic: The total size of the 15 globals exported is 72.55 MiB. The three 
largest globals are ‘...furrr_chunk_args’ (71.76 MiB of class ‘list’), ‘Models_WS’ (374.44 KiB
of class ‘list’) and ‘predictMaxEnt’ (331.60 KiB of class ‘function’)

is just giving you some additional information that might be helpful for troubleshooting, but it's not necessarily the reason for the problem. I decided to add "Post-mortem diagnostic" information to the error messages, because it sometimes helps me identify the problem, or rule out certain problems, when users reach out for help. For example, note that in this case the total size is only 72.55 MiB, so it's small and unlikely to cause issues. The default limit for future.globals.maxSize is 500 MiB. Sometimes this information show huge objects, or objects with non-exportable objects, which can be a clue.

[HenrikBengtsson]

(Reposting as a separate answer)

This has most likely nothing to do with furrr or even future. The fact that it cannot even launch a basic test future suggests you have a broken R setup. To me the warnings:

During startup - Warning messages:
1: package ‘utils’ in options("defaultPackages") was not found
2: package ‘stats’ in options("defaultPackages") was not found

look suspicious. What does

$ Rscript -e ".libPaths()"
$ Rscript --vanilla -e ".libPaths()"

output on that machine?

Also, do you have an .Rprofile file? What does that contain?

BTW, please use 'Code Block' here on GitHub to format your code you paste. It makes it much easier to read.

[elgabbas]

Thanks @HenrikBengtsson for your feedback. You were correct! It seems that there was a problem while installing R through conda (although R worked nicely with other packages).
I made a fresh installation of R (also via conda), and then was able to run future/furrrr nicely (although I needed to reduce the number of workers to make it run).

Cheers,
Ahmed

[HenrikBengtsson]

Good to hear.

Yes, it's not uncommon to see conda messing up things for HPC users - I'd say it's the most cause of issues I see people on HPC run into. Not only for R, but for lots of other things too. I know it's convenient, but it causes lots of head aches, so I'd stay away from it unless you truly have to use it.

[jakeberv]

I'm getting a similar error on a linux hpc machine. I'm using future_lapply inside some custom functions, which all work fine on an m1 max macbook pro.

On linux hpc:

Error: MultisessionFuture (future_lapply-2) failed to call gassign() on cluster RichSOCKnode #2 (PID 852085 on localhost ‘localhost’). The reason reported was ‘error writing to connection’. Post-mortem diagnostic: A process with this PID exists, which suggests that the localhost worker is still alive. The total size of the 11 globals exported is 5.18 MiB. The three largest globals are ‘...future.elements_ii’ (4.61 MiB of class ‘list’), ‘trait_data’ (382.78 KiB of class ‘numeric’) and ‘fitMvglsAndExtractGIC.formula’ (116.46 KiB of class ‘function’)

tguser@theorygroup20:/JAKE/shiftr$ Rscript -e ".libPaths()"
[1] "/home/tguser/R/x86_64-pc-linux-gnu-library/4.3"
[2] "/usr/local/lib/R/site-library"
[3] "/usr/lib/R/site-library"
[4] "/usr/lib/R/library"
tguser@theorygroup20:/JAKE/shiftr$ Rscript --vanilla -e ".libPaths()"
[1] "/home/tguser/R/x86_64-pc-linux-gnu-library/4.3"
[2] "/usr/local/lib/R/site-library"
[3] "/usr/lib/R/site-library"
[4] "/usr/lib/R/library"

sessionInfo()
R version 4.3.3 (2024-02-29)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.4 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/atlas/libblas.so.3.10.3
LAPACK: /usr/lib/x86_64-linux-gnu/atlas/liblapack.so.3.10.3; LAPACK version 3.10.0

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

time zone: Etc/UTC
tzcode source: system (glibc)

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] future.apply_1.11.2-9000 future_1.33.2-9000 mvMORPH_1.1.9
[4] subplex_1.8 corpcor_1.6.10 phytools_2.1-1
[7] maps_3.4.2 ape_5.7-1

loaded via a namespace (and not attached):
[1] Matrix_1.6-5 expm_0.999-9 compiler_4.3.3
[4] Rcpp_1.0.12 parallel_4.3.3 globals_0.16.3
[7] optimParallel_1.0-2 lattice_0.22-5 coda_0.19-4.1
[10] pbmcapply_1.5.1 generics_0.1.3 igraph_2.0.3
[13] iterators_1.0.14 MASS_7.3-60 dotCall64_1.1-1
[16] rlang_1.1.3 fastmatch_1.1-4 quadprog_1.5-8
[19] doParallel_1.0.17 cli_3.6.2 magrittr_2.0.3
[22] phangorn_2.11.1 combinat_0.0-8 digest_0.6.35
[25] foreach_1.5.2 grid_4.3.3 spam_2.10-0
[28] lifecycle_1.0.4 nlme_3.1-163 clusterGeneration_1.3.8
[31] mnormt_2.1.1 scatterplot3d_0.3-44 listenv_0.9.1
[34] numDeriv_2016.8-1.1 codetools_0.2-19 glassoFast_1.0.1
[37] parallelly_1.37.1 tools_4.3.3 pkgconfig_2.0.3

No ~/.Rprofile on this machine.

tried with
options(future.globals.onReference = "error")

Got this error
Error: MultisessionFuture (future_lapply-15) failed to call gassign() on cluster RichSOCKnode #15 (PID 894886 on localhost ‘localhost’). The reason reported was ‘error writing to connection’. Post-mortem diagnostic: A process with this PID exists, which suggests that the localhost worker is still alive. The total size of the 11 globals exported is 5.68 MiB. The three largest globals are ‘...future.elements_ii’ (5.19 MiB of class ‘list’), ‘trait_data’ (382.78 KiB of class ‘numeric’) and ‘...future.FUN’ (47.38 KiB of class ‘function’)

R is installed with apt-get and updated to the latest version. Getting the same error on two machines with the same hardware configuration.

Any idea?

Thanks,
Jacob