Move from rstan to cmdstanr

[danielinteractive]

I thought maybe this could be a good place to jot down pros/cons/risks/benefits of a potential move from rstan to cmdstanr for the rbmi package.

[danielinteractive]

Here are my current thoughts about this potential move:

• Motivation:
  • CRAN check results have continuously shown problems on the Linux Fedora/clang machine, and currently also for Debian/clang: https://cran.r-project.org/web/checks/check_results_rbmi.html (see below for the relevant part)
  • The idea is that these problems would disappear when using cmdstanr - essentially they will for sure disappear as long as cmdstanr is not on CRAN, because then we cannot even compile anything on CRAN anyway
    • Then the question might be whether we should just skip the problematic tests on CRAN machines to get the same result with less effort?
• Pros:
  • cmdstanr is stable and already used by many other packages (e.g. brms)
  • can use later versions of the Stan library
• Cons:
  • tests using Stan cannot be run on CRAN, because cmdstanr is not available on CRAN (and won't be available with 99% probability in the future, although it still seems to be "a goal", see here)
  • users have a harder time to install cmdstanr and cmdstan on their machines
  • cannot use precompilation and provide that via CRAN, but so far this has not been used anyway in rbmi, so not a real con

@gowerc @gravesti @bailliem @luwidmer @tobiasmuetze - what do you think?

---

Debian/clang:

Running the tests in ‘tests/testthat.R’ failed.
Complete output:
  > library(testthat)
  > library(rbmi)
  > 
  > test_check("rbmi")
  0,248,Js_of_ocaml__Js.Error,18,TypeError: not a function
      at c$ (<input>:5378:79)
      at dIb (<input>:25505:6)
  
  mmrm() registered as emmeans extension
  0,248,Js_of_ocaml__Js.Error,18,TypeError: not a function
      at c$ (<input>:5378:79)
      at dIb (<input>:25505:6)
  
  0,248,Js_of_ocaml__Js.Error,18,TypeError: not a function
      at c$ (<input>:5378:79)
      at dIb (<input>:25505:6)
  
  0,248,Js_of_ocaml__Js.Error,18,TypeError: not a function
      at c$ (<input>:5378:79)
      at dIb (<input>:25505:6)
  
  [ FAIL 4 | WARN 0 | SKIP 15 | PASS 1261 ]
  
  ══ Skipped tests (15) ══════════════════════════════════════════════════════════
  • On CRAN (1): 'test-mmrm.R:337:5'
  • is_full_test() is not TRUE (14): 'test-draws.R:115:5',
    'test-fullusage.R:43:5', 'test-fullusage.R:117:5', 'test-fullusage.R:209:5',
    'test-fullusage.R:300:5', 'test-fullusage.R:431:5', 'test-fullusage.R:520:5',
    'test-fullusage.R:675:5', 'test-mcmc.R:327:5', 'test-mcmc.R:504:5',
    'test-mcmc.R:635:5', 'test-parallel.R:8:5', 'test-reproducibility.R:9:5',
    'test-reproducibility.R:135:5'
  
  ══ Failed tests ════════════════════════════════════════════════════════════════
  ── Error ('test-draws.R:144:9'): bayes ─────────────────────────────────────────
  Error in `model_cppcode$errors`: $ operator is invalid for atomic vectors
  Backtrace:
      ▆
   1. ├─base::suppressWarnings(...) at test-draws.R:143:5
   2. │ └─base::withCallingHandlers(...)
   3. ├─rbmi::draws(d$dat, d$dat_ice, d$vars, meth, quiet = TRUE) at test-draws.R:144:9
   4. └─rbmi:::draws.bayes(d$dat, d$dat_ice, d$vars, meth, quiet = TRUE)
   5.   └─rbmi:::fit_mcmc(...)
   6.     └─rbmi:::get_stan_model()
   7.       └─rstan::stan_model(file = model_file, auto_write = TRUE, model_name = "rbmi_mmrm")
   8.         └─rstan::stanc(...)

Fedora/clang:

══ Failed tests ════════════════════════════════════════════════════════════════
  ── Error ('test-draws.R:144:9'): bayes ─────────────────────────────────────────
  Error in `fit_mcmc(designmat = model_df_scaled[, -1, drop = FALSE], outcome = model_df_scaled[, 
      1, drop = TRUE], group = data2[[vars$group]], visit = data2[[vars$visit]], 
      subjid = data2[[vars$subjid]], method = method, quiet = quiet)`: cannot allocate vector of size 1042968.4 Gb
  Backtrace:
      ▆
   1. ├─base::suppressWarnings(...) at test-draws.R:143:5
   2. │ └─base::withCallingHandlers(...)
   3. ├─rbmi::draws(d$dat, d$dat_ice, d$vars, meth, quiet = TRUE) at test-draws.R:144:9
   4. └─rbmi:::draws.bayes(d$dat, d$dat_ice, d$vars, meth, quiet = TRUE)

[gravesti]

Just to note that the error message from debian-clang is still from 1.4.0, so I expect that the 0,248,Js_of_ocaml__Js.Error,18,TypeError: not a function will be resolved.

Outstanding is the fedora-clang error cannot allocate vector of size 1042968.4 Gb which we have seen before on that machine but it resolved itself without any change from us.

I'm not sure if it would be appreciated to just skip the tests.

[danielinteractive]

I just thought about this again and checked the CRAN check results, and "siehe da", now we have:

So fedora-clang has resolved itself already, while debian-clang is still waiting for its refresh.

Therefore I would say the need to move to cmdstanr, at least for the CRAN check motivation, is not that pressing.

Even if CRAN checks would be consistently an issue with say fedora-clang or any of the exotic Linux configurations on CRAN, I am not sure we should jump to cmdstanr:

• It should not be for the side effect of just not having any CRAN checks with Stan compilation any longer, because then we could just directly skip certain tests now.
• If there was a way to reproduce the exotic Linux problems e.g. via special Docker images etc., and then we could find there that cmdstanr's Stan version does improve the situation, then it would be a good reason to move to cmdstanr.

[gravesti]

This is a nice surprise and I agree it does remove the immediate motivation.

[gowerc]

I mean I won't lie, side stepping the CRAN issues was a big motivator for making this change, outside of that though there are a few reasons that I think it is still worth considering:

cmdstanr:

• provides significantly faster compile times
• Is less prone to crashes / needing to recompile the model for safety
• Is the recommend tool to use by the Stan team (I get the impression they are focussing development effort on cmdstanr now and not rstan)

Regarding the CRAN issues:

Reproducing the CRAN machines isn't really feasible, many of the installation steps aren't documented and worse some of the documented steps are inaccurate. Building the non-standard compiler from source takes ~6-7 hours and most of the time isn't able to replicate the errors anyway. The amount of time we've sunk trying to work out CRAN problems is significant and I'd really prefer to take steps to avoid this happening again in the future personally. (Though I appreciate I am no longer technically the maintainer so I guess Isaac gets the final say here now 😓 )