Hanging downloads in byFileAOP?

[mbjoseph]

Function

byFileAOP

Describe the bug

The byFileAOP function will hang if a file download hangs.

To Reproduce

I discovered this while trying to get some leaf area index mosaic data over Niwot Ridge:

library(neonUtilities)
byFileAOP("DP3.30012.001", site = "NIWO", check.size = FALSE)

For me, this downloads about 14% of the files, and then hangs.

Expected behavior

I expected all the files to download, rather than getting stuck with a subset of the files downloaded. If one file cannot be downloaded/is unavailable, I would expect a warning or error.

System (please complete the following information):

• OS: [e.g. iOS] Ubuntu
• OS Version: [e.g. 22] 18.04
• R Version: [e.g. 3.5.1] 3.6.2

Additional context

I think this may be an issue that is separate from #69, because there are no errors that arise in the download (hence the infinite while loop of #69 may be irrelevant).

In trying to debug this issue, I think I tracked the problem down to this call to the downloader::download() function, which goes unresponsive for (at least) one of the files: https://github.com/NEONScience/NEON-utilities/blob/master/neonUtilities/R/byFileAOP.R#L139-L141

If that call to download() never returns anything, then the error handling below is irrelevant. I'm not totally sure what the best solution here might be...

> sessionInfo()
R version 3.6.2 (2019-12-12)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] neonUtilities_1.3.3

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.3       rstudioapi_0.10  magrittr_1.5     hms_0.5.3        tidyselect_0.2.5 R6_2.4.1         rlang_0.4.2     
 [8] stringr_1.4.0    httr_1.4.1       dplyr_0.8.3      tools_3.6.2      packrat_0.5.0    gtools_3.8.1     digest_0.6.23   
[15] assertthat_0.2.1 tibble_2.1.3     lifecycle_0.1.0  crayon_1.3.4     purrr_0.3.3      readr_1.3.1      tidyr_1.0.2     
[22] vctrs_0.2.1      curl_4.3         zeallot_0.1.0    glue_1.3.1       gdata_2.18.0     stringi_1.4.5    downloader_0.4  
[29] compiler_3.6.2   pillar_1.4.3     backports_1.1.5  jsonlite_1.6     lubridate_1.7.4  pkgconfig_2.0.3 

[cklunch]

@mbjoseph Thanks for the alert! I get the same behavior, including the halt at 14%, which makes me think it's a specific file that's causing the problem. But the function should be robust to that, at first glance I don't understand why it's hanging there. We'll get back to you soon.

[cklunch]

@mbjoseph I'm no longer able to replicate this, neither is @NateMietk . Can you give it another try and let us know if it's working on your end now?

[mbjoseph]

Same here @cklunch - I get the expected behavior now. Thanks for checking in on this!

[cklunch]

@mbjoseph Glad to hear it's resolved! I'll leave this issue open for now, in case it happens again. My best guess is that it was related to some issues we were having with the caches, but that's just a guess.