Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Potentially investigate why dv_data gets built during 2_process #46

Open
lindsayplatt opened this issue Nov 7, 2018 · 1 comment
Open

Comments

@lindsayplatt
Copy link
Contributor

lindsayplatt commented Nov 7, 2018

I was experiencing some strange rebuilding behavior.

When 1_fetch/out/dv_data.rds.ind is rebuilt, text is printed to the console to share progress. When I ran scipiper::scmake(remake_file = '2_process.yml') and 1_fetch/out/dv_data.rds.ind was rebuilt, the same text about progress for fetching dv data would be repeated during the 2_process/out/dv_stats.rds.ind step. I was able to pinpoint that the re-downloading of dv data was happening at the line of code that read in the dv data with dv_data <- readRDS(scipiper::sc_retrieve(dv_data_ind, remake_file = '1_fetch.yml')) in 2_process/src/process_dv_stats.R.

This does not happen when 2_process/out/dv_stats.rds.ind is built and remake did not need to rebuild 1_fetch/out/dv_data.rds.ind. @aappling-usgs pointed to the potential for a known scipiper issue (??) to be the cause, but more investigation is needed.

For now, this is not a priority. It is annoying because the same data download is happening twice but we need to move on for now and might return when that step would cause more of an issue as we expand the time period for the animation (see #36).

@lindsayplatt
Copy link
Contributor Author

Running these separately did not result in the rebuild issue:

scipiper::scmake('1_fetch/out/dv_data.rds.ind', remake_file = '1_fetch.yml')
scipiper::scmake('2_process/out/dv_stats.rds.ind', remake_file = '2_process.yml')

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant