Using Datadeps

[willtebbutt]

I also wonder whether DataDeps.jl wouldn't make more sense for this project than RemoteFiles.jl because the data you're working with is largely static. Is this something that has been considered already?

[Alexander-Barth]

I started to use RemoteFiles.jlbecause I was already familiar with it from another project, and it worked quite well for me. The maintainer has been quite reactive. But I am open to change. Unfortunately, I have no experience with DataDeps.jl but I would be interested to hear if you have any experience with DataDeps.jl that you can share.

Let me know if you want to join JuliaGeo, so that you make such contributions more easily.

[willtebbutt]

I've got a bit of experience using DataDeps.jl to set up GHCNData.jl -- it was a very straightforward experience and @oxinabox (the core maintainer) is also very responsive. The core advantage is that it downloads the data once, and only downloads again if the data disappears. My brief experience with RemoteFiles is that it re-downloads regularly, maybe I'm mistaken though?

Ironically I suspect I might actually have been better off using RemoteFiles.jl for that particular package because the GHCN daily data is updated regularly.

Happy to join JuliaGeo!

[oxinabox]

Yes, one of DataDeps' core assumptions about data is that it never changes.
RemotesFiles' core assumption seems to be that it changes often.

If ever I did the DataDepsPaths plan it would probably support data changing.
(Check HTTP header and see if that has changed and if so re-download).
I am never going to have time to do that plan though

Using Artifacts standard library is another option.
(Much newer than either of the aforementioned)
Though data is a secondary concern, and it kind of assumes you self-host your data and can make sure it is a gzipped tarball.
It also assumes data never changes.

[Alexander-Barth]

As far as I know, RemoteFiles.jl allows you to set :never as updating frequency (which is also the default). Maybe I miss something?

In Earth Science (and my field oceanography) some datasets are updated continously. Such dataset are currently not used in GeoDatasets. @willtebbutt, I am wondering of you found a solution within GHCNData.jl for the case if some data is added (or corrected)?

[oxinabox]

As far as I know, RemoteFiles.jl allows you to set :never as updating frequency (which is also the default).

today I learnt

[willtebbutt]

As far as I know, RemoteFiles.jl allows you to set :never as updating frequency (which is also the default). Maybe I miss something?

Ah, interesting, I missed that.

@willtebbutt, I am wondering of you found a solution within GHCNData.jl for the case if some data is added (or corrected)?

DataDeps won't offer a solution to this unfortuntely (GHCNData is something of a WIP that I've thrown together hastily for a project I'm working on where I don't actually want the data to update, so it has a couple of rough edges, this being one of them). As I say, I should probably switch that particular package over to RemoteFiles 😂

edit: maybe this package is just completely fine using RemoteFiles then. Hmmm....

[Alexander-Barth]

@willtebbutt I just added you as a member. Let me know if you need more rights ...
Maybe I need to configure also something in GeoDatasets.jl to give you direct access (bear with me while I figure this stuff out :-))