Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

We should implement a thredds crawler ingestor command in geospaas.nansat_ingestor #40

Closed
mortenwh opened this issue Nov 7, 2018 · 6 comments
Closed

We should implement a thredds crawler ingestor command in geospaas.nansat_ingestor #40

mortenwh opened this issue Nov 7, 2018 · 6 comments
Assignees
@mortenwh
Labels
enhancement

Comments

@mortenwh
Copy link
Contributor

mortenwh commented Nov 7, 2018

No description provided.

mortenwh added a commit that referenced this issue Nov 7, 2018
@mortenwh
#40: initial version of command ingest_thredds_crawl
6893dbb
mortenwh added a commit that referenced this issue Nov 7, 2018
@mortenwh
#40: thredds crawler is crawling - remains to see if it works correct…
3d90759 
…ly. This is time consuming...
mortenwh added a commit that referenced this issue Nov 7, 2018
@mortenwh
#40: ingestor using thredds crawler seems to work and adds datasets t…
df6490d 
…o the db - give as specific as possible urls (e.g., to the date of interest), otherwise it will take very long time
@mortenwh
Copy link
Contributor Author

mortenwh commented Nov 7, 2018

Tested on Sentinel-2 data from the Norwegian ground segment, e.g.:

./manage.py ingest_thredds_crawl http://nbstds.met.no/thredds/catalog/NBS/S2B/2018/07/catalog.html --date 2018/07/17

or

./manage.py ingest_thredds_crawl http://nbstds.met.no/thredds/catalog/NBS/S2B/2018/07/17/catalog.html --filename S2B_MSIL1C_20180717T095029_N0206_R079_T34VFM_20180717T115424.nc

The latter is much faster...

mortenwh added a commit that referenced this issue Nov 8, 2018
@mortenwh
#40: corrected print statement that was following python2.7 syntax
f1a07bc
@mortenwh mortenwh self-assigned this Nov 10, 2018
@mortenwh mortenwh mentioned this issue Nov 12, 2018
Merged
mortenwh added a commit that referenced this issue Nov 22, 2018
@mortenwh
#40: catch AttributeError caused by non-standard(?)/erroneous(?) netcdf
d4ea0e5
mortenwh added a commit that referenced this issue Jan 1, 2019
@mortenwh
#40: while installing geospaas on a postgis server it turned out that…
fabc4ea 
… many model fields of the vocabularies app had too few characters, i.e., max_length was too low. Corrected the max_length numbers and added two automatically created migrations (one in viewer and one in catalog).
mortenwh added a commit that referenced this issue Feb 7, 2019
@mortenwh
#40: deleted outdated migrations
c63769a
@akorosov
Copy link
Member

akorosov commented Feb 13, 2019 •
edited
Loading 

./manage.py ingest_thredds_crawl http://nbstds.met.no/thredds/catalog/NBS/S2B/2018/07/17/catalog.html --filename S2B_MSIL1C_20180717T095029_N0206_R079_T34VFM_20180717T115424.nc

This one didn't work for me. It find nothing:

(base) root@0294d0ad74aa:/src# project/manage.py ingest_thredds_crawl http://nbstds.met.no/thredds/catalog/NBS/S2B/2018/07/17/catalog.html --filename S2B_MSIL1C_20180717T095029_N0206_R079_T34VFM_20180717T115424.nc
2019-02-13 13:55:24,061 - [INFO] Crawling: http://nbstds.met.no/thredds/catalog/NBS/S2B/2018/07/17/catalog.html
2019-02-13 13:55:48,434 - [INFO] Ignoring dataset based on 'selects'.  ID: nbs/S2B/2018/07/17/S2B_MSIL1C_20180717T144749_N0206_R082_T35XML_20180717T195515.nc

..... many filenames come here .....

2019-02-13 13:55:48,483 - [INFO] Ignoring dataset based on 'selects'.  ID: nbs/S2B/2018/07/17/S2B_MSIL1C_20180717T095029_N0206_R079_T34VCH_20180717T133928.nc
HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140212812998400:
  #000: H5F.c line 509 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1400 in H5F__open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5Fint.c line 1548 in H5F_open(): unable to open file: name = 'http://nbstds.met.no/thredds/dodsC/NBS/S2B/2018/07/17/S2B_MSIL1C_20180717T095029_N0206_R079_T34VFM_20180717T115424.nc', tent_flags = 0
    major: File accessibilty
    minor: Unable to open file
  #003: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
Successfully added metadata of 0 datasets

@akorosov
Copy link
Member

The same with another option. After some file which look OK:

2019-02-13 14:10:43,343 - [INFO] Ignoring dataset based on 'selects'.  ID: nbs/S2B/2018/07/01/S2B_MSIL1C_20180701T093039_N0206_R136_T35VMH_20180701T113337.nc
2019-02-13 14:10:43,344 - [INFO] Ignoring dataset based on 'selects'.  ID: nbs/S2B/2018/07/01/S2B_MSIL1C_20180701T093039_N0206_R136_T35VMF_20180701T113337.nc
2019-02-13 14:10:43,344 - [INFO] Ignoring dataset based on 'selects'.  ID: nbs/S2B/2018/07/01/S2B_MSIL1C_20180701T093039_N0206_R136_T35VME_20180701T113337.nc
2019-02-13 14:10:43,344 - [INFO] Ignoring dataset based on 'selects'.  ID: nbs/S2B/2018/07/01/S2B_MSIL1C_20180701T093039_N0206_R136_T35VLE_20180701T113337.nc
2019-02-13 14:10:43,344 - [INFO] Ignoring dataset based on 'selects'.  ID: nbs/S2B/2018/07/01/S2B_MSIL1C_20180701T093039_N0206_R136_T35VLD_20180701T113337.nc

Come many errors:

HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140197406467840:
  #000: H5F.c line 509 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1400 in H5F__open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5Fint.c line 1548 in H5F_open(): unable to open file: name = 'http://nbstds.met.no/thredds/dodsC/NBS/S2B/2018/07/17/S2B_MSIL1C_20180717T144749_N0206_R082_T35XML_20180717T195515.nc', tent_flags = 0
    major: File accessibilty
    minor: Unable to open file
  #003: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140197406467840:
  #000: H5F.c line 509 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1400 in H5F__open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5Fint.c line 1548 in H5F_open(): unable to open file: name = 'http://nbstds.met.no/thredds/dodsC/NBS/S2B/2018/07/17/S2B_MSIL1C_20180717T144749_N0206_R082_T35XMK_20180717T195515.nc', tent_flags = 0
    major: File accessibilty
    minor: Unable to open file
  #003: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140197406467840:
  #000: H5F.c line 509 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1400 in H5F__open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5Fint.c line 1548 in H5F_open(): unable to open file: name = 'http://nbstds.met.no/thredds/dodsC/NBS/S2B/2018/07/17/S2B_MSIL1C_20180717T144749_N0206_R082_T33XXL_20180717T195515.nc', tent_flags = 0
    major: File accessibilty
    minor: Unable to open file
  #003: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 140197406467840:
  #000: H5F.c line 509 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1400 in H5F__open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5Fint.c line 1548 in H5F_open(): unable to open file: name = 'http://nbstds.met.no/thredds/dodsC/NBS/S2B/2018/07/17/S2B_MSIL1C_20180717T144749_N0206_R082_T33XXK_20180717T195515.nc', tent_flags = 0
    major: File accessibilty
    minor: Unable to open file
  #003: H5FD.c line 734 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object

Maybe a timeout should be added before ds, cr = NansatDataset.objects.get_or_create(url) to not crash the opendap server ?

mortenwh added a commit that referenced this issue Feb 14, 2019
@mortenwh
#40: removed debugging lines
64547e9
mortenwh added a commit that referenced this issue Feb 14, 2019
@mortenwh
#40: working on tests but for some reason the mocking of validate_uri…
8bd16c7 
… does not work
mortenwh added a commit that referenced this issue Feb 15, 2019
@mortenwh
#40: tests for crawl function in nansat_ingestor/management/commands/…
ee52725 
…ingest_thredds_crawl.py and some changes in the function
mortenwh added a commit that referenced this issue Feb 15, 2019
@mortenwh
#40: test command ingest_thredds_crawl
3896548
mortenwh added a commit that referenced this issue Feb 15, 2019
@mortenwh
#40: added installation of thredds_crawler to travis CI
26ca613
mortenwh added a commit that referenced this issue Feb 23, 2019
@mortenwh
#40: removed old duplicate migrations
de3bbc0
mortenwh added a commit that referenced this issue Feb 23, 2019
@mortenwh
#40: after merge with issue #24 - do not raise error in the crawler
6dbbf21
mortenwh added a commit that referenced this issue Feb 24, 2019
@mortenwh
#40 and #24: I discovered that a valid uri to an opendap stream retur…
8b5ed10 
…ned html 400 error although netCDF4.Dataset works fine. If request.status!=200, I therefore try to open with netCDF4.Dataset. If that works, no error is raised..
mortenwh added a commit that referenced this issue May 9, 2019
@mortenwh
#40: update test to catching OSError instead of ConnectionError
c161fc4
mortenwh added a commit that referenced this issue May 9, 2019
@mortenwh
#40: changed from conda-forge to default
e1e484c
mortenwh added a commit that referenced this issue May 9, 2019
@mortenwh
#40: install nansat and pythesint from conda-forge
7559dcd
mortenwh added a commit that referenced this issue May 9, 2019
@mortenwh
#40: install nansat and pythesint from conda-forge - and confirm chan…
6a35693 
…ges with --yes
mortenwh added a commit that referenced this issue May 9, 2019
@mortenwh
#40: numpy issue at travis installation. Try solving by 'conda update…
1d7572f 
… --all'
mortenwh added a commit that referenced this issue May 9, 2019
@mortenwh
#40: numpy issue at travis installation. Try solving by 'conda update…
d8d8daa 
… --all --yes'
mortenwh added a commit that referenced this issue May 10, 2019
@mortenwh
#40: install libspatialite on travis
d10f924
mortenwh added a commit that referenced this issue May 10, 2019
@mortenwh
#40: django etc from conda-forge
5cd49b7
mortenwh added a commit that referenced this issue May 10, 2019
@mortenwh
Revert "#40: django etc from conda-forge"
dceb9e4 
This reverts commit 5cd49b7.
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
Revert "#40: django etc from conda-forge"
057f442 
This reverts commit 5cd49b7.
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
Revert "#40: install libspatialite on travis"
79bbaa9 
This reverts commit d10f924.
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
Revert "#40: numpy issue at travis installation. Try solving by 'cond…
ee1fd52 
…a update --all --yes'"

This reverts commit d8d8daa.
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
Revert "#40: numpy issue at travis installation. Try solving by 'cond…
357ff61 
…a update --all'"

This reverts commit 1d7572f.
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
Revert "#40: install nansat and pythesint from conda-forge - and conf…
2a89fbe 
…irm changes with --yes"

This reverts commit 6a35693.
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
Revert "#40: install nansat and pythesint from conda-forge"
3560d65 
This reverts commit 7559dcd.
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
Revert "#40: changed from conda-forge to default"
a0eb014 
This reverts commit e1e484c.
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: Try to install from packages listed in packages.json file. This …
5b62c39 
…file contains the installed packages in a working version of the vm. Hopefully this solves the broken installation at travis...
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: forgot the '--yes'...
e446d33
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
Revert "#40: forgot the '--yes'..."
bca5485 
This reverts commit a7b83c7.
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
Revert "#40: Try to install from packages listed in packages.json fil…
7ac1d43 
…e. This file contains the installed packages in a working version of the vm. Hopefully this solves the broken installation at travis..."

This reverts commit 81c60fb.
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: enabled ssh and X forwarding
1bd9b3c
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: adjust uri validation to handle errors with fillvalue (#fillmism…
e65d7c5 
…atch)
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: Added some string constants to use for new fields in DatasetURI
213e04e
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: added name and service fields to DatasetURI model, and made migr…
3c17ddd 
…ations
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: set DatasetURI.service to OPENDAP_SERVICE or LOCAL_FILE_SERVICE …
026bd2f 
…(and similar for DatasetURI.name) when a new dataset is ingested.
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: added all services for a given Dataset as DatasetURIs, and made …
265fb38 
…tests for the crawl function
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: don't validate uri when saving DatasetURI, try/except clause in …
af2f2e7 
…crawl command, and changed a docstring in nansat_ingestor/managers.py
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: removed migration to update DatasetURI with name and service
97ce588
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: Added service strings in catalog.managers and updated docstring …
ec6ff89 
…in ingest_thredds_crawl command
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: updated tests to accomodate change in ingest_thredds_crawl command
d0130fe
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: projlib is not found on travis - adding some debugging
e09b076
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: travis installation - do not use conda-forge
0c5b129
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: travis installation - use conda-forge for nansat and pythesint
0656429
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: try to update all paclages after installation of nansat
383be5d
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: undid last change
f0abc8c
mortenwh added a commit that referenced this issue Jul 25, 2019
@mortenwh
#40: added a test to increase coverage, and updated the function to w…
90eb450 
…ork correctly..
@ninsbl
Copy link

ninsbl commented Oct 1, 2021

Hei, I have been using thredds_crawler towards NBS too, but for me that regularly causes 502 or 504 errors. Did you notice that too? I tried to address thatin: ioos/thredds_crawler#29

But I am not sure if the library is actively maintained.... I saw that @akorosov contributed to it earlier. Do you know more or an alternative to thredds_crawler if that is no longer actively maintained (there is also anonther un-commented PR from may)? Maybe that is a dead horse?

@mortenwh
Copy link
Contributor Author

mortenwh commented Oct 26, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mortenwh mortenwh

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mortenwh @ninsbl @akorosov