ESM-Intake Catalogue Error

[shreyatrivedi26]

I recently moved to work on my school's HPC and encountered the following problem with accessing CMIP6 catalogues:

import intake

col = intake.open_esm_datastore("https://raw.githubusercontent.com/NCAR/intake-esm-datastore/master/catalogs/pangeo-cmip6.json")

I get the following error:

---------------------------------------------------------------------------
gaierror                                  Traceback (most recent call last)
~/.local/lib/python3.9/site-packages/aiohttp/connector.py in _create_direct_connection(self, req, traces, timeout, client_error)
   1151             #  across all connections.
-> 1152             hosts = await asyncio.shield(host_resolved)
   1153         except asyncio.CancelledError:

~/.local/lib/python3.9/site-packages/aiohttp/connector.py in _resolve_host(self, host, port, traces)
    873 
--> 874                 addrs = await self._resolver.resolve(host, port, family=self._family)
    875                 if traces:

~/.local/lib/python3.9/site-packages/aiohttp/resolver.py in resolve(self, hostname, port, family)
     32     ) -> List[Dict[str, Any]]:
---> 33         infos = await self._loop.getaddrinfo(
     34             hostname,

/opt/ohpc/pub/eb-gcc103/software/Python/3.9.5-GCCcore-10.3.0/lib/python3.9/asyncio/base_events.py in getaddrinfo(self, host, port, family, type, proto, flags)
    855 
--> 856         return await self.run_in_executor(
    857             None, getaddr_func, host, port, family, type, proto, flags)

/opt/ohpc/pub/eb-gcc103/software/Python/3.9.5-GCCcore-10.3.0/lib/python3.9/concurrent/futures/thread.py in run(self)
     51         try:
---> 52             result = self.fn(*self.args, **self.kwargs)
     53         except BaseException as exc:

/opt/ohpc/pub/eb-gcc103/software/Python/3.9.5-GCCcore-10.3.0/lib/python3.9/socket.py in getaddrinfo(host, port, family, type, proto, flags)
    952     addrlist = []
--> 953     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    954         af, socktype, proto, canonname, sa = res

gaierror: [Errno -2] Name or service not known

The above exception was the direct cause of the following exception:

ClientConnectorError                      Traceback (most recent call last)
~/.local/lib/python3.9/site-packages/fsspec/implementations/http.py in _info(self, url, **kwargs)
    416                 info.update(
--> 417                     await _file_info(
    418                         self.encode_url(url),

~/.local/lib/python3.9/site-packages/fsspec/implementations/http.py in _file_info(url, session, size_policy, **kwargs)
    832     elif size_policy == "get":
--> 833         r = await session.get(url, allow_redirects=ar, **kwargs)
    834     else:

~/.local/lib/python3.9/site-packages/aiohttp/client.py in _request(self, method, str_or_url, params, data, json, cookies, headers, skip_auto_headers, auth, allow_redirects, max_redirects, compress, chunked, expect100, raise_for_status, read_until_eof, proxy, proxy_auth, timeout, verify_ssl, fingerprint, ssl_context, ssl, proxy_headers, trace_request_ctx, read_bufsize)
    535                             assert self._connector is not None
--> 536                             conn = await self._connector.connect(
    537                                 req, traces=traces, timeout=real_timeout

~/.local/lib/python3.9/site-packages/aiohttp/connector.py in connect(self, req, traces, timeout)
    539             try:
--> 540                 proto = await self._create_connection(req, traces, timeout)
    541                 if self._closed:

~/.local/lib/python3.9/site-packages/aiohttp/connector.py in _create_connection(self, req, traces, timeout)
    900         else:
--> 901             _, proto = await self._create_direct_connection(req, traces, timeout)
    902 

~/.local/lib/python3.9/site-packages/aiohttp/connector.py in _create_direct_connection(self, req, traces, timeout, client_error)
   1165             # it is problem of resolving proxy ip itself
-> 1166             raise ClientConnectorError(req.connection_key, exc) from exc
   1167 

ClientConnectorError: Cannot connect to host raw.githubusercontent.com:443 ssl:default [Name or service not known]

The above exception was the direct cause of the following exception:

FileNotFoundError                         Traceback (most recent call last)
/tmp/ipykernel_120139/2362624247.py in <module>
----> 1 dataframe = intake.open_esm_datastore("https://raw.githubusercontent.com/NCAR/intake-esm-datastore/master/catalogs/pangeo-cmip6.json")
      2 # cat_url = "https://storage.googleapis.com/cmip6/pangeo-cmip6.json" #not a local but object storage
      3 # col = intake.open_esm_datastore(cat_url)
      4 # # col

~/.local/lib/python3.9/site-packages/intake_esm/core.py in __init__(self, obj, progressbar, sep, registry, read_csv_kwargs, columns_with_iterables, storage_options, **intake_kwargs)
    105             self.esmcat = ESMCatalogModel.from_dict(obj)
    106         else:
--> 107             self.esmcat = ESMCatalogModel.load(
    108                 obj, storage_options=self.storage_options, read_csv_kwargs=read_csv_kwargs
    109             )

~/.local/lib/python3.9/site-packages/intake_esm/cat.py in load(cls, json_file, storage_options, read_csv_kwargs)
    251         _mapper = fsspec.get_mapper(json_file, **storage_options)
    252 
--> 253         with fsspec.open(json_file, **storage_options) as fobj:
    254             data = json.loads(fobj.read())
    255             if 'last_updated' not in data:

~/.local/lib/python3.9/site-packages/fsspec/core.py in __enter__(self)
    100         mode = self.mode.replace("t", "").replace("b", "") + "b"
    101 
--> 102         f = self.fs.open(self.path, mode=mode)
    103 
    104         self.fobjects = [f]

~/.local/lib/python3.9/site-packages/fsspec/spec.py in open(self, path, mode, block_size, cache_options, compression, **kwargs)
   1197         else:
   1198             ac = kwargs.pop("autocommit", not self._intrans)
-> 1199             f = self._open(
   1200                 path,
   1201                 mode=mode,

~/.local/lib/python3.9/site-packages/fsspec/implementations/http.py in _open(self, path, mode, block_size, autocommit, cache_type, cache_options, size, **kwargs)
    354         kw["asynchronous"] = self.asynchronous
    355         kw.update(kwargs)
--> 356         size = size or self.info(path, **kwargs)["size"]
    357         session = sync(self.loop, self.set_session)
    358         if block_size and size:

~/.local/lib/python3.9/site-packages/fsspec/asyn.py in wrapper(*args, **kwargs)
    113     def wrapper(*args, **kwargs):
    114         self = obj or args[0]
--> 115         return sync(self.loop, func, *args, **kwargs)
    116 
    117     return wrapper

~/.local/lib/python3.9/site-packages/fsspec/asyn.py in sync(loop, func, timeout, *args, **kwargs)
     98         raise FSTimeoutError from return_result
     99     elif isinstance(return_result, BaseException):
--> 100         raise return_result
    101     else:
    102         return return_result

~/.local/lib/python3.9/site-packages/fsspec/asyn.py in _runner(event, coro, result, timeout)
     53         coro = asyncio.wait_for(coro, timeout=timeout)
     54     try:
---> 55         result[0] = await coro
     56     except Exception as ex:
     57         result[0] = ex

~/.local/lib/python3.9/site-packages/fsspec/implementations/http.py in _info(self, url, **kwargs)
    428                 if policy == "get":
    429                     # If get failed, then raise a FileNotFoundError
--> 430                     raise FileNotFoundError(url) from exc
    431                 logger.debug(str(exc))
    432 

FileNotFoundError: https://raw.githubusercontent.com/NCAR/intake-esm-datastore/master/catalogs/pangeo-cmip6.json

Based on one of the previous related error threads, I am adding the following information as well:

import intake_esm
print(intake_esm.__version__)

2023.4.20.post6

import sys
print(sys.executable)

/opt/ohpc/pub/eb-gcc103/software/Python/3.9.5-GCCcore-10.3.0/bin/python

print(fsspec.__version__)
2023.5.0

[andersy005]

@shreyatrivedi26, can you confirm that your HPC nodes allow public internet access? i'm asking because of this section in the error message:

/opt/ohpc/pub/eb-gcc103/software/Python/3.9.5-GCCcore-10.3.0/lib/python3.9/concurrent/futures/thread.py in run(self)
     51         try:
---> 52             result = self.fn(*self.args, **self.kwargs)
     53         except BaseException as exc:

/opt/ohpc/pub/eb-gcc103/software/Python/3.9.5-GCCcore-10.3.0/lib/python3.9/socket.py in getaddrinfo(host, port, family, type, proto, flags)
    952     addrlist = []
--> 953     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    954         af, socktype, proto, canonname, sa = res

gaierror: [Errno -2] Name or service not known

This error typically occurs when the DNS resolution fails, i.e., the domain name system (DNS) is unable to resolve the IP address of the server (in this case this would be GitHub) you are trying to access

[shreyatrivedi26]

Thanks for the reply!
I inquired and its exactly what you pointed it out. I will get it resolved now!