Merge branch 'develop'

.github/workflows/test.yml

@@ -87,6 +87,7 @@ jobs:
 
     - name: Publish Unit Test Results
       uses: EnricoMi/publish-unit-test-result-action@v1
+      continue-on-error: true
       with:
         files: artifacts/**/*junit-report.xml
     - name: Publish Coverage Report

CHANGES.rst

@@ -1,6 +1,15 @@
 Release history
 ---------------
 
+2.7.0 (2022-11-29)
+++++++++++++++++++
+
+* Fetch external product types before searching for an unkown product type (:pull:`559`)
+* Handle local assets in :class:`~eodag.plugins.download.http.HTTPDownload` plugin (:pull:`561`)
+* Fetch external product types only for given provider if one is specified (:pull:`557`)
+* Fixed request error handling during :meth:`~eodag.api.core.EODataAccessGateway.search_all` (:pull:`554`)
+* Various minor fixes and improvements (:pull:`555`)(:pull:`558`)(:pull:`562`)
+
 2.6.2 (2022-11-15)
 ++++++++++++++++++
 

CONTRIBUTING.rst

@@ -55,13 +55,13 @@ To run the default test suite in parallel:
 
     tox -p
 
-To only run end-to-end test:
+To only run end-to-end tests:
 
 .. code-block:: bash
 
     tox -- tests/test_end_to_end.py
 
-To run the entire tests (units, integraton and end-to-end):
+To run the entire tests (units, integration and end-to-end):
 
 .. code-block:: bash
 

docs/cli_user_guide.rst

@@ -79,9 +79,9 @@ Then you can start playing with it:
         --all \
         --storage my_search.geojson
 
-The request above searches for `S2_MSI_L1C` product types in a given bounding box, in January 2018. The command fetches internally all
-the products that match these criteria. Without ``--all``, it would only fetch the products found on the first result page.
-It finally saves the results in a GeoJSON file.
+The request above searches for `S2_MSI_L1C` product types in a given bounding box, in January 2018. The command fetches
+internally all the products that match these criteria. Without ``--all``, it would only fetch the products found on the
+first result page. It finally saves the results in a GeoJSON file.
 
 You can pass arguments to a cruncher on the command line by doing this (example with using ``FilterOverlap`` cruncher
 which takes ``minimum_overlap`` as argument):
@@ -92,19 +92,19 @@ which takes ``minimum_overlap`` as argument):
                      --cruncher FilterOverlap \
                      --cruncher-args FilterOverlap minimum_overlap 10
 
-The request above means : "Give me all the products of type `S2_MSI_L1C`, use ``FilterOverlap`` to keep only those products
-that are contained in the bbox I gave you, or whose spatial extent overlaps at least 10% (``minimum_overlap``) of the surface
-of this bbox"
+The request above means : "Give me all the products of type `S2_MSI_L1C`, use ``FilterOverlap`` to keep only those
+products that are contained in the bbox I gave you, or whose spatial extent overlaps at least 10% (``minimum_overlap``)
+of the surface of this bbox".
 
-You can use ``eaodag search`` with custom parameters. Custom parameters will be used as is in the query string search sent
-to the provider. For instance, if you want to add foo=1 and bar=2 to the previous query:
+You can use ``eaodag search`` with custom query parameters. Custom query parameters will be used as is in the query
+string search sent to the provider. For instance, if you want to add foo=1 and bar=2 to the previous query:
 
 .. code-block:: console
 
         eodag search -f my_conf.yml -b 1 43 2 44 -s 2018-01-01 -e 2018-01-31 -p S2_MSI_L1C \
                      --cruncher FilterOverlap \
-                     --cruncher-args FilterOverlap minimum_overlap 1 \
-                     --custom "foo=1&bar=2"
+                     --cruncher-args FilterOverlap minimum_overlap 10 \
+                     --query "foo=1&bar=2"
 
 * To download the result of a previous call to ``search``:
 
@@ -131,8 +131,8 @@ to the provider. For instance, if you want to add foo=1 and bar=2 to the previou
 
         eodag list --no-fetch
 
-* EODAG can fetch providers (all or only a given one) to discover available product types, using the following command. It
-  will store result in a JSON file (defaults to `ext_product_types.json`):
+* EODAG can fetch providers (all or only a given one) to discover available product types, using the following command.
+  It will store result in a JSON file (defaults to `ext_product_types.json`):
 
 .. code-block:: console
 

eodag/api/core.py

@@ -56,6 +56,7 @@
 )
 from eodag.utils.exceptions import (
     AuthenticationError,
+    MisconfiguredError,
     NoMatchingProductType,
     PluginImplementationError,
     UnsupportedProvider,
@@ -540,13 +541,32 @@ def fetch_product_types_list(self, provider=None):
         """
         if provider is not None and provider not in self.providers_config:
             return
-        # check if any provider has already been fetched for product types
-        # (no need go get ext_product_types conf)
-        already_fetched = False
-        for provider, provider_config in self.providers_config.items():
-            if getattr(provider_config, "product_types_fetched", False):
-                already_fetched = True
-                break
+
+        # providers discovery confs that are fetchable
+        providers_discovery_configs_fetchable = {}
+        # check if any provider has not already been fetched for product types
+        already_fetched = True
+        for provider_to_fetch, provider_config in (
+            {provider: self.providers_config[provider]}.items()
+            if provider
+            else self.providers_config.items()
+        ):
+            # get discovery conf
+            if hasattr(provider_config, "search"):
+                provider_search_config = provider_config.search
+            elif hasattr(provider_config, "api"):
+                provider_search_config = provider_config.api
+            else:
+                continue
+            discovery_conf = getattr(
+                provider_search_config, "discover_product_types", {}
+            )
+            if discovery_conf.get("fetch_url", None):
+                providers_discovery_configs_fetchable[
+                    provider_to_fetch
+                ] = discovery_conf
+                if not getattr(provider_config, "product_types_fetched", False):
+                    already_fetched = False
 
         if not already_fetched:
             # get ext_product_types conf
@@ -559,24 +579,23 @@ def fetch_product_types_list(self, provider=None):
 
                 if not ext_product_types_conf:
                     # empty ext_product_types conf
-                    ext_product_types_conf = self.discover_product_types()
+                    discover_kwargs = dict(provider=provider) if provider else {}
+                    ext_product_types_conf = self.discover_product_types(
+                        **discover_kwargs
+                    )
 
             # update eodag product types list with new conf
             self.update_product_types_list(ext_product_types_conf)
 
+        # Compare current provider with default one to see if it has been modified
+        # and product types list would need to be fetched
+
         # get ext_product_types conf for user modified providers
         default_providers_config = load_default_config()
-        for provider, user_provider_config in self.providers_config.items():
-            # user discover_product_types conf
-            if hasattr(user_provider_config, "search"):
-                user_provider_search_config = user_provider_config.search
-            elif hasattr(user_provider_config, "api"):
-                user_provider_search_config = user_provider_config.api
-            else:
-                continue
-            user_discovery_conf = getattr(
-                user_provider_search_config, "discover_product_types", {}
-            )
+        for (
+            provider,
+            user_discovery_conf,
+        ) in providers_discovery_configs_fetchable.items():
             # default discover_product_types conf
             if provider in default_providers_config:
                 default_provider_config = default_providers_config[provider]
@@ -648,11 +667,20 @@ def discover_product_types(self, provider=None):
                         search_plugin.provider
                     )
                     if callable(getattr(auth_plugin, "authenticate", None)):
-                        search_plugin.auth = auth_plugin.authenticate()
+                        try:
+                            search_plugin.auth = auth_plugin.authenticate()
+                        except (AuthenticationError, MisconfiguredError) as e:
+                            logger.warning(
+                                f"Could not authenticate on {provider}: {str(e)}"
+                            )
+                            ext_product_types_conf[provider] = None
+                            continue
                     else:
-                        raise AuthenticationError(
-                            f"Could not authenticate using {auth_plugin} plugin"
+                        logger.warning(
+                            f"Could not authenticate on {provider} using {auth_plugin} plugin"
                         )
+                        ext_product_types_conf[provider] = None
+                        continue
 
                 ext_product_types_conf[
                     provider
@@ -1077,6 +1105,17 @@ def search_all(
             )
         except NoMatchingProductType:
             product_type = GENERIC_PRODUCT_TYPE
+        else:
+            # fetch product types list if product_type is unknown
+            if (
+                product_type
+                not in self._plugins_manager.product_type_to_provider_config_map.keys()
+            ):
+                logger.debug(
+                    f"Fetching external product types sources to find {product_type} product type"
+                )
+                self.fetch_product_types_list()
+
         search_plugin = next(
             self._plugins_manager.get_search_plugins(product_type=product_type)
         )
@@ -1132,7 +1171,12 @@ def _search_by_id(self, uid, provider=None, **kwargs):
                   of EO products retrieved (0 or 1)
         :rtype: tuple(:class:`~eodag.api.search_result.SearchResult`, int)
         """
-        for plugin in self._plugins_manager.get_search_plugins(provider=provider):
+        get_search_plugins_kwargs = dict(
+            provider=provider, product_type=kwargs.get("productType", None)
+        )
+        for plugin in self._plugins_manager.get_search_plugins(
+            **get_search_plugins_kwargs
+        ):
             logger.info(
                 "Searching product with id '%s' on provider: %s", uid, plugin.provider
             )
@@ -1245,6 +1289,16 @@ def _prepare_search(
             kwargs.pop(arg, None)
         del kwargs["locations"]
 
+        # fetch product types list if product_type is unknown
+        if (
+            product_type
+            not in self._plugins_manager.product_type_to_provider_config_map.keys()
+        ):
+            logger.debug(
+                f"Fetching external product types sources to find {product_type} product type"
+            )
+            self.fetch_product_types_list()
+
         search_plugin = next(
             self._plugins_manager.get_search_plugins(product_type=product_type)
         )

eodag/plugins/authentication/keycloak.py

@@ -39,21 +39,21 @@ def authenticate(self):
         Makes authentication request
         """
         self.validate_config_credentials()
-        response = self.session.post(
-            self.TOKEN_URL_TEMPLATE.format(
-                auth_base_uri=self.config.auth_base_uri.rstrip("/"),
-                realm=self.config.realm,
-            ),
-            data={
-                "client_id": self.config.client_id,
-                "client_secret": self.config.client_secret,
-                "username": self.config.credentials["username"],
-                "password": self.config.credentials["password"],
-                "grant_type": self.GRANT_TYPE,
-            },
-            timeout=HTTP_REQ_TIMEOUT,
-        )
         try:
+            response = self.session.post(
+                self.TOKEN_URL_TEMPLATE.format(
+                    auth_base_uri=self.config.auth_base_uri.rstrip("/"),
+                    realm=self.config.realm,
+                ),
+                data={
+                    "client_id": self.config.client_id,
+                    "client_secret": self.config.client_secret,
+                    "username": self.config.credentials["username"],
+                    "password": self.config.credentials["password"],
+                    "grant_type": self.GRANT_TYPE,
+                },
+                timeout=HTTP_REQ_TIMEOUT,
+            )
             response.raise_for_status()
         except requests.RequestException as e:
             # check if error is identified as auth_error in provider conf