Move allow_* flags to Campaign (#423)

This PR builds upon #412 and finalizes the metadata migration from
search spaces / recommenders to `Campaign`, making `Campaign` the only
stateful class:

* All `allow_*` flags are now passed directly to `Campaign`,
strengthening the role of `Campaign` as _the_ class for metadata
handling / tracking the progress of an experimentation path.
* A corresponding deprecation mechanism is set into place.
* The documentation gets updated.
* `allow_repeated_recommendations` is renamed to
`allow_recommending_already_recommended` because i) the original name
was imprecise in that it also suggested that repetitions are disallowed
_within_ a batch and ii) the new follows the same pattern as the other
two flags.
* The flags are now context-aware, i.e. setting flags in contexts where
they have no effect now raises an error. This avoids surprises on the
user side, which have been reported multiple times.
* Replaces the private class `AnnotatedSubspaceDiscrete` with a new
private class `FilteredSubspaceDiscrete` taking over the original role
but without the necessity of being metadata aware.

### Coming next 
The changes introduced here and in #423 bring significant improvements
from a user interface perspective in that:
* Metadata handling now happens entirely behind the scenes
* No more fiddling with internal low-level objects (like setting Boolean
values in metadata dataframes) is necessary to control candidate
generation is necessary. Instead, there is now an interface that enables
control via user-level objects such as `Constraint` objects and
candidates dataframes.
* The `pandas` part of the current discrete search space implementation
is less entangled in the code base

This paves the way for upcoming enhancements:
* A `SubspaceDiscreteProtocol` is already in sight, which allows
seamless integration of other backends (`polars`, databases, etc) and
will help us complete the ongoing `polars` transition.
* An improved user interface for manipulating existing state spaces a la
`SubspaceDiscrete.filter(constraints)`, which can also become the
backbone of the current constraints logic

.lockfiles/py310-dev.lock

@@ -891,6 +891,7 @@ sphinx==8.1.3
     #   sphinx-autodoc-typehints
     #   sphinx-basic-ng
     #   sphinx-copybutton
+    #   sphinx-design
     #   sphinx-paramlinks
     #   sphinxcontrib-bibtex
 sphinx-autodoc-typehints==2.5.0
@@ -899,6 +900,8 @@ sphinx-basic-ng==1.0.0b2
     # via furo
 sphinx-copybutton==0.5.2
     # via baybe (pyproject.toml)
+sphinx-design==0.6.1
+    # via baybe (pyproject.toml)
 sphinx-paramlinks==0.6.0
     # via baybe (pyproject.toml)
 sphinxcontrib-applehelp==1.0.8

CHANGELOG.md

@@ -38,6 +38,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `MetaRecommender`s can now be composed of other `MetaRecommender`s
 - For performance reasons, search space manipulation using `polars` is no longer
   guaranteed to produce the same row order as the corresponding `pandas` operations
+- `allow_repeated_recommendations` has been renamed to 
+  `allow_recommending_already_recommended` and is now `True` by default
 
 ### Fixed
 - Rare bug arising from degenerate `SubstanceParameter.comp_df` rows that caused
@@ -54,6 +56,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   `SingleTargetObjective` no longer erroneously maximizes it
 - Improvement-based Monte Carlo acquisition functions now use the correct
   reference value in minimization mode
+- `allow_*` flags are now context-aware, i.e. setting them in a context where they are
+  irrelevant now raises an error instead of passing silently
 
 ### Removed
 - `botorch_function_wrapper` utility for creating lookup callables
@@ -73,6 +77,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   exclusively handled by the `Campaign` class.
 - `get_current_recommender` and `get_next_recommender` of `MetaRecommender` have become
   obsolete and calling them is no longer possible
+- Passing `allow_*` flags to recommenders is no longer supported since the necessary
+  metadata required for the flags is no longer available at that level. The
+  functionality has been taken over by `Campaign`.
 
 ## [0.11.3] - 2024-11-06
 ### Fixed

baybe/campaign.py

@@ -4,27 +4,28 @@
 
 import gc
 import json
-from collections.abc import Collection
+from collections.abc import Callable, Collection
 from functools import reduce
-from typing import TYPE_CHECKING
+from typing import TYPE_CHECKING, Any
 
 import cattrs
 import numpy as np
 import pandas as pd
-from attrs import define, evolve, field
+from attrs import Attribute, Factory, define, evolve, field, fields
 from attrs.converters import optional
 from attrs.validators import instance_of
 from typing_extensions import override
 
 from baybe.constraints.base import DiscreteConstraint
-from baybe.exceptions import IncompatibilityError
+from baybe.exceptions import IncompatibilityError, NotEnoughPointsLeftError
 from baybe.objectives.base import Objective, to_objective
 from baybe.parameters.base import Parameter
 from baybe.recommenders.base import RecommenderProtocol
 from baybe.recommenders.meta.base import MetaRecommender
 from baybe.recommenders.meta.sequential import TwoPhaseMetaRecommender
 from baybe.recommenders.pure.bayesian.base import BayesianRecommender
-from baybe.searchspace._annotated import AnnotatedSubspaceDiscrete
+from baybe.recommenders.pure.nonpredictive.base import NonPredictiveRecommender
+from baybe.searchspace._filtered import FilteredSubspaceDiscrete
 from baybe.searchspace.core import (
     SearchSpace,
     SearchSpaceType,
@@ -39,7 +40,7 @@
     telemetry_record_recommended_measurement_percentage,
     telemetry_record_value,
 )
-from baybe.utils.basic import is_all_instance
+from baybe.utils.basic import UNSPECIFIED, UnspecifiedType, is_all_instance
 from baybe.utils.boolean import eq_dataframe
 from baybe.utils.dataframe import filter_df, fuzzy_row_match
 from baybe.utils.plotting import to_string
@@ -54,6 +55,38 @@
 _METADATA_COLUMNS = [_RECOMMENDED, _MEASURED, _EXCLUDED]
 
 
+def _make_allow_flag_default_factory(
+    default: bool,
+) -> Callable[[Campaign], bool | UnspecifiedType]:
+    """Make a default factory for allow_* flags."""
+
+    def default_allow_flag(campaign: Campaign) -> bool | UnspecifiedType:
+        """Attrs-compatible default factory for allow_* flags."""
+        if campaign.searchspace.type is SearchSpaceType.DISCRETE:
+            return default
+        return UNSPECIFIED
+
+    return default_allow_flag
+
+
+def _validate_allow_flag(campaign: Campaign, attribute: Attribute, value: Any) -> None:
+    """Attrs-compatible validator for context-aware validation of allow_* flags."""
+    match campaign.searchspace.type:
+        case SearchSpaceType.DISCRETE:
+            if not isinstance(value, bool):
+                raise ValueError(
+                    f"For search spaces of '{SearchSpaceType.DISCRETE}', "
+                    f"'{attribute.name}' must be a Boolean."
+                )
+        case _:
+            if value is not UNSPECIFIED:
+                raise ValueError(
+                    f"For search spaces of type other than "
+                    f"'{SearchSpaceType.DISCRETE}', '{attribute.name}' cannot be set "
+                    f"since the flag is meaningless in such contexts.",
+                )
+
+
 @define
 class Campaign(SerialMixin):
     """Main class for interaction with BayBE.
@@ -88,6 +121,36 @@ class Campaign(SerialMixin):
     )
     """The employed recommender"""
 
+    allow_recommending_already_measured: bool | UnspecifiedType = field(
+        default=Factory(
+            _make_allow_flag_default_factory(default=True), takes_self=True
+        ),
+        validator=_validate_allow_flag,
+        kw_only=True,
+    )
+    """Allow to recommend experiments that were already measured earlier.
+    Can only be set for discrete search spaces."""
+
+    allow_recommending_already_recommended: bool | UnspecifiedType = field(
+        default=Factory(
+            _make_allow_flag_default_factory(default=True), takes_self=True
+        ),
+        validator=_validate_allow_flag,
+        kw_only=True,
+    )
+    """Allow to recommend experiments that were already recommended earlier.
+    Can only be set for discrete search spaces."""
+
+    allow_recommending_pending_experiments: bool | UnspecifiedType = field(
+        default=Factory(
+            _make_allow_flag_default_factory(default=False), takes_self=True
+        ),
+        validator=_validate_allow_flag,
+        kw_only=True,
+    )
+    """Allow pending experiments to be part of the recommendations.
+    Can only be set for discrete search spaces."""
+
     # Metadata
     _searchspace_metadata: pd.DataFrame = field(init=False, eq=eq_dataframe)
     """Metadata tracking the experimentation status of the search space."""
@@ -301,6 +364,9 @@ def toggle_discrete_candidates(  # noqa: DOC501
             A new dataframe containing the  discrete candidate set passing through the
             specified filter.
         """
+        # Clear cache
+        self._cached_recommendation = pd.DataFrame()
+
         df = self.searchspace.discrete.exp_rep
 
         if isinstance(constraints, pd.DataFrame):
@@ -371,21 +437,87 @@ def recommend(
             self._measurements_exp.fillna({"FitNr": self.n_fits_done}, inplace=True)
 
         # Prepare the search space according to the current campaign state
-        annotated_searchspace = evolve(
-            self.searchspace,
-            discrete=AnnotatedSubspaceDiscrete.from_subspace(
-                self.searchspace.discrete, self._searchspace_metadata
-            ),
-        )
+        if self.searchspace.type is SearchSpaceType.DISCRETE:
+            # TODO: This implementation should at some point be hidden behind an
+            #   appropriate public interface, like `SubspaceDiscrete.filter()`
+            mask_todrop = self._searchspace_metadata[_EXCLUDED].copy()
+            if not self.allow_recommending_already_recommended:
+                mask_todrop |= self._searchspace_metadata[_RECOMMENDED]
+            if not self.allow_recommending_already_measured:
+                mask_todrop |= self._searchspace_metadata[_MEASURED]
+            if (
+                not self.allow_recommending_pending_experiments
+                and pending_experiments is not None
+            ):
+                mask_todrop |= pd.merge(
+                    self.searchspace.discrete.exp_rep,
+                    pending_experiments,
+                    indicator=True,
+                    how="left",
+                )["_merge"].eq("both")
+            searchspace = evolve(
+                self.searchspace,
+                discrete=FilteredSubspaceDiscrete.from_subspace(
+                    self.searchspace.discrete, ~mask_todrop.to_numpy()
+                ),
+            )
+        else:
+            searchspace = self.searchspace
+
+        # Pending experiments should not be passed to non-predictive recommenders
+        # to avoid complaints about unused arguments, so we need to know of what
+        # type the next recommender will be
+        recommender = self.recommender
+        if isinstance(recommender, MetaRecommender):
+            recommender = recommender.get_non_meta_recommender(
+                batch_size,
+                searchspace,
+                self.objective,
+                self._measurements_exp,
+                pending_experiments,
+            )
+        is_nonpredictive = isinstance(recommender, NonPredictiveRecommender)
 
         # Get the recommended search space entries
-        rec = self.recommender.recommend(
-            batch_size,
-            annotated_searchspace,
-            self.objective,
-            self._measurements_exp,
-            pending_experiments,
-        )
+        try:
+            # NOTE: The `recommend` call must happen on `self.recommender` to update
+            #   potential inner states in case of meta recommenders!
+            rec = self.recommender.recommend(
+                batch_size,
+                searchspace,
+                self.objective,
+                self._measurements_exp,
+                None if is_nonpredictive else pending_experiments,
+            )
+        except NotEnoughPointsLeftError as ex:
+            # Aliases for code compactness
+            f = fields(Campaign)
+            ok_m = self.allow_recommending_already_measured
+            ok_r = self.allow_recommending_already_recommended
+            ok_p = self.allow_recommending_pending_experiments
+            ok_m_name = f.allow_recommending_already_measured.name
+            ok_r_name = f.allow_recommending_already_recommended.name
+            ok_p_name = f.allow_recommending_pending_experiments.name
+            no_blocked_pending_points = ok_p or (pending_experiments is None)
+
+            # If there are no candidate restrictions to be relaxed
+            if ok_m and ok_r and no_blocked_pending_points:
+                raise ex
+
+            # Otherwise, extract possible relaxations
+            solution = [
+                f"'{name}=True'"
+                for name, value in [
+                    (ok_m_name, ok_m),
+                    (ok_r_name, ok_r),
+                    (ok_p_name, no_blocked_pending_points),
+                ]
+                if not value
+            ]
+            message = solution[0] if len(solution) == 1 else " and/or ".join(solution)
+            raise NotEnoughPointsLeftError(
+                f"{str(ex)} Consider setting {message}."
+            ) from ex
 
         # Cache the recommendations
         self._cached_recommendation = rec.copy()

baybe/exceptions.py

@@ -33,6 +33,10 @@ class IncompatibleExplainerError(IncompatibilityError):
     """An explainer is incompatible with the data it is presented."""
 
 
+class IncompatibleArgumentError(IncompatibilityError):
+    """An incompatible argument was passed to a callable."""
+
+
 class NotEnoughPointsLeftError(Exception):
     """
     More recommendations are requested than there are viable parameter configurations

baybe/recommenders/base.py

@@ -68,6 +68,11 @@ def recommend(
             _surrogate_model=cattrs.override(rename="surrogate_model"),
             _current_recommender=cattrs.override(omit=False),
             _used_recommender_ids=cattrs.override(omit=False),
+            _deprecated_allow_repeated_recommendations=cattrs.override(omit=True),
+            _deprecated_allow_recommending_already_measured=cattrs.override(omit=True),
+            _deprecated_allow_recommending_pending_experiments=cattrs.override(
+                omit=True
+            ),
         ),
     ),
 )

baybe/recommenders/naive.py

@@ -1,11 +1,10 @@
 """Naive recommender for hybrid spaces."""
 
 import gc
-import warnings
 from typing import ClassVar
 
 import pandas as pd
-from attrs import define, evolve, field, fields
+from attrs import define, field
 from typing_extensions import override
 
 from baybe.objectives.base import Objective
@@ -48,36 +47,6 @@ class NaiveHybridSpaceRecommender(PureRecommender):
     """The recommender used for the continuous subspace. Default:
     :class:`baybe.recommenders.pure.bayesian.botorch.BotorchRecommender`"""
 
-    def __attrs_post_init__(self):
-        """Validate if flags are synchronized and overrides them otherwise."""
-        if (
-            flag := self.allow_recommending_already_measured
-        ) != self.disc_recommender.allow_recommending_already_measured:
-            warnings.warn(
-                f"The value of "
-                f"'{fields(self.__class__).allow_recommending_already_measured.name}' "
-                f"differs from what is specified in the discrete recommender. "
-                f"The value of the discrete recommender will be ignored."
-            )
-            self.disc_recommender = evolve(
-                self.disc_recommender,
-                allow_recommending_already_measured=flag,
-            )
-
-        if (
-            flag := self.allow_repeated_recommendations
-        ) != self.disc_recommender.allow_repeated_recommendations:
-            warnings.warn(
-                f"The value of "
-                f"'{fields(self.__class__).allow_repeated_recommendations.name}' "
-                f"differs from what is specified in the discrete recommender. "
-                f"The value of the discrete recommender will be ignored."
-            )
-            self.disc_recommender = evolve(
-                self.disc_recommender,
-                allow_repeated_recommendations=flag,
-            )
-
     @override
     def recommend(
         self,
@@ -89,8 +58,9 @@ def recommend(
     ) -> pd.DataFrame:
         from baybe.acquisition.partial import PartialAcquisitionFunction
 
-        if (not isinstance(self.disc_recommender, BayesianRecommender)) and (
-            not isinstance(self.disc_recommender, NonPredictiveRecommender)
+        disc_is_bayesian = isinstance(self.disc_recommender, BayesianRecommender)
+        if not disc_is_bayesian and not isinstance(
+            self.disc_recommender, NonPredictiveRecommender
         ):
             raise NotImplementedError(
                 """The discrete recommender should be either a Bayesian or a
@@ -110,7 +80,7 @@ def recommend(
                 searchspace=searchspace,
                 objective=objective,
                 measurements=measurements,
-                pending_experiments=pending_experiments,
+                pending_experiments=pending_experiments if disc_is_bayesian else None,
             )
 
         # We are in a hybrid setting now
@@ -122,12 +92,8 @@ def recommend(
         cont_part = searchspace.continuous.sample_uniform(1)
         cont_part_tensor = to_tensor(cont_part).unsqueeze(-2)
 
-        # Get discrete candidates. The metadata flags are ignored since the search space
-        # is hybrid
-        candidates_exp, _ = searchspace.discrete.get_candidates(
-            allow_repeated_recommendations=True,
-            allow_recommending_already_measured=True,
-        )
+        # Get discrete candidates
+        candidates_exp, _ = searchspace.discrete.get_candidates()
 
         # We now check whether the discrete recommender is bayesian.
         if isinstance(self.disc_recommender, BayesianRecommender):