Recommendation History, Deterministic?

[brandon-holt]

Hi, is recommendation for botorchrecommender supposed to be deterministic?

I find that if I save a campaign before every making recommendations, and request 1, 2, 3, etc. recommendations, the recs are preserved. For example, the first in the list of 2 and 3 is the same as in the list of 1. And the second in the list of 3 is the same as the second in the list of 2.

But when I save the campaign after having made any recommendations, the next batch does not preserve any of the historical recommendations and makes entirely new ones, overwriting the history. Why does it work this way? This seems pointless.

[Scienfitz]

Hi,
there are many aspects that could influence this, you need to provide more information.

• Did you freeze the random seed?
• Are you setting the sequential_continuous flag?
• Are you operating in a discrete, hybrid or continuous space?

Sequential Greedy Batch Recommendation
As example for the latter point: In discrete and hybrid spaces the botorch recommender always operates greedily. This means, when recommending a batch of N points, it first selects the best point, then conditions the acqf on that point, then selects the second point and so on. This naturally leads to the effect you describe in the second paragraph. In continuous spaces, the flag can change the outcome.

Cached Recommendations
Perhaps there's no immediate connection, but you should know that recommendations are cached when you work with Campaigns. So if you perform the exact same recommendation call twice, it should proved the exact same recommendation. It only recomputes in case you use a different batch size or new data were added etc.

To understand the potential connections to campaign saving that you describe, it would be good to have a minimal reproducing example.

[AdrianSosic]

Hi @brandon-holt 👋🏼 Generally speaking, unless you fix the random seed (e.g., via our set_random_seed/temporary_seed helpers), the output of recommenders is non-deterministic. Depending on the recommender setting and context, there can generally be several sources of randomness involved that would result in non-reproducible outcomes, such as the initial conditions for the optimization of the hyperparameters, or of the acquisition function.

HOWEVER: in most practical situations, where you condition the model on some data, this randomness should not matter too much since you'll likely end up in the same (local) optimum. That's why I think what you describe here could have a different cause. Unfortunately, I cannot completely follow what exact steps you've conducted. Would you be so kind to set up a minimal example demonstrating the behavior?

[AdrianSosic]

Oh, I see that @Scienfitz texted at the same time :D

[brandon-holt]

Thanks, this is helpful!