Choosing the best model

[cjacky475]

Hello,

I am trying to detect anomalous temperature points. I have several datasets manually labeled. Now my task is to find the best model that would give the highest recall, since missing anomalous temperature point is critical. Would it be fair:

1. Train various models on each dataset
2. predict() results on test set and compare with true labels
3. Select the model that has highest recall

The problem I think of is that if I used decision_function() and, let's say, used 0.3 threshold as anomalous, different models would give better results than models that I tested on predict(). Also, there are thousands of datasets and I would not know the correct threshold for each dataset.

So what would the best way to select the model?