Learner Metrics besides accuracy throw assertion error

[ebagjones]

Can you please clarify how we can use alternate metrics for binary classification problems?

learn = Learner(dls, model, metrics=accuracy, cbs=ShowGraph())

Virtually everything I've tried besides "accuracy" throws an assertion error like:
AssertionError: ==: 128 64

I understand that this is likely due to the "when inp and targ are the same size" requirement but I don't understand why inp and targ wouldn't be the same or how I can make them match.

[oguiza]

Hi @jefferyanderson,
I'm not sure where your issue is coming from. I've just tried the following code and it works well:

from tsai.all import *
X, y, splits = get_UCR_data('MoteStrain', split_data=False)
tfms = [None, TSClassification()]
batch_tfms = TSStandardize(by_sample=True)
dls = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
learn = ts_learner(dls, InceptionTimePlus, metrics=[accuracy, RocAucBinary(), APScoreBinary()], cbs=[ShowGraph()])
learn.fit_one_cycle(10, 1e-2)

It works both with the latest pip release (tsai 0.3.0) and also the GitHub release.

[ebagjones]

thanks @oguiza . what is the correct way to use precision and/or recall for a binary classification problem?

[oguiza]

You can use them as defined in the previous example:

from tsai.all import *
X, y, splits = get_UCR_data('MoteStrain', split_data=False)
tfms = [None, TSClassification()]
batch_tfms = TSStandardize(by_sample=True)
dls = get_ts_dls(X, y, splits=splits, tfms=tfms, batch_tfms=batch_tfms)
learn = ts_learner(dls, InceptionTimePlus, metrics=[accuracy, RocAucBinary(), APScoreBinary(), Precision(), Recall()], cbs=[ShowGraph()])
learn.fit_one_cycle(10, 1e-2)

Please, bear in mind that all these metrics come from fastai and sktime. There are only a few specific metrics we developed for tsai. You may be able to find them documented here.

[ebagjones]

Thanks @oguiza , sorry for the dumb question. In the documentation I only saw the options for multi-classification and had been trying those. It looks like it's working well now. Does the order of the metrics inform the weight/priority the learner places on the metric?

[oguiza]

There’re no dumb questions 😀
The order of the metrics is irrelevant.
Actually the learner only tries to reduce the training loss.
We just use validation loss and metrics to try to understand if what the model is learning is useful or not.