How to add lstm regressor as estimator in paired_ttest_5x2cv function

[Nikhitha74]

Hi,
I am trying to compare lstm model to a knn, but the lstm model estimator is showing problems with the shape of the input. Can someone suggest how to pass the lstm model to the paired_ttest_5x2cv function.

cr_model():
n_timesteps=15
n_features=3
model = Sequential()
model.add(LSTM(16, input_shape=(n_timesteps,n_features)))
model.add(Dense(1, activation='relu'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
return model

lstm_model=cr_model()

from sklearn.svm import SVR

svr_rbf = SVR(kernel= 'rbf', C= 1e2, gamma= 0.1)

t, p = paired_ttest_5x2cv(estimator1=svr_rbf,
estimator2=lstm_model,
X=X, y=y,
random_seed=1)

ValueError: Input 0 of layer "sequential_1" is incompatible with the layer: expected shape=(None, 15, 3), found shape=(None, 3)

How can I correct this error. Kindly help.

Regards,
Nikhitha.

[Nikhitha74]

Sorry i am comparing lstm with SVR. If i compare svr and KNN there is no issue, but wiht lstm i do not know how to fix the shape.

[rasbt]

It looks like the LSTM is taking in a 3D input? I think you may have to write a custom wrapper to reshare the 2D into a 3D input. With that, I mean you need a class around the LSTM that takes care of the reshaping.

[Nikhitha74]

Would you be kind enough to add an example code... or give any reference to this kind of class. the error is thrown at estimator2.fit. How do I access that code.

[Nikhitha74]

I have written a class. I am not sure whether this is the right approach. I am a beginner. Please excuse me for my coding skills. I am getting an error.

lstm_model=lstm_create(X,y)

class lstm_create:
def init(self,X,y):
self.X=X
self.y=y
features=3
#self.look_back=35
time_step=35
model = Sequential()
model.add(LSTM(16, input_shape=(time_step,features)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')

def fit(X,Y):
    dataX, dataY = [], []
    for i in range(time_step, len(X)):
        dataX.append(X[i-time_step:i,:])
        dataY.append(y[i,0])

    dataX, dataY = np.asarray(dataX), np.asarray(dataY)
    dataX = np.reshape(dataX, (dataX.shape[0], dataX.shape[1], dataX.shape[2]))
    hist=model.fit(dataX, dataY)
    return hist

t, p = paired_ttest_5x2cv(estimator1=svr_rbf,
estimator2=lstm_model,
X=Xsc,y=ysc,
random_seed=1)

It is throwing an error as below:
TypeError Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_21940\3071584853.py in
----> 1 t, p = paired_ttest_5x2cv(estimator1=svr_rbf,
2 estimator2=lstm_model,
3 X=Xsc,y=ysc,
4 random_seed=1)

~\anaconda3\lib\site-packages\mlxtend\evaluate\ttest.py in paired_ttest_5x2cv(estimator1, estimator2, X, y, scoring, random_seed)
319 X_1, X_2, y_1, y_2 = train_test_split(X, y, test_size=0.5, random_state=randint)
320
--> 321 score_diff_1 = score_diff(X_1, X_2, y_1, y_2)
322 score_diff_2 = score_diff(X_2, X_1, y_2, y_1)
323 score_mean = (score_diff_1 + score_diff_2) / 2.0

~\anaconda3\lib\site-packages\mlxtend\evaluate\ttest.py in score_diff(X_1, X_2, y_1, y_2)
308
309 estimator1.fit(X_1, y_1)
--> 310 estimator2.fit(X_1, y_1)
311 est1_score = scorer(estimator1, X_2, y_2)
312 est2_score = scorer(estimator2, X_2, y_2)

TypeError: fit() takes 2 positional arguments but 3 were given.
Any help would be appreciated.

Thanks,
Nikhitha

[rasbt]

Hi there,

I think the issue in your case is that the your model doesn't support model.fit(X, y). Or in other words, you defined a fit function, but it's not part of your model.

What would be required here is a class that has a fit function. The skeleton would be something like the following

class MyLSTM():
    def __init__(self, model, ...):
        self.model = ...
	...

    def fit(self, X, y):
        ...
        model.fit(...)

    def predict(X):
        ...
        model.predict(...)

[Nikhitha74]

Thank you for sharing this..
I could get past the predict method with the modifications suggested. However, the score is throwing an error. Kindly have a look at the error and give your valuable suggestions. Attaching the code below:

class MyLSTM():
def init(self, model, X,y):
self.model = model
self.X=X
self.y=y

def fit(self, X, y):
    features=3
    time_step=35
    dataX, dataY = [], []
    for i in range(time_step, len(X)):
        dataX.append(X[i-time_step:i,:])
        dataY.append(y[i,0])

    dataX=np.asarray(dataX)
    dataY = np.asarray(dataY)
    dataX = np.reshape(dataX, (dataX.shape[0], dataX.shape[1], dataX.shape[2]))
    self.model.fit(dataX, np.ravel(dataY))
    

def predict(self,X):
    features=3
    time_step=35
    dataX =[]
    for i in range(time_step, len(X)):
        dataX.append(X[i-time_step:i,:])
    dataX = np.asarray(dataX)
    dataX = np.reshape(dataX, (dataX.shape[0], dataX.shape[1], dataX.shape[2]))
    self.model.predict(dataX)

def cr_model():
features=3
#self.look_back=35
time_step=35
model = Sequential()
model.add(LSTM(16, input_shape=(time_step,features)))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
return model

ls_model=MyLSTM(cr_model(),Xsc,ysc)

t, p = paired_ttest_5x2cv(estimator1=svr_rbf,
estimator2=ls_model,
X=Xsc,y=ysc,
random_seed=1)

81/81 [==============================] - 2s 9ms/step - loss: 0.0364
81/81 [==============================] - 1s 3ms/step
C:\Users\Nikhitha\anaconda3\lib\site-packages\sklearn\metrics_regression.py:96: FutureWarning: Arrays of bytes/strings is being converted to decimal numbers if dtype='numeric'. This behavior is deprecated in 0.24 and will be removed in 1.1 (renaming of 0.26). Please convert your data to numeric values explicitly instead.
y_pred = check_array(y_pred, ensure_2d=False, dtype=dtype)

ValueError Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_9308\2485215778.py in
----> 1 t, p = paired_ttest_5x2cv(estimator1=svr_rbf,
2 estimator2=ls_model,
3 X=Xsc,y=ysc,
4 random_seed=1)

~\anaconda3\lib\site-packages\mlxtend\evaluate\ttest.py in paired_ttest_5x2cv(estimator1, estimator2, X, y, scoring, random_seed)
319 X_1, X_2, y_1, y_2 = train_test_split(X, y, test_size=0.5, random_state=randint)
320
--> 321 score_diff_1 = score_diff(X_1, X_2, y_1, y_2)
322 score_diff_2 = score_diff(X_2, X_1, y_2, y_1)
323 score_mean = (score_diff_1 + score_diff_2) / 2.0

~\anaconda3\lib\site-packages\mlxtend\evaluate\ttest.py in score_diff(X_1, X_2, y_1, y_2)
310 estimator2.fit(X_1, y_1)
311 est1_score = scorer(estimator1, X_2, y_2)
--> 312 est2_score = scorer(estimator2, X_2, y_2)
313 score_diff = est1_score - est2_score
314 return score_diff

~\anaconda3\lib\site-packages\sklearn\metrics_scorer.py in call(self, estimator, X, y_true, sample_weight)
214 Score function applied to prediction of estimator on X.
215 """
--> 216 return self._score(
217 partial(_cached_call, None),
218 estimator,

~\anaconda3\lib\site-packages\sklearn\metrics_scorer.py in _score(self, method_caller, estimator, X, y_true, sample_weight)
262 )
263 else:
--> 264 return self._sign * self._score_func(y_true, y_pred, **self._kwargs)
265
266

~\anaconda3\lib\site-packages\sklearn\metrics_regression.py in r2_score(y_true, y_pred, sample_weight, multioutput)
787 -3.0
788 """
--> 789 y_type, y_true, y_pred, multioutput = _check_reg_targets(
790 y_true, y_pred, multioutput
791 )

~\anaconda3\lib\site-packages\sklearn\metrics_regression.py in _check_reg_targets(y_true, y_pred, multioutput, dtype)
94 check_consistent_length(y_true, y_pred)
95 y_true = check_array(y_true, ensure_2d=False, dtype=dtype)
---> 96 y_pred = check_array(y_pred, ensure_2d=False, dtype=dtype)
97
98 if y_true.ndim == 1:

~\anaconda3\lib\site-packages\sklearn\utils\validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator)
798
799 if force_all_finite:
--> 800 _assert_all_finite(array, allow_nan=force_all_finite == "allow-nan")
801
802 if ensure_min_samples > 0:

~\anaconda3\lib\site-packages\sklearn\utils\validation.py in _assert_all_finite(X, allow_nan, msg_dtype)
112 ):
113 type_err = "infinity" if allow_nan else "NaN, infinity"
--> 114 raise ValueError(
115 msg_err.format(
116 type_err, msg_dtype if msg_dtype is not None else X.dtype

ValueError: Input contains NaN, infinity or a value too large for dtype('float64').