How to use use_probas=True with StackingCVClassifier()

[skuma17]

i tried creating a stacking model like:

clf1 = LogisticRegression(random_state=42)
clf2 = RandomForestClassifier(n_estimators=30, random_state=42) # just for demo have taken 10 trees
xgb = XGBClassifier(use_label_encoder=False,learning_rate=0.02, objective='binary:logistic',n_estimators=30, nthread=2,eval_metric='logloss')
#use_label_encoder=False

stacking_demo = StackingCVClassifier(classifiers=[clf1, clf2], meta_classifier=xgb, use_probas=True, shuffle=True, random_state=42 )

for clf, label in zip([clf1, clf2, stacking_demo],
['lr',
'Random Forest',
'StackingClassifier']):

scores = model_selection.cross_val_score(clf, x_train, y_train, cv=3, scoring='accuracy')
print("Accuracy: %0.2f (+/- %0.2f) [%s]" % (scores.mean(), scores.std(), label))

when i run this code: i get an error like:

C:\Users\tsati\anaconda3\lib\site-packages\sklearn\model_selection_validation.py:985: RuntimeWarning: Number of classes in training fold (2) does not match total number of classes (1). Results may not be appropriate for your use case. To fix this, use a cross-validation technique resulting in properly stratified folds
warnings.warn('Number of classes in training fold ({}) does '
C:\Users\tsati\anaconda3\lib\site-packages\sklearn\model_selection_validation.py:610: FitFailedWarning: Estimator fit failed. The score on this train-test partition for these parameters will be set to nan. Details:
Traceback (most recent call last):
File "C:\Users\tsati\anaconda3\lib\site-packages\sklearn\model_selection_validation.py", line 593, in _fit_and_score
estimator.fit(X_train, y_train, **fit_params)
File "C:\Users\tsati\anaconda3\lib\site-packages\mlxtend\classifier\stacking_cv_classification.py", line 238, in fit
prediction = cross_val_predict(
File "C:\Users\tsati\anaconda3\lib\site-packages\sklearn\utils\validation.py", line 63, in inner_f
return f(*args, **kwargs)
File "C:\Users\tsati\anaconda3\lib\site-packages\sklearn\model_selection_validation.py", line 866, in cross_val_predict
predictions = parallel(delayed(_fit_and_predict)(
File "C:\Users\tsati\anaconda3\lib\site-packages\joblib\parallel.py", line 1041, in call
if self.dispatch_one_batch(iterator):
File "C:\Users\tsati\anaconda3\lib\site-packages\joblib\parallel.py", line 859, in dispatch_one_batch
self._dispatch(tasks)
File "C:\Users\tsati\anaconda3\lib\site-packages\joblib\parallel.py", line 777, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File "C:\Users\tsati\anaconda3\lib\site-packages\joblib_parallel_backends.py", line 208, in apply_async
result = ImmediateResult(func)
File "C:\Users\tsati\anaconda3\lib\site-packages\joblib_parallel_backends.py", line 572, in init
self.results = batch()
File "C:\Users\tsati\anaconda3\lib\site-packages\joblib\parallel.py", line 262, in call
return [func(*args, **kwargs)
File "C:\Users\tsati\anaconda3\lib\site-packages\joblib\parallel.py", line 262, in
return [func(*args, **kwargs)
File "C:\Users\tsati\anaconda3\lib\site-packages\sklearn\utils\fixes.py", line 222, in call
return self.function(*args, **kwargs)
File "C:\Users\tsati\anaconda3\lib\site-packages\sklearn\model_selection_validation.py", line 962, in _fit_and_predict
predictions = _enforce_prediction_order(
File "C:\Users\tsati\anaconda3\lib\site-packages\sklearn\model_selection_validation.py", line 1019, in _enforce_prediction_order
predictions_for_all_classes[:, classes] = predictions
IndexError: index 1 is out of bounds for axis 1 with size 1

[rasbt]

Maybe it has something do do with the dataset formatting. Could you print the output from

import numpy as np

print('x_train', x_train)
print('y_train', y_train)
print('y_train labels', np.unique(y_train))

[rasbt]

Hm, not sure, but I suspect it's something about the dataset. The code works fine for me on another dataset.

from sklearn import model_selection
from sklearn.model_selection import train_test_split
from sklearn import datasets


data = datasets.load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = \
    train_test_split(X, y, test_size=0.3, random_state=1, stratify=y)

X_train_sub, X_valid, y_train_sub, y_valid = \
    train_test_split(X_train, y_train, test_size=0.2, random_state=1, stratify=y_train)

############

from xgboost import XGBClassifier
from mlxtend.classifier import StackingCVClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

clf1 = LogisticRegression(random_state=42)
clf2 = RandomForestClassifier(n_estimators=30, random_state=42) # just for demo have taken 10 trees
xgb = XGBClassifier(use_label_encoder=False,learning_rate=0.02, objective='binary:logistic',n_estimators=30, nthread=2,eval_metric='logloss')
#use_label_encoder=False

stacking_demo = StackingCVClassifier(classifiers=[clf1, clf2], meta_classifier=xgb, use_probas=True, shuffle=True, random_state=42 )

scores = model_selection.cross_val_score(stacking_demo, X_train, y_train, cv=3, scoring='accuracy')
scores

[rasbt]

Can you double check

print('y_train.values.shape', y_train.values.shape)

that is is a 1D array as expected?

[skuma17]

its a 1-dimensional array:
y_train.values.shape (870146, 1)

[skuma17]

with y_train=y_train.loc[:,1]

the issue is now resolved.

[rasbt]

Glad that we found the issue!