{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Out Of Fold Meta-Model Stacking Pipeline 🥳\nThis notebook is a continuation of my [previous notebook](https://www.kaggle.com/amoghjrules/intro-to-stacking-averaging-base-models).\nI use a custom dataset here. If you want to visualise how it was made, checkout my notebook [here](https://www.kaggle.com/amoghjrules/encode-like-there-s-no-tomorrow)"},{"metadata":{},"cell_type":"markdown","source":"### What are out of fold predictions?\nThe main idea behind the structure of a stacked generalization is to use one or more first level models, make predictions using these models and then use these predictions as features to fit one or more second level models on top. To avoid overfitting, we use cross-validation to predict the OOF (out-of-fold) part of the training set. [source](https://towardsdatascience.com/automate-stacking-in-python-fc3e7834772e)\n\n![image](https://i.postimg.cc/T32XpjhC/stacking2.png)"},{"metadata":{},"cell_type":"markdown","source":"## Importing pre-encoded data from my public dataset\n\nSince our data is pre-encoded we can start working on it right away "},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\ntest = pd.read_csv(\"../input/catindat2encoded/test_encoded.csv\")\ntrain = pd.read_csv(\"../input/catindat2encoded/train_encoded.csv\")\ntest_id = pd.read_csv(\"../input/cat-in-the-dat-ii/sample_submission.csv\")['id']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"target = train['target']\ntrain.drop(['target'], axis = 1, inplace = True)\n# test.drop(['id'], axis = 1, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Importing libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import StratifiedKFold, KFold, cross_validate\nfrom sklearn.linear_model import LogisticRegression, ElasticNet, SGDClassifier\nfrom sklearn.ensemble import GradientBoostingClassifier, AdaBoostClassifier","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Defining the function which will be our pipeline\nHere we build a out of fold training set and a testing set \n\nDividing the trainset using a stratified kfold split, we are generating a train set. Next, we generate a test set which consists of the predictions of the pre-existing models"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"def generate_oof_trainset( train, test, target, strat_kfold, models,):\n\n    oof_train = pd.DataFrame() # Initializing empty data frame\n    \n    count = 0\n    print(train.shape, target.shape)\n\n    for train_id, test_id in strat_kfold.split(train, target):\n        count += 1\n        print(\"Current fold number is :\", count)\n        xtrain, xtest = train.iloc[train_id], train.iloc[test_id]\n        ytrain, ytest = target.iloc[train_id], target.iloc[test_id]\n        \n        curr_split = [None]*(len(models)+1) # Initializing list of lists to save all predictions for a split from all models for the current split\n        \n        for i in tqdm(range(len(models))):\n            \n            model = models[i]\n            model.fit(xtrain, ytrain)\n            \n            curr_split[i] = model.predict_proba(xtest)[:,1]      \n            \n        curr_split[-1] = ytest\n        oof_train = pd.concat([oof_train,pd.DataFrame(curr_split).T], ignore_index= True)\n    \n    oof_test = [None]*len(models)\n    for i, model in enumerate(models):\n        model.fit( train, target)\n        oof_test[i] = model.predict_proba(test)[:,1]\n    oof_test = pd.DataFrame(oof_test)\n    return oof_train, oof_test\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we fit the generated trainset and perform cross validation."},{"metadata":{"trusted":true},"cell_type":"code","source":"from tqdm import tqdm\nstrat_kfold = StratifiedKFold( n_splits =2, shuffle = \n              True)\n\nlog_reg = LogisticRegression(random_state = 0)\ngbr = GradientBoostingClassifier(\n        max_depth=6,\n        n_estimators=35,\n        warm_start=False,\n        random_state=42)\nadar = AdaBoostClassifier(n_estimators=100, random_state=0)\n\nmodels = [ log_reg, gbr, adar ]\ntrain_generated, test_generated = generate_oof_trainset( train, test, target, strat_kfold, models)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then we fit the meta classifier onto the generated testset"},{"metadata":{"trusted":true},"cell_type":"code","source":"lr_clf = LogisticRegression()\ntarget = train_generated[train_generated.columns[-1]]\ntrain_generated.drop([train_generated.columns[-1]], axis = 1 , inplace = True)\n\ncv_results = cross_validate(lr_clf,\n                            train_generated.values,\n                            target.values,\n                            cv = 3,\n                            scoring = 'roc_auc',\n                            verbose = 1,\n                            return_train_score = True,\n                            return_estimator = True)\n\nprint(\"Fit time :\", cv_results['fit_time'].sum(),\"secs\")\nprint(\"Score time :\", cv_results['score_time'].sum(),\"secs\")\nprint(\"Train score :\", cv_results['train_score'].mean())\nprint(\"Test score :\", cv_results['test_score'].mean())   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lr_clf.fit(train_generated.values, target.values)\npreds = lr_clf.predict(test_generated.T.values)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## If you learnt something from this notebook, do upvote the kernel 😁"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}