{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Overview\nThis is an explorative study on NCAA Men’s Datasets. Our focus is on the odd for an underdog to defeat a higher ranking team in NCAA tournaments. Public rating info since 2002-2003 season was provided by Kenneth Massey on the competition page; therefore, this project covers the trounaments from 2003 through 2019. The rankings right before the trounaments, usually on Day 133 since the season beginning, are the fundations to decide which team is an underdog. After data engineering and data creation, I'll create a stacking ensamble model by following Arthur Tok's steps in his famous Notebook(https://www.kaggle.com/arthurtok/introduction-to-ensembling-stacking-in-python).  "},{"metadata":{},"cell_type":"markdown","source":"# Data Engineering\n\nBefore we dive into the amazing modeling magics, feature engineering is always worth the sweats if fruitful outcomes are desired. We are going to exam the list of data sets provided by the site and trying to form the useful features by masagging the data. \n1. Team Data\n2. Season Info\n3. Seeds Info\n4. Regular Season Results: Year by Year, Game Details\n5. Public Rating"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import gc\nimport matplotlib.pylab as plt\nplt.style.use('seaborn-dark-palette')\nimport numpy as np\nimport os\nimport pandas as pd\nfrom sklearn.preprocessing import StandardScaler\n\nDIR = '../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Team data presents the different college teams and each school is uniquely identified by a 4 digit id number."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Team Data\nMTeams = pd.read_csv(f'{DIR}/MDataFiles_Stage1/MTeams.csv')\nprint(MTeams.shape)\nprint(MTeams.isnull().sum())\nMTeams.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Season file identifies when the season began and certain season-level properties."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Season info\nMSeasons = pd.read_csv(f'{DIR}/MDataFiles_Stage1/MSeasons.csv')\nprint(MSeasons.shape)\nprint(MSeasons.isnull().sum())\nMSeasons.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The first letter of Seed indicates which region the team was in and the next two digits tell you the seed within the region. For play-in teams, there is a fourth character (a or b) to further distinguish the seeds, since teams that face each other in the play-in games will have seeds with the same first three characters. The \"a\" and \"b\" are assigned based on which Team ID is lower numerically. Therefore, Variable 'SeedConference' and 'SeedOrder' are created to indicate Conference and Seed respectively.   "},{"metadata":{"trusted":true},"cell_type":"code","source":"#Seeds Info\n#separate the seeds and the conferences\nMNCAATourneySeed = pd.read_csv(f'{DIR}/MDataFiles_Stage1/MNCAATourneySeeds.csv')\nprint(MNCAATourneySeed.shape)\nprint(MNCAATourneySeed.isnull().sum())\nMNCAATourneySeeds = MNCAATourneySeed.merge(MTeams, how = 'left', left_on='TeamID', right_on='TeamID')\nMNCAATourneySeeds['SeedConference'] = 'Region'+MNCAATourneySeeds['Seed'].str.slice(stop=1)\nMNCAATourneySeeds['SeedOrder'] = MNCAATourneySeeds['Seed'].str.slice(start=1, stop=3).astype(int)\nMNCAATourneySeeds.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#double check to see if the seeds are in the rage of 1 and 16.\nMNCAATourneySeeds['SeedOrder'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Regular Seaon file identifies the game-by-game results since the 1985 season. We are going to bring team name to the data set and calculate the win rate of the season for each team."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Regular Season --Team Year by Year\nMRegularSeasonCompactResult = pd.read_csv(f'{DIR}/MDataFiles_Stage1/MRegularSeasonCompactResults.csv')\nMRegularSeasonCompactResults = MRegularSeasonCompactResult.merge(MTeams[['TeamName','TeamID']], how='left', left_on='WTeamID', right_on='TeamID')\\\n                                .drop('TeamID', axis=1)\\\n                                .rename(columns={\"TeamName\":\"WTeamName\"})\\\n                                .merge(MTeams[['TeamName','TeamID']], how='left', left_on='LTeamID', right_on='TeamID')\\\n                                .drop('TeamID', axis=1)\\\n                                .rename(columns={\"TeamName\":\"LTeamName\"})\nfreq_win_yr = MRegularSeasonCompactResults.groupby(['Season','WTeamID','WTeamName'])['WTeamID'].count().sort_values(ascending=False)\nfreq_lose_yr = MRegularSeasonCompactResults.groupby(['Season','LTeamID','LTeamName'])['LTeamID'].count().sort_values(ascending=False)\nMRegularSeasonTeamResultsYr = pd.concat([freq_win_yr, freq_lose_yr], axis=1)\nprint(MRegularSeasonTeamResultsYr.shape)\nMRegularSeasonTeamResultsYr.fillna(0,inplace=True)\nMRegularSeasonTeamResultsYr.index.set_names(['Season','TeamID','TeamName'],inplace=True)\nMRegularSeasonTeamResultsYr.rename(columns={'WTeamID':'win','LTeamID':'loss'}, inplace=True)\nMRegularSeasonTeamResultsYr['compact']=MRegularSeasonTeamResultsYr['win'] + MRegularSeasonTeamResultsYr['loss']\nMRegularSeasonTeamResultsYr['WinRate'] = MRegularSeasonTeamResultsYr['win']/MRegularSeasonTeamResultsYr['compact']\nMRegularSeasonTeamResultsYr.reset_index(inplace=True)\nMRegularSeasonTeamResultsYr.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MRegularSeasonTeamResultsYr.isnull().sum().to_frame(name = 'missing').T","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Regular Season Detail file provides team-level box scores for many regular seasons of historical data, starting with the 2003 season. All games listed in the MRegularSeasonCompactResults file since the 2003 season should exactly be present in the MRegularSeasonDetailedResults file. As you can see, the stats are recorded with 'W' the win team and 'L' the lost team. By rearranging the records, we are going to compute the average number for each metric. We can learn the team's offense and defense capability from those statistics."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Regular Season -- Game Details\nMRegularSeasonDetail = pd.read_csv(f'{DIR}/MDataFiles_Stage1/MRegularSeasonDetailedResults.csv')\nprint(MRegularSeasonDetail.shape)\nMRegularSeasonDetail.head()\nprint(MRegularSeasonDetail.columns)\nwcol = [col for col in MRegularSeasonDetail if (col.startswith('W')) & (col !='WLoc') or (col=='Season')or (col=='DayNum') ]\nprint(wcol)\nlcol = [col for col in MRegularSeasonDetail if col.startswith('L') or (col=='Season')or (col=='DayNum')]\nprint(lcol)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"rename = [w[1:] for w in wcol if w.startswith('W')]\n\nwteam = MRegularSeasonDetail[wcol].copy()\nwteam.columns = ['Season','DayNum']+rename\nwteam['result']='W'\nwteam['LScore']=MRegularSeasonDetail['LScore']\nprint(len(wteam))\n\nlteam = MRegularSeasonDetail[lcol].copy()\nlteam.columns = ['Season','DayNum']+rename\nlteam['result']='L'\nlteam['LScore']=MRegularSeasonDetail['WScore']\nprint(len(lteam))\n\nMRegularSeasonDetails = pd.concat([wteam, lteam])\nprint(len(MRegularSeasonDetails))\n\nMRegularSeasonDetails['FG_avg'] = MRegularSeasonDetails.FGM/MRegularSeasonDetails.FGA\nMRegularSeasonDetails['FG3_avg'] = MRegularSeasonDetails.FGM3/MRegularSeasonDetails.FGA3\nMRegularSeasonDetails['FGM2'] = MRegularSeasonDetails.FGM-MRegularSeasonDetails.FGM3\nMRegularSeasonDetails['FGA2'] = MRegularSeasonDetails.FGA-MRegularSeasonDetails.FGA3\nMRegularSeasonDetails['FG2_avg'] = MRegularSeasonDetails.FGM2/MRegularSeasonDetails.FGA2\nMRegularSeasonDetails['FT_avg'] = MRegularSeasonDetails.FTM/MRegularSeasonDetails.FTA\nMRegularSeasonDetails['TR'] = MRegularSeasonDetails.OR + MRegularSeasonDetails.DR\nMRegularSeasonDetails.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MRegularSeasonTeamBox = MRegularSeasonDetails.groupby(['Season','TeamID']).mean().reset_index()\n\nMRegularSeasonTeamBox.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Public Rating file provides weekly team rankings for dozens of top rating systems - Pomeroy, Sagarin, RPI, ESPN, etc., since the 2002-2003 season. The medians of the latest ratings by the systeams before the tournaments are considered the final rankings.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"#Public Rating\nMMasseyOrdinals = pd.read_csv(f'{DIR}/MDataFiles_Stage1/MMasseyOrdinals.csv')\nMMasseyOrdinals.sort_values(by=['Season', 'TeamID','SystemName','RankingDayNum'], inplace=True)\nMMasseyOrdinals.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"prior_tourney = MMasseyOrdinals.query(\"RankingDayNum <= 133\")\ncomb=prior_tourney.groupby(['Season','TeamID','SystemName']).size().reset_index().rename(columns={0:'count'})\ncomb.shape\nmax_rankingdaynum = prior_tourney.groupby(['Season','TeamID','SystemName']).agg({'RankingDayNum':'max'}).reset_index()\nmax_rankingdaynum.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MMasseyOrdinalsPriorTourney = prior_tourney.merge(max_rankingdaynum, how='inner', left_on=['Season','TeamID','SystemName','RankingDayNum'], right_on=['Season','TeamID','SystemName','RankingDayNum'])\nprint(MMasseyOrdinalsPriorTourney.shape) #has to have 307393 combos\nMMasseyOrdinalsPriorTourney.query('TeamID==1102 & Season==2003')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MMasseyOrdinalsMedian = MMasseyOrdinalsPriorTourney.groupby(['Season','TeamID'])['OrdinalRank'].median().reset_index()\nprint(MMasseyOrdinalsMedian.shape)\nMMasseyOrdinalsMedian.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Creation\n\nOur goal of this section is to form the final datasets to train our models. First, we look at the dependent variable, which is the results of the tournaments. Next, the underdogs are decided by their ranking medians. After the fields are concatenated, standardization is conducted to make the numeric attributes have a 0 mean and unit variance. Lastly, the data before year 2015 is treated as the traing set and year 2015-2019 is the test set.\n\n1. Tournament Results\n2. Underdogs\n3. Training/Test sets\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Tournaments\nMNCAATourneyCompactResult = pd.read_csv(f'{DIR}/MDataFiles_Stage1/MNCAATourneyCompactResults.csv')\nprint(MNCAATourneyCompactResult.shape)\nprint(MNCAATourneyCompactResult.isnull().sum())\nMNCAATourneyCompactResults = MNCAATourneyCompactResult.merge(MTeams[['TeamName','TeamID']], how='left', left_on='WTeamID', right_on='TeamID')\\\n                                .drop('TeamID', axis=1)\\\n                                .rename(columns={\"TeamName\":\"WTeamName\"})\\\n                                .merge(MTeams[['TeamName','TeamID']], how='left', left_on='LTeamID', right_on='TeamID')\\\n                                .drop('TeamID', axis=1)\\\n                                .rename(columns={\"TeamName\":\"LTeamName\"})\nMNCAATourneyCompactResults['Diff_Score'] = MNCAATourneyCompactResults['WScore'] - MNCAATourneyCompactResults['LScore']\nMNCAATourneyCompactResults.sort_values(by='Diff_Score', ascending=False, inplace=True)\nMNCAATourneyCompactResults.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# decide the underdogs and collect the fields\nMNCAATourneyCompactResults.sort_values(by=['Season','DayNum'], inplace=True)\n\nMNCAATourney_ = MNCAATourneyCompactResults.merge(MMasseyOrdinalsMedian, how='left', left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'])\nMNCAATourney = MNCAATourney_.merge(MMasseyOrdinalsMedian, how='left', left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'])\nMNCAATourney.rename(columns={'OrdinalRank_x':'WTeamRank','OrdinalRank_y':'LTeamRank', 'TeamID_x':'T1','TeamID_y':'T2'}, inplace=True)\nMNCAATourney.loc[MNCAATourney['WTeamRank'] < MNCAATourney['LTeamRank'], 'T1']=MNCAATourney['LTeamID']\nMNCAATourney.loc[MNCAATourney['WTeamRank'] < MNCAATourney['LTeamRank'], 'T2']=MNCAATourney['WTeamID']\nMNCAATourney['label'] = np.where(MNCAATourney['T1']==MNCAATourney['WTeamID'], 1,0)\nprint(MNCAATourney.shape)\nMNCAATourney.tail()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MNCAATourney[MNCAATourney.Season>=2003].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def gen_TeamBoxDict(tag):\n    TeamBoxDict = { k:tag+v for (k,v) in zip(MRegularSeasonTeamBox.columns, MRegularSeasonTeamBox.columns) if (k != 'Season' and k != 'DayNum')}  \n    return(TeamBoxDict)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def grab_col(tag,dataframe):\n    df = dataframe.merge(MMasseyOrdinalsMedian, how='left', left_on=['Season', tag], right_on=['Season', 'TeamID'])\\\n        .rename(columns={'OrdinalRank':tag+'OrdinalRank'})\\\n        .merge(MNCAATourneySeeds[['Season', 'TeamID','SeedConference','SeedOrder']], how='left', left_on=['Season', tag], right_on=['Season', 'TeamID'])\\\n        .rename(columns={'SeedConference':tag+'SeedConference','SeedOrder':tag+'SeedOrder'})\\\n        .merge(MRegularSeasonTeamResultsYr[['Season','TeamID','WinRate']], how='left', left_on=['Season',tag], right_on=['Season','TeamID'])\\\n        .rename(columns={'WinRate':tag+'SeasonWinRate'})\\\n        .merge(MRegularSeasonTeamBox, how='left', left_on=['Season', tag], right_on=['Season', 'TeamID'])\\\n        .rename(columns=gen_TeamBoxDict(tag))\\\n        .rename(columns={'DayNum_x':'DayNum'})\n    df.drop(columns=[col for col in df if col.startswith('TeamID')], inplace=True)\n    df.drop(columns='DayNum_y', inplace=True)\n    return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_ = MNCAATourney.query('Season >= 2003')[['label','Season','DayNum','T1','T2']]\ndf1 = grab_col('T1',df_)\ndf2 = grab_col('T2',df_)\ndf = df1.merge(df2, how='inner',left_on=['label','Season','DayNum','T1','T2'],right_on=['label','Season','DayNum','T1','T2'])\ndf['OrdinalRankDiff']=df['T2OrdinalRank'] - df['T1OrdinalRank']\ndf['SeedOrderDiff']=df['T2SeedOrder'] - df['T1SeedOrder']\ndf.drop(columns={'T1SeedConference','T2SeedConference'}, inplace=True)\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.set_option('display.max_rows', df.shape[0]+1)\ndf.isnull().sum().to_frame(name = 'missing')\npd.set_option('display.max_rows', 5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# separate traning and test sets\nsc = StandardScaler()\ndf_training_ = df.loc[df.Season < 2015, df.columns != 'label']\nlabel_training = df.loc[df.Season < 2015,'label']\ndf_training_.set_index(['Season','DayNum','T1','T2'], inplace=True)\ndf_training=pd.DataFrame(sc.fit_transform(df_training_),columns = df_training_.columns)\ndf_training.head()\ndf_training.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_test_ = df.loc[df.Season >= 2015, df.columns != 'label']\nlabel_test = df.loc[df.Season >= 2015,'label']\ndf_test_.set_index(['Season','DayNum','T1','T2'], inplace=True)\ndf_test = pd.DataFrame(sc.transform (df_test_),columns = df_test_.columns)\ndf_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#check the defeat rate in the training data\nlabel_training.value_counts(normalize=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predictive Modeling\nThe following codes are inspired by Arthor Tok's notebook on Titanic Competition. This is a stacking ensamble model, including two stages: basic classification and prediction. First, we use basic classifiers to perform the predictions. Those predictions will be the new set of features to train the next classifier.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"import sklearn\nfrom sklearn.ensemble import (RandomForestClassifier, AdaBoostClassifier, GradientBoostingClassifier, ExtraTreesClassifier)\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.svm import SVC\nfrom sklearn.model_selection import KFold\nimport xgboost as xgb","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Arthor Tok wrote a class SklearnHelper that allows one to extend the inbuilt methods and the classifiers will action the same way on each methods. Setting seeds is to ensure the results will stay the same each time we run the codes. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Some useful parameters which will come in handy later on\nntrain = df_training.shape[0]\nntest = df_test.shape[0]\nSEED = 20200325 # for reproducibility\nNFOLDS = 5 # set folds for out-of-fold prediction\nkf = KFold(n_splits=NFOLDS, random_state=100, shuffle=True)\n\n# Class to extend the Sklearn classifier\nclass SklearnHelper(object):\n    def __init__(self, clf, seed=0, params=None):\n        params['random_state'] = seed\n        self.clf = clf(**params)\n\n    def train(self, x_train, y_train):\n        self.clf.fit(x_train, y_train)\n\n    def predict(self, x):\n        return self.clf.predict(x)\n    \n    def fit(self,x,y):\n        return self.clf.fit(x,y)\n    \n    def feature_importances(self,x,y):\n        print(self.clf.fit(x,y).feature_importances_)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Define a method to perform k fold cross validation, conduct the training, and make predictions of the test set on each iteration."},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_oof(clf, x_train, y_train, x_test):\n    oof_train = np.zeros((ntrain,))\n    oof_test = np.zeros((ntest,))\n    oof_test_skf = np.empty((NFOLDS, ntest))\n    \n    for i, (train_index, test_index) in enumerate(kf.split(oof_train)):\n        x_tr = x_train[train_index]\n        y_tr = y_train[train_index]\n        x_te = x_train[test_index]\n\n        clf.train(x_tr, y_tr)\n\n        oof_train[test_index] = clf.predict(x_te)\n        oof_test_skf[i, :] = clf.predict(x_test)\n\n    oof_test[:] = oof_test_skf.mean(axis=0)\n    return oof_train.reshape(-1, 1), oof_test.reshape(-1, 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There will be 6 basic classifiers to contribute in the 1st stage classification.\n1. Logistic Regression classifier\n2. Support Vector Machine\n3. Random Forest classifier\n4. Extra Trees classifier\n5. AdaBoost classifer\n6. Gradient Boosting classifer"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Put in our parameters for said classifiers\n# Random Forest parameters\n\n# Logistic Regression \nlr_params = {'C': 1}\n\n# Support Vector \nsvc_params = {\n    'kernel' : 'linear',\n    'C' : 1\n    }\n\n# Random Forest\nrf_params = {\n    'n_jobs': -1,\n    'n_estimators': 500,\n    'warm_start': False, \n    'max_depth': 6,\n    'min_samples_leaf': 2,\n    'max_features' : 'sqrt',\n    'verbose': 0\n}\n\n# Extra Trees\net_params = {\n    'n_jobs': -1,\n    'n_estimators':500,\n    'max_depth': 8,\n    'min_samples_leaf': 2,\n    'verbose': 0\n}\n\n# AdaBoost\nada_params = {\n    'n_estimators': 500,\n    'learning_rate' : 0.75\n}\n\n# Gradient Boosting\ngb_params = {\n    'n_estimators': 500,\n     #'max_features': 0.2,\n    'max_depth': 5,\n    'min_samples_leaf': 2,\n    'verbose': 0\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create 6 objects\nrf = SklearnHelper(clf=RandomForestClassifier, seed=SEED, params=rf_params)\net = SklearnHelper(clf=ExtraTreesClassifier, seed=SEED, params=et_params)\nada = SklearnHelper(clf=AdaBoostClassifier, seed=SEED, params=ada_params)\ngb = SklearnHelper(clf=GradientBoostingClassifier, seed=SEED, params=gb_params)\nsvc = SklearnHelper(clf=SVC, seed=SEED, params=svc_params)\nlr = SklearnHelper(clf=LogisticRegression, seed=SEED, params=lr_params)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create Numpy arrays of train, test and target ( Survived) dataframes to feed into our models\ny_train = label_training.ravel()\nx_train = df_training.values # Creates an array of the train data\nx_test = df_test.values # Creats an array of the test data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create our OOF train and test predictions. These base results will be used as new features\nlr_oof_train, lr_oof_test = get_oof(lr,x_train, y_train, x_test) # Logistic Regression Classifier\nsvc_oof_train, svc_oof_test = get_oof(svc,x_train, y_train, x_test) # Support Vector Classifier\nrf_oof_train, rf_oof_test = get_oof(rf,x_train, y_train, x_test) # Random Forest\net_oof_train, et_oof_test = get_oof(et, x_train, y_train, x_test) # Extra Trees\nada_oof_train, ada_oof_test = get_oof(ada, x_train, y_train, x_test) # AdaBoost \ngb_oof_train, gb_oof_test = get_oof(gb,x_train, y_train, x_test) # Gradient Boost\n\nprint(\"Training is complete\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"base_predictions_train = pd.DataFrame( {\n    'LR': lr_oof_train.ravel(),\n    'SVM': svc_oof_train.ravel(),\n    'RandomForest': rf_oof_train.ravel(),\n    'ExtraTrees': et_oof_train.ravel(),\n    'AdaBoost': ada_oof_train.ravel(),\n    'GradientBoost': gb_oof_train.ravel()\n    })\nbase_predictions_train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We call an XGBClassifier and fit it to the first-level train and target data and use the learned model to predict the test data."},{"metadata":{"trusted":true},"cell_type":"code","source":"#form 2nd stage training/test sets\nx_train = np.concatenate(( et_oof_train, rf_oof_train, ada_oof_train, gb_oof_train, svc_oof_train, lr_oof_train), axis=1)\nx_test = np.concatenate(( et_oof_test, rf_oof_test, ada_oof_test, gb_oof_test, svc_oof_test, lr_oof_test), axis=1)\n\n#conduct 2nd level learning model via XGBoost\ngbm = xgb.XGBClassifier(\n    #learning_rate = 0.02,\n     n_estimators= 2000,\n     max_depth= 4,\n     min_child_weight= 2,\n     #gamma=1,\n     gamma=0.9,                        \n     subsample=0.8,\n     colsample_bytree=0.8,\n     objective= 'binary:logistic',\n     nthread= -1,\n     scale_pos_weight=1).fit(x_train, y_train)\npredictions = gbm.predict(x_test)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, the accuracy of the stacking ensamble model is around 74%."},{"metadata":{"trusted":true},"cell_type":"code","source":"gbm.score(x_test, label_test)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}