{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"## Regarding messed up leader board, a mistake I made and how to make your validation score and learder board score correctly reflect your model \nThis kernel is written due to a wrong answer I gave to a comment under another kaggler's kernel. So to make it right and cosidering that I was also lost in the same wrong path for a while, I think it's necessary to show where is the problem, what it can cause and how to properly handle it.  \n\nTo be brief, there are leakage problems in some kernels here. And I think this is part of the reason there are some god like score in leader board right now. This causes some obstacle in evaluate the performance of model. Below is the layout of this kernel:  \n* [Data Prep](#dp)\n* [Issue](#i)\n* [Comparison Code](#cc)\n* [Conclusion](http://)"},{"metadata":{},"cell_type":"markdown","source":"# <a name =\"dp\"></a>Data Prep"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom sklearn.model_selection import KFold,StratifiedKFold\nfrom sklearn.metrics import roc_auc_score, log_loss\nimport lightgbm as lgb\nimport string","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '../input/google-cloud-ncaa-march-madness-2020-division-1-womens-tournament/'\nregular_result_brief=pd.read_csv(path+'WDataFiles_Stage1/WRegularSeasonCompactResults.csv')\ntour_result_brief=pd.read_csv(path+'WDataFiles_Stage1/WNCAATourneyCompactResults.csv')\ntour_seeds=pd.read_csv(path+'WDataFiles_Stage1/WNCAATourneySeeds.csv')\nseasons=pd.read_csv(path+'WDataFiles_Stage1/WSeasons.csv')\nsub_test=pd.read_csv(path+'WSampleSubmissionStage1_2020.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"regular_result_brief.drop(['WLoc','NumOT'],axis=1,inplace=True)\ntour_result_brief.drop(['WLoc','NumOT'],axis=1,inplace=True)\nrwin=regular_result_brief[['Season', 'DayNum', 'WTeamID', 'WScore']]\nrwin.columns=['Season', 'DayNum', 'TeamID', 'Score']\nrlos=regular_result_brief[['Season', 'DayNum', 'LTeamID', 'LScore']]\nrlos.columns=['Season', 'DayNum', 'TeamID', 'Score']\nregular_score=pd.concat((rwin,rlos)).reset_index(drop=True)\nregular_score=regular_score.groupby(['Season','TeamID'],as_index=False).mean()\nregular_score.head(7)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tour_seeds['Seed']=tour_seeds['Seed'].apply(lambda x: int(x[1:]))\ndef myMerge(indf):\n    df=indf[['Season', 'WTeamID', 'LTeamID']]\n    df=df.merge(tour_seeds,left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'], how='left')\n    df.rename(columns={'Seed':'WSeed'}, inplace=True)\n    df = df.drop('TeamID', axis=1)\n    df = df.merge(tour_seeds, left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], how='left')\n    df.rename(columns={'Seed':'LSeed'}, inplace=True)\n    df = df.drop('TeamID', axis=1)\n    df = pd.merge(df, regular_score, left_on=['Season', 'WTeamID'], right_on=['Season', 'TeamID'], how='left')\n    df.rename(columns={'Score':'WScore','DayNum':'WDayNum'}, inplace=True)\n    df = df.drop(['TeamID'], axis=1)\n    df = pd.merge(df, regular_score, left_on=['Season', 'LTeamID'], right_on=['Season', 'TeamID'], how='left')\n    df.rename(columns={'Score':'LScore','DayNum':'LDayNum'}, inplace=True)\n    df = df.drop(['TeamID'], axis=1)\n    return df\ntour_merge=myMerge(tour_result_brief)\ntour_merge.head(7)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tour_win=tour_merge.copy()\ntour_los=tour_merge.copy()\ntour_win.columns=[c.replace('W','T1_').replace('L','T2_') for c in tour_win.columns]\ntour_los.columns=[c.replace('W','T2_').replace('L','T1_') for c in tour_los.columns]\ntour_win['Pred']=1\ntour_los['Pred']=0\ntour_all=pd.concat((tour_win, tour_los),sort=True).reset_index(drop=True)\ntour_all","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_test['Season'] = sub_test['ID'].map(lambda x: int(x[:4]))\nsub_test['WTeamID'] = sub_test['ID'].map(lambda x: int(x[5:9]))\nsub_test['LTeamID'] = sub_test['ID'].map(lambda x: int(x[10:14]))\ntest=myMerge(sub_test)\ntest.columns=[c.replace('W','T1_').replace('L','T2_') for c in test.columns]\ntest.head(7)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_dungeon=['ID', 'Pred', 'T1_TeamID', 'T2_TeamID']\nfeatures=[c for c in tour_all.columns if c not in feature_dungeon]\ntarget=tour_all.Pred\ntrain=tour_all[features]\ntest=test[features]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('train shape: {}, test shape: {}'.format(train.shape,test.shape))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# <a name =\"i\"></a>Issue\nUp until now every thing is fine. Then some kernel did a n fold cross validation on this entire taining set and get the prediction from test set. Using this procedure and the data above, my submission can get about 0.268 on leader board. But if you look back the competition in 2018 and 2019, the leader score 0.4 and 0.32 respectively.  \nThe reason that I have a low log loss here is that some of the test data have duplication in training set. Due to the same reason, another thing you can observe is that your leader board score have a strong link with the training loss instead validation loss. Also The other thing you can see is training log loss and leader board score excitedly react to the increase of number of folds, whereas validation loss doesn't move much."},{"metadata":{},"cell_type":"markdown","source":"# <a name =\"cc\"></a>Comparision Code\nBelow is the similar code in some kenel. Here I use folds=100 to save time(I have seen even larger number) "},{"metadata":{"trusted":true},"cell_type":"code","source":"params = {\n        'learning_rate': 0.01,\n        'feature_fraction': 0.5,\n        'min_data_in_leaf' : 12,\n        'max_depth': 8,\n        'objective': 'binary',\n        'metric': 'binary_logloss',\n        'n_jobs': -1,\n        'feature_fraction_seed': 42,\n        'bagging_seed': 42,\n        'boosting_type': 'gbdt',\n        'verbosity': -1,\n        'is_unbalance': False,\n        'boost_from_average': False}\ntraintion = np.zeros(len(train))\nvalidation = np.zeros(len(train))\npredictions = np.zeros(len(test))\nfeature_importance_df = pd.DataFrame()\nfolds = KFold(n_splits=100, shuffle=True, random_state=1990)\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(train)):\n    print(\"fold n°{}\".format(fold_))\n    train_x=train.iloc[trn_idx].reset_index(drop=True)\n    valid_x=train.iloc[val_idx].reset_index(drop=True)\n    target_train=target.iloc[trn_idx].reset_index(drop=True)\n    target_valid=target.iloc[val_idx].reset_index(drop=True)\n    trn_data = lgb.Dataset(train_x,\n                           label=target_train,\n                          )\n    val_data = lgb.Dataset(valid_x,\n                           label=target_valid,\n                          )\n\n    num_round = 1000000\n    clf = lgb.train(params,\n                    trn_data,\n                    num_round,\n                    valid_sets = [trn_data, val_data],\n                    verbose_eval=False,\n                    early_stopping_rounds = 1000)\n    traintion[trn_idx] += clf.predict(train_x, num_iteration=clf.best_iteration)/(folds.n_splits-1)\n    validation[val_idx] = clf.predict(valid_x, num_iteration=clf.best_iteration)\n    \n    fold_importance_df = pd.DataFrame()\n    fold_importance_df[\"feature\"] = features\n    fold_importance_df[\"importance\"] = clf.feature_importance(importance_type='gain')\n    fold_importance_df[\"fold\"] = fold_ + 1\n    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n    \n    predictions += clf.predict(test, num_iteration=clf.best_iteration) / folds.n_splits\nprint(\"Train AUC score: {:<8.5f}\".format(log_loss(target,traintion)))\nprint(\"Valid AUC score: {:<8.5f}\".format(log_loss(target,validation)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# <a name =\"c\"></a>Conclusion\nThe solution is simple. Just using data from previous seasons to predict the season of interest. For example, using data from 2010-2014 seasons in training set build model and predict season 2015 and put it in submission file, using 2010-2015 to do 2016 so on so forth.\nThen you will find your validation score makes much sense with respect to your leader board score. \nAnd if you want to compare your model to others and have an idea of how it is, you can reference previous competitions, since current public leader board is a bit messed up.  \n[2018 leader board](https://www.kaggle.com/c/womens-machine-learning-competition-2018/leaderboard)  \n[2019 leader board](https://www.kaggle.com/c/womens-machine-learning-competition-2019/leaderboard)\n\n#### Hope this post helped you! "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}