{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!pip install BorutaShap","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:03.077627Z","iopub.execute_input":"2022-05-15T19:45:03.078126Z","iopub.status.idle":"2022-05-15T19:45:14.367961Z","shell.execute_reply.started":"2022-05-15T19:45:03.078031Z","shell.execute_reply":"2022-05-15T19:45:14.367104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Introduction","metadata":{}},{"cell_type":"markdown","source":"Football is the most popular sport around the world. It is a team sport played with a spherical ball between two teams of 11 players. The game is played on a rectangular field called a pitch with a goal at each end. The objective of the game is to score more goals than the opposition by moving the ball beyond the goal line into the opposing goal, usually within a time frame of 90 or more minutes. ","metadata":{}},{"cell_type":"markdown","source":"## Metrics for the project","metadata":{}},{"cell_type":"markdown","source":"The goal of the competition is to predict the probabilities for each of the target classes - home, away and draw. The metrics for evaluation for the project is the multinomial log loss which measures the quality of the predicted probabilities. ","metadata":{}},{"cell_type":"markdown","source":"## Preparing the environment","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-05-15T19:45:14.370795Z","iopub.execute_input":"2022-05-15T19:45:14.371061Z","iopub.status.idle":"2022-05-15T19:45:15.321914Z","shell.execute_reply.started":"2022-05-15T19:45:14.371025Z","shell.execute_reply":"2022-05-15T19:45:15.321205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv('../input/football-match-probability-prediction/train.csv')\ntest = pd.read_csv('../input/football-match-probability-prediction/test.csv')\ntarget_scores = pd.read_csv('../input/football-match-probability-prediction/train_target_and_scores.csv')\n\ntrain.shape, test.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:15.323045Z","iopub.execute_input":"2022-05-15T19:45:15.323309Z","iopub.status.idle":"2022-05-15T19:45:24.600276Z","shell.execute_reply.started":"2022-05-15T19:45:15.323275Z","shell.execute_reply":"2022-05-15T19:45:24.599435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploratory Data Analysis","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:24.602490Z","iopub.execute_input":"2022-05-15T19:45:24.602858Z","iopub.status.idle":"2022-05-15T19:45:24.642292Z","shell.execute_reply.started":"2022-05-15T19:45:24.602815Z","shell.execute_reply":"2022-05-15T19:45:24.641637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see the distribution of the target we're trying to predict","metadata":{}},{"cell_type":"code","source":"train['target'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:24.643407Z","iopub.execute_input":"2022-05-15T19:45:24.643796Z","iopub.status.idle":"2022-05-15T19:45:24.674021Z","shell.execute_reply.started":"2022-05-15T19:45:24.643759Z","shell.execute_reply":"2022-05-15T19:45:24.673077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(x='target', data=train).set(xlabel='Target', ylabel='Density')","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:24.675829Z","iopub.execute_input":"2022-05-15T19:45:24.676201Z","iopub.status.idle":"2022-05-15T19:45:25.000754Z","shell.execute_reply.started":"2022-05-15T19:45:24.676162Z","shell.execute_reply":"2022-05-15T19:45:25.000040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we see from above, we have three target values - away, home and draw. home team means the home team won, away means the away team won and draw means they had a tie in the game. \n\nLet's see the kind of league games this dataset contains","metadata":{}},{"cell_type":"code","source":"train['league_name'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:25.002121Z","iopub.execute_input":"2022-05-15T19:45:25.002377Z","iopub.status.idle":"2022-05-15T19:45:25.028078Z","shell.execute_reply.started":"2022-05-15T19:45:25.002345Z","shell.execute_reply":"2022-05-15T19:45:25.027348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Majority of the matches are from Premier League, Club Friendlies and Primera Division","metadata":{}},{"cell_type":"markdown","source":"### Functions to help with EDA","metadata":{}},{"cell_type":"markdown","source":"Let's write some functions to help with the process of exploratory data analysis","metadata":{}},{"cell_type":"code","source":"def missing_values_table(df):\n    \"\"\"Returns a dataframe of number of missing entries per column in df\"\"\"\n    miss_val = df.isnull().sum()\n    miss_val_percent = (100 * df.isnull().sum())/len(df)\n    miss_val_table = pd.concat([miss_val, miss_val_percent], axis=1)\n    miss_val_table = miss_val_table.rename(columns={0: 'Missing values', 1: 'Percent of missing values'})\n    \n    # sort by percent of missing values for only cols with missing values\n    miss_val_table = miss_val_table[miss_val_table.iloc[:, 1] != 0].sort_values(by='Percent of missing values', ascending=False).round(1)\n    print(f'The dataframe has {str(df.shape[1])} columns.\\nThere are {str(miss_val_table.shape[0])} columns with missing values')\n    return miss_val_table","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:25.029528Z","iopub.execute_input":"2022-05-15T19:45:25.030000Z","iopub.status.idle":"2022-05-15T19:45:25.038573Z","shell.execute_reply.started":"2022-05-15T19:45:25.029962Z","shell.execute_reply":"2022-05-15T19:45:25.037788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def numeric_distribution_plot(df):\n    \"\"\"Plots a seaborn displot for all numeric features in the dataset\"\"\"\n    num_features = df.select_dtypes(exclude='object').columns # only include numeric features\n    print(f\"There are {len(num_features)} numeric features in the dataset\")\n    for feature in num_features[:20]: # restrics the plot to only 20 cols\n        plt.figure(figsize=(12, 5))\n        sns.displot(data=df, x=feature)\n        plt.show()\n\ndef categorical_feature_plot(df):\n    \"\"\"Plots a countplot for all categorical features present in the dataframe\"\"\"\n    cat_features = df.select_dtypes(include='object').columns\n    print(f\"There are {len(cat_features)} categorical columns in df\")\n    for feature in cat_features:\n        plt.figure(figsize=(12,5))\n        sns.countplot(x = feature, data=df)\n        plt.tight_layout()\n        plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:25.040165Z","iopub.execute_input":"2022-05-15T19:45:25.040511Z","iopub.status.idle":"2022-05-15T19:45:25.049164Z","shell.execute_reply.started":"2022-05-15T19:45:25.040472Z","shell.execute_reply":"2022-05-15T19:45:25.048405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"missing_values_table(train)[:20]","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:25.053560Z","iopub.execute_input":"2022-05-15T19:45:25.053841Z","iopub.status.idle":"2022-05-15T19:45:25.703494Z","shell.execute_reply.started":"2022-05-15T19:45:25.053781Z","shell.execute_reply":"2022-05-15T19:45:25.702592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Reading from the documentation of the dataset, we have two types of features in the train dataset - descriptive features and historical features. The descriptive features are point-in-time descriptions of the match that has to be predicted. Historical features contain past information of the 10 games previously played by the Home and the Away team. The historical features contain the keyword ```history``` in them. \n\nIn the dataframe above for the missing values, we see that most of these features are historical features. Let's see a few of them","metadata":{}},{"cell_type":"code","source":"temp = train[['away_team_history_coach_10', 'home_team_history_coach_10', 'away_team_history_coach_9', 'home_team_history_coach_9']]\ntemp.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:25.704990Z","iopub.execute_input":"2022-05-15T19:45:25.705966Z","iopub.status.idle":"2022-05-15T19:45:25.722076Z","shell.execute_reply.started":"2022-05-15T19:45:25.705918Z","shell.execute_reply":"2022-05-15T19:45:25.721106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The features above refer to the coach id of the home & away teams respectively in the 10th and 9th teams.\nOther descriptions of the historical data present in the dataset...\n\n* ```home_team_history_match_date_{i}``` - date of the last ith match played by home\n* ```home_team_history_is_play_home_{i}``` - if 1, the home team played home\n* ```home_team_history_is_cup_{i}``` - If 1, the match was a cup competition.\n* ```home_team_history_goal_{i}``` - The number of goals scored by the Home team on its last i-th match.\n* ```home_team_history_opponent_goal_{i}``` - The number of goals conceded by the Home team on its last i-th match.\n* ```home_team_history_rating_{i}``` - The rating of the Home team on its last i-th match (pre match rating).\n* ```home_team_history_opponent_rating_{i}``` - The rating of the opponent team on Home team last i-th match (pre match rating).\n* ```home_team_history_coach_{i}``` - The coach id of the Home team on its last i-th match.\n* ```home_team_history_league_id_{i}``` - The league name id by the Home team on its last i-th match.\n\nThe feature definitions refer to the away team if the prefix home is replaced with away. \n\nFor each of the 9 features above, we have 9 more for the away teams and each of the features are available for the 10 matches in the dataset. Based on this, we're expecting to have 18 x 10 = 180 historical features.\n\nFrom the competition documentation on the ratings: 'ratings are calculated by Octosport and correspond to the pre-match team's relative strength. It is provided for all 10 historical matches. But, we do not provide the team's rating for the match you have to predict. The reason being these ratings will already contain the winning probabilities information'.\n\nLet's see some of these features to gain a concrete idea of what they mean... Let i = 1, the most recent match\n ","metadata":{}},{"cell_type":"code","source":"temp = train[['home_team_name', 'away_team_name', 'home_team_history_match_date_1', 'away_team_history_match_date_1', 'home_team_history_is_play_home_1',\n              'home_team_history_is_cup_1', 'home_team_history_goal_1', 'home_team_history_opponent_goal_1', 'home_team_history_rating_1', 'home_team_history_opponent_rating_1',\n             'home_team_history_coach_1', 'home_team_history_league_id_1']]\ntemp.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:25.723602Z","iopub.execute_input":"2022-05-15T19:45:25.724011Z","iopub.status.idle":"2022-05-15T19:45:25.754877Z","shell.execute_reply.started":"2022-05-15T19:45:25.723968Z","shell.execute_reply":"2022-05-15T19:45:25.754043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"temp.info()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:25.756423Z","iopub.execute_input":"2022-05-15T19:45:25.756726Z","iopub.status.idle":"2022-05-15T19:45:25.822631Z","shell.execute_reply.started":"2022-05-15T19:45:25.756686Z","shell.execute_reply":"2022-05-15T19:45:25.821830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From this we can see that the date columns in the dataframe are in type object, we'll need to convert them to datetime type for future work.\n\nFor now, let's look at the descriptive features. They should be 10\n\n* ```id``` - the id of the game\n* ```target``` - The variable you have to predict the probabilities only available in the train set.\n* ```home_team_name``` - The name of the Home the team. Hidden in test set, see this discussion\n* ```away_team_name``` - The name of the Away the team. Hidden in test set, see this discussion\n* ```match_date``` - The match date (UTC).\n* ```league_name``` - The league name.\n* ```league_id``` - The league id. Note that league names can be identical for two differents id.\n* ```is_cup``` - If the value is 1 the match is played for a cup compettion.\n* ```home_team_coach_id``` - The id of the Home team coach.\n* ```away_team_coach_id``` - The id of the Away team coach.\n","metadata":{}},{"cell_type":"code","source":"temp = train[['id', 'target', 'home_team_name', 'away_team_name', 'match_date', 'league_name', 'league_id', \n             'is_cup', 'home_team_coach_id', 'away_team_coach_id']]\ntemp.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:25.823715Z","iopub.execute_input":"2022-05-15T19:45:25.824010Z","iopub.status.idle":"2022-05-15T19:45:25.848058Z","shell.execute_reply.started":"2022-05-15T19:45:25.823969Z","shell.execute_reply":"2022-05-15T19:45:25.847229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see if the test dataset has similar proportion of missing entries per column as test,","metadata":{}},{"cell_type":"code","source":"missing_values_table(test)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:25.849344Z","iopub.execute_input":"2022-05-15T19:45:25.850205Z","iopub.status.idle":"2022-05-15T19:45:26.257838Z","shell.execute_reply.started":"2022-05-15T19:45:25.850160Z","shell.execute_reply":"2022-05-15T19:45:26.257106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we see they're similar. Also, the features in both train and test do not have a high portion of missing values so we do not drop them. A personal threshold is 80 - 90%. Let's take a look at the target_scores dataset.","metadata":{}},{"cell_type":"code","source":"target_scores.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:26.259485Z","iopub.execute_input":"2022-05-15T19:45:26.259797Z","iopub.status.idle":"2022-05-15T19:45:26.271719Z","shell.execute_reply.started":"2022-05-15T19:45:26.259757Z","shell.execute_reply":"2022-05-15T19:45:26.270681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This dataframe contains information on the scoreline of each game in the training set. We'll decide later in the ML pipeline if adding these features will increase the predictive power of the models built. ","metadata":{}},{"cell_type":"code","source":"cols = train.select_dtypes(exclude='object').columns\n\nlen(cols)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:26.273340Z","iopub.execute_input":"2022-05-15T19:45:26.273663Z","iopub.status.idle":"2022-05-15T19:45:26.330735Z","shell.execute_reply.started":"2022-05-15T19:45:26.273621Z","shell.execute_reply":"2022-05-15T19:45:26.329691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"numeric_distribution_plot(train)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:26.332602Z","iopub.execute_input":"2022-05-15T19:45:26.332895Z","iopub.status.idle":"2022-05-15T19:45:35.524803Z","shell.execute_reply.started":"2022-05-15T19:45:26.332857Z","shell.execute_reply":"2022-05-15T19:45:35.524077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Deriving insights from data","metadata":{}},{"cell_type":"code","source":"# let's encode the target class\n\ntrain['target'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:35.526063Z","iopub.execute_input":"2022-05-15T19:45:35.526687Z","iopub.status.idle":"2022-05-15T19:45:35.551678Z","shell.execute_reply.started":"2022-05-15T19:45:35.526642Z","shell.execute_reply":"2022-05-15T19:45:35.551008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\nencoder = LabelEncoder()\nencoder.fit(train['target'])\n\ntrain['target'] = encoder.transform(train['target'])\n\ntrain['target'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:35.552987Z","iopub.execute_input":"2022-05-15T19:45:35.553443Z","iopub.status.idle":"2022-05-15T19:45:35.726865Z","shell.execute_reply.started":"2022-05-15T19:45:35.553407Z","shell.execute_reply":"2022-05-15T19:45:35.726201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the above, we see the classes are encoded 0 - away, 1 - draw and 2 - home. \n\nNow, we can check the correlation of the numeric features with the target.","metadata":{}},{"cell_type":"code","source":"corr = train.corr()['target']\n\nprint(\"The 15 features with the highest correlations with the target are: \\n\", corr.sort_values(ascending=False).head(10))\nprint('\\n')\nprint(\"The 15 features with the lowest correlations with the target are: \\n\", corr.sort_values(ascending=False).tail(10))","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:35.728225Z","iopub.execute_input":"2022-05-15T19:45:35.728635Z","iopub.status.idle":"2022-05-15T19:45:44.193944Z","shell.execute_reply.started":"2022-05-15T19:45:35.728594Z","shell.execute_reply":"2022-05-15T19:45:44.192454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Clearly the ratings features are important as they dominate the top 15 and bottom 15 correlations with the target. \n\nWhat do these correlations mean?\nFrom the target we encoded, we know that 0 refers to away wins, 1 refers to draws and 2 refers to home wins. The higher correlations with the target like home_team_history_rating_2 and home_team_history_rating_1 refer to likeliness of draws and home wins. For example, games with higer home_team_history_rating_2 will more likely have home wins and draws. \n\nSimilarly, the lower correlations with the target like away_team_history_rating_1 amd home_team_history_opponent_rating_1 with the target refer to likeliness of draws and away wins. \n\nLet's have a kde plot of some of the features to see how it affects the different classes clearly","metadata":{}},{"cell_type":"code","source":"plt.style.use('fivethirtyeight')\nsns.set(rc = {'figure.figsize':(10,8)})\n\ndef kde_plot(df, col):\n    \n    # print mean information of col by target\n    away_mean = df.loc[df['target'] == 0, col].mean()\n    draws_mean = df.loc[df['target'] == 1, col].mean()\n    home_mean = df.loc[df['target'] == 2, col].mean()\n    \n    print(f\"The average {col} for the home wins: {round(home_mean, 2)}\")\n    print(f\"The average {col} for the away wins: {round(away_mean, 2)}\")\n    print(f\"The average {col} for the draws: {round(draws_mean, 2)}\")\n    \n    # kde distribution for each of the target classes\n    sns.kdeplot(df.loc[df['target'] == 0, col], label = 'target == away')\n    sns.kdeplot(df.loc[df['target'] == 1, col], label = 'target == draws')\n    sns.kdeplot(df.loc[df['target'] == 2, col], label = 'target == home')\n    \n    plt.title(f\"Distribution of {col} by target\")\n    plt.xlabel(f\"{col}\")\n    plt.ylabel('Density')\n    plt.legend()\n    plt.tight_layout(h_pad = 2.5)\n    ","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:44.195231Z","iopub.execute_input":"2022-05-15T19:45:44.195557Z","iopub.status.idle":"2022-05-15T19:45:44.209002Z","shell.execute_reply.started":"2022-05-15T19:45:44.195511Z","shell.execute_reply":"2022-05-15T19:45:44.208121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### home_team_history_rating_2","metadata":{}},{"cell_type":"code","source":"kde_plot(train, 'home_team_history_rating_2')","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:44.210749Z","iopub.execute_input":"2022-05-15T19:45:44.214474Z","iopub.status.idle":"2022-05-15T19:45:45.578123Z","shell.execute_reply.started":"2022-05-15T19:45:44.214438Z","shell.execute_reply":"2022-05-15T19:45:45.577357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['home_team_history_rating_2'].describe()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:45.582927Z","iopub.execute_input":"2022-05-15T19:45:45.585277Z","iopub.status.idle":"2022-05-15T19:45:45.606214Z","shell.execute_reply.started":"2022-05-15T19:45:45.585229Z","shell.execute_reply":"2022-05-15T19:45:45.605350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let's see a boxplot of this feature\n\nsns.boxplot(x='home_team_history_rating_2', data=train)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:45.610649Z","iopub.execute_input":"2022-05-15T19:45:45.613082Z","iopub.status.idle":"2022-05-15T19:45:45.967927Z","shell.execute_reply.started":"2022-05-15T19:45:45.613036Z","shell.execute_reply":"2022-05-15T19:45:45.967159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's observe the outliers from this feature - home_team_history_rating_2...","metadata":{}},{"cell_type":"code","source":"train.loc[train['home_team_history_rating_2'] == train['home_team_history_rating_2'].max()]","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:45.969418Z","iopub.execute_input":"2022-05-15T19:45:45.969682Z","iopub.status.idle":"2022-05-15T19:45:46.001547Z","shell.execute_reply.started":"2022-05-15T19:45:45.969645Z","shell.execute_reply":"2022-05-15T19:45:46.000721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let's see the ratings that are above 20 - arbitrarily chosen value above the top of the boxplot and plot the kde\n\ntemp = train.loc[train['home_team_history_rating_2'] > 20]\n\noutlier_ratingmean = temp['home_team_history_rating_2'].mean()\nprint(f\"The average home_team_history_rating_2 of the outliers: {round(outlier_ratingmean, 2)}\")\n\nkde_plot(temp, 'home_team_history_rating_2')","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:46.003191Z","iopub.execute_input":"2022-05-15T19:45:46.003601Z","iopub.status.idle":"2022-05-15T19:45:46.584390Z","shell.execute_reply.started":"2022-05-15T19:45:46.003554Z","shell.execute_reply":"2022-05-15T19:45:46.583627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the investigation above, we see that the outliers have a similar distribution to the general dataset. I have decided to not remove these outlier values since a. they're minorities compared to the full dataset (mean of 23.95 vs 6.65) b. they do not alter the distribution of our full dataset.\n\nIn both cases, there is a very slight difference in the plots shown in the different classes and as reported by the correlation matrix, we see that the feature contributes to the target but not very strongly. We see a slightly higher average in the home wins for the feature compared to the draws and away wins as predicted by the correlation matrix.\n\nLet's see the case of a negative correlation (more away wins)","metadata":{}},{"cell_type":"markdown","source":"#### home_team_history_opponent_rating_1","metadata":{}},{"cell_type":"code","source":"kde_plot(train, 'home_team_history_opponent_rating_1')","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:46.591914Z","iopub.execute_input":"2022-05-15T19:45:46.592487Z","iopub.status.idle":"2022-05-15T19:45:47.564768Z","shell.execute_reply.started":"2022-05-15T19:45:46.592446Z","shell.execute_reply":"2022-05-15T19:45:47.564028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['home_team_history_opponent_rating_1'].describe()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:47.566314Z","iopub.execute_input":"2022-05-15T19:45:47.566840Z","iopub.status.idle":"2022-05-15T19:45:47.583944Z","shell.execute_reply.started":"2022-05-15T19:45:47.566797Z","shell.execute_reply":"2022-05-15T19:45:47.583091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see similar observations here as well. Just like before, we see slight correlations with the contributions to the target. In this case, the mean of away wins for the feature is slightly higher than for home and draws, as already explained. ","metadata":{}},{"cell_type":"markdown","source":"Since the ratings feature is very important, we should aggregate them and see how much of a contribution to the target, the aggregate makes","metadata":{}},{"cell_type":"markdown","source":"### aggregating the historical ratings features","metadata":{}},{"cell_type":"code","source":"train.columns[160:180]","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:47.585525Z","iopub.execute_input":"2022-05-15T19:45:47.585851Z","iopub.status.idle":"2022-05-15T19:45:47.592810Z","shell.execute_reply.started":"2022-05-15T19:45:47.585808Z","shell.execute_reply":"2022-05-15T19:45:47.591985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"homerating_features = ['home_team_history_rating_1', 'home_team_history_rating_2', 'home_team_history_rating_3', 'home_team_history_rating_4', \n                      'home_team_history_rating_5', 'home_team_history_rating_6', 'home_team_history_rating_7', 'home_team_history_rating_8', \n                      'home_team_history_rating_9', 'home_team_history_rating_10']\n\nawayrating_features = ['away_team_history_rating_1', 'away_team_history_rating_2', 'away_team_history_rating_3', 'away_team_history_rating_4', \n                      'away_team_history_rating_5', 'away_team_history_rating_6', 'away_team_history_rating_7', 'away_team_history_rating_8', \n                      'away_team_history_rating_9', 'away_team_history_rating_10']\n\nhomeopponent_features = ['home_team_history_opponent_rating_1', 'home_team_history_opponent_rating_2', 'home_team_history_opponent_rating_3', \n                          'home_team_history_opponent_rating_4', 'home_team_history_opponent_rating_5', 'home_team_history_opponent_rating_6', \n                          'home_team_history_opponent_rating_7', 'home_team_history_opponent_rating_8', 'home_team_history_opponent_rating_9', \n                          'home_team_history_opponent_rating_10']\n\nawayopponent_features = ['away_team_history_opponent_rating_1', 'away_team_history_opponent_rating_2', 'away_team_history_opponent_rating_3', \n                        'away_team_history_opponent_rating_4', 'away_team_history_opponent_rating_5', 'away_team_history_opponent_rating_6', \n                        'away_team_history_opponent_rating_7', 'away_team_history_opponent_rating_8', 'away_team_history_opponent_rating_9', \n                        'away_team_history_opponent_rating_10']\n\n# let's create the aggregate of these features in the train and test\ntrain['home_rating_norm'] = train[homerating_features].mean(axis=1)\ntrain['away_rating_norm'] = train[awayrating_features].mean(axis=1)\ntrain['home_opponent_norm'] = train[homeopponent_features].mean(axis=1)\ntrain['away_opponent_norm'] = train[awayopponent_features].mean(axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:47.594508Z","iopub.execute_input":"2022-05-15T19:45:47.595088Z","iopub.status.idle":"2022-05-15T19:45:47.962839Z","shell.execute_reply.started":"2022-05-15T19:45:47.595040Z","shell.execute_reply":"2022-05-15T19:45:47.961913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# for the test\ntest['home_rating_norm'] = test[homerating_features].mean(axis=1)\ntest['away_rating_norm'] = test[awayrating_features].mean(axis=1)\ntest['home_opponent_norm'] = test[homeopponent_features].mean(axis=1)\ntest['away_opponent_norm'] = test[awayopponent_features].mean(axis=1)\n\ntrain.shape, test.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:47.964511Z","iopub.execute_input":"2022-05-15T19:45:47.964797Z","iopub.status.idle":"2022-05-15T19:45:48.215316Z","shell.execute_reply.started":"2022-05-15T19:45:47.964757Z","shell.execute_reply":"2022-05-15T19:45:48.214587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let's see the correlations again\ncorr = train.corr()['target']\n\nprint(\"The 15 features with the highest correlations with the target are: \\n\", corr.sort_values(ascending=False).head(10))\nprint('\\n')\nprint(\"The 15 features with the lowest correlations with the target are: \\n\", corr.sort_values(ascending=False).tail(10))","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:48.216493Z","iopub.execute_input":"2022-05-15T19:45:48.216952Z","iopub.status.idle":"2022-05-15T19:45:56.600672Z","shell.execute_reply.started":"2022-05-15T19:45:48.216909Z","shell.execute_reply":"2022-05-15T19:45:56.599831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we see from the correlation matrix above, the newly added 4 features - home_rating_norm, away_opponent_norm, away_rating_norm and home_opponent_norm are slightly more positively and negatively correlated with the target than the features alone. Let's have a kde plot for these new features added to gain moe insight.","metadata":{}},{"cell_type":"code","source":"kde_plot(train, 'home_rating_norm')","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:56.603037Z","iopub.execute_input":"2022-05-15T19:45:56.603727Z","iopub.status.idle":"2022-05-15T19:45:57.483651Z","shell.execute_reply.started":"2022-05-15T19:45:56.603688Z","shell.execute_reply":"2022-05-15T19:45:57.482998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kde_plot(train, 'away_rating_norm')","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:57.484777Z","iopub.execute_input":"2022-05-15T19:45:57.486313Z","iopub.status.idle":"2022-05-15T19:45:58.378465Z","shell.execute_reply.started":"2022-05-15T19:45:57.486271Z","shell.execute_reply":"2022-05-15T19:45:58.377753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kde_plot(train, 'home_opponent_norm')","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:58.379518Z","iopub.execute_input":"2022-05-15T19:45:58.379908Z","iopub.status.idle":"2022-05-15T19:45:59.261628Z","shell.execute_reply.started":"2022-05-15T19:45:58.379869Z","shell.execute_reply":"2022-05-15T19:45:59.260957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kde_plot(train, 'away_opponent_norm')","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:45:59.263010Z","iopub.execute_input":"2022-05-15T19:45:59.263463Z","iopub.status.idle":"2022-05-15T19:46:00.186669Z","shell.execute_reply.started":"2022-05-15T19:45:59.263425Z","shell.execute_reply":"2022-05-15T19:46:00.185988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Additional Insights from the data","metadata":{}},{"cell_type":"markdown","source":"We have derived some insights from the rating features available in the data given and have added features based on these insights. Let's see some the historical features available in the datset from which we could gain more insights and engineer the features.","metadata":{}},{"cell_type":"code","source":"# let's see the dataframe again\n\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:00.188042Z","iopub.execute_input":"2022-05-15T19:46:00.188520Z","iopub.status.idle":"2022-05-15T19:46:00.215808Z","shell.execute_reply.started":"2022-05-15T19:46:00.188482Z","shell.execute_reply":"2022-05-15T19:46:00.215114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looking at the features description given, the features that should contribute to the correlation of the target should be the ```home_team_history_goal-{i}``` and the ```home_team_history_opponent_goal_{i}``` and the same for the away teams. Before we look at the correlations these features might have with the target or even aggregating them, let's understand what type of information they contain","metadata":{}},{"cell_type":"code","source":"temp = ['home_team_history_goal_1', 'away_team_history_goal_1','home_team_history_opponent_goal_1', 'away_team_history_opponent_goal_1']\n\ntrain[temp].head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:00.216861Z","iopub.execute_input":"2022-05-15T19:46:00.217191Z","iopub.status.idle":"2022-05-15T19:46:00.328756Z","shell.execute_reply.started":"2022-05-15T19:46:00.217141Z","shell.execute_reply":"2022-05-15T19:46:00.328044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* ```home_team_history_goal_1``` refers to the number of goals scored by the home team in the last 1 game\n* ```away_team_history_goal_1``` refers to the number of goals scored by the away team in the last 1 game\n* ```home_team_history_opponent_goal_1``` refers to the number of goals conceded by the home team in the last 1 game\n* ```away_team_history_opponent_goal_1``` refers to the number of goals conceded by the away team in the last 1 game","metadata":{}},{"cell_type":"code","source":"# let's see the correlations of these features with the target\n\ncorr1 = train['target'].corr(train['home_team_history_goal_1'])\ncorr2 = train['target'].corr(train['away_team_history_goal_1'])\ncorr3 = train['target'].corr(train['home_team_history_opponent_goal_1'])\ncorr4 = train['target'].corr(train['away_team_history_opponent_goal_1'])\n\nl = [corr1, corr2, corr3, corr4]\nprint(l)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:00.329995Z","iopub.execute_input":"2022-05-15T19:46:00.330247Z","iopub.status.idle":"2022-05-15T19:46:00.347449Z","shell.execute_reply.started":"2022-05-15T19:46:00.330212Z","shell.execute_reply":"2022-05-15T19:46:00.346735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These features have even weaker correlations with the target. Everything discussed about positive and negative correlations with the target earlier hold for these features as well.","metadata":{}},{"cell_type":"code","source":"# let's aggregate the features\n\nhomehistory_goals = ['home_team_history_goal_1', 'home_team_history_goal_2', 'home_team_history_goal_3', 'home_team_history_goal_4',\n                     'home_team_history_goal_5', 'home_team_history_goal_6', 'home_team_history_goal_7', 'home_team_history_goal_8',\n                     'home_team_history_goal_9', 'home_team_history_goal_10']\n\nawayhistory_goals = ['away_team_history_goal_1', 'away_team_history_goal_2', 'away_team_history_goal_3', 'away_team_history_goal_4', \n                    'away_team_history_goal_5', 'away_team_history_goal_6', 'away_team_history_goal_7', 'away_team_history_goal_8', \n                    'away_team_history_goal_9', 'away_team_history_goal_10']\n\nhomeopponent_goals = ['home_team_history_opponent_goal_1', 'home_team_history_opponent_goal_2', 'home_team_history_opponent_goal_3', \n                     'home_team_history_opponent_goal_4', 'home_team_history_opponent_goal_5', 'home_team_history_opponent_goal_6', \n                     'home_team_history_opponent_goal_7', 'home_team_history_opponent_goal_8', 'home_team_history_opponent_goal_9', \n                     'home_team_history_opponent_goal_10']\n\nawayopponent_goals = ['away_team_history_opponent_goal_1', 'away_team_history_opponent_goal_2', 'away_team_history_opponent_goal_3', \n                     'away_team_history_opponent_goal_4', 'away_team_history_opponent_goal_5', 'away_team_history_opponent_goal_6', \n                     'away_team_history_opponent_goal_7', 'away_team_history_opponent_goal_8', 'away_team_history_opponent_goal_9',\n                     'away_team_history_opponent_goal_10']\n\n# let's aggregate the features for train and test\n\ntrain['home_goals_norm'] = train[homehistory_goals].mean(axis=1)\ntrain['away_goals_norm'] = train[awayhistory_goals].mean(axis=1)\ntrain['home_goalsopp_norm'] = train[homeopponent_goals].mean(axis=1)\ntrain['away_goalsopp_norm'] = train[awayopponent_goals].mean(axis=1)\n\ntest['home_goals_norm'] = test[homehistory_goals].mean(axis=1)\ntest['away_goals_norm'] = test[awayhistory_goals].mean(axis=1)\ntest['home_goalsopp_norm'] = test[homeopponent_goals].mean(axis=1)\ntest['away_goalsopp_norm'] = test[awayopponent_goals].mean(axis=1)\n\ntrain.shape, test.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:00.348719Z","iopub.execute_input":"2022-05-15T19:46:00.348970Z","iopub.status.idle":"2022-05-15T19:46:01.014645Z","shell.execute_reply.started":"2022-05-15T19:46:00.348936Z","shell.execute_reply":"2022-05-15T19:46:01.013975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:01.016030Z","iopub.execute_input":"2022-05-15T19:46:01.016597Z","iopub.status.idle":"2022-05-15T19:46:01.045834Z","shell.execute_reply.started":"2022-05-15T19:46:01.016554Z","shell.execute_reply":"2022-05-15T19:46:01.045128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let's see if they show up in the correlation now\n\n# let's see the correlations again\ncorr = train.corr()['target']\n\nprint(\"The 15 features with the highest correlations with the target are: \\n\", corr.sort_values(ascending=False).head(10))\nprint('\\n')\nprint(\"The 15 features with the lowest correlations with the target are: \\n\", corr.sort_values(ascending=False).tail(10))","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:01.046932Z","iopub.execute_input":"2022-05-15T19:46:01.047238Z","iopub.status.idle":"2022-05-15T19:46:09.568941Z","shell.execute_reply.started":"2022-05-15T19:46:01.047204Z","shell.execute_reply":"2022-05-15T19:46:09.566997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we see now, the features contribute to the target as they both show up in both the positive and negative correlations.","metadata":{}},{"cell_type":"markdown","source":"#### Dropping date features","metadata":{}},{"cell_type":"markdown","source":"Now, it's unlikely that date features will contribute to the predictive power of the model we're trying to build. The date features refer to the dates the current match was played and to the dates the previous i matches were played. We'll drop all of these columns from the datasets","metadata":{}},{"cell_type":"code","source":"datefeatures = ['match_date', 'home_team_history_match_date_1', 'home_team_history_match_date_2', 'home_team_history_match_date_3', 'home_team_history_match_date_4',\n               'home_team_history_match_date_5', 'home_team_history_match_date_6', 'home_team_history_match_date_7', 'home_team_history_match_date_8', 'home_team_history_match_date_9', 'home_team_history_match_date_10', \n               'away_team_history_match_date_1', 'away_team_history_match_date_2', 'away_team_history_match_date_3', 'away_team_history_match_date_4', 'away_team_history_match_date_5', \n               'away_team_history_match_date_6', 'away_team_history_match_date_7', 'away_team_history_match_date_8', 'away_team_history_match_date_9', 'away_team_history_match_date_10']\n\ntrain = train.drop(columns=datefeatures)\ntest = test.drop(columns=datefeatures)\n\ntrain.shape, test.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:09.570278Z","iopub.execute_input":"2022-05-15T19:46:09.570552Z","iopub.status.idle":"2022-05-15T19:46:09.850385Z","shell.execute_reply.started":"2022-05-15T19:46:09.570514Z","shell.execute_reply":"2022-05-15T19:46:09.849652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### historical is_cup and is_play_home","metadata":{}},{"cell_type":"code","source":"temp = ['home_team_history_is_cup_1', 'away_team_history_is_cup_1', 'home_team_history_is_play_home_1',\n            'away_team_history_is_play_home_1']\n\ntrain[temp].head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:09.851688Z","iopub.execute_input":"2022-05-15T19:46:09.852096Z","iopub.status.idle":"2022-05-15T19:46:09.867943Z","shell.execute_reply.started":"2022-05-15T19:46:09.852058Z","shell.execute_reply":"2022-05-15T19:46:09.867105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let's see their correlations with the target\n\ncorr1 = train['target'].corr(train['home_team_history_is_cup_1'])\ncorr2 = train['target'].corr(train['away_team_history_is_cup_1'])\ncorr3 = train['target'].corr(train['home_team_history_is_play_home_1'])\ncorr4 = train['target'].corr(train['away_team_history_is_play_home_1'])\n\nl = [corr1, corr2, corr3, corr4]\nprint(l)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:09.869183Z","iopub.execute_input":"2022-05-15T19:46:09.869539Z","iopub.status.idle":"2022-05-15T19:46:09.886003Z","shell.execute_reply.started":"2022-05-15T19:46:09.869504Z","shell.execute_reply":"2022-05-15T19:46:09.885360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Correlations are low as observed with other features, and positive and negative signs are as described for others. \nthe ```home_team_history_is_cup_{i}``` refers to if the home team last i games were cup competition and the ```home_team_history_is_play_home_{i}``` refers to if the home team last i games were played at home and similar definitions for the away prefixes. These features have a value of 1 if True and 0 for False. Let's aggregate the features. I've decided to use the median for these features since they're flags (1 or 0)","metadata":{}},{"cell_type":"code","source":"home_iscup = ['home_team_history_is_cup_1', 'home_team_history_is_cup_2', 'home_team_history_is_cup_3', 'home_team_history_is_cup_4', \n             'home_team_history_is_cup_5', 'home_team_history_is_cup_6', 'home_team_history_is_cup_7', 'home_team_history_is_cup_8', \n             'home_team_history_is_cup_9', 'home_team_history_is_cup_10']\n\naway_iscup = ['away_team_history_is_cup_1', 'away_team_history_is_cup_2', 'away_team_history_is_cup_3', 'away_team_history_is_cup_4', \n             'away_team_history_is_cup_5', 'away_team_history_is_cup_6', 'away_team_history_is_cup_7', 'away_team_history_is_cup_8', \n             'away_team_history_is_cup_9', 'away_team_history_is_cup_10']\n\nhome_ishome = ['home_team_history_is_play_home_1', 'home_team_history_is_play_home_2', 'home_team_history_is_play_home_3', 'home_team_history_is_play_home_4', \n              'home_team_history_is_play_home_5', 'home_team_history_is_play_home_6', 'home_team_history_is_play_home_7', 'home_team_history_is_play_home_8', \n              'home_team_history_is_play_home_9', 'home_team_history_is_play_home_10']\n\naway_ishome = ['away_team_history_is_play_home_1', 'away_team_history_is_play_home_2', 'away_team_history_is_play_home_3', 'away_team_history_is_play_home_4', \n              'away_team_history_is_play_home_5', 'away_team_history_is_play_home_6', 'away_team_history_is_play_home_7', 'away_team_history_is_play_home_8',\n              'away_team_history_is_play_home_9', 'away_team_history_is_play_home_10']\n\ntrain['home_iscup_norm'] = train[home_iscup].median(axis=1)\ntrain['away_iscup_norm'] = train[away_iscup].median(axis=1)\ntrain['home_ishome_norm'] = train[home_ishome].median(axis=1)\ntrain['away_ishome_norm'] = train[away_ishome].median(axis=1)\n\ntest['home_iscup_norm'] = test[home_iscup].median(axis=1)\ntest['away_iscup_norm'] = test[away_iscup].median(axis=1)\ntest['home_ishome_norm'] = test[home_ishome].median(axis=1)\ntest['away_ishome_norm'] = test[away_ishome].median(axis=1)\n\ntrain.shape, test.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:09.887125Z","iopub.execute_input":"2022-05-15T19:46:09.887607Z","iopub.status.idle":"2022-05-15T19:46:10.770825Z","shell.execute_reply.started":"2022-05-15T19:46:09.887572Z","shell.execute_reply":"2022-05-15T19:46:10.770029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:10.772082Z","iopub.execute_input":"2022-05-15T19:46:10.772341Z","iopub.status.idle":"2022-05-15T19:46:10.801406Z","shell.execute_reply.started":"2022-05-15T19:46:10.772308Z","shell.execute_reply":"2022-05-15T19:46:10.800703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let's see the correlations again\ncorr = train.corr()['target']\n\nprint(\"The 15 features with the highest correlations with the target are: \\n\", corr.sort_values(ascending=False).head(10))\nprint('\\n')\nprint(\"The 15 features with the lowest correlations with the target are: \\n\", corr.sort_values(ascending=False).tail(10))","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:10.802647Z","iopub.execute_input":"2022-05-15T19:46:10.802865Z","iopub.status.idle":"2022-05-15T19:46:20.021276Z","shell.execute_reply.started":"2022-05-15T19:46:10.802832Z","shell.execute_reply":"2022-05-15T19:46:20.020488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These new features do not show up in our top 15 and bottom 15 correlations with the target. This means their correlations are much weaker compared to the other features.\n\nSince, we're done with the transforming the historical features to gather better insights from the data given, let's define the new features created.\n\n* ```home_rating_norm``` refers to the average of ratings of the home team of the current match in their last 10 games\n* ```away_rating_norm``` refers to the average of ratings of the away team of the current match in their last 10 games\n* ```home_opponent_norm``` refers to the average of ratings of the opponents of the home team of the current match in the last 10 games\n* ```away_opponent_norm``` refers to the average of ratings of the opponents of the away team of the current match in the last 10 games\n* ```home_goals_norm``` refers to the average of the goals scored by the home team of the current match in their last 10 games\n* ```away_goals_norm``` refers to the average of the goals scored by the away team of the current match in their last 10 games\n* ```home_goalsopp_norm``` refers to the average of the goals scored by the opponents of the home team in their last 10 games\n* ```away_goalsopp_norm``` refers to the average of the goals scored by the opponents of the away team in their last 10 games\n* ```home_iscup_norm``` refers to average of flag times games played by the home team of the current match played a cup comp in their last 10 games\n* ```away_iscup_norm``` refers to the average of flag times games played by the away team of the current match played a cup comp in their last 10 games\n* ```home_ishome_norm``` refers to the average of flag times the home team of the current match played at home in their last 10 games\n* ```away_ishome_norm``` refers to the average of flag times the away team of the current match played at home in their last 10 games","metadata":{}},{"cell_type":"markdown","source":"### Categorical Encoding","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:20.022675Z","iopub.execute_input":"2022-05-15T19:46:20.023133Z","iopub.status.idle":"2022-05-15T19:46:20.052747Z","shell.execute_reply.started":"2022-05-15T19:46:20.023092Z","shell.execute_reply":"2022-05-15T19:46:20.052045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# the is_cup feature\nencoder = LabelEncoder()\nencoder.fit(train['is_cup'])\n\ntrain['is_cup'] = encoder.transform(train['is_cup'])\ntrain.shape, test.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:20.054038Z","iopub.execute_input":"2022-05-15T19:46:20.054383Z","iopub.status.idle":"2022-05-15T19:46:20.092598Z","shell.execute_reply.started":"2022-05-15T19:46:20.054344Z","shell.execute_reply":"2022-05-15T19:46:20.091767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"encoder = LabelEncoder()\ntest['is_cup'] = encoder.fit_transform(test['is_cup'])","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:20.094076Z","iopub.execute_input":"2022-05-15T19:46:20.094383Z","iopub.status.idle":"2022-05-15T19:46:20.102272Z","shell.execute_reply.started":"2022-05-15T19:46:20.094334Z","shell.execute_reply":"2022-05-15T19:46:20.101539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:20.103603Z","iopub.execute_input":"2022-05-15T19:46:20.103856Z","iopub.status.idle":"2022-05-15T19:46:20.132763Z","shell.execute_reply.started":"2022-05-15T19:46:20.103823Z","shell.execute_reply":"2022-05-15T19:46:20.132081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols = train.select_dtypes(include='object').columns\n\nprint(len(cols))\ncols","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:20.133994Z","iopub.execute_input":"2022-05-15T19:46:20.134487Z","iopub.status.idle":"2022-05-15T19:46:20.254433Z","shell.execute_reply.started":"2022-05-15T19:46:20.134453Z","shell.execute_reply":"2022-05-15T19:46:20.253478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These features would not be included in modelling. So, there's no need to encode them","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:20.255777Z","iopub.execute_input":"2022-05-15T19:46:20.256092Z","iopub.status.idle":"2022-05-15T19:46:20.284313Z","shell.execute_reply.started":"2022-05-15T19:46:20.256054Z","shell.execute_reply":"2022-05-15T19:46:20.283541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Drop the ids","metadata":{}},{"cell_type":"code","source":"id_cols = ['home_team_history_coach_1', 'home_team_history_coach_2', 'home_team_history_coach_3', 'home_team_history_coach_4', 'home_team_history_coach_5', \n          'home_team_history_coach_6', 'home_team_history_coach_7', 'home_team_history_coach_8', 'home_team_history_coach_9', 'home_team_history_coach_10', \n          'home_team_history_league_id_1', 'home_team_history_league_id_2', 'home_team_history_league_id_3', 'home_team_history_league_id_4', 'home_team_history_league_id_5', \n          'home_team_history_league_id_6', 'home_team_history_league_id_7', 'home_team_history_league_id_8', 'home_team_history_league_id_9', 'home_team_history_league_id_10', \n          'league_id', 'home_team_coach_id', 'away_team_coach_id']\n\ntrain = train.drop(columns=id_cols)\ntest = test.drop(columns=id_cols)\ntrain.shape, test.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:20.285651Z","iopub.execute_input":"2022-05-15T19:46:20.286185Z","iopub.status.idle":"2022-05-15T19:46:20.434759Z","shell.execute_reply.started":"2022-05-15T19:46:20.286134Z","shell.execute_reply":"2022-05-15T19:46:20.434046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ML Models","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:20.436015Z","iopub.execute_input":"2022-05-15T19:46:20.436260Z","iopub.status.idle":"2022-05-15T19:46:20.462966Z","shell.execute_reply.started":"2022-05-15T19:46:20.436226Z","shell.execute_reply":"2022-05-15T19:46:20.462336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"target = train['target']\ny_train = np.array(target)\n\nX_train = train.drop(columns=['id', 'target', 'home_team_name', 'away_team_name', 'league_name'])\nX_test = test.drop(columns=['id', 'home_team_name', 'away_team_name', 'league_name'])\nX_train.shape, X_test.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:20.464137Z","iopub.execute_input":"2022-05-15T19:46:20.464458Z","iopub.status.idle":"2022-05-15T19:46:20.540688Z","shell.execute_reply.started":"2022-05-15T19:46:20.464425Z","shell.execute_reply":"2022-05-15T19:46:20.539746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.pipeline import Pipeline\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.preprocessing import StandardScaler\n\npipeline = Pipeline(steps=[\n    ('imputer', SimpleImputer(strategy='mean')), \n    ('scaler', StandardScaler())\n])\n\npipeline.fit(X_train)\nX_train = pipeline.transform(X_train)\nX_test = pipeline.transform(X_test)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:20.541823Z","iopub.execute_input":"2022-05-15T19:46:20.545246Z","iopub.status.idle":"2022-05-15T19:46:21.880077Z","shell.execute_reply.started":"2022-05-15T19:46:20.545208Z","shell.execute_reply":"2022-05-15T19:46:21.879224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Baseline Models","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import LogisticRegression\n\nmodel = LogisticRegression(multi_class='multinomial', solver='lbfgs')\nmodel.fit(X_train, y_train)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:21.881579Z","iopub.execute_input":"2022-05-15T19:46:21.881816Z","iopub.status.idle":"2022-05-15T19:46:28.910911Z","shell.execute_reply.started":"2022-05-15T19:46:21.881784Z","shell.execute_reply":"2022-05-15T19:46:28.909857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred = model.predict(X_test)\ny_pred[:5]","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:28.916448Z","iopub.execute_input":"2022-05-15T19:46:28.919179Z","iopub.status.idle":"2022-05-15T19:46:28.988093Z","shell.execute_reply.started":"2022-05-15T19:46:28.919095Z","shell.execute_reply":"2022-05-15T19:46:28.987219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predict_proba = model.predict_proba(X_test)\npredict_proba[:5]","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:28.993230Z","iopub.execute_input":"2022-05-15T19:46:28.995580Z","iopub.status.idle":"2022-05-15T19:46:29.071076Z","shell.execute_reply.started":"2022-05-15T19:46:28.995532Z","shell.execute_reply":"2022-05-15T19:46:29.070369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predict_proba[:5,0]","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:29.076009Z","iopub.execute_input":"2022-05-15T19:46:29.078929Z","iopub.status.idle":"2022-05-15T19:46:29.091325Z","shell.execute_reply.started":"2022-05-15T19:46:29.078858Z","shell.execute_reply":"2022-05-15T19:46:29.090216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that the predict_proba properly arranged the predicted probabilities - class 0, 1 and 2...","metadata":{}},{"cell_type":"code","source":"ids = test['id']\n\n# let's make a dataframe for submission\n\nsubmission = pd.DataFrame({'id': ids, 'home': predict_proba[:, 2], 'draw': predict_proba[:, 1], 'away': predict_proba[:, 0]})\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:29.093237Z","iopub.execute_input":"2022-05-15T19:46:29.093802Z","iopub.status.idle":"2022-05-15T19:46:29.115818Z","shell.execute_reply.started":"2022-05-15T19:46:29.093757Z","shell.execute_reply":"2022-05-15T19:46:29.114823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('baseline.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:29.117439Z","iopub.execute_input":"2022-05-15T19:46:29.117731Z","iopub.status.idle":"2022-05-15T19:46:29.677530Z","shell.execute_reply.started":"2022-05-15T19:46:29.117691Z","shell.execute_reply":"2022-05-15T19:46:29.676774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This gave a public score of 1.01170 on the Kaggle board.","metadata":{}},{"cell_type":"markdown","source":"### RandomForestClassifier","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\nmodel = RandomForestClassifier(n_estimators=1000, random_state=42, n_jobs=-1, max_depth=4, max_features=6)\nmodel.fit(X_train, y_train)\npredict_proba_rnd = model.predict_proba(X_test)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:46:29.678806Z","iopub.execute_input":"2022-05-15T19:46:29.679067Z","iopub.status.idle":"2022-05-15T19:48:00.241363Z","shell.execute_reply.started":"2022-05-15T19:46:29.679032Z","shell.execute_reply":"2022-05-15T19:48:00.240549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predict_proba_rnd","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:48:00.242730Z","iopub.execute_input":"2022-05-15T19:48:00.242972Z","iopub.status.idle":"2022-05-15T19:48:00.249925Z","shell.execute_reply.started":"2022-05-15T19:48:00.242939Z","shell.execute_reply":"2022-05-15T19:48:00.249136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame({'id': ids, 'home': predict_proba_rnd[:, 2], 'draw': predict_proba_rnd[:, 1], 'away': predict_proba_rnd[:, 0]})\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:48:00.251617Z","iopub.execute_input":"2022-05-15T19:48:00.252120Z","iopub.status.idle":"2022-05-15T19:48:00.269269Z","shell.execute_reply.started":"2022-05-15T19:48:00.252079Z","shell.execute_reply":"2022-05-15T19:48:00.268588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('randomforestclassifier.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:48:00.270377Z","iopub.execute_input":"2022-05-15T19:48:00.270610Z","iopub.status.idle":"2022-05-15T19:48:00.723549Z","shell.execute_reply.started":"2022-05-15T19:48:00.270578Z","shell.execute_reply":"2022-05-15T19:48:00.722825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This had a score of 1.03777 on the Kaggle board which performed worse than the baseline logisticregression","metadata":{}},{"cell_type":"markdown","source":"Let's tune the Randomforestclassifier to for a better score","metadata":{}},{"cell_type":"markdown","source":"### XGBoost","metadata":{}},{"cell_type":"code","source":"from sklearn import model_selection\nimport xgboost as xgb\nX_train_xgb, X_valid, y_train_xgb, y_valid = model_selection.train_test_split(X_train, y_train, train_size=0.8, random_state=42)\n\n\nd_train = xgb.DMatrix(X_train_xgb, y_train_xgb)\nd_test = xgb.DMatrix(X_valid, y_valid)\n","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:48:00.724790Z","iopub.execute_input":"2022-05-15T19:48:00.725022Z","iopub.status.idle":"2022-05-15T19:48:03.994318Z","shell.execute_reply.started":"2022-05-15T19:48:00.724991Z","shell.execute_reply":"2022-05-15T19:48:03.993705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"params_1 = {\n    'booster': 'gbtree', 'max_depth': 5, \n    'learning_rate': 0.025, 'min_split_loss': 0.01, 'min_child_weight': 5,'subsample': 0.8,\n    'colsample_bytree': 0.25, 'alpha': 2, 'lambda': 3, 'objective': 'multi:softprob',\n    'num_class': 3, 'eval_metric':'mlogloss'\n    }\n\nnum_round = 1000\nevallist = [(d_train, 'train'), (d_test, 'eval')]\nbst = xgb.train(params_1, d_train, num_round, evallist, early_stopping_rounds=20, verbose_eval=50)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:48:03.995501Z","iopub.execute_input":"2022-05-15T19:48:03.996044Z","iopub.status.idle":"2022-05-15T19:53:23.692746Z","shell.execute_reply.started":"2022-05-15T19:48:03.996005Z","shell.execute_reply":"2022-05-15T19:53:23.692077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_test_d = xgb.DMatrix(X_test)\npredict_proba_xgb = bst.predict(X_test_d, iteration_range=(0, bst.best_iteration))\npredict_proba_xgb","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:53:23.696720Z","iopub.execute_input":"2022-05-15T19:53:23.698505Z","iopub.status.idle":"2022-05-15T19:53:25.559267Z","shell.execute_reply.started":"2022-05-15T19:53:23.698467Z","shell.execute_reply":"2022-05-15T19:53:25.558590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame({'id': ids, 'home': predict_proba_xgb[:, 2], 'draw': predict_proba_xgb[:, 1], \n                           'away': predict_proba_xgb[:, 0]})\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:53:25.563096Z","iopub.execute_input":"2022-05-15T19:53:25.565017Z","iopub.status.idle":"2022-05-15T19:53:25.582530Z","shell.execute_reply.started":"2022-05-15T19:53:25.564979Z","shell.execute_reply":"2022-05-15T19:53:25.581853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('xgb.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:53:25.583810Z","iopub.execute_input":"2022-05-15T19:53:25.584060Z","iopub.status.idle":"2022-05-15T19:53:25.968371Z","shell.execute_reply.started":"2022-05-15T19:53:25.584028Z","shell.execute_reply":"2022-05-15T19:53:25.967660Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This gave a score of 1.00841 on the Kaggle board","metadata":{}},{"cell_type":"markdown","source":"### Neural Network","metadata":{}},{"cell_type":"code","source":"X_train.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:53:25.969497Z","iopub.execute_input":"2022-05-15T19:53:25.969744Z","iopub.status.idle":"2022-05-15T19:53:25.977195Z","shell.execute_reply.started":"2022-05-15T19:53:25.969712Z","shell.execute_reply":"2022-05-15T19:53:25.976104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow import keras\nfrom keras import models, layers\n\n# convert y_train to categorical\ny_traint = keras.utils.to_categorical(y_train, 3)\n\n\n# create a neural network using Sequential API\n\nmodel = models.Sequential(name='footballnn')\nmodel.add(layers.Dense(100, activation='relu', input_shape=X_train.shape[1:]))\nmodel.add(layers.Dense(100, activation='relu'))\nmodel.add(layers.Dense(3, activation='softmax'))\n\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics='accuracy')","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:53:25.978681Z","iopub.execute_input":"2022-05-15T19:53:25.979208Z","iopub.status.idle":"2022-05-15T19:53:35.756251Z","shell.execute_reply.started":"2022-05-15T19:53:25.979165Z","shell.execute_reply":"2022-05-15T19:53:35.754161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit(X_train, y_traint, epochs=20, validation_split=0.2, callbacks = keras.callbacks.EarlyStopping(patience=5), \n                   )","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:53:35.757814Z","iopub.execute_input":"2022-05-15T19:53:35.758091Z","iopub.status.idle":"2022-05-15T19:54:23.872977Z","shell.execute_reply.started":"2022-05-15T19:53:35.758053Z","shell.execute_reply":"2022-05-15T19:54:23.872154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred = model.predict(X_test)\npred","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:54:23.874625Z","iopub.execute_input":"2022-05-15T19:54:23.874862Z","iopub.status.idle":"2022-05-15T19:54:26.659976Z","shell.execute_reply.started":"2022-05-15T19:54:23.874829Z","shell.execute_reply":"2022-05-15T19:54:26.659192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame({'id': ids, 'home': pred[:, 2], 'draw': pred[:, 1], 'away': pred[:, 0]})\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:54:26.661188Z","iopub.execute_input":"2022-05-15T19:54:26.661759Z","iopub.status.idle":"2022-05-15T19:54:26.677691Z","shell.execute_reply.started":"2022-05-15T19:54:26.661716Z","shell.execute_reply":"2022-05-15T19:54:26.676773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('nn.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:54:26.679413Z","iopub.execute_input":"2022-05-15T19:54:26.679831Z","iopub.status.idle":"2022-05-15T19:54:27.061811Z","shell.execute_reply.started":"2022-05-15T19:54:26.679784Z","shell.execute_reply":"2022-05-15T19:54:27.061106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This performed worse than the baseline model with a score of 1.05249","metadata":{}},{"cell_type":"code","source":"# let's create an ensemble of the predictions\n\npredictions = (predict_proba + predict_proba_rnd + predict_proba_xgb + pred)/4\npredictions","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:54:27.063076Z","iopub.execute_input":"2022-05-15T19:54:27.063334Z","iopub.status.idle":"2022-05-15T19:54:27.071913Z","shell.execute_reply.started":"2022-05-15T19:54:27.063302Z","shell.execute_reply":"2022-05-15T19:54:27.070732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.DataFrame({'id': ids, 'home': predictions[:, 2], 'draw': predictions[:, 1], 'away': predictions[:, 0]})\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:54:27.080031Z","iopub.execute_input":"2022-05-15T19:54:27.080344Z","iopub.status.idle":"2022-05-15T19:54:27.096097Z","shell.execute_reply.started":"2022-05-15T19:54:27.080311Z","shell.execute_reply":"2022-05-15T19:54:27.095114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('ensembledd.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:54:27.097582Z","iopub.execute_input":"2022-05-15T19:54:27.097954Z","iopub.status.idle":"2022-05-15T19:54:27.733990Z","shell.execute_reply.started":"2022-05-15T19:54:27.097913Z","shell.execute_reply":"2022-05-15T19:54:27.733119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This gave a score of 1.00983 on the board","metadata":{}},{"cell_type":"markdown","source":"The XGBoost model gave the best performance on the Kaggle board so we'll use it henceforth for the feature selection process. \nLet's define a function for easy reusability","metadata":{}},{"cell_type":"code","source":"def train_xgboost(X_train, y_train, X_test):\n    X_train_xgb, X_valid, y_train_xgb, y_valid = model_selection.train_test_split(X_train, y_train, train_size=0.8, random_state=42)\n    \n    d_train = xgb.DMatrix(X_train_xgb, y_train_xgb)\n    d_test = xgb.DMatrix(X_valid, y_valid)\n    \n    params_1 = {\n    'booster': 'gbtree', 'max_depth': 5, \n    'learning_rate': 0.025, 'min_split_loss': 0.01, 'min_child_weight': 5,'subsample': 0.8,\n    'colsample_bytree': 0.25, 'alpha': 2, 'lambda': 3, 'objective': 'multi:softprob',\n    'num_class': 3, 'eval_metric':'mlogloss'\n    }\n    \n    num_round = 1000\n    evallist = [(d_train, 'train'), (d_test, 'eval')]\n    bst = xgb.train(params_1, d_train, num_round, evallist, early_stopping_rounds=20, verbose_eval=50)\n    \n    X_test_d = xgb.DMatrix(X_test)\n    predict_proba_xgb = bst.predict(X_test_d, iteration_range=(0, bst.best_iteration))\n    \n    submission = pd.DataFrame({'id': ids, 'home': predict_proba_xgb[:, 2], 'draw': predict_proba_xgb[:, 1], \n                           'away': predict_proba_xgb[:, 0]})\n    display(submission.head())\n    \n    return submission","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:56:53.299575Z","iopub.execute_input":"2022-05-15T19:56:53.299837Z","iopub.status.idle":"2022-05-15T19:56:53.308618Z","shell.execute_reply.started":"2022-05-15T19:56:53.299809Z","shell.execute_reply":"2022-05-15T19:56:53.307863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Feature Selection","metadata":{}},{"cell_type":"markdown","source":"#### SelectKBEST","metadata":{}},{"cell_type":"code","source":"X_train_x = train.drop(columns=['id', 'target', 'home_team_name', 'away_team_name', 'league_name'])\nX_train_x = X_train_x.fillna(X_train_x.mean())\nX_test_x = test.drop(columns=['id', 'home_team_name', 'away_team_name', 'league_name'])\nX_test_x = X_test_x.fillna(X_train_x.mean())\ny_train_x = train['target']","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:05:24.951893Z","iopub.execute_input":"2022-05-15T20:05:24.952203Z","iopub.status.idle":"2022-05-15T20:05:25.412730Z","shell.execute_reply.started":"2022-05-15T20:05:24.952164Z","shell.execute_reply":"2022-05-15T20:05:25.411898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.feature_selection import SelectKBest, f_classif\n\ndef select_kbest(X_train, y_train, X_test, score_func=f_classif, k=70, transform=1):\n    \"\"\"Uses scikit-learn's SelectKBest to perform feature selection of columns\n    \n    Args:  \n        X_train: df of features in the train dataset, without the target\n        y_train: df of the target set in the train dataset\n        X_test: df of features in the test dataset without the target\n        score_func: used to score the features for univariate classification, defaults to chi2 score\n        k : number of features to select, defaults to 5\n        transform: decides if to transform X_train in terms of the kbest features. 1/0\n\n    Returns:\n           prints a df of best features in order of descending scores\n           if transform, returns X_train with only the features selected\n    \"\"\"\n    bestfeatures = SelectKBest(score_func=score_func, k=k)\n    bestfeatures.fit(X_train, y_train)\n    \n    scores = pd.Series(bestfeatures.scores_)\n    columns = pd.Series(X_train.columns)\n    \n    df = pd.concat([columns, scores], axis=1)\n    df = df.rename(columns={0: 'Columns', 1: 'Scores'})\n    display(df.sort_values(by='Scores', ascending=False))\n    \n    if transform == 1:\n        X_train = bestfeatures.transform(X_train)\n        X_test = bestfeatures.transform(X_test)\n    \n    return X_train, X_test","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:06:05.246504Z","iopub.execute_input":"2022-05-15T20:06:05.246757Z","iopub.status.idle":"2022-05-15T20:06:05.256938Z","shell.execute_reply.started":"2022-05-15T20:06:05.246730Z","shell.execute_reply":"2022-05-15T20:06:05.256141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_tr, X_test_tr = select_kbest(X_train_x, y_train_x, X_test_x)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:06:08.990386Z","iopub.execute_input":"2022-05-15T20:06:08.990848Z","iopub.status.idle":"2022-05-15T20:06:09.448654Z","shell.execute_reply.started":"2022-05-15T20:06:08.990812Z","shell.execute_reply":"2022-05-15T20:06:09.447833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_tr.shape","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:07:06.271618Z","iopub.execute_input":"2022-05-15T20:07:06.271872Z","iopub.status.idle":"2022-05-15T20:07:06.277189Z","shell.execute_reply.started":"2022-05-15T20:07:06.271845Z","shell.execute_reply":"2022-05-15T20:07:06.276188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = train_xgboost(X_train_tr, y_train_x, X_test_tr)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:08:23.097085Z","iopub.execute_input":"2022-05-15T20:08:23.097681Z","iopub.status.idle":"2022-05-15T20:11:57.638949Z","shell.execute_reply.started":"2022-05-15T20:08:23.097639Z","shell.execute_reply":"2022-05-15T20:11:57.638139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('xgbfeatureselected.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:15:17.025557Z","iopub.execute_input":"2022-05-15T20:15:17.025851Z","iopub.status.idle":"2022-05-15T20:15:17.407835Z","shell.execute_reply.started":"2022-05-15T20:15:17.025821Z","shell.execute_reply":"2022-05-15T20:15:17.406942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This had a score of 1.01025 on the Kaggle board. This is just slightly below the Xgboost model trained on the full dataset","metadata":{}},{"cell_type":"markdown","source":"### Collinear Features","metadata":{}},{"cell_type":"code","source":"def remove_collinear_features(train, test, threshold=0.8):\n    \"\"\"Computes the correlation matrix for the train dataset and removes highly collinear features\n\n    above_threshold_features = {feature1: (feature1, feature2, feature3), etc}\n    where feature1, feature2, feature3 are features very highly correlated with feature1 (feature1 trivially so as corr=1)\n    \n    Returns: \n            train & test with highly correlated features dropped\n            \"\"\"\n    corr = train.corr()  # compute the correlation matrix\n    above_threshold_features = {} # dictionary to hold highly correlated features\n    # record only correlations above threshold\n    for col in corr:\n        above_threshold_features[col] = list(corr.index[corr[col] > threshold]) \n    \n    cols_to_remove = []\n    cols_seen = []\n    cols_to_remove_pair = []\n\n    for key, value in above_threshold_features.items():\n        cols_seen.append(key)\n        \n        for x in value:\n            if x == key:\n                next\n            else:\n                # only remove one in a pair\n                if x not in cols_seen:\n                    cols_to_remove.append(x)\n                    cols_to_remove_pair.append(key)\n    \n    cols_to_remove = list(set(cols_to_remove))\n    print('Number of columns to remove: ', len(cols_to_remove))\n\n    train = train.drop(columns=cols_to_remove)\n    test = test.drop(columns=cols_to_remove)\n\n    return train, test","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:20:03.464873Z","iopub.execute_input":"2022-05-15T20:20:03.465194Z","iopub.status.idle":"2022-05-15T20:20:03.473695Z","shell.execute_reply.started":"2022-05-15T20:20:03.465157Z","shell.execute_reply":"2022-05-15T20:20:03.472944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_tr, X_test_tr = remove_collinear_features(X_train_x, X_test_x)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:21:01.024561Z","iopub.execute_input":"2022-05-15T20:21:01.024822Z","iopub.status.idle":"2022-05-15T20:21:07.331181Z","shell.execute_reply.started":"2022-05-15T20:21:01.024796Z","shell.execute_reply":"2022-05-15T20:21:07.330291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let's train the xgboost model\nsubmission = train_xgboost(X_train_tr, y_train_x, X_test_tr)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:21:51.560483Z","iopub.execute_input":"2022-05-15T20:21:51.560732Z","iopub.status.idle":"2022-05-15T20:27:46.337669Z","shell.execute_reply.started":"2022-05-15T20:21:51.560705Z","shell.execute_reply":"2022-05-15T20:27:46.336928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('xgboostcollinear.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:28:11.404416Z","iopub.execute_input":"2022-05-15T20:28:11.404930Z","iopub.status.idle":"2022-05-15T20:28:11.923451Z","shell.execute_reply.started":"2022-05-15T20:28:11.404888Z","shell.execute_reply":"2022-05-15T20:28:11.922090Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This had a score of 1.00798... the best score yet","metadata":{}},{"cell_type":"markdown","source":"## Feature Importance","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier\n\nmodel = RandomForestClassifier(n_estimators=1000, random_state=42, n_jobs=-1, max_depth=5)\nmodel.fit(X_train, y_train)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:51:27.188514Z","iopub.execute_input":"2022-05-15T20:51:27.188770Z","iopub.status.idle":"2022-05-15T20:54:41.799294Z","shell.execute_reply.started":"2022-05-15T20:51:27.188743Z","shell.execute_reply":"2022-05-15T20:54:41.798507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_importances = model.feature_importances_\nfeature_importances","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:54:41.801014Z","iopub.execute_input":"2022-05-15T20:54:41.801270Z","iopub.status.idle":"2022-05-15T20:54:42.113343Z","shell.execute_reply.started":"2022-05-15T20:54:41.801237Z","shell.execute_reply":"2022-05-15T20:54:42.112473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train = train.drop(columns=['id', 'target', 'home_team_name', 'away_team_name', 'league_name'])\nfeature_importances = pd.DataFrame({'feature': list(X_train.columns), 'importance': feature_importances})","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:55:05.858579Z","iopub.execute_input":"2022-05-15T20:55:05.858833Z","iopub.status.idle":"2022-05-15T20:55:05.904993Z","shell.execute_reply.started":"2022-05-15T20:55:05.858805Z","shell.execute_reply":"2022-05-15T20:55:05.903990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_feature_importances(df):\n    \"\"\"Plot importances returned by a model. This works with any measure of feature importances provided that higher is better\n     Args:\n        df (dataframe): feature importances. Must have the features in a column\n        called `features` and the importances in a column called `importance\n\n    Returns: shows a plot of the 15 most importance features\n        df (dataframe): feature importances sorted by importance (highest to lowest) \n        with a column for normalized importance\n  \"\"\"\n    df = df.sort_values('importance', ascending=False).reset_index()\n  \n  # Normalize the feature importances to add up to one\n    df['importance_normalized'] = df['importance'] / df['importance'].sum()\n\n  # Make a horizontal bar chart of feature importances\n    plt.figure(figsize = (10, 6))\n    ax = plt.subplot()\n  \n  # Need to reverse the index to plot most important on top\n    ax.barh(list(reversed(list(df.index[:15]))), df['importance_normalized'].head(15), \n            align = 'center', edgecolor = 'k')\n    # Set the yticks and labels\n    ax.set_yticks(list(reversed(list(df.index[:15]))))\n    ax.set_yticklabels(df['feature'].head(15))\n    \n  # Plot labeling\n    plt.xlabel('Normalized Importance'); plt.title('Feature Importances')\n    plt.show()\n\n    return df","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:55:09.004085Z","iopub.execute_input":"2022-05-15T20:55:09.004552Z","iopub.status.idle":"2022-05-15T20:55:09.013296Z","shell.execute_reply.started":"2022-05-15T20:55:09.004515Z","shell.execute_reply":"2022-05-15T20:55:09.012081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_feature_importances(feature_importances)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T20:55:10.357759Z","iopub.execute_input":"2022-05-15T20:55:10.358316Z","iopub.status.idle":"2022-05-15T20:55:10.667631Z","shell.execute_reply.started":"2022-05-15T20:55:10.358276Z","shell.execute_reply":"2022-05-15T20:55:10.666977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### BorutaShap","metadata":{}},{"cell_type":"code","source":"X_train_boruta = train.drop(columns=['id', 'target', 'home_team_name', 'away_team_name', 'league_name'])\ny_train_boruta = train['target']","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:15:19.432855Z","iopub.execute_input":"2022-05-15T19:15:19.433356Z","iopub.status.idle":"2022-05-15T19:15:19.483748Z","shell.execute_reply.started":"2022-05-15T19:15:19.433319Z","shell.execute_reply":"2022-05-15T19:15:19.482821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train_boruta = X_train_boruta.fillna(X_train_boruta.mean())","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:17:56.197737Z","iopub.execute_input":"2022-05-15T19:17:56.198496Z","iopub.status.idle":"2022-05-15T19:17:56.431586Z","shell.execute_reply.started":"2022-05-15T19:17:56.198457Z","shell.execute_reply":"2022-05-15T19:17:56.430795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from BorutaShap import BorutaShap\n\nfeature_selector = BorutaShap(importance_measure='shap', classification=True)\nfeature_selector.fit(X_train_boruta, y_train_boruta, n_trials=20, random_state=42)","metadata":{"execution":{"iopub.status.busy":"2022-05-15T19:17:59.734904Z","iopub.execute_input":"2022-05-15T19:17:59.735537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_Selector.plot(which_features='all', figsize=(16,12))","metadata":{},"execution_count":null,"outputs":[]}]}