{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"***Please, if you have found this notebook interesting and helpful, don't forget to upvote :). It's always helpful!***","metadata":{}},{"cell_type":"code","source":"import pandas as pd \nimport numpy as np \nimport matplotlib.pyplot as plt \nimport seaborn as sns \n%matplotlib inline","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.set_option('display.max_columns', None)\n#pd.reset_option('all')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_path = 'C:/Users/redha.cherif_artefac/Desktop/Redha-s-Coding-Lab/Kaggle_competitions/Football_Match_Probability_Prediction/Input_files/'\n\ntrain = pd.read_csv(data_path + 'train.csv')\ntest = pd.read_csv(data_path + 'test.csv')\nsub_sample = pd.read_csv(data_path + 'sample_submission.csv')\ntrain_target_and_scores = pd.read_csv(data_path + 'train_target_and_scores.csv')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Train/test split understanding**<br>\nTrain set is composed of data from December 2010 to May 2021 (1y 5 months) whereas test set is going from May 2021 to November 2021","metadata":{}},{"cell_type":"code","source":"\"\"\" We have 1y and 5 months of data in train \n    We have 7 months of data in test \n    The train and test have been splitted according to time \"\"\"\n    \nprint('Train dates : from {} to {}'.format(train['match_date'].min(), train['match_date'].max()))\nprint('Test dates : from {} to {}'.format(test['match_date'].min(), test['match_date'].max()))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Target undestanding** <br>\nThanks to ALIFIA GHANTIWALA and his notebook : <br>\nhttps://www.kaggle.com/code/aliphya/football-match-probability-prediction-eda#Home-and-Away-team-name <br><br>\n\n- Most of targets are 'home' meaning that most of our predictions consist into getting the prediction of the home team to win. <br>\nAt least, 3 algorithms would have to be used to predict the probability of the home team to win, the probability of the away team to win and the probability of a draw.\n- **Important** : For each game in the test set, we'd have to predict the probability for the home team to win, the probability for the away team to win and the probability of a draw.<br>\nHence, the submission folder should have 218133 predictions organised in a 72 711 by 3 matrix.","metadata":{}},{"cell_type":"code","source":"sns.set_theme(style=\"darkgrid\")\norder = train['target'].value_counts(ascending=False).index\n\nax = sns.countplot(x='target',data=train,order=order)\n\nax.bar_label(container=ax.containers[0],labels=train.target.value_counts(ascending=False).values)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# These graphs don't give much information about the predictions to perform \n\nsns.set_theme(style='darkgrid')\nfig, ax = plt.subplots(1,2,figsize=(15,5))\n\n#Target distribution for top 5 most frequent home teams\nhome_team_list = train.home_team_name.value_counts(ascending=False).head(5).index\ntarget_dist_hteam = sns.countplot(ax=ax[0], x='home_team_name',hue='target',data=train,order=home_team_list)\nax[0].set_title('Target distribution for home teams')\ntarget_dist_hteam.legend(loc='upper right',bbox_to_anchor=(1.21,1))\n\n#Target distribution for top 5 most frequent away teams\naway_team_list = train.home_team_name.value_counts(ascending=False).head(5).index\ntarget_dist_ateam = sns.countplot(ax=ax[1], x='away_team_name',hue='target',data=train,order=away_team_list)\nax[1].set_title('Target distribution for away teams')\ntarget_dist_ateam.legend(loc='upper right',bbox_to_anchor=(1.21,1))\n\nplt.subplots_adjust(wspace=0.3)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Basic analytics for train and test sets**<br>\nThanks to ALIFIA GHANTIWALA and his notebook : <br>\nhttps://www.kaggle.com/code/aliphya/football-match-probability-prediction-eda#Home-and-Away-team-name <br><br>\n\n- There are 10 214 teams in the training set and 10 427 in the test (home + away teams) => There are 213 more teams in the test compared to the train <br>\nPotential explanation : it can be explained by the fact that new teams data have been recorded within 7 months (test set) which have not been in the training set.\n- Besides of the test having more teams, the proportion is different ie 2% of the teams present in the test are not present in the train at all (e.g Azerbaijan U19, Everton U21, Islami Kalkelea)\n- There are 9813 home teams and 9892 away teams in the training set vs 9925 and 9982 in the test set\n- In the 5 most frequent teams, 3 teams are different comparing train and test : Al Ittihad, Liverpool and Al Ahli. \n\n**Conclusions:** <br> \n- Regarding the validation set, a strategy might be a take the same size as the test set (7 months) and try to choose 2% of teams which won't be in the train set.","metadata":{}},{"cell_type":"code","source":"total_nb_of_teams_train = train[['home_team_name','away_team_name']].stack().drop_duplicates().count()\ntotal_nb_of_teams_test = test[['home_team_name','away_team_name']].stack().drop_duplicates().count()\nprint(f'The total number of teams in the training set (home + away) is equal to {total_nb_of_teams_train}')\nprint(f'The total number of teams in the test set (home + away) is equal to {total_nb_of_teams_test}')\nprint(f'There are {total_nb_of_teams_test - total_nb_of_teams_train} more teams in the test set compared to the train')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'The number of home teams in the training set is : {train.home_team_name.nunique()}')\nprint(f'The number of home teams in the test set is : {test.home_team_name.nunique()}')\nprint()\nprint(f'The number of away teams in the training set is : {train.away_team_name.nunique()}')\nprint(f'The number of away teams in the test set is : {test.away_team_name.nunique()}')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('The 5 most frequent teams in the training set :')\ntrain[['home_team_name','away_team_name']].stack().value_counts().head(5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('The 5 most frequent teams in the test set :')\ntest[['home_team_name','away_team_name']].stack().value_counts().head(5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'The 5 most frequent home teams in the training set :')\ntrain.home_team_name.value_counts().head(5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'The 5 most frequent home teams in the test set :')\ntest.home_team_name.value_counts().head(5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'The 5 most frequent away teams in the training set :')\ntrain.away_team_name.value_counts().head(5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'The 5 most frequent away teams in the test set :')\ntest.away_team_name.value_counts().head(5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Which teams are in the test set but aren't in the training one ? \n\ntotal_nb_of_teams_train = train[['home_team_name','away_team_name']].stack().drop_duplicates()\ntotal_nb_of_teams_test = test[['home_team_name','away_team_name']].stack().drop_duplicates()\n\ntotal_nb_of_teams_train = set(total_nb_of_teams_train)\ntotal_nb_of_teams_test = set(total_nb_of_teams_test)\ndiff = total_nb_of_teams_test - total_nb_of_teams_train\nprint(f'There are {len(diff)} teams which are in the test set but are not in the train')\nprint(f'It represents {round(len(diff)/len(test)*100,1)}% hence out of 100 teams in the test set, 2 are not present in the train')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Footbalistic performance analysis**\n\n- Section in progress","metadata":{}},{"cell_type":"code","source":"#Take the score column from train_target_and_scores\ntrain = train.merge(train_target_and_scores[['id','score']],how='left',on='id')\n\n# Split score columns into two to create a column HasWon \ngoals = train.score.str.split('-',n=1,expand=True)\ntrain['home_goals'] = goals[0]\ntrain['away_goals'] = goals[1]\n\n# Create a column which gives the last match result \ntrain.loc[train['home_goals'] > train['away_goals'], 'home_victory_match'] = 1\ntrain.loc[train['home_goals'] == train['away_goals'], 'home_victory_match'] = 0\ntrain.loc[train['home_goals'] < train['away_goals'], 'home_victory_match'] = 0\n\ntrain.loc[train['home_goals'] > train['away_goals'], 'home_draw_match'] = 0\ntrain.loc[train['home_goals'] == train['away_goals'], 'home_draw_match'] = 1\ntrain.loc[train['home_goals'] < train['away_goals'], 'home_draw_match'] = 0\n\n# Relocate score columns\nsixth_column = train.pop('score')\ntrain.insert(5,'score',sixth_column)\n\nseventh_column = train.pop('home_goals')\ntrain.insert(6,'home_goals',seventh_column)\n\neighth_column = train.pop('away_goals')\ntrain.insert(7,'away_goals',eighth_column)\n\nnineth_column = train.pop('home_victory_match')\ntrain.insert(8,'home_victory_match',nineth_column)\n\ntenth_column = train.pop('home_draw_match')\ntrain.insert(9,'home_draw_match',tenth_column)\n\n#Contact train and test\ndata = pd.concat(objs=[train,test])\n\n#Drop unuseful columns \ndata.drop(labels=['league_id','home_team_coach_id','away_team_coach_id'],axis=1, inplace=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Loop to concatenate columns to get the scores of the last 10 matches of home teams \ni = 1 \nfor cols in data.columns :\n    if 'home_team_history_goal' in cols : \n        i = str(i)\n        data['home_team_history_score_' + i] = data['home_team_history_goal_' + i].astype(str) + '-' + data['home_team_history_opponent_goal_' + i].astype(str)\n        i = int(i)\n        i = i + 1","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Loop to concatenate columns to get the scores of the last 10 matches of away teams \ni = 1 \nfor cols in data.columns :\n    if 'away_team_history_goal' in cols : \n        i = str(i)\n        data['away_team_history_score_' + i] = data['away_team_history_goal_' + i].astype(str) + '-' + data['away_team_history_opponent_goal_' + i].astype(str)\n        i = int(i)\n        i = i + 1","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Loop to create new victory and raw columns for home teams\ni = 1 \nfor cols in data.columns :\n    if 'home_team_history_goal' in cols :\n        data.loc[data['home_team_history_goal_' + str(i)] > data['home_team_history_opponent_goal_' + str(i)], 'home_victory_match_'+str(i)] = 1\n        data.loc[data['home_team_history_goal_' + str(i)] == data['home_team_history_opponent_goal_' + str(i)], 'home_victory_match_'+str(i)] = 0\n        data.loc[data['home_team_history_goal_' + str(i)] < data['home_team_history_opponent_goal_' + str(i)], 'home_victory_match_'+str(i)] = 0\n\n        data.loc[data['home_team_history_goal_' + str(i)] > data['home_team_history_opponent_goal_' + str(i)], 'home_draw_match_'+str(i)] = 0\n        data.loc[data['home_team_history_goal_' + str(i)] == data['home_team_history_opponent_goal_' + str(i)], 'home_draw_match_'+str(i)] = 1\n        data.loc[data['home_team_history_goal_' + str(i)] < data['home_team_history_opponent_goal_' + str(i)], 'home_draw_match_'+str(i)] = 0\n        \n        i = i + 1","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Loop to create new victory and raw columns for away teams\ni = 1 \nfor cols in data.columns :\n    if 'away_team_history_goal' in cols :\n        data.loc[data['away_team_history_goal_' + str(i)] > data['away_team_history_opponent_goal_' + str(i)], 'away_victory_match_'+str(i)] = 1\n        data.loc[data['away_team_history_goal_' + str(i)] == data['away_team_history_opponent_goal_' + str(i)], 'away_victory_match_'+str(i)] = 0\n        data.loc[data['away_team_history_goal_' + str(i)] < data['away_team_history_opponent_goal_' + str(i)], 'away_victory_match_'+str(i)] = 0\n\n        data.loc[data['away_team_history_goal_' + str(i)] > data['away_team_history_opponent_goal_' + str(i)], 'away_draw_match_'+str(i)] = 0\n        data.loc[data['away_team_history_goal_' + str(i)] == data['away_team_history_opponent_goal_' + str(i)], 'away_draw_match_'+str(i)] = 1\n        data.loc[data['away_team_history_goal_' + str(i)] < data['away_team_history_opponent_goal_' + str(i)], 'away_draw_match_'+str(i)] = 0\n        \n        i = i + 1","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Delete columns which allowed to build the previous features  \nfor cols in data.columns :\n    if 'home_team_history_goal' in cols :\n        data.drop(labels=[cols],axis=1,inplace=True)\n    if 'away_team_history_goal' in cols :\n        data.drop(labels=[cols],axis=1,inplace=True)    \n    if 'home_team_history_opponent_goal' in cols :\n        data.drop(labels=[cols],axis=1,inplace=True)   \n    if 'away_team_history_opponent_goal' in cols :\n        data.drop(labels=[cols],axis=1,inplace=True)    ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_copy = data.copy()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Size down the data_copy df to make it more understandable \nfor cols in data_copy.columns:\n    if 'team_history_is_play_' in cols :\n        data_copy.drop(labels=[cols],axis=1,inplace=True)\n    if 'team_history_is_cup_' in cols :\n        data_copy.drop(labels=[cols],axis=1,inplace=True)\n    if 'team_history_coach_' in cols :\n        data_copy.drop(labels=[cols],axis=1,inplace=True)\n    if 'team_history_league_id' in cols :\n        data_copy.drop(labels=[cols],axis=1,inplace=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_copy.head(2)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split the data_copy into train/test \ntrain_copy = data_copy[:len(train)]\ntest_copy = data_copy[len(train):]\n\n# Look at season matches and cup ones alone\ntrain_copy_season = train_copy[train_copy['is_cup'] == False]\ntrain_copy_cup = train_copy[train_copy['is_cup'] == True]\n\ntest_copy_season = test_copy[test_copy['is_cup'] == False]\ntest_copy_cup = test_copy[test_copy['is_cup'] == True]","metadata":{},"execution_count":null,"outputs":[]}]}