{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-03-12T11:53:59.350802Z","iopub.execute_input":"2022-03-12T11:53:59.351465Z","iopub.status.idle":"2022-03-12T11:53:59.391933Z","shell.execute_reply.started":"2022-03-12T11:53:59.351351Z","shell.execute_reply":"2022-03-12T11:53:59.390868Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This notebook starts from [this](https://www.kaggle.com/aliphya/football-match-probability-prediction-eda), many thanks to the author [Alifia Ghantiwala](https://www.kaggle.com/aliphya)","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/football-match-probability-prediction/train.csv\")\ntest = pd.read_csv(\"/kaggle/input/football-match-probability-prediction/test.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:53:59.394434Z","iopub.execute_input":"2022-03-12T11:53:59.395105Z","iopub.status.idle":"2022-03-12T11:54:10.267529Z","shell.execute_reply.started":"2022-03-12T11:53:59.395052Z","shell.execute_reply":"2022-03-12T11:54:10.266524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:10.270991Z","iopub.execute_input":"2022-03-12T11:54:10.271453Z","iopub.status.idle":"2022-03-12T11:54:11.437664Z","shell.execute_reply.started":"2022-03-12T11:54:10.271403Z","shell.execute_reply":"2022-03-12T11:54:11.436635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:11.439517Z","iopub.execute_input":"2022-03-12T11:54:11.440152Z","iopub.status.idle":"2022-03-12T11:54:11.547658Z","shell.execute_reply.started":"2022-03-12T11:54:11.440103Z","shell.execute_reply":"2022-03-12T11:54:11.546767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.describe()","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:11.550197Z","iopub.execute_input":"2022-03-12T11:54:11.550431Z","iopub.status.idle":"2022-03-12T11:54:12.763729Z","shell.execute_reply.started":"2022-03-12T11:54:11.550403Z","shell.execute_reply":"2022-03-12T11:54:12.763092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Descriptive columns\n* target - The variable you have to predict the probabilities only available in the train set.\n* home_team_name - The name of the Home the team.\n* away_team_name - The name of the Away the team.\n* match_date - The match date (UTC).\n* league_name - The league name.\n* league_id - The league id. Note that league names can be identical for two differents id.\n* is_cup - If the value is 1 the match is played for a cup compettion.\n* home_team_coach_id - The id of the Home team coach.\n* away_team_coach_id - The id of the Away team coach.\n\n\nHistorical home team features\n\n* home_team_history_match_date_{i} - The date of the last i-th match played by Home team.\n* home_team_history_is_play_home_{i} - If 1, the Home team played home.\n* home_team_history_is_cup_{i} - If 1, the match was a cup competition.\n* home_team_history_goal_{i} - The number of goals scored by the Home team on its last i-th match.\n* home_team_history_opponent_goal_{i} - The number of goals conceded by the Home team on its last i-th match.\n* home_team_history_rating_{i} - The rating of the Home team on its last i-th match (pre match rating).\n* home_team_history_opponent_rating_{i} - The rating of the opponent team on Home team last i-th match (pre match rating).\n* home_team_history_coach_{i} - The coach id of the Home team on its last i-th match.\n* home_team_history_league_id_{i} - The league name id by the Home team on its last i-th match.\n\n\nHistorical away team features\n\n* away_team_history_match_date_{i} - The date of the last i-th match played by Away team.\n* away_team_history_is_play_home_{i} - If 1, the Away team played home.\n* away_team_history_is_cup_{i} - If 1, the match was a cup competition.\n* away_team_history_goal_{i} - The number of goals scored by the Away team on its last i-th match.\n* away_team_history_opponent_goal_{i} - The number of goals conceded by the Away team on its last i-th match.\n* away_team_history_rating_{i} - The rating of the Away team on its last i-th match (pre match rating).\n* away_team_history_opponent_rating_{i} - The rating of the opponent team on Away team last i-th match (pre match rating).\n* away_team_history_coach_{i} - The coach id of the Away team on its last i-th match.\n* away_team_history_league_id_{i} - The league name id played by the Away on its last i-th match.","metadata":{}},{"cell_type":"markdown","source":"**We have to predict the probabilty of the home team winning, probability of the away team winning and the probability of a draw.**\n\n**Let us first try to gain some insights into the data**","metadata":{}},{"cell_type":"markdown","source":"# Feature exploration","metadata":{}},{"cell_type":"markdown","source":"### Team Coach History","metadata":{}},{"cell_type":"code","source":"coaches = train.filter(regex=(\"home_team_history_coach_[0-9]\")).stack().groupby(level=0).apply(lambda x: x.unique().tolist())","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:12.765092Z","iopub.execute_input":"2022-03-12T11:54:12.765582Z","iopub.status.idle":"2022-03-12T11:54:20.525553Z","shell.execute_reply.started":"2022-03-12T11:54:12.765549Z","shell.execute_reply":"2022-03-12T11:54:20.524714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"coaches.map(lambda x: len(x)).value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:20.526748Z","iopub.execute_input":"2022-03-12T11:54:20.526967Z","iopub.status.idle":"2022-03-12T11:54:20.598171Z","shell.execute_reply.started":"2022-03-12T11:54:20.526940Z","shell.execute_reply":"2022-03-12T11:54:20.596008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Team League History","metadata":{}},{"cell_type":"code","source":"league_ids = train.filter(regex=(\"home_team_history_league_id_[0-9]\")).stack().groupby(level=0).apply(lambda x: x.unique().tolist())","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:20.599284Z","iopub.execute_input":"2022-03-12T11:54:20.599853Z","iopub.status.idle":"2022-03-12T11:54:29.616657Z","shell.execute_reply.started":"2022-03-12T11:54:20.599813Z","shell.execute_reply":"2022-03-12T11:54:29.615687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"league_ids.map(lambda x: len(x)).value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:29.618431Z","iopub.execute_input":"2022-03-12T11:54:29.619034Z","iopub.status.idle":"2022-03-12T11:54:29.696054Z","shell.execute_reply.started":"2022-03-12T11:54:29.618987Z","shell.execute_reply":"2022-03-12T11:54:29.694976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Missing Data","metadata":{}},{"cell_type":"markdown","source":"We have a lot of historical data missing, some descriptive columns like is cup, league name, away team name also have some missing data.\n\nWe would have to think of a strategy to impute this missing data.\n\nWe do not have any feature with more than 30% of its data being missing.","metadata":{}},{"cell_type":"code","source":"count_missing = pd.DataFrame([train.isna().mean(),test.isna().mean()]).T\ncount_missing = count_missing.rename(columns={0: \"train_missing\", 1: \"test_missing\"})","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:29.697786Z","iopub.execute_input":"2022-03-12T11:54:29.698025Z","iopub.status.idle":"2022-03-12T11:54:30.259818Z","shell.execute_reply.started":"2022-03-12T11:54:29.697997Z","shell.execute_reply":"2022-03-12T11:54:30.258948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"count_missing.query(\"train_missing > 0 and train_missing < 0.1\").plot(kind=\"barh\", figsize=(15, 30), title=\"% of Values Missing\")","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:30.261221Z","iopub.execute_input":"2022-03-12T11:54:30.261873Z","iopub.status.idle":"2022-03-12T11:54:34.183595Z","shell.execute_reply.started":"2022-03-12T11:54:30.261835Z","shell.execute_reply":"2022-03-12T11:54:34.182804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check if the missing data are contiguous, this can happen by creating the time series for the first games in time order","metadata":{}},{"cell_type":"code","source":"!pip install missingno\nimport missingno as msno","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:34.184814Z","iopub.execute_input":"2022-03-12T11:54:34.185129Z","iopub.status.idle":"2022-03-12T11:54:45.578570Z","shell.execute_reply.started":"2022-03-12T11:54:34.185093Z","shell.execute_reply":"2022-03-12T11:54:45.577685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(train.filter(regex=(\"[a-zA-Z]*_1$\")))","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:45.581256Z","iopub.execute_input":"2022-03-12T11:54:45.581673Z","iopub.status.idle":"2022-03-12T11:54:47.016761Z","shell.execute_reply.started":"2022-03-12T11:54:45.581619Z","shell.execute_reply":"2022-03-12T11:54:47.015772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(train.filter(regex=(\"[a-zA-Z]*_3$\")))","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:47.020727Z","iopub.execute_input":"2022-03-12T11:54:47.021073Z","iopub.status.idle":"2022-03-12T11:54:48.427711Z","shell.execute_reply.started":"2022-03-12T11:54:47.021030Z","shell.execute_reply":"2022-03-12T11:54:48.426669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(train.filter(regex=(\"[a-zA-Z]*_6$\")))","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:48.429101Z","iopub.execute_input":"2022-03-12T11:54:48.429780Z","iopub.status.idle":"2022-03-12T11:54:49.884837Z","shell.execute_reply.started":"2022-03-12T11:54:48.429740Z","shell.execute_reply":"2022-03-12T11:54:49.883199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(train.filter(regex=(\"[a-zA-Z]*_9$\")))","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:49.886498Z","iopub.execute_input":"2022-03-12T11:54:49.886817Z","iopub.status.idle":"2022-03-12T11:54:51.311486Z","shell.execute_reply.started":"2022-03-12T11:54:49.886784Z","shell.execute_reply":"2022-03-12T11:54:51.310495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Sample analysis, keeping the history index fixed, seems to fairly confirm the contiguity hypothesis. Moreover the number of missing value increase with the index of series.","metadata":{}},{"cell_type":"code","source":"msno.matrix(train.filter(regex=(\"home_team_history_match_date_[0-9]+$\")))","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:51.313029Z","iopub.execute_input":"2022-03-12T11:54:51.313353Z","iopub.status.idle":"2022-03-12T11:54:52.430887Z","shell.execute_reply.started":"2022-03-12T11:54:51.313308Z","shell.execute_reply":"2022-03-12T11:54:52.429744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(train.filter(regex=(\"home_team_history_opponent_goal_[0-9]+$\")))","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:52.432241Z","iopub.execute_input":"2022-03-12T11:54:52.432489Z","iopub.status.idle":"2022-03-12T11:54:53.399611Z","shell.execute_reply.started":"2022-03-12T11:54:52.432460Z","shell.execute_reply":"2022-03-12T11:54:53.398762Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.matrix(train.filter(regex=(\"home_team_history_league_id_[0-9]+$\")))","metadata":{"execution":{"iopub.status.busy":"2022-03-12T11:54:53.400729Z","iopub.execute_input":"2022-03-12T11:54:53.401026Z","iopub.status.idle":"2022-03-12T11:54:54.354121Z","shell.execute_reply.started":"2022-03-12T11:54:53.400996Z","shell.execute_reply":"2022-03-12T11:54:54.353334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Keeping the feature fixed we also have a similar result which confirms the hypothesis.","metadata":{}},{"cell_type":"markdown","source":"This leads us to think that even in the test set (which we will not analyze, since it must be considered as obscure) it may have missing contiguous values that grow with the increase of the index, so it is good not to delete the rows with missing values, to generalize better on partial data.","metadata":{}},{"cell_type":"markdown","source":"## to be continued...","metadata":{}}]}