{ "id": 322014, "title": "Training model without imputing missing values", "url": "/competitions/football-match-probability-prediction/discussion/322014", "author_name": "Tomas Uždavinys", "post_date": "2022-04-29T17:54:05.375000", "votes": 3, "comment_count": 2, "views": 0, "content": "

Hi, this is my first comment ever😅

\n

Data for this competition was rather messy- large number of rows had missing values. I saw that most public notebooks used some sort of missing data imputation techniques. I really like expression “garbage in garbage out” [1], so I wanted to create model without imputing missing values. My results are presented here: https://www.kaggle.com/uzdavinys/log-regression-with-no-imputed-nan-values

\n

We have 2 types of features:
\ni) Information about current match (date, league, ect),
\nii) Information about historical matches for 2 opposing teams.

\n

Features in the second group can be split into 9 sub-groups:

\n\n

Considering max 10 historical games for both teams, we can have up to 180 historical features. There might be matches, there information is available only for last 2 games and for other cases we have all information for both home and away team’s last’s 10 matches. In the best case scenario I would like to know information about all 9 sub-group features for last n games.

\n

From the pie chat (see notebook) one can see that only 58.3% of train dataset rows had no missing values for all 9 features and 18.1% of data had at least one missing data point. Model for these rows can’t be trained without imputing at least one missing value. For the test dataset this proportion is even larger 60.7% and 31.5%, respectively.

\n

Let’s try to minimize requirements for features- we would like to know the number of goals score in last n matches. Results are much better, only 1.79% (1985 matches) for train dataset didn’t have information about number of goals scored. Of course, the real problem if much deeper….

\n

\"\"

\n

Out of 1985 matches, 400 don’t have any information about home and away games, but 157 matches have information only about last 1 home game, and 158 matches have information about 1 away game. To make things even more complicated, out of selected 1985 matches, 375 games have information about last 10 home games and 0 away games (heat map in the notebook). If you wanted to train model for these 375 games you can either

\n

i) exclude those 10 historical home games from your analysis (train model with no information about historical games),
\nii) use data only from last 10 historical home games, i.e. train model as if you know nothing about away team.

\n

I could use 100% of my train data to train models there no historical data was used. For matches where information is known about last 1 game, I could use 98.21% of train data and so on.

\n

My idea was instead of imputing missing values I will create many log. regression models for different amount of available not missing data. Off course, the more information about last n games is known the less matches will be used for training, i.e. try to make predictions with highest amount of available info.

\n

I used cross-validation (CV) to test different features. First, I selected 58.3% of train dataset with no missing data and always did 5-fold CV using different features. I used boxplot (with plotly to interactively see quartile values) and compared different sets of features using log-loss as validation score. In notebook function **select_features ** explains how each feature was generated.

\n

In my first attempt, I investigated the number of goals and date (number of days since the game we are making predicts). Instead of using the number of goals scored by each team at nth match, I tried using goal difference as a feature.

\n

The more data was provide for the model (number of historical games) the more accurate my average validation score became. Using goal difference instead of number of goals scored reduced validation score. The best approach was to use number of goals scored by home, away and their opponents with day difference since nth match.

\n

In my second attempt I used last 1 game of away or home team and then increased number of home or away games, respectively. Again, the more information we have, the lower validation score became.

\n

If I must choose whatever to train model with 1 away game and 10 home historical games (orange box plots) or 10 away and 10 home games (blue boxes), I will always choose the second. Here is the image \"\".

\n

This means that for cases where information is available for n=2 away games and n=6 home games, it might be worth to train model with asymmetrical number of features instead of using last 2 games for home and away sides.

\n

In my last attempt I investigated remaining features. The best improvement came from using ratings. Other notable improvements were achieved by using is cup and same league features than using just goals and dates.

\n

While I didn’t manage to get <1 public LB score, I believe this notebook shows that maybe creating LSTM’s for different number of historical games could produce better results than training on whole dataset with imputed missing values. Of course, it’s much faster to train 100 log. regression models than one LSTM. So, this was just an idea.

\n

[1] https://www.youtube.com/watch?v=pcmUdXIJQ74&list=PLPnZfvKID1Sje5jWxt-4CSZD7bUI4gSPS&index=26

", "messages": [ { "id": 1771984, "postDate": "2022-04-29T17:54:05.377Z", "content": "

Hi, this is my first comment ever😅

\n

Data for this competition was rather messy- large number of rows had missing values. I saw that most public notebooks used some sort of missing data imputation techniques. I really like expression “garbage in garbage out” [1], so I wanted to create model without imputing missing values. My results are presented here: https://www.kaggle.com/uzdavinys/log-regression-with-no-imputed-nan-values

\n

We have 2 types of features:
\ni) Information about current match (date, league, ect),
\nii) Information about historical matches for 2 opposing teams.

\n

Features in the second group can be split into 9 sub-groups:

\n\n

Considering max 10 historical games for both teams, we can have up to 180 historical features. There might be matches, there information is available only for last 2 games and for other cases we have all information for both home and away team’s last’s 10 matches. In the best case scenario I would like to know information about all 9 sub-group features for last n games.

\n

From the pie chat (see notebook) one can see that only 58.3% of train dataset rows had no missing values for all 9 features and 18.1% of data had at least one missing data point. Model for these rows can’t be trained without imputing at least one missing value. For the test dataset this proportion is even larger 60.7% and 31.5%, respectively.

\n

Let’s try to minimize requirements for features- we would like to know the number of goals score in last n matches. Results are much better, only 1.79% (1985 matches) for train dataset didn’t have information about number of goals scored. Of course, the real problem if much deeper….

\n

\"\"

\n

Out of 1985 matches, 400 don’t have any information about home and away games, but 157 matches have information only about last 1 home game, and 158 matches have information about 1 away game. To make things even more complicated, out of selected 1985 matches, 375 games have information about last 10 home games and 0 away games (heat map in the notebook). If you wanted to train model for these 375 games you can either

\n

i) exclude those 10 historical home games from your analysis (train model with no information about historical games),
\nii) use data only from last 10 historical home games, i.e. train model as if you know nothing about away team.

\n

I could use 100% of my train data to train models there no historical data was used. For matches where information is known about last 1 game, I could use 98.21% of train data and so on.

\n

My idea was instead of imputing missing values I will create many log. regression models for different amount of available not missing data. Off course, the more information about last n games is known the less matches will be used for training, i.e. try to make predictions with highest amount of available info.

\n

I used cross-validation (CV) to test different features. First, I selected 58.3% of train dataset with no missing data and always did 5-fold CV using different features. I used boxplot (with plotly to interactively see quartile values) and compared different sets of features using log-loss as validation score. In notebook function **select_features ** explains how each feature was generated.

\n

In my first attempt, I investigated the number of goals and date (number of days since the game we are making predicts). Instead of using the number of goals scored by each team at nth match, I tried using goal difference as a feature.

\n

The more data was provide for the model (number of historical games) the more accurate my average validation score became. Using goal difference instead of number of goals scored reduced validation score. The best approach was to use number of goals scored by home, away and their opponents with day difference since nth match.

\n

In my second attempt I used last 1 game of away or home team and then increased number of home or away games, respectively. Again, the more information we have, the lower validation score became.

\n

If I must choose whatever to train model with 1 away game and 10 home historical games (orange box plots) or 10 away and 10 home games (blue boxes), I will always choose the second. Here is the image \"\".

\n

This means that for cases where information is available for n=2 away games and n=6 home games, it might be worth to train model with asymmetrical number of features instead of using last 2 games for home and away sides.

\n

In my last attempt I investigated remaining features. The best improvement came from using ratings. Other notable improvements were achieved by using is cup and same league features than using just goals and dates.

\n

While I didn’t manage to get <1 public LB score, I believe this notebook shows that maybe creating LSTM’s for different number of historical games could produce better results than training on whole dataset with imputed missing values. Of course, it’s much faster to train 100 log. regression models than one LSTM. So, this was just an idea.

\n

[1] https://www.youtube.com/watch?v=pcmUdXIJQ74&list=PLPnZfvKID1Sje5jWxt-4CSZD7bUI4gSPS&index=26

", "rawMarkdown": "Hi, this is my first comment ever😅 \n\nData for this competition was rather messy- large number of rows had missing values. I saw that most public notebooks used some sort of missing data imputation techniques. I really like expression “garbage in garbage out” [1], so I wanted to create model without imputing missing values. My results are presented here: https://www.kaggle.com/uzdavinys/log-regression-with-no-imputed-nan-values\n\nWe have 2 types of features:\ni)\tInformation about current match (date, league, ect),\nii)\tInformation about historical matches for 2 opposing teams.\n\nFeatures in the second group can be split into 9 sub-groups:\n- match_date,\n- is_play_home,\n- is_cup,\n- goal,\n- opponent_goal,\n- rating,\n- opponent_rating,\n- coach,\n- league_id.\n\nConsidering max 10 historical games for both teams, we can have up to 180 historical features. There might be matches, there information is available only for last 2 games and for other cases we have all information for both home and away team’s last’s 10 matches. In the best case scenario I would like to know information about all 9 sub-group features for last n games.\n\nFrom the pie chat (see notebook) one can see that only 58.3% of train dataset rows had no missing values for all 9 features and 18.1% of data had at least one missing data point. Model for these rows can’t be trained without imputing at least one missing value. For the test dataset this proportion is even larger 60.7% and 31.5%, respectively.\n\nLet’s try to minimize requirements for features- we would like to know the number of goals score in last n matches. Results are much better, only 1.79% (1985 matches) for train dataset didn’t have information about number of goals scored. Of course, the real problem if much deeper…. \n\n![](https://www.dropbox.com/s/5vo3519sx1cy51e/fig4.PNG?raw=1)\n\nOut of 1985 matches, 400 don’t have any information about home and away games, but 157 matches have information only about last 1 home game, and 158 matches have information about 1 away game. To make things even more complicated, out of selected 1985 matches, 375 games have information about last 10 home games and 0 away games (heat map in the notebook). If you wanted to train model for these 375 games you can either\n\ni) exclude those 10 historical home games from your analysis (train model with no information about historical games),\nii) use data only from last 10 historical home games, i.e. train model as if you know nothing about away team.\n\nI could use 100% of my train data to train models there no historical data was used. For matches where information is known about last 1 game, I could use 98.21% of train data and so on. \n\nMy idea was instead of imputing missing values I will create many log. regression models for different amount of available not missing data. Off course, the more information about last n games is known the less matches will be used for training, i.e. try to make predictions with highest amount of available info.\n\nI used cross-validation (CV) to test different features. First, I selected 58.3% of train dataset with no missing data and always did 5-fold CV using different features. I used boxplot (with plotly to interactively see quartile values) and compared different sets of features using log-loss as validation score. In notebook function **select_features ** explains how each feature was generated.\n\nIn my first attempt, I investigated the number of goals and date (number of days since the game we are making predicts). Instead of using the number of goals scored by each team at nth match, I tried using goal difference as a feature.\n\nThe more data was provide for the model (number of historical games) the more accurate my average validation score became. Using goal difference instead of number of goals scored reduced validation score. The best approach was to use number of goals scored by home, away and their opponents with day difference since nth match.\n\nIn my second attempt I used last 1 game of away or home team and then increased number of home or away games, respectively. Again, the more information we have, the lower validation score became.\n\nIf I must choose whatever to train model with 1 away game and 10 home historical games (orange box plots) or 10 away and 10 home games (blue boxes), I will always choose the second. Here is the image ![](https://www.dropbox.com/s/2vb4usk41eof52u/fig3.PNG?raw=1). \n\nThis means that for cases where information is available for n=2 away games and n=6 home games, it might be worth to train model with asymmetrical number of features instead of using last 2 games for home and away sides.\n\nIn my last attempt I investigated remaining features. The best improvement came from using ratings. Other notable improvements were achieved by using is cup and same league features than using just goals and dates.\n\nWhile I didn’t manage to get <1 public LB score, I believe this notebook shows that maybe creating LSTM’s for different number of historical games could produce better results than training on whole dataset with imputed missing values. Of course, it’s much faster to train 100 log. regression models than one LSTM. So, this was just an idea.\n\n[1] https://www.youtube.com/watch?v=pcmUdXIJQ74&list=PLPnZfvKID1Sje5jWxt-4CSZD7bUI4gSPS&index=26\n\n\n", "votes": 3 }, { "id": 1774559, "postDate": "2022-05-02T09:13:53.390Z", "rawMarkdown": "", "votes": 2, "isDeleted": true, "replies": [ { "id": 1777515, "postDate": "2022-05-04T16:23:09.677Z", "content": "

Great that you found it interesting! I tested LSTM myself without imputing or masking any missing values, i.e. I trained my 7 different models on different amount of missing data :D For rows with large proportion of missing values (n>=8), I used log. regression to make predictions.

\n

The public LSTM notebook by FERNANDO SERAQUEVENCE which masked missing values had a score of 0.99661. My best LB score using LSTM without imputing or masking missing values was 0.99765 (I engineered slightly different features). Results are rather similar. It will be interesting to compare these results after whole test dataset will be released.

\n

The biggest drawback of my approach is long training times- you need more models. I think this dataset/competition presented a really good environment to at least test this approach (I ran out of GPU quota really fast :D).

", "rawMarkdown": "Great that you found it interesting! I tested LSTM myself without imputing or masking any missing values, i.e. I trained my 7 different models on different amount of missing data :D For rows with large proportion of missing values (n>=8), I used log. regression to make predictions.\n\nThe public LSTM notebook by FERNANDO SERAQUEVENCE which masked missing values had a score of 0.99661. My best LB score using LSTM without imputing or masking missing values was 0.99765 (I engineered slightly different features). Results are rather similar. It will be interesting to compare these results after whole test dataset will be released.\n\nThe biggest drawback of my approach is long training times- you need more models. I think this dataset/competition presented a really good environment to at least test this approach (I ran out of GPU quota really fast :D). ", "votes": 1 } ] } ], "comments": [ { "id": 1774559, "author_name": "", "author_url": "", "post_date": "2022-05-02T09:13:53.390000", "content": "", "votes": 2, "replies": [ { "id": 1777515, "author_name": "Tomas Uždavinys", "author_url": "", "post_date": "2022-05-04T16:23:09.677000", "content": "

Great that you found it interesting! I tested LSTM myself without imputing or masking any missing values, i.e. I trained my 7 different models on different amount of missing data :D For rows with large proportion of missing values (n>=8), I used log. regression to make predictions.

\n

The public LSTM notebook by FERNANDO SERAQUEVENCE which masked missing values had a score of 0.99661. My best LB score using LSTM without imputing or masking missing values was 0.99765 (I engineered slightly different features). Results are rather similar. It will be interesting to compare these results after whole test dataset will be released.

\n

The biggest drawback of my approach is long training times- you need more models. I think this dataset/competition presented a really good environment to at least test this approach (I ran out of GPU quota really fast :D).

", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1771984": "Hi, this is my first comment ever😅 \n\nData for this competition was rather messy- large number of rows had missing values. I saw that most public notebooks used some sort of missing data imputation techniques. I really like expression “garbage in garbage out” [1], so I wanted to create model without imputing missing values. My results are presented here: https://www.kaggle.com/uzdavinys/log-regression-with-no-imputed-nan-values\n\nWe have 2 types of features:\ni)\tInformation about current match (date, league, ect),\nii)\tInformation about historical matches for 2 opposing teams.\n\nFeatures in the second group can be split into 9 sub-groups:\n- match_date,\n- is_play_home,\n- is_cup,\n- goal,\n- opponent_goal,\n- rating,\n- opponent_rating,\n- coach,\n- league_id.\n\nConsidering max 10 historical games for both teams, we can have up to 180 historical features. There might be matches, there information is available only for last 2 games and for other cases we have all information for both home and away team’s last’s 10 matches. In the best case scenario I would like to know information about all 9 sub-group features for last n games.\n\nFrom the pie chat (see notebook) one can see that only 58.3% of train dataset rows had no missing values for all 9 features and 18.1% of data had at least one missing data point. Model for these rows can’t be trained without imputing at least one missing value. For the test dataset this proportion is even larger 60.7% and 31.5%, respectively.\n\nLet’s try to minimize requirements for features- we would like to know the number of goals score in last n matches. Results are much better, only 1.79% (1985 matches) for train dataset didn’t have information about number of goals scored. Of course, the real problem if much deeper…. \n\n![](https://www.dropbox.com/s/5vo3519sx1cy51e/fig4.PNG?raw=1)\n\nOut of 1985 matches, 400 don’t have any information about home and away games, but 157 matches have information only about last 1 home game, and 158 matches have information about 1 away game. To make things even more complicated, out of selected 1985 matches, 375 games have information about last 10 home games and 0 away games (heat map in the notebook). If you wanted to train model for these 375 games you can either\n\ni) exclude those 10 historical home games from your analysis (train model with no information about historical games),\nii) use data only from last 10 historical home games, i.e. train model as if you know nothing about away team.\n\nI could use 100% of my train data to train models there no historical data was used. For matches where information is known about last 1 game, I could use 98.21% of train data and so on. \n\nMy idea was instead of imputing missing values I will create many log. regression models for different amount of available not missing data. Off course, the more information about last n games is known the less matches will be used for training, i.e. try to make predictions with highest amount of available info.\n\nI used cross-validation (CV) to test different features. First, I selected 58.3% of train dataset with no missing data and always did 5-fold CV using different features. I used boxplot (with plotly to interactively see quartile values) and compared different sets of features using log-loss as validation score. In notebook function **select_features ** explains how each feature was generated.\n\nIn my first attempt, I investigated the number of goals and date (number of days since the game we are making predicts). Instead of using the number of goals scored by each team at nth match, I tried using goal difference as a feature.\n\nThe more data was provide for the model (number of historical games) the more accurate my average validation score became. Using goal difference instead of number of goals scored reduced validation score. The best approach was to use number of goals scored by home, away and their opponents with day difference since nth match.\n\nIn my second attempt I used last 1 game of away or home team and then increased number of home or away games, respectively. Again, the more information we have, the lower validation score became.\n\nIf I must choose whatever to train model with 1 away game and 10 home historical games (orange box plots) or 10 away and 10 home games (blue boxes), I will always choose the second. Here is the image ![](https://www.dropbox.com/s/2vb4usk41eof52u/fig3.PNG?raw=1). \n\nThis means that for cases where information is available for n=2 away games and n=6 home games, it might be worth to train model with asymmetrical number of features instead of using last 2 games for home and away sides.\n\nIn my last attempt I investigated remaining features. The best improvement came from using ratings. Other notable improvements were achieved by using is cup and same league features than using just goals and dates.\n\nWhile I didn’t manage to get <1 public LB score, I believe this notebook shows that maybe creating LSTM’s for different number of historical games could produce better results than training on whole dataset with imputed missing values. Of course, it’s much faster to train 100 log. regression models than one LSTM. So, this was just an idea.\n\n[1] https://www.youtube.com/watch?v=pcmUdXIJQ74&list=PLPnZfvKID1Sje5jWxt-4CSZD7bUI4gSPS&index=26\n\n\n", "1774559": "" } }