{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**Exponentially Weighted Ensemble**\n\nThis model is an ensemble consisting of a weighted average of models from public notebooks. As such, it continues the long tradition of ensembling in Kaggle competitions. The technique chosen here is one that optimises a single parameter in order to determine the weights of the component models, and hence may be less vulnerable to overfitting to the public LB data than are regressions with maybe only one fewer parameter than models. Nonetheless, we give more weight to better models, so we may outperform a mean-based or median-based averaging approach. \n\nEach model represented in the ensemble is weighted by a factor that depends in an exponentially decaying manner on its public LB score. Specifically, each model is given an exponential weight according to\n\nexp(b*(S-x))\n\nwhere x is the public LB score of that model. Larger scores are worse, hence the weights get smaller as x increases and get larger as x decreases. Thus, the best models have the largest weights and make the largest contributions to the ensemble.\n\nThe parameter b is the one meaningfully adjustable parameter of the model, the larger b is then the faster the weights decay as the score gets worse. S is a calibration parameter defined such that if S is set to the best single model score then the highest unnormalised weight exp(b*(x-S)) is 1.0, which is convenient, but not essential.\n\nThe sum of the unnormalised weights is called q. Once all weights have been calculated, these weights are normalised by dividing them all by q.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"**Ask a friend**\n\nThis is like asking your friends each to predict the outcome of every game, and then scaling their guesses by how much you reckon each one knows about football. Here, however, we have the public LB scores as a guideline for how knowledgable each available notebook is. We give a higher weight to those we trust more.","metadata":{}},{"cell_type":"markdown","source":"**Pundit or Prophet?**\n\nWe know that this competition requires 'predictions' to be made after the event. Thus, it is inherently vulnerable to both accidental leaks or deliberatev cheats. The hosts have explained why this is at it is, and if this were a Featured Competition with medals and larger prizes, it would be designed differently.\n\nThis notebook has from v16 onwards turned the internet off and does not look up past results from the training data.\n\nIn order to ensure that all predictions in a given row are for the same match, all files are sorted and ordered by 'id'. This does not imply use of information from later dates in the predictions.","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Notebooks included**\n\n[TOP 5: Football prob prediction - LSTM_v02 (GPU)](https://www.kaggle.com/code/seraquevence/top-5-football-prob-prediction-lstm-v02-gpu) by @seraquevence (v38 0.99661)\n\n[TOP 5: Football prob prediction - LSTM_v01](https://www.kaggle.com/code/seraquevence/top-5-football-prob-prediction-lstm-v01) by @seraquevence (v76 0.99945; v104 0.99695)\n\n[LSTM and Feature Engineering-Top8](https://www.kaggle.com/code/ravi07bec/lstm-and-feature-engineering-top8) by @ravi07bec (v10 1.00523)\n\n[Football Match Probability Prediction-LSTM starter](https://www.kaggle.com/code/igorkf/football-match-probability-prediction-lstm-starter) by @igorkf (v6 1.00598)\n\n[Football Results Prediction](https://www.kaggle.com/code/henriqueweber/football-results-prediction) by @henriqueweber (v5 1.01021)\n\n[Football Match Prediction XGBoost Baseline](https://www.kaggle.com/code/yousseftaoudi/football-match-prediction-xgboost-baseline) by @yousseftaoudi (v31 1.01122)\n\n[Football-Match-Prediction](https://www.kaggle.com/code/shaggy11/football-match-prediction) by @shaggy11 (v5 1.01325)\n\n[Football Probability Prediction](https://www.kaggle.com/code/purvansharora/football-probability-prediction) by @purvansharora (v11 1.01340)\n\n[Logistic Regression Submission Example](https://www.kaggle.com/code/octosportio/logistic-regression-submission-example) by @octosportio (v1 1.01760)\n\n[Investigation into rating feature octosport ](https://www.kaggle.com/code/curiosityquotient/investigation-into-rating-feature-octosport) by @curiosityquotient (v5 1.05541; v8 1.05272)\n\n[Constant Probability Baseline ](https://www.kaggle.com/code/jbomitchell/constant-probability-baseline) by @jbomitchell (v3 1.0757)\n\n[xG (expected goals) with simple sklearn models](https://www.kaggle.com/code/uzdavinys/xg-expected-goals-with-simple-sklearn-models) by @uzdavinys (v5 1.02505)\n\n[football-match-probability-prediction Baseline](https://www.kaggle.com/code/hoangnguyen719/football-match-probability-prediction-baseline) by @hoangnguyen719 (v1 1.09861)\n\n[Ensemble: Football prob prediction - LSTM+LGBM_v01](https://www.kaggle.com/code/seraquevence/ensemble-football-prob-prediction-lstm-lgbm-v01) by seraquevence (v15 0.99708)\n\n[notebook931e348ab2](https://www.kaggle.com/code/arjunjanamatti/notebook931e348ab2) by @arjunjanamatti (v4 1.01931)","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv('../input/football-match-probability-prediction/sample_submission.csv')\nsub.sort_values(by=['id'], inplace=True)\nsub0 = pd.read_csv('../input/football-match-probability-prediction/sample_submission.csv')\nsub0.sort_values(by=['id'], inplace=True)\nsub1 = pd.read_csv('../input/football-results-prediction/099945_seraquevence_v76_submission.csv')\nsub1.sort_values(by=['id'], inplace=True)\nsub2 = pd.read_csv('../input/football-results-prediction/100523_ravi07bec_v10_submission.csv')\nsub2.sort_values(by=['id'], inplace=True)\nsub3 = pd.read_csv('../input/football-results-prediction/100598_igorkf_v6_submission.csv')\nsub3.sort_values(by=['id'], inplace=True)\nsub4 = pd.read_csv('../input/football-results-prediction/101021_henriqueweber_v5_submission.csv')\nsub4.sort_values(by=['id'], inplace=True)\nsub5 = pd.read_csv('../input/football-results-prediction/101122_yousseftaoudi_v31_submission.csv')\nsub5.sort_values(by=['id'], inplace=True)\nsub6 = pd.read_csv('../input/football-results-prediction/101325_shaggy11_v6_submission.csv')\nsub6.sort_values(by=['id'], inplace=True)\nsub7 = pd.read_csv('../input/football-results-prediction/101340_purvansharora_v11_submission.csv')\nsub7.sort_values(by=['id'], inplace=True)\nsub8 = pd.read_csv('../input/football-results-prediction/101760_octosportio_v1_submission.csv')\nsub8.sort_values(by=['id'], inplace=True)\nsub9 = pd.read_csv('../input/football-results-prediction/105541_curiosityquotient_v5_submission.csv')\nsub9.sort_values(by=['id'], inplace=True)\nsub10 = pd.read_csv('../input/football-results-prediction/107057_jbomitchell_v3_submission.csv')\nsub10.sort_values(by=['id'], inplace=True)\nsub11 = pd.read_csv('../input/football-results-prediction/099695_seraquevence_v104_submission.csv')\nsub11.sort_values(by=['id'], inplace=True)\nsub12 = pd.read_csv('../input/football-results-prediction/099661_seraquevence_v38_submission.csv')\nsub12.sort_values(by=['id'], inplace=True)\nsub13 = pd.read_csv('../input/football-results-prediction/102505_uzdavinys_v5_submission.csv')\nsub13.sort_values(by=['id'], inplace=True)\nsub14 = pd.read_csv('../input/football-results-prediction/109681_hoangnguyen719_v1_submission.csv')\nsub14.sort_values(by=['id'], inplace=True)\nsub15 = pd.read_csv('../input/football-results-prediction/105272_curiosityquotient_v8_submission.csv')\nsub15.sort_values(by=['id'], inplace=True)\nsub16 = pd.read_csv('../input/football-results-prediction/101931_arjunjanamatti_v4_submission.csv')\nsub16.sort_values(by=['id'], inplace=True)\nsub17 = pd.read_csv('../input/football-results-prediction/099708_seraquevence_v15_submission.csv')\nsub17.sort_values(by=['id'], inplace=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Setting the parameters**\n\nWe can get an approximate feel for a suitable value of b by considering the difference in LB score for which we would want a second model to have a factor of e (~2.72) smaller contribution to the ensemble than the best available model. If we want the contribution to drop of by a factor of e for each 0.01 deterioration in score, then we would set b to 100.0. In practice, there is some trial and error involved in keeping b close to optimal as the ensemble progresses throughout the competition. \n\nS is set to the score of the best component model; this makes the maximum weight 1.0. Forgetting to reset S when a new best score is available will not have a significant impact.\n\nq is the sum of the weights. We increment it as we add the contribution of each new model.","metadata":{}},{"cell_type":"code","source":"b = 490.0\nS = 0.99661\nq = 0.0","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Iterating over component models to set 'home' probability**\n\nWe now iterate over each of the component models' 'home' probabilities, multiplying each one by the corresponding exponential weighting factor, and keeping track of the running sum and running sum of weights.\n\nAt the end of this process, we divide the final sum of weighted 'home' probabilities by the sum of weights to obtain a suitably scaled ensemble predicted 'home' probability.","metadata":{}},{"cell_type":"markdown","source":"**Non-zero sum game**\n\nBecause 'home' probabilities are on average higher than away probabilites and more home than away fans attend a typical match, on average more fans see their team win than see their team lose. ","metadata":{}},{"cell_type":"code","source":"sub['home'] = sub1['home']*np.exp(b*(S-0.99945))\nq = q + np.exp(b*(S-0.99945))\nsub['home'] = sub['home'] + sub2['home']*np.exp(b*(S-1.00523))\nq = q + np.exp(b*(S-1.00523))\nsub['home'] = sub['home'] + sub3['home']*np.exp(b*(S-1.00598))\nq = q + np.exp(b*(S-1.00598))\nsub['home'] = sub['home'] + sub4['home']*np.exp(b*(S-1.01021))\nq = q + np.exp(b*(S-1.01021))\nsub['home'] = sub['home'] + sub5['home']*np.exp(b*(S-1.01122))\nq = q + np.exp(b*(S-1.01122))\nsub['home'] = sub['home'] + sub6['home']*np.exp(b*(S-1.01325))\nq = q + np.exp(b*(S-1.01325))\nsub['home'] = sub['home'] + sub7['home']*np.exp(b*(S-1.01340))\nq = q + np.exp(b*(S-1.01340))\nsub['home'] = sub['home'] + sub8['home']*np.exp(b*(S-1.01760))\nq = q + np.exp(b*(S-1.01760))\nsub['home'] = sub['home'] + sub9['home']*np.exp(b*(S-1.05541))\nq = q + np.exp(b*(S-1.05541))\nsub['home'] = sub['home'] + sub10['home']*np.exp(b*(S-1.07057))\nq = q + np.exp(b*(S-1.07057))\nsub['home'] = sub['home'] + sub11['home']*np.exp(b*(S-0.99695))\nq = q + np.exp(b*(S-0.99695))\nsub['home'] = sub['home'] + sub12['home']*np.exp(b*(S-0.99661))\nq = q + np.exp(b*(S-0.99661))\nsub['home'] = sub['home'] + sub13['home']*np.exp(b*(S-1.02505))\nq = q + np.exp(b*(S-1.02505))\nsub['home'] = sub['home'] + sub14['home']*np.exp(b*(S-1.09681))\nq = q + np.exp(b*(S-1.09681))\nsub['home'] = sub['home'] + sub15['home']*np.exp(b*(S-1.05272))\nq = q + np.exp(b*(S-1.05272))\nsub['home'] = sub['home'] + sub16['home']*np.exp(b*(S-1.01931))\nq = q + np.exp(b*(S-1.01931))\nsub['home'] = sub['home'] + sub17['home']*np.exp(b*(S-0.99708))\nq = q + np.exp(b*(S-0.99708))\nsub['home'] = sub['home']/q\nprint(q)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Because we are going to reuse the same variable q when we iterate over the away probabilities, we must remember to reset q to 0.0.","metadata":{}},{"cell_type":"code","source":"q = 0.0","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Iterating over component models to set 'away' probability**\n\nWe now iterate over each of the component models' 'away' probabilities, multiplying each one by the corresponding exponential weighting factor, and keeping track of the running sum and running sum of weights.\n\nAt the end of this process, we divide the final sum of weighted 'away' probabilities by the sum of weights to obtain a suitably scaled ensemble predicted 'away' probability.","metadata":{}},{"cell_type":"code","source":"sub['away'] = sub1['away']*np.exp(b*(S-0.99945))\nq = q + np.exp(b*(S-0.99945))\nsub['away'] = sub['away'] + sub2['away']*np.exp(b*(S-1.00523))\nq = q + np.exp(b*(S-1.00523))\nsub['away'] = sub['away'] + sub3['away']*np.exp(b*(S-1.00598))\nq = q + np.exp(b*(S-1.00598))\nsub['away'] = sub['away'] + sub4['away']*np.exp(b*(S-1.01021))\nq = q + np.exp(b*(S-1.01021))\nsub['away'] = sub['away'] + sub5['away']*np.exp(b*(S-1.01122))\nq = q + np.exp(b*(S-1.01122))\nsub['away'] = sub['away'] + sub6['away']*np.exp(b*(S-1.01325))\nq = q + np.exp(b*(S-1.01325))\nsub['away'] = sub['away'] + sub7['away']*np.exp(b*(S-1.01340))\nq = q + np.exp(b*(S-1.01340))\nsub['away'] = sub['away'] + sub8['away']*np.exp(b*(S-1.01760))\nq = q + np.exp(b*(S-1.01760))\nsub['away'] = sub['away'] + sub9['away']*np.exp(b*(S-1.05541))\nq = q + np.exp(b*(S-1.05541))\nsub['away'] = sub['away'] + sub10['away']*np.exp(b*(S-1.07057))\nq = q + np.exp(b*(S-1.07057))\nsub['away'] = sub['away'] + sub11['away']*np.exp(b*(S-0.99695))\nq = q + np.exp(b*(S-0.99695))\nsub['away'] = sub['away'] + sub12['away']*np.exp(b*(S-0.99661))\nq = q + np.exp(b*(S-0.99661))\nsub['away'] = sub['away'] + sub13['away']*np.exp(b*(S-1.02505))\nq = q + np.exp(b*(S-1.02505))\nsub['away'] = sub['away'] + sub14['away']*np.exp(b*(S-1.09681))\nq = q + np.exp(b*(S-1.09681))\nsub['away'] = sub['away'] + sub15['away']*np.exp(b*(S-1.05272))\nq = q + np.exp(b*(S-1.05272))\nsub['away'] = sub['away'] + sub16['away']*np.exp(b*(S-1.01931))\nq = q + np.exp(b*(S-1.01931))\nsub['away'] = sub['away'] + sub17['away']*np.exp(b*(S-0.99708))\nq = q + np.exp(b*(S-0.99708))\nsub['away'] = sub['away']/q\nprint(q)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For tidiness, we will reset q to 0.0 again. Perhaps, we may wish to rewrite our code in a later version to calculate the 'draw' probability iteratively, and it would be easy to forget to reset q.","metadata":{}},{"cell_type":"code","source":"q = 0.0","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The 'draw' probability**\n\nThe 'draw' probability is often the smallest of the three, and simple three-class classification models may therefore predict rather few draws. This isn't particularly significant here, as all our predictions are probabilistic.\n\nPredicting draws was historically important in the UK because of the football pools, where life-changing amounts of money like £100,000 (in the 1970s, that was a fortune) could be won by picking eight score draws on the 'Treble Chance'. The football results would conclude with something like: \"There were 31 home wins, 15 away wins and 9 draws, of which 8 were score draws. Telegram claims are required for 23 or 24 points and the dividend forecast is a possible jackpot.\"\n\nAlthough we could compute the 'draw' probabilities iteratively as with 'home' and 'away', here we simply use the 'probabilities must sum to one' principle to infer them.","metadata":{}},{"cell_type":"code","source":"sub['draw'] = 1.0 - sub['home'] - sub['away']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Empirical Shift**\n\nHere we test the effect of adding an empirical uniform shift to the predicted probabilities. This is designed to cover the possibility that there is a systematic non-optimality in the balance between the 'home', 'away', and 'draw' probabilities predicted by the models. In version 1 of this notebook, we found that transferring 0.001 probability from 'home' to 'draw' slightly improved the score on the public LB; version 9 showed an improvement when 0.001 was shifted from 'home' to 'away'. The downside is that there is a risk of overfitting.","metadata":{}},{"cell_type":"code","source":"sub['home'] = sub['home'] - 0.001\nsub['draw'] = sub['draw'] + 0.001\n\nsub['home'] = sub['home'] - 0.0015\nsub['away'] = sub['away'] + 0.0015","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"** Clipping **\n\nFor the predictions, I make a couple of changes to the code.\n\nFirstly, I clip the base home, away and draw probabilities to lie within the ranges given by:\n\nsub['home'] = np.clip(sub['home'], 0.1, 0.95)\n\nsub['away'] = np.clip(sub['away'], 0.05, 0.9)\n\nsub['draw'] = np.clip(sub['draw'], 0.125, 0.325)\n\nThese values are based on the plots by @curiosityquotient and the tendency of log loss to punish predictions that are both extreme and wrong.\n\nI also normalise the predictions such that the three final probabilities sum to one:\n\nsub0['total'] = sub['home'] + sub['away'] + sub['draw']\n\nsub['home'] = sub['home']/sub0['total']\n\nsub['away'] = sub['away']/sub0['total']\n\nsub['draw'] = sub['draw']/sub0['total']","metadata":{}},{"cell_type":"code","source":"sub['home'] = np.clip(sub['home'], 0.1, 0.95)\n\nsub['away'] = np.clip(sub['away'], 0.05, 0.9)\n\nsub['draw'] = np.clip(sub['draw'], 0.125, 0.325)\n\nsub0['total'] = sub['home'] + sub['away'] + sub['draw']\n\nsub['home'] = sub['home']/sub0['total']\n\nsub['away'] = sub['away']/sub0['total']\n\nsub['draw'] = sub['draw']/sub0['total']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Submission**\n\nWe write the predicted probabilities to a .csv file and print a few of them. It is important to check that these look sensible, are contained within the interval 0 -> 1, and sum to 1.0 for each match.","metadata":{}},{"cell_type":"code","source":"sub.to_csv('submission.csv', index=False)\nsub.head(10)","metadata":{},"execution_count":null,"outputs":[]}]}