{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**Exponentially Weighted Ensemble**\n\nThis model is an ensemble consisting of a weighted average of models from public notebooks. As such, it continues the long tradition of ensembling in Kaggle competitions. The technique chosen here is one that optimises a single parameter in order to determine the weights of the component models, and hence may be less vulnerable to overfitting to the public LB data than are regressions with maybe only one fewer parameter than models. Nonetheless, we give more weight to better models, so we may outperform a mean-based or median-based averaging approach. \n\nEach model represented in the ensemble is weighted by a factor that depends in an exponentially decaying manner on its public LB score. Specifically, each model is given an exponential weight according to\n\nexp(b*(S-x))\n\nwhere x is the public LB score of that model. Larger scores are worse, hence the weights get smaller as x increases and get larger as x decreases. Thus, the best models have the largest weights and make the largest contributions to the ensemble.\n\nThe parameter b is the one meaningfully adjustable parameter of the model, the larger b is then the faster the weights decay as the score gets worse. S is a calibration parameter defined such that if S is set to the best single model score then the highest unnormalised weight exp(b*(x-S)) is 1.0, which is convenient, but not essential.\n\nThe sum of the unnormalised weights is called q. Once all weights have been calculated, these weights are normalised by dividing them all by q.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"**Analogy to Boltzmann weighting in statistical thermodynamics**\n\nThe models in this ensemble can be thought of as corresponding to quantum states in statistical thermodynamics. Evidently, b is analogous to 1/kT in statistical thermodynamics and behaves like a reciprocal temperature. In thermodynamics textbooks this parameter is often called beta.\n\nThe LB score x is analogous to energy. Since higher LB scores are bad in this competition the weights, which are effectively Boltzmann factors, decrease with increasing x. The opposite sign convention would have been used in a competition where higher LB scores are good. S is a calibration parameter defined such that if S is set to the best single model score then the highest weight is 1.0, which is convenient, but not essential.\n\nThe sum of the weights q is analogous to the partition function in statistical thermodynamics, and measures an effective number of contributing models. This is analogous to how the partition function, if calibrated such that the highest weight is 1.0, measures the number of thermally accessible quantum states.","metadata":{}},{"cell_type":"markdown","source":"**Ask a friend**\n\nThis is like asking your friends each to predict the outcome of every game, and then scaling their guesses by how much you reckon each one knows about football. Here, however, we have the public LB scores as a guideline for how knowledgable each available notebook is. We give a higher weight to those we trust more.","metadata":{}},{"cell_type":"markdown","source":"**Pundit or Prophet?**\n\nWe know that this competition requires 'predictions' to be made after the event. Thus, it is inherently vulnerable to both accidental leaks and deliberate cheats. The hosts have explained why this is at it is, and if this were a Featured Competition with medals and larger prizes, it would be designed differently.\n\nThis notebook has from v16 onwards turned the internet off and does not look up past results from the training data.\n\nIn order to ensure that all predictions in a given row are for the same match, all files are sorted and ordered by 'id'. This does not imply use of information from later dates in the predictions, which we are careful to avoid. Any public notebook suspected of using future information is excluded from this ensemble, so none of the 'Data Leakage' notebooks are utilised.","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Notebooks included**\n\n[TOP 5: Football prob prediction - LSTM_v02 (GPU)](https://www.kaggle.com/code/seraquevence/top-5-football-prob-prediction-lstm-v02-gpu) by @seraquevence (v38 0.99661)\n\n[TOP 5: Football prob prediction - LSTM_v01](https://www.kaggle.com/code/seraquevence/top-5-football-prob-prediction-lstm-v01) by @seraquevence (v76 0.99945; v104 0.99695)\n\n[LSTM and Feature Engineering-Top8](https://www.kaggle.com/code/ravi07bec/lstm-and-feature-engineering-top8) by @ravi07bec (v10 1.00523)\n\n[Football Match Probability Prediction-LSTM starter](https://www.kaggle.com/code/igorkf/football-match-probability-prediction-lstm-starter) by @igorkf (v6 1.00598)\n\n[Football Results Prediction](https://www.kaggle.com/code/henriqueweber/football-results-prediction) by @henriqueweber (v5 1.01021)\n\n[Football Match Prediction XGBoost Baseline](https://www.kaggle.com/code/yousseftaoudi/football-match-prediction-xgboost-baseline) by @yousseftaoudi (v31 1.01122)\n\n[Football-Match-Prediction](https://www.kaggle.com/code/shaggy11/football-match-prediction) by @shaggy11 (v5 1.01325)\n\n[Football Probability Prediction](https://www.kaggle.com/code/purvansharora/football-probability-prediction) by @purvansharora (v11 1.01340)\n\n[Logistic Regression Submission Example](https://www.kaggle.com/code/octosportio/logistic-regression-submission-example) by @octosportio (v1 1.01760)\n\n[Investigation into rating feature octosport ](https://www.kaggle.com/code/curiosityquotient/investigation-into-rating-feature-octosport) by @curiosityquotient (v5 1.05541; v8 1.05272)\n\n[Rating model after @curiosityquotient & @szeyeung](https://www.kaggle.com/code/jbomitchell/rating-model-after-curiosityquotient-szeyeung) by @jbomitchell (v4 1.04849)\n\n[Constant Probability Baseline ](https://www.kaggle.com/code/jbomitchell/constant-probability-baseline) by @jbomitchell (v3 1.0757)\n\n[xG (expected goals) with simple sklearn models](https://www.kaggle.com/code/uzdavinys/xg-expected-goals-with-simple-sklearn-models) by @uzdavinys (v5 1.02505)\n\n[football-match-probability-prediction Baseline](https://www.kaggle.com/code/hoangnguyen719/football-match-probability-prediction-baseline) by @hoangnguyen719 (v1 1.09861)\n\n[Ensemble: Football prob prediction - LSTM+LGBM_v01](https://www.kaggle.com/code/seraquevence/ensemble-football-prob-prediction-lstm-lgbm-v01) by seraquevence (v15 0.99708)\n\n[notebook931e348ab2](https://www.kaggle.com/code/arjunjanamatti/notebook931e348ab2) by @arjunjanamatti (v4 1.01931)\n\n[Football Match: Simple implementation LightGBM](https://www.kaggle.com/code/aboriginal3153/football-match-simple-implementation-lightgbm) by @aboriginal3153 (v4 1.00729)\n\n[TensorFlow Transformer: Football prob pred_v01](https://www.kaggle.com/code/seraquevence/tensorflow-transformer-football-prob-pred-v01) by @seraquevence (v56 1.00023)\n\n[TOP X: Football prob prediction - LGBM_v01](https://www.kaggle.com/code/seraquevence/top-x-football-prob-prediction-lgbm-v01) by @seraquevence (v50 1.00303; v97 1.00199)\n\n[Simple LightGBM with Aggregated Features](https://www.kaggle.com/code/tarique7/simple-lightgbm-with-aggregated-features) by @tarique7 (v1 1.00340)\n\n[Football Match Probability Prediction](https://www.kaggle.com/code/shariful07/football-match-probability-prediction) by @shariful07 (v4 1.01126)\n\n[Log. regression with no imputed NaN values](https://www.kaggle.com/code/uzdavinys/log-regression-with-no-imputed-nan-values) by @uzdavinys (v10 1.01059)\n\n[notebooke77cdbd24b](https://www.kaggle.com/code/arjunjanamatti/notebooke77cdbd24b) by @arjunjanamatti (v15 1.01282)","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv('../input/football-match-probability-prediction/sample_submission.csv')\nsub.sort_values(by=['id'], inplace=True)\nsub1 = pd.read_csv('../input/football-results-prediction/099945_seraquevence_v76_submission.csv')\nsub1.sort_values(by=['id'], inplace=True)\nsub2 = pd.read_csv('../input/football-results-prediction/100523_ravi07bec_v10_submission.csv')\nsub2.sort_values(by=['id'], inplace=True)\nsub3 = pd.read_csv('../input/football-results-prediction/100598_igorkf_v6_submission.csv')\nsub3.sort_values(by=['id'], inplace=True)\nsub4 = pd.read_csv('../input/football-results-prediction/101021_henriqueweber_v5_submission.csv')\nsub4.sort_values(by=['id'], inplace=True)\nsub5 = pd.read_csv('../input/football-results-prediction/101122_yousseftaoudi_v31_submission.csv')\nsub5.sort_values(by=['id'], inplace=True)\nsub6 = pd.read_csv('../input/football-results-prediction/101325_shaggy11_v6_submission.csv')\nsub6.sort_values(by=['id'], inplace=True)\nsub7 = pd.read_csv('../input/football-results-prediction/101340_purvansharora_v11_submission.csv')\nsub7.sort_values(by=['id'], inplace=True)\nsub8 = pd.read_csv('../input/football-results-prediction/101760_octosportio_v1_submission.csv')\nsub8.sort_values(by=['id'], inplace=True)\nsub9 = pd.read_csv('../input/football-results-prediction/105541_curiosityquotient_v5_submission.csv')\nsub9.sort_values(by=['id'], inplace=True)\nsub10 = pd.read_csv('../input/football-results-prediction/107057_jbomitchell_v3_submission.csv')\nsub10.sort_values(by=['id'], inplace=True)\nsub11 = pd.read_csv('../input/football-results-prediction/099695_seraquevence_v104_submission.csv')\nsub11.sort_values(by=['id'], inplace=True)\nsub12 = pd.read_csv('../input/football-results-prediction/099661_seraquevence_v38_submission.csv')\nsub12.sort_values(by=['id'], inplace=True)\nsub13 = pd.read_csv('../input/football-results-prediction/102505_uzdavinys_v5_submission.csv')\nsub13.sort_values(by=['id'], inplace=True)\nsub14 = pd.read_csv('../input/football-results-prediction/109681_hoangnguyen719_v1_submission.csv')\nsub14.sort_values(by=['id'], inplace=True)\nsub15 = pd.read_csv('../input/football-results-prediction/105272_curiosityquotient_v8_submission.csv')\nsub15.sort_values(by=['id'], inplace=True)\nsub16 = pd.read_csv('../input/football-results-prediction/101931_arjunjanamatti_v4_submission.csv')\nsub16.sort_values(by=['id'], inplace=True)\nsub17 = pd.read_csv('../input/football-results-prediction/099708_seraquevence_v15_submission.csv')\nsub17.sort_values(by=['id'], inplace=True)\nsub18 = pd.read_csv('../input/football-results-prediction/100729_aboriginal3153_v4_submission.csv')\nsub18.sort_values(by=['id'], inplace=True)\nsub19 = pd.read_csv('../input/football-results-prediction/100023_seraquevence_v56_submission.csv')\nsub19.sort_values(by=['id'], inplace=True)\nsub20 = pd.read_csv('../input/football-results-prediction/100303_seraquevence_v50_submission.csv')\nsub20.sort_values(by=['id'], inplace=True)\nsub21 = pd.read_csv('../input/football-results-prediction/104849_jbomitchell_v4_submission.csv')\nsub21.sort_values(by=['id'], inplace=True)\nsub22 = pd.read_csv('../input/football-results-prediction/100340_tarique7_v1_submission.csv')\nsub22.sort_values(by=['id'], inplace=True)\nsub23 = pd.read_csv('../input/football-results-prediction/101126_shariful07_v4_submission.csv')\nsub23.sort_values(by=['id'], inplace=True)\nsub24 = pd.read_csv('../input/football-results-prediction/101059_uzdavinys_v10_submission.csv')\nsub24.sort_values(by=['id'], inplace=True)\nsub25 = pd.read_csv('../input/football-results-prediction/100199_seraquevence_v97_submission.csv')\nsub25.sort_values(by=['id'], inplace=True)\nsub26 = pd.read_csv('../input/football-results-prediction/101282_arjunjanamatti_v15_submission.csv')\nsub26.sort_values(by=['id'], inplace=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Setting the parameters**\n\nWe can get an approximate feel for a suitable value of b by considering the difference in LB score for which we would want a second model to have a factor of e (~2.72) smaller contribution to the ensemble than the best available model. If we want the contribution to drop of by a factor of e for each 0.01 deterioration in score, then we would set b to 100.0. In practice, there is some trial and error involved in keeping b close to optimal as the ensemble progresses throughout the competition. \n\nS is set to the score of the best component model; this makes the maximum weight 1.0. Forgetting to reset S when a new best score is available amongst the input models will not have a significant impact.\n\nq is the sum of the weights. We increment it as we add the contribution of each new model.","metadata":{}},{"cell_type":"code","source":"b = 611.0\nS = 0.99661\nq = 0.0","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Iterating over component models to set 'home' probability**\n\nWe now iterate over each of the component models' 'home' probabilities, multiplying each one by the corresponding exponential weighting factor, and keeping track of the running sum and running sum of weights.\n\nAt the end of this process, we divide the final sum of weighted 'home' probabilities by the sum of weights to obtain a suitably scaled ensemble predicted 'home' probability.","metadata":{}},{"cell_type":"markdown","source":"**Non-zero sum game**\n\nBecause 'home' probabilities are on average higher than away probabilites and more home than away fans attend a typical match, on average more fans see their team win than see their team lose. ","metadata":{}},{"cell_type":"code","source":"sub['home'] = sub1['home']*np.exp(b*(S-0.99945))\nq = q + np.exp(b*(S-0.99945))\nsub['home'] = sub['home'] + sub2['home']*np.exp(b*(S-1.00523))\nq = q + np.exp(b*(S-1.00523))\nsub['home'] = sub['home'] + sub3['home']*np.exp(b*(S-1.00598))\nq = q + np.exp(b*(S-1.00598))\nsub['home'] = sub['home'] + sub4['home']*np.exp(b*(S-1.01021))\nq = q + np.exp(b*(S-1.01021))\nsub['home'] = sub['home'] + sub5['home']*np.exp(b*(S-1.01122))\nq = q + np.exp(b*(S-1.01122))\nsub['home'] = sub['home'] + sub6['home']*np.exp(b*(S-1.01325))\nq = q + np.exp(b*(S-1.01325))\nsub['home'] = sub['home'] + sub7['home']*np.exp(b*(S-1.01340))\nq = q + np.exp(b*(S-1.01340))\nsub['home'] = sub['home'] + sub8['home']*np.exp(b*(S-1.01760))\nq = q + np.exp(b*(S-1.01760))\nsub['home'] = sub['home'] + sub9['home']*np.exp(b*(S-1.05541))\nq = q + np.exp(b*(S-1.05541))\nsub['home'] = sub['home'] + sub10['home']*np.exp(b*(S-1.07057))\nq = q + np.exp(b*(S-1.07057))\nsub['home'] = sub['home'] + sub11['home']*np.exp(b*(S-0.99695))\nq = q + np.exp(b*(S-0.99695))\nsub['home'] = sub['home'] + sub12['home']*np.exp(b*(S-0.99661))\nq = q + np.exp(b*(S-0.99661))\nsub['home'] = sub['home'] + sub13['home']*np.exp(b*(S-1.02505))\nq = q + np.exp(b*(S-1.02505))\nsub['home'] = sub['home'] + sub14['home']*np.exp(b*(S-1.09681))\nq = q + np.exp(b*(S-1.09681))\nsub['home'] = sub['home'] + sub15['home']*np.exp(b*(S-1.05272))\nq = q + np.exp(b*(S-1.05272))\nsub['home'] = sub['home'] + sub16['home']*np.exp(b*(S-1.01931))\nq = q + np.exp(b*(S-1.01931))\nsub['home'] = sub['home'] + sub17['home']*np.exp(b*(S-0.99708))\nq = q + np.exp(b*(S-0.99708))\nsub['home'] = sub['home'] + sub18['home']*np.exp(b*(S-1.00729))\nq = q + np.exp(b*(S-1.00729))\nsub['home'] = sub['home'] + sub19['home']*np.exp(b*(S-1.00023))\nq = q + np.exp(b*(S-1.00023))\nsub['home'] = sub['home'] + sub20['home']*np.exp(b*(S-1.00303))\nq = q + np.exp(b*(S-1.00303))\nsub['home'] = sub['home'] + sub21['home']*np.exp(b*(S-1.04849))\nq = q + np.exp(b*(S-1.04849))\nsub['home'] = sub['home'] + sub22['home']*np.exp(b*(S-1.00340))\nq = q + np.exp(b*(S-1.00340))\nsub['home'] = sub['home'] + sub23['home']*np.exp(b*(S-1.01126))\nq = q + np.exp(b*(S-1.01126))\nsub['home'] = sub['home'] + sub24['home']*np.exp(b*(S-1.01059))\nq = q + np.exp(b*(S-1.01059))\nsub['home'] = sub['home'] + sub25['home']*np.exp(b*(S-1.00199))\nq = q + np.exp(b*(S-1.00199))\nsub['home'] = sub['home'] + sub26['home']*np.exp(b*(S-1.01282))\nq = q + np.exp(b*(S-1.01282))\nsub['home'] = sub['home']/q\nprint(q)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Resetting q to zero**\n\nBecause we are going to reuse the same variable q when we iterate over the away probabilities, we must remember to reset q to 0.0.","metadata":{}},{"cell_type":"code","source":"q = 0.0","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Iterating over component models to set 'away' probability**\n\nWe now iterate over each of the component models' 'away' probabilities, multiplying each one by the corresponding exponential weighting factor, and keeping track of the running sum and running sum of weights.\n\nAt the end of this process, we divide the final sum of weighted 'away' probabilities by the sum of weights to obtain a suitably scaled ensemble predicted 'away' probability.","metadata":{}},{"cell_type":"code","source":"sub['away'] = sub1['away']*np.exp(b*(S-0.99945))\nq = q + np.exp(b*(S-0.99945))\nsub['away'] = sub['away'] + sub2['away']*np.exp(b*(S-1.00523))\nq = q + np.exp(b*(S-1.00523))\nsub['away'] = sub['away'] + sub3['away']*np.exp(b*(S-1.00598))\nq = q + np.exp(b*(S-1.00598))\nsub['away'] = sub['away'] + sub4['away']*np.exp(b*(S-1.01021))\nq = q + np.exp(b*(S-1.01021))\nsub['away'] = sub['away'] + sub5['away']*np.exp(b*(S-1.01122))\nq = q + np.exp(b*(S-1.01122))\nsub['away'] = sub['away'] + sub6['away']*np.exp(b*(S-1.01325))\nq = q + np.exp(b*(S-1.01325))\nsub['away'] = sub['away'] + sub7['away']*np.exp(b*(S-1.01340))\nq = q + np.exp(b*(S-1.01340))\nsub['away'] = sub['away'] + sub8['away']*np.exp(b*(S-1.01760))\nq = q + np.exp(b*(S-1.01760))\nsub['away'] = sub['away'] + sub9['away']*np.exp(b*(S-1.05541))\nq = q + np.exp(b*(S-1.05541))\nsub['away'] = sub['away'] + sub10['away']*np.exp(b*(S-1.07057))\nq = q + np.exp(b*(S-1.07057))\nsub['away'] = sub['away'] + sub11['away']*np.exp(b*(S-0.99695))\nq = q + np.exp(b*(S-0.99695))\nsub['away'] = sub['away'] + sub12['away']*np.exp(b*(S-0.99661))\nq = q + np.exp(b*(S-0.99661))\nsub['away'] = sub['away'] + sub13['away']*np.exp(b*(S-1.02505))\nq = q + np.exp(b*(S-1.02505))\nsub['away'] = sub['away'] + sub14['away']*np.exp(b*(S-1.09681))\nq = q + np.exp(b*(S-1.09681))\nsub['away'] = sub['away'] + sub15['away']*np.exp(b*(S-1.05272))\nq = q + np.exp(b*(S-1.05272))\nsub['away'] = sub['away'] + sub16['away']*np.exp(b*(S-1.01931))\nq = q + np.exp(b*(S-1.01931))\nsub['away'] = sub['away'] + sub17['away']*np.exp(b*(S-0.99708))\nq = q + np.exp(b*(S-0.99708))\nsub['away'] = sub['away'] + sub18['away']*np.exp(b*(S-1.00729))\nq = q + np.exp(b*(S-1.00729))\nsub['away'] = sub['away'] + sub19['away']*np.exp(b*(S-1.00023))\nq = q + np.exp(b*(S-1.00023))\nsub['away'] = sub['away'] + sub20['away']*np.exp(b*(S-1.00303))\nq = q + np.exp(b*(S-1.00303))\nsub['away'] = sub['away'] + sub21['away']*np.exp(b*(S-1.04849))\nq = q + np.exp(b*(S-1.04849))\nsub['away'] = sub['away'] + sub22['away']*np.exp(b*(S-1.00340))\nq = q + np.exp(b*(S-1.00340))\nsub['away'] = sub['away'] + sub23['away']*np.exp(b*(S-1.01126))\nq = q + np.exp(b*(S-1.01126))\nsub['away'] = sub['away'] + sub24['away']*np.exp(b*(S-1.01059))\nq = q + np.exp(b*(S-1.01059))\nsub['away'] = sub['away'] + sub25['away']*np.exp(b*(S-1.00199))\nq = q + np.exp(b*(S-1.00199))\nsub['away'] = sub['away'] + sub26['away']*np.exp(b*(S-1.01282))\nq = q + np.exp(b*(S-1.01282))\nsub['away'] = sub['away']/q\nprint(q)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Resetting q to zero again**\n\nFor tidiness, we will reset q to 0.0 again. Perhaps, we may wish to rewrite our code in a later version to calculate the 'draw' probability iteratively, and it would be easy to forget to reset q.","metadata":{}},{"cell_type":"code","source":"q = 0.0","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**The 'draw' probability**\n\nThe 'draw' probability is often the smallest of the three, and simple three-class classification models may therefore predict rather few draws. This isn't particularly significant here, as all our predictions are probabilistic.\n\nPredicting draws was historically important in the UK because of the football pools, where life-changing amounts of money like £100,000 (in the 1970s, that was a fortune) could be won by picking eight score draws on the 'Treble Chance'. The football results would conclude with something like: \"There were 31 home wins, 15 away wins and 9 draws, of which 8 were score draws. Telegram claims are required for 23 or 24 points and the dividend forecast is a possible jackpot.\"\n\nAlthough we could compute the 'draw' probabilities iteratively as with 'home' and 'away', here we simply use the 'probabilities must sum to one' principle to infer them.","metadata":{}},{"cell_type":"code","source":"sub['draw'] = 1.0 - sub['home'] - sub['away']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Submission**\n\nWe write the predicted probabilities to a .csv file and print a few of them. It is important to check that these look sensible, are contained within the interval 0 -> 1, and sum to 1.0 for each match.","metadata":{}},{"cell_type":"code","source":"sub.to_csv('submission.csv', index=False)\nsub.head(10)","metadata":{},"execution_count":null,"outputs":[]}]}