{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Introduction**\n\nThis essay is about the unreasonable effectiveness of ensemble learning in Kaggle Competitions. \n\nEnsemble learning refers to machine learning methods where multiple models are combined to make more accurate predictions. \n\nUnreasonable effectiveness refers to the mysteries behind competitors substantially improving their rankings (on Kaggle leaderboards) by combining several lower-ranking models.\n\nWhile the essay title assumes the premise that ensemble learning is unreasonably effective, this is a position that is argued for and evidenced in the main body. It then goes further to reflect on whether the current popularity of ensemble learning is good for the Kaggle or machine learning community.","metadata":{}},{"cell_type":"markdown","source":"# **1. The effectiveness of ensemble learning**\n\n[Ensemble models are considered to have advantages over a single model in these dimensions](https://www.kaggle.com/discussions/general/263786):\n- performance-wise: an ensemble reduces the variance component of the prediction error by adding bias;\n- robustness-wise: an ensemble reduces reliance on any single model's predictions, making it better equipped to handle noisy data (e.g. outliers). \n\nIt is applicable across a wide range of data science and machine learning problems: clustering, regression, classification etc.\n\nIn this part, we first survey some of the ensemble learning techniques presently used, and review the progress made in the past two years (i.e. 2021-2023). Then, we explore the prevalence of ensemble learning in Kaggle Competitions. Finally, its effectiveness is highlighted, and is argued to be \"unreasonable\" at times. ","metadata":{}},{"cell_type":"markdown","source":"## **1.1 What are the current ensemble learning techniques?**\n\nTypically, models of different natures are chosen to be part of the ensemble as they can supposedly cover for each other's shortcomings. Common models chosen are: \n- Gradient Boosted Decision Trees (GBDT): the consensus number one for structured data;\n- other tree-based models: e.g. Extremely Randomized Trees (ERT), Random Forest, Regularized Greedy Forest (RGF);\n- neural networks;\n- linear regression models;\n- clustering algorithms: e.g. k-means;\n- Field-aware Factorization Machines (FFM).\n\nNote that decision tree-based models can themselves be considered ensemble learning since they involve building many different predictive trees and aggregating their outputs in some way, but the ensemble learning discussed in this essay refers to ensembling *methods* rather than ensembling-based *algorithms*.\n\nModel outputs can be aggregated using the following techniques:\n1. *Blending: averaging, weighted averaging and rank averaging*<br>\n    The simplest way to combine different models is by averaging their outputs. This is sometimes known as \"blending\".<br>\n    \n    The weights given to each model can be assigned either explicitly (weighted averaging), or implicitly by a [ranking script (rank averaging)](https://www.kaggle.com/code/fizzbuzz/how-to-perform-rank-averaging/script). The latter technique involves [ranking models by performance, and giving the more accurate models greater weight](https://towardsdatascience.com/ensemble-averaging-improve-machine-learning-performance-by-voting-246106c753ee).<br>\n    \n    In Kaggle Competitions, whether and how to blend model outputs are often determined by observing the public leaderboard (LB) scores. This is known as [\"leaderboard probing\"](https://www.kaggle.com/competitions/tabular-playground-series-nov-2022/discussion/363968), and is to be used with caution since it can result in overfitting on the public test dataset.<br>\n    \n    Optimization modules such as **scipy.optimize**, **optuna** or **hyperopt** can be used to find the optimal blend by taking into account cross-validation metrics such as Area Under the ROC Curve (AUC).\n\n2. *Voting*<br>\n    For classification tasks, an alternative approach is having the models \"vote\" on the outcome. A simple method is \"majority voting\", where the class that is predicted by the majority of models is chosen as the final prediction.\n\n3. *The classical trio: bagging, boosting and stacking*<br>\n    Bagging is also known as \"bootstrap aggregating\". It involves training multiple models on different subsets of the training data, and then averaging their predictions.\n\n    Boosting involves sequentially training models, with each new model focusing on the errors made by its predecessors. An implementation of boosting is AdaBoost: short for \"adaptive boosting\", it [adjusts model weights based on the performance of past models](https://www.kaggle.com/discussions/general/413803). Boosting techniques can be used together with Random Forests to further improve their efficacy.\n\n    Stacking involves feeding the predictions of various models as inputs to a higher-level model. The latter is sometimes known as a \"meta-model\", and can be a GBDT, neural network or any other type of model.\n\nIn recent years, there have been more discussion on and experimentation with these ensemble learning paradigms:\n\n- [adaptive ensemble learning](\nhttps://doi.org/10.48550/arXiv.2304.02653): e.g. AdaBoost and Gradient Boosting;\n- [ensemble deep learning](\nhttps://doi.org/10.48550/arXiv.2104.02395): e.g. using neural networks to train neural networks;\n- [federated ensemble learning](https://doi.org/10.48550/arXiv.2206.03852): large-scale training of deep learning models on-device on disjoint subsets of the training data;\n- utilizing AutoML: e.g. Google AdaNet, which [learns a neural architecture from an ensemble of subnetworks](https://jrodthoughts.medium.com/google-adanet-is-a-lightweight-automl-framework-for-building-ensemble-learning-models-cce2b55114b6).","metadata":{}},{"cell_type":"markdown","source":"## **1.2 How popular is ensemble learning in Kaggle Competitions?**\n\nWhile we would like to have a definitive answer to this question, it is hard to answer even if you allow for the setting of an arbitrary metric. \n\nIf a rough estimate is acceptable, let us look at the workload involved in determining the prevalence of ensemble learning in Kaggle Competitions in the past two years from publicly available data. We will be using the [metadata from recent Kaggle Competition Writeups](https://www.kaggle.com/competitions/2023-kaggle-ai-report/data) provided by Kaggle.","metadata":{}},{"cell_type":"code","source":"# Import libraries\nimport pandas as pd\nimport numpy as np\nimport datetime as dt\nimport matplotlib.pyplot as plt\n\n# Retrieve Kaggle writeups data\ncomps = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv')\n\n# Extract competitions in the past two years\n# The starting date is the date this essay was first drafted\nstart_date = dt.datetime(2021, 6, 4) \nend_date = dt.datetime(2023, 6, 4)\ncomps['Competition Launch Date'] = pd.to_datetime(comps['Competition Launch Date'])\nfiltered = comps[(comps['Competition Launch Date'] >= start_date) \n                 & (comps['Competition Launch Date'] <= end_date)]\nprint(\"Number of competitions with writeups in the past 2 years:\", \n      len(filtered['Title of Competition'].unique()))","metadata":{"execution":{"iopub.status.busy":"2023-07-13T13:01:34.181640Z","iopub.execute_input":"2023-07-13T13:01:34.182173Z","iopub.status.idle":"2023-07-13T13:01:34.475782Z","shell.execute_reply.started":"2023-07-13T13:01:34.182138Z","shell.execute_reply":"2023-07-13T13:01:34.474168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"57 competitions are a lot. How about looking at the latest 10.","metadata":{}},{"cell_type":"code","source":"# Get the last ten competitions\nlatest_comps = filtered['Title of Competition'].unique()[-10:]\nprint(\"\\nThe latest 10 competitions:\")\n# Number the outputs\nfor i, name in enumerate(latest_comps):\n    print(f\"{i+1}) {name}\")","metadata":{"execution":{"iopub.status.busy":"2023-07-13T11:50:44.140901Z","iopub.execute_input":"2023-07-13T11:50:44.141440Z","iopub.status.idle":"2023-07-13T11:50:44.150187Z","shell.execute_reply.started":"2023-07-13T11:50:44.141408Z","shell.execute_reply":"2023-07-13T11:50:44.148863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Crop the chosen competitions\nlatest_data = filtered[filtered['Title of Competition'].isin(latest_comps)]\n\n# Reset index and start from 1 for easier reference\nlatest_data = latest_data.reset_index()\nlatest_data.index += 1\nprint(\"Number of writeups from the latest 10 competitions to evaluate:\", len(latest_data))","metadata":{"execution":{"iopub.status.busy":"2023-07-13T13:01:19.341080Z","iopub.execute_input":"2023-07-13T13:01:19.341637Z","iopub.status.idle":"2023-07-13T13:01:19.353258Z","shell.execute_reply.started":"2023-07-13T13:01:19.341603Z","shell.execute_reply":"2023-07-13T13:01:19.351527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us see how many of these 144 writeups involve ensemble learning. We will be adding a new column **\"uses_ensemble\"** to the filtered metadata dataframe that has the value 1 when ensemble was used and 0 otherwise. \n\nNote that a writeup is only considered as using ensemble if the final solution is an ensemble learning solution. A solution does not count as using ensemble if:<br>\n- ensembling was only experimented with, and was ultimately dropped from the final solution;<br>\n- the competition is, or can be approached as, multi-objective (e.g. [OTTO – Multi-Objective Recommender System](https://www.kaggle.com/competitions/otto-recommender-system/overview)), and a single model was used to complete each objective;\n- it is a [retriever-reranker two-stage model](https://www.kaggle.com/c/39585/discussion/394896) and each stage was completed with a single model;\n- it is a non-machine learning solution.\n\nThese criteria were set because the thesis of this essay requires finding testaments to ensemble learning's incredible effectiveness.","metadata":{}},{"cell_type":"code","source":"# Encoding: 1 for \"yes\", 0 for \"no\"\nuses_ensemble = [1,1,1,1,1,1,1,1,1,0,\n                 0,1,0,0,0,1,1,1,1,0,\n                 1,0,0,1,1,0,1,1,1,0,\n                 1,1,1,0,1,1,1,1,0,1,\n                 0,0,1,1,1,1,1,1,1,1,\n                 0,1,1,1,0,1,1,1,1,1,\n                 1,1,1,1,1,1,1,1,1,0,\n                 0,0,0,0,1,1,1,1,1,1,\n                 1,1,1,1,1,1,1,1,1,1,\n                 1,1,0,1,1,1,0,0,1,1,\n                 1,0,1,1,1,1,1,1,1,1,\n                 1,1,1,1,1,0,0,1,0,0,\n                 0,1,1,0,1,1,1,0,0,1,\n                 1,1,1,1,1,1,1,1,1,0,\n                 1,1,1,1\n                ]\n\n# Uncomment to view all writeup URLs for checking\n# pd.options.display.max_rows = 144\n# writeups = latest_data['Writeup URL']\n# print(writeups)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plug the labels into the dataframe\nlatest_data['uses_ensemble'] = uses_ensemble\nlatest_data.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-13T11:56:27.827636Z","iopub.execute_input":"2023-07-13T11:56:27.828160Z","iopub.status.idle":"2023-07-13T11:56:27.860696Z","shell.execute_reply.started":"2023-07-13T11:56:27.828124Z","shell.execute_reply":"2023-07-13T11:56:27.859584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have our data! Let us explore it.","metadata":{}},{"cell_type":"code","source":"# View the number of writeups for each competition\nwriteup_count = latest_data.groupby('Title of Competition')['uses_ensemble'].count()\nwriteup_count","metadata":{"execution":{"iopub.status.busy":"2023-07-13T11:59:40.258135Z","iopub.execute_input":"2023-07-13T11:59:40.258555Z","iopub.status.idle":"2023-07-13T11:59:40.269650Z","shell.execute_reply.started":"2023-07-13T11:59:40.258526Z","shell.execute_reply":"2023-07-13T11:59:40.268231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# View the number of writeups using ensemble by competition\nensemble_sum = latest_data.groupby('Title of Competition')['uses_ensemble'].sum()\nensemble_sum","metadata":{"execution":{"iopub.status.busy":"2023-07-13T11:59:30.143150Z","iopub.execute_input":"2023-07-13T11:59:30.143627Z","iopub.status.idle":"2023-07-13T11:59:30.158160Z","shell.execute_reply.started":"2023-07-13T11:59:30.143595Z","shell.execute_reply":"2023-07-13T11:59:30.157267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The 0 really stands out. Why not double check.","metadata":{}},{"cell_type":"code","source":"# Select all entries for \"Santa 2022 - The Christmas Card Conundrum\"\nselect = latest_data[latest_data['Title of Competition'] \n                     == 'Santa 2022 - The Christmas Card Conundrum']\n# Narrow the selection for easier viewing\nselect[['Title of Competition', 'Title of Writeup', 'Writeup URL', 'uses_ensemble']]","metadata":{"execution":{"iopub.status.busy":"2023-07-13T12:03:03.915115Z","iopub.execute_input":"2023-07-13T12:03:03.915617Z","iopub.status.idle":"2023-07-13T12:03:03.931875Z","shell.execute_reply.started":"2023-07-13T12:03:03.915584Z","shell.execute_reply":"2023-07-13T12:03:03.930746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems that deep learning just was not suitable for that competition. Let us see how popular ensemble learning is in all the competition writeups.","metadata":{}},{"cell_type":"code","source":"# Calculate the percentage of writeups using ensemble \nnumer = sum(uses_ensemble)\ndenom = len(uses_ensemble)\npercent = round(numer/denom*100, 2)\nprint(f\"{percent}% ({numer} out of {denom}) of the writeups used ensemble learning\")","metadata":{"execution":{"iopub.status.busy":"2023-07-13T12:03:22.715532Z","iopub.execute_input":"2023-07-13T12:03:22.716762Z","iopub.status.idle":"2023-07-13T12:03:22.724317Z","shell.execute_reply.started":"2023-07-13T12:03:22.716706Z","shell.execute_reply":"2023-07-13T12:03:22.722963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us plot the proportion of writeups using ensemble learning in a graph to get a sense of its popularity across different types of competitions.","metadata":{}},{"cell_type":"code","source":"# Get the proportions\ndata = ensemble_sum.values/writeup_count.values\n# Sort by descending order\nsort_idx = np.argsort(data)\ntitles = writeup_count.index[sort_idx]\nvals = data[sort_idx]\n# Create horizontal bar chart\nplt.barh(titles, vals, height=0.5, color='blue')\nplt.yticks(rotation=0)\nplt.ylabel('Competition Name')\nplt.xlabel('Proportion of writeups with ensemble')\nplt.title('Ensemble Usage by Competition')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-07-13T11:59:58.964015Z","iopub.execute_input":"2023-07-13T11:59:58.964458Z","iopub.status.idle":"2023-07-13T11:59:59.372132Z","shell.execute_reply.started":"2023-07-13T11:59:58.964427Z","shell.execute_reply":"2023-07-13T11:59:59.371135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We observe that only 3 out of the 10 competitions have the proportion of writeups using ensemble under 50%. In contrast, there were 2 competitions with 100% ensemble writeups. \n\nIn total, ~76% (110 over 144) of the writeups used ensemble learning.\n\nOf course, since some competitions have many more writeups than others, the relative length of the bars must be taken with a grain of salt since they could be shaken up with more data (or if some of the writeups were misclassified).\n\nConsidering the overall trend, though, it is safe to say that ensemble learning is highly popular except in a small number of competitions.","metadata":{}},{"cell_type":"markdown","source":"## **1.3 How effective is ensemble learning?**\n\nThe effectiveness of ensemble learning in Kaggle Competitions depends on the nature of the problem and the competition objectives. There are some competitions where the ability to amalgamate a large number of models is the discriminant:\n- [\"Otto Group Product Classification Challenge\"](https://www.kaggle.com/c/otto-group-product-classification-challenge): the 1st place solution involved [stacking over 33 models](https://www.kaggle.com/competitions/otto-group-product-classification-challenge/discussion/14335);\n\n- [\"Home Credit Default Risk\"](https://www.kaggle.com/competitions/home-credit-default-risk): the 2nd place went to a [team of 12 Kagglers](https://www.kaggle.com/competitions/home-credit-default-risk/discussion/64722). It demonstrated the use of a technique called \"adversarial stochastic blending\";\n\n- [\"Feedback Prize - English Language Learning\"](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/overview): the 1st place solution was the work of [a team of three ensembling over a dozen pre-trained transformer models with different pooling techniques and max lengths](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369457).\n\nWhile huge ensembles have worked wonders in some competitions, smaller ensembles trained on well-curated data and with careful cross-validation have outperformed larger ones in others:\n- [\"CommonLit Readability Prize\"](https://www.kaggle.com/competitions/commonlitreadabilityprize/overview): the 1st place winner trained [5 Bert models on external data](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/257844); 3rd place was an [ensemble of 3 models with simple averaging and blending techniques](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258095); while the 6th place solution consisted of a [stack of 9 transformer models and a Gaussian Process Regression (GPR) model](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258554).\n\nAs we have observed from the previous section, there are also competitions where non-ensemble learning (and sometimes even non-machine learning!) solutions came out on top:\n- [\"G2Net Detecting Continuous Gravitational Waves\"](https://www.kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves/overview): the 1st place solution [summed power along various signal patterns](https://www.kaggle.com/c/37077/discussion/375910);\n- [\"Santa 2022 - The Christmas Card Conundrum\"](https://www.kaggle.com/competitions/santa-2022/overview): [beam search](https://www.kaggle.com/c/41879/discussion/379167) was the magic potion;\n- [\"March Machine Learning Mania 2023\"](https://www.kaggle.com/competitions/march-machine-learning-mania-2023/overview) was won with just [XGBoost, the LeBron James of machine learning algorithms](https://www.kaggle.com/c/46801/discussion/399553). \n\nInterestingly, in the [\"IceCube - Neutrinos in Deep Ice\" competition](https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/overview), the 2nd place winner remarked that their [linear ensemble of 6 models gave only insignificant improvements](https://www.kaggle.com/c/38257/discussion/402882) over their single model submissions; while the 13th place writeup remarked that they had separate submissions with single models that had comparable LB scores, and that they consider the [ensemble effect to be weak](https://www.kaggle.com/c/38257/discussion/405563).\n\nIn short, ensemble learning is definitely something to try when you are aiming for the top, but it does not necessarily improve results and is not a must for winning competitions. Its usefulness depends on the nature of the competition and your problem-solving approach. \n\nAlso, ensembles are not the larger and more complex the better. Diligent cross-validation and engineering rigor are needed to find the sweet spot.","metadata":{}},{"cell_type":"markdown","source":"## **1.4 Can ensemble learning be said to be unreasonably effective?**\n\nThe fact that by combining multiple weaker models, we can typically produce a stronger, more robust model that is better able to generalize to the test dataset is not what is considered unreasonable. \n\nThere are many theories from research on statistical inferencing and machine learning that show why this is generically the case, such as the [\"No Free Lunch\" Theorem](https://towardsdatascience.com/what-no-free-lunch-really-means-in-machine-learning-85493215625d#) and [\"Wisdom of Crowds\"](https://doi.org/10.48550/arXiv.1605.04074). Other supporting theories are centered on [bias-variance decomposition; statistical, computational, and representational arguments; and diversity](https://arxiv.org/pdf/2104.02395.pdf).\n\nHow exactly a particular machine learning strategy benefits from ensemble learning, however, is shrouded in mystery. From the writeups of Kagglers surveyed in the previous section, we see that their *modus operandi* when ensembling models is intuition-based and results-focused; with forays into theory usually speculative and inconclusive.  \n\nThe typical workflow is training dozens of models and stacking or blending them together in various combinations, finding which ones work best through leaderboard probing or cross-validation (CV), and choosing the one with the best LB or CV score, or striking a balance between them. \n\nOf course, pragmatism is always the driver behind choosing any solution over the other in Kaggle Competitions (\"trust your CV\" is the mantra), but it seems that the discovery of the incredible effectiveness of ensemble learning turned it up to eleven.\n\nThe dominance of ensemble learning in most Kaggle Competitions has caused some Kagglers to think that they must try ensembling to remain competitive, even when they do not have a clear rationale or direction for doing so. So they form teams, build dozens of models, and submit a multitude of ensembles with different configurations to probe the leaderboard for direction.\n\nSome Kagglers, after extensively searching for ensemble solutions, found that their painstakingly ensembled models performed no better than their initial single model submissions.\n\nMore often than not though, ad hoc ensembling works. You rank considerably higher than you would have had had you not tediously manufactured many more models. This is what this essay finds \"unreasonably effective\".","metadata":{}},{"cell_type":"markdown","source":"# **2. Reflections**\n\nHaving established that Kaggle Competitions highly rewards ensemble learning solutions that may not be able to justify their use of ensemble learning, it is worth reflecting on whether it is good for the Kaggle or machine learning community that ensemble learning is so popular.\n\nAs with any other machine learning technique, it is important to ensure that ensemble learning is used appropriately and transparently. This requires careful consideration of the trade-offs between results and explainability, and the impact of high-performing but costly (computational power-wise, money-wise and time-wise) models in Kaggle Competitions.","metadata":{}},{"cell_type":"markdown","source":"## **2.1 Concerns about explainability and transparency**<br>\n\nEnsemble models often face [challenges when it comes to explainability](https://arxiv.org/abs/2105.06791). Since ensembling greatly increases model complexity, it can be difficult to understand how and why some aggregations of model outputs improve prediction results more than others, especially when the individual models in the ensemble are quite opaque themselves.\n\nWhen the mechanisms by which models make decisions are poorly understood, they can be said to have a [lack of transparency](https://crcs.seas.harvard.edu/sites/projects.iq.harvard.edu/files/crcs/files/ai4sg_2020_paper_62.pdf): looking from the outside in, you cannot justify to the end user how they came to their conclusions. This can make it difficult to endorse ensemble learning in real-world usage beyond sheer pragmatism.\n\nAn issue about transparency specific to Kaggle Competitions is the utilization of public notebooks from other Kagglers in ensemble solutions. While post-competition writeups usually give credit to other people's notebooks if their models were used in ensembles, only a small portion of competition submissions are accompanied by writeups and public source code. This makes it hard to police plagiarism in the Kaggle community, and give Kagglers who contribute decent performing models early in the competition their due credit.","metadata":{}},{"cell_type":"markdown","source":"## **2.2 Concerns about the fairness of Kaggle Competitions**\n\nSome Kagglers are of the view that Kaggle code competitions have come to be about running as many models as possible within the time limit, and that this gives an [unfair advantage to teams having larger storage limits](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/249689), making it much more difficult to win solo. \n\nA reply by Kaggle Grandmaster Chun Ming Lee refers to competitions that are virtually impossible to win without ensembling an obscene volume of models as [\"exercises in compute/ensembling\"](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/249689#1369528) \n\nThis sentiment is understandable when you observe that some award-winning solutions require ensembling dozens of models, which incentivizes optimizing computing power to train models and forming large teams, inadvertently raising the barriers of competition so that those unwilling to spend for more computing resources or do social networking to form teams with other experienced competitors are penalized. \n\nIf we take the point of view that Kaggle Competitions should be about in-depth understanding of the problem, data exploration, feature engineering, model selection and fine-tuning, then the unreasonable effectiveness of ensemble learning has clearly tipped the scales in favor of factors unrelated to machine learning technical expertise.\n\nHowever, if we consider [Kaggle Competitions to be about finding the best practical solutions to a given problem, then it would not make sense to restrict ensemble learning strategies](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/249689#1369557). ","metadata":{}},{"cell_type":"markdown","source":"## **2.3 Concerns about the gap between competition and application**\n\nIn industry, it is generally unacceptable to train and deploy many models only for small boosts in performance due to the [increased maintenance burden](https://www.featurespace.com/newsroom/challenges-in-building-production-grade-machine-learning-models/). In Kaggle Competitions, however, marginal differences in scores can have substantial impact on rankings, and thus justify the exponential increase in resource expenditure. \n\nThis gap between what is required to be successful in a Kaggle Competition and what is prudent for an industry practitioner is striking. Some of the highly touted ensemble learning solutions seen in Kaggle Competitions may not be viable solutions in real-life applications. \n\nThankfully, Kaggle is aware of this, and has [introduced resource constraints to some of their recent competitions to ensure that the submitted models are \"production grade\"](https://www.kaggle.com/code/crodoc/relevance-of-kaggle-competitions-for-real-life), such as:\n- [limiting computation to CPU-only](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/overview/code-requirements);\n- [only accepting certain types of (computationally efficient) models](https://www.kaggle.com/competitions/asl-fingerspelling/overview/evaluation);\n- [having parallel efficiency rankings](https://www.kaggle.com/competitions/feedback-prize-effectiveness/overview/description). ","metadata":{}},{"cell_type":"markdown","source":"## **2.4 Personal recommendations**\n\nAs more and more machine learning solutions are being deployed for corporate and consumer usage, [\"AI ethics\" has entered the public consciousness and stirred up heated debates](https://aiindex.stanford.edu/wp-content/uploads/2023/04/HAI_AI-Index-Report-2023_CHAPTER_3.pdf). Many are concerned about the integrity and reliability of machine learning-based systems, services and products.\n\nBuilding a highly accurate and robust model is only half the solution, an equally challenging part is justifying the methods used and ensuring their fairness. Machine learning explainability is thus critical, and equal effort should be expended on explaining why and how a model works.\n\nSince model performance can be easily quantified according to objectives that all competition participants can agree on, it is naturally the most suitable metric to use when doling out rewards and medals. However, this results-first stance may reward black-box approaches in which the winners may not be able to clearly articulate why their techniques were so effective. \n\nI believe that giving separate awards to post-competition writeups that can use sound theory to support their models can be a way to balance the scales, encourage Kagglers to demystify their machine learning strategies, and contribute to the body of knowledge in the wider machine learning community. ","metadata":{}},{"cell_type":"markdown","source":"# **Conclusion**\n\nIt is often said that machine learning is half science, half art. This is particularly relevant to ensemble learning. When in doubt, try blending some models. \n\nEnsemble learning has landed on virtually every Kaggle veteran's checklist of things to try out in the quest to climb up the leaderboard. It can be unreasonably effective in that understanding and justifying your ensembling strategy is not needed to reap its benefits.\n\nOutside of machine learning competitions, however, the opaque nature of the resultant solutions is problematic. While pragmatic experimentation is king in Kaggle Competitions, being able to \"show your working\" is highly valued in real-world applications. More effort in machine learning explainability is needed to close the gap.","metadata":{}},{"cell_type":"markdown","source":"# **References** \n\n**Articles**<br>\nBarsacchi, M. (2022). Challenges in Building Production-Grade Machine Learning Models. Featurespace. Retrieved from https://www.featurespace.com/newsroom/challenges-in-building-production-grade-machine-learning-models/<br>\nLim, Y. (2022). Ensemble Averaging — Improve Machine Learning Performance by Voting. Towards Data Science. Retrieved from https://towardsdatascience.com/ensemble-averaging-improve-machine-learning-performance-by-voting-246106c753ee<br>\nMavuduru, A. (2020). What “no free lunch” really means in machine learning. Retrieved from https://towardsdatascience.com/what-no-free-lunch-really-means-in-machine-learning-85493215625d#<br>\nRodriguez, J. (2021). Google AdaNet is a Lightweight, AutoML Framework for Building Ensemble Learning Models. Retrieved from https://jrodthoughts.medium.com/google-adanet-is-a-lightweight-automl-framework-for-building-ensemble-learning-models-cce2b55114b6<br>\n\n\n**Book**<br>\nKadowaki, D., Sakata, R., Hosaka, K., & Hiramatsu, Y. (2019). Kaggle De Katsu Data Bunseki No Gijutsu. Gijitsu-Hyoron Co., Ltd., Tokyo, Japan. (Chapter 7).<br>\n\n**Papers**<br>\nAlizadeh, H., Yousefnezhad, M., & Minaei Bidgoli, B. (2016). Wisdom of Crowds Cluster Ensemble. https://doi.org/10.48550/arXiv.1605.04074<br>\nGanaie, M. A., Hu, M., Malik, A. K., Tanveer, M., & Suganthan, P. N. (2022). Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence.\nhttps://doi.org/10.48550/arXiv.2104.02395<br>\nHejazinia, M., Huba, D., Leontiadis, I., Maeng, K., Malek, M., Melis, L., Mironov, I., Nasr, M., Wang, K., & Wu, C.-J. (2022). FEL: High Capacity Learning for Recommendation and Ranking via Federated Ensemble Learning. https://doi.org/10.48550/arXiv.2206.03852<br>\nMungoli, N. (2023). Adaptive Ensemble Learning: Boosting Model Performance through Intelligent Feature Fusion in Deep Neural Networks. https://doi.org/10.48550/arXiv.2304.02653<br>\nNgo, H. (2023). Chapter 3: Technical AI Ethics. In Artificial Intelligence Index Report 2023. Retrieved from https://aiindex.stanford.edu/wp-content/uploads/2023/04/HAI_AI-Index-Report-2023_CHAPTER_3.pdf<br>\nWatson, M., Hasan, B. A. S., & Al Moubayed, N. (2021). Agree to Disagree: When Deep Learning Models With Identical Architectures Produce Distinct Explanations. https://doi.org/10.48550/arXiv.2105.06791<br>\nZhou, Y., & Kantarcioglu, M. (2020). On transparency of machine learning models: A position paper. In AI for Social Good Workshop. Retrieved from https://crcs.seas.harvard.edu/sites/projects.iq.harvard.edu/files/crcs/files/ai4sg_2020_paper_62.pdf<br>\n\n**Kaggle Competitions**<br>\nBossan, B., Feigl, J., & Kan, W. (2015). Otto Group Product Classification Challenge. Kaggle. Retrieved from https://kaggle.com/competitions/otto-group-product-classification-challenge<br>\nChow, A., Cameron, G., Georg, M., Sherwood, M., Culliton, P., Sepah, S., Dane, S., & Starner, T. (2023). Google - American Sign Language Fingerspelling Recognition. Kaggle. Retrieved from https://kaggle.com/competitions/asl-fingerspelling<br>\nChow, A., Heinrich, L., Eller, P., Ørsøe, R., & Dane, S. (2023). IceCube - Neutrinos in Deep Ice. Kaggle. Retrieved from https://kaggle.com/competitions/icecube-neutrinos-in-deep-ice<br>\nFranklin, A., Gagnon, D., Maggie, Benner, M., Rambis, N., Baffour, P., Culliton, P., Crossley, S., & ulrichboser (2023). Predict Student Performance from Game Play. Kaggle. Retrieved from https://kaggle.com/competitions/predict-student-performance-from-game-play<br>\nFranklin, A., Maggie, Benner, M., Rambis, N., Baffour, P., Holbrook, R., Crossley, S., & ulrichboser (2022). Feedback Prize - Predicting Effective Arguments. Kaggle. Retrieved from https://kaggle.com/competitions/feedback-prize-effectiveness<br>\nHoward, A., & Holbrook, R. (2022). Santa 2022 - The Christmas Card Conundrum. Kaggle. Retrieved from https://kaggle.com/competitions/santa-2022<br>\nMalatinszky, A., Heintz, A., asiegel, Harris, H., Choi, J. S., Maggie, Culliton, P., & Crossley, S. (2021). CommonLit Readability Prize. Kaggle. Retrieved from https://kaggle.com/competitions/commonlitreadabilityprize<br>\nMessenger, C., inversion, Maggie, Williams, M. J., & Tenorio, R. (2022). G2Net Detecting Continuous Gravitational Waves. Kaggle. Retrieved from https://kaggle.com/competitions/g2net-detecting-continuous-gravitational-waves<br>\nMontoya, A., inversion, Odintsov, K., & Kotek, M. (2018). Home Credit Default Risk. Kaggle. Retrieved from https://kaggle.com/competitions/home-credit-default-risk<br>\nSonas, J., Maggie, & Cukierski, W. (2023). March Machine Learning Mania 2023. Kaggle. Retrieved from https://kaggle.com/competitions/march-machine-learning-mania-2023<br>\nTunguz, B., Dieter, Heads or Tails, Kapoor, K., Pandey, P., Mooney, P., Culliton, P., Mulla, R., Bhutani, S., & Cukierski, W. (2023). 2023 Kaggle AI Report. Kaggle. Retrieved from https://kaggle.com/competitions/2023-kaggle-ai-report<br>\nWand, A., inversion, Normann, P., Baumeister, S., & Wilm, T. (2022). OTTO – Multi-Objective Recommender System. Kaggle. Retrieved from https://kaggle.com/competitions/otto-recommender-system<br>","metadata":{}},{"cell_type":"markdown","source":"**Submission to the 2023 Kaggle AI Report competition**","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')\nsub.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"vals = [\"'Kaggle Competitions'\", \n        \"'https://www.kaggle.com/yeemeitsang/unreasonably-effective-ensemble-learning'\",\n        \"'https://www.kaggle.com/code/crodoc/relevance-of-kaggle-competitions-for-real-life/comments#2332709'\",\n        \"'https://www.kaggle.com/code/juanricardop/2023-ai-report-kaggle-competitions/comments#2332792'\",\n        \"'https://www.kaggle.com/code/insiyajafferji/kaggle-competitions-for-biological-scientists/comments#2332887'\"]\nsub.value = vals\nsub.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}