{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# How to Win a Kaggle Competition\n\n###### Topic: Kaggle Competitions\n###### Author: Darek Kłeczek\n######   \n\nI compete on Kaggle. I like learning about new problems and experimenting with different methods. I like the adrenaline rush while checking the leaderboard score. And obviously, I like winning. \n\nWhen I join a competition, I research winning solutions from past similar competitions. It takes a lot of time to read and digest them, but it's an incredible source of ideas and knowledge. But what if we could learn from *all* the competitions? We've been given a list of Kaggle writeups in this competition, but there are so many of them! If only we could find a way to extract some structured data and analyze it... Well, it turns out that large language models (LLMs) [1] can help us extract structured data from unstructured writeups. \n\nIn this essay, I'll start by providing a quick overview of the process I used to collect the data. I will then present several insights from analyzing the dataset. The focus is to understand what the community has learned over the past 2 years of working and experimenting with Kaggle competitions. Finally, I'll mention some ideas for future research.\n\nOne of my personal lessons from Kaggle is that more data usually helps, so I will leverage all the writeups provided in the competition dataset. This data extraction method has only become possible with the recent advances in LLMs, so we couldn't have done it more than 2 years ago. ","metadata":{}},{"cell_type":"markdown","source":"# Data Collection Using LLMs\n\nI used OpenAI LLMs [1,2] to first collect a list of methods from each winning writeup, and then clean up the method categories. As input dataset, I used a combination of structured data from [Kaggle Solutions](https://farid.one/kaggle-solutions/) [3] and this competition's dataset (the dataset is filtered to write-ups that are featured in both sources only). The process is described in [this article](https://wandb.ai/darek/llmapps/reports/Using-LLMs-to-Extract-Structured-Data-OpenAI-Function-Calling-in-Action--Vmlldzo0Nzc0MzQ3) [4], and the code is shared in [this notebook](https://www.kaggle.com/code/thedrcat/using-llms-to-extract-structured-data) [5]. I also published the resulting [dataset](https://www.kaggle.com/datasets/thedrcat/kaggle-winning-solutions-methods).\n\nThe original data processing consisted of two steps. First, I input each solution writeup to an LLM with a prompt requesting it to return a list of machine learning methods mentioned in that specific writeup. This results in a rather noisy list, so in the second step I concatenate all the methods and ask an LLM to standardize this list, mapping from the original method to a more structured, cleaner category. You can see below a subset of data that I generated. The column `methods` contains the noisy, original method, and the column `cleaned_methods` contains the cleaner category. ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\ndetails = pd.read_csv('/kaggle/input/kaggle-winning-solutions-methods/kaggle_winning_solutions_methods_detail.csv')\ndetails[['competition_name', 'place', 'year', 'methods', 'cleaned_methods']].head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:48:39.744685Z","iopub.execute_input":"2023-07-16T21:48:39.744976Z","iopub.status.idle":"2023-07-16T21:48:41.465013Z","shell.execute_reply.started":"2023-07-16T21:48:39.744956Z","shell.execute_reply":"2023-07-16T21:48:41.464186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looking into the specific, detailed methods is still helpful to understand the popularity of the bigger categories. I originally displayed raw data generated by LLM in the detailed charts. The reviewers pointed out that noise, such as different spellings of the same method name, made the charts harder to read. To address this, I performed manual cleanup. I spent many hours on this process, which is documented in [this notebook](https://www.kaggle.com/code/thedrcat/kaggle-winning-solution-methods-review/) [6], and I feel there's still noise in the data - but it should be good enough to capture trends.\n\nWhile cleaning up the data, I noted some LLM categorizations that I disagreed with. For example, `ReduceLROnPlateau` was listed under `Optimizers` category, and I moved it under `Training Methods`. I encourage the readers to be conscious of potential miscategorizations by the LLM or the human author and point them out in the comments!","metadata":{}},{"cell_type":"markdown","source":"# Most Popular Methods\n\nLet's start with the most basic question: what are the most popular methods used in Kaggle winning solutions descriptions? ","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\nfrom matplotlib import pyplot as plt\n\ndf = pd.read_csv('/kaggle/input/kaggle-winning-solutions-methods/kaggle_winning_solutions_methods.csv')\n\nfig, ax = plt.subplots(figsize=(8, 6))\nsns.countplot(data=df, x='cleaned_methods', order=df.cleaned_methods.value_counts().head(20).index, ax=ax)\n\nax.set_title('Most Popular Methods')\nax.set_xlabel(\"Method name\")\nax.set_ylabel(\"Number of solutions featuring method\")\nax.tick_params(axis='x', rotation=90)\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:48:41.466297Z","iopub.execute_input":"2023-07-16T21:48:41.466524Z","iopub.status.idle":"2023-07-16T21:48:42.305870Z","shell.execute_reply.started":"2023-07-16T21:48:41.466504Z","shell.execute_reply":"2023-07-16T21:48:42.305103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The first place, *Ensemble Methods* is perhaps unsurprising. I've personally experienced the frenzy of teaming up to blend diverse solutions and the effort to stuff as many models as possible within the notebook inference limit. The remaining methods are very interesting though! Let's try to uncover some more details there. ","metadata":{}},{"cell_type":"markdown","source":"# Unpacking Data Augmentation\n\nThe second place surprised me, to be honest. But maybe it shouldn't? In real life, data is often the most critical aspect of building a good predictive model. On Kaggle, we're usually given the dataset, but we can augment it! I've always used data augmentation for deep learning, including some [fancy solutions for NLP](https://www.kaggle.com/competitions/chaii-hindi-and-tamil-question-answering/discussion/287923) models [7]. Let's see what's behind this category by listing the detailed methods extracted from each description. ","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(8, 6))\nsns.countplot(data=details[details.cleaned_methods == 'Data Augmentation'], y='methods', order=details[details.cleaned_methods == 'Data Augmentation'].methods.value_counts().head(15).index, ax=ax)\n\nax.set_title('Granular Data Augmentation Methods')\nax.set_xlabel(\"Number of solutions featuring method\")\nax.set_ylabel(\"Method name\")\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:48:42.306871Z","iopub.execute_input":"2023-07-16T21:48:42.307203Z","iopub.status.idle":"2023-07-16T21:48:42.604565Z","shell.execute_reply.started":"2023-07-16T21:48:42.307182Z","shell.execute_reply":"2023-07-16T21:48:42.603784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Unfortunately, the LLM wasn't very granular in most of the cases. Beyond the generic category, we can see the importance of test time augmentation, a number of popular vision transforms, and finally external data additions. There's definitely more research to be done here! ","metadata":{}},{"cell_type":"markdown","source":"# CNNs vs RNNs vs Transformers\n\nThere's a popular belief that [transformers are taking over](https://twitter.com/karpathy/status/1468370605229547522?lang=en) as the default neural network architecture. Looking at our data, they don't seem as dominant though! Is this a matter of timing? Transformers are fairly new, so maybe the statistics haven't caught up yet... Let's plot the percentage of solutions each year that mention the most popular deep learning architectures: Transformers [8], Convolutional Neural Networks (CNNs) [9, 10], and Recurrent Neural Networks (RNNs) [11]. \n\n*Note: In the first version of this essay Transfomers were behind both CNNs and RNNs in terms of popularity. After manual data cleanup & review, Transformers have overtaken RNNs, but CNNs got even more popular.*","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(8, 6))\n\ndf['cnn'] = df.cleaned_methods.str.contains('CNNs')\ndf['transformer'] = df.cleaned_methods.str.contains('Transformers')\ndf['rnn'] = df.cleaned_methods.str.contains('RNNs')\n\ndf.groupby('year').cnn.mean().plot(kind='line', ax=ax)\ndf.groupby('year').transformer.mean().plot(kind='line', ax=ax)\ndf.groupby('year').rnn.mean().plot(kind='line', ax=ax)\n\nax.legend(['CNNs', 'Transformers', 'RNNs'])\nax.set_title('Popularity of Methods Over Time')\nax.set_xlabel(\"Competition year\")\nax.set_ylabel(\"% solutions featuring method\")\nplt.tight_layout()\nplt.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:48:42.606384Z","iopub.execute_input":"2023-07-16T21:48:42.606638Z","iopub.status.idle":"2023-07-16T21:48:42.907290Z","shell.execute_reply.started":"2023-07-16T21:48:42.606617Z","shell.execute_reply":"2023-07-16T21:48:42.906530Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well, this is interesting! We can see how deep learning (DL) entered the Kaggle scene in 2016 with CNNs first, soon followed by RNNs. Transformers started appearing in 2019, and peaked in 2022. As Transformers grew, RNNs started declining, and CNNs slowed down growth first, and then declined abruptly in 2022, which is when Transformer became the most popular DL architecture. The data for 2023 is still incomplete, so we're yet to see if the CNNs will make a comeback, which is indicated by the partial data we have. ","metadata":{}},{"cell_type":"markdown","source":"# Enter GBDT \n\nI started this analysis with the three architectures listed above, and then decided to add Gradient Boosting (GBDT) [12] to the chart. The results made me remember the [Godzilla vs Kong vs Doge meme](https://www.reddit.com/r/MemeRestoration/comments/qz1019/godzilla_vs_kong_vs_doge_bonk_meme_template_blank/). GBDT for the win!","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(8, 6))\n\ndf['gbdt'] = df.cleaned_methods.str.contains('Gradient Boosting')\n\ndf.groupby('year').gbdt.mean().plot(kind='line', ax=ax)\ndf.groupby('year').cnn.mean().plot(kind='line', ax=ax)\ndf.groupby('year').transformer.mean().plot(kind='line', ax=ax)\ndf.groupby('year').rnn.mean().plot(kind='line', ax=ax)\n\nax.legend(['GBDTs', 'CNNs', 'Transformers', 'RNNs'])\nax.set_title('Popularity of Methods Over Time')\nax.set_xlabel(\"Competition year\")\nax.set_ylabel(\"% solutions featuring method (log scale)\")\nax.set_yscale('log')  # Set y-axis to logarithmic scale\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:48:43.347921Z","iopub.execute_input":"2023-07-16T21:48:43.348389Z","iopub.status.idle":"2023-07-16T21:48:43.803022Z","shell.execute_reply.started":"2023-07-16T21:48:43.348362Z","shell.execute_reply":"2023-07-16T21:48:43.802237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The dominance of gradient boosting methods before the deep learning era is probably not surprising, especially given the popularity of tabular competitions back then. It then declined, potentially as Kaggle added more CV and NLP competitions. But it shot back up starting 2022 (notice log scale on the above chart). This can't be explained by tabular competitions alone. I have used GBDT in combination with Transformers before, for example in an [NLP competition](https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235) [13]. I wondered how other Kagglers combined GBDT with DL architectures. Below is a random sample of solutions that used both GBDT and one of the DL architectures.","metadata":{}},{"cell_type":"code","source":"combined_methods = df.groupby(['year', 'competition_name', 'place', 'link']).agg({'cleaned_methods' : list}).reset_index()\ncombined_methods['both'] = combined_methods.cleaned_methods.apply(lambda x: 'Gradient Boosting' in x and ('Transformers' in x or 'CNNs' in x or 'RNNs' in x))\nsample = combined_methods[combined_methods.both == True].sample(n=5, random_state=1)\nfor i, (_, row) in enumerate(sample.iterrows()):\n    print(f'[{14 + i}] {row.place}st place in {row.competition_name} ({row.year}): {row.link}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:48:43.804257Z","iopub.execute_input":"2023-07-16T21:48:43.804743Z","iopub.status.idle":"2023-07-16T21:48:43.853278Z","shell.execute_reply.started":"2023-07-16T21:48:43.804715Z","shell.execute_reply":"2023-07-16T21:48:43.852474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Reviewing these solutions, we can see gradient boosting methods (usually XGBoost or LightGBM) used in combination with DL methods (CNNs or Transformers). GBDT is used to combine structured and unstructured data [14, 15] and for stacking on top of an ensemble of base DL models [16, 17, 18]. While the bitter lesson [24] says that general methods leveraging computation are the most effective in the long term, Kagglers still find creative ways of improving the results, such as reducing dimensionality of a CNN feature map via t-SNE and then feeding it as features into a LightGBM model [17]. ","metadata":{}},{"cell_type":"markdown","source":"# Tuning is important, but there are good defaults!\n\nBoth experience and the high level data above tell me that you need to be really good at tuning models to win a Kaggle competition. But what do we need to tune? Are there sensible defaults we can use? Let's explore optimizers and loss functions in detail.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(8, 6))\nsns.countplot(data=details[details.cleaned_methods == 'Optimizers'], y='methods', order=details[details.cleaned_methods == 'Optimizers'].methods.value_counts().head(10).index, ax=ax)\n\nax.set_title('Granular Optimizers Methods')\nax.set_xlabel(\"Number of solutions featuring method\")\nax.set_ylabel(\"Method name\")\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:48:43.854445Z","iopub.execute_input":"2023-07-16T21:48:43.854747Z","iopub.status.idle":"2023-07-16T21:48:44.070636Z","shell.execute_reply.started":"2023-07-16T21:48:43.854721Z","shell.execute_reply":"2023-07-16T21:48:44.069853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(8, 6))\nsns.countplot(data=details[details.cleaned_methods == 'Loss Functions'], y='methods', order=details[details.cleaned_methods == 'Loss Functions'].methods.value_counts().head(10).index, ax=ax)\n\nax.set_title('Granular Loss Functions')\nax.set_xlabel(\"Number of solutions featuring method\")\nax.set_ylabel(\"Method name\")\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:48:44.071463Z","iopub.execute_input":"2023-07-16T21:48:44.071673Z","iopub.status.idle":"2023-07-16T21:48:44.317641Z","shell.execute_reply.started":"2023-07-16T21:48:44.071653Z","shell.execute_reply":"2023-07-16T21:48:44.316829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lot's of people mention optimizers and solution descriptions in the writeups, but most default to popular choices. Adam family [19, 20] rules the optimizer landscape. Other than focal loss [21] (imbalanced data is reality on Kaggle too!), most solutions go with standard CE/BCE/MSE loss functions.","metadata":{}},{"cell_type":"markdown","source":"# What's the magic?\n\nKaggle competitors often search for magic, but can we see it in our data? Let's try to unpack somewhat mysterious *Labeling Techniques* and *Post-processing* categories. ","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(8, 6))\nsns.countplot(data=details[details.cleaned_methods == 'Labeling Techniques'], y='methods', order=details[details.cleaned_methods == 'Labeling Techniques'].methods.value_counts().head(10).index, ax=ax)\n\nax.set_title('Granular Labeling Techniques')\nax.set_xlabel(\"Number of solutions featuring method\")\nax.set_ylabel(\"Method name\")\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:49:37.363063Z","iopub.execute_input":"2023-07-16T21:49:37.363516Z","iopub.status.idle":"2023-07-16T21:49:37.521985Z","shell.execute_reply.started":"2023-07-16T21:49:37.363494Z","shell.execute_reply":"2023-07-16T21:49:37.521444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# as you can see from the line below, even after a few hours of cleanup I still find duplicates :S\ndetails['methods'] = details['methods'].apply(lambda x: 'Post-processing' if x == 'Postprocessing' else x)\nfig, ax = plt.subplots(figsize=(8, 6))\nsns.countplot(data=details[details.cleaned_methods == 'Post-processing'], y='methods', order=details[details.cleaned_methods == 'Post-processing'].methods.value_counts().head(10).index, ax=ax)\n\nax.set_title('Granular Post-processing Methods')\nax.set_xlabel(\"Number of solutions featuring method\")\nax.set_ylabel(\"Method name\")\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:48:44.846846Z","iopub.execute_input":"2023-07-16T21:48:44.847153Z","iopub.status.idle":"2023-07-16T21:48:45.114255Z","shell.execute_reply.started":"2023-07-16T21:48:44.847125Z","shell.execute_reply":"2023-07-16T21:48:45.113414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ah, so labeling techniques category is mostly about pseudo-labels, with a good dose of label smoothing [22]. Do these fit in the same category? I thought they're different at first, but maybe it's all about regularization? Should this be merged with *Data Augmentation*? Anyways, keep them in your Kaggle arsenal!\n\nPost-processing unveils a bit of mystery by mentioning thresholds and averaging, but still remains vague... If there's magic to be found, I'd bet it's there!\n\nAs a side note, I couldn't find a canonical citation for pseudo-labels. I feel this technique - using a model to generate labels on unlabeled data, and then using that to train a new model - has been really made popular and proven on Kaggle. Let's see what the earliest discussion is that contains pseudo-label method in our dataset. I'm not making a claim it's the first one, or should be the citation, but felt it's worth to highlight it! [23]","metadata":{}},{"cell_type":"code","source":"pseudo = details[details.methods == 'Pseudo labeling'].sort_values('nm').head(1)\nfor i, (_, row) in enumerate(pseudo.iterrows()):\n    print(f'[{23 + i}] {row.place}st place in {row.competition_name} ({row.year}): {row.link}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T21:48:45.115619Z","iopub.execute_input":"2023-07-16T21:48:45.115904Z","iopub.status.idle":"2023-07-16T21:48:45.125311Z","shell.execute_reply.started":"2023-07-16T21:48:45.115877Z","shell.execute_reply":"2023-07-16T21:48:45.124306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusions\n\nHere's my personal checklist I'll keep handy for new Kaggle competitions:\n1. Data is key - get the right data and augment it\n2. Don't forget GBDT, CNNs and RNNs! Transformers will probably be top of mind anyway :) \n3. Stick to good defaults (optimizers, loss functions)\n4. Try pseudo-labels\n5. Look for post-processing tricks\n6. Finally, ensemble for the win!\n","metadata":{}},{"cell_type":"markdown","source":"# Future Research\n\nI feel we've just scratched the surface here! There's a lot to be explored, starting with a more careful dataset creation. Can you find out more about data augmentation or post-processing by better prompting an LLM? There's way more insights to be discovered there. Another angle is using more data we collected. I didn't look at the place associated with a given solution at all. Can we predict the place based on the methods used? What would feature importances look like for such a model? You can use my [dataset](https://www.kaggle.com/datasets/thedrcat/kaggle-winning-solutions-methods), the [code](https://www.kaggle.com/code/thedrcat/using-llms-to-extract-structured-data), or just go explore on your own. If you write this up, I'd love to read it myself!","metadata":{}},{"cell_type":"markdown","source":"# Acknowledgments\n\nThank you to the reviewers for providing feedback on the first version of this essay, especially Leonie, CroDoc, Yury Kashnitsky, Yash Sarwaiya.","metadata":{}},{"cell_type":"markdown","source":"# AI-Generated Content Disclosure\n\nLLMs were used to generate the dataset, as described in the essay. I also used ChatGPT [2] to help generate and refactor the code to produce charts - essentially a natural language interface to `matplotlib`. The essay was written by me. ","metadata":{}},{"cell_type":"markdown","source":"# References\n\n1. OpenAI. (2023). GPT-4 Technical Report.\n2. OpenAI. (2023). GPT-3.5 [Computer software]. Retrieved from https://platform.openai.com/docs/models/gpt-3-5\n3. Farid Rashidi. (2023). Kaggle Solutions (website). Retrieved from https://farid.one/kaggle-solutions/\n4. Darek Kłeczek. (2023). Using LLMs to Extract Structured Data - OpenaAI Function Calling in Action. Retrieved from https://wandb.ai/darek/llmapps/reports/Using-LLMs-to-Extract-Structured-Data-OpenAI-Function-Calling-in-Action--Vmlldzo0Nzc0MzQ3\n5. Darek Kłeczek. (2023). Using LLMs to Extract Structured Data. [Kaggle notebook](https://www.kaggle.com/code/thedrcat/using-llms-to-extract-structured-data)\n6. Darek Kłeczek. (2023). Kaggle Winning Solution Methods Review. [Kaggle notebook](https://www.kaggle.com/code/thedrcat/kaggle-winning-solution-methods-review/)\n7. Darek Kłeczek. (2021). chaii: [1st Place Solution](https://www.kaggle.com/competitions/chaii-hindi-and-tamil-question-answering/discussion/287923)\n8. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, & Illia Polosukhin. (2017). Attention Is All You Need.\n9. Lecun, Y., & Bengio, Y. (1995). Convolutional Networks for Images, Speech, and Time-Series. \n10. Krizhevsky, A., Sutskever, I., & Hinton, G. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Neural Information Processing Systems, 25.\n11. Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Comput., 9(8), 1735–1780.\n12. Jerome H. Friedman (2001). Greedy function approximation: A gradient boosting machine.. The Annals of Statistics, 29(5), 1189 – 1232.\n13. Shujun. (2022). Feedback Prize - Evaluating Student Writing: [3rd Place Solution w code and notebook](https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313235)\n14. Phaedrus. (2022). Foursquare - Location Matching: [21st place solution](https://www.kaggle.com/c/foursquare-location-matching/discussion/335857)\n15. Kirderf. (2023). RSNA Screening Mammography Breast Cancer Detection: [31st place solution](https://www.kaggle.com/c/rsna-breast-cancer-detection/discussion/391286)\n16. D.Imanishi. (2021). NFL Health & Safety - Helmet Assignment: [11th place solution](https://www.kaggle.com/c/nfl-health-and-safety-helmet-assignment/discussion/285156)\n17. KaizaburoChubachi. (2020). ALASKA2 Image Steganalysis: [3rd place solution](https://www.kaggle.com/c/alaska2-image-steganalysis/discussion/168870)\n18. Y.Nakama. (2022). Feedback Prize - Predicting Effective Arguments: [5st place solution](https://www.kaggle.com/c/feedback-prize-effectiveness/discussion/347387)\n19. Diederik P. Kingma, & Jimmy Ba. (2017). Adam: A Method for Stochastic Optimization. \n20. Ilya Loshchilov, & Frank Hutter. (2019). Decoupled Weight Decay Regularization. \n21. Lin, T.Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal Loss for Dense Object Detection. In 2017 IEEE International Conference on Computer Vision (ICCV) (pp. 2999-3007).\n22. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the Inception Architecture for Computer Vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2818-2826).\n23. Dong-Hyun Lee. (2013). Challenges in Representation Learning; The Black Box Learning Challenge: [2nd place entry (code+paper)](https://www.kaggle.com/c/challenges-in-representation-learning-the-black-box-learning-challenge/discussion/4726)\n24. Sutton, R. (2019). The bitter lesson. Incomplete Ideas (blog), 13(1).\n\n## Submission","metadata":{}},{"cell_type":"code","source":"# submission - to be edited with peer feedback\nimport pandas as pd\nsub = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/sample_submission.csv')\nsub.iloc[0,1] = \"'Kaggle Competitions'\"\nsub.iloc[1,1] = \"'https://www.kaggle.com/code/thedrcat/how-to-win-a-kaggle-competition'\"\nsub.iloc[2,1] = \"'https://www.kaggle.com/code/iamleonie/towards-green-ai/comments#2332800'\"\nsub.iloc[3,1] = \"'https://www.kaggle.com/code/kashnitsky/essay-competition-cards-recent-nlp-competitions/comments#2334693'\"\nsub.iloc[4,1] = \"'https://www.kaggle.com/code/crodoc/relevance-of-kaggle-competitions-for-real-life/comments#2332801'\"\nsub.to_csv('submission.csv', index=False)\nsub.head()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-16T21:48:45.126825Z","iopub.execute_input":"2023-07-16T21:48:45.127155Z","iopub.status.idle":"2023-07-16T21:48:45.149738Z","shell.execute_reply.started":"2023-07-16T21:48:45.127126Z","shell.execute_reply":"2023-07-16T21:48:45.148882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}