{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
Kaggle 2023:
The Advancement of Kaggle Compettitions Over Years
\n\n
\n \"Trulli\"\n
\n\n\n
ESSAY BY: DENIS MURIUNGI
","metadata":{}},{"cell_type":"markdown","source":"\n\n

TABLE OF CONTENT

\n\n

0 INTRODUCTION

\n\n

1 RISE OF GENERAL PURPOSE MODELS

\n\n

2 CONTINUED EFFICIENCY IMPROVEMENT FOR ML

\n\n\n

3 GROWING BENEFITS OF ML IN SCIENCE,HEALTH AND SUSTAINABILITY

\n\n

4 BENEFIT OF ML TO INDIVIDUALS AND THE COMMUNITY

\n\n

5 DEEPER AND BROADER UNDERSTANDING OF ML

\n\n

6 FINAL CONCLUSIONS

\n\n

7 REFERENCES

\n\n

8 SUBMISSION

","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport re\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom IPython.display import HTML","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T12:53:33.476713Z","iopub.execute_input":"2023-07-12T12:53:33.480303Z","iopub.status.idle":"2023-07-12T12:53:34.236763Z","shell.execute_reply.started":"2023-07-12T12:53:33.480255Z","shell.execute_reply":"2023-07-12T12:53:34.235747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"writeups_data = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv')\nwriteups_data.head()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-07-12T12:53:34.239060Z","iopub.execute_input":"2023-07-12T12:53:34.239577Z","iopub.status.idle":"2023-07-12T12:53:34.640745Z","shell.execute_reply.started":"2023-07-12T12:53:34.239538Z","shell.execute_reply":"2023-07-12T12:53:34.639553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"writeups_data['SVM'] = writeups_data['Writeup'].str.count('SVM',flags=re.IGNORECASE) + writeups_data['Writeup'].str.count('support vector machine',flags=re.IGNORECASE) > 0\nwriteups_data['Random Forest'] = writeups_data['Writeup'].str.count('random forest',flags=re.IGNORECASE) + writeups_data['Writeup'].str.count('random forest',flags=re.IGNORECASE) > 0\nwriteups_data['Neural Networks'] = writeups_data['Writeup'].str.count('deep learning',flags=re.IGNORECASE) + writeups_data['Writeup'].str.count('neural network',flags=re.IGNORECASE) > 0\nwriteups_data['GBM'] = writeups_data['Writeup'].str.count('gbm',flags=re.IGNORECASE) + writeups_data['Writeup'].str.count('gradient boosting machine',flags=re.IGNORECASE) + writeups_data['Writeup'].str.count('xgboost',flags=re.IGNORECASE) + writeups_data['Writeup'].str.count('catboost',flags=re.IGNORECASE) + writeups_data['Writeup'].str.count('lightgbm',flags=re.IGNORECASE)> 0","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T12:53:34.642192Z","iopub.execute_input":"2023-07-12T12:53:34.642497Z","iopub.status.idle":"2023-07-12T12:53:36.687336Z","shell.execute_reply.started":"2023-07-12T12:53:34.642471Z","shell.execute_reply":"2023-07-12T12:53:36.686405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

\n\n# 0 INTRODUCTION\n
","metadata":{}},{"cell_type":"markdown","source":"In the fast-paced world of data science and machine learning, Kaggle has emerged as a vibrant platform that ignites the competitive spirit of data enthusiasts and drives innovation. Over the past two years, Kaggle competitions have undergone significant advancements, transforming the landscape of data-driven problem-solving and fostering a global community of like-minded individuals passionate about pushing the boundaries of what is possible.
\n\n Background Information
\n Kaggle, was founded in 2010, has evolved from its early days as a data science competition platform and a comprehensive forum for sharing, learning, and collaborating. The platform's remarkable growth and development have been fueled by a combination of factors, including advancements in technology, increasing accessibility of data, and the emergence of diverse global talent.This essay aims to explore the remarkable advancements witnessed in Kaggle competitions over the past two years, highlighting the key factors that have contributed to its success and impact. We will delve into the transformative role of new techniques, the expansion of the Kaggle community, and the platform's efforts to foster collaboration and knowledge sharing. Moreover, we will examine the implications of these advancements on the field of data science and the opportunities they present for aspiring data practitioners.","metadata":{}},{"cell_type":"markdown","source":"

\n\n# 1 RISE OF GENERAL PURPOSE MODELS\n
","metadata":{}},{"cell_type":"markdown","source":"For the last few years we have witnessed immense change in the field of machine learning at the kaggle community. One of them being the development of large models and their usage by kagglers to solve real world problems.General-purpose models such as transformer based models like BERT have gained attention due to their ability perform well across a wide range of tasks. In the last couple of years models in the language domain have grown from billion of parameters trained on ten billions of token data. These increases in dataset and model size have led to significant increases in accurancy for a wide variety of language tasks. Most of these advanced models have shown state-of-the-art results in language understanding benchmarks and open-ended conversational capibilities. One exciting capability demonstrated by these models is being able to generalize to new language tasks with relatively little training.\n\nThe transformer models are also having a major impact in image,video and audio dataset,as shown from kaggle previous competitions such Google - Isolated Sign Language Recognition,BirdCLEF 2023 and many more. Transformers for image recognition and video classification are achieving state-of-the-art results on many competitions. This has demonstrated that co-training models on both image and video datasets can improve performance on both image and video tasks compared to video dataset alone. ","metadata":{}},{"cell_type":"markdown","source":"
\n \"Trulli\"\n
\nBar graphs showing the rise of transformers in both computer vision and natural language processing(NLP) fields on Kaggle. Image Credit: Kaggle 2022: Cloud racing, Ethical AI rising by NGHI HUYNH \n\nThe dataset used for creating the above visualization was the Kaggle Machine Learning & Data Science Survey dataset from the years 2018,2019,2020,2021 and 2022","metadata":{}},{"cell_type":"markdown","source":"The usage of transformers in machine learning has experienced a significant growth in Kaggle over the years.In computer vision,the adoption of transformers started to rise noticeably from 2021 to 2022.This indicates an increasing recognition of the effectiveness of transformers in task related to visual data analysis. Similarly,in the domain of natural language processing(NLP), the usage of transformers on Kaggle has been on the rise since 2019. Moreover, from 2021 to 2021 the growth of transformers usage still continues to accelerate. This growth can be attributed to several factors such as the release on new transformer architectures and the availability of pretrained models ","metadata":{}},{"cell_type":"markdown","source":"

\n\n# 2 CONTINUED EFFICIENCY IMPROVEMENT FOR ML\n
","metadata":{}},{"cell_type":"markdown","source":"One of the most vital aspect of improving efficiency in machine learning on Kaggle competitions is the development and utilization of advanced algorithms. Competition participants strive to devise novel algorithms that reduce computational complexity,enhance training speed and optimize performance.Moreover,techniques such as gradient boosting,deep learning architectures and ensembling methods have been refined over time, enabling more efficieny in model development.\n","metadata":{}},{"cell_type":"code","source":"algorithms_used = writeups_data[['SVM','Random Forest','Neural Networks','GBM']].sum()/len(writeups_data)\nax = algorithms_used.plot(kind='bar')\nax.set_ylabel(\"% of Writeups\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T12:53:36.688565Z","iopub.execute_input":"2023-07-12T12:53:36.688883Z","iopub.status.idle":"2023-07-12T12:53:36.967915Z","shell.execute_reply.started":"2023-07-12T12:53:36.688856Z","shell.execute_reply":"2023-07-12T12:53:36.967088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"According to the writeups data provided for this competition,reveals an interesting trend in the use of gradient boosting algorithms to enhance participants' efficiency and improve their chances of winning. This can be indicated by the trend shown on the bar graph above.","metadata":{}},{"cell_type":"code","source":"writeups_data['Date of Writeup'] = pd.to_datetime(writeups_data['Date of Writeup'])\n# Extract the year from the 'Date of writeup' column\nwriteups_data['Year'] = writeups_data['Date of Writeup'].dt.year\n# Filter the dataset for the years 2021, 2022, and 2023\nfiltered_df = writeups_data[writeups_data['Year'].isin([2021, 2022, 2023])]\ngrouped_data = filtered_df.groupby('Year')[['SVM', 'Random Forest', 'Neural Networks', 'GBM']].mean() * 100\n\n# Plot line graphs for each algorithm\nax = grouped_data.plot(kind='line', marker='o', figsize=(10, 6))\nax.set_xlabel(\"Year\")\nax.set_ylabel(\"% of Writeups\")\nax.set_title(\"Percentage of Writeups mentioning Algorithms over Time\")\n\nplt.xticks(grouped_data.index) # Set x-ticks to be the years in the dataset\nplt.legend(loc='upper right') # Display the legend\n\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T12:53:36.970398Z","iopub.execute_input":"2023-07-12T12:53:36.970936Z","iopub.status.idle":"2023-07-12T12:53:37.590116Z","shell.execute_reply.started":"2023-07-12T12:53:36.970903Z","shell.execute_reply":"2023-07-12T12:53:37.589091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The line graph above plotted using the writeups data, indicates that the the gradient boosting algorithm and neural networks have adopted a significant growth from the year 2021 to 2023. Gradient boosting algorithms such as XGBoost,CatBoost and LightGBM ,have gained recognition in Kaggle competition such as playground series, for their exceptional efficiency and effectiveness in various machine learning tasks.","metadata":{}},{"cell_type":"code","source":"# Get the top 7 solutions based on the 'Title of Writeup' column\ntop_solutions = filtered_df['Title of Writeup'].value_counts().head(7).index\n\n# Create a subset of the filtered dataset containing only the top solutions\nsubset_df = filtered_df[filtered_df['Title of Writeup'].isin(top_solutions)]\n\n# Group the subset dataset by the 'Title of Writeup' column and calculate the percentage of writeups mentioning each algorithm\ngrouped_data = subset_df.groupby('Title of Writeup')[['SVM', 'Random Forest', 'Neural Networks', 'GBM']].mean() * 100\n\n# Plot line graphs for each algorithm\nax = grouped_data.plot(kind='line', marker='o', figsize=(12, 6))\nax.set_xlabel(\"Solution\")\nax.set_ylabel(\"% of Writeups\")\nax.set_title(\"Percentage of Writeups mentioning Algorithms for Top Solutions\")\n\nplt.xticks(range(7), top_solutions, rotation=45) # Set x-ticks to be the top solutions\nplt.legend(loc='upper right') # Display the legend\n\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T12:53:37.591401Z","iopub.execute_input":"2023-07-12T12:53:37.591724Z","iopub.status.idle":"2023-07-12T12:53:37.922054Z","shell.execute_reply.started":"2023-07-12T12:53:37.591697Z","shell.execute_reply":"2023-07-12T12:53:37.921283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The graph above depicting the top winners/solutions in the Kaggle competitions reveals a noteworthy trend ,all the top winners have employed gradient boosting algorithms to secure their victories. This finding shows the significance of gradient boosting algorithms in achieving success in Kaggle competitions.","metadata":{}},{"cell_type":"markdown","source":"Hardware acceleration\n\nEfficient utilization of hardware resources is pivotal in reducing training time and improving parformance in machine learning.Kaggle competition participants have leveraged advancement in hardware acceleration techniques to gain a competitive edge. Graphics Processing Unit(GPUs) and more recently, specialized hardware like Tensor Processing Units(TPUs) , have revolutionized machine learning computation in Kaggle competitions.\n","metadata":{}},{"cell_type":"code","source":"filtered_data = writeups_data[writeups_data['Date of Writeup'].dt.year.isin([2020, 2021, 2022])]\n# Create a dictionary to store the usage counts\nusage_counts = {\n 'Accelerator': ['GPU', 'TPU', 'CPU'],\n 'Usage Count': [\n writeups_data['Writeup'].str.contains('gpu', case=False).sum(),\n writeups_data['Writeup'].str.contains('tpu', case=False).sum(),\n writeups_data['Writeup'].str.contains('cpu', case=False).sum()\n ]\n}\n\n# Create a DataFrame from the usage counts dictionary\nusage_df = pd.DataFrame(usage_counts)\n\n# Define CSS styles for the table\nstyles = [\n {'selector': 'table', 'props': [('border-collapse', 'collapse')]},\n {'selector': 'th, td', 'props': [('border', '1px solid #ddd'), ('padding', '8px')]},\n {'selector': 'th', 'props': [('background-color', '#f2f2f2')]},\n {'selector': 'tr:nth-child(even)', 'props': [('background-color', '#f9f9f9')]},\n {'selector': 'tr:hover', 'props': [('background-color', '#e5e5e5')]},\n {'selector': 'th:first-child, td:first-child', 'props': [('border-left', 'none')]},\n {'selector': 'th:last-child, td:last-child', 'props': [('border-right', 'none')]},\n {'selector': 'th', 'props': [('text-align', 'left')]},\n {'selector': 'caption', 'props': [('caption-side', 'bottom')]}\n]\n\n# Create a styled HTML table\nhtml_table = (\n usage_df.style\n .set_table_styles(styles)\n .set_caption('Table 1: Showing Accelerators Usage')\n .set_properties(**{'text-align': 'center'})\n)\n\n# Display the HTML table\ndisplay(HTML(html_table.render()))\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T12:53:37.923485Z","iopub.execute_input":"2023-07-12T12:53:37.924024Z","iopub.status.idle":"2023-07-12T12:53:38.462858Z","shell.execute_reply.started":"2023-07-12T12:53:37.923995Z","shell.execute_reply":"2023-07-12T12:53:38.461836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Automation and Pipeline Optimization\n\nAutomation plays a vital in machine learning workflows on Kaggle competitions.Competitions participants have leveraged automation tools and frameworks to streamline their model development pipelines. Pipeline optimization libraries like Optuna,GridSearchCV,RandomizedSearchCV,Bayesian Optimization ,Sequential Model Based Optimization, Genetic Algorithms and Hyperopt have been used for hyperparameter tuning, searching for the optimal configurations to improve model performance.Through the automation of time-consuming and repetitive tasks,Kaggle competition participants can focus on higher level problem solving and iterate faster,resulting in increased efficiency.","metadata":{}},{"cell_type":"code","source":"# Define the hyperparameter tuning methods\ntuning_methods = [\n 'GridSearchCV',\n 'RandomizedSearchCV',\n 'Bayesian Optimization',\n 'Sequential Model Based Optimization (skopt)',\n 'Optuna',\n 'Genetic Algorithms',\n 'Hyperopt'\n]\n\n# Create an empty dictionary to store the usage counts\nusage_counts = {method: [0, 0, 0] for method in tuning_methods}\n\n# Iterate over the filtered data and update the usage counts\nfor _, row in filtered_data.iterrows():\n for method in tuning_methods:\n if method.lower() in row['Writeup'].lower():\n usage_counts[method][row['Date of Writeup'].year - 2020] += 1\n\n# Convert the dictionary to a DataFrame\nusage_df = pd.DataFrame.from_dict(usage_counts)\n\n# Plot the line graph\nax = usage_df.plot(kind='line', marker='o', figsize=(10, 6))\n\n# Set the title and labels\nax.set_title('Usage of Hyperparameter Tuning Methods (2020-2022)')\nax.set_xlabel('Year')\nax.set_ylabel('Usage Count')\n\n# Customize the x-axis tick labels\nax.set_xticks(range(len(filtered_data['Date of Writeup'].dt.year.unique())))\nax.set_xticklabels(['2020', '2021', '2022'])\n\n# Add a legend\nax.legend(tuning_methods, loc='center left', bbox_to_anchor=(1, 0.5))\n\n# Display the graph\nplt.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T12:53:38.464237Z","iopub.execute_input":"2023-07-12T12:53:38.464553Z","iopub.status.idle":"2023-07-12T12:53:39.257029Z","shell.execute_reply.started":"2023-07-12T12:53:38.464526Z","shell.execute_reply":"2023-07-12T12:53:39.256213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

\n\n# 3 GROWING BENEFITS OF ML IN SCIENCE,HEALTH AND SUSTAINABILITY\n
","metadata":{}},{"cell_type":"markdown","source":"Over years Machine Learing(ML) has proved to be a transformative technology with far reaching implications across various domains. In Kaggle competitions , machine learning has emerged as a powerful tool in the fields of science, health and sustainability. This has drove various innovation in these fields delivering significant benefits. This, can be proved further by the data shown in the table below which was done using this competition writeups data.","metadata":{}},{"cell_type":"code","source":"# Convert the 'Writeup' column to lowercase for case-insensitive matching\nwriteups_data['Writeup'] = writeups_data['Writeup'].str.lower()\n\n# Calculate the frequencies for each field\nfield_freq = {\n 'Health': writeups_data['Writeup'].str.count('health').sum(),\n 'Technology': writeups_data['Writeup'].str.count('technology').sum(),\n 'Environment Sustainability': writeups_data['Writeup'].str.count('environment|sustainability').sum(),\n 'Education': writeups_data['Writeup'].str.count('education').sum(),\n 'Others': writeups_data['Writeup'].str.count('health|technology|environment|sustainability|education').eq(0).sum()\n}\n\n# Create the contingency table DataFrame\ncontingency_df = pd.DataFrame(field_freq, index=['Frequency'])\n\n# Transpose the DataFrame\ncontingency_df = contingency_df.T\n\n# Reset the index\ncontingency_df.reset_index(inplace=True)\ncontingency_df.rename(columns={'index': 'Field'}, inplace=True)\n\n# Display the contingency table\nHTML(contingency_df.to_html(index=False))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T12:53:39.258248Z","iopub.execute_input":"2023-07-12T12:53:39.258541Z","iopub.status.idle":"2023-07-12T12:53:39.865203Z","shell.execute_reply.started":"2023-07-12T12:53:39.258515Z","shell.execute_reply":"2023-07-12T12:53:39.864208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Advancements In Scientific Research\n\nMachine Learing(ML) techniques applied in Kaggle competitions have facilitated breakthroughs in scientific research. Ranging, from astronomy to particle physics, machine learning has helped analyze and study complex data, accelerating discoveries and yielding new insights. In Kaggle Competitions, participants have leverage various machine learning algorithms to analyze astronomical data,detect patterns in particle physics experiments and identify potential drug candidates in pharmaceutical research","metadata":{}},{"cell_type":"code","source":"# Filter competitions related to science and research\nscience_competitions = writeups_data[writeups_data['Title of Competition'].str.contains('science|research', case=False, na=False)]\n\n# Count the frequency of each competition title\ncompetition_freq = science_competitions['Title of Competition'].value_counts()\n\n# Limit the number of competition titles to show on the plot (optional)\nnum_competitions_to_show = 10\ntop_competitions = competition_freq.head(num_competitions_to_show)\n\n# Plot the frequency of competitions\nplt.figure(figsize=(10, 6))\nax = top_competitions.plot(kind='bar', color='teal')\nax.set_xlabel('Competition Title')\nax.set_ylabel('Frequency')\nax.set_title('Frequency of Competitions in Science and Research')\n\n# Add frequency labels on top of each bar\nfor i, freq in enumerate(top_competitions):\n ax.text(i, freq + 5, str(freq), ha='center')\n\nplt.xticks(rotation=45)\nplt.tight_layout()\nplt.show()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T12:53:39.866624Z","iopub.execute_input":"2023-07-12T12:53:39.866920Z","iopub.status.idle":"2023-07-12T12:53:40.258642Z","shell.execute_reply.started":"2023-07-12T12:53:39.866894Z","shell.execute_reply":"2023-07-12T12:53:40.257530Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Health\n\nKaggle competitions have showcased innovative machine learning solutions that aid in early diseases detection, improve medical imaging analysis and predict patient health outcomes.For example , the competition focussed on RSNA Screening Mammography Breast Cancer Detection have led to the development of machine learning models capable of detecting abnormalities in X-ray images and mammograms. These advancements not only enhance healthcare but also have the potential to save lives and reduce healthcare costs. As shown, in the bar graph below, there is a clear indication that Kaggle competitions are furthering growth in the health sector.","metadata":{}},{"cell_type":"code","source":"# Filter competitions related to health and medicine\nhealth_competitions = writeups_data[writeups_data['Title of Competition'].str.contains('health|medicine', case=False, na=False)]\n\n# Count the frequency of each competition title\ncompetition_freq = health_competitions['Title of Competition'].value_counts()\n\n# Limit the number of competition titles to show on the plot (optional)\nnum_competitions_to_show = 10\ntop_competitions = competition_freq.head(num_competitions_to_show)\n\n# Plot the frequency of competitions\nplt.figure(figsize=(10, 6))\nax = top_competitions.plot(kind='bar', color='teal')\nax.set_xlabel('Competition Title')\nax.set_ylabel('Frequency')\nax.set_title('Frequency of Competitions in Health and Medicine')\n\n# Add frequency labels on top of each bar\nfor i, freq in enumerate(top_competitions):\n ax.text(i, freq + 5, str(freq), ha='center')\n\nplt.xticks(rotation=45)\nplt.tight_layout()\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-12T12:53:40.260068Z","iopub.execute_input":"2023-07-12T12:53:40.260503Z","iopub.status.idle":"2023-07-12T12:53:40.551813Z","shell.execute_reply.started":"2023-07-12T12:53:40.260466Z","shell.execute_reply":"2023-07-12T12:53:40.550722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Environmental Sustainability\n\nKaggle competitions have witnessed the application of machine learning to tackle issues such as climate change, biodiversity conservation and energy efficiency.Machine learning models have been utilized to predict climate patterns, optimize energy consumption and analyze statellite imagery to monitor deforestation and biodiversity loss. The insights gained from Kaggle competitions contribute to the global effort towards a more sustainable future,helping to mitigate environmental risks amd promote responsible stewardship of natural resources.","metadata":{}},{"cell_type":"markdown","source":"

\n\n# 4 BENEFIT OF ML TO INDIVIDUALS AND THE COMMUNITY\n
","metadata":{}},{"cell_type":"markdown","source":"Renowed for their data driven challenges, Kaggle competitions have played a crucial role in showcasing the potential advantages of machine learning. This section explores the diverse benefits of machine learning to individuals and the community,drawing insights from the previous kaggle competitions and the innovative solutions they have generated.\n\nEmpowering Individuals With Data Literacy\n\nKaggle competitions provide a platform for individuals to develop and enhance their data literacy skills. Participants acquire hands-on experience in handling real world dataset, refining their ability to extract meaningful insights and make informed decisions. Through these, competition individuals develop proficiency in data preprocessing,feature engineering and model development,empowering them with valuable data-driven skills which can be applied beyond the realm of Kaggle. This improved data literacy fosters a culture of informed decision making and equip individuals to contribute meaningfully to their communities.\n\n![Kaggle leading in the number of dataset created and sharing](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhICA4Cni_7mE_ntz8pmTIRu3iZmcaeKn9SrfNs5GW3gHPF7JDfyhxT7G9UCt72b0DNIaw2-dtUrhPzUUWNceuNM2LIB9J4MovWKmscYzTvpcNge1fOnFvzMlgP78w2LCa-ipMD-ODIcbSjRzc8uj3kEDPZas3pokUoyrA37TKyXNwYCTfHQfBBhBUaUvY/w597-h454/kaggle_top.png)\n\nA piechart showing the distribution of dataset among various machine learning and data science platforms. Image Credit: Trail through 'Law of the Jungle' by PRANAV941\n\nAccording to the pie chart show above, which was create using the Kaggle Machine Learning & Data Science Survey dataset from the years 2018,2019,2020,2021 and 2022, Kaggle emerges as the leading platform,accounting for the 39% of data shared. This statistic highlights the significant role that the Kaggle competitions play in fostering data literacy and driving advancement in the field of machine learning.\n\nHealth\n\nKaggle competitions ahve showcased innovative machine learning solutions that aid in early diseases detection, improve medical imaging analysis and predict patient health outcomes.For example , the competition focussed on RSNA Screening Mammography Breast Cancer Detection have led to the development of machine learning models capable of detecting abnormalities in X-ray images and mammograms. These advancements not only enhance healthcare but also have the potential to save lives and reduce healthcare costs.\n\nCareeer Advancement and Professional Development\n\nParticipating in Kaggle competitions and achieving success can have a significant impact on individuals' career trajectories. These competitions provide a showcase for participants to demonstrate their skills and expertise, often attracting attention of potential employers or collaborators. Additionally, the interactive nature of Kaggle competitions encourages continous learning and keeps individuals updated of the latest machine learning techniques and methodologies, thus contributing to their career development.\n\n![Trends in Data science and data engineering](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS_qGv5wpTt8IJyUl1eWpCGna6-BtPGsmYOuMJIAs5cLDh_JfxH2z8DmVWm1Pk7y-In0-EALbwSLGCqCghLoYihJQBsauISbmXpvd8twVexR2EiEKQV4muSFWz33MVU7X5hK_KMzfHKhDbsIgqYULjJM88WsaqKMuKabbKxYjFkKMSdUcq-Z9gtTQrxHo/w655-h325/career_advancement.png)\n\nLine graph showing the variation between people who want to become data scientist and data analysts. Image Credit: Trail through 'Law of the Jungle' by PRANAV941 \n\nAs shown by the line graphs above, the number of individuals aspiring to become data scientists and data analysts reveals a notable surge from 2021 to 2022\n\nOpen Source Contributions and Reproducible Research\n\nKaggle competitions emphasize open source contribution and reproductibility,which benefit both the individuals and the wider community. Participants often share their code, methodologies and models, allowing others to replicate and build upon their work. This open collaboration fosters a culture of transparency,enabling community to collectively enhance the state-of-the-art in machine learning.","metadata":{}},{"cell_type":"markdown","source":"

\n\n# 5 DEEPER AND BROADER UNDERSTANDING OF ML\n
","metadata":{}},{"cell_type":"markdown","source":"Continued utilization of machine learning and data science through Kaggle competitions contributes to an evolving comprehension of machine learning itself. The fundamental objective in machine learning research is to enhance our understanding of data, and one way to achieve this is by identifying and examining anomalous data points.\n \n\n\nKaggle competition enable participants to gain a deeper understanding of machine learning through hands-on experience and active problem solving.By analyzing anomalous data,examining model behaviour,addressing challenges and collaborating with peers,participants develop insights into machine learning principles and limitations.This increased understanding of machine learning helps participants to make informed decisions and improve their performance in future competitions.","metadata":{}},{"cell_type":"markdown","source":"

\n\n# 6 FINAL CONCLUSIONS\n
","metadata":{}},{"cell_type":"markdown","source":"Kaggle competitions have significantly contributed to the advancement and broader understanding of machine learning. Participants gain hands-on experience,refine their skills and collaborate with peers,leading to insights in machine learning techniques,architectures,feature engineering,ensemble methods,intepretability and optimization. These competitions empower individuals,promote knowledge sharing and drive innovation in machine learning.As machine learning become more pervasive, the lessons learned from kaggle competitions remain essential for driving machine learning advancement and unlocking it full potential.","metadata":{}},{"cell_type":"markdown","source":"

\n\n# 7 REFERENCES\n
","metadata":{}},{"cell_type":"markdown","source":"Latest advancements in machine learning
\nAdvanced lectures on AI and Machine Learning
\nIs Kaggle Good For Machine Learning
\nA Beginner's Guide to Kaggle for Data Science
\nTrail through 'Law of the Jungle'
\nKaggle 2022: Cloud racing, Ethical AI rising\n
\n ✔️TOP Machine Learning Algorithms -Beginner💥
\n Comprehensive Guide on Feature Selection
\n When to use kaggle accelerators
\n Topic 10. Gradient Boosting
\n When to use CPUs vs GPUs vs TPUs in a Kaggle Competition?
\n Hyperparameter Tuning in Machine Learning","metadata":{}},{"cell_type":"markdown","source":"

\n\n# 8 SUBMISSION\n
","metadata":{}},{"cell_type":"code","source":"submission = pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/sample_submission.csv\")\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-12T12:53:40.553181Z","iopub.execute_input":"2023-07-12T12:53:40.553899Z","iopub.status.idle":"2023-07-12T12:53:40.569678Z","shell.execute_reply.started":"2023-07-12T12:53:40.553866Z","shell.execute_reply":"2023-07-12T12:53:40.568971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.loc[0]['value'] = 'Kaggle competitions'\nsubmission.loc[1]['value'] = 'https://www.kaggle.com/code/denismunene/the-advancement-of-kaggle-competitions-over-years?scriptVersionId=135253931#3-GROWING-BENEFITS-OF-ML-IN-SCIENCE,HEALTH-AND-SUSTAINABILITY'\nsubmission.loc[2]['value'] = 'https://www.kaggle.com/docs/competitions'\nsubmission.loc[3]['value'] = 'https://www.kaggle.com/code/paultimothymooney/list-of-every-kaggle-competition-updated-monthly'\nsubmission.loc[4]['value'] = 'https://www.kaggle.com/code/ashagutlapalli/evolution-of-kaggle'\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-12T12:53:40.570702Z","iopub.execute_input":"2023-07-12T12:53:40.571569Z","iopub.status.idle":"2023-07-12T12:53:40.582305Z","shell.execute_reply.started":"2023-07-12T12:53:40.571539Z","shell.execute_reply":"2023-07-12T12:53:40.581196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2023-07-12T12:53:40.585361Z","iopub.execute_input":"2023-07-12T12:53:40.585708Z","iopub.status.idle":"2023-07-12T12:53:40.598219Z","shell.execute_reply.started":"2023-07-12T12:53:40.585680Z","shell.execute_reply":"2023-07-12T12:53:40.597204Z"},"trusted":true},"execution_count":null,"outputs":[]}]}