{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Kaggle is the Global Data Science Classroom! 🌎📈

\n

2023 Kaggle AI Report

\n\n# Introduction\n\nAs a professor dedicated to the growth and achievement of my students, one of my main success criteria is implementing the most captivating and effective teaching methods available. I am confident that this principle is shared by countless dedicated educators who are committed to providing an exceptional learning experience for their students. A question often asked is: how can we employ modern technology in our curriculums? Through extensive research and experimentation, I am excited to present a compelling solution in this report about how universities - globally - are utilizing Kaggle to transform the classroom by enhancing student engagement, promoting collaboration, and fostering a deeper understanding of the subject matter.\n\nThis report primarily targets two main audiences:\n\n* **The Kaggle Community** - This report will demonstrate how Kaggle Competitions have changed in the past two years with the introduction of Community Competitions and how there is a new audience who is not as interested in competing for prizes and kudos, but rather, creating and participating in learning exercises in a classroom setting.\n* **STEM Educators** - This report will give educators a background on Kaggle and how other educators are using Kaggle to promote science communication in their classrooms and abroad.\n\nMy hope is this report will inspire people on how scientific topics can be communicated with modern technology and create a more inclusive and engaging experience for all.\n\n# Kaggle Notebooks are 21st-Century Textbooks\n\nKaggle Notebooks are modernizing how knowledge is communicated from teacher to student. In my view, Kaggle Notebooks can act as a better replacement to the traditional textbook model for today’s classrooms.\n\n## What is Kaggle?\nKaggle acts not only as a cloud-computing platform but also as an online community for data scientists, machine learning practitioners, and AI enthusiasts. It provides a wide range of resources - which at a high-level include Kaggle Notebooks, Kaggle Datasets, Kaggle Competitions, and community forums [1].\n\nKaggle was founded in April 2010 by Anthony Goldbloom [2] and since then has amasses a broad social community of over 14 million registered users [3].\n\n## What are Kaggle Notebooks?\n\nKaggle Notebooks are open-source web applications for creating and sharing content on the Kaggle platform - a cloud computational environment. This content can be live code, narrative text, data visualizations, or equations. There are three types of Kaggle Notebooks: Jupyter Notebooks (usually referred to as just “notebooks”), RMarkdown Scripts, and Scripts. By far the most popular is Jupyter Notebooks, so when I say “Kaggle Notebook”, I am mainly referring to this type.\n\n
\"Kaggle\n

A screenshot of a Kaggle Notebook running on the Kaggle Platform.

\n

Content is segregated into cells containing formatted text explanations, executable code, and output visualization.\n

\n
\n\n## How do Kaggle Notebooks Provide a Better Learning Environment?\n\nWhen you think about teaching material to students, specifically in the domain of STEM, there are three major challenges you encounter [4]:\n\n* **Hardware** - Each student may have a different device and operating system (Windows, MacOS, Linux). How can you ensure consistency when installing software and during runtime?\n* **Troubleshooting** - Can you give each student technical support? What about in a remote setting? What if the scale is massive (1000+ students?)\n* **Programming Expertise** - Every student comes in with a different skill set, so the content must be approachable by anyone.\n\nKaggle Notebooks solve these problems by being a completely reproducible computational environment that can be shared in a single file and run in web browsers. Content is organized into cells, which are atomic building blocks that can contain code or formatted text (in the form of Markdown). Cells can be run individually or as sequentially, which gives students immediate feedback and also allows for step-by-step analysis. From the instructor's view, it allows them to describe the process and then empower the students to execute it on their own.\n\n\nWhen paired with the entire cloud-computing platform of Kaggle, the technology becomes even more powerful by giving all students the same underlying operating system and hardware. Moreover, Kaggle gives students the opportunity to collaborate, share thoughts and ideas, and maintain versions of their notebooks.\n\n# Educators are using Kaggle to Transform the Classroom\nKaggle gives tools to empower educators through self-hosted community competitions, engaging datasets, tooling, and more. \n\n## Kaggle Competitions\n\nOne of the primary functions of Kaggle is the ability to participate in and host machine learning and data analytics competitions in which participants compete to find the best predictive models for a given dataset. There are two main types of competitions \"Featured\" and \"Community\", but there are other types as well which include \"Research\" and \"Playground\". Featured competitions are typically hosted by Kaggle in partnership with businesses or univerities and marketed on the front-page of Kaggle.com, while Community competitions are self-hosted by individuals or organizations and can be discovered through browsing the broader list of competitions. One example is the most famous Kaggle Competition - Titanic: Machine Learning from Disaster [5] - launched in 2012, in which participants build predictive models to determine which passengers survived the Titanic shipwreck. Kaggle Competitions also often come with incentives to participate such as monetary prizes [6], job opportunities [7], and “swag” (which includes t-shirts, hoodies, stickers, or other promotional materials).\n\nIn Kaggle competitions, it is customary for winners to share their code, allowing the broader community to benefit from their approaches. Notably, exposure to state-of-the-art algorithms has been introduced to the community through these competitions, such as deep neural networks [8] and XGBoost [9].\n\n### Kaggle inClass and Community Competitions\n\nKaggle inClass was first introduced in 2012 as a self-service platform to conduct competitions [10]. The idea behind inClass was for universities to host competitions for students, which had features such as private competitions, limited members, and a simple setup process. In these competitions, performance metrics and leaderboards were given to students. This allowed them to assess their model in comparison to their peers. Furthermore, they were given multiple submission attempts, which gave them the chance to make experimental changes and test further hypotheses.\n\n
\"Kaggle\n

Banner from the 2012 launch of Kaggle inClass.

\n
\n\nIn 2021, inClass was rebranded to Community Competitions and extended to the entire Kaggle Community [10], which broadened educational opportunities to those operating outside of the academic system.\n\n### Explosion of Community Competitions\n\nOvertime, and specifically after the launch of Kaggle inClass, the amount of Community Competitions has greatly outpaced that of the other types of competitions. The ease of setup and use has been an attractive model for educators and data science practitioners to host competitions for learning and for fun.\n\nOf the **5596** competitions in the dataset provided by Kaggle for this report, **5053 (90%)** of competitions are *Community* driven.\n\nThe below figure demonstrates the evolution of Kaggle Competition types over time. \n\nWe can see that sometime in 2015, Community Competitions began to take over Feature Competition in number launched and still is continously growing.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport pandas as pd\nimport plotly.express as px\nimport numpy as np\nimport datetime\n\n# Read in dataset\ndf = pd.read_csv('/kaggle/input/meta-kaggle/Competitions.csv')\n\n# Print total competitions\nprint(f'Total competitions {len(df)}')\n\n# Print frequency table of competition types\nprint(df['HostSegmentTitle'].value_counts())\n\n# Convert the 'EnabledDate' column to datetime if it is not already\ndf['EnabledDate'] = pd.to_datetime(df['EnabledDate'])\n\n# Filter the DataFrame to include data from 2010 onwards\ndf_filtered = df[df['EnabledDate'].dt.year >= 2010]\n\n# Sort the filtered DataFrame by 'EnabledDate'\ndf_sorted = df_filtered.sort_values('EnabledDate')\n\n# Calculate the running total of occurrences for each 'HostSegmentTitle' over time\ndf_sorted['RunningCount'] = df_sorted.groupby('HostSegmentTitle').cumcount() + 1\n\n# Apply logarithmic scaling to the 'RunningCount' values\ndf_sorted['ScaledRunningCount'] = np.log1p(df_sorted['RunningCount'])\n\n# Create the plot using Plotly Express\nfig = px.line(df_sorted, x='EnabledDate', y='ScaledRunningCount', color='HostSegmentTitle',\n labels={'EnabledDate': 'Enabled Date', 'ScaledRunningCount': 'Scaled Running Count', 'HostSegmentTitle': 'Competition Type'},\n title='Running Total Count of Competitions Over Time (Log Scale)')\n\n\n# Add a clear sign indicating the launch of Kaggle in-Class\nfig.add_vline(x=datetime.datetime.strptime(\"2012-01-03\", \"%Y-%m-%d\").timestamp() * 1000, line_width=2, line_dash='dash', line_color='red',\n annotation_text='Kaggle in-Class Launch', annotation_position='top left')\n\n# Configure hover data to display the 'Title' and 'RunningCount' columns\nfig.update_traces(hovertemplate='Title: %{text}
Running Count: %{y}
Date: %{x}',\n text=df_sorted['Title'])\n\n# Show the plot\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T18:10:39.606965Z","iopub.execute_input":"2023-07-04T18:10:39.607933Z","iopub.status.idle":"2023-07-04T18:10:43.335925Z","shell.execute_reply.started":"2023-07-04T18:10:39.607884Z","shell.execute_reply":"2023-07-04T18:10:43.335052Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Research on the Competitions Effect upon Students\n\nThere is a body of evidence collected by the academic community that competitions have a positive effect in the classroom. Some of the benefits discovered include:\n\n* **More Student Engagement** Competitions generate motivation and excitement in students, by providing a goal-oriented environment. The increased engagement can lead to higher levels of participation and active learning in the classroom [11].\n* **Practical Problem-Solving Skills** Competitions provide opportunities for students to apply the knowledge and skills they have learned in a real-world context. This practical application can enhance their understanding and retention of concepts [12].\n* **Teamwork and Collaboration** Team-based competitions require students to work in a collaborative manner, which promotes effective communication and cooperation [13].\n* **Increased Knowledge Exploration** Competitions can introduce new topics, technologies, or approaches beyond the scope of typical classroom lectures. For students, not only can this expand their knowledge base but also encourage some further explorations they might not otherwise have been inticed to do [12].\n\nMoreover, there has been research done using the *Kaggle platform* itself. Polak and Cooke [14] utilized Kaggle inClass competitions and found that students that performed in a regression competition on Melbourne housing data [15] had higher total exam scores and increased engagement in the classroom. In addition, there has also been research done by Cardoso et al. that has found that Jupyter Notebooks, which are heavily used on the Kaggle platform, are an effective way to teach STEM, which is notoriously difficult and complex to explain, to students [16]. \n\n
\"Boxplot\n

From the Polak and Cook paper - the performance for regression question relative to total exam score for students who did and did not do the regression data competition in Kaggle's Statistical Thinking Competition.

\n
","metadata":{}},{"cell_type":"markdown","source":"# Educators are using Kaggle Across the Globe\n\nKaggle has provided a dataset in this report, from this, using GPT-based Question and Answering methods, I have found that **470** unique universities are using Kaggle for data science and machine learning competitions and **1092** competitions in total, meaning many are returning users to the Kaggle platform.These competitions have spanned from Computer Vision to NLP to Regression models. Educators have utilized both Kaggle managed datasets and custom datasets for students to work with.\n\nSome selected examples of these competitions include:\n* **Learning to Count (University of Barcelona, Spain)** - Competition to create a machine learning model that can count the number of instances of an object in an image [17]\n* **UNIFESP Chest CT Fatty Liver Competition (The Federal University of SĂŁo Paulo, Brazil)** - Competition to create an algorithm to detect fatty liver on Chest CT Scans [18]\n* **DAI_5004_01_AMLDL_Competition (Sungkyunkwan University, South Korea)** - Competition to classify chest X-rays with bacterial, viral, or no pneumonia [19].\n\nDisplayed below is a global map pinpointing the geographic locations and their competitions.","metadata":{}},{"cell_type":"code","source":"import folium\n\n# Read CSV\ndf = pd.read_csv(\"/kaggle/input/binder-datasets/kaggle-ai-report/Competitions.csv\")\n\nprint(f\"Unique number of universities using Kaggle: {len(df['UniversityName'].unique())}\")\nprint(f\"Number of university competitions: {len(df['UniversityName'])}\")\n\n# Create a map centered at (0, 0) to show the whole world\nm = folium.Map(location=[0, 0], zoom_start=2)\n\n# Iterate over the DataFrame rows and add markers to the map\nfor index, row in df.iterrows():\n location = [row['Latitude'], row['Longitude']]\n title = row['Title']\n host_segment = row['HostSegmentTitle']\n popup_text = f\"{title}
{host_segment}
{row['UniversityName']}\"\n\n # Choose marker color based on host_segment\n if host_segment == 'Community':\n icon_color = 'green'\n elif host_segment == 'Featured':\n icon_color = 'red'\n elif host_segment == 'Research':\n icon_color = 'blue'\n elif host_segment == 'Recruitment':\n icon_color = 'purple'\n elif host_segment == 'Getting Started':\n icon_color = 'darkred'\n elif host_segment == 'Prospect':\n icon_color = 'pink'\n else:\n icon_color = 'beige'\n\n # Create a marker with the specified color\n marker = folium.Marker(location=location, popup=popup_text, icon=folium.Icon(color=icon_color))\n marker.add_to(m)\n\n# Add a custom control button to reset the zoom\nreset_zoom_button = \"\"\"\n \n \n \"\"\"\n\nm.get_root().html.add_child(folium.Element(reset_zoom_button))\n\n# Display the map\nm","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-04T18:10:43.337668Z","iopub.execute_input":"2023-07-04T18:10:43.338202Z","iopub.status.idle":"2023-07-04T18:10:46.171629Z","shell.execute_reply.started":"2023-07-04T18:10:43.33817Z","shell.execute_reply":"2023-07-04T18:10:46.170487Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# The Future of Education?\n\nAccording the the United States Bureau of Labor Statistics, employment of computer and information research scientists is projected to grow **36%** from 2021 to 2031, which is much faster than the average of all job professions [20].\n\nOne of the challenges with such growth is ensuring that those that enter the workforce are educated and prepared to do their daily job activities. I believe that this is a major opportunity to explore these new educational offers provided by cloud-computing and the Kaggle platform.\n\nThe question for us as a community is how to best utilize these resources to our advantage - how can we create new engaging materials and exciting prospects for future students? This is where I believe the competitive and educational landscape of Kaggle will come into play. The user base is continously expanding and the technologies are improving. There is a body of evidence supporting that these new methods have a positive effect on students. New paths will be forged for educators who want to take non-traditional approachs to teaching to make a large impact.","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\n\n# Define the data\ncategories = ['Data Scientists', 'Computer and Information Research Scientists', 'Total, All Occupations']\nemployment_growth = [36, 21, 5.3]\n\n# Set the plot size\nplt.figure(figsize=(12, 6))\n\n# Create the bar plot\nax = sns.barplot(x=categories, y=employment_growth)\n\n# Add percentage labels on the bars\nfor i, v in enumerate(employment_growth):\n ax.text(i, v + 0.1, f'{v}%', ha='center', va='bottom', color='black', fontweight='bold')\n\n# Set the title and labels\nplt.title('Data Scientist Employment Growth - Percent Change in Employment, Projected 2021 to 2031')\n\n# Remove y-axis tick labels\nplt.yticks([])\n\n# Display the plot\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-07-04T18:29:44.639744Z","iopub.execute_input":"2023-07-04T18:29:44.640652Z","iopub.status.idle":"2023-07-04T18:29:44.865809Z","shell.execute_reply.started":"2023-07-04T18:29:44.640599Z","shell.execute_reply":"2023-07-04T18:29:44.864608Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# References\n\n[1] Kaggle. (2023). Kaggle: Your Home for Data Science. Retrieved June 28, 2023, from Kaggle: https://www.kaggle.com/\n\n[2] Lardinois, Frederic; Lynley, Matthew; Mannes, John (2017-03-08). \"Google is acquiring data science community Kaggle\". TechCrunch.\n\n[3] Kaggle. (2023, June 23). We've officially passed 14M community members on Kaggle!. LinkedIn. Retrieved from https://www.linkedin.com/posts/kaggle_weve-officially-passed-14m-community-members-activity-7078063761030291456-39tW?utm_source=share&utm_medium=member_desktop\n\n[4] Runceanu, Adrian & Runceanu, Mihaela-Ana. (2016). CHALLENGES IN TEACHING PROGRAMMING AND ALGORITHMS. 4120-4126. 10.21125/inted.2016.2003.\n\n[5] Cukierski, W. (2012). Titanic - Machine Learning from Disaster. Kaggle. Retrieved from https://kaggle.com/competitions/titanic\n\n[6] Howard, J. (2013). POWERDOT Awarded 500,000 and Announcing Heritage Health Prize 2.0. http://blog.kaggle.com/2013/06/03/powerdot-awarded-500000-and-announcing-heritage-health-prize-2-0/\n\n[7] Novy, A., CH1Mercedes, Drescher, C., Pfaundler, C., KOESIM, & Cukierski, W. (2017). Mercedes-Benz Greener Manufacturing. Kaggle. Retrieved from https://kaggle.com/competitions/mercedes-benz-greener-manufacturing\n\n[8] Hinton, G., and Dahl, G. (2012). “Deep Learning How I Did It: Merck 1st Place Interview,” No Free Hunch by Kaggle Team. Retrieve from: http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it-merck-1st-place-interview/.\n\n[9] Chen, T., & Guestrin, C. (2016). XGBoost. Retrieved from https://xgboost.ai/\n\n[10] Kaggle. @kaggle. (2022, June 22). \"As was the case until late last year, Community Competitions were then called \"Kaggle inClass\". The self-serve platform was used exclusively by educators looking to offer a ML competition as a class project. Here’s a banner from its original launch in 2012\". Retrieved from https://webcache.googleusercontent.com/search?q=cache:LCLkP3FxL0AJ:https://twitter.com/kaggle/status/1542658094856974337&cd=9&hl=en&ct=clnk&gl=us\n\n[11] Van Nuland, S.E., Roach, V.A., Wilson, T.D. and Belliveau, D.J. (2015), Head to head: The role of academic competition in undergraduate anatomical education. American Association of Anatomists, 8: 404-412. https://doi.org/10.1002/ase.1498\n\n[12] Cheung, S. Y., & Ng, K. Y. (2021). Application of the Educational Game to Enhance Student Learning. Frontiers in Education, 6. Retrieved from https://www.frontiersin.org/articles/10.3389/feduc.2021.623793\n\n[13] Johnson, D. W., & Johnson, R. T. (2009). An Educational Psychology Success Story: Social Interdependence Theory and Cooperative Learning. Educational Researcher, 38(5), 365–379. https://doi.org/10.3102/0013189X09339057\n\n[14] Julia Polak & Dianne Cook (2021) A Study on Student Performance, Engagement, and Experience With Kaggle InClass data Challenges, Journal of Statistics and Data Science Education, 29:1, 63-70, DOI: 10.1080/10691898.2021.1892554 \n\n[15] Melbourne Housing Market. (2016). Melbourne housing clearance data from Jan 2016. Retrieve from: https://www.kaggle.com/datasets/anthonypino/melbourne-housing-market\n\n[16] Cardoso, Alberto & LeitĂŁo, Joaquim & Teixeira, CĂ©sar. (2019). Using the Jupyter Notebook as a Tool to Support the Teaching and Learning Processes in Engineering Courses. Retrieved from: https://link.springer.com/chapter/10.1007/978-3-030-11935-5_22\n\n[17] Oriol, S. S. (2021). Learning to Count. Kaggle. Retrieved from https://kaggle.com/competitions/learning-to-count\n\n[18] Augusto Sarquis Serpa, F., & FelipeKitamura. (2022). UNIFESP Chest CT Fatty Liver Competition. Kaggle. Retrieved from https://kaggle.com/competitions/unifesp-fatty-liver\n\n[19] Yoon, J. (2021). DAI_5004_01_AMLDL_Competition. Kaggle. Retrieved from https://kaggle.com/competitions/dai-5004-01-amldl-competition\n\n[20] Bureau of Labor Statistics, U.S. Department of Labor, Occupational Outlook Handbook, Data Scientists,\nat https://www.bls.gov/ooh/math/data-scientists.htm (visited June 02, 2023).","metadata":{}}]}