## Overview For this assignment, we will be looking at edX online course data from Harvard and MIT to predict completion of an online course based on the activity of a student. We have provided a training dataset and test dataset that you will use to train and evaluate your model (described in detail in the Data tab). You will get more practice doing machine learning in the "real world" in the assignment this week. For many ML tasks, you will be given a dataset and your goal is to find the model that will do best in the future by using that dataset to train the model. Instead of risking deploying a bad model into production, a great test to verify that the model will work in the future is to evaluate it based on its accuracy on data it has never seen before. For this assignment, we will do just that! You will have access to a training set with labels and a test set without labels. Your task is to train a model that will do optimally in the future, which will be assessed by looking at your predictions on this test dataset you haven't seen the labels for. Kaggle is a good medium for simulating what it is like to deploy models in the real world! It assesses your model on these unseen examples and gives you an indication of how you will do by reporting your accuracy on a portion of the test set. This is seen as the "leaderboard" when you submit. The leaderboard is only meant to give you an indication of how your model might do in the future, but the final evaluation will be on data that has never been seen before or used to calculate leaderboard positions. This means you should take the leaderboard scores with a grain of salt since the final evaluation will be on a different dataset. The idea of being assessed on data you have never looked at before might seem a bit unnerving at first, but this is exactly the scenario that happens whenever an ML practitioner deploys a new model. Uncertainty is unavoidable in the world, so we need to use the ML best practice we have learned in the class so far as to assure us that our model will do well in the future. As a note, this assignment is NOT graded as a competition. The Kaggle leaderboard is just supposed to be a fun way for you to compare your model against your colleagues and the rankings are not used in grading. All that matters is how your model does in the end, not how it competes with everyone else's (more on grading in the Evaluation section). ## Usernames When you make a submission on Kaggle, by default it will use your username on the leaderboard. If you would like to use something else, you may change your "team name" in the Team tab which will change what shows up in the leaderboard. Regardless if you change the name or not, you must include your username and/or team name in programming assignment 5 on Gradescope so we can identify which submission is yours. User/team names must not be offensive and should be appropriate. If we are not able to find which submission on Kaggle is yours, you will not receive credit for what you submit on Kaggle. ## Group Work You will be allowed to form teams for this assignment! This assignment should be doable alone, but we thought this was a good opportunity to let you explore working on machine learning in a team if that interests you. We recommend grouping up with your peers since you will learn a lot more from the experience! You can form teams of up to 3 students. Kaggle lets you make submissions as a team and Gradescope allows you to turn in your code/report as a group as well. See Gradescope for more. When working with a group, you can fully collaborate on the programming portion (the report and the Kaggle competition), but you should do the concept portions individually like normal. ## Starter Code There is minimal starter code for this assignment [here](https://colab.research.google.com/drive/1_VcpxCBmOzqrHcqdCoKNuovcYqUh7F75?usp=sharing). Note that you will need to write everything based on what we have done in previous assignments. It is recommended to work off of this starter notebook as it has section headings and instructions for all necessary components of the programming report. ## Programming Report Requirements The programming portion of this assignment involves writing code to train a model that accurately predicts whether students received their certification in the course for future student data examples of the same format. You also need to turn in your notebook code on Gradescope (it could be several notebooks). Make sure they explain the process of what you are doing (use markdown cells), together with the prediction submissions file. Your notebook should clearly demonstrate the components listed below, and you should explain what you are doing and the interpretation of your results in Markdown (or Text) cells in your Jupyter notebook. The starter code has section headings for each key component. Here are several components we expect you to complete in your homework and should be described in markdown cells along with the code you used to complete them: * Your team name as it appears here on Kaggle. * Run at least 2 different types of classifiers (different methods: you can additionally run different versions of individual methods with different parameters and features) for prediction * Make a visualization that lets you compare the models you trained. Briefly discuss what you are inferring from this plot. * Discuss what features you used and whether you did some transformations on them. What features seemed important for your final model? * Explain what did you do to improve your initial predictions. Did it help? * Consider and discuss the ethical implications of using the model you trained. Imagine you were hired to work as a data scientist for an online education platform (exciting, right?). They want to use your model to predict which students they should tailor their course material towards so they can maximize their profits. Their idea is to use the model to help predict which student groups (i.e. from specific countries or certain educational backgrounds) would make them the most money by getting the most participants to complete their paid certificate program. Are there any ethical considerations we should think about before acting on that plan? The starter notebook includes section headings for each of the instructions above. ## Submitting For this assignment, you will use Kaggle to turn in your model predictions, but everything else (Notebook, programming, concept questions) should be submitted on Gradescope. This will differ from regular assignments since we will not submit code on EdStem. ## Logistics Please do not use the Discussion tab on Kaggle, but instead, use EdStem so there is only one place to look. If you use Kaggle's Kernels to develop your code, please make sure they are private so that you are not in violation of our academic misconduct policy. ## Coding Environment We recommend getting used to a coding environment outside of EdStem for this environment. We recommend three possible options: 1. Use Google Collaboratory (or Colab for short). Colab is Google's "Google Docs for Jupyter Notebooks". It works quite well and is very popular. You can use this starter notebook [here](https://colab.research.google.com/drive/1_VcpxCBmOzqrHcqdCoKNuovcYqUh7F75?usp=sharing). 2. Use a "kernel" on Kaggle (click the Code tab above). This will be another Jupyter Notebook view that is built by the Kaggle team. This is another great option since it's integrated in Kaggle, but make sure you don't share your notebook publicly as that would be a violation of our academic honesty policy. 3. Download Python and set up Jupyter Notebook on your computer locally. We recommend installing Python with [Anaconda](https://www.anaconda.com/download) and launching Jupyter Notebooks from there. ## Grading For information about how your solution will be evaluated and graded, see the Evaluation section. ## Due Date This Kaggle assignment for HW5 is due on **Thursday 5/16 11:59PM PST**. If you decide to use late days (up to two), you can submit up till Saturday 5/18 11:59PM PST. However, this Kaggle competition will be open until the end of the month, in case you want to play around with it more even after the homework is due.