{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# *Sample Submission*"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"df = pd.read_csv('../input/ashrae-energy-prediction/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.to_csv('submission.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# *Data Leakage Exercise*"},{"metadata":{},"cell_type":"markdown","source":"Most people find target leakage very tricky until they've thought about it for a long time.\n\nSo, before trying to think about leakage in the housing price example, we'll go through a few examples in other applications. \n\nThings will feel more familiar once you come back to a question about house prices."},{"metadata":{},"cell_type":"markdown","source":"reference: https://www.kaggle.com/alexisbcook/data-leakage"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set up code checking\nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.ml_intermediate.ex7 import *\nprint(\"Setup Complete\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### *1. The Data Science of Shoelaces*"},{"metadata":{},"cell_type":"markdown","source":"Nike has hired you as a data science consultant to help them save money on shoe materials. \n\nYour first assignment is to review a model one of their employees built to predict how many shoelaces they'll need each month. \n\nThe features going into the machine learning model include:\n- The current month (January, February, etc)\n- Advertising expenditures in the previous month\n- Various macroeconomic features (like the unemployment rate) as of the beginning of the current month\n- The amount of leather they ended up using in the current month\n\nThe results show the model is almost perfectly accurate if you include the feature about how much leather they used. \n\nBut it is only moderately accurate if you leave that feature out. \n\nYou realize this is because the amount of leather they use is a perfect indicator of how many shoes they produce, which in turn tells you how many shoelaces they need.\n\nDo you think the _leather used_ feature constitutes a source of data leakage? \n\nIf your answer is \"it depends,\" what does it depend on?\n\nAfter you have thought about your answer, check it against the solution below."},{"metadata":{"trusted":true},"cell_type":"code","source":"q_1.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### *2. Return of the Shoelaces*"},{"metadata":{},"cell_type":"markdown","source":"You have a new idea. \n\nYou could use the amount of leather Nike ordered (rather than the amount they actually used) leading up to a given month as a predictor in your shoelace model.\n\nDoes this change your answer about whether there is a leakage problem? \n\nIf you answer \"it depends,\" what does it depend on?"},{"metadata":{"trusted":true},"cell_type":"code","source":"q_2.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### *3. Getting Rich With Cryptocurrencies?*"},{"metadata":{},"cell_type":"markdown","source":"You saved Nike so much money that they gave you a bonus. \n\nCongratulations.\n\nYour friend, who is also a data scientist, says he has built a model that will let you turn your bonus into millions of dollars. \n\nSpecifically, his model predicts the price of a new cryptocurrency (like Bitcoin, but a newer one) one day ahead of the moment of prediction. \n\nHis plan is to purchase the cryptocurrency whenever the model says the price of the currency (in dollars) is about to go up.\n\nThe most important features in his model are:\n- Current price of the currency\n- Amount of the currency sold in the last 24 hours\n- Change in the currency price in the last 24 hours\n- Change in the currency price in the last 1 hour\n- Number of new tweets in the last 24 hours that mention the currency\n\nThe value of the cryptocurrency in dollars has fluctuated up and down by over \\$100 in the last year, and yet his model's average error is less than \\$1. \n\nHe says this is proof his model is accurate, and you should invest with him, buying the currency whenever the model says it is about to go up.\n\nIs he right? \n\nIf there is a problem with his model, what is it?"},{"metadata":{"trusted":true},"cell_type":"code","source":"q_3.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### *4. Preventing Infections*"},{"metadata":{},"cell_type":"markdown","source":"An agency that provides healthcare wants to predict which patients from a rare surgery are at risk of infection, so it can alert the nurses to be especially careful when following up with those patients.\n\nYou want to build a model. \n\nEach row in the modeling dataset will be a single patient who received the surgery, and the prediction target will be whether they got an infection.\n\nSome surgeons may do the procedure in a manner that raises or lowers the risk of infection. \n\nBut how can you best incorporate the surgeon information into the model?\n\nYou have a clever idea. \n1. Take all surgeries by each surgeon and calculate the infection rate among those surgeons.\n2. For each patient in the data, find out who the surgeon was and plug in that surgeon's average infection rate as a feature.\n\nDoes this pose any target leakage issues?\n\nDoes it pose any train-test contamination issues?"},{"metadata":{"trusted":true},"cell_type":"code","source":"q_4.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### *5. Housing Prices*"},{"metadata":{},"cell_type":"markdown","source":"You will build a model to predict housing prices.  \n\nThe model will be deployed on an ongoing basis, to predict the price of a new house when a description is added to a website.  \n\nHere are four features that could be used as predictors.\n1. Size of the house (in square meters)\n2. Average sales price of homes in the same neighborhood\n3. Latitude and longitude of the house\n4. Whether the house has a basement\n\nYou have historic data to train and validate the model.\n\nWhich of the features is most likely to be a source of leakage?"},{"metadata":{"trusted":true},"cell_type":"code","source":"q_5.hint()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"q_5.solution()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### *Conclusion*"},{"metadata":{},"cell_type":"markdown","source":"Leakage is a hard and subtle issue. \n\nYou should be proud if you picked up on the issues in these examples.\n\nThere is still a lot of room to build knowledge and experience. \n\nTry out 'ASHRAE - Great Energy Predictor III': https://www.kaggle.com/c/ashrae-energy-prediction \n\nAgain, Congratulations!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}