{"cells":[{"cell_type":"markdown","metadata":{"_cell_guid":"cb207525-2807-81af-adab-a2c27b95cab7"},"source":"Can we predict anomalies using the numerical feature density? It doesn't look like it..."},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"74e5dd47-d604-6354-7502-4afe7b4af08e"},"outputs":[],"source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport sys\nfrom collections import defaultdict\nimport random"},{"cell_type":"markdown","metadata":{"_cell_guid":"e4ab94c1-f12e-cc04-9a24-1d7c9aed4879"},"source":"Author = Eliot Andres"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"e45d27d0-27ff-6c42-0675-ddcb48931e76"},"outputs":[],"source":"sys.version"},{"cell_type":"markdown","metadata":{"_cell_guid":"427db241-5f74-b9c0-9775-05d765cb1a89"},"source":"**Loading the numerical dataset in chunks:**\n   "},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"9d1e6f25-b2f3-a30f-a1da-660e782fcfeb"},"outputs":[],"source":"def df_load(file, batch_size, skip = 1, usecolsCustom = None):\n    ''' Build dataframe by iterating over chunks. Option to skip chunks and\n        therefore read in less data. '''\n\n    reader = pd.read_csv(file, nrows=batch_size,\n                         dtype=np.float32, usecols=usecolsCustom)\n\n    '''df = pd.concat((chunk for i, chunk in enumerate(reader) if i % skip == 0))'''\n\n    return reader"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"91b6f12d-9cc6-8d96-68cd-05a708053b9f"},"outputs":[],"source":"input_path = '../input/'\ndf_num = df_load(input_path+'train_numeric.csv',\n                 batch_size=100000, skip=20, usecolsCustom=['Id', 'Response'])"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"2a393a97-c68d-909a-a24a-55d1351c8382"},"outputs":[],"source":"df_date = df_load(input_path+'train_date.csv',\n                 batch_size=100000, skip=20)"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"c1b8f01f-9cd7-a3b0-5694-6d836c3021de"},"outputs":[],"source":"df_date.head()"},{"cell_type":"markdown","metadata":{"_cell_guid":"8078436c-13fb-1f19-9efb-dd4b0aac215a"},"source":"**Create a set (for fast lookup) containing  parts with defects**"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"12740ec2-fb77-154c-3903-f3d911e6b45c"},"outputs":[],"source":"responsesTemp = []\nfor index, row in df_num.iterrows():\n    if bool(row['Response']):\n        responsesTemp.append(int(row['Id']))\nresponses = set(responsesTemp)"},{"cell_type":"markdown","metadata":{"_cell_guid":"1b8010d9-866b-9dec-b19b-d1134fceb518"},"source":"**Store the timestamps at each stations. Split in 2, parts with defects and parts without. Note that we don't know when the part got the defect**"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"1d2d812a-1e6f-44d4-e426-ff45f7f99e54"},"outputs":[],"source":"stations = defaultdict(list)\nstationsdefaults = defaultdict(list)\nfor index, row in df_date.iterrows():\n    current_station = ''\n    for column in df_date:\n        if(column != 'Id'):\n            if (current_station != column.split('_')[1]):\n                current_station = column.split('_')[1]\n                stations[current_station]\n                if not np.isnan(row[column]):\n                    if (row['Id'] in responses):\n                        stationsdefaults[current_station].append(row[column])\n                    else:\n                        stations[current_station].append(row[column])  "},{"cell_type":"markdown","metadata":{"_cell_guid":"5e775d1c-ff94-bef6-c0c0-d51b8c66487c"},"source":"**Plot the timestamps for each station. We use jittering(the random coefficient) to see things better**"},{"cell_type":"code","execution_count":null,"metadata":{"_cell_guid":"c37e93e3-dc90-7a40-02d8-d50bb178d001"},"outputs":[],"source":"for j in range(51):\n    values = [(random.random()*0.5 + 0.5) for i in range(len(stationsdefaults['S' + str(j)]))]\n    plt.plot(stationsdefaults['S' + str(j)], values, marker='*', ls='', markersize='1')\n\n    values = [random.random()*0.5 for i in range(len(stations['S' + str(j)]))]\n    plt.plot(stations['S' + str(j)], values, marker='*', ls='', markersize='1')\n    plt.title('Station S' + str(j))\n    \n    plt.show()\n   "}],"metadata":{"_change_revision":0,"_is_fork":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.2"}},"nbformat":4,"nbformat_minor":0}