{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Data Preprocessing Only Using Pandas and a Normal Notebook Without Accelerator\n\nI tried to preprocess the data **only with Pandas and without any other 'techniques' and accelerator**.<br/>A big weakness is the high memory usage if you do not pay attention on it. \n<br/>I'm a newbie and it's my first shared notebook. This code needs a lot less than the normal memory of Kaggle Notebooks provides.\n<br/>Please contact me, if you see a big misstake.\n\n<br/> You can use the preprocessed data files in the output.\n\n<br/>Computed on my local machine. On Kaggle need the code a lot more time (ca. 22 minutes).<br /> But the finished preprocessed files in the output directory.\n<br/>\n### If you like it, i will be very thanksfull for a comment or vote :-)","metadata":{}},{"cell_type":"code","source":"import warnings\nimport pandas as pd\n\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2023-10-12T19:32:18.309301Z","iopub.execute_input":"2023-10-12T19:32:18.309644Z","iopub.status.idle":"2023-10-12T19:32:18.776652Z","shell.execute_reply.started":"2023-10-12T19:32:18.309617Z","shell.execute_reply":"2023-10-12T19:32:18.775353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series = pd.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/train_series.parquet')","metadata":{"execution":{"iopub.status.busy":"2023-10-12T19:32:18.778907Z","iopub.execute_input":"2023-10-12T19:32:18.780196Z","iopub.status.idle":"2023-10-12T19:33:31.116894Z","shell.execute_reply.started":"2023-10-12T19:32:18.780162Z","shell.execute_reply":"2023-10-12T19:33:31.115449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def Info_Train_Series():\n    print(f\"Info:\")\n    print(train_series.info())\n    print(f\"Memory Usage:\\n\\n{train_series.memory_usage(deep=True)}\")\n    print(\"------------------------------\")\n    print(f\"Memory Usage Summary: {train_series.memory_usage(deep=True).sum() / (1024 * 1024)}\")","metadata":{"execution":{"iopub.status.busy":"2023-10-12T19:33:31.118364Z","iopub.execute_input":"2023-10-12T19:33:31.119442Z","iopub.status.idle":"2023-10-12T19:33:31.125749Z","shell.execute_reply.started":"2023-10-12T19:33:31.119399Z","shell.execute_reply":"2023-10-12T19:33:31.124477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def RangeSeries(column):\n    print(f\"Min: {train_series[column].min()}\")\n    print(f\"Max: {train_series[column].max()}\")\n    print(f\"Type: {train_series[column].dtype}\")\n    print(f\"NaN: {train_series[column].isna().sum()}\")","metadata":{"execution":{"iopub.status.busy":"2023-10-12T19:33:31.127312Z","iopub.execute_input":"2023-10-12T19:33:31.128759Z","iopub.status.idle":"2023-10-12T19:33:31.141809Z","shell.execute_reply.started":"2023-10-12T19:33:31.128715Z","shell.execute_reply":"2023-10-12T19:33:31.140830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Preprocess train_series dataframe\n\nFirst take an overview.","metadata":{}},{"cell_type":"code","source":"train_series.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-12T19:33:31.147238Z","iopub.execute_input":"2023-10-12T19:33:31.148156Z","iopub.status.idle":"2023-10-12T19:33:31.218168Z","shell.execute_reply.started":"2023-10-12T19:33:31.148108Z","shell.execute_reply":"2023-10-12T19:33:31.217101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_series.tail()","metadata":{"execution":{"iopub.status.busy":"2023-10-12T19:33:31.219820Z","iopub.execute_input":"2023-10-12T19:33:31.220506Z","iopub.status.idle":"2023-10-12T19:33:31.234576Z","shell.execute_reply.started":"2023-10-12T19:33:31.220468Z","shell.execute_reply":"2023-10-12T19:33:31.233217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We detect two **objects**.<br />\nObjects need a lot of memory.<br />How much, we can see below.","metadata":{}},{"cell_type":"code","source":"Info_Train_Series()","metadata":{"execution":{"iopub.status.busy":"2023-10-12T19:33:31.235851Z","iopub.execute_input":"2023-10-12T19:33:31.236604Z","iopub.status.idle":"2023-10-12T19:34:31.783422Z","shell.execute_reply.started":"2023-10-12T19:33:31.236577Z","shell.execute_reply":"2023-10-12T19:34:31.781969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before the preprocess the data frame using 19767 MB.","metadata":{}},{"cell_type":"markdown","source":"### Preprocess\n\nFirst we preprocess the **series_id**.<br />The column contains only unique id's for each accelerometer series, so we cast this type to **categorize**.<br />Later we use this to make the used memory even smaller.","metadata":{}},{"cell_type":"code","source":"train_series['series_id'] = train_series['series_id'].astype('category')\nInfo_Train_Series()","metadata":{"execution":{"iopub.status.busy":"2023-10-12T19:34:31.785055Z","iopub.execute_input":"2023-10-12T19:34:31.785467Z","iopub.status.idle":"2023-10-12T19:35:11.664388Z","shell.execute_reply.started":"2023-10-12T19:34:31.785432Z","shell.execute_reply":"2023-10-12T19:35:11.662790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Amazing, this one step encreased the memory down to 11592 MB, 8175 MB fewer then before, thats more then **40%**!","metadata":{}},{"cell_type":"code","source":"len(train_series['series_id'].unique())","metadata":{"execution":{"iopub.status.busy":"2023-10-12T19:35:11.665938Z","iopub.execute_input":"2023-10-12T19:35:11.666359Z","iopub.status.idle":"2023-10-12T19:35:12.378207Z","shell.execute_reply.started":"2023-10-12T19:35:11.666316Z","shell.execute_reply":"2023-10-12T19:35:12.377389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For a further step we make the unique ids even smaller, with an allocate count.<br />\nThis will be a smaller type and needs fewer memory. Because thes are only 277 different ids or now **categories**.<br />\nFirst we make a data frame with the ids and allocate smaller ids named **num_id**.","metadata":{}},{"cell_type":"code","source":"series_id_cat = pd.DataFrame()\nseries_id_cat['series_id'] = train_series.series_id\nseries_id_cat['num_id'] = train_series.series_id.cat.codes\nseries_id_cat = series_id_cat.drop_duplicates()\nseries_id_cat","metadata":{"execution":{"iopub.status.busy":"2023-10-12T19:35:12.379567Z","iopub.execute_input":"2023-10-12T19:35:12.380449Z","iopub.status.idle":"2023-10-12T19:35:20.868254Z","shell.execute_reply.started":"2023-10-12T19:35:12.380422Z","shell.execute_reply":"2023-10-12T19:35:20.866629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Because of the limited memory, we work in steps.<br />\nTo create this data frame we build at first a temporary data frame with the same columns.","metadata":{}},{"cell_type":"code","source":"train_series_temp = pd.DataFrame(columns=train_series.columns)\ntrain_series_temp","metadata":{"execution":{"iopub.status.busy":"2023-10-12T19:35:20.869946Z","iopub.execute_input":"2023-10-12T19:35:20.871381Z","iopub.status.idle":"2023-10-12T19:35:20.885420Z","shell.execute_reply.started":"2023-10-12T19:35:20.871346Z","shell.execute_reply":"2023-10-12T19:35:20.884054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we work in steps, simply we use the **series_id** as divider.<br />They are not constant parts, but enough to let the needed memory not too much encrease.<br/><br/> In this step we cast the second object, the **timestamp**, to a datetime type.<br />This will reduce the needed memory significant.","metadata":{}},{"cell_type":"code","source":"for series_id in series_id_cat['series_id']:\n    df_temp = train_series.loc[train_series['series_id'] == series_id]\n    df_temp['series_id'] = df_temp.series_id.cat.codes\n    df_temp['timestamp'] = pd.to_datetime(df_temp['timestamp'], format='ISO8601', utc=True)  \n    train_series_temp = train_series_temp._append(df_temp)\n\ntrain_series_temp","metadata":{"execution":{"iopub.status.busy":"2023-10-12T19:35:20.887716Z","iopub.execute_input":"2023-10-12T19:35:20.888412Z","iopub.status.idle":"2023-10-12T20:05:04.623338Z","shell.execute_reply.started":"2023-10-12T19:35:20.888376Z","shell.execute_reply":"2023-10-12T20:05:04.622215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we assign the temporary data frame to the original data frame and delete the temporary.","metadata":{}},{"cell_type":"code","source":"train_series = train_series_temp.copy()\ntrain_series.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:05:04.624924Z","iopub.execute_input":"2023-10-12T20:05:04.626450Z","iopub.status.idle":"2023-10-12T20:05:24.536539Z","shell.execute_reply.started":"2023-10-12T20:05:04.626413Z","shell.execute_reply":"2023-10-12T20:05:24.535573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train_series_temp","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:05:24.540809Z","iopub.execute_input":"2023-10-12T20:05:24.541175Z","iopub.status.idle":"2023-10-12T20:05:24.546545Z","shell.execute_reply.started":"2023-10-12T20:05:24.541148Z","shell.execute_reply":"2023-10-12T20:05:24.545498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check the 'new' dataframe.","metadata":{}},{"cell_type":"code","source":"Info_Train_Series()","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:05:24.547994Z","iopub.execute_input":"2023-10-12T20:05:24.548416Z","iopub.status.idle":"2023-10-12T20:06:22.321269Z","shell.execute_reply.started":"2023-10-12T20:05:24.548380Z","shell.execute_reply":"2023-10-12T20:06:22.319938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After that we can the 'new' objects in the temporary dataframe cast to appropriate types.","metadata":{}},{"cell_type":"code","source":"RangeSeries('step')","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:06:22.323113Z","iopub.execute_input":"2023-10-12T20:06:22.323443Z","iopub.status.idle":"2023-10-12T20:06:43.575583Z","shell.execute_reply.started":"2023-10-12T20:06:22.323416Z","shell.execute_reply":"2023-10-12T20:06:43.574794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We know that the range from **series_id** is between 0 and 277 that implements we can use a UInt16 as type.<br/>**step** can cast to an UInt32 because the maximum is > 4294967295 (maximum value of UInt32).","metadata":{}},{"cell_type":"code","source":"train_series['series_id'] = train_series['series_id'].astype('uint16')\ntrain_series['step'] = train_series['step'].astype('uint32')\nInfo_Train_Series()","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:06:43.576458Z","iopub.execute_input":"2023-10-12T20:06:43.576713Z","iopub.status.idle":"2023-10-12T20:06:53.381427Z","shell.execute_reply.started":"2023-10-12T20:06:43.576691Z","shell.execute_reply":"2023-10-12T20:06:53.380457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After that we have a memory usage of 3661 MB!<br/>But one unnecessary memory hogs is the Index. A simple step is save without index and read in again.","metadata":{}},{"cell_type":"code","source":"train_series.to_parquet('/kaggle/working/new_train_series.parquet', index=False)\ntrain_series = pd.read_parquet('/kaggle/working//new_train_series.parquet')\nInfo_Train_Series()","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:06:53.382924Z","iopub.execute_input":"2023-10-12T20:06:53.383243Z","iopub.status.idle":"2023-10-12T20:07:18.701503Z","shell.execute_reply.started":"2023-10-12T20:06:53.383218Z","shell.execute_reply":"2023-10-12T20:07:18.700300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Wow! This is amazing, we decreased the needed memory to 2685 MB. Thats 13.58 percent of the origin data frame!<br/>**84 %** fewer and 17082 MB!<br/><br/>Not only is it less, but now every work with this data frame will also be faster.","metadata":{}},{"cell_type":"markdown","source":"## Preprocess Train Events","metadata":{}},{"cell_type":"code","source":"def Info_Train_Events():\n    print(f\"Info:\")\n    print(train_events.info())\n    print(\"------------------------------\")\n    print(f\"Memory Usage:\\n\\n{train_events.memory_usage(deep=True)}\")\n    print(\"------------------------------\")\n    print(f\"Memory Usage Summary: {train_events.memory_usage(deep=True).sum() / (1024 * 1024)}\")","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:18.702804Z","iopub.execute_input":"2023-10-12T20:07:18.703740Z","iopub.status.idle":"2023-10-12T20:07:18.709032Z","shell.execute_reply.started":"2023-10-12T20:07:18.703713Z","shell.execute_reply":"2023-10-12T20:07:18.708062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def RangeEvent(column):\n    print(f\"Min: {train_events[column].min()}\")\n    print(f\"Max: {train_events[column].max()}\")\n    print(f\"Type: {train_events[column].dtype}\")\n    print(f\"NaN: {train_events[column].isna().sum()}\")","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:18.710361Z","iopub.execute_input":"2023-10-12T20:07:18.710849Z","iopub.status.idle":"2023-10-12T20:07:18.723135Z","shell.execute_reply.started":"2023-10-12T20:07:18.710823Z","shell.execute_reply":"2023-10-12T20:07:18.721902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events = pd.read_csv('/kaggle/input/child-mind-institute-detect-sleep-states/train_events.csv')\ntrain_events.head","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:18.724394Z","iopub.execute_input":"2023-10-12T20:07:18.725030Z","iopub.status.idle":"2023-10-12T20:07:18.774925Z","shell.execute_reply.started":"2023-10-12T20:07:18.725004Z","shell.execute_reply":"2023-10-12T20:07:18.773830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's take an overview.","metadata":{}},{"cell_type":"code","source":"Info_Train_Events()","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:18.776268Z","iopub.execute_input":"2023-10-12T20:07:18.776540Z","iopub.status.idle":"2023-10-12T20:07:18.802973Z","shell.execute_reply.started":"2023-10-12T20:07:18.776517Z","shell.execute_reply":"2023-10-12T20:07:18.801963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have three objects and big memory needed types.<br/> It is not very much needed Memory, but let's look what we can optimize.","metadata":{}},{"cell_type":"markdown","source":"First we change the **series_id** to the identically assigned numbers from the **train_series** data frame. ","metadata":{}},{"cell_type":"code","source":"RangeEvent('series_id')","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:18.804341Z","iopub.execute_input":"2023-10-12T20:07:18.804627Z","iopub.status.idle":"2023-10-12T20:07:18.814872Z","shell.execute_reply.started":"2023-10-12T20:07:18.804601Z","shell.execute_reply":"2023-10-12T20:07:18.813789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events = train_events.replace(dict(zip(series_id_cat['series_id'], series_id_cat['num_id'])))\ntrain_events['series_id'] = train_events['series_id'].astype('uint16')\ntrain_events.head","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:18.816373Z","iopub.execute_input":"2023-10-12T20:07:18.816846Z","iopub.status.idle":"2023-10-12T20:07:19.615040Z","shell.execute_reply.started":"2023-10-12T20:07:18.816820Z","shell.execute_reply":"2023-10-12T20:07:19.613760Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Info_Train_Events()","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:19.616656Z","iopub.execute_input":"2023-10-12T20:07:19.616920Z","iopub.status.idle":"2023-10-12T20:07:19.639761Z","shell.execute_reply.started":"2023-10-12T20:07:19.616898Z","shell.execute_reply":"2023-10-12T20:07:19.638728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok, almost 1 MB fewer. That's more than **30 %**!<br/>Let's look about the NaN, they are in the step and/or timestamp columns.<br/>Without this values we can not work. We must drop this rows.","metadata":{}},{"cell_type":"code","source":"train_events = train_events.dropna()\ntrain_events.head(10000)","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:19.641271Z","iopub.execute_input":"2023-10-12T20:07:19.642296Z","iopub.status.idle":"2023-10-12T20:07:19.663125Z","shell.execute_reply.started":"2023-10-12T20:07:19.642258Z","shell.execute_reply":"2023-10-12T20:07:19.662011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Info_Train_Events()","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:19.664415Z","iopub.execute_input":"2023-10-12T20:07:19.664675Z","iopub.status.idle":"2023-10-12T20:07:19.686506Z","shell.execute_reply.started":"2023-10-12T20:07:19.664654Z","shell.execute_reply":"2023-10-12T20:07:19.685140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"Wow, there was a lot of NaN's in this data frame.<br/>The data frame is almost 50 % smaller.<br/>Next we process the other types.","metadata":{}},{"cell_type":"markdown","source":"The **event** are only 'wakeup' or 'onset'. Simplify them.","metadata":{}},{"cell_type":"code","source":"train_events['event'] = train_events['event'].astype('category')\ntrain_events['event'] = train_events.event.cat.codes\ntrain_events.head","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:19.688551Z","iopub.execute_input":"2023-10-12T20:07:19.689028Z","iopub.status.idle":"2023-10-12T20:07:19.713467Z","shell.execute_reply.started":"2023-10-12T20:07:19.688986Z","shell.execute_reply":"2023-10-12T20:07:19.712339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"RangeEvent('event')","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:19.714689Z","iopub.execute_input":"2023-10-12T20:07:19.715135Z","iopub.status.idle":"2023-10-12T20:07:19.723035Z","shell.execute_reply.started":"2023-10-12T20:07:19.715094Z","shell.execute_reply":"2023-10-12T20:07:19.721925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next the **timestamp**.","metadata":{}},{"cell_type":"code","source":"train_events['timestamp'] = pd.to_datetime(train_events['timestamp'], format=\"ISO8601\", utc=True)","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:19.725031Z","iopub.execute_input":"2023-10-12T20:07:19.725495Z","iopub.status.idle":"2023-10-12T20:07:19.779618Z","shell.execute_reply.started":"2023-10-12T20:07:19.725457Z","shell.execute_reply":"2023-10-12T20:07:19.778436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The **night** seems to be using a large type. Let's take a smaller one.","metadata":{}},{"cell_type":"code","source":"RangeEvent('night')","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:19.780850Z","iopub.execute_input":"2023-10-12T20:07:19.781493Z","iopub.status.idle":"2023-10-12T20:07:19.790021Z","shell.execute_reply.started":"2023-10-12T20:07:19.781453Z","shell.execute_reply":"2023-10-12T20:07:19.788814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events['night'] = train_events['night'].astype('uint8')","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:19.794330Z","iopub.execute_input":"2023-10-12T20:07:19.796029Z","iopub.status.idle":"2023-10-12T20:07:19.809510Z","shell.execute_reply.started":"2023-10-12T20:07:19.795982Z","shell.execute_reply":"2023-10-12T20:07:19.808347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next the **step** is not a float, so change to an integer.","metadata":{}},{"cell_type":"code","source":"RangeEvent('step')","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:19.811685Z","iopub.execute_input":"2023-10-12T20:07:19.813403Z","iopub.status.idle":"2023-10-12T20:07:19.829796Z","shell.execute_reply.started":"2023-10-12T20:07:19.813364Z","shell.execute_reply":"2023-10-12T20:07:19.828446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events['step'] = train_events['step'].astype('uint32')\nInfo_Train_Events()","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:19.832034Z","iopub.execute_input":"2023-10-12T20:07:19.832893Z","iopub.status.idle":"2023-10-12T20:07:19.861410Z","shell.execute_reply.started":"2023-10-12T20:07:19.832850Z","shell.execute_reply":"2023-10-12T20:07:19.860177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We are almost **92.5 %** smaller then the origin!!!","metadata":{}},{"cell_type":"code","source":"train_events","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:19.863197Z","iopub.execute_input":"2023-10-12T20:07:19.864206Z","iopub.status.idle":"2023-10-12T20:07:19.883890Z","shell.execute_reply.started":"2023-10-12T20:07:19.864163Z","shell.execute_reply":"2023-10-12T20:07:19.882757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_events.to_csv('/kaggle/working/events_new.csv')","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:19.885358Z","iopub.execute_input":"2023-10-12T20:07:19.886407Z","iopub.status.idle":"2023-10-12T20:07:19.999175Z","shell.execute_reply.started":"2023-10-12T20:07:19.886366Z","shell.execute_reply":"2023-10-12T20:07:19.998048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After saving this data, we have done a lot of work... <br/>I hope you can follow my steps.<br/>\nIn further steps we need more validations and plots to understand the data.\n <br/>Contact me if you see an issue or you know an better way.","metadata":{}},{"cell_type":"markdown","source":"The last one step is to signal when the preprocess is ready. :-)","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport IPython.display as ipd\n\n\nbeep = np.sin(2*np.pi*400*np.arange(10000*2)/10000)\nipd.Audio(beep, rate=10000, autoplay=True)","metadata":{"execution":{"iopub.status.busy":"2023-10-12T20:07:20.000715Z","iopub.execute_input":"2023-10-12T20:07:20.001357Z","iopub.status.idle":"2023-10-12T20:07:20.083440Z","shell.execute_reply.started":"2023-10-12T20:07:20.001319Z","shell.execute_reply":"2023-10-12T20:07:20.082219Z"},"trusted":true},"execution_count":null,"outputs":[]}]}