{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":53666,"databundleVersionId":6589269,"sourceType":"competition"},{"sourceId":6507308,"sourceType":"datasetVersion","datasetId":3706667}],"dockerImageVersionId":30558,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!pip install -Uqq fastbook ","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T22:26:46.919049Z","iopub.execute_input":"2023-09-29T22:26:46.919392Z","iopub.status.idle":"2023-09-29T22:27:00.510577Z","shell.execute_reply.started":"2023-09-29T22:26:46.919362Z","shell.execute_reply":"2023-09-29T22:27:00.509555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Introduction\n\nThe “Zzzs” you catch each night are crucial for your overall health. Sleep affects everything from your development to cognitive functioning. Even so, research into sleep has proved challenging, due to the lack of naturalistic data capture alongside accurate annotation. If data science could help researchers better analyze wrist-worn accelerometer data for sleep monitoring, sleep experts could more easily conduct large-scale studies of sleep, thus improving the understanding of sleep's importance and function.\n\nCurrent approaches for annotating sleep data include sleep logs, which are the gold standard for detecting the onset of sleep. However, they are impractical for many participants to use reliably, and fail to capture the nuanced difference between heading to bed and falling asleep (or, conversely, waking up and getting out of bed). Heuristic-based software is another solution that attempts to identify sleep windows, though these rely on human-engineered features of sleep (i.e. arm angle) that vary across individuals and don’t accurately summarize the sleep windows that experts can visually detect from their data. With improved tools to analyze sleep data on a large scale, researchers can explore the relationship between sleep and mood/behavioral difficulties. This knowledge can lead to more targeted interventions and treatment strategies.\n\nCompetition host Child Mind Institute (CMI) transforms the lives of children and families struggling with mental health and learning disorders by giving them the help they need. CMI has become the leading independent nonprofit in children’s mental health by providing gold-standard evidence-based care, delivering educational resources to millions of families each year, training educators in underserved communities, and developing tomorrow’s breakthrough treatments.\n\nEstablished with a foundational grant from the Stavros Niarchos Foundation (SNF), the SNF Global Center for Child and Adolescent Mental Health at the Child Mind Institute works to accelerate global collaboration on under-researched areas of children’s mental health and expand worldwide access to culturally appropriate trainings, resources, and treatment. A major goal of the SNF Global Center is to expand innovations in clinical assessment and intervention, to include building, testing, and deploying new technologies to augment mental health care and research, including mobile apps, sensors, and analytical tools.","metadata":{}},{"cell_type":"markdown","source":"# Import Important Libraries","metadata":{}},{"cell_type":"code","source":"from fastbook import *\nfrom datetime import datetime\nfrom pandas.api.types import is_string_dtype, is_numeric_dtype, is_categorical_dtype\nfrom pandas.plotting import scatter_matrix\nfrom fastai.tabular.all import *\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.tree import DecisionTreeRegressor\nimport plotly.express as px\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nplt.style.use('ggplot') \nimport plotly.express as px","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-09-29T22:27:00.513301Z","iopub.execute_input":"2023-09-29T22:27:00.513751Z","iopub.status.idle":"2023-09-29T22:27:07.507254Z","shell.execute_reply.started":"2023-09-29T22:27:00.513709Z","shell.execute_reply":"2023-09-29T22:27:07.506463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading the Data and Basic EDA","metadata":{}},{"cell_type":"code","source":"# Load the dataset\ndf =  pd.read_parquet(\"/kaggle/input/zzzs-lightweight-training-dataset-target/Zzzs_train_multi.parquet\")\n#test =  pd.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet')\ntrain = df.copy()\n# Check first 5 columns\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:27:07.508497Z","iopub.execute_input":"2023-09-29T22:27:07.508937Z","iopub.status.idle":"2023-09-29T22:27:18.383289Z","shell.execute_reply.started":"2023-09-29T22:27:07.508910Z","shell.execute_reply":"2023-09-29T22:27:18.382551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Columns in the Dataset","metadata":{}},{"cell_type":"code","source":"# Check the columns in the Dataset\ndf.columns","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:27:18.385266Z","iopub.execute_input":"2023-09-29T22:27:18.385564Z","iopub.status.idle":"2023-09-29T22:27:18.392193Z","shell.execute_reply.started":"2023-09-29T22:27:18.385540Z","shell.execute_reply":"2023-09-29T22:27:18.390984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## The Datatype of the Various Columns","metadata":{}},{"cell_type":"code","source":"df.info()","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:27:18.393271Z","iopub.execute_input":"2023-09-29T22:27:18.393514Z","iopub.status.idle":"2023-09-29T22:27:18.416698Z","shell.execute_reply.started":"2023-09-29T22:27:18.393494Z","shell.execute_reply":"2023-09-29T22:27:18.415557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Check for nulls","metadata":{}},{"cell_type":"code","source":"# Number of nulls in each columns\ndf.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:27:18.418191Z","iopub.execute_input":"2023-09-29T22:27:18.418551Z","iopub.status.idle":"2023-09-29T22:27:20.166915Z","shell.execute_reply.started":"2023-09-29T22:27:18.418519Z","shell.execute_reply":"2023-09-29T22:27:20.165922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.describe()","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:27:20.168397Z","iopub.execute_input":"2023-09-29T22:27:20.168730Z","iopub.status.idle":"2023-09-29T22:27:22.004817Z","shell.execute_reply.started":"2023-09-29T22:27:20.168691Z","shell.execute_reply":"2023-09-29T22:27:22.003783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.hist()\nplt.suptitle('Distribution of the Variables');","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:27:22.005954Z","iopub.execute_input":"2023-09-29T22:27:22.006263Z","iopub.status.idle":"2023-09-29T22:27:24.250737Z","shell.execute_reply.started":"2023-09-29T22:27:22.006237Z","shell.execute_reply":"2023-09-29T22:27:24.249705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The target ('awake') is imbalanced","metadata":{}},{"cell_type":"code","source":"print(df.groupby('awake').size().sort_values())","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:27:24.252063Z","iopub.execute_input":"2023-09-29T22:27:24.252843Z","iopub.status.idle":"2023-09-29T22:27:24.587931Z","shell.execute_reply.started":"2023-09-29T22:27:24.252806Z","shell.execute_reply":"2023-09-29T22:27:24.587226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#scatter_matrix(df);\n","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:27:24.591947Z","iopub.execute_input":"2023-09-29T22:27:24.592650Z","iopub.status.idle":"2023-09-29T22:27:24.595973Z","shell.execute_reply.started":"2023-09-29T22:27:24.592620Z","shell.execute_reply":"2023-09-29T22:27:24.595206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"None of the variables has a clear relationship.","metadata":{}},{"cell_type":"code","source":"df.boxplot(column='step', by='awake')\nplt.title('Boxplot of Step by Awake')\n\n# Adjust the layout and display the plot\nplt.tight_layout()\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:27:24.596964Z","iopub.execute_input":"2023-09-29T22:27:24.597410Z","iopub.status.idle":"2023-09-29T22:27:42.090572Z","shell.execute_reply.started":"2023-09-29T22:27:24.597386Z","shell.execute_reply":"2023-09-29T22:27:42.089897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Basic Feature Engineering","metadata":{}},{"cell_type":"markdown","source":"We create new columns to detect the time and date features. Create a time_of_day feature to categorize the time period (morning, afternoon and night).\nWe can go on to create the other columns we need. For example, the hour of the day will be a better predictor of sleep tham just time of the day. \n","metadata":{}},{"cell_type":"code","source":"# dependent variables\ndep_var = 'awake'","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:27:42.091753Z","iopub.execute_input":"2023-09-29T22:27:42.092245Z","iopub.status.idle":"2023-09-29T22:27:42.095586Z","shell.execute_reply.started":"2023-09-29T22:27:42.092219Z","shell.execute_reply":"2023-09-29T22:27:42.094632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['timestamp'] = pd.to_datetime(df['timestamp'], format=\"%Y-%m-%dT%H:%M:%S%z\", errors='coerce', utc=True)\n\n\n# Function to categorize the hour into time periods\ndef categorize_time(datetime_obj):\n    hour = datetime_obj.hour\n    if 6 <= hour < 12:\n        return \"Morning\"\n    elif 12 <= hour < 18:\n        return \"Afternoon\"\n    else:\n        return \"Night\"\n\n# Apply the categorization function to the datetime column\ndf['time_of_day'] = df['timestamp'].apply(categorize_time)\n\n# Creating time series column with fastbook\ndf = add_datepart(df, 'timestamp')\n","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:27:42.097138Z","iopub.execute_input":"2023-09-29T22:27:42.097522Z","iopub.status.idle":"2023-09-29T22:29:54.557182Z","shell.execute_reply.started":"2023-09-29T22:27:42.097488Z","shell.execute_reply":"2023-09-29T22:29:54.556106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pivot_table = df.pivot_table(index='time_of_day', columns='awake', aggfunc='size', fill_value=0)\npivot_table","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:29:54.558720Z","iopub.execute_input":"2023-09-29T22:29:54.559424Z","iopub.status.idle":"2023-09-29T22:29:56.404804Z","shell.execute_reply.started":"2023-09-29T22:29:54.559383Z","shell.execute_reply":"2023-09-29T22:29:56.403856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"grouped_data = df.groupby(['time_of_day', 'awake']).size().unstack(fill_value=0)\n\n# Calculate the total count of samples for each time of day\ntotal_counts = grouped_data.sum(axis=1)\n\n# Create a donut plot\nfig, ax = plt.subplots()\n\n# Plot the pie chart (outer ring)\nax.pie(total_counts, labels=total_counts.index, autopct='%1.1f%%', startangle=90,\n                    wedgeprops=dict(width=0.4, edgecolor='w'))\n\n\nax.set_title('Awake/Sleep Distribution Throughout the Day')\n\n# Equal aspect ratio ensures that pie is drawn as a circle\nax.axis('equal')\n\n# Display the plot\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:29:56.406154Z","iopub.execute_input":"2023-09-29T22:29:56.406554Z","iopub.status.idle":"2023-09-29T22:29:58.432519Z","shell.execute_reply.started":"2023-09-29T22:29:56.406528Z","shell.execute_reply":"2023-09-29T22:29:58.431307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see what our dataset looks like","metadata":{}},{"cell_type":"code","source":"df","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:29:58.434411Z","iopub.execute_input":"2023-09-29T22:29:58.434959Z","iopub.status.idle":"2023-09-29T22:29:58.483950Z","shell.execute_reply.started":"2023-09-29T22:29:58.434935Z","shell.execute_reply":"2023-09-29T22:29:58.482661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check the new columns we have created","metadata":{}},{"cell_type":"code","source":"' '.join(o for o in df.columns if o.startswith('time'))","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:29:58.486137Z","iopub.execute_input":"2023-09-29T22:29:58.487158Z","iopub.status.idle":"2023-09-29T22:29:58.493506Z","shell.execute_reply.started":"2023-09-29T22:29:58.487108Z","shell.execute_reply":"2023-09-29T22:29:58.492859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The column names are self explanatory.\n\nIn the future versions of the notebook, i will go on to make lag and rolling averages of the original features.\n","metadata":{}},{"cell_type":"markdown","source":"This is a baseline notebook so I won't be focusing on modelling for now. In future version, I go deep into the feature engineering and model selection. For now, I will borrow features and use the same models used in  [@zotovaa notebook](https://www.kaggle.com/code/zotovaa/cmi-gradient-boosting-insomnia-research).\n","metadata":{}},{"cell_type":"code","source":"def make_features(df):\n    # parse the timestamp and create an \"hour\" feature\n   # df['timestamp'] = pd.to_datetime(df['timestamp']).apply(lambda t: t.tz_localize(None))\n    #df[\"hour\"] = df[\"timestamp\"].dt.hour\n    \n    periods = 20\n    df[\"anglez\"] = abs(df[\"anglez\"])\n    df[\"anglez_diff\"] = df.groupby('series_id')['anglez'].diff(periods=periods).fillna(method=\"bfill\").astype('float16')\n    df[\"enmo_diff\"] = df.groupby('series_id')['enmo'].diff(periods=periods).fillna(method=\"bfill\").astype('float16')\n    df[\"anglez_rolling_mean\"] = df[\"anglez\"].rolling(periods,center=True).mean().fillna(method=\"bfill\").fillna(method=\"ffill\").astype('float16')\n    df[\"enmo_rolling_mean\"] = df[\"enmo\"].rolling(periods,center=True).mean().fillna(method=\"bfill\").fillna(method=\"ffill\").astype('float16')\n    df[\"anglez_rolling_max\"] = df[\"anglez\"].rolling(periods,center=True).max().fillna(method=\"bfill\").fillna(method=\"ffill\").astype('float16')\n    df[\"enmo_rolling_max\"] = df[\"enmo\"].rolling(periods,center=True).max().fillna(method=\"bfill\").fillna(method=\"ffill\").astype('float16')\n    df[\"anglez_rolling_std\"] = df[\"anglez\"].rolling(periods,center=True).std().fillna(method=\"bfill\").fillna(method=\"ffill\").astype('float16')\n    df[\"enmo_rolling_std\"] = df[\"enmo\"].rolling(periods,center=True).std().fillna(method=\"bfill\").fillna(method=\"ffill\").astype('float16')\n    df[\"anglez_diff_rolling_mean\"] = df[\"anglez_diff\"].rolling(periods,center=True).mean().fillna(method=\"bfill\").fillna(method=\"ffill\").astype('float16')\n    df[\"enmo_diff_rolling_mean\"] = df[\"enmo_diff\"].rolling(periods,center=True).mean().fillna(method=\"bfill\").fillna(method=\"ffill\").astype('float16')\n    df[\"anglez_diff_rolling_max\"] = df[\"anglez_diff\"].rolling(periods,center=True).max().fillna(method=\"bfill\").fillna(method=\"ffill\").astype('float16')\n    df[\"enmo_diff_rolling_max\"] = df[\"enmo_diff\"].rolling(periods,center=True).max().fillna(method=\"bfill\").fillna(method=\"ffill\").astype('float16')\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:29:58.494497Z","iopub.execute_input":"2023-09-29T22:29:58.495003Z","iopub.status.idle":"2023-09-29T22:29:58.507093Z","shell.execute_reply.started":"2023-09-29T22:29:58.494977Z","shell.execute_reply":"2023-09-29T22:29:58.505980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df  = make_features(df)\ndf.columns","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:29:58.508510Z","iopub.execute_input":"2023-09-29T22:29:58.509788Z","iopub.status.idle":"2023-09-29T22:30:11.419294Z","shell.execute_reply.started":"2023-09-29T22:29:58.509754Z","shell.execute_reply":"2023-09-29T22:30:11.418453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check the datatype of the new columns.","metadata":{}},{"cell_type":"code","source":"df.info()","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:11.420580Z","iopub.execute_input":"2023-09-29T22:30:11.420979Z","iopub.status.idle":"2023-09-29T22:30:11.432637Z","shell.execute_reply.started":"2023-09-29T22:30:11.420951Z","shell.execute_reply":"2023-09-29T22:30:11.431528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's load in the test data","metadata":{}},{"cell_type":"code","source":"test = pd.read_parquet('/kaggle/input/child-mind-institute-detect-sleep-states/test_series.parquet')\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:11.434080Z","iopub.execute_input":"2023-09-29T22:30:11.434565Z","iopub.status.idle":"2023-09-29T22:30:11.464437Z","shell.execute_reply.started":"2023-09-29T22:30:11.434534Z","shell.execute_reply":"2023-09-29T22:30:11.463815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = test.copy()","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:11.465851Z","iopub.execute_input":"2023-09-29T22:30:11.466225Z","iopub.status.idle":"2023-09-29T22:30:11.471026Z","shell.execute_reply.started":"2023-09-29T22:30:11.466191Z","shell.execute_reply":"2023-09-29T22:30:11.470032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.columns","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:11.472136Z","iopub.execute_input":"2023-09-29T22:30:11.472469Z","iopub.status.idle":"2023-09-29T22:30:11.484620Z","shell.execute_reply.started":"2023-09-29T22:30:11.472447Z","shell.execute_reply":"2023-09-29T22:30:11.483529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = make_features(test)\ntest['timestamp'] = pd.to_datetime(test['timestamp'], format=\"%Y-%m-%dT%H:%M:%S%z\", errors='coerce', utc=True)\ntest['time_of_day'] = test['timestamp'].apply(categorize_time)\ntest = add_datepart(test, 'timestamp')","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:11.485971Z","iopub.execute_input":"2023-09-29T22:30:11.486998Z","iopub.status.idle":"2023-09-29T22:30:11.523433Z","shell.execute_reply.started":"2023-09-29T22:30:11.486964Z","shell.execute_reply":"2023-09-29T22:30:11.522753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f' The number of columns in the train dataset is {len(df.columns)} while that of the test dataset is {len(test.columns)}')","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:11.525106Z","iopub.execute_input":"2023-09-29T22:30:11.525868Z","iopub.status.idle":"2023-09-29T22:30:11.531718Z","shell.execute_reply.started":"2023-09-29T22:30:11.525831Z","shell.execute_reply":"2023-09-29T22:30:11.530732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Now we want to ensure that (since this is time series) our validation and test.\n\nFirst we start by checking the length of both the train and test test.","metadata":{}},{"cell_type":"code","source":"print(f'The number of rows in the train dataset is {len(df)} while that of the test dataset is {len(test)}')","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:11.533142Z","iopub.execute_input":"2023-09-29T22:30:11.533489Z","iopub.status.idle":"2023-09-29T22:30:11.543142Z","shell.execute_reply.started":"2023-09-29T22:30:11.533458Z","shell.execute_reply":"2023-09-29T22:30:11.542273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df['timestamp'].min(), test_df['timestamp'].max()\n","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:11.544144Z","iopub.execute_input":"2023-09-29T22:30:11.544808Z","iopub.status.idle":"2023-09-29T22:30:11.556866Z","shell.execute_reply.started":"2023-09-29T22:30:11.544773Z","shell.execute_reply":"2023-09-29T22:30:11.555838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's find that particular index","metadata":{}},{"cell_type":"code","source":"idx = train['timestamp'][(train['timestamp'] == train['timestamp'][len(test_df)])].index.max()\nidx","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:11.561378Z","iopub.execute_input":"2023-09-29T22:30:11.561798Z","iopub.status.idle":"2023-09-29T22:30:12.859039Z","shell.execute_reply.started":"2023-09-29T22:30:11.561772Z","shell.execute_reply":"2023-09-29T22:30:12.858060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"splits = (L(range(idx, len(train))),L(range(idx)))\nsplits","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:12.860171Z","iopub.execute_input":"2023-09-29T22:30:12.860469Z","iopub.status.idle":"2023-09-29T22:30:13.537572Z","shell.execute_reply.started":"2023-09-29T22:30:12.860445Z","shell.execute_reply":"2023-09-29T22:30:13.536539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.options.mode.chained_assignment=None","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:13.538580Z","iopub.execute_input":"2023-09-29T22:30:13.538862Z","iopub.status.idle":"2023-09-29T22:30:13.543093Z","shell.execute_reply.started":"2023-09-29T22:30:13.538841Z","shell.execute_reply":"2023-09-29T22:30:13.542008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Split the continuous and categorical variables using fastbook. We also want to normalize the continuous features and encode the categorical features.","metadata":{}},{"cell_type":"code","source":"cont,cat = cont_cat_split(df, 1, dep_var=dep_var)\n\nprocs = [Normalize, Categorify]\n","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:13.544054Z","iopub.execute_input":"2023-09-29T22:30:13.544753Z","iopub.status.idle":"2023-09-29T22:30:14.224477Z","shell.execute_reply.started":"2023-09-29T22:30:13.544724Z","shell.execute_reply":"2023-09-29T22:30:14.223480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"to = TabularPandas(df, procs, cat, cont, dep_var, y_block=CategoryBlock(),\n                   splits=splits, inplace=True, reduce_memory=True)","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:14.225529Z","iopub.execute_input":"2023-09-29T22:30:14.225836Z","iopub.status.idle":"2023-09-29T22:30:35.288376Z","shell.execute_reply.started":"2023-09-29T22:30:14.225812Z","shell.execute_reply":"2023-09-29T22:30:35.287541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(to.train),len(to.valid)","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:35.289601Z","iopub.execute_input":"2023-09-29T22:30:35.290014Z","iopub.status.idle":"2023-09-29T22:30:38.935883Z","shell.execute_reply.started":"2023-09-29T22:30:35.289979Z","shell.execute_reply":"2023-09-29T22:30:38.934931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train,y_train = to.train.xs,to.train.y\nvalid_x,valid_y = to.valid.xs,to.valid.y","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:38.937089Z","iopub.execute_input":"2023-09-29T22:30:38.937365Z","iopub.status.idle":"2023-09-29T22:30:46.774212Z","shell.execute_reply.started":"2023-09-29T22:30:38.937341Z","shell.execute_reply":"2023-09-29T22:30:46.773065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import xgboost as xgb\nfrom sklearn.experimental import enable_halving_search_cv \nfrom sklearn.model_selection import (HalvingRandomSearchCV, KFold)\nfrom scipy.stats import randint","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:46.775800Z","iopub.execute_input":"2023-09-29T22:30:46.776636Z","iopub.status.idle":"2023-09-29T22:30:46.944308Z","shell.execute_reply.started":"2023-09-29T22:30:46.776607Z","shell.execute_reply":"2023-09-29T22:30:46.943499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cv = KFold(n_splits=3, shuffle=True, random_state=42)\n\nclf = xgb.XGBClassifier(random_state=42)\n\ncv = KFold(n_splits=3, shuffle=True, random_state=42)\nparam_distributions = {\"max_depth\": [3, 6, 18],\n                       'learning_rate': [0.01, 0.1, 0.3],\n                       \"min_samples_split\": randint(2, 11)}\nsearch = HalvingRandomSearchCV(clf, param_distributions,\n                               resource='n_estimators',\n                               cv=cv, \n                               scoring='neg_mean_absolute_error',\n                               max_resources=10,\n                               random_state=42,\n                              verbose=2).fit(X_train,y_train)\nsearch.best_params_  ","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:30:46.945441Z","iopub.execute_input":"2023-09-29T22:30:46.945816Z","iopub.status.idle":"2023-09-29T22:34:55.547304Z","shell.execute_reply.started":"2023-09-29T22:30:46.945792Z","shell.execute_reply":"2023-09-29T22:34:55.546257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"search.best_score_","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:34:55.548611Z","iopub.execute_input":"2023-09-29T22:34:55.549291Z","iopub.status.idle":"2023-09-29T22:34:55.555551Z","shell.execute_reply.started":"2023-09-29T22:34:55.549266Z","shell.execute_reply":"2023-09-29T22:34:55.554551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = xgb.XGBClassifier(n_estimators = 9, max_depth=3, learning_rate=0.01, min_samples_split = 2)","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:34:55.556922Z","iopub.execute_input":"2023-09-29T22:34:55.557434Z","iopub.status.idle":"2023-09-29T22:34:55.566735Z","shell.execute_reply.started":"2023-09-29T22:34:55.557408Z","shell.execute_reply":"2023-09-29T22:34:55.565713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.fit(X_train, y_train,\n        eval_set=[(X_train, y_train), (valid_x,valid_y)],\n        verbose=100)","metadata":{"execution":{"iopub.status.busy":"2023-09-29T22:34:55.568278Z","iopub.execute_input":"2023-09-29T22:34:55.569082Z","iopub.status.idle":"2023-09-29T22:35:29.049756Z","shell.execute_reply.started":"2023-09-29T22:34:55.569051Z","shell.execute_reply":"2023-09-29T22:35:29.048795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}