{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Notebook description\n\nIn this notebook you will find:\n* Research of features from https://www.kaggle.com/code/smnuruzzaman/feature-engineering-and-ensemble\n* Removing redundant features\n* Creating and testing lag features\n* Creating and testing features obtained by arithmetic operations from the best ones in the previous step (aka bruteforce FE)\n\nIn all steps I will use 5-fold CV with lightgbm to get feature importance.","metadata":{}},{"cell_type":"markdown","source":"## Load data","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nfrom itertools import groupby\nfrom sklearn.model_selection import KFold, train_test_split\nfrom sklearn.metrics import f1_score, accuracy_score\nfrom pandas.api.types import is_datetime64_ns_dtype\n\nimport lightgbm as lgb\nfrom joblib import Parallel, delayed\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport gc\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:44:04.362440Z","iopub.execute_input":"2023-10-21T20:44:04.362828Z","iopub.status.idle":"2023-10-21T20:44:08.454072Z","shell.execute_reply.started":"2023-10-21T20:44:04.362776Z","shell.execute_reply":"2023-10-21T20:44:08.452843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TRAIN_PATH = '/kaggle/input/zzzs-lightweight-training-dataset-target/Zzzs_train.parquet'","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:44:08.456275Z","iopub.execute_input":"2023-10-21T20:44:08.456928Z","iopub.status.idle":"2023-10-21T20:44:08.461761Z","shell.execute_reply.started":"2023-10-21T20:44:08.456896Z","shell.execute_reply":"2023-10-21T20:44:08.460400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"series_id  = pd.read_parquet(TRAIN_PATH, columns=['series_id'])\nseries_id = series_id['series_id'].unique()\n\nweird_series = ['31011ade7c0a', 'a596ad0b82aa']\nseries_id = [s for s in series_id if s not in weird_series]","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:44:08.463342Z","iopub.execute_input":"2023-10-21T20:44:08.463924Z","iopub.status.idle":"2023-10-21T20:44:11.219754Z","shell.execute_reply.started":"2023-10-21T20:44:08.463894Z","shell.execute_reply":"2023-10-21T20:44:11.218667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Selection\n## Basic Features","metadata":{}},{"cell_type":"markdown","source":"Mostly features from https://www.kaggle.com/code/smnuruzzaman/feature-engineering-and-ensemble. \n\nI removed the **median** as they are very close to the means, you can try with them.","metadata":{}},{"cell_type":"code","source":"def basic_fe(df, periods):\n    \n    df['series_id'] = df['series_id'].astype('category')\n    df['timestamp'] = pd.to_datetime(df['timestamp']).apply(lambda t: t.tz_localize(None))\n    df['hour'] = df[\"timestamp\"].dt.hour\n    \n    df.sort_values(['timestamp'], inplace=True)\n    df.set_index('timestamp', inplace=True)\n    \n    df['lids'] = np.maximum(0., df['enmo'] - 0.02)\n    df['lids'] = df['lids'].rolling(f'{120*5}s', center=True, min_periods=1).agg('sum')\n    df['lids'] = 100 / (df['lids'] + 1)\n    df['lids'] = df['lids'].rolling(f'{360*5}s', center=True, min_periods=1).agg('mean').astype(np.float16)\n    \n    df[\"enmo\"] = (df[\"enmo\"]*1000).astype(np.int16)\n    df[\"anglez\"] = df[\"anglez\"].astype(np.int16)\n    df[\"anglezdiffabs\"] = df[\"anglez\"].diff().abs().astype(np.float16)\n    \n    for col in ['enmo', 'anglez', 'anglezdiffabs']:\n        for p in periods:\n            rol_args = {'window':f'{p+5}s', 'min_periods':10, 'center':True}\n            \n            for agg in ['mean', 'max', 'min']:\n                df[f'{col}_{p}_{agg}'] = df[col].rolling(**rol_args).agg(agg).astype(np.float16).values\n                gc.collect()\n            # There was an overflow error when trying to cast to float16, so I decided to leave it as float32\n            for agg in ['var']:\n                df[f'{col}_{p}_{agg}'] = df[col].rolling(**rol_args).agg(agg).astype(np.float32).values\n                gc.collect()\n\n            \n            df[f'{col}_{p}_amplit'] = df[f'{col}_{p}_max'] - df[f'{col}_{p}_min']\n            df[f'{col}_diff_{p}_max'] = df[f'{col}_{p}_max'].diff().abs().rolling(**rol_args).max().astype(np.float16)\n            df[f'{col}_diff_{p}_mean'] = df[f'{col}_{p}_min'].diff().abs().rolling(**rol_args).mean().astype(np.float16)\n            gc.collect()\n \n    df.reset_index(inplace=True)\n    df.dropna(inplace=True)\n    gc.collect()\n\n    return df\n","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:44:11.222000Z","iopub.execute_input":"2023-10-21T20:44:11.222354Z","iopub.status.idle":"2023-10-21T20:44:11.241615Z","shell.execute_reply.started":"2023-10-21T20:44:11.222326Z","shell.execute_reply":"2023-10-21T20:44:11.240170Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def feat_eng_by_id(idx, periods):\n    \n    from warnings import simplefilter \n    simplefilter(action=\"ignore\", category=pd.errors.PerformanceWarning)\n    \n    df = pd.read_parquet(TRAIN_PATH, filters=[('series_id','=',idx)])\n    df['awake'] = df['awake'].astype(np.int8)\n    df = basic_fe(df, periods)\n    \n    return df","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:44:11.242755Z","iopub.execute_input":"2023-10-21T20:44:11.243753Z","iopub.status.idle":"2023-10-21T20:44:11.276394Z","shell.execute_reply.started":"2023-10-21T20:44:11.243720Z","shell.execute_reply":"2023-10-21T20:44:11.275025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"periods = [60, 300, 600, 3600] ","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:44:11.278204Z","iopub.execute_input":"2023-10-21T20:44:11.278599Z","iopub.status.idle":"2023-10-21T20:44:11.291618Z","shell.execute_reply.started":"2023-10-21T20:44:11.278568Z","shell.execute_reply":"2023-10-21T20:44:11.290349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\ndf = Parallel(n_jobs=-2)(delayed(feat_eng_by_id)(i, periods) for i in series_id)\ndf = pd.concat(df, ignore_index=True)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:44:11.293133Z","iopub.execute_input":"2023-10-21T20:44:11.294146Z","iopub.status.idle":"2023-10-21T20:47:53.073069Z","shell.execute_reply.started":"2023-10-21T20:44:11.294096Z","shell.execute_reply":"2023-10-21T20:47:53.072003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Correlation heatmap between all features would take too long to compute, so let's compute it for the feature sets corresponding to each time window.","metadata":{}},{"cell_type":"code","source":"len(df.columns)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:47:53.085742Z","iopub.execute_input":"2023-10-21T20:47:53.086156Z","iopub.status.idle":"2023-10-21T20:47:53.097611Z","shell.execute_reply.started":"2023-10-21T20:47:53.086121Z","shell.execute_reply":"2023-10-21T20:47:53.096270Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(df.filter(like=f'_{periods[0]}_').columns)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:47:53.102033Z","iopub.execute_input":"2023-10-21T20:47:53.102387Z","iopub.status.idle":"2023-10-21T20:47:55.299429Z","shell.execute_reply.started":"2023-10-21T20:47:53.102347Z","shell.execute_reply":"2023-10-21T20:47:55.298406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features = [col for col in df.columns if col not in ['timestamp', 'series_id', 'step', 'awake']]","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:47:55.300530Z","iopub.execute_input":"2023-10-21T20:47:55.300831Z","iopub.status.idle":"2023-10-21T20:47:55.305559Z","shell.execute_reply.started":"2023-10-21T20:47:55.300789Z","shell.execute_reply":"2023-10-21T20:47:55.304557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for p in periods:\n    plt.figure(figsize=(12,12))\n    plt.title(f\"Correlation heatmap for features with period {p}[s]\")\n    sns.heatmap(df[features].filter(like=f'_{p}_').corr(),\n    vmin=-1, vmax=1, center=0,\n    cmap=sns.color_palette(\"flare\", as_cmap=True),\n    square=True)\n    plt.show() ","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:47:55.306656Z","iopub.execute_input":"2023-10-21T20:47:55.307495Z","iopub.status.idle":"2023-10-21T20:49:57.439219Z","shell.execute_reply.started":"2023-10-21T20:47:55.307465Z","shell.execute_reply":"2023-10-21T20:49:57.438050Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Insights:\n1. **enmo\\_{p}\\_max** and **enmo\\_{p}\\_amplit** are highly correlated for each aggregation window, which is logical since enmo is normally fluctuating around zero, with a limited number of sharp spikes. It's worth dropping one of them.\n2. **anglezdiffabs\\_{p}\\_max** and **anglezdiffabs\\_{p}\\_amplit** are highly correlated for each aggregation window.\n3. **anglezdiffabs\\_3600\\_min** and **anglezdiffabs\\_600\\_min** are just columns of zeros (see below), useless feature aggregation for large periods. \n4. **anglezdiffabs\\_diff\\_3600_mean** and **anglezdiffabs\\_diff\\_600\\_mean** - same issue.\n5. **enmo\\_3600\\_min** and **enmo\\_600\\_min** values are also all close to zero and make no sense.","metadata":{}},{"cell_type":"code","source":"features_to_drop = ['anglezdiffabs_3600_min', 'anglezdiffabs_600_min', 'anglezdiffabs_diff_3600_mean', 'anglezdiffabs_diff_600_mean',\n                        'enmo_3600_min', 'enmo_600_min']\nfor p in periods:\n    features_to_drop.append(f'anglezdiffabs_{p}_amplit')\n    features_to_drop.append(f'enmo_{p}_amplit')","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:49:57.440435Z","iopub.execute_input":"2023-10-21T20:49:57.440745Z","iopub.status.idle":"2023-10-21T20:49:57.447149Z","shell.execute_reply.started":"2023-10-21T20:49:57.440719Z","shell.execute_reply":"2023-10-21T20:49:57.446131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.anglezdiffabs_3600_min.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:49:57.448660Z","iopub.execute_input":"2023-10-21T20:49:57.449037Z","iopub.status.idle":"2023-10-21T20:49:57.579933Z","shell.execute_reply.started":"2023-10-21T20:49:57.449009Z","shell.execute_reply":"2023-10-21T20:49:57.578789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.anglezdiffabs_600_min.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:49:57.581491Z","iopub.execute_input":"2023-10-21T20:49:57.582033Z","iopub.status.idle":"2023-10-21T20:49:57.709453Z","shell.execute_reply.started":"2023-10-21T20:49:57.582004Z","shell.execute_reply":"2023-10-21T20:49:57.708248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:49:57.710910Z","iopub.execute_input":"2023-10-21T20:49:57.711245Z","iopub.status.idle":"2023-10-21T20:49:57.847771Z","shell.execute_reply.started":"2023-10-21T20:49:57.711221Z","shell.execute_reply":"2023-10-21T20:49:57.846977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.anglezdiffabs_diff_300_mean.value_counts().head(5)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:49:57.848834Z","iopub.execute_input":"2023-10-21T20:49:57.849175Z","iopub.status.idle":"2023-10-21T20:49:57.993677Z","shell.execute_reply.started":"2023-10-21T20:49:57.849139Z","shell.execute_reply":"2023-10-21T20:49:57.992556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.anglezdiffabs_diff_3600_mean.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:49:57.995286Z","iopub.execute_input":"2023-10-21T20:49:57.995624Z","iopub.status.idle":"2023-10-21T20:49:58.123270Z","shell.execute_reply.started":"2023-10-21T20:49:57.995595Z","shell.execute_reply":"2023-10-21T20:49:58.122005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.anglezdiffabs_diff_600_mean.value_counts().head(5)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:49:58.124788Z","iopub.execute_input":"2023-10-21T20:49:58.125231Z","iopub.status.idle":"2023-10-21T20:49:58.255153Z","shell.execute_reply.started":"2023-10-21T20:49:58.125192Z","shell.execute_reply":"2023-10-21T20:49:58.254010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:49:58.256536Z","iopub.execute_input":"2023-10-21T20:49:58.256985Z","iopub.status.idle":"2023-10-21T20:49:58.364571Z","shell.execute_reply.started":"2023-10-21T20:49:58.256943Z","shell.execute_reply":"2023-10-21T20:49:58.363446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.enmo_3600_min.value_counts().head(5)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:49:58.366048Z","iopub.execute_input":"2023-10-21T20:49:58.366461Z","iopub.status.idle":"2023-10-21T20:49:58.504547Z","shell.execute_reply.started":"2023-10-21T20:49:58.366419Z","shell.execute_reply":"2023-10-21T20:49:58.503289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Top features by correlation with the target:","metadata":{}},{"cell_type":"code","source":"corr_with_target = df[features].corrwith(df['awake'])\ncorr_with_target.abs().sort_values().tail(50)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:49:58.506997Z","iopub.execute_input":"2023-10-21T20:49:58.507913Z","iopub.status.idle":"2023-10-21T20:50:33.817337Z","shell.execute_reply.started":"2023-10-21T20:49:58.507880Z","shell.execute_reply":"2023-10-21T20:50:33.816176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features_for_heatmap = corr_with_target.abs().sort_values().tail(27).index.to_list()\nfeatures_for_heatmap = [f for f in features_for_heatmap if f not in ['anglezdiffabs_3600_min','anglezdiffabs_diff_3600_mean']]","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:50:33.818551Z","iopub.execute_input":"2023-10-21T20:50:33.818873Z","iopub.status.idle":"2023-10-21T20:50:33.824509Z","shell.execute_reply.started":"2023-10-21T20:50:33.818845Z","shell.execute_reply":"2023-10-21T20:50:33.823781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(14,14))\nplt.title(f\"Correlation heatmap\")\nsns.heatmap(df[features_for_heatmap].corr(),\nvmin=-1, vmax=1, center=0,\ncmap=sns.color_palette(\"flare\", as_cmap=True),\nsquare=True)\nplt.show() ","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:52:14.909839Z","iopub.execute_input":"2023-10-21T20:52:14.910233Z","iopub.status.idle":"2023-10-21T20:52:43.176725Z","shell.execute_reply.started":"2023-10-21T20:52:14.910204Z","shell.execute_reply":"2023-10-21T20:52:43.175843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training","metadata":{}},{"cell_type":"markdown","source":"Let's train ensemble to check if our assumptions about features are valid.","metadata":{}},{"cell_type":"code","source":"# To spped up the process\ndf = df[::50]","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:52:43.178292Z","iopub.execute_input":"2023-10-21T20:52:43.179110Z","iopub.status.idle":"2023-10-21T20:52:43.184100Z","shell.execute_reply.started":"2023-10-21T20:52:43.179078Z","shell.execute_reply":"2023-10-21T20:52:43.183114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgb_params = {\n'boosting_type':'gbdt',\n'num_leaves':35,\n'max_depth':6,\n'learning_rate':0.03,\n'n_estimators':850,\n'subsample_for_bin':200000,\n'min_child_weight':0.001,\n'min_child_samples':20,\n'subsample':0.9,\n'reg_alpha':0.05,\n'reg_lambda':0.05,\n'verbose': -1\n}","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:52:43.185497Z","iopub.execute_input":"2023-10-21T20:52:43.185952Z","iopub.status.idle":"2023-10-21T20:52:43.195594Z","shell.execute_reply.started":"2023-10-21T20:52:43.185918Z","shell.execute_reply":"2023-10-21T20:52:43.194698Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_model(df, cv):\n    \n    drop_cols = ['series_id', 'step', 'timestamp']\n    X, y = df.drop(columns=drop_cols+['awake']), df['awake']\n    gc.collect()\n    \n    cv_split = cv.split(X)\n    models = []\n    mean_f1 = 0\n    mean_acc = 0\n    \n    for i, (train_index, test_index) in enumerate(cv_split):\n\n        X_train, y_train = X.iloc[train_index], y.iloc[train_index]\n        X_test, y_test = X.iloc[test_index], y.iloc[test_index]\n        \n        print(f\"Fitting model {i+1} of {cv.n_splits}\")\n        \n        model = lgb.LGBMClassifier(random_state=42, n_jobs=-1, **lgb_params)\n        model.fit(X_train, y_train)\n        preds = model.predict(X_test)\n        f1 = f1_score(y_test, preds)\n        acc = accuracy_score(y_test, preds)\n        print(f\"Fold {i+1} finished training\\n F1:{f1_score(y_test, preds)} | Acc:{accuracy_score(y_test, preds)}\")\n        mean_f1 += f1\n        mean_acc += acc\n        models.append(model)\n        \n    print('-'*50)\n    print(f\"Training finished\\nF1: {mean_f1 / cv.n_splits}, Acc: {mean_acc / cv.n_splits}\")   \n    gc.collect()\n    \n    return models","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:52:43.197868Z","iopub.execute_input":"2023-10-21T20:52:43.198684Z","iopub.status.idle":"2023-10-21T20:52:43.210106Z","shell.execute_reply.started":"2023-10-21T20:52:43.198652Z","shell.execute_reply":"2023-10-21T20:52:43.208942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nprint(f\"Fitting model with all {len(features)} features\")\n\nkf = KFold(n_splits=5, shuffle=True, random_state=42)\nmodels = train_model(df, kf)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:52:43.211696Z","iopub.execute_input":"2023-10-21T20:52:43.212054Z","iopub.status.idle":"2023-10-21T20:57:16.613420Z","shell.execute_reply.started":"2023-10-21T20:52:43.212026Z","shell.execute_reply":"2023-10-21T20:57:16.612218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's investigate feature importances of models","metadata":{}},{"cell_type":"code","source":"fi = pd.DataFrame([pd.Series(model.feature_importances_, index=features) for model in models])","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:57:16.614626Z","iopub.execute_input":"2023-10-21T20:57:16.614943Z","iopub.status.idle":"2023-10-21T20:57:16.626034Z","shell.execute_reply.started":"2023-10-21T20:57:16.614916Z","shell.execute_reply":"2023-10-21T20:57:16.624950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fi.head()","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:57:16.627592Z","iopub.execute_input":"2023-10-21T20:57:16.628005Z","iopub.status.idle":"2023-10-21T20:57:16.654590Z","shell.execute_reply.started":"2023-10-21T20:57:16.627975Z","shell.execute_reply":"2023-10-21T20:57:16.653476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_importance = fi.mean(axis=0).sort_values()","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:57:16.656428Z","iopub.execute_input":"2023-10-21T20:57:16.656870Z","iopub.status.idle":"2023-10-21T20:57:16.663414Z","shell.execute_reply.started":"2023-10-21T20:57:16.656831Z","shell.execute_reply":"2023-10-21T20:57:16.662171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15,8))\nplt.xticks(rotation='vertical')\nsns.barplot(x=feature_importance[-15:].index, y=feature_importance[-15:].values)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:57:16.665212Z","iopub.execute_input":"2023-10-21T20:57:16.665631Z","iopub.status.idle":"2023-10-21T20:57:17.130810Z","shell.execute_reply.started":"2023-10-21T20:57:16.665594Z","shell.execute_reply":"2023-10-21T20:57:17.129720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Least important features\nfeature_importance.head(20)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:57:17.135496Z","iopub.execute_input":"2023-10-21T20:57:17.135856Z","iopub.status.idle":"2023-10-21T20:57:17.145242Z","shell.execute_reply.started":"2023-10-21T20:57:17.135829Z","shell.execute_reply":"2023-10-21T20:57:17.144047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.drop(columns=features_to_drop)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:57:17.146711Z","iopub.execute_input":"2023-10-21T20:57:17.147386Z","iopub.status.idle":"2023-10-21T20:57:17.545196Z","shell.execute_reply.started":"2023-10-21T20:57:17.147356Z","shell.execute_reply":"2023-10-21T20:57:17.544264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features = [col for col in df.columns if col not in ['timestamp', 'series_id', 'step', 'awake']]","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:57:17.546622Z","iopub.execute_input":"2023-10-21T20:57:17.546943Z","iopub.status.idle":"2023-10-21T20:57:17.552215Z","shell.execute_reply.started":"2023-10-21T20:57:17.546916Z","shell.execute_reply":"2023-10-21T20:57:17.551130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nprint(f\"Fitting model with {len(features)} features\")\n\nkf = KFold(n_splits=5, shuffle=True, random_state=42)\nmodels = train_model(df, kf)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T20:57:17.554061Z","iopub.execute_input":"2023-10-21T20:57:17.554447Z","iopub.status.idle":"2023-10-21T21:01:26.762362Z","shell.execute_reply.started":"2023-10-21T20:57:17.554411Z","shell.execute_reply":"2023-10-21T21:01:26.760891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Same score with less features and less computation time. Nice.","metadata":{}},{"cell_type":"code","source":"fi = pd.DataFrame([pd.Series(model.feature_importances_, index=features) for model in models])\nfeature_importance = fi.mean(axis=0).sort_values()\nplt.figure(figsize=(15,8))\nplt.xticks(rotation='vertical')\nsns.barplot(x=feature_importance[-15:].index, y=feature_importance[-15:].values)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T21:01:26.763470Z","iopub.execute_input":"2023-10-21T21:01:26.763990Z","iopub.status.idle":"2023-10-21T21:01:27.225115Z","shell.execute_reply.started":"2023-10-21T21:01:26.763960Z","shell.execute_reply":"2023-10-21T21:01:27.224259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering","metadata":{}},{"cell_type":"markdown","source":"## Lag Features\nNow let's add some new features and check their impact. I want to try lag features. Note that it's okay to use \"information from the future\" like **shift(-1)**, because the organizer is going to use the model on previously collected data for labeling.","metadata":{}},{"cell_type":"code","source":"for col in df.columns.to_list():\n    if col not in ['hour', 'series_id', 'step', 'timestamp', 'awake']:\n        df[f'{col}_lag_-1'] = df[col].shift(-1).astype(\"float16\")\n        df[f'{col}_lag_1'] = df[col].shift(1).astype(\"float16\")","metadata":{"execution":{"iopub.status.busy":"2023-10-21T21:01:27.226178Z","iopub.execute_input":"2023-10-21T21:01:27.227144Z","iopub.status.idle":"2023-10-21T21:01:27.429884Z","shell.execute_reply.started":"2023-10-21T21:01:27.227109Z","shell.execute_reply":"2023-10-21T21:01:27.428870Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features = [col for col in df.columns if col not in ['timestamp', 'series_id', 'step', 'awake']]","metadata":{"execution":{"iopub.status.busy":"2023-10-21T21:01:27.431115Z","iopub.execute_input":"2023-10-21T21:01:27.431420Z","iopub.status.idle":"2023-10-21T21:01:27.436808Z","shell.execute_reply.started":"2023-10-21T21:01:27.431392Z","shell.execute_reply":"2023-10-21T21:01:27.435881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nprint(f\"Fitting model with {len(features)} features\")\n\nkf = KFold(n_splits=5, shuffle=True, random_state=42)\nmodels = train_model(df, kf)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T21:01:27.438109Z","iopub.execute_input":"2023-10-21T21:01:27.439367Z","iopub.status.idle":"2023-10-21T21:12:16.755755Z","shell.execute_reply.started":"2023-10-21T21:01:27.439335Z","shell.execute_reply":"2023-10-21T21:12:16.754439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It works (accuracy increased), but training the model started to take much longer. Let's try to remove some of the generated new features that have the lowest feature importance.","metadata":{}},{"cell_type":"code","source":"fi = pd.DataFrame([pd.Series(model.feature_importances_, index=features) for model in models])\nfeature_importance = fi.mean(axis=0).sort_values()\nplt.figure(figsize=(15,8))\nplt.xticks(rotation='vertical')\nsns.barplot(x=feature_importance[-15:].index, y=feature_importance[-15:].values)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T21:12:16.757600Z","iopub.execute_input":"2023-10-21T21:12:16.758779Z","iopub.status.idle":"2023-10-21T21:12:17.249703Z","shell.execute_reply.started":"2023-10-21T21:12:16.758715Z","shell.execute_reply":"2023-10-21T21:12:17.248858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_importance.filter(like='_lag_1').head(25)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T21:12:17.250935Z","iopub.execute_input":"2023-10-21T21:12:17.251785Z","iopub.status.idle":"2023-10-21T21:12:17.262415Z","shell.execute_reply.started":"2023-10-21T21:12:17.251752Z","shell.execute_reply":"2023-10-21T21:12:17.261152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_importance.filter(like='_lag_-1').head(25)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T21:12:17.263675Z","iopub.execute_input":"2023-10-21T21:12:17.264021Z","iopub.status.idle":"2023-10-21T21:12:17.280133Z","shell.execute_reply.started":"2023-10-21T21:12:17.263991Z","shell.execute_reply":"2023-10-21T21:12:17.278952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features_to_drop = feature_importance.filter(like='_lag_1').head(25).index.to_list() +\\\n                   feature_importance.filter(like='_lag_-1').head(25).index.to_list()","metadata":{"execution":{"iopub.status.busy":"2023-10-21T21:12:17.281306Z","iopub.execute_input":"2023-10-21T21:12:17.281623Z","iopub.status.idle":"2023-10-21T21:12:17.290672Z","shell.execute_reply.started":"2023-10-21T21:12:17.281596Z","shell.execute_reply":"2023-10-21T21:12:17.289604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.drop(columns=features_to_drop)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T21:12:17.294105Z","iopub.execute_input":"2023-10-21T21:12:17.294517Z","iopub.status.idle":"2023-10-21T21:12:17.761659Z","shell.execute_reply.started":"2023-10-21T21:12:17.294489Z","shell.execute_reply":"2023-10-21T21:12:17.760747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nprint(f\"Fitting model with {len(df.columns)-4} features\")\n\nkf = KFold(n_splits=5, shuffle=True, random_state=42)\nmodels = train_model(df, kf)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T21:12:17.762946Z","iopub.execute_input":"2023-10-21T21:12:17.763583Z","iopub.status.idle":"2023-10-21T21:21:03.646427Z","shell.execute_reply.started":"2023-10-21T21:12:17.763547Z","shell.execute_reply":"2023-10-21T21:21:03.645207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Accuracy score with 50 removed features is the same.\n\nNow let's try to combine top performing features with each other using basic arithmetic operations.","metadata":{}},{"cell_type":"code","source":"fi = pd.DataFrame([pd.Series(model.feature_importances_, index=model.booster_.feature_name()) for model in models])\nfeature_importance = fi.mean(axis=0).sort_values()","metadata":{"execution":{"iopub.status.busy":"2023-10-21T21:21:03.647985Z","iopub.execute_input":"2023-10-21T21:21:03.648982Z","iopub.status.idle":"2023-10-21T21:21:03.666751Z","shell.execute_reply.started":"2023-10-21T21:21:03.648937Z","shell.execute_reply":"2023-10-21T21:21:03.665846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feature_importance.tail(20)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T21:21:03.668349Z","iopub.execute_input":"2023-10-21T21:21:03.668651Z","iopub.status.idle":"2023-10-21T21:21:03.680015Z","shell.execute_reply.started":"2023-10-21T21:21:03.668624Z","shell.execute_reply":"2023-10-21T21:21:03.678828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"candidates = feature_importance.tail(20).index.to_list()\nfor feature1 in candidates:\n    df[f'{feature1}_SQR'] = df[feature1] * df[feature1]\n    for feature2 in candidates:\n        if feature1 != feature2:\n            df[f'{feature1}_MUL_{feature2}'] = df[feature1] * df[feature2]\n            df[f'{feature1}_DIV_{feature2}'] = df[feature1] / df[feature2]        ","metadata":{"execution":{"iopub.status.busy":"2023-10-21T21:21:03.681866Z","iopub.execute_input":"2023-10-21T21:21:03.682284Z","iopub.status.idle":"2023-10-21T21:21:08.208880Z","shell.execute_reply.started":"2023-10-21T21:21:03.682246Z","shell.execute_reply":"2023-10-21T21:21:08.207854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features = [col for col in df.columns if col not in ['timestamp', 'series_id', 'step', 'awake']]","metadata":{"execution":{"iopub.status.busy":"2023-10-21T22:21:13.105714Z","iopub.execute_input":"2023-10-21T22:21:13.106176Z","iopub.status.idle":"2023-10-21T22:21:13.113077Z","shell.execute_reply.started":"2023-10-21T22:21:13.106142Z","shell.execute_reply":"2023-10-21T22:21:13.111894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nprint(f\"Fitting model with {len(features)} features\")\n\nkf = KFold(n_splits=5, shuffle=True, random_state=42)\nmodels = train_model(df, kf)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T21:26:08.103815Z","iopub.execute_input":"2023-10-21T21:26:08.104249Z","iopub.status.idle":"2023-10-21T22:16:55.572117Z","shell.execute_reply.started":"2023-10-21T21:26:08.104217Z","shell.execute_reply":"2023-10-21T22:16:55.570761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Before\n#F1: 0.9880503108878849, Acc: 0.9843715584218957\n#After\n#F1: 0.9900570304275906, Acc: 0.9869845275025252","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There is a clear increase in the model's metrics, but most of the features generated in this way are obviously worthless. Let's take out 80% and see what happens.","metadata":{}},{"cell_type":"code","source":"fi = pd.DataFrame([pd.Series(model.feature_importances_, index=features) for model in models])\nfeature_importance = fi.mean(axis=0).sort_values()\nplt.figure(figsize=(15,8))\nplt.xticks(rotation='vertical')\nsns.barplot(x=feature_importance[-15:].index, y=feature_importance[-15:].values)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T22:21:16.672833Z","iopub.execute_input":"2023-10-21T22:21:16.673279Z","iopub.status.idle":"2023-10-21T22:21:17.264142Z","shell.execute_reply.started":"2023-10-21T22:21:16.673246Z","shell.execute_reply":"2023-10-21T22:21:17.262879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features_to_drop = feature_importance.head(800).index.to_list()","metadata":{"execution":{"iopub.status.busy":"2023-10-21T22:28:44.771116Z","iopub.execute_input":"2023-10-21T22:28:44.771565Z","iopub.status.idle":"2023-10-21T22:28:44.780559Z","shell.execute_reply.started":"2023-10-21T22:28:44.771532Z","shell.execute_reply":"2023-10-21T22:28:44.779154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_light = df.drop(columns=features_to_drop)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T22:28:46.530056Z","iopub.execute_input":"2023-10-21T22:28:46.530459Z","iopub.status.idle":"2023-10-21T22:28:46.921923Z","shell.execute_reply.started":"2023-10-21T22:28:46.530427Z","shell.execute_reply":"2023-10-21T22:28:46.920900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_light.shape","metadata":{"execution":{"iopub.status.busy":"2023-10-21T22:28:48.222412Z","iopub.execute_input":"2023-10-21T22:28:48.222866Z","iopub.status.idle":"2023-10-21T22:28:48.230543Z","shell.execute_reply.started":"2023-10-21T22:28:48.222820Z","shell.execute_reply":"2023-10-21T22:28:48.229338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features = [col for col in df_light.columns if col not in ['timestamp', 'series_id', 'step', 'awake']]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nprint(f\"Fitting model with {len(features)} features\")\n\nkf = KFold(n_splits=5, shuffle=True, random_state=42)\nmodels = train_model(df_light, kf)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T22:29:06.859836Z","iopub.execute_input":"2023-10-21T22:29:06.860502Z","iopub.status.idle":"2023-10-21T22:40:07.624232Z","shell.execute_reply.started":"2023-10-21T22:29:06.860459Z","shell.execute_reply":"2023-10-21T22:40:07.623009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"80% of the generated features are removed and the model shows the same high accuracy.","metadata":{}},{"cell_type":"code","source":"fi = pd.DataFrame([pd.Series(model.feature_importances_, index=features) for model in models])\nfeature_importance = fi.mean(axis=0).sort_values()\nplt.figure(figsize=(15,8))\nplt.xticks(rotation='vertical')\nsns.barplot(x=feature_importance[-15:].index, y=feature_importance[-15:].values)","metadata":{"execution":{"iopub.status.busy":"2023-10-21T22:50:59.986629Z","iopub.execute_input":"2023-10-21T22:50:59.987607Z","iopub.status.idle":"2023-10-21T22:51:00.572184Z","shell.execute_reply.started":"2023-10-21T22:50:59.987566Z","shell.execute_reply":"2023-10-21T22:51:00.571150Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Final set of features:","metadata":{}},{"cell_type":"code","source":"features","metadata":{"execution":{"iopub.status.busy":"2023-10-21T22:55:43.539339Z","iopub.execute_input":"2023-10-21T22:55:43.539810Z","iopub.status.idle":"2023-10-21T22:55:43.554594Z","shell.execute_reply.started":"2023-10-21T22:55:43.539761Z","shell.execute_reply":"2023-10-21T22:55:43.553420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A few observations: \n* Among the remaining features, most belong to the 3600 second window. Aggregations over shorter periods are also present, though.\n* The models \"liked\" features normalized by **hour**, maybe you should add them, even if you don't intend to use the whole set derived here. \n* This notebook has many degrees of freedom: you could try to remove lag(-1) or for example add another time window for aggregation at the very beginning. \n\nGood luck with your experiments.","metadata":{}},{"cell_type":"markdown","source":"Please leave an upvote if you found this work useful. It motivates me to post new versions. Feel free to suggest new ideas for experiments, I will try to include the most interesting things in this notebook.","metadata":{}}]}