{"metadata":{"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":61145,"databundleVersionId":7047669,"sourceType":"competition"}],"dockerImageVersionId":30646,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"papermill":{"default_parameters":{},"duration":8816.051734,"end_time":"2024-01-29T07:16:03.940045","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-01-29T04:49:07.888311","version":"2.4.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Loading Libraries and Datasets","metadata":{"papermill":{"duration":0.013798,"end_time":"2024-01-29T04:49:11.324968","exception":false,"start_time":"2024-01-29T04:49:11.31117","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install -q iterative-stratification","metadata":{"papermill":{"duration":13.360127,"end_time":"2024-01-29T04:49:24.69885","exception":false,"start_time":"2024-01-29T04:49:11.338723","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:15.557874Z","iopub.execute_input":"2024-03-14T15:16:15.558250Z","iopub.status.idle":"2024-03-14T15:16:32.854845Z","shell.execute_reply.started":"2024-03-14T15:16:15.558219Z","shell.execute_reply":"2024-03-14T15:16:32.853507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#NOTES: In this section all the required libraries have been imported. Common libraries in ML as\n# Tensorflow, pandas","metadata":{"execution":{"iopub.status.busy":"2024-03-14T15:16:32.858155Z","iopub.execute_input":"2024-03-14T15:16:32.859026Z","iopub.status.idle":"2024-03-14T15:16:32.866160Z","shell.execute_reply.started":"2024-03-14T15:16:32.858974Z","shell.execute_reply":"2024-03-14T15:16:32.865147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport polars as pl\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport tensorflow as tf\nimport os\nimport optuna\nimport holidays\nimport copy\n\nfrom category_encoders import OneHotEncoder, MEstimateEncoder, CatBoostEncoder, BinaryEncoder\nfrom iterstrat.ml_stratifiers import MultilabelStratifiedKFold, RepeatedMultilabelStratifiedKFold\nfrom sklearn.cluster import KMeans\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.model_selection import StratifiedKFold, RepeatedStratifiedKFold, TimeSeriesSplit\nfrom sklearn.decomposition import PCA\nfrom sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier\nfrom sklearn.ensemble import GradientBoostingClassifier, HistGradientBoostingClassifier\nfrom sklearn.ensemble import VotingClassifier\nfrom sklearn.impute import KNNImputer, SimpleImputer\nfrom sklearn.metrics import accuracy_score, roc_auc_score, roc_curve, make_scorer\nfrom sklearn.metrics.pairwise import euclidean_distances\nfrom sklearn.naive_bayes import GaussianNB, BernoulliNB\nfrom sklearn.pipeline import make_pipeline, Pipeline\nfrom sklearn.base import BaseEstimator, TransformerMixin, ClassifierMixin, clone\nfrom sklearn.multioutput import ClassifierChain\nfrom sklearn.preprocessing import FunctionTransformer, StandardScaler, PowerTransformer, LabelEncoder\nfrom sklearn.linear_model import LogisticRegression, RidgeClassifier\nfrom sklearn.svm import SVC\nfrom sklearn.discriminant_analysis import LinearDiscriminantAnalysis\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.inspection import permutation_importance\nfrom scipy.cluster.hierarchy import dendrogram, linkage\nfrom scipy.spatial.distance import squareform\nfrom xgboost import XGBClassifier, XGBRFClassifier\nfrom lightgbm import LGBMClassifier\nfrom catboost import CatBoostClassifier\nfrom sklearn.feature_selection import SequentialFeatureSelector\n\nsns.set_theme(style = 'white', palette = 'viridis')\npal = sns.color_palette('viridis')\n\npd.set_option('display.max_rows', 100)","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":19.724847,"end_time":"2024-01-29T04:49:44.43698","exception":false,"start_time":"2024-01-29T04:49:24.712133","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:32.872390Z","iopub.execute_input":"2024-03-14T15:16:32.872737Z","iopub.status.idle":"2024-03-14T15:16:53.375241Z","shell.execute_reply.started":"2024-03-14T15:16:32.872710Z","shell.execute_reply":"2024-03-14T15:16:53.374121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#NOTES: Data preprocessing work was started in this section. Most of them related to datetime type variable. \n#This could be useful to split date time features like year, month and day this improves performance of \n# ML algorithm","metadata":{"execution":{"iopub.status.busy":"2024-03-14T15:16:53.376268Z","iopub.execute_input":"2024-03-14T15:16:53.376595Z","iopub.status.idle":"2024-03-14T15:16:53.382244Z","shell.execute_reply.started":"2024-03-14T15:16:53.376567Z","shell.execute_reply":"2024-03-14T15:16:53.380819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preprocessor = [\n    pl.col('dT').cast(pl.Float32),\n    \n    pl.col('ts_oxygen_saturation').cast(pl.Float32),\n    pl.col('ts_resting_heart_rate').cast(pl.Float32),\n    \n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z'),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.year().alias('time_year').cast(pl.UInt16),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.month().alias('time_month').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.week().alias('time_week').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.weekday().alias('time_weekday').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.ordinal_day().alias('time_day_of_year').cast(pl.UInt16),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.day().alias('time_day').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.hour().alias('time_hour').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.minute().alias('time_minute').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.second().alias('time_second').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.microsecond().alias('time_micro').cast(pl.UInt32),\n    \n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z'),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.year().alias('loc_year').cast(pl.UInt16),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.month().alias('loc_month').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.week().alias('loc_week').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.weekday().alias('loc_weekday').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.ordinal_day().alias('loc_day_of_year').cast(pl.UInt16),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.day().alias('loc_day').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.hour().alias('loc_hour').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.minute().alias('loc_minute').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.second().alias('loc_second').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.microsecond().alias('loc_micro').cast(pl.UInt32),\n    \n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z'),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.year().alias('start_year').cast(pl.UInt16),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.month().alias('start_month').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.week().alias('start_week').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.weekday().alias('start_weekday').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.ordinal_day().alias('start_day_of_year').cast(pl.UInt16),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.day().alias('start_day').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.hour().alias('start_hour').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.minute().alias('start_minute').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.second().alias('start_second').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.microsecond().alias('start_micro').cast(pl.UInt32)\n]\n\npreprocessor_2 = [\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z'),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.year().alias('year').cast(pl.UInt16),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.month().alias('month').cast(pl.UInt8),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.day().alias('day').cast(pl.UInt8),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.hour().alias('hour').cast(pl.UInt8),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.minute().alias('minute').cast(pl.UInt8),\n    \n    pl.col('value').cast(pl.Float32)\n]\n\nduration_calculator = [\n    (pl.duration(days = pl.col('time_day_of_year'), hours = pl.col('time_hour'), minutes = pl.col('time_minute'), seconds = pl.col('time_second'), microseconds = pl.col('time_micro')) -\n    pl.duration(days = pl.col('start_day_of_year'), hours = pl.col('start_hour'), minutes = pl.col('start_minute'), seconds = pl.col('start_second'), microseconds = pl.col('start_micro'))).dt.total_microseconds().alias('time_start_difference'),\n    (pl.duration(days = pl.col('time_day_of_year'), hours = pl.col('time_hour'), minutes = pl.col('time_minute'), seconds = pl.col('time_second'), microseconds = pl.col('time_micro')) -\n    pl.duration(days = pl.col('loc_day_of_year'), hours = pl.col('loc_hour'), minutes = pl.col('loc_minute'), seconds = pl.col('loc_second'), microseconds = pl.col('loc_micro'))).dt.total_microseconds().alias('time_loc_difference'),\n    (pl.duration(days = pl.col('loc_day_of_year'), hours = pl.col('loc_hour'), minutes = pl.col('loc_minute'), seconds = pl.col('loc_second'), microseconds = pl.col('loc_micro')) -\n     pl.duration(days = pl.col('start_day_of_year'), hours = pl.col('start_hour'), minutes = pl.col('start_minute'), seconds = pl.col('start_second'), microseconds = pl.col('start_micro'))).dt.total_microseconds().alias('start_loc_difference')\n]","metadata":{"papermill":{"duration":0.064479,"end_time":"2024-01-29T04:49:44.521031","exception":false,"start_time":"2024-01-29T04:49:44.456552","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.384487Z","iopub.execute_input":"2024-03-14T15:16:53.385016Z","iopub.status.idle":"2024-03-14T15:16:53.458206Z","shell.execute_reply.started":"2024-03-14T15:16:53.384968Z","shell.execute_reply":"2024-03-14T15:16:53.457010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#NOTES: Separate dataset in two sets. Train set to train ML algorithm and Test set to figure out \n#performance of ML algorithm created before","metadata":{"execution":{"iopub.status.busy":"2024-03-14T15:16:53.461111Z","iopub.execute_input":"2024-03-14T15:16:53.462006Z","iopub.status.idle":"2024-03-14T15:16:53.466810Z","shell.execute_reply.started":"2024-03-14T15:16:53.461963Z","shell.execute_reply":"2024-03-14T15:16:53.465695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/cozie_responses_and_physiological_data_training.csv').\\\n        with_columns(preprocessor).with_columns(duration_calculator)\ntest = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/cozie_responses_and_physiological_data_test_public.csv').\\\n       with_columns(preprocessor).with_columns(duration_calculator)","metadata":{"papermill":{"duration":0.082503,"end_time":"2024-01-29T04:49:44.616906","exception":false,"start_time":"2024-01-29T04:49:44.534403","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.468643Z","iopub.execute_input":"2024-03-14T15:16:53.469590Z","iopub.status.idle":"2024-03-14T15:16:53.528772Z","shell.execute_reply.started":"2024-03-14T15:16:53.469549Z","shell.execute_reply":"2024-03-14T15:16:53.527899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"air_temp = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_air-temperature.csv').\\\n           melt('').\\\n           with_columns(preprocessor_2).\\\n           group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n           agg(\n               pl.col('value').mean().alias('mean_air_temp'),\n               pl.col('value').std().alias('std_air_temp'),\n           )\nrainfall = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_rainfall.csv').\\\n           melt('').\\\n           with_columns(preprocessor_2).\\\n           group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n           agg(\n               pl.col('value').mean().alias('mean_rainfall'),\n               pl.col('value').std().alias('std_rainfall'),\n           )\nhumidity = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_relative-humidity.csv').\\\n           melt('').\\\n           with_columns(preprocessor_2).\\\n           group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n           agg(\n               pl.col('value').mean().alias('mean_humidity'),\n               pl.col('value').std().alias('std_humidity'),\n           )\nstations = pl.read_csv('/kaggle/input/cool-quiet-city-competition/weather_stations.csv')\nwind_direction = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_wind-direction.csv').\\\n                 melt('').\\\n                 with_columns(preprocessor_2).\\\n                 group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n                 agg(\n                     pl.col('value').mean().alias('mean_wind_direction'),\n                     pl.col('value').std().alias('std_wind_direction')\n                 )\nwind_speed = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_wind-speed.csv').\\\n             melt('').\\\n             with_columns(preprocessor_2).\\\n             group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n             agg(\n                 pl.col('value').mean().alias('mean_wind_speed'),\n                 pl.col('value').std().alias('std_wind_speed'),\n             )","metadata":{"papermill":{"duration":0.102406,"end_time":"2024-01-29T04:49:44.732804","exception":false,"start_time":"2024-01-29T04:49:44.630398","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.530474Z","iopub.execute_input":"2024-03-14T15:16:53.531265Z","iopub.status.idle":"2024-03-14T15:16:53.610225Z","shell.execute_reply.started":"2024-03-14T15:16:53.531225Z","shell.execute_reply":"2024-03-14T15:16:53.609081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset Information","metadata":{"papermill":{"duration":0.01312,"end_time":"2024-01-29T04:49:44.759911","exception":false,"start_time":"2024-01-29T04:49:44.746791","status":"completed"},"tags":[]}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def desc_dataset_pl(df):\n    return pl.concat([\n        pl.DataFrame({'key' : df.columns, 'dataset_length' : len(df)}),\n        df.null_count().transpose().rename({'column_0' : 'null'}),\n        df.approx_n_unique().transpose().rename({'column_0' : 'nunique'}),\n        df.mean().transpose().rename({'column_0' : 'mean'}),\n        df.std().transpose().rename({'column_0' : 'std'}),\n        df.median().transpose().rename({'column_0' : 'median'}),\n        df.min().transpose().rename({'column_0' : 'min'}),\n        df.max().transpose().rename({'column_0' : 'max'})\n    ], how = 'horizontal').with_columns(\n        (pl.col('dataset_length') - pl.col('null')).alias('count'),\n        (pl.col('null') / pl.col('dataset_length') * 100).alias('%null'),\n        (pl.col('nunique') / pl.col('dataset_length') * 100).alias('%unique')\n    ).drop('dataset_length').to_pandas()","metadata":{"papermill":{"duration":0.025358,"end_time":"2024-01-29T04:49:44.798669","exception":false,"start_time":"2024-01-29T04:49:44.773311","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.616631Z","iopub.execute_input":"2024-03-14T15:16:53.617504Z","iopub.status.idle":"2024-03-14T15:16:53.627489Z","shell.execute_reply.started":"2024-03-14T15:16:53.617466Z","shell.execute_reply":"2024-03-14T15:16:53.626640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#desc_dataset_pl(train)","metadata":{"papermill":{"duration":0.0217,"end_time":"2024-01-29T04:49:44.834261","exception":false,"start_time":"2024-01-29T04:49:44.812561","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.628978Z","iopub.execute_input":"2024-03-14T15:16:53.629595Z","iopub.status.idle":"2024-03-14T15:16:53.639801Z","shell.execute_reply.started":"2024-03-14T15:16:53.629565Z","shell.execute_reply":"2024-03-14T15:16:53.638438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#desc_dataset_pl(test)","metadata":{"papermill":{"duration":0.021945,"end_time":"2024-01-29T04:49:44.869571","exception":false,"start_time":"2024-01-29T04:49:44.847626","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.641654Z","iopub.execute_input":"2024-03-14T15:16:53.642057Z","iopub.status.idle":"2024-03-14T15:16:53.649620Z","shell.execute_reply.started":"2024-03-14T15:16:53.642026Z","shell.execute_reply":"2024-03-14T15:16:53.648610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TS Features Aggregation","metadata":{"papermill":{"duration":0.013129,"end_time":"2024-01-29T04:49:44.960247","exception":false,"start_time":"2024-01-29T04:49:44.947118","status":"completed"},"tags":[]}},{"cell_type":"code","source":"df_all = pl.concat([train, test], how = 'vertical')","metadata":{"papermill":{"duration":0.025587,"end_time":"2024-01-29T04:49:44.999204","exception":false,"start_time":"2024-01-29T04:49:44.973617","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.651345Z","iopub.execute_input":"2024-03-14T15:16:53.652062Z","iopub.status.idle":"2024-03-14T15:16:53.662152Z","shell.execute_reply.started":"2024-03-14T15:16:53.652020Z","shell.execute_reply":"2024-03-14T15:16:53.661195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#NOTES: Some extra features has been agregated to the dataset in this section in order to have more \n# relevant info about dataset\n","metadata":{"execution":{"iopub.status.busy":"2024-03-14T15:16:53.663693Z","iopub.execute_input":"2024-03-14T15:16:53.664527Z","iopub.status.idle":"2024-03-14T15:16:53.673308Z","shell.execute_reply.started":"2024-03-14T15:16:53.664487Z","shell.execute_reply":"2024-03-14T15:16:53.672310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"survey_counter_day = df_all.group_by(\n    ['id_participant', 'time_year', 'time_month', 'time_day']\n).agg(\n    pl.col('id_participant').count().alias('survey_count_per_day').cast(pl.UInt16)\n)\nsurvey_counter_hour = df_all.group_by(\n    ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour']\n).agg(\n    pl.col('id_participant').count().alias('survey_count_per_hour').cast(pl.UInt16)\n)\nsurvey_counter_minute = df_all.group_by(\n    ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour', 'time_minute']\n).agg(\n    pl.col('id_participant').count().alias('survey_count_per_minute').cast(pl.UInt8)\n)","metadata":{"papermill":{"duration":0.025501,"end_time":"2024-01-29T04:49:45.038174","exception":false,"start_time":"2024-01-29T04:49:45.012673","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.674583Z","iopub.execute_input":"2024-03-14T15:16:53.674960Z","iopub.status.idle":"2024-03-14T15:16:53.686950Z","shell.execute_reply.started":"2024-03-14T15:16:53.674893Z","shell.execute_reply":"2024-03-14T15:16:53.686016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rest_heart_rate_agg = df_all.group_by(\n    ['time_year', 'time_week', 'id_participant']\n).agg(\n    pl.col('ts_resting_heart_rate').mean().alias('mean_resting_heart_rate'),\n    pl.col('ts_resting_heart_rate').std().alias('std_resting_heart_rate')\n)\noxygen_saturation_agg = df_all.group_by(\n    ['time_year', 'time_week', 'id_participant']\n).agg(\n    pl.col('ts_oxygen_saturation').mean().alias('mean_oxygen_saturation'),\n    pl.col('ts_oxygen_saturation').std().alias('std_oxygen_saturation')\n)\nstand_time_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_stand_time').mean().alias('mean_stand_time'),\n    pl.col('ts_stand_time').std().alias('std_stand_time')\n)\nstep_count_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_step_count').mean().alias('mean_step_count'),\n    pl.col('ts_step_count').std().alias('std_step_count')\n)\nwalking_distance_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_walking_distance').mean().alias('mean_walking_distance'),\n    pl.col('ts_walking_distance').std().alias('std_walking_distance')\n)\naudio_exposure_environment_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_audio_exposure_environment').mean().alias('mean_audio_exposure_environment'),\n    pl.col('ts_audio_exposure_environment').std().alias('std_audio_exposure_environment')\n)\nheart_rate_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_heart_rate').mean().alias('mean_heart_rate'),\n    pl.col('ts_heart_rate').std().alias('std_heart_rate')\n)","metadata":{"papermill":{"duration":0.030034,"end_time":"2024-01-29T04:49:45.081786","exception":false,"start_time":"2024-01-29T04:49:45.051752","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.688100Z","iopub.execute_input":"2024-03-14T15:16:53.689084Z","iopub.status.idle":"2024-03-14T15:16:53.706452Z","shell.execute_reply.started":"2024-03-14T15:16:53.689053Z","shell.execute_reply":"2024-03-14T15:16:53.705282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.join(\n    rest_heart_rate_agg, on = ['time_year', 'time_week', 'id_participant'], how = 'left'\n).join(\n    oxygen_saturation_agg, on = ['time_year', 'time_week', 'id_participant'], how = 'left'\n).join(\n    stand_time_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    step_count_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    walking_distance_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    audio_exposure_environment_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    heart_rate_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    survey_counter_day, on = ['id_participant', 'time_year', 'time_month', 'time_day'], how = 'left'\n).join(\n    survey_counter_hour, on = ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour'], how = 'left'\n).join(\n    survey_counter_minute, on = ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'], how = 'left'\n)\ntest = test.join(\n    rest_heart_rate_agg, on = ['time_year', 'time_week', 'id_participant'], how = 'left'\n).join(\n    oxygen_saturation_agg, on = ['time_year', 'time_week', 'id_participant'], how = 'left'\n).join(\n    stand_time_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    step_count_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    walking_distance_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    audio_exposure_environment_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    heart_rate_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    survey_counter_day, on = ['id_participant', 'time_year', 'time_month', 'time_day'], how = 'left'\n).join(\n    survey_counter_hour, on = ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour'], how = 'left'\n).join(\n    survey_counter_minute, on = ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'], how = 'left'\n)","metadata":{"papermill":{"duration":0.040233,"end_time":"2024-01-29T04:49:45.135455","exception":false,"start_time":"2024-01-29T04:49:45.095222","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.708462Z","iopub.execute_input":"2024-03-14T15:16:53.709256Z","iopub.status.idle":"2024-03-14T15:16:53.738223Z","shell.execute_reply.started":"2024-03-14T15:16:53.709216Z","shell.execute_reply":"2024-03-14T15:16:53.737014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Station and Weather","metadata":{"papermill":{"duration":0.012909,"end_time":"2024-01-29T04:49:45.16168","exception":false,"start_time":"2024-01-29T04:49:45.148771","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#NOTES: Some functions was defined. First of them find the closest station in a dataset called stations\n# in terms of latitude and longitude. Last function takes a DataFrame with latitude and longitude columns\n# formats these columns then finds the nearest weather station to each set of coordinates, and returns the \n# result in a new column","metadata":{"execution":{"iopub.status.busy":"2024-03-14T15:16:53.739567Z","iopub.execute_input":"2024-03-14T15:16:53.740178Z","iopub.status.idle":"2024-03-14T15:16:53.744096Z","shell.execute_reply.started":"2024-03-14T15:16:53.740146Z","shell.execute_reply":"2024-03-14T15:16:53.743147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def find_closest_station(x):\n    if x[0] == None: return None\n    latitude = float(x[0][:18])\n    longitude = float(x[0][19:])\n    return stations.row(stations.select(\n        distance = (pl.col('latitude') - latitude) ** 2 + (pl.col('longitude') - longitude) ** 2\n    )['distance'].arg_min())[1]","metadata":{"papermill":{"duration":0.022635,"end_time":"2024-01-29T04:49:45.197578","exception":false,"start_time":"2024-01-29T04:49:45.174943","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.745230Z","iopub.execute_input":"2024-03-14T15:16:53.746085Z","iopub.status.idle":"2024-03-14T15:16:53.759450Z","shell.execute_reply.started":"2024-03-14T15:16:53.746053Z","shell.execute_reply":"2024-03-14T15:16:53.758453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def map_station(df):\n    return df.select(pl.col('ws_latitude').cast(pl.Utf8).map_elements(\n        lambda x: x + '0' * (18 - len(x))\n    ) + ' ' + pl.col('ws_longitude').cast(pl.Utf8).map_elements(\n        lambda x: x + '0' * (18 - len(x))\n    )).map_rows(find_closest_station)['map']","metadata":{"papermill":{"duration":0.02323,"end_time":"2024-01-29T04:49:45.234313","exception":false,"start_time":"2024-01-29T04:49:45.211083","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.760717Z","iopub.execute_input":"2024-03-14T15:16:53.761841Z","iopub.status.idle":"2024-03-14T15:16:53.772244Z","shell.execute_reply.started":"2024-03-14T15:16:53.761798Z","shell.execute_reply":"2024-03-14T15:16:53.771002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.join(\n    air_temp,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    rainfall,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    humidity,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    wind_direction,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    wind_speed,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute'])\n\ntest = test.join(\n    air_temp,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    rainfall,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    humidity,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    wind_direction,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    wind_speed,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute'])","metadata":{"papermill":{"duration":0.04683,"end_time":"2024-01-29T04:49:45.295134","exception":false,"start_time":"2024-01-29T04:49:45.248304","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.774088Z","iopub.execute_input":"2024-03-14T15:16:53.774848Z","iopub.status.idle":"2024-03-14T15:16:53.809386Z","shell.execute_reply.started":"2024-03-14T15:16:53.774812Z","shell.execute_reply":"2024-03-14T15:16:53.808138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Correlation","metadata":{"papermill":{"duration":0.01345,"end_time":"2024-01-29T04:49:45.322014","exception":false,"start_time":"2024-01-29T04:49:45.308564","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#NOTES: Correlation helps to find out which features are related to each other in order to avoid \n# oversizing in dataset","metadata":{"execution":{"iopub.status.busy":"2024-03-14T15:16:53.810745Z","iopub.execute_input":"2024-03-14T15:16:53.811258Z","iopub.status.idle":"2024-03-14T15:16:53.823744Z","shell.execute_reply.started":"2024-03-14T15:16:53.811228Z","shell.execute_reply":"2024-03-14T15:16:53.822620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def heatmap(dataset, label = None):\n    corr = dataset.corr(method = 'pearson')\n    plt.figure(figsize = (45, 45), dpi = 300)\n    mask = np.zeros_like(corr)\n    mask[np.triu_indices_from(mask)] = True\n    sns.heatmap(corr, mask = mask, cmap = 'viridis', annot = True, annot_kws = {'size' : 11})\n    plt.title(f'{label} Dataset Correlation Matrix\\n', fontsize = 25, weight = 'bold')\n    plt.show()","metadata":{"papermill":{"duration":0.022915,"end_time":"2024-01-29T04:49:45.358148","exception":false,"start_time":"2024-01-29T04:49:45.335233","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.825342Z","iopub.execute_input":"2024-03-14T15:16:53.826092Z","iopub.status.idle":"2024-03-14T15:16:53.834156Z","shell.execute_reply.started":"2024-03-14T15:16:53.826061Z","shell.execute_reply":"2024-03-14T15:16:53.833162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preparation","metadata":{"papermill":{"duration":0.013031,"end_time":"2024-01-29T04:49:45.384604","exception":false,"start_time":"2024-01-29T04:49:45.371573","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#NOTES: Next sections many ML algorithms has been trained with some specs applied","metadata":{"execution":{"iopub.status.busy":"2024-03-14T15:16:53.835979Z","iopub.execute_input":"2024-03-14T15:16:53.836676Z","iopub.status.idle":"2024-03-14T15:16:53.844777Z","shell.execute_reply.started":"2024-03-14T15:16:53.836636Z","shell.execute_reply":"2024-03-14T15:16:53.843997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = train.filter(pl.col('q_location').is_not_null()).drop('id_unique', 'time', 'ws_timestamp_location', 'ws_timestamp_start', 'ts_oxygen_saturation', 'ts_resting_heart_rate', 'ts_stand_time', 'ts_step_count', 'ts_walking_distance', 'ts_heart_rate', 'ts_audio_exposure_environment').collect()\nX = X.with_columns(stations = map_station(X)).to_pandas()\ny = X[['q_earphones', 'q_noise_kind', 'q_noise_nearby', 'q_thermal_preference']]\nX = X.drop(['q_earphones', 'q_noise_kind', 'q_noise_nearby', 'q_thermal_preference'], axis = 1)\n\nsubmission = test.filter(pl.col('q_location').is_not_null()).drop('q_earphones', 'q_noise_kind', 'q_noise_nearby', 'q_thermal_preference', 'time', 'ws_timestamp_location', 'ws_timestamp_start', 'ts_oxygen_saturation', 'ts_resting_heart_rate', 'ts_stand_time', 'ts_step_count', 'ts_walking_distance', 'ts_heart_rate', 'ts_audio_exposure_environment').collect()\nsubmission = submission.with_columns(stations = map_station(submission)).to_pandas()\ntest_pd = submission.drop('id_unique', axis = 1)\ntest_pd.loc[3178, 'time_loc_difference'] += 1e6 * 3600 * 24 * 365\ntest_pd.loc[3178, 'start_loc_difference'] -= 1e6 * 3600 * 24 * 365\n\ncategorical_features = list(X.select_dtypes('object'))\n\nle = {}\n\nfor col in y.columns:\n    le[col] = LabelEncoder().fit(y[col])\n    y[col] = le[col].transform(y[col])\n    \nacc_scorer = make_scorer(accuracy_score)\nseed = 42\ntf.keras.utils.set_random_seed(seed)\n\nkf = MultilabelStratifiedKFold(n_splits = 5, random_state = seed, shuffle = True)\n#kf = TimeSeriesSplit()","metadata":{"papermill":{"duration":45.476571,"end_time":"2024-01-29T04:50:30.874598","exception":false,"start_time":"2024-01-29T04:49:45.398027","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:16:53.846190Z","iopub.execute_input":"2024-03-14T15:16:53.846752Z","iopub.status.idle":"2024-03-14T15:17:43.701782Z","shell.execute_reply.started":"2024-03-14T15:16:53.846721Z","shell.execute_reply":"2024-03-14T15:17:43.700771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering","metadata":{"papermill":{"duration":0.013352,"end_time":"2024-01-29T04:50:30.901792","exception":false,"start_time":"2024-01-29T04:50:30.88844","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class NumericalImputer(BaseEstimator, TransformerMixin):\n    \n    def fit(self, x, y = None):\n        self.imputer = SimpleImputer().fit(x.select_dtypes(['int', 'float']), y)\n        return self\n    \n    def transform(self, x):\n        x_copy = x.copy()\n        x_copy.loc[:, list(x_copy.select_dtypes(['int', 'float']))] = self.imputer.transform(x_copy.loc[:, list(x_copy.select_dtypes(['int', 'float']))])\n        return x_copy","metadata":{"papermill":{"duration":0.024644,"end_time":"2024-01-29T04:50:31.195005","exception":false,"start_time":"2024-01-29T04:50:31.170361","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:17:43.703041Z","iopub.execute_input":"2024-03-14T15:17:43.703362Z","iopub.status.idle":"2024-03-14T15:17:43.713623Z","shell.execute_reply.started":"2024-03-14T15:17:43.703334Z","shell.execute_reply":"2024-03-14T15:17:43.711932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Adversarial Validation","metadata":{"papermill":{"duration":0.013686,"end_time":"2024-01-29T04:50:31.222264","exception":false,"start_time":"2024-01-29T04:50:31.208578","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#thanks to @carlmcbrideellis\n#https://www.kaggle.com/code/carlmcbrideellis/what-is-adversarial-validation\n\ndef adversarial_validation(dataset_1 = train, dataset_2 = test, label = 'Train-Test'):\n\n    adv_train = dataset_1.copy()\n    adv_test = dataset_2.copy()\n\n    adv_train['is_test'] = 0\n    adv_test['is_test'] = 1\n\n    adv = pd.concat([adv_train, adv_test], ignore_index = True)\n\n    adv_shuffled = adv.sample(frac = 1)\n\n    adv_X = adv_shuffled.drop('is_test', axis = 1)\n    adv_y = adv_shuffled.is_test\n\n    skf = StratifiedKFold(n_splits = 5, random_state = 42, shuffle = True)\n\n    val_scores = []\n    predictions = np.zeros(len(adv))\n\n    for fold, (train_idx, val_idx) in enumerate(skf.split(adv_X, adv_y)):\n    \n        adv_lr = make_pipeline(\n            MEstimateEncoder(cols = list(categorical_features)), \n            #FeatureDropper(cols = ['ws_survey_count', 'id_participant']),\n            SimpleImputer(),\n            XGBClassifier(random_state = seed, tree_method = 'hist')\n        )\n        adv_lr.fit(adv_X.iloc[train_idx], adv_y.iloc[train_idx])\n        \n        val_preds = adv_lr.predict_proba(adv_X.iloc[val_idx])[:, 1]\n        predictions[val_idx] = val_preds\n        val_score = roc_auc_score(adv_y.iloc[val_idx], val_preds)\n        val_scores.append(val_score)\n    \n    fpr, tpr, _ = roc_curve(adv['is_test'], predictions)\n    \n    plt.figure(figsize = (10, 10), dpi = 300)\n    sns.lineplot(x=[0, 1], y=[0, 1], linestyle=\"--\", label=\"Indistinguishable Datasets\")\n    sns.lineplot(x=fpr, y=tpr, label=\"Adversarial Validation Classifier\")\n    plt.title(f'{label} Validation = {np.mean(val_scores):.5f}', weight = 'bold', size = 17)\n    plt.xlabel('False Positive Rate')\n    plt.ylabel('True Positive Rate')\n    plt.show()","metadata":{"papermill":{"duration":0.028635,"end_time":"2024-01-29T04:50:31.264265","exception":false,"start_time":"2024-01-29T04:50:31.23563","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:17:43.715650Z","iopub.execute_input":"2024-03-14T15:17:43.716138Z","iopub.status.idle":"2024-03-14T15:17:43.733431Z","shell.execute_reply.started":"2024-03-14T15:17:43.716091Z","shell.execute_reply":"2024-03-14T15:17:43.732348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#adversarial_validation(X, test_pd)","metadata":{"papermill":{"duration":0.021817,"end_time":"2024-01-29T04:50:31.299491","exception":false,"start_time":"2024-01-29T04:50:31.277674","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:17:43.741012Z","iopub.execute_input":"2024-03-14T15:17:43.741865Z","iopub.status.idle":"2024-03-14T15:17:43.749809Z","shell.execute_reply.started":"2024-03-14T15:17:43.741824Z","shell.execute_reply":"2024-03-14T15:17:43.748988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Cross Validation","metadata":{"papermill":{"duration":0.012986,"end_time":"2024-01-29T04:50:31.325875","exception":false,"start_time":"2024-01-29T04:50:31.312889","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def cross_val_score(estimator, cv = kf, label = '', show_importance = False):\n    \n    val_scores = []\n    \n    feature_importances_table = pd.DataFrame({'value' : 0}, index = list(X.columns))\n    \n    for fold, (train_idx, val_idx) in enumerate(cv.split(X, y)):\n        \n        X_train = X.iloc[train_idx]\n        y_train = y.iloc[train_idx]\n        \n        X_val = X.iloc[val_idx]\n        y_val = y.iloc[val_idx]\n        \n        val_preds = np.zeros((len(X_val), 4))\n        val_score = 0\n        \n        for i in range(4):\n            model = clone(estimator).fit(X_train, y_train.iloc[:, i])\n            val_preds[:, i] = model.predict(X_val)\n            val_score += accuracy_score(y_val.iloc[:, i], val_preds[:, i]) / 4\n            if show_importance:\n                feature_importances_table['value'] += permutation_importance(model, X_val, y_val.iloc[:, i], random_state = seed, scoring = 'accuracy', n_repeats = 5).importances_mean / 4 / cv.get_n_splits()\n            \n        val_scores.append(val_score)\n    \n    print(f'Val Score: {np.mean(val_scores):.5f} ± {np.std(val_scores):.5f} | {label}')\n    \n    if show_importance:\n        plt.figure(figsize = (20, 50))\n        plt.title('Feature Importances', weight = 'bold', size = 25)\n        sns.barplot(feature_importances_table.sort_values('value', ascending = False).T, orient = 'h', palette = 'viridis')\n        plt.show()\n    \n    return val_scores, feature_importances_table","metadata":{"papermill":{"duration":0.032302,"end_time":"2024-01-29T04:50:31.373358","exception":false,"start_time":"2024-01-29T04:50:31.341056","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:17:43.751476Z","iopub.execute_input":"2024-03-14T15:17:43.752216Z","iopub.status.idle":"2024-03-14T15:17:43.767567Z","shell.execute_reply.started":"2024-03-14T15:17:43.752174Z","shell.execute_reply":"2024-03-14T15:17:43.766477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def cross_val_score_proba(estimator, col, cv = kf, label = ''):\n    \n    val_scores = []\n    val_predictions = np.zeros((len(X), y.loc[:, col].nunique()))\n    test_predictions = np.zeros((len(test_pd), y.loc[:, col].nunique()))\n    \n    for fold, (train_idx, val_idx) in enumerate(cv.split(X, y)):\n        \n        X_train = X.iloc[train_idx].reset_index(drop = True)\n        y_train = y.iloc[train_idx].reset_index(drop = True)\n        \n        X_val = X.iloc[val_idx].reset_index(drop = True)\n        y_val = y.iloc[val_idx].reset_index(drop = True)\n        \n        model = clone(estimator).fit(X_train, y_train[col])\n        val_preds = model.predict_proba(X_val)\n        val_score = accuracy_score(y_val[col], val_preds.argmax(axis = 1))\n            \n        val_scores.append(val_score)\n        val_predictions[val_idx] = val_preds\n        test_predictions += model.predict_proba(test_pd) / cv.get_n_splits()\n        \n        #print(f'Fold Score: {val_score:.5f}')\n    \n    print(f'Val Score: {np.mean(val_scores):.5f} ± {np.std(val_scores):.5f} | {label}')\n    \n    return val_scores, val_predictions, test_predictions","metadata":{"papermill":{"duration":0.026822,"end_time":"2024-01-29T04:50:31.413927","exception":false,"start_time":"2024-01-29T04:50:31.387105","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:17:43.768982Z","iopub.execute_input":"2024-03-14T15:17:43.769337Z","iopub.status.idle":"2024-03-14T15:17:43.784667Z","shell.execute_reply.started":"2024-03-14T15:17:43.769307Z","shell.execute_reply":"2024-03-14T15:17:43.783471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof_list = {col : [] for col in list(y)}\npredict_list = {col : [] for col in list(y)}","metadata":{"papermill":{"duration":0.022826,"end_time":"2024-01-29T04:50:31.450679","exception":false,"start_time":"2024-01-29T04:50:31.427853","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:17:43.786114Z","iopub.execute_input":"2024-03-14T15:17:43.786482Z","iopub.status.idle":"2024-03-14T15:17:43.799828Z","shell.execute_reply.started":"2024-03-14T15:17:43.786452Z","shell.execute_reply":"2024-03-14T15:17:43.798693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Logistic Regression","metadata":{"papermill":{"duration":0.013066,"end_time":"2024-01-29T04:50:31.477414","exception":false,"start_time":"2024-01-29T04:50:31.464348","status":"completed"},"tags":[]}},{"cell_type":"code","source":"log_models = {}\n\nfor col in list(y):\n    log_models[col] = make_pipeline(\n        MEstimateEncoder(cols = categorical_features),\n        SimpleImputer(),\n        StandardScaler(),\n        PowerTransformer(),\n        LogisticRegression(random_state = seed, max_iter = 10000000)\n    )\n    _, oof_preds, _ = cross_val_score_proba(log_models[col], col)\n    oof_list[col].append(oof_preds)\n    \nprint('')","metadata":{"papermill":{"duration":35.58775,"end_time":"2024-01-29T04:51:07.07856","exception":false,"start_time":"2024-01-29T04:50:31.49081","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:17:43.801332Z","iopub.execute_input":"2024-03-14T15:17:43.801695Z","iopub.status.idle":"2024-03-14T15:18:27.517738Z","shell.execute_reply.started":"2024-03-14T15:17:43.801662Z","shell.execute_reply":"2024-03-14T15:18:27.516138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# SVM","metadata":{"papermill":{"duration":0.022879,"end_time":"2024-01-29T04:51:07.125295","exception":false,"start_time":"2024-01-29T04:51:07.102416","status":"completed"},"tags":[]}},{"cell_type":"code","source":"svm_models = {}\n\nfor col in list(y):\n    svm_models[col] = make_pipeline(\n        MEstimateEncoder(cols = categorical_features),\n        SimpleImputer(),\n        StandardScaler(),\n        PowerTransformer(),\n        SVC(random_state = seed, probability = True, max_iter = 1000000)\n    )\n    _, oof_preds, _ = cross_val_score_proba(svm_models[col], col)\n    oof_list[col].append(oof_preds)\n    \nprint('')","metadata":{"papermill":{"duration":222.322245,"end_time":"2024-01-29T04:54:49.47075","exception":false,"start_time":"2024-01-29T04:51:07.148505","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:18:27.520333Z","iopub.execute_input":"2024-03-14T15:18:27.521718Z","iopub.status.idle":"2024-03-14T15:22:59.697532Z","shell.execute_reply.started":"2024-03-14T15:18:27.521656Z","shell.execute_reply":"2024-03-14T15:22:59.696607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# LightGBM","metadata":{"papermill":{"duration":0.014521,"end_time":"2024-01-29T04:54:49.49914","exception":false,"start_time":"2024-01-29T04:54:49.484619","status":"completed"},"tags":[]}},{"cell_type":"code","source":"lgb_models = {}\n\nfor col in list(y):\n    lgb_models[col] = make_pipeline(\n        MEstimateEncoder(cols = categorical_features),\n        SimpleImputer(),\n        LGBMClassifier(random_state = seed)\n    )\n    _, oof_preds, _ = cross_val_score_proba(lgb_models[col], col)\n    oof_list[col].append(oof_preds)\n    \nprint('')","metadata":{"papermill":{"duration":108.777442,"end_time":"2024-01-29T04:56:38.368714","exception":false,"start_time":"2024-01-29T04:54:49.591272","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:22:59.698848Z","iopub.execute_input":"2024-03-14T15:22:59.699984Z","iopub.status.idle":"2024-03-14T15:24:52.906261Z","shell.execute_reply.started":"2024-03-14T15:22:59.699950Z","shell.execute_reply":"2024-03-14T15:24:52.905087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CatBoost","metadata":{"papermill":{"duration":0.014134,"end_time":"2024-01-29T04:56:38.397087","exception":false,"start_time":"2024-01-29T04:56:38.382953","status":"completed"},"tags":[]}},{"cell_type":"code","source":"cb_models = {}\n\nfor col in list(y):\n    cb_models[col] = make_pipeline(\n        NumericalImputer(),\n        CatBoostClassifier(random_state = seed, verbose = 0, cat_features = categorical_features)\n    )\n    _, oof_preds, _ = cross_val_score_proba(cb_models[col], col)\n    oof_list[col].append(oof_preds)\n\nprint('')","metadata":{"papermill":{"duration":2203.741797,"end_time":"2024-01-29T05:33:22.153384","exception":false,"start_time":"2024-01-29T04:56:38.411587","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T15:24:52.907395Z","iopub.execute_input":"2024-03-14T15:24:52.907730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cb_bayes_models = {}\n\nfor col in list(y):\n    cb_bayes_models[col] = make_pipeline(\n        NumericalImputer(),\n        CatBoostClassifier(random_state = seed, verbose = 0, bootstrap_type = 'Bayesian', cat_features = categorical_features)\n    )\n    _, oof_preds, _ = cross_val_score_proba(cb_bayes_models[col], col)\n    oof_list[col].append(oof_preds)\n    \nprint('')","metadata":{"papermill":{"duration":2181.626697,"end_time":"2024-01-29T06:09:43.794465","exception":false,"start_time":"2024-01-29T05:33:22.167768","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cb_bernoulli_models = {}\n\nfor col in list(y):\n    cb_bernoulli_models[col] = make_pipeline(\n        NumericalImputer(),\n        CatBoostClassifier(random_state = seed, verbose = 0, bootstrap_type = 'Bernoulli', cat_features = categorical_features)\n    )\n    _, oof_preds, _ = cross_val_score_proba(cb_bernoulli_models[col], col)\n    oof_list[col].append(oof_preds)\n    \nprint('')","metadata":{"papermill":{"duration":2013.17832,"end_time":"2024-01-29T06:43:16.987367","exception":false,"start_time":"2024-01-29T06:09:43.809047","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# XGBoost","metadata":{"papermill":{"duration":0.014571,"end_time":"2024-01-29T06:43:17.016813","exception":false,"start_time":"2024-01-29T06:43:17.002242","status":"completed"},"tags":[]}},{"cell_type":"code","source":"xgb_models = {}\n\nfor col in list(y):\n    xgb_models[col] = make_pipeline(\n        MEstimateEncoder(cols = categorical_features),\n        SimpleImputer(),\n        XGBClassifier(random_state = seed, tree_method = 'hist')\n    )\n    _, oof_preds, _ = cross_val_score_proba(xgb_models[col], col)\n    oof_list[col].append(oof_preds)\n    \nprint('')","metadata":{"papermill":{"duration":133.699844,"end_time":"2024-01-29T06:45:30.812195","exception":false,"start_time":"2024-01-29T06:43:17.112351","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Ensemble NN","metadata":{"papermill":{"duration":0.014948,"end_time":"2024-01-29T06:45:30.842435","exception":false,"start_time":"2024-01-29T06:45:30.827487","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def cross_val_score_nn(estimator, col, cv = kf, label = ''):\n    \n    X_oof = oof_list[col]\n        \n    model = clone(estimator).fit(X_oof, y[col])\n    val_preds = model.predict_proba(X_oof)\n    val_score = accuracy_score(y[col], val_preds.argmax(axis = 1))\n    \n    print(f'Val Score: {val_score:.5f} | {label}')\n    \n    return val_score","metadata":{"papermill":{"duration":0.025169,"end_time":"2024-01-29T06:45:30.88277","exception":false,"start_time":"2024-01-29T06:45:30.857601","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class TensorFlower(BaseEstimator, ClassifierMixin):\n    \n    def fit(self, x, y):\n        \n        tf.keras.utils.set_random_seed(seed)\n        tf.config.experimental.enable_op_determinism()\n        \n        inputs = []\n        norm_inputs = []\n        \n        multiply = []\n        \n        for i in range(len(x)):\n            inputs.append(tf.keras.Input((len(np.unique(y)), )))\n            norm_inputs.append(tf.keras.layers.BatchNormalization()(inputs[i]))\n            \n        concat = tf.keras.layers.Concatenate()(norm_inputs)\n        \n        for i in range(len(x)):\n            dense = tf.keras.layers.Dense(len(np.unique(y)))(concat)\n            norm = tf.keras.layers.BatchNormalization()(dense)\n            act = tf.keras.activations.softmax(norm)\n            multiply.append(tf.keras.layers.Multiply()([inputs[i], act]))\n            \n        outputs = tf.keras.layers.Add()(multiply)\n        \n        self.model = tf.keras.Model(inputs, outputs)\n        self.model.compile(loss = 'sparse_categorical_crossentropy', optimizer = tf.keras.optimizers.AdamW(1e-4))\n        \n        self.model.fit(x, y, epochs = 50, verbose = 0)\n        \n        return self\n    \n    def predict_proba(self, x):\n        \n        return self.model.predict(x, verbose = 0)\n    \n    def predict(self, x):\n        \n        return np.argmax(self.predict_proba(x), axis = 1)","metadata":{"papermill":{"duration":0.029221,"end_time":"2024-01-29T06:45:30.927107","exception":false,"start_time":"2024-01-29T06:45:30.897886","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in list(y):\n    cross_val_score_nn(TensorFlower(), col)\n    \nprint('\\n')","metadata":{"papermill":{"duration":163.27622,"end_time":"2024-01-29T06:48:14.218501","exception":false,"start_time":"2024-01-29T06:45:30.942281","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction and Submission","metadata":{"papermill":{"duration":0.015352,"end_time":"2024-01-29T06:48:14.322618","exception":false,"start_time":"2024-01-29T06:48:14.307266","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# NOTES: final section used all ML algorithm trained before to predict results about test dataset. \n# Finally all algorithms has been grouped in ensembler model to increase performance.","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in list(y):\n    predict_list[col].append(log_models[col].fit(X, y[col]).predict_proba(test_pd))\n    predict_list[col].append(svm_models[col].fit(X, y[col]).predict_proba(test_pd))\n    predict_list[col].append(lgb_models[col].fit(X, y[col]).predict_proba(test_pd))\n    predict_list[col].append(cb_models[col].fit(X, y[col]).predict_proba(test_pd))\n    predict_list[col].append(cb_bayes_models[col].fit(X, y[col]).predict_proba(test_pd))\n    predict_list[col].append(cb_bernoulli_models[col].fit(X, y[col]).predict_proba(test_pd))\n    predict_list[col].append(xgb_models[col].fit(X, y[col]).predict_proba(test_pd))","metadata":{"papermill":{"duration":1492.512251,"end_time":"2024-01-29T07:13:06.850299","exception":false,"start_time":"2024-01-29T06:48:14.338048","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in list(y):\n    ensembler = TensorFlower().fit(oof_list[col], y[col])\n    submission[col] = ensembler.predict(predict_list[col])\n    submission[col] = le[col].inverse_transform(submission[col].astype(np.int8))\n\nsubmission[['id_unique'] + list(y)].to_csv('cool_city.csv', index = False)","metadata":{"papermill":{"duration":173.734399,"end_time":"2024-01-29T07:16:00.601369","exception":false,"start_time":"2024-01-29T07:13:06.86697","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Citation\n\n- Clayton Miller, Matias Quintana, Mario Frei, Yun Xuan Chua, Chun Fu, Bianca Picchetti, Winston Yap, Adrian Chong, and Filip Biljecki. 2023. Introducing the Cool, Quiet City Competition: Predicting Smartwatch-Reported Heat and Noise with Digital Twin Metrics. In Proceedings of the 10th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys '23). Association for Computing Machinery, New York, NY, USA, 298–299. https://doi.org/10.1145/3600100.3626269","metadata":{"papermill":{"duration":0.015561,"end_time":"2024-01-29T07:16:00.632974","exception":false,"start_time":"2024-01-29T07:16:00.617413","status":"completed"},"tags":[]}}]}