{"metadata":{"kaggle":{"accelerator":"none","dataSources":[{"sourceId":61145,"databundleVersionId":7047669,"sourceType":"competition"}],"dockerImageVersionId":30646,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"papermill":{"default_parameters":{},"duration":8816.051734,"end_time":"2024-01-29T07:16:03.940045","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-01-29T04:49:07.888311","version":"2.4.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# STUDENT: Wilfredo Cubas Pajuelo - 20185238\n# TASK 03\n\nThis notebook was created and modified to participate in a Kaggle competition - \"Cool, Quiet City Competition\". In this opportunity, we have taken as a guide, the code of the participant who came in second place. The code will be explained block by block and some sections will be commented at the end of the notebook.\n\nI will follow a similar format to the one I followed in task 02. As I commented previously, I have already followed some AI courses in which we were taught how to make predictions, so some comments will be my own knowledge, and others will be support from forums and ChatGPT.","metadata":{}},{"cell_type":"markdown","source":"# Loading Libraries and Datasets","metadata":{"papermill":{"duration":0.013798,"end_time":"2024-01-29T04:49:11.324968","exception":false,"start_time":"2024-01-29T04:49:11.31117","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install -q iterative-stratification\n\n# This line of code installs the Python package \"iterative-stratification\". Once installed, \n# this package can be imported and used in Python scripts for tasks related to iterative stratification,\n# which is a technique commonly used in machine learning for splitting datasets into train and test\n# sets while preserving class distributions.","metadata":{"papermill":{"duration":13.360127,"end_time":"2024-01-29T04:49:24.69885","exception":false,"start_time":"2024-01-29T04:49:11.338723","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:18:00.876343Z","iopub.execute_input":"2024-03-14T21:18:00.876797Z","iopub.status.idle":"2024-03-14T21:18:15.648513Z","shell.execute_reply.started":"2024-03-14T21:18:00.876764Z","shell.execute_reply":"2024-03-14T21:18:15.647162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport polars as pl\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport tensorflow as tf\nimport os\nimport optuna\nimport holidays\nimport copy\n\nfrom category_encoders import OneHotEncoder, MEstimateEncoder, BinaryEncoder\nfrom iterstrat.ml_stratifiers import MultilabelStratifiedKFold, RepeatedMultilabelStratifiedKFold\nfrom sklearn.cluster import KMeans\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.model_selection import StratifiedKFold, RepeatedStratifiedKFold, TimeSeriesSplit\nfrom sklearn.decomposition import PCA\nfrom sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier\nfrom sklearn.ensemble import GradientBoostingClassifier, HistGradientBoostingClassifier\nfrom sklearn.ensemble import VotingClassifier\nfrom sklearn.impute import KNNImputer, SimpleImputer\nfrom sklearn.metrics import accuracy_score, roc_auc_score, roc_curve, make_scorer\nfrom sklearn.metrics.pairwise import euclidean_distances\nfrom sklearn.naive_bayes import GaussianNB, BernoulliNB\nfrom sklearn.pipeline import make_pipeline, Pipeline\nfrom sklearn.base import BaseEstimator, TransformerMixin, ClassifierMixin, clone\nfrom sklearn.multioutput import ClassifierChain\nfrom sklearn.preprocessing import FunctionTransformer, StandardScaler, PowerTransformer, LabelEncoder\nfrom sklearn.linear_model import LogisticRegression, RidgeClassifier\nfrom sklearn.svm import SVC\nfrom sklearn.discriminant_analysis import LinearDiscriminantAnalysis\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.inspection import permutation_importance\nfrom scipy.cluster.hierarchy import dendrogram, linkage\nfrom scipy.spatial.distance import squareform\nfrom xgboost import XGBClassifier, XGBRFClassifier\nfrom lightgbm import LGBMClassifier\nfrom sklearn.feature_selection import SequentialFeatureSelector\n\nsns.set_theme(style = 'white', palette = 'viridis')\npal = sns.color_palette('viridis')\n\npd.set_option('display.max_rows', 100)\n\n# This code snippet imports essential Python libraries and modules commonly utilized in data analysis,\n# machine learning, and visualization tasks. It covers a wide range of functionalities, including data\n# manipulation, visualization, model building, hyperparameter optimization, and evaluation.","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":19.724847,"end_time":"2024-01-29T04:49:44.43698","exception":false,"start_time":"2024-01-29T04:49:24.712133","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:18:23.206007Z","iopub.execute_input":"2024-03-14T21:18:23.206424Z","iopub.status.idle":"2024-03-14T21:18:23.226582Z","shell.execute_reply.started":"2024-03-14T21:18:23.206390Z","shell.execute_reply":"2024-03-14T21:18:23.225451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preprocessor = [\n    pl.col('dT').cast(pl.Float32),\n    \n    pl.col('ts_oxygen_saturation').cast(pl.Float32),\n    pl.col('ts_resting_heart_rate').cast(pl.Float32),\n    \n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z'),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.year().alias('time_year').cast(pl.UInt16),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.month().alias('time_month').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.week().alias('time_week').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.weekday().alias('time_weekday').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.ordinal_day().alias('time_day_of_year').cast(pl.UInt16),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.day().alias('time_day').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.hour().alias('time_hour').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.minute().alias('time_minute').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.second().alias('time_second').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.microsecond().alias('time_micro').cast(pl.UInt32),\n    \n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z'),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.year().alias('loc_year').cast(pl.UInt16),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.month().alias('loc_month').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.week().alias('loc_week').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.weekday().alias('loc_weekday').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.ordinal_day().alias('loc_day_of_year').cast(pl.UInt16),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.day().alias('loc_day').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.hour().alias('loc_hour').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.minute().alias('loc_minute').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.second().alias('loc_second').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.microsecond().alias('loc_micro').cast(pl.UInt32),\n    \n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z'),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.year().alias('start_year').cast(pl.UInt16),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.month().alias('start_month').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.week().alias('start_week').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.weekday().alias('start_weekday').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.ordinal_day().alias('start_day_of_year').cast(pl.UInt16),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.day().alias('start_day').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.hour().alias('start_hour').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.minute().alias('start_minute').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.second().alias('start_second').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.microsecond().alias('start_micro').cast(pl.UInt32)\n]\n\npreprocessor_2 = [\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z'),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.year().alias('year').cast(pl.UInt16),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.month().alias('month').cast(pl.UInt8),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.day().alias('day').cast(pl.UInt8),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.hour().alias('hour').cast(pl.UInt8),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.minute().alias('minute').cast(pl.UInt8),\n    \n    pl.col('value').cast(pl.Float32)\n]\n\nduration_calculator = [\n    (pl.duration(days = pl.col('time_day_of_year'), hours = pl.col('time_hour'), minutes = pl.col('time_minute'), seconds = pl.col('time_second'), microseconds = pl.col('time_micro')) -\n    pl.duration(days = pl.col('start_day_of_year'), hours = pl.col('start_hour'), minutes = pl.col('start_minute'), seconds = pl.col('start_second'), microseconds = pl.col('start_micro'))).dt.total_microseconds().alias('time_start_difference'),\n    (pl.duration(days = pl.col('time_day_of_year'), hours = pl.col('time_hour'), minutes = pl.col('time_minute'), seconds = pl.col('time_second'), microseconds = pl.col('time_micro')) -\n    pl.duration(days = pl.col('loc_day_of_year'), hours = pl.col('loc_hour'), minutes = pl.col('loc_minute'), seconds = pl.col('loc_second'), microseconds = pl.col('loc_micro'))).dt.total_microseconds().alias('time_loc_difference'),\n    (pl.duration(days = pl.col('loc_day_of_year'), hours = pl.col('loc_hour'), minutes = pl.col('loc_minute'), seconds = pl.col('loc_second'), microseconds = pl.col('loc_micro')) -\n     pl.duration(days = pl.col('start_day_of_year'), hours = pl.col('start_hour'), minutes = pl.col('start_minute'), seconds = pl.col('start_second'), microseconds = pl.col('start_micro'))).dt.total_microseconds().alias('start_loc_difference')\n]\n\n# These lists of expressions are likely used to preprocess datetime and numerical data in a Polars dataframe,\n# preparing it for further analysis or machine learning tasks.","metadata":{"papermill":{"duration":0.064479,"end_time":"2024-01-29T04:49:44.521031","exception":false,"start_time":"2024-01-29T04:49:44.456552","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:18:27.038060Z","iopub.execute_input":"2024-03-14T21:18:27.039106Z","iopub.status.idle":"2024-03-14T21:18:27.084655Z","shell.execute_reply.started":"2024-03-14T21:18:27.039058Z","shell.execute_reply":"2024-03-14T21:18:27.082942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/cozie_responses_and_physiological_data_training.csv').\\\n        with_columns(preprocessor).with_columns(duration_calculator)\ntest = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/cozie_responses_and_physiological_data_test_public.csv').\\\n       with_columns(preprocessor).with_columns(duration_calculator)\n\n# This code efficiently reads and preprocesses the training and test data, preparing them for further analysis or machine learning tasks.","metadata":{"papermill":{"duration":0.082503,"end_time":"2024-01-29T04:49:44.616906","exception":false,"start_time":"2024-01-29T04:49:44.534403","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:18:31.372218Z","iopub.execute_input":"2024-03-14T21:18:31.372877Z","iopub.status.idle":"2024-03-14T21:18:31.384576Z","shell.execute_reply.started":"2024-03-14T21:18:31.372835Z","shell.execute_reply":"2024-03-14T21:18:31.383537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"air_temp = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_air-temperature.csv').\\\n           melt('').\\\n           with_columns(preprocessor_2).\\\n           group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n           agg(\n               pl.col('value').mean().alias('mean_air_temp'),\n               pl.col('value').std().alias('std_air_temp'),\n           )\nrainfall = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_rainfall.csv').\\\n           melt('').\\\n           with_columns(preprocessor_2).\\\n           group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n           agg(\n               pl.col('value').mean().alias('mean_rainfall'),\n               pl.col('value').std().alias('std_rainfall'),\n           )\nhumidity = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_relative-humidity.csv').\\\n           melt('').\\\n           with_columns(preprocessor_2).\\\n           group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n           agg(\n               pl.col('value').mean().alias('mean_humidity'),\n               pl.col('value').std().alias('std_humidity'),\n           )\nstations = pl.read_csv('/kaggle/input/cool-quiet-city-competition/weather_stations.csv')\nwind_direction = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_wind-direction.csv').\\\n                 melt('').\\\n                 with_columns(preprocessor_2).\\\n                 group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n                 agg(\n                     pl.col('value').mean().alias('mean_wind_direction'),\n                     pl.col('value').std().alias('std_wind_direction')\n                 )\nwind_speed = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_wind-speed.csv').\\\n             melt('').\\\n             with_columns(preprocessor_2).\\\n             group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n             agg(\n                 pl.col('value').mean().alias('mean_wind_speed'),\n                 pl.col('value').std().alias('std_wind_speed'),\n             )\n\n# This code efficiently reads weather-related data from CSV files, converts it into a suitable format, and\n# aggregates it based on specific time intervals, making it ready for further analysis or integration with\n# other datasets.","metadata":{"papermill":{"duration":0.102406,"end_time":"2024-01-29T04:49:44.732804","exception":false,"start_time":"2024-01-29T04:49:44.630398","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:18:33.699665Z","iopub.execute_input":"2024-03-14T21:18:33.700076Z","iopub.status.idle":"2024-03-14T21:18:33.724658Z","shell.execute_reply.started":"2024-03-14T21:18:33.700030Z","shell.execute_reply":"2024-03-14T21:18:33.723520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset Information","metadata":{"papermill":{"duration":0.01312,"end_time":"2024-01-29T04:49:44.759911","exception":false,"start_time":"2024-01-29T04:49:44.746791","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def desc_dataset_pl(df):\n    return pl.concat([\n        pl.DataFrame({'key' : df.columns, 'dataset_length' : len(df)}),\n        df.null_count().transpose().rename({'column_0' : 'null'}),\n        df.approx_n_unique().transpose().rename({'column_0' : 'nunique'}),\n        df.mean().transpose().rename({'column_0' : 'mean'}),\n        df.std().transpose().rename({'column_0' : 'std'}),\n        df.median().transpose().rename({'column_0' : 'median'}),\n        df.min().transpose().rename({'column_0' : 'min'}),\n        df.max().transpose().rename({'column_0' : 'max'})\n    ], how = 'horizontal').with_columns(\n        (pl.col('dataset_length') - pl.col('null')).alias('count'),\n        (pl.col('null') / pl.col('dataset_length') * 100).alias('%null'),\n        (pl.col('nunique') / pl.col('dataset_length') * 100).alias('%unique')\n    ).drop('dataset_length').to_pandas()\n\n# This function provides a concise and informative summary of the dataset's characteristics, which can be\n# useful for exploratory data analysis and understanding the data's distribution and quality.","metadata":{"papermill":{"duration":0.025358,"end_time":"2024-01-29T04:49:44.798669","exception":false,"start_time":"2024-01-29T04:49:44.773311","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:18:36.399748Z","iopub.execute_input":"2024-03-14T21:18:36.400380Z","iopub.status.idle":"2024-03-14T21:18:36.409970Z","shell.execute_reply.started":"2024-03-14T21:18:36.400347Z","shell.execute_reply":"2024-03-14T21:18:36.408950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#desc_dataset_pl(train)","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:44.827491Z","iopub.status.busy":"2024-01-29T04:49:44.82704Z","iopub.status.idle":"2024-01-29T04:49:44.832046Z","shell.execute_reply":"2024-01-29T04:49:44.830934Z"},"papermill":{"duration":0.0217,"end_time":"2024-01-29T04:49:44.834261","exception":false,"start_time":"2024-01-29T04:49:44.812561","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#desc_dataset_pl(test)","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:44.862552Z","iopub.status.busy":"2024-01-29T04:49:44.862102Z","iopub.status.idle":"2024-01-29T04:49:44.867229Z","shell.execute_reply":"2024-01-29T04:49:44.866164Z"},"papermill":{"duration":0.021945,"end_time":"2024-01-29T04:49:44.869571","exception":false,"start_time":"2024-01-29T04:49:44.847626","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TS Features Aggregation","metadata":{"papermill":{"duration":0.013129,"end_time":"2024-01-29T04:49:44.960247","exception":false,"start_time":"2024-01-29T04:49:44.947118","status":"completed"},"tags":[]}},{"cell_type":"code","source":"df_all = pl.concat([train, test], how = 'vertical')\n\n# Data are concatenated to facilitate manipulation and analysis of the combined data set.","metadata":{"papermill":{"duration":0.025587,"end_time":"2024-01-29T04:49:44.999204","exception":false,"start_time":"2024-01-29T04:49:44.973617","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:18:39.283779Z","iopub.execute_input":"2024-03-14T21:18:39.284152Z","iopub.status.idle":"2024-03-14T21:18:39.290337Z","shell.execute_reply.started":"2024-03-14T21:18:39.284124Z","shell.execute_reply":"2024-03-14T21:18:39.288951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"survey_counter_day = df_all.group_by(\n    ['id_participant', 'time_year', 'time_month', 'time_day']\n).agg(\n    pl.col('id_participant').count().alias('survey_count_per_day').cast(pl.UInt16)\n)\nsurvey_counter_hour = df_all.group_by(\n    ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour']\n).agg(\n    pl.col('id_participant').count().alias('survey_count_per_hour').cast(pl.UInt16)\n)\nsurvey_counter_minute = df_all.group_by(\n    ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour', 'time_minute']\n).agg(\n    pl.col('id_participant').count().alias('survey_count_per_minute').cast(pl.UInt8)\n)\n\n# These calculations provide insights into the frequency of survey responses at different\n# time intervals for each participant, facilitating temporal analysis of the survey data.","metadata":{"papermill":{"duration":0.025501,"end_time":"2024-01-29T04:49:45.038174","exception":false,"start_time":"2024-01-29T04:49:45.012673","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:18:41.557035Z","iopub.execute_input":"2024-03-14T21:18:41.557435Z","iopub.status.idle":"2024-03-14T21:18:41.567036Z","shell.execute_reply.started":"2024-03-14T21:18:41.557402Z","shell.execute_reply":"2024-03-14T21:18:41.565795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rest_heart_rate_agg = df_all.group_by(\n    ['time_year', 'time_week', 'id_participant']\n).agg(\n    pl.col('ts_resting_heart_rate').mean().alias('mean_resting_heart_rate'),\n    pl.col('ts_resting_heart_rate').std().alias('std_resting_heart_rate')\n)\noxygen_saturation_agg = df_all.group_by(\n    ['time_year', 'time_week', 'id_participant']\n).agg(\n    pl.col('ts_oxygen_saturation').mean().alias('mean_oxygen_saturation'),\n    pl.col('ts_oxygen_saturation').std().alias('std_oxygen_saturation')\n)\nstand_time_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_stand_time').mean().alias('mean_stand_time'),\n    pl.col('ts_stand_time').std().alias('std_stand_time')\n)\nstep_count_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_step_count').mean().alias('mean_step_count'),\n    pl.col('ts_step_count').std().alias('std_step_count')\n)\nwalking_distance_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_walking_distance').mean().alias('mean_walking_distance'),\n    pl.col('ts_walking_distance').std().alias('std_walking_distance')\n)\naudio_exposure_environment_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_audio_exposure_environment').mean().alias('mean_audio_exposure_environment'),\n    pl.col('ts_audio_exposure_environment').std().alias('std_audio_exposure_environment')\n)\nheart_rate_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_heart_rate').mean().alias('mean_heart_rate'),\n    pl.col('ts_heart_rate').std().alias('std_heart_rate')\n)\n\n# These calculations provide aggregated statistics for each physiological and activity-related\n# feature over different time intervals, facilitating analysis and interpretation of trends in\n# the data.","metadata":{"papermill":{"duration":0.030034,"end_time":"2024-01-29T04:49:45.081786","exception":false,"start_time":"2024-01-29T04:49:45.051752","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:18:44.639746Z","iopub.execute_input":"2024-03-14T21:18:44.640157Z","iopub.status.idle":"2024-03-14T21:18:44.656100Z","shell.execute_reply.started":"2024-03-14T21:18:44.640127Z","shell.execute_reply":"2024-03-14T21:18:44.655040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.join(\n    rest_heart_rate_agg, on = ['time_year', 'time_week', 'id_participant'], how = 'left'\n).join(\n    oxygen_saturation_agg, on = ['time_year', 'time_week', 'id_participant'], how = 'left'\n).join(\n    stand_time_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    step_count_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    walking_distance_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    audio_exposure_environment_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    heart_rate_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    survey_counter_day, on = ['id_participant', 'time_year', 'time_month', 'time_day'], how = 'left'\n).join(\n    survey_counter_hour, on = ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour'], how = 'left'\n).join(\n    survey_counter_minute, on = ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'], how = 'left'\n)\ntest = test.join(\n    rest_heart_rate_agg, on = ['time_year', 'time_week', 'id_participant'], how = 'left'\n).join(\n    oxygen_saturation_agg, on = ['time_year', 'time_week', 'id_participant'], how = 'left'\n).join(\n    stand_time_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    step_count_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    walking_distance_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    audio_exposure_environment_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    heart_rate_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    survey_counter_day, on = ['id_participant', 'time_year', 'time_month', 'time_day'], how = 'left'\n).join(\n    survey_counter_hour, on = ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour'], how = 'left'\n).join(\n    survey_counter_minute, on = ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'], how = 'left'\n)\n\n# These join operations merge the aggregated statistics with the original datasets, providing\n# additional features for analysis and modeling. The left joins ensure that all rows from the original\n# datasets are retained, even if there are no matching rows in the aggregated statistics datasets.","metadata":{"papermill":{"duration":0.040233,"end_time":"2024-01-29T04:49:45.135455","exception":false,"start_time":"2024-01-29T04:49:45.095222","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:18:48.012599Z","iopub.execute_input":"2024-03-14T21:18:48.013020Z","iopub.status.idle":"2024-03-14T21:18:48.043139Z","shell.execute_reply.started":"2024-03-14T21:18:48.012987Z","shell.execute_reply":"2024-03-14T21:18:48.041732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Station and Weather","metadata":{"papermill":{"duration":0.012909,"end_time":"2024-01-29T04:49:45.16168","exception":false,"start_time":"2024-01-29T04:49:45.148771","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def find_closest_station(x):\n    if x[0] == None: return None\n    latitude = float(x[0][:18])\n    longitude = float(x[0][19:])\n    return stations.row(stations.select(\n        distance = (pl.col('latitude') - latitude) ** 2 + (pl.col('longitude') - longitude) ** 2\n    )['distance'].arg_min())[1]\n\n# This function efficiently finds the closest weather station to a given set of coordinates by computing\n# the distances and selecting the station with the minimum distance.","metadata":{"papermill":{"duration":0.022635,"end_time":"2024-01-29T04:49:45.197578","exception":false,"start_time":"2024-01-29T04:49:45.174943","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:18:51.718287Z","iopub.execute_input":"2024-03-14T21:18:51.718801Z","iopub.status.idle":"2024-03-14T21:18:51.727897Z","shell.execute_reply.started":"2024-03-14T21:18:51.718761Z","shell.execute_reply":"2024-03-14T21:18:51.726201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def map_station(df):\n    return df.select(pl.col('ws_latitude').cast(pl.Utf8).map_elements(\n        lambda x: x + '0' * (18 - len(x))\n    ) + ' ' + pl.col('ws_longitude').cast(pl.Utf8).map_elements(\n        lambda x: x + '0' * (18 - len(x))\n    )).map_rows(find_closest_station)['map']\n\n# This function efficiently maps latitude and longitude coordinates to the closest weather station using\n# the find_closest_station function.","metadata":{"papermill":{"duration":0.02323,"end_time":"2024-01-29T04:49:45.234313","exception":false,"start_time":"2024-01-29T04:49:45.211083","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:18:54.531572Z","iopub.execute_input":"2024-03-14T21:18:54.531966Z","iopub.status.idle":"2024-03-14T21:18:54.538987Z","shell.execute_reply.started":"2024-03-14T21:18:54.531935Z","shell.execute_reply":"2024-03-14T21:18:54.537791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.join(\n    air_temp,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    rainfall,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    humidity,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    wind_direction,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    wind_speed,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute'])\n\ntest = test.join(\n    air_temp,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    rainfall,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    humidity,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    wind_direction,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    wind_speed,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute'])\n\n# These steps merge weather-related features into the datasets based on the corresponding timestamp information.\n# After each join operation, the columns representing year, month, day, hour, and minute from the weather-related\n# dataframes are dropped as they are no longer needed. This process ensures that the datasets are enriched with\n# weather information for further analysis or modeling.","metadata":{"papermill":{"duration":0.04683,"end_time":"2024-01-29T04:49:45.295134","exception":false,"start_time":"2024-01-29T04:49:45.248304","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:18:56.678653Z","iopub.execute_input":"2024-03-14T21:18:56.679461Z","iopub.status.idle":"2024-03-14T21:18:56.714197Z","shell.execute_reply.started":"2024-03-14T21:18:56.679429Z","shell.execute_reply":"2024-03-14T21:18:56.713265Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Correlation","metadata":{"papermill":{"duration":0.01345,"end_time":"2024-01-29T04:49:45.322014","exception":false,"start_time":"2024-01-29T04:49:45.308564","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def heatmap(dataset, label = None):\n    corr = dataset.corr(method = 'pearson')\n    plt.figure(figsize = (45, 45), dpi = 300)\n    mask = np.zeros_like(corr)\n    mask[np.triu_indices_from(mask)] = True\n    sns.heatmap(corr, mask = mask, cmap = 'viridis', annot = True, annot_kws = {'size' : 11})\n    plt.title(f'{label} Dataset Correlation Matrix\\n', fontsize = 25, weight = 'bold')\n    plt.show()\n    \n# This function provides a convenient way to visualize the correlation structure within a dataset, helping\n# to identify patterns and relationships between different features.","metadata":{"papermill":{"duration":0.022915,"end_time":"2024-01-29T04:49:45.358148","exception":false,"start_time":"2024-01-29T04:49:45.335233","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:18:59.235483Z","iopub.execute_input":"2024-03-14T21:18:59.235901Z","iopub.status.idle":"2024-03-14T21:18:59.243518Z","shell.execute_reply.started":"2024-03-14T21:18:59.235870Z","shell.execute_reply":"2024-03-14T21:18:59.242157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preparation","metadata":{"papermill":{"duration":0.013031,"end_time":"2024-01-29T04:49:45.384604","exception":false,"start_time":"2024-01-29T04:49:45.371573","status":"completed"},"tags":[]}},{"cell_type":"code","source":"X = train.filter(pl.col('q_location').is_not_null()).drop('id_unique', 'time', 'ws_timestamp_location', 'ws_timestamp_start', 'ts_oxygen_saturation', 'ts_resting_heart_rate', 'ts_stand_time', 'ts_step_count', 'ts_walking_distance', 'ts_heart_rate', 'ts_audio_exposure_environment').collect()\nX = X.with_columns(stations = map_station(X)).to_pandas()\ny = X[['q_earphones', 'q_noise_kind', 'q_noise_nearby', 'q_thermal_preference']]\nX = X.drop(['q_earphones', 'q_noise_kind', 'q_noise_nearby', 'q_thermal_preference'], axis = 1)\n\nsubmission = test.filter(pl.col('q_location').is_not_null()).drop('q_earphones', 'q_noise_kind', 'q_noise_nearby', 'q_thermal_preference', 'time', 'ws_timestamp_location', 'ws_timestamp_start', 'ts_oxygen_saturation', 'ts_resting_heart_rate', 'ts_stand_time', 'ts_step_count', 'ts_walking_distance', 'ts_heart_rate', 'ts_audio_exposure_environment').collect()\nsubmission = submission.with_columns(stations = map_station(submission)).to_pandas()\ntest_pd = submission.drop('id_unique', axis = 1)\ntest_pd.loc[3178, 'time_loc_difference'] += 1e6 * 3600 * 24 * 365\ntest_pd.loc[3178, 'start_loc_difference'] -= 1e6 * 3600 * 24 * 365\n\ncategorical_features = list(X.select_dtypes('object'))\n\nle = {}\n\nfor col in y.columns:\n    le[col] = LabelEncoder().fit(y[col])\n    y[col] = le[col].transform(y[col])\n    \nacc_scorer = make_scorer(accuracy_score)\nseed = 42\ntf.keras.utils.set_random_seed(seed)\n\nkf = MultilabelStratifiedKFold(n_splits = 5, random_state = seed, shuffle = True)\n#kf = TimeSeriesSplit()\n\n# The code prepares the data, sets up evaluation metrics and cross-validation strategies, and makes necessary\n# configurations for building and evaluating machine learning models for multilabel classification tasks.","metadata":{"papermill":{"duration":45.476571,"end_time":"2024-01-29T04:50:30.874598","exception":false,"start_time":"2024-01-29T04:49:45.398027","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:19:01.977765Z","iopub.execute_input":"2024-03-14T21:19:01.978150Z","iopub.status.idle":"2024-03-14T21:19:50.550533Z","shell.execute_reply.started":"2024-03-14T21:19:01.978122Z","shell.execute_reply":"2024-03-14T21:19:50.549298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering","metadata":{"papermill":{"duration":0.013352,"end_time":"2024-01-29T04:50:30.901792","exception":false,"start_time":"2024-01-29T04:50:30.88844","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class NumericalImputer(BaseEstimator, TransformerMixin):\n    \n    def fit(self, x, y = None):\n        self.imputer = SimpleImputer().fit(x.select_dtypes(['int', 'float']), y)\n        return self\n    \n    def transform(self, x):\n        x_copy = x.copy()\n        x_copy.loc[:, list(x_copy.select_dtypes(['int', 'float']))] = self.imputer.transform(x_copy.loc[:, list(x_copy.select_dtypes(['int', 'float']))])\n        return x_copy\n    \n# This custom transformer class provides a way to impute missing values in numerical columns using scikit-learn's\n# SimpleImputer while maintaining compatibility with scikit-learn's pipeline and workflow.","metadata":{"papermill":{"duration":0.024644,"end_time":"2024-01-29T04:50:31.195005","exception":false,"start_time":"2024-01-29T04:50:31.170361","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:19:56.116570Z","iopub.execute_input":"2024-03-14T21:19:56.116989Z","iopub.status.idle":"2024-03-14T21:19:56.124428Z","shell.execute_reply.started":"2024-03-14T21:19:56.116958Z","shell.execute_reply":"2024-03-14T21:19:56.123620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Adversarial Validation","metadata":{"papermill":{"duration":0.013686,"end_time":"2024-01-29T04:50:31.222264","exception":false,"start_time":"2024-01-29T04:50:31.208578","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#thanks to @carlmcbrideellis\n#https://www.kaggle.com/code/carlmcbrideellis/what-is-adversarial-validation\n\ndef adversarial_validation(dataset_1 = train, dataset_2 = test, label = 'Train-Test'):\n\n    adv_train = dataset_1.copy()\n    adv_test = dataset_2.copy()\n\n    adv_train['is_test'] = 0\n    adv_test['is_test'] = 1\n\n    adv = pd.concat([adv_train, adv_test], ignore_index = True)\n\n    adv_shuffled = adv.sample(frac = 1)\n\n    adv_X = adv_shuffled.drop('is_test', axis = 1)\n    adv_y = adv_shuffled.is_test\n\n    skf = StratifiedKFold(n_splits = 5, random_state = 42, shuffle = True)\n\n    val_scores = []\n    predictions = np.zeros(len(adv))\n\n    for fold, (train_idx, val_idx) in enumerate(skf.split(adv_X, adv_y)):\n    \n        adv_lr = make_pipeline(\n            MEstimateEncoder(cols = list(categorical_features)), \n            #FeatureDropper(cols = ['ws_survey_count', 'id_participant']),\n            SimpleImputer(),\n            XGBClassifier(random_state = seed, tree_method = 'hist')\n        )\n        adv_lr.fit(adv_X.iloc[train_idx], adv_y.iloc[train_idx])\n        \n        val_preds = adv_lr.predict_proba(adv_X.iloc[val_idx])[:, 1]\n        predictions[val_idx] = val_preds\n        val_score = roc_auc_score(adv_y.iloc[val_idx], val_preds)\n        val_scores.append(val_score)\n    \n    fpr, tpr, _ = roc_curve(adv['is_test'], predictions)\n    \n    plt.figure(figsize = (10, 10), dpi = 300)\n    sns.lineplot(x=[0, 1], y=[0, 1], linestyle=\"--\", label=\"Indistinguishable Datasets\")\n    sns.lineplot(x=fpr, y=tpr, label=\"Adversarial Validation Classifier\")\n    plt.title(f'{label} Validation = {np.mean(val_scores):.5f}', weight = 'bold', size = 17)\n    plt.xlabel('False Positive Rate')\n    plt.ylabel('True Positive Rate')\n    plt.show()\n    \n# This function provides a visual assessment of the similarity between two datasets based on their distribution\n# characteristics, particularly useful in scenarios like domain adaptation or dataset shift detection.","metadata":{"papermill":{"duration":0.028635,"end_time":"2024-01-29T04:50:31.264265","exception":false,"start_time":"2024-01-29T04:50:31.23563","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:20:01.411252Z","iopub.execute_input":"2024-03-14T21:20:01.414612Z","iopub.status.idle":"2024-03-14T21:20:01.428830Z","shell.execute_reply.started":"2024-03-14T21:20:01.414568Z","shell.execute_reply":"2024-03-14T21:20:01.427630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#adversarial_validation(X, test_pd)","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:50:31.293053Z","iopub.status.busy":"2024-01-29T04:50:31.292603Z","iopub.status.idle":"2024-01-29T04:50:31.297277Z","shell.execute_reply":"2024-01-29T04:50:31.296116Z"},"papermill":{"duration":0.021817,"end_time":"2024-01-29T04:50:31.299491","exception":false,"start_time":"2024-01-29T04:50:31.277674","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Cross Validation","metadata":{}},{"cell_type":"code","source":"def cross_val_score(estimator, cv = kf, label = '', show_importance = False):\n    \n    val_scores = []\n    \n    feature_importances_table = pd.DataFrame({'value' : 0}, index = list(X.columns))\n    \n    for fold, (train_idx, val_idx) in enumerate(cv.split(X, y)):\n        \n        X_train = X.iloc[train_idx]\n        y_train = y.iloc[train_idx]\n        \n        X_val = X.iloc[val_idx]\n        y_val = y.iloc[val_idx]\n        \n        val_preds = np.zeros((len(X_val), 4))\n        val_score = 0\n        \n        for i in range(4):\n            model = clone(estimator).fit(X_train, y_train.iloc[:, i])\n            val_preds[:, i] = model.predict(X_val)\n            val_score += accuracy_score(y_val.iloc[:, i], val_preds[:, i]) / 4\n            if show_importance:\n                feature_importances_table['value'] += permutation_importance(model, X_val, y_val.iloc[:, i], random_state = seed, scoring = 'accuracy', n_repeats = 5).importances_mean / 4 / cv.get_n_splits()\n            \n        val_scores.append(val_score)\n    \n    print(f'Val Score: {np.mean(val_scores):.5f} ± {np.std(val_scores):.5f} | {label}')\n    \n    if show_importance:\n        plt.figure(figsize = (20, 50))\n        plt.title('Feature Importances', weight = 'bold', size = 25)\n        sns.barplot(feature_importances_table.sort_values('value', ascending = False).T, orient = 'h', palette = 'viridis')\n        plt.show()\n    \n    return val_scores, feature_importances_table\n\n# This function provides a comprehensive evaluation of the model's performance and can also help in understanding\n# the importance of different features in the dataset.","metadata":{"execution":{"iopub.status.busy":"2024-03-14T21:21:18.396316Z","iopub.execute_input":"2024-03-14T21:21:18.396768Z","iopub.status.idle":"2024-03-14T21:21:18.411285Z","shell.execute_reply.started":"2024-03-14T21:21:18.396734Z","shell.execute_reply":"2024-03-14T21:21:18.409812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def cross_val_score_proba(estimator, col, cv = kf, label = ''):\n    \n    val_scores = []\n    val_predictions = np.zeros((len(X), y.loc[:, col].nunique()))\n    test_predictions = np.zeros((len(test_pd), y.loc[:, col].nunique()))\n    \n    for fold, (train_idx, val_idx) in enumerate(cv.split(X, y)):\n        \n        X_train = X.iloc[train_idx].reset_index(drop = True)\n        y_train = y.iloc[train_idx].reset_index(drop = True)\n        \n        X_val = X.iloc[val_idx].reset_index(drop = True)\n        y_val = y.iloc[val_idx].reset_index(drop = True)\n        \n        model = clone(estimator).fit(X_train, y_train[col])\n        val_preds = model.predict_proba(X_val)\n        val_score = accuracy_score(y_val[col], val_preds.argmax(axis = 1))\n            \n        val_scores.append(val_score)\n        val_predictions[val_idx] = val_preds\n        test_predictions += model.predict_proba(test_pd) / cv.get_n_splits()\n        \n        #print(f'Fold Score: {val_score:.5f}')\n    \n    print(f'Val Score: {np.mean(val_scores):.5f} ± {np.std(val_scores):.5f} | {label}')\n    \n    return val_scores, val_predictions, test_predictions\n\n# This function is useful for models that output probabilities for each class label, such as softmax classifiers.\n# It allows for a more detailed analysis of the model's performance beyond simple accuracy scores.","metadata":{"execution":{"iopub.status.busy":"2024-03-14T21:21:36.587894Z","iopub.execute_input":"2024-03-14T21:21:36.588299Z","iopub.status.idle":"2024-03-14T21:21:36.599080Z","shell.execute_reply.started":"2024-03-14T21:21:36.588267Z","shell.execute_reply":"2024-03-14T21:21:36.598121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof_list = {col : [] for col in list(y)}\npredict_list = {col : [] for col in list(y)}\n\n# Both dictionaries allow to organize and store separate predictions for each column of target variables,\n# which can be useful for analysis and evaluation purposes.","metadata":{"execution":{"iopub.status.busy":"2024-03-14T21:21:39.703956Z","iopub.execute_input":"2024-03-14T21:21:39.704387Z","iopub.status.idle":"2024-03-14T21:21:39.710199Z","shell.execute_reply.started":"2024-03-14T21:21:39.704354Z","shell.execute_reply":"2024-03-14T21:21:39.708843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# LightGBM","metadata":{"papermill":{"duration":0.014521,"end_time":"2024-01-29T04:54:49.49914","exception":false,"start_time":"2024-01-29T04:54:49.484619","status":"completed"},"tags":[]}},{"cell_type":"code","source":"lgb_models = {}\n\nfor col in list(y):\n    lgb_models[col] = make_pipeline(\n        MEstimateEncoder(cols = categorical_features),\n        SimpleImputer(),\n        LGBMClassifier(random_state = seed)\n    )\n    _, oof_preds, _ = cross_val_score_proba(lgb_models[col], col)\n    oof_list[col].append(oof_preds)\n    \nprint('')\n\n# This loop efficiently trains LGBM classifiers for each target column in the dataset, utilizing techniques\n# such as mean encoding for categorical features and handling missing values before fitting the model.","metadata":{"papermill":{"duration":108.777442,"end_time":"2024-01-29T04:56:38.368714","exception":false,"start_time":"2024-01-29T04:54:49.591272","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:21:42.159179Z","iopub.execute_input":"2024-03-14T21:21:42.159659Z","iopub.status.idle":"2024-03-14T21:23:34.626713Z","shell.execute_reply.started":"2024-03-14T21:21:42.159519Z","shell.execute_reply":"2024-03-14T21:23:34.625601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# XGBoost","metadata":{"papermill":{"duration":0.014571,"end_time":"2024-01-29T06:43:17.016813","exception":false,"start_time":"2024-01-29T06:43:17.002242","status":"completed"},"tags":[]}},{"cell_type":"code","source":"xgb_models = {}\n\nfor col in list(y):\n    xgb_models[col] = make_pipeline(\n        MEstimateEncoder(cols = categorical_features),\n        SimpleImputer(),\n        XGBClassifier(random_state = seed, tree_method = 'hist')\n    )\n    _, oof_preds, _ = cross_val_score_proba(xgb_models[col], col)\n    oof_list[col].append(oof_preds)\n    \nprint('')\n\n# It does the same as the previous block, but using XGBoost.","metadata":{"papermill":{"duration":133.699844,"end_time":"2024-01-29T06:45:30.812195","exception":false,"start_time":"2024-01-29T06:43:17.112351","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:23:40.646695Z","iopub.execute_input":"2024-03-14T21:23:40.647069Z","iopub.status.idle":"2024-03-14T21:25:56.766113Z","shell.execute_reply.started":"2024-03-14T21:23:40.647040Z","shell.execute_reply":"2024-03-14T21:25:56.764799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Ensemble NN","metadata":{"papermill":{"duration":0.014948,"end_time":"2024-01-29T06:45:30.842435","exception":false,"start_time":"2024-01-29T06:45:30.827487","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def cross_val_score_nn(estimator, col, cv = kf, label = ''):\n    \n    X_oof = oof_list[col]\n        \n    model = clone(estimator).fit(X_oof, y[col])\n    val_preds = model.predict_proba(X_oof)\n    val_score = accuracy_score(y[col], val_preds.argmax(axis = 1))\n    \n    print(f'Val Score: {val_score:.5f} | {label}')\n    \n    return val_score\n\n# This function facilitates the evaluation of a neural network model using cross-validation and prints\n# the validation score for monitoring model performance.","metadata":{"papermill":{"duration":0.025169,"end_time":"2024-01-29T06:45:30.88277","exception":false,"start_time":"2024-01-29T06:45:30.857601","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:26:07.739885Z","iopub.execute_input":"2024-03-14T21:26:07.740302Z","iopub.status.idle":"2024-03-14T21:26:07.749018Z","shell.execute_reply.started":"2024-03-14T21:26:07.740268Z","shell.execute_reply":"2024-03-14T21:26:07.747638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class TensorFlower(BaseEstimator, ClassifierMixin):\n    \n    def fit(self, x, y):\n        \n        tf.keras.utils.set_random_seed(seed)\n        tf.config.experimental.enable_op_determinism()\n        \n        inputs = []\n        norm_inputs = []\n        \n        multiply = []\n        \n        for i in range(len(x)):\n            inputs.append(tf.keras.Input((len(np.unique(y)), )))\n            norm_inputs.append(tf.keras.layers.BatchNormalization()(inputs[i]))\n            \n        concat = tf.keras.layers.Concatenate()(norm_inputs)\n        \n        for i in range(len(x)):\n            dense = tf.keras.layers.Dense(len(np.unique(y)))(concat)\n            norm = tf.keras.layers.BatchNormalization()(dense)\n            act = tf.keras.activations.softmax(norm)\n            multiply.append(tf.keras.layers.Multiply()([inputs[i], act]))\n            \n        outputs = tf.keras.layers.Add()(multiply)\n        \n        self.model = tf.keras.Model(inputs, outputs)\n        self.model.compile(loss = 'sparse_categorical_crossentropy', optimizer = tf.keras.optimizers.AdamW(1e-4))\n        \n        self.model.fit(x, y, epochs = 50, verbose = 0)\n        \n        return self\n    \n    def predict_proba(self, x):\n        \n        return self.model.predict(x, verbose = 0)\n    \n    def predict(self, x):\n        \n        return np.argmax(self.predict_proba(x), axis = 1)\n    \n# The TensorFlower class provides an implementation of a custom neural network classifier using TensorFlow,\n# allowing for training, prediction of class probabilities, and prediction of class labels.","metadata":{"papermill":{"duration":0.029221,"end_time":"2024-01-29T06:45:30.927107","exception":false,"start_time":"2024-01-29T06:45:30.897886","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:26:12.235123Z","iopub.execute_input":"2024-03-14T21:26:12.235562Z","iopub.status.idle":"2024-03-14T21:26:12.254105Z","shell.execute_reply.started":"2024-03-14T21:26:12.235515Z","shell.execute_reply":"2024-03-14T21:26:12.252983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in list(y):\n    cross_val_score_nn(TensorFlower(), col)\n    \nprint('\\n')\n\n# This approach allows the performance of the TensorFlower model to be evaluated for each target\n# variable separately using cross-validation.","metadata":{"papermill":{"duration":163.27622,"end_time":"2024-01-29T06:48:14.218501","exception":false,"start_time":"2024-01-29T06:45:30.942281","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:26:15.677590Z","iopub.execute_input":"2024-03-14T21:26:15.678002Z","iopub.status.idle":"2024-03-14T21:27:49.924242Z","shell.execute_reply.started":"2024-03-14T21:26:15.677971Z","shell.execute_reply":"2024-03-14T21:27:49.922917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction and Submission","metadata":{"papermill":{"duration":0.015352,"end_time":"2024-01-29T06:48:14.322618","exception":false,"start_time":"2024-01-29T06:48:14.307266","status":"completed"},"tags":[]}},{"cell_type":"code","source":"for col in list(y):\n    predict_list[col].append(lgb_models[col].fit(X, y[col]).predict_proba(test_pd))\n    predict_list[col].append(xgb_models[col].fit(X, y[col]).predict_proba(test_pd))\n    \n# This approach allows generating predictions for each model (LightGBM and XGBoost) on the test data\n# set for each column of the target variable separately.","metadata":{"papermill":{"duration":1492.512251,"end_time":"2024-01-29T07:13:06.850299","exception":false,"start_time":"2024-01-29T06:48:14.338048","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:27:56.225048Z","iopub.execute_input":"2024-03-14T21:27:56.225459Z","iopub.status.idle":"2024-03-14T21:28:51.744106Z","shell.execute_reply.started":"2024-03-14T21:27:56.225427Z","shell.execute_reply":"2024-03-14T21:28:51.742814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in list(y):\n    ensembler = TensorFlower().fit(oof_list[col], y[col])\n    submission[col] = ensembler.predict(predict_list[col])\n    submission[col] = le[col].inverse_transform(submission[col].astype(np.int8))\n\nsubmission[['id_unique'] + list(y)].to_csv('cool_city.csv', index = False)\n\n# This approach allows pooling the predictions of several models and generating final\n# predictions for each column of target variables.","metadata":{"papermill":{"duration":173.734399,"end_time":"2024-01-29T07:16:00.601369","exception":false,"start_time":"2024-01-29T07:13:06.86697","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-14T21:33:44.695717Z","iopub.execute_input":"2024-03-14T21:33:44.696151Z","iopub.status.idle":"2024-03-14T21:35:18.358651Z","shell.execute_reply.started":"2024-03-14T21:33:44.696117Z","shell.execute_reply":"2024-03-14T21:35:18.357735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusions\nIt has followed practically the same sequence that I learned in the previous courses. First, a small exploratory analysis, the feature engineering to then integrate all the data. Then, the variables for prediction are defined and the test and training groups are separated. Finally, based on models, predictions are made. On this occasion, I chose to work with XGBoost because it is something I have used before. Something similar happened with LightGBM, I took it into account since you mentioned in class that it is a very robust model for many applications. Finally, you get as output a file called \"cool_city.csv\" inside the /kaggle/working/ folder.","metadata":{}},{"cell_type":"markdown","source":"# Citation\n\n- Clayton Miller, Matias Quintana, Mario Frei, Yun Xuan Chua, Chun Fu, Bianca Picchetti, Winston Yap, Adrian Chong, and Filip Biljecki. 2023. Introducing the Cool, Quiet City Competition: Predicting Smartwatch-Reported Heat and Noise with Digital Twin Metrics. In Proceedings of the 10th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys '23). Association for Computing Machinery, New York, NY, USA, 298–299. https://doi.org/10.1145/3600100.3626269\n- Iqbal Syah Akbar. 2023. #2nd Place | Cool Quiet City Competition (Python · Cool, Quiet City Competition).","metadata":{"papermill":{"duration":0.015561,"end_time":"2024-01-29T07:16:00.632974","exception":false,"start_time":"2024-01-29T07:16:00.617413","status":"completed"},"tags":[]}}]}