{"metadata":{"kaggle":{"accelerator":"none","dataSources":[{"sourceId":61145,"databundleVersionId":7047669,"sourceType":"competition"}],"dockerImageVersionId":30646,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"papermill":{"default_parameters":{},"duration":8816.051734,"end_time":"2024-01-29T07:16:03.940045","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-01-29T04:49:07.888311","version":"2.4.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Loading Libraries and Datasets","metadata":{"papermill":{"duration":0.013798,"end_time":"2024-01-29T04:49:11.324968","exception":false,"start_time":"2024-01-29T04:49:11.31117","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install -q iterative-stratification","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:11.35445Z","iopub.status.busy":"2024-01-29T04:49:11.353892Z","iopub.status.idle":"2024-01-29T04:49:24.69621Z","shell.execute_reply":"2024-01-29T04:49:24.694869Z"},"papermill":{"duration":13.360127,"end_time":"2024-01-29T04:49:24.69885","exception":false,"start_time":"2024-01-29T04:49:11.338723","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport polars as pl\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport tensorflow as tf\nimport os\nimport optuna\nimport holidays\nimport copy\n\nfrom category_encoders import OneHotEncoder, MEstimateEncoder, CatBoostEncoder, BinaryEncoder\nfrom iterstrat.ml_stratifiers import MultilabelStratifiedKFold, RepeatedMultilabelStratifiedKFold\nfrom sklearn.cluster import KMeans\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.model_selection import StratifiedKFold, RepeatedStratifiedKFold, TimeSeriesSplit\nfrom sklearn.decomposition import PCA\nfrom sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier\nfrom sklearn.ensemble import GradientBoostingClassifier, HistGradientBoostingClassifier\nfrom sklearn.ensemble import VotingClassifier\nfrom sklearn.impute import KNNImputer, SimpleImputer\nfrom sklearn.metrics import accuracy_score, roc_auc_score, roc_curve, make_scorer\nfrom sklearn.metrics.pairwise import euclidean_distances\nfrom sklearn.naive_bayes import GaussianNB, BernoulliNB\nfrom sklearn.pipeline import make_pipeline, Pipeline\nfrom sklearn.base import BaseEstimator, TransformerMixin, ClassifierMixin, clone\nfrom sklearn.multioutput import ClassifierChain\nfrom sklearn.preprocessing import FunctionTransformer, StandardScaler, PowerTransformer, LabelEncoder\nfrom sklearn.linear_model import LogisticRegression, RidgeClassifier\nfrom sklearn.svm import SVC\nfrom sklearn.discriminant_analysis import LinearDiscriminantAnalysis\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.inspection import permutation_importance\nfrom scipy.cluster.hierarchy import dendrogram, linkage\nfrom scipy.spatial.distance import squareform\nfrom xgboost import XGBClassifier, XGBRFClassifier\nfrom lightgbm import LGBMClassifier\nfrom catboost import CatBoostClassifier\nfrom sklearn.feature_selection import SequentialFeatureSelector\n\nsns.set_theme(style = 'white', palette = 'viridis')\npal = sns.color_palette('viridis')\n\npd.set_option('display.max_rows', 100)","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2024-01-29T04:49:24.728023Z","iopub.status.busy":"2024-01-29T04:49:24.727004Z","iopub.status.idle":"2024-01-29T04:49:44.432738Z","shell.execute_reply":"2024-01-29T04:49:44.431464Z"},"papermill":{"duration":19.724847,"end_time":"2024-01-29T04:49:44.43698","exception":false,"start_time":"2024-01-29T04:49:24.712133","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This code imports various data science libraries such as NumPy, Pandas, TensorFlow, and scikit-learn, along with visualization libraries like Matplotlib and Seaborn. It also imports models and tools for machine learning tasks such as imputation, encoding, clustering, and classification. It sets visualization preferences and adjusts Pandas display options.","metadata":{}},{"cell_type":"code","source":"preprocessor = [\n    pl.col('dT').cast(pl.Float32),\n    \n    pl.col('ts_oxygen_saturation').cast(pl.Float32),\n    pl.col('ts_resting_heart_rate').cast(pl.Float32),\n    \n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z'),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.year().alias('time_year').cast(pl.UInt16),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.month().alias('time_month').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.week().alias('time_week').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.weekday().alias('time_weekday').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.ordinal_day().alias('time_day_of_year').cast(pl.UInt16),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.day().alias('time_day').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.hour().alias('time_hour').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.minute().alias('time_minute').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.second().alias('time_second').cast(pl.UInt8),\n    pl.col('time').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.microsecond().alias('time_micro').cast(pl.UInt32),\n    \n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z'),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.year().alias('loc_year').cast(pl.UInt16),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.month().alias('loc_month').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.week().alias('loc_week').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.weekday().alias('loc_weekday').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.ordinal_day().alias('loc_day_of_year').cast(pl.UInt16),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.day().alias('loc_day').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.hour().alias('loc_hour').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.minute().alias('loc_minute').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.second().alias('loc_second').cast(pl.UInt8),\n    pl.col('ws_timestamp_location').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.microsecond().alias('loc_micro').cast(pl.UInt32),\n    \n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z'),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.year().alias('start_year').cast(pl.UInt16),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.month().alias('start_month').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.week().alias('start_week').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.weekday().alias('start_weekday').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.ordinal_day().alias('start_day_of_year').cast(pl.UInt16),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.day().alias('start_day').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.hour().alias('start_hour').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.minute().alias('start_minute').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.second().alias('start_second').cast(pl.UInt8),\n    pl.col('ws_timestamp_start').str.to_datetime('%Y-%m-%d %H:%M:%S%.6f%Z').dt.microsecond().alias('start_micro').cast(pl.UInt32)\n]\n\npreprocessor_2 = [\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z'),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.year().alias('year').cast(pl.UInt16),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.month().alias('month').cast(pl.UInt8),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.day().alias('day').cast(pl.UInt8),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.hour().alias('hour').cast(pl.UInt8),\n    pl.col('').str.to_datetime('%Y-%m-%d %H:%M:%S%Z').dt.minute().alias('minute').cast(pl.UInt8),\n    \n    pl.col('value').cast(pl.Float32)\n]\n\nduration_calculator = [\n    (pl.duration(days = pl.col('time_day_of_year'), hours = pl.col('time_hour'), minutes = pl.col('time_minute'), seconds = pl.col('time_second'), microseconds = pl.col('time_micro')) -\n    pl.duration(days = pl.col('start_day_of_year'), hours = pl.col('start_hour'), minutes = pl.col('start_minute'), seconds = pl.col('start_second'), microseconds = pl.col('start_micro'))).dt.total_microseconds().alias('time_start_difference'),\n    (pl.duration(days = pl.col('time_day_of_year'), hours = pl.col('time_hour'), minutes = pl.col('time_minute'), seconds = pl.col('time_second'), microseconds = pl.col('time_micro')) -\n    pl.duration(days = pl.col('loc_day_of_year'), hours = pl.col('loc_hour'), minutes = pl.col('loc_minute'), seconds = pl.col('loc_second'), microseconds = pl.col('loc_micro'))).dt.total_microseconds().alias('time_loc_difference'),\n    (pl.duration(days = pl.col('loc_day_of_year'), hours = pl.col('loc_hour'), minutes = pl.col('loc_minute'), seconds = pl.col('loc_second'), microseconds = pl.col('loc_micro')) -\n     pl.duration(days = pl.col('start_day_of_year'), hours = pl.col('start_hour'), minutes = pl.col('start_minute'), seconds = pl.col('start_second'), microseconds = pl.col('start_micro'))).dt.total_microseconds().alias('start_loc_difference')\n]","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:44.473314Z","iopub.status.busy":"2024-01-29T04:49:44.472851Z","iopub.status.idle":"2024-01-29T04:49:44.518446Z","shell.execute_reply":"2024-01-29T04:49:44.517346Z"},"papermill":{"duration":0.064479,"end_time":"2024-01-29T04:49:44.521031","exception":false,"start_time":"2024-01-29T04:49:44.456552","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This code defines data preprocessing pipelines using Polars for datetime conversion, type casting, and feature engineering. It extracts various temporal components such as year, month, day, and time differences between different timestamps. These pipelines are crucial for preparing temporal data for analysis and modeling tasks.","metadata":{}},{"cell_type":"code","source":"train = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/cozie_responses_and_physiological_data_training.csv').\\\n        with_columns(preprocessor).with_columns(duration_calculator)\ntest = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/cozie_responses_and_physiological_data_test_public.csv').\\\n       with_columns(preprocessor).with_columns(duration_calculator)","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:44.550384Z","iopub.status.busy":"2024-01-29T04:49:44.549628Z","iopub.status.idle":"2024-01-29T04:49:44.614301Z","shell.execute_reply":"2024-01-29T04:49:44.613233Z"},"papermill":{"duration":0.082503,"end_time":"2024-01-29T04:49:44.616906","exception":false,"start_time":"2024-01-29T04:49:44.534403","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This code reads CSV files containing training and test data using Polars. It applies data preprocessing pipelines defined earlier to convert datetime columns, cast data types, and calculate time differences. These processed datasets are then stored in the variables train and test for further analysis or modeling tasks.","metadata":{}},{"cell_type":"code","source":"air_temp = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_air-temperature.csv').\\\n           melt('').\\\n           with_columns(preprocessor_2).\\\n           group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n           agg(\n               pl.col('value').mean().alias('mean_air_temp'),\n               pl.col('value').std().alias('std_air_temp'),\n           )\nrainfall = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_rainfall.csv').\\\n           melt('').\\\n           with_columns(preprocessor_2).\\\n           group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n           agg(\n               pl.col('value').mean().alias('mean_rainfall'),\n               pl.col('value').std().alias('std_rainfall'),\n           )\nhumidity = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_relative-humidity.csv').\\\n           melt('').\\\n           with_columns(preprocessor_2).\\\n           group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n           agg(\n               pl.col('value').mean().alias('mean_humidity'),\n               pl.col('value').std().alias('std_humidity'),\n           )\nstations = pl.read_csv('/kaggle/input/cool-quiet-city-competition/weather_stations.csv')\nwind_direction = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_wind-direction.csv').\\\n                 melt('').\\\n                 with_columns(preprocessor_2).\\\n                 group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n                 agg(\n                     pl.col('value').mean().alias('mean_wind_direction'),\n                     pl.col('value').std().alias('std_wind_direction')\n                 )\nwind_speed = pl.scan_csv('/kaggle/input/cool-quiet-city-competition/weather_wind-speed.csv').\\\n             melt('').\\\n             with_columns(preprocessor_2).\\\n             group_by(['year', 'month', 'day', 'hour', 'minute']).\\\n             agg(\n                 pl.col('value').mean().alias('mean_wind_speed'),\n                 pl.col('value').std().alias('std_wind_speed'),\n             )","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:44.645486Z","iopub.status.busy":"2024-01-29T04:49:44.645055Z","iopub.status.idle":"2024-01-29T04:49:44.730161Z","shell.execute_reply":"2024-01-29T04:49:44.729179Z"},"papermill":{"duration":0.102406,"end_time":"2024-01-29T04:49:44.732804","exception":false,"start_time":"2024-01-29T04:49:44.630398","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This code reads multiple CSV files containing weather data using Polars. It applies data preprocessing pipelines to convert datetime columns, calculates statistical aggregates such as mean and standard deviation, and groups the data by year, month, day, hour, and minute. The processed weather data is stored for analysis.","metadata":{}},{"cell_type":"markdown","source":"# Dataset Information","metadata":{"papermill":{"duration":0.01312,"end_time":"2024-01-29T04:49:44.759911","exception":false,"start_time":"2024-01-29T04:49:44.746791","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def desc_dataset_pl(df):\n    return pl.concat([\n        pl.DataFrame({'key' : df.columns, 'dataset_length' : len(df)}),\n        df.null_count().transpose().rename({'column_0' : 'null'}),\n        df.approx_n_unique().transpose().rename({'column_0' : 'nunique'}),\n        df.mean().transpose().rename({'column_0' : 'mean'}),\n        df.std().transpose().rename({'column_0' : 'std'}),\n        df.median().transpose().rename({'column_0' : 'median'}),\n        df.min().transpose().rename({'column_0' : 'min'}),\n        df.max().transpose().rename({'column_0' : 'max'})\n    ], how = 'horizontal').with_columns(\n        (pl.col('dataset_length') - pl.col('null')).alias('count'),\n        (pl.col('null') / pl.col('dataset_length') * 100).alias('%null'),\n        (pl.col('nunique') / pl.col('dataset_length') * 100).alias('%unique')\n    ).drop('dataset_length').to_pandas()","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:44.788755Z","iopub.status.busy":"2024-01-29T04:49:44.78797Z","iopub.status.idle":"2024-01-29T04:49:44.796042Z","shell.execute_reply":"2024-01-29T04:49:44.794832Z"},"papermill":{"duration":0.025358,"end_time":"2024-01-29T04:49:44.798669","exception":false,"start_time":"2024-01-29T04:49:44.773311","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This function desc_dataset_pl(df) describes a dataset by calculating various statistics such as null count, number of unique values, mean, standard deviation, median, minimum, and maximum for each column. It returns a Pandas DataFrame containing these statistics along with counts and percentages of null and unique values for each column.","metadata":{}},{"cell_type":"code","source":"#desc_dataset_pl(train)","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:44.827491Z","iopub.status.busy":"2024-01-29T04:49:44.82704Z","iopub.status.idle":"2024-01-29T04:49:44.832046Z","shell.execute_reply":"2024-01-29T04:49:44.830934Z"},"papermill":{"duration":0.0217,"end_time":"2024-01-29T04:49:44.834261","exception":false,"start_time":"2024-01-29T04:49:44.812561","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#desc_dataset_pl(test)","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:44.862552Z","iopub.status.busy":"2024-01-29T04:49:44.862102Z","iopub.status.idle":"2024-01-29T04:49:44.867229Z","shell.execute_reply":"2024-01-29T04:49:44.866164Z"},"papermill":{"duration":0.021945,"end_time":"2024-01-29T04:49:44.869571","exception":false,"start_time":"2024-01-29T04:49:44.847626","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TS Features Aggregation","metadata":{"papermill":{"duration":0.013129,"end_time":"2024-01-29T04:49:44.960247","exception":false,"start_time":"2024-01-29T04:49:44.947118","status":"completed"},"tags":[]}},{"cell_type":"code","source":"df_all = pl.concat([train, test], how = 'vertical')","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:44.989314Z","iopub.status.busy":"2024-01-29T04:49:44.98836Z","iopub.status.idle":"2024-01-29T04:49:44.996438Z","shell.execute_reply":"2024-01-29T04:49:44.995253Z"},"papermill":{"duration":0.025587,"end_time":"2024-01-29T04:49:44.999204","exception":false,"start_time":"2024-01-29T04:49:44.973617","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"survey_counter_day = df_all.group_by(\n    ['id_participant', 'time_year', 'time_month', 'time_day']\n).agg(\n    pl.col('id_participant').count().alias('survey_count_per_day').cast(pl.UInt16)\n)\nsurvey_counter_hour = df_all.group_by(\n    ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour']\n).agg(\n    pl.col('id_participant').count().alias('survey_count_per_hour').cast(pl.UInt16)\n)\nsurvey_counter_minute = df_all.group_by(\n    ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour', 'time_minute']\n).agg(\n    pl.col('id_participant').count().alias('survey_count_per_minute').cast(pl.UInt8)\n)","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:45.027876Z","iopub.status.busy":"2024-01-29T04:49:45.027499Z","iopub.status.idle":"2024-01-29T04:49:45.035787Z","shell.execute_reply":"2024-01-29T04:49:45.034553Z"},"papermill":{"duration":0.025501,"end_time":"2024-01-29T04:49:45.038174","exception":false,"start_time":"2024-01-29T04:49:45.012673","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rest_heart_rate_agg = df_all.group_by(\n    ['time_year', 'time_week', 'id_participant']\n).agg(\n    pl.col('ts_resting_heart_rate').mean().alias('mean_resting_heart_rate'),\n    pl.col('ts_resting_heart_rate').std().alias('std_resting_heart_rate')\n)\noxygen_saturation_agg = df_all.group_by(\n    ['time_year', 'time_week', 'id_participant']\n).agg(\n    pl.col('ts_oxygen_saturation').mean().alias('mean_oxygen_saturation'),\n    pl.col('ts_oxygen_saturation').std().alias('std_oxygen_saturation')\n)\nstand_time_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_stand_time').mean().alias('mean_stand_time'),\n    pl.col('ts_stand_time').std().alias('std_stand_time')\n)\nstep_count_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_step_count').mean().alias('mean_step_count'),\n    pl.col('ts_step_count').std().alias('std_step_count')\n)\nwalking_distance_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_walking_distance').mean().alias('mean_walking_distance'),\n    pl.col('ts_walking_distance').std().alias('std_walking_distance')\n)\naudio_exposure_environment_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_audio_exposure_environment').mean().alias('mean_audio_exposure_environment'),\n    pl.col('ts_audio_exposure_environment').std().alias('std_audio_exposure_environment')\n)\nheart_rate_agg = df_all.group_by(\n    ['time_year', 'time_month', 'time_day', 'id_participant']\n).agg(\n    pl.col('ts_heart_rate').mean().alias('mean_heart_rate'),\n    pl.col('ts_heart_rate').std().alias('std_heart_rate')\n)","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:45.067365Z","iopub.status.busy":"2024-01-29T04:49:45.06693Z","iopub.status.idle":"2024-01-29T04:49:45.079419Z","shell.execute_reply":"2024-01-29T04:49:45.078292Z"},"papermill":{"duration":0.030034,"end_time":"2024-01-29T04:49:45.081786","exception":false,"start_time":"2024-01-29T04:49:45.051752","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This code aggregates physiological data from a DataFrame (df_all) by grouping it based on time periods and participant IDs. For each group, it calculates the mean and standard deviation of various physiological measures such as resting heart rate, oxygen saturation, stand time, step count, walking distance, and audio exposure environment.","metadata":{}},{"cell_type":"code","source":"train = train.join(\n    rest_heart_rate_agg, on = ['time_year', 'time_week', 'id_participant'], how = 'left'\n).join(\n    oxygen_saturation_agg, on = ['time_year', 'time_week', 'id_participant'], how = 'left'\n).join(\n    stand_time_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    step_count_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    walking_distance_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    audio_exposure_environment_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    heart_rate_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    survey_counter_day, on = ['id_participant', 'time_year', 'time_month', 'time_day'], how = 'left'\n).join(\n    survey_counter_hour, on = ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour'], how = 'left'\n).join(\n    survey_counter_minute, on = ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'], how = 'left'\n)\ntest = test.join(\n    rest_heart_rate_agg, on = ['time_year', 'time_week', 'id_participant'], how = 'left'\n).join(\n    oxygen_saturation_agg, on = ['time_year', 'time_week', 'id_participant'], how = 'left'\n).join(\n    stand_time_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    step_count_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    walking_distance_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    audio_exposure_environment_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    heart_rate_agg, on = ['time_year', 'time_month', 'time_day', 'id_participant'], how = 'left'\n).join(\n    survey_counter_day, on = ['id_participant', 'time_year', 'time_month', 'time_day'], how = 'left'\n).join(\n    survey_counter_hour, on = ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour'], how = 'left'\n).join(\n    survey_counter_minute, on = ['id_participant', 'time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'], how = 'left'\n)","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:45.111225Z","iopub.status.busy":"2024-01-29T04:49:45.110544Z","iopub.status.idle":"2024-01-29T04:49:45.133044Z","shell.execute_reply":"2024-01-29T04:49:45.132171Z"},"papermill":{"duration":0.040233,"end_time":"2024-01-29T04:49:45.135455","exception":false,"start_time":"2024-01-29T04:49:45.095222","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This code joins aggregated physiological data with the training and test datasets based on specific time periods and participant IDs. It incorporates features such as resting heart rate, oxygen saturation, stand time, step count, walking distance, audio exposure environment, and heart rate along with survey response counts at different time granularities.","metadata":{}},{"cell_type":"markdown","source":"# Station and Weather","metadata":{"papermill":{"duration":0.012909,"end_time":"2024-01-29T04:49:45.16168","exception":false,"start_time":"2024-01-29T04:49:45.148771","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def find_closest_station(x):\n    if x[0] == None: return None\n    latitude = float(x[0][:18])\n    longitude = float(x[0][19:])\n    return stations.row(stations.select(\n        distance = (pl.col('latitude') - latitude) ** 2 + (pl.col('longitude') - longitude) ** 2\n    )['distance'].arg_min())[1]","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:45.190188Z","iopub.status.busy":"2024-01-29T04:49:45.189745Z","iopub.status.idle":"2024-01-29T04:49:45.195654Z","shell.execute_reply":"2024-01-29T04:49:45.194866Z"},"papermill":{"duration":0.022635,"end_time":"2024-01-29T04:49:45.197578","exception":false,"start_time":"2024-01-29T04:49:45.174943","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def map_station(df):\n    return df.select(pl.col('ws_latitude').cast(pl.Utf8).map_elements(\n        lambda x: x + '0' * (18 - len(x))\n    ) + ' ' + pl.col('ws_longitude').cast(pl.Utf8).map_elements(\n        lambda x: x + '0' * (18 - len(x))\n    )).map_rows(find_closest_station)['map']","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:45.226315Z","iopub.status.busy":"2024-01-29T04:49:45.225646Z","iopub.status.idle":"2024-01-29T04:49:45.231975Z","shell.execute_reply":"2024-01-29T04:49:45.230868Z"},"papermill":{"duration":0.02323,"end_time":"2024-01-29T04:49:45.234313","exception":false,"start_time":"2024-01-29T04:49:45.211083","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.join(\n    air_temp,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    rainfall,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    humidity,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    wind_direction,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    wind_speed,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute'])\n\ntest = test.join(\n    air_temp,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    rainfall,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    humidity,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    wind_direction,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute']).join(\n    wind_speed,\n    left_on = ['time_year', 'time_month', 'time_day', 'time_hour', 'time_minute'],\n    right_on = ['year', 'month', 'day', 'hour', 'minute'],\n    how = 'left'\n).drop(['year', 'month', 'day', 'hour', 'minute'])","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:45.263619Z","iopub.status.busy":"2024-01-29T04:49:45.263154Z","iopub.status.idle":"2024-01-29T04:49:45.292589Z","shell.execute_reply":"2024-01-29T04:49:45.291397Z"},"papermill":{"duration":0.04683,"end_time":"2024-01-29T04:49:45.295134","exception":false,"start_time":"2024-01-29T04:49:45.248304","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This code joins weather data (air temperature, rainfall, humidity, wind direction, and wind speed) with the training and test datasets based on specific time periods. It aligns the weather data with the corresponding timestamps and drops redundant columns after the join operation.","metadata":{}},{"cell_type":"markdown","source":"# Correlation","metadata":{"papermill":{"duration":0.01345,"end_time":"2024-01-29T04:49:45.322014","exception":false,"start_time":"2024-01-29T04:49:45.308564","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def heatmap(dataset, label = None):\n    corr = dataset.corr(method = 'pearson')\n    plt.figure(figsize = (45, 45), dpi = 300)\n    mask = np.zeros_like(corr)\n    mask[np.triu_indices_from(mask)] = True\n    sns.heatmap(corr, mask = mask, cmap = 'viridis', annot = True, annot_kws = {'size' : 11})\n    plt.title(f'{label} Dataset Correlation Matrix\\n', fontsize = 25, weight = 'bold')\n    plt.show()","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:45.350271Z","iopub.status.busy":"2024-01-29T04:49:45.349844Z","iopub.status.idle":"2024-01-29T04:49:45.356055Z","shell.execute_reply":"2024-01-29T04:49:45.355023Z"},"papermill":{"duration":0.022915,"end_time":"2024-01-29T04:49:45.358148","exception":false,"start_time":"2024-01-29T04:49:45.335233","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preparation","metadata":{"papermill":{"duration":0.013031,"end_time":"2024-01-29T04:49:45.384604","exception":false,"start_time":"2024-01-29T04:49:45.371573","status":"completed"},"tags":[]}},{"cell_type":"code","source":"X = train.filter(pl.col('q_location').is_not_null()).drop('id_unique', 'time', 'ws_timestamp_location', 'ws_timestamp_start', 'ts_oxygen_saturation', 'ts_resting_heart_rate', 'ts_stand_time', 'ts_step_count', 'ts_walking_distance', 'ts_heart_rate', 'ts_audio_exposure_environment').collect()\nX = X.with_columns(stations = map_station(X)).to_pandas()\ny = X[['q_earphones', 'q_noise_kind', 'q_noise_nearby', 'q_thermal_preference']]\nX = X.drop(['q_earphones', 'q_noise_kind', 'q_noise_nearby', 'q_thermal_preference'], axis = 1)\n\nsubmission = test.filter(pl.col('q_location').is_not_null()).drop('q_earphones', 'q_noise_kind', 'q_noise_nearby', 'q_thermal_preference', 'time', 'ws_timestamp_location', 'ws_timestamp_start', 'ts_oxygen_saturation', 'ts_resting_heart_rate', 'ts_stand_time', 'ts_step_count', 'ts_walking_distance', 'ts_heart_rate', 'ts_audio_exposure_environment').collect()\nsubmission = submission.with_columns(stations = map_station(submission)).to_pandas()\ntest_pd = submission.drop('id_unique', axis = 1)\ntest_pd.loc[3178, 'time_loc_difference'] += 1e6 * 3600 * 24 * 365\ntest_pd.loc[3178, 'start_loc_difference'] -= 1e6 * 3600 * 24 * 365\n\ncategorical_features = list(X.select_dtypes('object'))\n\nle = {}\n\nfor col in y.columns:\n    le[col] = LabelEncoder().fit(y[col])\n    y[col] = le[col].transform(y[col])\n    \nacc_scorer = make_scorer(accuracy_score)\nseed = 42\ntf.keras.utils.set_random_seed(seed)\n\nkf = MultilabelStratifiedKFold(n_splits = 5, random_state = seed, shuffle = True)\n#kf = TimeSeriesSplit()","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:49:45.413792Z","iopub.status.busy":"2024-01-29T04:49:45.413369Z","iopub.status.idle":"2024-01-29T04:50:30.871529Z","shell.execute_reply":"2024-01-29T04:50:30.870515Z"},"papermill":{"duration":45.476571,"end_time":"2024-01-29T04:50:30.874598","exception":false,"start_time":"2024-01-29T04:49:45.398027","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineering","metadata":{"papermill":{"duration":0.013352,"end_time":"2024-01-29T04:50:30.901792","exception":false,"start_time":"2024-01-29T04:50:30.88844","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class NumericalImputer(BaseEstimator, TransformerMixin):\n    \n    def fit(self, x, y = None):\n        self.imputer = SimpleImputer().fit(x.select_dtypes(['int', 'float']), y)\n        return self\n    \n    def transform(self, x):\n        x_copy = x.copy()\n        x_copy.loc[:, list(x_copy.select_dtypes(['int', 'float']))] = self.imputer.transform(x_copy.loc[:, list(x_copy.select_dtypes(['int', 'float']))])\n        return x_copy","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:50:31.185711Z","iopub.status.busy":"2024-01-29T04:50:31.18525Z","iopub.status.idle":"2024-01-29T04:50:31.192492Z","shell.execute_reply":"2024-01-29T04:50:31.191281Z"},"papermill":{"duration":0.024644,"end_time":"2024-01-29T04:50:31.195005","exception":false,"start_time":"2024-01-29T04:50:31.170361","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Adversarial Validation","metadata":{"papermill":{"duration":0.013686,"end_time":"2024-01-29T04:50:31.222264","exception":false,"start_time":"2024-01-29T04:50:31.208578","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#thanks to @carlmcbrideellis\n#https://www.kaggle.com/code/carlmcbrideellis/what-is-adversarial-validation\n\ndef adversarial_validation(dataset_1 = train, dataset_2 = test, label = 'Train-Test'):\n\n    adv_train = dataset_1.copy()\n    adv_test = dataset_2.copy()\n\n    adv_train['is_test'] = 0\n    adv_test['is_test'] = 1\n\n    adv = pd.concat([adv_train, adv_test], ignore_index = True)\n\n    adv_shuffled = adv.sample(frac = 1)\n\n    adv_X = adv_shuffled.drop('is_test', axis = 1)\n    adv_y = adv_shuffled.is_test\n\n    skf = StratifiedKFold(n_splits = 5, random_state = 42, shuffle = True)\n\n    val_scores = []\n    predictions = np.zeros(len(adv))\n\n    for fold, (train_idx, val_idx) in enumerate(skf.split(adv_X, adv_y)):\n    \n        adv_lr = make_pipeline(\n            MEstimateEncoder(cols = list(categorical_features)), \n            #FeatureDropper(cols = ['ws_survey_count', 'id_participant']),\n            SimpleImputer(),\n            XGBClassifier(random_state = seed, tree_method = 'hist')\n        )\n        adv_lr.fit(adv_X.iloc[train_idx], adv_y.iloc[train_idx])\n        \n        val_preds = adv_lr.predict_proba(adv_X.iloc[val_idx])[:, 1]\n        predictions[val_idx] = val_preds\n        val_score = roc_auc_score(adv_y.iloc[val_idx], val_preds)\n        val_scores.append(val_score)\n    \n    fpr, tpr, _ = roc_curve(adv['is_test'], predictions)\n    \n    plt.figure(figsize = (10, 10), dpi = 300)\n    sns.lineplot(x=[0, 1], y=[0, 1], linestyle=\"--\", label=\"Indistinguishable Datasets\")\n    sns.lineplot(x=fpr, y=tpr, label=\"Adversarial Validation Classifier\")\n    plt.title(f'{label} Validation = {np.mean(val_scores):.5f}', weight = 'bold', size = 17)\n    plt.xlabel('False Positive Rate')\n    plt.ylabel('True Positive Rate')\n    plt.show()","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:50:31.25084Z","iopub.status.busy":"2024-01-29T04:50:31.250391Z","iopub.status.idle":"2024-01-29T04:50:31.261576Z","shell.execute_reply":"2024-01-29T04:50:31.260396Z"},"papermill":{"duration":0.028635,"end_time":"2024-01-29T04:50:31.264265","exception":false,"start_time":"2024-01-29T04:50:31.23563","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#adversarial_validation(X, test_pd)","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:50:31.293053Z","iopub.status.busy":"2024-01-29T04:50:31.292603Z","iopub.status.idle":"2024-01-29T04:50:31.297277Z","shell.execute_reply":"2024-01-29T04:50:31.296116Z"},"papermill":{"duration":0.021817,"end_time":"2024-01-29T04:50:31.299491","exception":false,"start_time":"2024-01-29T04:50:31.277674","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Cross Validation","metadata":{"papermill":{"duration":0.012986,"end_time":"2024-01-29T04:50:31.325875","exception":false,"start_time":"2024-01-29T04:50:31.312889","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def cross_val_score(estimator, cv = kf, label = '', show_importance = False):\n    \n    val_scores = []\n    \n    feature_importances_table = pd.DataFrame({'value' : 0}, index = list(X.columns))\n    \n    for fold, (train_idx, val_idx) in enumerate(cv.split(X, y)):\n        \n        X_train = X.iloc[train_idx]\n        y_train = y.iloc[train_idx]\n        \n        X_val = X.iloc[val_idx]\n        y_val = y.iloc[val_idx]\n        \n        val_preds = np.zeros((len(X_val), 4))\n        val_score = 0\n        \n        for i in range(4):\n            model = clone(estimator).fit(X_train, y_train.iloc[:, i])\n            val_preds[:, i] = model.predict(X_val)\n            val_score += accuracy_score(y_val.iloc[:, i], val_preds[:, i]) / 4\n            if show_importance:\n                feature_importances_table['value'] += permutation_importance(model, X_val, y_val.iloc[:, i], random_state = seed, scoring = 'accuracy', n_repeats = 5).importances_mean / 4 / cv.get_n_splits()\n            \n        val_scores.append(val_score)\n    \n    print(f'Val Score: {np.mean(val_scores):.5f} ± {np.std(val_scores):.5f} | {label}')\n    \n    if show_importance:\n        plt.figure(figsize = (20, 50))\n        plt.title('Feature Importances', weight = 'bold', size = 25)\n        sns.barplot(feature_importances_table.sort_values('value', ascending = False).T, orient = 'h', palette = 'viridis')\n        plt.show()\n    \n    return val_scores, feature_importances_table","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:50:31.36077Z","iopub.status.busy":"2024-01-29T04:50:31.360342Z","iopub.status.idle":"2024-01-29T04:50:31.370741Z","shell.execute_reply":"2024-01-29T04:50:31.369556Z"},"papermill":{"duration":0.032302,"end_time":"2024-01-29T04:50:31.373358","exception":false,"start_time":"2024-01-29T04:50:31.341056","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This function cross_val_score() performs cross-validation using a specified estimator and splitting strategy. It calculates validation scores for each fold, optionally displays feature importances, and returns validation scores along with feature importance table. It's particularly useful for model evaluation and feature analysis in machine learning tasks.","metadata":{}},{"cell_type":"code","source":"def cross_val_score_proba(estimator, col, cv = kf, label = ''):\n    \n    val_scores = []\n    val_predictions = np.zeros((len(X), y.loc[:, col].nunique()))\n    test_predictions = np.zeros((len(test_pd), y.loc[:, col].nunique()))\n    \n    for fold, (train_idx, val_idx) in enumerate(cv.split(X, y)):\n        \n        X_train = X.iloc[train_idx].reset_index(drop = True)\n        y_train = y.iloc[train_idx].reset_index(drop = True)\n        \n        X_val = X.iloc[val_idx].reset_index(drop = True)\n        y_val = y.iloc[val_idx].reset_index(drop = True)\n        \n        model = clone(estimator).fit(X_train, y_train[col])\n        val_preds = model.predict_proba(X_val)\n        val_score = accuracy_score(y_val[col], val_preds.argmax(axis = 1))\n            \n        val_scores.append(val_score)\n        val_predictions[val_idx] = val_preds\n        test_predictions += model.predict_proba(test_pd) / cv.get_n_splits()\n        \n        #print(f'Fold Score: {val_score:.5f}')\n    \n    print(f'Val Score: {np.mean(val_scores):.5f} ± {np.std(val_scores):.5f} | {label}')\n    \n    return val_scores, val_predictions, test_predictions","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:50:31.402929Z","iopub.status.busy":"2024-01-29T04:50:31.402469Z","iopub.status.idle":"2024-01-29T04:50:31.411459Z","shell.execute_reply":"2024-01-29T04:50:31.410289Z"},"papermill":{"duration":0.026822,"end_time":"2024-01-29T04:50:31.413927","exception":false,"start_time":"2024-01-29T04:50:31.387105","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof_list = {col : [] for col in list(y)}\npredict_list = {col : [] for col in list(y)}","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:50:31.443615Z","iopub.status.busy":"2024-01-29T04:50:31.442567Z","iopub.status.idle":"2024-01-29T04:50:31.44809Z","shell.execute_reply":"2024-01-29T04:50:31.446878Z"},"papermill":{"duration":0.022826,"end_time":"2024-01-29T04:50:31.450679","exception":false,"start_time":"2024-01-29T04:50:31.427853","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Logistic Regression","metadata":{"papermill":{"duration":0.013066,"end_time":"2024-01-29T04:50:31.477414","exception":false,"start_time":"2024-01-29T04:50:31.464348","status":"completed"},"tags":[]}},{"cell_type":"code","source":"log_models = {}\n\nfor col in list(y):\n    log_models[col] = make_pipeline(\n        MEstimateEncoder(cols = categorical_features),\n        SimpleImputer(),\n        StandardScaler(),\n        PowerTransformer(),\n        LogisticRegression(random_state = seed, max_iter = 10000000)\n    )\n    _, oof_preds, _ = cross_val_score_proba(log_models[col], col)\n    oof_list[col].append(oof_preds)\n    \nprint('')","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:50:31.505844Z","iopub.status.busy":"2024-01-29T04:50:31.505452Z","iopub.status.idle":"2024-01-29T04:51:07.051792Z","shell.execute_reply":"2024-01-29T04:51:07.050549Z"},"papermill":{"duration":35.58775,"end_time":"2024-01-29T04:51:07.07856","exception":false,"start_time":"2024-01-29T04:50:31.49081","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# SVM","metadata":{"papermill":{"duration":0.022879,"end_time":"2024-01-29T04:51:07.125295","exception":false,"start_time":"2024-01-29T04:51:07.102416","status":"completed"},"tags":[]}},{"cell_type":"code","source":"svm_models = {}\n\nfor col in list(y):\n    svm_models[col] = make_pipeline(\n        MEstimateEncoder(cols = categorical_features),\n        SimpleImputer(),\n        StandardScaler(),\n        PowerTransformer(),\n        SVC(random_state = seed, probability = True, max_iter = 1000000)\n    )\n    _, oof_preds, _ = cross_val_score_proba(svm_models[col], col)\n    oof_list[col].append(oof_preds)\n    \nprint('')","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:51:07.171536Z","iopub.status.busy":"2024-01-29T04:51:07.170838Z","iopub.status.idle":"2024-01-29T04:54:49.455731Z","shell.execute_reply":"2024-01-29T04:54:49.454592Z"},"papermill":{"duration":222.322245,"end_time":"2024-01-29T04:54:49.47075","exception":false,"start_time":"2024-01-29T04:51:07.148505","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# LightGBM","metadata":{"papermill":{"duration":0.014521,"end_time":"2024-01-29T04:54:49.49914","exception":false,"start_time":"2024-01-29T04:54:49.484619","status":"completed"},"tags":[]}},{"cell_type":"code","source":"lgb_models = {}\n\nfor col in list(y):\n    lgb_models[col] = make_pipeline(\n        MEstimateEncoder(cols = categorical_features),\n        SimpleImputer(),\n        LGBMClassifier(random_state = seed)\n    )\n    _, oof_preds, _ = cross_val_score_proba(lgb_models[col], col)\n    oof_list[col].append(oof_preds)\n    \nprint('')","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:54:49.606797Z","iopub.status.busy":"2024-01-29T04:54:49.606385Z","iopub.status.idle":"2024-01-29T04:56:38.351815Z","shell.execute_reply":"2024-01-29T04:56:38.350504Z"},"papermill":{"duration":108.777442,"end_time":"2024-01-29T04:56:38.368714","exception":false,"start_time":"2024-01-29T04:54:49.591272","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CatBoost","metadata":{"papermill":{"duration":0.014134,"end_time":"2024-01-29T04:56:38.397087","exception":false,"start_time":"2024-01-29T04:56:38.382953","status":"completed"},"tags":[]}},{"cell_type":"code","source":"cb_models = {}\n\nfor col in list(y):\n    cb_models[col] = make_pipeline(\n        NumericalImputer(),\n        CatBoostClassifier(random_state = seed, verbose = 0, cat_features = categorical_features)\n    )\n    _, oof_preds, _ = cross_val_score_proba(cb_models[col], col)\n    oof_list[col].append(oof_preds)\n\nprint('')","metadata":{"execution":{"iopub.execute_input":"2024-01-29T04:56:38.427938Z","iopub.status.busy":"2024-01-29T04:56:38.427559Z","iopub.status.idle":"2024-01-29T05:33:22.136091Z","shell.execute_reply":"2024-01-29T05:33:22.134848Z"},"papermill":{"duration":2203.741797,"end_time":"2024-01-29T05:33:22.153384","exception":false,"start_time":"2024-01-29T04:56:38.411587","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cb_bayes_models = {}\n\nfor col in list(y):\n    cb_bayes_models[col] = make_pipeline(\n        NumericalImputer(),\n        CatBoostClassifier(random_state = seed, verbose = 0, bootstrap_type = 'Bayesian', cat_features = categorical_features)\n    )\n    _, oof_preds, _ = cross_val_score_proba(cb_bayes_models[col], col)\n    oof_list[col].append(oof_preds)\n    \nprint('')","metadata":{"execution":{"iopub.execute_input":"2024-01-29T05:33:22.184556Z","iopub.status.busy":"2024-01-29T05:33:22.183677Z","iopub.status.idle":"2024-01-29T06:09:43.783276Z","shell.execute_reply":"2024-01-29T06:09:43.781812Z"},"papermill":{"duration":2181.626697,"end_time":"2024-01-29T06:09:43.794465","exception":false,"start_time":"2024-01-29T05:33:22.167768","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cb_bernoulli_models = {}\n\nfor col in list(y):\n    cb_bernoulli_models[col] = make_pipeline(\n        NumericalImputer(),\n        CatBoostClassifier(random_state = seed, verbose = 0, bootstrap_type = 'Bernoulli', cat_features = categorical_features)\n    )\n    _, oof_preds, _ = cross_val_score_proba(cb_bernoulli_models[col], col)\n    oof_list[col].append(oof_preds)\n    \nprint('')","metadata":{"execution":{"iopub.execute_input":"2024-01-29T06:09:43.826052Z","iopub.status.busy":"2024-01-29T06:09:43.824999Z","iopub.status.idle":"2024-01-29T06:43:16.970497Z","shell.execute_reply":"2024-01-29T06:43:16.969416Z"},"papermill":{"duration":2013.17832,"end_time":"2024-01-29T06:43:16.987367","exception":false,"start_time":"2024-01-29T06:09:43.809047","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# XGBoost","metadata":{"papermill":{"duration":0.014571,"end_time":"2024-01-29T06:43:17.016813","exception":false,"start_time":"2024-01-29T06:43:17.002242","status":"completed"},"tags":[]}},{"cell_type":"code","source":"xgb_models = {}\n\nfor col in list(y):\n    xgb_models[col] = make_pipeline(\n        MEstimateEncoder(cols = categorical_features),\n        SimpleImputer(),\n        XGBClassifier(random_state = seed, tree_method = 'hist')\n    )\n    _, oof_preds, _ = cross_val_score_proba(xgb_models[col], col)\n    oof_list[col].append(oof_preds)\n    \nprint('')","metadata":{"execution":{"iopub.execute_input":"2024-01-29T06:43:17.129141Z","iopub.status.busy":"2024-01-29T06:43:17.128714Z","iopub.status.idle":"2024-01-29T06:45:30.799625Z","shell.execute_reply":"2024-01-29T06:45:30.798193Z"},"papermill":{"duration":133.699844,"end_time":"2024-01-29T06:45:30.812195","exception":false,"start_time":"2024-01-29T06:43:17.112351","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Ensemble NN","metadata":{"papermill":{"duration":0.014948,"end_time":"2024-01-29T06:45:30.842435","exception":false,"start_time":"2024-01-29T06:45:30.827487","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def cross_val_score_nn(estimator, col, cv = kf, label = ''):\n    \n    X_oof = oof_list[col]\n        \n    model = clone(estimator).fit(X_oof, y[col])\n    val_preds = model.predict_proba(X_oof)\n    val_score = accuracy_score(y[col], val_preds.argmax(axis = 1))\n    \n    print(f'Val Score: {val_score:.5f} | {label}')\n    \n    return val_score","metadata":{"execution":{"iopub.execute_input":"2024-01-29T06:45:30.875169Z","iopub.status.busy":"2024-01-29T06:45:30.874371Z","iopub.status.idle":"2024-01-29T06:45:30.880556Z","shell.execute_reply":"2024-01-29T06:45:30.879807Z"},"papermill":{"duration":0.025169,"end_time":"2024-01-29T06:45:30.88277","exception":false,"start_time":"2024-01-29T06:45:30.857601","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class TensorFlower(BaseEstimator, ClassifierMixin):\n    \n    def fit(self, x, y):\n        \n        tf.keras.utils.set_random_seed(seed)\n        tf.config.experimental.enable_op_determinism()\n        \n        inputs = []\n        norm_inputs = []\n        \n        multiply = []\n        \n        for i in range(len(x)):\n            inputs.append(tf.keras.Input((len(np.unique(y)), )))\n            norm_inputs.append(tf.keras.layers.BatchNormalization()(inputs[i]))\n            \n        concat = tf.keras.layers.Concatenate()(norm_inputs)\n        \n        for i in range(len(x)):\n            dense = tf.keras.layers.Dense(len(np.unique(y)))(concat)\n            norm = tf.keras.layers.BatchNormalization()(dense)\n            act = tf.keras.activations.softmax(norm)\n            multiply.append(tf.keras.layers.Multiply()([inputs[i], act]))\n            \n        outputs = tf.keras.layers.Add()(multiply)\n        \n        self.model = tf.keras.Model(inputs, outputs)\n        self.model.compile(loss = 'sparse_categorical_crossentropy', optimizer = tf.keras.optimizers.AdamW(1e-4))\n        \n        self.model.fit(x, y, epochs = 50, verbose = 0)\n        \n        return self\n    \n    def predict_proba(self, x):\n        \n        return self.model.predict(x, verbose = 0)\n    \n    def predict(self, x):\n        \n        return np.argmax(self.predict_proba(x), axis = 1)","metadata":{"execution":{"iopub.execute_input":"2024-01-29T06:45:30.915267Z","iopub.status.busy":"2024-01-29T06:45:30.914483Z","iopub.status.idle":"2024-01-29T06:45:30.924814Z","shell.execute_reply":"2024-01-29T06:45:30.924047Z"},"papermill":{"duration":0.029221,"end_time":"2024-01-29T06:45:30.927107","exception":false,"start_time":"2024-01-29T06:45:30.897886","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in list(y):\n    cross_val_score_nn(TensorFlower(), col)\n    \nprint('\\n')","metadata":{"execution":{"iopub.execute_input":"2024-01-29T06:45:30.959625Z","iopub.status.busy":"2024-01-29T06:45:30.958636Z","iopub.status.idle":"2024-01-29T06:48:14.201221Z","shell.execute_reply":"2024-01-29T06:48:14.200024Z"},"papermill":{"duration":163.27622,"end_time":"2024-01-29T06:48:14.218501","exception":false,"start_time":"2024-01-29T06:45:30.942281","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction and Submission","metadata":{"papermill":{"duration":0.015352,"end_time":"2024-01-29T06:48:14.322618","exception":false,"start_time":"2024-01-29T06:48:14.307266","status":"completed"},"tags":[]}},{"cell_type":"code","source":"for col in list(y):\n    predict_list[col].append(log_models[col].fit(X, y[col]).predict_proba(test_pd))\n    predict_list[col].append(svm_models[col].fit(X, y[col]).predict_proba(test_pd))\n    predict_list[col].append(lgb_models[col].fit(X, y[col]).predict_proba(test_pd))\n    predict_list[col].append(cb_models[col].fit(X, y[col]).predict_proba(test_pd))\n    predict_list[col].append(cb_bayes_models[col].fit(X, y[col]).predict_proba(test_pd))\n    predict_list[col].append(cb_bernoulli_models[col].fit(X, y[col]).predict_proba(test_pd))\n    predict_list[col].append(xgb_models[col].fit(X, y[col]).predict_proba(test_pd))","metadata":{"execution":{"iopub.execute_input":"2024-01-29T06:48:14.355437Z","iopub.status.busy":"2024-01-29T06:48:14.355036Z","iopub.status.idle":"2024-01-29T07:13:06.847387Z","shell.execute_reply":"2024-01-29T07:13:06.846229Z"},"papermill":{"duration":1492.512251,"end_time":"2024-01-29T07:13:06.850299","exception":false,"start_time":"2024-01-29T06:48:14.338048","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This code iterates over each column in the target variable y. For each column, it predicts probabilities using various models (logistic regression, support vector machines, LightGBM, CatBoost, CatBoost with Bayes features, CatBoost with Bernoulli features, and XGBoost) and appends the results to a list. This likely aims to compare model performances for each target variable column.","metadata":{}},{"cell_type":"code","source":"for col in list(y):\n    ensembler = TensorFlower().fit(oof_list[col], y[col])\n    submission[col] = ensembler.predict(predict_list[col])\n    submission[col] = le[col].inverse_transform(submission[col].astype(np.int8))\n\nsubmission[['id_unique'] + list(y)].to_csv('cool_city.csv', index = False)","metadata":{"execution":{"iopub.execute_input":"2024-01-29T07:13:06.888262Z","iopub.status.busy":"2024-01-29T07:13:06.887491Z","iopub.status.idle":"2024-01-29T07:16:00.598198Z","shell.execute_reply":"2024-01-29T07:16:00.597125Z"},"papermill":{"duration":173.734399,"end_time":"2024-01-29T07:16:00.601369","exception":false,"start_time":"2024-01-29T07:13:06.86697","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This code iterates over each column in the target variable y. It creates an ensembler model using TensorFlower with out-of-fold predictions (oof_list) and true labels y. Then, it generates predictions using the model for the test data (predict_list), inverse transforms the predictions, and saves the results to a CSV file named 'cool_city.csv'.","metadata":{}},{"cell_type":"markdown","source":"## Reflection\n\nThe complete code showcases a comprehensive machine learning pipeline for predictive modeling in a city environment dataset. It encompasses data preprocessing, feature engineering, model training using various algorithms like logistic regression, SVM, ensemble methods like LightGBM, CatBoost, and XGBoost, as well as advanced techniques like ensembling with TensorFlower. Additionally, it incorporates cross-validation, feature importance analysis, and submission generation. This holistic approach ensures robustness and accuracy in predicting various aspects of city life, making it a powerful tool for urban planning, infrastructure optimization, and understanding citizen well-being.","metadata":{}},{"cell_type":"markdown","source":"# Citation\n\n- Clayton Miller, Matias Quintana, Mario Frei, Yun Xuan Chua, Chun Fu, Bianca Picchetti, Winston Yap, Adrian Chong, and Filip Biljecki. 2023. Introducing the Cool, Quiet City Competition: Predicting Smartwatch-Reported Heat and Noise with Digital Twin Metrics. In Proceedings of the 10th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys '23). Association for Computing Machinery, New York, NY, USA, 298–299. https://doi.org/10.1145/3600100.3626269","metadata":{"papermill":{"duration":0.015561,"end_time":"2024-01-29T07:16:00.632974","exception":false,"start_time":"2024-01-29T07:16:00.617413","status":"completed"},"tags":[]}}]}