{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":61145,"databundleVersionId":7047669,"sourceType":"competition"},{"sourceId":7758478,"sourceType":"datasetVersion","datasetId":4537055}],"dockerImageVersionId":30664,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Here is the code for Cool Quiet City Cmpetition<br>\nI don't like overweight model with vague parameters and rather prefer models where each parameter gives meaningful input.<br>\nIt's not a very competitive approach but during feature research I've got a lot of insights on Singapore climate, environmental noise and much more. <br>","metadata":{}},{"cell_type":"markdown","source":"# 1.Data loading.\nSimple code without  any new ideas","metadata":{}},{"cell_type":"code","source":"#module for conversion from lat-lon to geohash index\n!pip install pygeohash\n\nimport numpy as np \nimport pandas as pd \nimport pyproj\nimport pygeohash\nimport os\nfrom xgboost import XGBClassifier\nfrom pyproj import Transformer\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.model_selection import train_test_split\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dirname = '/kaggle/input/cool-quiet-city-competition'\n\n#read cozie_responses with date parsing\ndf_trains   = pd.read_csv(os.path.join(dirname, 'cozie_responses_and_physiological_data_training.csv'),\n                        parse_dates=['time', 'ws_timestamp_location', 'ws_timestamp_start'], index_col = 0)\n\n\ndf_solution = pd.read_csv(os.path.join(dirname, 'cozie_responses_and_physiological_data_test_public.csv'),\n                        parse_dates=['time', 'ws_timestamp_location', 'ws_timestamp_start'], index_col = 0)\n\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#merge train and test subsets for easier feature engeneering\ndf_trains['q_noise_nearby'].fillna('None',inplace = True)\ndf_trains['source']='train'\ndf_solution['source']='solution'\n\ndf_everything = pd.concat([df_trains, df_solution])\ndf_everything['time_sec'] = df_everything.index\ndf_everything['time_sec']  = df_everything['time_sec'].astype('int64')/10**9\n\n#Keep only microsurveys and not health and activity data\ndf_all = df_everything[df_everything.ws_survey_count.notna()].copy()\nprint('Total amount of microsurveys: ',len(df_all))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#It's always better to keep in metrical coordinates than in degrees when you are working with relatively small area.\n#It's not critical on Equator (where lon and lat have rel. the same scale), but very important in polar regions\ntransformer = Transformer.from_crs(\"EPSG:4326\", \"EPSG:32648\")\n\ndf_all['X'] = transformer.transform(df_all['ws_latitude'], df_all['ws_longitude'])[0]\ndf_all['Y'] = transformer.transform(df_all['ws_latitude'], df_all['ws_longitude'])[1]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"At the beginning it seems that noise is a very subjective parameter that depends on a personality and current mood not less than on absolute acoustic pressure. For sure, there will be general agreement of high noise near the airplane and highway and low noise in 2 AM in the residential area, <br>\nBut there are many cases when exactly the same noise could be evaluated differently by different people and even by one person after a bad day, for instance. <be>\nFrom one side task seems to be very intriguing - to create a model than predict not only real noise, but likely human attitude regarding ambient sound. <br>\nOn the other hand only 9431 microsurvey response looks unsofficient for such ambitious task...\n\n\n","metadata":{}},{"cell_type":"markdown","source":"<h1>2. Data examination</h1>\n\n<h2>2.1 Microsurvey questions</h2>\n\n<h3>2.1.1 Location </h3>","metadata":{}},{"cell_type":"markdown","source":"Let's start with microsurvey queston and have a looks what we could do with them.\nThere are two questions where second question depends on the response on the first one.\nAnd there are two possibilities - \n* Use dummies for all questions. This option doesn't looks good because of the question dependence. And a lot of NaN values as a result\n* Use one-hot encoder for every unique combination from three questions.\n\nI choosed second one and also group answers in a greater classes to avoid classes that have very low occurence\n","metadata":{}},{"cell_type":"code","source":"df_all['joined_loc'] = np.where(df_all['q_location'] == 'Transportation', df_all['q_location_transport'], df_all['q_location_office'])\ndf_all['joined_loc'].replace(to_replace = 'Not applicable', value =  '', inplace = True)\ndf_all['joined_loc'] = df_all['q_location'] +' ' + df_all['joined_loc']\n\ndf_location = pd.crosstab(df_all['joined_loc'],df_all['q_noise_nearby'], normalize = 'index')\ndf_noise_kind = pd.crosstab(df_all['joined_loc'],df_all['q_noise_kind'], normalize = 'index')\ndf_location = pd.merge(df_all['joined_loc'].value_counts(), df_location, left_index = True, right_index = True)\ndf_location = pd.merge(df_noise_kind, df_location, left_index = True, right_index = True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"By the treeplot we can estimate representation of each answer and size of the train subset.<br>\nFancy plotly implementation allows to zoom on any groop by clicking it","metadata":{}},{"cell_type":"code","source":"import plotly.express as px\nfig = px.treemap(df_all, path=[ 'q_location', 'joined_loc','source'])\nfig.update_layout(margin = dict(t=50, l=25, r=25, b=25))\nfig.data[0].textinfo = 'label + value'\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that Transportation subgroup is rather small and we probably could not get meaningful predictions based on the groups with 100 or less samples.<br>\nSo let's have a look on the prediction classes ratio for groups with small representation and union them  based on their closiness. \nFor this I've choosed a PCA analysis","metadata":{}},{"cell_type":"code","source":"from sklearn.decomposition import PCA\n\n\npca = PCA(n_components=2)\ncomponents = pca.fit_transform(df_location.drop('count', axis = 1))\n\nfig = px.scatter(components, x=0, y=1, color=df_location.index, size = df_location['count'].values, size_max = 100, text = df_location.index, title='PCA for noise')\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we look on two subcategories - ***Indoor-Office*** and ***Transport*** to see can we unite some of the subclasses<br>\n**Transport**<br>\nOn the plot we can see that Bus and Train categories are quite close to each other and we can join them. <br>\nAlso vague \"Transportation - other\" category is very close to the Outdoor group, which is reasonable, <br>\nas probably this category means bike, motorbike, skooter or any other vehicle when you are not isolated from the environmental noise. OK, unite them too!<br>\n**Indoor**<br>\nOn the screen center we can see dense group of four inputs- ***Indoor - Office*** with *Large open plain, Small shared, Cubicies* and ***Indoor - Class***.<br>\nThey are not identical but my feeling that all all of them are very similar relatevely to the environmental noise and 100-250 samples per group are definitely not enough to find real tiny differences.\nThey also will be united for the model!\n","metadata":{}},{"cell_type":"code","source":"def get_location(row):\n    if row['q_location'] in [ 'Indoor - Home', 'Outdoor', 'Indoor - Other'] :\n        return(row['q_location'])\n    elif row['q_location'] in ['Indoor - Class']:\n        return('Shared office')\n    elif row['q_location'] == 'Indoor - Office':\n        if row['q_location_office'] in ['Small shared', 'Large open plan','Cubicles']:\n            return('Shared office')\n        if row['q_location_office'] == 'Conference room':\n            return('Conference room')\n        else:\n            return('Ind office')\n    elif row['q_location'] == 'Transportation':\n        if row['q_location_transport'] in ['Car','Other','Taxi']:\n            return('Car')\n        elif row['q_location_transport'] in ['Bus','Train']:\n            return('Public transport')\ndf_all['detailed_loc_noise'] = df_all.apply(get_location, axis = 1)\ndf_all['detailed_loc_noise'] = df_all['detailed_loc_noise'].astype(\"category\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I am a bit unsure about headphones using as it could be similar with noise, but also could have specific ratio and it has to be carefully checked. ","metadata":{}},{"cell_type":"code","source":"subset = df_all.query('source == \"train\"')\ndf_headphones = pd.crosstab(subset['joined_loc'],subset['q_earphones'], normalize = 'index')\ndf_headphones = pd.merge(subset['joined_loc'].value_counts(), df_headphones, left_index = True, right_index = True)\n\n\npca = PCA(n_components=2)\ncomponents = pca.fit_transform(df_headphones.drop('count', axis = 1))\n\nfig = px.scatter(components, x=0, y=1, color=df_headphones.index, size = df_headphones['count'].values, size_max = 100, text = df_headphones.index, title='PCA for headphones')\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Great! It looks very similar, Bus and Train together, and different options of Offices also grouped together. So let's keep the same grouping for headphone prediction ","metadata":{}},{"cell_type":"code","source":"subset = df_all.query('source == \"train\"')\ndf_temp = pd.crosstab(subset['joined_loc'],subset['q_thermal_preference'], normalize = 'index')\ndf_temp = pd.merge(subset['joined_loc'].value_counts(), df_temp, left_index = True, right_index = True)\n\npca = PCA(n_components=2)\ncomponents = pca.fit_transform(df_temp.drop('count', axis = 1))\n\nfig = px.scatter(components, x=0, y=1, color=df_temp.index, size = df_temp['count'].values, size_max = 100, text = df_temp.index, title='PCA for temperature')\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Thermal preference looks different from the noise. <br>\nThere is an obvious difference between ***Train*** and ***Bus***, so it's not possible to unite them<br>\nOnly two groups could be united - ***Office - Individual*** and ***Indoor - Home***, that have perfect sense as in both cases you have your own AC.<br>\n","metadata":{}},{"cell_type":"code","source":"def get_location(row):\n    if row['q_location'] == 'Indoor - Office' and row['q_location_office'] == 'Office - Individual':\n        return('Personal AC')\n    elif row['q_location'] == 'Indoor - Home':\n        return('Personal AC')\n    else:\n        return('joined_loc')\ndf_all['detailed_loc_temp'] = df_all.apply(get_location, axis = 1)\ndf_all['detailed_loc_temp'] = df_all['detailed_loc_temp'].astype(\"category\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<h3>2.1.2 Group question </h3>","metadata":{}},{"cell_type":"code","source":"df_all['joined_activity'] = df_all['q_alone_group'] +' ' + df_all['q_activity_category_group']\n\nimport plotly.express as px\nfig = px.treemap(df_all, path=[ 'q_alone_group', 'q_activity_category_group','source'])\nfig.update_layout(margin = dict(t=50, l=25, r=25, b=25))\nfig.data[0].textinfo = 'label + value'\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well, all the groups looks rather represantative","metadata":{}},{"cell_type":"code","source":"df_group = pd.crosstab(df_all['joined_activity'],df_all['q_noise_nearby'], normalize = 'index')\ndf_noise_kind = pd.crosstab(df_all['joined_activity'],df_all['q_noise_kind'], normalize = 'index')\ndf_group = pd.merge(df_all['joined_activity'].value_counts(), df_group, left_index = True, right_index = True)\ndf_group = pd.merge(df_noise_kind, df_group, left_index = True, right_index = True)\n\nfrom sklearn.decomposition import PCA\n\n\npca = PCA(n_components=2)\ncomponents = pca.fit_transform(df_group.drop('count', axis = 1))\n\nfig = px.scatter(components, x=0, y=1, color=df_group.index, size = df_group['count'].values, size_max = 100, text = df_group.index, title='PCA for noise')\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Seems all the groups are unique except two - **Other** and **Socialise**.\nThere will be no harm to join them. \nSame works for q_earphones. But not for the q_thermal preference","metadata":{}},{"cell_type":"code","source":"df_group = pd.crosstab(df_all['joined_activity'],df_all['q_thermal_preference'], normalize = 'index')\ndf_group = pd.merge(df_all['joined_activity'].value_counts(), df_group, left_index = True, right_index = True)\n\nfrom sklearn.decomposition import PCA\n\n\npca = PCA(n_components=2)\ncomponents = pca.fit_transform(df_group.drop('count', axis = 1))\n\nfig = px.scatter(components, x=0, y=1, color=df_group.index, size = df_group['count'].values, size_max = 100, text = df_group.index, title='PCA for thermal preference')\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_all['detailed_group_noise'] = df_all['joined_activity'].apply(lambda x: 'Group party' if x in ['Group Socialize', 'Group Other'] else x)\ndf_all['detailed_group_noise']  = df_all['detailed_group_noise'].astype('category')\ndf_all['id_participant']  = df_all['id_participant'].astype('category')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<h2>2.2 Date and time of the microsurvey </h2><br>\nObviously the time and day of the week are the very important things for noise estimation<br>\nAnd time also should reflect thermal_preference. <br>\nLet's have a look<br>","metadata":{}},{"cell_type":"code","source":"#Weekend has to be more quiet time than weekdays\ndf_all['dayofweek'] = df_all.index.dayofweek.astype('category')\ndf_all['dayofyear'] = df_all.index.dayofyear\n\n\n#There is a natural gap in the deep night so no need to convert it to circular sin-cos values\ndf_all['minutes'] = df_all.index.hour*60 + df_all.index.minute\ndf_all['minutes'] = df_all['minutes'].astype('int')\n\nimport matplotlib.pyplot as plt\nplt.hist(df_all['minutes'])\n\ndf_all['hour']= df_all.index.hour\ndef get_dayperiod(hour):\n    if 7<=hour<9: return('morning')\n    elif 9<=hour<21: return('day')\n    elif 21<=hour<23: return('evening')\n    else:\n        return('night')\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<h2>2.3 Coordinates of microsurvey </h2><br>\nGeographic location is a very important feature, but using just lat and lon values has no any sence, as this values itself are not related to noise anyhow. <br>\nI choosed between three approaches:<br>\n* Geohash<br>\n* Parsed real adress from openstreetmap<br>\n* K-means for coordinates to split all the dataset in a reasonable amount of groups<br>\nAll of them showed similar results of 5-10% better than just raw lat lon. I stopped on an option with parsed real adress\n\n","metadata":{}},{"cell_type":"code","source":"df_all['adress_geohash'] = df_all.apply(lambda x: pygeohash.encode(x.ws_latitude, x.ws_longitude, precision=7), axis = 1).astype('category')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#This cell take a lot of time because of the requests to the OSM server\n#The generated feature not using in the prediction anymore\n\n'''\nfrom geopy.geocoders import Nominatim\ngeolocator = Nominatim(user_agent=\"place your name\")\nadress_dict = {}\nq= 0 \ndef get_adress(row):\n    global q\n    q+=1\n    if q % 500 == 0: \n        print(q)\n    lat = round(row.ws_latitude,4)\n    lon = round(row.ws_longitude,4)\n    if (lat,lon) not in adress_dict.keys():\n        location = geolocator.reverse(\"{},{}\".format(lat,lon))\n        adress_dict[(lat,lon)] = location\n        return(location)\n    else:\n        return(adress_dict[(lat,lon)])\ndf_all['adress'] = df_all.apply(get_adress, axis = 1)\ndf_all['adress_cat'] = df_all['adress'].apply(str)\n'''","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2.4 Weather \n\nChallenge organisators provided extensive dataset of weather conditions.<br>\nWe need weather information for two cases:<br>\nPredict specific noise_kind - ***Weather*** <br>\nPredict air conditions for **q_thermal_preference**<br>\n\n","metadata":{}},{"cell_type":"markdown","source":"<h2>2.4.1 Noise prediction</h2>\nFor the noise prediction it turned out that noise related to weather caused only by thunderstorms and <br>\nthunderstorm itself unfortunately doesn't correlate with any weather parameters. Sometimes you have obvious lightning without rain and wind<br>\nSometimes you have severe wind without any lightning<br>\nSo only one possibility is to check responses of the other participants and trying to predict response ***Weather*** by closiness to someone who already state about noise related to weather<br>","metadata":{}},{"cell_type":"code","source":"df_thunder = df_all.query('q_noise_kind == \"Weather\"')\ndef get_closest_thunder(time):\n    df_thunder['time_diff'] = df_thunder['time_sec'] - time\n    closest = df_thunder['time_diff'].min()\n    return(closest)\ndf_all['nearest_thunder'] = df_all['time_sec'].apply(get_closest_thunder)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<h2>2.4.2 Thermal_preference prediction</h2>\nInfluence of the actual air temperature and humidity have curiosly small influence on q_thermal_preference prediction. <br>\nProbably it is because the air temperature in Singapore have a very small variation. And the bigger variance come from the knowledge did respondent take his jacket today or not.<br>\nBut anyway we will calculate Heat index that combine both humidity and temperature in one variable.","metadata":{}},{"cell_type":"code","source":"from pyproj import Transformer\ntransformer = Transformer.from_crs(\"EPSG:4326\", \"EPSG:32648\")\n\n#Load the weather stations data\ndf_stations = pd.read_csv(os.path.join(dirname,'weather_stations.csv')).drop(columns=['Unnamed: 0'])\ndf_temperature = pd.read_csv(os.path.join(dirname,'weather_air-temperature.csv'), parse_dates=True, index_col=[0])\ndf_humidity = pd.read_csv(os.path.join(dirname,'weather_relative-humidity.csv'), parse_dates=True, index_col=[0])\n\n#I don't like lat lon and in case of small areas prefer to convert it to X Y as it have much more sense.\ndf_stations['X']= transformer.transform(df_stations['latitude'], df_stations['longitude'])[0]\ndf_stations['Y']= transformer.transform(df_stations['latitude'], df_stations['longitude'])[1]\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Take only stations that exist in both humidity and temperature dataframes\ncommon_stations = list(set(df_temperature.columns) & set(df_humidity.columns))\ncommon_stations.sort()\nprint(common_stations)\ndf_stations_temp_subset = df_stations[df_stations['id'].isin(common_stations)]\ndf_stations_temp_subset.sort_values('id', inplace = True)\n\n#We keep stations coordinates in numpy to speedify calculation of distances between observation point and weather stations\narray_stations_temp = np.array(df_stations_temp_subset[['X','Y']])\n\n\n#Make numpy-array instead of dataframes to speedify significantly the calculations\nhum = np.array(df_humidity[common_stations].resample(rule='100s').mean())\ntemp = np.array(df_temperature[common_stations].resample(rule='100s').mean())\n\n#We keep indexes and columns of the dataframe to join them back after calculations of numpy arrays\nresampled = df_humidity.resample(rule='100s').mean()\nresampled = resampled[common_stations]\n\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Calculate Heat Index based on wikipedia formula\nc1 = -8.78469475556\nc2 = 1.61139411\nc3 = 2.33854883889\nc4 = -0.14611605\nc5 = -0.012308094\nc6 = -0.0164248277778\nc7 = 2.211732e-3\nc8 = 7.2546e-4\nc9 = -3.582e-6\n\nHI = c1 + c2 * temp + c3 * hum + c4 *temp * hum + c5 * temp**2 + c6 * hum**2 + c7 * temp**2 * hum + c8 * temp * hum**2 + c9 * temp**2 * hum**2\nHI_df = pd.DataFrame(HI, index = resampled.index, columns = resampled.columns)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#As we had 100 second resampling now we have to calculate specific column to perform join of initial dataframe with HI dataframe\ndf_all['time_from_beginning_temp'] = df_all.index - resampled.iloc[0].name\ndf_all['time_from_beginning_temp'] = df_all['time_from_beginning_temp'].apply(lambda x: x.total_seconds())\ndf_all['time_from_beginning_temp'] =  df_all['time_from_beginning_temp'] // 100\n\n\n#Function for searching 2 closest station and return statistics base on them\ndef get_closest(row, measurements_array, stations_array, agg_func,timestamp):\n    distance_series = ((stations_array[:,0] - row.X)**2 + (stations_array[:,1] - row.Y)**2)**0.5\n    closest_stat = distance_series.argsort()[:2]\n    distance = distance_series[closest_stat]\n    timestamp = int(row[timestamp])\n    if agg_func =='sum':\n        values = np.sum(measurements_array[timestamp,closest_stat], axis = 0)\n    if agg_func =='max':       \n        values = measurements_array[timestamp,closest_stat]\n    if agg_func =='mean':       \n        values = np.mean(measurements_array[timestamp-1:timestamp+1,closest_stat], axis = 0)\n    \n    values = np.nan_to_num(values, nan = np.nanmean(values))\n    res = sum( values * distance) / sum(distance)\n    return(res)\n\ndf_all['HI'] = df_all.apply(get_closest,axis = 1, args = (HI,array_stations_temp,'mean','time_from_beginning_temp'))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<h2>2.5 Data from wearable sensors</h2>","metadata":{}},{"cell_type":"code","source":"df_audio = df_everything[df_everything['ts_audio_exposure_environment'].notna()]\n#df_all.drop('ts_audio_exposure_environment_y',axis = 'columns', inplace = True)\n\ndf_all_new = pd.DataFrame(columns = df_all.columns)\nfor id_part in df_all['id_participant'].unique():\n    sub = pd.merge_asof(df_all.query('id_participant == @id_part'),df_audio.query('id_participant == @id_part')['ts_audio_exposure_environment'],left_index = True, right_index = True,tolerance = pd.Timedelta(minutes = 60))\n\n    df_all_new = pd.concat([df_all_new,sub])\ndf_all = df_all_new.copy()\n\ndf_all['ts_audio_exposure_environment'] = df_all['ts_audio_exposure_environment'].astype('float')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Idea is to get mean value of a pulse in a last 10 minutes before the response\n#It should be represented as an absolute and z-score value\n\nmean_dict = dict(df_everything.groupby('id_participant')['ts_heart_rate'].mean())\nstd_dict = dict(df_everything.groupby('id_participant')['ts_heart_rate'].std())\ndf_pulse = df_everything[df_everything['ts_heart_rate'].notna()]\n\n\npart_dict = {}\nfor name in df_everything['id_participant'].unique():\n    part_dict[name] = df_pulse.query('id_participant == @name')\n\ndef get_closest_health(row):\n    res = part_dict[row.id_participant].query(' 0<@row.time_sec - time_sec <600')\n\n    mean_10T_BPM = res['ts_heart_rate'].mean()\n    #print(mean_10T_BPM,mean_dict[row.id_participant],std_dict[row.id_participant])\n    mean_10T_BPM_zscore = (mean_10T_BPM - mean_dict[row.id_participant])/ std_dict[row.id_participant]\n    return(mean_10T_BPM, mean_10T_BPM_zscore)\n\ndf_all[['mean_10T_BPM','mean_10T_BPM_zscore']] = df_all.apply(get_closest_health, axis = 1, result_type = 'expand')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<h1>3 Model</h1>","metadata":{}},{"cell_type":"code","source":"\n#Final preparations....\ndf_all['id_participant'] = df_all['id_participant'].astype(\"category\")\ndf_all['detailed_loc_temp'] = df_all['detailed_loc_temp'].astype(\"category\")\ndf_all['detailed_loc_noise'] = df_all['detailed_loc_noise'].astype(\"category\")\ndf_all['detailed_group_noise'] = df_all['detailed_group_noise'].astype(\"category\")\ndf_all['joined_activity'] = df_all['joined_activity'].astype(\"category\")\ndf_all['dayofweek'] = df_all['dayofweek'].astype(\"category\")\ndf_all['dayofyear'] = df_all['dayofyear'].astype(\"int\")\ndf_all['adress_geohash'] = df_all['adress_geohash'].astype('category')\ndf_all['id_participant'] = df_all['id_participant'].astype(\"category\")\ndf_all['minutes'] = df_all['minutes'].astype(\"int\")\ndf_train = df_all[df_all.source == 'train'].copy()\ndf_solution = df_all[df_all.source == 'solution'].copy()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# encoding categorical variables  copy from the initial notebook\n#As I use XGboost no need to encode X features, only outcome\ndict_cat = {}\ncat_vars = ['q_noise_kind','q_noise_nearby','q_earphones','q_thermal_preference']\nfor var in cat_vars:\n    print(var)\n    # convert categorical variables into 'category' type\n    df_train[var] = df_train[var].astype('category')\n    df_solution[var] = df_solution[var].astype('category')\n    \n    # dictionary to keep track of the variable mapping: {variable-name: {number1: string1, number2: string2, ...}\n    dict_cat[var] = dict(enumerate(df_train[var].cat.categories))\n\n    # encode variables, missing values are encoded as -1\n    df_train[var] = df_train[var].cat.codes\n    df_solution[var] = df_solution[var].cat.codes","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Specify set of parameters for each question response prediction\nthermal_set = ['id_participant','HI','detailed_loc_temp','minutes', 'mean_10T_BPM_zscore','dayofyear'] \nearphones_set = ['ts_audio_exposure_environment','id_participant','detailed_loc_noise','minutes','adress_geohash','detailed_group_noise']\nnoise_nearby_set = ['adress_geohash','dayperiod','id_participant','detailed_loc_noise','detailed_group_noise','nearest_thunder']\nnoise_kind_set = ['adress_geohash','id_participant','detailed_loc_noise','minutes','dayofweek','detailed_group_noise','nearest_thunder']\n\nX = df_train.copy()\ny = df_train[['id_unique','id_participant','q_noise_kind','q_noise_nearby','q_earphones','q_thermal_preference']]\nX_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trial = False\nif trial == False:\n    X_train = df_train.copy()\n    X_test = df_solution.copy()\nif trial == True:\n    X_train, X_test, = train_test_split(df_train, test_size=0.2)\ndf_final = X_test[['id_unique','id_participant']].copy()\nprint(X_train.shape)\n\nacc_score = []\nparameter = 'q_noise_kind'\ny_train = X_train[parameter]\nclf_nk = XGBClassifier( enable_categorical=True, objective= 'multi: softprob', learning_rate =  0.05, max_depth =  2, subsample = 1.0)\nclf_nk.fit(X_train[noise_kind_set], y_train)\ndf_final[parameter] = clf_nk.predict(X_test[noise_kind_set])\n\nif trial == True:\n    score = accuracy_score(X_test[parameter], df_final[parameter])\n    print('Noise_kind score: ', score)\n    acc_score.append(score)\ndf_final[parameter] = df_final[parameter].replace(dict_cat[parameter])\n\nparameter = 'q_noise_nearby'\ny_train = X_train[parameter]\nclf_nn = XGBClassifier( enable_categorical=True, objective= 'multi: softprob', learning_rate =  0.05, max_depth =  2, subsample = 1.0)\nclf_nn.fit(X_train[noise_nearby_set], y_train)\ndf_final[parameter] = clf_nn.predict(X_test[noise_nearby_set])\n\nif trial == True:\n    score = accuracy_score(X_test[parameter], df_final[parameter])\n    print('Noise_nearby score: ', score)\n    acc_score.append(score)\ndf_final[parameter] = df_final[parameter].replace(dict_cat[parameter])\n\nparameter = 'q_thermal_preference'\ny_train = X_train[parameter]\nclf_tp = XGBClassifier( enable_categorical=True, objective= 'multi: softprob', learning_rate =  0.05, max_depth =  2, subsample = 1.0)\nclf_tp.fit(X_train[thermal_set], y_train)\ndf_final[parameter] = clf_tp.predict(X_test[thermal_set])\n\nif trial == True:\n    score = accuracy_score(X_test[parameter], df_final[parameter])\n    print('Thermal preference score: ', score)\n    acc_score.append(score)\ndf_final[parameter] = df_final[parameter].replace(dict_cat[parameter])\n\nparameter = 'q_earphones'\ny_train = X_train[parameter]\nclf_e = XGBClassifier( enable_categorical=True, objective= 'multi: softprob', learning_rate =  0.05, max_depth =  2, subsample = 1.0)\nclf_e.fit(X_train[earphones_set], y_train)\ndf_final[parameter] = clf_e.predict(X_test[earphones_set])\n\nif trial == True:\n    score = accuracy_score(X_test[parameter], df_final[parameter])\n    print('Earphones score: ', score)\n    acc_score.append(score)\ndf_final[parameter] = df_final[parameter].replace(dict_cat[parameter])\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n#Prepare data for submission\nsubmission = df_final[['id_unique', 'q_noise_nearby', 'q_noise_kind', 'q_earphones', 'q_thermal_preference']].set_index('id_unique')\nsubmission.sort_index(inplace = True)\n# Write dataframe to csv file\nsubmission.sort_index().to_csv('submission_all_train4.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}