{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":61145,"databundleVersionId":7047669,"sourceType":"competition"}],"dockerImageVersionId":30665,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Applications","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"This notebook has to be created through the competition","metadata":{}},{"cell_type":"markdown","source":"## Table of Contents\n* [Read data](#Read-data)\n* [Data pre-processing](#Data-pre-processing)\n* [Map of all weather stations that measure air temperature](#Map-of-all-weather-stations-that-measure-air-temperature)\n* [What's next?](#What's-next)","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\nimport math","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:10:05.179542Z","iopub.execute_input":"2024-03-14T16:10:05.180196Z","iopub.status.idle":"2024-03-14T16:10:09.207805Z","shell.execute_reply.started":"2024-03-14T16:10:05.180167Z","shell.execute_reply":"2024-03-14T16:10:09.206732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Read data","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv('/kaggle/input/cool-quiet-city-competition/cozie_responses_and_physiological_data_training.csv', index_col=[0], \n                       keep_default_na=False, na_values=[''], parse_dates=['time', 'ws_timestamp_location', 'ws_timestamp_start'], \n                       date_format='%Y-%m-%d %H:%M:%S.%f%z')\ndf_solution = pd.read_csv('/kaggle/input/cool-quiet-city-competition/cozie_responses_and_physiological_data_test_public.csv', index_col=[0],\n                          keep_default_na=False, na_values=[''], parse_dates=['time', 'ws_timestamp_location', 'ws_timestamp_start'], \n                          date_format='%Y-%m-%d %H:%M:%S.%f%z')\ndf_rain = pd.read_csv('/kaggle/input/cool-quiet-city-competition/weather_rainfall.csv', parse_dates=True, index_col=[0])\ndf_temperature = pd.read_csv('/kaggle/input/cool-quiet-city-competition/weather_air-temperature.csv', parse_dates=True, index_col=[0])\ndf_humidity = pd.read_csv('/kaggle/input/cool-quiet-city-competition/weather_relative-humidity.csv', parse_dates=True, index_col=[0])\ndf_wind_speed = pd.read_csv('/kaggle/input/cool-quiet-city-competition/weather_wind-speed.csv', parse_dates=True, index_col=[0])\ndf_wind_direction = pd.read_csv('/kaggle/input/cool-quiet-city-competition/weather_wind-direction.csv', parse_dates=True, index_col=[0])\ndf_stations = pd.read_csv('/kaggle/input/cool-quiet-city-competition/weather_stations.csv').drop(columns=['Unnamed: 0'])","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:10:09.209682Z","iopub.execute_input":"2024-03-14T16:10:09.210071Z","iopub.status.idle":"2024-03-14T16:12:14.908541Z","shell.execute_reply.started":"2024-03-14T16:10:09.210045Z","shell.execute_reply":"2024-03-14T16:12:14.907588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* **df_train**: Reads a CSV file.\n* **df_solution**: Reads a CSV file and converts certain columns to date and time objects.\n* **df_rain**, df_temperature, df_humidity, df_wind_speed, df_wind_direction: Each of these DataFrames reads a corresponding CSV file containing different weather data (rainfall, temperature, humidity, wind speed, and wind direction) from the specified location. These DataFrames also convert the date column to date and time objects.\n* **df_stations**: Reads a CSV file named 'weather_stations.csv' from the specified location.","metadata":{}},{"cell_type":"code","source":"df_stations.info()","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:14.914846Z","iopub.execute_input":"2024-03-14T16:12:14.915520Z","iopub.status.idle":"2024-03-14T16:12:14.946034Z","shell.execute_reply.started":"2024-03-14T16:12:14.915483Z","shell.execute_reply":"2024-03-14T16:12:14.944875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data pre-processing\n\n### Check dates\n\nWhen checking the last date of different kinds of dataframes it becomes clear that the watch survey responses only last until Jan 21, 2023, while the weather data lasts until July 3, 2023.","metadata":{}},{"cell_type":"code","source":"print(df_train.index.max())\nprint(df_solution.index.max())\nprint(df_rain.index.max())","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:14.947381Z","iopub.execute_input":"2024-03-14T16:12:14.947681Z","iopub.status.idle":"2024-03-14T16:12:14.971548Z","shell.execute_reply.started":"2024-03-14T16:12:14.947654Z","shell.execute_reply":"2024-03-14T16:12:14.970358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We don't really need the weather data after Jan 21, 2023, so we might just as well get rid of it.","metadata":{}},{"cell_type":"code","source":"df_rain = df_rain[df_rain.index<'2023-01-22']\ndf_temperature = df_temperature[df_temperature.index<'2023-01-22']\ndf_humidity = df_humidity[df_humidity.index<'2023-01-22']\ndf_wind_speed = df_wind_speed[df_wind_speed.index<'2023-01-22']\ndf_wind_direction = df_wind_direction[df_wind_direction.index<'2023-01-22']","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:14.973532Z","iopub.execute_input":"2024-03-14T16:12:14.974076Z","iopub.status.idle":"2024-03-14T16:12:15.016999Z","shell.execute_reply.started":"2024-03-14T16:12:14.974017Z","shell.execute_reply":"2024-03-14T16:12:15.016104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Check weather data","metadata":{}},{"cell_type":"code","source":"# Show new shortened air temperature dataframe\ndf_temperature.describe()","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:15.018205Z","iopub.execute_input":"2024-03-14T16:12:15.018512Z","iopub.status.idle":"2024-03-14T16:12:15.175932Z","shell.execute_reply.started":"2024-03-14T16:12:15.018485Z","shell.execute_reply":"2024-03-14T16:12:15.174894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It looks like there is no temperature data available in before Jan 21, 2023 for weather station S117. So, we might just as well remove that column","metadata":{}},{"cell_type":"code","source":"df_temperature = df_temperature.drop(columns=['S117'])\n# Get dimensions of the air temperature data\ndf_temperature.shape","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:15.177438Z","iopub.execute_input":"2024-03-14T16:12:15.177888Z","iopub.status.idle":"2024-03-14T16:12:15.189738Z","shell.execute_reply.started":"2024-03-14T16:12:15.177847Z","shell.execute_reply":"2024-03-14T16:12:15.188305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The weather dataframes are rather large. This can make the plotting of the data slow and the output images large. To make it a bit lighter, we are going to resample the data hourly.","metadata":{"execution":{"iopub.status.busy":"2024-03-12T21:09:04.412698Z","iopub.execute_input":"2024-03-12T21:09:04.413286Z","iopub.status.idle":"2024-03-12T21:09:04.419237Z","shell.execute_reply.started":"2024-03-12T21:09:04.413253Z","shell.execute_reply":"2024-03-12T21:09:04.41811Z"}}},{"cell_type":"code","source":"# Resample weather data\ndf_temperature_average = df_temperature.resample('60min').mean()\ndf_rain_sum = df_rain.resample('60min').sum() # note: for rain the data is aggregated with a sum instead of the mean\ndf_humidity_average = df_humidity.resample('60min').mean()\ndf_wind_speed_average = df_wind_speed.resample('60min').mean()\ndf_wind_direction_average = df_wind_direction.resample('60min').mean()\n\n# Check size of resulting dataframe\ndf_temperature_average.shape","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:15.191323Z","iopub.execute_input":"2024-03-14T16:12:15.191657Z","iopub.status.idle":"2024-03-14T16:12:15.288488Z","shell.execute_reply.started":"2024-03-14T16:12:15.191630Z","shell.execute_reply":"2024-03-14T16:12:15.287381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Map watch survey questions to question ID\n","metadata":{}},{"cell_type":"code","source":"ws_questions = {\"q_noise_nearby\": \"Noise distractions nearby? (without earphones)\",\n                \"q_noise_kind\": \"What kind of noise?\",\n                \"q_earphones\": \"Wearing earphones?\",\n                \"q_thermal_preference\": \"Thermally, what do you prefer now?\",\n                \"q_location\": \"Where are you?\",\n                \"q_location_office\": \"What kind of office?\",\n                \"q_location_transport\": \"What kind of transport?\",\n                \"q_alone_group\": \"Alone or in a group?\",\n                \"q_activity_category_alone\": \"Category of activity? (alone)\",\n                \"q_activity_category_group\": \"Category of activity? (group)\"}","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:15.292355Z","iopub.execute_input":"2024-03-14T16:12:15.292706Z","iopub.status.idle":"2024-03-14T16:12:15.298705Z","shell.execute_reply.started":"2024-03-14T16:12:15.292677Z","shell.execute_reply":"2024-03-14T16:12:15.297573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plot weather data\n\n### Plot weather weather data time-series¶\n\nBelow we are going to plot the air temperature data from all weather stations. I used the Matplotlib method plt.plot() instead of sns.lineplot(). The reason is that plt.plot() shows the gaps in the data, while sns.lineplot() draws straight lines accross the gaps.\n","metadata":{}},{"cell_type":"code","source":"cols_num = 1\nrows_num = len(df_temperature_average.columns)\ncol_current = 0\nrow_current = 0 \nfig, axs = plt.subplots(rows_num, cols_num, figsize=(20,3*rows_num))\nplt.subplots_adjust(hspace = 0.5)\n\nfor i, col in enumerate(df_temperature_average.columns):\n    row_current = i\n    axs[row_current].plot(df_temperature_average.index, df_temperature_average[col])\n    station_name = df_stations[df_stations['id']==col]['name'].values[0]\n    axs[row_current].set_title(f'Weather Station {col}: {station_name}')\n    axs[row_current].set_ylabel('Air Temperature [°C]')","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:15.300306Z","iopub.execute_input":"2024-03-14T16:12:15.300667Z","iopub.status.idle":"2024-03-14T16:12:21.381940Z","shell.execute_reply.started":"2024-03-14T16:12:15.300637Z","shell.execute_reply":"2024-03-14T16:12:21.380893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* **Variable Definition**: Variables are defined for the number of subplot columns, the number of rows (based on the number of columns in the DataFrame `df_temperature_average`), and position variables `col_current` and `row_current`.\n\n* **Subplot Creation**: The `plt.subplots()` function is used to create a figure and a series of subplots. The number of rows and columns, as well as the figure size (`figsize`), are specified. Additionally, `plt.subplots_adjust()` is used to adjust the vertical space between the subplots (`hspace`).\n\n* **Iteration through DataFrame Columns**: A `for` loop is used to iterate over each column of the DataFrame `df_temperature_average`. Within the loop: The current index is assigned to `row_current`. The average air temperature of the current station is plotted on the corresponding subplot using `axs[row_current].plot()`. The name of the current weather station is obtained using the DataFrame `df_stations` and set as the title of the subplot with `axs[row_current].set_title()`. The y-axis label is set to 'Air Temperature [°C]' using `axs[row_current].set_ylabel()`.","metadata":{}},{"cell_type":"markdown","source":"Since, we are going to plot more weather data, we might just as well wrap the above code for plotting in to a function.","metadata":{}},{"cell_type":"code","source":"# Function that plots weather data fromm all weather stations\ndef plot_weather(df, ylabel):\n    cols_num = 1\n    rows_num = len(df.columns)\n    col_current = 0\n    row_current = 0 \n    fig, axs = plt.subplots(rows_num, cols_num, figsize=(20,3*rows_num))\n    plt.subplots_adjust(hspace = 0.5)\n\n    for i, col in enumerate(df.columns):\n        row_current = i\n        axs[row_current].plot(df.index, df[col])\n        station_name = df_stations[df_stations['id']==col]['name'].values[0]\n        axs[row_current].set_title(f'Weather Station {col}: {station_name}')\n        axs[row_current].set_ylabel(ylabel)\n    \n    return fig, axs","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:21.383247Z","iopub.execute_input":"2024-03-14T16:12:21.383552Z","iopub.status.idle":"2024-03-14T16:12:21.392344Z","shell.execute_reply.started":"2024-03-14T16:12:21.383526Z","shell.execute_reply":"2024-03-14T16:12:21.391063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Temperature","metadata":{}},{"cell_type":"code","source":"fig, axs = plot_weather(df_temperature_average, 'Temperature [°C]')","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:21.393476Z","iopub.execute_input":"2024-03-14T16:12:21.393780Z","iopub.status.idle":"2024-03-14T16:12:27.510314Z","shell.execute_reply.started":"2024-03-14T16:12:21.393754Z","shell.execute_reply":"2024-03-14T16:12:27.509283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Rainfall","metadata":{}},{"cell_type":"code","source":"fig, axs = plot_weather(df_rain_sum, 'Rainfall [mm/h]')","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:27.511641Z","iopub.execute_input":"2024-03-14T16:12:27.512445Z","iopub.status.idle":"2024-03-14T16:12:52.504158Z","shell.execute_reply.started":"2024-03-14T16:12:27.512413Z","shell.execute_reply":"2024-03-14T16:12:52.503210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Relative humidity","metadata":{}},{"cell_type":"code","source":"fig, axs = plot_weather(df_humidity_average, 'Relative humidity [%]')\n","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:52.505385Z","iopub.execute_input":"2024-03-14T16:12:52.505751Z","iopub.status.idle":"2024-03-14T16:12:58.605821Z","shell.execute_reply.started":"2024-03-14T16:12:52.505724Z","shell.execute_reply":"2024-03-14T16:12:58.604855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Map of all weather stations that measure air temperature","metadata":{}},{"cell_type":"code","source":"# Have a look at the weather station meta data\ndf_stations.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:58.607282Z","iopub.execute_input":"2024-03-14T16:12:58.607963Z","iopub.status.idle":"2024-03-14T16:12:58.619270Z","shell.execute_reply.started":"2024-03-14T16:12:58.607932Z","shell.execute_reply":"2024-03-14T16:12:58.618176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_stations.shape","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:58.620536Z","iopub.execute_input":"2024-03-14T16:12:58.620820Z","iopub.status.idle":"2024-03-14T16:12:58.630369Z","shell.execute_reply.started":"2024-03-14T16:12:58.620781Z","shell.execute_reply":"2024-03-14T16:12:58.629452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Have a look at the air temperature data\ndf_temperature.columns","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:58.631633Z","iopub.execute_input":"2024-03-14T16:12:58.631939Z","iopub.status.idle":"2024-03-14T16:12:58.642408Z","shell.execute_reply.started":"2024-03-14T16:12:58.631906Z","shell.execute_reply":"2024-03-14T16:12:58.641490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note, the list of all weather stations is much longer then the list with weather stations that can measure air temperature. Hence, we need to pre-process the data a bit and only pick the weather stations that can measure temperature data.","metadata":{}},{"cell_type":"code","source":"# Initialize a new dataframe for creating the map\ndf_map = pd.DataFrame(columns=['latitude', 'longitude', 'text'])\n\n\n# Go through each column of the temperature dataframe\nfor col in df_temperature.columns:\n    # Get the weather station name based on the station id\n    station_name = df_stations.loc[df_stations.id==col].name.values[0]\n    \n    # Create a new row, with latitude, longitude, hover text, and marker color as columns\n    df_row = pd.DataFrame([{'latitude': df_stations.loc[df_stations.id==col].latitude.values[0], \n                            'longitude': df_stations.loc[df_stations.id==col].longitude.values[0], \n                            'text': f'Station ID: {col} <br>Station name: {station_name} <br>Measurement: Rain',\n                            'color': 'rgb(0, 128, 255)'}])\n    \n    # Append new row to map dataframe\n    df_map = pd.concat([df_map, df_row])\n    \ndf_map.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:58.643849Z","iopub.execute_input":"2024-03-14T16:12:58.644127Z","iopub.status.idle":"2024-03-14T16:12:58.705712Z","shell.execute_reply.started":"2024-03-14T16:12:58.644103Z","shell.execute_reply":"2024-03-14T16:12:58.704856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The pre-processing for the maps is done. Now, we can go and actually create the map chart.","metadata":{}},{"cell_type":"code","source":"# Create trace and add markers to map\ntrace = go.Scattermapbox(lat=df_map.latitude, \n                         lon=df_map.longitude,\n                         mode='markers',\n                         marker=dict(size=10,\n                                     color=df_map['color'],\n                                     opacity=0.8),\n                         hovertemplate = \"\",\n                         text=df_map['text'])\n\n# Define the layout for the map\nlayout = go.Layout(\n    margin={\"l\": 0, \"r\": 0, \"t\": 0, \"b\": 0},\n    mapbox=dict(\n        style='carto-positron', # Alternative: 'open-street-map',\n        center=dict(lat=df_map.latitude.median(),\n                    lon=df_map.longitude.median()),\n        zoom=10,\n        ),\n    height=700, width=900)\n\n# Create figure\nfig = go.Figure(data=[trace], layout=layout)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2024-03-14T16:12:58.707112Z","iopub.execute_input":"2024-03-14T16:12:58.707434Z","iopub.status.idle":"2024-03-14T16:12:59.250329Z","shell.execute_reply.started":"2024-03-14T16:12:58.707395Z","shell.execute_reply":"2024-03-14T16:12:59.249293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note, you can hovering the cursor over the markers will reveal some meta information about the weather station.","metadata":{}},{"cell_type":"markdown","source":"* **Trace Creation**: The `go.Scattermapbox()` function is used to create a scatter trace on a Mapbox map. The latitude and longitude of the markers on the map are specified using the data from the `df_map` DataFrame. Marker properties are configured, such as size, color, and opacity based on the data from the `df_map` DataFrame.\n\n* **Layout Definition**: The map layout is defined using the `go.Layout()` function. Graph margins are set to 0 so there is no space between the graph's edge and the container. The map style is configured as 'carto-positron' for a clear map style. The center of the map is set to the median of the latitudes and longitudes from the data in the `df_map` DataFrame. The initial zoom level of the map is set to 10. The dimensions of the graph are specified as 700 pixels high and 900 pixels wide.\n\n* **Figure Creation**: The `go.Figure()` function is used to create the graph figure, with the trace and layout as arguments.\n\n* The figure is displayed using the `show()` method.","metadata":{}},{"cell_type":"markdown","source":"## What's next?","metadata":{}},{"cell_type":"markdown","source":"How do I continue this? Where to start?!\n\nGO TO THE COMPETITION SITE!\n\n[https://www.kaggle.com/competitions/cool-quiet-city-competition](https://www.kaggle.com/competitions/cool-quiet-city-competition)","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}