{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n

🌡 New York City Daily Weather Data 2019 🌡

\n

Check the dataset here

\n\n","metadata":{}},{"cell_type":"markdown","source":"
Context
\n\n
This dataset was gathered to be used in the Big Data Derby 2022 competition. However, it can obviously be used for any other purposes. Horses are affected by weather conditions, so knowing the day's temperature, snowfall, precipitation, etc will definitely be useful.
\n \n
Content
\n\n
This notebook is a basic EDA of the weather data collected from the National Weather Service. It contains daily data from all days in 2019. It contains for each day the minimum temperature, maximum temperature, average temperature, precipitation, new snow fall, and current snow depth. The temperature is measured in Fahrenheit and the depth is measured in inches. T means that there is a trace of precipitation.\n
\n","metadata":{}},{"cell_type":"markdown","source":"
Import libraries
\n","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\nimport os\nimport pandas as pd\nimport plotly.express as px","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-08-17T23:14:38.016064Z","iopub.execute_input":"2022-08-17T23:14:38.016806Z","iopub.status.idle":"2022-08-17T23:14:39.562423Z","shell.execute_reply.started":"2022-08-17T23:14:38.016673Z","shell.execute_reply":"2022-08-17T23:14:39.561513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Load Data
\n","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(\"../input/new-york-city-weather-data-2019/nyc_temperature.csv\", sep = ',')\nprint(\"\\033[1;30;25m Rows:\\033[0;0m\" + f\"{df.shape[0]}\")\nprint(\"\\033[1;32;25m Columns:\\033[0;0m\" + f\"{df.shape[1]}\")\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-17T23:39:21.917737Z","iopub.execute_input":"2022-08-17T23:39:21.918084Z","iopub.status.idle":"2022-08-17T23:39:21.939386Z","shell.execute_reply.started":"2022-08-17T23:39:21.918059Z","shell.execute_reply":"2022-08-17T23:39:21.938454Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-17T23:35:08.181591Z","iopub.execute_input":"2022-08-17T23:35:08.182010Z","iopub.status.idle":"2022-08-17T23:35:08.208768Z","shell.execute_reply.started":"2022-08-17T23:35:08.181972Z","shell.execute_reply":"2022-08-17T23:35:08.207853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Data pre-processing
\n","metadata":{}},{"cell_type":"code","source":"def fahrenheit_to_celsius(x):\n x = (x-32)*(5/9)\n return x\n\ndf[\"date\"] = pd.to_datetime(df[\"date\"], format='%d/%m/%y')\ndf[\"tmax_c\"] = df[\"tmax\"].apply(lambda x: fahrenheit_to_celsius(x))\ndf[\"tmin_c\"] = df[\"tmin\"].apply(lambda x: fahrenheit_to_celsius(x))\ndf[\"tavg_c\"] = df[\"tavg\"].apply(lambda x: fahrenheit_to_celsius(x))\ndf[\"departure_c\"] = df[\"departure\"].apply(lambda x: fahrenheit_to_celsius(x))\ndf[\"precipitation\"].replace(to_replace=\"T\", value=0, inplace = True)\ndf[\"new_snow\"].replace(to_replace=\"T\", value=0, inplace = True)\ndf[\"snow_depth\"].replace(to_replace=\"T\", value=0, inplace = True)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-17T23:14:39.605766Z","iopub.execute_input":"2022-08-17T23:14:39.606146Z","iopub.status.idle":"2022-08-17T23:14:39.647461Z","shell.execute_reply.started":"2022-08-17T23:14:39.606119Z","shell.execute_reply":"2022-08-17T23:14:39.646127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Daily temperature in Fahrenheit
\n","metadata":{}},{"cell_type":"code","source":"from plotly.subplots import make_subplots\nimport plotly.graph_objects as go\n\nfig = make_subplots(rows=2,\n cols=2)\nrc = [[1,1],[1,2],[2,1],[2,2]]\ncol_names = [\"tmax\", \"tmin\", \"tavg\", \"departure\"]\ncolors = [\"#FF0000\", \"#00FFFB\", \"#FCFCFC\", \"#1FFF00\"]\nnames = [\"tmax\", \"tmin\", \"tavg\", \"departure\"]\ntitle_texts = [\"Max Temperature (°F)\",\"Min Temperature (°F)\", \"Average Temperature (°F)\", \"Departure Temperature (°F)\"]\n\n# ITERATE OVER DATAFRAMES\nfor i in range(4):\n # Add one subplot\n fig.add_trace(\n go.Scatter(x = df[\"date\"],\n y= df[col_names[i]],\n name=names[i],\n marker=dict(color=colors[i]),\n hovertemplate = title_texts[i] + ' %{y:.2f}',\n mode='lines+markers' \n ), \n row=rc[i][0], col=rc[i][1] \n )\n fig.update_xaxes(\n title_text=title_texts[i],\n title_font_color=colors[i],\n tickfont_color=colors[i], \n row=rc[i][0], col=rc[i][1]) \n fig.update_yaxes(\n title_text = \"Frequency\", \n title_font_color=colors[i],\n tickfont_color=colors[i], \n row=rc[i][0], col=rc[i][1]) \n \nfig.update_layout(height=800,\n width=1100,\n title_text=\"Daily Temperature in Fahrenheit\",\n template=\"plotly_dark\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-17T23:14:39.653389Z","iopub.execute_input":"2022-08-17T23:14:39.654473Z","iopub.status.idle":"2022-08-17T23:14:40.949108Z","shell.execute_reply.started":"2022-08-17T23:14:39.654424Z","shell.execute_reply":"2022-08-17T23:14:40.946750Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Daily temperature in Celsius
\n","metadata":{}},{"cell_type":"code","source":"from plotly.subplots import make_subplots\nimport plotly.graph_objects as go\n\nfig = make_subplots(rows=2,\n cols=2)\nrc = [[1,1],[1,2],[2,1],[2,2]]\ncol_names = [\"tmax_c\", \"tmin_c\", \"tavg_c\", \"departure_c\"]\ncolors = [\"#FF0000\", \"#00FFFB\", \"#FCFCFC\", \"#1FFF00\"]\nnames = [\"tmax\", \"tmin\", \"tavg\", \"departure\"]\ntitle_texts = [\"Max Temperature (°C)\",\"Min Temperature (°C)\", \"Average Temperature (°C)\", \"Departure Temperature (°C)\"]\n\n# ITERATE OVER DATAFRAMES\nfor i in range(4):\n # Add one subplot\n fig.add_trace(\n go.Scatter(x = df[\"date\"],\n y= df[col_names[i]],\n name=names[i],\n marker=dict(color=colors[i]),\n hovertemplate = title_texts[i] + ' %{y:.2f}',\n mode='lines+markers' \n ), \n row=rc[i][0], col=rc[i][1] \n )\n fig.update_xaxes(\n title_text=title_texts[i],\n title_font_color=colors[i],\n tickfont_color=colors[i], \n row=rc[i][0], col=rc[i][1]) \n fig.update_yaxes(\n title_text = \"Frequency\", \n title_font_color=colors[i],\n tickfont_color=colors[i], \n row=rc[i][0], col=rc[i][1]) \n \nfig.update_layout(height=800,\n width=1100,\n title_text=\"Daily Temperature in Celsius\",\n template=\"plotly_dark\")\nfig.show()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2022-08-17T23:14:40.950725Z","iopub.execute_input":"2022-08-17T23:14:40.951121Z","iopub.status.idle":"2022-08-17T23:14:41.108420Z","shell.execute_reply.started":"2022-08-17T23:14:40.951093Z","shell.execute_reply":"2022-08-17T23:14:41.107527Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Temperature in Fahrenheit Histograms
\n","metadata":{}},{"cell_type":"code","source":"from plotly.subplots import make_subplots\nimport plotly.graph_objects as go\n\nfig = make_subplots(rows=2,\n cols=2)\n\nrc = [[1,1],[1,2],[2,1],[2,2]]\ncol_names = [\"tmax\", \"tmin\", \"tavg\", \"departure\"]\ncolors = [\"#FF0000\", \"#00FFFB\", \"#FCFCFC\", \"#1FFF00\"]\nnames = [\"tmax\", \"tmin\", \"tavg\", \"departure\"]\ntitle_texts = [\"Max Temperature (°F)\",\"Min Temperature (°F)\", \"Average Temperature (°F)\", \"Departure Temperature (°F)\"]\n\n# ITERATE OVER DATAFRAMES\nfor i in range(4):\n # Add one subplot\n fig.add_trace(\n go.Histogram(x=df[col_names[i]],\n name=names[i],\n hovertemplate = title_texts[i] + ' %{y:.2f}',\n marker=dict(color=colors[i])\n ), \n row=rc[i][0], col=rc[i][1] \n )\n fig.update_xaxes(\n title_text=title_texts[i],\n title_font_color=colors[i],\n tickfont_color=colors[i], \n row=rc[i][0], col=rc[i][1]) \n fig.update_yaxes(\n title_text = \"Frequency\", \n title_font_color=colors[i], \n tickfont_color=colors[i], \n row=rc[i][0], col=rc[i][1]) \n \nfig.update_layout(height=800,\n width=1000,\n title_text=\"Temperature information in Fahrenheit\",\n template=\"plotly_dark\",\n xaxis=dict(color=\"#FF9300\"),\n yaxis=dict(color=\"#FF9300\"))\nfig.show()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2022-08-17T23:14:41.109480Z","iopub.execute_input":"2022-08-17T23:14:41.109845Z","iopub.status.idle":"2022-08-17T23:14:41.250051Z","shell.execute_reply.started":"2022-08-17T23:14:41.109814Z","shell.execute_reply":"2022-08-17T23:14:41.248404Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Temperature in Celsius Histograms
\n","metadata":{}},{"cell_type":"code","source":"from plotly.subplots import make_subplots\nimport plotly.graph_objects as go\n\nfig = make_subplots(rows=2,\n cols=2)\nrc = [[1,1],[1,2],[2,1],[2,2]]\ncol_names = [\"tmax_c\", \"tmin_c\", \"tavg_c\", \"departure_c\"]\ncolors = [\"#FF0000\", \"#00FFFB\", \"#FCFCFC\", \"#1FFF00\"]\nnames = [\"tmax\", \"tmin\", \"tavg\", \"departure\"]\ntitle_texts = [\"Max Temperature (°C)\",\"Min Temperature (°C)\", \"Average Temperature (°C)\", \"Departure Temperature (°C)\"]\n\n# ITERATE OVER DATAFRAMES\nfor i in range(4):\n # Add one subplot\n fig.add_trace(\n go.Histogram(x=df[col_names[i]],\n name=names[i],\n hovertemplate = title_texts[i] + ' %{y:.2f}',\n marker=dict(color=colors[i])\n ), \n row=rc[i][0], col=rc[i][1] \n )\n fig.update_xaxes(\n title_text=title_texts[i],\n title_font_color=colors[i],\n tickfont_color=colors[i], \n row=rc[i][0], col=rc[i][1]) \n fig.update_yaxes(\n title_text = \"Frequency\", \n title_font_color=colors[i], \n tickfont_color=colors[i], \n row=rc[i][0], col=rc[i][1]) \n \nfig.update_layout(height=800,\n width=1000,\n title_text=\"Temperature information in Celsius\",\n template=\"plotly_dark\",\n xaxis=dict(color=\"#FF9300\"),\n yaxis=dict(color=\"#FF9300\"))\nfig.show()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2022-08-17T23:14:41.251539Z","iopub.execute_input":"2022-08-17T23:14:41.252726Z","iopub.status.idle":"2022-08-17T23:14:41.383569Z","shell.execute_reply.started":"2022-08-17T23:14:41.252642Z","shell.execute_reply":"2022-08-17T23:14:41.382832Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Min - Average - Max Temperature Plot (Fahrenheit)
\n","metadata":{}},{"cell_type":"code","source":"fig = go.Figure()\n# first I add a trace for every x\nfig.add_trace(go.Scatter(x=df[\"date\"],\n y=df[\"tmin\"],\n mode=\"markers\",\n showlegend=False,\n marker=dict(color=\"blue\",\n size=6)))\n\nfig.add_trace(go.Scatter(x=df[\"date\"],\n y=df[\"tavg\"],\n mode=\"markers\",\n showlegend=False,\n marker=dict(color=\"yellow\",\n size=6)))\n\nfig.add_trace(go.Scatter(x=df[\"date\"],\n y=df[\"tmax\"],\n mode=\"markers\",\n showlegend=False,\n marker=dict(color=\"red\",\n size=6)))\n\nfor i, row in df.iterrows():\n if row[\"tmin\"]!=row[\"tmax\"]:\n fig.add_shape(\n dict(type=\"line\",\n x0=row[\"date\"],\n x1=row[\"date\"],\n y0=row[\"tmin\"],\n y1=row[\"tmax\"],\n line=dict(\n color=\"black\",\n width=2)\n )\n )\nfig.update_layout(title=\"Min - Average - Max Temperature Graph\",\n title_x=0.5, \n xaxis_title=\"Date\",\n yaxis_title=\"Temperature (°F)\"\n)\nfig.show()","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2022-08-17T23:14:41.384493Z","iopub.execute_input":"2022-08-17T23:14:41.384953Z","iopub.status.idle":"2022-08-17T23:15:10.994204Z","shell.execute_reply.started":"2022-08-17T23:14:41.384927Z","shell.execute_reply":"2022-08-17T23:15:10.991722Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Min - Average - Max Temperature Plot (Celsius)
\n","metadata":{}},{"cell_type":"code","source":"fig = go.Figure()\n# first I add a trace for every x\nfig.add_trace(go.Scatter(x=df[\"date\"],\n y=df[\"tmin_c\"],\n mode=\"markers\",\n showlegend=False,\n marker=dict(color=\"blue\",\n size=6)))\n\nfig.add_trace(go.Scatter(x=df[\"date\"],\n y=df[\"tavg_c\"],\n mode=\"markers\",\n showlegend=False,\n marker=dict(color=\"yellow\",\n size=6)))\n\nfig.add_trace(go.Scatter(x=df[\"date\"],\n y=df[\"tmax_c\"],\n mode=\"markers\",\n showlegend=False,\n marker=dict(color=\"red\",\n size=6)))\n\nfor i, row in df.iterrows():\n if row[\"tmin_c\"]!=row[\"tmax_c\"]:\n fig.add_shape(\n dict(type=\"line\",\n x0=row[\"date\"],\n x1=row[\"date\"],\n y0=row[\"tmin_c\"],\n y1=row[\"tmax_c\"],\n line=dict(\n color=\"black\",\n width=2)\n )\n )\nfig.update_layout(title=\"Min - Average - Max Temperature Graph\",\n title_x=0.5, \n xaxis_title=\"Date\",\n yaxis_title=\"Temperature (°C)\"\n)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-17T23:15:40.061109Z","iopub.execute_input":"2022-08-17T23:15:40.061382Z","iopub.status.idle":"2022-08-17T23:16:09.408550Z","shell.execute_reply.started":"2022-08-17T23:15:40.061358Z","shell.execute_reply":"2022-08-17T23:16:09.407469Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Daily Precipitation
\n","metadata":{}},{"cell_type":"code","source":"fig = px.line(df, x=\"date\", y=\"precipitation\",\n title='Daily precipitation (in inches)',\n color_discrete_sequence=[\"#FFFF00\"], # LINE COLOR\n template=\"plotly_dark\")\n\n## Another way:\n# fig.update_xaxes(title_font_color=\"#FF9300\") # X AXIS COLOR\n# fig.update_yaxes(title_font_color=\"#FF9300\") # Y AXIS COLOR\n\nfig.update_layout(\n title_font_color=\"#FF9300\", # TITLE FONT COLOR\n xaxis=dict(color=\"#FF9300\"), # X AXIS COLOR\n yaxis=dict(color=\"#FF9300\") # Y AXIS COLOR\n)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-17T23:20:47.117421Z","iopub.execute_input":"2022-08-17T23:20:47.117840Z","iopub.status.idle":"2022-08-17T23:20:47.228702Z","shell.execute_reply.started":"2022-08-17T23:20:47.117811Z","shell.execute_reply":"2022-08-17T23:20:47.227517Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
New snow (inches)
\n","metadata":{}},{"cell_type":"code","source":"fig = px.line(df, x=\"date\", y=\"new_snow\",\n title='New snow (in inches)',\n color_discrete_sequence=[\"#FFFF00\"], \n template=\"plotly_dark\")\n\n\nfig.update_layout(\n title_font_color=\"#FF9300\", \n xaxis=dict(color=\"#FF9300\"), \n yaxis=dict(color=\"#FF9300\") \n)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-17T23:21:23.247480Z","iopub.execute_input":"2022-08-17T23:21:23.248072Z","iopub.status.idle":"2022-08-17T23:21:23.310575Z","shell.execute_reply.started":"2022-08-17T23:21:23.248046Z","shell.execute_reply":"2022-08-17T23:21:23.309733Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Snow depth (inches)
\n","metadata":{"execution":{"iopub.status.busy":"2022-08-17T23:25:03.977198Z","iopub.execute_input":"2022-08-17T23:25:03.977561Z","iopub.status.idle":"2022-08-17T23:25:03.986228Z","shell.execute_reply.started":"2022-08-17T23:25:03.977537Z","shell.execute_reply":"2022-08-17T23:25:03.984951Z"}}},{"cell_type":"code","source":"fig = px.line(df, x=\"date\", y=\"snow_depth\",\n title='Snow depth (in inches)',\n color_discrete_sequence=[\"#FFFF00\"], \n template=\"plotly_dark\")\n\nfig.update_layout(\n title_font_color=\"#FF9300\", \n xaxis=dict(color=\"#FF9300\"), \n yaxis=dict(color=\"#FF9300\") \n)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-08-17T23:25:47.991084Z","iopub.execute_input":"2022-08-17T23:25:47.991536Z","iopub.status.idle":"2022-08-17T23:25:48.063519Z","shell.execute_reply.started":"2022-08-17T23:25:47.991501Z","shell.execute_reply":"2022-08-17T23:25:48.061746Z"},"jupyter":{"source_hidden":true},"trusted":true},"execution_count":null,"outputs":[]}]}