\n\n1. [Context](#CONTEXT)\n2. [Project Overview](#PROJECTOVERVIEW)\n3. [Metric](#METRIC)\n4. [Objective](#OBJECTIVE)\n5. [Glimpse of Data](#PREPARATION)\n * [Import packages](#IMPORT)\n * [Auxiliar Functions](#FUNCTION)\n6. [Explore the power plant data](#PLANTDATA)\n * [Glimpse of Data - Power Plants](#GLIMPSEDATA)\n * [Understanding the estimation gwh by primary fuel](#U1)\n * [Visuals for each primary fuel](#VISUALS)\n * [Understanding the power plants](#U2)\n7. [Visualizations](#VISUALS)\n * [Amount estimated generation gwh per primary fuel](#V1)\n * [Amount capacity mw per primary fue](#V2)\n * [Histogram and Curve Plot for each Primary Fuel](#V3)\n * [Violin Plot Primary Fuel](#V4)\n * [Pollute Fuel Type X Non Pollute Fuel ](#V5)\n8. [Explore the NO2 emissions data](#P2)\n * [Sentinel 5P](#S5P) \n * [Zonal N02 statistics around each power plant](#P2) \n9. [Simple Factor Modeling Example](#SFM) \n10. [Understanding Sentinel-5P OFFL NO2](#US5P)\n * [Outlier Analysis](#OULIER)\n11. [Explore the weather data](#WEATHERDATA) \n12. [Modeling](#MODELING) \n * [Emission Factor - non pollute fuel types](#NONP)\n * [Emission Factor - pollute fuel types](#P)\n13. [General findings](#GF) \n14. [About the data](#ABOUTDATA) \n15. [The End](#END) \n\n "},{"metadata":{},"cell_type":"markdown","source":"\n#
Context
\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n\nCurrent emissions factors methodologies are based on **time-consuming** data collection and may include errors derived from a lack of access to granular datasets, inability to refresh data on a frequent basis, overly general modeling assumptions, and inaccurate reporting of emissions sources like fuel consumption. \n\n\n#
Project Overview
\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n\nThe Environmental Insights Explorer team at Google is keen to gather insights on ways to improve calculations of global emissions factors for sub-national regions. The ultimate goal of this challenge is to test if **calculations of emissions factors using remote sensing techniques** are possible and on par with calculations of emissions factors from current methodologies.\n\n"},{"metadata":{},"cell_type":"markdown","source":" \n\n#
The Metric
\n
The general equation for emissions estimation is:
\n\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n\n### E = A x EF x (1-ER/100)\nwhere:\n\nE = emissions;\nA = activity rate;\nEF = emission factor,\nand ER =overall emission reduction efficiency, % therefore\n\n### EF = E / [A x (1-ER/100)]\n\n- To simplify things a bit, I'll reduce that equation to: EF = E / A\n\n- Simplified Emissions Factor = Emissions / Activity Rate\n\n** Which again can be similified to the following:\nSimplified Emissions Factor = (Measure of NO2 emissions) / (Quanity of electricity generated)\nWe can find a \"measure of NO2 emissions\" from the Sentinel-5p dataset. Likewise, we can find a measure of (quanitity of electricity generated) from the power plant dataset. To demonstrate how to get started with the data, we will load and preview those two data sources and then we will calculate a simplified emissions factor for a single power plant on the island of Vieques in Puerto Rico.**"},{"metadata":{},"cell_type":"markdown","source":"\n#
Objective
\n\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n\nThis Data Science for Good Competition intends to use remote sensing techniques to understand Environmental Emissions. Since the whole concept of Satellite Imagery and can be a little overwhelming, this is just an introductory kernel, where I try to explain the various terms and datasets related to satellite Imagery and anomaly detection.\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"\n
How green energy will change our future
\n\n\nHow will green energy change our future? What will our future look like with green energy? The growth of green energy goes together with change. Our future will not only include green energy, but our future will also be shaped by it\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"%%HTML\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Suppress warnings \nimport warnings\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\nwarnings.filterwarnings(\"ignore\", category=UserWarning)\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)\nfrom IPython.display import HTML\n\nHTML('')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
Glimpse of Data
\n\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)"},{"metadata":{},"cell_type":"markdown","source":"#
Imports
\n\n[Back to Table of Contents](#top)\n\nWe are using a stack: ``numpy``, ``pandas``, ``sklearn``, ``matplotlib``, ``rasterio``, ``plotly``."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom ast import literal_eval\nimport rasterio as rio\nimport plotly.graph_objs as go\n\nfrom shapely.geometry import Point\nimport geopandas as gpd\nimport tifffile as tiff\nimport folium\nimport os, random\nimport seaborn as sns\nimport plotly\nimport plotly.express as px\nimport plotly.figure_factory as ff\nimport plotly.graph_objs as go\nimport glob\nfrom tqdm import tqdm_notebook as tqdm\n\nfrom plotly.offline import init_notebook_mode, iplot, plot\n\nimport numpy as np\nimport pandas as pd\npd.set_option('max_columns', 200)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#
Auxiliar Functions
\n\n[Back to Table of Contents](#top)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Graphs\ndef hist_curve(primary_fuel_type, hasType=True):\n if hasType==True:\n x1 = power_plants_df[power_plants['primary_fuel']==primary_fuel_type][\"capacity_mw\"].values\n x2 = power_plants_df[power_plants['primary_fuel']==primary_fuel_type][\"estimated_generation_gwh\"].values\n hist_data = [x1, x2]\n group_labels = ['estimated_generation_gwh', 'capacity_mw']\n colors = ['#A56CC1', '#A6ACEC', '#63F5EF']\n fig = ff.create_distplot(hist_data, group_labels, colors=colors, bin_size=.2, show_rug=False)\n fig['layout'].update(title='Hist and Curve Plot')\n plotly.offline.iplot(fig, filename='Hist and Curve')\n else:\n x1 = power_plants_df[\"capacity_mw\"].values\n x2 = power_plants_df[\"estimated_generation_gwh\"].values\n hist_data = [x1, x2]\n group_labels = ['primary_fuel', 'capacity_mw']\n colors = ['#A56CC1', '#A6ACEC', '#63F5EF']\n fig = ff.create_distplot(hist_data, group_labels, colors=colors, bin_size=.2, show_rug=False)\n fig['layout'].update(title='Hist and Curve Plot')\n plotly.offline.iplot(fig, filename='Hist and Curve')\n \n\ndef violin_plot(feature):\n data = []\n for i in range(0,len(pd.unique(power_plants_df['primary_fuel']))):\n trace = {\n \"type\": 'violin',\n \"x\": power_plants_df['primary_fuel'][power_plants_df['primary_fuel'] == pd.unique(power_plants_df['primary_fuel'])[i]],\n \"y\": power_plants_df[feature][power_plants_df['primary_fuel'] == pd.unique(power_plants_df['primary_fuel'])[i]],\n \"name\": pd.unique(power_plants_df['primary_fuel'])[i],\n \"box\": {\n \"visible\": True\n },\n \"meanline\": {\n \"visible\": True\n }\n }\n data.append(trace)\n fig = {\n \"data\": data,\n \"layout\" : {\n \"title\": \"Primary Fuel Types\",\n \"yaxis\": {\n \"zeroline\": False,\n }\n }\n }\n plotly.offline.iplot(fig, filename='violin/multiple', validate = False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_points_on_map(dataframe,begin_index,end_index,latitude_column,latitude_value,longitude_column,longitude_value,zoom):\n df = dataframe[begin_index:end_index]\n location = [latitude_value,longitude_value]\n plot = folium.Map(location=location,zoom_start=zoom, tiles= 'Stamen Terrain')\n color={ 'Hydro' : 'lightblue', 'Solar' : 'orange', 'Oil' : 'darkblue', 'Coal' : 'black', 'Gas' : 'lightgray', 'Wind' : 'green' }\n for i in range(0,len(df)):\n popup = folium.Popup(str(df.primary_fuel[i:i+1])) \n folium.Marker([df[latitude_column].iloc[i],\n df[longitude_column].iloc[i]],\n popup=popup,icon=folium.Icon(icon_color='red',icon ='bolt',prefix='fa',color=color[df.primary_fuel.iloc[i]])).add_to(plot)\n \n return(plot)\n\ndef overlay_image_on_puerto_rico(file_name,band_layer):\n band = rio.open(file_name).read(band_layer)\n m = folium.Map([lat, lon], zoom_start=8)\n folium.raster_layers.ImageOverlay(\n image=band,\n bounds = [[18.6,-67.3,],[17.9,-65.2]],\n colormap=lambda x: (1, 0, 0, x),\n ).add_to(m)\n return m\n\ndef plot_scaled(file_name):\n vmin, vmax = np.nanpercentile(file_name, (5,95)) # 5-95% stretch\n img_plt = plt.imshow(file_name, cmap='gray', vmin=vmin, vmax=vmax)\n plt.show()\n\ndef split_column_into_new_columns(dataframe,column_to_split,new_column_one,begin_column_one,end_column_one):\n for i in range(0, len(dataframe)):\n dataframe.loc[i, new_column_one] = dataframe.loc[i, column_to_split][begin_column_one:end_column_one]\n return dataframe\n\ndef get_lon_from_geo(str_):\n dict_ = literal_eval(str_)\n coordinates = dict_['coordinates']\n lon = coordinates[0]\n return lon\n\ndef get_lat_from_geo(str_):\n dict_ = literal_eval(str_)\n coordinates = dict_['coordinates']\n lat = coordinates[1]\n return lat\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Somefunctions for NO2, Whether, Satellite plots "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def plotInfoNO2(img_path_id):\n # NO2 Info\n# https://www.kaggle.com/jyesawtellrickson/understanding-sentinel-5p-offl-no2 \n dataExample = tiff.imread(img_path_id)\n data_nn = dataExample[:, :177, :] # no null values\n titles = ['NO2_column_number_density',\n 'tropospheric_NO2_column_number_density', \n 'stratospheric_NO2_column_number_density',\n 'NO2_slant_column_number_density']\n f = plt.figure()\n f.set_size_inches(8, 8)\n for i in range(4):\n plt.subplot(2, 2, i+1)\n sns.heatmap(data_nn[:, :, i], cbar=False)\n f\n ax = plt.gca()\n ax.get_xaxis().set_visible(False)\n ax.get_yaxis().set_visible(False)\n plt.title(titles[i], fontsize=10)\n for i in range(4):\n print('\\n {}: {:.2E} mol/m^2'.format(titles[i], np.nanmean(data_nn[:, :, i])))\n print(' ratio stratospheric_NO2_column_number_density by tropospheric_NO2_column_number_density mol/m^2 ', ( np.nanmean(data_nn[:, :, 2])/ np.nanmean(data_nn[:, :, 1]) ) )\n\ndef plotInfoNO2_(img_path_id):\n # NO2 Info\n dataExample = tiff.imread(img_path_id)\n data_nn = dataExample[:, :177, :] # no null values\n titles = ['NO2_column_number_density',\n 'tropospheric_NO2_column_number_density', \n 'stratospheric_NO2_column_number_density',\n 'NO2_slant_column_number_density']\n for i in range(4):\n print('\\n {}: {:.2E} mol/m^2'.format(titles[i], np.nanmean(data_nn[:, :, i])))\n print(' ratio stratospheric_NO2_column_number_density by tropospheric_NO2_column_number_density mol/m^2 ', ( np.nanmean(data_nn[:, :, 2])/ np.nanmean(data_nn[:, :, 1]) ) )\n print(\"__\"*10)\ndef plotInfoWheatherInfo(img_path_id):\n # Wheather Info\n dataExample = tiff.imread(img_path_id)\n data_nn = dataExample[:, :177, :] # no null values\n \n titles = ['tropopause_pressure', 'absorbing_aerosol_index', 'cloud_fraction']\n f = plt.figure()\n f.set_size_inches(12,4)\n for i in range(3):\n plt.subplot(1, 3, i+1)\n sns.heatmap(dataExample[:, :, 4+i], cbar=False)\n f\n ax = plt.gca()\n ax.get_xaxis().set_visible(False)\n ax.get_yaxis().set_visible(False)\n plt.title(titles[i], fontsize=16)\n for i in range(3):\n print('{}: {:.2f}'.format(titles[i], np.nanmean(data_nn[:, :, i+4])), '\\n ')\n print('_'*10)\ndef plotInfoSatelliteInfo(img_path_id):\n # Satellite info\n dataExample = tiff.imread(img_path_id)\n data_nn = dataExample[:, :177, :] # no null values\n\n titles = ['sensor_altitude', 'sensor_azimuth_angle', 'sensor_zenith_angle',\n 'solar_azimuth_angle', 'solar_zenith_angle']\n\n f = plt.figure()\n f.set_size_inches(12, 8)\n for i in range(5):\n plt.subplot(2, 3, i+1)\n sns.heatmap(data_nn[:, :, 7+i], cbar=False)\n f\n ax = plt.gca()\n ax.get_xaxis().set_visible(False)\n ax.get_yaxis().set_visible(False)\n plt.title(titles[i], fontsize=16)\n for i in range(5):\n print('{}: {:.2f}'.format(titles[i], np.nanmean(data_nn[:, :, i+7])))\n print('_'*10)\n\n ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n#
\n\n[Back to Table of Contents](#top)"},{"metadata":{},"cell_type":"markdown","source":"We have a lot of columns are empty."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df = power_plants.sort_values('capacity_mw',ascending=False).reset_index()\npower_plants_df[['name','latitude','longitude','primary_fuel','capacity_mw','estimated_generation_gwh']]\n\npower_plants_df['prod_hrs_p_day']=power_plants_df.estimated_generation_gwh*1000/(power_plants_df.capacity_mw*365)\npower_plants_df['prod_hrs_p_hour']=power_plants_df.estimated_generation_gwh*1000/(power_plants_df.capacity_mw*365)*24\npower_plants_df['prod_hrs_p_minute']=power_plants_df.estimated_generation_gwh*1000/(power_plants_df.capacity_mw*365)*24*60\n\npower_plants_df[['name','latitude','longitude','primary_fuel','capacity_mw','estimated_generation_gwh','prod_hrs_p_day','prod_hrs_p_hour','prod_hrs_p_minute']]\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"ordered by primary_fuel"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df[['name','latitude','longitude','primary_fuel','capacity_mw','estimated_generation_gwh','prod_hrs_p_day','prod_hrs_p_hour','prod_hrs_p_minute']].groupby(['primary_fuel']).head(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some useful information about the data."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Visuals for each primary fuel - estimated_generation_gwh\n "},{"metadata":{},"cell_type":"markdown","source":"\n#
Understanding the estimation gwh by primary fuel
\n\n\n[Back to Table of Contents](#top)\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"bar_plot_df = pd.DataFrame(power_plants_df.groupby(['primary_fuel'])['estimated_generation_gwh'].sum().nlargest())\nbar_plot_df = bar_plot_df.reset_index()\nbar_plot_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
Visuals for each primary fuel
\n\n[Back to Table of Contents](#top)\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"bar_plot_df_ = pd.DataFrame(power_plants_df.groupby(['primary_fuel'])['capacity_mw'].sum().nlargest())\nbar_plot_df_ = bar_plot_df_.reset_index()\nbar_plot_df_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df.groupby(['primary_fuel']).agg({'estimated_generation_gwh': ['nunique', 'sum', 'mean', 'max', 'min', 'std'], 'capacity_mw' : ['nunique', 'sum', 'mean', 'max', 'min', 'std']}).reset_index()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- **Oil** fuel power plants have the highest mean capacity, followed by the single **Coal** power plant\n- **Coal** fuel power plants have the highest mean estimated generation, followed by **Oil** power plant"},{"metadata":{},"cell_type":"markdown","source":"\n#
PREPA Analysis - The most important plants
\nThe most commom value in owner - PREPA \n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"power_plants_df['owner'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"power_plants_df[['name','latitude','longitude','primary_fuel','capacity_mw','estimated_generation_gwh', 'owner']][power_plants_df['owner']=='PREPA'].sort_values('estimated_generation_gwh', ascending = False).groupby(['primary_fuel','capacity_mw']).head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- The power plant Aguirre have the highest mean capacity and estimated generation for **Oil**.\n- The power plant Dos Bocas have the highest mean capacity estimated generation for **Hydro**.\n"},{"metadata":{},"cell_type":"markdown","source":"\n
get lat/lon from .geo column.
"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df['lon'] = power_plants_df['.geo'].map(get_lon_from_geo)\npower_plants_df['lat'] = power_plants_df['.geo'].map(get_lat_from_geo)\npower_plants_df.drop(columns=['.geo'], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"generation_gwh_years = [\"generation_gwh_2013\", \n \"generation_gwh_2014\", \n \"generation_gwh_2015\", \n \"generation_gwh_2016\",\n \"generation_gwh_2017\"]\npower_plants_df.loc[:, generation_gwh_years].sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Drop these columns, not usefulthere is no gwh information for the given years. "},{"metadata":{},"cell_type":"markdown","source":"> power_plants_df.head()\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df.head(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
Understanding the power plants
\n\n[Back to Table of Contents](#top)\n"},{"metadata":{},"cell_type":"markdown","source":"Duplicate values\nCheck how many duplicate values exists per columns"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"%%time\nfeatures = power_plants_df[['name','latitude','longitude','primary_fuel','capacity_mw','estimated_generation_gwh','gppd_idnr','name','owner','primary_fuel','source','url','wepp_id']].columns.values\nunique_max = []\nfor feature in features:\n values = power_plants_df[feature].value_counts()\n unique_max.append([feature, values.max(), values.idxmax()])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"np.transpose((pd.DataFrame(unique_max, columns=['Feature', 'Max duplicates', 'Value'])).\\\n sort_values(by = 'Max duplicates', ascending=False).head(50))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"geolocation_source.unique()"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df[\"geolocation_source\"].unique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"geolocation_source.value_counts()"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df[\"geolocation_source\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"gppd_idnr.unique()"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df[\"gppd_idnr\"].unique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"name.unique()"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df[\"name\"].unique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"name.value_counts()"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df[\"name\"].value_counts().head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"source.unique()"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df[\"source\"].unique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"source.value_counts()"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df[\"source\"].value_counts().head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"url.unique()"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df[\"url\"].unique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"url.value_counts()\nthe most commom url is: http://energia.pr.gov/datos/plantas/ \n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df[\"url\"].value_counts().head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Drop collumns"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"to_drop = [\"generation_gwh_2013\", \"generation_gwh_2014\", \"generation_gwh_2015\", \"generation_gwh_2016\",\"generation_gwh_2017\", \n \"other_fuel1\",\"other_fuel2\",\"other_fuel3\",\n \"geolocation_source\",\"year_of_capacity_data\",\n \"country\", \"country_long\"]\npower_plants_df = power_plants_df.drop(to_drop, axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n#
Visualizations
\n\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)"},{"metadata":{},"cell_type":"markdown","source":"\n#
Amount estimated generation gwh per primary fuel
\n\n[Back to Table of Contents](#top)\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#making barplot\nbar_plot=px.bar(bar_plot_df, x='primary_fuel', y='estimated_generation_gwh', color='estimated_generation_gwh', color_continuous_scale='Reds')\n#making layout changes\nbar_plot.update_layout(xaxis_tickangle=-45, title_text='Amount estimated generation gwh per primary fuel ')\n#outputing plot\nbar_plot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
Amount capacity mw per primary fuel
\n\n\n[Back to Table of Contents](#top)\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#making barplot\nbar_plot_=px.bar(bar_plot_df_, x='primary_fuel', y='capacity_mw', color='capacity_mw', color_continuous_scale='Reds')\n#making layout changes\nbar_plot_.update_layout(xaxis_tickangle=-45, title_text='Amount capacity mw per primary fuel ')\n#outputing plot\nbar_plot_.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
The primary fuel Oil and Gas have the most amount capacity mw
\n\n[Back to Table of Contents](#top)\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hist_curve('Hydro')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
The estimated generation gwh for Hydro have some high values
\n\n[Back to Table of Contents](#top)\n"},{"metadata":{},"cell_type":"markdown","source":"### Capacity_mw"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"violin_plot(\"capacity_mw\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
The Oil have a higher mean value of capacity mw for primary fuel
Pollute Fuel Type Coal, Oil, Gas X Non Pollute Fuel Type Hydro, Wind, Solar
\n\n[Back to Table of Contents](#top)\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def plot_dist_col(column):\n p_type_df = power_plants_df[power_plants_df['primary_fuel'].isin(['Coal', 'Oil', 'Gas'])]\n np_type_df = power_plants_df[power_plants_df['primary_fuel'].isin(['Hydro', 'Wind', 'Solar'])]\n\n '''plot dist curves for train and test weather data for the given column name'''\n fig, ax = plt.subplots(figsize=(10, 10))\n sns.distplot(p_type_df[column].dropna(), color='green', ax=ax).set_title(column, fontsize=16)\n sns.distplot(np_type_df[column].dropna(), color='purple', ax=ax).set_title(column, fontsize=16)\n plt.xlabel(column, fontsize=15)\n plt.legend(['Pollute Fuel Type', 'Non Pollute Fuel Type'])\n plt.show()\nplot_dist_col('estimated_generation_gwh')\nplot_dist_col('capacity_mw')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
Explore the NO2 emissions data
\n\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n\nnO2 emissions data\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"image = '/kaggle/input/ds4g-environmental-insights-explorer/eie_data/s5p_no2/s5p_no2_20180807T161744_20180813T174606.tif'\noverlay_image_on_puerto_rico(image,band_layer=7)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
Sentinel 5P
\n\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n\n Let's now look at the remote sensing data, starting from [**Sentinel 5P OFFL NO2**](https://developers.google.com/earth-engine/datasets/catalog/COPERNICUS_S5P_OFFL_L3_NO2) by [EU/ESA/Copernicus](https://sentinel.esa.int/web/sentinel/user-guides/sentinel-5p-tropomi/document-library)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/gpoulain/eda-ef-with-time-series-earth-engine\nsentinel_path = \"../input/ds4g-environmental-insights-explorer/eie_data/s5p_no2/\"\nimage_name = random.choice(os.listdir(sentinel_path))\nimage_path = sentinel_path + image_name\nimage = rio.open(image_path)\n\nbands = image.count\nprint(f\"There are {bands} bands\")\n\nfor i in image.indexes:\n desc = image.descriptions[i-1]\n print(f'{i}: {desc}')\n\nprint('\\n')\nprint(f'Example of:{image_name}')\n\nfig, ax = plt.subplots(3,4,figsize=(20,6))\nband=1\nfor n in range(3):\n for m in range(4):\n image_bandi = image.read(band)\n desc = image.descriptions[band-1]\n ax[n,m].set_title(desc)\n ax[n,m].imshow(image_bandi, cmap=\"Reds\")\n ax[n,m].grid(False)\n band += 1\nprint('image_path ',image_path)\nfig.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Quantity of Electricity Generated"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"power_plants_ = power_plants.sort_values('capacity_mw',ascending=False).reset_index()\nprint( power_plants_[['name','latitude','longitude','primary_fuel','capacity_mw','estimated_generation_gwh']][29:30] )\n\nquantity_of_electricity_generated = power_plants_['estimated_generation_gwh'][29:30].values\nprint('Quantity of Electricity Generated: ', quantity_of_electricity_generated)\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import geopandas as gpd\ngeometry_power_plants_Texport = [Point(x,y) for x,y in zip(power_plants_df['lon'], power_plants_df['lat'])]\ngeometry_power_plants_Texport = gpd.GeoDataFrame(power_plants_df, crs = {'init': 'epsg: 4326'}, geometry = geometry_power_plants_Texport)\ngeometry_power_plants_Texport.to_file('Global_Power_Plants_gdf.geojson', driver='GeoJSON')\npower_plants_path = '/kaggle/input/geolocated-power-plants-geojson/Global_Power_Plants_gdf.geojson'\n\n# from https://www.kaggle.com/maxlenormand/simplified-emission-for-each-plant-wip\nbuffered_power_plants = geometry_power_plants_Texport.copy()\nbuffered_power_plants['geometry'] = geometry_power_plants_Texport.geometry.buffer(0.05)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n
Zonal N02 statistics around each power plant
\n\n\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n \n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"lat=18.200178; lon=-66.664513 #puerto rico\nMap_ = plot_points_on_map(buffered_power_plants,0,425,'latitude',lat,'longitude',lon,9)\nfor power_plants in range(buffered_power_plants.shape[0]):\n folium.GeoJson(buffered_power_plants.geometry[power_plants]).add_to(Map_)\nMap_","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
Simple Factor Modeling Example
\n \n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n \n This is a simple modeling. Using each image for extract NO2 value."},{"metadata":{},"cell_type":"markdown","source":"\n#
gldas
\n#### Total elems - gldas [Global Forecast System 384-Hour Predicted Atmosphere Data](https://developers.google.com/earth-engine/datasets/catalog/NOAA_GFS0P25) by NOAA/NCEP/EMC"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"gldas_path_gldas = \"../input/ds4g-environmental-insights-explorer/eie_data/gldas/\"\nlist = os.listdir(gldas_path_gldas) # dir is your directory path\nnumber_files_gldas = len(list)\nprint ('total elements gldas ',number_files_gldas)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"test_data_gldas = [random.choice(os.listdir(gldas_path_gldas)) for _ in range(number_files_gldas)]\ntest_data_gldas[:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
s5p_no2
\n> some images in s5p_no2"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"s5p_no2_path = \"../input/ds4g-environmental-insights-explorer/eie_data/s5p_no2/\"\nno2_path = glob.glob(s5p_no2_path)\n\nlist = os.listdir(gldas_path_gldas) # dir is your directory path\nnumber_files = len(list)\nprint ('total elements s5p_no2',number_files)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Total elems - s5p_no2"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"test_data_s5p_no2 = [random.choice(os.listdir(s5p_no2_path)) for _ in range(number_files)]\ntest_data_s5p_no2[:5]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Example of Emission Factor using image image4"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"image4 = '/kaggle/input/ds4g-environmental-insights-explorer/eie_data/s5p_no2/s5p_no2_20180807T161744_20180813T174606.tif'\ndef EmissionF(image):\n mean_no2_emission_image4 = np.mean(tiff.imread(image)[:, :, 0 : 1])\n print (\"{:.8f}\".format(mean_no2_emission_image4))\n Emission_Factor = float(mean_no2_emission_image4/quantity_of_electricity_generated)\n Emission_Factor\n print (\"Emission factor for Vieques {:.12f} mol * h / m^2 * gw\".format(Emission_Factor))\n return Emission_Factor \nEmissionF(image4) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
Emission factor s5p_no2, Using just 10 images
\n> some results is empty, so there is no recordings for given date range "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"k=0\ntNull =0\nfor i in range((number_files)):\n mean_no2_emission_image4 = np.mean(tiff.imread( s5p_no2_path+test_data_s5p_no2[i] )[:, :, 0 : 1])\n Emission_Factor = float(mean_no2_emission_image4/quantity_of_electricity_generated)\n if pd.isnull(Emission_Factor):\n tNull+=1\n else:\n print (i,\"Emission factor for Vieques {:.12f} mol * h / m^2 * gw\".format(Emission_Factor))\n k+=1\n if k==10:\n break\nprint('\\n\\n total empty ',tNull)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
s5p_no2 -Null data
\n## Emission factor for ~66% of the values is null.\nThe next step is to create a more advanced Model for Emission Factor"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(tNull/number_files*100,'of the dataset is null. len data ',number_files, ' null values')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
Understanding Sentinel-5P OFFL NO2
\n\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n\n#
Outlier Analysis
\n\n In this section, let's analyze some results looking for strange values, or 'outliers'. But first is necessary to understand the measurements of NO2 in the image below.\n "},{"metadata":{},"cell_type":"markdown","source":"## This notebook explains the NO2 anomalies:\n[🌏🌿Green Future: Anomaly Analysis & Time Series - A Deep Analysis](href=\"https://www.kaggle.com/caesarlupum/ds4g-anomaly-analysis)\n"},{"metadata":{},"cell_type":"markdown","source":"\n\n#
Understanding four different measurements of NO2 for each image
\n"},{"metadata":{},"cell_type":"markdown","source":"\nsource: https://scied.ucar.edu/imagecontent/stratosphere-diagram.\n The stratosphere is the second layer (going upward) of Earth's atmosphere. It is above the troposphere and below the mesosphere. The ozone layer is within the stratosphere. The temperature gets warmer as you go higher in the stratosphere\n "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# NO2 Measures\nplotInfoNO2(s5p_no2_path+test_data_s5p_no2[540])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# plotInfoNO2(s5p_no2_path+test_data_s5p_no2[539])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Empty arrays for Emission_Factor and mean_no2_emission_imageArray\nEF_ratioStratosphericTroposphericArray = np.array([])\nmean_no2_emission_imageArray = np.array([])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Some measurements of NO2 for 10 images."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"k=0\ntNull =0\nfor i in range((number_files)):\n mean_no2_emission_image = np.mean(tiff.imread( s5p_no2_path+test_data_s5p_no2[i] )[:, :, 0 : 1]) # Using NO2_column_number_density\n Emission_Factor = float(mean_no2_emission_image/quantity_of_electricity_generated)\n measuresNO2 = plotInfoNO2_(s5p_no2_path+test_data_s5p_no2[i])\n \n EF_ratioStratosphericTroposphericArray = np.append(EF_ratioStratosphericTroposphericArray, np.array( Emission_Factor ))\n mean_no2_emission_imageArray = np.append(mean_no2_emission_imageArray, np.array( mean_no2_emission_image ))\n if pd.isnull(Emission_Factor):\n tNull+=1\n else:\n print (i,\"Emission factor for Vieques {:.12f} mol * h / m^2 * gw\".format(Emission_Factor))\n k+=1\n if k==10:\n break\nprint('\\n ',tNull,' of ',number_files,' measures for Emission_Factor are empty ')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> The ratio stratospheric_NO2_column_number_density by tropospheric_NO2_column_number_density are similar in the most values."},{"metadata":{},"cell_type":"markdown","source":"ratio Stratospheric Tropospheric Array"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"EF_ratioStratosphericTroposphericArray","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" mean no2 emission image Array"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":" mean_no2_emission_imageArray","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Creating a 2 dimensional numpy array\ndf_ = np.array( [EF_ratioStratosphericTroposphericArray, mean_no2_emission_imageArray] )\n# print(df_)\n# Creating pandas dataframe from numpy array\ndfEF_meanNo2 = pd.DataFrame({'Emission_Factor': df_[ 0], 'mean_no2_emission_image': df_[1]})\nprint(dfEF_meanNo2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print (\"Emission factor for Vieques {:.12f} mol * h / m^2 * gw\".format(dfEF_meanNo2.Emission_Factor[1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Aha** - we already knew that"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"corrs = dfEF_meanNo2.corr(); corrs","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
Explore the weather data
\n\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"image = '/kaggle/input/ds4g-environmental-insights-explorer/eie_data/gldas/gldas_20180702_1500.tif'\noverlay_image_on_puerto_rico(image,band_layer=5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n#
Wheather Information
\n"},{"metadata":{},"cell_type":"markdown","source":"\n### Bands\n- tropopause_pressure\ttopopause pressure\n- absorbing_aerosol_index\tAerosol index (at wavelengths 354/388, i.e. the OMI pair) from the AER_AI level 2 product. See [Level 2 Algorithms - Aerosol Index](https://earth.esa.int/web/sentinel/technical-guides/sentinel-5p/level-2/aerosol-index).)\n\n The Aerosol Index (AI) is a qualitative index indicating the presence of elevated layers of aerosols with significant absorption. The main aerosol types that cause signals detected in the AI are desert dust, biomass burning and volcanic ash plumes. An advantage of the AI is that it can be derived for clear as well as (partly) cloudy ground pixels.\n(https://earth.esa.int/web/sentinel/technical-guides/sentinel-5p/level-2/aerosol-index).\n\n- cloud_fraction\tEffective cloud fraction. See the [Sentinel 5P L2 Input/Output Data Definition Spec](https://sentinels.copernicus.eu/documents/247904/3119978/Sentinel-5P-Level-2-Input-Output-Data-Definition), p.220 \n \n Effective cloud fraction retrieved from the O2 A-band.\nThe effective cloud fraction is the radiometric equivalent cloud fraction of a satellite pixel\nassuming a fixed cloud albedo, usually 0.8. By definition the effective cloud fraction times\nthe assumed cloud albedo plus the cloud-free surface and atmosphere contributions yields\na TOA reflectance that agrees with the observed TOA reflectance\n\nsource: https://gee.stac.cloud/22Xh3VjjQHaEAfFGXRb4ko1DnpuvqpFCT2qByfnr7wb?t=bands"},{"metadata":{},"cell_type":"markdown","source":"\n
\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plotInfoSatelliteInfo(s5p_no2_path+test_data_s5p_no2[540]) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plotInfoSatelliteInfo(s5p_no2_path+test_data_s5p_no2[539]) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Just as different seats in a theater provide different perspectives on a performance, different Earth orbits give satellites varying perspectives, each valuable for different reasons. Some seem to hover over a single spot, providing a constant view of one face of the Earth, while others circle the planet, zipping over many different places in a day.\n\nFor more info: https://earthobservatory.nasa.gov/features/OrbitsCatalog"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"\n
Simple Modeling
\n\n\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n\n"},{"metadata":{},"cell_type":"markdown","source":"\n
\n\n\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def nonPolluteFT():\n nonpolluteEF_ = np.array([])\n k=0\n tNull =0\n for i in range((number_files)):\n np_fuel_types = ['Hydro', 'Wind', 'Solar']\n # non pollute fuel types\n np_type_df = power_plants_df[power_plants_df['primary_fuel'].isin(np_fuel_types)]\n np_type_sum = np_type_df['estimated_generation_gwh'].sum()\n sum_no2_emission = np.sum(tiff.imread( \n s5p_no2_path+test_data_s5p_no2[i] \n )[:, :, 0 : 4])\n sum_no2_emission_oe = sum_no2_emission\n # use the simplified emission factor formula\n factor = sum_no2_emission_oe / np_type_sum\n \n nonpolluteEF_ = np.append(nonpolluteEF_, np.array( factor ))\n if pd.isnull(factor):\n pass\n else:\n print(f'Simplified emissions factor Non Pollute fule types in Puerto Rico is {factor} mol * h / m^2 * gw')\n k+=1\n if k==10:\n break\n return nonpolluteEF_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"nonPolluteFT()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n
Emission Factor - pollute fuel types
\n\n\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def polluteFT():\n polluteEF_ = np.array([])\n k=0\n tNull =0\n for i in range((number_files)): \n p_fuel_types = ['Coal', 'Oil', 'Gas']\n # only consider Npollute fuel types\n p_type_df = power_plants_df[power_plants_df['primary_fuel'].isin(p_fuel_types)]\n # sum the electricity generation\n p_type_sum = p_type_df['estimated_generation_gwh'].sum()\n sum_no2_emission = np.sum(tiff.imread(\n s5p_no2_path+test_data_s5p_no2[539] \n )[:, :, 0 : 4])\n sum_no2_emission_oe = sum_no2_emission\n factor = sum_no2_emission_oe / p_type_sum\n \n polluteEF_ = np.append(polluteEF_, np.array( factor ))\n if pd.isnull(factor):\n pass\n else:\n print(f'Simplified emissions factor Pollute fule types in Puerto Rico is {factor} mol * h / m^2 * gw')\n k+=1\n if k==10: # just 10 cases- test\n break\n return polluteEF_\npolluteFT()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_ = np.array( [polluteFT(), nonPolluteFT()] )\ndfEF_meanNo2 = pd.DataFrame({'Pollute Emission_Factor': df_[ 0], 'Non Pollute Emission_Factor': df_[1]})\ndfEF_meanNo2.sort_values(['Pollute Emission_Factor','Non Pollute Emission_Factor'], ascending=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### We can see the simple emission for Pollute and no Pollute primary fuels."},{"metadata":{},"cell_type":"markdown","source":"\n
Prepare Data for Modeling
\n\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n\n[factor model](https://www.kaggle.com/ragnar123/exploratory-data-analysis-and-factor-model-idea) Upvote this\n"},{"metadata":{},"cell_type":"markdown","source":"NO2 Data"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"no2_path = '/kaggle/input/ds4g-environmental-insights-explorer/eie_data/s5p_no2/*'\nno2_pictures_path = glob.glob(no2_path)\nlen(no2_pictures_path)\nprint('We have {} pictures of the Copernicus Sentinel'.format(len(no2_pictures_path)))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# this function will help us extract the no2 emission data in a tabular way\ndef read_s5p_no2_pictures_data_fill(only_no2_emissions = True):\n s5p_no2_pictures = []\n for num, i in tqdm(enumerate(no2_pictures_path), total = 387):\n temp_s5p_no2_pictures = {'start_date': [], 'end_date': [], 'data': []}\n temp_s5p_no2_pictures['start_date'] = no2_pictures_path[num][76:84]\n temp_s5p_no2_pictures['end_date'] = no2_pictures_path[num][92:100]\n # only no2 emissions\n if only_no2_emissions:\n image = tiff.imread(i)[:, :, 0 : 4]\n band1 = pd.DataFrame(image[: ,: , 0]).interpolate()\n band1.fillna(band1.mean(), inplace = True)\n band2 = pd.DataFrame(image[: ,: , 1]).interpolate()\n band2.fillna(band2.mean(), inplace = True)\n band3 = pd.DataFrame(image[: ,: , 2]).interpolate()\n band3.fillna(band3.mean(), inplace = True)\n band4 = pd.DataFrame(image[: ,: , 3]).interpolate()\n band4.fillna(band4.mean(), inplace = True)\n image = np.dstack((band1, band2, band3, band4))\n temp_s5p_no2_pictures['data'] = image\n temp_s5p_no2_pictures['no2_emission_sum'] = np.sum(image)\n temp_s5p_no2_pictures['no2_emission_mean'] = np.average(image)\n temp_s5p_no2_pictures['no2_emission_std'] = np.std(image)\n temp_s5p_no2_pictures['no2_emission_max'] = np.max(image)\n temp_s5p_no2_pictures['no2_emission_min'] = np.min(image)\n s5p_no2_pictures.append(temp_s5p_no2_pictures)\n # all Copernicus data\n else:\n temp_s5p_no2_pictures['data'] = tiff.imread(i)\n s5p_no2_pictures.append(temp_s5p_no2_pictures)\n s5p_no2_pictures = pd.DataFrame(s5p_no2_pictures)\n s5p_no2_pictures['start_date'] = pd.to_datetime(s5p_no2_pictures['start_date'])\n s5p_no2_pictures['end_date'] = pd.to_datetime(s5p_no2_pictures['end_date'])\n s5p_no2_pictures.sort_values('start_date', inplace = True)\n s5p_no2_pictures.reset_index(drop = True, inplace = True)\n return s5p_no2_pictures\n\ns5p_no2_pictures_df_fill = read_s5p_no2_pictures_data_fill()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Drop Duplicate"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"s5p_no2_pictures_stats_fill = s5p_no2_pictures_df_fill[[col for col in s5p_no2_pictures_df_fill.columns if col not in ['data']]]\ndel s5p_no2_pictures_df_fill\n# drop nan values and check again for duplicate columns\ns5p_no2_pictures_stats_fill = s5p_no2_pictures_stats_fill[s5p_no2_pictures_stats_fill['start_date']!='2019-04-15'].dropna()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"s5p_no2_pictures_stats_fill.to_csv(\"s5p_no2_pictures_stats.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"weather data"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"weather_path = '/kaggle/input/ds4g-environmental-insights-explorer/eie_data/gfs/*'\nweather_pictures_path = glob.glob(weather_path)\nlen(weather_pictures_path)\nprint('We have {} pictures of the global forecast system'.format(len(weather_pictures_path)))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# this function will help us extract weather pictures in a tabular way\ndef read_weather_data():\n weather_pictures = []\n for num, i in tqdm(enumerate(weather_pictures_path), total = len(weather_pictures_path)):\n temp_weather_pictures = {'date': [], 'temperature_2m_above_ground': [], 'specific_humidity_2m_above_ground': [], 'relative_humidity_2m_above_ground': [], \n 'u_component_of_wind_10m_above_ground': [], 'v_component_of_wind_10m_above_ground': [], 'total_precipitation_surface': []}\n temp_weather_pictures['date'] = weather_pictures_path[num][68:-6]\n temp_weather_pictures['date'] = weather_pictures_path[num][68:-6]\n image = tiff.imread(i)\n temp_weather_pictures['temperature_2m_above_ground'] = image[ : , : , 0]\n temp_weather_pictures['specific_humidity_2m_above_ground'] = image[ : , : , 1]\n temp_weather_pictures['relative_humidity_2m_above_ground'] = image[ : , : , 2]\n temp_weather_pictures['u_component_of_wind_10m_above_ground'] = image[ : , : , 3]\n temp_weather_pictures['v_component_of_wind_10m_above_ground'] = image[ : , : , 4]\n temp_weather_pictures['total_precipitation_surface'] = image[ : , : , 5]\n temp_weather_pictures['temperature_2m_above_ground_mean'] = np.average(image[ : , : , 0])\n temp_weather_pictures['specific_humidity_2m_above_ground_mean'] = np.average(image[ : , : , 1])\n temp_weather_pictures['relative_humidity_2m_above_ground_mean'] = np.average(image[ : , : , 2])\n temp_weather_pictures['u_component_of_wind_10m_above_ground_mean'] = np.average(image[ : , : , 3])\n temp_weather_pictures['v_component_of_wind_10m_above_ground_mean'] = np.average(image[ : , : , 4])\n temp_weather_pictures['total_precipitation_surface_mean'] = np.average(image[ : , : , 5])\n \n weather_pictures.append(temp_weather_pictures)\n \n weather_pictures = pd.DataFrame(weather_pictures)\n weather_pictures['date'] = pd.to_datetime(weather_pictures['date'], infer_datetime_format = True)\n weather_pictures.sort_values('date', inplace = True)\n weather_pictures.reset_index(drop = True, inplace = True)\n return weather_pictures\n\nweather_pictures_df = read_weather_data()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"weather_pictures_df.to_csv(\"weather_pictures_df.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Merge NO2 and weather data"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Weather data have all the dates, on the other hand some days in the N02 dataframe are missing \nno2_weather = s5p_no2_pictures_stats_fill[['start_date', 'no2_emission_sum']].merge(weather_pictures_df, left_on = 'start_date', right_on = 'date', how = 'left')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Save the data\n> Now we can use this data for analysis. You can see An Anomaly and Time-series Analysis using this clean dataset [here](https://www.kaggle.com/caesarlupum/green-future-anomaly-analysis-time-series)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"no2_weather.to_csv(\"no2_weather.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\n\n#
General findings
\n[Back to Table of Contents](#top)\n\n\n\n \n### Articles\n\n#### [Joint Learning from Earth Observation and OpenStreetMap Data to Get Faster Better Semantic Maps](https://arxiv.org/abs/1705.06057)\n\n#### [Air Emissions Factors and Quantification](https://www.epa.gov/air-emissions-factors-and-quantification/basic-information-air-emissions-factors-and-quantification#About%20Emissions%20Factors) \n\n#### [How much carbon dioxide is produced per kilowatthour of U.S. electricity generation?](https://www.eia.gov/tools/faqs/faq.php?id=74&t=11)\n\n#### [The Need for Alternate PM2.5 Emission Factors for Gas-Fired Combustion Unit](https://www.powermag.com/the-need-for-alternate-pm2-5-emission-factors-for-gas-fired-combustion-units/)\n\n#### [How to Calculate Air Emissions ?](https://www.enveraconsulting.com/how-to-calculate-air-emissions/)\n\n### Videos\n\n#### [US energy startup builds power plant that produces no carbon emissions](https://youtu.be/4_w9BId_Pms?t=2)\n\n#### [AIR POLLUTION](https://youtu.be/_dTtvtlct9k)\n\n#### [WATCH: NO2 pollution rocketing thorugh Europe pushed by mild weather](https://youtu.be/9ZeaP8ljc9E)\n\n#### [NASA ARSET: Remote Sensing of NO2 with OMI, Part 1/3](https://youtu.be/rvSjD-vGI4k)\n\n#### [NASA ARSET: Introducing TROPOMI - High Resolution NO2 Observations from Space, Part 2/3](https://youtu.be/-yOInEUJTYM)\n\n#### [NASA ARSET: Python Tools for Analyzing NO2, Part 3/3](https://youtu.be/MxBg5ZMH2nM)\n### Public Dataset\n\n[Atmospheric Composition Analysis Group](http://fizz.phys.dal.ca/~atmos/martin/?page_id=100#)\n> GEOS-Chem Community Mission: to advance understanding of human and natural influences on the environment through a comprehensive, state-of-the-science, readily accessible global model of atmospheric composition.\n\n[Global Power Plant Database](http://datasets.wri.org/dataset/globalpowerplantdatabase)\n\n[Geographic 2D CRS used in World](https://georepository.com/crs_4326/WGS-84.html)\n\n(https://www.gov.uk/government/organisations/defence-science-and-technology-laboratory)\n\n### Remote Sensing Tools\n- [QGIS- A Free and Open Source Geographic Information System](https://www.qgis.org/en/site/)\n- [Manifold GIS Systems](https://gisgeography.com/manifold-gis-systems/)\n- [MapInfo Professional] (https://gisgeography.com/mapinfo-professional-pitney-bowes/)\n- [ArcGIS Pro](https://gisgeography.com/arcgis-pro/)\n\n\n### Just useful links\n-Measure, plan, and act to reduce emissions. https://insights.sustainability.google/\n Mapping emissions\nExplore Google’s data to discover the impacts and opportunities in your city\n\n- https://www.cosmiqworks.org/\n > Since 2015, CosmiQ Works has focused on the geospatial analytics market and provided technical insights, targeted research, and reports. \n\n- [The Spacenet Challenge](https://i.ho.lc/winning-solution-for-the-spacenet-challenge-joint-learning-with-openstreetmap.html)\n > Winning Solution for the Spacenet Challenge: Joint Learning with OpenStreetMap\n\n### Libs - Satellite Imagery\n- [Solaris](https://github.com/CosmiQ/solaris)\n > An open source ML pipeline for overhead imagery by CosmiQ Works \n\n### Projects\n- [SATELLITE IMAGERY MULTISCALE RAPID DETECTION WITH WINDOWED NETWORKS (SIMRDWN) REPOSITORY](https://www.cosmiqworks.org/category/archived-projects/)\n\n- [SPACENET 6 Multimodal Data Analysis for Travel Routing and Time Estimation](https://www.cosmiqworks.org/category/current-projects/)\n\n- [SATELLITE IMAGERY MULTISCALE RAPID DETECTION WITH WINDOWED NETWORKS (SIMRDWN) REPOSITORY](https://www.cosmiqworks.org/category/archived-projects/)\n\n-[COMET TIME SERIES (TS)](https://www.cosmiqworks.org/archived-projects/comet-time-series-ts/)\n\n\n### Just some useful Codes\n#### [Overview of the EIE Analytics Challenge](https://www.kaggle.com/paultimothymooney/overview-of-the-eie-analytics-challenge) by @paultimothymooney \n\n#### [How to get started with the Earth Engine data](https://www.kaggle.com/paultimothymooney/how-to-get-started-with-the-earth-engine-data) by @paultimothymooney \n\n[🌏🌿DS4G: Go to the Green Future!] (https://www.kaggle.com/caesarlupum/ds4g-go-to-the-green-future) by @caesarlupum \n\nSimilar comp. [Satellite Imagery]\n[Dstl Satellite Imagery Feature Detection](https://www.kaggle.com/c/dstl-satellite-imagery-feature-detection/overview)\n\n"},{"metadata":{},"cell_type":"markdown","source":" \n\n#
About the data
\n\n\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n\n\n[Global Power Plant database ](https://developers.google.com/earth-engine/datasets/catalog/WRI_GPPD_power_plants) by WRI\n> Description\nThe Global Power Plant Database is a comprehensive, open source database of power plants around the world. It centralizes power plant data to make it easier to navigate, compare and draw insights for one’s own analysis. The database covers approximately 30,000 power plants from 164 countries and includes thermal plants (e.g. coal, gas, oil, nuclear, biomass, waste, geothermal) and renewables (e.g. hydro, wind, solar). Each power plant is geolocated and entries contain information on plant capacity, generation, ownership, and fuel type. It will be continuously updated as data becomes available.\n\n[Sentinel 5P OFFL NO2](https://developers.google.com/earth-engine/datasets/catalog/COPERNICUS_S5P_OFFL_L3_NO2) by [EU/ESA/Copernicus](https://sentinel.esa.int/web/sentinel/user-guides/sentinel-5p-tropomi/document-library)\n> Sentinel-5 Precursor\nSentinel-5 Precursor is a satellite launched on 13 October 2017 by the European Space Agency to monitor air pollution. The onboard sensor is frequently referred to as Tropomi (TROPOspheric Monitoring Instrument). The OFFL/NO2 is a dataset that provides offline high-resolution imagery of **NO2 concentration**.\n\n[Global Forecast System 384-Hour Predicted Atmosphere Data](https://developers.google.com/earth-engine/datasets/catalog/NOAA_GFS0P25) by NOAA/NCEP/EMC\n> The Global Forecast System (GFS) is a weather forecast model produced by the National Centers for Environmental Prediction (NCEP). The GFS dataset consists of selected model outputs (described below) as gridded forecast variables. The 384-hour forecasts, with 3-hour forecast interval, are made at 6-hour temporal resolution (i.e. updated four times daily). Use the 'creation_time' and 'forecast_time' properties to select data of interest.\n\n[Global Land Data Assimilation System](https://developers.google.com/earth-engine/datasets/catalog/NASA_GLDAS_V021_NOAH_G025_T3H) by NASA\n> Global Land Data Assimilation System (GLDAS) ingests satellite and ground-based observational data products. Using advanced land surface modeling and data assimilation techniques, it generates optimal fields of land surface states and fluxes. his dataset provided by NASA ingest satellite.\n\nParticipants may also consider using other public datasets related to trade commodities for fuel types, total fuel consumed, and/or data from the [US Energy Information Agency (EIA)](https://www.eia.gov/state/data.php?sid=RQ#CarbonDioxideEmissions)."},{"metadata":{},"cell_type":"markdown","source":"\n#
End Notebook
\n[Back to Table of Contents](#top)\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}