{"cells":[{"metadata":{},"cell_type":"markdown","source":"# DS4G - Geospatial Analysis, Exploring alternatives for emission factor"},{"metadata":{},"cell_type":"markdown","source":"Lot of great kernals like\n\nhttps://www.kaggle.com/paultimothymooney/overview-of-the-eie-analytics-challenge\nhttps://www.kaggle.com/paultimothymooney/how-to-get-started-with-the-earth-engine-data\nhttps://www.kaggle.com/ragnar123/exploratory-data-analysis-and-factor-model-idea\nhttps://www.kaggle.com/parulpandey/understanding-the-data-wip\n\nhave already stated the details/videos on green environment, emission factor analysis, emission sources and remote sensing process of acquiring images and data"},{"metadata":{},"cell_type":"markdown","source":"Also referred these discussion which are very useful for this project \n\nhttps://www.kaggle.com/c/ds4g-environmental-insights-explorer/discussion/130055\nhttps://www.kaggle.com/c/ds4g-environmental-insights-explorer/discussion/130221\n"},{"metadata":{},"cell_type":"markdown","source":"## Why do we use Google Earth Engine\n"},{"metadata":{},"cell_type":"markdown","source":"Google's mission is to organize the world's information and make it universally accessible and useful. In line with this mission, Earth Engine organizes geospatial information and makes it available for analysis. More generally, Google strives to make the world a better place through the use of technology."},{"metadata":{},"cell_type":"markdown","source":"**About Google Earth Engine\n\nGoogle Earth Engine is a cloud-based platform for planetary-scale environmental data analysis. **"},{"metadata":{},"cell_type":"markdown","source":"**The below statements are taken from kaggle project description page **\n\nhttps://www.kaggle.com/c/ds4g-environmental-insights-explorer/overview"},{"metadata":{"colab_type":"text","id":"ZqQ0WNGv608i"},"cell_type":"markdown","source":"## PROJECT OVERVIEW\n\nDevelop a methodology to calculate an average historical emissions factor of electricity generated for a sub-national region, using remote sensing data and techniques."},{"metadata":{},"cell_type":"markdown","source":"## PROBLEM STATEMENT\n\nCurrent emissions factors methodologies are based on time-consuming data collection and may include errors derived from a lack of access to granular datasets,inability to refresh data on a frequent basis, overly general modeling assumptions, and inaccurate reporting of emissions sources like fuel consumption. \n\nTo develop a methodology to calculate an average historical emissions factor for electricity generation in a sub-national region.\n"},{"metadata":{},"cell_type":"markdown","source":"## DATASET PROVIDED\nInitial list of datasets covering the geographic boundary of ***Puerto Rico*** to serve as the foundation for this analysis. As an island, there are fewer confounding factors from nearby areas. \nCalculate the annual historical emission factor\n1. Bonus points will be awarded for smaller time slices of the average historical emissions factors, such as one per month for the 12-month period\n2. Additional bonus points will be awarded for participants that develop methodologies for calculating **marginal emissions factors** for the sub-national region."},{"metadata":{},"cell_type":"markdown","source":"### What is Marginal Emission factor"},{"metadata":{},"cell_type":"markdown","source":"** Marginal emission factor (MEF) is an effective tool for estimating incremental changes in carbon emissions as a result of a change in demand. ... Under this plan, the power sector—responsible for 27% of carbon emissions and constituting the single largest source of carbon in 2010—must cut emissions to almost zero by 2050 **"},{"metadata":{},"cell_type":"markdown","source":"https://www.tmrow.com/blog/marginal-emissions-what-they-are-and-when-to-use-them"},{"metadata":{},"cell_type":"markdown","source":"** LOADING BASIC LIBRARIES **"},{"metadata":{"trusted":true,"id":"EUzPzb76gNEy","colab_type":"code","colab":{}},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport os\nimport rasterio as rio\nimport folium \nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"** Referred from @paulmooney's kernal [https://www.kaggle.com/paultimothymooney/how-to-get-started-with-the-earth-engine-data](http://) **"},{"metadata":{"colab_type":"text","id":"6DZldYs2608k"},"cell_type":"markdown","source":"**Step 1: Explore the power plant data**"},{"metadata":{"_kg_hide-input":true,"colab_type":"code","id":"weDvKvFh608l","trusted":true,"colab":{}},"cell_type":"code","source":"\ndef plot_points_on_map(dataframe,begin_index,end_index,latitude_column,latitude_value,longitude_column,longitude_value,zoom):\n    df = dataframe[begin_index:end_index]\n    location = [latitude_value,longitude_value]\n    plot = folium.Map(location=location,zoom_start=zoom)\n    for i in range(0,len(df)):\n        popup = folium.Popup(str(df.primary_fuel[i:i+1]))\n        folium.Marker([df[latitude_column].iloc[i],df[longitude_column].iloc[i]],popup=popup).add_to(plot)\n    return(plot)\n\ndef overlay_image_on_puerto_rico(file_name,band_layer):\n    band = rio.open(file_name).read(band_layer)\n    m = folium.Map([lat, lon], zoom_start=8)\n    folium.raster_layers.ImageOverlay(\n        image=band,\n        bounds = [[18.6,-67.3,],[17.9,-65.2]],\n        colormap=lambda x: (1, 0, 0, x),\n    ).add_to(m)\n    return m\n\ndef plot_scaled(file_name):\n    vmin, vmax = np.nanpercentile(file_name, (5,95))  # 5-95% stretch\n    img_plt = plt.imshow(file_name, cmap='gray', vmin=vmin, vmax=vmax)\n    plt.show()\n\ndef split_column_into_new_columns(dataframe,column_to_split,new_column_one,begin_column_one,end_column_one):\n    for i in range(0, len(dataframe)):\n        dataframe.loc[i, new_column_one] = dataframe.loc[i, column_to_split][begin_column_one:end_column_one]\n    return dataframe","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.set_option('max_columns', 30)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"djPHyfR4gNFy","colab_type":"code","colab":{}},"cell_type":"code","source":"power_plants = pd.read_csv('/kaggle/input/ds4g-environmental-insights-explorer/eie_data/gppd/gppd_120_pr.csv')\n\npower_plants.head(35)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"colab_type":"code","id":"K1DlwAqB608o","trusted":true,"colab":{}},"cell_type":"code","source":"power_plants = split_column_into_new_columns(power_plants,'.geo','latitude',50,66)\npower_plants = split_column_into_new_columns(power_plants,'.geo','longitude',31,48)\npower_plants['latitude'] = power_plants['latitude'].astype(float)\na = np.array(power_plants['latitude'].values.tolist()) # 18 instead of 8\npower_plants['latitude'] = np.where(a < 10, a+10, a).tolist() \nlat=18.200178; lon=-66.664513\nplot_points_on_map(power_plants,0,425,'latitude',lat,'longitude',lon,9)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**The dataset gppd_120_pr.csv contains 24 columns, we include only those are required as of now**"},{"metadata":{"trusted":true},"cell_type":"code","source":"power_plants.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"0Tt8fFTpgNGB","colab_type":"code","colab":{}},"cell_type":"code","source":"import pandas_profiling\neda_analysis = pandas_profiling.ProfileReport(power_plants)\neda_analysis.to_file('eie_analysis.html')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"zyWZxLY3gNGo","colab_type":"code","colab":{}},"cell_type":"code","source":"power_plants.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"Z3pX0CDBgNGF","colab_type":"code","colab":{}},"cell_type":"code","source":"power_plants.describe(include = 'all')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"colab_type":"code","id":"0eDVlNlC608r","trusted":true,"colab":{}},"cell_type":"code","source":"#power_plants_df = power_plants.sort_values('estimated_generation_gwh',ascending=False).reset_index()\ndsg_df = power_plants[['name','latitude','longitude','primary_fuel','capacity_mw','estimated_generation_gwh','source','owner','country','commissioning_year','year_of_capacity_data']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dsg_df.head(25)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dsg_df.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(25,15))\nsns.barplot(x='capacity_mw', y='estimated_generation_gwh', hue='primary_fuel', data=dsg_df[dsg_df['primary_fuel'].isin(['Coal','Oil','Gas'])])\n#plt.set_xticklabels(a.get_xticklabels(), rotation=45)\nplt.ylabel('Estimated Generation')\nplt.title('Capacity Vs Estimated Generation');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dsg_df_corr = dsg_df.corr()\ndsg_df_corr = sns.heatmap(dsg_df_corr, cmap=\"Accent\",  annot= True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"nPvE6juegNGJ","colab_type":"code","colab":{}},"cell_type":"code","source":"from matplotlib import style\nfrom matplotlib.pyplot import pie,show","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"qWyTgLplgNGN","colab_type":"code","colab":{}},"cell_type":"code","source":"power_plants_df_fueltype = power_plants.groupby (['primary_fuel']).agg({'capacity_mw': 'sum',\n                                                'estimated_generation_gwh': 'sum'           \n                                                }).reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"7YdT94hkgNGQ","colab_type":"code","colab":{}},"cell_type":"code","source":"power_plants_df_fueltype[['primary_fuel','capacity_mw','estimated_generation_gwh']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"PbqUg6CPgNGU","colab_type":"code","colab":{}},"cell_type":"code","source":"plt.figure(figsize=(22,15))\nplt.title(\"Fuel Generation from different power plants\")\ncolors = ['green', 'orange', 'pink', 'c', 'm', 'y']\npower_plants['primary_fuel'].value_counts().plot(kind='pie', colors=colors, \n autopct='%1.1f%%',\ncounterclock=False, shadow=True)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"5UnsGvUhgNGY","colab_type":"code","colab":{}},"cell_type":"code","source":"power_plants['primary_fuel'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"byHf6uspgNGb","colab_type":"code","colab":{}},"cell_type":"code","source":"%matplotlib inline \n\nfig, ax = plt.subplots(figsize=[8,6])\nplt.title ('Various Power Plants Capacity (in MW)')\nplt.xlabel('Fuel Type')\nplt.ylabel('Capacity mw');\nax.bar(power_plants_df_fueltype['primary_fuel'], power_plants_df_fueltype['capacity_mw'], color='YGR')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"RXemgQtsgNGg","colab_type":"code","colab":{}},"cell_type":"code","source":"import seaborn as sns\nsns.jointplot(x=\"capacity_mw\", y=\"estimated_generation_gwh\", data=power_plants_df_fueltype, kind = 'kde', color = 'orange')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"XlSxSKp4gNGl","colab_type":"code","colab":{}},"cell_type":"code","source":"df = power_plants[['name','latitude','longitude','primary_fuel','capacity_mw','estimated_generation_gwh']]\nd=df.corr()\nplt.figure(figsize=(10,7))\na = sns.heatmap(d, cmap=\"viridis\",  annot= True)\na.Title = 'Fuel Type power Generation - Heatmap';\nrotx = a.set_xticklabels(a.get_xticklabels(), rotation=45)\nroty = a.set_yticklabels(a.get_yticklabels(), rotation=45)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from skimage.io import imread\nimage = imread('/kaggle/input/ds4g-environmental-insights-explorer/eie_data/s5p_no2/s5p_no2_20180708T172237_20180714T190743.tif')\nprint (image.shape)\nplt.imshow(image[:,:,0], cmap = 'cool')\nplt.axes = False","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image = imread('/kaggle/input/ds4g-environmental-insights-explorer/eie_data/gfs/gfs_2018070400.tif')\nprint (image.shape)\nplt.imshow(image[:,:,2], cmap = 'viridis')","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"G1ca13u6608t"},"cell_type":"markdown","source":"** Explore the nO2 emissions data**"},{"metadata":{},"cell_type":"markdown","source":"**Sentinel-5P OFFL NO2: Offline Nitrogen Dioxide - NO2_column_number_density**"},{"metadata":{"_kg_hide-input":true,"colab_type":"code","id":"BLzBQe_N608u","trusted":true,"colab":{}},"cell_type":"code","source":"image = '/kaggle/input/ds4g-environmental-insights-explorer/eie_data/s5p_no2/s5p_no2_20180714T170945_20180720T185244.tif'\nimage_band = rio.open(image).read(1)\nplot_scaled(image_band)\noverlay_image_on_puerto_rico(image,band_layer=1)","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"1cWq-KgX608w"},"cell_type":"markdown","source":"** Explore the weather data**"},{"metadata":{},"cell_type":"markdown","source":"**GLDAS-2.1: Global Land Data Assimilation System  -  **"},{"metadata":{"_kg_hide-input":true,"colab_type":"code","id":"L7M5yI_9608x","trusted":true,"colab":{}},"cell_type":"code","source":"image = '/kaggle/input/ds4g-environmental-insights-explorer/eie_data/gldas/gldas_20180702_1500.tif'\nimage_band = rio.open(image).read(3)\nplot_scaled(image_band)\n\nimage = '/kaggle/input/ds4g-environmental-insights-explorer/eie_data/gfs/gfs_2018072118.tif'\nimage_band = rio.open(image).read(3)\nplot_scaled(image_band)\n\noverlay_image_on_puerto_rico(image,band_layer=3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**** Work In Progress ....****"}],"metadata":{"colab":{"name":"DS4G-Emission Factor-Analytics-EDA.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}