{"cells":[{"metadata":{},"cell_type":"markdown","source":"# OVERVIEW\n\nThe Environmental Insights Explorer team at Google is keen to gather insights on ways to improve calculations of global emissions factors of sub-national regions.The ultimate goal of this challenge is to test if calculations of emission factor using remote sensing technique are possible.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Remote Sensing\n  \n Remote sensing is the acquistion of information about an object or phenomenon without making physical contact with the object and thus in contrast to on site observation,especially the earth. Remote sensing is used in numerous fields, including geography,land surveying and most Earth Sciene disciplines; it also has military,intelligence,commercial,economic,planning, and humanitarian applications.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Remote Sensing Techniques\n \nRemote sensing utilizes satellite and/or  airborne based sensors to collect information about a given object or area. Remote sensing data collection methods can be passive or active.Passive sensors(e.g., spetral imagers) detect natural radition that is emitted or reflected by the object or area being observed. In active remote sensing energy is emitted and resultant signal that is reflected back is measured.\n![](http://th.bing.com/th/id/OIP.cr73jDfOvCNKcH3-1jcSXgHaFj?w=223&h=180&c=7&o=5&dpr=2.5&pid=1.7)","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Earth science data & analysis\n\nThe following datasets have been provided as a starter kit to get started with the competition.\n\n1.**Sentinel-5P Carbon Monoxide**\nConcentrations of Carbon monoxide(CO) and water vapor. CO is an important atmospheric trace gas for our understanding of tropospheric chemistry. Main sources of CO are combustion of fossil fuels,biomass burning, and atmospheric oxidation of methane and other hydrocarbons.\n\nSentinel-5 Percursor is statelite launched on 13 october by the European Space Agency to monitor air pollution.The onboard sensor is frequently referred to as Tropomi.\n\n2**Sentinel-5P Nitrogen Dioxide**\nTroposheric and Stratospheric nitrogen dioxide concentration. Nitrogen dioxide enters the atmosphere as a result of anthropogenic activities such as fossil fuel combustion and biomass burning as well as natural processes including microbiological processes in soils,wildfires and lighting.\n\nSentinel-5 Precursor is a satellite launched on 13 october 2017 by the European Space Ageny to monitor air pollution.The onboard sensor is frequently referred to as Tropomi.\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"**Importing  necessary libraries**","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nimport datetime as dt\nfrom datetime import datetime\n# color pallette\ncnf, dth, rec, act = '#393e46', '#ff2e63', '#21bf73', '#fe9801'\nimport folium \nfrom folium import plugins\nimport plotly.express as px\nimport rasterio as rio\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Global Power Plant Database (GPPD)\n\nThe Global Power Plant Database is a comprehensive, open source database of power plants around the world.It centralizes power plant data to make it easier to navigate,compare and draw insights. Each power plant is geolocated and entries contain information on plant capcity, generation,ownership and fueltype. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"power_plant=pd.read_csv('/kaggle/input/ds4g-environmental-insights-explorer/eie_data/gppd/gppd_120_pr.csv')\ndisplay(power_plant.shape)\ndisplay(power_plant.columns)\ndisplay(power_plant['system:index'].is_unique)\ndisplay(power_plant.head())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"power_plant.set_index('system:index',inplace=True)\npower_plant.head().T","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's understand some of the attributes :\n* **capcity_mw**- electrical generating capcity in megawatts\n* **commissioning year** - year of plant operation \n* **country_long**- full name of country\n* **estimated_generation_gwh** - estimated annual electricity generation in gigawatt-hours\n* **generation_gwh_2013**- generation of electricity in gigawatt-hours for the year2013\n* **geolocation_source** - Dataset provider\n* **gppd_idnr**- It is id of Global Power Plant Database under department of lllinois Natural Resourcessource\n* **name**- name or title of powerplant\n* **primary_fuel**-energy source used in primary eletricity generation \n* **weep_id**- a reference to a unique plant identifier in the widely used PLATTS-WEPP database\n* **year_of_capcity_data**-year the capcity information was reported\n* **source**- entity reporting the data\n* **owner**- majority shareholder of the powerplant\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"display(\"Database Country\")\ndisplay(power_plant['country'].value_counts())\ndisplay(\"ID of Global Power Plant Database\")\ndisplay(power_plant['gppd_idnr'].value_counts())\ndisplay(\"Stockholder of power plant\")\ndisplay(power_plant['owner'].value_counts())\ndisplay(\"Source to generated electricity\")\ndisplay(power_plant['source'].value_counts())\ndisplay(\"Types of primary fuel  used :\")\ndisplay(power_plant['primary_fuel'].value_counts())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"power_plant['name'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#plot primary fuel used\nsns.countplot(power_plant['primary_fuel'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Year of establishment of plant\npower_plant['commissioning_year'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#contribution of Stock Holder\npower_plant['owner'].value_counts(ascending=True).plot(kind='barh',title='Contribution of stock holders ')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#types of source\npower_plant['source'].value_counts(ascending=True).plot(kind='barh',title='Types of source')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"temp=power_plant.groupby('commissioning_year')['estimated_generation_gwh','capacity_mw'].sum().reset_index()\ntemp=temp[temp['commissioning_year']==max(temp['commissioning_year'])].reset_index(drop=True)\ntm=temp.melt(id_vars=\"commissioning_year\",value_vars=[\"estimated_generation_gwh\",\"capacity_mw\"])\ntemp.head()\nfig=px.treemap(tm,path=[\"variable\"],values=\"value\",height=225,width=1200,color_discrete_sequence=[act,rec])\nfig.data[0].textinfo='label+text+value'\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Estimated generation growth from commissioning year\ntemp=power_plant.groupby('commissioning_year')['estimated_generation_gwh','capacity_mw'].sum().reset_index()\ntemp=temp.melt(id_vars=\"commissioning_year\",value_vars=[\"estimated_generation_gwh\",\"capacity_mw\"],var_name='Year',value_name='Count')\ntemp.head()\nfig=px.area(temp,x='commissioning_year',y='Count',color='Year',height=600,title='Production  over time',color_discrete_sequence=[rec,dth])\nfig.update_layout(xaxis_rangeslider_visible=True)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"full_grouped=power_plant.groupby(['source','primary_fuel',])['capacity_mw','estimated_generation_gwh'].sum().reset_index()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"temp_1=full_grouped.sort_values(by='estimated_generation_gwh',ascending=False)\ntemp_1=temp_1.reset_index(drop=True)\ntemp_1.style.background_gradient(cmap='Blues')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#total electricity generation in gigwatthour for one year\ntotal_gen=power_plant['estimated_generation_gwh'].sum()\nprint('Total Generation :'+'{:.3f}'.format(total_gen)+'GW')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# percentage of total generation in gigawatthour\ngeneration = (power_plant.groupby(['primary_fuel'])['estimated_generation_gwh'].sum()).to_frame()\ngeneration = generation.sort_values('estimated_generation_gwh',ascending=False)\ngeneration['percentage_of_total'] = (generation['estimated_generation_gwh']/total_gen)*100\ngeneration","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.gcf()\nfig.set_size_inches(10, 6)\ncolors = ['dodgerblue', 'plum', '#F0A30A','#8c564b','orange','green','yellow'] \ngeneration['percentage_of_total'].plot(kind='bar',color=colors)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\ngeneration = (power_plant.groupby(['source'])['estimated_generation_gwh'].sum()).to_frame()\ngeneration = generation.sort_values('estimated_generation_gwh',ascending=False)\ngeneration['percentage_of_total'] = (generation['estimated_generation_gwh']/total_gen)*100\ngeneration","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#total production capcity \ntotal_cap=power_plant['capacity_mw'].sum()\nprint('Total Capcity :'+'{:.3f}'.format(total_cap)+'MW')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"capcity = (power_plant.groupby(['primary_fuel'])['capacity_mw'].sum()).to_frame()\ncapcity = capcity.sort_values('capacity_mw',ascending=False)\ncapcity['percentage_of_total'] = (capcity['capacity_mw']/total_cap)*100\ncapcity","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.gcf()\nfig.set_size_inches(10, 6)\ncolors = ['dodgerblue', 'plum', '#F0A30A','#8c564b','orange','green','yellow'] \ncapcity['percentage_of_total'].plot(kind='bar',color=colors)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"capcity = (power_plant.groupby(['source'])['capacity_mw'].sum()).to_frame()\ncapcity = capcity.sort_values('capacity_mw',ascending=False)\ncapcity['percentage_of_total'] = (capcity['capacity_mw']/total_cap)*100\ncapcity","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Second Part\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_points_on_map(dataframe,begin_index,end_index,latitude_column,latitude_value,longitude_column,longitude_value,zoom):\n    df = dataframe[begin_index:end_index]\n    location = [latitude_value,longitude_value]\n    plot = folium.Map(location=location,zoom_start=zoom)\n    for i in range(0,len(df)):\n        popup = folium.Popup(str(df.primary_fuel[i:i+1]))\n        folium.Marker([df[latitude_column].iloc[i],df[longitude_column].iloc[i]],popup=popup).add_to(plot)\n    return(plot)\n\ndef overlay_image_on_puerto_rico(file_name,band_layer):\n    band = rio.open(file_name).read(band_layer)\n    m = folium.Map([lat, lon], zoom_start=8)\n    folium.raster_layers.ImageOverlay(\n        image=band,\n        bounds = [[18.6,-67.3,],[17.9,-65.2]],\n        colormap=lambda x: (1, 0, 0, x),\n    ).add_to(m)\n    return m\n\ndef plot_scaled(file_name):\n    vmin, vmax = np.nanpercentile(file_name, (5,95))  # 5-95% stretch\n    img_plt = plt.imshow(file_name, cmap='gray', vmin=vmin, vmax=vmax)\n    plt.show()\n\ndef split_column_into_new_columns(dataframe,column_to_split,new_column_one,begin_column_one,end_column_one):\n    for i in range(0, len(dataframe)):\n        dataframe.loc[i, new_column_one] = dataframe.loc[i, column_to_split][begin_column_one:end_column_one]\n    return dataframe\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"power_plant = pd.read_csv('/kaggle/input/ds4g-environmental-insights-explorer/eie_data/gppd/gppd_120_pr.csv')\npower_plant = split_column_into_new_columns(power_plant,'.geo','latitude',50,66)\npower_plant = split_column_into_new_columns(power_plant,'.geo','longitude',31,48)\npower_plant['latitude'] = power_plant['latitude'].astype(float)\na = np.array(power_plant['latitude'].values.tolist()) # 18 instead of 8\npower_plant['latitude'] = np.where(a < 10, a+10, a).tolist() \nlat=18.200178; lon=-66.664513\nplot_points_on_map(power_plant,0,425,'latitude',lat,'longitude',lon,9)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Explore the weather data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"image='/kaggle/input/ds4g-environmental-insights-explorer/eie_data/gldas/gldas_20181210_0600.tif'\nimage_band=rio.open(image).read(1)\nplot_scaled(image_band)\noverlay_image_on_puerto_rico(image,band_layer=1)\nimage='/kaggle/input/ds4g-environmental-insights-explorer/eie_data/gfs/gfs_2019051106.tif'\nimage_band=rio.open(image).read(1)\nplot_scaled(image_band)\noverlay_image_on_puerto_rico(image,band_layer=1)\nimage='/kaggle/input/ds4g-environmental-insights-explorer/eie_data/gfs/gfs_2019031218.tif'\nimage_band=rio.open(image).read(1)\nplot_scaled(image_band)\noverlay_image_on_puerto_rico(image,band_layer=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Check the Automated land use classification","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# checck random image:\nimage='../input/inputravi/l3-ne43h01-094-059-01feb2013-band2.tif'\nimage_band=rio.open(image).read(1)\nplot_scaled(image_band)\noverlay_image_on_puerto_rico(image,band_layer=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Connect to the Google Earth Engine API","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"from kaggle_secrets import UserSecretsClient\nfrom google.oauth2.credentials import Credentials\nimport ee\nimport folium\n\ndef add_ee_layer(self, ee_image_object, vis_params, name):\n  # https://github.com/google/earthengine-api/blob/master/python/examples/ipynb/ee-api-colab-setup.ipynb\n  map_id_dict = ee.Image(ee_image_object).getMapId(vis_params)\n  folium.raster_layers.TileLayer(\n    tiles = map_id_dict['tile_fetcher'].url_format,\n    attr = 'Map Data &copy; <a href=\"https://earthengine.google.com/\">Google Earth Engine</a>',\n    name = name,\n    overlay = True,\n    control = True\n  ).add_to(self)\n\ndef plot_ee_data_on_map(dataset,column,begin_date,end_date,minimum_value,maximum_value,latitude,longitude,zoom):\n    # https://github.com/google/earthengine-api/blob/master/python/examples/ipynb/ee-api-colab-setup.ipynb\n    folium.Map.add_ee_layer = add_ee_layer\n    vis_params = {\n      'min': minimum_value,\n      'max': maximum_value,\n      'palette': ['006633', 'E5FFCC', '662A00', 'D8D8D8', 'F5F5F5']}\n    my_map = folium.Map(location=[latitude,longitude], zoom_start=zoom, height=500)\n    s5p = ee.ImageCollection(dataset).filterDate(\n        begin_date, end_date)\n    my_map.add_ee_layer(s5p.first().select(column), vis_params, 'Color')\n    my_map.add_child(folium.LayerControl())\n    display(my_map)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After registering at [https://earthengine.google.com/signup/,](http://earthengine.google.com/signup/,) navigate to the add on menu of the notebook editor and create a new user secret called \"earth_engine\" that contains the refresh token from ee.Authenticate().This step only need to run once. YOu can generate the refresh_token by running the following line in a new code cell:ee.Authenticate().Next, follow the instructions ,paste the value into the input box, run the following command in  new code cell:**!cat ~/.config/earthengine/credentials**.This should return to you a refresh token that can then be saved as your Kagggle user secret.\n\nIn summary:\n\n* Step 0: Register your account using both of the following two links:[Links#1](https://earthengine.google.com/signup/) ,[Link#2](https://docs.google.com/forms/d/e/1FAIpQLScFk_pkrrDDF4O8imsEBMaryLDU-Ghf44eHbgujIAl_SXJTJQ/viewform)    \n* Step 1: Open an internet-enabled notebbok and then retrieve your token by opening the ee.Authenticate() link in a new tab.You will also ned to copy/paste that value into the relavant input box.\n* Step2: After completing Step1,retrieve your refresh-token by running **!cat ~/.config/earthengine/credentials**\n* step3: Save your refresh-token as kaggle user secret\n* Step 4: Run the code snippet that contains ee.Initialize()\n\n\nSteps #0,#1,#2, and #3 only need to be perfoemed once(for the initial setup).\n\n\nStep #4 is run every time that you run your code. Hopefully that helps!","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}