{"cells":[{"metadata":{"id":"x0b46VULTw3T","colab_type":"text"},"cell_type":"markdown","source":"# DS4G : Environmental Insights Explorer\n# Exploring Alternatives for Emission Factor Calculations\n# (Domain - Environment) (Analytics Prediction)"},{"metadata":{"id":"rbePaaeoZVuf","colab_type":"text"},"cell_type":"markdown","source":"## The entire solution is divided into following notebooks\n\n####1. [DS4G Environmental Insights Explorer - introduction.(Part 1)](https://www.kaggle.com/meenakshiramaswamy/ds4g-eie-no2emission-earthenginereading)\n\n\n####2. [Exploration on data points, detailed analysis and correlation.(Part 2)](https://www.kaggle.com/meenakshiramaswamy/ds4g-eie-no2emission-eda/)\n\n####3. [Emission factor calculation and predictive model building.(Part 3)](https://www.kaggle.com/meenakshiramaswamy/ds4g-eie-no2emission-calcef-buildmodel/) \n\n"},{"metadata":{"id":"_HcMVBq8bKtg","colab_type":"text"},"cell_type":"markdown","source":"#### [Part 2 - Exploration of Data points and analysis] is subdivided into following topics\n\n\n* Loading the necessary libraries \n* Utility Functions\n* Reading the global power plant data\n* Calculate the input energy required for the powerplants\n* Convert the longitude, latitude into geometry points\n* Organize the image data into datasets\n* Find the co-relation between S5P, GFS and GLDAS images, influencing parameters on no2 density \n\n\n"},{"metadata":{"id":"Hg1PknTz5Sb8","colab_type":"text"},"cell_type":"markdown","source":" pr = ee.Geometry.Polygon(\n [[[-67.32297404549217, 18.563112930177304],\n [-67.32297404549217, 17.903121359128956],\n [-65.19437297127342, 17.903121359128956],\n [-65.19437297127342, 18.563112930177304]]], null, false);"},{"metadata":{"id":"yLzp0IdL5Sch","colab_type":"text"},"cell_type":"markdown","source":"### Import all packages and dataset"},{"metadata":{"id":"qR479WV5MLrJ","colab_type":"text"},"cell_type":"markdown","source":"#### Tifffile: used to read the GeoTIFF files of satellite imagery and convert to python numpy arrays for further calculations"},{"metadata":{"id":"iCGB1RW1N-Ua","colab_type":"text"},"cell_type":"markdown","source":"#### geopandas: The goal of GeoPandas is to make working with geospatial data in python easier. It combines the capabilities of pandas and shapely, providing geospatial operations in pandas and a high-level interface to multiple geometries to shapely"},{"metadata":{"id":"kt6hOHBvOIjK","colab_type":"text"},"cell_type":"markdown","source":"#### rasterstats : Python module for summarizing geospatial raster datasets based on vector geometries. "},{"metadata":{"trusted":true,"id":"EUzPzb76gNEy","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"### Folium is a Python Library that can allow us to visualize spatial data Manipulate your data in Python, then visualize it in a Leaflet map via Folium ...\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport math as m\n\nfrom datetime import datetime\n\nimport os\nimport glob\n\nimport rasterio as rio\nimport folium \n\nimport geopandas\nimport tifffile as tiff\nfrom folium import plugins\nfrom shapely.geometry import Point\nimport rasterstats\nfrom rasterstats import zonal_stats, point_query","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"HJU3uTye5SdE","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"pd.set_option('max_columns', 500)\npd.set_option('max_rows', 500)\nimport warnings\nwarnings.filterwarnings('ignore')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#Required only for Colab and Jypter NB\ndef configure_plotly_browser_state():\n import IPython\n display(IPython.core.display.HTML('''\n \n \n '''))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"LIDs4Md05SdL","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"from IPython.display import display\nfrom IPython.core.interactiveshell import InteractiveShell\nInteractiveShell.ast_node_interactivity = \"all\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from plotly import tools, subplots\nimport plotly.offline as py\npy.init_notebook_mode(connected = True)\nimport plotly.graph_objs as go\nimport plotly.express as px\nconfigure_plotly_browser_state()","execution_count":null,"outputs":[]},{"metadata":{"colab_type":"text","id":"6DZldYs2608k"},"cell_type":"markdown","source":"## Utility functions"},{"metadata":{"trusted":true,"id":"cAtUYfnu5Sdc","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"def plot_points_on_map(dataframe,latitude_value,longitude_value,zoom):\n #df = dataframe[begin_index:end_index]\n location = [latitude_value,longitude_value]\n fuelColor = {'Coal': 'darkred',\n 'Oil': 'black',\n 'Gas': 'lightgray',\n 'Hydro': 'lightblue',\n 'Solar': 'orange',\n 'Wind': 'green'\n }\n Map = folium.Map(location=location,zoom_start=zoom)\n for i in range(0,dataframe.shape[0]):\n fuel = dataframe['primary_fuel'][i]\n lat = dataframe['latitude'][i]\n lon = dataframe['longitude'][i]\n name = dataframe['name'][i]\n #data = dataframe[i]\n popup = \"{}, geolocation : [{},{}], {} \".format(fuel,lat, lon, name)\n #popup = pd.DataFrame(data).to_html()\n color = fuelColor[fuel]\n folium.Marker([lat,lon],\n popup=popup,\n icon=folium.Icon(color=color, icon_color='white', icon='bolt', prefix='fa')\n ).add_to(Map) \n \n\n return Map\n\n ","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"colab_type":"code","id":"weDvKvFh608l","trusted":true,"colab":{}},"cell_type":"code","source":"def overlay_image_on_puerto_rico(file_name,band_layer,lat,lon,zoom):\n band = rio.open(file_name).read(band_layer)\n m = folium.Map([lat, lon], zoom_start=zoom)\n folium.raster_layers.ImageOverlay(\n image=band,\n bounds = [[18.6,-67.3,],[17.9,-65.2]],\n colormap=lambda x: (1, 0, 0, x),\n ).add_to(m)\n return m\n\ndef plot_scaled(file_name):\n vmin, vmax = np.nanpercentile(file_name, (5,95)) # 5-95% stretch\n img_plt = plt.imshow(file_name, cmap='gray', vmin=vmin, vmax=vmax)\n plt.show()\n\ndef split_column_into_new_columns(dataframe,column_to_split,new_column_one,begin_column_one,end_column_one):\n for i in range(0, len(dataframe)):\n dataframe.loc[i, new_column_one] = dataframe.loc[i, column_to_split][begin_column_one:end_column_one]\n return dataframe","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"qRGt3YQD5SeW","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"import json\n\ndef string_to_dict(dict_string):\n # Convert to proper json format (from here: https://stackoverflow.com/questions/39169718/convert-string-to-dict-then-access-keyvalues-how-to-access-data-in-a-class)\n dict_string = dict_string.replace(\"'\", '\"').replace('u\"', '\"')\n return json.loads(dict_string)['coordinates']\n\n","execution_count":null,"outputs":[]},{"metadata":{"id":"EBCE3eJ98WDt","colab_type":"text"},"cell_type":"markdown","source":"### We have been provided with the geographic boundary of Puerto Rico to calculate the emission factor from power plants"},{"metadata":{"id":"O19u1TkN5Sef","colab_type":"text"},"cell_type":"markdown","source":"## Read the power plant data"},{"metadata":{"trusted":true,"id":"djPHyfR4gNFy","colab_type":"code","outputId":"6f365468-687f-4a02-82c8-d7d4464722fd","executionInfo":{"status":"ok","timestamp":1584775205168,"user_tz":-330,"elapsed":26291,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"colab":{"base_uri":"https://localhost:8080/","height":313},"_kg_hide-input":true},"cell_type":"code","source":"#for kaggle\npower_plants = pd.read_csv('/kaggle/input/ds4g-environmental-insights-explorer/eie_data/gppd/gppd_120_pr.csv')\ndata_path = '/kaggle/input/ds4g-environmental-insights-explorer'\n# For colab\n#data_path = '/content/drive/My Drive/Kaggle-DS4G'\n#power_plants = pd.read_csv(data_path+'/eie_data/gppd/gppd_120_pr.csv')\n\npower_plants.head(5)\n","execution_count":null,"outputs":[]},{"metadata":{"id":"merlAWZler6I","colab_type":"text"},"cell_type":"markdown","source":"### Descriptive satistics on the datafram shows the NaN and missing values in categorical and numerical columns"},{"metadata":{"id":"wZhfplVh-YEF","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":432},"outputId":"8c1071d8-d35d-4b78-fad0-6b992ad2b74d","executionInfo":{"status":"ok","timestamp":1584775205169,"user_tz":-330,"elapsed":26254,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants.describe (include = 'all')","execution_count":null,"outputs":[]},{"metadata":{"id":"Bbn5YF8v5Ser","colab_type":"text"},"cell_type":"markdown","source":"Arrangement of Coordinates"},{"metadata":{"_kg_hide-input":true,"colab_type":"code","id":"K1DlwAqB608o","trusted":true,"colab":{}},"cell_type":"code","source":"power_plants = split_column_into_new_columns(power_plants,'.geo','latitude',50,66)\npower_plants = split_column_into_new_columns(power_plants,'.geo','longitude',31,48)\npower_plants['latitude'] = power_plants['latitude'].astype(float)\na = np.array(power_plants['latitude'].values.tolist()) # 18 instead of 8\npower_plants['latitude'] = np.where(a < 10, a+10, a).tolist() \npower_plants['longitude'] = power_plants['longitude'].astype(float)\npower_plants['coord'] = power_plants['.geo'].apply(string_to_dict)\n#power_plants.head()","execution_count":null,"outputs":[]},{"metadata":{"id":"k0GygtEK5Se7","colab_type":"text"},"cell_type":"markdown","source":"**The dataset gppd_120_pr.csv contains 24 columns, we include only those are required as of now**"},{"metadata":{"trusted":true,"id":"Z3pX0CDBgNGF","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df = power_plants[['capacity_mw', \n 'commissioning_year', \n 'country', \n 'estimated_generation_gwh', \n 'source', 'name', \n 'owner', 'primary_fuel','latitude','longitude','wepp_id','coord']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15,17))\npower_plants_df.groupby(['estimated_generation_gwh','primary_fuel']).sum()['capacity_mw'].plot(kind='bar')\nplt.ylabel('capacity_mw');\nplt.title('Fossil power plants grouped by power generation , fuel type');","execution_count":null,"outputs":[]},{"metadata":{"id":"UOMaDabK3bwb","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":893},"outputId":"2a82db95-fdf3-45e4-9861-d4a3882e0bfc","executionInfo":{"status":"ok","timestamp":1584775206114,"user_tz":-330,"elapsed":27133,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(15,15))\nsns.barplot(x='capacity_mw', y='estimated_generation_gwh', hue='primary_fuel', data= power_plants_df[power_plants_df['primary_fuel'].isin(['Coal','Oil','Gas'])])\n#plt.set_xticklabels(a.get_xticklabels(), rotation=45)\nplt.ylabel('Estimated Generation')\nplt.title('Capacity Vs Estimated Generation of fossil power plants');","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"\npower_plants_df_fueltype = power_plants.groupby(['primary_fuel']).agg({'capacity_mw':'sum',\n 'estimated_generation_gwh':'sum'}).reset_index()\npower_plants_df_fueltype\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**** Some power plants are in same location but fueled by either fuel oil or gas, Like Aguirre has both Gas and Oil plants"},{"metadata":{"id":"1pqaWHuY3dH-","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":608},"outputId":"9e40a367-f76a-40b4-dbf3-e7e3f094318a","executionInfo":{"status":"ok","timestamp":1584775207203,"user_tz":-330,"elapsed":28099,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(15,10))\nplt.title(\"Fuel Generation % from various type of power plants\")\ncolors = ['g', 'orange', 'pink', 'c', 'm', 'y']\npower_plants_df['primary_fuel'].value_counts().plot(kind='pie', colors=colors, \n autopct='%1.1f%%',\ncounterclock=False, shadow=True)","execution_count":null,"outputs":[]},{"metadata":{"id":"VM-kUG8_4G96","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":421},"outputId":"c57905b2-934a-4455-dbec-a1f09d483101","executionInfo":{"status":"ok","timestamp":1584775207204,"user_tz":-330,"elapsed":28062,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"%matplotlib inline \n\nfig, ax = plt.subplots(figsize=[8,6])\nplt.title ('Various Power Plants Capacity (in MW)')\nplt.xlabel('Fuel Type')\nplt.ylabel('Capacity mw');\nax.bar(power_plants_df_fueltype['primary_fuel'], power_plants_df_fueltype['capacity_mw'], color='YGR')","execution_count":null,"outputs":[]},{"metadata":{"id":"gXNUG-2a4UW8","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":138},"outputId":"172f180e-1d51-4d33-a715-d5c2420fe806","executionInfo":{"status":"ok","timestamp":1584775207205,"user_tz":-330,"elapsed":28024,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants['primary_fuel'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"id":"xmdOL0DpOaQ6","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"### Using Geopandas we convert the coordinates into geometry and plot them in the map\n'''\npp_gdf = geopandas.GeoDataFrame(\n power_plants_df, geometry=geopandas.points_from_xy(power_plants_df.longitude, power_plants_df.latitude))\n\nworld = geopandas.read_file(geopandas.datasets.get_path('naturalearth_lowres'))\n\n# We restrict to Puerto Rica.\nax = world[world.continent == 'Puerto Rica'].plot(\n color='white', edgecolor='black')\n\n# We can now plot our ``GeoDataFrame``.\npp_gdf.plot(ax=ax, color='magenta')\n\nplt.show()\npp_gdf['name'] = pp_gdf['name'] + ':' + pp_gdf['primary_fuel']\n'''\npower_plants_fossil_df = power_plants_df[power_plants_df['primary_fuel'].isin(['Coal','Oil','Gas'])]","execution_count":null,"outputs":[]},{"metadata":{"id":"nju1Z3gGgOFS","colab_type":"text"},"cell_type":"markdown","source":"#### The dataset contains both fossil and renewable powerplants. We split them into two datasets fossil and green plants"},{"metadata":{"id":"x2e9hbNlZkKF","colab_type":"text"},"cell_type":"markdown","source":"#### Source : Google \n#### Typical thermal efficiency for utility-scale electrical generators is around 37% for coal and oil-fired plants, and 56 – 60% (LEV) for combined-cycle gas-fired plants. Plants designed to achieve peak efficiency while operating at capacity will be less efficient when operating off-design (i.e. temperatures too low.)"},{"metadata":{"id":"Es43BoHKXLib","colab_type":"text"},"cell_type":"markdown","source":"#### Our PP dataset has only one Coal plant , A E S Corp located at the southern part of the country. The conventional steam coal is used (as per eie.gov), we define the efficiency of the different fuel type PP as per (https://www.eia.gov/electricity/annualhtml/epa_08_01.html\n#### the average heating rate of coal for year 2018 is 10481, so efficiency would be 3412/10481 = 32%\n#### petroleum, natural gas are 11095 and 7821, hence effiency would be 30%, 43% respectively"},{"metadata":{"id":"LJUXLZQ-7Ev2","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def pp_efficiency(x,y):\n if x == 'Coal':\n y = 32\n elif x == 'Oil':\n y = 30\n elif x == 'Gas' :\n y = 43\n else : #nuclear\n y = 32 \n return y","execution_count":null,"outputs":[]},{"metadata":{"id":"Xq6KrDht17mZ","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_fossil_df['Efficiency_percent'] = \"\"\npower_plants_fossil_df['Efficiency_percent'] = power_plants_fossil_df.apply(lambda x: pp_efficiency(x.primary_fuel, x.Efficiency_percent), axis=1)","execution_count":null,"outputs":[]},{"metadata":{"id":"YEBZ4UY85SfX","colab_type":"text"},"cell_type":"markdown","source":"\n#### We measure the input energy in terms of thermal units , Btu. In order to get the (output) , how much input energy is needed for the powerplant? We multiply estimated generated gwh by 1000000 to convert to kwh,\n#### 1 kwh = 3412 Btu"},{"metadata":{"id":"-x-ahVhciS-2","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def input_Btu(x,y):\n if x == 'Coal':\n y = y * 1000000 * 10481 \n elif x == 'Oil':\n y = y * 1000000 * 11095 \n elif x == 'Gas' :\n y = y * 1000000 * 7821 \n else : #nuclear\n y = y * 1000000 * 10455 \n return y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"uZSnWQ-K5Sfb","colab_type":"code","colab":{}},"cell_type":"code","source":"# Reference https://www.eia.gov/tools/faqs/faq.php?id=107&t=3\n#power_plants_fossil_df['input_energy_gwh'] = (power_plants_fossil_df['estimated_generation_gwh'] / (power_plants_fossil_df['Efficiency_percent']*100))\n#power_plants_fossil_df['input_energy_Btu'] = power_plants_fossil_df['input_energy_gwh'] * 3412 * 1000000\n","execution_count":null,"outputs":[]},{"metadata":{"id":"2EfUtUkxgTe0","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_fossil_df['input_energy_Btu'] = \"\"\npower_plants_fossil_df['input_energy_Btu'] = power_plants_fossil_df.apply(lambda x: input_Btu(x.primary_fuel, x.estimated_generation_gwh), axis=1)","execution_count":null,"outputs":[]},{"metadata":{"id":"aN2uV2hUpIp-","colab_type":"text"},"cell_type":"markdown","source":"\n\n#### capacity utilization of a powerplant/day is nothing but (estimated_generation_gwh * 1000 (mwh) /(365 multiplied by 24))\n\n* Assuming on stream days as 365 days"},{"metadata":{"id":"QkcJ70x2pGav","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Convert gwh into mwh, find the capacity utilization per day\npower_plants_fossil_df['capacity_utilization'] = (power_plants_fossil_df ['estimated_generation_gwh']/ power_plants_fossil_df['capacity_mw'] * 1000 ) / (365*24)","execution_count":null,"outputs":[]},{"metadata":{"id":"EF-Us0tLivDM","colab_type":"text"},"cell_type":"markdown","source":"\n1. Organic nitrogen compounds are found in coal and oil in concentrations of \na few tenths to a few percent by weight\n\n2. Bituminous coal contains 1–2 percent nitrogen, and United States crude oil approximately 0.05–0.5 percent nitrogen (Demski et al. 1973).\n\n3. Natural gas, while containing up to 4 percent nitrogen gas, does not contain any significant organic nitrogen (Perry et al. 1963). Because organic nitrogen compounds have relatively high molecular weights, they tend to be concentrated in the residual and heavy oil fractions during distillation.\n\n4. Nitrogen oxides are produced during combustion by the oxidation of organic nitrogen compounds in fossil fuels and by the thermal fixation of atmospheric nitrogen gas, N2.\n\nRefer : https://www.nap.edu/read/10840/chapter/18#716"},{"metadata":{"id":"8kK3lC0h2dKr","colab_type":"text"},"cell_type":"markdown","source":"#### The emission factor of fossil fuels are measured in gms per mmBtu, So the conversion would be dividing the input energy in mmBtu by 1000000 and multiplying the emissions per mmBtu according to each fuel type\n\nRefer : https://www.epa.gov/sites/production/files/2015-07/documents/emission-factors_2014.pdf"},{"metadata":{"id":"T2H3bQFnWRFG","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# no2 emission in gms per input energy in Btu for powerplant , convert into mmBtu, then apply the emission / gm value\ndef nox_emission_gms(x,y):\n if x == 'Coal':\n y = ( y / 1000000 ) * 1.6\n elif x == 'Oil':\n y = ( y / 1000000 ) * 0.6\n elif x == 'Gas':\n y = ( y / 1000000 ) * 0.1\n return y\n","execution_count":null,"outputs":[]},{"metadata":{"id":"WjiEsvVdXtRD","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_fossil_df['no2_Emission_for_input_Btu_in_gms'] = \"\"\npower_plants_fossil_df['no2_Emission_for_input_Btu_in_gms'] = power_plants_fossil_df.apply(lambda x: nox_emission_gms(x.primary_fuel, x.input_energy_Btu), axis=1)","execution_count":null,"outputs":[]},{"metadata":{"id":"pbo9SVgOtLxc","colab_type":"text"},"cell_type":"markdown","source":""},{"metadata":{"id":"3b8lsjfcXlt3","colab_type":"text"},"cell_type":"markdown","source":"### No2 / mols is 46.0055 during conversion from grams to moles"},{"metadata":{"id":"kPL2b_BVomgZ","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"power_plants_fossil_df['no2_emission_in_tons'] = power_plants_fossil_df['no2_Emission_for_input_Btu_in_gms'] / 1000000\npower_plants_fossil_df['no2_emission_in_mols'] = power_plants_fossil_df['no2_Emission_for_input_Btu_in_gms'] / 46.0055","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"Upwwr7795Sfl","colab_type":"code","outputId":"3e4228b4-2100-4d34-f8c6-3460378b2a7b","executionInfo":{"status":"ok","timestamp":1584775207220,"user_tz":-330,"elapsed":27822,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"colab":{"base_uri":"https://localhost:8080/","height":550},"_kg_hide-input":true},"cell_type":"code","source":"power_plants_fossil_df[['capacity_mw','estimated_generation_gwh','name','primary_fuel','input_energy_Btu',#'capacity_utilization',\n 'no2_Emission_for_input_Btu_in_gms','no2_emission_in_tons', 'no2_emission_in_mols']]","execution_count":null,"outputs":[]},{"metadata":{"id":"aTHa1JfHsWW-","colab_type":"code","outputId":"5821bed2-3812-40c2-da51-73a916076ba3","executionInfo":{"status":"ok","timestamp":1584775207220,"user_tz":-330,"elapsed":27788,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"colab":{"base_uri":"https://localhost:8080/","height":433},"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"power_plants_fossil_df.info()","execution_count":null,"outputs":[]},{"metadata":{"id":"VNgakA_avj1A","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":893},"outputId":"c48d06a6-7d87-4f74-958c-94009ba3ee36","executionInfo":{"status":"ok","timestamp":1584775208388,"user_tz":-330,"elapsed":28921,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"'''\nplt.figure(figsize=(25,15))\nsns.barplot(x='no2_in_mols', y='no2_Emission_for_input_Btu_in_gms', hue='primary_fuel', data=power_plants_fossil_df[power_plants_fossil_df['primary_fuel'].isin(['Coal','Oil','Gas'])])\n#plt.set_xticklabels(a.get_xticklabels(), rotation=45)\nplt.ylabel('Estimated Generation')\nplt.title('no2_in_mols Vs no2_Emission_for_input_Btu_in_gms of fossil power plants');\n'''","execution_count":null,"outputs":[]},{"metadata":{"id":"JeH08wxdzbKK","colab_type":"text"},"cell_type":"markdown","source":"#### Gas power plants are less pollutants compared to oil and Coal plants"},{"metadata":{"trusted":true,"id":"W6OSGirj5Sfv","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"power_plants_renew_df = power_plants_df[power_plants_df['primary_fuel'].isin(['Wind','Hydro','Solar'])]\npower_plants_renew_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"4I1oxKc35SgG","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"power_plants_df.to_csv('power_plants.csv')\npower_plants_fossil_df.to_csv('fossil.csv')\npower_plants_renew_df.to_csv('renew.csv')\n#pp_gdf.to_csv('pp_gdf.csv')","execution_count":null,"outputs":[]},{"metadata":{"id":"Drttgbm75SgP","colab_type":"text"},"cell_type":"markdown","source":"https://www.kaggle.com/vpatricio/ds4g-where-does-the-no2-come-from"},{"metadata":{"id":"BfRvahaC5SgQ","colab_type":"text"},"cell_type":"markdown","source":"### For fiscal year 2019, \n### petroleum fueled 40% of Puerto Rico’s total electricity generation \n### natural gas accounted for 39%. \n### Coal continued to fuel 18% of generation, \n### while renewables supplied 2.3%."},{"metadata":{"id":"jGMu-Q0g5SgY","colab_type":"text"},"cell_type":"markdown","source":"## 329 Kwh /kg of coal. 1MWH = 1000 Kwh, =1000 ÷0.329 =3039.5kgs or 3040kgs of coal is required for 1MWH generation of electrical power."},{"metadata":{"trusted":true,"id":"3XgcJXov5Sga","colab_type":"code","outputId":"15f76e89-76f5-4741-b049-613373bfa605","executionInfo":{"status":"ok","timestamp":1584775208392,"user_tz":-330,"elapsed":28839,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"colab":{"base_uri":"https://localhost:8080/","height":136},"_kg_hide-input":true},"cell_type":"code","source":"power_plants_fossil_df[power_plants_fossil_df['primary_fuel'] == ('Coal')]","execution_count":null,"outputs":[]},{"metadata":{"id":"L2fwqoG5ojWc","colab_type":"text"},"cell_type":"markdown","source":"### Total Capacity of fossil power plants\n#### Oil, Gas, Coal "},{"metadata":{"trusted":true,"id":"Ng8NktKR5Sgd","colab_type":"code","outputId":"d28d1498-84ed-4322-8886-0a21c81fe5dc","executionInfo":{"status":"ok","timestamp":1584775208394,"user_tz":-330,"elapsed":28808,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"colab":{"base_uri":"https://localhost:8080/","height":104},"_kg_hide-input":true},"cell_type":"code","source":"\nprint ('Oil: ', power_plants_fossil_df.loc[power_plants_fossil_df.primary_fuel=='Oil','capacity_mw'].sum() )\nprint ('Gas :',power_plants_fossil_df.loc[power_plants_fossil_df.primary_fuel=='Gas','capacity_mw'].sum() )\nprint ('Coal:',power_plants_fossil_df.loc[power_plants_fossil_df.primary_fuel=='Coal','capacity_mw'].sum() )","execution_count":null,"outputs":[]},{"metadata":{"id":"MxfUfMcD5ShE","colab_type":"text"},"cell_type":"markdown","source":"### Organize the data from the image files"},{"metadata":{"id":"gcXJUF9R5ShF","colab_type":"text"},"cell_type":"markdown","source":"#### Need to extract the date details from the file names, GFS , GLDAS contains single date with time, whereas s5p contains both starting and ending datetime"},{"metadata":{"trusted":true,"id":"d8K0jYJh5ShI","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"def convert_df( dff, j):\n name, ext = dff['filename'][j].split('.')\n #print (name, ext)\n df_name = name[0:3]\n split_str = name.split('_')\n \n if df_name in 'gld' :\n dff['dataset'][j] = split_str[0]\n dff['date_from'][j] = split_str[1]\n dff['time_f'][j] = split_str[2]\n elif df_name in 'gfs':\n dff['dataset'][j] = split_str[0]\n dff['date_from'][j] = split_str[1][:-2]\n dff['time_f'][j] = split_str[1][8:]+'00'\n elif df_name in 's5p':\n dff['dataset'][j] = split_str[0]+'_'+split_str[1]\n dff['date_from'][j] = split_str[2][:8]\n dff['date_to'][j] = split_str[3][:8]\n dff['time_f'][j] = split_str[2][9:]\n dff['time_t'][j] = split_str[3][9:]\n #dff['no2_mean'] = np.mean(tiff.imread(image)[:, :, 0 : 4])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"mYSkl7SF5ShM","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"gldas_files = os.listdir(data_path + '/eie_data/gldas')\ndf_gldas = pd.DataFrame(gldas_files)\ndf_gldas.rename(columns={0:'filename'}, inplace=True)\ndf_gldas['dataset'] = \"\"\ndf_gldas['date_from']=\"\"\ndf_gldas['time_f']=\"\"\n\nfor i in range(df_gldas.shape[0]):\n convert_df(df_gldas, i)\ndf_gldas.head() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"ZTHLqhFU5ShO","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"gfs_files = os.listdir(data_path + '/eie_data/gfs')\ndf_gfs = pd.DataFrame(gfs_files)\ndf_gfs.rename(columns={0:'filename'}, inplace=True)\ndf_gfs['dataset'] = \"\"\ndf_gfs['date_from']=\"\"\ndf_gfs['time_f']=\"\"\n\nfor i in range(df_gfs.shape[0]):\n convert_df(df_gfs, i)\ndf_gfs.head(5) ","execution_count":null,"outputs":[]},{"metadata":{"id":"FqwUxkVJ5ShV","colab_type":"text"},"cell_type":"markdown","source":"### Sentinel-5P OFFL NO2: Offline Nitrogen Dioxide"},{"metadata":{"trusted":true,"id":"OtvSNpQF5ShW","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"image = data_path + '/eie_data/s5p_no2/s5p_no2_20180701T161259_20180707T175356.tif'\nim = rio.open(image)\nim.descriptions\nim.dataset_mask()\nim.units","execution_count":null,"outputs":[]},{"metadata":{"id":"HYdt9yWuz2hC","colab_type":"text"},"cell_type":"markdown","source":"### We can find the coordinates for individual positions within each image by running the following code snippet:"},{"metadata":{"id":"-9NantxDyWvY","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def preview_meta_data(file_name):\n with rio.open(file_name) as img_filename:\n print('Bounding Box:',img_filename.bounds)\n print('\\nCoordinates of Top Left Corner: ',img_filename.transform * (0,0))\n print('\\nCoordinates of Bottom Right Corner: ',img_filename.transform * (img_filename.width,img_filename.height))","execution_count":null,"outputs":[]},{"metadata":{"id":"XKTSsSYd2on-","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"preview_meta_data(image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"7jvd_pO95Shj","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"s5p_files = os.listdir(data_path + '/eie_data/s5p_no2')\ndf_s5p_no2 = pd.DataFrame(s5p_files)\ndf_s5p_no2.rename(columns={0:'filename'}, inplace=True)\ndf_s5p_no2['dataset'] = \"\"\ndf_s5p_no2['date_from']=\"\"\ndf_s5p_no2['date_to']=\"\"\ndf_s5p_no2['time_f']=\"\"\ndf_s5p_no2['time_t']=\"\"\n\nfor i in range(df_s5p_no2.shape[0]):\n convert_df(df_s5p_no2, i)\n #print (df_s5p_no2)\ndf_s5p_no2.head(5) \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true,"id":"wm6f9xU45Shl","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"#print (os.listdir(data_path + '/eie_data/s5p_no2'))\n\ndf_s5p_mean_no2 = pd.DataFrame(s5p_files)\ndf_s5p_mean_no2.rename(columns={0:'filename'}, inplace=True)\ndf_s5p_mean_no2['no2_mean'] = \"\"\ndf_s5p_mean_no2['NO2_col_density'] = \"\"\ndf_s5p_mean_no2['tropospheric_NO2'] = \"\"\ndf_s5p_mean_no2['stratospheric_NO2'] = \"\"\ndf_s5p_mean_no2['NO2_slant_col_density'] = \"\"\ndf_s5p_mean_no2['tropopause_pressure_mean'] = \"\"\ndf_s5p_mean_no2['aai_mean'] = \"\"\ndf_s5p_mean_no2['cloud_fraction_mean'] = \"\"\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"yYCCrs6D5Sho","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"s5p_no2_path = glob.glob(data_path+'/eie_data/s5p_no2/*')\nfor (i, image) in enumerate(s5p_no2_path):\n #df_s5p_mean_no2['img_no2_data'][i] = tiff.imread(image)[:, :, 0 : 4] # first 4 bands\n #df_s5p_mean_no2['img_data'][i] = tiff.imread(image)[:, :, 5 : 8] # all bands\n img = tiff.imread(image)\n #print (img)\n df_s5p_mean_no2['no2_mean'][i] = np.nanmean(img[:, :, 0 : 4])\n df_s5p_mean_no2['NO2_col_density'][i] = img[:, :, 0 ]\n df_s5p_mean_no2['tropospheric_NO2'][i] = np.nanmean(img[:, :, 1 ])\n df_s5p_mean_no2['stratospheric_NO2'][i] = img[:, :, 2 ]\n df_s5p_mean_no2['NO2_slant_col_density'][i] = img[:, :, 3 ]\n df_s5p_mean_no2['tropopause_pressure_mean'] [i] = np.nanmean(img[:, : , 4])\n df_s5p_mean_no2['aai_mean'][i] = np.nanmean(img[:, :, 5 ]) \n df_s5p_mean_no2['cloud_fraction_mean'][i] = np.nanmean(img[:,:,6])\n #print (\"{:.8f}\".format(x))\n","execution_count":null,"outputs":[]},{"metadata":{"id":"Oei45YLZ9qtV","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"print (df_s5p_mean_no2['NO2_col_density'][0])\nprint (np.flip(df_s5p_mean_no2['NO2_col_density'][0]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"0gEE07eW5Shz","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"df_s5p_no2 = pd.merge(df_s5p_no2, df_s5p_mean_no2, left_on='filename', right_on='filename')\n\ndf_s5p_no2['filename']= df_s5p_no2['filename'].astype(str)\ndf_s5p_no2['dataset'] = df_s5p_no2['dataset'].astype(str)\ndf_s5p_no2['date_from'] = pd.to_datetime(df_s5p_no2['date_from'])\ndf_s5p_no2['date_to'] = pd.to_datetime(df_s5p_no2['date_to'])\n\ndf_s5p_no2['tropopause_pressure_mean'] = df_s5p_no2['tropopause_pressure_mean'].fillna(\"\")\ndf_s5p_no2['aai_mean'] = df_s5p_no2['aai_mean'].fillna(\"\")\ndf_s5p_no2['cloud_fraction_mean'] = df_s5p_no2['cloud_fraction_mean'].fillna(\"\")\ndf_s5p_no2['no2_mean'] = df_s5p_no2['no2_mean'].fillna(\"\")\n#df_s5p_no2['tropospheric_NO2'] = df_s5p_no2['tropospheric_NO2'].fillna(\"\")\n\ndf_s5p_no2.sort_values('date_from', inplace = True)\ndf_s5p_no2.reset_index(drop = True, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"id":"StGXi5Zu5Sh1","colab_type":"text"},"cell_type":"markdown","source":"**Check for Not Null values**"},{"metadata":{"trusted":true,"id":"frCoAkbH5Sh7","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"\ns5p_no2_df = df_s5p_no2[['filename','dataset','date_from','date_to','no2_mean','tropospheric_NO2','tropopause_pressure_mean','aai_mean','cloud_fraction_mean']].copy()\n\nsentinel_no_nan_df = s5p_no2_df[s5p_no2_df['no2_mean'] != \"\"]\n\nsentinel_nan_df = s5p_no2_df[s5p_no2_df['no2_mean'] == \"\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"r1CnGwN45SiW","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"def line_plot(df, x, y, title, width, height, col):\n trace = go.Scatter(\n x = df[x],\n y = df[y],\n mode='lines+markers',\n name=title,\n marker = dict(\n color = col, \n ), \n )\n \n \n layout = go.Layout(\n title = go.layout.Title(\n text = title,\n x = 0.05\n ),\n font = dict(size = 10),\n width = width,\n height = height,\n )\n \n data = [trace]\n fig = go.Figure(data = data, layout = layout)\n py.iplot(fig, filename = 'line_plot')\n ","execution_count":null,"outputs":[]},{"metadata":{"id":"rDCIbw2q4QfJ","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"configure_plotly_browser_state() # To be used in the cell when using Colab or Jypter NB\nline_plot(sentinel_no_nan_df, 'date_from', 'tropospheric_NO2', 'NO2 emission - tropospheric_NO2 ', 1000, 600, 'lightblue')\n\nline_plot(sentinel_no_nan_df, 'date_from', 'no2_mean', 'NO2 emission - all NO2 ', 1000, 600, 'purple')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"1qIyp9z45SiZ","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"configure_plotly_browser_state() # To be used in the cell when using Colab or Jypter NB\n\n#line_plot(df_s5p_test, 'start_date', 'no2_mean', 'NO2 emission - mean excluding NaN', 1000, 500)\n \nline_plot(sentinel_no_nan_df, 'date_from', 'aai_mean', 'NO2 emission - AAI ', 1000, 600, 'orange')\n\n#line_plot(sentinel_no_nan_df, 'date_from', 'cloud_fraction_mean', 'NO2 emission - Cloud fraction excluding NaN', 1000, 600, 'maroon')\n\nline_plot(sentinel_no_nan_df, 'date_from', 'tropopause_pressure_mean', 'NO2 emission - tropopause pressure ', 1000, 600, 'green')\n\n","execution_count":null,"outputs":[]},{"metadata":{"id":"9MeyotYEFskU","colab_type":"text"},"cell_type":"markdown","source":"### More Q&A pls refer the below link \nhttps://www.kaggle.com/c/ds4g-environmental-insights-explorer/discussion/129991"},{"metadata":{"id":"rcFr-G2_5Sif","colab_type":"text"},"cell_type":"markdown","source":"### GFS: Global Forecast System 384-Hour Predicted Atmosphere Data"},{"metadata":{"trusted":true,"id":"oL0i_hAh5Sif","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"image = data_path+'/eie_data/gfs/gfs_2018070306.tif'\nim = rio.open(image)\nim.descriptions\nim.dataset_mask()\nim.units\nim.bounds","execution_count":null,"outputs":[]},{"metadata":{"id":"M4IFk1lQ2H9h","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"preview_meta_data(data_path+'/eie_data/gfs/gfs_2018070306.tif')","execution_count":null,"outputs":[]},{"metadata":{"id":"n5_SnfTHVTlL","colab_type":"text"},"cell_type":"markdown","source":"### Lets explore the gfs image band details"},{"metadata":{"trusted":true,"id":"6q8ilDpk5Sih","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"df_gfs_data = pd.DataFrame(gfs_files)\ndf_gfs_data.rename(columns={0:'filename'}, inplace=True)\n\ndf_gfs_data['temp_2m_above_ground_mean'] = \"\"\ndf_gfs_data['sp_humidity_2m_above_ground_mean'] = \"\"\ndf_gfs_data['rh_2m_above_ground_mean'] = \"\"\ndf_gfs_data['u_wind_10m_above_ground_mean'] = \"\"\ndf_gfs_data['v_wind_10m_above_ground_mean'] = \"\"\ndf_gfs_data['wind_velocity'] = \"\"\ndf_gfs_data['tot_percip_surface_mean'] = \"\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"UZA0FR-35Sik","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"gfs_path = glob.glob(data_path+'/eie_data/gfs/*')\n#gfs_path\nfor i, image in enumerate(gfs_path):\n img = tiff.imread(image)\n #df_gfs_data['temperature_2m_above_ground'][i] = img[:, :, 0 ]\n #df_gfs_data['specific_humidity_2m_above_ground'][i] = img[:, :, 1 ]\n #df_gfs_data['relative_humidity_2m_above_ground'][i] = img[:, :, 2 ]\n #df_gfs_data['u_component_of_wind_10m_above_ground'][i] = img[:, :, 3 ]\n #df_gfs_data['v_component_of_wind_10m_above_ground'][i] = img[:, :, 4] \n #df_gfs_data['total_precipitation_surface'][i] = img[:, :, 5] \n df_gfs_data['temp_2m_above_ground_mean'][i] = np.nanmean(img[:, :, 0 ])\n df_gfs_data['sp_humidity_2m_above_ground_mean'][i] = np.nanmean(img[:, :, 1 ])\n df_gfs_data['rh_2m_above_ground_mean'][i] = np.nanmean(img[:, :, 2 ])\n u_val = np.nanmean(img[:, :, 3 ])\n v_val = np.nanmean(img[:, :, 4 ])\n df_gfs_data['u_wind_10m_above_ground_mean'][i] = u_val\n df_gfs_data['v_wind_10m_above_ground_mean'][i] = v_val \n df_gfs_data['wind_velocity'][i] = m.sqrt ( m.pow(u_val,2 ) + m.pow(v_val, 2) ) \n df_gfs_data['tot_percip_surface_mean'][i] = np.nanmean(img[:, :, 5] )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"xKFTSVf35SjA","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"gfs_df = pd.merge(df_gfs, df_gfs_data, left_on='filename', right_on='filename')\n\ngfs_df_copy = gfs_df[['filename','dataset','date_from','time_f','temp_2m_above_ground_mean','sp_humidity_2m_above_ground_mean',\n 'rh_2m_above_ground_mean','u_wind_10m_above_ground_mean','v_wind_10m_above_ground_mean', 'wind_velocity',\n 'tot_percip_surface_mean']].copy()\n\ngfs_df_copy['filename']= gfs_df_copy['filename'].astype(str)\ngfs_df_copy['dataset'] = gfs_df_copy['dataset'].astype(str)\ngfs_df_copy['date_from'] = pd.to_datetime(gfs_df_copy['date_from'])\ngfs_df_copy['time_f'] = gfs_df_copy['time_f'].astype(str)\ngfs_df_copy['temp_2m_above_ground_mean'] = gfs_df_copy['temp_2m_above_ground_mean'].astype(float)\ngfs_df_copy['sp_humidity_2m_above_ground_mean'] = gfs_df_copy['sp_humidity_2m_above_ground_mean'].astype(float)\ngfs_df_copy['rh_2m_above_ground_mean'] = gfs_df_copy['rh_2m_above_ground_mean'].astype(float)\ngfs_df_copy['u_wind_10m_above_ground_mean'] = gfs_df_copy['u_wind_10m_above_ground_mean'].astype(float)\n\ngfs_df_copy['v_wind_10m_above_ground_mean'] = gfs_df_copy['v_wind_10m_above_ground_mean'].astype(float)\ngfs_df_copy['wind_velocity'] = gfs_df_copy['wind_velocity'].astype(float)\n\ngfs_df_copy.sort_values('date_from', inplace = True)\ngfs_df_copy.reset_index(drop = True, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"k6YwZPIU5SjK","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"configure_plotly_browser_state() # To be used in the cell when using Colab or Jypter NB\n\nline_plot(gfs_df_copy, 'date_from', 'temp_2m_above_ground_mean', 'GFS-temp_2m_above_ground_mean', 1200, 600, 'indigo')\n\n#line_plot(gfs_df_copy, 'date_from', 'sp_humidity_2m_above_ground_mean', 'GFS-sp_humidity_2m_above_ground_mean', 1200, 600, 'pink')\n\nline_plot(gfs_df_copy, 'date_from', 'rh_2m_above_ground_mean', 'GFS-rh_2m_above_ground_mean', 1200, 600, 'aquamarine')\n\n\nline_plot(gfs_df_copy, 'date_from', 'wind_velocity', 'GFS-wind-velocity', 1200, 600, 'orange')\n\nline_plot(gfs_df_copy, 'date_from', 'tot_percip_surface_mean', 'GFS-tot_percip_surface_mean', 1200, 600, 'lightblue')","execution_count":null,"outputs":[]},{"metadata":{"id":"CYcUmoavCoAs","colab_type":"text"},"cell_type":"markdown","source":"### gfs has no missing data. So we merge the sentinel df without nan values to gfs df to find the correlation between these two datasets"},{"metadata":{"trusted":true,"id":"dI7EkmOI5SjT","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"no2_dfs_merge_df = pd.DataFrame()\nno2_dfs_merge_df = pd.merge(gfs_df_copy, sentinel_no_nan_df, left_on='date_from', right_on='date_from')\n\nno2_dfs_weather_df = no2_dfs_merge_df[['date_from','time_f','temp_2m_above_ground_mean','sp_humidity_2m_above_ground_mean',\n 'rh_2m_above_ground_mean','u_wind_10m_above_ground_mean','v_wind_10m_above_ground_mean','wind_velocity',\n 'tot_percip_surface_mean', 'tropospheric_NO2','aai_mean','tropopause_pressure_mean','cloud_fraction_mean', 'no2_mean']].copy()","execution_count":null,"outputs":[]},{"metadata":{"id":"d4yAwAad6W6H","colab_type":"text"},"cell_type":"markdown","source":"#### GFS provides 4 readings i.e every 6 hours data"},{"metadata":{"id":"63AVj9MBDMXS","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"no2_dfs_weather_df['temp_2m_above_ground_mean'] = no2_dfs_weather_df['temp_2m_above_ground_mean'].astype(float)\nno2_dfs_weather_df['sp_humidity_2m_above_ground_mean'] = no2_dfs_weather_df['sp_humidity_2m_above_ground_mean'].astype(float)\nno2_dfs_weather_df['rh_2m_above_ground_mean'] = no2_dfs_weather_df['rh_2m_above_ground_mean'].astype(float)\nno2_dfs_weather_df['u_wind_10m_above_ground_mean'] = no2_dfs_weather_df['u_wind_10m_above_ground_mean'].astype(float)\n\nno2_dfs_weather_df['v_wind_10m_above_ground_mean'] = no2_dfs_weather_df['v_wind_10m_above_ground_mean'].astype(float)\nno2_dfs_weather_df['wind_velocity'] = no2_dfs_weather_df['wind_velocity'].astype(float)\nno2_dfs_weather_df['tot_percip_surface_mean'] = no2_dfs_weather_df['tot_percip_surface_mean'].astype(float)\nno2_dfs_weather_df['tropospheric_NO2'] = no2_dfs_weather_df['tropospheric_NO2'].astype(float)\n","execution_count":null,"outputs":[]},{"metadata":{"id":"C9yAJ16XJZ-A","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"no2_dfs_weather_df['time_f'] = no2_dfs_weather_df['time_f'].astype(float)\nno2_dfs_weather_df['temp_2m_above_ground_mean'] = no2_dfs_weather_df['temp_2m_above_ground_mean'].astype(float)\nno2_dfs_weather_df['sp_humidity_2m_above_ground_mean'] = no2_dfs_weather_df['sp_humidity_2m_above_ground_mean'].astype(float)\n\nno2_dfs_weather_df['rh_2m_above_ground_mean'] = no2_dfs_weather_df['rh_2m_above_ground_mean'].astype(float)\nno2_dfs_weather_df['u_wind_10m_above_ground_mean'] = no2_dfs_weather_df['u_wind_10m_above_ground_mean'].astype(float)\n\nno2_dfs_weather_df['v_wind_10m_above_ground_mean'] = no2_dfs_weather_df['v_wind_10m_above_ground_mean'].astype(float)\nno2_dfs_weather_df['wind_velocity'] = no2_dfs_weather_df['wind_velocity'] .astype(float)\nno2_dfs_weather_df['tot_percip_surface_mean'] = no2_dfs_weather_df['tot_percip_surface_mean'].astype(float)\n\n\nno2_dfs_weather_df['aai_mean'] = no2_dfs_weather_df['aai_mean'].astype(float)\nno2_dfs_weather_df['tropopause_pressure_mean'] = no2_dfs_weather_df['tropopause_pressure_mean'].astype(float)\nno2_dfs_weather_df['cloud_fraction_mean'] = no2_dfs_weather_df['cloud_fraction_mean'].astype(float)\nno2_dfs_weather_df['no2_mean'] = no2_dfs_weather_df['no2_mean'].astype(float)\nno2_dfs_weather_df['tropospheric_NO2'] = no2_dfs_weather_df['tropospheric_NO2'].astype(float)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### **However, the reaction of NO with tropospheric ambient concentrations of O3 (ozone) to form NO2 is rapid**."},{"metadata":{"id":"tVH_AGeNI-YP","colab_type":"text"},"cell_type":"markdown","source":"#### To check the wind speed and direction in the noon and correlate with no2\n\n"},{"metadata":{"id":"juUTNj5EFfzP","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"no2_dfs_weather_df[(no2_dfs_weather_df.time_f == 1200.0) & (no2_dfs_weather_df.date_from >= '2019-05-01') & (no2_dfs_weather_df.date_from <= '2019-05-10')]","execution_count":null,"outputs":[]},{"metadata":{"id":"99V_vHwYAPEq","colab_type":"text"},"cell_type":"markdown","source":"### To check the wind speed and direction at midnight and correlate with no2"},{"metadata":{"id":"G2ARHwpSAH1o","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"no2_dfs_weather_df[(no2_dfs_weather_df.time_f == 0000.0) & (no2_dfs_weather_df.date_from >= '2019-05-01') & (no2_dfs_weather_df.date_from <= '2019-05-02')]","execution_count":null,"outputs":[]},{"metadata":{"id":"QIEjcKNQiale","colab_type":"text"},"cell_type":"markdown","source":"#### To check all the parameters affecting NO2 for couple of days"},{"metadata":{"id":"FEKSxzE8iimI","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"no2_dfs_weather_df[ (no2_dfs_weather_df.date_from >= '2019-05-01') & (no2_dfs_weather_df.date_from <= '2019-05-02')]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"id":"Yrj8PTAj5Sjm","colab_type":"code","colab":{},"_kg_hide-input":true},"cell_type":"code","source":"corr_weather_and_no2 = no2_dfs_weather_df.corr()\n\nplt.figure(figsize=(20,13))\na = sns.heatmap(corr_weather_and_no2, annot = True, cmap = 'inferno')\na.Title = 'NO2 - Weather Data Correlation';\nrotx = a.set_xticklabels(a.get_xticklabels(), rotation=90)\nroty = a.set_yticklabels(a.get_yticklabels(), rotation=45)\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"id":"-K0Lirc15Sjo","colab_type":"text"},"cell_type":"markdown","source":"#### NO2 is positively correlated with temparature 2m above ground, aerosol index , tropopause pressure and specific humidity, wind velocity"},{"metadata":{"id":"i5MCzmUWv9gU","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"sns.jointplot(x=\"tropospheric_NO2\", y=\"sp_humidity_2m_above_ground_mean\", data=no2_dfs_weather_df,kind = 'kde', color=\"purple\")","execution_count":null,"outputs":[]},{"metadata":{"id":"a3KPBaXF92Fb","colab_type":"text"},"cell_type":"markdown","source":"### GLDAS-2.1: Global Land Data Assimilation System"},{"metadata":{"id":"7tGGiIho93c3","colab_type":"code","colab":{},"trusted":true},"cell_type":"code","source":"image = data_path + '/eie_data/gldas/gldas_20190528_0000.tif'\nim = rio.open(image)\nim.descriptions\nim.count\nim.tags(ns='IMAGE_STRUCTURE')","execution_count":null,"outputs":[]},{"metadata":{"id":"m1ykcx-RVpnK","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"preview_meta_data(image)","execution_count":null,"outputs":[]},{"metadata":{"id":"lw7j8tmp99fo","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_gldas_data = pd.DataFrame(gldas_files)\ndf_gldas_data.rename(columns={0:'filename'}, inplace=True)\n\ndf_gldas_data['Qair_f_inst_mean'] = \"\" #Specific humidity\ndf_gldas_data['Rainf_f_tavg_mean']= \"\" #Rain\ndf_gldas_data['Tair_f_inst_mean'] = \"\" #Air temperature\ndf_gldas_data['Wind_f_inst_mean'] = \"\" #Wind Speed","execution_count":null,"outputs":[]},{"metadata":{"id":"OOp3F5kU987e","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"gldas_path = glob.glob(data_path+'/eie_data/gldas/*')\nfor i, image in enumerate(gldas_path):\n #df_gfs_data['no2_mean'][i] = np.average(tiff.imread(image)[:, :, 0 : 4])\n img = tiff.imread(image)\n #df_gldas_data['Qair_f_inst'][i] = img[:, :, 3 ]\n #df_gldas_data['Rainf_f_tavg'][i] = img[:, : , 7]\n #df_gldas_data['Tair_f_inst'][i] = img[:, :, 10 ]\n #df_gldas_data['Wind_f_inst'][i] = img[:, :, 11 ]\n df_gldas_data['Qair_f_inst_mean'][i] = np.nanmean(img[:, :, 3]) \n df_gldas_data['Rainf_f_tavg_mean'][i] = np.nanmean(img[:,:,7])\n df_gldas_data['Tair_f_inst_mean'][i] = np.nanmean(img[:, :, 10 ])\n df_gldas_data['Wind_f_inst_mean'][i] = np.nanmean(img[:, :, 11 ])","execution_count":null,"outputs":[]},{"metadata":{"id":"BQi06yrM-O22","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_gldas = pd.merge(df_gldas, df_gldas_data, left_on='filename', right_on='filename')\n\ndf_gldas['filename']= df_gldas['filename'].astype(str)\ndf_gldas['dataset'] = df_gldas['dataset'].astype(str)\ndf_gldas['date_from'] = pd.to_datetime(df_gldas['date_from'])\ndf_gldas['time_f'] = df_gldas['time_f'].astype(float)\n\ndf_gldas.sort_values('date_from', inplace = True)\ndf_gldas.reset_index(drop = True, inplace = True)\n\nno2_gldas_merge_df = pd.DataFrame()\n\nno2_gldas_merge_df = pd.merge(df_gldas, no2_dfs_weather_df, left_on= 'date_from', right_on = 'date_from', how = \"inner\")\n\nno2_gldas_merge_df.set_index('date_from','time_f')","execution_count":null,"outputs":[]},{"metadata":{"id":"Q-sCGgwkTTK0","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"no2_dfs_gldas_df = no2_gldas_merge_df[['date_from','time_f_x',#gldas time\n 'time_f_y',#gfs time\n 'temp_2m_above_ground_mean',\n 'sp_humidity_2m_above_ground_mean',\n 'rh_2m_above_ground_mean',\n # 'u_wind_10m_above_ground_mean','v_wind_10m_above_ground_mean',\n 'wind_velocity',\n 'tot_percip_surface_mean', \n 'aai_mean',\n 'tropopause_pressure_mean',\n 'cloud_fraction_mean', \n 'tropospheric_NO2',\n 'no2_mean',\n 'Rainf_f_tavg_mean',\n 'Qair_f_inst_mean',\n 'Tair_f_inst_mean',\n 'Wind_f_inst_mean']].copy()\n\n\nno2_dfs_gldas_df['Qair_f_inst_mean'] = no2_dfs_gldas_df['Qair_f_inst_mean'].astype(float)\nno2_dfs_gldas_df['Rainf_f_tavg_mean'] = no2_dfs_gldas_df['Rainf_f_tavg_mean'].astype(float)\nno2_dfs_gldas_df['Tair_f_inst_mean'] = no2_dfs_gldas_df['Tair_f_inst_mean'].astype(float)\nno2_dfs_gldas_df['Wind_f_inst_mean'] = no2_dfs_gldas_df['Wind_f_inst_mean'].astype(float)\nno2_dfs_gldas_df['time_f_x'] = no2_dfs_gldas_df['time_f_x'].astype(float)\nno2_dfs_gldas_df['time_f_y'] = no2_dfs_gldas_df['time_f_y'].astype(float)","execution_count":null,"outputs":[]},{"metadata":{"id":"oowgCZj4I8lK","colab_type":"code","colab":{},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#no2_dfs_gldas_df.info()\nno2_dfs_gldas_df.to_csv('no2_dfs_gldas_df.csv')","execution_count":null,"outputs":[]},{"metadata":{"id":"KrMud4Vf_xtK","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":1000},"outputId":"4fdc7fe4-a0b9-4627-e299-a02ead15af09","executionInfo":{"status":"ok","timestamp":1584773550893,"user_tz":-330,"elapsed":339711,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"all_df_corr = no2_dfs_gldas_df.corr()\n\nplt.figure(figsize=(20,16))\na = sns.heatmap(all_df_corr, annot = True, cmap = 'gist_earth')\na.Title = 'NO2 # Weather # Land Data Correlation';\nrotx = a.set_xticklabels(a.get_xticklabels(), rotation=90)\nroty = a.set_yticklabels(a.get_yticklabels(), rotation=45)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"RuWOV_uQjm_2","colab_type":"text"},"cell_type":"markdown","source":"#### There is a positive corelation between tropospheric no2 and 'Tair_f_inst_mean','Wind_f_inst_mean', 'sp_humidity_2m_above_ground_mean', wind velocity, temp_2m_above_ground_mean and tropopause pressure"},{"metadata":{"id":"iOMc3fi57M7T","colab_type":"text"},"cell_type":"markdown","source":"#### GLDAS records data every 3 hours, so we have 8 image readings per day"},{"metadata":{},"cell_type":"markdown","source":"#### Lets choose some randam date, take the data at midnight and 12 noon (30-May-2019)"},{"metadata":{"id":"EqpG1EtI1Pua","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":1000},"outputId":"ee8c347f-56b8-4b4f-d901-48388b89ff04","executionInfo":{"status":"ok","timestamp":1584773550894,"user_tz":-330,"elapsed":339680,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"no2_dfs_gldas_df[(no2_dfs_gldas_df.date_from == '2019-05-30') & (no2_dfs_gldas_df.time_f_x == 0.0) & (no2_dfs_gldas_df.time_f_y == 0.0)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"no2_dfs_gldas_df[(no2_dfs_gldas_df.date_from == '2019-05-30') & (no2_dfs_gldas_df.time_f_x == 1200.0) & (no2_dfs_gldas_df.time_f_y == 1200.0)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### There is no significant impact in the tropospheric no2 density, lets have a look at 6 am data"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"no2_dfs_gldas_df[(no2_dfs_gldas_df.date_from == '2019-05-30') & (no2_dfs_gldas_df.time_f_x == 600.0) & (no2_dfs_gldas_df.time_f_y == 600.0)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"tropospheric_NO2 value 0.00001"},{"metadata":{},"cell_type":"markdown","source":"Lets analyse another data for the month of december 2018 (28-Dec-2018)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"no2_dfs_gldas_df[(no2_dfs_gldas_df.date_from == '2018-12-28') & (no2_dfs_gldas_df.time_f_x == 0.0) & (no2_dfs_gldas_df.time_f_y == 0.0)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"no2_dfs_gldas_df[(no2_dfs_gldas_df.date_from == '2018-12-28') & (no2_dfs_gldas_df.time_f_x == 1200.0) & (no2_dfs_gldas_df.time_f_y == 1200.0)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"no2_dfs_gldas_df[(no2_dfs_gldas_df.date_from == '2018-12-28') & (no2_dfs_gldas_df.time_f_x == 600.0) & (no2_dfs_gldas_df.time_f_y == 600.0)]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**There is decrease on trophospheric no2 and all-no2-mean value due to wind ( 28-Dec-2018), so this confirms that no2 density is positively correlated to wind speed.\ntropospheric_NO2 value 0.000009"},{"metadata":{"id":"fsGKDKkAv9R2","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":622},"outputId":"4050c031-82d2-4395-d87b-873e37683e02","executionInfo":{"status":"ok","timestamp":1584773550894,"user_tz":-330,"elapsed":339648,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nsns.distplot(no2_dfs_gldas_df.tropospheric_NO2)","execution_count":null,"outputs":[]},{"metadata":{"id":"JHJVki-vwPZn","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":622},"outputId":"eaf9fde4-6354-473f-cdf2-5137cc9ba2bb","executionInfo":{"status":"ok","timestamp":1584773550895,"user_tz":-330,"elapsed":339617,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20, 10))\nsns.scatterplot(x=\"tropospheric_NO2\", y=\"wind_velocity\",data=no2_dfs_gldas_df, hue=\"tropospheric_NO2\" )","execution_count":null,"outputs":[]},{"metadata":{"id":"FSqiSVaWr6Pc","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":622},"outputId":"7e21839c-7342-4f94-b8a8-ba22f3b62d24","executionInfo":{"status":"ok","timestamp":1584773551957,"user_tz":-330,"elapsed":340648,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20, 10))\nsns.scatterplot(x=\"tropospheric_NO2\", y=\"tot_percip_surface_mean\",data=no2_dfs_gldas_df, hue=\"tropospheric_NO2\" ,color = 'orange' )","execution_count":null,"outputs":[]},{"metadata":{"id":"aZxD3FPfhSKJ","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":476},"outputId":"4c413901-844f-491e-b3f8-ce3fd9813895","executionInfo":{"status":"ok","timestamp":1584773553309,"user_tz":-330,"elapsed":341968,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#plt.figure(figsize=(20, 10))\n#sns.jointplot(x=\"tropospheric_NO2\", y=\"v_wind_10m_above_ground_mean\", data=no2_dfs_gldas_df, kind=\"reg\", color = 'green')","execution_count":null,"outputs":[]},{"metadata":{"id":"GDi_vbPltUhs","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":476},"outputId":"50a27d8c-b29b-48f3-e93b-43a6dc1df963","executionInfo":{"status":"ok","timestamp":1584773559462,"user_tz":-330,"elapsed":348090,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(20, 10))\nsns.jointplot(x=\"tropospheric_NO2\", y=\"Wind_f_inst_mean\", data=no2_dfs_gldas_df,kind = 'kde', color=\"maroon\")","execution_count":null,"outputs":[]},{"metadata":{"id":"YMcnkDeYt6i5","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":476},"outputId":"8fe71c36-8bcd-4119-afec-758c4df8de24","executionInfo":{"status":"ok","timestamp":1584773560327,"user_tz":-330,"elapsed":348925,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(20, 10))\nsns.jointplot(x=\"tropospheric_NO2\", y=\"temp_2m_above_ground_mean\", data=no2_dfs_gldas_df ,kind=\"hex\")\n","execution_count":null,"outputs":[]},{"metadata":{"id":"eLPMI8VPre3p","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":476},"outputId":"afc2f055-f8d9-4ec7-cd28-9aa69e4d1dba","executionInfo":{"status":"ok","timestamp":1584773567269,"user_tz":-330,"elapsed":355837,"user":{"displayName":"Meenakshi Ramaswamy","photoUrl":"https://lh4.googleusercontent.com/-3sXdlOXrEr8/AAAAAAAAAAI/AAAAAAAACC8/ngMx9EiU3T4/s64/photo.jpg","userId":"09422112087559126593"}},"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(20, 10))\nsns.jointplot(x=\"tropospheric_NO2\", y=\"Rainf_f_tavg_mean\", data=no2_dfs_gldas_df,kind = 'kde', color=\"lightgreen\")","execution_count":null,"outputs":[]},{"metadata":{"id":"AdekdG1i5SlV","colab_type":"text"},"cell_type":"markdown","source":"#### Next kernel discusses the emission factor calculation and model building , Follow [Part 3](https://www.kaggle.com/meenakshiramaswamy/ds4g-eie-no2emission-calcef-buildmodel/)"}],"metadata":{"colab":{"name":"DS4G-Part-2.ipynb","provenance":[{"file_id":"12cEi4Uz6rS50GvqiAVC3q-9X6A3QE-Dz","timestamp":1584632451131},{"file_id":"1e_y8GV4f_i2cdBmHrln2LdJRucSccTfR","timestamp":1584533637402},{"file_id":"1uSRpNrAyduJ8RuWYd9o0RDVfsfLvBS19","timestamp":1584279176959}],"collapsed_sections":[],"machine_shape":"hm"},"kernelspec":{"name":"python3","display_name":"Python 3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"},"accelerator":"GPU"},"nbformat":4,"nbformat_minor":4}