{"cells":[{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"%%HTML\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n# DS4G: Environmental Insights Explorer 🌏
\n### Exploring alternatives for emissions factor calculations\n \n\n[source](https://finance.yahoo.com/news/googles-chief-sustainability-officer-strives-for-ever-greater-efficiencies-in-a-highpowered-world-212906260.html?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&guce_referrer_sig=AQAAAJOy7waEjgMr5fwa-FvOUrbAVhcF-pSuSGMOMTV01KFsTc19_IGQsQkXpDCJLzsS1x0SK32JBnsfN0zzw5ZRecjmvDyk5CQSCmNZcUKPF_i-DZCf26YKK_HHjsTFQgqWGKKsX2y9nKe1wal-J9YLGtwsxRg9hBvbds4Cz3sNEbE9) "},{"metadata":{},"cell_type":"markdown","source":"\n\n# Table of contents
\n\nTeam: Green Future
\nAuthors: Name, Kaggle ID, email\n - Crislânio Macêdo, caesarlupum, crislanio.ufc@gmail.com\n - Maxime Lenormand, maxlenormand, maxime.lenormand@ipsa.fr\n\n Key Objectives:
\nThe Environmental Insights Explorer team at Google is keen to gather insights on ways to improve calculations of global emissions factors for sub-national regions. The ultimate goal of this challenge is to test if calculations of emissions factors using remote sensing techniques are possible and on par with calculations of emissions factors from current methodologies.\n\nNitrogen oxides (NOx ) are toxic air pollutants and play an\nimportant role in tropospheric chemistry as precursors of tropospheric ozone and secondary aerosols [(Jacob et al., 1996;\nSeinfeld and Pandis, 2006)](https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/96JD00336).Bottom-up emission inventories depend on information of fuel consumptions and emission factors, which are subject\nto substantial uncertainties [Primary air pollutant emissions of coal-fired power plants in China]-(Butler et al., 2008; Zhao et al.,2011). A significant improvement in accuracy of emission inventories for power plants has been achieved by the installation of continuous emissions monitoring systems (CEMSs).\n\nThis Data Science for Good Competition intends to use remote sensing techniques to understand Environmental Emissions.\nThis work is organized as follows. We start from the data description and proceed with the exploratory analysis, modeling data and then improvements of the emissions factors for Regions and primary_fuel. We detailed method explanation is provided, key findings on the emissions factors calculation in Puerto Rico are reported and anomaly detection.\n\nWe also propose some paths of thoughts on the militstions of the insights the available data can allow us to gain. We wish to make this analysis as in depth as possible but without extrapolating too much from data that isn't always perfect.\n\nPresentation: [Slides](https://docs.google.com/presentation/d/e/2PACX-1vRNrjO0oPb0nchSrPJs6z2bKG2GRWcp2lxvTPwjetNODFP3wCg80tBSPrCkQpQwUkIkxK4S7_CqrWHv/pub?start=false&loop=false&delayms=3000)\n\nConsiderations
\n1. [Limits of remote sensing techniques & available data](#L1)\n 1.1 [Remote Sensing is take measures from far away](#L2)\n 1.2 [Weather data is tricky](##L2)\n 1.3 [Each sensor is different](##L3)\n 1.4 [High-quality data from complex topics is hard to find and validate](##L4)\n 1.5 [Some good news after all!](##L5)\n\n Submissions:
\nFollowing are parts of Kernels Submissions in order:\n\n"},{"metadata":{},"cell_type":"markdown","source":"\n🌏🌿Green Future: Analysis and Solution
\n"},{"metadata":{},"cell_type":"markdown","source":"\nApproach Overview
\n[Back to Table of Contents](#top)\n\n[General Findinds](#theend)\n\nAir Quality Management is an important area and influences a lot of decisions taken by countries. But how does one ascertain the Air quality of a place? This is done by calculating the Emissions Factor of that area. Thus, an emission factor is a coefficient that converts any activity's data into GHG emissions. This factor attempts to relate the quantity of a pollutant released to the atmosphere with an activity associated with the release of that pollutant.\nFrom the challenge, the description says: Current emissions factors methodologies are based on time-consuming data collection and may include errors derived from a lack of access to granular datasets, inability to refresh data on a frequent basis, overly general modeling assumptions, and inaccurate reporting of emissions sources like fuel consumption.\n\nIn the **1° kernel**, we have a gentle introduction about the data with insights, shows challenges of remote sensing and simple modeling.\n\nIn the **2° kernel**, Its create a GeoJSON file. GeoJSONs are a great format to be able to open points very easily. They are basically just geo-located JSON files, which prevent us from having to retrieve the location info from each point with a self-made function each time we want to plot / access to points. Should add that there are more benefits to making handling GeoJSONs instead of CSVs. For example, we can very easily drop them in QGIS and compare them to any rasters we might have.\n\nIn the **3° kernel**, investigated the presence of NO2 concentration in air, considering its constant increase over days, years.\nOwing to accurate future air quality estimates, the need for detecting the anomalously high increase in the concentration of pollutants cannot be adjourned. This study is helpful in educating the government for decision making and people about spatiotemporal, geographical, and economic conditions responsible for anomalously high NO2 concentrations in air. In this work, we modeling the solution and analyze the impacts of air pollution for each region in Porto Rico for each primary_fuel in the year.\nIn this notebook we investigated the presence of NO2 concentration in air, considering its constant increase over days, years. Owing to accurate future air quality estimates, the need for detecting the anomalously high increase in the concentration of pollutants cannot be adjourned. This study is helpful in educating the government for decision making and people about spatiotemporal, geographical, and economic conditions responsible for anomalously high NO2 concentrations in air. In this work, we modeling the solution and analyze the impacts of air pollution for each region in Porto Rico for each primary_fuel in the year.\n\n\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"\n\nLimits of Remote Sensing Techniques & Available Data
\n[Back to Table of Contents](#top)\n\n[Next](#L2)\n\nIn any analysis, it is important to take into consideration what kind of data one is using and keep in mind the limitations as well as possible flaws this data might have. It can range from incomplete datasets, inaccurate measurements to plain wrong information. Remote Sensing is a complex methodology; emissions is another complex topic. It is important to keep at mind and acknowledge the boundaries to a study made using the data available. This notebook tries to cover some of these and proposes some thoughts on the possibilities given by the available data."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport geopandas as gpd\nimport os\nfrom datetime import datetime","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n\nRemote Sensing is take measures from far away
\n[Back to Table of Contents](#top)\n\n[Next](#L3)\n\nRemote Sensing, by it's very nature, is a method to remotely (as in from a distance) measure something. This is important to keep in mind as there might be things between what we are measuring and the sensor used to make the measurement.\n\nThe biggest impact of this is caused by the atmosphere, as it is composed of multiple different gases, it absorbs some wavelengths, and changes the celeriry of light (commonly noted *c*). Here is a diagram of how different wavelenghts get affected by this (Here on Landsat satellites sensors)\n\n\n(Taken from [this website sharing some information about Landsat 8](https://www.geoimage.com.au/satellite/landsat-8))\n\nWhile this is usually taken into account in pre-processing, the exact composition of the atmosphere at the precise moment an image was taken cannot be known. Pre-processing techniques to take this into account are well known and still manage to provide accurate estimations for most use cases. \n\nClouds and human pollution can also create some distortion in measurements taken remotely by satellites. [Haze correction](https://geol260.academic.wlu.edu/course-notes/image-processing/haze-correction/) is for example of one the ways to try to prevent this."},{"metadata":{},"cell_type":"markdown","source":"\n\nWeather data is tricky
\n[Back to Table of Contents](#top)\n\n[Next](#L4)\n\nThe weather data available for this competition came from the [Global Forecast System](https://www.emc.ncep.noaa.gov/emc/pages/numerical_forecast_systems/gfs.php) and the [Global Land Data Assimilation System](https://ldas.gsfc.nasa.gov/gldas/). It is important to keep in mind that these are NOT ground truth data. These are based on real measured data but they are mostly models and simulations used to create a global model. This means that different regions might have different accuracies and that this data has some \"noise\" in it. While during this competition we did try to make the most out of the available data, it is important to take all these results with a grain of salt as they are not the perfect ground truth that we might be more familiarized with here on Kaggle.\n\nGFS data ,for example, is as the name implies, forecast data. That means there is a confidence measure that should come with it, or an uncertainty. As we forecast further into the future our confidence usually goes down, and so all measurements are not equal, or comparable. "},{"metadata":{},"cell_type":"markdown","source":"\n\nEach sensor is different
\n[Back to Table of Contents](#top)\n\n[Next](#L5)\n\nJust like any model should have data used for inference as close to the training data used, sensors used to create remote sensing datasets are different in many ways, and not always easily comparable.\n\nOne very simple example is to compare the two weather datasets that we have:\n\n\n\nWe can see that these two rasters have different spatial resolution. This makes comparing the different data sources difficult as the values inside the rasters are spatial aggregates over the entire zone covered by the pixel. Keep in mind these pixels are multiple kilometers wide!\n\nIt also seems like GLDAS has a datapoint for every 3 hours while GFS is every 6 from their respective documentation. In this case it could (in theory) be easily solved by taking measurements two by two on GLDAS and computing their average. This can easily be checked:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Make a dataframe containing all the GLDAS raster paths and dates\ndf_gldas = pd.DataFrame()\nfiles=[]\ncaptured_datetime = []\nfor dirname, _, filenames in os.walk('/kaggle/input/ds4g-environmental-insights-explorer/eie_data/gldas'):\n for filename in filenames:\n files.append(os.path.join(dirname, filename))\n captured_datetime.append(datetime.strptime(filename.split('_')[-2],'%Y%m%d').date())\n \ndf_gldas['GLDAS_path'] = files\ndf_gldas['Captured_datetime'] = captured_datetime\n\n# Sort dataframe by ascending time\ndf_gldas.sort_values('Captured_datetime', inplace=True)\ndf_gldas.reset_index(inplace=True, drop=True)\n\nprint(f'There are {df_gldas.shape[0]} rasters from GLDAS')\n\nstart_date = df_gldas['Captured_datetime'].iloc[0]\nend_date = df_gldas['Captured_datetime'].iloc[-1]\nspan = end_date - start_date\n\n\nprint(f'GLDAS data range from: {start_date} to {end_date} ({end_date - start_date})')\nprint(f'This makes an average of {round((df_gldas.shape[0])/(span.days), 1)} images per day')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Make a dataframe containing all the GLDAS raster paths and dates\ndf_gfs = pd.DataFrame()\nfiles=[]\ncaptured_datetime = []\nfor dirname, _, filenames in os.walk('/kaggle/input/ds4g-environmental-insights-explorer/eie_data/gfs'):\n for filename in filenames:\n files.append(os.path.join(dirname, filename))\n# print(filename.split('_')[-1][:-4])\n# print(datetime.strptime(filename.split('_')[-1][:-4],'%Y%m%d'))\n captured_datetime.append(datetime.strptime(filename.split('_')[-1][:-6],'%Y%m%d').date())\n \ndf_gfs['GFS_path'] = files\ndf_gfs['Captured_datetime'] = captured_datetime\n\n# Sort dataframe by ascending time\ndf_gfs.sort_values('Captured_datetime', inplace=True)\ndf_gfs.reset_index(inplace=True, drop=True)\n\nprint(f'There are {df_gfs.shape[0]} rasters from GFS')\n\nstart_date = df_gfs['Captured_datetime'].iloc[0]\nend_date = df_gfs['Captured_datetime'].iloc[-1]\nspan = end_date - start_date\n\n\nprint(f'GFS data range from: {start_date} to {end_date} ({end_date - start_date})')\nprint(f'This makes an average of {round((df_gfs.shape[0])/(span.days), 1)} images per day')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can slowly see, it gets very tricky to compare these two datasets when they should have overlapping data. Also as mentioned above these are models/forecasts so the methods used to get these data to differ, leading to even more differences. This is one of the big challenges of remote sensing."},{"metadata":{},"cell_type":"markdown","source":"\nHigh-quality data from complex topics is hard to find and validate
\n[Back to Table of Contents](#top)\n\n[Next](#L6)\n\nIt is already complicated to use remote sensing data, but it is as challenging to find solid ground truth data to validate a model on. It has been shown in this competition multiple times that [there were some big flaws in the generated power dataset](https://www.kaggle.com/c/ds4g-environmental-insights-explorer/discussion/130537). This brings to one of the biggest challenges in remote sensing and in lots of different fields of data science: getting high quality data is the biggest part of the job. This complex problem of emissions modeling is no different quite the opposite. Not being able to trust the very data we are given to model makes the whole challenge about finding good, reliable data in the first place. This is beyond our skills however, as neither of us were professionals in emissions modelling nor measurement."},{"metadata":{},"cell_type":"markdown","source":"\nSome good news after all!
\n[Back to Table of Contents](#top)\n\nWhile remote sensing is really tricky, Sentinel 5 is a very useful sensor, as has been shown in the news:\n\n\n[source](https://earthobservatory.nasa.gov/images/146362/airborne-nitrogen-dioxide-plummets-over-china)\n\nThe feasibility of modeling the emissions in Puerto Rico using only freely available remains questionable after taking all these points into consideration. But we create a useful modeling show anomalies events and use forecast modeling over the year for sub-region and primary fuels. Sentinel 5 and the use of qualitative weather data remain an essential tool to help understand the changes in air pollution in populated areas as the recent drastic drops have showed us.\n
"},{"metadata":{},"cell_type":"markdown","source":"# General Findings:"},{"metadata":{},"cell_type":"markdown","source":"## Papers\n- [NOx lifetimes and emissions of cities and power plants in polluted\nbackground estimated by satellite observations](https://www.atmos-chem-phys.net/16/5283/2016/acp-16-5283-2016.pdf)\n- [Inconsistent decadal variations between surface and free tropospheric\n2 nitrogen oxides over United States](https://www.atmos-chem-phys-discuss.net/acp-2017-382/acp-2017-382.pdf)\n- [Direct observation of changing NOx lifetime in North American cities](https://authors.library.caltech.edu/99738/2/aax6832-Laughner-SM.pdf)\n- [Background information about the Row Anomaly in OMI](http://projects.knmi.nl/omi/research/product/rowanomaly-background.php)\n-[Detection of anomalous nitrogen dioxide (NO2) concentration in urban air of India using proximity and clustering methods](https://www.tandfonline.com/doi/abs/10.1080/10962247.2019.1577314?scroll=top&needAccess=true&journalCode=uawm20)\n\n[Recommended Procedures for Development of\nEmissions Factors and Use of the WebFIRE\nDatabase -PG 77 outlier](https://www3.epa.gov/ttn/chief/efpac/procedures/procedures81213.pdf)\n\n\n- [Automatic Quality Control of the Meteosat First Generation\nMeasurements\n](https://www.atmos-meas-tech-discuss.net/amt-2019-249/amt-2019-249.pdf)\n\n-[Finding and Detection of Outlier Regions in Satellite ](Image http://www.ipcsit.com/vol11/1-ICNEE2011-N002.pdf)\n\n# Useful Links - Outlier detection\n- [dixon test outliers python](https://plot.ly/python/v3/outlier-test/)\n- [Rosner's test outlier python](https://www.hs.uni-hamburg.de/DE/Ins/Per/Czesla/PyA/PyA/pyaslDoc/aslDoc/outlier.html)\n- [Comparison of Methods for detecting Outliers](https://www.researchgate.net/publication/283476235_Comparison_of_Methods_for_detecting_Outliers)\n\n## Useful Links\n- [Satellite Sensors (0.31m - 2m](https://www.satimagingcorp.com/satellite-sensors/)\n- [How To Compute Satellite Image Statistics](https://towardsdatascience.com/how-to-compute-satellite-image-statistics-and-use-it-in-pandas-81864a489144)\n-[Satellite imagery access and analysis in Python & Jupyter notebooks](https://towardsdatascience.com/satellite-imagery-access-and-analysis-in-python-jupyter-notebooks-387971ece84b)\n\n-[60 questions about Raster Image ](https://www.researchgate.net/topic/Raster-Image)\n\n## Videos\n- [Automated anomaly detection in satellite imagery](https://youtu.be/b8cDu5CoLVU?t=20)\n- [Combining satellite imagery and machine learning to predict poverty](https://youtu.be/DafZSeIGLNE)\n \n\n### Useful Videos\n- [Deep Learning and the Analysis of Satellite Imagery](https://youtu.be/F7dM5vmMqdk)\n Orbital Insight is a Geospatial Big Data company leveraging the rapidly growing availability of satellite, UAV, and other geospatial data sources, to understand and characterize socio-economic trends at global, regional, and hyper-local scales. In this talk Boris discusses the satellite imagery domain, how it’s evolving, and the various advantages and challenges of working with such imagery. \n\n## Libs\n- [Resources for deep learning with satellite & aerial imagery](https://github.com/robmarkcole/satellite-image-deep-learning)\n-[A curated list of resources focused on Machine Learning in Geospatial Data Science](https://github.com/deepVector/geospatial-machine-learning)\n-[Libraries for a modern geospatial workflow](https://gist.github.com/jacquestardie/0d1c0cb413b3b9b06edf)\n\n## Topics\n- 🌏🌿DS4G: Go to the Green Future!\nhttps://www.kaggle.com/c/ds4g-environmental-insights-explorer/discussion\n- [Official Thread]: Questions about competition setup, rules, submissions, etc\nhttps://www.kaggle.com/c/ds4g-environmental-insights-explorer/discussion/129991\n- A short guide to satellite imagery for Data Scientists\nhttps://www.kaggle.com/c/ds4g-environmental-insights-explorer/discussion/130221\n\n- Anomaly Detection in Satellite imagery 📶\nhttps://www.kaggle.com/c/ds4g-environmental-insights-explorer/discussion/131201\n- AMA with the Competition Hosts\nhttps://www.kaggle.com/c/ds4g-environmental-insights-explorer/discussion/134727\n\n👍 ✔️ Thanks Everyone✔️\n"},{"metadata":{},"cell_type":"markdown","source":"\n\n# End Notebook
\n[Back to Table of Contents](#top)\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}