{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Understanding Sentinel-5P OFFL NO2\n\nBefore we get into modelling, it's critical we understand exactly what the data represents. Unlike many other competitions on Kaggle (that I've done before), this one requires a lot scientific background, and the datasets are complex.\n\nLet's look at the Sentinel-5P OFFL NO2 dataset as it is one of the most important ones for predicting emission factors at distance.\n\nFirstly, what does the name even mean? \n\nSentinel-5P OFFL NO2:\n* Sentinel-5P: This is the name of the satellite, the Sentinel-5 Precursor which was launched on 13 October 2017 by the European Space Agency to monitor air pollution. (pictured below)\n* OFFL: Offline data. This is in contrast to NRTI, near real time data which is basically more recent data.\n* NO2: Nitrogen dioxide. Nitrogen oxides (NO2 and NO) are important trace gases in the Earth’s atmosphere, present in both the troposphere and the stratosphere. They enter the atmosphere as a result of anthropogenic activities (in our case, power generation) and natural processes (e.g. wildfires and lightning, we'll have to watch out for these events). Note that here, NO2 is used to represent concentrations of collective nitrogen oxides because during daytime, i.e. in the presence of sunlight, a photochemical cycle involving ozone (O3) converts NO into NO2 and vice versa on a timescale of minutes. So our final values won't just be NO2.\n\n<img src=\"https://airbus-h.assetsadobe2.com/is/image/content/dam/products-and-solutions/space/earth-observation/sentinel/Sentinel-5-Precursor-inOrbit-Copyright-Max-Alexander-Airbus2017.jpg?wid=1920&fit=fit,1&qlt=85,0\" alt=\"Drawing\" style=\"width: 600px;\"/>\n\nImage from airbus: <https://airbus-h.assetsadobe2.com/is/image/content/dam/products-and-solutions/space/earth-observation/sentinel/Sentinel-5-Precursor-inOrbit-Copyright-Max-Alexander-Airbus2017.jpg>\n\nYou can also check out this inspiring video: https://youtu.be/zP0UVKAwdMc.\nOr blog post from the Sentinel team themselves: https://medium.com/sentinel-hub/measuring-air-pollution-from-space-7492f5dad7bc.\n\nInformation from: <https://developers.google.com/earth-engine/datasets/catalog/COPERNICUS_S5P_OFFL_L3_NO2>.\n\n\nYou might also be wondering how a satellite even records NO2 levels. Satellites measure the concentration of particles in the atmosphere by observing how much light reaches the surface of the Earth and how much is reflected off the aerosols. The measurement is called aerosol optical depth or aerosol optical thickness.\n\nInformation from: <https://terra.nasa.gov/citizen-science/air-quality/part-ii-track-pollution-from-space>.\n\n\nAlright, that's step one. Now we know roughly what to expect from the data. Let's open a sample image."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import tifffile as tiff\n\nimport numpy as np\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Import Data\n\nWe'll work with just the first image for now, and assume that similar conclusions can be taken about the other images."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"data_path = '/kaggle/input/ds4g-environmental-insights-explorer'\n\nimage = '/eie_data/s5p_no2/s5p_no2_20190629T174803_20190705T194117.tif'\n\ndata = tiff.imread(data_path + image)\n\nprint('Data shape:', data.shape)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('Data sample')\nprint(data[:2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Eek, those are a lot of NaNs, and the numbers are varying from 10^-5 to 10^1, what does it all mean!\n\n## Measurements Overview\n\nJumping back to our dataset description, we can find further information.\n\nThe dataset is divided into bands, where each band represents something that was measured. In this case, we have 12 bands. These bands are each measured at different locations (arg degrees).\n\nSo above we saw we have 148 x 475 x 12. We have 12 different measurements each with 148 x 475 = 70,300 locations.\n\nLet's see just one location:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"data[0][0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As noted before, we have a wide range of values. Let's list exactly what they mean.\n\n![](https://i.imgur.com/I2xNQ1Y.png)\n\nThe first column is good to see - it's the concentration of NO2 (in mols) per square meter, which is exactly what we're looking for. The creators of this dataset have done all the hard work in calculating the number for us.\n\nWe could just stop here, but let's have a look at the other values.\n\nThe next three measurements are related to different parts of the NO2 measurement which can be split out, noticeably we see that the majority of the NO2 sits in the troposhere.\n\nThe next three measurements describe some weather conditions.\n\nThe last five measurements describe the information about the satellites altitude and angle.\n\n\n### Null Values\n\nBefore we look at the detailed values, do you remember seeing the NaN values above? Let's check how many there are - and try to work out why."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure()\nf.set_size_inches(12, 9)\nfor i in range(12):\n    plt.subplot(3, 4, i+1)\n    sns.heatmap(data[:, :, i], cbar=False)\n    f\n    ax = plt.gca()\n    ax.get_xaxis().set_visible(False)\n    ax.get_yaxis().set_visible(False)\n    # https://seaborn.pydata.org/generated/seaborn.heatmap.html","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So we can see that at least half of the values are missing, and they're consistent across the different measurements. This is the boundary for the latitude / longitude that this dataset contains.\n\nLet's calculate the exact value."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('{:.0f}% of the dataset is null'.format(\n    np.isnan(data[:, :, 0]).sum() / np.multiply(*data[:, :, 0].shape)*100\n))\n\nprint('Last measurement at y index {}'.format(np.argwhere(np.isnan(data[0, :, 0])).min()-1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Detailed Look\n\nNow let's take a detailed look at the parts of data (ignoring the nulls)."},{"metadata":{"trusted":true},"cell_type":"code","source":"data_nn = data[:, :177, :]  # no nulls","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### NO2\n\nWe saw four different measurements of NO2, how do the differ?"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"titles = ['NO2_column_number_density',\n          'tropospheric_NO2_column_number_density', \n          'stratospheric_NO2_column_number_density',\n          'NO2_slant_column_number_density']\n\nf = plt.figure()\nf.set_size_inches(8, 8)\nfor i in range(4):\n    plt.subplot(2, 2, i+1)\n    sns.heatmap(data_nn[:, :, i], cbar=False)\n    f\n    ax = plt.gca()\n    ax.get_xaxis().set_visible(False)\n    ax.get_yaxis().set_visible(False)\n    plt.title(titles[i], fontsize=10)\n    # https://seaborn.pydata.org/generated/seaborn.heatmap.html","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for i in range(4):\n    print('{}: {:.2E} mol/m^2'.format(titles[i], np.nanmean(data_nn[:, :, i])))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Three of the four measurements all look basically the same, which suggests that our assumption to simply take the first value is good. The outlier is the stratoshperic value which seems to gradually increase from top to bottom. It doesn't affect the first measurement since the magnitude is very small.\n\nBonus (for those who need a refresher): the Troposhere is the lowest level of the atmoshpere, it is where humans live, and where weather happens. The Stratosphere sits above, has a much lower air density / pressure and contains the ozone layer. See the image below.\n\n![](https://scied.ucar.edu/sites/default/files/images/large_image_for_image_content/atmosphere_layers_diagram_720x440.jpg)\n\nImage from the Center for Science Education: <https://scied.ucar.edu>."},{"metadata":{},"cell_type":"markdown","source":"We see that the slant column number density is slightly higher than the first, since the former is multiplied by the total air mass factor which reduces it (not sure what that is though).\n\nLet's just reiterate that this first measurement NO2_column_number_density appears to be the measurement we want to use for this competition. For example, I could say that during this period the average concentration was 6.636E-05 mol / m^2 which can be fed straight into our Emissions Factor equation \n\n$$EF = \\frac{E}{A  \\times (1-ER/100)}$$\n\nas E. We can then combine it with our Activity data from the powerplant dataset.\n\n\n### Weather\n\nLet's look at the weather:\n* tropopause_pressure\n* absorbing_aerosol_index\n* cloud_fraction"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"titles = ['tropopause_pressure', 'absorbing_aerosol_index', 'cloud_fraction']\nf = plt.figure()\nf.set_size_inches(12,4)\nfor i in range(3):\n    plt.subplot(1, 3, i+1)\n    sns.heatmap(data_nn[:, :, 4+i], cbar=False)\n    f\n    ax = plt.gca()\n    ax.get_xaxis().set_visible(False)\n    ax.get_yaxis().set_visible(False)\n    plt.title(titles[i], fontsize=16)\n    # https://seaborn.pydata.org/generated/seaborn.heatmap.html","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for i in range(3):\n    print('{}: {:.2f}'.format(titles[i], np.nanmean(data_nn[:, :, i+4])))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see from the latter two figures that there was some cloud coverage on this day towards the center-right. The tropopause pressure is mostly reverse correlated, but not quite exactly.\n\nThe weather information is important because it tells us if our data can be reliable or not. For example: cloud-covered scenes or partially snow/ice covered scenes can cause problems.\n\nInformation from: <http://www.tropomi.eu/sites/default/files/files/publicSentinel-5P-Nitrogen-Dioxide-Level-2-Product-Readme-File.pdf>."},{"metadata":{},"cell_type":"markdown","source":"### Satellite Information\n\nFinally, we can look at the satellite information:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"titles = ['sensor_altitude', 'sensor_azimuth_angle', 'sensor_zenith_angle',\n          'solar_azimuth_angle', 'solar_zenith_angle']\n\nf = plt.figure()\nf.set_size_inches(12, 8)\nfor i in range(5):\n    plt.subplot(2, 3, i+1)\n    sns.heatmap(data_nn[:, :, 7+i], cbar=False)\n    f\n    ax = plt.gca()\n    ax.get_xaxis().set_visible(False)\n    ax.get_yaxis().set_visible(False)\n    plt.title(titles[i], fontsize=16)\n    # https://seaborn.pydata.org/generated/seaborn.heatmap.html","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for i in range(5):\n    print('{}: {:.2f}'.format(titles[i], np.nanmean(data_nn[:, :, i+7])))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Not much to see here, but interestingly, our satellite is sitting at 838km (relative to WSG84). WSG84 appears to be the center of the Earth  which means our satellite would be >800km above the surface. This seems to be way too high, but I don't really know why at the moment.\n\n\nWGS84 notes: https://confluence.qps.nl/qinsy/latest/en/world-geodetic-system-1984-wgs84-182618391.html\nOrbits notes: https://earthobservatory.nasa.gov/features/OrbitsCatalog"},{"metadata":{},"cell_type":"markdown","source":"## Conclusion\n\nLooking into the data, we've understood that:\n* the data has a large amount of nulls\n* each band represents a different type of measurement (some more valuable than others), and\n* how we can get an estimate of the NO2 concentration from a TIFF file for use in this competition\n\nAs a next step, we can begin to investigate:\n* temporal effects from other TIFF files\n* other correlations between measurements\n\nFurther reading:\n* Readme for the dataset: <http://www.tropomi.eu/sites/default/files/files/publicSentinel-5P-Nitrogen-Dioxide-Level-2-Product-Readme-File.pdf>"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"hide_input":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.9"},"toc":{"base_numbering":1,"nav_menu":{},"number_sections":true,"sideBar":true,"skip_h1_title":true,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":false,"toc_position":{},"toc_section_display":true,"toc_window_display":true},"varInspector":{"cols":{"lenName":16,"lenType":16,"lenVar":40},"kernels_config":{"python":{"delete_cmd_postfix":"","delete_cmd_prefix":"del ","library":"var_list.py","varRefreshCmd":"print(var_dic_list())"},"r":{"delete_cmd_postfix":") ","delete_cmd_prefix":"rm(","library":"var_list.r","varRefreshCmd":"cat(var_dic_list()) "}},"types_to_exclude":["module","function","builtin_function_or_method","instance","_Feature"],"window_display":false}},"nbformat":4,"nbformat_minor":1}