{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Data Measurement Frequency\n\nIt states in the competition: \"Bonus points for smaller time slices of the average historical emissions factors, such as one per month for the 12-month period.\", so it's important to use (and understand) data that has certain time scales.\n\nIn the provided datasets, what sort of measurement level are we provided with? Is it daily, weekly, yearly? Do we have consistent data or missing data?"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import os\n\nfrom datetime import datetime, timedelta\n\nimport pandas as pd\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\ndata_path = '/kaggle/input/ds4g-environmental-insights-explorer'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load Data\n\nThere are four provided datasets:\n\n* Global Power Plant database by WRI\n* Sentinel 5P OFFL NO2 by EU/ESA/Copernicus\n* Global Forecast System 384-Hour Predicted Atmosphere Data by NOAA/NCEP/EMC\n* Global Land Data Assimilation System by NASA\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"os.listdir(data_path + '/eie_data')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the GLDAS, GFS and S5P NO2 datasets, we can get the timing from the names.\n\nThey're formatted as: \n\n- gldas_20180901_1500.tif\n- gfs_2019030900.tif\n- s5p_no2_20181101T173802_20181107T192402\n\nFor s5p we will take the first date."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"gldas_files = os.listdir(data_path + '/eie_data/gldas')\ngldas_dates = [datetime.strptime(g, 'gldas_%Y%m%d_%H%M.tif') for g in gldas_files]\n\ngfs_files = os.listdir(data_path + '/eie_data/gfs')\ngfs_dates = [datetime.strptime(g, 'gfs_%Y%m%d%H.tif') for g in gfs_files]\n\ns5p_files = os.listdir(data_path + '/eie_data/s5p_no2')\ns5p_dates = [datetime.strptime(g[:16], 's5p_no2_%Y%m%d') for g in s5p_files]\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The GPPD dataset has just one csv. We can see in the columns that this is yearly data."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"pd.read_csv(data_path + '/eie_data/gppd/gppd_120_pr.csv').columns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Get all the dates in one dataframe."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"all_dates = pd.DataFrame(columns=['dataset', 'datetime']).append(\n    pd.DataFrame(gldas_dates, columns=['datetime']) \\\n        .assign(dataset = 'gldas'), sort=True\n).append(\n    pd.DataFrame(gfs_dates, columns=['datetime']) \\\n        .assign(dataset = 'gfs'), sort=True\n).append(\n    pd.DataFrame(s5p_dates, columns=['datetime']) \\\n        .assign(dataset = 's5p'), sort=True\n).append(\n    pd.DataFrame([datetime(y, 1, 1) for y in range(2013, 2018)], columns=['datetime']) \\\n    .assign(dataset = 'gppd'), sort=True\n).assign(date = lambda x: x.datetime.apply(lambda x: x.date()))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"all_dates","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##  Analysis"},{"metadata":{},"cell_type":"markdown","source":"Some basic info, grouped by date:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"all_dates.groupby('dataset').date.agg(\n    min=min,\n    max=max,\n    measurement_period=  lambda x: (x.max()-x.min()).days+1,\n    measurement_count= 'count',\n    measurements_per_day= lambda x: x.count() / ((x.max()-x.min()).days+1)\n).transpose()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So GFS, GLDAS and S5P NO2 are one year's worth of data from July 2018 until end of June 2019, though S5P NO2 appears to be missing one day.\n\nGPPD is yearly data from 2013 until 2017.\n\nStraight away we see that the emissions data doesn't align with the activity data. We'll have to do some basic interpolation.\n\nWe also see that GFS and GLDAS appear to have a steady amount of 4 and 8 measurements per day, respectively while S5P NO2 has something more uncommon."},{"metadata":{},"cell_type":"markdown","source":"## Visualisation\n\nWe can now visualise these:"},{"metadata":{"scrolled":false,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Get the date index to work with\ndaily_data = pd.date_range('2018-07-01', '2019-06-30').to_frame() \\\n    .merge(all_dates.groupby(['dataset', 'date']).date.count().unstack(level=0),\n        left_index=True,\n        right_index=True) \\\n    .drop(columns=[0, 'gppd'], axis=1)\n\nsns.heatmap(daily_data.transpose())\nfig = plt.gcf()\nfig.set_size_inches(11,3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As noticed above, we see that we have steady measurements for GFS and GLDAS, but S5P is erratic, ranging from 0 measurements to 4."},{"metadata":{},"cell_type":"markdown","source":"## Conclusion\n\nSome problems we identified:\n * the activity data (GPPD) doesn't overlap in the same timeframe as the emissions data (others)\n * the data for S5P NO2 is not evenly spaced, and even has missing dates "}],"metadata":{"hide_input":false,"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.9"},"toc":{"base_numbering":1,"nav_menu":{},"number_sections":true,"sideBar":true,"skip_h1_title":true,"title_cell":"Table of Contents","title_sidebar":"Contents","toc_cell":false,"toc_position":{},"toc_section_display":true,"toc_window_display":true},"varInspector":{"cols":{"lenName":16,"lenType":16,"lenVar":40},"kernels_config":{"python":{"delete_cmd_postfix":"","delete_cmd_prefix":"del ","library":"var_list.py","varRefreshCmd":"print(var_dic_list())"},"r":{"delete_cmd_postfix":") ","delete_cmd_prefix":"rm(","library":"var_list.r","varRefreshCmd":"cat(var_dic_list()) "}},"types_to_exclude":["module","function","builtin_function_or_method","instance","_Feature"],"window_display":false}},"nbformat":4,"nbformat_minor":1}