{"cells":[{"metadata":{},"cell_type":"markdown","source":"

EIA-923 validation, NOx emissions reference and NOx Emissions Factor reference

\nThis notebook intends to:\n\n- validate EIA-923 as a source for Power Plant fuel consumption and energy generation information (at the month level), extracted from the EIA-923 Annual Report for 2018 (EIA is U.S. Energy Information Administration) \n- provide a reference for NOx emissions in the larger Power Plants in Puerto Rico, in order to be compared later (hopefully, before the end of the competition) with the emissions inferred from data provided by the satellites. For this purpose, EIA-923 fuel consumption information is complemented with standard Emission Factors obtained from an eGRID document (eGRID belongs to EPA, U.S. Environmental Protection Agency)\n- incorporate this information into the original dataset gppd_120_pr.csv\n\nAfter applying this information the conclusion is:

The estimated reference NOx Emissions Factor for the whole grid during the year 2018 is 2.114 ton/GWh, or 4.228 lb/MWh

\n\nNote: Due to unavailability of information for the year 2019, the numbers are provided just for 2018, and consequently just the second half is inside the period under study.\n\n

Validation of EIA-923 information

\nIn a previous Notebook (https://www.kaggle.com/ajulian/gppd-120-pr-csv-and-the-capacity-factor/) I showed the provided Power Plant dataset (gppd_120_pr.csv) is reliable except for energy generation, thus I decided to look for another data source.\n\nEIA-923 is a reliable information source for energy generation and fuel consumption in the US, and has a good level of granularity: the information is provided for every month and every production area in each Power Plant.\n\nHowever, when applied to our case, Puerto Rico in the period July 2018 - June 2019, some issues should be mentioned:\n- information from Puerto Rico began being included in EIA-923 in 2017, and with a red note: \"*Generation and fuel consumption data collected from Puerto Rico power plants may be incomplete or may contain errors (...)*\" \n- the information for the rest of the US is aggregated in EIA-923 on a monthly basis, with a delay of two months (Dec 2019 info has been included on Feb 2020); however, the information from Puerto Rico is included later and for the whole year, so currently (March, 2020) there is no information yet about Puerto Rico for 2019.\n\n

EIA923_PR_2018.csv

\nEIA923_PR_2018.csv is an extract of the sheet \"Page 1 Puerto Rico 2018\" in EIA923_Schedules_2_3_4_5_M_12_2018_Final_Revision.xlsx, which is one of the files of the 2018 EIA-923 Annual Report (f923_2018.zip) in https://www.eia.gov/electricity/data/eia923/ (credit: this file was linked by @pcjimmmy in https://www.kaggle.com/c/ds4g-environmental-insights-explorer/discussion/131395)"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nmonths = [\"January\", \"February\", \"March\", \"April\", \"May\", \"June\", \n \"July\", \"August\", \"September\", \"October\", \"November\", \"December\"]\neia923_df = pd.read_csv(\"../input/eia923-pr-2018-v0/EIA923_PR_2018_v0.csv\", sep=\";\")\neia923_df = eia923_df.drop(columns=\"YEAR\")\neia923_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the original Excel file there is also a sheet \"Page 7 File Layout\" detailing all the possible values in the columns. Some values that occur in Puerto Rico are:\n- **Reported Prime Mover**: CA (Steam part of a Combined-Cycle plant), CT (Combustion turbine of a Combined-Cycle plant), IC (Internal Combustion Engine), ST (Steam Turbine); there are also values for renewable energy, such as PV (PhotoVoltaic) for Solar plants, WT (Wind Turbine), HY (Hydraulic Turbine)\n- **Reported Fuel Type Code**: BIT (Bituminous coal), DFO (Distillate Fuel Oil), RFO (Residual Fuel Oil), NG (Natural Gas); regarding renewable energy, we have WAT (Water at a conventional Hydroelectric plant), WND (Wind)\n- **Physical Unit Label (of the fuel)**: typical values are \"mcf\" (thousands of cubic feet) for gas, \"short tons\" for coal and \"barrels\" for oil\n\nThere are also twelve \"Quantity _\" columns for the fuel consumption monthly values and twelve \"Netgen _\" columns for the energy generation monthly values.\n

Capacity Factor consistency

\nFirst, we will do the same as in the previous Notebook https://www.kaggle.com/ajulian/gppd-120-pr-csv-and-the-capacity-factor/: check the Capacity Factors for all the sub-plants are < 1. Thus, we must get the capacity values from gppd_120_pr.csv; they are at the Power Plant level, while generation data in EIA-923 is at the sub-plant level, so first me must group values."},{"metadata":{"trusted":true},"cell_type":"code","source":"gppd = pd.read_csv('../input/ds4g-environmental-insights-explorer/eie_data/gppd/gppd_120_pr.csv')\n# EIA-923 has rows at the sub-plant month level, thus first must be grouped.\n# However, since the plant names must be fixed because they do not match among both files, I will only \n# check Capacity Factors for the six largest Power Plants\n\n# We keep the Generation annual columns and group it by Power Plant \ntmp_eia923_df = eia923_df.groupby(\"Plant Name\", as_index=False)[\"Net Generation MWh\"].sum()\ntmp_eia923_df = tmp_eia923_df.sort_values('Net Generation MWh', ascending=False).reset_index(drop=True)\n# Check Capacity Factor\ntmp_eia923_df[\"capacity_mw\"] = 0.0\ntmp_eia923_df.iloc[0, 2] = gppd[gppd[\"name\"]==\"Costa Sur\"][\"capacity_mw\"].sum() # \"Costa Sur Plant\" in EIA-923\ntmp_eia923_df.iloc[1, 2] = gppd[gppd[\"name\"]==\"EcoEléctrica\"][\"capacity_mw\"].sum() # EcoElectrica (no accent)\ntmp_eia923_df.iloc[2, 2] = gppd[gppd[\"name\"]==\"San Juan CC\"][\"capacity_mw\"].sum() # Central San Juan Plant\ntmp_eia923_df.iloc[3, 2] = gppd[gppd[\"name\"]==\"A.E.S. Corp.\"][\"capacity_mw\"].sum() # AES Puerto Rico\ntmp_eia923_df.iloc[4, 2] = gppd[gppd[\"name\"]==\"Aguirre\"][\"capacity_mw\"].sum() # Aguirre Plant\ntmp_eia923_df.iloc[5, 2] = gppd[gppd[\"name\"]==\"Palo Seco\"][\"capacity_mw\"].sum() # Palo Seco Plant\ntmp_eia923_df = tmp_eia923_df[:6]\ntmp_eia923_df[\"capacity_factor\"] = tmp_eia923_df[\"Net Generation MWh\"]/(tmp_eia923_df[\"capacity_mw\"]*24*365)\ntmp_eia923_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see the Capacity Factor is < 1 for the larger Power Plants, thus we can consider the EIA-923 consistent from the energy generation point of view. \n\nLet's display scatter plots of Electricity generated versus Fuel for some Power plants to check additional consistency.\n

Electricity generated versus Fuel

"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# Let's check generated power vs gas in EcoEléctrica, a gas Combined-Cycle Power Plant\neia923_eco = eia923_df.iloc[2, 4:-2]\neco_fuel = eia923_eco[:12]\neco_power = eia923_eco[12:]\nplt.scatter(eco_fuel, eco_power)\nplt.xlabel(\"Monthly Natural Gas Consumption (mcf)\")\nplt.ylabel(\"Monthly Electricity generated (MWh)\")\nplt.title(\"Gas Consumption vs. Electricity Generated in EcoEléctrica PP\")\nplt.show()\n\n# Same with the coal Power Plant, from AES Corp\neia923_aes = eia923_df.iloc[4, 4:-2]\naes_fuel = eia923_aes[1:12]\naes_power = eia923_aes[13:]\nplt.scatter(aes_fuel, aes_power)\nplt.xlabel(\"Monthly Coal Consumption (tons)\")\nplt.ylabel(\"Monthly Electricity generated (MWh)\")\nplt.title(\"Coal Consumption vs. Electricity Generated in A.E.S. Corp. PP\")\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see the relationship between Fuel Consumption and Electricity Generated is quite linear, which suggests the fuel consumption values in the EIA-923 dataset for Puerto Rico are reliable enough to apply standard Emissions Factor and obtain a reference for the NOx emissions and the Emissions Factor."},{"metadata":{},"cell_type":"markdown","source":"

NOX emissions reference

\nIn general, developing a new methodology requires as much validation against current approaches as possible. In our case of remote sensing of NOx emissions, a typical reference would be CEMS (Continuous Emission Monitoring Systems) installed in Power Plants, whose information is provided periodically and can be found in an EIA-923 file for US Power Plants; unfortunately, Puerto Rico is not (yet) included.\n\nAnother option is seek for some engineering tests performed during a limited period. I did find a 2016 report with emissions information per Power Plant in Puerto Rico, but unfortunately it does not monitorize NO2 ([\"Perfil de Emisiones Tóxicas de Puerto Rico 2016\", in Spanish](https://estadisticas.pr/files/Publicaciones/IEPR_Perfil_de_Emisiones_Toxicas_2016.pdf)).\n\nThus, the only reference I could think of is applying the corresponsing Emission Factors to the monthly fuel quantities consumption in the EIA, and aggregate the emissions in the same Power Plant.\n\n

EF_eGRID.csv

\nThe Emission Factors in EF_eGRID.csv have been gathered from [\"The Emissions & Generation Resource Integrated Database\"](https://www.epa.gov/sites/production/files/2020-01/documents/egrid2018_technical_support_document.pdf), Table C-2 in page 107 and onwards (this file was mentioned by @pcjimmmy in https://www.kaggle.com/c/ds4g-environmental-insights-explorer/discussion/131395).\n\nThe Emissions & Generation Resource Integrated Database (eGRID) is a comprehensive source of data on the environmental characteristics of almost all electric power generated in the United States, including Emission Factors. "},{"metadata":{"_cell_guid":"","_uuid":"","trusted":true},"cell_type":"code","source":"ef_egrid_df = pd.read_csv(\"../input/emissions-factors-egrid/EF_eGRID.csv\", sep=\";\")\nef_egrid_df = ef_egrid_df.drop(columns=\"Boiler firing type\") # N/A in Puerto Rico data\nef_egrid_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we can found the following columns:\n- **Primer mover**: same as \"Reported Primer Mover\" in EIA-923\n- **Primary fuel type**: same as \"Reported Fuel Type Code\" in EIA-923\n- **Nox Emission Factor**: actual Emission Factor values\n- (Emission) **Numerator Unit**: standard Emission Factors are expressed tipically as a fraction, with an Emission Unit in the Numerator and a Fuel Unit in the Denominator; the typical Emission Unit is the pound (lb)\n- (Fuel) **Denominator Unit**: same as \"Physical Unit Label\" in the EIA-923 file, with values mcf, (short) tons, barrels (there is one EF in MMBtu, millions of British Termal Units, a heat unit, but for Natural Gas it is almost equivalent to one mcf)\n\nLuckily, the code values in \"Primer mover\" and \"Primary fuel type\" in the eGRID file are the same as in \"Reported Primer Mover\" and \"Reported Fuel Type Code\" in the EIA file, so we can perform a **left outer join** to apply Emission Factors in the eGRID dataframe to the sub-plants in the EIA-923 dataframe based on the mover and fuel type. "},{"metadata":{"trusted":true},"cell_type":"code","source":"eia923_egrid_df = pd.merge(eia923_df, ef_egrid_df, how=\"left\", \n left_on=['Reported Prime Mover', 'Reported Fuel Type Code'], \n right_on=['Primer mover', 'Primary fuel type'])\n# Let's remove duplicated columns\neia923_egrid_df = eia923_egrid_df.drop(columns=[\"Primer mover\", \"Primary fuel type\"])\neia923_egrid_df.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are rows with NaN values in the \"NOx Emission Factor\", which correspond to emission-free renewable energy plants. We will keep only the fossil-fuel sub-plants (reminder from the EIA-923 file description, at the beginning of the Notebook: NG => Natural Gas, BIT => Bituminous coal, DFO => Distillate Fuel Oil, RFO => Residual Fuel Oil):"},{"metadata":{"trusted":true},"cell_type":"code","source":"eia923_egrid_df = eia923_egrid_df[eia923_egrid_df[\"Reported Fuel Type Code\"].isin([\"NG\", \"BIT\", \"DFO\", \"RFO\"])]\neia923_egrid_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Now we can compute the emissions (short tons) per sub-plant

\nEmission Factors Numerator Unit is pound (lb), so we have to divide by 2000 to obtain (short) tons."},{"metadata":{"trusted":true},"cell_type":"code","source":"for month in months:\n eia923_egrid_df[\"Emissions ton \" + month] = \\\n eia923_egrid_df[\"Quantity \" + month] * eia923_egrid_df[\"Nox Emission Factor\"]/2000 # 1 (short) ton = 2000 pounds (lb)\neia923_egrid_df.head(10) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Final steps:

\n- group by Power Plant\n- compute Emissions for 2018 (ton) per Power Plant\n- plot the eight most NO2 emissions releasing Power Plants \n- plot the eight largest Power Generating Plants\n- incorporate all this information into the original Power Plant dataset, gppd_120_pr.csv\n- calculate the NOx Emissions Factor"},{"metadata":{"trusted":true},"cell_type":"code","source":"# We keep the Generation and Emissions monthly columns and group them by Power Plant \ncols_idx = eia923_egrid_df.columns\nkeep_cols = [cols_idx[i] for i in range(len(cols_idx)) if (cols_idx[i][:3] in (\"Emi\", \"Net\"))]\neia923_egrid_df = eia923_egrid_df.groupby(\"Plant Name\", as_index=False)[keep_cols].sum()\n\n# Now we can compute \"Electricity Generated (GWh)\" and \"Emissions (ton)\" per Power Plant and the whole year\nfor month in months:\n if month==\"January\": # initialize Emissions Total\n eia923_egrid_df[\"Electricity Generated 2018 (GWh)\"] = eia923_egrid_df[\"Netgen January\"]/1000\n eia923_egrid_df[\"Emissions 2018 (ton)\"] = eia923_egrid_df[\"Emissions ton January\"] \n else:\n eia923_egrid_df[\"Electricity Generated 2018 (GWh)\"] += eia923_egrid_df[\"Netgen \" + month]/1000\n eia923_egrid_df[\"Emissions 2018 (ton)\"] += eia923_egrid_df[\"Emissions ton \" + month]\n\n# Sort by total Emissions and plot\neia923_egrid_df = eia923_egrid_df.sort_values('Emissions 2018 (ton)', ascending=False).reset_index(drop=True)\neia923_egrid_df.loc[0, \"Plant Name\"] = \"San Juan\"\n\nimport seaborn as sns\nsns.barplot(y=eia923_egrid_df.loc[:7, \"Plant Name\"], \n x=eia923_egrid_df.loc[:7, \"Emissions 2018 (ton)\"], orient=\"h\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nsns.barplot(y=eia923_egrid_df.loc[:7, \"Plant Name\"], \n x=eia923_egrid_df.loc[:7, \"Electricity Generated 2018 (GWh)\"], orient=\"h\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that the two largest Power Plants, EcoEléctrica and Costa Sur (both Natural Gas fueled), are far less contaminant than San Juan (Oil fueled) and AES (Coal fueled).\n\nNow we can add all this (real) electricity generation and (estimated) emissions information to the original dataset, gppd_120_pr.csv, after performing the same fuel-type filtering."},{"metadata":{"trusted":true},"cell_type":"code","source":"gppd_EF = gppd[gppd[\"primary_fuel\"].isin([\"Oil\", \"Gas\", \"Coal\"])]\ngppd_EF = gppd_EF.groupby(\"name\", as_index=False).agg({\"capacity_mw\": \"sum\", \"commissioning_year\": \"first\",\n \"primary_fuel\": \"first\", \"owner\": \"first\", \".geo\": \"first\"})\ngppd_EF = gppd_EF.sort_values([\"capacity_mw\"], ascending=False).reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

NOx Emissions Factor reference

Now we can sum up the NOx emissions and electricity generation for the whole grid in 2018 and estimate the NOx Emissions Factor reference:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"The estimated annual NOx emissions for the whole grid are\", round(eia923_egrid_df[\"Emissions 2018 (ton)\"].sum(), 1), \"tons\")\ngen_GWh_2018 = eia923_df[\"Net Generation MWh\"].sum()/1000\nprint(\"The estimated annual electricity generation for the whole grid is\", round(gen_GWh_2018, 1), \"GWh\")\nEF_2018_ton_GWh = eia923_egrid_df[\"Emissions 2018 (ton)\"].sum() / gen_GWh_2018\nEF_2018_lb_MWh = EF_2018_ton_GWh*2000/1000\nprint(\"The estimated NOx Emissions Factor reference for 2018 is\", round(EF_2018_ton_GWh, 3), \"ton/GWh, or\", round(EF_2018_lb_MWh, 3), \"lb/MWh\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gppd_EF = gppd_EF.loc[:7].sort_values(\"name\").reset_index(drop=True)\neia923_egrid_eight_df = eia923_egrid_df.loc[:7].sort_values(\"Plant Name\").reset_index(drop=True)\ngppd_EF = pd.concat([gppd_EF, eia923_egrid_eight_df], axis=1).drop(columns=[\"Plant Name\", \"Net Generation MWh\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Last note

\nWe have complemented the original Power Plants dataset with (real) power generation and (estimated) emissions per Power Plant, for the year 2018 and for each month, which will eventually help in development/tuning/validation of monthly Emission Factors.\n\nWe have also calculated a reference NOx Emissions Factor of 2.114 ton/GWh, or 4.228 lb/MWh.\n\nWe now can save the dataset into /kaggle/working/ for future use:"},{"metadata":{"trusted":true},"cell_type":"code","source":"gppd_EF.to_csv(\"/kaggle/working/gppd_120_pr_ef.csv\", index=False)\n\ntest_df = pd.read_csv(\"/kaggle/working/gppd_120_pr_ef.csv\")\ntest_df","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}