{"cells":[{"metadata":{},"cell_type":"markdown","source":"<h2>Summary</h2> \nIn order to get reasonable electricity generation values from the provided Power Plant dataset (gppd_120_pr.csv), the \"estimated_generation_gwh\" values should be fixed. Just the code in the following cell is necessary; the rest of the Notebook is the justification.\n\n<h4>Update: some data forensics</h4>\nThe Kaggle Power Plant dataset for Puerto Rico is a subset of the [GPPD (Global Power Plant Database)](http://datasets.wri.org/dataset/globalpowerplantdatabase), which is a wonderful effort to have information from as many Power Plants in the world as possible. The GPPD data for US Power Plants comes in turn from the EIA-923 reports, which seem very reliable; however, I have not seen information from Puerto Rico Power Plants in EIA-923 reports previous to 2017; thus, in case Puerto Rico's information in the GPPD was populated previous to 2017, it may be a combination of sources. The geopositioning data seems ok, and the \"capacity_mw\" and \"primary_fuel\" data match those in [\"Tabla de datos\"](http://http://energia.pr.gov/datos/plantas/), from PREPA, Puerto Rico Electric Power Authority (Autoridad de Energía Eléctrica, in Spanish). However, the \"estimated_generation_gwh\" values seem invented, as I justify below."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nsource_capacity_factors = {\"Coal\": 0.55, \"Hydro\": 0.40, \"Gas\": 0.80, \"Oil\": 0.64, \"Solar\": 0.25, \"Wind\": 0.30, \"Nuclear\": 0.85}\n\n# \"force_fix\": - if False, the source_capacity_factors dictionary values are applied only to the \n#                \"estimated_generation_gwh\" values whose Capacity Factor is > 1\n#              - if True, all the \"estimated_generation_gwh\" values are fixed with the source_capacity_factors dictionary values\ndef fix_estimated_generation(gpp_df, source_capacity_factors, force_fix=False):\n    gpp_df[\"capacity_factor\"] = np.where(gpp_df[\"capacity_mw\"] > 0, gpp_df[\"estimated_generation_gwh\"] / (gpp_df[\"capacity_mw\"]*24*365/1000), 0)\n    for idx in range(gpp_df.shape[0]):\n        if (gpp_df.loc[idx, 'capacity_factor'] > 1) or force_fix: \n            gpp_df.loc[idx, 'capacity_factor'] = source_capacity_factors[gpp_df.loc[idx, \"primary_fuel\"]]\n            gpp_df.loc[idx, 'estimated_generation_gwh'] = gpp_df.loc[idx, \"capacity_factor\"] * gpp_df.loc[idx, \"capacity_mw\"] * 24*365/1000\n    return gpp_df\n\n# Uncomment next line if you have not read yet the Power Plants file\n# global_power_plants = pd.read_csv('../input/ds4g-environmental-insights-explorer/eie_data/gppd/gppd_120_pr.csv')\n# global_power_plants = fix_estimated_generation(global_power_plants, source_capacity_factors)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2>The issue</h2>\nWhile browsing the nice analysis in https://www.kaggle.com/parulpandey/understanding-the-data-wip, I realized there seemed to be some inconsistency between the \"capacity_mw\" and \"estimated_generation_gwh\" columns. If we represent the \"capacity_mw\" by source (Coal, Oil, Hydro... in \"primary_fuel\" column), we get the following plot:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nglobal_power_plants = pd.read_csv('../input/ds4g-environmental-insights-explorer/eie_data/gppd/gppd_120_pr.csv')\ntotal_capacity_mw = global_power_plants['capacity_mw'].sum()\nprint('Total Installed Capacity: '+'{:.2f}'.format(total_capacity_mw) + ' MW')\ncapacity = (global_power_plants.groupby(['primary_fuel'])['capacity_mw'].sum()).to_frame()\ncapacity = capacity.sort_values('capacity_mw',ascending=False)\ncapacity['percentage_of_total'] = (capacity['capacity_mw']/total_capacity_mw)*100\ncapacity['percentage_of_total'].plot(kind='bar',color=['orange', 'yellow', 'black', 'orange','cyan','blue'], \n                                     title=\"Capacity per fuel type (%)\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"On the other hand, representing the \"estimated_generation_gwh\" by source shows the following plot:"},{"metadata":{"trusted":true},"cell_type":"code","source":"total_estimated_generation_gwh = global_power_plants['estimated_generation_gwh'].sum()\nprint('Total Estimated Generation per year: '+'{:.2f}'.format(total_estimated_generation_gwh) + ' GWh')\nestimated_generation = (global_power_plants.groupby(['primary_fuel'])['estimated_generation_gwh'].sum()).to_frame()\nestimated_generation = estimated_generation.sort_values('estimated_generation_gwh',ascending=False)\nestimated_generation['percentage_of_total'] = (estimated_generation['estimated_generation_gwh']/total_estimated_generation_gwh)*100\nestimated_generation['percentage_of_total'].plot(kind='bar',color=['orange', 'yellow', 'black', 'orange','cyan','blue'],\n                                                title=\"Annual generation per fuel type (%)\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This means **coal Power Plants in Puerto Rico (in fact, there is just one), whose capacity is less than 10% of the total, generate more than 90% of the total electricity???** This seemed inconsistent, so I checked the data: the estimated generation for the coal Power Plant is 450,562.69 GWh. \n\nHowever, according to the [original source of Power Plant data](http://datasets.wri.org/dataset/globalpowerplantdatabase), the biggest Power Plant in the world, the hydro one in Three Gorges Dam (China) generated 92,452.6 GWh in a year, which is 5 times less!!\n\nI guessed there should be a concept relating the capacity and the energy generation, and it does exist: the Capacity Factor(https://en.wikipedia.org/wiki/Capacity_factor) is \"the unitless ratio of an actual electrical energy output over a given period of time to the maximum possible electrical energy output over that period\". This means the Capacity Factor should be 0 < CF < 1.\n\n*Note: in order to have the ratio properly unitless, the units in the numerator and denominator must be the same; to obtain the \"maximum possible electrical energy output\" in a year means we put this Capacity in MW (MegaWatt),  to produce energy during a year, thus we must multiply by 24 hours/day and 365 days/year, and then divide by 1000 to get the \"maximum possible electrical energy output\" in a year in GWh (GigaWatt hour)\n*\n\nThus I added a capacity_factor column:"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"global_power_plants[\"capacity_factor\"] = global_power_plants[\"estimated_generation_gwh\"]/(global_power_plants[\"capacity_mw\"]*24*365/1000)\nglobal_power_plants[[\"name\", \"capacity_mw\", \"primary_fuel\", \"estimated_generation_gwh\", \"capacity_factor\"]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see the Capacity Factor is 113 (11300%) for that coal Power Plant, which is not possible.\n\nAlso, the capacity factor for the hydro Power Plants is 5.2 (5200%), which is neither possible. However, this is much less important since those are very low capacity plants compared to Oil and Coal Power Plants.\n\nFinally, we can see all the plants of a given primary_fuel have EXACTLY the same Capacity Factor; since the \"capacity_mw\" values are particular for each plant, this means the \"estimated_generation_gwh\" are not real but have been computed to maintain a certain ratio (a Capacity Factor, but not always with reasonable values) with their corresponding \"capacity_mw\" values.\n\nConsequently, we must choose reasonable Capacity Factors per source and fix the estimated generation accordingly; the following Capacity Factor ranges are typical in the USA (just as a reference):\n* Coal: between 50-60% (sometimes even 70%); I chose 55%\n* Hydro: 40-50%, but can be much less with long drought periods; I chose 40%\n\nFor the rest, the CF values calculated from the capacity and generation data provided may stay as they are, since the CF is < 1, or can also be fixed (I have added an optional force_fix flag for this):\n* Gas: 80%\n* Oil: 64%\n* Solar: 25%; the CF value extracted from values in gppd_120_pr.csv, 5%, is very low, difficult to make those plants profitable\n* Wind: 30%; the CF value extracted from values in gppd_120_pr.csv, 2%, is very low, difficult to make those plants profitable\n* Nuclear: 85%; there are no nuclear power plants in Puerto Rico, but just in case another dataset is provided which has nuclear power plants\n\nI wrote the lines of code to fix it (dictionary source_capacity_factors and function fix_estimated_generation) at the top of the NoteBook.\n\nNow the fixed estimated generation plot is:"},{"metadata":{"trusted":true},"cell_type":"code","source":"global_power_plants = fix_estimated_generation(global_power_plants, source_capacity_factors)\nglobal_power_plants[[\"name\", \"capacity_mw\", \"primary_fuel\", \"estimated_generation_gwh\", \"capacity_factor\"]]    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"total_estimated_generation_gwh = global_power_plants['estimated_generation_gwh'].sum()\nprint('Total Estimated Generation per year: '+'{:.2f}'.format(total_estimated_generation_gwh) + ' GWh')\nestimated_generation = (global_power_plants.groupby(['primary_fuel'])['estimated_generation_gwh'].sum()).to_frame()\nestimated_generation = estimated_generation.sort_values('estimated_generation_gwh',ascending=False)\nestimated_generation['percentage_of_total'] = (estimated_generation['estimated_generation_gwh']/total_estimated_generation_gwh)*100\nestimated_generation['percentage_of_total'].plot(kind='bar',color=['orange', 'yellow', 'black', 'orange','cyan','blue'],\n                                                title=\"Annual generation per fuel type (%)\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now the generation plot is much more consistent with the capacity plot, as expected.\n\nAlso, the total year estimated generation has reduced from 486860.88 GWh to 33913.69 GWh (14 times less!!). This can have impact in the NO2 emission factor attributed to Power Plants since they are 14 times less guilty than with the provided data, compared to other sources such as cars."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}