{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"microsoft":{"ms_spell_check":{"ms_spell_check_language":"de"},"host":{"AzureML":{"notebookHasBeenCompleted":true}}},"nteract":{"version":"nteract-front-end@1.0.0"},"kernel_info":{"name":"python310-sdkv2"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":66052,"databundleVersionId":7284431,"sourceType":"competition"}],"dockerImageVersionId":30635,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport random\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"jupyter":{"source_hidden":false,"outputs_hidden":false},"nteract":{"transient":{"deleting":false}},"collapsed":false,"execution":{"iopub.status.busy":"2024-01-25T09:09:19.403724Z","iopub.execute_input":"2024-01-25T09:09:19.404457Z","iopub.status.idle":"2024-01-25T09:09:19.760681Z","shell.execute_reply.started":"2024-01-25T09:09:19.404420Z","shell.execute_reply":"2024-01-25T09:09:19.759671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Read the data","metadata":{"nteract":{"transient":{"deleting":false}}}},{"cell_type":"code","source":"raw_train_data = pd.read_csv('/kaggle/input/ai-serving-grid-stability/train.csv', index_col='id', parse_dates=['Datum_Uhrzeit_CET'])\nraw_test_data = pd.read_csv('/kaggle/input/ai-serving-grid-stability/test.csv', index_col='id', parse_dates=['Datum_Uhrzeit_CET'])","metadata":{"execution":{"iopub.status.busy":"2024-01-25T09:09:27.279030Z","iopub.execute_input":"2024-01-25T09:09:27.279535Z","iopub.status.idle":"2024-01-25T09:09:58.121317Z","shell.execute_reply.started":"2024-01-25T09:09:27.279502Z","shell.execute_reply":"2024-01-25T09:09:58.119173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Explore the data","metadata":{"nteract":{"transient":{"deleting":false}}}},{"cell_type":"code","source":"# The Dataset contains data from two different control areas. Make sure to separate them when handling the data ;-)\ncontrol_area_1 = raw_train_data[raw_train_data['controlArea']==1].copy()\ncontrol_area_2 = raw_train_data[raw_train_data['controlArea']==2].copy()\n\ncontrol_area_1[['correctedDemand', 'aFRRactivation', 'aFRRrequest']][1000:2000].plot()\ncontrol_area_2[['correctedDemand', 'aFRRactivation', 'aFRRrequest']][1000:2000].plot()","metadata":{"jupyter":{"source_hidden":false,"outputs_hidden":false},"nteract":{"transient":{"deleting":false}},"gather":{"logged":1705939892302},"collapsed":false,"execution":{"iopub.status.busy":"2024-01-25T09:10:57.079906Z","iopub.execute_input":"2024-01-25T09:10:57.080271Z","iopub.status.idle":"2024-01-25T09:10:58.171633Z","shell.execute_reply.started":"2024-01-25T09:10:57.080242Z","shell.execute_reply":"2024-01-25T09:10:58.170127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Feature engineering\n#### Business context aFRR\nThe aFRR process is a complex, real-time process and a tool for TSOs to balance supply and demand in the electricity grid. In the Policy on Load Frequency Control and Reserves ENTSO-E published an overview, how the different load frequency control and reserve processes depend on each other, see page 20 in [Policy](https://eepublicdownloads.entsoe.eu/clean-documents/Publications/SOC/safa/1_-_Policy_on_Load-Frequency_Control_and_Reserves.pdf).\n\nThe overview also detailes how features from our data set are related to each other and, furthermore, which features are exchanged with the PICASSO platform.\n\n##### Usage for feature engineering\nThese insights can help us to understand the relationship between feature better and thus, we can use the knowledge to improve our feature engineering which in turn support our models to better predict anomalies.","metadata":{"nteract":{"transient":{"deleting":false}}}},{"cell_type":"markdown","source":"#### Example 1: Calculate Demand and correctedDemand from other features\nUsing the overview presented above, we can derive that the Demand is derived by the LFCInput, aFRRactivation minus correction. The correctedDemand equals the Demand plus correction. ","metadata":{"nteract":{"transient":{"deleting":false}}}},{"cell_type":"code","source":"# from the policy we can derive a realtionshio between Demand, LFCInput and aFRRactivation\ncontrol_area_1['calculated_Demand'] = control_area_1['aFRRactivation'] + control_area_1['LFCInput'] - control_area_1['correction']\ncontrol_area_1['calculated_correctedDemand'] = control_area_1['aFRRactivation'] + control_area_1['LFCInput']\ncontrol_area_2['calculated_Demand'] = control_area_2['aFRRactivation'] + control_area_2['LFCInput'] - control_area_2['correction']\ncontrol_area_2['calculated_correctedDemand'] = control_area_2['aFRRactivation'] + control_area_2['LFCInput']\n\ncontrol_area_1[['Demand', 'calculated_Demand', 'correctedDemand', 'calculated_correctedDemand']][1000:2000].plot()\ncontrol_area_2[['Demand', 'calculated_Demand', 'correctedDemand', 'calculated_correctedDemand']][1000:2000].plot()","metadata":{"jupyter":{"source_hidden":false,"outputs_hidden":false},"nteract":{"transient":{"deleting":false}},"gather":{"logged":1706031209624},"collapsed":false,"execution":{"iopub.status.busy":"2024-01-25T09:11:02.290432Z","iopub.execute_input":"2024-01-25T09:11:02.290846Z","iopub.status.idle":"2024-01-25T09:11:03.121129Z","shell.execute_reply.started":"2024-01-25T09:11:02.290813Z","shell.execute_reply":"2024-01-25T09:11:03.119870Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Example 2: Relationship between aFRRrequest and aFRRactivation\nDue to physical restrictions of power plants, the regulatory framework on aFRR allows balancing service providers (BSP) to react on a new setpoint in a specified timeframe. E.g., in Germany BSPs need to fully activate a requested setpoint after 5 minutes for aFRR. We can use this knowledge to better predict the aFRRactivation from the aFRRrequest.","metadata":{"nteract":{"transient":{"deleting":false}}}},{"cell_type":"code","source":"# smooth aFRRrequest with window size of 5 minutes as a first approximation\ncontrol_area_1['smooth_aFRRrequest'] = control_area_1['aFRRrequest'].rolling(75).mean()\ncontrol_area_2['smooth_aFRRrequest'] = control_area_2['aFRRrequest'].rolling(75).mean()\n\ncontrol_area_1[['aFRRrequest', 'aFRRactivation', 'smooth_aFRRrequest']][1000:2000].plot()\ncontrol_area_2[['aFRRrequest', 'aFRRactivation', 'smooth_aFRRrequest']][1000:2000].plot()","metadata":{"jupyter":{"source_hidden":false,"outputs_hidden":false},"nteract":{"transient":{"deleting":false}},"gather":{"logged":1705940060424},"collapsed":false,"execution":{"iopub.status.busy":"2024-01-25T09:13:57.254683Z","iopub.execute_input":"2024-01-25T09:13:57.255045Z","iopub.status.idle":"2024-01-25T09:13:58.401268Z","shell.execute_reply.started":"2024-01-25T09:13:57.255017Z","shell.execute_reply":"2024-01-25T09:13:58.399935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that our window size of 5 minutes is already a good approximation for control area 1, while for control area 2, the aFRRactivation seems to be adjusting quicker to the aFRRrequest. This behavior could have different explanations. First, it might be that in control area 2 a different regulation applies, second, the power plant portfolio in this control area just allows physically for quicker responses than in control area 1, or third, the calculation of the aFRRrequest (via the LFC controller) is different and has already a smoothing factor included. \n\nAs a next step, we will try to optimize our approximation reflecting the differences between the control areas.","metadata":{"nteract":{"transient":{"deleting":false}}}},{"cell_type":"code","source":"# Optimize power change period (PCP) and response time\n# Approach: Simple brute force algorithm\n# Simplified assumption: aFRR by BSPs is activated following a constant ramp up / down time\ndef optimize_window_size_smoothed_aFRRrequest(train_data, window_size_min, window_size_max, response_time_min=0, response_time_max=0, objective_quantile_p=0.05 ):\n    \n    # set index to optimize on\n    indexes_opt = train_data.index\n    \n    # initiate variables\n    objective_opt = np.inf\n    response_time_opt = None\n    window_size_opt = None\n    objectives = list()\n    response_times = list()\n    window_sizes = list()\n    # Optimize window size and response time for rolling mean using simple brute force approach\n    # Loop through range of possible reaction times\n    for response_time in range(response_time_min, response_time_max + 1):\n        # Loop through range of possible window sizes\n        for window_size in range(window_size_min, window_size_max + 1):\n            # Calculate simulated aFRR activation using rolling mean with current window size and response time\n            aFRRactivation_sim = train_data['aFRRrequest'].rolling(window_size - response_time, min_periods=1).mean().shift(response_time)\n            # Calculate differences of simulated and actual aFRR activation\n            diffs = (aFRRactivation_sim.loc[indexes_opt] - train_data.loc[indexes_opt, 'aFRRactivation'])\n            # Calculate objective as interquantile distance\n            objective = diffs.quantile(1.0 - objective_quantile_p) - diffs.quantile(objective_quantile_p)\n            objectives.append(objective)\n            response_times.append(response_time)\n            window_sizes.append(window_size)\n         \n            # Save best fitting solution obtained so far\n            if (objective < objective_opt):\n                objective_opt = objective\n                response_time_opt = response_time\n                window_size_opt = window_size\n    opt_results = pd.DataFrame(data=objectives, index=window_sizes, columns=['objective'])\n    return (window_size_opt, response_time_opt, objective_opt), opt_results\n\n# Run algorithm for both control areas\nopt_tuple_control_area_1, opt_df_contorl_area_1 = optimize_window_size_smoothed_aFRRrequest(control_area_1, 10, 100)\nprint('control area 1: window_size_opt, response_time, objective_opt: ', opt_tuple_control_area_1)\nopt_tuple_control_area_2, opt_df_contorl_area_2 = optimize_window_size_smoothed_aFRRrequest(control_area_2, 10, 100)\nprint('control area 2: window_size_opt, response_time, objective_opt: ', opt_tuple_control_area_2)","metadata":{"jupyter":{"source_hidden":false,"outputs_hidden":false},"nteract":{"transient":{"deleting":false}},"gather":{"logged":1706084010212},"collapsed":false,"execution":{"iopub.status.busy":"2024-01-25T09:12:05.785920Z","iopub.execute_input":"2024-01-25T09:12:05.786385Z","iopub.status.idle":"2024-01-25T09:12:43.162654Z","shell.execute_reply.started":"2024-01-25T09:12:05.786350Z","shell.execute_reply":"2024-01-25T09:12:43.161768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can plot the optimized approximation:","metadata":{"nteract":{"transient":{"deleting":false}}}},{"cell_type":"code","source":"# smooth aFRRrequest with window size of 5 minutes as a first approximation\ncontrol_area_1['smooth_aFRRrequest'] = control_area_1['aFRRrequest'].rolling(81).mean()\ncontrol_area_2['smooth_aFRRrequest'] = control_area_2['aFRRrequest'].rolling(16).mean()\n\ncontrol_area_1[['aFRRrequest', 'aFRRactivation', 'smooth_aFRRrequest']][1000:2000].plot()\ncontrol_area_2[['aFRRrequest', 'aFRRactivation', 'smooth_aFRRrequest']][1000:2000].plot()","metadata":{"jupyter":{"source_hidden":false,"outputs_hidden":false},"nteract":{"transient":{"deleting":false}},"gather":{"logged":1706084010927},"collapsed":false,"execution":{"iopub.status.busy":"2024-01-25T09:13:19.841182Z","iopub.execute_input":"2024-01-25T09:13:19.841556Z","iopub.status.idle":"2024-01-25T09:13:20.509519Z","shell.execute_reply.started":"2024-01-25T09:13:19.841525Z","shell.execute_reply":"2024-01-25T09:13:20.507705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Next steps: How to leverage engineered features\nThe new engineered features can be used to either add them to the data set on which our model is trained or to create specific rule-based algorithms that enhance our statistical model.","metadata":{"nteract":{"transient":{"deleting":false}}}}]}