{"cells":[{"metadata":{},"cell_type":"markdown","source":"# How Big is your Model Error? A comparison between RMSLE and MAPE\n\nIn this competition, the evaluation metric used (Root Mean Square Logarithmic Error) doesn't provide a good intuition about the magnitude of the error of the models. In this kernel, we will make a very simple experiment to put RMSLE in relation to MAPE (Mean Absolute Percentage Error).\n\n### The experiment\nFor each value of the target variable (meter_reading), we will create a new value picked from a random normal distribution, centered in the original value. We will call this new set of values \"y_noise\". This will simulate the predictions of an unbiased model for our target variable. We will repeat this procedure many times, increasing the variance of the normal distribution at each iteration, and saving the values of RMSLE and MAPE obtained. This sampling procedure will allow us to create a map of the relationship between RMSLE and MAPE for our dataset.\n\n#### * Note about MAPE\nThe target variable in this dataset has a lot of zeros. The traditional implementation of MAPE can't be used when the ground truth has zeros, because you get an x/0 indetermination. However, there is a known alternative to solve this problem, consisting in replace each actual value of the series in the original formula by the average of all actual values of that series. This is the same as dividing the sum of absolute differences by the sum of actual values, and is sometimes referred to as WAPE (Weighted Absolute Percentage Error)."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom sklearn.metrics import mean_squared_log_error","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Metrics definition"},{"metadata":{"trusted":true},"cell_type":"code","source":"def RMSLE(y_true, y_pred):\n    \n    if len(y_pred[y_pred<0])>0:\n        y_pred = np.clip(y_pred, 0, None)\n    \n    return np.sqrt(mean_squared_log_error(y_true, y_pred))\n\n\n\n\ndef MAPE(y_true, y_pred):\n    \n    if len(y_true[y_true==0])>0: # Use WAPE if there are zeros\n        return sum(np.abs(y_true - y_pred)) / sum(y_true) * 100\n        \n    return np.mean(np.abs((y_true - y_pred) / y_true)) * 100","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load data"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/ashrae-energy-prediction/train.csv', usecols=['meter_reading'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The experiment"},{"metadata":{"trusted":true},"cell_type":"code","source":"rmsle = list()\nmape = list()\n\ny = df['meter_reading'].values\n\nfor dev in np.arange(0, 1, 0.01):\n    \n    y_noise = np.random.normal(y, dev*y)\n    \n    rmsle.append(RMSLE(y, y_noise))\n    mape.append(MAPE(y, y_noise))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_results = pd.DataFrame([mape, rmsle], index=['MAPE', 'RMSLE']).T\ndf_results.sort_values('MAPE', inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Plot results"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,16))\nplt.plot(df_results['MAPE'].values, df_results['RMSLE'].values)\nplt.xlabel('MAPE %')\nplt.ylabel('RMSLE')\nplt.grid(True, axis='both')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Conclusions\nAt the moment of writing this kernel, the leaderboard top 100 scores are in the range of 1.08 - 1.21 RMSLE. As we can see, this is equivalent to a MAPE in the 40% - 50% range. \n\nThere is noise in the mapping obtained between the two metrics, as expected. The variance of that noise grows with the MAPE. This can be due to the non linearities we introduced in the metrics functions: for the RMSLE, we are clipping to 0 the negative y_noise values, while for the MAPE these values contribute to the error in their full magnitude (check the metrics code at the beginning of this kernel). At each iteration, the number of values clipped in the calculus of RMSLE grows randomly but not monotonically, giving as result the growing irregular pattern between MAPE and RMSLE observed. "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}