{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"**...On Training and Public Leaderboard**\n\n...kind of a low-hanging fruit, but since I didn't find it anywhere else here we go.\n\nInteresting that the all zeroes benchmark gives such different results on training and public leaderboard data.  I'd like to compare the rate of positive revenue cases and the conditional mean revenue of those, between training and public leaderboard data. On the public leaderboard data these have to be computed indirectly through rmse like so:\n\n- The target is nearly dichotomous having values 0 and values around the conditional mean revenue of clients with positive revenues.\n- Say we predict a fixed value x as revenue for all clients, then rmse is equal to y=sqrt(m/n*a^2-2*m/n*)a*x+x^2), where a is the conditional mean revenue and m is the number of clients with positive revenues.\n- To compute m and a we need two pairs of values (x,y), for the public leaderboard e.g. we are already given the pair (0 , 1.7804) for the all zeroes benchmark.\n\nI'll check validity of this approach on the training data, and then compute rate and mean implicitly on the public leaderboard data."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# thx for import: kernel by Julián Peller\n\nimport os\nimport json\nimport numpy as np\nimport pandas as pd\nfrom pandas.io.json import json_normalize\n\ndef load_df(csv_path='../input/train.csv', nrows=None):\n    JSON_COLUMNS = ['device', 'geoNetwork', 'totals', 'trafficSource']\n    \n    df = pd.read_csv(csv_path, \n                     converters={column: json.loads for column in JSON_COLUMNS}, \n                     dtype={'fullVisitorId': 'str'}, # Important!!\n                     nrows=nrows)\n    \n    for column in JSON_COLUMNS:\n        column_as_df = json_normalize(df[column])\n        column_as_df.columns = [f\"{column}.{subcolumn}\" for subcolumn in column_as_df.columns]\n        df = df.drop(column, axis=1).merge(column_as_df, right_index=True, left_index=True)\n    print(f\"Loaded {os.path.basename(csv_path)}. Shape: {df.shape}\")\n    return df\n\ndf_train = load_df()\ndf_test = load_df(\"../input/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"382dfa1a9df8fd6143a44ee452e2b299a5f0a379"},"cell_type":"code","source":"# compute rate and conditional revenue on test data:\ndf_train['totals.transactionRevenue']=df_train['totals.transactionRevenue'].astype(float)\ntrain=pd.DataFrame()\ntrain['target']=np.log1p(df_train.groupby('fullVisitorId')['totals.transactionRevenue'].sum())\nprint('Explicit')\nprint('Cases: '+str((train.target>0).mean())+', Conditional Mean: '+str(train[train.target>0].target.mean()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"98151c1e7c91f0f95e6a54993e756ecd3f7b64a2"},"cell_type":"code","source":"# compute rate and conditional mean through rmse:\nn=len(train)\ny0=np.round(np.sqrt(((train.target-0)**2).mean()),4)\ny1=np.round(np.sqrt(((train.target-1)**2).mean()),4)\na=2*y0**2/(y0**2-y1**2+1)\nm=n*y0**2/a**2\nprint('Implicit')\nprint('Cases: '+str(m/n)+', Conditional Mean: '+str(a))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"df27535b3ab1819b3d4f9a57adbafe3388d3928f"},"cell_type":"markdown","source":"Appears to be fit for purpose... let's turn to the public leaderboard data.\nWe already have the all-zeroes benchmark. Let's check the rmse of another constant and compute implicit figures."},{"metadata":{"trusted":true,"_uuid":"980410f0543d77ca8488920277fc3188c43f0dc4"},"cell_type":"code","source":"# get leaderboard rmse for x=1\n# submission=pd.read_csv('../input/sample_submission.csv',index_col=[0])\n# submission['PredictedLogRevenue']=1\n# submission.to_csv('submission1.csv')\n# this actually returns a leaderboard rmse of 1.9529","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ce8101e3037084e611850b128df928f74bfb0de5"},"cell_type":"code","source":"n=0.3*len(df_test)\ny0=1.7804\ny1=1.9529\na=2*y0**2/(y0**2-y1**2+1)\nm=n*y0**2/a**2\nprint('Implicit on Public Leaderboard')\nprint('Cases: '+str(m/n)+', Conditional Mean: '+str(a))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7304552126ece24e61bf1cc9a20ba7876fbc9a14"},"cell_type":"markdown","source":"Ok, maybe not that surprising. The reason for the difference in the performance of the all zeroes benchmark is the diminuished revenue rate. The product of both should be close to the best performance by a constant prediction. Maybe one could check the mean prediction and shift accordingly to improve a model... depends on  the type of public/private split of the test data."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}