{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\ntqdm.pandas()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocessing","metadata":{}},{"cell_type":"code","source":"# load train data\ndf_train = pd.read_csv(\"../input/coleridgeinitiative-show-us-the-data/train.csv\")\nprint(df_train.shape)\ndf_train.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# label encoding\ndf_label = df_train.value_counts('cleaned_label').reset_index()\ndf_label.columns = ['label', 'count']\ndf_label['-count'] = -df_label['count']\ndf_label['label'] = df_label['label'].apply(lambda x: x.strip())\ndf_label = df_label.sort_values(['-count', 'label']).reset_index(drop=True)\ndf_label['target'] = np.arange(len(df_label))\nprint(df_label.shape)\ndf_label.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train['target'] = df_train['cleaned_label'].progress_apply(lambda x: \n    df_label['target'][df_label['label']==x.strip()].values[0]\n)\ndf_train.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# aggregation\ndf_tmp = df_train.groupby('Id')['target'].agg(lambda x: np.array(x)).reset_index()\ndf_tmp.columns = ['Id', 'target']\ndf_train_agg = df_train[df_train['Id'].duplicated()==False].reset_index(drop=True)\ndf_train_agg = pd.merge(df_train_agg['Id'], df_tmp, on='Id')\ndf_train_agg['target'] = df_train_agg['target'].apply(lambda x: x.reshape(-1))\nprint(df_train_agg.shape)\ndf_train_agg.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Metric implementaion","metadata":{}},{"cell_type":"code","source":"def calc_score(y_true, y_pred, beta=0.5):\n    TP = 0\n    FP = 0\n    FN = 0\n    for i in range(len(y_true)):\n        y_true_i = y_true[i]\n        y_pred_i = y_pred[i]\n        for j in range(len(y_true_i)):\n            if y_true_i[j] in y_pred_i:\n                TP += 1\n            else:\n                FN += 1\n        for j in range(len(y_pred_i)):\n            if y_pred_i[j] not in y_true_i:\n                FP += 1\n    F_beta = (1+beta**2)*TP/((1+beta**2)*TP + beta**2*FP + FN)\n    return F_beta","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Let's do math!","metadata":{}},{"cell_type":"markdown","source":"Let $T$ be the number of sample in the train data. Let $N$ be the number of positive labels in the train data. \nThen the average number of positive labels for each sample of the train data can be written as $N/T$.\n","metadata":{}},{"cell_type":"code","source":"T = len(df_train_agg)\nN = len(df_train)\nN_per_T = N/T\nprint(\"N/T: {:.6f}\".format(N_per_T))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The metric of this competition is $F_\\beta$, \n$$ F_{\\beta}(P) = \\frac{(1+\\beta^2)TP}{(1+\\beta^2)TP + \\beta^2FP + FN}, \\tag{1}$$\nwhere $P$ is the prediction and $TP$, $FP$ and $FN$ is the number of true positive, false positive and false negative respectively.  \nLet $n_0$ be the number of target 0 in the train data and let $P_0$ be the prediction that predicts all samples as 0. Then, \n\n$$ F_{\\beta}(P_0) = \\frac{(1+\\beta^2)n_0}{(1+\\beta^2)n_0 + \\beta^2(T-n_0) + (N-n_0)} = \\frac{(1+\\beta^2)n_0}{N+\\beta^2T } \\tag{2}$$\n\nAs same as above, let $n_1$ be the number of target 1 in the train data and let $P_1$ be the prediction that predicts all samples as 1. Then,\n\n$$ F_{\\beta}(P_1) = \\frac{(1+\\beta^2)n_1}{N+\\beta^2T} \\tag{3}$$\n\nMoreover, let $P_{0,1}$ be the prediction that predicts all samples as 0 and 1. Then,\n\n$$ F_{\\beta}(P_{0,1}) = \\frac{(1+\\beta^2)(n_0 + n_1)}{(1+\\beta^2)(n_0+n_1) + \\beta^2(T-n_0-n_1) + (2N-n_0-n_1)} = \\frac{(1+\\beta^2)(n_0+n_1)}{N+2\\beta^2T} \\tag{4}$$\n\n$F_{\\beta}(P_0)$, $F_{\\beta}(P_1)$ and $F_{\\beta}(P_{0,1}) $ are calcurated in the next cell.","metadata":{}},{"cell_type":"code","source":"# predict all data as 0\npred0 = np.ones([len(df_train_agg), 1])*0\nF_beta0 = calc_score(df_train_agg['target'].values, pred0)\nprint(\"F_beta0 : {:.6f}\".format(F_beta0))\n\n# predict all data as 1\npred1 = np.ones([len(df_train_agg), 1])*1\nF_beta1 = calc_score(df_train_agg['target'].values, pred1)\nprint(\"F_beta1 : {:.6f}\".format(F_beta1))\n\n\n# predict all data as [0,1]\npred01 = np.zeros([len(df_train_agg), 2])\npred01[:,1] = 1\nF_beta01 = calc_score(df_train_agg['target'].values, pred01)\nprint(\"F_beta01: {:.6f}\".format(F_beta01))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the equation (2), (3) and (4) we have, \n\n\n$$ F_{\\beta}(P_0)(N+\\beta^2T) = (1+\\beta^2)n_0 \\tag{5}$$\n\n$$ F_{\\beta}(P_1)(N+\\beta^2T) = (1+\\beta^2)n_1 \\tag{6}$$\n\n$$ F_{\\beta}(P_{0,1})(N+2\\beta^2T) = (1+\\beta^2)(n_0+n_1). \\tag{7}$$\n\nBy equation (5) + (6), we have, \n\n\n$$ (F_{\\beta}(P_0)+F_{\\beta}(P_1))(N+\\beta^2T) = (1+\\beta^2)(n_0+n_1) \\tag{8}$$\n\nFrom equation (7) and (8), we have,\n\n$$ (F_{\\beta}(P_0)+F_{\\beta}(P_1))(N+\\beta^2T) = F_{\\beta}(P_{0,1})(N+2\\beta^2T) \\tag{9}$$\n\nBy transforming (9), we have,\n\n\n$$ \\frac{N}{T} = \\beta^2\\frac{2F_{\\beta}(P_{0,1})-F_{\\beta}(P_0)-F_{\\beta}(P_1)}{F_{\\beta}(P_0)+F_{\\beta}(P_1)-F_{\\beta}(P_{0,1})} \\tag{10}$$\n\nOK. Using equation (10), we can calcurate $N/T$ with $F_{\\beta}(P_0)$, $F_{\\beta}(P_1)$ and $F_{\\beta}(P_{0,1})$. Let's check it.\n","metadata":{}},{"cell_type":"code","source":"def calc_N_per_T(F_beta0, F_beta1, F_beta01, beta=0.5):\n    return (beta**2)*(2*F_beta01-F_beta0-F_beta1)/(F_beta0+F_beta1-F_beta01)\n\ntmp = calc_N_per_T(F_beta0, F_beta1, F_beta01)\nprint('estimated N/T: {:.6f}'.format(tmp))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looks Fine! We can do same calcuration on the test data by submitting $P_0$, $P_1$ and $P_{0,1}$. I submitted them and got results below. ","metadata":{}},{"cell_type":"code","source":"F_beta0_test = 0.022\nF_beta1_test = 0.014\nF_beta01_test = 0.026","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's calcurate $N/T$ of the test data!","metadata":{}},{"cell_type":"code","source":"tmp = calc_N_per_T(F_beta0_test, F_beta1_test, F_beta01_test)\nprint('estimated N/T of the test data: {:.6f}'.format(tmp))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we have the average number of positive labels for each sample of the test data. But the result is surprising. It's much less than that of the train data. Is it correct? Or is there something wrong with my equation or implementation? If you find any problems with my code, please let me know.","metadata":{}},{"cell_type":"markdown","source":"# make submission with prediction of all [0, 1]","metadata":{}},{"cell_type":"code","source":"# make submisttion with P_0,1\ndf_sub = pd.read_csv(\"../input/coleridgeinitiative-show-us-the-data/sample_submission.csv\")\ndf_sub['PredictionString'] = \"{}|{}\".format(df_label['label'][0], df_label['label'][1])\ndf_sub.to_csv(\"submission.csv\", index=None)\ndf_sub.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}