Coleridge Initiative : Evaluation Process
\n\n####There has been discussions regarding the evaluation process. Seems like there were some issues with LB or the metric because of which the competition got 0.992 score on the LB within 12 hours of launch. I followed the discussions and concluded that initially the metric wasn't considering recall. But now the issue has been fixed and they are using FBeta (0.5) metric. In this notebook, I have demonstrate the evaluation metric in detail.
\n\n### ⚠️ Before moving forward, I recommend you to look into the Competition details and data, if not already. You can follow this notebook - [Starter 🌟: Competition, Data, EDA and Modelling🚀](https://www.kaggle.com/pashupatigupta/starter-competition-data-eda-and-modelling)","metadata":{}},{"cell_type":"markdown","source":"#### A brief overview of competition before going ahead\n\n> The objective of the competition is to identify the mention of datasets within scientific publications. Your predictions will be short excerpts from the publications that appear to note a dataset.\n\nSo, our predictions are dataset names used in a particular publication. Now, a particular publication can use multiple datasets so we will be predicting all the datasets use in the publication. Multiple predictions are delineated with a pipe (|) character in the submission file.\n\nSo, our prediction will look like - ABC|PQR|XYZ\n\nAnd so will look the ground truth - PQR|BNM|XYZ|DEF\n\nNow how to evaluate the such prediction against the given ground truth? The competion says using Jaccard based FBeta score. What's this? Let's find out. ","metadata":{}},{"cell_type":"code","source":"import os\nimport re\nimport json\nimport pandas as pd\nimport numpy as np\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nsns.set(style=\"whitegrid\")\nfrom wordcloud import WordCloud, STOPWORDS\nimport plotly.graph_objects as go\n\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Reading the training dataset","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')\ndf.sample(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, the content of every publication is provided in a json file. We'll read the text of a publication from the json file and put it in the train dataframe","metadata":{}},{"cell_type":"code","source":"def get_text(filename, test=False):\n if test:\n df = pd.read_json('../input/coleridgeinitiative-show-us-the-data/test/{}.json'.format(filename))\n else:\n df = pd.read_json('../input/coleridgeinitiative-show-us-the-data/train/{}.json'.format(filename))\n text = \" \".join(list(df['text']))\n return text","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['text'] = df['Id'].apply(get_text)\ndf.sample(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let's split this datafeame into development set and validation set.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\ndev, val = train_test_split(df, test_size=0.1, random_state=42)\nprint(\"Development Shape : \", dev.shape)\nprint(\"Validation Shape : \", val.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Cool! Here \"cleaned_label\" is our target variable. (As per my understanding)\n\n> The \"cleaned_label\" column in validation set will work as ground truth. We'll make prediction on valication set. And then, we'll understand the evaluation metric. (Since we'll be having both predictions and ground truth)","metadata":{}},{"cell_type":"markdown","source":"### Simple Baseline Model\n\nI'm directly copying the baseline model from my previous notebook. You can look into this [Notebook](https://www.kaggle.com/pashupatigupta/starter-competition-data-eda-and-modelling) if you want the explanation","metadata":{}},{"cell_type":"markdown","source":"> Note that ALL ground truth texts have been cleaned for matching purposes using the following code:\n\nSo we'll be using this function to clean our texts wherever needed.","metadata":{}},{"cell_type":"code","source":"def clean_text(txt):\n return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def baseline_model(dev, val):\n \n print(\"Running...\")\n datasets_titles = [x.lower() for x in set(dev['dataset_title'].unique()).union(set(dev['dataset_label'].unique()))]\n\n print(\"Preparing Validation set...\")\n val_id = val['Id'].unique()\n ids = []\n texts = []\n gt = []\n\n for ed in val_id:\n tdf = val[val['Id'] == ed]\n gt_label = \"|\".join(tdf['cleaned_label'].tolist())\n text = tdf['text'].tolist()[0]\n ids.append(ed)\n gt.append(gt_label)\n texts.append(text)\n\n pval = pd.DataFrame({'Id':ids, 'text':texts, 'ground_truth':gt})\n\n #print(pval.shape)\n\n print(\"Generating predictions...\")\n labels = []\n for index in pval['Id']:\n publication_text = pval[pval['Id'] == index].text.str.cat(sep='\\n').lower()\n #print(publication_text)\n label = []\n for dataset_title in datasets_titles:\n if dataset_title in publication_text:\n label.append(clean_text(dataset_title))\n labels.append('|'.join(label))\n\n pval['prediction'] = labels\n print(\"Done!\")\n \n return pval","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"output = baseline_model(dev, val)\noutput.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"### Making another copy of predictions for future use\nsub_df = baseline_model(dev, val)","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Superb! We have got the predictions and ground truths. Let's understand the Evaluation metric which is Jaccard similarity based FBeta (0.5). \n\nBefore this we'll first understand Jaccard Similarity and FBeta separately. The steps will be like -\n- Jaccard Similarity\n- FBeta Score\n- Jaccard similarity based FBeta score","metadata":{}},{"cell_type":"markdown","source":"##1. Jaccard Similarity
\n\nDefinition - The Jaccard similarity measures similarity between finite sample sets, and is defined as the size of the intersection divided by the size of the union of the sample sets.\n\n\n\nThe definition is pretty simple and we can apply in any case if we have defined sets. But How it is applied in our case?\n\n> So, in our case we have sentences (collection of words) ans we can split a sentence on white space to make a python set. Then we can apply jaccard similarity. Simple! \n\nLet's impliment it now!","metadata":{}},{"cell_type":"code","source":"def jaccard(str1, str2): \n a = set(str1.lower().split()) \n b = set(str2.lower().split())\n c = a.intersection(b)\n return float(len(c)) / (len(a) + len(b) - len(c))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check it on some examples -","metadata":{}},{"cell_type":"markdown","source":"#### 1.1 On Similar sentences","metadata":{}},{"cell_type":"code","source":"jaccard(\"The cat is on the mat\", \"The cat is on the table\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 1.2 On exact same sentence","metadata":{}},{"cell_type":"code","source":"jaccard(\"This is a pen\", \"This is a pen\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 1.3 On totally different sentence","metadata":{}},{"cell_type":"code","source":"jaccard(\"I am going home\", \"India is a beautiful country\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Cool, we can see jaccard is working fine.","metadata":{}},{"cell_type":"markdown","source":"##2. FBeta Score
","metadata":{}},{"cell_type":"markdown","source":"> Assuming that you are already familiar with true positive, true, negative, false positive, false negative.\n\n#### **Definition** - The F-score is a measure of a model's performance. It is calculated from the precision and recall of the test, where the precision is the number of true positive results divided by the number of all positive results, including those not identified correctly, and the recall is the number of true positive results divided by the number of all samples that should have been identified as positive.\n\nBelow is the formula for Precision and Recall - \n\n\n\nAnd below is the formula for F-Score\n\n\n\nNote : F-Score is often referred as F1-Score. They are the same.","metadata":{}},{"cell_type":"markdown","source":"#### Notice that the definition above is F-Score not FBeta score. Now, Let's look into FBeta\n\n**FBeta Score** : It a F-Score that uses a positive real factor β, where β is chosen such that recall is considered β times as important as precision. The formula for FBeta is give as - \n\n\n\nLet's implement it!!","metadata":{}},{"cell_type":"code","source":"def get_precision_recall(tp, fp, fn):\n precision = tp / (tp+fp)\n recall = tp / (tp + fn)\n return precision, recall\n\ndef fbeta_score(precision, recall, beta):\n fbeta = (1+(beta*beta))*((precision*recall)/( (beta*beta*precision) + recall))\n return fbeta","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check it on some examples (We are passing beta = 0.5 because this competition uses beta = 0.5)","metadata":{}},{"cell_type":"markdown","source":"#### 2.1 When precision equals to recall","metadata":{}},{"cell_type":"code","source":"fbeta_score(0.8, 0.8, 0.5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 2.2 When precision greater than recall","metadata":{}},{"cell_type":"code","source":"fbeta_score(0.8, 0.5, 0.5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 2.3 When precision less than recall","metadata":{}},{"cell_type":"code","source":"fbeta_score(0.5, 0.8, 0.5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well well well! We can see that beta=0.5 give more importance to precision. So this competition's evaluation is baised towards precision.","metadata":{}},{"cell_type":"markdown","source":"##3. Jaccard Similarity based FBeta Score
","metadata":{}},{"cell_type":"markdown","source":"### Now, the main question is \"How FBeta Score is calculated in our case?\"\n\nAnswer is pretty simple - All we need is true positives, false positives and false negatives. And the compition provides the rules to find these.\n\n#### Rules mentioned in the Evaluation section of the competion ","metadata":{}},{"cell_type":"markdown","source":"For each publication's set of predictions, a token-based Jaccard score is calculated for each potential prediction / ground truth pair. The prediction with the highest score for a given ground truth is matched with that ground truth.\n\n- Predicted strings for each publication are sorted alphabetically and processed in that order. Any scoring ties are resolved on the basis of that sort.\n- Any matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are counted as true positives (TP), the remainder as false positives (FP).\n- Any unmatched predictions are counted as false positives (FP).\n- Any ground truths with no nearest predictions are counted as false negatives (FN).\n\nAll TP, FP and FN across all samples are used to calculate a final micro F0.5 score. (Note that a micro F score does precisely this, creating one pool of TP, FP and FN that is used to calculate a score for the entire set of predictions.)","metadata":{}},{"cell_type":"markdown","source":"Disclaimer : This implementation is solely based on my understanding. If yours understanding is different let me know - we can discuss. This implementation is NOT official.
","metadata":{}},{"cell_type":"code","source":"def coleridge_initiative_jaccard(ground_truth, prediction, verbose=True):\n gts = ground_truth.split('|')\n pds = sorted(prediction.split('|'))\n if verbose:\n print(\"Ground truth : \" , gts)\n print(\"Prediction : \", pds)\n \n js_scores = []\n cf_matrix = []\n \n #### Counting True Positives (TP) and False Positives (FP)\n \n for pd in pds:\n score = -1\n for gt in gts:\n js = jaccard(pd, gt)\n if js > score:\n score = js\n if score >= 0.5:\n js_scores.append(score)\n cf_matrix.append(\"TP\")\n else:\n js_scores.append(score)\n cf_matrix.append(\"FP\")\n \n \n #### Counting False Negatives (FN)\n \n for gt in gts:\n score = -1\n for pd in pds:\n js = jaccard(gt, pd)\n if js > score:\n score = js\n if score == 0:\n js_scores.append(score)\n cf_matrix.append(\"FN\")\n \n return js_scores, \" \".join(cf_matrix)\n ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check the function","metadata":{}},{"cell_type":"code","source":"coleridge_initiative_jaccard(\"this data|that dataset|xyz\", \"which data|no dataset|that dataset\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's apply it on our dataset","metadata":{}},{"cell_type":"code","source":"output['evaluation'] = output.apply(lambda x: coleridge_initiative_jaccard(x['ground_truth'], x['prediction'], verbose=False), axis=1)\noutput['js_scores'] = output['evaluation'].apply(lambda x : x[0])\noutput['pred_type'] = output['evaluation'].apply(lambda x : x[1])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"output.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let's write a function to count TP, FP, FN","metadata":{}},{"cell_type":"code","source":"def get_count_tp_fp_fn(prediction, verbose=True):\n preds = prediction.split(\" \")\n if verbose:\n print(preds)\n tpc = 0\n fpc = 0\n fnc = 0\n for pred in preds:\n if pred == \"TP\":\n tpc = tpc + 1\n elif pred == \"FP\":\n fpc = fpc + 1\n elif pred == \"FN\":\n fnc = fnc + 1\n return [tpc, fpc, fnc]\n\ndef make_col_tp_fp_fn(df, col):\n df['TP'] = df[col].apply(lambda x : x[0])\n df['FP'] = df[col].apply(lambda x : x[1])\n df['FN'] = df[col].apply(lambda x : x[2])\n return df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check this function","metadata":{}},{"cell_type":"code","source":"get_count_tp_fp_fn(\"TP TP FP FN\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's apply it on our dataset","metadata":{}},{"cell_type":"code","source":"output['tp_fp_fn'] = output['pred_type'].apply(lambda x : get_count_tp_fp_fn(x, verbose=False))\noutput = make_col_tp_fp_fn(output, 'tp_fp_fn')\noutput.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tp = sum(output['TP'])\nfp = sum(output['FP'])\nfn = sum(output['FN'])\n\nprint(\"True Positives (TP) : \", tp)\nprint(\"False Positives (FP) : \", fp)\nprint(\"False Negatives (FN) : \", fn)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Finally we have got the TP, FP, FN to calculate FBeta score !!","metadata":{}},{"cell_type":"code","source":"precision, recall = get_precision_recall(tp, fp, fn)\nprint(\"Precision : \", precision)\nprint(\"Recall : \", recall)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fbeta = fbeta_score(precision, recall, 0.5)\nprint(\"FBeta Score : \", fbeta)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Yay! We got the metric value.\n\nThis is the overall process of evaluation for this competition.\n\n####I have made this Notebook public so that fellow kagglers can understand the evaluation process better. Thanks for viewing this notebook. If you found this helpful consider UPVOTING it. Also, please correct me if you think I'm wrong anywhere.
","metadata":{}},{"cell_type":"markdown","source":"## EDIT 1 (31/03/2021)\n\nAdding this score_df_coleridge_initiative function to score your predictions dataframe using a single function. You can also use this as objective function / loss function while model building.","metadata":{}},{"cell_type":"code","source":"def score_df_coleridge_initiative(output, gt_col, pred_col, beta=0.5, verbose=True):\n \n '''\n This function will calculate the FBeta score for Coleridge Initiative competition \n if given appropriate arguments\n \n Arguments - \n output - Your submission dataframe that has both ground truth and prediction columns.\n gt_col - This is the column name of ground truth column.\n pred_col - This is the column name of predictions column.\n beta - Beta value to calculate FBeta score.\n \n Returns - \n This function will return the FBeta (beta=0.5) score.\n \n ## Set verbose = True to print logs \n '''\n \n ### Jaccard Similarity\n output['evaluation'] = output.apply(lambda x: coleridge_initiative_jaccard(x[gt_col], x[pred_col], verbose=False), axis=1)\n output['js_scores'] = output['evaluation'].apply(lambda x : x[0])\n output['pred_type'] = output['evaluation'].apply(lambda x : x[1])\n \n ### TP, FP and FN \n output['tp_fp_fn'] = output['pred_type'].apply(lambda x : get_count_tp_fp_fn(x, verbose=False))\n output = make_col_tp_fp_fn(output, 'tp_fp_fn')\n \n tp = sum(output['TP'])\n fp = sum(output['FP'])\n fn = sum(output['FN'])\n precision, recall = get_precision_recall(tp, fp, fn)\n fbeta = fbeta_score(precision, recall, 0.5)\n \n if verbose:\n print(\"True Positives (TP) : \", tp)\n print(\"False Positives (FP) : \", fp)\n print(\"False Negatives (FN) : \", fn)\n print(\"Precision : \", precision)\n print(\"Recall : \", recall)\n print(\"FBeta Score : \", fbeta)\n display(output.head())\n\n return fbeta","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"score_df_coleridge_initiative(sub_df, \"ground_truth\", \"prediction\", beta=0.5, verbose=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}