{"cells":[{"metadata":{},"cell_type":"markdown","source":"This notebook explains my understanding of how the F0.5 metric is computed in this competition. I have a post in the discussion forum - https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230457 - to confirm my understanding.\n\nThere is a class at the end of this notebook which encapsulates the entire evaluation logic.\n\nPlease feel free to share your thoughts and questions.\n\nReference: https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation"},{"metadata":{},"cell_type":"markdown","source":"## Imports and utils"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import re\nimport numpy as np\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Utils\ndef jaccard(str1, str2): \n    a = set(str1.lower().split()) \n    b = set(str2.lower().split())\n    c = a.intersection(b)\n    return float(len(c)) / (len(a) + len(b) - len(c))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Util for displaying (ground-truth) - predictions matrix as DataFrame\ndef display_sample_matrix(np_mat):\n    return pd.DataFrame(np_mat, columns=y_pred_labels, index=y_true_labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Sample ground-truth and prediction\nLet's use this sample ground-truth and predictions to walk through the evaluation steps"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# '|'.join(train_df[train_df.Id == 'ecc1b24e-35b3-4b7e-8c54-c9d42bef16a9'].cleaned_label.values)\ny_true_str = ('baccalaureate and beyond longitudinal study|baccalaureate and beyond|beginning postsecondary student|education longitudinal study|national education longitudinal study')\n\ny_pred_str = 'postsecondary student|education longitudinal study|xyz'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Split label-strings into labels"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_true_labels = y_true_str.split('|')\ny_pred_labels = sorted(y_pred_str.split('|'))\ny_pred_labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Compute Jaccard similarity for each prediction & ground-truth pair"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get Jaccard-matrix with ground-truths as rows and predictions in columns \ngt_p_jaccards = np.array([[jaccard(pred_label, true_label) for pred_label in y_pred_labels] for true_label in y_true_labels])\ndisplay_sample_matrix(gt_p_jaccards)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Get matched predictions matrix\n\nFor each ground-truth label, identify prediction which matches it. A matrix is created to represent matched ground-truths and predictions.\n\nResult matrix characteristics:\n- ground-truths are represented by rows and predictions by columns\n- if there is a match for a ground truth, the corresponding row will be binary array having 1 at the position of the predicted label. Otherwise, it will be an array of 0's\n- if there is a tie in the best predictions' jaccard scores, the one which comes earlier alphabetically would be chosen"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_matched_preds(jacc_scores):\n    true_pred_flags = np.zeros(len(jacc_scores))\n    \n    best_jacc_score = jacc_scores.max()\n    \n    if best_jacc_score < 0.5:\n        '''\n        If there are no matches, return an array of zeros indicating no match\n        Note: A prediction / ground truth pair is considered a match if Jaccard score is less than 0.5\n        '''\n        return true_pred_flags\n    \n    # If there is a match, identify position of best prediction\n    best_score_positions = np.nonzero(jacc_scores == best_jacc_score)[0]\n    best_score_pos = best_score_positions[0]  # Note: requires columns to be sorted alphabetically\n    true_pred_flags[best_score_pos] = 1\n    return true_pred_flags\n    \nmatched_preds_mt = np.apply_along_axis(get_matched_preds, 1, gt_p_jaccards)\ndisplay_sample_matrix(matched_preds_mt)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Compute F-Beta\nReference: https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation\n\nI have a post in the discussion forum - https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230457 - to confirm my understanding for the formulae.\n\n**Note:**\n- Any matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are counted as true positives (TP), the remainder as false positives (FP).\n- Any unmatched predictions are counted as false positives (FP).\n- Any ground truths with no nearest predictions are counted as false negatives (FN).\n\nAll TP, FP and FN across all samples are used to calculate a final micro F0.5 score. (Note that a micro F score does precisely this, creating one pool of TP, FP and FN that is used to calculate a score for the entire set of predictions.)"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_tp = matched_preds_mt.sum()  # No. of matched predictions and ground-truths\nsample_fp = (~matched_preds_mt.any(axis=0)).sum()  # No. of predictions (columns) without any matches\nsample_fn = (~matched_preds_mt.any(axis=1)).sum()  # No. of ground-truths (rows) without any matches","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Combine TP, FP and FN of all samples and compute F0.5 score"},{"metadata":{"trusted":true},"cell_type":"code","source":"tp = sum([sample_tp])\nfp = sum([sample_fp])\nfn = sum([sample_fn])\n\nprecision = tp / (tp + fp)\nrecall = tp / (tp + fn)\n\nbeta = 0.5\nfbeta = (1 + beta**2) * precision * recall / ((beta**2 * precision) + recall)\nfbeta","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ColeridgeEvaluation class\nThis class encapsulates all the above evaluation logic"},{"metadata":{"trusted":true},"cell_type":"code","source":"class ColeridgeEvaluation:\n    @classmethod\n    def evaluate_samples(cls, y_true_list, y_pred_list):\n        tp_fp_fn_list = []\n        \n        # Compute TP, FP, FN for each sample\n        for sample_y_true, sample_y_pred in zip(y_true_list, y_pred_list):\n            sample_tp_fp_fn = cls.evaluate_sample(sample_y_true, sample_y_pred)\n            tp_fp_fn_list.append(sample_tp_fp_fn)\n        \n        # Compute F0.5\n        tp, fp, fn = np.array(tp_fp_fn_list).sum(axis=0)\n        sample_fbeta = cls.compute_fbeta(tp, fp, fn)\n        return sample_fbeta\n    \n    @classmethod\n    def evaluate_sample(cls, y_true_str, y_pred_str):\n        # Split label-strings into labels\n        y_true_labels = y_true_str.split('|')\n        y_pred_labels = sorted(y_pred_str.split('|'))\n        \n        # -- Compute Jaccard similarity for each prediction & ground-truth pair --\n        # Get Jaccard-matrix with ground-truths as rows and predictions in columns \n        gt_p_jaccards = np.array([[cls.jaccard(pred_label, true_label) for pred_label in y_pred_labels] \n                                      for true_label in y_true_labels])\n\n        # Binarize matched-predictions\n        matched_preds_mt = np.apply_along_axis(cls._get_matched_preds, 1, gt_p_jaccards)\n        \n        # Return sample's tp, fp, fn\n        return cls.compute_sample_cf_metrics(matched_preds_mt)\n    \n    @classmethod\n    def jaccard(cls, str1, str2): \n        a = set(str1.lower().split()) \n        b = set(str2.lower().split())\n        c = a.intersection(b)\n        return float(len(c)) / (len(a) + len(b) - len(c))\n    \n    @classmethod\n    def _get_matched_preds(cls, jacc_scores):\n        true_pred_flags = np.zeros(len(jacc_scores))\n\n        best_jacc_score = jacc_scores.max()\n\n        if best_jacc_score < 0.5:\n            '''\n            If there no matches, return an array of zeros indicating no match\n            Note: A prediction / ground truth pair is considered a match if Jaccard score is less than 0.5\n            '''\n            return true_pred_flags\n\n        # If there is a match, identify position of the best prediction\n        best_score_positions = np.nonzero(jacc_scores == best_jacc_score)[0]\n        best_score_pos = best_score_positions[0]  # Get first position of best-score predictions\n        true_pred_flags[best_score_pos] = 1\n        return true_pred_flags\n    \n    @classmethod\n    def compute_sample_cf_metrics(cls, matched_preds_mt, beta=0.5):\n        sample_tp = matched_preds_mt.sum()  # No. of matched predictions and ground-truths\n        sample_fp = (~matched_preds_mt.any(axis=0)).sum()  # No. of predictions (columns) without any matches\n        sample_fn = (~matched_preds_mt.any(axis=1)).sum()  # No. of ground-truths (rows) without any matches\n        return sample_tp, sample_fp, sample_fn\n        \n    @classmethod\n    def compute_fbeta(cls, tp, fp, fn, beta=0.5):\n        precision = tp / (tp + fp)\n        recall = tp / (tp + fn)\n\n        fbeta = (1 + beta**2) * precision * recall / ((beta**2 * precision) + recall)\n        return fbeta","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ColeridgeEvaluation.evaluate_samples([y_true_str], [y_pred_str])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}