Hi all, after going through the Evaluation page and the discussions, I'm still not clear what exactly is the implementation of the metric. So I wrote my own based on my understanding. Please do verify and let me know in the comment if my understanding is right or wrong.
\nTrue_label = [\"covid 19 open research data\", \"high school longitudinal study\"]\nprediction = [\"19 open research data\", \"covid 19 open research dataset\", \"agni\"]\nThese are my assumptions.
\nOne true label can be mapped to only one predicted string. While looping If it finds a first match(Jaccard>=0.5), then the same True label cannot be matched to another prediction which could be very close to it later in the loop .i.e.
\ni.e. True_Label[0] ''covid 19 open research data\" is matched to only Prediction[0] \"covid 19 open research dataset\". So Prediction[0] is TP
Similarly the predictions cannot be matched with True labels which are already matched.
\ni.e. Prediction[1] \"covid 19 open research dataset\" cannot be matched to True_Label[0] ''covid 19 open research data\" since it's already matched with Prediction[0] \"19 open research data\". So Prediction[1] if FP
Prediction[2] \"agni\" does not have any close match so it's FP
True Label[1] \"high school longitudinal study\" does not have a match so it's FN
All the FP, FN, TP are summed-up together from each sample to finally calculate the Precision and Recall and finally the F0.5
overall_TP = 0\noverall_FP = 0\noverall_FN = 0\n\nfor Id in final_prediction:\n print(Id)\n\n # Sort the actuals\n actual = final_prediction[Id]['actual'].copy()\n actual.sort()\n\n # Sort the Predictions\n prediction = final_prediction[Id]['predicted'].copy()\n prediction.sort()\n\n # Initialize the TP, FP, FN for the sample\n sample_TP = 0\n sample_FP = 0\n sample_FN = len(actual) #Assume none of the true label is matched initially\n\n # Loop the ground Truth first\n for actu_index in range(len(actual)):\n if len(prediction) !=0:\n\n # Loop through each of the predictions\n for pred_index in range(len(prediction)):\n\n # Find the Jaccard score.\n jaccard_score = jaccard(actual[actu_index], prediction[pred_index])\n\n if jaccard_score>= 0.5: \n # When matched add 1 to True Positive \n # and remove 1 from False Negative\n sample_TP += 1 \n sample_FN -= 1 \n\n # Once matched, that particular prediction \n # should no longer be used, so remove it\n prediction.remove(prediction[pred_index])\n\n #Since only one True dataset can match with \n # one predicted dataset, Leave the loop.\n break\n\n # All other predictions left which are not matched goes to FP\n sample_FP = len(prediction)\n\n print(f'TP:{sample_TP}, FP: {sample_FP}, FN:{sample_FN}')\n\n # Sample all the samples TP, FP, FN to get the overall TP, FP, FN\n overall_TP += sample_TP\n overall_FP += sample_FP\n overall_FN += sample_FN\n\n print(f'overall_TP:{overall_TP}, overall_FP: {overall_FP}, overall_FN:{overall_FN}')\n print(\"\")\n\n# Calculate Percision, Recall and F0.5 Score\nPrecision = overall_TP/(overall_TP+overall_FP)\nRecall = overall_TP/(overall_TP+overall_FN)\n\nF_0_5 = ((1.25 * Precision) * Recall) / ((0.25*Precision) + Recall)\nprint(Precision, Recall, F_0_5)\nExample:
\nfinal_prediction = {'sample1': {'actual':['agricultural resource management survey',\n 'baccalaureate and beyond', \n 'beginning postsecondary student'\n ], \n 'predicted':['baccalaureate and']\n },\n 'sample2': {'actual':['characterizing health associated risks and your baseline disease in sars cov 2 charybdis',\n 'genome sequence of sars cov 2'\n ], \n 'predicted':['genome sequence', \n 'characterizing health associated risks and your baseline disease']\n }\n }\nsample1\nTP:1, FP: 0, FN:2\noverall_TP:1, overall_FP: 0, overall_FN:2\n\nsample2\nTP:1, FP: 1, FN:1\noverall_TP:2, overall_FP: 1, overall_FN:3\n\nPrecision: 0.6666666666666666 Recall: 0.4 F0.5: 0.5882352941176471\nHi all, after going through the Evaluation page and the discussions, I'm still not clear what exactly is the implementation of the metric. So I wrote my own based on my understanding. Please do verify and let me know in the comment if my understanding is right or wrong.
\nTrue_label = [\"covid 19 open research data\", \"high school longitudinal study\"]\nprediction = [\"19 open research data\", \"covid 19 open research dataset\", \"agni\"]\nThese are my assumptions.
\nOne true label can be mapped to only one predicted string. While looping If it finds a first match(Jaccard>=0.5), then the same True label cannot be matched to another prediction which could be very close to it later in the loop .i.e.
\ni.e. True_Label[0] ''covid 19 open research data\" is matched to only Prediction[0] \"covid 19 open research dataset\". So Prediction[0] is TP
Similarly the predictions cannot be matched with True labels which are already matched.
\ni.e. Prediction[1] \"covid 19 open research dataset\" cannot be matched to True_Label[0] ''covid 19 open research data\" since it's already matched with Prediction[0] \"19 open research data\". So Prediction[1] if FP
Prediction[2] \"agni\" does not have any close match so it's FP
True Label[1] \"high school longitudinal study\" does not have a match so it's FN
All the FP, FN, TP are summed-up together from each sample to finally calculate the Precision and Recall and finally the F0.5
overall_TP = 0\noverall_FP = 0\noverall_FN = 0\n\nfor Id in final_prediction:\n print(Id)\n\n # Sort the actuals\n actual = final_prediction[Id]['actual'].copy()\n actual.sort()\n\n # Sort the Predictions\n prediction = final_prediction[Id]['predicted'].copy()\n prediction.sort()\n\n # Initialize the TP, FP, FN for the sample\n sample_TP = 0\n sample_FP = 0\n sample_FN = len(actual) #Assume none of the true label is matched initially\n\n # Loop the ground Truth first\n for actu_index in range(len(actual)):\n if len(prediction) !=0:\n\n # Loop through each of the predictions\n for pred_index in range(len(prediction)):\n\n # Find the Jaccard score.\n jaccard_score = jaccard(actual[actu_index], prediction[pred_index])\n\n if jaccard_score>= 0.5: \n # When matched add 1 to True Positive \n # and remove 1 from False Negative\n sample_TP += 1 \n sample_FN -= 1 \n\n # Once matched, that particular prediction \n # should no longer be used, so remove it\n prediction.remove(prediction[pred_index])\n\n #Since only one True dataset can match with \n # one predicted dataset, Leave the loop.\n break\n\n # All other predictions left which are not matched goes to FP\n sample_FP = len(prediction)\n\n print(f'TP:{sample_TP}, FP: {sample_FP}, FN:{sample_FN}')\n\n # Sample all the samples TP, FP, FN to get the overall TP, FP, FN\n overall_TP += sample_TP\n overall_FP += sample_FP\n overall_FN += sample_FN\n\n print(f'overall_TP:{overall_TP}, overall_FP: {overall_FP}, overall_FN:{overall_FN}')\n print(\"\")\n\n# Calculate Percision, Recall and F0.5 Score\nPrecision = overall_TP/(overall_TP+overall_FP)\nRecall = overall_TP/(overall_TP+overall_FN)\n\nF_0_5 = ((1.25 * Precision) * Recall) / ((0.25*Precision) + Recall)\nprint(Precision, Recall, F_0_5)\nExample:
\nfinal_prediction = {'sample1': {'actual':['agricultural resource management survey',\n 'baccalaureate and beyond', \n 'beginning postsecondary student'\n ], \n 'predicted':['baccalaureate and']\n },\n 'sample2': {'actual':['characterizing health associated risks and your baseline disease in sars cov 2 charybdis',\n 'genome sequence of sars cov 2'\n ], \n 'predicted':['genome sequence', \n 'characterizing health associated risks and your baseline disease']\n }\n }\nsample1\nTP:1, FP: 0, FN:2\noverall_TP:1, overall_FP: 0, overall_FN:2\n\nsample2\nTP:1, FP: 1, FN:1\noverall_TP:2, overall_FP: 1, overall_FN:3\n\nPrecision: 0.6666666666666666 Recall: 0.4 F0.5: 0.5882352941176471\nThanks for sharing
", "rawMarkdown": "Thanks for sharing", "votes": null } ], "comments": [ { "id": 1357723, "author_name": "ddiiiieeennndd", "author_url": "", "post_date": "06/19/2021 23:22:41", "content": "Thanks for sharing
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1347611": "Hi all, after going through the Evaluation page and the discussions, I'm still not clear what exactly is the implementation of the metric. So I wrote my own based on my understanding. Please do verify and let me know in the comment if my understanding is right or wrong.\n```\nTrue_label = [\"covid 19 open research data\", \"high school longitudinal study\"]\nprediction = [\"19 open research data\", \"covid 19 open research dataset\", \"agni\"]\n```\nThese are my assumptions. \n- One true label can be mapped to only one predicted string. While looping If it finds a first match(Jaccard>=0.5), then the same True label cannot be matched to another prediction which could be very close to it later in the loop .i.e. \ni.e. True_Label[0] ''covid 19 open research data\" is matched to only Prediction[0] \"covid 19 open research dataset\". So Prediction[0] is TP\n\n- Similarly the predictions cannot be matched with True labels which are already matched.\ni.e. Prediction[1] \"covid 19 open research dataset\" cannot be matched to True_Label[0] ''covid 19 open research data\" since it's already matched with Prediction[0] \"19 open research data\". So Prediction[1] if FP\n\n- Prediction[2] \"agni\" does not have any close match so it's FP\n\n- True Label[1] \"high school longitudinal study\" does not have a match so it's FN\n\n- All the FP, FN, TP are summed-up together from each sample to finally calculate the Precision and Recall and finally the F0.5\n\n\n```\noverall_TP = 0\noverall_FP = 0\noverall_FN = 0\n\nfor Id in final_prediction:\n print(Id)\n \n # Sort the actuals\n actual = final_prediction[Id]['actual'].copy()\n actual.sort()\n\n # Sort the Predictions\n prediction = final_prediction[Id]['predicted'].copy()\n prediction.sort()\n \n # Initialize the TP, FP, FN for the sample\n sample_TP = 0\n sample_FP = 0\n sample_FN = len(actual) #Assume none of the true label is matched initially\n \n # Loop the ground Truth first\n for actu_index in range(len(actual)):\n if len(prediction) !=0:\n \n # Loop through each of the predictions\n for pred_index in range(len(prediction)):\n \n # Find the Jaccard score.\n jaccard_score = jaccard(actual[actu_index], prediction[pred_index])\n\n if jaccard_score>= 0.5: \n # When matched add 1 to True Positive \n # and remove 1 from False Negative\n sample_TP += 1 \n sample_FN -= 1 \n \n # Once matched, that particular prediction \n # should no longer be used, so remove it\n prediction.remove(prediction[pred_index])\n \n #Since only one True dataset can match with \n # one predicted dataset, Leave the loop.\n break\n \n # All other predictions left which are not matched goes to FP\n sample_FP = len(prediction)\n \n print(f'TP:{sample_TP}, FP: {sample_FP}, FN:{sample_FN}')\n \n # Sample all the samples TP, FP, FN to get the overall TP, FP, FN\n overall_TP += sample_TP\n overall_FP += sample_FP\n overall_FN += sample_FN\n\n print(f'overall_TP:{overall_TP}, overall_FP: {overall_FP}, overall_FN:{overall_FN}')\n print(\"\")\n\n# Calculate Percision, Recall and F0.5 Score\nPrecision = overall_TP/(overall_TP+overall_FP)\nRecall = overall_TP/(overall_TP+overall_FN)\n\nF_0_5 = ((1.25 * Precision) * Recall) / ((0.25*Precision) + Recall)\nprint(Precision, Recall, F_0_5)\n```\n\nExample:\n######### Example Input \n\n```\nfinal_prediction = {'sample1': {'actual':['agricultural resource management survey',\n 'baccalaureate and beyond', \n 'beginning postsecondary student'\n ], \n 'predicted':['baccalaureate and']\n },\n 'sample2': {'actual':['characterizing health associated risks and your baseline disease in sars cov 2 charybdis',\n 'genome sequence of sars cov 2'\n ], \n 'predicted':['genome sequence', \n 'characterizing health associated risks and your baseline disease']\n }\n }\n\n```\n\n######### output\n\n```\nsample1\nTP:1, FP: 0, FN:2\noverall_TP:1, overall_FP: 0, overall_FN:2\n\nsample2\nTP:1, FP: 1, FN:1\noverall_TP:2, overall_FP: 1, overall_FN:3\n\nPrecision: 0.6666666666666666 Recall: 0.4 F0.5: 0.5882352941176471\n```", "1357723": "Thanks for sharing" }, "source": "meta" }