~Initially, I had some trouble understanding how the two-step evaluation would actually work.
\nSo, I'll break it down here a little for others to refer too.
For a given publication, we are asked to find the datasets used in that research paper by the author.
\nThese are short excerpts from the publications that would appear to mention a dataset. Since a publication can refer to multiple datasets, you can also include names of multiple datasets, separated by ' | '. Similarly, even ground truth for a publication will have multiple datasets.
So for a given publication, we have a set of predictions separated by \" | \" along with the ground truth labels(currently they are given as multiple entries in train.csv for a single publication, but with some transformations, you can get it in the same format).
\nThe evaluation step for this competition comes in two parts:
\ndef jaccard_similarity(s1, s2):l1 = s1.split(\" \")l2 = s2.split(\" \") intersection = len(list(set(l1).intersection(l2)))union = (len(l1) + len(l2)) - intersectionreturn float(intersection) / unionIt simply looks at the prediction and the ground label to see which words are common and which are not and would assign a score between 0(least similar) and 1(most similar).
\nThe prediction with the highest similarity score for a given ground truth is matched with that ground truth. other predictions for that ground truth will be called unmatched. If there is a tie for the highest similarity score, then it will be resolved by alphabetically sorting the predicting strings.
Update: There is no F0 score anymore.
\nBefore computing the F0.5 score, we'll have to compute the elements of it as follows:
\n\n\nDoes it mean a ground truth for which there are no TP(no matched prediction above 0.5)?
\n
\n Or it hints at the case when there is no prediction at all ?
It would be great if someone could clarify this in the comments.
", "messages": [ { "id": "1250724", "postDate": "03/24/2021 08:25:28", "content": "~Initially, I had some trouble understanding how the two-step evaluation would actually work.
\nSo, I'll break it down here a little for others to refer too.
For a given publication, we are asked to find the datasets used in that research paper by the author.
\nThese are short excerpts from the publications that would appear to mention a dataset. Since a publication can refer to multiple datasets, you can also include names of multiple datasets, separated by ' | '. Similarly, even ground truth for a publication will have multiple datasets.
So for a given publication, we have a set of predictions separated by \" | \" along with the ground truth labels(currently they are given as multiple entries in train.csv for a single publication, but with some transformations, you can get it in the same format).
\nThe evaluation step for this competition comes in two parts:
\ndef jaccard_similarity(s1, s2):l1 = s1.split(\" \")l2 = s2.split(\" \") intersection = len(list(set(l1).intersection(l2)))union = (len(l1) + len(l2)) - intersectionreturn float(intersection) / unionIt simply looks at the prediction and the ground label to see which words are common and which are not and would assign a score between 0(least similar) and 1(most similar).
\nThe prediction with the highest similarity score for a given ground truth is matched with that ground truth. other predictions for that ground truth will be called unmatched. If there is a tie for the highest similarity score, then it will be resolved by alphabetically sorting the predicting strings.
Update: There is no F0 score anymore.
\nBefore computing the F0.5 score, we'll have to compute the elements of it as follows:
\n\n\nDoes it mean a ground truth for which there are no TP(no matched prediction above 0.5)?
\n
\n Or it hints at the case when there is no prediction at all ?
It would be great if someone could clarify this in the comments.
", "rawMarkdown": "~Initially, I had some trouble understanding how the two-step evaluation would actually work.\nSo, I'll break it down here a little for others to refer too.\n\nFor a given publication, we are asked to find the datasets used in that research paper by the author.\nThese are short excerpts from the publications that would appear to mention a dataset. Since a publication can refer to multiple datasets, you can also include names of multiple datasets, separated by ' | '. Similarly, even ground truth for a publication will have multiple datasets.\n\nSo for a given publication, we have a set of predictions separated by \" | \" along with the ground truth labels(currently they are given as multiple entries in train.csv for a single publication, but with some transformations, you can get it in the same format).\n\nThe evaluation step for this competition comes in two parts:\n- **Jaccard Score**: It is a similarity score calculated for each pair of prediction and ground truth for a particular publication. The code for it is pretty straightforward:\n`def jaccard_similarity(s1, s2):`\n `l1 = s1.split(\" \")`\n `l2 = s2.split(\" \")` \n `intersection = len(list(set(l1).intersection(l2)))`\n `union = (len(l1) + len(l2)) - intersection`\n ` return float(intersection) / union`\n\nIt simply looks at the prediction and the ground label to see which words are common and which are not and would assign a score between 0(least similar) and 1(most similar).\n\nThe prediction with the highest similarity score for a given ground truth is `matched` with that ground truth. other predictions for that ground truth will be called `unmatched`. If there is a tie for the highest similarity score, then it will be resolved by alphabetically sorting the predicting strings.\n\n~~- **F0 score**: It is a measure of precision derived from True Positives(TP) and False Positives(FP).~~\n\n~~`F0 score = TP/ (TP + FP)`~~\n\n*Update: There is no F0 score anymore.*\n\n- **F0.5 Score** : I believe F 0.5 is a measure between F1(which weights precision and recall equally) and F0(which is just precision). F 0.5 will take into account False Negatives( unlike F0) but will be biased towards precision.\n\nBefore computing the F0.5 score, we'll have to compute the elements of it as follows:\n- Matched predictions with Jaccard Score greater than or equal to 0.5 are counted as TP, the rest matched predictions(below 0.5) will be FP.\n- Predictions unmatched for a ground truth will also be counted as FP.\n- Any ground truths with no nearest predictions are counted as false negatives (FN): I am still not sure about what this means. \n> Does it mean a ground truth for which there are no TP(no matched prediction above 0.5)?\nOr it hints at the case when there is no prediction at all ?\n\nIt would be great if someone could clarify this in the comments.", "votes": null }, { "id": "1250753", "postDate": "03/24/2021 08:48:59", "content": "There is no F1 score here, it is F0 instead. F0 is precision, it is TP / (TP + FP)
", "rawMarkdown": "There is no F1 score here, it is F0 instead. F0 is precision, it is TP / (TP + FP)", "votes": null }, { "id": "1250799", "postDate": "03/24/2021 09:27:06", "content": "Thanks for pointing it out. I don't know how I missed that.
", "rawMarkdown": "Thanks for pointing it out. I don't know how I missed that.", "votes": null }, { "id": "1251605", "postDate": "03/25/2021 00:10:13", "content": "The use of the F0 score is why string literal matching on dataset title was generating 100% scores. This has now been fixed. We are now using the F0.5 score, which balances Precision and Recall, but is biased towards Precision.
\n\n\nYou mention FN in the evaluation page, but the metric F0 does not use it?
\nThe description will change, but to note that we are moving to a Beta = 0.5. FNs are important here, but precision moreso.
\n
https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228393#1251248
", "rawMarkdown": "The use of the F0 score is why string literal matching on dataset title was generating 100% scores. This has now been fixed. We are now using the F0.5 score, which balances Precision and Recall, but is biased towards Precision.\n\n> You mention FN in the evaluation page, but the metric F0 does not use it? \n>\n> The description will change, but to note that we are moving to a Beta = 0.5. FNs are important here, but precision moreso.\n\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228393#1251248", "votes": null }, { "id": "1251608", "postDate": "03/25/2021 00:12:01", "content": "I also asked a similar question here, and got the following resonses:
\n\n\nIn this competition, predictions are drawn from the documents. PredictionString is being compared against extracts from publications that indicate a dataset was mentioned in that publication. The strings that predictions are being compared against have been cleaned, as indicated in the Evaluation page.
\nYes! There are a huge number of labels in test that are not mentioned in train at all. So the task is less to match strings, and more to create an NLP solution for generalizing the \"can I identify datasets mentioned in this document\" problem.
\nCorrect. cleaned_label represents examples of (cleaned) extracted text we're attempting to predict for the training data.
\n
https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228463
", "rawMarkdown": "I also asked a similar question here, and got the following resonses:\n\n> In this competition, predictions are drawn from the documents. PredictionString is being compared against extracts from publications that indicate a dataset was mentioned in that publication. The strings that predictions are being compared against have been cleaned, as indicated in the Evaluation page.\n>\n> Yes! There are a huge number of labels in test that are not mentioned in train at all. So the task is less to match strings, and more to create an NLP solution for generalizing the \"can I identify datasets mentioned in this document\" problem.\n>\n> Correct. cleaned_label represents examples of (cleaned) extracted text we're attempting to predict for the training data.\n\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228463", "votes": null }, { "id": "1257468", "postDate": "03/30/2021 22:04:02", "content": "I'm also really confused about the same part. It's not clear what \"no nearest predictions\" means.
\nIn addition to the two possibilties OP mentioned, there's also a third possibility that \"ground truth with no nearest predictions\" means all predictions have 0 Jaccard score. But this seems a little bit strange given that they list an explicit rule for how to break ties. This third possibility would be kind of an odd special case/exception to the tiebreaking rule.
\nI tried to code up the three possiblities. I'm numbering the rules, quoting from the evaluation page:
\ndef f_beta(tp, fp, fn, beta=0.5):\n return (1 + beta ** 2) * tp / ((1 + beta ** 2) * tp + beta ** 2 * fn + fp)\n\n\ndef match_predictions_v1(targets, predictions):\n \"\"\"assume that \"no matched predictions\" means the match is below 0.5\"\"\"\n if len(predictions) == 0:\n return 0\n\n predictions = sorted(predictions)\n matches = set()\n tp = 0\n fp = 0\n fn = 0\n\n for tgt in targets:\n score, best_pred = sorted((jaccard(tgt, pred), pred) for pred in predictions)[-1] # rule 1\n if score >= 0.5:\n tp += 1 # rule 2\n matches.add(best_pred)\n else:\n fp += 1 # rule 2a\n fn += 1 # by assumption\n\n fp += len(set(predictions) - matches) # rule 3\n return f_beta(tp, fp, fn)\n\n\ndef match_predictions_v2(targets, predictions):\n \"\"\"assume that \"no matched predictions\" means no predictions at all\"\"\"\n if len(predictions) == 0:\n return 0\n\n predictions = sorted(predictions)\n matches = set()\n tp = 0\n fp = 0\n fn = 0\n\n for tgt in targets:\n score, best_pred = sorted((jaccard(tgt, pred), pred) for pred in predictions)[-1] # rule 1\n if score >= 0.5:\n tp += 1 # rule 2a\n matches.add(best_pred)\n else:\n fp += 1 # rule 2b\n\n fp += len(set(predictions) - matches) # rule 3\n return f_beta(tp, fp, fn)\n\n\ndef match_predictions_v3(targets, predictions):\n \"\"\"assume that \"no matched predictions\" means all predictions have 0 score\"\"\"\n if len(predictions) == 0:\n return 0\n\n predictions = sorted(predictions)\n matches = set()\n tp = 0\n fp = 0\n fn = 0\n for tgt in targets:\n scores = ((jaccard(tgt, pred), pred) for pred in predictions)\n if all(score == 0 for score, _ in scores):\n fn += 1\n continue\n\n score, best_pred = sorted(scores)[-1]\n if score >= 0.5:\n tp += 1 # rule 2a\n matches.add(best_pred)\n else:\n fp += 1 # rule 2b\n\n fp += len(set(predictions) - matches) # rule 3\n return f_beta(tp, fp, fn)\nIf someone wants to double check my work here that would be great 😄
", "rawMarkdown": "I'm also really confused about the same part. It's not clear what \"no nearest predictions\" means.\n\nIn addition to the two possibilties OP mentioned, there's also a third possibility that \"ground truth with no nearest predictions\" means all predictions have 0 Jaccard score. But this seems a little bit strange given that they list an explicit rule for how to break ties. This third possibility would be kind of an odd special case/exception to the tiebreaking rule.\n\nI tried to code up the three possiblities. I'm numbering the rules, quoting from the evaluation page:\n\n1. Predicted strings for each publication are sorted alphabetically and processed in that order. Any\n scoring ties are resolved on the basis of that sort.\n2. a) Any matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are\n counted as true positives (TP),\n b) the remainder as false positives (FP).\n3. Any unmatched predictions are counted as false positives (FP).\n4. Any ground truths with no nearest predictions are counted as false negatives (FN).\n\n```python\ndef f_beta(tp, fp, fn, beta=0.5):\n return (1 + beta ** 2) * tp / ((1 + beta ** 2) * tp + beta ** 2 * fn + fp)\n\n\ndef match_predictions_v1(targets, predictions):\n \"\"\"assume that \"no matched predictions\" means the match is below 0.5\"\"\"\n if len(predictions) == 0:\n return 0\n\n predictions = sorted(predictions)\n matches = set()\n tp = 0\n fp = 0\n fn = 0\n\n for tgt in targets:\n score, best_pred = sorted((jaccard(tgt, pred), pred) for pred in predictions)[-1] # rule 1\n if score >= 0.5:\n tp += 1 # rule 2\n matches.add(best_pred)\n else:\n fp += 1 # rule 2a\n fn += 1 # by assumption\n\n fp += len(set(predictions) - matches) # rule 3\n return f_beta(tp, fp, fn)\n\n\ndef match_predictions_v2(targets, predictions):\n \"\"\"assume that \"no matched predictions\" means no predictions at all\"\"\"\n if len(predictions) == 0:\n return 0\n\n predictions = sorted(predictions)\n matches = set()\n tp = 0\n fp = 0\n fn = 0\n\n for tgt in targets:\n score, best_pred = sorted((jaccard(tgt, pred), pred) for pred in predictions)[-1] # rule 1\n if score >= 0.5:\n tp += 1 # rule 2a\n matches.add(best_pred)\n else:\n fp += 1 # rule 2b\n\n fp += len(set(predictions) - matches) # rule 3\n return f_beta(tp, fp, fn)\n\n\ndef match_predictions_v3(targets, predictions):\n \"\"\"assume that \"no matched predictions\" means all predictions have 0 score\"\"\"\n if len(predictions) == 0:\n return 0\n\n predictions = sorted(predictions)\n matches = set()\n tp = 0\n fp = 0\n fn = 0\n for tgt in targets:\n scores = ((jaccard(tgt, pred), pred) for pred in predictions)\n if all(score == 0 for score, _ in scores):\n fn += 1\n continue\n\n score, best_pred = sorted(scores)[-1]\n if score >= 0.5:\n tp += 1 # rule 2a\n matches.add(best_pred)\n else:\n fp += 1 # rule 2b\n\n fp += len(set(predictions) - matches) # rule 3\n return f_beta(tp, fp, fn)\n\n```\n\nIf someone wants to double check my work here that would be great 😄", "votes": null }, { "id": "1343272", "postDate": "06/10/2021 06:21:31", "content": "The problem is that we do not know what the groundtruth is. Without knowing that it is not possible to calculate valuations.
", "rawMarkdown": "The problem is that we do not know what the groundtruth is. Without knowing that it is not possible to calculate valuations.", "votes": null } ], "comments": [ { "id": 1250753, "author_name": "cpmpml", "author_url": "", "post_date": "03/24/2021 08:48:59", "content": "There is no F1 score here, it is F0 instead. F0 is precision, it is TP / (TP + FP)
", "votes": null, "replies": [ { "id": 1250799, "author_name": "ibtesama", "author_url": "", "post_date": "03/24/2021 09:27:06", "content": "Thanks for pointing it out. I don't know how I missed that.
", "votes": null, "replies": [] }, { "id": 1251605, "author_name": "jamesmcguigan", "author_url": "", "post_date": "03/25/2021 00:10:13", "content": "The use of the F0 score is why string literal matching on dataset title was generating 100% scores. This has now been fixed. We are now using the F0.5 score, which balances Precision and Recall, but is biased towards Precision.
\n\n\nYou mention FN in the evaluation page, but the metric F0 does not use it?
\nThe description will change, but to note that we are moving to a Beta = 0.5. FNs are important here, but precision moreso.
\n
https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228393#1251248
", "votes": null, "replies": [] } ] }, { "id": 1251608, "author_name": "jamesmcguigan", "author_url": "", "post_date": "03/25/2021 00:12:01", "content": "I also asked a similar question here, and got the following resonses:
\n\n\nIn this competition, predictions are drawn from the documents. PredictionString is being compared against extracts from publications that indicate a dataset was mentioned in that publication. The strings that predictions are being compared against have been cleaned, as indicated in the Evaluation page.
\nYes! There are a huge number of labels in test that are not mentioned in train at all. So the task is less to match strings, and more to create an NLP solution for generalizing the \"can I identify datasets mentioned in this document\" problem.
\nCorrect. cleaned_label represents examples of (cleaned) extracted text we're attempting to predict for the training data.
\n
https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228463
", "votes": null, "replies": [] }, { "id": 1257468, "author_name": "mskoh52", "author_url": "", "post_date": "03/30/2021 22:04:02", "content": "I'm also really confused about the same part. It's not clear what \"no nearest predictions\" means.
\nIn addition to the two possibilties OP mentioned, there's also a third possibility that \"ground truth with no nearest predictions\" means all predictions have 0 Jaccard score. But this seems a little bit strange given that they list an explicit rule for how to break ties. This third possibility would be kind of an odd special case/exception to the tiebreaking rule.
\nI tried to code up the three possiblities. I'm numbering the rules, quoting from the evaluation page:
\ndef f_beta(tp, fp, fn, beta=0.5):\n return (1 + beta ** 2) * tp / ((1 + beta ** 2) * tp + beta ** 2 * fn + fp)\n\n\ndef match_predictions_v1(targets, predictions):\n \"\"\"assume that \"no matched predictions\" means the match is below 0.5\"\"\"\n if len(predictions) == 0:\n return 0\n\n predictions = sorted(predictions)\n matches = set()\n tp = 0\n fp = 0\n fn = 0\n\n for tgt in targets:\n score, best_pred = sorted((jaccard(tgt, pred), pred) for pred in predictions)[-1] # rule 1\n if score >= 0.5:\n tp += 1 # rule 2\n matches.add(best_pred)\n else:\n fp += 1 # rule 2a\n fn += 1 # by assumption\n\n fp += len(set(predictions) - matches) # rule 3\n return f_beta(tp, fp, fn)\n\n\ndef match_predictions_v2(targets, predictions):\n \"\"\"assume that \"no matched predictions\" means no predictions at all\"\"\"\n if len(predictions) == 0:\n return 0\n\n predictions = sorted(predictions)\n matches = set()\n tp = 0\n fp = 0\n fn = 0\n\n for tgt in targets:\n score, best_pred = sorted((jaccard(tgt, pred), pred) for pred in predictions)[-1] # rule 1\n if score >= 0.5:\n tp += 1 # rule 2a\n matches.add(best_pred)\n else:\n fp += 1 # rule 2b\n\n fp += len(set(predictions) - matches) # rule 3\n return f_beta(tp, fp, fn)\n\n\ndef match_predictions_v3(targets, predictions):\n \"\"\"assume that \"no matched predictions\" means all predictions have 0 score\"\"\"\n if len(predictions) == 0:\n return 0\n\n predictions = sorted(predictions)\n matches = set()\n tp = 0\n fp = 0\n fn = 0\n for tgt in targets:\n scores = ((jaccard(tgt, pred), pred) for pred in predictions)\n if all(score == 0 for score, _ in scores):\n fn += 1\n continue\n\n score, best_pred = sorted(scores)[-1]\n if score >= 0.5:\n tp += 1 # rule 2a\n matches.add(best_pred)\n else:\n fp += 1 # rule 2b\n\n fp += len(set(predictions) - matches) # rule 3\n return f_beta(tp, fp, fn)\nIf someone wants to double check my work here that would be great 😄
", "votes": null, "replies": [] }, { "id": 1343272, "author_name": "paoloantonuccio", "author_url": "", "post_date": "06/10/2021 06:21:31", "content": "The problem is that we do not know what the groundtruth is. Without knowing that it is not possible to calculate valuations.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1250724": "~Initially, I had some trouble understanding how the two-step evaluation would actually work.\nSo, I'll break it down here a little for others to refer too.\n\nFor a given publication, we are asked to find the datasets used in that research paper by the author.\nThese are short excerpts from the publications that would appear to mention a dataset. Since a publication can refer to multiple datasets, you can also include names of multiple datasets, separated by ' | '. Similarly, even ground truth for a publication will have multiple datasets.\n\nSo for a given publication, we have a set of predictions separated by \" | \" along with the ground truth labels(currently they are given as multiple entries in train.csv for a single publication, but with some transformations, you can get it in the same format).\n\nThe evaluation step for this competition comes in two parts:\n- **Jaccard Score**: It is a similarity score calculated for each pair of prediction and ground truth for a particular publication. The code for it is pretty straightforward:\n`def jaccard_similarity(s1, s2):`\n `l1 = s1.split(\" \")`\n `l2 = s2.split(\" \")` \n `intersection = len(list(set(l1).intersection(l2)))`\n `union = (len(l1) + len(l2)) - intersection`\n ` return float(intersection) / union`\n\nIt simply looks at the prediction and the ground label to see which words are common and which are not and would assign a score between 0(least similar) and 1(most similar).\n\nThe prediction with the highest similarity score for a given ground truth is `matched` with that ground truth. other predictions for that ground truth will be called `unmatched`. If there is a tie for the highest similarity score, then it will be resolved by alphabetically sorting the predicting strings.\n\n~~- **F0 score**: It is a measure of precision derived from True Positives(TP) and False Positives(FP).~~\n\n~~`F0 score = TP/ (TP + FP)`~~\n\n*Update: There is no F0 score anymore.*\n\n- **F0.5 Score** : I believe F 0.5 is a measure between F1(which weights precision and recall equally) and F0(which is just precision). F 0.5 will take into account False Negatives( unlike F0) but will be biased towards precision.\n\nBefore computing the F0.5 score, we'll have to compute the elements of it as follows:\n- Matched predictions with Jaccard Score greater than or equal to 0.5 are counted as TP, the rest matched predictions(below 0.5) will be FP.\n- Predictions unmatched for a ground truth will also be counted as FP.\n- Any ground truths with no nearest predictions are counted as false negatives (FN): I am still not sure about what this means. \n> Does it mean a ground truth for which there are no TP(no matched prediction above 0.5)?\nOr it hints at the case when there is no prediction at all ?\n\nIt would be great if someone could clarify this in the comments.", "1250753": "There is no F1 score here, it is F0 instead. F0 is precision, it is TP / (TP + FP)", "1250799": "Thanks for pointing it out. I don't know how I missed that.", "1251605": "The use of the F0 score is why string literal matching on dataset title was generating 100% scores. This has now been fixed. We are now using the F0.5 score, which balances Precision and Recall, but is biased towards Precision.\n\n> You mention FN in the evaluation page, but the metric F0 does not use it? \n>\n> The description will change, but to note that we are moving to a Beta = 0.5. FNs are important here, but precision moreso.\n\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228393#1251248", "1251608": "I also asked a similar question here, and got the following resonses:\n\n> In this competition, predictions are drawn from the documents. PredictionString is being compared against extracts from publications that indicate a dataset was mentioned in that publication. The strings that predictions are being compared against have been cleaned, as indicated in the Evaluation page.\n>\n> Yes! There are a huge number of labels in test that are not mentioned in train at all. So the task is less to match strings, and more to create an NLP solution for generalizing the \"can I identify datasets mentioned in this document\" problem.\n>\n> Correct. cleaned_label represents examples of (cleaned) extracted text we're attempting to predict for the training data.\n\nhttps://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228463", "1257468": "I'm also really confused about the same part. It's not clear what \"no nearest predictions\" means.\n\nIn addition to the two possibilties OP mentioned, there's also a third possibility that \"ground truth with no nearest predictions\" means all predictions have 0 Jaccard score. But this seems a little bit strange given that they list an explicit rule for how to break ties. This third possibility would be kind of an odd special case/exception to the tiebreaking rule.\n\nI tried to code up the three possiblities. I'm numbering the rules, quoting from the evaluation page:\n\n1. Predicted strings for each publication are sorted alphabetically and processed in that order. Any\n scoring ties are resolved on the basis of that sort.\n2. a) Any matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are\n counted as true positives (TP),\n b) the remainder as false positives (FP).\n3. Any unmatched predictions are counted as false positives (FP).\n4. Any ground truths with no nearest predictions are counted as false negatives (FN).\n\n```python\ndef f_beta(tp, fp, fn, beta=0.5):\n return (1 + beta ** 2) * tp / ((1 + beta ** 2) * tp + beta ** 2 * fn + fp)\n\n\ndef match_predictions_v1(targets, predictions):\n \"\"\"assume that \"no matched predictions\" means the match is below 0.5\"\"\"\n if len(predictions) == 0:\n return 0\n\n predictions = sorted(predictions)\n matches = set()\n tp = 0\n fp = 0\n fn = 0\n\n for tgt in targets:\n score, best_pred = sorted((jaccard(tgt, pred), pred) for pred in predictions)[-1] # rule 1\n if score >= 0.5:\n tp += 1 # rule 2\n matches.add(best_pred)\n else:\n fp += 1 # rule 2a\n fn += 1 # by assumption\n\n fp += len(set(predictions) - matches) # rule 3\n return f_beta(tp, fp, fn)\n\n\ndef match_predictions_v2(targets, predictions):\n \"\"\"assume that \"no matched predictions\" means no predictions at all\"\"\"\n if len(predictions) == 0:\n return 0\n\n predictions = sorted(predictions)\n matches = set()\n tp = 0\n fp = 0\n fn = 0\n\n for tgt in targets:\n score, best_pred = sorted((jaccard(tgt, pred), pred) for pred in predictions)[-1] # rule 1\n if score >= 0.5:\n tp += 1 # rule 2a\n matches.add(best_pred)\n else:\n fp += 1 # rule 2b\n\n fp += len(set(predictions) - matches) # rule 3\n return f_beta(tp, fp, fn)\n\n\ndef match_predictions_v3(targets, predictions):\n \"\"\"assume that \"no matched predictions\" means all predictions have 0 score\"\"\"\n if len(predictions) == 0:\n return 0\n\n predictions = sorted(predictions)\n matches = set()\n tp = 0\n fp = 0\n fn = 0\n for tgt in targets:\n scores = ((jaccard(tgt, pred), pred) for pred in predictions)\n if all(score == 0 for score, _ in scores):\n fn += 1\n continue\n\n score, best_pred = sorted(scores)[-1]\n if score >= 0.5:\n tp += 1 # rule 2a\n matches.add(best_pred)\n else:\n fp += 1 # rule 2b\n\n fp += len(set(predictions) - matches) # rule 3\n return f_beta(tp, fp, fn)\n\n```\n\nIf someone wants to double check my work here that would be great 😄", "1343272": "The problem is that we do not know what the groundtruth is. Without knowing that it is not possible to calculate valuations." }, "source": "meta" }