I share a function for calculating the evaluation metric.
\nfrom typing import List\nimport numpy as np\n\n\ndef compute_fbeta(y_true: List[List[str]],\n y_pred: List[List[str]],\n beta: float = 0.5) -> float:\n \"\"\"Compute the Jaccard-based micro FBeta score.\n\n References\n ----------\n - https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation\n \"\"\"\n\n def _jaccard_similarity(str1: str, str2: str) -> float:\n a = set(str1.split()) \n b = set(str2.split())\n c = a.intersection(b)\n return float(len(c)) / (len(a) + len(b) - len(c))\n\n tp = 0 # true positive\n fp = 0 # false positive\n fn = 0 # false negative\n for ground_truth_list, predicted_string_list in zip(y_true, y_pred):\n predicted_string_list_sorted = sorted(predicted_string_list)\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n predicted_string_list_sorted.pop(matched_idx)\n tp += 1\n else:\n fn += 1\n fp += len(predicted_string_list_sorted)\n\n tp *= (1 + beta ** 2)\n fn *= beta ** 2\n fbeta_score = tp / (tp + fp + fn)\n return fbeta_score\nIf I'm wrong, please let me know!
", "messages": [ { "id": 1260182, "postDate": "2021-04-01T23:06:51.410Z", "content": "I share a function for calculating the evaluation metric.
\nfrom typing import List\nimport numpy as np\n\n\ndef compute_fbeta(y_true: List[List[str]],\n y_pred: List[List[str]],\n beta: float = 0.5) -> float:\n \"\"\"Compute the Jaccard-based micro FBeta score.\n\n References\n ----------\n - https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation\n \"\"\"\n\n def _jaccard_similarity(str1: str, str2: str) -> float:\n a = set(str1.split()) \n b = set(str2.split())\n c = a.intersection(b)\n return float(len(c)) / (len(a) + len(b) - len(c))\n\n tp = 0 # true positive\n fp = 0 # false positive\n fn = 0 # false negative\n for ground_truth_list, predicted_string_list in zip(y_true, y_pred):\n predicted_string_list_sorted = sorted(predicted_string_list)\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n predicted_string_list_sorted.pop(matched_idx)\n tp += 1\n else:\n fn += 1\n fp += len(predicted_string_list_sorted)\n\n tp *= (1 + beta ** 2)\n fn *= beta ** 2\n fbeta_score = tp / (tp + fp + fn)\n return fbeta_score\nIf I'm wrong, please let me know!
", "rawMarkdown": "I share a function for calculating the evaluation metric.\n\n```\nfrom typing import List\nimport numpy as np\n\n\ndef compute_fbeta(y_true: List[List[str]],\n y_pred: List[List[str]],\n beta: float = 0.5) -> float:\n \"\"\"Compute the Jaccard-based micro FBeta score.\n\n References\n ----------\n - https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation\n \"\"\"\n\n def _jaccard_similarity(str1: str, str2: str) -> float:\n a = set(str1.split()) \n b = set(str2.split())\n c = a.intersection(b)\n return float(len(c)) / (len(a) + len(b) - len(c))\n\n tp = 0 # true positive\n fp = 0 # false positive\n fn = 0 # false negative\n for ground_truth_list, predicted_string_list in zip(y_true, y_pred):\n predicted_string_list_sorted = sorted(predicted_string_list)\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n predicted_string_list_sorted.pop(matched_idx)\n tp += 1\n else:\n fn += 1\n fp += len(predicted_string_list_sorted)\n\n tp *= (1 + beta ** 2)\n fn *= beta ** 2\n fbeta_score = tp / (tp + fp + fn)\n return fbeta_score\n```\n\nIf I'm wrong, please let me know!", "votes": 59 }, { "id": 1306968, "postDate": "2021-05-14T07:12:31.287Z", "content": "Hi @sakami, I have seen many changes proposed in the comments. Is the code above the final version ?
", "rawMarkdown": "Hi @sakami, I have seen many changes proposed in the comments. Is the code above the final version ?", "votes": 2 }, { "id": 1264262, "postDate": "2021-04-06T03:13:46.543Z", "content": "@philculliton I would like to know the implementation behind the Jaccard similarity. I made an experiment where I change the order of the words in one prediction for each publication in the test dataset, and the score was different(0.001 lower with fixed labels) and non of the Jaccard similarity function that I've seen get a different result when the words in a sentence have different order. The objective of the competition is to get new labels different than the ones fixed in the training data so I think is important to clarify if this new predictions have to come in an specific order
\nExample(code from @cpmpml ):
\ndef jaccard(s1, s2):\n l1 = set( s1.split(\" \"))\n l2 = set(s2.split(\" \")) \n intersection = len(l1.intersection(l2))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union
\njaccard('alzheimers disease neuroimaging initiative','initiative disease neuroimaging alzheimers')
\n1
But it seems that in the backend different order in the words results in a different outcome, or at least is what I got from the experiment, please give us an update on the evaluation score or let me know if somehow I am mistaken
", "rawMarkdown": "@philculliton I would like to know the implementation behind the Jaccard similarity. I made an experiment where I change the order of the words in one prediction for each publication in the test dataset, and the score was different(0.001 lower with fixed labels) and non of the Jaccard similarity function that I've seen get a different result when the words in a sentence have different order. The objective of the competition is to get new labels different than the ones fixed in the training data so I think is important to clarify if this new predictions have to come in an specific order\n\nExample(code from @cpmpml ):\n`def jaccard(s1, s2):\n l1 = set( s1.split(\" \"))\n l2 = set(s2.split(\" \")) \n intersection = len(l1.intersection(l2))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union`\n`jaccard('alzheimers disease neuroimaging initiative','initiative disease neuroimaging alzheimers')`\n`1`\n\nBut it seems that in the backend different order in the words results in a different outcome, or at least is what I got from the experiment, please give us an update on the evaluation score or let me know if somehow I am mistaken", "votes": 2, "replies": [ { "id": 1265285, "postDate": "2021-04-06T19:00:50.267Z", "content": "The example code from CPMP does not remove empty entries, which is something the metric does. The example code on the evaluation page does this.
\nThe order of words is not a factor in Jaccard, but the order of the predictions could be a factor in the matching between predictions and ground truth. As mentioned on the Evaluation page, we order the prediction strings alphabetically for each sample, and then each is compared against all ground truth strings. It's possible that by reordering words you are changing that order for some predictions, which would change the evaluation order, and potentially impact scoring.
\nIt's recommended that you extract text directly from the documents as your predictions.
", "rawMarkdown": "The example code from CPMP does not remove empty entries, which is something the metric does. The example code on the evaluation page does this.\n\nThe order of words is not a factor in Jaccard, but the order of the predictions could be a factor in the matching between predictions and ground truth. As mentioned on the Evaluation page, we order the prediction strings alphabetically for each sample, and then each is compared against all ground truth strings. It's possible that by reordering words you are changing that order for some predictions, which would change the evaluation order, and potentially impact scoring.\n\nIt's recommended that you extract text directly from the documents as your predictions." }, { "id": 1265320, "postDate": "2021-04-06T19:40:59.677Z", "content": "\n\nThe example code from CPMP does not remove empty entries, which is something the metric does. > The example code on the evaluation page does this.
\n
@philculliton Can you expand on this, for instance with an example? I honestly don't understand what you mean. What is confusing also is that the python code on the evaluation page does not do the same thing as the C# code you shared recently.
", "rawMarkdown": "> The example code from CPMP does not remove empty entries, which is something the metric does. > The example code on the evaluation page does this.\n\n@philculliton Can you expand on this, for instance with an example? I honestly don't understand what you mean. What is confusing also is that the python code on the evaluation page does not do the same thing as the C# code you shared recently.\n" }, { "id": 1265342, "postDate": "2021-04-06T20:11:01.880Z", "content": "@cpmpml - thanks! Yes. Your implementation splits on spaces (i.e. .split(\" \")) which produces empty string entries vs. the Evaluation version (which matches up with the C# metric's removal of empty entries) via .split(). Any strings with trailing / extra spaces will produce an empty entry in the sets in your code.
Would you mind providing an example where the C# example doesn't match up with the Python code? I walked through it with the other data scientists the other day and we couldn't see a gap, but we could definitely have missed something.
", "rawMarkdown": "@cpmpml - thanks! Yes. Your implementation splits on spaces (i.e. `.split(\" \")`) which produces empty string entries vs. the Evaluation version (which matches up with the C# metric's removal of empty entries) via `.split()`. Any strings with trailing / extra spaces will produce an empty entry in the sets in your code.\n\nWould you mind providing an example where the C# example doesn't match up with the Python code? I walked through it with the other data scientists the other day and we couldn't see a gap, but we could definitely have missed something.", "votes": 1 }, { "id": 1265349, "postDate": "2021-04-06T20:21:59.923Z", "content": "Tx, I'll check where split() and split(' ') differ. But if all is needed is to replace split(' ') by split() in my code, then I'm more than happy to do it.
\nRe your question, I see you changed the code in the evaluation page without telling us… The code you have now is correct IMHO.
\nI thought we still had this code in the evaluation page:
\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len(list(set(l1).intersection(l2)))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\nI took split(' ') from this code actually.
Ahhh, okay! Yep - I changed that right away when you brought it up, failed to mention it explicitly in the thread.
\nThanks very much for digging through it with me. I think that replacing split(\" \") with split() should do the trick, as split() handles repeated separators (I'm not a super big fan of functions doing very different things implicitly, but there you have it).
@philculliton first of all, thank you for your commitment.
\nI would like to do the double confirmation regarding your below comment.
\n As mentioned on the Evaluation page, we order the prediction strings alphabetically for each sample
\nDoes it mean alphabetically reverse order, which means that z starting is first?
From the below snapshot, I guess it so. (Because s is alphabetically behind compared to n, but it comes first)
\nId,PredictionString
\n01e4e08c-ffea-45a7-adde-6a0c0ad755fc,space location data|national space objects|national space dataset
My other question is about the pipe (|) character. I already questioned in another discussion. But here looks the right place.
I have tested with the above string.
\nstr1 = \"space location data|national space objects|national space dataset\"\nset(str1.lower().split())\n{'data', 'dataset|space', 'location', 'national', 'objects|national', 'space'}\nI guess that the reason for split() is to get the meaningful word in any sentence.
\nBut for the pipe character, it gives not a meaningful word like 'dataset|space' from my point of view.
Cheers,
\nSONG
I have same two questions that Mr. SONG @dtmanager1979 asked, if anyone can help please!
", "rawMarkdown": "I have same two questions that Mr. SONG @dtmanager1979 asked, if anyone can help please!" } ] }, { "id": 1334551, "postDate": "2021-06-03T15:29:09.517Z", "content": "Hi, late to the party but I believe there is a small issue with this scoring algoritm.
\nNamely, as the loop is constructed on 'ground_truth', it fails to find exact matching pairs, if an earlier ground_truth in the loop has a j-score above .5 for that prediction ('that prediction': the prediction that actually had an exact match (or a higher jaccard match)).
This might be more than a cosmetic issue if the exact match can not find a match in predictions, and the earlier match actually had another match above .5, although lower than the argmax.
\nI hope what I say makes sense.
\nFeel free to correct or challange this.
\nI have revised the scoring methodology but am cautious to share, because I want to make sure I am not missing something, before I stir up things :-)
I wanted to attach an example screenshot but kaggle currently doesnt support direct attachments.
\nSo here is the link if you are interested:[
\nhttps://drive.google.com/file/d/1zNtVzlS-UeyQ_15uTxJ1P5XlWzSR8sWm/view?usp=sharing
\n
\nCheers,
@sakami I think you are missing out to count the GTs as FN if they can't be matched to any predictions. Or am I missing something?
", "rawMarkdown": "@sakami I think you are missing out to count the GTs as FN if they can't be matched to any predictions. Or am I missing something?" }, { "id": 1269062, "postDate": "2021-04-10T05:36:39.470Z", "content": "Thank you for sharing!
\nI found an implementation mistake.
\nIf the most similar in a loop of ground_truth_list of which similarity < 0.5, it's not used in a next process of the loop because it's popped before checking its similarity. \nThe most similar label should only be poped if its similarity >= 0.5.
Sorry, I was wrong.
\nAny prediction should not be poped because it may be matched more than one ground truth.
def compute_fbeta(y_true: List[List[str]],\n y_pred: List[List[str]],\n beta: float = 0.5) -> float:\n \"\"\"Compute the Jaccard-based micro FBeta score.\n\n References\n ----------\n - https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation\n \"\"\"\n\n def _jaccard_similarity(str1: str, str2: str) -> float:\n a = set(str1.split()) \n b = set(str2.split())\n c = a.intersection(b)\n return float(len(c)) / (len(a) + len(b) - len(c))\n\n tp = 0 # true positive\n fp = 0 # false positive\n fn = 0 # false negative\n for ground_truth_list, predicted_string_list in zip(y_true, y_pred):\n predicted_string_list_sorted = sorted(predicted_string_list)\n correct_prediction = []\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n correct_prediction.append(predicted_string_list_sorted[matched_idx])\n else:\n fn += 1\n tp += len(correct_prediction)\n fp += max(len(predicted_string_list_sorted) - len(set(correct_prediction)), 0)\n\n tp *= (1 + beta ** 2)\n fn *= beta ** 2\n fbeta_score = tp / (tp + fp + fn)\n return fbeta_score\nThank you for your comments, Shota!
\n@philculliton Could you tell us about the metric, is it possible for one predicted string to match more than one ground truth?
\nI agree with Shota, one prediction should be able to match multiple ground truth labels.
Thank you, sakami and Shota! I agree with you that one prediction should be able to match multiple ground truth labels. I'm not convinced that the description in Evaluation supports this though, but I find it hard to tell for sure. It would be ideal to see the evaluation code (regardless of which language it is written in) in order to fully verify the approach and a Python implementation of it.
", "rawMarkdown": "Thank you, sakami and Shota! I agree with you that one prediction should be able to match multiple ground truth labels. I'm not convinced that the description in Evaluation supports this though, but I find it hard to tell for sure. It would be ideal to see the evaluation code (regardless of which language it is written in) in order to fully verify the approach and a Python implementation of it.", "votes": 4 }, { "id": 1270599, "postDate": "2021-04-11T19:33:37.560Z", "content": "@spshota I also think that each prediction can be matched with more than one GT. Thanks for changing that in the code!
\nI might be wrong, but isn't it now possible that one prediction is counted more than once as FP in your code?
", "rawMarkdown": "@spshota I also think that each prediction can be matched with more than one GT. Thanks for changing that in the code!\n\nI might be wrong, but isn't it now possible that one prediction is counted more than once as FP in your code?\n\n", "votes": 1 }, { "id": 1271153, "postDate": "2021-04-12T10:54:42.930Z", "content": "@hannes82 Thank you for your comment!
\nYou are right. I've fixed the code in a previous comment.
Before
\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n correct_count += 1\n else:\n fp += 1\n tp += correct_count\n fp += max(len(predicted_string_list_sorted) - correct_count, 0)\nAfter
\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n correct_count += 1\n tp += correct_count\n fp += max(len(predicted_string_list_sorted) - correct_count, 0)\n@spshota Your code might need another small adjustment: it subtracts a certain prediction more than once from your count of FP (in case the prediction matches with more than one GT).
", "rawMarkdown": "@spshota Your code might need another small adjustment: it subtracts a certain prediction more than once from your count of FP (in case the prediction matches with more than one GT).", "votes": 1 }, { "id": 1273006, "postDate": "2021-04-14T02:21:43.597Z", "content": "@hannes82 Thank you.
\nI've fixed the code in a previous comment.
Before
\n for ground_truth_list, predicted_string_list in zip(y_true, y_pred):\n predicted_string_list_sorted = sorted(predicted_string_list)\n correct_count = 0\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n correct_count += 1\n tp += correct_count\n fp += max(len(predicted_string_list_sorted) - correct_count, 0)\nAfter
\n for ground_truth_list, predicted_string_list in zip(y_true, y_pred):\n predicted_string_list_sorted = sorted(predicted_string_list)\n correct_prediction = []\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n correct_prediction.append(predicted_string_list_sorted[matched_idx])\n tp += len(correct_prediction)\n fp += max(len(predicted_string_list_sorted) - len(set(correct_prediction)), 0)\n@spshota Thank you for changing it again! Maybe you can also consider my comment to @sakami on the top of the comments, I think it also applies to your code.
", "rawMarkdown": "@spshota Thank you for changing it again! Maybe you can also consider my comment to @sakami on the top of the comments, I think it also applies to your code.", "votes": 1 }, { "id": 1275159, "postDate": "2021-04-16T02:38:22.763Z", "content": "@hannes82 Thank you for your comment.
\nI've fixed the code again.
Before
\nfor ground_truth_list, predicted_string_list in zip(y_true, y_pred):\n predicted_string_list_sorted = sorted(predicted_string_list)\n correct_prediction = []\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n correct_prediction.append(predicted_string_list_sorted[matched_idx])\n tp += len(correct_prediction)\n fp += max(len(predicted_string_list_sorted) - len(set(correct_prediction)), 0)\nAfter
\nfor ground_truth_list, predicted_string_list in zip(y_true, y_pred):\n predicted_string_list_sorted = sorted(predicted_string_list)\n correct_prediction = []\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n correct_prediction.append(predicted_string_list_sorted[matched_idx])\n else:\n fn += 1 \n tp += len(correct_prediction)\n fp += max(len(predicted_string_list_sorted) - len(set(correct_prediction)), 0)\n@spshota Great! This does the trick. Where I am not sure how the evaluation metric is set up is the following: Let's say two predictions match a given GT with a jaccard score >= 0.5, but can only one of them be matched with that GT?
\nThe code is written in this way now, but it is not clear from the evaluation page and unfortunately we still didn't get any clarification on this.
@spshota I think one prediction can only be matched with one GT.
\nI somehow missed this response:
\n\n", "rawMarkdown": "@spshota I think one prediction can only be matched with one GT.\n\nI somehow missed this response:\n> Hi! Good question. If your two predictions are both separate mentions of that dataset in the document in question, then you should make both predictions. The ground truth includes multiple different mentions of the same dataset in a document. As mentioned in the Evaluation page, multiple predictions are checked against each of the multiple ground truths, with the highest Jaccard score determining a \"match\", and a threshold determining whether that match is a true or false positive. So basically - those two mentions are potentially separate classes that just happen to point to the same dataset. If you think they both exist in the document, make that prediction.", "votes": 3 }, { "id": 1283964, "postDate": "2021-04-25T12:53:33.540Z", "content": "Hi! Good question. If your two predictions are both separate mentions of that dataset in the document in question, then you should make both predictions. The ground truth includes multiple different mentions of the same dataset in a document. As mentioned in the Evaluation page, multiple predictions are checked against each of the multiple ground truths, with the highest Jaccard score determining a \"match\", and a threshold determining whether that match is a true or false positive. So basically - those two mentions are potentially separate classes that just happen to point to the same dataset. If you think they both exist in the document, make that prediction.
\n
@hannes82 Thank you for your comment!
\nI also think one prediction can only be matched with one ground truth based on the results of the following little experiment.
I replaced the labels of prediction with the ones that had the most known labels with a jaccard score of 0.5 or higher.
\n(After the replacement, I made the labels unique to remove the same label.)
\nIf a prediction matched more than one ground truth, my score should remain the same.
\nBut It dropped drastically.
Is the conclusion that the code in the initial post for this topic most likely represents the scoring algorithm that Kaggle is using?
", "rawMarkdown": "Is the conclusion that the code in the initial post for this topic most likely represents the scoring algorithm that Kaggle is using?", "votes": 2 }, { "id": 1284280, "postDate": "2021-04-25T18:52:26.170Z", "content": "Yes exactly :-) Except that here it should say FN instead of FP in the last line, right?
\n\n", "rawMarkdown": "Yes exactly :-) Except that here it should say FN instead of FP in the last line, right?\n\n> if similarity_scores[matched_idx] >= 0.5:\n tp += 1\n else:\n fp += 1", "votes": 2 }, { "id": 1284419, "postDate": "2021-04-25T22:23:49.577Z", "content": "\nif similarity_scores[matched_idx] >= 0.5:\n tp += 1\n else:\n fp += 1\n
Yes, fp += 1 should be fn += 1 in the last line.
I think this code need a small adjustment.
\nThe most similar prediction which has the jaccard score < 0.5 should continue to be used in the next process in the loop.
if similarity_scores[matched_idx] >= 0.5:\n predicted_string_list_sorted.pop(matched_idx)\n tp += 1\n else:\n fn += 1\nYes you're right. Thanks!
", "rawMarkdown": "Yes you're right. Thanks!", "votes": 1 }, { "id": 1284481, "postDate": "2021-04-26T01:19:46.740Z", "content": "Thank you @hannes82 @spshota @trentb, I corrected the parts you pointed out.
", "rawMarkdown": "Thank you @hannes82 @spshota @trentb, I corrected the parts you pointed out.", "votes": 4 }, { "id": 1284500, "postDate": "2021-04-26T02:24:51.690Z", "content": "\n\nThe prediction with the highest score for a given ground truth is matched with that ground truth
\n
After matched with GT, the prediction should be ignored even the Jaccard score < 0.5. matched here mean nearest but not mean Jaccard similarity >= 0.5.
\n\n\nAny matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are counted as true positives (TP), the remainder as false positives (FP).
\n
Matched pair (PRED, GT) is TP if its Jaccard similarity >= 0.5 else it's FP. The FN is only for the case len(GT_LIST) > len(PRED_LIST) then FN = len(GT_LIST) - len(PRED_LIST).
@dathudeptrai Thank you for pointing this out! I think you are right with your first point.
\nYour second point could also be the case if \"nearest predictions\" means that. But does that really make sense? A GT should also count as FN if no prediction with a Jaccard score >= 0.5 with that GT exists, shouldn't it?
", "rawMarkdown": "@dathudeptrai Thank you for pointing this out! I think you are right with your first point.\n\nYour second point could also be the case if \"nearest predictions\" means that. But does that really make sense? A GT should also count as FN if no prediction with a Jaccard score >= 0.5 with that GT exists, shouldn't it?" }, { "id": 1284961, "postDate": "2021-04-26T13:02:49.277Z", "content": "@dathudeptrai I also think you are right with your first point.
\nRegarding your second point, I also think you are right based on the Evaluation description. I think a case could be made for counting these cases as FNs (as @hannes82 has done), but the simplest reading of the Evaluation description indicates these cases are counted as FPs.
\nWe could really use the examples that the Kaggle Staff have said they will provide to add clarity.
", "rawMarkdown": "@dathudeptrai I also think you are right with your first point.\n\nRegarding your second point, I also think you are right based on the Evaluation description. I think a case could be made for counting these cases as FNs (as @hannes82 has done), but the simplest reading of the Evaluation description indicates these cases are counted as FPs.\n\nWe could really use the examples that the Kaggle Staff have said they will provide to add clarity.", "votes": 1 }, { "id": 1285005, "postDate": "2021-04-26T13:43:42.210Z", "content": "I’d settle for a clear definition of what a “data set” is at this point.
", "rawMarkdown": "I’d settle for a clear definition of what a “data set” is at this point. \n\n", "votes": 3 }, { "id": 1285014, "postDate": "2021-04-26T13:54:24.497Z", "content": "", "rawMarkdown": "@leecming 😄" }, { "id": 1285027, "postDate": "2021-04-26T14:04:50.153Z", "content": "@leecming :))) It's hard to classify (organization, program, system, method, … ) and \"dataset citation\" even with my knowledge after reading a sentence :))).
", "rawMarkdown": "@leecming :))) It's hard to classify (organization, program, system, method, ... ) and \"dataset citation\" even with my knowledge after reading a sentence :))). " }, { "id": 1310031, "postDate": "2021-05-16T12:58:47.083Z", "content": "Hi I am still very confused about the description about the evaluation script. So with your metric script, a GT-Prediction pair has possibility to be both FP and FN?
", "rawMarkdown": "Hi I am still very confused about the description about the evaluation script. So with your metric script, a GT-Prediction pair has possibility to be both FP and FN?" }, { "id": 1313885, "postDate": "2021-05-18T19:09:28.420Z", "content": "@sakami so I am just confused because from your final evaluation description, a matched prediction with jaccard<0.5 is counted as both FN and FP?
", "rawMarkdown": "@sakami so I am just confused because from your final evaluation description, a matched prediction with jaccard<0.5 is counted as both FN and FP?" } ] }, { "id": 1262259, "postDate": "2021-04-04T03:09:46.213Z", "content": "~~This is wrong, I explained why in a previous post:
\nEdited given that the code in evaluation page has been fixed.
", "rawMarkdown": "~~This is wrong, I explained why in[ a previous post](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/228814):\n\n\nEdited given that the code in evaluation page has been fixed.", "replies": [ { "id": 1262957, "postDate": "2021-04-04T23:38:42.457Z", "content": "Thank you for your comments, CPMP!
\nThen, which is the correct implementation?
\nI took it from the evaluation page, and I think it produces the same result as your second python code.
No it does not produce the same as my second code. The difference shows when some words are repeated.
\nThe correct implementation is the C# one given by @philculliton in a comment there. The closest Python code is my second one, with sets everywhere. I say closest because there are some extra data prep steps in C# code,
", "rawMarkdown": "No it does not produce the same as my second code. The difference shows when some words are repeated.\n\nThe correct implementation is the C# one given by @philculliton in a comment there. The closest Python code is my second one, with sets everywhere. I say closest because there are some extra data prep steps in C# code," }, { "id": 1263575, "postDate": "2021-04-05T14:14:46.400Z", "content": "Hi @cpmpml - thanks! Under what conditions does it produce different results? Just looking for a repro sample so I can test. Your second example looks good, with the exception that .split(\" \") as used there will produce empty entries, where .split() will not.
I didn't noticed you change the code in the evaluation page. It looks good now indeed.
\nGiven I asked you and you did not respond I wrongly assumed you didn't answer my ask ;)
Thanks @cpmpml - as you can imagine, a lot of things to keep track of, just lost sight of answering that one! My fault.
", "rawMarkdown": "Thanks @cpmpml - as you can imagine, a lot of things to keep track of, just lost sight of answering that one! My fault.", "votes": 1 }, { "id": 1265411, "postDate": "2021-04-06T21:26:33.090Z", "content": "All that matters is that the code is now correct!
", "rawMarkdown": "All that matters is that the code is now correct!" } ] }, { "id": 1262201, "postDate": "2021-04-03T23:57:52.773Z", "content": "I only have a question:
\nwhy there are multiple strings-lists in one predicted_string_list?
You have to submit multiple predictions for each publication.
\nFor example,
Id,PredictionString\n000e04d6-d6ef-442f-b070-4309493221ba,space objects dataset|small objects data\n0176e38e-2286-4ea2-914f-0583808a98aa,small objects dataset\n...\nTherefore, in this case, the following code will work:
\ny_true = [['space objects dataset', 'small objects data'], ['small objects dataset'], ...]\ny_pred = [['space objects dataset', 'small objects data'], ['small objects dataset'], ...]\nfbeta_score = compute_fbeta(y_true, y_pred, beta=0.5)\nHi @sakami, I have seen many changes proposed in the comments. Is the code above the final version ?
", "votes": 2, "replies": [] }, { "id": 1264262, "author_name": "Carlos Sneyder Yepes Echeverri", "author_url": "", "post_date": "2021-04-06T03:13:46.543000", "content": "@philculliton I would like to know the implementation behind the Jaccard similarity. I made an experiment where I change the order of the words in one prediction for each publication in the test dataset, and the score was different(0.001 lower with fixed labels) and non of the Jaccard similarity function that I've seen get a different result when the words in a sentence have different order. The objective of the competition is to get new labels different than the ones fixed in the training data so I think is important to clarify if this new predictions have to come in an specific order
\nExample(code from @cpmpml ):
\ndef jaccard(s1, s2):\n l1 = set( s1.split(\" \"))\n l2 = set(s2.split(\" \")) \n intersection = len(l1.intersection(l2))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union
\njaccard('alzheimers disease neuroimaging initiative','initiative disease neuroimaging alzheimers')
\n1
But it seems that in the backend different order in the words results in a different outcome, or at least is what I got from the experiment, please give us an update on the evaluation score or let me know if somehow I am mistaken
", "votes": 2, "replies": [ { "id": 1265285, "author_name": "Phil Culliton", "author_url": "", "post_date": "2021-04-06T19:00:50.267000", "content": "The example code from CPMP does not remove empty entries, which is something the metric does. The example code on the evaluation page does this.
\nThe order of words is not a factor in Jaccard, but the order of the predictions could be a factor in the matching between predictions and ground truth. As mentioned on the Evaluation page, we order the prediction strings alphabetically for each sample, and then each is compared against all ground truth strings. It's possible that by reordering words you are changing that order for some predictions, which would change the evaluation order, and potentially impact scoring.
\nIt's recommended that you extract text directly from the documents as your predictions.
", "votes": 0, "replies": [] }, { "id": 1265320, "author_name": "CPMP", "author_url": "", "post_date": "2021-04-06T19:40:59.677000", "content": "\n\nThe example code from CPMP does not remove empty entries, which is something the metric does. > The example code on the evaluation page does this.
\n
@philculliton Can you expand on this, for instance with an example? I honestly don't understand what you mean. What is confusing also is that the python code on the evaluation page does not do the same thing as the C# code you shared recently.
", "votes": 0, "replies": [] }, { "id": 1265342, "author_name": "Phil Culliton", "author_url": "", "post_date": "2021-04-06T20:11:01.880000", "content": "@cpmpml - thanks! Yes. Your implementation splits on spaces (i.e. .split(\" \")) which produces empty string entries vs. the Evaluation version (which matches up with the C# metric's removal of empty entries) via .split(). Any strings with trailing / extra spaces will produce an empty entry in the sets in your code.
Would you mind providing an example where the C# example doesn't match up with the Python code? I walked through it with the other data scientists the other day and we couldn't see a gap, but we could definitely have missed something.
", "votes": 1, "replies": [] }, { "id": 1265349, "author_name": "CPMP", "author_url": "", "post_date": "2021-04-06T20:21:59.923000", "content": "Tx, I'll check where split() and split(' ') differ. But if all is needed is to replace split(' ') by split() in my code, then I'm more than happy to do it.
\nRe your question, I see you changed the code in the evaluation page without telling us… The code you have now is correct IMHO.
\nI thought we still had this code in the evaluation page:
\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len(list(set(l1).intersection(l2)))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\nI took split(' ') from this code actually.
Ahhh, okay! Yep - I changed that right away when you brought it up, failed to mention it explicitly in the thread.
\nThanks very much for digging through it with me. I think that replacing split(\" \") with split() should do the trick, as split() handles repeated separators (I'm not a super big fan of functions doing very different things implicitly, but there you have it).
@philculliton first of all, thank you for your commitment.
\nI would like to do the double confirmation regarding your below comment.
\n As mentioned on the Evaluation page, we order the prediction strings alphabetically for each sample
\nDoes it mean alphabetically reverse order, which means that z starting is first?
From the below snapshot, I guess it so. (Because s is alphabetically behind compared to n, but it comes first)
\nId,PredictionString
\n01e4e08c-ffea-45a7-adde-6a0c0ad755fc,space location data|national space objects|national space dataset
My other question is about the pipe (|) character. I already questioned in another discussion. But here looks the right place.
I have tested with the above string.
\nstr1 = \"space location data|national space objects|national space dataset\"\nset(str1.lower().split())\n{'data', 'dataset|space', 'location', 'national', 'objects|national', 'space'}\nI guess that the reason for split() is to get the meaningful word in any sentence.
\nBut for the pipe character, it gives not a meaningful word like 'dataset|space' from my point of view.
Cheers,
\nSONG
I have same two questions that Mr. SONG @dtmanager1979 asked, if anyone can help please!
", "votes": 0, "replies": [] } ] }, { "id": 1334551, "author_name": "GUNER", "author_url": "", "post_date": "2021-06-03T15:29:09.517000", "content": "Hi, late to the party but I believe there is a small issue with this scoring algoritm.
\nNamely, as the loop is constructed on 'ground_truth', it fails to find exact matching pairs, if an earlier ground_truth in the loop has a j-score above .5 for that prediction ('that prediction': the prediction that actually had an exact match (or a higher jaccard match)).
This might be more than a cosmetic issue if the exact match can not find a match in predictions, and the earlier match actually had another match above .5, although lower than the argmax.
\nI hope what I say makes sense.
\nFeel free to correct or challange this.
\nI have revised the scoring methodology but am cautious to share, because I want to make sure I am not missing something, before I stir up things :-)
I wanted to attach an example screenshot but kaggle currently doesnt support direct attachments.
\nSo here is the link if you are interested:[
\nhttps://drive.google.com/file/d/1zNtVzlS-UeyQ_15uTxJ1P5XlWzSR8sWm/view?usp=sharing
\n
\nCheers,
@sakami I think you are missing out to count the GTs as FN if they can't be matched to any predictions. Or am I missing something?
", "votes": 0, "replies": [] }, { "id": 1269062, "author_name": "Shota", "author_url": "", "post_date": "2021-04-10T05:36:39.470000", "content": "Thank you for sharing!
\nI found an implementation mistake.
\nIf the most similar in a loop of ground_truth_list of which similarity < 0.5, it's not used in a next process of the loop because it's popped before checking its similarity. \nThe most similar label should only be poped if its similarity >= 0.5.
Sorry, I was wrong.
\nAny prediction should not be poped because it may be matched more than one ground truth.
def compute_fbeta(y_true: List[List[str]],\n y_pred: List[List[str]],\n beta: float = 0.5) -> float:\n \"\"\"Compute the Jaccard-based micro FBeta score.\n\n References\n ----------\n - https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation\n \"\"\"\n\n def _jaccard_similarity(str1: str, str2: str) -> float:\n a = set(str1.split()) \n b = set(str2.split())\n c = a.intersection(b)\n return float(len(c)) / (len(a) + len(b) - len(c))\n\n tp = 0 # true positive\n fp = 0 # false positive\n fn = 0 # false negative\n for ground_truth_list, predicted_string_list in zip(y_true, y_pred):\n predicted_string_list_sorted = sorted(predicted_string_list)\n correct_prediction = []\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n correct_prediction.append(predicted_string_list_sorted[matched_idx])\n else:\n fn += 1\n tp += len(correct_prediction)\n fp += max(len(predicted_string_list_sorted) - len(set(correct_prediction)), 0)\n\n tp *= (1 + beta ** 2)\n fn *= beta ** 2\n fbeta_score = tp / (tp + fp + fn)\n return fbeta_score\nThank you for your comments, Shota!
\n@philculliton Could you tell us about the metric, is it possible for one predicted string to match more than one ground truth?
\nI agree with Shota, one prediction should be able to match multiple ground truth labels.
Thank you, sakami and Shota! I agree with you that one prediction should be able to match multiple ground truth labels. I'm not convinced that the description in Evaluation supports this though, but I find it hard to tell for sure. It would be ideal to see the evaluation code (regardless of which language it is written in) in order to fully verify the approach and a Python implementation of it.
", "votes": 4, "replies": [] }, { "id": 1270599, "author_name": "Hannes Öhler", "author_url": "", "post_date": "2021-04-11T19:33:37.560000", "content": "@spshota I also think that each prediction can be matched with more than one GT. Thanks for changing that in the code!
\nI might be wrong, but isn't it now possible that one prediction is counted more than once as FP in your code?
", "votes": 1, "replies": [] }, { "id": 1271153, "author_name": "Shota", "author_url": "", "post_date": "2021-04-12T10:54:42.930000", "content": "@hannes82 Thank you for your comment!
\nYou are right. I've fixed the code in a previous comment.
Before
\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n correct_count += 1\n else:\n fp += 1\n tp += correct_count\n fp += max(len(predicted_string_list_sorted) - correct_count, 0)\nAfter
\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n correct_count += 1\n tp += correct_count\n fp += max(len(predicted_string_list_sorted) - correct_count, 0)\n@spshota Your code might need another small adjustment: it subtracts a certain prediction more than once from your count of FP (in case the prediction matches with more than one GT).
", "votes": 1, "replies": [] }, { "id": 1273006, "author_name": "Shota", "author_url": "", "post_date": "2021-04-14T02:21:43.597000", "content": "@hannes82 Thank you.
\nI've fixed the code in a previous comment.
Before
\n for ground_truth_list, predicted_string_list in zip(y_true, y_pred):\n predicted_string_list_sorted = sorted(predicted_string_list)\n correct_count = 0\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n correct_count += 1\n tp += correct_count\n fp += max(len(predicted_string_list_sorted) - correct_count, 0)\nAfter
\n for ground_truth_list, predicted_string_list in zip(y_true, y_pred):\n predicted_string_list_sorted = sorted(predicted_string_list)\n correct_prediction = []\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n correct_prediction.append(predicted_string_list_sorted[matched_idx])\n tp += len(correct_prediction)\n fp += max(len(predicted_string_list_sorted) - len(set(correct_prediction)), 0)\n@spshota Thank you for changing it again! Maybe you can also consider my comment to @sakami on the top of the comments, I think it also applies to your code.
", "votes": 1, "replies": [] }, { "id": 1275159, "author_name": "Shota", "author_url": "", "post_date": "2021-04-16T02:38:22.763000", "content": "@hannes82 Thank you for your comment.
\nI've fixed the code again.
Before
\nfor ground_truth_list, predicted_string_list in zip(y_true, y_pred):\n predicted_string_list_sorted = sorted(predicted_string_list)\n correct_prediction = []\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n correct_prediction.append(predicted_string_list_sorted[matched_idx])\n tp += len(correct_prediction)\n fp += max(len(predicted_string_list_sorted) - len(set(correct_prediction)), 0)\nAfter
\nfor ground_truth_list, predicted_string_list in zip(y_true, y_pred):\n predicted_string_list_sorted = sorted(predicted_string_list)\n correct_prediction = []\n for ground_truth in sorted(ground_truth_list):\n if len(predicted_string_list_sorted) == 0:\n fn += 1\n else:\n similarity_scores = [\n _jaccard_similarity(ground_truth, predicted_string)\n for predicted_string in predicted_string_list_sorted\n ]\n matched_idx = np.argmax(similarity_scores)\n if similarity_scores[matched_idx] >= 0.5:\n correct_prediction.append(predicted_string_list_sorted[matched_idx])\n else:\n fn += 1 \n tp += len(correct_prediction)\n fp += max(len(predicted_string_list_sorted) - len(set(correct_prediction)), 0)\n@spshota Great! This does the trick. Where I am not sure how the evaluation metric is set up is the following: Let's say two predictions match a given GT with a jaccard score >= 0.5, but can only one of them be matched with that GT?
\nThe code is written in this way now, but it is not clear from the evaluation page and unfortunately we still didn't get any clarification on this.
@spshota I think one prediction can only be matched with one GT.
\nI somehow missed this response:
\n\n", "votes": 3, "replies": [] }, { "id": 1283964, "author_name": "Shota", "author_url": "", "post_date": "2021-04-25T12:53:33.540000", "content": "Hi! Good question. If your two predictions are both separate mentions of that dataset in the document in question, then you should make both predictions. The ground truth includes multiple different mentions of the same dataset in a document. As mentioned in the Evaluation page, multiple predictions are checked against each of the multiple ground truths, with the highest Jaccard score determining a \"match\", and a threshold determining whether that match is a true or false positive. So basically - those two mentions are potentially separate classes that just happen to point to the same dataset. If you think they both exist in the document, make that prediction.
\n
@hannes82 Thank you for your comment!
\nI also think one prediction can only be matched with one ground truth based on the results of the following little experiment.
I replaced the labels of prediction with the ones that had the most known labels with a jaccard score of 0.5 or higher.
\n(After the replacement, I made the labels unique to remove the same label.)
\nIf a prediction matched more than one ground truth, my score should remain the same.
\nBut It dropped drastically.
Is the conclusion that the code in the initial post for this topic most likely represents the scoring algorithm that Kaggle is using?
", "votes": 2, "replies": [] }, { "id": 1284280, "author_name": "Hannes Öhler", "author_url": "", "post_date": "2021-04-25T18:52:26.170000", "content": "Yes exactly :-) Except that here it should say FN instead of FP in the last line, right?
\n\n", "votes": 2, "replies": [] }, { "id": 1284419, "author_name": "Trent", "author_url": "", "post_date": "2021-04-25T22:23:49.577000", "content": "\nif similarity_scores[matched_idx] >= 0.5:\n tp += 1\n else:\n fp += 1\n
Yes, fp += 1 should be fn += 1 in the last line.
I think this code need a small adjustment.
\nThe most similar prediction which has the jaccard score < 0.5 should continue to be used in the next process in the loop.
if similarity_scores[matched_idx] >= 0.5:\n predicted_string_list_sorted.pop(matched_idx)\n tp += 1\n else:\n fn += 1\nYes you're right. Thanks!
", "votes": 1, "replies": [] }, { "id": 1284481, "author_name": "sakami", "author_url": "", "post_date": "2021-04-26T01:19:46.740000", "content": "Thank you @hannes82 @spshota @trentb, I corrected the parts you pointed out.
", "votes": 4, "replies": [] }, { "id": 1284500, "author_name": "Nguyen Quan Anh Minh", "author_url": "", "post_date": "2021-04-26T02:24:51.690000", "content": "\n\nThe prediction with the highest score for a given ground truth is matched with that ground truth
\n
After matched with GT, the prediction should be ignored even the Jaccard score < 0.5. matched here mean nearest but not mean Jaccard similarity >= 0.5.
\n\n\nAny matched predictions where the Jaccard score meets or exceeds the threshold of 0.5 are counted as true positives (TP), the remainder as false positives (FP).
\n
Matched pair (PRED, GT) is TP if its Jaccard similarity >= 0.5 else it's FP. The FN is only for the case len(GT_LIST) > len(PRED_LIST) then FN = len(GT_LIST) - len(PRED_LIST).
@dathudeptrai Thank you for pointing this out! I think you are right with your first point.
\nYour second point could also be the case if \"nearest predictions\" means that. But does that really make sense? A GT should also count as FN if no prediction with a Jaccard score >= 0.5 with that GT exists, shouldn't it?
", "votes": 0, "replies": [] }, { "id": 1284961, "author_name": "Trent", "author_url": "", "post_date": "2021-04-26T13:02:49.277000", "content": "@dathudeptrai I also think you are right with your first point.
\nRegarding your second point, I also think you are right based on the Evaluation description. I think a case could be made for counting these cases as FNs (as @hannes82 has done), but the simplest reading of the Evaluation description indicates these cases are counted as FPs.
\nWe could really use the examples that the Kaggle Staff have said they will provide to add clarity.
", "votes": 1, "replies": [] }, { "id": 1285005, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2021-04-26T13:43:42.210000", "content": "I’d settle for a clear definition of what a “data set” is at this point.
", "votes": 3, "replies": [] }, { "id": 1285014, "author_name": "Trent", "author_url": "", "post_date": "2021-04-26T13:54:24.497000", "content": "", "votes": 0, "replies": [] }, { "id": 1285027, "author_name": "Nguyen Quan Anh Minh", "author_url": "", "post_date": "2021-04-26T14:04:50.153000", "content": "@leecming :))) It's hard to classify (organization, program, system, method, … ) and \"dataset citation\" even with my knowledge after reading a sentence :))).
", "votes": 0, "replies": [] }, { "id": 1310031, "author_name": "SchenbergZ", "author_url": "", "post_date": "2021-05-16T12:58:47.083000", "content": "Hi I am still very confused about the description about the evaluation script. So with your metric script, a GT-Prediction pair has possibility to be both FP and FN?
", "votes": 0, "replies": [] }, { "id": 1313885, "author_name": "SchenbergZ", "author_url": "", "post_date": "2021-05-18T19:09:28.420000", "content": "@sakami so I am just confused because from your final evaluation description, a matched prediction with jaccard<0.5 is counted as both FN and FP?
", "votes": 0, "replies": [] } ] }, { "id": 1262259, "author_name": "CPMP", "author_url": "", "post_date": "2021-04-04T03:09:46.213000", "content": "~~This is wrong, I explained why in a previous post:
\nEdited given that the code in evaluation page has been fixed.
", "votes": 0, "replies": [ { "id": 1262957, "author_name": "sakami", "author_url": "", "post_date": "2021-04-04T23:38:42.457000", "content": "Thank you for your comments, CPMP!
\nThen, which is the correct implementation?
\nI took it from the evaluation page, and I think it produces the same result as your second python code.
No it does not produce the same as my second code. The difference shows when some words are repeated.
\nThe correct implementation is the C# one given by @philculliton in a comment there. The closest Python code is my second one, with sets everywhere. I say closest because there are some extra data prep steps in C# code,
", "votes": 0, "replies": [] }, { "id": 1263575, "author_name": "Phil Culliton", "author_url": "", "post_date": "2021-04-05T14:14:46.400000", "content": "Hi @cpmpml - thanks! Under what conditions does it produce different results? Just looking for a repro sample so I can test. Your second example looks good, with the exception that .split(\" \") as used there will produce empty entries, where .split() will not.
I didn't noticed you change the code in the evaluation page. It looks good now indeed.
\nGiven I asked you and you did not respond I wrongly assumed you didn't answer my ask ;)
Thanks @cpmpml - as you can imagine, a lot of things to keep track of, just lost sight of answering that one! My fault.
", "votes": 1, "replies": [] }, { "id": 1265411, "author_name": "CPMP", "author_url": "", "post_date": "2021-04-06T21:26:33.090000", "content": "All that matters is that the code is now correct!
", "votes": 0, "replies": [] } ] }, { "id": 1262201, "author_name": "SchenbergZ", "author_url": "", "post_date": "2021-04-03T23:57:52.773000", "content": "I only have a question:
\nwhy there are multiple strings-lists in one predicted_string_list?
You have to submit multiple predictions for each publication.
\nFor example,
Id,PredictionString\n000e04d6-d6ef-442f-b070-4309493221ba,space objects dataset|small objects data\n0176e38e-2286-4ea2-914f-0583808a98aa,small objects dataset\n...\nTherefore, in this case, the following code will work:
\ny_true = [['space objects dataset', 'small objects data'], ['small objects dataset'], ...]\ny_pred = [['space objects dataset', 'small objects data'], ['small objects dataset'], ...]\nfbeta_score = compute_fbeta(y_true, y_pred, beta=0.5)\n