@philculliton the code in the evaluation tab is
\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len(list(set(l1).intersection(l2)))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\nDuplicates are removed to compute intersection, but they are not removed to compute union.
\nI would either remove the call to set, as in this,
\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len([s for s in l1 if s in l2])\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\nor use sets everywhere as in this
\ndef jaccard_similarity(s1, s2):\n l1 = set( s1.split(\" \"))\n l2 = set(s2.split(\" \")) \n intersection = len(l1.intersection(l2))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\nPlease let us know which implementation is used.
\nEdited after @aerdem4 comment.
\nEdit2: The code has been fixed in the evaluation tab. It is similar to my second one.
", "messages": [ { "id": "1253355", "postDate": "03/26/2021 16:31:08", "content": "@philculliton the code in the evaluation tab is
\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len(list(set(l1).intersection(l2)))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\nDuplicates are removed to compute intersection, but they are not removed to compute union.
\nI would either remove the call to set, as in this,
\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len([s for s in l1 if s in l2])\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\nor use sets everywhere as in this
\ndef jaccard_similarity(s1, s2):\n l1 = set( s1.split(\" \"))\n l2 = set(s2.split(\" \")) \n intersection = len(l1.intersection(l2))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\nPlease let us know which implementation is used.
\nEdited after @aerdem4 comment.
\nEdit2: The code has been fixed in the evaluation tab. It is similar to my second one.
", "rawMarkdown": "philculliton the code in the evaluation tab is\n\n```\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len(list(set(l1).intersection(l2)))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\n```\n\nDuplicates are removed to compute intersection, but they are not removed to compute union. \n\nI would either remove the call to set, as in this,\n\n```\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len([s for s in l1 if s in l2])\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\n```\n\nor use sets everywhere as in this\n\n```\ndef jaccard_similarity(s1, s2):\n l1 = set( s1.split(\" \"))\n l2 = set(s2.split(\" \")) \n intersection = len(l1.intersection(l2))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\n```\n\nPlease let us know which implementation is used.\n\nEdited after @aerdem4 comment.\n\nEdit2: The code has been fixed in the evaluation tab. It is similar to my second one.", "votes": null }, { "id": "1253363", "postDate": "03/26/2021 16:41:22", "content": "I think it is symmetrical but wrong. So intersection is correct, intersection function casts list to set and intersects but union part is wrong. [a, a] and [a, a] won't have perfect score as expected.
\nA unit test could prevent that. I recommend Kaggle Team and everyone writing unit tests.
", "rawMarkdown": "I think it is symmetrical but wrong. So intersection is correct, intersection function casts list to set and intersects but union part is wrong. [a, a] and [a, a] won't have perfect score as expected.\n\nA unit test could prevent that. I recommend Kaggle Team and everyone writing unit tests.", "votes": null }, { "id": "1253368", "postDate": "03/26/2021 16:46:29", "content": "Tx, updating the post.
\nWhat is weird is that Tweet Sentiment used a correct code for Jaccard:
\ndef jaccard(str1, str2): \n a = set(str1.lower().split()) \n b = set(str2.lower().split())\n c = a.intersection(b)\n return float(len(c)) / (len(a) + len(b) - len(c))\nI wonder why it is not reused.
", "rawMarkdown": "Tx, updating the post.\n\nWhat is weird is that [Tweet Sentiment](https://www.kaggle.com/c/tweet-sentiment-extraction/overview/evaluation) used a correct code for Jaccard:\n\n```\ndef jaccard(str1, str2): \n a = set(str1.lower().split()) \n b = set(str2.lower().split())\n c = a.intersection(b)\n return float(len(c)) / (len(a) + len(b) - len(c))\n\n```\nI wonder why it is not reused.", "votes": null }, { "id": "1253401", "postDate": "03/26/2021 17:26:25", "content": "First it was wrong in tweet sentiment. I guess the re-used the wrong implementation :D
", "rawMarkdown": "First it was wrong in tweet sentiment. I guess the re-used the wrong implementation :D", "votes": null }, { "id": "1253420", "postDate": "03/26/2021 17:49:33", "content": "in Tweet Sentiment the Python code was correct from the beginning ;) It was the backend in C# that had issues. Here maybe it is the reverse?
", "rawMarkdown": "in Tweet Sentiment the Python code was correct from the beginning ;) It was the backend in C# that had issues. Here maybe it is the reverse?", "votes": null }, { "id": "1256069", "postDate": "03/29/2021 14:22:34", "content": "@cpmpml - Thanks! You're correct, the Python implementation from Tweet Sentiment should be correct with the exception of the analog for StringSplitOptions.RemoveEmptyEntries below. I've switched the two. I might also simply post the C# code in the Evaluation page, which I've extracted this time to make it easier to share. I'm including it here for your reference.
@christofhenkel - thanks, you're welcome to take a look as well!
\npublic double CalculateJaccard(string true_string, string predicted_string)\n {\n Func<string, string[]> split = s => s.Split(new[] { ' ', '\\t', '\\r', '\\n', '\\f' }, StringSplitOptions.RemoveEmptyEntries);\n\n var currentTrueSplit = new HashSet<string>(split(true_string.ToLower()));\n var currentPredictedSplit = new HashSet<string>(split(predicted_string.ToLower()));\n\n var intersectionSize = new HashSet<string>(currentTrueSplit.Intersect(currentPredictedSplit, StringComparer.OrdinalIgnoreCase)).Count();\n\n var a = (float)currentTrueSplit.Count();\n var b = (float)currentPredictedSplit.Count();\n var c = (float)intersectionSize;\n\n return c / (a + b - c);\n }\n@aerdem4 - thanks. Agreed re: unit tests. In general we have a number for our metrics, but as with all unit tests that are not exhaustive (i.e. TDD) they tend to focus on certain areas. We do our best.
", "rawMarkdown": "cpmpml - Thanks! You're correct, the Python implementation from Tweet Sentiment should be correct with the exception of the analog for `StringSplitOptions.RemoveEmptyEntries` below. I've switched the two. I might also simply post the C# code in the Evaluation page, which I've extracted this time to make it easier to share. I'm including it here for your reference.\n\n@christofhenkel - thanks, you're welcome to take a look as well!\n\n``` \npublic double CalculateJaccard(string true_string, string predicted_string)\n {\n Func@philculliton This maps to my second python code, with sets everywhere,
", "rawMarkdown": "philculliton This maps to my second python code, with sets everywhere,", "votes": null }, { "id": "1256853", "postDate": "03/30/2021 09:56:02", "content": "@philculliton IMHO you should update the evaluation tab as the Python code there is definitely wrong and does not reflect what is in the C# backend.
", "rawMarkdown": "philculliton IMHO you should update the evaluation tab as the Python code there is definitely wrong and does not reflect what is in the C# backend.", "votes": null }, { "id": "1263643", "postDate": "04/05/2021 15:07:17", "content": "Hi @philculliton and @cpmpml, I have no C# compiler at hand but doesn't this C# implementation crash with a \"division by zero\" if both arguments are empty strings? If so this could maybe present problems when there is both no label and no prediction. If both arguments are empty strings the python code used in the tweet sentiment mentioned above seems to crash anyway. In case both strings are empty wikipedia says the jaccard should be 1.
", "rawMarkdown": "Hi @philculliton and @cpmpml, I have no C# compiler at hand but doesn't this C# implementation crash with a \"division by zero\" if both arguments are empty strings? If so this could maybe present problems when there is both no label and no prediction. If both arguments are empty strings the python code used in the tweet sentiment mentioned above seems to crash anyway. In case both strings are empty [wikipedia](https://en.wikipedia.org/wiki/Jaccard_index) says the jaccard should be 1.", "votes": null }, { "id": "1263684", "postDate": "04/05/2021 15:42:23", "content": "Don't ask me about this code ;)
\nBut yes, all shown implementations have issues when ground truth is empty. I therefore assume that all text have at least one mention…
", "rawMarkdown": "Don't ask me about this code ;)\n\nBut yes, all shown implementations have issues when ground truth is empty. I therefore assume that all text have at least one mention...", "votes": null }, { "id": "1264909", "postDate": "04/06/2021 13:52:39", "content": "@cpmpml and @julesvanligtenberg - that's correct. There are no empty ground truths in this competition. The handling of empty ground truths / predictions is handled outside of this function.
", "rawMarkdown": "cpmpml and @julesvanligtenberg - that's correct. There are no empty ground truths in this competition. The handling of empty ground truths / predictions is handled outside of this function.", "votes": null }, { "id": "1265063", "postDate": "04/06/2021 15:42:20", "content": "@philculliton Thanks for confirming.
", "rawMarkdown": "philculliton Thanks for confirming.", "votes": null }, { "id": "1265299", "postDate": "04/06/2021 19:18:40", "content": "@philculliton and @cpmpml Thanks!
", "rawMarkdown": "philculliton and @cpmpml Thanks!", "votes": null } ], "comments": [ { "id": 1253363, "author_name": "aerdem4", "author_url": "", "post_date": "03/26/2021 16:41:22", "content": "I think it is symmetrical but wrong. So intersection is correct, intersection function casts list to set and intersects but union part is wrong. [a, a] and [a, a] won't have perfect score as expected.
\nA unit test could prevent that. I recommend Kaggle Team and everyone writing unit tests.
", "votes": null, "replies": [ { "id": 1253368, "author_name": "cpmpml", "author_url": "", "post_date": "03/26/2021 16:46:29", "content": "Tx, updating the post.
\nWhat is weird is that Tweet Sentiment used a correct code for Jaccard:
\ndef jaccard(str1, str2): \n a = set(str1.lower().split()) \n b = set(str2.lower().split())\n c = a.intersection(b)\n return float(len(c)) / (len(a) + len(b) - len(c))\nI wonder why it is not reused.
", "votes": null, "replies": [] }, { "id": 1253401, "author_name": "christofhenkel", "author_url": "", "post_date": "03/26/2021 17:26:25", "content": "First it was wrong in tweet sentiment. I guess the re-used the wrong implementation :D
", "votes": null, "replies": [] }, { "id": 1253420, "author_name": "cpmpml", "author_url": "", "post_date": "03/26/2021 17:49:33", "content": "in Tweet Sentiment the Python code was correct from the beginning ;) It was the backend in C# that had issues. Here maybe it is the reverse?
", "votes": null, "replies": [] }, { "id": 1256069, "author_name": "philculliton", "author_url": "", "post_date": "03/29/2021 14:22:34", "content": "@cpmpml - Thanks! You're correct, the Python implementation from Tweet Sentiment should be correct with the exception of the analog for StringSplitOptions.RemoveEmptyEntries below. I've switched the two. I might also simply post the C# code in the Evaluation page, which I've extracted this time to make it easier to share. I'm including it here for your reference.
@christofhenkel - thanks, you're welcome to take a look as well!
\npublic double CalculateJaccard(string true_string, string predicted_string)\n {\n Func<string, string[]> split = s => s.Split(new[] { ' ', '\\t', '\\r', '\\n', '\\f' }, StringSplitOptions.RemoveEmptyEntries);\n\n var currentTrueSplit = new HashSet<string>(split(true_string.ToLower()));\n var currentPredictedSplit = new HashSet<string>(split(predicted_string.ToLower()));\n\n var intersectionSize = new HashSet<string>(currentTrueSplit.Intersect(currentPredictedSplit, StringComparer.OrdinalIgnoreCase)).Count();\n\n var a = (float)currentTrueSplit.Count();\n var b = (float)currentPredictedSplit.Count();\n var c = (float)intersectionSize;\n\n return c / (a + b - c);\n }\n@aerdem4 - thanks. Agreed re: unit tests. In general we have a number for our metrics, but as with all unit tests that are not exhaustive (i.e. TDD) they tend to focus on certain areas. We do our best.
", "votes": null, "replies": [] }, { "id": 1256093, "author_name": "cpmpml", "author_url": "", "post_date": "03/29/2021 14:37:34", "content": "@philculliton This maps to my second python code, with sets everywhere,
", "votes": null, "replies": [] }, { "id": 1256853, "author_name": "cpmpml", "author_url": "", "post_date": "03/30/2021 09:56:02", "content": "@philculliton IMHO you should update the evaluation tab as the Python code there is definitely wrong and does not reflect what is in the C# backend.
", "votes": null, "replies": [] }, { "id": 1263643, "author_name": "julesvanligtenberg", "author_url": "", "post_date": "04/05/2021 15:07:17", "content": "Hi @philculliton and @cpmpml, I have no C# compiler at hand but doesn't this C# implementation crash with a \"division by zero\" if both arguments are empty strings? If so this could maybe present problems when there is both no label and no prediction. If both arguments are empty strings the python code used in the tweet sentiment mentioned above seems to crash anyway. In case both strings are empty wikipedia says the jaccard should be 1.
", "votes": null, "replies": [] }, { "id": 1263684, "author_name": "cpmpml", "author_url": "", "post_date": "04/05/2021 15:42:23", "content": "Don't ask me about this code ;)
\nBut yes, all shown implementations have issues when ground truth is empty. I therefore assume that all text have at least one mention…
", "votes": null, "replies": [] }, { "id": 1264909, "author_name": "philculliton", "author_url": "", "post_date": "04/06/2021 13:52:39", "content": "@cpmpml and @julesvanligtenberg - that's correct. There are no empty ground truths in this competition. The handling of empty ground truths / predictions is handled outside of this function.
", "votes": null, "replies": [] }, { "id": 1265063, "author_name": "cpmpml", "author_url": "", "post_date": "04/06/2021 15:42:20", "content": "@philculliton Thanks for confirming.
", "votes": null, "replies": [] }, { "id": 1265299, "author_name": "julesvanligtenberg", "author_url": "", "post_date": "04/06/2021 19:18:40", "content": "@philculliton and @cpmpml Thanks!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1253355": "philculliton the code in the evaluation tab is\n\n```\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len(list(set(l1).intersection(l2)))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\n```\n\nDuplicates are removed to compute intersection, but they are not removed to compute union. \n\nI would either remove the call to set, as in this,\n\n```\ndef jaccard_similarity(s1, s2):\n l1 = s1.split(\" \")\n l2 = s2.split(\" \") \n intersection = len([s for s in l1 if s in l2])\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\n```\n\nor use sets everywhere as in this\n\n```\ndef jaccard_similarity(s1, s2):\n l1 = set( s1.split(\" \"))\n l2 = set(s2.split(\" \")) \n intersection = len(l1.intersection(l2))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\n```\n\nPlease let us know which implementation is used.\n\nEdited after @aerdem4 comment.\n\nEdit2: The code has been fixed in the evaluation tab. It is similar to my second one.", "1253363": "I think it is symmetrical but wrong. So intersection is correct, intersection function casts list to set and intersects but union part is wrong. [a, a] and [a, a] won't have perfect score as expected.\n\nA unit test could prevent that. I recommend Kaggle Team and everyone writing unit tests.", "1253368": "Tx, updating the post.\n\nWhat is weird is that [Tweet Sentiment](https://www.kaggle.com/c/tweet-sentiment-extraction/overview/evaluation) used a correct code for Jaccard:\n\n```\ndef jaccard(str1, str2): \n a = set(str1.lower().split()) \n b = set(str2.lower().split())\n c = a.intersection(b)\n return float(len(c)) / (len(a) + len(b) - len(c))\n\n```\nI wonder why it is not reused.", "1253401": "First it was wrong in tweet sentiment. I guess the re-used the wrong implementation :D", "1253420": "in Tweet Sentiment the Python code was correct from the beginning ;) It was the backend in C# that had issues. Here maybe it is the reverse?", "1256069": "cpmpml - Thanks! You're correct, the Python implementation from Tweet Sentiment should be correct with the exception of the analog for `StringSplitOptions.RemoveEmptyEntries` below. I've switched the two. I might also simply post the C# code in the Evaluation page, which I've extracted this time to make it easier to share. I'm including it here for your reference.\n\n@christofhenkel - thanks, you're welcome to take a look as well!\n\n``` \npublic double CalculateJaccard(string true_string, string predicted_string)\n {\n Func