{
  "id": 231977,
  "title": "Pipe Operator Problem for Jaccard Score",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/231977",
  "author_name": "",
  "post_date": "2021-04-11T15:24:43.044576500Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Dear all, </p>\n<p>regarding Jaccard-Score, I doubt that if the number of predicted dataset-label (PredictionString) is more than 2, there might be a problem. </p>\n<p>Here is my trial which is copied snapshot from Evaluation of Overview</p>\n<p>Id,PredictionString<br>\n01e4e08c-ffea-45a7-adde-6a0c0ad755fc,space location data|national space objects|national space dataset</p>\n<p>With this, I have tested with below two string.  Both are identical, only order matters.<br>\nstr1 = \"space location data|national space objects|national space dataset\"<br>\nstr2 = \"national space objects|national space dataset|space location data\"</p>\n<p>I assumed str1 is real, and str2 is predicted dataset-label (PredictionString).<br>\nAnd if I understood correctly, the Jaccard-Score should be 1 only when the order is different but identical. But the jaccard score is only 0.375 for the above case.</p>\n<p>And the main reason is as below.  <br>\nset(str1.lower().split()) = {'dataset', 'data|national', 'location', 'objects|national', 'space'}<br>\nset(str2.lower().split()) = {'data', 'dataset|space', 'location', 'national', 'objects|national', 'space'}<br>\nActually, Pipe Operator seems to be a not nice application for the split of the string.</p>\n<p>Do I understand wrongly? Is there any suggestion for it?</p>\n<p>Cheers,<br>\nSong</p>",
  "messages": [
    {
      "id": "1270370",
      "postDate": "04/11/2021 15:24:43",
      "content": "<p>Dear all, </p>\n<p>regarding Jaccard-Score, I doubt that if the number of predicted dataset-label (PredictionString) is more than 2, there might be a problem. </p>\n<p>Here is my trial which is copied snapshot from Evaluation of Overview</p>\n<p>Id,PredictionString<br>\n01e4e08c-ffea-45a7-adde-6a0c0ad755fc,space location data|national space objects|national space dataset</p>\n<p>With this, I have tested with below two string.  Both are identical, only order matters.<br>\nstr1 = \"space location data|national space objects|national space dataset\"<br>\nstr2 = \"national space objects|national space dataset|space location data\"</p>\n<p>I assumed str1 is real, and str2 is predicted dataset-label (PredictionString).<br>\nAnd if I understood correctly, the Jaccard-Score should be 1 only when the order is different but identical. But the jaccard score is only 0.375 for the above case.</p>\n<p>And the main reason is as below.  <br>\nset(str1.lower().split()) = {'dataset', 'data|national', 'location', 'objects|national', 'space'}<br>\nset(str2.lower().split()) = {'data', 'dataset|space', 'location', 'national', 'objects|national', 'space'}<br>\nActually, Pipe Operator seems to be a not nice application for the split of the string.</p>\n<p>Do I understand wrongly? Is there any suggestion for it?</p>\n<p>Cheers,<br>\nSong</p>",
      "rawMarkdown": "Dear all, \n\nregarding Jaccard-Score, I doubt that if the number of predicted dataset-label (PredictionString) is more than 2, there might be a problem. \n\nHere is my trial which is copied snapshot from Evaluation of Overview\n\nId,PredictionString\n01e4e08c-ffea-45a7-adde-6a0c0ad755fc,space location data|national space objects|national space dataset\n\nWith this, I have tested with below two string.  Both are identical, only order matters.\nstr1 = \"space location data|national space objects|national space dataset\"\nstr2 = \"national space objects|national space dataset|space location data\"\n\nI assumed str1 is real, and str2 is predicted dataset-label (PredictionString).\nAnd if I understood correctly, the Jaccard-Score should be 1 only when the order is different but identical. But the jaccard score is only 0.375 for the above case.\n\nAnd the main reason is as below.  \nset(str1.lower().split()) = {'dataset', 'data|national', 'location', 'objects|national', 'space'}\nset(str2.lower().split()) = {'data', 'dataset|space', 'location', 'national', 'objects|national', 'space'}\nActually, Pipe Operator seems to be a not nice application for the split of the string.\n\nDo I understand wrongly? Is there any suggestion for it?\n\nCheers,\nSong",
      "votes": null
    },
    {
      "id": "1270412",
      "postDate": "04/11/2021 16:18:32",
      "content": "<p>SONG, the Jaccard similarity function does not see pipe characters because there is code surrounding the call of the Jaccard similarity function (that has not been provided to us) that splits strings on the pipe characters before doing pairwise comparisons. In your example, pairwise calls will be made (e.g., jaccard_similarity(\"space location data\", \"national space objects\"), jaccard_similarity(\"space location data\", \"national space dataset\"), jaccard_similarity(\"national space objects\", \"national space objects\"), etc.), and then an F0.5 score will be determined from those results. There are still some questions about the exact, full evaluation algorithm, but a discussion about a Python implementation of the algorithm is <a href=\"https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230091\" target=\"_blank\">here</a>.</p>",
      "rawMarkdown": "SONG, the Jaccard similarity function does not see pipe characters because there is code surrounding the call of the Jaccard similarity function (that has not been provided to us) that splits strings on the pipe characters before doing pairwise comparisons. In your example, pairwise calls will be made (e.g., jaccard_similarity(\"space location data\", \"national space objects\"), jaccard_similarity(\"space location data\", \"national space dataset\"), jaccard_similarity(\"national space objects\", \"national space objects\"), etc.), and then an F0.5 score will be determined from those results. There are still some questions about the exact, full evaluation algorithm, but a discussion about a Python implementation of the algorithm is [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230091).",
      "votes": null
    },
    {
      "id": "1270591",
      "postDate": "04/11/2021 19:21:56",
      "content": "<p>Trent, thank you for your guide. I followed the given link from you and started a new discussion. <br>\nIn that link, I have already found a very meaningful discussion. Thank you again.</p>\n<p>Cheers,<br>\nSONG</p>",
      "rawMarkdown": "Trent, thank you for your guide. I followed the given link from you and started a new discussion. \nIn that link, I have already found a very meaningful discussion. Thank you again.\n\nCheers,\nSONG",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1270412,
      "author_name": "trentb",
      "author_url": "",
      "post_date": "04/11/2021 16:18:32",
      "content": "<p>SONG, the Jaccard similarity function does not see pipe characters because there is code surrounding the call of the Jaccard similarity function (that has not been provided to us) that splits strings on the pipe characters before doing pairwise comparisons. In your example, pairwise calls will be made (e.g., jaccard_similarity(\"space location data\", \"national space objects\"), jaccard_similarity(\"space location data\", \"national space dataset\"), jaccard_similarity(\"national space objects\", \"national space objects\"), etc.), and then an F0.5 score will be determined from those results. There are still some questions about the exact, full evaluation algorithm, but a discussion about a Python implementation of the algorithm is <a href=\"https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230091\" target=\"_blank\">here</a>.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1270591,
          "author_name": "dtmanager1979",
          "author_url": "",
          "post_date": "04/11/2021 19:21:56",
          "content": "<p>Trent, thank you for your guide. I followed the given link from you and started a new discussion. <br>\nIn that link, I have already found a very meaningful discussion. Thank you again.</p>\n<p>Cheers,<br>\nSONG</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1270370": "Dear all, \n\nregarding Jaccard-Score, I doubt that if the number of predicted dataset-label (PredictionString) is more than 2, there might be a problem. \n\nHere is my trial which is copied snapshot from Evaluation of Overview\n\nId,PredictionString\n01e4e08c-ffea-45a7-adde-6a0c0ad755fc,space location data|national space objects|national space dataset\n\nWith this, I have tested with below two string.  Both are identical, only order matters.\nstr1 = \"space location data|national space objects|national space dataset\"\nstr2 = \"national space objects|national space dataset|space location data\"\n\nI assumed str1 is real, and str2 is predicted dataset-label (PredictionString).\nAnd if I understood correctly, the Jaccard-Score should be 1 only when the order is different but identical. But the jaccard score is only 0.375 for the above case.\n\nAnd the main reason is as below.  \nset(str1.lower().split()) = {'dataset', 'data|national', 'location', 'objects|national', 'space'}\nset(str2.lower().split()) = {'data', 'dataset|space', 'location', 'national', 'objects|national', 'space'}\nActually, Pipe Operator seems to be a not nice application for the split of the string.\n\nDo I understand wrongly? Is there any suggestion for it?\n\nCheers,\nSong",
    "1270412": "SONG, the Jaccard similarity function does not see pipe characters because there is code surrounding the call of the Jaccard similarity function (that has not been provided to us) that splits strings on the pipe characters before doing pairwise comparisons. In your example, pairwise calls will be made (e.g., jaccard_similarity(\"space location data\", \"national space objects\"), jaccard_similarity(\"space location data\", \"national space dataset\"), jaccard_similarity(\"national space objects\", \"national space objects\"), etc.), and then an F0.5 score will be determined from those results. There are still some questions about the exact, full evaluation algorithm, but a discussion about a Python implementation of the algorithm is [here](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion/230091).",
    "1270591": "Trent, thank you for your guide. I followed the given link from you and started a new discussion. \nIn that link, I have already found a very meaningful discussion. Thank you again.\n\nCheers,\nSONG"
  },
  "source": "meta"
}