{
  "id": 248300,
  "title": "5th place solution",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/writeups/kramarenko-vladislav-5th-place-solution",
  "author_name": "",
  "post_date": "2021-06-23T05:58:00.423053300Z",
  "votes": 41,
  "comment_count": 9,
  "views": 0,
  "content": "<p>Thanks to the organizers for an interesting competition.</p>\n<h1><a href=\"https://www.kaggle.com/vlomme/coleridge-initiative-5th-place?scriptVersionId=66408633\" target=\"_blank\">My solution</a></h1>\n<p>I tried two approaches (no dataset label string matching)</p>\n<ul>\n<li>Regular expression - 0.5/0.56</li>\n<li>BERT NER - 0.37/0.52</li>\n</ul>\n<p>Ensemble, searching among known datasets and other approaches with BERT did not improve the results</p>\n<h2>Regular expressions</h2>\n<ul>\n<li>Looking for uppercase letters</li>\n<li>Looking for nearby words with these letters</li>\n<li>Just looking for words beginning with uppercase letters</li>\n<li>I memorize all found and search among other documents</li>\n</ul>\n<h2>BERT NER</h2>\n<ul>\n<li>I divide the document into sentences of 200-400 characters</li>\n<li>Select potential candidates</li>\n<li>I take 90% of sentences with tags and 10% without tags</li>\n<li>Use 3 classes - no class, first word and last word of dataset name</li>\n<li>Predict and select good candidates</li>\n</ul>",
  "messages": [
    {
      "id": "1361921",
      "postDate": "06/23/2021 05:58:00",
      "content": "<p>Thanks to the organizers for an interesting competition.</p>\n<h1><a href=\"https://www.kaggle.com/vlomme/coleridge-initiative-5th-place?scriptVersionId=66408633\" target=\"_blank\">My solution</a></h1>\n<p>I tried two approaches (no dataset label string matching)</p>\n<ul>\n<li>Regular expression - 0.5/0.56</li>\n<li>BERT NER - 0.37/0.52</li>\n</ul>\n<p>Ensemble, searching among known datasets and other approaches with BERT did not improve the results</p>\n<h2>Regular expressions</h2>\n<ul>\n<li>Looking for uppercase letters</li>\n<li>Looking for nearby words with these letters</li>\n<li>Just looking for words beginning with uppercase letters</li>\n<li>I memorize all found and search among other documents</li>\n</ul>\n<h2>BERT NER</h2>\n<ul>\n<li>I divide the document into sentences of 200-400 characters</li>\n<li>Select potential candidates</li>\n<li>I take 90% of sentences with tags and 10% without tags</li>\n<li>Use 3 classes - no class, first word and last word of dataset name</li>\n<li>Predict and select good candidates</li>\n</ul>",
      "rawMarkdown": "Thanks to the organizers for an interesting competition.\n\n# [My solution](https://www.kaggle.com/vlomme/coleridge-initiative-5th-place?scriptVersionId=66408633)\nI tried two approaches (no dataset label string matching)\n- Regular expression - 0.5/0.56\n- BERT NER - 0.37/0.52\n\nEnsemble, searching among known datasets and other approaches with BERT did not improve the results\n\n## Regular expressions\n- Looking for uppercase letters\n- Looking for nearby words with these letters\n- Just looking for words beginning with uppercase letters\n- I memorize all found and search among other documents\n\n## BERT NER\n- I divide the document into sentences of 200-400 characters\n- Select potential candidates\n- I take 90% of sentences with tags and 10% without tags\n- Use 3 classes - no class, first word and last word of dataset name\n- Predict and select good candidates",
      "votes": null
    },
    {
      "id": "1361941",
      "postDate": "06/23/2021 06:17:55",
      "content": "<p>Thanks for sharing the solution along with the code.</p>",
      "rawMarkdown": "Thanks for sharing the solution along with the code.",
      "votes": null
    },
    {
      "id": "1361984",
      "postDate": "06/23/2021 06:41:15",
      "content": "<p>Congrats on yet another solo gold!</p>",
      "rawMarkdown": "Congrats on yet another solo gold!",
      "votes": null
    },
    {
      "id": "1361987",
      "postDate": "06/23/2021 06:45:03",
      "content": "<p>Thanks. I'm sad that you didn't participate</p>",
      "rawMarkdown": "Thanks. I'm sad that you didn't participate",
      "votes": null
    },
    {
      "id": "1362012",
      "postDate": "06/23/2021 07:06:18",
      "content": "<p>Hey, congrats on gold! Had a quick look at the notebook you shared, came across the Regex portion where you used Levenshtein Distance for string similarity, is there a specific reason why you used this specific algorithm instead of any of its counterparts (Cosine Dist/Sim, Jaro, etc …)</p>",
      "rawMarkdown": "Hey, congrats on gold! Had a quick look at the notebook you shared, came across the Regex portion where you used Levenshtein Distance for string similarity, is there a specific reason why you used this specific algorithm instead of any of its counterparts (Cosine Dist/Sim, Jaro, etc ...)",
      "votes": null
    },
    {
      "id": "1362019",
      "postDate": "06/23/2021 07:09:55",
      "content": "<p>No. I just didn't want to repeat datasets. I think you can just delete it, the rating shouldn't go down</p>",
      "rawMarkdown": "No. I just didn't want to repeat datasets. I think you can just delete it, the rating shouldn't go down",
      "votes": null
    },
    {
      "id": "1362034",
      "postDate": "06/23/2021 07:16:58",
      "content": "<p>Ah cool. I saw the &lt;0.2 and I was a little confused. Then again, didn't really go into depth. Thanks man </p>",
      "rawMarkdown": "Ah cool. I saw the <0.2 and I was a little confused. Then again, didn't really go into depth. Thanks man",
      "votes": null
    },
    {
      "id": "1362064",
      "postDate": "06/23/2021 07:49:57",
      "content": "<p>Great method 👍, thanks a lot for the solution provided. The hand of the master is traced (regular expressions are our everything). Congratulations on the prize!💥</p>",
      "rawMarkdown": "Great method 👍, thanks a lot for the solution provided. The hand of the master is traced (regular expressions are our everything). Congratulations on the prize!💥",
      "votes": null
    },
    {
      "id": "1362707",
      "postDate": "06/23/2021 15:22:17",
      "content": "<p>create <a href=\"https://github.com/vlomme/Find-dataset-from-text\" target=\"_blank\">github</a></p>",
      "rawMarkdown": "create [github](https://github.com/vlomme/Find-dataset-from-text)",
      "votes": null
    },
    {
      "id": "1363636",
      "postDate": "06/24/2021 08:28:27",
      "content": "<p>Well, if I had received just one answer to my questions then I would have joined. It is good that in the end the LB isn't random.</p>\n<p>I am happy for you, getting two solo gold in unrelated domains is hard.</p>",
      "rawMarkdown": "Well, if I had received just one answer to my questions then I would have joined. It is good that in the end the LB isn't random.\n\n\nI am happy for you, getting two solo gold in unrelated domains is hard.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1361941,
      "author_name": "ssinyu",
      "author_url": "",
      "post_date": "06/23/2021 06:17:55",
      "content": "<p>Thanks for sharing the solution along with the code.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1361984,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "06/23/2021 06:41:15",
      "content": "<p>Congrats on yet another solo gold!</p>",
      "votes": null,
      "replies": [
        {
          "id": 1361987,
          "author_name": "vlomme",
          "author_url": "",
          "post_date": "06/23/2021 06:45:03",
          "content": "<p>Thanks. I'm sad that you didn't participate</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1363636,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "06/24/2021 08:28:27",
          "content": "<p>Well, if I had received just one answer to my questions then I would have joined. It is good that in the end the LB isn't random.</p>\n<p>I am happy for you, getting two solo gold in unrelated domains is hard.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1362012,
      "author_name": "toxicmaze",
      "author_url": "",
      "post_date": "06/23/2021 07:06:18",
      "content": "<p>Hey, congrats on gold! Had a quick look at the notebook you shared, came across the Regex portion where you used Levenshtein Distance for string similarity, is there a specific reason why you used this specific algorithm instead of any of its counterparts (Cosine Dist/Sim, Jaro, etc …)</p>",
      "votes": null,
      "replies": [
        {
          "id": 1362019,
          "author_name": "vlomme",
          "author_url": "",
          "post_date": "06/23/2021 07:09:55",
          "content": "<p>No. I just didn't want to repeat datasets. I think you can just delete it, the rating shouldn't go down</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1362034,
          "author_name": "toxicmaze",
          "author_url": "",
          "post_date": "06/23/2021 07:16:58",
          "content": "<p>Ah cool. I saw the &lt;0.2 and I was a little confused. Then again, didn't really go into depth. Thanks man </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1362064,
      "author_name": "kucherevskiy",
      "author_url": "",
      "post_date": "06/23/2021 07:49:57",
      "content": "<p>Great method 👍, thanks a lot for the solution provided. The hand of the master is traced (regular expressions are our everything). Congratulations on the prize!💥</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1362707,
      "author_name": "vlomme",
      "author_url": "",
      "post_date": "06/23/2021 15:22:17",
      "content": "<p>create <a href=\"https://github.com/vlomme/Find-dataset-from-text\" target=\"_blank\">github</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1361921": "Thanks to the organizers for an interesting competition.\n\n# [My solution](https://www.kaggle.com/vlomme/coleridge-initiative-5th-place?scriptVersionId=66408633)\nI tried two approaches (no dataset label string matching)\n- Regular expression - 0.5/0.56\n- BERT NER - 0.37/0.52\n\nEnsemble, searching among known datasets and other approaches with BERT did not improve the results\n\n## Regular expressions\n- Looking for uppercase letters\n- Looking for nearby words with these letters\n- Just looking for words beginning with uppercase letters\n- I memorize all found and search among other documents\n\n## BERT NER\n- I divide the document into sentences of 200-400 characters\n- Select potential candidates\n- I take 90% of sentences with tags and 10% without tags\n- Use 3 classes - no class, first word and last word of dataset name\n- Predict and select good candidates",
    "1361941": "Thanks for sharing the solution along with the code.",
    "1361984": "Congrats on yet another solo gold!",
    "1361987": "Thanks. I'm sad that you didn't participate",
    "1362012": "Hey, congrats on gold! Had a quick look at the notebook you shared, came across the Regex portion where you used Levenshtein Distance for string similarity, is there a specific reason why you used this specific algorithm instead of any of its counterparts (Cosine Dist/Sim, Jaro, etc ...)",
    "1362019": "No. I just didn't want to repeat datasets. I think you can just delete it, the rating shouldn't go down",
    "1362034": "Ah cool. I saw the <0.2 and I was a little confused. Then again, didn't really go into depth. Thanks man",
    "1362064": "Great method 👍, thanks a lot for the solution provided. The hand of the master is traced (regular expressions are our everything). Congratulations on the prize!💥",
    "1362707": "create [github](https://github.com/vlomme/Find-dataset-from-text)",
    "1363636": "Well, if I had received just one answer to my questions then I would have joined. It is good that in the end the LB isn't random.\n\n\nI am happy for you, getting two solo gold in unrelated domains is hard."
  },
  "source": "existing_discussion",
  "complete": true,
  "competition_id": 25925,
  "competition_slug": "coleridgeinitiative-show-us-the-data",
  "team_id": 6706875,
  "solution_url": "/competitions/coleridgeinitiative-show-us-the-data/writeups/kramarenko-vladislav-5th-place-solution"
}