{ "id": 248254, "title": "47th place solution - no training, no dataset label string matching", "url": "/competitions/coleridgeinitiative-show-us-the-data/writeups/tolgolas-47th-place-solution-no-training-no-datase", "author_name": "", "post_date": "2021-06-23T01:17:01.670Z", "votes": 28, "comment_count": 4, "views": 0, "content": "

Yes you read that right. All of the models I tried to train did worse than pre-trained models.

\n

Inspired by this publication about MRC for NER, I tested many question-answering models on the Hugging Face model hub to see which ones were able to extract the dataset name from a sentence. I ended up using an electra model trained on squad v2.

\n

I trained a distilbert, roberta, and fasttext model to be able to tell if a sentence contained a datatset, but sadly a regex search for words like \"Data\" or \"Survey\" did the best. I think the advantage of regex is that it has very high precision compared to the others. I'm sure that if I knew how to fine-tune a q-a model better I could have gone much higher up the leaderboard. Not bad considering it had no training…

\n

In brief

\n
    \n
  1. String match to find sentences mentioning dataset.
    2. \n
  2. Use question-answering model to extract name if probability above a certain threshold. e.g. \"What is the data source used?\"
    3. \n
\n

Pretty crazy that this helped me go up 1400 positions from public to private!

\n

Notebook here

", "messages": [ { "id": "1361627", "postDate": "06/23/2021 01:04:38", "content": "

Yes you read that right. All of the models I tried to train did worse than pre-trained models.

\n

Inspired by this publication about MRC for NER, I tested many question-answering models on the Hugging Face model hub to see which ones were able to extract the dataset name from a sentence. I ended up using an electra model trained on squad v2.

\n

I trained a distilbert, roberta, and fasttext model to be able to tell if a sentence contained a datatset, but sadly a regex search for words like \"Data\" or \"Survey\" did the best. I think the advantage of regex is that it has very high precision compared to the others. I'm sure that if I knew how to fine-tune a q-a model better I could have gone much higher up the leaderboard. Not bad considering it had no training…

\n

In brief

\n
    \n
  1. String match to find sentences mentioning dataset.
    2. \n
  2. Use question-answering model to extract name if probability above a certain threshold. e.g. \"What is the data source used?\"
    3. \n
\n

Pretty crazy that this helped me go up 1400 positions from public to private!

\n

Notebook here

", "rawMarkdown": "Yes you read that right. All of the models I tried to train did worse than pre-trained models.\n\nInspired by [this publication about MRC for NER](https://arxiv.org/pdf/1910.11476.pdf), I tested many question-answering models on the [Hugging Face model hub](https://huggingface.co/models?pipeline_tag=question-answering) to see which ones were able to extract the dataset name from a sentence. I ended up using an [electra model trained on squad v2](https://huggingface.co/ahotrod/electra_large_discriminator_squad2_512?context=The+inverse+association+between+bilirubin+and+risk+of+coronary+disease+was+analyzed+in+the+European+study+%22Prospective+Epidemiological+Study+of+Myocardial+Infarction+PRIME%22+30.&question=What+is+the+name+of+study+used%3F).\n\nI trained a distilbert, roberta, and fasttext model to be able to tell if a sentence contained a datatset, but sadly a regex search for words like \"Data\" or \"Survey\" did the best. I think the advantage of regex is that it has very high precision compared to the others. I'm sure that if I knew how to fine-tune a q-a model better I could have gone much higher up the leaderboard. Not bad considering it had no training... \n\nIn brief\n1. String match to find sentences mentioning dataset.\n2. Use question-answering model to extract name if probability above a certain threshold. e.g. \"What is the data source used?\"\n\nPretty crazy that this helped me go up 1400 positions from public to private! \n\n[Notebook here](https://www.kaggle.com/nbroad/coleridge-regex-electra)", "votes": null }, { "id": "1361630", "postDate": "06/23/2021 01:08:57", "content": "

Very intuitive approach. Congratulations !!!

", "rawMarkdown": "Very intuitive approach. Congratulations !!!", "votes": null }, { "id": "1361632", "postDate": "06/23/2021 01:09:55", "content": "

Wow, I never thought of anything along these lines. A really simple structure but effective 🤣. Congrats!

", "rawMarkdown": "Wow, I never thought of anything along these lines. A really simple structure but effective 🤣. Congrats!", "votes": null }, { "id": "1361781", "postDate": "06/23/2021 04:10:33", "content": "

So stable performance

", "rawMarkdown": "So stable performance", "votes": null }, { "id": "1361994", "postDate": "06/23/2021 06:52:37", "content": "

fwiw, i spent a fair amount of time trying to use QA models (fine-tuned and frozen) without any luck

", "rawMarkdown": "fwiw, i spent a fair amount of time trying to use QA models (fine-tuned and frozen) without any luck", "votes": null } ], "comments": [ { "id": 1361630, "author_name": "kartikbhargav", "author_url": "", "post_date": "06/23/2021 01:08:57", "content": "

Very intuitive approach. Congratulations !!!

", "votes": null, "replies": [] }, { "id": 1361632, "author_name": "toxicmaze", "author_url": "", "post_date": "06/23/2021 01:09:55", "content": "

Wow, I never thought of anything along these lines. A really simple structure but effective 🤣. Congrats!

", "votes": null, "replies": [] }, { "id": 1361781, "author_name": "yus002", "author_url": "", "post_date": "06/23/2021 04:10:33", "content": "

So stable performance

", "votes": null, "replies": [] }, { "id": 1361994, "author_name": "leecming", "author_url": "", "post_date": "06/23/2021 06:52:37", "content": "

fwiw, i spent a fair amount of time trying to use QA models (fine-tuned and frozen) without any luck

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1361627": "Yes you read that right. All of the models I tried to train did worse than pre-trained models.\n\nInspired by [this publication about MRC for NER](https://arxiv.org/pdf/1910.11476.pdf), I tested many question-answering models on the [Hugging Face model hub](https://huggingface.co/models?pipeline_tag=question-answering) to see which ones were able to extract the dataset name from a sentence. I ended up using an [electra model trained on squad v2](https://huggingface.co/ahotrod/electra_large_discriminator_squad2_512?context=The+inverse+association+between+bilirubin+and+risk+of+coronary+disease+was+analyzed+in+the+European+study+%22Prospective+Epidemiological+Study+of+Myocardial+Infarction+PRIME%22+30.&question=What+is+the+name+of+study+used%3F).\n\nI trained a distilbert, roberta, and fasttext model to be able to tell if a sentence contained a datatset, but sadly a regex search for words like \"Data\" or \"Survey\" did the best. I think the advantage of regex is that it has very high precision compared to the others. I'm sure that if I knew how to fine-tune a q-a model better I could have gone much higher up the leaderboard. Not bad considering it had no training... \n\nIn brief\n1. String match to find sentences mentioning dataset.\n2. Use question-answering model to extract name if probability above a certain threshold. e.g. \"What is the data source used?\"\n\nPretty crazy that this helped me go up 1400 positions from public to private! \n\n[Notebook here](https://www.kaggle.com/nbroad/coleridge-regex-electra)", "1361630": "Very intuitive approach. Congratulations !!!", "1361632": "Wow, I never thought of anything along these lines. A really simple structure but effective 🤣. Congrats!", "1361781": "So stable performance", "1361994": "fwiw, i spent a fair amount of time trying to use QA models (fine-tuned and frozen) without any luck" }, "source": "meta" }