Even though there was quite a large drop (-13) in private leaderboard, I got a nice score (maximum 0.410) through public leaderboard probing. I used simple string matching and RE matching. Although I contained BERT in my final submission, it didn't make a significant lift for me.
\nI googled the existing dataset, and cherry-picked the datasets which I think are likely to be contained in test data. Although the amount was small (about 2kb), I can make a huge lift-up in public leaderboard using this external dataset.
\nAt first I mapped the original tokens in the texts into a token type (acronym, a word that starts with capital letter, etc) through Regular Expression. Then I used RE again to find phrases which are likely to contain dataset names, and filtered the phrases based on the indicators (dataset, database, survey, etc).
\nThese LB probing indeed led to overfitting, however. It's clear that using proper ML models combined with these methods will make a better result. My notebook will be made public as soon as I check up the references and all other stuffs.
", "messages": [ { "id": 1420714, "postDate": "2021-08-03T03:29:16.307Z", "content": "Even though there was quite a large drop (-13) in private leaderboard, I got a nice score (maximum 0.410) through public leaderboard probing. I used simple string matching and RE matching. Although I contained BERT in my final submission, it didn't make a significant lift for me.
\nI googled the existing dataset, and cherry-picked the datasets which I think are likely to be contained in test data. Although the amount was small (about 2kb), I can make a huge lift-up in public leaderboard using this external dataset.
\nAt first I mapped the original tokens in the texts into a token type (acronym, a word that starts with capital letter, etc) through Regular Expression. Then I used RE again to find phrases which are likely to contain dataset names, and filtered the phrases based on the indicators (dataset, database, survey, etc).
\nThese LB probing indeed led to overfitting, however. It's clear that using proper ML models combined with these methods will make a better result. My notebook will be made public as soon as I check up the references and all other stuffs.
", "rawMarkdown": "Even though there was quite a large drop (-13) in private leaderboard, I got a nice score (maximum 0.410) through public leaderboard probing. I used simple string matching and RE matching. Although I contained BERT in my final submission, it didn't make a significant lift for me.\n\n### 1. Simple String Matching\nI googled the existing dataset, and cherry-picked the datasets which I think are likely to be contained in test data. Although the amount was small (about 2kb), I can make a huge lift-up in public leaderboard using this external dataset.\n\n### 2. Regular Expression Matching\nAt first I mapped the original tokens in the texts into a token type (acronym, a word that starts with capital letter, etc) through Regular Expression. Then I used RE again to find phrases which are likely to contain dataset names, and filtered the phrases based on the indicators (dataset, database, survey, etc).\n\nThese LB probing indeed led to overfitting, however. It's clear that using proper ML models combined with these methods will make a better result. My notebook will be made public as soon as I check up the references and all other stuffs.", "votes": 1 } ], "comments": [], "raw_markdown_by_id": { "1420714": "Even though there was quite a large drop (-13) in private leaderboard, I got a nice score (maximum 0.410) through public leaderboard probing. I used simple string matching and RE matching. Although I contained BERT in my final submission, it didn't make a significant lift for me.\n\n### 1. Simple String Matching\nI googled the existing dataset, and cherry-picked the datasets which I think are likely to be contained in test data. Although the amount was small (about 2kb), I can make a huge lift-up in public leaderboard using this external dataset.\n\n### 2. Regular Expression Matching\nAt first I mapped the original tokens in the texts into a token type (acronym, a word that starts with capital letter, etc) through Regular Expression. Then I used RE again to find phrases which are likely to contain dataset names, and filtered the phrases based on the indicators (dataset, database, survey, etc).\n\nThese LB probing indeed led to overfitting, however. It's clear that using proper ML models combined with these methods will make a better result. My notebook will be made public as soon as I check up the references and all other stuffs." } }