Thanks to Coleridge & Kaggle for organizing this competition, and congrats to the other prize winners and medallists. This was a challenging competition in contrast to the standard Kaggle NLP classification/regression task, and I hope Coleridge will consider sponsoring future competitions in this vein. I suspect a future iteration will have the kinks shaken out :)
\nCV strategy: It should be clear that there was no way to perform clean local validation. I used public LB while aware of its flaws.
\nI spent the first month of the competition working on an end-to-end Transformer solution with minimal text processing. However, it became clear that a tweaked Transformer approach was problematic - they’re a) unsuited for the long document sequences we’re dealing with, even with the specialized variants designed for such texts, and b) ended up requiring heavy pre/post-processing to perform well. My processing heuristics effectively turned into a model parallel to the main Transformer model. I began thinking about how to simplify this -
\nAny attempt to inject semantic context into my models seemed to worsen public LB accuracy. I suspected the annotation process was noisy and decided to focus on IDing datasets purely by title.
\nData sets in academic papers often have a regular structure - mixed cap words followed by an acronym. An example is “Baltimore Longitudinal Study of Aging (BLSA)”. I needed something lighter than a Transformer to search for them efficiently - the answer was the Schwartz-Hearst algorithm. Schwartz-Hearst is a non-learning string search algorithm used to extract strings from text in the form of “LONG-FORM (ACRONYM)” - it’s basically a more involved regexp search. A bonus from using this algorithm is that I could also use the acronym form of a given dataset for additional points.
\nThe third insight (hypothesis?) was that scraped data was unnecessary. I did not use any externally derived list of datasets or scraped data - my sense was that 1) positive examples of datasets weren’t as useful as negative examples, and 2) it wasn’t even clear whether these lists hewed to the dataset definition that annotators were using.
\nSchwartz-Hearst worked well at identifying named entities but there’d be many flagged candidates that weren’t data sets (e.g., organization names, scientific paraphernalia) so I needed a binary classifier to filter candidate strings. I hand-annotated (using my own judgment of what a dataset was) a miniature training corpus of positive and negative examples based on examples that Schwartz-Hearst IDed from training data and fine-tuned a roberta-base binary classifier to filter these named entities using the corpus. I’ll re-emphasize that this corpus was built wholly from the training data without any scraped data.
\nThe last piece of the puzzle was dealing with the fact that not all papers referred to datasets using the full “LONG-NAME (ACRONYM)” format. Many referenced only the LONG-FORM. A simple solution was to generate a dynamic table of datasets and search for LONG-FORM in documents. I noticed that thresholding this search on a minimum document frequency improved public LB. The intuition is that in any given document corpus, a dataset is probably referenced by multiple documents so the more frequently they’re referenced, the more likely they’re a hit and not a spurious find.
\nThis search + classify approach dramatically reduced inference time since I did not have to pass entire documents through a Transformer model. Inference on the test set took ~10 minutes vs >>1hr using end-to-end Transformer QA/NER models.
\nThe LONG-FORM (ACRONYM) string of “Singapore Longitudinal Study on Diabetes (SLSD)” is found in 64 documents
“Singapore Longitudinal Study on Diabetes” is classified by the Transformer binary classifier to be a dataset with probability 0.99, above the prediction threshold of 0.9
The document frequency of 64 exceeds the search threshold frequency of 50 - a search of LONG-NAME is conducted across all documents and another 200 documents are found to contain the label
For the combined 264 documents, the LONG-FORM is added as a prediction. Additionally, if any of the 264 documents contains the string ‘ SLSD ‘ (checked against the raw document texts), an additional prediction of the acronym is added
Note: this is a summary of the model logic, there are other rules applied that I’ve elided.
\nCode here
", "messages": [ { "id": 1361893, "postDate": "2021-06-23T05:38:40.840Z", "content": "Thanks to Coleridge & Kaggle for organizing this competition, and congrats to the other prize winners and medallists. This was a challenging competition in contrast to the standard Kaggle NLP classification/regression task, and I hope Coleridge will consider sponsoring future competitions in this vein. I suspect a future iteration will have the kinks shaken out :)
\nCV strategy: It should be clear that there was no way to perform clean local validation. I used public LB while aware of its flaws.
\nI spent the first month of the competition working on an end-to-end Transformer solution with minimal text processing. However, it became clear that a tweaked Transformer approach was problematic - they’re a) unsuited for the long document sequences we’re dealing with, even with the specialized variants designed for such texts, and b) ended up requiring heavy pre/post-processing to perform well. My processing heuristics effectively turned into a model parallel to the main Transformer model. I began thinking about how to simplify this -
\nAny attempt to inject semantic context into my models seemed to worsen public LB accuracy. I suspected the annotation process was noisy and decided to focus on IDing datasets purely by title.
\nData sets in academic papers often have a regular structure - mixed cap words followed by an acronym. An example is “Baltimore Longitudinal Study of Aging (BLSA)”. I needed something lighter than a Transformer to search for them efficiently - the answer was the Schwartz-Hearst algorithm. Schwartz-Hearst is a non-learning string search algorithm used to extract strings from text in the form of “LONG-FORM (ACRONYM)” - it’s basically a more involved regexp search. A bonus from using this algorithm is that I could also use the acronym form of a given dataset for additional points.
\nThe third insight (hypothesis?) was that scraped data was unnecessary. I did not use any externally derived list of datasets or scraped data - my sense was that 1) positive examples of datasets weren’t as useful as negative examples, and 2) it wasn’t even clear whether these lists hewed to the dataset definition that annotators were using.
\nSchwartz-Hearst worked well at identifying named entities but there’d be many flagged candidates that weren’t data sets (e.g., organization names, scientific paraphernalia) so I needed a binary classifier to filter candidate strings. I hand-annotated (using my own judgment of what a dataset was) a miniature training corpus of positive and negative examples based on examples that Schwartz-Hearst IDed from training data and fine-tuned a roberta-base binary classifier to filter these named entities using the corpus. I’ll re-emphasize that this corpus was built wholly from the training data without any scraped data.
\nThe last piece of the puzzle was dealing with the fact that not all papers referred to datasets using the full “LONG-NAME (ACRONYM)” format. Many referenced only the LONG-FORM. A simple solution was to generate a dynamic table of datasets and search for LONG-FORM in documents. I noticed that thresholding this search on a minimum document frequency improved public LB. The intuition is that in any given document corpus, a dataset is probably referenced by multiple documents so the more frequently they’re referenced, the more likely they’re a hit and not a spurious find.
\nThis search + classify approach dramatically reduced inference time since I did not have to pass entire documents through a Transformer model. Inference on the test set took ~10 minutes vs >>1hr using end-to-end Transformer QA/NER models.
\nThe LONG-FORM (ACRONYM) string of “Singapore Longitudinal Study on Diabetes (SLSD)” is found in 64 documents
“Singapore Longitudinal Study on Diabetes” is classified by the Transformer binary classifier to be a dataset with probability 0.99, above the prediction threshold of 0.9
The document frequency of 64 exceeds the search threshold frequency of 50 - a search of LONG-NAME is conducted across all documents and another 200 documents are found to contain the label
For the combined 264 documents, the LONG-FORM is added as a prediction. Additionally, if any of the 264 documents contains the string ‘ SLSD ‘ (checked against the raw document texts), an additional prediction of the acronym is added
Note: this is a summary of the model logic, there are other rules applied that I’ve elided.
\nCode here
", "rawMarkdown": "Thanks to Coleridge & Kaggle for organizing this competition, and congrats to the other prize winners and medallists. This was a challenging competition in contrast to the standard Kaggle NLP classification/regression task, and I hope Coleridge will consider sponsoring future competitions in this vein. I suspect a future iteration will have the kinks shaken out :)\n\n## TL;DR: Search + Classify\n\n1. Search for named entities using the Schwartz-Hearst algorithm\n2. Filter candidates using a fine-tuned roberta-base binary classifier\n3. Threshold and propagate candidates\n\nCV strategy: It should be clear that there was no way to perform clean local validation. I used public LB while aware of its flaws. \n\nI spent the first month of the competition working on an end-to-end Transformer solution with minimal text processing. However, it became clear that a tweaked Transformer approach was problematic - they’re a) unsuited for the long document sequences we’re dealing with, even with the specialized variants designed for such texts, and b) ended up requiring heavy pre/post-processing to perform well. My processing heuristics effectively turned into a model parallel to the main Transformer model. I began thinking about how to simplify this - \n\n#### Insight #1: No context needed\nAny attempt to inject semantic context into my models seemed to worsen public LB accuracy. I suspected the annotation process was noisy and decided to focus on IDing datasets purely by title. \n\n#### Insight #2: Search for LONG-FORM (ACRONYM) \nData sets in academic papers often have a regular structure - mixed cap words followed by an acronym. An example is “Baltimore Longitudinal Study of Aging (BLSA)”. I needed something lighter than a Transformer to search for them efficiently - the answer was the [Schwartz-Hearst algorithm](https://psb.stanford.edu/psb-online/proceedings/psb03/schwartz.pdf). Schwartz-Hearst is a non-learning string search algorithm used to extract strings from text in the form of “LONG-FORM (ACRONYM)” - it’s basically a more involved regexp search. A bonus from using this algorithm is that I could also use the acronym form of a given dataset for additional points. \n\n#### Insight #3: No scraped data needed\nThe third insight (hypothesis?) was that scraped data was unnecessary. I did not use any externally derived list of datasets or scraped data - my sense was that 1) positive examples of datasets weren’t as useful as negative examples, and 2) it wasn’t even clear whether these lists hewed to the dataset definition that annotators were using. \n\n#### Putting it all together\nSchwartz-Hearst worked well at identifying named entities but there’d be many flagged candidates that weren’t data sets (e.g., organization names, scientific paraphernalia) so I needed a binary classifier to filter candidate strings. I hand-annotated (using my own judgment of what a dataset was) a miniature training corpus of positive and negative examples based on examples that Schwartz-Hearst IDed from training data and fine-tuned a roberta-base binary classifier to filter these named entities using the corpus. I’ll re-emphasize that this corpus was built wholly from the training data without any scraped data. \n\nThe last piece of the puzzle was dealing with the fact that not all papers referred to datasets using the full “LONG-NAME (ACRONYM)” format. Many referenced only the LONG-FORM. A simple solution was to generate a dynamic table of datasets and search for LONG-FORM in documents. I noticed that thresholding this search on a minimum document frequency improved public LB. The intuition is that in any given document corpus, a dataset is probably referenced by multiple documents so the more frequently they’re referenced, the more likely they’re a hit and not a spurious find. \n\nThis search + classify approach dramatically reduced inference time since I did not have to pass entire documents through a Transformer model. Inference on the test set took ~10 minutes vs >>1hr using end-to-end Transformer QA/NER models. \n\n## End-to-end dummy example:\n1. The LONG-FORM (ACRONYM) string of “Singapore Longitudinal Study on Diabetes (SLSD)” is found in 64 documents\n\n2. “Singapore Longitudinal Study on Diabetes” is classified by the Transformer binary classifier to be a dataset with probability 0.99, above the prediction threshold of 0.9 \n\n3. The document frequency of 64 exceeds the search threshold frequency of 50 - a search of LONG-NAME is conducted across all documents and another 200 documents are found to contain the label\n\n4. For the combined 264 documents, the LONG-FORM is added as a prediction. Additionally, if any of the 264 documents contains the string ‘ SLSD ‘ (checked against the raw document texts), an additional prediction of the acronym is added\n\nNote: this is a summary of the model logic, there are other rules applied that I’ve elided.\n\nCode [here](https://github.com/leecming/kaggle_coleridge)", "votes": 96 }, { "id": 1367108, "postDate": "2021-06-27T12:39:57.363Z", "content": "Hi Chun Ming Lee! Congrats on the results!
\nLooking forward to the GitHub repository!
Updated my post to link to the relevant Github repo
", "rawMarkdown": "Updated my post to link to the relevant Github repo" } ] }, { "id": 1366418, "postDate": "2021-06-26T18:32:21.367Z", "content": "Very good, congrats.
", "rawMarkdown": "Very good, congrats. ", "votes": 1 }, { "id": 1366396, "postDate": "2021-06-26T18:01:32.113Z", "content": "Very intuitive. @leecming . Thanks for sharing.
", "rawMarkdown": "Very intuitive. @leecming . Thanks for sharing.", "votes": 1 }, { "id": 1364826, "postDate": "2021-06-25T08:38:43.280Z", "content": "Congrats again! King of NLP:))
", "rawMarkdown": "Congrats again! King of NLP:))", "votes": 1 }, { "id": 1364764, "postDate": "2021-06-25T07:27:22.487Z", "content": "Thanks for sharing your amazing work! Congrats :)
", "rawMarkdown": "Thanks for sharing your amazing work! Congrats :)", "votes": 1 }, { "id": 1362653, "postDate": "2021-06-23T15:02:02.073Z", "content": "Was rooting for you @leecming, congrats on second place and the clever approach!
", "rawMarkdown": "Was rooting for you @leecming, congrats on second place and the clever approach!", "votes": 1 }, { "id": 1362648, "postDate": "2021-06-23T14:58:02.813Z", "content": "Awesome explanation, thanks. And congrats on the 2nd place, just a thousandth away from first.
", "rawMarkdown": "Awesome explanation, thanks. And congrats on the 2nd place, just a thousandth away from first.", "votes": 1 }, { "id": 1362533, "postDate": "2021-06-23T13:51:29.713Z", "content": "Congratulations! Great work!
", "rawMarkdown": "Congratulations! Great work!", "votes": 1 }, { "id": 1362177, "postDate": "2021-06-23T08:53:33.657Z", "content": "Congrats on solo 2nd place prize @leecming and thanks for the writeup
", "rawMarkdown": "Congrats on solo 2nd place prize @leecming and thanks for the writeup ", "votes": 1 }, { "id": 1362058, "postDate": "2021-06-23T07:38:01.603Z", "content": "Great work!
", "rawMarkdown": "Great work!", "votes": 1 }, { "id": 1362043, "postDate": "2021-06-23T07:21:16.910Z", "content": "Thanks for sharing and Congrats @leecming !
", "rawMarkdown": "Thanks for sharing and Congrats @leecming !", "votes": 1 }, { "id": 1361982, "postDate": "2021-06-23T06:38:30.097Z", "content": "Thanks for sharing and Congrat! You deserve it. I got the same method candidate generate + classify finally(two days before final time), and didn't have much time to optimize.
", "rawMarkdown": "Thanks for sharing and Congrat! You deserve it. I got the same method candidate generate + classify finally(two days before final time), and didn't have much time to optimize.", "votes": 1, "replies": [ { "id": 1361998, "postDate": "2021-06-23T06:55:25.233Z", "content": "BTW, did your final solution include any string match? Does the public score keep improve align with the model improve? I spent much time selecting(models + rules) good labels from the training data and add to know labels for matching. And the public score stops increasing no matter how \"right\" label added.
", "rawMarkdown": "BTW, did your final solution include any string match? Does the public score keep improve align with the model improve? I spent much time selecting(models + rules) good labels from the training data and add to know labels for matching. And the public score stops increasing no matter how \"right\" label added.", "votes": 1 }, { "id": 1362003, "postDate": "2021-06-23T06:58:14.187Z", "content": "my observation was that you need negative examples more than you need positive examples i.e., string candidates that are NOT datasets. it's very easy to run into a plateau just throwing positive examples at a given model/approach
", "rawMarkdown": "my observation was that you need negative examples more than you need positive examples i.e., string candidates that are NOT datasets. it's very easy to run into a plateau just throwing positive examples at a given model/approach", "votes": 1 }, { "id": 1362013, "postDate": "2021-06-23T07:07:44.307Z", "content": "Thanks for your quick reply. I didn't think about this. No matter for reducing the FP or model training for classifying. Great observation!
", "rawMarkdown": "Thanks for your quick reply. I didn't think about this. No matter for reducing the FP or model training for classifying. Great observation!", "votes": 1 } ] }, { "id": 1361928, "postDate": "2021-06-23T06:05:09.117Z", "content": "Congratulations! I came to similar conclusions.
", "rawMarkdown": "Congratulations! I came to similar conclusions.", "votes": 1, "replies": [ { "id": 1361976, "postDate": "2021-06-23T06:33:27.093Z", "content": "It was puzzling to see all the scraping going on…
", "rawMarkdown": "It was puzzling to see all the scraping going on... ", "votes": 3 } ] }, { "id": 1361908, "postDate": "2021-06-23T05:52:29.880Z", "content": "Neat! Congrats on the result.
", "rawMarkdown": "Neat! Congrats on the result.", "votes": 1 }, { "id": 1362305, "postDate": "2021-06-23T11:18:46.283Z", "content": "Very clever solution, congratz !
", "rawMarkdown": "Very clever solution, congratz !", "votes": 2 }, { "id": 1362236, "postDate": "2021-06-23T09:43:38.470Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1361995, "postDate": "2021-06-23T06:53:04.813Z", "content": "Thanks for sharing. Congrats man!
", "rawMarkdown": "Thanks for sharing. Congrats man!", "votes": 1 }, { "id": 1361918, "postDate": "2021-06-23T05:57:18.633Z", "content": "Thank you for sharing, Congratulations!
", "rawMarkdown": "Thank you for sharing, Congratulations!", "votes": 1 } ], "comments": [ { "id": 1367108, "author_name": "KulishYurii", "author_url": "", "post_date": "2021-06-27T12:39:57.363000", "content": "Hi Chun Ming Lee! Congrats on the results!
\nLooking forward to the GitHub repository!
Updated my post to link to the relevant Github repo
", "votes": 0, "replies": [] } ] }, { "id": 1366418, "author_name": "Paolo Antonuccio", "author_url": "", "post_date": "2021-06-26T18:32:21.367000", "content": "Very good, congrats.
", "votes": 1, "replies": [] }, { "id": 1366396, "author_name": "Kaustubh B Bhargav", "author_url": "", "post_date": "2021-06-26T18:01:32.113000", "content": "Very intuitive. @leecming . Thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 1364826, "author_name": "MaChaogong", "author_url": "", "post_date": "2021-06-25T08:38:43.280000", "content": "Congrats again! King of NLP:))
", "votes": 1, "replies": [] }, { "id": 1364764, "author_name": "Yongdam Kim", "author_url": "", "post_date": "2021-06-25T07:27:22.487000", "content": "Thanks for sharing your amazing work! Congrats :)
", "votes": 1, "replies": [] }, { "id": 1362653, "author_name": "Rafi Hai", "author_url": "", "post_date": "2021-06-23T15:02:02.073000", "content": "Was rooting for you @leecming, congrats on second place and the clever approach!
", "votes": 1, "replies": [] }, { "id": 1362648, "author_name": "Pablo Carbó", "author_url": "", "post_date": "2021-06-23T14:58:02.813000", "content": "Awesome explanation, thanks. And congrats on the 2nd place, just a thousandth away from first.
", "votes": 1, "replies": [] }, { "id": 1362533, "author_name": "Ramesh Makwana", "author_url": "", "post_date": "2021-06-23T13:51:29.713000", "content": "Congratulations! Great work!
", "votes": 1, "replies": [] }, { "id": 1362177, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-06-23T08:53:33.657000", "content": "Congrats on solo 2nd place prize @leecming and thanks for the writeup
", "votes": 1, "replies": [] }, { "id": 1362058, "author_name": "KulishYurii", "author_url": "", "post_date": "2021-06-23T07:38:01.603000", "content": "Great work!
", "votes": 1, "replies": [] }, { "id": 1362043, "author_name": "Abderrahim Alakouche", "author_url": "", "post_date": "2021-06-23T07:21:16.910000", "content": "Thanks for sharing and Congrats @leecming !
", "votes": 1, "replies": [] }, { "id": 1361982, "author_name": "youzi", "author_url": "", "post_date": "2021-06-23T06:38:30.097000", "content": "Thanks for sharing and Congrat! You deserve it. I got the same method candidate generate + classify finally(two days before final time), and didn't have much time to optimize.
", "votes": 1, "replies": [ { "id": 1361998, "author_name": "youzi", "author_url": "", "post_date": "2021-06-23T06:55:25.233000", "content": "BTW, did your final solution include any string match? Does the public score keep improve align with the model improve? I spent much time selecting(models + rules) good labels from the training data and add to know labels for matching. And the public score stops increasing no matter how \"right\" label added.
", "votes": 1, "replies": [] }, { "id": 1362003, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2021-06-23T06:58:14.187000", "content": "my observation was that you need negative examples more than you need positive examples i.e., string candidates that are NOT datasets. it's very easy to run into a plateau just throwing positive examples at a given model/approach
", "votes": 1, "replies": [] }, { "id": 1362013, "author_name": "youzi", "author_url": "", "post_date": "2021-06-23T07:07:44.307000", "content": "Thanks for your quick reply. I didn't think about this. No matter for reducing the FP or model training for classifying. Great observation!
", "votes": 1, "replies": [] } ] }, { "id": 1361928, "author_name": "Kramarenko Vladislav", "author_url": "", "post_date": "2021-06-23T06:05:09.117000", "content": "Congratulations! I came to similar conclusions.
", "votes": 1, "replies": [ { "id": 1361976, "author_name": "Chun Ming Lee", "author_url": "", "post_date": "2021-06-23T06:33:27.093000", "content": "It was puzzling to see all the scraping going on…
", "votes": 3, "replies": [] } ] }, { "id": 1361908, "author_name": "CPMP", "author_url": "", "post_date": "2021-06-23T05:52:29.880000", "content": "Neat! Congrats on the result.
", "votes": 1, "replies": [] }, { "id": 1362305, "author_name": "Theo Viel", "author_url": "", "post_date": "2021-06-23T11:18:46.283000", "content": "Very clever solution, congratz !
", "votes": 2, "replies": [] }, { "id": 1362236, "author_name": "", "author_url": "", "post_date": "2021-06-23T09:43:38.470000", "content": "", "votes": 0, "replies": [] }, { "id": 1361995, "author_name": "maze508", "author_url": "", "post_date": "2021-06-23T06:53:04.813000", "content": "Thanks for sharing. Congrats man!
", "votes": 1, "replies": [] }, { "id": 1361918, "author_name": "SSinyu", "author_url": "", "post_date": "2021-06-23T05:57:18.633000", "content": "Thank you for sharing, Congratulations!
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1361893": "Thanks to Coleridge & Kaggle for organizing this competition, and congrats to the other prize winners and medallists. This was a challenging competition in contrast to the standard Kaggle NLP classification/regression task, and I hope Coleridge will consider sponsoring future competitions in this vein. I suspect a future iteration will have the kinks shaken out :)\n\n## TL;DR: Search + Classify\n\n1. Search for named entities using the Schwartz-Hearst algorithm\n2. Filter candidates using a fine-tuned roberta-base binary classifier\n3. Threshold and propagate candidates\n\nCV strategy: It should be clear that there was no way to perform clean local validation. I used public LB while aware of its flaws. \n\nI spent the first month of the competition working on an end-to-end Transformer solution with minimal text processing. However, it became clear that a tweaked Transformer approach was problematic - they’re a) unsuited for the long document sequences we’re dealing with, even with the specialized variants designed for such texts, and b) ended up requiring heavy pre/post-processing to perform well. My processing heuristics effectively turned into a model parallel to the main Transformer model. I began thinking about how to simplify this - \n\n#### Insight #1: No context needed\nAny attempt to inject semantic context into my models seemed to worsen public LB accuracy. I suspected the annotation process was noisy and decided to focus on IDing datasets purely by title. \n\n#### Insight #2: Search for LONG-FORM (ACRONYM) \nData sets in academic papers often have a regular structure - mixed cap words followed by an acronym. An example is “Baltimore Longitudinal Study of Aging (BLSA)”. I needed something lighter than a Transformer to search for them efficiently - the answer was the [Schwartz-Hearst algorithm](https://psb.stanford.edu/psb-online/proceedings/psb03/schwartz.pdf). Schwartz-Hearst is a non-learning string search algorithm used to extract strings from text in the form of “LONG-FORM (ACRONYM)” - it’s basically a more involved regexp search. A bonus from using this algorithm is that I could also use the acronym form of a given dataset for additional points. \n\n#### Insight #3: No scraped data needed\nThe third insight (hypothesis?) was that scraped data was unnecessary. I did not use any externally derived list of datasets or scraped data - my sense was that 1) positive examples of datasets weren’t as useful as negative examples, and 2) it wasn’t even clear whether these lists hewed to the dataset definition that annotators were using. \n\n#### Putting it all together\nSchwartz-Hearst worked well at identifying named entities but there’d be many flagged candidates that weren’t data sets (e.g., organization names, scientific paraphernalia) so I needed a binary classifier to filter candidate strings. I hand-annotated (using my own judgment of what a dataset was) a miniature training corpus of positive and negative examples based on examples that Schwartz-Hearst IDed from training data and fine-tuned a roberta-base binary classifier to filter these named entities using the corpus. I’ll re-emphasize that this corpus was built wholly from the training data without any scraped data. \n\nThe last piece of the puzzle was dealing with the fact that not all papers referred to datasets using the full “LONG-NAME (ACRONYM)” format. Many referenced only the LONG-FORM. A simple solution was to generate a dynamic table of datasets and search for LONG-FORM in documents. I noticed that thresholding this search on a minimum document frequency improved public LB. The intuition is that in any given document corpus, a dataset is probably referenced by multiple documents so the more frequently they’re referenced, the more likely they’re a hit and not a spurious find. \n\nThis search + classify approach dramatically reduced inference time since I did not have to pass entire documents through a Transformer model. Inference on the test set took ~10 minutes vs >>1hr using end-to-end Transformer QA/NER models. \n\n## End-to-end dummy example:\n1. The LONG-FORM (ACRONYM) string of “Singapore Longitudinal Study on Diabetes (SLSD)” is found in 64 documents\n\n2. “Singapore Longitudinal Study on Diabetes” is classified by the Transformer binary classifier to be a dataset with probability 0.99, above the prediction threshold of 0.9 \n\n3. The document frequency of 64 exceeds the search threshold frequency of 50 - a search of LONG-NAME is conducted across all documents and another 200 documents are found to contain the label\n\n4. For the combined 264 documents, the LONG-FORM is added as a prediction. Additionally, if any of the 264 documents contains the string ‘ SLSD ‘ (checked against the raw document texts), an additional prediction of the acronym is added\n\nNote: this is a summary of the model logic, there are other rules applied that I’ve elided.\n\nCode [here](https://github.com/leecming/kaggle_coleridge)", "1367108": "Hi Chun Ming Lee! Congrats on the results!\nLooking forward to the GitHub repository!", "1366418": "Very good, congrats. ", "1366396": "Very intuitive. @leecming . Thanks for sharing.", "1364826": "Congrats again! King of NLP:))", "1364764": "Thanks for sharing your amazing work! Congrats :)", "1362653": "Was rooting for you @leecming, congrats on second place and the clever approach!", "1362648": "Awesome explanation, thanks. And congrats on the 2nd place, just a thousandth away from first.", "1362533": "Congratulations! Great work!", "1362177": "Congrats on solo 2nd place prize @leecming and thanks for the writeup ", "1362058": "Great work!", "1362043": "Thanks for sharing and Congrats @leecming !", "1361982": "Thanks for sharing and Congrat! You deserve it. I got the same method candidate generate + classify finally(two days before final time), and didn't have much time to optimize.", "1361928": "Congratulations! I came to similar conclusions.", "1361908": "Neat! Congrats on the result.", "1362305": "Very clever solution, congratz !", "1362236": "", "1361995": "Thanks for sharing. Congrats man!", "1361918": "Thank you for sharing, Congratulations!" } }