Since I went through the huge shakedown (3rd->55th), this solution may not be the effective approach, but I will share my solution with you.
\nIf give me any feedback on this solution, I would be happy!!
\nI use the two stage predictions which is text matching and NER.
\nOf the 2nd Stage predictions, added the ones that are not similer to the 1st Stage predictions to the 1st Stage predictions.
\nThis score was lower than the 1st Stage score in public.
The following table is the public score and the private score of each stage.
\nMy inference notebook is here.
| Stage | \nPublic Score | \nPrivate Score | \n
|---|---|---|
| 1st | \n0.673 | \n0.319 | \n
| 2nd | \n0.636 | \n0.303 | \n
| 1st+2nd | \n0.672 | \n0.322 | \n
I added labels to the labels of training data by the following method.
\nI have adapted this solution only for train data, but the one I applied to test data as well was my best submit. (Public: 0.662, Private: 0.427)
Looking at the articles, we can find that most of the labels in the training data is used in the following structure.
\nI used regular expressions to extract the expressions for the bolded parts.
\"the\" + [several words written in headline style] + \"(\" + [an abbreviation of headline] + \")\"
\ne.g. the Alzheimer's Disease Neuroimaging Initiative (ADNI)
After the extraction, I counted DF values of the headline phrases.
\nI selected phrases by the rule-based approach.
\nOf the extracted phrases of 1-1, add the ones that match the following conditions to the labels for text matching.
I spent most of my time on this approach because I joined this competition to learn NER.
\nHowever, since my approach at this stage was strongly fitting to public and basically doing the same thing as text matching, the total score did not change much from the predictions of the first stage.
I generated train data for NER from the text divided into sentences as the follow.
\nPositive samples: text contains the labels of the train and 1-2 \nNegative samples: text contains the labels not selected in 1-2 \nELECTRA-base + Spatial Dropout + Bi-LSTM(using attention mask) + Dense(Activation: Softmax) + CRF(using attention mask)
\n| Item | \nValue | \n
|---|---|
| Optimizer | \nAdam | \n
| Learning rate | \n1.6e-4 | \n
| Sequence Length | \n128 | \n
| Spatial Dropout ratio | \n0.1 | \n
| Bi-LSTM output size | \n256 * 2 | \n
| Batch size | \n128 | \n
| Folds | \n4 | \n
| Epoch | \n100 | \n
| Patiance of early stopping | \n20 | \n
The labels predicted by the 3 folds was taken as the predictions of this stage.
\n(I tried to use the mean values of CRF's potentials, but they didn't work not well…)
Since I went through the huge shakedown (3rd->55th), this solution may not be the effective approach, but I will share my solution with you.
\nIf give me any feedback on this solution, I would be happy!!
\nI use the two stage predictions which is text matching and NER.
\nOf the 2nd Stage predictions, added the ones that are not similer to the 1st Stage predictions to the 1st Stage predictions.
\nThis score was lower than the 1st Stage score in public.
The following table is the public score and the private score of each stage.
\nMy inference notebook is here.
| Stage | \nPublic Score | \nPrivate Score | \n
|---|---|---|
| 1st | \n0.673 | \n0.319 | \n
| 2nd | \n0.636 | \n0.303 | \n
| 1st+2nd | \n0.672 | \n0.322 | \n
I added labels to the labels of training data by the following method.
\nI have adapted this solution only for train data, but the one I applied to test data as well was my best submit. (Public: 0.662, Private: 0.427)
Looking at the articles, we can find that most of the labels in the training data is used in the following structure.
\nI used regular expressions to extract the expressions for the bolded parts.
\"the\" + [several words written in headline style] + \"(\" + [an abbreviation of headline] + \")\"
\ne.g. the Alzheimer's Disease Neuroimaging Initiative (ADNI)
After the extraction, I counted DF values of the headline phrases.
\nI selected phrases by the rule-based approach.
\nOf the extracted phrases of 1-1, add the ones that match the following conditions to the labels for text matching.
I spent most of my time on this approach because I joined this competition to learn NER.
\nHowever, since my approach at this stage was strongly fitting to public and basically doing the same thing as text matching, the total score did not change much from the predictions of the first stage.
I generated train data for NER from the text divided into sentences as the follow.
\nPositive samples: text contains the labels of the train and 1-2 \nNegative samples: text contains the labels not selected in 1-2 \nELECTRA-base + Spatial Dropout + Bi-LSTM(using attention mask) + Dense(Activation: Softmax) + CRF(using attention mask)
\n| Item | \nValue | \n
|---|---|
| Optimizer | \nAdam | \n
| Learning rate | \n1.6e-4 | \n
| Sequence Length | \n128 | \n
| Spatial Dropout ratio | \n0.1 | \n
| Bi-LSTM output size | \n256 * 2 | \n
| Batch size | \n128 | \n
| Folds | \n4 | \n
| Epoch | \n100 | \n
| Patiance of early stopping | \n20 | \n
The labels predicted by the 3 folds was taken as the predictions of this stage.
\n(I tried to use the mean values of CRF's potentials, but they didn't work not well…)
Hi Shota! It was nice to see your name at the top of the rankings, it was like a beacon for us beginner kagglers. One of the approaches that I used to solve this problem was also NER, on manually marked up data I noticed that architectures based on transformers are worse at defining the context on the basis of which one can say whether the described dataset was used in the article or it is just mentioned. It's funny that the best solutions are based simply on regular expressions)) It was a great experience anyway!
", "rawMarkdown": "Hi Shota! It was nice to see your name at the top of the rankings, it was like a beacon for us beginner kagglers. One of the approaches that I used to solve this problem was also NER, on manually marked up data I noticed that architectures based on transformers are worse at defining the context on the basis of which one can say whether the described dataset was used in the article or it is just mentioned. It's funny that the best solutions are based simply on regular expressions)) It was a great experience anyway!", "votes": null }, { "id": "1362078", "postDate": "06/23/2021 08:02:06", "content": "Your performance on the public LB (especially before the re-annotation) definitely pushed me and many others to work very hard!
\nIt’s unfortunate it didn’t translate to private LB but tbh, not sure what else you could’ve done given we only had public LB as reference.
\nHopefully the experience here will help you for your next competition :)
", "rawMarkdown": "Your performance on the public LB (especially before the re-annotation) definitely pushed me and many others to work very hard!\n\nIt’s unfortunate it didn’t translate to private LB but tbh, not sure what else you could’ve done given we only had public LB as reference.\n\nHopefully the experience here will help you for your next competition :)", "votes": null }, { "id": "1362173", "postDate": "06/23/2021 08:52:00", "content": "@kucherevskiy
\nyou for your comment!
\n\nOne of the approaches that I used to solve this problem was also NER, on manually marked up data I noticed that architectures based on transformers are worse at defining the context on the basis of which one can say whether the described dataset was used in the article or it is just mentioned.
\n
I think so too.
\nUsing NER without rich context recognition, I think that it's almost the same as text matching.
\n\nIt's funny that the best solutions are based simply on regular expressions)) It was a great experience anyway!
\n
It's so sad.
\nBut I should have focused on this solution. lol
@leecming
\nThank you for your comment!
\nIt’s has been fun to fight on the higher rank of public LB for 3 months!!
\n(But, my private score was much different than yours lol.)
\n\nIt’s unfortunate it didn’t translate to private LB but tbh, not sure what else you could’ve done given we only had public LB as reference.
\n
Yes, I also think that was an important and difficult point.
\nI think the biggest reason for my defeat was that I fought without a guideline for tackling the problem.
I would like to make use of this experience in the next competition and aim for gold.
", "rawMarkdown": "leecming \nThank you for your comment!\nIt’s has been fun to fight on the higher rank of public LB for 3 months!!\n(But, my private score was much different than yours lol.)\n\n> It’s unfortunate it didn’t translate to private LB but tbh, not sure what else you could’ve done given we only had public LB as reference.\n\nYes, I also think that was an important and difficult point.\nI think the biggest reason for my defeat was that I fought without a guideline for tackling the problem.\n\nI would like to make use of this experience in the next competition and aim for gold.", "votes": null } ], "comments": [ { "id": 1362054, "author_name": "kucherevskiy", "author_url": "", "post_date": "06/23/2021 07:31:34", "content": "Hi Shota! It was nice to see your name at the top of the rankings, it was like a beacon for us beginner kagglers. One of the approaches that I used to solve this problem was also NER, on manually marked up data I noticed that architectures based on transformers are worse at defining the context on the basis of which one can say whether the described dataset was used in the article or it is just mentioned. It's funny that the best solutions are based simply on regular expressions)) It was a great experience anyway!
", "votes": null, "replies": [ { "id": 1362173, "author_name": "spshota", "author_url": "", "post_date": "06/23/2021 08:52:00", "content": "@kucherevskiy
\nyou for your comment!
\n\nOne of the approaches that I used to solve this problem was also NER, on manually marked up data I noticed that architectures based on transformers are worse at defining the context on the basis of which one can say whether the described dataset was used in the article or it is just mentioned.
\n
I think so too.
\nUsing NER without rich context recognition, I think that it's almost the same as text matching.
\n\nIt's funny that the best solutions are based simply on regular expressions)) It was a great experience anyway!
\n
It's so sad.
\nBut I should have focused on this solution. lol
Your performance on the public LB (especially before the re-annotation) definitely pushed me and many others to work very hard!
\nIt’s unfortunate it didn’t translate to private LB but tbh, not sure what else you could’ve done given we only had public LB as reference.
\nHopefully the experience here will help you for your next competition :)
", "votes": null, "replies": [ { "id": 1362187, "author_name": "spshota", "author_url": "", "post_date": "06/23/2021 09:03:52", "content": "@leecming
\nThank you for your comment!
\nIt’s has been fun to fight on the higher rank of public LB for 3 months!!
\n(But, my private score was much different than yours lol.)
\n\nIt’s unfortunate it didn’t translate to private LB but tbh, not sure what else you could’ve done given we only had public LB as reference.
\n
Yes, I also think that was an important and difficult point.
\nI think the biggest reason for my defeat was that I fought without a guideline for tackling the problem.
I would like to make use of this experience in the next competition and aim for gold.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1361963": "# Coleridge Initiative 55th place Solution \n\nSince I went through the huge shakedown (3rd->55th), this solution may not be the effective approach, but I will share my solution with you. \n\nIf give me any feedback on this solution, I would be happy!! \n\nI use the two stage predictions which is text matching and NER. \n\nOf the 2nd Stage predictions, added the ones that are not similer to the 1st Stage predictions to the 1st Stage predictions. \nThis score was lower than the 1st Stage score in public. \n\nThe following table is the public score and the private score of each stage. \nMy inference notebook is [here](https://www.kaggle.com/spshota/coleridge-text-matching-and-ner-cnt?scriptVersionId=66373085).\n\n|Stage|Public Score|Private Score|\n|---|---|---|\n|1st|0.673|0.319|\n|2nd|0.636|0.303|\n|1st+2nd|0.672|0.322|\n\n## 1st Stage: Text Matching with Additional Labels \nI added labels to the labels of training data by the following method. \nI have adapted this solution only for train data, but the one I applied to test data as well was my best submit. (Public: 0.662, Private: 0.427) \n\n### 1-1. Headline Style Phrases Extraction \n\nLooking at the articles, we can find that most of the labels in the training data is used in the following structure. \nI used regular expressions to extract the expressions for the bolded parts. \n\n\"the\" + **[several words written in headline style]** + \"(\" + [an abbreviation of headline] + \")\" \ne.g. the Alzheimer's Disease Neuroimaging Initiative (ADNI) \n\nAfter the extraction, I counted DF values of the headline phrases. \n\n### 1-2. Additional Labels Selection \n\nI selected phrases by the rule-based approach. \nOf the extracted phrases of 1-1, add the ones that match the following conditions to the labels for text matching. \n- DF value >= 20 \n- Contains one of the curated topic words \n- Not contains any of the curated stop words \n- Not similar to any of the labels of train data(jaccard similarity < 0.5)\n\n## 2nd Stage: Named Entity Recognition (NER) \nI spent most of my time on this approach because I joined this competition to learn NER. \nHowever, since my approach at this stage was strongly fitting to public and basically doing the same thing as text matching, the total score did not change much from the predictions of the first stage. \n\n### 2-1. Training Data for NER \nI generated train data for NER from the text divided into sentences as the follow. \n\n Positive samples: text contains the labels of the train and 1-2 \n Negative samples: text contains the labels not selected in 1-2 \n\n### 2-2. Model Architecture \nELECTRA-base + Spatial Dropout + Bi-LSTM(using attention mask) + Dense(Activation: Softmax) + CRF(using attention mask) \n\n### 2-3. Hyper Parameters \n\n|Item|Value|\n|---|---|\n|Optimizer|Adam|\n|Learning rate|1.6e-4|\n|Sequence Length|128|\n|Spatial Dropout ratio|0.1|\n|Bi-LSTM output size|256 * 2|\n|Batch size|128|\n|Folds|4|\n|Epoch|100|\n|Patiance of early stopping|20|\n\n### 2-4. Label Selection \n\nThe labels predicted by the 3 folds was taken as the predictions of this stage. \n(I tried to use the mean values of CRF's potentials, but they didn't work not well...)", "1362054": "Hi Shota! It was nice to see your name at the top of the rankings, it was like a beacon for us beginner kagglers. One of the approaches that I used to solve this problem was also NER, on manually marked up data I noticed that architectures based on transformers are worse at defining the context on the basis of which one can say whether the described dataset was used in the article or it is just mentioned. It's funny that the best solutions are based simply on regular expressions)) It was a great experience anyway!", "1362078": "Your performance on the public LB (especially before the re-annotation) definitely pushed me and many others to work very hard!\n\nIt’s unfortunate it didn’t translate to private LB but tbh, not sure what else you could’ve done given we only had public LB as reference.\n\nHopefully the experience here will help you for your next competition :)", "1362173": "kucherevskiy \nyou for your comment!\n\n> One of the approaches that I used to solve this problem was also NER, on manually marked up data I noticed that architectures based on transformers are worse at defining the context on the basis of which one can say whether the described dataset was used in the article or it is just mentioned.\n\nI think so too. \nUsing NER without rich context recognition, I think that it's almost the same as text matching.\n\n> It's funny that the best solutions are based simply on regular expressions)) It was a great experience anyway!\n\nIt's so sad. \nBut I should have focused on this solution. lol", "1362187": "leecming \nThank you for your comment!\nIt’s has been fun to fight on the higher rank of public LB for 3 months!!\n(But, my private score was much different than yours lol.)\n\n> It’s unfortunate it didn’t translate to private LB but tbh, not sure what else you could’ve done given we only had public LB as reference.\n\nYes, I also think that was an important and difficult point.\nI think the biggest reason for my defeat was that I fought without a guideline for tackling the problem.\n\nI would like to make use of this experience in the next competition and aim for gold." }, "source": "existing_discussion", "complete": true, "competition_id": 25925, "competition_slug": "coleridgeinitiative-show-us-the-data", "team_id": 6515854, "solution_url": "/competitions/coleridgeinitiative-show-us-the-data/writeups/shota-55th-place-solution" }