{ "id": 248253, "title": "1st solution: Matching the : Context Similarity via Deep Metric Learning and Beyond", "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/248253", "author_name": "", "post_date": "2021-06-23T00:59:44.599741100Z", "votes": 80, "comment_count": 22, "views": 0, "content": "

Thanks Kaggle for this exciting competition and our team (@suicaokhoailang). Congrats to me to become Kaggle Master and @suicaokhoailang to become Kaggle GM. Congratulate all winners -- we have learned a lot from this competition and all winners’ solutions !!.

\n

\"\"

\n

Here Is a summary of my solution:

\n\n

Bellow is the most important question that we should answer for ourselves before deciding on an approach.

\n

Do we need a context?

\n

Yes, a semantic context is very important for this kind of problem. As you can see, there are some solutions that don’t need semantic context and only needs dataset title. But those solutions are only suitable for this contest data since almost the dataset title includes keywords such as Study, Survey, Database, Dataset, … then the dataset title itself is good enough to classify if it’s a real dataset title or not. In this competition, we aim for a general solution that uses a semantic context to detect a dataset citation.

\n

How my model was trained?

\n
    \n
  1. An input text is randomly split into Support Set and Query Set.
    2. \n
  2. In Support Set, we use a MASK token to replace all dataset titles in an input text.
    3. \n
  3. Pass all input text from Support Set into a BertModel and extract MASK/NoMASK embedding.
    4. \n
  4. In Query Set, we pass a raw input into a shared BertModel used for Support Set to extract a sequence tokens embedding.
    5. \n
  5. MASK and NoMask embedding from Support/Query Set is passed into an ArcFace Metric Learning layer.
    6. \n
  6. A MASK embedding from the Support Set contains only a semantic context feature is used to calculate l2 norm cosine similarity with every token feature from the input text in the Query Set.
    7. \n
  7. A l2 norm cosine similarity will range from (-1, 1), we scale it into a range (-10, 10) then apply sigmoid activation before use it for training the NER model via BCE loss.
    8. \n
\n

How to use my trained model for the inference?

\n
    \n
  1. Pre-extract all MASK embedding from the given training set.
    2. \n
  2. Random selection K MASK embedding then average them to obtain the semantic context vector and perform step (6), (7) from the above training process.
    3. \n
\n

Post-Processing

\n

As @leecming said in his solution, a dataset is probably referenced by multiple documents so the more frequently they’re referenced, the more likely they’re a hit and not a spurious find. We use a threshold frequency of 2, which means if our model can detect that dataset title twice, we will get that dataset. We choose 2 because, in training data, there are many dataset titles mentioned only 1-2 times, we are afraid in the private dataset, such dataset titles will be many but we seem to be wrong, a higher threshold frequency gives better results on public and private leaderboards.

\n

Here is a summary of my solution, there are a few string rules-based that were used but did not affect the final result too much. The final submission is based on an ensemble of allenai/scibert and allenai/biomed_roberta. The code is done on Tensorflow 2.4 with V100 16gb version in AWS EC2.

\n

Inference Kernel: https://www.kaggle.com/dathudeptrai/biomed-roberta-scibert-base

\n
Thanks for reading !
", "messages": [ { "id": "1361626", "postDate": "06/23/2021 00:59:44", "content": "

Thanks Kaggle for this exciting competition and our team (@suicaokhoailang). Congrats to me to become Kaggle Master and @suicaokhoailang to become Kaggle GM. Congratulate all winners -- we have learned a lot from this competition and all winners’ solutions !!.

\n

\"\"

\n

Here Is a summary of my solution:

\n\n

Bellow is the most important question that we should answer for ourselves before deciding on an approach.

\n

Do we need a context?

\n

Yes, a semantic context is very important for this kind of problem. As you can see, there are some solutions that don’t need semantic context and only needs dataset title. But those solutions are only suitable for this contest data since almost the dataset title includes keywords such as Study, Survey, Database, Dataset, … then the dataset title itself is good enough to classify if it’s a real dataset title or not. In this competition, we aim for a general solution that uses a semantic context to detect a dataset citation.

\n

How my model was trained?

\n
    \n
  1. An input text is randomly split into Support Set and Query Set.
    2. \n
  2. In Support Set, we use a MASK token to replace all dataset titles in an input text.
    3. \n
  3. Pass all input text from Support Set into a BertModel and extract MASK/NoMASK embedding.
    4. \n
  4. In Query Set, we pass a raw input into a shared BertModel used for Support Set to extract a sequence tokens embedding.
    5. \n
  5. MASK and NoMask embedding from Support/Query Set is passed into an ArcFace Metric Learning layer.
    6. \n
  6. A MASK embedding from the Support Set contains only a semantic context feature is used to calculate l2 norm cosine similarity with every token feature from the input text in the Query Set.
    7. \n
  7. A l2 norm cosine similarity will range from (-1, 1), we scale it into a range (-10, 10) then apply sigmoid activation before use it for training the NER model via BCE loss.
    8. \n
\n

How to use my trained model for the inference?

\n
    \n
  1. Pre-extract all MASK embedding from the given training set.
    2. \n
  2. Random selection K MASK embedding then average them to obtain the semantic context vector and perform step (6), (7) from the above training process.
    3. \n
\n

Post-Processing

\n

As @leecming said in his solution, a dataset is probably referenced by multiple documents so the more frequently they’re referenced, the more likely they’re a hit and not a spurious find. We use a threshold frequency of 2, which means if our model can detect that dataset title twice, we will get that dataset. We choose 2 because, in training data, there are many dataset titles mentioned only 1-2 times, we are afraid in the private dataset, such dataset titles will be many but we seem to be wrong, a higher threshold frequency gives better results on public and private leaderboards.

\n

Here is a summary of my solution, there are a few string rules-based that were used but did not affect the final result too much. The final submission is based on an ensemble of allenai/scibert and allenai/biomed_roberta. The code is done on Tensorflow 2.4 with V100 16gb version in AWS EC2.

\n

Inference Kernel: https://www.kaggle.com/dathudeptrai/biomed-roberta-scibert-base

\n
Thanks for reading !
", "rawMarkdown": "Thanks Kaggle for this exciting competition and our team (@suicaokhoailang). Congrats to me to become Kaggle Master and @suicaokhoailang to become Kaggle GM. Congratulate all winners -- we have learned a lot from this competition and all winners’ solutions !!. \n\n![](https://i.imgur.com/9nuyRxE.png)\n\nHere Is a summary of my solution:\n- A shared Bert model for extract Context Embedding and Sequence Tokens Embedding.\n- An ArcFace Loss for training Mask/NoMask Embedding.\n- A BCE loss for training NER model to detect dataset citation in the input string.\n\nBellow is the most important question that we should answer for ourselves before deciding on an approach.\n\n### Do we need a context?\n\nYes, a semantic context is very important for this kind of problem. As you can see, there are some solutions that don’t need semantic context and only needs dataset title. But those solutions are only suitable for this contest data since almost the dataset title includes keywords such as Study, Survey, Database, Dataset, … then the dataset title itself is good enough to classify if it’s a real dataset title or not. In this competition, we aim for a general solution that uses a semantic context to detect a dataset citation.\n\n\n### How my model was trained?\n\n1. An input text is randomly split into Support Set and Query Set.\n2. In Support Set, we use a **MASK** token to replace all dataset titles in an input text.\n3. Pass all input text from Support Set into a BertModel and extract **MASK**/**NoMASK** embedding.\n4. In Query Set, we pass a raw input into a shared BertModel used for Support Set to extract a sequence tokens embedding.\n5. **MASK** and **NoMask** embedding from Support/Query Set is passed into an ArcFace Metric Learning layer.\n6. A **MASK** embedding from the Support Set contains only a semantic context feature is used to calculate l2 norm cosine similarity with every token feature from the input text in the Query Set.\n7. A l2 norm cosine similarity will range from (-1, 1), we scale it into a range (-10, 10) then apply sigmoid activation before use it for training the NER model via BCE loss.\n\n\n### How to use my trained model for the inference?\n\n1. Pre-extract all **MASK** embedding from the given training set.\n2. Random selection K **MASK** embedding then average them to obtain the semantic context vector and perform step (6), (7) from the above training process.\n \n\n### Post-Processing\n\nAs @leecming said in his solution, a dataset is probably referenced by multiple documents so the more frequently they’re referenced, the more likely they’re a hit and not a spurious find. We use a threshold frequency of 2, which means if our model can detect that dataset title twice, we will get that dataset. We choose 2 because, in training data, there are many dataset titles mentioned only 1-2 times, we are afraid in the private dataset, such dataset titles will be many but we seem to be wrong, a higher threshold frequency gives better results on public and private leaderboards.\n\nHere is a summary of my solution, there are a few string rules-based that were used but did not affect the final result too much. The final submission is based on an ensemble of allenai/scibert and allenai/biomed_roberta. The code is done on Tensorflow 2.4 with V100 16gb version in AWS EC2. \n\n\n**Inference Kernel**: https://www.kaggle.com/dathudeptrai/biomed-roberta-scibert-base\n\n##### Thanks for reading !", "votes": null }, { "id": "1361640", "postDate": "06/23/2021 01:19:53", "content": "

Congratulations! Cố lên anh ơi!

", "rawMarkdown": "Congratulations! Cố lên anh ơi!", "votes": null }, { "id": "1361671", "postDate": "06/23/2021 02:09:40", "content": "

Congrats on your hardwork paying off!

", "rawMarkdown": "Congrats on your hardwork paying off!", "votes": null }, { "id": "1361704", "postDate": "06/23/2021 02:57:45", "content": "

Congrats ✊

", "rawMarkdown": "Congrats ✊", "votes": null }, { "id": "1361768", "postDate": "06/23/2021 04:00:37", "content": "

Nghỉ mấy hôm rồi quay lại nhanh nha :))

", "rawMarkdown": "Nghỉ mấy hôm rồi quay lại nhanh nha :))", "votes": null }, { "id": "1361824", "postDate": "06/23/2021 04:40:53", "content": "

Congrats to the 1st place.

", "rawMarkdown": "Congrats to the 1st place.", "votes": null }, { "id": "1362169", "postDate": "06/23/2021 08:49:12", "content": "

Thanks @suicaokhoailang :D.

", "rawMarkdown": "Thanks @suicaokhoailang :D.", "votes": null }, { "id": "1362178", "postDate": "06/23/2021 08:54:57", "content": "

Wow amazing solution! Thanks for sharing!

", "rawMarkdown": "Wow amazing solution! Thanks for sharing!", "votes": null }, { "id": "1362197", "postDate": "06/23/2021 09:14:37", "content": "

Amazing work 😍! Congrats
\nP/s: rep me idol Anh Minh

", "rawMarkdown": "Amazing work 😍! Congrats\nP/s: rep me idol Anh Minh", "votes": null }, { "id": "1362222", "postDate": "06/23/2021 09:32:03", "content": "

great work. hope to collaborate with you again soon ^^

", "rawMarkdown": "great work. hope to collaborate with you again soon ^^", "votes": null }, { "id": "1362223", "postDate": "06/23/2021 09:32:06", "content": "

Just to understand few things: Is purpose of ARcFace loss is to ensure that embeddings for mask and statement from same document should be similar. What is the shape of Mask/No mask embeddings?
\nTrying to understand. Any description around intuition will help.
\nYour solution is quite innovative.

", "rawMarkdown": "Just to understand few things: Is purpose of ARcFace loss is to ensure that embeddings for mask and statement from same document should be similar. What is the shape of Mask/No mask embeddings?\nTrying to understand. Any description around intuition will help.\nYour solution is quite innovative.", "votes": null }, { "id": "1362266", "postDate": "06/23/2021 10:27:15", "content": "

Congrats 💪

", "rawMarkdown": "Congrats 💪", "votes": null }, { "id": "1362307", "postDate": "06/23/2021 11:19:48", "content": "

Very cool solution @dathudeptrai, congratz on the win !

", "rawMarkdown": "Very cool solution @dathudeptrai, congratz on the win !", "votes": null }, { "id": "1362538", "postDate": "06/23/2021 13:55:13", "content": "

Congratulations!

", "rawMarkdown": "Congratulations!", "votes": null }, { "id": "1363146", "postDate": "06/24/2021 00:43:39", "content": "

Hi @adityakumarsinha , the purpose of Arcface loss is to ensure that the embeddings for the MASK from all positive samples are similar. The shape of Mask/NoMask Embedding is [B, F], which is an average feature of all MASK/NoMASK token in the input text.

", "rawMarkdown": "Hi @adityakumarsinha , the purpose of Arcface loss is to ensure that the embeddings for the MASK from all positive samples are similar. The shape of Mask/NoMask Embedding is [B, F], which is an average feature of all MASK/NoMASK token in the input text.", "votes": null }, { "id": "1363148", "postDate": "06/24/2021 00:44:09", "content": "

Thanks @theoviel, hope to work with you again soon :D

", "rawMarkdown": "Thanks @theoviel, hope to work with you again soon :D", "votes": null }, { "id": "1363163", "postDate": "06/24/2021 01:39:26", "content": "

Congrats and nice solution !

", "rawMarkdown": "Congrats and nice solution !", "votes": null }, { "id": "1363182", "postDate": "06/24/2021 02:01:34", "content": "

@quanbk Thanks, hope we can work together in the future :)))

", "rawMarkdown": "quanbk Thanks, hope we can work together in the future :)))", "votes": null }, { "id": "1364178", "postDate": "06/24/2021 17:19:11", "content": "

Update: Surprisingly, remove the label from scispacy got a better result (0.588 on private), we should trust only our model :D. https://www.kaggle.com/dathudeptrai/biomed-roberta-scibert-base

", "rawMarkdown": "Update: Surprisingly, remove the label from scispacy got a better result (0.588 on private), we should trust only our model :D. https://www.kaggle.com/dathudeptrai/biomed-roberta-scibert-base", "votes": null }, { "id": "1367021", "postDate": "06/27/2021 11:51:32", "content": "

Congrats! Nice solution!

", "rawMarkdown": "Congrats! Nice solution!", "votes": null }, { "id": "1370095", "postDate": "06/29/2021 22:47:50", "content": "

Interesting solution, congrats!

", "rawMarkdown": "Interesting solution, congrats!", "votes": null }, { "id": "1370898", "postDate": "06/30/2021 13:27:37", "content": "

Nice solution, Congratulations!!

", "rawMarkdown": "Nice solution, Congratulations!!", "votes": null }, { "id": "1371506", "postDate": "07/01/2021 04:07:07", "content": "

Nice, congratulations…

", "rawMarkdown": "Nice, congratulations...", "votes": null } ], "comments": [ { "id": 1361640, "author_name": "ktgiahieu", "author_url": "", "post_date": "06/23/2021 01:19:53", "content": "

Congratulations! Cố lên anh ơi!

", "votes": null, "replies": [] }, { "id": 1361671, "author_name": "leecming", "author_url": "", "post_date": "06/23/2021 02:09:40", "content": "

Congrats on your hardwork paying off!

", "votes": null, "replies": [] }, { "id": 1361704, "author_name": "truonghoang", "author_url": "", "post_date": "06/23/2021 02:57:45", "content": "

Congrats ✊

", "votes": null, "replies": [] }, { "id": 1361768, "author_name": "duykhanh99", "author_url": "", "post_date": "06/23/2021 04:00:37", "content": "

Nghỉ mấy hôm rồi quay lại nhanh nha :))

", "votes": null, "replies": [] }, { "id": 1361824, "author_name": "youzi2016", "author_url": "", "post_date": "06/23/2021 04:40:53", "content": "

Congrats to the 1st place.

", "votes": null, "replies": [] }, { "id": 1362169, "author_name": "dathudeptrai", "author_url": "", "post_date": "06/23/2021 08:49:12", "content": "

Thanks @suicaokhoailang :D.

", "votes": null, "replies": [] }, { "id": 1362178, "author_name": "duykhanh99", "author_url": "", "post_date": "06/23/2021 08:54:57", "content": "

Wow amazing solution! Thanks for sharing!

", "votes": null, "replies": [] }, { "id": 1362197, "author_name": "quanbk", "author_url": "", "post_date": "06/23/2021 09:14:37", "content": "

Amazing work 😍! Congrats
\nP/s: rep me idol Anh Minh

", "votes": null, "replies": [ { "id": 1363182, "author_name": "dathudeptrai", "author_url": "", "post_date": "06/24/2021 02:01:34", "content": "

@quanbk Thanks, hope we can work together in the future :)))

", "votes": null, "replies": [] } ] }, { "id": 1362222, "author_name": "nddung", "author_url": "", "post_date": "06/23/2021 09:32:03", "content": "

great work. hope to collaborate with you again soon ^^

", "votes": null, "replies": [] }, { "id": 1362223, "author_name": "adityakumarsinha", "author_url": "", "post_date": "06/23/2021 09:32:06", "content": "

Just to understand few things: Is purpose of ARcFace loss is to ensure that embeddings for mask and statement from same document should be similar. What is the shape of Mask/No mask embeddings?
\nTrying to understand. Any description around intuition will help.
\nYour solution is quite innovative.

", "votes": null, "replies": [ { "id": 1363146, "author_name": "dathudeptrai", "author_url": "", "post_date": "06/24/2021 00:43:39", "content": "

Hi @adityakumarsinha , the purpose of Arcface loss is to ensure that the embeddings for the MASK from all positive samples are similar. The shape of Mask/NoMask Embedding is [B, F], which is an average feature of all MASK/NoMASK token in the input text.

", "votes": null, "replies": [] } ] }, { "id": 1362266, "author_name": "rin2401", "author_url": "", "post_date": "06/23/2021 10:27:15", "content": "

Congrats 💪

", "votes": null, "replies": [] }, { "id": 1362307, "author_name": "theoviel", "author_url": "", "post_date": "06/23/2021 11:19:48", "content": "

Very cool solution @dathudeptrai, congratz on the win !

", "votes": null, "replies": [ { "id": 1363148, "author_name": "dathudeptrai", "author_url": "", "post_date": "06/24/2021 00:44:09", "content": "

Thanks @theoviel, hope to work with you again soon :D

", "votes": null, "replies": [] } ] }, { "id": 1362538, "author_name": "ramesh084", "author_url": "", "post_date": "06/23/2021 13:55:13", "content": "

Congratulations!

", "votes": null, "replies": [] }, { "id": 1363163, "author_name": "mcggood", "author_url": "", "post_date": "06/24/2021 01:39:26", "content": "

Congrats and nice solution !

", "votes": null, "replies": [] }, { "id": 1364178, "author_name": "dathudeptrai", "author_url": "", "post_date": "06/24/2021 17:19:11", "content": "

Update: Surprisingly, remove the label from scispacy got a better result (0.588 on private), we should trust only our model :D. https://www.kaggle.com/dathudeptrai/biomed-roberta-scibert-base

", "votes": null, "replies": [] }, { "id": 1367021, "author_name": "duongprovipvip", "author_url": "", "post_date": "06/27/2021 11:51:32", "content": "

Congrats! Nice solution!

", "votes": null, "replies": [] }, { "id": 1370095, "author_name": "antonmaster007", "author_url": "", "post_date": "06/29/2021 22:47:50", "content": "

Interesting solution, congrats!

", "votes": null, "replies": [] }, { "id": 1370898, "author_name": "samarthgupta39", "author_url": "", "post_date": "06/30/2021 13:27:37", "content": "

Nice solution, Congratulations!!

", "votes": null, "replies": [] }, { "id": 1371506, "author_name": "laxmankusuma", "author_url": "", "post_date": "07/01/2021 04:07:07", "content": "

Nice, congratulations…

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1361626": "Thanks Kaggle for this exciting competition and our team (@suicaokhoailang). Congrats to me to become Kaggle Master and @suicaokhoailang to become Kaggle GM. Congratulate all winners -- we have learned a lot from this competition and all winners’ solutions !!. \n\n![](https://i.imgur.com/9nuyRxE.png)\n\nHere Is a summary of my solution:\n- A shared Bert model for extract Context Embedding and Sequence Tokens Embedding.\n- An ArcFace Loss for training Mask/NoMask Embedding.\n- A BCE loss for training NER model to detect dataset citation in the input string.\n\nBellow is the most important question that we should answer for ourselves before deciding on an approach.\n\n### Do we need a context?\n\nYes, a semantic context is very important for this kind of problem. As you can see, there are some solutions that don’t need semantic context and only needs dataset title. But those solutions are only suitable for this contest data since almost the dataset title includes keywords such as Study, Survey, Database, Dataset, … then the dataset title itself is good enough to classify if it’s a real dataset title or not. In this competition, we aim for a general solution that uses a semantic context to detect a dataset citation.\n\n\n### How my model was trained?\n\n1. An input text is randomly split into Support Set and Query Set.\n2. In Support Set, we use a **MASK** token to replace all dataset titles in an input text.\n3. Pass all input text from Support Set into a BertModel and extract **MASK**/**NoMASK** embedding.\n4. In Query Set, we pass a raw input into a shared BertModel used for Support Set to extract a sequence tokens embedding.\n5. **MASK** and **NoMask** embedding from Support/Query Set is passed into an ArcFace Metric Learning layer.\n6. A **MASK** embedding from the Support Set contains only a semantic context feature is used to calculate l2 norm cosine similarity with every token feature from the input text in the Query Set.\n7. A l2 norm cosine similarity will range from (-1, 1), we scale it into a range (-10, 10) then apply sigmoid activation before use it for training the NER model via BCE loss.\n\n\n### How to use my trained model for the inference?\n\n1. Pre-extract all **MASK** embedding from the given training set.\n2. Random selection K **MASK** embedding then average them to obtain the semantic context vector and perform step (6), (7) from the above training process.\n \n\n### Post-Processing\n\nAs @leecming said in his solution, a dataset is probably referenced by multiple documents so the more frequently they’re referenced, the more likely they’re a hit and not a spurious find. We use a threshold frequency of 2, which means if our model can detect that dataset title twice, we will get that dataset. We choose 2 because, in training data, there are many dataset titles mentioned only 1-2 times, we are afraid in the private dataset, such dataset titles will be many but we seem to be wrong, a higher threshold frequency gives better results on public and private leaderboards.\n\nHere is a summary of my solution, there are a few string rules-based that were used but did not affect the final result too much. The final submission is based on an ensemble of allenai/scibert and allenai/biomed_roberta. The code is done on Tensorflow 2.4 with V100 16gb version in AWS EC2. \n\n\n**Inference Kernel**: https://www.kaggle.com/dathudeptrai/biomed-roberta-scibert-base\n\n##### Thanks for reading !", "1361640": "Congratulations! Cố lên anh ơi!", "1361671": "Congrats on your hardwork paying off!", "1361704": "Congrats ✊", "1361768": "Nghỉ mấy hôm rồi quay lại nhanh nha :))", "1361824": "Congrats to the 1st place.", "1362169": "Thanks @suicaokhoailang :D.", "1362178": "Wow amazing solution! Thanks for sharing!", "1362197": "Amazing work 😍! Congrats\nP/s: rep me idol Anh Minh", "1362222": "great work. hope to collaborate with you again soon ^^", "1362223": "Just to understand few things: Is purpose of ARcFace loss is to ensure that embeddings for mask and statement from same document should be similar. What is the shape of Mask/No mask embeddings?\nTrying to understand. Any description around intuition will help.\nYour solution is quite innovative.", "1362266": "Congrats 💪", "1362307": "Very cool solution @dathudeptrai, congratz on the win !", "1362538": "Congratulations!", "1363146": "Hi @adityakumarsinha , the purpose of Arcface loss is to ensure that the embeddings for the MASK from all positive samples are similar. The shape of Mask/NoMask Embedding is [B, F], which is an average feature of all MASK/NoMASK token in the input text.", "1363148": "Thanks @theoviel, hope to work with you again soon :D", "1363163": "Congrats and nice solution !", "1363182": "quanbk Thanks, hope we can work together in the future :)))", "1364178": "Update: Surprisingly, remove the label from scispacy got a better result (0.588 on private), we should trust only our model :D. https://www.kaggle.com/dathudeptrai/biomed-roberta-scibert-base", "1367021": "Congrats! Nice solution!", "1370095": "Interesting solution, congrats!", "1370898": "Nice solution, Congratulations!!", "1371506": "Nice, congratulations..." }, "source": "meta" }