This is my first time participating in an NLP competition using BERT. I participate in this competition just to learn how to use BERT and how to do pretraining task.
\nI want to share a few hints about how to improve the pairwise models. My current pairwise model (0.839 LB) is from a single pairwise model using cosine similarity. It achieves 0.832 at the first epoch but did not improve much later. The CV-LB score correlates very well (0.005~0.007) difference.
As is shown from my local experiments, a pairwise model using the concatenation of markdown+code performs much better than a model using only cosine similarity. However, the inference time of a pairwise model using concatenation is too long. For the moment, using cosine similarity seems to be a better choice. The embeddings we get can also be used for other purposes.
\nThe followings are hints to improve a pairwise model using cosine similarity,
\n1, Pretraining
\npretraining improves my score of distilbert model by around 0.01.
\n2, Sampling strategy
\nI made a topic for it. Currently, I have no idea how to beat the 0.95 threshold strategy.
\n3, Better embedding for cells
\nAs is shown by many articles, [CLS] is not suitable for sentence similarity tasks. A typical better way is using mean pooling.
\n4, Better ranking strategy.
\nMy current way to get the rank of cells is using an empirical function created from nowhere by myself. There is obviously a better way to it.
\n5, Global context information
\nFor a pairwise model, we lack the information from global context. I believe this will be the key point that decides the final winner.
\n6, Post-processing
\nThis part is tricky. I have no experience in NLP before so I have no idea atm.
I will make notebooks for the improvements I made when I find it is suitable to publish them.
\nps: please upvote the notebooks if you find them helpful :D
This is my first time participating in an NLP competition using BERT. I participate in this competition just to learn how to use BERT and how to do pretraining task.
\nI want to share a few hints about how to improve the pairwise models. My current pairwise model (0.839 LB) is from a single pairwise model using cosine similarity. It achieves 0.832 at the first epoch but did not improve much later. The CV-LB score correlates very well (0.005~0.007) difference.
As is shown from my local experiments, a pairwise model using the concatenation of markdown+code performs much better than a model using only cosine similarity. However, the inference time of a pairwise model using concatenation is too long. For the moment, using cosine similarity seems to be a better choice. The embeddings we get can also be used for other purposes.
\nThe followings are hints to improve a pairwise model using cosine similarity,
\n1, Pretraining
\npretraining improves my score of distilbert model by around 0.01.
\n2, Sampling strategy
\nI made a topic for it. Currently, I have no idea how to beat the 0.95 threshold strategy.
\n3, Better embedding for cells
\nAs is shown by many articles, [CLS] is not suitable for sentence similarity tasks. A typical better way is using mean pooling.
\n4, Better ranking strategy.
\nMy current way to get the rank of cells is using an empirical function created from nowhere by myself. There is obviously a better way to it.
\n5, Global context information
\nFor a pairwise model, we lack the information from global context. I believe this will be the key point that decides the final winner.
\n6, Post-processing
\nThis part is tricky. I have no experience in NLP before so I have no idea atm.
I will make notebooks for the improvements I made when I find it is suitable to publish them.
\nps: please upvote the notebooks if you find them helpful :D
Thank you for shearing your tips.
", "rawMarkdown": "Thank you for shearing your tips.", "votes": null }, { "id": "1804792", "postDate": "05/29/2022 13:13:08", "content": "Thanks ,for sharing the tips. This will help to make the learning curve smooth.
", "rawMarkdown": "Thanks ,for sharing the tips. This will help to make the learning curve smooth.", "votes": null }, { "id": "1834608", "postDate": "06/27/2022 04:43:09", "content": "\"Using only cosine similarity\" - does this mean getting single sentence (CLS?) embeddings for each cell and then just applying a cos sim without any training and using those sim scores somehow to create the final rank or was there training involved in that using some cos-sim-loss or something ?
\nI was unable to find the pure cos sim notebook, if it was published can you share the link! ? Thanks!
Thank you for shearing your tips.
", "votes": null, "replies": [] }, { "id": 1804792, "author_name": "mahendrasinghrajpoot", "author_url": "", "post_date": "05/29/2022 13:13:08", "content": "Thanks ,for sharing the tips. This will help to make the learning curve smooth.
", "votes": null, "replies": [] }, { "id": 1834608, "author_name": "thedoors0", "author_url": "", "post_date": "06/27/2022 04:43:09", "content": "\"Using only cosine similarity\" - does this mean getting single sentence (CLS?) embeddings for each cell and then just applying a cos sim without any training and using those sim scores somehow to create the final rank or was there training involved in that using some cos-sim-loss or something ?
\nI was unable to find the pure cos sim notebook, if it was published can you share the link! ? Thanks!