Thanks a lot to the host and Kaggle for this interesting competition and congrats to the winners.
\nOur solution is a great collaborative effort.
\nThank you @crodoc @thedrcat @ivanaerlic @tascj0 for teaming up.
Our solution is an ensemble of several transformer based models, which take a full essay (including special tokens) or a concatenation of all discourse_text in the essay as input.
\nKey points of our solution are a powerful MLM pre-training and soft pseudo labeling on previous competition data. Our MLM pipeline always forced special tokens to be masked and predicted which was important to detect span boundaries.
For cross-validation we used MultilabelStratifiedKFold on discourse effectiveness and topic clusters to make efficient split on essays.
\nThese folds were really stable and CV to LB correlation was great.
\nMost of our blending submissions have the same private and public lb score.
Overall, we followed the token classification approach.
\nOur main modeling approach has input has follow:
\n\n[CLS] [cls_lead] Hi, i'm Isaac, i'm going to be writing about how this face on Mars is a natural landform or if there is life on Mars that made it. The story is about how NASA took a picture of Mars and a face was seen on the planet. NASA doesn't know if the landform was created by life on Mars, or if it is just a natural landform. [end_lead] [cls_position] On my perspective, I think that the face is a natural landform because I dont think that there is any life on Mars. In these next few paragraphs, I'll be talking about how I think that is is a natural landform [end_position]…..[SEP]
\n
We feed this sample to the backbone and take cls embeddings or mean pooling between cls and end tokens for each discourse text.
\ncls embeddings works better than mean pooling for us.
Model Backbones
\nOur best single model scored 0.562 with 8 min inference time.
\nThis single model finished 2nd place in the efficiency track and would make a gold medal score in the accuracy track.
\nUnfortunately this model was not included in any blend of the final submission for accuracy.
\nPlease refer to this post to read about our efficiency solution.
Removing Dropout
\nRandom mask augmentation (token dropout)
\nAre there any papers on these two technologies that I can refer to?
\nI also tried to use AWP to optimize my algorithm but nan appeared during training, don't know if you have encountered the same situation
No nan for us.
\nyou can use the code below to identify pytorch modules leading to nans.
def nan_hook(name):\n def hook(m, input, output):\n if not torch.isfinite(output).all():\n print(\"Invalid output in {}\".format(name))\n return hook\n\nfor name, module in model.named_modules():\n module.register_forward_hook(nan_hook(name))\nAny references to learn Adversarial Weight Perturbation?
", "votes": 0, "replies": [ { "id": 1915333, "author_name": "Vadim Irtlach", "author_url": "", "post_date": "2022-08-26T22:08:51.763000", "content": "Google :).
", "votes": -2, "replies": [] } ] } ], "comment_count": 4, "messages": [ { "id": 1914198, "postDate": "2022-08-25T20:25:16.113Z", "content": "Thanks a lot to the host and Kaggle for this interesting competition and congrats to the winners.
\nOur solution is a great collaborative effort.
\nThank you @crodoc @thedrcat @ivanaerlic @tascj0 for teaming up.
Our solution is an ensemble of several transformer based models, which take a full essay (including special tokens) or a concatenation of all discourse_text in the essay as input.
\nKey points of our solution are a powerful MLM pre-training and soft pseudo labeling on previous competition data. Our MLM pipeline always forced special tokens to be masked and predicted which was important to detect span boundaries.
For cross-validation we used MultilabelStratifiedKFold on discourse effectiveness and topic clusters to make efficient split on essays.
\nThese folds were really stable and CV to LB correlation was great.
\nMost of our blending submissions have the same private and public lb score.
Overall, we followed the token classification approach.
\nOur main modeling approach has input has follow:
\n\n[CLS] [cls_lead] Hi, i'm Isaac, i'm going to be writing about how this face on Mars is a natural landform or if there is life on Mars that made it. The story is about how NASA took a picture of Mars and a face was seen on the planet. NASA doesn't know if the landform was created by life on Mars, or if it is just a natural landform. [end_lead] [cls_position] On my perspective, I think that the face is a natural landform because I dont think that there is any life on Mars. In these next few paragraphs, I'll be talking about how I think that is is a natural landform [end_position]…..[SEP]
\n
We feed this sample to the backbone and take cls embeddings or mean pooling between cls and end tokens for each discourse text.
\ncls embeddings works better than mean pooling for us.
Model Backbones
\nOur best single model scored 0.562 with 8 min inference time.
\nThis single model finished 2nd place in the efficiency track and would make a gold medal score in the accuracy track.
\nUnfortunately this model was not included in any blend of the final submission for accuracy.
\nPlease refer to this post to read about our efficiency solution.
Removing Dropout
\nRandom mask augmentation (token dropout)
\nAre there any papers on these two technologies that I can refer to?
\nI also tried to use AWP to optimize my algorithm but nan appeared during training, don't know if you have encountered the same situation
No nan for us.
\nyou can use the code below to identify pytorch modules leading to nans.
def nan_hook(name):\n def hook(m, input, output):\n if not torch.isfinite(output).all():\n print(\"Invalid output in {}\".format(name))\n return hook\n\nfor name, module in model.named_modules():\n module.register_forward_hook(nan_hook(name))\nAny references to learn Adversarial Weight Perturbation?
", "rawMarkdown": "Any references to learn Adversarial Weight Perturbation?\n", "replies": [ { "id": 1915333, "postDate": "2022-08-26T22:08:51.763Z", "content": "Google :).
", "rawMarkdown": "Google :). ", "votes": -2 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:52:59Z", "competition_id": 35308, "competition_slug": "feedback-prize-effectiveness", "team_id": 8672529, "solution_url": "/competitions/feedback-prize-effectiveness/writeups/now-you-see-me-now-you-see-me-7th-place-solution" }