{ "id": 260939, "title": "Public&Private 37th solution [Team CM, T0m Part]", "content": "

Firstly, thanks to kaggle and hosts for an interesting competition, and to my teammate @teeyee314 , @nazarov , and @leecming . I learned a lot from them during this competition.
\nSince this is my first NLP competition, I started by reading BERT paper and thought a lot about how to train BERT stable and how to design the regression model.

\n

So, I'd like to discuss about what I've done to achieve the above.

\n

The followings are main idea of my part.

\n\n

Binary Pre-training

\n

I did an additional pre-training that was inspired by the roots of the competition target. They are actually Bradley-Terry values and from pairs of excerpts.
\nSo, I first designed binary-classification model, and then use the weights for main regression task.

\n

\n

Note:

\n\n

The pre-training weight boost my CV and LB by 0.002 ~ 0.01.

\n

Weighted Loss

\n

To make training more stable, I used the standard_error for calculating the training loss.
\nthe details are below,
\n1, scaling standard_error range {0. ~ 0.5}
\n2, calculating MSELoss
\n3, Weighting the each loss by using scaled standard_error
\n4, Mean and Backward

\n
# pytorch brief example code\n# scaling\nvalues = train[\"standard_error\"].values\ntrain[\"standard_error\"] = 0.5 * (values - np.min(values)) / (np.max(values) - np.min(values))\n~~~\n# training\ncriterion = nn.MSELoss(reduction='none')\n~~~\nloss = (criterion(output.squeeze(), labels) * (1 - errors)).mean()\nloss.backward()\n
\n

Custom Aggregation and Head for each hidden layer

\n

I added the regression head for each bert hidden layers, and then (learnable) weighted sum for predicting final target value.
\nEach aggregation and head weights are not shared.

\n

\n
class AggregationLayer(nn.Module):\n    def __init__(self, in_size, max_length):\n        super().__init__()\n        weights_init = torch.zeros(max_length).float()\n        self.seq_weights = torch.nn.Parameter(weights_init)\n        weights_init = torch.zeros(in_size).float()\n        weights_init.data[1:] = -1\n        self.dim_weights = torch.nn.Parameter(weights_init)\n\n    def forward(self, x):\n        w_dim = torch.softmax(self.dim_weights, dim=0)\n        w_seq = torch.softmax(self.seq_weights, dim=0)\n        x_dim = (x * w_dim).sum(1)\n        x_seq = (x * w_seq.unsqueeze(-1).expand(x.shape).float()).sum(1)\n        z = (x_dim + x_seq) / 2\n        return z\n
\n

Other tips

\n\n

Model

\n

With the above implementation, I achieved CV=0.4723 and LB=0.460 (roberta-large)

\n

After Team Merge

\n

I made a lot of models with pseudo labeling (external data).

", "comments": [ { "id": 1445492, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-08-04T09:06:28.273000", "content": "

Nice writeup, congrats on results and thanks for sharing @tomyanabe

", "votes": 2, "replies": [] }, { "id": 1445493, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-08-04T09:06:28.273000", "content": "

Nice writeup, congrats on results and thanks for sharing @tomyanabe

", "votes": 2, "replies": [] }, { "id": 1445227, "author_name": "", "author_url": "", "post_date": "2021-08-04T08:11:05.210000", "content": "", "votes": 0, "replies": [] }, { "id": 1442780, "author_name": "huiqin", "author_url": "", "post_date": "2021-08-04T00:20:23.233000", "content": "

Thanks for sharing! Great job.

", "votes": 2, "replies": [] } ], "comment_count": 4, "messages": [ { "id": 1442484, "postDate": "2021-08-03T23:02:15.610Z", "content": "

Firstly, thanks to kaggle and hosts for an interesting competition, and to my teammate @teeyee314 , @nazarov , and @leecming . I learned a lot from them during this competition.
\nSince this is my first NLP competition, I started by reading BERT paper and thought a lot about how to train BERT stable and how to design the regression model.

\n

So, I'd like to discuss about what I've done to achieve the above.

\n

The followings are main idea of my part.

\n\n

Binary Pre-training

\n

I did an additional pre-training that was inspired by the roots of the competition target. They are actually Bradley-Terry values and from pairs of excerpts.
\nSo, I first designed binary-classification model, and then use the weights for main regression task.

\n

\n

Note:

\n\n

The pre-training weight boost my CV and LB by 0.002 ~ 0.01.

\n

Weighted Loss

\n

To make training more stable, I used the standard_error for calculating the training loss.
\nthe details are below,
\n1, scaling standard_error range {0. ~ 0.5}
\n2, calculating MSELoss
\n3, Weighting the each loss by using scaled standard_error
\n4, Mean and Backward

\n
# pytorch brief example code\n# scaling\nvalues = train[\"standard_error\"].values\ntrain[\"standard_error\"] = 0.5 * (values - np.min(values)) / (np.max(values) - np.min(values))\n~~~\n# training\ncriterion = nn.MSELoss(reduction='none')\n~~~\nloss = (criterion(output.squeeze(), labels) * (1 - errors)).mean()\nloss.backward()\n
\n

Custom Aggregation and Head for each hidden layer

\n

I added the regression head for each bert hidden layers, and then (learnable) weighted sum for predicting final target value.
\nEach aggregation and head weights are not shared.

\n

\n
class AggregationLayer(nn.Module):\n    def __init__(self, in_size, max_length):\n        super().__init__()\n        weights_init = torch.zeros(max_length).float()\n        self.seq_weights = torch.nn.Parameter(weights_init)\n        weights_init = torch.zeros(in_size).float()\n        weights_init.data[1:] = -1\n        self.dim_weights = torch.nn.Parameter(weights_init)\n\n    def forward(self, x):\n        w_dim = torch.softmax(self.dim_weights, dim=0)\n        w_seq = torch.softmax(self.seq_weights, dim=0)\n        x_dim = (x * w_dim).sum(1)\n        x_seq = (x * w_seq.unsqueeze(-1).expand(x.shape).float()).sum(1)\n        z = (x_dim + x_seq) / 2\n        return z\n
\n

Other tips

\n\n

Model

\n

With the above implementation, I achieved CV=0.4723 and LB=0.460 (roberta-large)

\n

After Team Merge

\n

I made a lot of models with pseudo labeling (external data).

", "rawMarkdown": "Firstly, thanks to kaggle and hosts for an interesting competition, and to my teammate @teeyee314 , @nazarov , and @leecming . I learned a lot from them during this competition.\nSince this is my first NLP competition, I started by reading BERT paper and thought a lot about how to train BERT stable and how to design the regression model.\n\nSo, I'd like to discuss about what I've done to achieve the above.\n\nThe followings are main idea of my part.\n- Binary Pretraining using competition data.\n- High Probability Mixout (at the same time, Dropout=0 in bert layer)\n- Weighted Loss by using Standard_error\n- Custom Aggregation and Head for each hidden layer\n\n# Binary Pre-training\nI did an additional pre-training that was inspired by the roots of the competition target. They are actually Bradley-Terry values and from pairs of excerpts.\nSo, I first designed binary-classification model, and then use the weights for main regression task.\n\n\n\nNote:\n- The bert's weights are shared.\n- In train phase, pairs are randomly sampled.\n- In validation phase, the pairs are should be a fixed one.\n- There are many sampling strategy(ex, sample the pairs whose target difference is greater than a threshold), but random is the best.\n\nThe pre-training weight boost my CV and LB by 0.002 ~ 0.01.\n\n# Weighted Loss\nTo make training more stable, I used the standard_error for calculating the training loss.\nthe details are below,\n1, scaling standard_error range {0. ~ 0.5}\n2, calculating MSELoss\n3, Weighting the each loss by using scaled standard_error\n4, Mean and Backward\n```python\n# pytorch brief example code\n# scaling\nvalues = train[\"standard_error\"].values\ntrain[\"standard_error\"] = 0.5 * (values - np.min(values)) / (np.max(values) - np.min(values))\n~~~\n# training\ncriterion = nn.MSELoss(reduction='none')\n~~~\nloss = (criterion(output.squeeze(), labels) * (1 - errors)).mean()\nloss.backward()\n```\n\n# Custom Aggregation and Head for each hidden layer\nI added the regression head for each bert hidden layers, and then (learnable) weighted sum for predicting final target value.\nEach aggregation and head weights are not shared. \n\n\n\n```python\nclass AggregationLayer(nn.Module):\n def __init__(self, in_size, max_length):\n super().__init__()\n weights_init = torch.zeros(max_length).float()\n self.seq_weights = torch.nn.Parameter(weights_init)\n weights_init = torch.zeros(in_size).float()\n weights_init.data[1:] = -1\n self.dim_weights = torch.nn.Parameter(weights_init)\n\n def forward(self, x):\n w_dim = torch.softmax(self.dim_weights, dim=0)\n w_seq = torch.softmax(self.seq_weights, dim=0)\n x_dim = (x * w_dim).sum(1)\n x_seq = (x * w_seq.unsqueeze(-1).expand(x.shape).float()).sum(1)\n z = (x_dim + x_seq) / 2\n return z\n```\n\n\n# Other tips\n- add token randomly improved CV and LB (p=0.05)\n- Strong Mixout improved CV and LB (p=0.4) (this may be an effect of the dropout being zero.)\n- max_length = 300\n\n# Model\nWith the above implementation, I achieved CV=0.4723 and LB=0.460 (roberta-large)\n\n# After Team Merge\nI made a lot of models with pseudo labeling (external data).", "votes": 19 }, { "id": 1445492, "postDate": "2021-08-04T09:06:28.273Z", "content": "

Nice writeup, congrats on results and thanks for sharing @tomyanabe

", "rawMarkdown": "Nice writeup, congrats on results and thanks for sharing @tomyanabe ", "votes": 2 }, { "id": 1445493, "postDate": "2021-08-04T09:06:28.273Z", "content": "

Nice writeup, congrats on results and thanks for sharing @tomyanabe

", "rawMarkdown": "Nice writeup, congrats on results and thanks for sharing @tomyanabe ", "votes": 2 }, { "id": 1445227, "postDate": "2021-08-04T08:11:05.210Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1442780, "postDate": "2021-08-04T00:20:23.233Z", "content": "

Thanks for sharing! Great job.

", "rawMarkdown": "Thanks for sharing! Great job.", "votes": 2 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:49:19Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 6741571, "solution_url": "/competitions/commonlitreadabilityprize/writeups/cm-public-private-37th-solution-team-cm-t0m-part" }