{
  "id": 248386,
  "title": "44th place solution",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/writeups/coleridge-regression-44th-place-solution",
  "author_name": "",
  "post_date": "2021-06-23T13:11:54.067Z",
  "votes": 29,
  "comment_count": 4,
  "views": 0,
  "content": "<p><strong>Overview</strong></p>\n<p>My solution is a straight-forward Roberta NER model, which scored 0.497 public (#1400~)  -&gt; 0.347 private (#44). I initially wanted to use it as part of a two step pipeline to extract candidates, and then plug another model in charge of choosing whether to keep the dataset or not. <br>\nHowever, the nature of the public leaderboard and the impossibility to build a reliable CV made this competition extremely frustrating, so I decided to spend my free time on something else :)</p>\n<p><strong>Main points</strong></p>\n<ul>\n<li>Architecture : Roberta + concatenation of all the output layers + few convolutions + 0/1 output</li>\n<li>Loss : Dice loss, from <a href=\"https://arxiv.org/abs/1911.02855v3\" target=\"_blank\">https://arxiv.org/abs/1911.02855v3</a></li>\n<li>I built clusters using cooccurrences of datasets in the training data, this allowed me to have a (not so reliable) validation scheme with no overlap, but also to sample data more cleverly during training. </li>\n<li>I trained a first model using crops of 256 tokens containing at least one dataset. This approach biases the model towards predicting more datasets, and since the metric penalizes FPs more than FNs, I figured out this won't be enough</li>\n<li>So I used this first model to generate ~3000 candidates, that I manually reviewed. This was quite boring, and since it's hard to tell what organizers actually considered to be datasets, I mostly relied on keywords (i.e. survey, study, data, dataset &amp; more) for identification</li>\n<li>A second model was retrained using positive and negative examples from extracted candidates, which is the one I ended up using for my final submission</li>\n<li>I did a bunch of post processing in order to remove false positives. I don't really know how well each of these worked on LB but the first three worked on my CV :<ul>\n<li>Only keep predictions of length &gt; 10</li>\n<li>Only keep predictions where the maximum token probability is &gt; 0.99</li>\n<li>Only keep predictions that contains a keyword such as the one above</li>\n<li>Try to merge similar predictions in order not to predict the same dataset twice</li></ul></li>\n<li>Few ideas that didn't work and that I didn't already forget about :<ul>\n<li>Replacing datasets from the training data with datasets from external sources </li>\n<li>Using the previous Coleridge competition data</li></ul></li>\n</ul>\n<p>Feel free to ask any questions !<br>\nAlso, my inference code is available <a href=\"https://www.kaggle.com/theoviel/coleridge-ner-inference\" target=\"_blank\">here</a> if you wish to dig into the details.</p>",
  "messages": [
    {
      "id": "1362406",
      "postDate": "06/23/2021 12:26:41",
      "content": "<p><strong>Overview</strong></p>\n<p>My solution is a straight-forward Roberta NER model, which scored 0.497 public (#1400~)  -&gt; 0.347 private (#44). I initially wanted to use it as part of a two step pipeline to extract candidates, and then plug another model in charge of choosing whether to keep the dataset or not. <br>\nHowever, the nature of the public leaderboard and the impossibility to build a reliable CV made this competition extremely frustrating, so I decided to spend my free time on something else :)</p>\n<p><strong>Main points</strong></p>\n<ul>\n<li>Architecture : Roberta + concatenation of all the output layers + few convolutions + 0/1 output</li>\n<li>Loss : Dice loss, from <a href=\"https://arxiv.org/abs/1911.02855v3\" target=\"_blank\">https://arxiv.org/abs/1911.02855v3</a></li>\n<li>I built clusters using cooccurrences of datasets in the training data, this allowed me to have a (not so reliable) validation scheme with no overlap, but also to sample data more cleverly during training. </li>\n<li>I trained a first model using crops of 256 tokens containing at least one dataset. This approach biases the model towards predicting more datasets, and since the metric penalizes FPs more than FNs, I figured out this won't be enough</li>\n<li>So I used this first model to generate ~3000 candidates, that I manually reviewed. This was quite boring, and since it's hard to tell what organizers actually considered to be datasets, I mostly relied on keywords (i.e. survey, study, data, dataset &amp; more) for identification</li>\n<li>A second model was retrained using positive and negative examples from extracted candidates, which is the one I ended up using for my final submission</li>\n<li>I did a bunch of post processing in order to remove false positives. I don't really know how well each of these worked on LB but the first three worked on my CV :<ul>\n<li>Only keep predictions of length &gt; 10</li>\n<li>Only keep predictions where the maximum token probability is &gt; 0.99</li>\n<li>Only keep predictions that contains a keyword such as the one above</li>\n<li>Try to merge similar predictions in order not to predict the same dataset twice</li></ul></li>\n<li>Few ideas that didn't work and that I didn't already forget about :<ul>\n<li>Replacing datasets from the training data with datasets from external sources </li>\n<li>Using the previous Coleridge competition data</li></ul></li>\n</ul>\n<p>Feel free to ask any questions !<br>\nAlso, my inference code is available <a href=\"https://www.kaggle.com/theoviel/coleridge-ner-inference\" target=\"_blank\">here</a> if you wish to dig into the details.</p>",
      "rawMarkdown": "**Overview**\n\nMy solution is a straight-forward Roberta NER model, which scored 0.497 public (#1400~)  -> 0.347 private (#44). I initially wanted to use it as part of a two step pipeline to extract candidates, and then plug another model in charge of choosing whether to keep the dataset or not. \nHowever, the nature of the public leaderboard and the impossibility to build a reliable CV made this competition extremely frustrating, so I decided to spend my free time on something else :)\n\n**Main points**\n\n- Architecture : Roberta + concatenation of all the output layers + few convolutions + 0/1 output\n- Loss : Dice loss, from https://arxiv.org/abs/1911.02855v3\n- I built clusters using cooccurrences of datasets in the training data, this allowed me to have a (not so reliable) validation scheme with no overlap, but also to sample data more cleverly during training. \n- I trained a first model using crops of 256 tokens containing at least one dataset. This approach biases the model towards predicting more datasets, and since the metric penalizes FPs more than FNs, I figured out this won't be enough\n- So I used this first model to generate ~3000 candidates, that I manually reviewed. This was quite boring, and since it's hard to tell what organizers actually considered to be datasets, I mostly relied on keywords (i.e. survey, study, data, dataset & more) for identification\n- A second model was retrained using positive and negative examples from extracted candidates, which is the one I ended up using for my final submission\n- I did a bunch of post processing in order to remove false positives. I don't really know how well each of these worked on LB but the first three worked on my CV :\n  - Only keep predictions of length > 10\n  - Only keep predictions where the maximum token probability is > 0.99\n  - Only keep predictions that contains a keyword such as the one above\n  - Try to merge similar predictions in order not to predict the same dataset twice\n- Few ideas that didn't work and that I didn't already forget about :\n  - Replacing datasets from the training data with datasets from external sources \n  - Using the previous Coleridge competition data\n\nFeel free to ask any questions !\nAlso, my inference code is available [here](https://www.kaggle.com/theoviel/coleridge-ner-inference) if you wish to dig into the details.",
      "votes": null
    },
    {
      "id": "1683027",
      "postDate": "02/09/2022 14:14:07",
      "content": "<p><a href=\"https://www.kaggle.com/theoviel\" target=\"_blank\">@theoviel</a> <br>\nthank you for sharing!</p>\n<p>After checking your inference code, I think you are doing token classification. However, <a href=\"https://github.com/ShannonAI/dice_loss_for_NLP/blob/master/models/bert_query_ner.py\" target=\"_blank\">the official implementation</a> of dice loss uses start_logit and end_logit, which does not seem to be a token classification.</p>\n<p>Is it possible to share the loss function of the code used for training and the loss = loss_fn (** kwag) part?</p>\n<p>I'd appreciate it if you could reply.</p>",
      "rawMarkdown": "theoviel \nthank you for sharing!\n\nAfter checking your inference code, I think you are doing token classification. However, [the official implementation](https://github.com/ShannonAI/dice_loss_for_NLP/blob/master/models/bert_query_ner.py) of dice loss uses start_logit and end_logit, which does not seem to be a token classification.\n\nIs it possible to share the loss function of the code used for training and the loss = loss_fn (** kwag) part?\n\nI'd appreciate it if you could reply.",
      "votes": null
    },
    {
      "id": "1688034",
      "postDate": "02/13/2022 10:39:39",
      "content": "<p>Here's the implementation I use :</p>\n<pre><code>import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch import Tensor\nfrom typing import Optional\n\n\nclass DiceLoss(nn.Module):\n    \"\"\"\n    Dice coefficient for short, is an F1-oriented statistic used to gauge the similarity of two sets.\n    Given two sets A and B, the vanilla dice coefficient between them is given as follows:\n        Dice(A, B)  = 2 * True_Positive / (2 * True_Positive + False_Positive + False_Negative)\n                    = 2 * |A and B| / (|A| + |B|)\n    Math Function:\n        U-NET: https://arxiv.org/abs/1505.04597.pdf\n        dice_loss(p, y) = 1 - numerator / denominator\n            numerator = 2 * \\sum_{1}^{t} p_i * y_i + smooth\n            denominator = \\sum_{1}^{t} p_i + \\sum_{1} ^{t} y_i + smooth\n        if square_denominator is True, the denominator is \\sum_{1}^{t} (p_i ** 2) + \\sum_{1} ^{t} (y_i ** 2) + smooth\n        V-NET: https://arxiv.org/abs/1606.04797.pdf\n    Args:\n        smooth (float, optional): a manual smooth value for numerator and denominator.\n        square_denominator (bool, optional): [True, False], specifies whether to square the denominator in the loss function.\n        with_logits (bool, optional): [True, False], specifies whether the input tensor is normalized by Sigmoid/Softmax funcs.\n        ohem_ratio: max ratio of positive/negative, defautls to 0.0, which means no ohem.\n        alpha: dsc alpha\n    Shape:\n        - input: (*)\n        - target: (*)\n        - mask: (*) 0,1 mask for the input sequence.\n        - Output: Scalar loss\n    Examples:\n        &gt;&gt;&gt; loss = DiceLoss(with_logits=True, ohem_ratio=0.1)\n        &gt;&gt;&gt; input = torch.FloatTensor([2, 1, 2, 2, 1])\n        &gt;&gt;&gt; input.requires_grad=True\n        &gt;&gt;&gt; target = torch.LongTensor([0, 1, 0, 0, 0])\n        &gt;&gt;&gt; output = loss(input, target)\n        &gt;&gt;&gt; output.backward()\n    \"\"\"\n\n    def __init__(\n        self,\n        smooth: Optional[float] = 1e-4,\n        square_denominator: Optional[bool] = False,\n        with_logits: Optional[bool] = True,\n        ohem_ratio: float = 0.0,\n        alpha: float = 0.0,\n        reduction: Optional[str] = \"mean\",\n        index_label_position=True,\n    ) -&gt; None:\n        super(DiceLoss, self).__init__()\n\n        self.reduction = reduction\n        self.with_logits = with_logits\n        self.smooth = smooth\n        self.square_denominator = square_denominator\n        self.ohem_ratio = ohem_ratio\n        self.alpha = alpha\n        self.index_label_position = index_label_position\n\n    def forward(\n        self, input: Tensor, target: Tensor, mask: Optional[Tensor] = None\n    ) -&gt; Tensor:\n        loss = self._binary_class(input, target, mask=mask)\n\n        if self.reduction == \"mean\":\n            return loss.mean()\n        if self.reduction == \"sum\":\n            return loss.sum()\n        return loss\n\n    def _compute_dice_loss(self, flat_input, flat_target):\n        flat_input = ((1 - flat_input) ** self.alpha) * flat_input\n        intersection = torch.sum(flat_input * flat_target, -1)\n        if not self.square_denominator:\n            loss = 1 - (\n                (2 * intersection + self.smooth)\n                / (flat_input.sum() + flat_target.sum() + self.smooth)\n            )\n        else:\n            loss = 1 - (\n                (2 * intersection + self.smooth)\n                / (\n                    torch.sum(\n                        torch.square(\n                            flat_input,\n                        ),\n                        -1,\n                    )\n                    + torch.sum(torch.square(flat_target), -1)\n                    + self.smooth\n                )\n            )\n\n        return loss\n\n    def _binary_class(self, input, target, mask=None):\n        flat_input = input.view(-1)\n        flat_target = target.view(-1).float()\n        flat_input = torch.sigmoid(flat_input) if self.with_logits else flat_input\n\n        if mask is not None:\n            mask = mask.view(-1).float()\n            flat_input = flat_input * mask\n            flat_target = flat_target * mask\n        else:\n            mask = torch.ones_like(target)\n\n        if self.ohem_ratio &gt; 0:\n            pos_example = target &gt; 0.5\n            neg_example = target &lt;= 0.5\n            mask_neg_num = mask &lt;= 0.5\n\n            pos_num = pos_example.sum() - (pos_example &amp; mask_neg_num).sum()\n            neg_num = neg_example.sum()\n            keep_num = min(int(pos_num * self.ohem_ratio), neg_num)\n\n            neg_scores = torch.masked_select(flat_input, neg_example.bool())\n            neg_scores_sort, _ = torch.sort(\n                neg_scores,\n            )\n            threshold = neg_scores_sort[-keep_num + 1]\n            cond = (flat_input &gt; threshold) | pos_example.view(-1)\n            ohem_mask = torch.where(cond, 1, 0)\n            flat_input = flat_input * ohem_mask\n            flat_target = flat_target * ohem_mask\n\n        return self._compute_dice_loss(flat_input, flat_target)\n\n    def __str__(self):\n        return f\"Dice Loss smooth:{self.smooth}, ohem: {self.ohem_ratio}, alpha: {self.alpha}\"\n\n    def __repr__(self):\n        return str(self)\n</code></pre>",
      "rawMarkdown": "Here's the implementation I use :\n\n```\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch import Tensor\nfrom typing import Optional\n\n\nclass DiceLoss(nn.Module):\n    \"\"\"\n    Dice coefficient for short, is an F1-oriented statistic used to gauge the similarity of two sets.\n    Given two sets A and B, the vanilla dice coefficient between them is given as follows:\n        Dice(A, B)  = 2 * True_Positive / (2 * True_Positive + False_Positive + False_Negative)\n                    = 2 * |A and B| / (|A| + |B|)\n    Math Function:\n        U-NET: https://arxiv.org/abs/1505.04597.pdf\n        dice_loss(p, y) = 1 - numerator / denominator\n            numerator = 2 * \\sum_{1}^{t} p_i * y_i + smooth\n            denominator = \\sum_{1}^{t} p_i + \\sum_{1} ^{t} y_i + smooth\n        if square_denominator is True, the denominator is \\sum_{1}^{t} (p_i ** 2) + \\sum_{1} ^{t} (y_i ** 2) + smooth\n        V-NET: https://arxiv.org/abs/1606.04797.pdf\n    Args:\n        smooth (float, optional): a manual smooth value for numerator and denominator.\n        square_denominator (bool, optional): [True, False], specifies whether to square the denominator in the loss function.\n        with_logits (bool, optional): [True, False], specifies whether the input tensor is normalized by Sigmoid/Softmax funcs.\n        ohem_ratio: max ratio of positive/negative, defautls to 0.0, which means no ohem.\n        alpha: dsc alpha\n    Shape:\n        - input: (*)\n        - target: (*)\n        - mask: (*) 0,1 mask for the input sequence.\n        - Output: Scalar loss\n    Examples:\n        >>> loss = DiceLoss(with_logits=True, ohem_ratio=0.1)\n        >>> input = torch.FloatTensor([2, 1, 2, 2, 1])\n        >>> input.requires_grad=True\n        >>> target = torch.LongTensor([0, 1, 0, 0, 0])\n        >>> output = loss(input, target)\n        >>> output.backward()\n    \"\"\"\n\n    def __init__(\n        self,\n        smooth: Optional[float] = 1e-4,\n        square_denominator: Optional[bool] = False,\n        with_logits: Optional[bool] = True,\n        ohem_ratio: float = 0.0,\n        alpha: float = 0.0,\n        reduction: Optional[str] = \"mean\",\n        index_label_position=True,\n    ) -> None:\n        super(DiceLoss, self).__init__()\n\n        self.reduction = reduction\n        self.with_logits = with_logits\n        self.smooth = smooth\n        self.square_denominator = square_denominator\n        self.ohem_ratio = ohem_ratio\n        self.alpha = alpha\n        self.index_label_position = index_label_position\n\n    def forward(\n        self, input: Tensor, target: Tensor, mask: Optional[Tensor] = None\n    ) -> Tensor:\n        loss = self._binary_class(input, target, mask=mask)\n\n        if self.reduction == \"mean\":\n            return loss.mean()\n        if self.reduction == \"sum\":\n            return loss.sum()\n        return loss\n\n    def _compute_dice_loss(self, flat_input, flat_target):\n        flat_input = ((1 - flat_input) ** self.alpha) * flat_input\n        intersection = torch.sum(flat_input * flat_target, -1)\n        if not self.square_denominator:\n            loss = 1 - (\n                (2 * intersection + self.smooth)\n                / (flat_input.sum() + flat_target.sum() + self.smooth)\n            )\n        else:\n            loss = 1 - (\n                (2 * intersection + self.smooth)\n                / (\n                    torch.sum(\n                        torch.square(\n                            flat_input,\n                        ),\n                        -1,\n                    )\n                    + torch.sum(torch.square(flat_target), -1)\n                    + self.smooth\n                )\n            )\n\n        return loss\n\n    def _binary_class(self, input, target, mask=None):\n        flat_input = input.view(-1)\n        flat_target = target.view(-1).float()\n        flat_input = torch.sigmoid(flat_input) if self.with_logits else flat_input\n\n        if mask is not None:\n            mask = mask.view(-1).float()\n            flat_input = flat_input * mask\n            flat_target = flat_target * mask\n        else:\n            mask = torch.ones_like(target)\n\n        if self.ohem_ratio > 0:\n            pos_example = target > 0.5\n            neg_example = target <= 0.5\n            mask_neg_num = mask <= 0.5\n\n            pos_num = pos_example.sum() - (pos_example & mask_neg_num).sum()\n            neg_num = neg_example.sum()\n            keep_num = min(int(pos_num * self.ohem_ratio), neg_num)\n\n            neg_scores = torch.masked_select(flat_input, neg_example.bool())\n            neg_scores_sort, _ = torch.sort(\n                neg_scores,\n            )\n            threshold = neg_scores_sort[-keep_num + 1]\n            cond = (flat_input > threshold) | pos_example.view(-1)\n            ohem_mask = torch.where(cond, 1, 0)\n            flat_input = flat_input * ohem_mask\n            flat_target = flat_target * ohem_mask\n\n        return self._compute_dice_loss(flat_input, flat_target)\n\n    def __str__(self):\n        return f\"Dice Loss smooth:{self.smooth}, ohem: {self.ohem_ratio}, alpha: {self.alpha}\"\n\n    def __repr__(self):\n        return str(self)\n\n```",
      "votes": null
    },
    {
      "id": "1688147",
      "postDate": "02/13/2022 12:23:04",
      "content": "<p>What \"mask\" did you use for </p>\n<pre><code>loss = loss_fn(input, target,mask)\n</code></pre>\n<p>?</p>",
      "rawMarkdown": "What \"mask\" did you use for \n```\nloss = loss_fn(input, target,mask)\n```\n?",
      "votes": null
    },
    {
      "id": "1688169",
      "postDate": "02/13/2022 12:41:56",
      "content": "<p>I didn't use any</p>",
      "rawMarkdown": "I didn't use any",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1683027,
      "author_name": "abebe9849",
      "author_url": "",
      "post_date": "02/09/2022 14:14:07",
      "content": "<p><a href=\"https://www.kaggle.com/theoviel\" target=\"_blank\">@theoviel</a> <br>\nthank you for sharing!</p>\n<p>After checking your inference code, I think you are doing token classification. However, <a href=\"https://github.com/ShannonAI/dice_loss_for_NLP/blob/master/models/bert_query_ner.py\" target=\"_blank\">the official implementation</a> of dice loss uses start_logit and end_logit, which does not seem to be a token classification.</p>\n<p>Is it possible to share the loss function of the code used for training and the loss = loss_fn (** kwag) part?</p>\n<p>I'd appreciate it if you could reply.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1688034,
          "author_name": "theoviel",
          "author_url": "",
          "post_date": "02/13/2022 10:39:39",
          "content": "<p>Here's the implementation I use :</p>\n<pre><code>import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch import Tensor\nfrom typing import Optional\n\n\nclass DiceLoss(nn.Module):\n    \"\"\"\n    Dice coefficient for short, is an F1-oriented statistic used to gauge the similarity of two sets.\n    Given two sets A and B, the vanilla dice coefficient between them is given as follows:\n        Dice(A, B)  = 2 * True_Positive / (2 * True_Positive + False_Positive + False_Negative)\n                    = 2 * |A and B| / (|A| + |B|)\n    Math Function:\n        U-NET: https://arxiv.org/abs/1505.04597.pdf\n        dice_loss(p, y) = 1 - numerator / denominator\n            numerator = 2 * \\sum_{1}^{t} p_i * y_i + smooth\n            denominator = \\sum_{1}^{t} p_i + \\sum_{1} ^{t} y_i + smooth\n        if square_denominator is True, the denominator is \\sum_{1}^{t} (p_i ** 2) + \\sum_{1} ^{t} (y_i ** 2) + smooth\n        V-NET: https://arxiv.org/abs/1606.04797.pdf\n    Args:\n        smooth (float, optional): a manual smooth value for numerator and denominator.\n        square_denominator (bool, optional): [True, False], specifies whether to square the denominator in the loss function.\n        with_logits (bool, optional): [True, False], specifies whether the input tensor is normalized by Sigmoid/Softmax funcs.\n        ohem_ratio: max ratio of positive/negative, defautls to 0.0, which means no ohem.\n        alpha: dsc alpha\n    Shape:\n        - input: (*)\n        - target: (*)\n        - mask: (*) 0,1 mask for the input sequence.\n        - Output: Scalar loss\n    Examples:\n        &gt;&gt;&gt; loss = DiceLoss(with_logits=True, ohem_ratio=0.1)\n        &gt;&gt;&gt; input = torch.FloatTensor([2, 1, 2, 2, 1])\n        &gt;&gt;&gt; input.requires_grad=True\n        &gt;&gt;&gt; target = torch.LongTensor([0, 1, 0, 0, 0])\n        &gt;&gt;&gt; output = loss(input, target)\n        &gt;&gt;&gt; output.backward()\n    \"\"\"\n\n    def __init__(\n        self,\n        smooth: Optional[float] = 1e-4,\n        square_denominator: Optional[bool] = False,\n        with_logits: Optional[bool] = True,\n        ohem_ratio: float = 0.0,\n        alpha: float = 0.0,\n        reduction: Optional[str] = \"mean\",\n        index_label_position=True,\n    ) -&gt; None:\n        super(DiceLoss, self).__init__()\n\n        self.reduction = reduction\n        self.with_logits = with_logits\n        self.smooth = smooth\n        self.square_denominator = square_denominator\n        self.ohem_ratio = ohem_ratio\n        self.alpha = alpha\n        self.index_label_position = index_label_position\n\n    def forward(\n        self, input: Tensor, target: Tensor, mask: Optional[Tensor] = None\n    ) -&gt; Tensor:\n        loss = self._binary_class(input, target, mask=mask)\n\n        if self.reduction == \"mean\":\n            return loss.mean()\n        if self.reduction == \"sum\":\n            return loss.sum()\n        return loss\n\n    def _compute_dice_loss(self, flat_input, flat_target):\n        flat_input = ((1 - flat_input) ** self.alpha) * flat_input\n        intersection = torch.sum(flat_input * flat_target, -1)\n        if not self.square_denominator:\n            loss = 1 - (\n                (2 * intersection + self.smooth)\n                / (flat_input.sum() + flat_target.sum() + self.smooth)\n            )\n        else:\n            loss = 1 - (\n                (2 * intersection + self.smooth)\n                / (\n                    torch.sum(\n                        torch.square(\n                            flat_input,\n                        ),\n                        -1,\n                    )\n                    + torch.sum(torch.square(flat_target), -1)\n                    + self.smooth\n                )\n            )\n\n        return loss\n\n    def _binary_class(self, input, target, mask=None):\n        flat_input = input.view(-1)\n        flat_target = target.view(-1).float()\n        flat_input = torch.sigmoid(flat_input) if self.with_logits else flat_input\n\n        if mask is not None:\n            mask = mask.view(-1).float()\n            flat_input = flat_input * mask\n            flat_target = flat_target * mask\n        else:\n            mask = torch.ones_like(target)\n\n        if self.ohem_ratio &gt; 0:\n            pos_example = target &gt; 0.5\n            neg_example = target &lt;= 0.5\n            mask_neg_num = mask &lt;= 0.5\n\n            pos_num = pos_example.sum() - (pos_example &amp; mask_neg_num).sum()\n            neg_num = neg_example.sum()\n            keep_num = min(int(pos_num * self.ohem_ratio), neg_num)\n\n            neg_scores = torch.masked_select(flat_input, neg_example.bool())\n            neg_scores_sort, _ = torch.sort(\n                neg_scores,\n            )\n            threshold = neg_scores_sort[-keep_num + 1]\n            cond = (flat_input &gt; threshold) | pos_example.view(-1)\n            ohem_mask = torch.where(cond, 1, 0)\n            flat_input = flat_input * ohem_mask\n            flat_target = flat_target * ohem_mask\n\n        return self._compute_dice_loss(flat_input, flat_target)\n\n    def __str__(self):\n        return f\"Dice Loss smooth:{self.smooth}, ohem: {self.ohem_ratio}, alpha: {self.alpha}\"\n\n    def __repr__(self):\n        return str(self)\n</code></pre>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1688147,
          "author_name": "abebe9849",
          "author_url": "",
          "post_date": "02/13/2022 12:23:04",
          "content": "<p>What \"mask\" did you use for </p>\n<pre><code>loss = loss_fn(input, target,mask)\n</code></pre>\n<p>?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1688169,
          "author_name": "theoviel",
          "author_url": "",
          "post_date": "02/13/2022 12:41:56",
          "content": "<p>I didn't use any</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1362406": "**Overview**\n\nMy solution is a straight-forward Roberta NER model, which scored 0.497 public (#1400~)  -> 0.347 private (#44). I initially wanted to use it as part of a two step pipeline to extract candidates, and then plug another model in charge of choosing whether to keep the dataset or not. \nHowever, the nature of the public leaderboard and the impossibility to build a reliable CV made this competition extremely frustrating, so I decided to spend my free time on something else :)\n\n**Main points**\n\n- Architecture : Roberta + concatenation of all the output layers + few convolutions + 0/1 output\n- Loss : Dice loss, from https://arxiv.org/abs/1911.02855v3\n- I built clusters using cooccurrences of datasets in the training data, this allowed me to have a (not so reliable) validation scheme with no overlap, but also to sample data more cleverly during training. \n- I trained a first model using crops of 256 tokens containing at least one dataset. This approach biases the model towards predicting more datasets, and since the metric penalizes FPs more than FNs, I figured out this won't be enough\n- So I used this first model to generate ~3000 candidates, that I manually reviewed. This was quite boring, and since it's hard to tell what organizers actually considered to be datasets, I mostly relied on keywords (i.e. survey, study, data, dataset & more) for identification\n- A second model was retrained using positive and negative examples from extracted candidates, which is the one I ended up using for my final submission\n- I did a bunch of post processing in order to remove false positives. I don't really know how well each of these worked on LB but the first three worked on my CV :\n  - Only keep predictions of length > 10\n  - Only keep predictions where the maximum token probability is > 0.99\n  - Only keep predictions that contains a keyword such as the one above\n  - Try to merge similar predictions in order not to predict the same dataset twice\n- Few ideas that didn't work and that I didn't already forget about :\n  - Replacing datasets from the training data with datasets from external sources \n  - Using the previous Coleridge competition data\n\nFeel free to ask any questions !\nAlso, my inference code is available [here](https://www.kaggle.com/theoviel/coleridge-ner-inference) if you wish to dig into the details.",
    "1683027": "theoviel \nthank you for sharing!\n\nAfter checking your inference code, I think you are doing token classification. However, [the official implementation](https://github.com/ShannonAI/dice_loss_for_NLP/blob/master/models/bert_query_ner.py) of dice loss uses start_logit and end_logit, which does not seem to be a token classification.\n\nIs it possible to share the loss function of the code used for training and the loss = loss_fn (** kwag) part?\n\nI'd appreciate it if you could reply.",
    "1688034": "Here's the implementation I use :\n\n```\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch import Tensor\nfrom typing import Optional\n\n\nclass DiceLoss(nn.Module):\n    \"\"\"\n    Dice coefficient for short, is an F1-oriented statistic used to gauge the similarity of two sets.\n    Given two sets A and B, the vanilla dice coefficient between them is given as follows:\n        Dice(A, B)  = 2 * True_Positive / (2 * True_Positive + False_Positive + False_Negative)\n                    = 2 * |A and B| / (|A| + |B|)\n    Math Function:\n        U-NET: https://arxiv.org/abs/1505.04597.pdf\n        dice_loss(p, y) = 1 - numerator / denominator\n            numerator = 2 * \\sum_{1}^{t} p_i * y_i + smooth\n            denominator = \\sum_{1}^{t} p_i + \\sum_{1} ^{t} y_i + smooth\n        if square_denominator is True, the denominator is \\sum_{1}^{t} (p_i ** 2) + \\sum_{1} ^{t} (y_i ** 2) + smooth\n        V-NET: https://arxiv.org/abs/1606.04797.pdf\n    Args:\n        smooth (float, optional): a manual smooth value for numerator and denominator.\n        square_denominator (bool, optional): [True, False], specifies whether to square the denominator in the loss function.\n        with_logits (bool, optional): [True, False], specifies whether the input tensor is normalized by Sigmoid/Softmax funcs.\n        ohem_ratio: max ratio of positive/negative, defautls to 0.0, which means no ohem.\n        alpha: dsc alpha\n    Shape:\n        - input: (*)\n        - target: (*)\n        - mask: (*) 0,1 mask for the input sequence.\n        - Output: Scalar loss\n    Examples:\n        >>> loss = DiceLoss(with_logits=True, ohem_ratio=0.1)\n        >>> input = torch.FloatTensor([2, 1, 2, 2, 1])\n        >>> input.requires_grad=True\n        >>> target = torch.LongTensor([0, 1, 0, 0, 0])\n        >>> output = loss(input, target)\n        >>> output.backward()\n    \"\"\"\n\n    def __init__(\n        self,\n        smooth: Optional[float] = 1e-4,\n        square_denominator: Optional[bool] = False,\n        with_logits: Optional[bool] = True,\n        ohem_ratio: float = 0.0,\n        alpha: float = 0.0,\n        reduction: Optional[str] = \"mean\",\n        index_label_position=True,\n    ) -> None:\n        super(DiceLoss, self).__init__()\n\n        self.reduction = reduction\n        self.with_logits = with_logits\n        self.smooth = smooth\n        self.square_denominator = square_denominator\n        self.ohem_ratio = ohem_ratio\n        self.alpha = alpha\n        self.index_label_position = index_label_position\n\n    def forward(\n        self, input: Tensor, target: Tensor, mask: Optional[Tensor] = None\n    ) -> Tensor:\n        loss = self._binary_class(input, target, mask=mask)\n\n        if self.reduction == \"mean\":\n            return loss.mean()\n        if self.reduction == \"sum\":\n            return loss.sum()\n        return loss\n\n    def _compute_dice_loss(self, flat_input, flat_target):\n        flat_input = ((1 - flat_input) ** self.alpha) * flat_input\n        intersection = torch.sum(flat_input * flat_target, -1)\n        if not self.square_denominator:\n            loss = 1 - (\n                (2 * intersection + self.smooth)\n                / (flat_input.sum() + flat_target.sum() + self.smooth)\n            )\n        else:\n            loss = 1 - (\n                (2 * intersection + self.smooth)\n                / (\n                    torch.sum(\n                        torch.square(\n                            flat_input,\n                        ),\n                        -1,\n                    )\n                    + torch.sum(torch.square(flat_target), -1)\n                    + self.smooth\n                )\n            )\n\n        return loss\n\n    def _binary_class(self, input, target, mask=None):\n        flat_input = input.view(-1)\n        flat_target = target.view(-1).float()\n        flat_input = torch.sigmoid(flat_input) if self.with_logits else flat_input\n\n        if mask is not None:\n            mask = mask.view(-1).float()\n            flat_input = flat_input * mask\n            flat_target = flat_target * mask\n        else:\n            mask = torch.ones_like(target)\n\n        if self.ohem_ratio > 0:\n            pos_example = target > 0.5\n            neg_example = target <= 0.5\n            mask_neg_num = mask <= 0.5\n\n            pos_num = pos_example.sum() - (pos_example & mask_neg_num).sum()\n            neg_num = neg_example.sum()\n            keep_num = min(int(pos_num * self.ohem_ratio), neg_num)\n\n            neg_scores = torch.masked_select(flat_input, neg_example.bool())\n            neg_scores_sort, _ = torch.sort(\n                neg_scores,\n            )\n            threshold = neg_scores_sort[-keep_num + 1]\n            cond = (flat_input > threshold) | pos_example.view(-1)\n            ohem_mask = torch.where(cond, 1, 0)\n            flat_input = flat_input * ohem_mask\n            flat_target = flat_target * ohem_mask\n\n        return self._compute_dice_loss(flat_input, flat_target)\n\n    def __str__(self):\n        return f\"Dice Loss smooth:{self.smooth}, ohem: {self.ohem_ratio}, alpha: {self.alpha}\"\n\n    def __repr__(self):\n        return str(self)\n\n```",
    "1688147": "What \"mask\" did you use for \n```\nloss = loss_fn(input, target,mask)\n```\n?",
    "1688169": "I didn't use any"
  },
  "source": "existing_discussion",
  "complete": true,
  "competition_id": 25925,
  "competition_slug": "coleridgeinitiative-show-us-the-data",
  "team_id": 6535634,
  "solution_url": "/competitions/coleridgeinitiative-show-us-the-data/writeups/coleridge-regression-44th-place-solution"
}