{
  "id": 329370,
  "title": "Metric Learning Approach",
  "url": "/competitions/AI4Code/discussion/329370",
  "author_name": "The Devastator",
  "post_date": "2022-06-06T10:58:23.574000",
  "votes": 11,
  "comment_count": 0,
  "views": 0,
  "content": "<h1>Metric Learning Approach</h1>\n<p>I was reading some papers recently and It just hit me how useful it might be for this competition. </p>\n<p><a href=\"https://arxiv.org/abs/2104.08821\" target=\"_blank\">SimCSE: Simple Contrastive Learning of Sentence Embeddings</a></p>\n<blockquote>\n  <p>They use \"SimCLR-like\" contrastive setup on text using heavy dropout in the network architecture as augmentations. </p>\n</blockquote>\n<p>Basically speaking, this paper presents a simple contrastive learning framework that greatly advances state-of-the-art sentence embeddings. It takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previously supervised counterparts. They show that the contrastive learning objective better aligns positive pairs when supervised signals are available.</p>\n<h4>What does all of this has to do with the competition?</h4>\n<p>In the paper, they also use the same framework for aligning pairs of different samples from the training set together (based on a label from the training set).</p>\n<p><strong>My idea is to do the same for pairs of consecutive code &amp; markdown cells</strong></p>\n<h4>All of this is extremely easy to implement</h4>\n<ul>\n<li>You start with defining your language model encoder with dropout</li>\n</ul>\n<pre><code>bert = AutoModel.from_pretrained(\"bert-base-cased\")\n\nclass bert_encoder(nn.Module):\n    def __init__(self, bert, dropout_p = 0.5):\n        super(bert_encoder, self).__init__()\n        self.bert = bert\n        self.dropout = nn.Dropout(dropout_p)\n        self.linear = nn.Linear(768, fc_dim)\n\n    def forward(self, seq):\n        outputs, _= self.bert(seq)\n        return self.linear(outputs[:,0,:]) # Output from the encoder is 1 * batch * 768\n\nencoder = bert_encoder(bert, dropout_p)\ncross_encoding = nn.Linear(fc_dim*2, 512)\nencoder_fc = nn.Sequential(nn.Dropout(dropout_p), nn.Linear(fc_dim, fc_dim))\n</code></pre>\n<ul>\n<li>Then you implement the contrastive loss</li>\n</ul>\n<pre><code>def contrastive_loss(encoded1, encoded2, sim, margin=1.0):\n    # compute the Euclidean distance function\n    dist = F.pairwise_distance(encoded1, encoded2, keepdim=True)\n    dist_sq = dist * dist\n    dist_sq = dist_sq.t()\n    mdist = margin - dist\n    dist = torch.max(dist, mdist)\n    loss = torch.mean(sim-dist_sq)\n    return loss\n</code></pre>\n<ul>\n<li>Followed by SimCLR class</li>\n</ul>\n<pre><code>class SimCLR(torch.nn.Module):\n  def __init__(self, encoder):\n    super(SimCLR, self).__init__()\n    self.encoder = encoder\n    self.cd = nn.CosineSimilarity(dim=-1, eps=1e-6)\n\n  def encode(self, x):\n    features = self.encoding(x)\n    return features\n\n  def forward(self, target_1_seq, target_2_seq):\n    target_1_features = self.encoder(target_1_seq)\n    target_2_features = self.encoder(target_2_seq)\n    target_loss = contrastive_loss(target_1_features, target_2_features, self.cd(target_1_features, target_2_features))\n    return {'total_loss': target_loss}\n</code></pre>\n<p>Apologies for not implementing this properly in a notebook, I got a tight schedule this week.<br>\nI will do so in a public notebook in the next upcoming days. </p>\n<p>What do you think?</p>",
  "messages": [
    {
      "id": 1812912,
      "postDate": "2022-06-06T10:58:23.573Z",
      "content": "<h1>Metric Learning Approach</h1>\n<p>I was reading some papers recently and It just hit me how useful it might be for this competition. </p>\n<p><a href=\"https://arxiv.org/abs/2104.08821\" target=\"_blank\">SimCSE: Simple Contrastive Learning of Sentence Embeddings</a></p>\n<blockquote>\n  <p>They use \"SimCLR-like\" contrastive setup on text using heavy dropout in the network architecture as augmentations. </p>\n</blockquote>\n<p>Basically speaking, this paper presents a simple contrastive learning framework that greatly advances state-of-the-art sentence embeddings. It takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previously supervised counterparts. They show that the contrastive learning objective better aligns positive pairs when supervised signals are available.</p>\n<h4>What does all of this has to do with the competition?</h4>\n<p>In the paper, they also use the same framework for aligning pairs of different samples from the training set together (based on a label from the training set).</p>\n<p><strong>My idea is to do the same for pairs of consecutive code &amp; markdown cells</strong></p>\n<h4>All of this is extremely easy to implement</h4>\n<ul>\n<li>You start with defining your language model encoder with dropout</li>\n</ul>\n<pre><code>bert = AutoModel.from_pretrained(\"bert-base-cased\")\n\nclass bert_encoder(nn.Module):\n    def __init__(self, bert, dropout_p = 0.5):\n        super(bert_encoder, self).__init__()\n        self.bert = bert\n        self.dropout = nn.Dropout(dropout_p)\n        self.linear = nn.Linear(768, fc_dim)\n\n    def forward(self, seq):\n        outputs, _= self.bert(seq)\n        return self.linear(outputs[:,0,:]) # Output from the encoder is 1 * batch * 768\n\nencoder = bert_encoder(bert, dropout_p)\ncross_encoding = nn.Linear(fc_dim*2, 512)\nencoder_fc = nn.Sequential(nn.Dropout(dropout_p), nn.Linear(fc_dim, fc_dim))\n</code></pre>\n<ul>\n<li>Then you implement the contrastive loss</li>\n</ul>\n<pre><code>def contrastive_loss(encoded1, encoded2, sim, margin=1.0):\n    # compute the Euclidean distance function\n    dist = F.pairwise_distance(encoded1, encoded2, keepdim=True)\n    dist_sq = dist * dist\n    dist_sq = dist_sq.t()\n    mdist = margin - dist\n    dist = torch.max(dist, mdist)\n    loss = torch.mean(sim-dist_sq)\n    return loss\n</code></pre>\n<ul>\n<li>Followed by SimCLR class</li>\n</ul>\n<pre><code>class SimCLR(torch.nn.Module):\n  def __init__(self, encoder):\n    super(SimCLR, self).__init__()\n    self.encoder = encoder\n    self.cd = nn.CosineSimilarity(dim=-1, eps=1e-6)\n\n  def encode(self, x):\n    features = self.encoding(x)\n    return features\n\n  def forward(self, target_1_seq, target_2_seq):\n    target_1_features = self.encoder(target_1_seq)\n    target_2_features = self.encoder(target_2_seq)\n    target_loss = contrastive_loss(target_1_features, target_2_features, self.cd(target_1_features, target_2_features))\n    return {'total_loss': target_loss}\n</code></pre>\n<p>Apologies for not implementing this properly in a notebook, I got a tight schedule this week.<br>\nI will do so in a public notebook in the next upcoming days. </p>\n<p>What do you think?</p>",
      "rawMarkdown": "# Metric Learning Approach \n\nI was reading some papers recently and It just hit me how useful it might be for this competition. \n\n[SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://arxiv.org/abs/2104.08821)\n\n> They use \"SimCLR-like\" contrastive setup on text using heavy dropout in the network architecture as augmentations. \n\nBasically speaking, this paper presents a simple contrastive learning framework that greatly advances state-of-the-art sentence embeddings. It takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previously supervised counterparts. They show that the contrastive learning objective better aligns positive pairs when supervised signals are available.\n\n#### What does all of this has to do with the competition?\n\nIn the paper, they also use the same framework for aligning pairs of different samples from the training set together (based on a label from the training set).\n\n**My idea is to do the same for pairs of consecutive code & markdown cells**\n\n#### All of this is extremely easy to implement\n\n- You start with defining your language model encoder with dropout\n\n```python\n\nbert = AutoModel.from_pretrained(\"bert-base-cased\")\n\nclass bert_encoder(nn.Module):\n    def __init__(self, bert, dropout_p = 0.5):\n        super(bert_encoder, self).__init__()\n        self.bert = bert\n        self.dropout = nn.Dropout(dropout_p)\n        self.linear = nn.Linear(768, fc_dim)\n        \n    def forward(self, seq):\n        outputs, _= self.bert(seq)\n        return self.linear(outputs[:,0,:]) # Output from the encoder is 1 * batch * 768\n    \nencoder = bert_encoder(bert, dropout_p)\ncross_encoding = nn.Linear(fc_dim*2, 512)\nencoder_fc = nn.Sequential(nn.Dropout(dropout_p), nn.Linear(fc_dim, fc_dim))\n\n```\n\n- Then you implement the contrastive loss\n\n```python\ndef contrastive_loss(encoded1, encoded2, sim, margin=1.0):\n    # compute the Euclidean distance function\n    dist = F.pairwise_distance(encoded1, encoded2, keepdim=True)\n    dist_sq = dist * dist\n    dist_sq = dist_sq.t()\n    mdist = margin - dist\n    dist = torch.max(dist, mdist)\n    loss = torch.mean(sim-dist_sq)\n    return loss\n```\n\n- Followed by SimCLR class\n\n```python\nclass SimCLR(torch.nn.Module):\n  def __init__(self, encoder):\n    super(SimCLR, self).__init__()\n    self.encoder = encoder\n    self.cd = nn.CosineSimilarity(dim=-1, eps=1e-6)\n\n  def encode(self, x):\n    features = self.encoding(x)\n    return features\n\n  def forward(self, target_1_seq, target_2_seq):\n    target_1_features = self.encoder(target_1_seq)\n    target_2_features = self.encoder(target_2_seq)\n    target_loss = contrastive_loss(target_1_features, target_2_features, self.cd(target_1_features, target_2_features))\n    return {'total_loss': target_loss}\n\n```\n\n\nApologies for not implementing this properly in a notebook, I got a tight schedule this week.\nI will do so in a public notebook in the next upcoming days. \n\n\nWhat do you think?",
      "votes": 11
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1812912": "# Metric Learning Approach \n\nI was reading some papers recently and It just hit me how useful it might be for this competition. \n\n[SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://arxiv.org/abs/2104.08821)\n\n> They use \"SimCLR-like\" contrastive setup on text using heavy dropout in the network architecture as augmentations. \n\nBasically speaking, this paper presents a simple contrastive learning framework that greatly advances state-of-the-art sentence embeddings. It takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise. This simple method works surprisingly well, performing on par with previously supervised counterparts. They show that the contrastive learning objective better aligns positive pairs when supervised signals are available.\n\n#### What does all of this has to do with the competition?\n\nIn the paper, they also use the same framework for aligning pairs of different samples from the training set together (based on a label from the training set).\n\n**My idea is to do the same for pairs of consecutive code & markdown cells**\n\n#### All of this is extremely easy to implement\n\n- You start with defining your language model encoder with dropout\n\n```python\n\nbert = AutoModel.from_pretrained(\"bert-base-cased\")\n\nclass bert_encoder(nn.Module):\n    def __init__(self, bert, dropout_p = 0.5):\n        super(bert_encoder, self).__init__()\n        self.bert = bert\n        self.dropout = nn.Dropout(dropout_p)\n        self.linear = nn.Linear(768, fc_dim)\n        \n    def forward(self, seq):\n        outputs, _= self.bert(seq)\n        return self.linear(outputs[:,0,:]) # Output from the encoder is 1 * batch * 768\n    \nencoder = bert_encoder(bert, dropout_p)\ncross_encoding = nn.Linear(fc_dim*2, 512)\nencoder_fc = nn.Sequential(nn.Dropout(dropout_p), nn.Linear(fc_dim, fc_dim))\n\n```\n\n- Then you implement the contrastive loss\n\n```python\ndef contrastive_loss(encoded1, encoded2, sim, margin=1.0):\n    # compute the Euclidean distance function\n    dist = F.pairwise_distance(encoded1, encoded2, keepdim=True)\n    dist_sq = dist * dist\n    dist_sq = dist_sq.t()\n    mdist = margin - dist\n    dist = torch.max(dist, mdist)\n    loss = torch.mean(sim-dist_sq)\n    return loss\n```\n\n- Followed by SimCLR class\n\n```python\nclass SimCLR(torch.nn.Module):\n  def __init__(self, encoder):\n    super(SimCLR, self).__init__()\n    self.encoder = encoder\n    self.cd = nn.CosineSimilarity(dim=-1, eps=1e-6)\n\n  def encode(self, x):\n    features = self.encoding(x)\n    return features\n\n  def forward(self, target_1_seq, target_2_seq):\n    target_1_features = self.encoder(target_1_seq)\n    target_2_features = self.encoder(target_2_seq)\n    target_loss = contrastive_loss(target_1_features, target_2_features, self.cd(target_1_features, target_2_features))\n    return {'total_loss': target_loss}\n\n```\n\n\nApologies for not implementing this properly in a notebook, I got a tight schedule this week.\nI will do so in a public notebook in the next upcoming days. \n\n\nWhat do you think?"
  }
}