{
  "id": 336756,
  "title": "Decoupling Code and Markdown Representations w. Poly Encoders",
  "url": "/competitions/AI4Code/discussion/336756",
  "author_name": "Sarthak Bhatt",
  "post_date": "2022-07-12T21:14:08.724000",
  "votes": 8,
  "comment_count": 0,
  "views": 0,
  "content": "<p><a href=\"https://openreview.net/pdf?id=SkxgnnNFvH\" target=\"_blank\"><strong>Poly-Encoders</strong>: Architectures and Pre-training Strategies for Fast and Accurate <strong>Multi-Sentence Scoring</strong></a> @ ICLR 2020 by Meta AI</p>\n<p>There are on average 15.55 markdown cells per Jupyter notebook in the training data. Pointwise approaches like <a href=\"this\" target=\"_blank\">https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells</a> give good results but make for very inefficient inference and training. </p>\n<p>As all markdown cells in a given Jupyter notebook share the same code context, the inference could be made much faster by a decoupled processing of code context and markdown context. This means computing the representations of code cells only once per notebook and allowing all the markdown cells to reuse the computed representation. This is where Late interaction transformer architectures like Polyencoders and ColBERT come in. </p>\n<p>There is another advantage to decoupled representations. You could use a code specialized model like CodeBERT for code representations and a general model for markdown representations.</p>\n<p>In the Poly Encoder paper, the authors explore the task of multi-sentence scoring. Performance in this task is measured by prediction quality and prediction speed as scoring many sentences can be very slow. The authors study this problem in the context of information retrieval, where they achieve SOTA results.</p>\n<hr>\n<p><strong>My Proposal of a Late Interaction-based Architecture for Predicting Order of Jupyter Cells</strong></p>\n<p>The method uses two backbones w. a late interaction mechanism</p>\n<ul>\n<li><p><strong>Markdown Encoder</strong>: A general NLU model like RoBERTa or DeBERTa with a max sequence length of 64. Using a maximum sequence length of 64 is 64x faster during training and inference than using max sequence length of 512 because of O(N^2) dependency of self-attention (512/64)**2 = 64x faster. This markdown encoder is used for each markdown cell of a given Jupyter notebook. </p></li>\n<li><p><strong>Code Encoder</strong>: A code understanding model like CuBERT (which allows for a max sequence length of 2048) or CodeBERT w. a max seq length of 512. This is used to compute the representation of the code context of a notebook. This is only used once per Jupyter Notebook. All of the markdown cells in the notebook used the precomputed and cached code representations computed for the notebook.</p></li>\n<li><p><strong>Late-Interaction Mechanism</strong>: This is a cheap yet powerful interaction step between markdown encoder and code encoder. There are many ways to do this step. See the <a href=\"https://arxiv.org/pdf/2004.12832.pdf\" target=\"_blank\">ColBERT</a> paper for a detailed introduction to the late interaction mechanism.</p></li>\n</ul>",
  "messages": [
    {
      "id": 1853374,
      "postDate": "2022-07-12T21:14:08.723Z",
      "content": "<p><a href=\"https://openreview.net/pdf?id=SkxgnnNFvH\" target=\"_blank\"><strong>Poly-Encoders</strong>: Architectures and Pre-training Strategies for Fast and Accurate <strong>Multi-Sentence Scoring</strong></a> @ ICLR 2020 by Meta AI</p>\n<p>There are on average 15.55 markdown cells per Jupyter notebook in the training data. Pointwise approaches like <a href=\"this\" target=\"_blank\">https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells</a> give good results but make for very inefficient inference and training. </p>\n<p>As all markdown cells in a given Jupyter notebook share the same code context, the inference could be made much faster by a decoupled processing of code context and markdown context. This means computing the representations of code cells only once per notebook and allowing all the markdown cells to reuse the computed representation. This is where Late interaction transformer architectures like Polyencoders and ColBERT come in. </p>\n<p>There is another advantage to decoupled representations. You could use a code specialized model like CodeBERT for code representations and a general model for markdown representations.</p>\n<p>In the Poly Encoder paper, the authors explore the task of multi-sentence scoring. Performance in this task is measured by prediction quality and prediction speed as scoring many sentences can be very slow. The authors study this problem in the context of information retrieval, where they achieve SOTA results.</p>\n<hr>\n<p><strong>My Proposal of a Late Interaction-based Architecture for Predicting Order of Jupyter Cells</strong></p>\n<p>The method uses two backbones w. a late interaction mechanism</p>\n<ul>\n<li><p><strong>Markdown Encoder</strong>: A general NLU model like RoBERTa or DeBERTa with a max sequence length of 64. Using a maximum sequence length of 64 is 64x faster during training and inference than using max sequence length of 512 because of O(N^2) dependency of self-attention (512/64)**2 = 64x faster. This markdown encoder is used for each markdown cell of a given Jupyter notebook. </p></li>\n<li><p><strong>Code Encoder</strong>: A code understanding model like CuBERT (which allows for a max sequence length of 2048) or CodeBERT w. a max seq length of 512. This is used to compute the representation of the code context of a notebook. This is only used once per Jupyter Notebook. All of the markdown cells in the notebook used the precomputed and cached code representations computed for the notebook.</p></li>\n<li><p><strong>Late-Interaction Mechanism</strong>: This is a cheap yet powerful interaction step between markdown encoder and code encoder. There are many ways to do this step. See the <a href=\"https://arxiv.org/pdf/2004.12832.pdf\" target=\"_blank\">ColBERT</a> paper for a detailed introduction to the late interaction mechanism.</p></li>\n</ul>",
      "rawMarkdown": "[__Poly-Encoders__: Architectures and Pre-training Strategies for Fast and Accurate __Multi-Sentence Scoring__](https://openreview.net/pdf?id=SkxgnnNFvH) @ ICLR 2020 by Meta AI\n\nThere are on average 15.55 markdown cells per Jupyter notebook in the training data. Pointwise approaches like [https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells](this) give good results but make for very inefficient inference and training. \n\nAs all markdown cells in a given Jupyter notebook share the same code context, the inference could be made much faster by a decoupled processing of code context and markdown context. This means computing the representations of code cells only once per notebook and allowing all the markdown cells to reuse the computed representation. This is where Late interaction transformer architectures like Polyencoders and ColBERT come in. \n\nThere is another advantage to decoupled representations. You could use a code specialized model like CodeBERT for code representations and a general model for markdown representations.\n\nIn the Poly Encoder paper, the authors explore the task of multi-sentence scoring. Performance in this task is measured by prediction quality and prediction speed as scoring many sentences can be very slow. The authors study this problem in the context of information retrieval, where they achieve SOTA results.\n\n---\n__My Proposal of a Late Interaction-based Architecture for Predicting Order of Jupyter Cells__\n\nThe method uses two backbones w. a late interaction mechanism\n\n- __Markdown Encoder__: A general NLU model like RoBERTa or DeBERTa with a max sequence length of 64. Using a maximum sequence length of 64 is 64x faster during training and inference than using max sequence length of 512 because of O(N^2) dependency of self-attention (512/64)**2 = 64x faster. This markdown encoder is used for each markdown cell of a given Jupyter notebook. \n\n- __Code Encoder__: A code understanding model like CuBERT (which allows for a max sequence length of 2048) or CodeBERT w. a max seq length of 512. This is used to compute the representation of the code context of a notebook. This is only used once per Jupyter Notebook. All of the markdown cells in the notebook used the precomputed and cached code representations computed for the notebook.\n\n- __Late-Interaction Mechanism__: This is a cheap yet powerful interaction step between markdown encoder and code encoder. There are many ways to do this step. See the [ColBERT](https://arxiv.org/pdf/2004.12832.pdf) paper for a detailed introduction to the late interaction mechanism.",
      "votes": 8
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1853374": "[__Poly-Encoders__: Architectures and Pre-training Strategies for Fast and Accurate __Multi-Sentence Scoring__](https://openreview.net/pdf?id=SkxgnnNFvH) @ ICLR 2020 by Meta AI\n\nThere are on average 15.55 markdown cells per Jupyter notebook in the training data. Pointwise approaches like [https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells](this) give good results but make for very inefficient inference and training. \n\nAs all markdown cells in a given Jupyter notebook share the same code context, the inference could be made much faster by a decoupled processing of code context and markdown context. This means computing the representations of code cells only once per notebook and allowing all the markdown cells to reuse the computed representation. This is where Late interaction transformer architectures like Polyencoders and ColBERT come in. \n\nThere is another advantage to decoupled representations. You could use a code specialized model like CodeBERT for code representations and a general model for markdown representations.\n\nIn the Poly Encoder paper, the authors explore the task of multi-sentence scoring. Performance in this task is measured by prediction quality and prediction speed as scoring many sentences can be very slow. The authors study this problem in the context of information retrieval, where they achieve SOTA results.\n\n---\n__My Proposal of a Late Interaction-based Architecture for Predicting Order of Jupyter Cells__\n\nThe method uses two backbones w. a late interaction mechanism\n\n- __Markdown Encoder__: A general NLU model like RoBERTa or DeBERTa with a max sequence length of 64. Using a maximum sequence length of 64 is 64x faster during training and inference than using max sequence length of 512 because of O(N^2) dependency of self-attention (512/64)**2 = 64x faster. This markdown encoder is used for each markdown cell of a given Jupyter notebook. \n\n- __Code Encoder__: A code understanding model like CuBERT (which allows for a max sequence length of 2048) or CodeBERT w. a max seq length of 512. This is used to compute the representation of the code context of a notebook. This is only used once per Jupyter Notebook. All of the markdown cells in the notebook used the precomputed and cached code representations computed for the notebook.\n\n- __Late-Interaction Mechanism__: This is a cheap yet powerful interaction step between markdown encoder and code encoder. There are many ways to do this step. See the [ColBERT](https://arxiv.org/pdf/2004.12832.pdf) paper for a detailed introduction to the late interaction mechanism."
  }
}