{
  "id": 336753,
  "title": "Long-Range Modeling of Source Code Files with eWASH",
  "url": "/competitions/AI4Code/discussion/336753",
  "author_name": "",
  "post_date": "2022-07-12T20:29:14.243071500Z",
  "votes": 5,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Paper: <a href=\"https://arxiv.org/abs/2109.08780\" target=\"_blank\">Long-Range Modeling of Source Code Files with eWASH</a> by Microsoft Cloud</p>\n<p>Current Transformer-based models are unable to process long sequences because their self-attention operation is quadratic with respect to the number of input tokens. Some exceptions to this are Longformer, BigBird, and DeBERTa. This is important in this competition because most of the inputs take &gt;512 tokens.</p>\n<p>Limiting the context window means that the models will not be able to use the full context of long Jupyter notebooks and code cells.</p>\n<p>While there are many efforts (ex. Longformer) to extend the context window, eWASH is an architecture-independent approach that uses syntactic hierarchies of source code to incorporate the entire file-level (or in our case notebook level) context into a fixed-length window.</p>\n<p><a href=\"https://postimg.cc/nMPk0xym\" target=\"_blank\">eWASH Image Link</a></p>\n<p>eWASH assumes that names of entities at higher scopes (ex. function names, class names) are more relevant to the task than entities at lower scopes (ex. variables inside a function). After ordering each line by priority, the tokens are added till the token budget of the model context is filled.</p>\n<p>The authors used this method to achieve SOTA in the  CodeXGLUE benchmark. Although the method was introduced for code completion tasks, this could be relevant for this competition.</p>",
  "messages": [
    {
      "id": "1853334",
      "postDate": "07/12/2022 20:29:14",
      "content": "<p>Paper: <a href=\"https://arxiv.org/abs/2109.08780\" target=\"_blank\">Long-Range Modeling of Source Code Files with eWASH</a> by Microsoft Cloud</p>\n<p>Current Transformer-based models are unable to process long sequences because their self-attention operation is quadratic with respect to the number of input tokens. Some exceptions to this are Longformer, BigBird, and DeBERTa. This is important in this competition because most of the inputs take &gt;512 tokens.</p>\n<p>Limiting the context window means that the models will not be able to use the full context of long Jupyter notebooks and code cells.</p>\n<p>While there are many efforts (ex. Longformer) to extend the context window, eWASH is an architecture-independent approach that uses syntactic hierarchies of source code to incorporate the entire file-level (or in our case notebook level) context into a fixed-length window.</p>\n<p><a href=\"https://postimg.cc/nMPk0xym\" target=\"_blank\">eWASH Image Link</a></p>\n<p>eWASH assumes that names of entities at higher scopes (ex. function names, class names) are more relevant to the task than entities at lower scopes (ex. variables inside a function). After ordering each line by priority, the tokens are added till the token budget of the model context is filled.</p>\n<p>The authors used this method to achieve SOTA in the  CodeXGLUE benchmark. Although the method was introduced for code completion tasks, this could be relevant for this competition.</p>",
      "rawMarkdown": "Paper: [Long-Range Modeling of Source Code Files with eWASH](https://arxiv.org/abs/2109.08780) by Microsoft Cloud\n\nCurrent Transformer-based models are unable to process long sequences because their self-attention operation is quadratic with respect to the number of input tokens. Some exceptions to this are Longformer, BigBird, and DeBERTa. This is important in this competition because most of the inputs take >512 tokens.\n\nLimiting the context window means that the models will not be able to use the full context of long Jupyter notebooks and code cells.\n\nWhile there are many efforts (ex. Longformer) to extend the context window, eWASH is an architecture-independent approach that uses syntactic hierarchies of source code to incorporate the entire file-level (or in our case notebook level) context into a fixed-length window.\n\n[eWASH Image Link](https://postimg.cc/nMPk0xym)\n\neWASH assumes that names of entities at higher scopes (ex. function names, class names) are more relevant to the task than entities at lower scopes (ex. variables inside a function). After ordering each line by priority, the tokens are added till the token budget of the model context is filled.\n\nThe authors used this method to achieve SOTA in the  CodeXGLUE benchmark. Although the method was introduced for code completion tasks, this could be relevant for this competition.",
      "votes": null
    },
    {
      "id": "1853564",
      "postDate": "07/13/2022 00:47:17",
      "content": "<p>Were you able to find an implementation of ewash?</p>",
      "rawMarkdown": "Were you able to find an implementation of ewash?",
      "votes": null
    },
    {
      "id": "1855862",
      "postDate": "07/15/2022 01:44:23",
      "content": "<p>Thanks for sharing this paper! This is a really interesting approach that could be helpful for the competition. I'm curious to see if anyone has tried using this method and if it has helped improve their results.</p>",
      "rawMarkdown": "Thanks for sharing this paper! This is a really interesting approach that could be helpful for the competition. I'm curious to see if anyone has tried using this method and if it has helped improve their results.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1853564,
      "author_name": "brianyoon",
      "author_url": "",
      "post_date": "07/13/2022 00:47:17",
      "content": "<p>Were you able to find an implementation of ewash?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1855862,
      "author_name": "thedevastator",
      "author_url": "",
      "post_date": "07/15/2022 01:44:23",
      "content": "<p>Thanks for sharing this paper! This is a really interesting approach that could be helpful for the competition. I'm curious to see if anyone has tried using this method and if it has helped improve their results.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1853334": "Paper: [Long-Range Modeling of Source Code Files with eWASH](https://arxiv.org/abs/2109.08780) by Microsoft Cloud\n\nCurrent Transformer-based models are unable to process long sequences because their self-attention operation is quadratic with respect to the number of input tokens. Some exceptions to this are Longformer, BigBird, and DeBERTa. This is important in this competition because most of the inputs take >512 tokens.\n\nLimiting the context window means that the models will not be able to use the full context of long Jupyter notebooks and code cells.\n\nWhile there are many efforts (ex. Longformer) to extend the context window, eWASH is an architecture-independent approach that uses syntactic hierarchies of source code to incorporate the entire file-level (or in our case notebook level) context into a fixed-length window.\n\n[eWASH Image Link](https://postimg.cc/nMPk0xym)\n\neWASH assumes that names of entities at higher scopes (ex. function names, class names) are more relevant to the task than entities at lower scopes (ex. variables inside a function). After ordering each line by priority, the tokens are added till the token budget of the model context is filled.\n\nThe authors used this method to achieve SOTA in the  CodeXGLUE benchmark. Although the method was introduced for code completion tasks, this could be relevant for this competition.",
    "1853564": "Were you able to find an implementation of ewash?",
    "1855862": "Thanks for sharing this paper! This is a really interesting approach that could be helpful for the competition. I'm curious to see if anyone has tried using this method and if it has helped improve their results."
  },
  "source": "meta"
}