{
  "id": 336784,
  "title": "Evolution of Code Understanding Models after CodeBERT",
  "url": "/competitions/AI4Code/discussion/336784",
  "author_name": "",
  "post_date": "2022-07-12T23:31:02.659936900Z",
  "votes": 6,
  "comment_count": 5,
  "views": 0,
  "content": "<p><strong>GraphCodeBERT</strong><br>\nHuggingface Link: <a href=\"https://huggingface.co/microsoft/graphcodebert-base\" target=\"_blank\">https://huggingface.co/microsoft/graphcodebert-base</a><br>\n<a href=\"https://arxiv.org/pdf/2009.08366.pdf\" target=\"_blank\">GraphCodeBERT: Pre-training Code Representations with Data Flow</a> @ ICLR 2021</p>\n<p>CodeBERT regards a code snippet as a sequence of tokens and ignores the inherent structure of code. GraphCodeBERT is a pre-trained model for programming language that considers the inherent structure of code with a dataflow graph. If you have used TensorFlow, then you are probably already familiar with dataflow representations of a program. I have found GraphCodeBERT to always outperform CodeBERT in my experiments. </p>\n<hr>\n<p><strong>DOBF</strong><br>\n<a href=\"https://arxiv.org/pdf/2102.07492.pdf\" target=\"_blank\">DOBF: A Deobfuscation Pre-Training Objective for Programming Languages</a> @ NeurIPS 2021 by Meta AI</p>\n<p>The authors first obfuscate the source code and then ask the model to recover the original version of obfuscated source code. This leads to a 24% relative improvement in natural language code search.</p>\n<blockquote>\n  <p>In this paper, we introduce a new pre-training objective, DOBF, that leverages the structural aspect of programming languages and pre-trains a model to recover the original code. We show that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks.</p>\n</blockquote>\n<hr>\n<p><strong>UniXCoder</strong><br>\n<a href=\"https://arxiv.org/pdf/2203.03850.pdf\" target=\"_blank\">UniXcoder: Unified Cross-Modal Pre-training for Code Representation</a> @ ACL 2022<br>\nHuggingface Link: <a href=\"https://huggingface.co/microsoft/unixcoder-base\" target=\"_blank\">https://huggingface.co/microsoft/unixcoder-base</a></p>\n<p>UniXCoder can be used for both code understanding and generation. The model claims to achieve SOTA in many tasks but I have found it to underperform even CodeBERT. </p>\n<hr>\n<p>Other models: SynCoBERT, AstBERT, CLSEBERT</p>",
  "messages": [
    {
      "id": "1853498",
      "postDate": "07/12/2022 23:31:02",
      "content": "<p><strong>GraphCodeBERT</strong><br>\nHuggingface Link: <a href=\"https://huggingface.co/microsoft/graphcodebert-base\" target=\"_blank\">https://huggingface.co/microsoft/graphcodebert-base</a><br>\n<a href=\"https://arxiv.org/pdf/2009.08366.pdf\" target=\"_blank\">GraphCodeBERT: Pre-training Code Representations with Data Flow</a> @ ICLR 2021</p>\n<p>CodeBERT regards a code snippet as a sequence of tokens and ignores the inherent structure of code. GraphCodeBERT is a pre-trained model for programming language that considers the inherent structure of code with a dataflow graph. If you have used TensorFlow, then you are probably already familiar with dataflow representations of a program. I have found GraphCodeBERT to always outperform CodeBERT in my experiments. </p>\n<hr>\n<p><strong>DOBF</strong><br>\n<a href=\"https://arxiv.org/pdf/2102.07492.pdf\" target=\"_blank\">DOBF: A Deobfuscation Pre-Training Objective for Programming Languages</a> @ NeurIPS 2021 by Meta AI</p>\n<p>The authors first obfuscate the source code and then ask the model to recover the original version of obfuscated source code. This leads to a 24% relative improvement in natural language code search.</p>\n<blockquote>\n  <p>In this paper, we introduce a new pre-training objective, DOBF, that leverages the structural aspect of programming languages and pre-trains a model to recover the original code. We show that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks.</p>\n</blockquote>\n<hr>\n<p><strong>UniXCoder</strong><br>\n<a href=\"https://arxiv.org/pdf/2203.03850.pdf\" target=\"_blank\">UniXcoder: Unified Cross-Modal Pre-training for Code Representation</a> @ ACL 2022<br>\nHuggingface Link: <a href=\"https://huggingface.co/microsoft/unixcoder-base\" target=\"_blank\">https://huggingface.co/microsoft/unixcoder-base</a></p>\n<p>UniXCoder can be used for both code understanding and generation. The model claims to achieve SOTA in many tasks but I have found it to underperform even CodeBERT. </p>\n<hr>\n<p>Other models: SynCoBERT, AstBERT, CLSEBERT</p>",
      "rawMarkdown": "__GraphCodeBERT__\nHuggingface Link: https://huggingface.co/microsoft/graphcodebert-base\n[GraphCodeBERT: Pre-training Code Representations with Data Flow](https://arxiv.org/pdf/2009.08366.pdf) @ ICLR 2021\n\nCodeBERT regards a code snippet as a sequence of tokens and ignores the inherent structure of code. GraphCodeBERT is a pre-trained model for programming language that considers the inherent structure of code with a dataflow graph. If you have used TensorFlow, then you are probably already familiar with dataflow representations of a program. I have found GraphCodeBERT to always outperform CodeBERT in my experiments. \n\n---\n__DOBF__\n[DOBF: A Deobfuscation Pre-Training Objective for Programming Languages](https://arxiv.org/pdf/2102.07492.pdf) @ NeurIPS 2021 by Meta AI\n\nThe authors first obfuscate the source code and then ask the model to recover the original version of obfuscated source code. This leads to a 24% relative improvement in natural language code search.\n\n> In this paper, we introduce a new pre-training objective, DOBF, that leverages the structural aspect of programming languages and pre-trains a model to recover the original code. We show that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks.\n\n---\n__UniXCoder__\n[UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://arxiv.org/pdf/2203.03850.pdf) @ ACL 2022\nHuggingface Link: https://huggingface.co/microsoft/unixcoder-base\n\nUniXCoder can be used for both code understanding and generation. The model claims to achieve SOTA in many tasks but I have found it to underperform even CodeBERT. \n\n---\nOther models: SynCoBERT, AstBERT, CLSEBERT",
      "votes": null
    },
    {
      "id": "1853570",
      "postDate": "07/13/2022 00:56:26",
      "content": "<p>I also ran codebert, graphcodbert, and unixcoder, but unixcoder was the worst performance.<br>\nI wonder if the encoder-decoder model, such as bart and t5, needs to change the setting separately.<br>\nI lack knowledge of encoder-decoder models.😭😭 Do you happen to know anything?</p>\n<p>** The rest of the models don't seem to be visible on huggingface.</p>",
      "rawMarkdown": "I also ran codebert, graphcodbert, and unixcoder, but unixcoder was the worst performance.\nI wonder if the encoder-decoder model, such as bart and t5, needs to change the setting separately.\nI lack knowledge of encoder-decoder models.😭😭 Do you happen to know anything?\n\n\n** The rest of the models don't seem to be visible on huggingface.",
      "votes": null
    },
    {
      "id": "1855869",
      "postDate": "07/15/2022 01:54:22",
      "content": "<p>Thanks for your post.</p>\n<p>I think because GraphCodeBERT's considers the inherent structure of code with a dataflow graph, while CodeBERT regards a code snippet as a sequence of tokens and ignores the inherent structure of code it might be worth a shot if someone could get it right. Have you tried the other models mentioned in your post?</p>",
      "rawMarkdown": "Thanks for your post.\n\nI think because GraphCodeBERT's considers the inherent structure of code with a dataflow graph, while CodeBERT regards a code snippet as a sequence of tokens and ignores the inherent structure of code it might be worth a shot if someone could get it right. Have you tried the other models mentioned in your post?",
      "votes": null
    },
    {
      "id": "1858067",
      "postDate": "07/16/2022 16:08:53",
      "content": "<p>GraphCodeBERT is interesting approach. It is interesting, how to garantee correctness of generated code? Because language GPT models oftenly produce inconsistent text</p>",
      "rawMarkdown": "GraphCodeBERT is interesting approach. It is interesting, how to garantee correctness of generated code? Because language GPT models oftenly produce inconsistent text",
      "votes": null
    },
    {
      "id": "1863762",
      "postDate": "07/20/2022 13:50:02",
      "content": "<p>Thank you for nice info.<br>\nIn my same params experiment,  GraphCodeBERT (LB 0.8414) &gt; CodeBERT (LB 0.8398) &gt; UniXCoder (LB 0.795)<br>\nI haven't tried DOBF.</p>",
      "rawMarkdown": "Thank you for nice info.\nIn my same params experiment,  GraphCodeBERT (LB 0.8414) > CodeBERT (LB 0.8398) > UniXCoder (LB 0.795)\nI haven't tried DOBF.",
      "votes": null
    },
    {
      "id": "2013901",
      "postDate": "11/02/2022 07:44:28",
      "content": "<p>It takes too long to run, Unix code does not know why the experiment runs, but it does not run and no error is reported</p>",
      "rawMarkdown": "It takes too long to run, Unix code does not know why the experiment runs, but it does not run and no error is reported",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1853570,
      "author_name": "heuingttiang",
      "author_url": "",
      "post_date": "07/13/2022 00:56:26",
      "content": "<p>I also ran codebert, graphcodbert, and unixcoder, but unixcoder was the worst performance.<br>\nI wonder if the encoder-decoder model, such as bart and t5, needs to change the setting separately.<br>\nI lack knowledge of encoder-decoder models.😭😭 Do you happen to know anything?</p>\n<p>** The rest of the models don't seem to be visible on huggingface.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1855869,
      "author_name": "thedevastator",
      "author_url": "",
      "post_date": "07/15/2022 01:54:22",
      "content": "<p>Thanks for your post.</p>\n<p>I think because GraphCodeBERT's considers the inherent structure of code with a dataflow graph, while CodeBERT regards a code snippet as a sequence of tokens and ignores the inherent structure of code it might be worth a shot if someone could get it right. Have you tried the other models mentioned in your post?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1858067,
      "author_name": "zavodrobotov",
      "author_url": "",
      "post_date": "07/16/2022 16:08:53",
      "content": "<p>GraphCodeBERT is interesting approach. It is interesting, how to garantee correctness of generated code? Because language GPT models oftenly produce inconsistent text</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1863762,
      "author_name": "kojimar",
      "author_url": "",
      "post_date": "07/20/2022 13:50:02",
      "content": "<p>Thank you for nice info.<br>\nIn my same params experiment,  GraphCodeBERT (LB 0.8414) &gt; CodeBERT (LB 0.8398) &gt; UniXCoder (LB 0.795)<br>\nI haven't tried DOBF.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2013901,
      "author_name": "tanyx0901",
      "author_url": "",
      "post_date": "11/02/2022 07:44:28",
      "content": "<p>It takes too long to run, Unix code does not know why the experiment runs, but it does not run and no error is reported</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1853498": "__GraphCodeBERT__\nHuggingface Link: https://huggingface.co/microsoft/graphcodebert-base\n[GraphCodeBERT: Pre-training Code Representations with Data Flow](https://arxiv.org/pdf/2009.08366.pdf) @ ICLR 2021\n\nCodeBERT regards a code snippet as a sequence of tokens and ignores the inherent structure of code. GraphCodeBERT is a pre-trained model for programming language that considers the inherent structure of code with a dataflow graph. If you have used TensorFlow, then you are probably already familiar with dataflow representations of a program. I have found GraphCodeBERT to always outperform CodeBERT in my experiments. \n\n---\n__DOBF__\n[DOBF: A Deobfuscation Pre-Training Objective for Programming Languages](https://arxiv.org/pdf/2102.07492.pdf) @ NeurIPS 2021 by Meta AI\n\nThe authors first obfuscate the source code and then ask the model to recover the original version of obfuscated source code. This leads to a 24% relative improvement in natural language code search.\n\n> In this paper, we introduce a new pre-training objective, DOBF, that leverages the structural aspect of programming languages and pre-trains a model to recover the original code. We show that models pre-trained with DOBF significantly outperform existing approaches on multiple downstream tasks.\n\n---\n__UniXCoder__\n[UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://arxiv.org/pdf/2203.03850.pdf) @ ACL 2022\nHuggingface Link: https://huggingface.co/microsoft/unixcoder-base\n\nUniXCoder can be used for both code understanding and generation. The model claims to achieve SOTA in many tasks but I have found it to underperform even CodeBERT. \n\n---\nOther models: SynCoBERT, AstBERT, CLSEBERT",
    "1853570": "I also ran codebert, graphcodbert, and unixcoder, but unixcoder was the worst performance.\nI wonder if the encoder-decoder model, such as bart and t5, needs to change the setting separately.\nI lack knowledge of encoder-decoder models.😭😭 Do you happen to know anything?\n\n\n** The rest of the models don't seem to be visible on huggingface.",
    "1855869": "Thanks for your post.\n\nI think because GraphCodeBERT's considers the inherent structure of code with a dataflow graph, while CodeBERT regards a code snippet as a sequence of tokens and ignores the inherent structure of code it might be worth a shot if someone could get it right. Have you tried the other models mentioned in your post?",
    "1858067": "GraphCodeBERT is interesting approach. It is interesting, how to garantee correctness of generated code? Because language GPT models oftenly produce inconsistent text",
    "1863762": "Thank you for nice info.\nIn my same params experiment,  GraphCodeBERT (LB 0.8414) > CodeBERT (LB 0.8398) > UniXCoder (LB 0.795)\nI haven't tried DOBF.",
    "2013901": "It takes too long to run, Unix code does not know why the experiment runs, but it does not run and no error is reported"
  },
  "source": "meta"
}