{
  "id": 333408,
  "title": "Error Analysis: Visualize CodeBERT's Attention heads",
  "url": "/competitions/AI4Code/discussion/333408",
  "author_name": "",
  "post_date": "2022-06-26T11:50:25.642469500Z",
  "votes": 11,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi all,</p>\n<p>Something I haven't seen here yet is attention <strong>Error analysis</strong>.<br>\nWe all use CodeBERT, just like any other transformer its attention heads can be visualized and debugged. <br>\nThis is incredibly useful since using this approach we can poke into its inner workings and understand what makes it tick (sometimes in a bad way).</p>\n<p><strong>This is what it looks like on an example from the competition's data</strong><br>\n<img src=\"https://i.ibb.co/1vXkw5w/code-bert.png\" alt=\"\"></p>\n<p>I've made a notebook so you could play around with this if you want. </p>\n<p>It is right <a href=\"https://www.kaggle.com/code/thedevastator/error-analysis-visualize-codebert-s-attention?scriptVersionId=99355509\" target=\"_blank\">here</a></p>\n<p>Enjoy! </p>",
  "messages": [
    {
      "id": "1833838",
      "postDate": "06/26/2022 11:50:25",
      "content": "<p>Hi all,</p>\n<p>Something I haven't seen here yet is attention <strong>Error analysis</strong>.<br>\nWe all use CodeBERT, just like any other transformer its attention heads can be visualized and debugged. <br>\nThis is incredibly useful since using this approach we can poke into its inner workings and understand what makes it tick (sometimes in a bad way).</p>\n<p><strong>This is what it looks like on an example from the competition's data</strong><br>\n<img src=\"https://i.ibb.co/1vXkw5w/code-bert.png\" alt=\"\"></p>\n<p>I've made a notebook so you could play around with this if you want. </p>\n<p>It is right <a href=\"https://www.kaggle.com/code/thedevastator/error-analysis-visualize-codebert-s-attention?scriptVersionId=99355509\" target=\"_blank\">here</a></p>\n<p>Enjoy! </p>",
      "rawMarkdown": "Hi all,\n\nSomething I haven't seen here yet is attention **Error analysis**.\nWe all use CodeBERT, just like any other transformer its attention heads can be visualized and debugged. \nThis is incredibly useful since using this approach we can poke into its inner workings and understand what makes it tick (sometimes in a bad way).\n\n**This is what it looks like on an example from the competition's data**\n![](https://i.ibb.co/1vXkw5w/code-bert.png)\n\nI've made a notebook so you could play around with this if you want. \n\nIt is right [here](https://www.kaggle.com/code/thedevastator/error-analysis-visualize-codebert-s-attention?scriptVersionId=99355509)\n\n\nEnjoy!",
      "votes": null
    },
    {
      "id": "1834051",
      "postDate": "06/26/2022 15:22:01",
      "content": "<p>Got a newbie question: is it common or practical to use this finding which word matters or not in other NLP task?</p>",
      "rawMarkdown": "Got a newbie question: is it common or practical to use this finding which word matters or not in other NLP task?",
      "votes": null
    },
    {
      "id": "1837109",
      "postDate": "06/29/2022 10:53:17",
      "content": "<p>Generally - not. However, maybe it will be useful information for business tasks. </p>",
      "rawMarkdown": "Generally - not. However, maybe it will be useful information for business tasks.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1834051,
      "author_name": "snaker",
      "author_url": "",
      "post_date": "06/26/2022 15:22:01",
      "content": "<p>Got a newbie question: is it common or practical to use this finding which word matters or not in other NLP task?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1837109,
          "author_name": "vad13irt",
          "author_url": "",
          "post_date": "06/29/2022 10:53:17",
          "content": "<p>Generally - not. However, maybe it will be useful information for business tasks. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1833838": "Hi all,\n\nSomething I haven't seen here yet is attention **Error analysis**.\nWe all use CodeBERT, just like any other transformer its attention heads can be visualized and debugged. \nThis is incredibly useful since using this approach we can poke into its inner workings and understand what makes it tick (sometimes in a bad way).\n\n**This is what it looks like on an example from the competition's data**\n![](https://i.ibb.co/1vXkw5w/code-bert.png)\n\nI've made a notebook so you could play around with this if you want. \n\nIt is right [here](https://www.kaggle.com/code/thedevastator/error-analysis-visualize-codebert-s-attention?scriptVersionId=99355509)\n\n\nEnjoy!",
    "1834051": "Got a newbie question: is it common or practical to use this finding which word matters or not in other NLP task?",
    "1837109": "Generally - not. However, maybe it will be useful information for business tasks."
  },
  "source": "meta"
}