{
  "id": 330540,
  "title": "about codebert tokenizer question",
  "url": "/competitions/AI4Code/discussion/330540",
  "author_name": "",
  "post_date": "2022-06-13T03:34:56.948856800Z",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I have some question about Codebert tokenizer (aka roberta tokenizer) token id (1437)  ['Ġ']</p>\n<p>is need information in our model?  </p>\n<p>test as below:<br>\nencode_plus<br>\n<img alt=\"\" src=\"https://i.ibb.co/f81qGkC/tokenizer.png\"><br>\n<img src=\"https://imgur.com/a/k48Ad4M\" alt=\"\"></p>\n<p><img alt=\"\" src=\"https://i.ibb.co/CM9WsZv/text1.png\"><br>\nencode_plus -&gt;<br>\n[21714, 1215, 36807, 10975, 42666, 1215, 42800, 8174, 32001, 1640, <strong>1437, 1437, 1437, 1437</strong>, 29959, 10799, 49487, 1549, 6, 1549, 43, 4839, 2968, 90, 4, 12005, 43048]</p>\n<p>convert id to tokens -&gt; <br>\n['train', '<em>', 'df', '[', 'num', '</em>', 'feat', '].', 'hist', '(', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġfig', 'size', '=(', '16', ',', '16', ')', 'Ġ)', 'Ġpl', 't', '.', 'show', '()']</p>\n<p><img alt=\"\" src=\"https://i.ibb.co/fnfNGjV/text2.png\"></p>\n<p>encode_plus -&gt;<br>\n[0, 4684, 5423, 35, <strong>1437, 1437, 1437, 1437, 1437, 1437, 1437, 1437, 1437</strong>, 3816, 27148, 25153, 30529, 1640, 13367, 3256, <strong>1437, 1437, 1437, 1437, 1437, 1437, 1437, 1437,</strong> 3457, 2580, 594, 2]</p>\n<p>convert id to tokens -&gt; <br>\n['']</p>\n<p>I think some text need to do more preprocess to filter useless information</p>",
  "messages": [
    {
      "id": "1818667",
      "postDate": "06/13/2022 03:34:56",
      "content": "<p>I have some question about Codebert tokenizer (aka roberta tokenizer) token id (1437)  ['Ġ']</p>\n<p>is need information in our model?  </p>\n<p>test as below:<br>\nencode_plus<br>\n<img alt=\"\" src=\"https://i.ibb.co/f81qGkC/tokenizer.png\"><br>\n<img src=\"https://imgur.com/a/k48Ad4M\" alt=\"\"></p>\n<p><img alt=\"\" src=\"https://i.ibb.co/CM9WsZv/text1.png\"><br>\nencode_plus -&gt;<br>\n[21714, 1215, 36807, 10975, 42666, 1215, 42800, 8174, 32001, 1640, <strong>1437, 1437, 1437, 1437</strong>, 29959, 10799, 49487, 1549, 6, 1549, 43, 4839, 2968, 90, 4, 12005, 43048]</p>\n<p>convert id to tokens -&gt; <br>\n['train', '<em>', 'df', '[', 'num', '</em>', 'feat', '].', 'hist', '(', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġfig', 'size', '=(', '16', ',', '16', ')', 'Ġ)', 'Ġpl', 't', '.', 'show', '()']</p>\n<p><img alt=\"\" src=\"https://i.ibb.co/fnfNGjV/text2.png\"></p>\n<p>encode_plus -&gt;<br>\n[0, 4684, 5423, 35, <strong>1437, 1437, 1437, 1437, 1437, 1437, 1437, 1437, 1437</strong>, 3816, 27148, 25153, 30529, 1640, 13367, 3256, <strong>1437, 1437, 1437, 1437, 1437, 1437, 1437, 1437,</strong> 3457, 2580, 594, 2]</p>\n<p>convert id to tokens -&gt; <br>\n['']</p>\n<p>I think some text need to do more preprocess to filter useless information</p>",
      "rawMarkdown": "I have some question about Codebert tokenizer (aka roberta tokenizer) token id (1437)  ['Ġ']\n\nis need information in our model?  \n\n\ntest as below:\nencode_plus\n<img alt=\"\" src=\"https://i.ibb.co/f81qGkC/tokenizer.png\">\n![](https://imgur.com/a/k48Ad4M)\n\n\n\n\n\n<img alt=\"\" src=\"https://i.ibb.co/CM9WsZv/text1.png\">\nencode_plus ->\n[21714, 1215, 36807, 10975, 42666, 1215, 42800, 8174, 32001, 1640, **1437, 1437, 1437, 1437**, 29959, 10799, 49487, 1549, 6, 1549, 43, 4839, 2968, 90, 4, 12005, 43048]\n\nconvert id to tokens -> \n['train', '_', 'df', '[', 'num', '_', 'feat', '].', 'hist', '(', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġfig', 'size', '=(', '16', ',', '16', ')', 'Ġ)', 'Ġpl', 't', '.', 'show', '()']\n\n\n<img alt=\"\" src=\"https://i.ibb.co/fnfNGjV/text2.png\">\n\nencode_plus ->\n[0, 4684, 5423, 35, **1437, 1437, 1437, 1437, 1437, 1437, 1437, 1437, 1437**, 3816, 27148, 25153, 30529, 1640, 13367, 3256, **1437, 1437, 1437, 1437, 1437, 1437, 1437, 1437,** 3457, 2580, 594, 2]\n\nconvert id to tokens -> \n['<s>', 'class', 'ĠData', ':', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġdef', 'Ġ__', 'init', '__', '(', 'self', '):', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ' ,'Ġtests', 'ets', 'et', '</s>']\n\nI think some text need to do more preprocess to filter useless information",
      "votes": null
    },
    {
      "id": "1819047",
      "postDate": "06/13/2022 11:48:00",
      "content": "<p>There are many meaningless tokens in vocabulary of codebert tokenizer.👀</p>",
      "rawMarkdown": "There are many meaningless tokens in vocabulary of codebert tokenizer.👀",
      "votes": null
    },
    {
      "id": "1820882",
      "postDate": "06/15/2022 04:40:28",
      "content": "<p>In addition to the tokens shown in the example, I know that there are several strange-looking tokens. Personally, I think it would be okay to ignore it enough and use codbert.</p>",
      "rawMarkdown": "In addition to the tokens shown in the example, I know that there are several strange-looking tokens. Personally, I think it would be okay to ignore it enough and use codbert.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1819047,
      "author_name": "yuanzhezhou",
      "author_url": "",
      "post_date": "06/13/2022 11:48:00",
      "content": "<p>There are many meaningless tokens in vocabulary of codebert tokenizer.👀</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1820882,
      "author_name": "heuingttiang",
      "author_url": "",
      "post_date": "06/15/2022 04:40:28",
      "content": "<p>In addition to the tokens shown in the example, I know that there are several strange-looking tokens. Personally, I think it would be okay to ignore it enough and use codbert.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1818667": "I have some question about Codebert tokenizer (aka roberta tokenizer) token id (1437)  ['Ġ']\n\nis need information in our model?  \n\n\ntest as below:\nencode_plus\n<img alt=\"\" src=\"https://i.ibb.co/f81qGkC/tokenizer.png\">\n![](https://imgur.com/a/k48Ad4M)\n\n\n\n\n\n<img alt=\"\" src=\"https://i.ibb.co/CM9WsZv/text1.png\">\nencode_plus ->\n[21714, 1215, 36807, 10975, 42666, 1215, 42800, 8174, 32001, 1640, **1437, 1437, 1437, 1437**, 29959, 10799, 49487, 1549, 6, 1549, 43, 4839, 2968, 90, 4, 12005, 43048]\n\nconvert id to tokens -> \n['train', '_', 'df', '[', 'num', '_', 'feat', '].', 'hist', '(', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġfig', 'size', '=(', '16', ',', '16', ')', 'Ġ)', 'Ġpl', 't', '.', 'show', '()']\n\n\n<img alt=\"\" src=\"https://i.ibb.co/fnfNGjV/text2.png\">\n\nencode_plus ->\n[0, 4684, 5423, 35, **1437, 1437, 1437, 1437, 1437, 1437, 1437, 1437, 1437**, 3816, 27148, 25153, 30529, 1640, 13367, 3256, **1437, 1437, 1437, 1437, 1437, 1437, 1437, 1437,** 3457, 2580, 594, 2]\n\nconvert id to tokens -> \n['<s>', 'class', 'ĠData', ':', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġdef', 'Ġ__', 'init', '__', '(', 'self', '):', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ', 'Ġ' ,'Ġtests', 'ets', 'et', '</s>']\n\nI think some text need to do more preprocess to filter useless information",
    "1819047": "There are many meaningless tokens in vocabulary of codebert tokenizer.👀",
    "1820882": "In addition to the tokens shown in the example, I know that there are several strange-looking tokens. Personally, I think it would be okay to ignore it enough and use codbert."
  },
  "source": "meta"
}