{ "id": 336983, "title": "How to use CuBert", "url": "/competitions/AI4Code/discussion/336983", "author_name": "", "post_date": "2022-07-13T22:55:55.423129700Z", "votes": 20, "comment_count": 5, "views": 0, "content": "

I saw some posts about wanting to use CuBert so I added this dataset with the weights and tokenizer: https://www.kaggle.com/datasets/nbroad/hf-cubert

\n

There are three versions (512, 1024, 2048 sequence lengths) trained on python code.

\n

This notebook shows how to use it through transformers in PyTorch: https://www.kaggle.com/nbroad/using-cubert/

\n

There might be some bugs, so please share if you notice anything.

\n

Also, there isn't a fast tokenizer, so it is just a python implementation (not rust).

\n

If you are wondering what CuBert is, see the Google Research repo here: https://github.com/google-research/google-research/tree/master/cubert

\n

Credit to this post for showing how to load it in transformers: https://github.com/google-research/google-research/issues/582#issuecomment-794260254

", "messages": [ { "id": "1854685", "postDate": "07/13/2022 22:55:55", "content": "

I saw some posts about wanting to use CuBert so I added this dataset with the weights and tokenizer: https://www.kaggle.com/datasets/nbroad/hf-cubert

\n

There are three versions (512, 1024, 2048 sequence lengths) trained on python code.

\n

This notebook shows how to use it through transformers in PyTorch: https://www.kaggle.com/nbroad/using-cubert/

\n

There might be some bugs, so please share if you notice anything.

\n

Also, there isn't a fast tokenizer, so it is just a python implementation (not rust).

\n

If you are wondering what CuBert is, see the Google Research repo here: https://github.com/google-research/google-research/tree/master/cubert

\n

Credit to this post for showing how to load it in transformers: https://github.com/google-research/google-research/issues/582#issuecomment-794260254

", "rawMarkdown": "I saw some posts about wanting to use CuBert so I added this dataset with the weights and tokenizer: https://www.kaggle.com/datasets/nbroad/hf-cubert\n\nThere are three versions (512, 1024, 2048 sequence lengths) trained on python code.\n\nThis notebook shows how to use it through `transformers` in PyTorch: https://www.kaggle.com/nbroad/using-cubert/\n\nThere might be some bugs, so please share if you notice anything.\n\nAlso, there isn't a fast tokenizer, so it is just a python implementation (not rust).\n\nIf you are wondering what CuBert is, see the Google Research repo here: https://github.com/google-research/google-research/tree/master/cubert\n\nCredit to this post for showing how to load it in `transformers`: https://github.com/google-research/google-research/issues/582#issuecomment-794260254", "votes": null }, { "id": "1854817", "postDate": "07/14/2022 03:59:38", "content": "

Thank you very much.

", "rawMarkdown": "Thank you very much.", "votes": null }, { "id": "1855361", "postDate": "07/14/2022 14:15:25", "content": "

I'll experiment with it. Thank you very much and I will vote for the dataset.

", "rawMarkdown": "I'll experiment with it. Thank you very much and I will vote for the dataset.", "votes": null }, { "id": "1857963", "postDate": "07/16/2022 14:44:11", "content": "

From what I can tell the tokenization is far too slow to be usable as is. A good opportunity for someone to write a faster tokenizer and take advantage for the competition ;)

", "rawMarkdown": "From what I can tell the tokenization is far too slow to be usable as is. A good opportunity for someone to write a faster tokenizer and take advantage for the competition ;)", "votes": null }, { "id": "1858434", "postDate": "07/16/2022 23:05:48", "content": "

I did some profiling to see what is making it so slow. My initial results indicate that this function is the first bottleneck. I don't really understand why there is so much code because it seems like it could be replaced with a one-liner.

\n
def split_long_token(token_string,\n                     max_output_token_length):\n  \"\"\"Splits a token losslessly to some maximum length per component.\n\n  A long token is split into multiple tokens. For instance, `'bcd'` with\n  `max_output_token_length=2` will become `['bc', 'd']`. No sentinel or other\n  split mark is added at this stage.\n\n  A token is assumed to be non-empty.\n\n  Args:\n    token_string: The token.\n    max_output_token_length: Maximum length of an output token.\n\n  Returns:\n    List of split tokens.\n\n  Raises:\n    ValueError: if `token` is empty.\n  \"\"\"\n  if not token_string:\n    raise ValueError('Expected %r to be non-empty' % token_string)\n\n  whole_token_length = len(token_string)\n  remainder_length = whole_token_length % max_output_token_length\n  even_parts = list(\n      map(\n          # ...join together...\n          ''.join,\n          zip(\n              # `max_output_token_length` copies of the iterator of\n              # whole_token's characters. zip will draw from the same iterator\n              # and return `max_output_token_length` tuples of characters from\n              # `whole_token`.\n              *[iter(token_string)] * max_output_token_length)))\n  remainder_part = ([token_string[-remainder_length:]]\n                    if remainder_length else [])\n  split_token = even_parts + remainder_part\n  assert split_token, ('while wrapping >>%s<< into >%r<' %\n                       (token_string, split_token))\n  assert all([\n      len(t) <= max_output_token_length for t in split_token\n  ]), ('Got split_token >>>%r<<<, which contains tokens longer than %d.' %\n       (split_token, max_output_token_length))\n  return split_token\n
\n

One liner

\n
return [token_string[i:i+max_output_token_length] for i in range(0, len(token_string), max_output_token_length)]\n
\n

This can speed it up a lot, and there are likely other bottlenecks that I haven't found just yet.

", "rawMarkdown": "I did some profiling to see what is making it so slow. My initial results indicate that this function is the first bottleneck. I don't really understand why there is so much code because it seems like it could be replaced with a one-liner.\n\n```python\ndef split_long_token(token_string,\n max_output_token_length):\n \"\"\"Splits a token losslessly to some maximum length per component.\n\n A long token is split into multiple tokens. For instance, `'bcd'` with\n `max_output_token_length=2` will become `['bc', 'd']`. No sentinel or other\n split mark is added at this stage.\n\n A token is assumed to be non-empty.\n\n Args:\n token_string: The token.\n max_output_token_length: Maximum length of an output token.\n\n Returns:\n List of split tokens.\n\n Raises:\n ValueError: if `token` is empty.\n \"\"\"\n if not token_string:\n raise ValueError('Expected %r to be non-empty' % token_string)\n\n whole_token_length = len(token_string)\n remainder_length = whole_token_length % max_output_token_length\n even_parts = list(\n map(\n # ...join together...\n ''.join,\n zip(\n # `max_output_token_length` copies of the iterator of\n # whole_token's characters. zip will draw from the same iterator\n # and return `max_output_token_length` tuples of characters from\n # `whole_token`.\n *[iter(token_string)] * max_output_token_length)))\n remainder_part = ([token_string[-remainder_length:]]\n if remainder_length else [])\n split_token = even_parts + remainder_part\n assert split_token, ('while wrapping >>%s<< into >%r<' %\n (token_string, split_token))\n assert all([\n len(t) <= max_output_token_length for t in split_token\n ]), ('Got split_token >>>%r<<<, which contains tokens longer than %d.' %\n (split_token, max_output_token_length))\n return split_token\n\n```\n\nOne liner\n\n```python\nreturn [token_string[i:i+max_output_token_length] for i in range(0, len(token_string), max_output_token_length)]\n```\n\nThis can speed it up a lot, and there are likely other bottlenecks that I haven't found just yet.", "votes": null }, { "id": "1859717", "postDate": "07/17/2022 22:05:28", "content": "

Nice info!! Will check this out.. thanks for sharing.

", "rawMarkdown": "Nice info!! Will check this out.. thanks for sharing.", "votes": null } ], "comments": [ { "id": 1854817, "author_name": "locbaop", "author_url": "", "post_date": "07/14/2022 03:59:38", "content": "

Thank you very much.

", "votes": null, "replies": [] }, { "id": 1855361, "author_name": "heuingttiang", "author_url": "", "post_date": "07/14/2022 14:15:25", "content": "

I'll experiment with it. Thank you very much and I will vote for the dataset.

", "votes": null, "replies": [] }, { "id": 1857963, "author_name": "happycube", "author_url": "", "post_date": "07/16/2022 14:44:11", "content": "

From what I can tell the tokenization is far too slow to be usable as is. A good opportunity for someone to write a faster tokenizer and take advantage for the competition ;)

", "votes": null, "replies": [ { "id": 1858434, "author_name": "nbroad", "author_url": "", "post_date": "07/16/2022 23:05:48", "content": "

I did some profiling to see what is making it so slow. My initial results indicate that this function is the first bottleneck. I don't really understand why there is so much code because it seems like it could be replaced with a one-liner.

\n
def split_long_token(token_string,\n                     max_output_token_length):\n  \"\"\"Splits a token losslessly to some maximum length per component.\n\n  A long token is split into multiple tokens. For instance, `'bcd'` with\n  `max_output_token_length=2` will become `['bc', 'd']`. No sentinel or other\n  split mark is added at this stage.\n\n  A token is assumed to be non-empty.\n\n  Args:\n    token_string: The token.\n    max_output_token_length: Maximum length of an output token.\n\n  Returns:\n    List of split tokens.\n\n  Raises:\n    ValueError: if `token` is empty.\n  \"\"\"\n  if not token_string:\n    raise ValueError('Expected %r to be non-empty' % token_string)\n\n  whole_token_length = len(token_string)\n  remainder_length = whole_token_length % max_output_token_length\n  even_parts = list(\n      map(\n          # ...join together...\n          ''.join,\n          zip(\n              # `max_output_token_length` copies of the iterator of\n              # whole_token's characters. zip will draw from the same iterator\n              # and return `max_output_token_length` tuples of characters from\n              # `whole_token`.\n              *[iter(token_string)] * max_output_token_length)))\n  remainder_part = ([token_string[-remainder_length:]]\n                    if remainder_length else [])\n  split_token = even_parts + remainder_part\n  assert split_token, ('while wrapping >>%s<< into >%r<' %\n                       (token_string, split_token))\n  assert all([\n      len(t) <= max_output_token_length for t in split_token\n  ]), ('Got split_token >>>%r<<<, which contains tokens longer than %d.' %\n       (split_token, max_output_token_length))\n  return split_token\n
\n

One liner

\n
return [token_string[i:i+max_output_token_length] for i in range(0, len(token_string), max_output_token_length)]\n
\n

This can speed it up a lot, and there are likely other bottlenecks that I haven't found just yet.

", "votes": null, "replies": [] } ] }, { "id": 1859717, "author_name": "cid007", "author_url": "", "post_date": "07/17/2022 22:05:28", "content": "

Nice info!! Will check this out.. thanks for sharing.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1854685": "I saw some posts about wanting to use CuBert so I added this dataset with the weights and tokenizer: https://www.kaggle.com/datasets/nbroad/hf-cubert\n\nThere are three versions (512, 1024, 2048 sequence lengths) trained on python code.\n\nThis notebook shows how to use it through `transformers` in PyTorch: https://www.kaggle.com/nbroad/using-cubert/\n\nThere might be some bugs, so please share if you notice anything.\n\nAlso, there isn't a fast tokenizer, so it is just a python implementation (not rust).\n\nIf you are wondering what CuBert is, see the Google Research repo here: https://github.com/google-research/google-research/tree/master/cubert\n\nCredit to this post for showing how to load it in `transformers`: https://github.com/google-research/google-research/issues/582#issuecomment-794260254", "1854817": "Thank you very much.", "1855361": "I'll experiment with it. Thank you very much and I will vote for the dataset.", "1857963": "From what I can tell the tokenization is far too slow to be usable as is. A good opportunity for someone to write a faster tokenizer and take advantage for the competition ;)", "1858434": "I did some profiling to see what is making it so slow. My initial results indicate that this function is the first bottleneck. I don't really understand why there is so much code because it seems like it could be replaced with a one-liner.\n\n```python\ndef split_long_token(token_string,\n max_output_token_length):\n \"\"\"Splits a token losslessly to some maximum length per component.\n\n A long token is split into multiple tokens. For instance, `'bcd'` with\n `max_output_token_length=2` will become `['bc', 'd']`. No sentinel or other\n split mark is added at this stage.\n\n A token is assumed to be non-empty.\n\n Args:\n token_string: The token.\n max_output_token_length: Maximum length of an output token.\n\n Returns:\n List of split tokens.\n\n Raises:\n ValueError: if `token` is empty.\n \"\"\"\n if not token_string:\n raise ValueError('Expected %r to be non-empty' % token_string)\n\n whole_token_length = len(token_string)\n remainder_length = whole_token_length % max_output_token_length\n even_parts = list(\n map(\n # ...join together...\n ''.join,\n zip(\n # `max_output_token_length` copies of the iterator of\n # whole_token's characters. zip will draw from the same iterator\n # and return `max_output_token_length` tuples of characters from\n # `whole_token`.\n *[iter(token_string)] * max_output_token_length)))\n remainder_part = ([token_string[-remainder_length:]]\n if remainder_length else [])\n split_token = even_parts + remainder_part\n assert split_token, ('while wrapping >>%s<< into >%r<' %\n (token_string, split_token))\n assert all([\n len(t) <= max_output_token_length for t in split_token\n ]), ('Got split_token >>>%r<<<, which contains tokens longer than %d.' %\n (split_token, max_output_token_length))\n return split_token\n\n```\n\nOne liner\n\n```python\nreturn [token_string[i:i+max_output_token_length] for i in range(0, len(token_string), max_output_token_length)]\n```\n\nThis can speed it up a lot, and there are likely other bottlenecks that I haven't found just yet.", "1859717": "Nice info!! Will check this out.. thanks for sharing." }, "source": "meta" }