Learning and Evaluating Contextual Embedding of Source Code by Google Research
\nGithub Repo Link (Unfortunately CuBERT weights and tokenizer are not available at huggingface/models)
\nCuBERT is a concurrent work to CodeBERT. However, it has 4 main advantages over CodeBERT, thanks to the compute power available to Google researchers:
\nCuBERT has a model checkpoint for a sequence length of 2048, unlike CodeBERT which can only fit 512 tokens in the input context. This is important in the competition because most Jupyter notebooks do not fit in 512 tokens.
CuBERT is the only code understanding model I could find that has a BERT-large configuration. All of the other models like CodeBERT, GraphCodeBERT, and UniXCoder only have BERT-base configurations. I don't know why this is the case?
CodeBERT initializes its training from RoBERTa-base model. This means it uses the general-purpose vocabulary which might not be suited for programming tasks. In contrast, CuBERT learns program vocabulary from scratch (see section 3.3 in the paper). For a detailed comparison of domain-specific pretraining (like CuBERT) vs continual pretraining (like CodeBERT), see the paper [1].
CuBERT is a Python-only model unlike CodeBERT which is trained on 6 different programming languages. CodeBERT is trained on 1.1 million python functions, while CuBERT is trained on 7.4 million Python files
[1] Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
", "messages": [ { "id": "1853419", "postDate": "07/12/2022 21:41:37", "content": "Learning and Evaluating Contextual Embedding of Source Code by Google Research
\nGithub Repo Link (Unfortunately CuBERT weights and tokenizer are not available at huggingface/models)
\nCuBERT is a concurrent work to CodeBERT. However, it has 4 main advantages over CodeBERT, thanks to the compute power available to Google researchers:
\nCuBERT has a model checkpoint for a sequence length of 2048, unlike CodeBERT which can only fit 512 tokens in the input context. This is important in the competition because most Jupyter notebooks do not fit in 512 tokens.
CuBERT is the only code understanding model I could find that has a BERT-large configuration. All of the other models like CodeBERT, GraphCodeBERT, and UniXCoder only have BERT-base configurations. I don't know why this is the case?
CodeBERT initializes its training from RoBERTa-base model. This means it uses the general-purpose vocabulary which might not be suited for programming tasks. In contrast, CuBERT learns program vocabulary from scratch (see section 3.3 in the paper). For a detailed comparison of domain-specific pretraining (like CuBERT) vs continual pretraining (like CodeBERT), see the paper [1].
CuBERT is a Python-only model unlike CodeBERT which is trained on 6 different programming languages. CodeBERT is trained on 1.1 million python functions, while CuBERT is trained on 7.4 million Python files
[1] Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
", "rawMarkdown": "[Learning and Evaluating Contextual Embedding of Source Code](https://arxiv.org/pdf/2001.00059.pdf) by Google Research\n\n[Github Repo Link](https://github.com/google-research/google-research/tree/master/cubert) (Unfortunately CuBERT weights and tokenizer are not available at huggingface/models)\n\n---\nCuBERT is a concurrent work to CodeBERT. However, it has 4 main advantages over CodeBERT, thanks to the compute power available to Google researchers: \n\n1. CuBERT has a model checkpoint for a sequence length of 2048, unlike CodeBERT which can only fit 512 tokens in the input context. This is important in the competition because most Jupyter notebooks do not fit in 512 tokens. \n\n2. CuBERT is the only code understanding model I could find that has a BERT-large configuration. All of the other models like CodeBERT, GraphCodeBERT, and UniXCoder only have BERT-base configurations. I don't know why this is the case?\n\n3. CodeBERT initializes its training from RoBERTa-base model. This means it uses the general-purpose vocabulary which might not be suited for programming tasks. In contrast, CuBERT learns program vocabulary from scratch (see section 3.3 in the paper). For a detailed comparison of domain-specific pretraining (like CuBERT) vs continual pretraining (like CodeBERT), see the paper [1].\n\n4. CuBERT is a Python-only model unlike CodeBERT which is trained on 6 different programming languages. CodeBERT is trained on 1.1 million python _functions_, while CuBERT is trained on 7.4 million Python _files_ \n\n---\n[1] [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://arxiv.org/pdf/2007.15779.pdf)", "votes": null }, { "id": "1855863", "postDate": "07/15/2022 01:45:41", "content": "Thanks for the helpful post. Question, since you said that Google Research's CuBERT is a concurrent work to CodeBERT, would you know if there are any differences between these two when it comes to the applications that they can be used for or are they pretty much the same? Also, is there a reason why you think that CuBERT is better than CodeBERT? Thanks in advance!
", "rawMarkdown": "Thanks for the helpful post. Question, since you said that Google Research's CuBERT is a concurrent work to CodeBERT, would you know if there are any differences between these two when it comes to the applications that they can be used for or are they pretty much the same? Also, is there a reason why you think that CuBERT is better than CodeBERT? Thanks in advance!", "votes": null }, { "id": "1861653", "postDate": "07/19/2022 06:34:52", "content": "We tried using this model for fine-tuning but cubert-512 will take more than 30 hours per epoch with A100…
", "rawMarkdown": "We tried using this model for fine-tuning but cubert-512 will take more than 30 hours per epoch with A100...", "votes": null } ], "comments": [ { "id": 1855863, "author_name": "thedevastator", "author_url": "", "post_date": "07/15/2022 01:45:41", "content": "Thanks for the helpful post. Question, since you said that Google Research's CuBERT is a concurrent work to CodeBERT, would you know if there are any differences between these two when it comes to the applications that they can be used for or are they pretty much the same? Also, is there a reason why you think that CuBERT is better than CodeBERT? Thanks in advance!
", "votes": null, "replies": [] }, { "id": 1861653, "author_name": "conan1024", "author_url": "", "post_date": "07/19/2022 06:34:52", "content": "We tried using this model for fine-tuning but cubert-512 will take more than 30 hours per epoch with A100…
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1853419": "[Learning and Evaluating Contextual Embedding of Source Code](https://arxiv.org/pdf/2001.00059.pdf) by Google Research\n\n[Github Repo Link](https://github.com/google-research/google-research/tree/master/cubert) (Unfortunately CuBERT weights and tokenizer are not available at huggingface/models)\n\n---\nCuBERT is a concurrent work to CodeBERT. However, it has 4 main advantages over CodeBERT, thanks to the compute power available to Google researchers: \n\n1. CuBERT has a model checkpoint for a sequence length of 2048, unlike CodeBERT which can only fit 512 tokens in the input context. This is important in the competition because most Jupyter notebooks do not fit in 512 tokens. \n\n2. CuBERT is the only code understanding model I could find that has a BERT-large configuration. All of the other models like CodeBERT, GraphCodeBERT, and UniXCoder only have BERT-base configurations. I don't know why this is the case?\n\n3. CodeBERT initializes its training from RoBERTa-base model. This means it uses the general-purpose vocabulary which might not be suited for programming tasks. In contrast, CuBERT learns program vocabulary from scratch (see section 3.3 in the paper). For a detailed comparison of domain-specific pretraining (like CuBERT) vs continual pretraining (like CodeBERT), see the paper [1].\n\n4. CuBERT is a Python-only model unlike CodeBERT which is trained on 6 different programming languages. CodeBERT is trained on 1.1 million python _functions_, while CuBERT is trained on 7.4 million Python _files_ \n\n---\n[1] [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://arxiv.org/pdf/2007.15779.pdf)", "1855863": "Thanks for the helpful post. Question, since you said that Google Research's CuBERT is a concurrent work to CodeBERT, would you know if there are any differences between these two when it comes to the applications that they can be used for or are they pretty much the same? Also, is there a reason why you think that CuBERT is better than CodeBERT? Thanks in advance!", "1861653": "We tried using this model for fine-tuning but cubert-512 will take more than 30 hours per epoch with A100..." }, "source": "meta" }