{ "id": 326473, "title": "Pairwise model baseline", "url": "/competitions/AI4Code/discussion/326473", "author_name": "yuanzhe zhou", "post_date": "2022-05-22T16:46:00.657000", "votes": 84, "comment_count": 23, "views": 0, "content": "

I posted my pairwise model baseline (~ 0.817) and inference notebooks here, the main idea is using bert model to determine if a markdown and a code is related,

\n

[CLS] + markdown + [SEP] + code + [SEP]

\n\n
\n

pointwise model notebooks:

\n\n

EDIT1
\nPS: Pairwise model is more suitable for accurate rank task. Here I used only a bert-small model. From my local experiments, larger models perform much better.

\n

EDIT2
\nA pairwise model can achieve 0.832 score with 1 epoch training.

", "messages": [ { "id": 1798042, "postDate": "2022-05-22T16:46:00.657Z", "content": "

I posted my pairwise model baseline (~ 0.817) and inference notebooks here, the main idea is using bert model to determine if a markdown and a code is related,

\n

[CLS] + markdown + [SEP] + code + [SEP]

\n\n
\n

pointwise model notebooks:

\n\n

EDIT1
\nPS: Pairwise model is more suitable for accurate rank task. Here I used only a bert-small model. From my local experiments, larger models perform much better.

\n

EDIT2
\nA pairwise model can achieve 0.832 score with 1 epoch training.

", "rawMarkdown": "I posted my pairwise model baseline (~ 0.817) and inference notebooks here, the main idea is using bert model to determine if a markdown and a code is related, \n\n[CLS] + markdown + [SEP] + code + [SEP]\n\n\n* [Pretrain](https://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-pretrain/notebook)\n* [Training](https://www.kaggle.com/yuanzhezhou/ai4code-pairwise-bertsmall-training)\n* [Inference](https://www.kaggle.com/yuanzhezhou/ai4code-pairwise-bertsmall-inference)\n\n----------------------------------------------------------------------------------------------\npointwise model notebooks:\n* [inference](https://www.kaggle.com/code/yuanzhezhou/ai4code-distilbert-inference-777)\n* [training](https://www.kaggle.com/code/yuanzhezhou/ai4code-distilbert-training-777)\n\nEDIT1\nPS: Pairwise model is more suitable for accurate rank task. Here I used only a bert-small model. From my local experiments, larger models perform much better.\n\nEDIT2\nA pairwise model can achieve 0.832 score with 1 epoch training.", "votes": 84 }, { "id": 1800858, "postDate": "2022-05-25T09:12:10.337Z", "content": "

Thanks! I would like to know why you choose distill bert rather than bert_base or other PLM

", "rawMarkdown": "Thanks! I would like to know why you choose distill bert rather than bert_base or other PLM", "votes": 1, "replies": [ { "id": 1800966, "postDate": "2022-05-25T10:29:26.077Z", "content": "

Hello, I used part of the code from this notebook, in which distilbert was used.

\n

At the same time, to use larger models, we need to add more information to our inputs, like global context information, etc. Distilbert is enough as the baseline. To improve it, you can try larger models.

", "rawMarkdown": "Hello, I used part of the code from[ this notebook](https://www.kaggle.com/code/aerdem4/ai4code-pytorch-distilbert-baseline), in which distilbert was used. \n\nAt the same time, to use larger models, we need to add more information to our inputs, like global context information, etc. Distilbert is enough as the baseline. To improve it, you can try larger models." }, { "id": 1800991, "postDate": "2022-05-25T10:45:58.310Z", "content": "

Thanks for your reply, it's helpful~

", "rawMarkdown": "Thanks for your reply, it's helpful~" } ] }, { "id": 1798613, "postDate": "2022-05-23T07:46:44.583Z", "content": "

Thanks :-) I think I was trying to do something like this, but apparently I was missing multiple pieces of the puzzle.

", "rawMarkdown": "Thanks :-) I think I was trying to do something like this, but apparently I was missing multiple pieces of the puzzle. ", "votes": 1, "replies": [ { "id": 1798642, "postDate": "2022-05-23T08:53:34.217Z", "content": "

I hope it inspires you :).

", "rawMarkdown": "I hope it inspires you :).", "votes": 1 } ] }, { "id": 1798585, "postDate": "2022-05-23T07:07:23.083Z", "content": "

Good solution! One question: How do you select the best checkpoint in train? How many epochs in train ?

", "rawMarkdown": "Good solution! One question: How do you select the best checkpoint in train? How many epochs in train ?", "votes": 1, "replies": [ { "id": 1798638, "postDate": "2022-05-23T08:50:22.563Z", "content": "

I did not save checkpoint by steps, because for this method, validation takes long time. So I only save for last step of each epochs.
\n3~4 epochs.

", "rawMarkdown": "I did not save checkpoint by steps, because for this method, validation takes long time. So I only save for last step of each epochs.\n3~4 epochs.", "votes": 3 }, { "id": 1798806, "postDate": "2022-05-23T11:40:23.863Z", "content": "

Thanks. I will try it.

", "rawMarkdown": "Thanks. I will try it. " } ] }, { "id": 1798453, "postDate": "2022-05-23T03:47:34.517Z", "content": "

I think I am most happy that you cleaned up the old code more though. 👍

", "rawMarkdown": "I think I am most happy that you cleaned up the old code more though. 👍", "votes": 1, "replies": [ { "id": 1798643, "postDate": "2022-05-23T08:54:08.040Z", "content": "

Thanks you for your reply. I'm glad you find it helpful. :)

", "rawMarkdown": "Thanks you for your reply. I'm glad you find it helpful. :)", "votes": 1 } ] }, { "id": 1798445, "postDate": "2022-05-23T03:39:41.797Z", "content": "

Your generosity is amazing. I hope you win this competition.

", "rawMarkdown": "Your generosity is amazing. I hope you win this competition.", "votes": 1 }, { "id": 1798442, "postDate": "2022-05-23T03:37:50.537Z", "content": "

Thanks for this! Great approach to solve this problem.

", "rawMarkdown": "Thanks for this! Great approach to solve this problem.", "votes": 1 }, { "id": 1830197, "postDate": "2022-06-23T08:59:08.953Z", "content": "

This is my first attempt at NLP competitions. I have some questions

\n
    \n
  1. What do the terms [CLS], [SEP] mean and what is their signifance.
    2. \n
  2. Looking at the data, for every markdown cell, there are many cells for the code,
    \nthen how do you choose the cell to perform this operation [CLS] + markdown + [SEP] + code + [SEP]. An illustration could be useful.
    3. \n
", "rawMarkdown": "This is my first attempt at NLP competitions. I have some questions\n1. What do the terms [CLS], [SEP] mean and what is their signifance.\n2. Looking at the data, for every markdown cell, there are many cells for the code,\nthen how do you choose the cell to perform this operation [CLS] + markdown + [SEP] + code + [SEP]. An illustration could be useful.", "replies": [ { "id": 1836025, "postDate": "2022-06-28T09:39:15.323Z", "content": "

[CLS] and [SEP] are artificial tokens that are respectively inserted before the first sequence of tokens and between the first and second sequences.
\nA Pairwise Comparison is the process of comparing candidates in pairs to judge which of each candidate is preferred overall. Therefore, [CLS] + markdown + [SEP] + code + [SEP] is using to compare relevance between markdown and code. The [SEP] between markdown and code is here to help Bert understand which token belong to markdown and which token belong to code.

", "rawMarkdown": "`[CLS]` and `[SEP]` are artificial tokens that are respectively inserted before the first sequence of tokens and between the first and second sequences. \nA Pairwise Comparison is the process of comparing candidates in pairs to judge which of each candidate is preferred overall. Therefore, `[CLS] + markdown + [SEP] + code + [SEP]` is using to compare relevance between markdown and code. The `[SEP]` between markdown and code is here to help Bert understand which token belong to markdown and which token belong to code." } ] }, { "id": 1817586, "postDate": "2022-06-11T15:12:21.330Z", "content": "

Thank you very much！This is really helpful!

", "rawMarkdown": "Thank you very much！This is really helpful!" }, { "id": 1812373, "postDate": "2022-06-05T18:10:11.107Z", "content": "

Is the pairwise model a better choice for more optimization in the future?

", "rawMarkdown": "Is the pairwise model a better choice for more optimization in the future?" }, { "id": 1808061, "postDate": "2022-06-01T14:08:24.653Z", "content": "

Thanks for sharing @yuanzhezhou. Regarding your comment below, could you clarify what was used to get 0.817 model -Is it a part of training data for 1epoch ?

\n
A pairwise model can achieve 0.832 score with 1 epoch training.\n
", "rawMarkdown": "Thanks for sharing @yuanzhezhou. Regarding your comment below, could you clarify what was used to get 0.817 model -Is it a part of training data for 1epoch ?\n\n```\nA pairwise model can achieve 0.832 score with 1 epoch training.\n\n```\n", "replies": [ { "id": 1808164, "postDate": "2022-06-01T15:22:55.387Z", "content": "

You have all you need to get 0.817 in my training notebook.

\n

There are hints to improve the performance of pairwise models in another topic post by me.

", "rawMarkdown": "You have all you need to get 0.817 in my training notebook.\n\nThere are hints to improve the performance of pairwise models in another topic post by me.", "votes": 1 } ] }, { "id": 1804524, "postDate": "2022-05-29T05:40:58.090Z", "content": "

Thanks for your share.
\nCould you show me the device config to train this model and if you use all data to train?

", "rawMarkdown": "Thanks for your share. \nCould you show me the device config to train this model and if you use all data to train?", "replies": [ { "id": 1808175, "postDate": "2022-06-01T15:28:11.373Z", "content": "

50G RAM and 16G GPU memory

", "rawMarkdown": "50G RAM and 16G GPU memory", "votes": 1 } ] }, { "id": 1801089, "postDate": "2022-05-25T12:25:45.430Z", "content": "

Good inspiration！

", "rawMarkdown": "Good inspiration！" }, { "id": 1800442, "postDate": "2022-05-25T01:25:25.510Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1799290, "postDate": "2022-05-23T21:10:05.667Z", "content": "

Thank you for sharing code!

", "rawMarkdown": "Thank you for sharing code!", "votes": 1 } ], "comments": [ { "id": 1800858, "author_name": "Lin Xi", "author_url": "", "post_date": "2022-05-25T09:12:10.337000", "content": "

Thanks! I would like to know why you choose distill bert rather than bert_base or other PLM

", "votes": 1, "replies": [ { "id": 1800966, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2022-05-25T10:29:26.077000", "content": "

Hello, I used part of the code from this notebook, in which distilbert was used.

\n

At the same time, to use larger models, we need to add more information to our inputs, like global context information, etc. Distilbert is enough as the baseline. To improve it, you can try larger models.

", "votes": 0, "replies": [] }, { "id": 1800991, "author_name": "Lin Xi", "author_url": "", "post_date": "2022-05-25T10:45:58.310000", "content": "

Thanks for your reply, it's helpful~

", "votes": 0, "replies": [] } ] }, { "id": 1798613, "author_name": "Lucas Morin", "author_url": "", "post_date": "2022-05-23T07:46:44.583000", "content": "

Thanks :-) I think I was trying to do something like this, but apparently I was missing multiple pieces of the puzzle.

", "votes": 1, "replies": [ { "id": 1798642, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2022-05-23T08:53:34.217000", "content": "

I hope it inspires you :).

", "votes": 1, "replies": [] } ] }, { "id": 1798585, "author_name": "huiqin", "author_url": "", "post_date": "2022-05-23T07:07:23.083000", "content": "

Good solution! One question: How do you select the best checkpoint in train? How many epochs in train ?

", "votes": 1, "replies": [ { "id": 1798638, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2022-05-23T08:50:22.563000", "content": "

I did not save checkpoint by steps, because for this method, validation takes long time. So I only save for last step of each epochs.
\n3~4 epochs.

", "votes": 3, "replies": [] }, { "id": 1798806, "author_name": "huiqin", "author_url": "", "post_date": "2022-05-23T11:40:23.863000", "content": "

Thanks. I will try it.

", "votes": 0, "replies": [] } ] }, { "id": 1798453, "author_name": "", "author_url": "", "post_date": "2022-05-23T03:47:34.517000", "content": "

I think I am most happy that you cleaned up the old code more though. 👍

", "votes": 1, "replies": [ { "id": 1798643, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2022-05-23T08:54:08.040000", "content": "

Thanks you for your reply. I'm glad you find it helpful. :)

", "votes": 1, "replies": [] } ] }, { "id": 1798445, "author_name": "", "author_url": "", "post_date": "2022-05-23T03:39:41.797000", "content": "

Your generosity is amazing. I hope you win this competition.

", "votes": 1, "replies": [] }, { "id": 1798442, "author_name": "Chenglu", "author_url": "", "post_date": "2022-05-23T03:37:50.537000", "content": "

Thanks for this! Great approach to solve this problem.

", "votes": 1, "replies": [] }, { "id": 1830197, "author_name": "samu2505", "author_url": "", "post_date": "2022-06-23T08:59:08.953000", "content": "

This is my first attempt at NLP competitions. I have some questions

\n
    \n
  1. What do the terms [CLS], [SEP] mean and what is their signifance.
    2. \n
  2. Looking at the data, for every markdown cell, there are many cells for the code,
    \nthen how do you choose the cell to perform this operation [CLS] + markdown + [SEP] + code + [SEP]. An illustration could be useful.
    3. \n
", "votes": 0, "replies": [ { "id": 1836025, "author_name": "biubiubiu~", "author_url": "", "post_date": "2022-06-28T09:39:15.323000", "content": "

[CLS] and [SEP] are artificial tokens that are respectively inserted before the first sequence of tokens and between the first and second sequences.
\nA Pairwise Comparison is the process of comparing candidates in pairs to judge which of each candidate is preferred overall. Therefore, [CLS] + markdown + [SEP] + code + [SEP] is using to compare relevance between markdown and code. The [SEP] between markdown and code is here to help Bert understand which token belong to markdown and which token belong to code.

", "votes": 0, "replies": [] } ] }, { "id": 1817586, "author_name": "yrjjlili", "author_url": "", "post_date": "2022-06-11T15:12:21.330000", "content": "

Thank you very much！This is really helpful!

", "votes": 0, "replies": [] }, { "id": 1812373, "author_name": "aCode", "author_url": "", "post_date": "2022-06-05T18:10:11.107000", "content": "

Is the pairwise model a better choice for more optimization in the future?

", "votes": 0, "replies": [] }, { "id": 1808061, "author_name": "Vee", "author_url": "", "post_date": "2022-06-01T14:08:24.653000", "content": "

Thanks for sharing @yuanzhezhou. Regarding your comment below, could you clarify what was used to get 0.817 model -Is it a part of training data for 1epoch ?

\n
A pairwise model can achieve 0.832 score with 1 epoch training.\n
", "votes": 0, "replies": [ { "id": 1808164, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2022-06-01T15:22:55.387000", "content": "

You have all you need to get 0.817 in my training notebook.

\n

There are hints to improve the performance of pairwise models in another topic post by me.

", "votes": 1, "replies": [] } ] }, { "id": 1804524, "author_name": "chuan", "author_url": "", "post_date": "2022-05-29T05:40:58.090000", "content": "

Thanks for your share.
\nCould you show me the device config to train this model and if you use all data to train?

", "votes": 0, "replies": [ { "id": 1808175, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2022-06-01T15:28:11.373000", "content": "

50G RAM and 16G GPU memory

", "votes": 1, "replies": [] } ] }, { "id": 1801089, "author_name": "ROZZZZ", "author_url": "", "post_date": "2022-05-25T12:25:45.430000", "content": "

Good inspiration！

", "votes": 0, "replies": [] }, { "id": 1800442, "author_name": "", "author_url": "", "post_date": "2022-05-25T01:25:25.510000", "content": "", "votes": 1, "replies": [] }, { "id": 1799290, "author_name": "Making TARS", "author_url": "", "post_date": "2022-05-23T21:10:05.667000", "content": "

Thank you for sharing code!

", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1798042": "I posted my pairwise model baseline (~ 0.817) and inference notebooks here, the main idea is using bert model to determine if a markdown and a code is related, \n\n[CLS] + markdown + [SEP] + code + [SEP]\n\n\n* [Pretrain](https://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-pretrain/notebook)\n* [Training](https://www.kaggle.com/yuanzhezhou/ai4code-pairwise-bertsmall-training)\n* [Inference](https://www.kaggle.com/yuanzhezhou/ai4code-pairwise-bertsmall-inference)\n\n----------------------------------------------------------------------------------------------\npointwise model notebooks:\n* [inference](https://www.kaggle.com/code/yuanzhezhou/ai4code-distilbert-inference-777)\n* [training](https://www.kaggle.com/code/yuanzhezhou/ai4code-distilbert-training-777)\n\nEDIT1\nPS: Pairwise model is more suitable for accurate rank task. Here I used only a bert-small model. From my local experiments, larger models perform much better.\n\nEDIT2\nA pairwise model can achieve 0.832 score with 1 epoch training.", "1800858": "Thanks! I would like to know why you choose distill bert rather than bert_base or other PLM", "1798613": "Thanks :-) I think I was trying to do something like this, but apparently I was missing multiple pieces of the puzzle. ", "1798585": "Good solution! One question: How do you select the best checkpoint in train? How many epochs in train ?", "1798453": "I think I am most happy that you cleaned up the old code more though. 👍", "1798445": "Your generosity is amazing. I hope you win this competition.", "1798442": "Thanks for this! Great approach to solve this problem.", "1830197": "This is my first attempt at NLP competitions. I have some questions\n1. What do the terms [CLS], [SEP] mean and what is their signifance.\n2. Looking at the data, for every markdown cell, there are many cells for the code,\nthen how do you choose the cell to perform this operation [CLS] + markdown + [SEP] + code + [SEP]. An illustration could be useful.", "1817586": "Thank you very much！This is really helpful!", "1812373": "Is the pairwise model a better choice for more optimization in the future?", "1808061": "Thanks for sharing @yuanzhezhou. Regarding your comment below, could you clarify what was used to get 0.817 model -Is it a part of training data for 1epoch ?\n\n```\nA pairwise model can achieve 0.832 score with 1 epoch training.\n\n```\n", "1804524": "Thanks for your share. \nCould you show me the device config to train this model and if you use all data to train?", "1801089": "Good inspiration！", "1800442": "", "1799290": "Thank you for sharing code!" } }