From my few experiments, it seemed that random sampling 5% of negative pairs gave the best result. If I sample negative pairs (code cells) by count of similar words compared to markdown cell, the result become worse.
\nHow could we build our negative pairs for supervised learning for pairwise models?
", "messages": [ { "id": "1800330", "postDate": "05/24/2022 22:56:23", "content": "From my few experiments, it seemed that random sampling 5% of negative pairs gave the best result. If I sample negative pairs (code cells) by count of similar words compared to markdown cell, the result become worse.
\nHow could we build our negative pairs for supervised learning for pairwise models?
", "rawMarkdown": "From my few experiments, it seemed that **random sampling 5% of negative pairs gave the best result**. If I sample negative pairs (code cells) by count of similar words compared to markdown cell, the result become worse. \n\nHow could we build our negative pairs for supervised learning for pairwise models?", "votes": null }, { "id": "1803946", "postDate": "05/28/2022 11:10:02", "content": "I'm also interested in that theme, but I don't understand the title.
\nCan you tell me the meaning of the negative pair?
We need to constuct labels for supervised learning. We regard a related code and markdown as positivev pair, a non-related code and markdown are negative pairs.
", "rawMarkdown": "We need to constuct labels for supervised learning. We regard a related code and markdown as positivev pair, a non-related code and markdown are negative pairs.", "votes": null }, { "id": "1804427", "postDate": "05/29/2022 00:34:02", "content": "I understand the terms,
\nbut I don't know how to distinguish between negative and positive….
\nIs it the part that corresponds to the \"def generate_triplet(df, mode='train')\" your notebook https://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-training?
yes, it generates training samples here.
", "rawMarkdown": "yes, it generates training samples here.", "votes": null }, { "id": "1804605", "postDate": "05/29/2022 08:36:11", "content": "How can I try this approach? Is negative sampling is done only for the train dataset or for both train and test data?
", "rawMarkdown": "How can I try this approach? Is negative sampling is done only for the train dataset or for both train and test data?", "votes": null }, { "id": "1804719", "postDate": "05/29/2022 11:34:25", "content": "You can use my code here to train a pairwise model. https://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-training
\nPairwise model can achieve 84+ score with 1 epoch training, I believe it can be improved much furthur.
", "rawMarkdown": "You can use my code here to train a pairwise model. https://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-training\n\nPairwise model can achieve 84+ score with 1 epoch training, I believe it can be improved much furthur.", "votes": null }, { "id": "1805700", "postDate": "05/30/2022 12:43:57", "content": "I checked the code of the pairwise model.
\nMy understanding is that I'm using ranks when building training data,
\nso I think it leaks in validation.
Is there something wrong?
", "rawMarkdown": "I checked the code of the pairwise model. \nMy understanding is that I'm using ranks when building training data, \nso I think it leaks in validation. \n\nIs there something wrong?", "votes": null }, { "id": "1805822", "postDate": "05/30/2022 14:31:58", "content": "No, I use rank to tell if 2 cells are related (difference of position=1). And rank is calculated for cell_ids belonging to a given id.
", "rawMarkdown": "No, I use rank to tell if 2 cells are related (difference of position=1). And rank is calculated for cell_ids belonging to a given id.", "votes": null }, { "id": "1806859", "postDate": "05/31/2022 14:01:05", "content": "would you please share how much data and batch size you use, and how long 1 epoch takes? thanks~
", "rawMarkdown": "would you please share how much data and batch size you use, and how long 1 epoch takes? thanks~", "votes": null }, { "id": "1806871", "postDate": "05/31/2022 14:14:10", "content": "Thank you for your reply.
\nIt's a little difficult, so let's actually analyze your public notebook,
\ntry and error,
\nand think again.
I used batch size as large as possible. The training time for 1 epoch depends on my settings, usually 3~8 hours.
", "rawMarkdown": "I used batch size as large as possible. The training time for 1 epoch depends on my settings, usually 3~8 hours.", "votes": null }, { "id": "1806878", "postDate": "05/31/2022 14:21:09", "content": "thanks for reply, I tried to use all the training data, and it shows 66 hours for 1 epoch on 16g GPU memory lol😅
", "rawMarkdown": "thanks for reply, I tried to use all the training data, and it shows 66 hours for 1 epoch on 16g GPU memory lol😅", "votes": null }, { "id": "1806885", "postDate": "05/31/2022 14:30:32", "content": "reducing seq_len to 64 or less should help.
", "rawMarkdown": "reducing seq_len to 64 or less should help.", "votes": null }, { "id": "1806912", "postDate": "05/31/2022 15:04:36", "content": "thanks I will try it
", "rawMarkdown": "thanks I will try it", "votes": null }, { "id": "1840351", "postDate": "07/02/2022 07:24:47", "content": "Hi guy, This approach is pretty useful for us. Do you think that we should use a dataset that has the number of positive and negative pairwise are about the same for training pupose?
", "rawMarkdown": "Hi guy, This approach is pretty useful for us. Do you think that we should use a dataset that has the number of positive and negative pairwise are about the same for training pupose?", "votes": null } ], "comments": [ { "id": 1803946, "author_name": "kunihikofurugori", "author_url": "", "post_date": "05/28/2022 11:10:02", "content": "I'm also interested in that theme, but I don't understand the title.
\nCan you tell me the meaning of the negative pair?
We need to constuct labels for supervised learning. We regard a related code and markdown as positivev pair, a non-related code and markdown are negative pairs.
", "votes": null, "replies": [] }, { "id": 1804427, "author_name": "kunihikofurugori", "author_url": "", "post_date": "05/29/2022 00:34:02", "content": "I understand the terms,
\nbut I don't know how to distinguish between negative and positive….
\nIs it the part that corresponds to the \"def generate_triplet(df, mode='train')\" your notebook https://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-training?
yes, it generates training samples here.
", "votes": null, "replies": [] }, { "id": 1805700, "author_name": "kunihikofurugori", "author_url": "", "post_date": "05/30/2022 12:43:57", "content": "I checked the code of the pairwise model.
\nMy understanding is that I'm using ranks when building training data,
\nso I think it leaks in validation.
Is there something wrong?
", "votes": null, "replies": [] }, { "id": 1805822, "author_name": "yuanzhezhou", "author_url": "", "post_date": "05/30/2022 14:31:58", "content": "No, I use rank to tell if 2 cells are related (difference of position=1). And rank is calculated for cell_ids belonging to a given id.
", "votes": null, "replies": [] }, { "id": 1806871, "author_name": "kunihikofurugori", "author_url": "", "post_date": "05/31/2022 14:14:10", "content": "Thank you for your reply.
\nIt's a little difficult, so let's actually analyze your public notebook,
\ntry and error,
\nand think again.
How can I try this approach? Is negative sampling is done only for the train dataset or for both train and test data?
", "votes": null, "replies": [ { "id": 1804719, "author_name": "yuanzhezhou", "author_url": "", "post_date": "05/29/2022 11:34:25", "content": "You can use my code here to train a pairwise model. https://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-training
\nPairwise model can achieve 84+ score with 1 epoch training, I believe it can be improved much furthur.
", "votes": null, "replies": [] }, { "id": 1806859, "author_name": "please1993", "author_url": "", "post_date": "05/31/2022 14:01:05", "content": "would you please share how much data and batch size you use, and how long 1 epoch takes? thanks~
", "votes": null, "replies": [] }, { "id": 1806872, "author_name": "yuanzhezhou", "author_url": "", "post_date": "05/31/2022 14:15:36", "content": "I used batch size as large as possible. The training time for 1 epoch depends on my settings, usually 3~8 hours.
", "votes": null, "replies": [] }, { "id": 1806878, "author_name": "please1993", "author_url": "", "post_date": "05/31/2022 14:21:09", "content": "thanks for reply, I tried to use all the training data, and it shows 66 hours for 1 epoch on 16g GPU memory lol😅
", "votes": null, "replies": [] }, { "id": 1806885, "author_name": "yuanzhezhou", "author_url": "", "post_date": "05/31/2022 14:30:32", "content": "reducing seq_len to 64 or less should help.
", "votes": null, "replies": [] }, { "id": 1806912, "author_name": "please1993", "author_url": "", "post_date": "05/31/2022 15:04:36", "content": "thanks I will try it
", "votes": null, "replies": [] } ] }, { "id": 1840351, "author_name": "nguynthhi", "author_url": "", "post_date": "07/02/2022 07:24:47", "content": "Hi guy, This approach is pretty useful for us. Do you think that we should use a dataset that has the number of positive and negative pairwise are about the same for training pupose?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1800330": "From my few experiments, it seemed that **random sampling 5% of negative pairs gave the best result**. If I sample negative pairs (code cells) by count of similar words compared to markdown cell, the result become worse. \n\nHow could we build our negative pairs for supervised learning for pairwise models?", "1803946": "I'm also interested in that theme, but I don't understand the title.\nCan you tell me the meaning of the negative pair?", "1803996": "We need to constuct labels for supervised learning. We regard a related code and markdown as positivev pair, a non-related code and markdown are negative pairs.", "1804427": "I understand the terms, \nbut I don't know how to distinguish between negative and positive....\nIs it the part that corresponds to the \"def generate_triplet(df, mode='train')\" your notebook https://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-training?", "1804451": "yes, it generates training samples here.", "1804605": "How can I try this approach? Is negative sampling is done only for the train dataset or for both train and test data?", "1804719": "You can use my code here to train a pairwise model. https://www.kaggle.com/code/yuanzhezhou/ai4code-pairwise-bertsmall-training\n\nPairwise model can achieve 84+ score with 1 epoch training, I believe it can be improved much furthur.", "1805700": "I checked the code of the pairwise model. \nMy understanding is that I'm using ranks when building training data, \nso I think it leaks in validation. \n\nIs there something wrong?", "1805822": "No, I use rank to tell if 2 cells are related (difference of position=1). And rank is calculated for cell_ids belonging to a given id.", "1806859": "would you please share how much data and batch size you use, and how long 1 epoch takes? thanks~", "1806871": "Thank you for your reply. \nIt's a little difficult, so let's actually analyze your public notebook, \ntry and error, \nand think again.", "1806872": "I used batch size as large as possible. The training time for 1 epoch depends on my settings, usually 3~8 hours.", "1806878": "thanks for reply, I tried to use all the training data, and it shows 66 hours for 1 epoch on 16g GPU memory lol😅", "1806885": "reducing seq_len to 64 or less should help.", "1806912": "thanks I will try it", "1840351": "Hi guy, This approach is pretty useful for us. Do you think that we should use a dataset that has the number of positive and negative pairwise are about the same for training pupose?" }, "source": "meta" }