{ "id": 343724, "title": "How long did you train? Did you finish training?", "url": "/competitions/AI4Code/discussion/343724", "author_name": "", "post_date": "2022-08-12T09:52:53.168655600Z", "votes": 11, "comment_count": 17, "views": 0, "content": "

Our best model (deberta-v3 base) was trained for 7 days, but I had to stop it on the last day of the competition. I feel like it never stopped improving.

\n

It is also possible that it never stopped overfitting since there were leaks on CV and public LB. I guess some public LB notebooks were \"similar\" enough (e.g. same competition and approach) to notebooks from train dataset.

\n

How long did you train and did you finish training?

", "messages": [ { "id": "1895660", "postDate": "08/12/2022 09:52:53", "content": "

Our best model (deberta-v3 base) was trained for 7 days, but I had to stop it on the last day of the competition. I feel like it never stopped improving.

\n

It is also possible that it never stopped overfitting since there were leaks on CV and public LB. I guess some public LB notebooks were \"similar\" enough (e.g. same competition and approach) to notebooks from train dataset.

\n

How long did you train and did you finish training?

", "rawMarkdown": "Our best model (deberta-v3 base) was trained for 7 days, but I had to stop it on the last day of the competition. I feel like it never stopped improving.\n\nIt is also possible that it never stopped overfitting since there were leaks on CV and public LB. I guess some public LB notebooks were \"similar\" enough (e.g. same competition and approach) to notebooks from train dataset.\n\nHow long did you train and did you finish training?", "votes": null }, { "id": "1895721", "postDate": "08/12/2022 10:45:01", "content": "

It depends, my context rank model using deberta-v3-small, markdown 128 * 3 + code 40 * 25 , running 9 epochs, need 2 days using 4*A100.
\nBut there are light methods, actually I learned from one of the top teams who just using pretrained embedding as cell inputs and do listwise learning can be very fast for training(just several hours), it also need little time to infer.

", "rawMarkdown": "It depends, my context rank model using deberta-v3-small, markdown 128 * 3 + code 40 * 25 , running 9 epochs, need 2 days using 4*A100.\nBut there are light methods, actually I learned from one of the top teams who just using pretrained embedding as cell inputs and do listwise learning can be very fast for training(just several hours), it also need little time to infer.", "votes": null }, { "id": "1895725", "postDate": "08/12/2022 10:54:19", "content": "

Wow, the week of training!!! How much did you gain from such long training? We trained for 2 days ~3-5 epochs.

", "rawMarkdown": "Wow, the week of training!!! How much did you gain from such long training? We trained for 2 days ~3-5 epochs.", "votes": null }, { "id": "1895755", "postDate": "08/12/2022 11:21:41", "content": "

I was definitely down a few echelons, but my last model trained for about 16hrs before my last submission (which was melded with an .8375 model to get to .8463) - interestingly a 24hr trained model came in slightly under that.

\n

The 'leaks' make sense since a lot of the notebooks are from ongoing Getting Started compettiions - possibly even more likely in the test set, since they won't get excluded because they were in this one.

", "rawMarkdown": "I was definitely down a few echelons, but my last model trained for about 16hrs before my last submission (which was melded with an .8375 model to get to .8463) - interestingly a 24hr trained model came in slightly under that.\n\nThe 'leaks' make sense since a lot of the notebooks are from ongoing Getting Started compettiions - possibly even *more* likely in the test set, since they won't get excluded because they were in this one.", "votes": null }, { "id": "1895936", "postDate": "08/12/2022 12:53:04", "content": "

The training was slow. My best score was on the last validation.

\n

It is possible my scheduler was not that good. It was hard to find good hyperparameters since it took so long to train an epoch.

", "rawMarkdown": "The training was slow. My best score was on the last validation.\n\nIt is possible my scheduler was not that good. It was hard to find good hyperparameters since it took so long to train an epoch.", "votes": null }, { "id": "1895938", "postDate": "08/12/2022 12:58:20", "content": "

The embedding approach didn't work for me. It is possible I had a bug somewhere.

\n

4*A100 in 2 days sounds like more than 7 days on 1 RTX 3090.

\n

@goldenlock did adding 3 MD cells help in score or just inference speed? We had 128 MD + 23 code cells with 28 length. Using length 24 for code cells gave similar results ( @init27 was training with 24 in parallel, but started later).

\n

Our inference was 7.5h for deberta v3 base. Hope it won't crash on rerun. Hope similar number of notebooks is less than 20% more notebooks.

", "rawMarkdown": "The embedding approach didn't work for me. It is possible I had a bug somewhere.\n\n4*A100 in 2 days sounds like more than 7 days on 1 RTX 3090.\n\n@goldenlock did adding 3 MD cells help in score or just inference speed? We had 128 MD + 23 code cells with 28 length. Using length 24 for code cells gave similar results ( @init27 was training with 24 in parallel, but started later).\n\nOur inference was 7.5h for deberta v3 base. Hope it won't crash on rerun. Hope similar number of notebooks is less than 20% more notebooks.", "votes": null }, { "id": "1896111", "postDate": "08/12/2022 15:33:57", "content": "

Before using it for fine-tuning, did you pretrain the DeBERTa model using MLM or anything similar?
\nI am asking because I also tried to use DeBERTa out of the box for fine-tuning (I was out of time for MLM..) but the results were worse than CodeGraphBert..

\n

As for training time: A single model training took around 3.5 days for me using 512 seq length with CodeGraphBert (GPUs: 2x Titan RTX).
\nI didn't even finish training my final model in time (I will late-submit it sometime this week to know for myself)..

", "rawMarkdown": "Before using it for fine-tuning, did you pretrain the DeBERTa model using MLM or anything similar? \nI am asking because I also tried to use DeBERTa out of the box for fine-tuning (I was out of time for MLM..) but the results were worse than CodeGraphBert.. \n\n**As for training time:** A single model training took around 3.5 days for me using 512 seq length with CodeGraphBert (GPUs: 2x Titan RTX).\nI didn't even finish training my final model in time (I will late-submit it sometime this week to know for myself)..", "votes": null }, { "id": "1896147", "postDate": "08/12/2022 16:10:12", "content": "

@crodoc By using group strategy of similar markdowns I could improve the result using 3 markdowns. But mainly for infer speed improvement. Below is a comparation without using recall model to choose 40 codes for training, if adding that the gain may be larger. Though not verified but I think there might be gain especially for adjacent markdowns.
\n\"\"
\nI think your rerun will pass🤠 I don't think there will be big difference and my solution also run 7.5h…. Personally I think 1h buffer is large enough：）

", "rawMarkdown": "crodoc By using group strategy of similar markdowns I could improve the result using 3 markdowns. But mainly for infer speed improvement. Below is a comparation without using recall model to choose 40 codes for training, if adding that the gain may be larger. Though not verified but I think there might be gain especially for adjacent markdowns.\n![](https://i.postimg.cc/rFdWMZ7d/1.png)\nI think your rerun will pass🤠 I don't think there will be big difference and my solution also run 7.5h.... Personally I think 1h buffer is large enough：）", "votes": null }, { "id": "1896170", "postDate": "08/12/2022 16:19:15", "content": "

Interesting I found deberta-v3-small much better then CodeBert and CodeGraphBert and antother thing is CodeBert and CodeGraphBert does not support length > 512 , if you wish to use long sequence you have to hack it using LSG method, maybe that hurt the perf, not sure.

", "rawMarkdown": "Interesting I found deberta-v3-small much better then CodeBert and CodeGraphBert and antother thing is CodeBert and CodeGraphBert does not support length > 512 , if you wish to use long sequence you have to hack it using LSG method, maybe that hurt the perf, not sure.", "votes": null }, { "id": "1896172", "postDate": "08/12/2022 16:20:03", "content": "

Exactly same for me. I also trained my model for a week and still on the half way of training (for both w/ and w/o external data).
\nI also had an issue at the negative sampling size caused by lacking of GPU memory during training, so I had no choice but using much smaller size of negative sampling for the large model than the one for the small model (on the small model the bigger sampling size always gets the better score). If I could use A100…

", "rawMarkdown": "Exactly same for me. I also trained my model for a week and still on the half way of training (for both w/ and w/o external data). \nI also had an issue at the negative sampling size caused by lacking of GPU memory during training, so I had no choice but using much smaller size of negative sampling for the large model than the one for the small model (on the small model the bigger sampling size always gets the better score). If I could use A100...", "votes": null }, { "id": "1896193", "postDate": "08/12/2022 16:34:48", "content": "

I gave up the competition when it took me 1 day on an A100 (Colab Pro+) to only pre-train the Deberta-v3-base with 2.5 epochs. My best decision in this year so far, never felt relieved more than that before 😎

", "rawMarkdown": "I gave up the competition when it took me 1 day on an A100 (Colab Pro+) to only pre-train the Deberta-v3-base with 2.5 epochs. My best decision in this year so far, never felt relieved more than that before 😎", "votes": null }, { "id": "1896199", "postDate": "08/12/2022 16:39:53", "content": "

Congratulations on your 1st place! Yes definetely better hardware matters a lot for fast develop, and 9 neg pariwise much beter then 4 neg I found in the last days but could only train fewer epochs to submit, especially for the model using all training data. We all wish we could have thought to add more kaggle external data, I have tried github notebooks but not much improvment and not used it. So you have done great work on this and I think the advatage is still huge for new data, as I verified using 100% training data the perf can improve about 0.01 then using 80%.

", "rawMarkdown": "Congratulations on your 1st place! Yes definetely better hardware matters a lot for fast develop, and 9 neg pariwise much beter then 4 neg I found in the last days but could only train fewer epochs to submit, especially for the model using all training data. We all wish we could have thought to add more kaggle external data, I have tried github notebooks but not much improvment and not used it. So you have done great work on this and I think the advatage is still huge for new data, as I verified using 100% training data the perf can improve about 0.01 then using 80%.", "votes": null }, { "id": "1896258", "postDate": "08/12/2022 17:45:43", "content": "

I should have tried using deberta v3 small with more code cells. And joined the competition earlier 😅

", "rawMarkdown": "I should have tried using deberta v3 small with more code cells. And joined the competition earlier 😅", "votes": null }, { "id": "1896261", "postDate": "08/12/2022 17:47:10", "content": "

Didn't want to use more than 1 MD cells since it took more tokens (128) than code cellds (28).

", "rawMarkdown": "Didn't want to use more than 1 MD cells since it took more tokens (128) than code cellds (28).", "votes": null }, { "id": "1896413", "postDate": "08/12/2022 19:53:58", "content": "

five days for one epoch with my rtx 2060. It's a nightmare

", "rawMarkdown": "five days for one epoch with my rtx 2060. It's a nightmare", "votes": null }, { "id": "1896444", "postDate": "08/12/2022 20:25:22", "content": "

What is LSG method?

", "rawMarkdown": "What is LSG method?", "votes": null }, { "id": "1896551", "postDate": "08/12/2022 22:59:03", "content": "

@crodoc Seems most top teams used much more code cells, no matter what method they use, more code cells(context) like 128+ is the key for win.

", "rawMarkdown": "crodoc Seems most top teams used much more code cells, no matter what method they use, more code cells(context) like 128+ is the key for win.", "votes": null }, { "id": "1896552", "postDate": "08/12/2022 22:59:58", "content": "

@cabbage972 here https://github.com/ccdv-ai/convert_checkpoint_to_lsg

", "rawMarkdown": "cabbage972 here https://github.com/ccdv-ai/convert_checkpoint_to_lsg", "votes": null } ], "comments": [ { "id": 1895721, "author_name": "goldenlock", "author_url": "", "post_date": "08/12/2022 10:45:01", "content": "

It depends, my context rank model using deberta-v3-small, markdown 128 * 3 + code 40 * 25 , running 9 epochs, need 2 days using 4*A100.
\nBut there are light methods, actually I learned from one of the top teams who just using pretrained embedding as cell inputs and do listwise learning can be very fast for training(just several hours), it also need little time to infer.

", "votes": null, "replies": [ { "id": 1895938, "author_name": "crodoc", "author_url": "", "post_date": "08/12/2022 12:58:20", "content": "

The embedding approach didn't work for me. It is possible I had a bug somewhere.

\n

4*A100 in 2 days sounds like more than 7 days on 1 RTX 3090.

\n

@goldenlock did adding 3 MD cells help in score or just inference speed? We had 128 MD + 23 code cells with 28 length. Using length 24 for code cells gave similar results ( @init27 was training with 24 in parallel, but started later).

\n

Our inference was 7.5h for deberta v3 base. Hope it won't crash on rerun. Hope similar number of notebooks is less than 20% more notebooks.

", "votes": null, "replies": [] }, { "id": 1896147, "author_name": "goldenlock", "author_url": "", "post_date": "08/12/2022 16:10:12", "content": "

@crodoc By using group strategy of similar markdowns I could improve the result using 3 markdowns. But mainly for infer speed improvement. Below is a comparation without using recall model to choose 40 codes for training, if adding that the gain may be larger. Though not verified but I think there might be gain especially for adjacent markdowns.
\n\"\"
\nI think your rerun will pass🤠 I don't think there will be big difference and my solution also run 7.5h…. Personally I think 1h buffer is large enough：）

", "votes": null, "replies": [] }, { "id": 1896258, "author_name": "crodoc", "author_url": "", "post_date": "08/12/2022 17:45:43", "content": "

I should have tried using deberta v3 small with more code cells. And joined the competition earlier 😅

", "votes": null, "replies": [] }, { "id": 1896261, "author_name": "crodoc", "author_url": "", "post_date": "08/12/2022 17:47:10", "content": "

Didn't want to use more than 1 MD cells since it took more tokens (128) than code cellds (28).

", "votes": null, "replies": [] }, { "id": 1896551, "author_name": "goldenlock", "author_url": "", "post_date": "08/12/2022 22:59:03", "content": "

@crodoc Seems most top teams used much more code cells, no matter what method they use, more code cells(context) like 128+ is the key for win.

", "votes": null, "replies": [] } ] }, { "id": 1895725, "author_name": "vad13irt", "author_url": "", "post_date": "08/12/2022 10:54:19", "content": "

Wow, the week of training!!! How much did you gain from such long training? We trained for 2 days ~3-5 epochs.

", "votes": null, "replies": [ { "id": 1895936, "author_name": "crodoc", "author_url": "", "post_date": "08/12/2022 12:53:04", "content": "

The training was slow. My best score was on the last validation.

\n

It is possible my scheduler was not that good. It was hard to find good hyperparameters since it took so long to train an epoch.

", "votes": null, "replies": [] } ] }, { "id": 1895755, "author_name": "happycube", "author_url": "", "post_date": "08/12/2022 11:21:41", "content": "

I was definitely down a few echelons, but my last model trained for about 16hrs before my last submission (which was melded with an .8375 model to get to .8463) - interestingly a 24hr trained model came in slightly under that.

\n

The 'leaks' make sense since a lot of the notebooks are from ongoing Getting Started compettiions - possibly even more likely in the test set, since they won't get excluded because they were in this one.

", "votes": null, "replies": [] }, { "id": 1896111, "author_name": "thedevastator", "author_url": "", "post_date": "08/12/2022 15:33:57", "content": "

Before using it for fine-tuning, did you pretrain the DeBERTa model using MLM or anything similar?
\nI am asking because I also tried to use DeBERTa out of the box for fine-tuning (I was out of time for MLM..) but the results were worse than CodeGraphBert..

\n

As for training time: A single model training took around 3.5 days for me using 512 seq length with CodeGraphBert (GPUs: 2x Titan RTX).
\nI didn't even finish training my final model in time (I will late-submit it sometime this week to know for myself)..

", "votes": null, "replies": [ { "id": 1896170, "author_name": "goldenlock", "author_url": "", "post_date": "08/12/2022 16:19:15", "content": "

Interesting I found deberta-v3-small much better then CodeBert and CodeGraphBert and antother thing is CodeBert and CodeGraphBert does not support length > 512 , if you wish to use long sequence you have to hack it using LSG method, maybe that hurt the perf, not sure.

", "votes": null, "replies": [] }, { "id": 1896444, "author_name": "cabbage972", "author_url": "", "post_date": "08/12/2022 20:25:22", "content": "

What is LSG method?

", "votes": null, "replies": [] }, { "id": 1896552, "author_name": "goldenlock", "author_url": "", "post_date": "08/12/2022 22:59:58", "content": "

@cabbage972 here https://github.com/ccdv-ai/convert_checkpoint_to_lsg

", "votes": null, "replies": [] } ] }, { "id": 1896172, "author_name": "yufuin", "author_url": "", "post_date": "08/12/2022 16:20:03", "content": "

Exactly same for me. I also trained my model for a week and still on the half way of training (for both w/ and w/o external data).
\nI also had an issue at the negative sampling size caused by lacking of GPU memory during training, so I had no choice but using much smaller size of negative sampling for the large model than the one for the small model (on the small model the bigger sampling size always gets the better score). If I could use A100…

", "votes": null, "replies": [ { "id": 1896199, "author_name": "goldenlock", "author_url": "", "post_date": "08/12/2022 16:39:53", "content": "

Congratulations on your 1st place! Yes definetely better hardware matters a lot for fast develop, and 9 neg pariwise much beter then 4 neg I found in the last days but could only train fewer epochs to submit, especially for the model using all training data. We all wish we could have thought to add more kaggle external data, I have tried github notebooks but not much improvment and not used it. So you have done great work on this and I think the advatage is still huge for new data, as I verified using 100% training data the perf can improve about 0.01 then using 80%.

", "votes": null, "replies": [] } ] }, { "id": 1896193, "author_name": "shinomoriaoshi", "author_url": "", "post_date": "08/12/2022 16:34:48", "content": "

I gave up the competition when it took me 1 day on an A100 (Colab Pro+) to only pre-train the Deberta-v3-base with 2.5 epochs. My best decision in this year so far, never felt relieved more than that before 😎

", "votes": null, "replies": [] }, { "id": 1896413, "author_name": "sanghuynh1501", "author_url": "", "post_date": "08/12/2022 19:53:58", "content": "

five days for one epoch with my rtx 2060. It's a nightmare

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1895660": "Our best model (deberta-v3 base) was trained for 7 days, but I had to stop it on the last day of the competition. I feel like it never stopped improving.\n\nIt is also possible that it never stopped overfitting since there were leaks on CV and public LB. I guess some public LB notebooks were \"similar\" enough (e.g. same competition and approach) to notebooks from train dataset.\n\nHow long did you train and did you finish training?", "1895721": "It depends, my context rank model using deberta-v3-small, markdown 128 * 3 + code 40 * 25 , running 9 epochs, need 2 days using 4*A100.\nBut there are light methods, actually I learned from one of the top teams who just using pretrained embedding as cell inputs and do listwise learning can be very fast for training(just several hours), it also need little time to infer.", "1895725": "Wow, the week of training!!! How much did you gain from such long training? We trained for 2 days ~3-5 epochs.", "1895755": "I was definitely down a few echelons, but my last model trained for about 16hrs before my last submission (which was melded with an .8375 model to get to .8463) - interestingly a 24hr trained model came in slightly under that.\n\nThe 'leaks' make sense since a lot of the notebooks are from ongoing Getting Started compettiions - possibly even *more* likely in the test set, since they won't get excluded because they were in this one.", "1895936": "The training was slow. My best score was on the last validation.\n\nIt is possible my scheduler was not that good. It was hard to find good hyperparameters since it took so long to train an epoch.", "1895938": "The embedding approach didn't work for me. It is possible I had a bug somewhere.\n\n4*A100 in 2 days sounds like more than 7 days on 1 RTX 3090.\n\n@goldenlock did adding 3 MD cells help in score or just inference speed? We had 128 MD + 23 code cells with 28 length. Using length 24 for code cells gave similar results ( @init27 was training with 24 in parallel, but started later).\n\nOur inference was 7.5h for deberta v3 base. Hope it won't crash on rerun. Hope similar number of notebooks is less than 20% more notebooks.", "1896111": "Before using it for fine-tuning, did you pretrain the DeBERTa model using MLM or anything similar? \nI am asking because I also tried to use DeBERTa out of the box for fine-tuning (I was out of time for MLM..) but the results were worse than CodeGraphBert.. \n\n**As for training time:** A single model training took around 3.5 days for me using 512 seq length with CodeGraphBert (GPUs: 2x Titan RTX).\nI didn't even finish training my final model in time (I will late-submit it sometime this week to know for myself)..", "1896147": "crodoc By using group strategy of similar markdowns I could improve the result using 3 markdowns. But mainly for infer speed improvement. Below is a comparation without using recall model to choose 40 codes for training, if adding that the gain may be larger. Though not verified but I think there might be gain especially for adjacent markdowns.\n![](https://i.postimg.cc/rFdWMZ7d/1.png)\nI think your rerun will pass🤠 I don't think there will be big difference and my solution also run 7.5h.... Personally I think 1h buffer is large enough：）", "1896170": "Interesting I found deberta-v3-small much better then CodeBert and CodeGraphBert and antother thing is CodeBert and CodeGraphBert does not support length > 512 , if you wish to use long sequence you have to hack it using LSG method, maybe that hurt the perf, not sure.", "1896172": "Exactly same for me. I also trained my model for a week and still on the half way of training (for both w/ and w/o external data). \nI also had an issue at the negative sampling size caused by lacking of GPU memory during training, so I had no choice but using much smaller size of negative sampling for the large model than the one for the small model (on the small model the bigger sampling size always gets the better score). If I could use A100...", "1896193": "I gave up the competition when it took me 1 day on an A100 (Colab Pro+) to only pre-train the Deberta-v3-base with 2.5 epochs. My best decision in this year so far, never felt relieved more than that before 😎", "1896199": "Congratulations on your 1st place! Yes definetely better hardware matters a lot for fast develop, and 9 neg pariwise much beter then 4 neg I found in the last days but could only train fewer epochs to submit, especially for the model using all training data. We all wish we could have thought to add more kaggle external data, I have tried github notebooks but not much improvment and not used it. So you have done great work on this and I think the advatage is still huge for new data, as I verified using 100% training data the perf can improve about 0.01 then using 80%.", "1896258": "I should have tried using deberta v3 small with more code cells. And joined the competition earlier 😅", "1896261": "Didn't want to use more than 1 MD cells since it took more tokens (128) than code cellds (28).", "1896413": "five days for one epoch with my rtx 2060. It's a nightmare", "1896444": "What is LSG method?", "1896551": "crodoc Seems most top teams used much more code cells, no matter what method they use, more code cells(context) like 128+ is the key for win.", "1896552": "cabbage972 here https://github.com/ccdv-ai/convert_checkpoint_to_lsg" }, "source": "meta" }