{
  "id": 360501,
  "title": "1st Place Solution",
  "url": "/competitions/AI4Code/writeups/hyd-1st-place-solution",
  "author_name": "",
  "post_date": "2023-12-12T03:30:26.123Z",
  "votes": 62,
  "comment_count": 28,
  "views": 0,
  "content": "<p><strong>Overview</strong></p>\n<p>I used single list-wise deberta-v3-large trained on all competition data. Input text is cells separated with [SEP] and [CLS] tokens.</p>\n<p><strong>Code</strong></p>\n<ul>\n<li>training code: <a href=\"https://github.com/hydantess/2022-Google-AI4Code-1st-Place-Solution\" target=\"_blank\">https://github.com/hydantess/2022-Google-AI4Code-1st-Place-Solution</a></li>\n<li>inference notebook: <a href=\"https://www.kaggle.com/code/hydantess/ai4code0729?scriptVersionId=103060429\" target=\"_blank\">https://www.kaggle.com/code/hydantess/ai4code0729?scriptVersionId=103060429</a></li>\n</ul>\n<p><strong>What Worked</strong></p>\n<ul>\n<li>MLM</li>\n<li>preprocessing<ul>\n<li>Reasoning and context: It makes text shorter, so we can handle longer text. </li></ul></li>\n<li>use deberta rather than codebert</li>\n<li>+lstm head</li>\n<li>bigger cell_cnt and seq_length<ul>\n<li>Reasoning and context: metric function cares more about long text. </li></ul></li>\n<li>+postprocessing: predict code and markdown both, then sort code by ground truth.<ul>\n<li>Reasoning and context: more reasonable than only predict markdown, and concat with code cells. </li></ul></li>\n</ul>\n<p><strong>What Didn’t Work</strong></p>\n<ul>\n<li>translation: facebook/mbart-large-50-many-to-one-mmt<ul>\n<li>Reasoning and context: most notebooks are in english. Or I did something wrong.</li></ul></li>\n<li>+code rank to dense layers<ul>\n<li>Reasoning and context:  I should try to concat text with rank information. Model will learn better.</li></ul></li>\n<li>extract embedding, then train a transformer model<ul>\n<li>Reasoning and context: I should pre-train the extracting model first.</li></ul></li>\n<li>xlm-robert-large or mdeberta-v3-base<ul>\n<li>Reasoning and context: much worse than deberta-v3-large. so can not ensemble directly.</li></ul></li>\n</ul>\n<p><strong>Important Citations</strong></p>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/ilyaryabov/fastttext-sorting-with-cosine-distance-algo\" target=\"_blank\">https://www.kaggle.com/code/ilyaryabov/fastttext-sorting-with-cosine-distance-algo</a> by ilyaryabov</li>\n<li><a href=\"https://www.kaggle.com/competitions/AI4Code/discussion/327626#1804178\" target=\"_blank\">https://www.kaggle.com/competitions/AI4Code/discussion/327626#1804178</a> by Nguyentuananh</li>\n</ul>\n<p><strong>Additional Context</strong></p>\n<ul>\n<li>loss is MAE.</li>\n<li>MLM on training data for 15 epochs,  max_length is 1024. It takes 3 days. </li>\n<li>training for 10 epochs, max_length is 2048. It takes 7 days. I think set max_length to 5120 will greatly improve model performance but I have not enough A100~</li>\n<li>inference: max_length is 5120. It takes 6 hours.</li>\n<li>I used 1 * A100(80GB)  for the last two weeks and 1 * 3090(24GB) at the early stage. </li>\n<li>Future work:<ul>\n<li>concat text with rank information: flag + rank_1 + cell_1 + flag+ rank_2 + cell_2 + … + flag+ rank_n + cell_n</li>\n<li>multi languages</li>\n<li>extra training data</li>\n<li>Adversarial Weight Perturbations (AWP), but too time-consuming</li></ul></li>\n</ul>\n<p><strong>Thanks and Acknowledgements:</strong></p>\n<p>Thanks to competition organizers for hosting such an amazing NLP competition. And thanks to the<br>\ngreat notebooks and discussions, I learned a lot. </p>\n<p><strong>Team Members:</strong></p>\n<ul>\n<li><a href=\"https://www.kaggle.com/hydantess\" target=\"_blank\">@hydantess</a> <br>\n.</li>\n</ul>",
  "messages": [
    {
      "id": "1991325",
      "postDate": "10/17/2022 05:26:51",
      "content": "<p><strong>Overview</strong></p>\n<p>I used single list-wise deberta-v3-large trained on all competition data. Input text is cells separated with [SEP] and [CLS] tokens.</p>\n<p><strong>Code</strong></p>\n<ul>\n<li>training code: <a href=\"https://github.com/hydantess/2022-Google-AI4Code-1st-Place-Solution\" target=\"_blank\">https://github.com/hydantess/2022-Google-AI4Code-1st-Place-Solution</a></li>\n<li>inference notebook: <a href=\"https://www.kaggle.com/code/hydantess/ai4code0729?scriptVersionId=103060429\" target=\"_blank\">https://www.kaggle.com/code/hydantess/ai4code0729?scriptVersionId=103060429</a></li>\n</ul>\n<p><strong>What Worked</strong></p>\n<ul>\n<li>MLM</li>\n<li>preprocessing<ul>\n<li>Reasoning and context: It makes text shorter, so we can handle longer text. </li></ul></li>\n<li>use deberta rather than codebert</li>\n<li>+lstm head</li>\n<li>bigger cell_cnt and seq_length<ul>\n<li>Reasoning and context: metric function cares more about long text. </li></ul></li>\n<li>+postprocessing: predict code and markdown both, then sort code by ground truth.<ul>\n<li>Reasoning and context: more reasonable than only predict markdown, and concat with code cells. </li></ul></li>\n</ul>\n<p><strong>What Didn’t Work</strong></p>\n<ul>\n<li>translation: facebook/mbart-large-50-many-to-one-mmt<ul>\n<li>Reasoning and context: most notebooks are in english. Or I did something wrong.</li></ul></li>\n<li>+code rank to dense layers<ul>\n<li>Reasoning and context:  I should try to concat text with rank information. Model will learn better.</li></ul></li>\n<li>extract embedding, then train a transformer model<ul>\n<li>Reasoning and context: I should pre-train the extracting model first.</li></ul></li>\n<li>xlm-robert-large or mdeberta-v3-base<ul>\n<li>Reasoning and context: much worse than deberta-v3-large. so can not ensemble directly.</li></ul></li>\n</ul>\n<p><strong>Important Citations</strong></p>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/ilyaryabov/fastttext-sorting-with-cosine-distance-algo\" target=\"_blank\">https://www.kaggle.com/code/ilyaryabov/fastttext-sorting-with-cosine-distance-algo</a> by ilyaryabov</li>\n<li><a href=\"https://www.kaggle.com/competitions/AI4Code/discussion/327626#1804178\" target=\"_blank\">https://www.kaggle.com/competitions/AI4Code/discussion/327626#1804178</a> by Nguyentuananh</li>\n</ul>\n<p><strong>Additional Context</strong></p>\n<ul>\n<li>loss is MAE.</li>\n<li>MLM on training data for 15 epochs,  max_length is 1024. It takes 3 days. </li>\n<li>training for 10 epochs, max_length is 2048. It takes 7 days. I think set max_length to 5120 will greatly improve model performance but I have not enough A100~</li>\n<li>inference: max_length is 5120. It takes 6 hours.</li>\n<li>I used 1 * A100(80GB)  for the last two weeks and 1 * 3090(24GB) at the early stage. </li>\n<li>Future work:<ul>\n<li>concat text with rank information: flag + rank_1 + cell_1 + flag+ rank_2 + cell_2 + … + flag+ rank_n + cell_n</li>\n<li>multi languages</li>\n<li>extra training data</li>\n<li>Adversarial Weight Perturbations (AWP), but too time-consuming</li></ul></li>\n</ul>\n<p><strong>Thanks and Acknowledgements:</strong></p>\n<p>Thanks to competition organizers for hosting such an amazing NLP competition. And thanks to the<br>\ngreat notebooks and discussions, I learned a lot. </p>\n<p><strong>Team Members:</strong></p>\n<ul>\n<li><a href=\"https://www.kaggle.com/hydantess\" target=\"_blank\">@hydantess</a> <br>\n.</li>\n</ul>",
      "rawMarkdown": "**Overview**\n\nI used single list-wise deberta-v3-large trained on all competition data. Input text is cells separated with [SEP] and [CLS] tokens.\n\n\n**Code**\n- training code: https://github.com/hydantess/2022-Google-AI4Code-1st-Place-Solution\n- inference notebook: https://www.kaggle.com/code/hydantess/ai4code0729?scriptVersionId=103060429\n\n**What Worked**\n- MLM\n- preprocessing\n   - Reasoning and context: It makes text shorter, so we can handle longer text. \n- use deberta rather than codebert\n- +lstm head\n- bigger cell_cnt and seq_length\n   - Reasoning and context: metric function cares more about long text. \n- +postprocessing: predict code and markdown both, then sort code by ground truth.\n   - Reasoning and context: more reasonable than only predict markdown, and concat with code cells. \n\n**What Didn’t Work**\n- translation: facebook/mbart-large-50-many-to-one-mmt\n   - Reasoning and context: most notebooks are in english. Or I did something wrong.\n- +code rank to dense layers\n   - Reasoning and context:  I should try to concat text with rank information. Model will learn better.\n- extract embedding, then train a transformer model\n - Reasoning and context: I should pre-train the extracting model first.\n- xlm-robert-large or mdeberta-v3-base\n   - Reasoning and context: much worse than deberta-v3-large. so can not ensemble directly.\n\n\n**Important Citations**\n- https://www.kaggle.com/code/ilyaryabov/fastttext-sorting-with-cosine-distance-algo by ilyaryabov\n- https://www.kaggle.com/competitions/AI4Code/discussion/327626#1804178 by Nguyentuananh\n\n**Additional Context**\n- loss is MAE.\n- MLM on training data for 15 epochs,  max_length is 1024. It takes 3 days. \n- training for 10 epochs, max_length is 2048. It takes 7 days. I think set max_length to 5120 will greatly improve model performance but I have not enough A100~\n- inference: max_length is 5120. It takes 6 hours.\n-  I used 1 * A100(80GB)  for the last two weeks and 1 * 3090(24GB) at the early stage. \n-  Future work:\n    - concat text with rank information: flag + rank_1 + cell_1 + flag+ rank_2 + cell_2 + … + flag+ rank_n + cell_n\n    - multi languages\n    - extra training data\n    - Adversarial Weight Perturbations (AWP), but too time-consuming\n\n**Thanks and Acknowledgements:**\n\nThanks to competition organizers for hosting such an amazing NLP competition. And thanks to the\ngreat notebooks and discussions, I learned a lot. \n\n**Team Members:**\n- @hydantess \n .",
      "votes": null
    },
    {
      "id": "1991867",
      "postDate": "10/17/2022 12:26:16",
      "content": "<p>Congratulations on your solo gold and your current 1st place position. Good luck to you for the final reruns. <br>\nFirst of all, thank you for your solution write-up. I got two follow-up questions:<br>\n(1) Did you check the model results with and without text preprocessing? Did this step help you a lot?<br>\n(2) Considering that you used deberta-v3-large and the number of days it took you for training:<br>\nWere you able to perform hyperparameter tuning? <br>\nWere you working on a sub-sampled dataset to speed up your experiments?</p>",
      "rawMarkdown": "Congratulations on your solo gold and your current 1st place position. Good luck to you for the final reruns. \nFirst of all, thank you for your solution write-up. I got two follow-up questions:\n(1) Did you check the model results with and without text preprocessing? Did this step help you a lot?\n(2) Considering that you used deberta-v3-large and the number of days it took you for training:\nWere you able to perform hyperparameter tuning? \nWere you working on a sub-sampled dataset to speed up your experiments?",
      "votes": null
    },
    {
      "id": "1991900",
      "postDate": "10/17/2022 12:43:33",
      "content": "<p>1) It makes text shorter, so we can handle long text. Otherwise 5120 is not enough.<br>\n2) Yes. I only sample 30000 notebooks for fast experiment at the early stage. </p>",
      "rawMarkdown": "1) It makes text shorter, so we can handle long text. Otherwise 5120 is not enough.\n2) Yes. I only sample 30000 notebooks for fast experiment at the early stage.",
      "votes": null
    },
    {
      "id": "1992040",
      "postDate": "10/17/2022 14:01:46",
      "content": "<p>Your solution, especially the max length of the input, blew up my mind 😄… I hope, you won't have an OOM error at the end 🙏</p>",
      "rawMarkdown": "Your solution, especially the max length of the input, blew up my mind 😄... I hope, you won't have an OOM error at the end 🙏",
      "votes": null
    },
    {
      "id": "1992107",
      "postDate": "10/17/2022 14:35:28",
      "content": "<p>I tested it before. So it's fine.</p>",
      "rawMarkdown": "I tested it before. So it's fine.",
      "votes": null
    },
    {
      "id": "1992279",
      "postDate": "10/17/2022 16:08:50",
      "content": "<p>Congrats! and thank for sharing your solution.<br>\nI have one question here.<br>\nWhat's the output of the model? I am confused because from your code the output dimension is 1, and the previous context indicates the target is more than one dimensions. Or is it list-wise or point-wise model? </p>",
      "rawMarkdown": "Congrats! and thank for sharing your solution.\nI have one question here.\nWhat's the output of the model? I am confused because from your code the output dimension is 1, and the previous context indicates the target is more than one dimensions. Or is it list-wise or point-wise model?",
      "votes": null
    },
    {
      "id": "1992320",
      "postDate": "10/17/2022 16:24:53",
      "content": "<p>List-wise. </p>",
      "rawMarkdown": "List-wise.",
      "votes": null
    },
    {
      "id": "1999565",
      "postDate": "10/22/2022 13:24:58",
      "content": "<p>Now you are the top of the leaderboard. Great. </p>",
      "rawMarkdown": "Now you are the top of the leaderboard. Great.",
      "votes": null
    },
    {
      "id": "2001636",
      "postDate": "10/24/2022 07:43:34",
      "content": "<p>Congrats!<br>\nCan you explain your listwise method more detailly? Or just throw me a document or website?</p>",
      "rawMarkdown": "Congrats!\nCan you explain your listwise method more detailly? Or just throw me a document or website?",
      "votes": null
    },
    {
      "id": "2001875",
      "postDate": "10/24/2022 11:08:06",
      "content": "<p>Are you using percentile ranks or just the absolute ranks as target?</p>",
      "rawMarkdown": "Are you using percentile ranks or just the absolute ranks as target?",
      "votes": null
    },
    {
      "id": "2001970",
      "postDate": "10/24/2022 12:36:12",
      "content": "<p>percentile ranks. </p>",
      "rawMarkdown": "percentile ranks.",
      "votes": null
    },
    {
      "id": "2002162",
      "postDate": "10/24/2022 14:51:14",
      "content": "<p>Thanks! Another question: <br>\nThe texts are \"flag + cell_1 + flag + cell_2 + … + flag + cell_n\". Does each cell have a # tokens quota? Did you truncate certain cells if it is too long? Or did you select shorter cells only such that 128 cells just need 2048 tokens? </p>\n<p>Perhaps a better question should be: How did you make sure 2048 tokens can include 128 cells?</p>",
      "rawMarkdown": "Thanks! Another question: \nThe texts are \"flag + cell_1 + flag + cell_2 + … + flag + cell_n\". Does each cell have a # tokens quota? Did you truncate certain cells if it is too long? Or did you select shorter cells only such that 128 cells just need 2048 tokens? \n\nPerhaps a better question should be: How did you make sure 2048 tokens can include 128 cells?",
      "votes": null
    },
    {
      "id": "2002185",
      "postDate": "10/24/2022 15:15:50",
      "content": "<ol>\n<li>Cells are truncated and length is smaller than 128.</li>\n<li>Each cell's length starts from 1 and for-looper all cells, step = 4, then break when total length is greater than certain value.</li>\n</ol>",
      "rawMarkdown": "1. Cells are truncated and length is smaller than 128.\n2. Each cell's length starts from 1 and for-looper all cells, step = 4, then break when total length is greater than certain value.",
      "votes": null
    },
    {
      "id": "2002379",
      "postDate": "10/24/2022 18:44:32",
      "content": "<p>When training, in the worst case (128 cells), max_length of a cell is just 2048/128 = 16?</p>",
      "rawMarkdown": "When training, in the worst case (128 cells), max_length of a cell is just 2048/128 = 16?",
      "votes": null
    },
    {
      "id": "2002612",
      "postDate": "10/25/2022 00:27:16",
      "content": "<p>Yes, you are right.</p>",
      "rawMarkdown": "Yes, you are right.",
      "votes": null
    },
    {
      "id": "2026340",
      "postDate": "11/12/2022 00:58:48",
      "content": "<p>Congratulations on winning the challenge!</p>\n<p>An impressive result, especially for the single model without ensembling.</p>",
      "rawMarkdown": "Congratulations on winning the challenge!\n\nAn impressive result, especially for the single model without ensembling.",
      "votes": null
    },
    {
      "id": "2026398",
      "postDate": "11/12/2022 03:19:04",
      "content": "<p>Thanks. Your poly-encoder like approach is also very impressive. I tried translating and using xlm-roberta model but failed!</p>",
      "rawMarkdown": "Thanks. Your poly-encoder like approach is also very impressive. I tried translating and using xlm-roberta model but failed!",
      "votes": null
    },
    {
      "id": "2026759",
      "postDate": "11/12/2022 10:22:12",
      "content": "<p>Congratulations to <a href=\"https://www.kaggle.com/hydantess\" target=\"_blank\">@hydantess</a> and all the prize winners!<br>\nYou are the geniuses!!!</p>",
      "rawMarkdown": "Congratulations to @hydantess and all the prize winners!\nYou are the geniuses!!!",
      "votes": null
    },
    {
      "id": "2026857",
      "postDate": "11/12/2022 11:38:01",
      "content": "<p>You are the genius too! I am very happy competing with you~</p>",
      "rawMarkdown": "You are the genius too! I am very happy competing with you~",
      "votes": null
    },
    {
      "id": "2027101",
      "postDate": "11/12/2022 14:51:55",
      "content": "<p>Congrats on entering top10! How do you deal with multiple languages, with deberta models?</p>",
      "rawMarkdown": "Congrats on entering top10! How do you deal with multiple languages, with deberta models?",
      "votes": null
    },
    {
      "id": "2027120",
      "postDate": "11/12/2022 15:07:51",
      "content": "<p>I didn't deal with multi languages situation, so just list it in feature work.<br>\nI think Dmytro Poplavskiy's poly-encoder like approach may help.</p>",
      "rawMarkdown": "I didn't deal with multi languages situation, so just list it in feature work.\nI think Dmytro Poplavskiy's poly-encoder like approach may help.",
      "votes": null
    },
    {
      "id": "2027642",
      "postDate": "11/13/2022 03:45:56",
      "content": "<p>Oh, I think it should be future work, hhh.</p>",
      "rawMarkdown": "Oh, I think it should be future work, hhh.",
      "votes": null
    },
    {
      "id": "2027644",
      "postDate": "11/13/2022 03:49:43",
      "content": "<p>I see, hhh.</p>",
      "rawMarkdown": "I see, hhh.",
      "votes": null
    },
    {
      "id": "2035545",
      "postDate": "11/19/2022 02:20:50",
      "content": "<p>I tried to address the multiple language notebooks, I used either the same shared codebert model for code and md cells or the codebert for code and sentence-transformers/paraphrase-multilingual-mpnet-base-v2 for md.</p>\n<p>The results (from the local validation, on one of folds):<br>\nshared codebert: All notebooks: 0.9113; English notebooks: 0.9164; Non-English notebooks: 0.8652<br>\ncodebert+mpnet: All notebooks: 0.9088; English notebooks: 0.9117; Non-English notebooks: 0.8825</p>\n<p>The multilingual sentence ensembling mpnet model is trained on language pairs and allows to reduce the score gap between English and non-English notebooks but it has the worse combined score (most notebooks are English).</p>\n<p>I initially planned to do a different ensemble based on the language, but later I found the impact of ensembling codebert2 and codebert+mpnet models produces a much bigger score jump and decided to use the same large ensemble for all notebooks, regardless of the language.</p>",
      "rawMarkdown": "I tried to address the multiple language notebooks, I used either the same shared codebert model for code and md cells or the codebert for code and sentence-transformers/paraphrase-multilingual-mpnet-base-v2 for md.\n\nThe results (from the local validation, on one of folds):\nshared codebert: All notebooks: 0.9113; English notebooks: 0.9164; Non-English notebooks: 0.8652\ncodebert+mpnet: All notebooks: 0.9088; English notebooks: 0.9117; Non-English notebooks: 0.8825\n\nThe multilingual sentence ensembling mpnet model is trained on language pairs and allows to reduce the score gap between English and non-English notebooks but it has the worse combined score (most notebooks are English).\n\nI initially planned to do a different ensemble based on the language, but later I found the impact of ensembling codebert2 and codebert+mpnet models produces a much bigger score jump and decided to use the same large ensemble for all notebooks, regardless of the language.",
      "votes": null
    },
    {
      "id": "2035576",
      "postDate": "11/19/2022 03:06:14",
      "content": "<p>Did you try to use deberta rather than codebert. Deberta is much better in my exp even without MLM. <br>\nOr maybe our approaches are too different.</p>",
      "rawMarkdown": "Did you try to use deberta rather than codebert. Deberta is much better in my exp even without MLM. \nOr maybe our approaches are too different.",
      "votes": null
    },
    {
      "id": "2036606",
      "postDate": "11/20/2022 01:07:56",
      "content": "<p>I only tried the combination of codebert for code plus deberta-v3-large for markdown, it was usually the best performing model but it was 2-3x slower to train and predict. With my final configuration/hyperparameters, the performance difference was not very large so I included a few extra folds of faster models instead. Probably it was worth to have deberta in at least one submission (I kept one submission with a smaller ensemble with more lightweight models, just in case the larger model would face a huge notebook and fail with OOM. I also had fallback model for the larger ensemble).</p>",
      "rawMarkdown": "I only tried the combination of codebert for code plus deberta-v3-large for markdown, it was usually the best performing model but it was 2-3x slower to train and predict. With my final configuration/hyperparameters, the performance difference was not very large so I included a few extra folds of faster models instead. Probably it was worth to have deberta in at least one submission (I kept one submission with a smaller ensemble with more lightweight models, just in case the larger model would face a huge notebook and fail with OOM. I also had fallback model for the larger ensemble).",
      "votes": null
    },
    {
      "id": "2040350",
      "postDate": "11/22/2022 23:45:50",
      "content": "<p>I tried to train with the deberta-v3-base and deberta-v3-large used for both code end markdown, deberta-v3-base performance seems to be similar to codebert but deberta-v3-large is significantly better while being about 4x slower.</p>\n<p>During the early stages of training, the score difference is about 2%, but I expect it to reduce to below 1% (judging from the similar behavior of my previous experiment with deverta-v3-large for md and codebert for code).</p>",
      "rawMarkdown": "I tried to train with the deberta-v3-base and deberta-v3-large used for both code end markdown, deberta-v3-base performance seems to be similar to codebert but deberta-v3-large is significantly better while being about 4x slower.\n\nDuring the early stages of training, the score difference is about 2%, but I expect it to reduce to below 1% (judging from the similar behavior of my previous experiment with deverta-v3-large for md and codebert for code).",
      "votes": null
    },
    {
      "id": "2040366",
      "postDate": "11/23/2022 00:52:11",
      "content": "<p>Nice conclusion! Deberta-v3-base is slightly better than codebert in my exp. And deberta-v3-large is very slow, I have no time to do ensemble.</p>",
      "rawMarkdown": "Nice conclusion! Deberta-v3-base is slightly better than codebert in my exp. And deberta-v3-large is very slow, I have no time to do ensemble.",
      "votes": null
    },
    {
      "id": "2048353",
      "postDate": "11/29/2022 11:29:11",
      "content": "<p>Thank you! Can you explain in more detail <br>\n1) what is your input representation of a notebook (x)? <br>\n2) what is model output (num_classes=1)? <br>\n3) how this output converts to total ordering?<br>\n4) What is your input to deberta during MLM? Is it all cells concatenated (after preproc)?</p>",
      "rawMarkdown": "Thank you! Can you explain in more detail \n1) what is your input representation of a notebook (x)? \n2) what is model output (num_classes=1)? \n3) how this output converts to total ordering?\n4) What is your input to deberta during MLM? Is it all cells concatenated (after preproc)?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1991867,
      "author_name": "gregorlied",
      "author_url": "",
      "post_date": "10/17/2022 12:26:16",
      "content": "<p>Congratulations on your solo gold and your current 1st place position. Good luck to you for the final reruns. <br>\nFirst of all, thank you for your solution write-up. I got two follow-up questions:<br>\n(1) Did you check the model results with and without text preprocessing? Did this step help you a lot?<br>\n(2) Considering that you used deberta-v3-large and the number of days it took you for training:<br>\nWere you able to perform hyperparameter tuning? <br>\nWere you working on a sub-sampled dataset to speed up your experiments?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1991900,
          "author_name": "hydantess",
          "author_url": "",
          "post_date": "10/17/2022 12:43:33",
          "content": "<p>1) It makes text shorter, so we can handle long text. Otherwise 5120 is not enough.<br>\n2) Yes. I only sample 30000 notebooks for fast experiment at the early stage. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1992040,
      "author_name": "vad13irt",
      "author_url": "",
      "post_date": "10/17/2022 14:01:46",
      "content": "<p>Your solution, especially the max length of the input, blew up my mind 😄… I hope, you won't have an OOM error at the end 🙏</p>",
      "votes": null,
      "replies": [
        {
          "id": 1992107,
          "author_name": "hydantess",
          "author_url": "",
          "post_date": "10/17/2022 14:35:28",
          "content": "<p>I tested it before. So it's fine.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1992279,
      "author_name": "kylewu",
      "author_url": "",
      "post_date": "10/17/2022 16:08:50",
      "content": "<p>Congrats! and thank for sharing your solution.<br>\nI have one question here.<br>\nWhat's the output of the model? I am confused because from your code the output dimension is 1, and the previous context indicates the target is more than one dimensions. Or is it list-wise or point-wise model? </p>",
      "votes": null,
      "replies": [
        {
          "id": 1992320,
          "author_name": "hydantess",
          "author_url": "",
          "post_date": "10/17/2022 16:24:53",
          "content": "<p>List-wise. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2048353,
          "author_name": "andreybazhan",
          "author_url": "",
          "post_date": "11/29/2022 11:29:11",
          "content": "<p>Thank you! Can you explain in more detail <br>\n1) what is your input representation of a notebook (x)? <br>\n2) what is model output (num_classes=1)? <br>\n3) how this output converts to total ordering?<br>\n4) What is your input to deberta during MLM? Is it all cells concatenated (after preproc)?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1999565,
      "author_name": "fiemfager",
      "author_url": "",
      "post_date": "10/22/2022 13:24:58",
      "content": "<p>Now you are the top of the leaderboard. Great. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2001636,
      "author_name": "infiniteemo",
      "author_url": "",
      "post_date": "10/24/2022 07:43:34",
      "content": "<p>Congrats!<br>\nCan you explain your listwise method more detailly? Or just throw me a document or website?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2001875,
      "author_name": "css919",
      "author_url": "",
      "post_date": "10/24/2022 11:08:06",
      "content": "<p>Are you using percentile ranks or just the absolute ranks as target?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2001970,
          "author_name": "hydantess",
          "author_url": "",
          "post_date": "10/24/2022 12:36:12",
          "content": "<p>percentile ranks. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2002162,
          "author_name": "css919",
          "author_url": "",
          "post_date": "10/24/2022 14:51:14",
          "content": "<p>Thanks! Another question: <br>\nThe texts are \"flag + cell_1 + flag + cell_2 + … + flag + cell_n\". Does each cell have a # tokens quota? Did you truncate certain cells if it is too long? Or did you select shorter cells only such that 128 cells just need 2048 tokens? </p>\n<p>Perhaps a better question should be: How did you make sure 2048 tokens can include 128 cells?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2002185,
          "author_name": "hydantess",
          "author_url": "",
          "post_date": "10/24/2022 15:15:50",
          "content": "<ol>\n<li>Cells are truncated and length is smaller than 128.</li>\n<li>Each cell's length starts from 1 and for-looper all cells, step = 4, then break when total length is greater than certain value.</li>\n</ol>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2002379,
          "author_name": "css919",
          "author_url": "",
          "post_date": "10/24/2022 18:44:32",
          "content": "<p>When training, in the worst case (128 cells), max_length of a cell is just 2048/128 = 16?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2002612,
          "author_name": "hydantess",
          "author_url": "",
          "post_date": "10/25/2022 00:27:16",
          "content": "<p>Yes, you are right.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2026340,
      "author_name": "dmytropoplavskiy",
      "author_url": "",
      "post_date": "11/12/2022 00:58:48",
      "content": "<p>Congratulations on winning the challenge!</p>\n<p>An impressive result, especially for the single model without ensembling.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2026398,
          "author_name": "hydantess",
          "author_url": "",
          "post_date": "11/12/2022 03:19:04",
          "content": "<p>Thanks. Your poly-encoder like approach is also very impressive. I tried translating and using xlm-roberta model but failed!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2026759,
      "author_name": "jarfo1",
      "author_url": "",
      "post_date": "11/12/2022 10:22:12",
      "content": "<p>Congratulations to <a href=\"https://www.kaggle.com/hydantess\" target=\"_blank\">@hydantess</a> and all the prize winners!<br>\nYou are the geniuses!!!</p>",
      "votes": null,
      "replies": [
        {
          "id": 2026857,
          "author_name": "hydantess",
          "author_url": "",
          "post_date": "11/12/2022 11:38:01",
          "content": "<p>You are the genius too! I am very happy competing with you~</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2027101,
      "author_name": "zzy990106",
      "author_url": "",
      "post_date": "11/12/2022 14:51:55",
      "content": "<p>Congrats on entering top10! How do you deal with multiple languages, with deberta models?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2027120,
          "author_name": "hydantess",
          "author_url": "",
          "post_date": "11/12/2022 15:07:51",
          "content": "<p>I didn't deal with multi languages situation, so just list it in feature work.<br>\nI think Dmytro Poplavskiy's poly-encoder like approach may help.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2027642,
          "author_name": "zzy990106",
          "author_url": "",
          "post_date": "11/13/2022 03:45:56",
          "content": "<p>Oh, I think it should be future work, hhh.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2027644,
          "author_name": "hydantess",
          "author_url": "",
          "post_date": "11/13/2022 03:49:43",
          "content": "<p>I see, hhh.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2035545,
          "author_name": "dmytropoplavskiy",
          "author_url": "",
          "post_date": "11/19/2022 02:20:50",
          "content": "<p>I tried to address the multiple language notebooks, I used either the same shared codebert model for code and md cells or the codebert for code and sentence-transformers/paraphrase-multilingual-mpnet-base-v2 for md.</p>\n<p>The results (from the local validation, on one of folds):<br>\nshared codebert: All notebooks: 0.9113; English notebooks: 0.9164; Non-English notebooks: 0.8652<br>\ncodebert+mpnet: All notebooks: 0.9088; English notebooks: 0.9117; Non-English notebooks: 0.8825</p>\n<p>The multilingual sentence ensembling mpnet model is trained on language pairs and allows to reduce the score gap between English and non-English notebooks but it has the worse combined score (most notebooks are English).</p>\n<p>I initially planned to do a different ensemble based on the language, but later I found the impact of ensembling codebert2 and codebert+mpnet models produces a much bigger score jump and decided to use the same large ensemble for all notebooks, regardless of the language.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2035576,
          "author_name": "hydantess",
          "author_url": "",
          "post_date": "11/19/2022 03:06:14",
          "content": "<p>Did you try to use deberta rather than codebert. Deberta is much better in my exp even without MLM. <br>\nOr maybe our approaches are too different.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2036606,
          "author_name": "dmytropoplavskiy",
          "author_url": "",
          "post_date": "11/20/2022 01:07:56",
          "content": "<p>I only tried the combination of codebert for code plus deberta-v3-large for markdown, it was usually the best performing model but it was 2-3x slower to train and predict. With my final configuration/hyperparameters, the performance difference was not very large so I included a few extra folds of faster models instead. Probably it was worth to have deberta in at least one submission (I kept one submission with a smaller ensemble with more lightweight models, just in case the larger model would face a huge notebook and fail with OOM. I also had fallback model for the larger ensemble).</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2040350,
          "author_name": "dmytropoplavskiy",
          "author_url": "",
          "post_date": "11/22/2022 23:45:50",
          "content": "<p>I tried to train with the deberta-v3-base and deberta-v3-large used for both code end markdown, deberta-v3-base performance seems to be similar to codebert but deberta-v3-large is significantly better while being about 4x slower.</p>\n<p>During the early stages of training, the score difference is about 2%, but I expect it to reduce to below 1% (judging from the similar behavior of my previous experiment with deverta-v3-large for md and codebert for code).</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2040366,
          "author_name": "hydantess",
          "author_url": "",
          "post_date": "11/23/2022 00:52:11",
          "content": "<p>Nice conclusion! Deberta-v3-base is slightly better than codebert in my exp. And deberta-v3-large is very slow, I have no time to do ensemble.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1991325": "**Overview**\n\nI used single list-wise deberta-v3-large trained on all competition data. Input text is cells separated with [SEP] and [CLS] tokens.\n\n\n**Code**\n- training code: https://github.com/hydantess/2022-Google-AI4Code-1st-Place-Solution\n- inference notebook: https://www.kaggle.com/code/hydantess/ai4code0729?scriptVersionId=103060429\n\n**What Worked**\n- MLM\n- preprocessing\n   - Reasoning and context: It makes text shorter, so we can handle longer text. \n- use deberta rather than codebert\n- +lstm head\n- bigger cell_cnt and seq_length\n   - Reasoning and context: metric function cares more about long text. \n- +postprocessing: predict code and markdown both, then sort code by ground truth.\n   - Reasoning and context: more reasonable than only predict markdown, and concat with code cells. \n\n**What Didn’t Work**\n- translation: facebook/mbart-large-50-many-to-one-mmt\n   - Reasoning and context: most notebooks are in english. Or I did something wrong.\n- +code rank to dense layers\n   - Reasoning and context:  I should try to concat text with rank information. Model will learn better.\n- extract embedding, then train a transformer model\n - Reasoning and context: I should pre-train the extracting model first.\n- xlm-robert-large or mdeberta-v3-base\n   - Reasoning and context: much worse than deberta-v3-large. so can not ensemble directly.\n\n\n**Important Citations**\n- https://www.kaggle.com/code/ilyaryabov/fastttext-sorting-with-cosine-distance-algo by ilyaryabov\n- https://www.kaggle.com/competitions/AI4Code/discussion/327626#1804178 by Nguyentuananh\n\n**Additional Context**\n- loss is MAE.\n- MLM on training data for 15 epochs,  max_length is 1024. It takes 3 days. \n- training for 10 epochs, max_length is 2048. It takes 7 days. I think set max_length to 5120 will greatly improve model performance but I have not enough A100~\n- inference: max_length is 5120. It takes 6 hours.\n-  I used 1 * A100(80GB)  for the last two weeks and 1 * 3090(24GB) at the early stage. \n-  Future work:\n    - concat text with rank information: flag + rank_1 + cell_1 + flag+ rank_2 + cell_2 + … + flag+ rank_n + cell_n\n    - multi languages\n    - extra training data\n    - Adversarial Weight Perturbations (AWP), but too time-consuming\n\n**Thanks and Acknowledgements:**\n\nThanks to competition organizers for hosting such an amazing NLP competition. And thanks to the\ngreat notebooks and discussions, I learned a lot. \n\n**Team Members:**\n- @hydantess \n .",
    "1991867": "Congratulations on your solo gold and your current 1st place position. Good luck to you for the final reruns. \nFirst of all, thank you for your solution write-up. I got two follow-up questions:\n(1) Did you check the model results with and without text preprocessing? Did this step help you a lot?\n(2) Considering that you used deberta-v3-large and the number of days it took you for training:\nWere you able to perform hyperparameter tuning? \nWere you working on a sub-sampled dataset to speed up your experiments?",
    "1991900": "1) It makes text shorter, so we can handle long text. Otherwise 5120 is not enough.\n2) Yes. I only sample 30000 notebooks for fast experiment at the early stage.",
    "1992040": "Your solution, especially the max length of the input, blew up my mind 😄... I hope, you won't have an OOM error at the end 🙏",
    "1992107": "I tested it before. So it's fine.",
    "1992279": "Congrats! and thank for sharing your solution.\nI have one question here.\nWhat's the output of the model? I am confused because from your code the output dimension is 1, and the previous context indicates the target is more than one dimensions. Or is it list-wise or point-wise model?",
    "1992320": "List-wise.",
    "1999565": "Now you are the top of the leaderboard. Great.",
    "2001636": "Congrats!\nCan you explain your listwise method more detailly? Or just throw me a document or website?",
    "2001875": "Are you using percentile ranks or just the absolute ranks as target?",
    "2001970": "percentile ranks.",
    "2002162": "Thanks! Another question: \nThe texts are \"flag + cell_1 + flag + cell_2 + … + flag + cell_n\". Does each cell have a # tokens quota? Did you truncate certain cells if it is too long? Or did you select shorter cells only such that 128 cells just need 2048 tokens? \n\nPerhaps a better question should be: How did you make sure 2048 tokens can include 128 cells?",
    "2002185": "1. Cells are truncated and length is smaller than 128.\n2. Each cell's length starts from 1 and for-looper all cells, step = 4, then break when total length is greater than certain value.",
    "2002379": "When training, in the worst case (128 cells), max_length of a cell is just 2048/128 = 16?",
    "2002612": "Yes, you are right.",
    "2026340": "Congratulations on winning the challenge!\n\nAn impressive result, especially for the single model without ensembling.",
    "2026398": "Thanks. Your poly-encoder like approach is also very impressive. I tried translating and using xlm-roberta model but failed!",
    "2026759": "Congratulations to @hydantess and all the prize winners!\nYou are the geniuses!!!",
    "2026857": "You are the genius too! I am very happy competing with you~",
    "2027101": "Congrats on entering top10! How do you deal with multiple languages, with deberta models?",
    "2027120": "I didn't deal with multi languages situation, so just list it in feature work.\nI think Dmytro Poplavskiy's poly-encoder like approach may help.",
    "2027642": "Oh, I think it should be future work, hhh.",
    "2027644": "I see, hhh.",
    "2035545": "I tried to address the multiple language notebooks, I used either the same shared codebert model for code and md cells or the codebert for code and sentence-transformers/paraphrase-multilingual-mpnet-base-v2 for md.\n\nThe results (from the local validation, on one of folds):\nshared codebert: All notebooks: 0.9113; English notebooks: 0.9164; Non-English notebooks: 0.8652\ncodebert+mpnet: All notebooks: 0.9088; English notebooks: 0.9117; Non-English notebooks: 0.8825\n\nThe multilingual sentence ensembling mpnet model is trained on language pairs and allows to reduce the score gap between English and non-English notebooks but it has the worse combined score (most notebooks are English).\n\nI initially planned to do a different ensemble based on the language, but later I found the impact of ensembling codebert2 and codebert+mpnet models produces a much bigger score jump and decided to use the same large ensemble for all notebooks, regardless of the language.",
    "2035576": "Did you try to use deberta rather than codebert. Deberta is much better in my exp even without MLM. \nOr maybe our approaches are too different.",
    "2036606": "I only tried the combination of codebert for code plus deberta-v3-large for markdown, it was usually the best performing model but it was 2-3x slower to train and predict. With my final configuration/hyperparameters, the performance difference was not very large so I included a few extra folds of faster models instead. Probably it was worth to have deberta in at least one submission (I kept one submission with a smaller ensemble with more lightweight models, just in case the larger model would face a huge notebook and fail with OOM. I also had fallback model for the larger ensemble).",
    "2040350": "I tried to train with the deberta-v3-base and deberta-v3-large used for both code end markdown, deberta-v3-base performance seems to be similar to codebert but deberta-v3-large is significantly better while being about 4x slower.\n\nDuring the early stages of training, the score difference is about 2%, but I expect it to reduce to below 1% (judging from the similar behavior of my previous experiment with deverta-v3-large for md and codebert for code).",
    "2040366": "Nice conclusion! Deberta-v3-base is slightly better than codebert in my exp. And deberta-v3-large is very slow, I have no time to do ensemble.",
    "2048353": "Thank you! Can you explain in more detail \n1) what is your input representation of a notebook (x)? \n2) what is model output (num_classes=1)? \n3) how this output converts to total ordering?\n4) What is your input to deberta during MLM? Is it all cells concatenated (after preproc)?"
  },
  "source": "existing_discussion",
  "complete": true,
  "competition_id": 35887,
  "competition_slug": "AI4Code",
  "team_id": 8610635,
  "solution_url": "/competitions/AI4Code/writeups/hyd-1st-place-solution"
}