{ "id": 343595, "title": "4th solution", "url": "/competitions/AI4Code/writeups/gezi-4th-solution", "author_name": "", "post_date": "2022-11-21T12:25:24.357Z", "votes": 43, "comment_count": 4, "views": 0, "content": "

Recall model (infer 2h+, 1h using 2 T4)

\n\n

Pairwise rank model (infer 2h+)

\n\n

Context rank model (infer 2.5h+)
\nModified from https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells

\n\n

GBDT model to combine above 3 models score (infer 5m）

\n\n

Just learned deberta-v3-large with large inputs can be much better , more context(more codes like 128+) is a key for better result,^_^, next time will do more experiments at first to choose the correct direction.
\nI did not use external data finally, though I tried but not with much improvement, the direction is wrong I should have tried to use more kaggle notebooks, I found so much duplicates in the public kaggle dataset published so not try it also.
\nThanks kaggle , this is really an intersting game! Also want to thanks for wandb and tabnine, they make life easier for doing experiments and coding.

\n

code
\ninference to get 9170 for 4th place: https://www.kaggle.com/code/goldenlock/reproduce-v1
\npairwise inference to get 8936: https://www.kaggle.com/code/goldenlock/pairwise-recall
\ntraining: https://www.kaggle.com/datasets/goldenlock/ai4code
\nhttps://github.com/chenghuige/Google-AI4Code-Understand-Code-in-Python-Notebooks

", "messages": [ { "id": "1895078", "postDate": "08/12/2022 00:27:54", "content": "

Recall model (infer 2h+, 1h using 2 T4)

\n\n

Pairwise rank model (infer 2h+)

\n\n

Context rank model (infer 2.5h+)
\nModified from https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells

\n\n

GBDT model to combine above 3 models score (infer 5m）

\n\n

Just learned deberta-v3-large with large inputs can be much better , more context(more codes like 128+) is a key for better result,^_^, next time will do more experiments at first to choose the correct direction.
\nI did not use external data finally, though I tried but not with much improvement, the direction is wrong I should have tried to use more kaggle notebooks, I found so much duplicates in the public kaggle dataset published so not try it also.
\nThanks kaggle , this is really an intersting game! Also want to thanks for wandb and tabnine, they make life easier for doing experiments and coding.

\n

code
\ninference to get 9170 for 4th place: https://www.kaggle.com/code/goldenlock/reproduce-v1
\npairwise inference to get 8936: https://www.kaggle.com/code/goldenlock/pairwise-recall
\ntraining: https://www.kaggle.com/datasets/goldenlock/ai4code
\nhttps://github.com/chenghuige/Google-AI4Code-Understand-Code-in-Python-Notebooks

", "rawMarkdown": "**Recall model** (infer 2h+, 1h using 2 T4)\n- pairwise, two tower, with backbone sentence-transformers/all-mpnet-base-v2\n- continued mlm on training data\n- predicting for earch markdown wether a code is just after it (also a fake last code with content 'the end of notebook' added) \n- 9 negative selected withn the same notebook (better comparing to using 4 both offline and online)\n- for code, encode itsself and before 2 and after 2 codes \n- **crossentropy loss with cosine / learned temperature**(finally about 0.002) as input logits\n- offline tau score 897 (averge the recall score for each code position)\n- late submission score is 8936\n- **recall model is a key model, as rank model and context rank model all using its output**\n- The main drawback here is the recall model if using 9 negatives will be gpu costly, I used 8*v100, and it cost about 30 h for 1 model training on all train data. \n\n**Pairwise rank model** (infer 2h+)\n- pairwsie , concat , with backbone deberta-v3-small \n- only infer for those recall model max prob < 0.8, so infer(rerank) about 50% markdowns.\n- offline tau score 905\n\n**Context rank model** (infer 2.5h+)\nModified from https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells \n- backbone deberta-v3-small\n- **3 markdowns as a group (infer at once for each instance)** \nthese groups slected using recall model, to select markdowns with similar code scores to a group.\n- **40 codes selected by recall model (train using recall model OOF**) \n this is also a key point, for training using neg sample with sample prob provided by recall model, for infer just choose top recall prob codes. And after contest finish I learned that most top teams used much more codes, more context/codes is a key point for win.\nTrain just using spaced selection of codes with some distoration (randomness) and then infer with recall model selected top codes can improve 0.1+. I have tested 1 context model before online from 8806(without recall model) to 8892(using recall model)\nThen if train using recall model selected, or with probablity of sampling will be even better 0.05+.\n- add OOF recall model score as inputs to bert improve 0.004 \n not sure if overfit, so I selected 1 sub with it 9147 and another witout it 9143, I made some mistakes and no time to retrain, just finetune 3 epochs for models with OOF score, so this OOF method should have improved much more I think should be 2k+, you could see below huge improvement from just the first epoch . \nFinally we see improment from 9162 to 9170.\n![](https://i.postimg.cc/TYZ5hH0S/Qd-AG4-Bsuv-FVh-C5-M.png)\n- multiple losses( classification for num_classes 100 + MAE for regression to preidct the relative rank for each markdown)\n- offline tau score 916(interesting, this is the classification output） and tau score 914(regression output)\n\n**GBDT model to combine above 3 models score** (infer 5m） \n- used catboost and lgb, it helps improve about 6k online.\n- offline tau score 922, online 9147\ntop features show below: \nxpc means pairwise concat rank model \nxc means context rank model \nxp means pairwise two tower model (recall model) \n![](https://i.postimg.cc/kMYWbRSt/2.png)\n\nJust learned deberta-v3-large with large inputs can be much better , more context(more codes like 128+) is a key for better result,^_^, next time will do more experiments at first to choose the correct direction. \nI did not use external data finally, though I tried but not with much improvement, the direction is wrong I should have tried to use more kaggle notebooks, I found so much duplicates in the public kaggle dataset published so not try it also.\nThanks kaggle , this is really an intersting game! Also want to thanks for wandb and tabnine, they make life easier for doing experiments and coding.\n\n**code**\ninference to get 9170 for 4th place: https://www.kaggle.com/code/goldenlock/reproduce-v1\npairwise inference to get 8936: https://www.kaggle.com/code/goldenlock/pairwise-recall\ntraining: https://www.kaggle.com/datasets/goldenlock/ai4code \nhttps://github.com/chenghuige/Google-AI4Code-Understand-Code-in-Python-Notebooks", "votes": null }, { "id": "1895082", "postDate": "08/12/2022 00:31:41", "content": "

Congrats on the public 8th!
\nI'll ask clarification questions after I've gotten some sleep :D

", "rawMarkdown": "Congrats on the public 8th! \nI'll ask clarification questions after I've gotten some sleep :D", "votes": null }, { "id": "1896501", "postDate": "08/12/2022 21:14:01", "content": "

@goldenlock Thanks for the writeup! I hope you could answer a few questions.

\n

Recall model:

\n
    \n
  1. For continued pretraining, did you use regular MLM or MPNet?
    2. \n
  2. Does the model learn separate weights for code and markdown?
    3. \n
  3. How did you combine the encoding of a code cell with the encodings of it's neighboring code cells to create the final cell code representation?
    4. \n
\n

Ensembling:
\nCould you explain more about the features you used there? e.g. abs_cls2_reg_diff, var_sim, etc.

", "rawMarkdown": "goldenlock Thanks for the writeup! I hope you could answer a few questions.\n\nRecall model:\n1. For continued pretraining, did you use regular MLM or MPNet?\n2. Does the model learn separate weights for code and markdown?\n3. How did you combine the encoding of a code cell with the encodings of it's neighboring code cells to create the final cell code representation?\n\nEnsembling:\nCould you explain more about the features you used there? e.g. abs_cls2_reg_diff, var_sim, etc.", "votes": null }, { "id": "1896544", "postDate": "08/12/2022 22:39:46", "content": "

@cabbage972

\n
    \n
  1. regular mlm on mpnet
    2. \n
  2. Learn exactly the same weights for code and markdown, only difference is the inputs for bert
    3. \n
  3. Like this
    \ncode_cell-2[SEP]code_cell-1[SEP]int(code_cell_pct_rank * 1000)[SEP]Num_cells[SEP]Num_code_cells[SEP]current_code_cell[SEP]code_cell+1[SEP]code_cell+2
    \nif like no code_cell+1 then leave it as 'after' and if no code_cell-1 leave it as 'before'
    4. \n
  4. The ensembling mainly combine the outputs score of the 3 models.
    \nFor recall model, pred means the code cell which is neareast to the markdown
    \nFor recall model, cls pred means the sum of each code_cell_position * softmax_prob, cls pred tau score is higher then pred tau score
    \nsim means cosine sim of code and markdown, so var_sim means np.var(all the codes cosine sims for specific markdown)
    \nFor pairwise concat model, pred and cls pred means the same as recall model
    \nFor context renak model, as I used classification (100 classes) and regression.
    \nSo pred, and reg_pred both means regression prediction. as in the figure listed xc_reg_pred.
    \ncls_pred means classification prediction (sum of each 100 class position * softmax_prob)
    \ncls2_pred means the classification prediction of postion class with the argmax softmax_prob
    \nSo abs_cls2_reg_diff means for context model abs(cls2_pred - reg_pred)
    5. \n
", "rawMarkdown": "cabbage972 \n1. regular mlm on mpnet\n2. Learn exactly the same weights for code and markdown, only difference is the inputs for bert \n3. Like this \ncode_cell-2[SEP]code_cell-1[SEP]int(code_cell_pct_rank * 1000)[SEP]Num_cells[SEP]Num_code_cells[SEP]current_code_cell[SEP]code_cell+1[SEP]code_cell+2\nif like no code_cell+1 then leave it as 'after' and if no code_cell-1 leave it as 'before'\n4. The ensembling mainly combine the outputs score of the 3 models. \nFor recall model, pred means the code cell which is neareast to the markdown \nFor recall model, cls pred means the sum of each code_cell_position * softmax_prob, cls pred tau score is higher then pred tau score\nsim means cosine sim of code and markdown, so var_sim means np.var(all the codes cosine sims for specific markdown)\nFor pairwise concat model, pred and cls pred means the same as recall model\nFor context renak model, as I used classification (100 classes) and regression.\nSo pred, and reg_pred both means regression prediction. as in the figure listed xc_reg_pred.\ncls_pred means classification prediction (sum of each 100 class position * softmax_prob)\ncls2_pred means the classification prediction of postion class with the argmax softmax_prob\nSo abs_cls2_reg_diff means for context model abs(cls2_pred - reg_pred)", "votes": null }, { "id": "1896586", "postDate": "08/13/2022 00:04:50", "content": "

Thank you for the detailed answers!

", "rawMarkdown": "Thank you for the detailed answers!", "votes": null } ], "comments": [ { "id": 1895082, "author_name": "init27", "author_url": "", "post_date": "08/12/2022 00:31:41", "content": "

Congrats on the public 8th!
\nI'll ask clarification questions after I've gotten some sleep :D

", "votes": null, "replies": [] }, { "id": 1896501, "author_name": "cabbage972", "author_url": "", "post_date": "08/12/2022 21:14:01", "content": "

@goldenlock Thanks for the writeup! I hope you could answer a few questions.

\n

Recall model:

\n
    \n
  1. For continued pretraining, did you use regular MLM or MPNet?
    2. \n
  2. Does the model learn separate weights for code and markdown?
    3. \n
  3. How did you combine the encoding of a code cell with the encodings of it's neighboring code cells to create the final cell code representation?
    4. \n
\n

Ensembling:
\nCould you explain more about the features you used there? e.g. abs_cls2_reg_diff, var_sim, etc.

", "votes": null, "replies": [ { "id": 1896544, "author_name": "goldenlock", "author_url": "", "post_date": "08/12/2022 22:39:46", "content": "

@cabbage972

\n
    \n
  1. regular mlm on mpnet
    2. \n
  2. Learn exactly the same weights for code and markdown, only difference is the inputs for bert
    3. \n
  3. Like this
    \ncode_cell-2[SEP]code_cell-1[SEP]int(code_cell_pct_rank * 1000)[SEP]Num_cells[SEP]Num_code_cells[SEP]current_code_cell[SEP]code_cell+1[SEP]code_cell+2
    \nif like no code_cell+1 then leave it as 'after' and if no code_cell-1 leave it as 'before'
    4. \n
  4. The ensembling mainly combine the outputs score of the 3 models.
    \nFor recall model, pred means the code cell which is neareast to the markdown
    \nFor recall model, cls pred means the sum of each code_cell_position * softmax_prob, cls pred tau score is higher then pred tau score
    \nsim means cosine sim of code and markdown, so var_sim means np.var(all the codes cosine sims for specific markdown)
    \nFor pairwise concat model, pred and cls pred means the same as recall model
    \nFor context renak model, as I used classification (100 classes) and regression.
    \nSo pred, and reg_pred both means regression prediction. as in the figure listed xc_reg_pred.
    \ncls_pred means classification prediction (sum of each 100 class position * softmax_prob)
    \ncls2_pred means the classification prediction of postion class with the argmax softmax_prob
    \nSo abs_cls2_reg_diff means for context model abs(cls2_pred - reg_pred)
    5. \n
", "votes": null, "replies": [] }, { "id": 1896586, "author_name": "cabbage972", "author_url": "", "post_date": "08/13/2022 00:04:50", "content": "

Thank you for the detailed answers!

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1895078": "**Recall model** (infer 2h+, 1h using 2 T4)\n- pairwise, two tower, with backbone sentence-transformers/all-mpnet-base-v2\n- continued mlm on training data\n- predicting for earch markdown wether a code is just after it (also a fake last code with content 'the end of notebook' added) \n- 9 negative selected withn the same notebook (better comparing to using 4 both offline and online)\n- for code, encode itsself and before 2 and after 2 codes \n- **crossentropy loss with cosine / learned temperature**(finally about 0.002) as input logits\n- offline tau score 897 (averge the recall score for each code position)\n- late submission score is 8936\n- **recall model is a key model, as rank model and context rank model all using its output**\n- The main drawback here is the recall model if using 9 negatives will be gpu costly, I used 8*v100, and it cost about 30 h for 1 model training on all train data. \n\n**Pairwise rank model** (infer 2h+)\n- pairwsie , concat , with backbone deberta-v3-small \n- only infer for those recall model max prob < 0.8, so infer(rerank) about 50% markdowns.\n- offline tau score 905\n\n**Context rank model** (infer 2.5h+)\nModified from https://www.kaggle.com/code/suicaokhoailang/stronger-baseline-with-code-cells \n- backbone deberta-v3-small\n- **3 markdowns as a group (infer at once for each instance)** \nthese groups slected using recall model, to select markdowns with similar code scores to a group.\n- **40 codes selected by recall model (train using recall model OOF**) \n this is also a key point, for training using neg sample with sample prob provided by recall model, for infer just choose top recall prob codes. And after contest finish I learned that most top teams used much more codes, more context/codes is a key point for win.\nTrain just using spaced selection of codes with some distoration (randomness) and then infer with recall model selected top codes can improve 0.1+. I have tested 1 context model before online from 8806(without recall model) to 8892(using recall model)\nThen if train using recall model selected, or with probablity of sampling will be even better 0.05+.\n- add OOF recall model score as inputs to bert improve 0.004 \n not sure if overfit, so I selected 1 sub with it 9147 and another witout it 9143, I made some mistakes and no time to retrain, just finetune 3 epochs for models with OOF score, so this OOF method should have improved much more I think should be 2k+, you could see below huge improvement from just the first epoch . \nFinally we see improment from 9162 to 9170.\n![](https://i.postimg.cc/TYZ5hH0S/Qd-AG4-Bsuv-FVh-C5-M.png)\n- multiple losses( classification for num_classes 100 + MAE for regression to preidct the relative rank for each markdown)\n- offline tau score 916(interesting, this is the classification output） and tau score 914(regression output)\n\n**GBDT model to combine above 3 models score** (infer 5m） \n- used catboost and lgb, it helps improve about 6k online.\n- offline tau score 922, online 9147\ntop features show below: \nxpc means pairwise concat rank model \nxc means context rank model \nxp means pairwise two tower model (recall model) \n![](https://i.postimg.cc/kMYWbRSt/2.png)\n\nJust learned deberta-v3-large with large inputs can be much better , more context(more codes like 128+) is a key for better result,^_^, next time will do more experiments at first to choose the correct direction. \nI did not use external data finally, though I tried but not with much improvement, the direction is wrong I should have tried to use more kaggle notebooks, I found so much duplicates in the public kaggle dataset published so not try it also.\nThanks kaggle , this is really an intersting game! Also want to thanks for wandb and tabnine, they make life easier for doing experiments and coding.\n\n**code**\ninference to get 9170 for 4th place: https://www.kaggle.com/code/goldenlock/reproduce-v1\npairwise inference to get 8936: https://www.kaggle.com/code/goldenlock/pairwise-recall\ntraining: https://www.kaggle.com/datasets/goldenlock/ai4code \nhttps://github.com/chenghuige/Google-AI4Code-Understand-Code-in-Python-Notebooks", "1895082": "Congrats on the public 8th! \nI'll ask clarification questions after I've gotten some sleep :D", "1896501": "goldenlock Thanks for the writeup! I hope you could answer a few questions.\n\nRecall model:\n1. For continued pretraining, did you use regular MLM or MPNet?\n2. Does the model learn separate weights for code and markdown?\n3. How did you combine the encoding of a code cell with the encodings of it's neighboring code cells to create the final cell code representation?\n\nEnsembling:\nCould you explain more about the features you used there? e.g. abs_cls2_reg_diff, var_sim, etc.", "1896544": "cabbage972 \n1. regular mlm on mpnet\n2. Learn exactly the same weights for code and markdown, only difference is the inputs for bert \n3. Like this \ncode_cell-2[SEP]code_cell-1[SEP]int(code_cell_pct_rank * 1000)[SEP]Num_cells[SEP]Num_code_cells[SEP]current_code_cell[SEP]code_cell+1[SEP]code_cell+2\nif like no code_cell+1 then leave it as 'after' and if no code_cell-1 leave it as 'before'\n4. The ensembling mainly combine the outputs score of the 3 models. \nFor recall model, pred means the code cell which is neareast to the markdown \nFor recall model, cls pred means the sum of each code_cell_position * softmax_prob, cls pred tau score is higher then pred tau score\nsim means cosine sim of code and markdown, so var_sim means np.var(all the codes cosine sims for specific markdown)\nFor pairwise concat model, pred and cls pred means the same as recall model\nFor context renak model, as I used classification (100 classes) and regression.\nSo pred, and reg_pred both means regression prediction. as in the figure listed xc_reg_pred.\ncls_pred means classification prediction (sum of each 100 class position * softmax_prob)\ncls2_pred means the classification prediction of postion class with the argmax softmax_prob\nSo abs_cls2_reg_diff means for context model abs(cls2_pred - reg_pred)", "1896586": "Thank you for the detailed answers!" }, "source": "meta" }