![\"\"](\"https://i.ibb.co/pRgRJ8Y/Selection-898.png\")
Hi guys,
\nI just finished going through all the winning solutions of 4 different code competitions (not all from Kaggle).
Here are all the tricks they used.
\nEnjoy yourself!
\n\nClassification: Line by line, some lines have bugs
\n
2 Stage Pipeline
\n
Super simple pipeline
\nTFIDF (ngrams=(1,2))
LogisticRegression
Blended with the rest of the team
Models
\nInput & output
\nTraining
\nPost-processing
\nInjected Special Token
\nMany Models Ensemble
\n\n\nNote: always select a single fold.
\n
\n\nTask: Translate english to bash syntax
\n
Transformer + Architecture Improvements
\nCustom Tokenizer
\nFound by hyperparam search that they can shrink the vocabulary size by 90% and it even improves the results
Heuristic improved beam search
\n\n\nwhy didnt they just summarize thisssssss???? @#$@#%
\n
Augmentations
\nBacktranslated (only the english prompt) using fairseq pretrained models, used only predictions with BLEU higher then a threshold (to avoid training their model with the mistakes of the fairseq model)
\nData Generation
\nGenerated more data from the manpage: Commands were created by concatenating cmd name & some random flags,
\nChanged some common pipline endings that were \"too easy\" such as \"wc\" & \"how many\"
Preprocessing
\nPostprocessing
\n\n\nAgain, reversing their code
\n
Fairseq Model
\nTwo models
\nTree RNN Tensorflow Model
\nRight here
They also have many variations in the code with multiple heads and so on. They probably didn't delete the past experiments they tried.
\n\nTask: predict formatting errors in source code
\n
Preprocessing
\nManually defined 22 regex expressions for automaticlly detect common source code errors.
Used the regex to clean manually a high quality dataset by hand then trained a generative language model only on those samples.
[this is a really cool idea..]
\nAdded common meaningful words to the tokenizer such as \"True\" \"False\" \"if\"..
Optimized n-grams size (7 was optimal) so the distribution of tokens for \"broken/not broken\" would by as close as possible to 50%/50% then they used these token to train a balanced XGBoost classifier and an 1-class SVM for outlier detection.
Ensemble: prediction char by char (mean) with all models, then activate the regex on the \"most problematic\" parts of the text as post processing.
\n\nTask: replace lines in a given code: On a pair (code line, code file) predict for each line in \"code file\" the probability so that this is the line that needs to be replaced by \"code line\" [sort of \"similarity\" to code line]
\n
Treated as a search/ranking
\nPost-processing
\nHere's a table that summarizes our results:
\n\n\nThis was a bit messy to follow since they didn't have a full writeup. This is what I managed to extract from conversation over the pull request.
\n
Had simple string similarity version that turned out to be a really strong baseline
Hand crafted features for a tree based model: High-level AST of each line (basically the type of each code element, VARIABLE, CLASS, STRING, COMMENT, NUMBER, TOKEN, KEYWORD)
Cool Trick One idea: because code tends to be added to the end of the file, and new code tend to be more buggy than old code.
LambdaMART using RankLib
Used standard text similarity metrics
\n\nNo writeup, reversed the code
\n
Used:
\nstring matching
parse checking
decision rules
heuristics
Hi guys,
\nI just finished going through all the winning solutions of 4 different code competitions (not all from Kaggle).
Here are all the tricks they used.
\nEnjoy yourself!
\n\nClassification: Line by line, some lines have bugs
\n
2 Stage Pipeline
\n
Super simple pipeline
\nTFIDF (ngrams=(1,2))
LogisticRegression
Blended with the rest of the team
Models
\nInput & output
\nTraining
\nPost-processing
\nInjected Special Token
\nMany Models Ensemble
\n\n\nNote: always select a single fold.
\n
\n\nTask: Translate english to bash syntax
\n
Transformer + Architecture Improvements
\nCustom Tokenizer
\nFound by hyperparam search that they can shrink the vocabulary size by 90% and it even improves the results
Heuristic improved beam search
\n\n\nwhy didnt they just summarize thisssssss???? @#$@#%
\n
Augmentations
\nBacktranslated (only the english prompt) using fairseq pretrained models, used only predictions with BLEU higher then a threshold (to avoid training their model with the mistakes of the fairseq model)
\nData Generation
\nGenerated more data from the manpage: Commands were created by concatenating cmd name & some random flags,
\nChanged some common pipline endings that were \"too easy\" such as \"wc\" & \"how many\"
Preprocessing
\nPostprocessing
\n\n\nAgain, reversing their code
\n
Fairseq Model
\nTwo models
\nTree RNN Tensorflow Model
\nRight here
They also have many variations in the code with multiple heads and so on. They probably didn't delete the past experiments they tried.
\n\nTask: predict formatting errors in source code
\n
Preprocessing
\nManually defined 22 regex expressions for automaticlly detect common source code errors.
Used the regex to clean manually a high quality dataset by hand then trained a generative language model only on those samples.
[this is a really cool idea..]
\nAdded common meaningful words to the tokenizer such as \"True\" \"False\" \"if\"..
Optimized n-grams size (7 was optimal) so the distribution of tokens for \"broken/not broken\" would by as close as possible to 50%/50% then they used these token to train a balanced XGBoost classifier and an 1-class SVM for outlier detection.
Ensemble: prediction char by char (mean) with all models, then activate the regex on the \"most problematic\" parts of the text as post processing.
\n\nTask: replace lines in a given code: On a pair (code line, code file) predict for each line in \"code file\" the probability so that this is the line that needs to be replaced by \"code line\" [sort of \"similarity\" to code line]
\n
Treated as a search/ranking
\nPost-processing
\nHere's a table that summarizes our results:
\n\n\nThis was a bit messy to follow since they didn't have a full writeup. This is what I managed to extract from conversation over the pull request.
\n
Had simple string similarity version that turned out to be a really strong baseline
Hand crafted features for a tree based model: High-level AST of each line (basically the type of each code element, VARIABLE, CLASS, STRING, COMMENT, NUMBER, TOKEN, KEYWORD)
Cool Trick One idea: because code tends to be added to the end of the file, and new code tend to be more buggy than old code.
LambdaMART using RankLib
Used standard text similarity metrics
\n\nNo writeup, reversed the code
\n
Used:
\nstring matching
parse checking
decision rules
heuristics
Thanks for putting this out! Just one quick question, why should you only select a single fold? Isn't using multiple per model just like a big pasting ensemble for the model?
", "rawMarkdown": "Thanks for putting this out! Just one quick question, why should you only select a single fold? Isn't using multiple per model just like a big pasting ensemble for the model?", "votes": null }, { "id": "1820868", "postDate": "06/15/2022 04:25:32", "content": "Thank you so much for looking into so many references. I think it will be a lot of study. Thank you. @thedevastator 👍👍
", "rawMarkdown": "Thank you so much for looking into so many references. I think it will be a lot of study. Thank you. @thedevastator 👍👍", "votes": null } ], "comments": [ { "id": 1818622, "author_name": "leoxu1243876", "author_url": "", "post_date": "06/13/2022 01:36:40", "content": "Thanks for putting this out! Just one quick question, why should you only select a single fold? Isn't using multiple per model just like a big pasting ensemble for the model?
", "votes": null, "replies": [] }, { "id": 1820868, "author_name": "heuingttiang", "author_url": "", "post_date": "06/15/2022 04:25:32", "content": "Thank you so much for looking into so many references. I think it will be a lot of study. Thank you. @thedevastator 👍👍
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1818521": "# Tricks From Winning Solutions Of Other Code Competitions\n\n\nHi guys, \nI just finished going through all the winning solutions of 4 different code competitions (not all from Kaggle). \n\nHere are all the tricks they used.\nEnjoy yourself!\n\n\n### [Where there's code, there's bug](https://www.kaggle.com/competitions/wherethereiscodethereisbug/overview) - Last week\n\n> Classification: Line by line, some lines have bugs\n\n\n##### [4th solution](https://www.kaggle.com/competitions/wherethereiscodethereisbug/discussion/328738)\n\n**2 Stage Pipeline**\n- **Backbone** - CodeBERTa-small-v1 - applied to each code line - chose one fold (out of the 4)\n- **Linear classifier** runs on concatenation of the backbone pooler outputs\n\n\n##### [4th solution - Giba's approach](https://www.kaggle.com/code/karthickp6/giba-s-approach-tfidf-logistic-reg-lb-0-86538)\n\n\n\n**Super simple pipeline**\n\n- TFIDF (ngrams=(1,2))\n\n- LogisticRegression \n\n- Blended with the rest of the team \n\n\n##### [6th solution](https://www.kaggle.com/competitions/wherethereiscodethereisbug/discussion/328738)\n\n**Models**\n\n- \"microsoft/codebert-base\" (2 folds)\n- \"huggingface/CodeBERTa-small-v1\" (2x 1 fold + 1x 2 folds)\n- \"flax-sentence-embeddings/st-codesearch-distilroberta-base\" (1 fold)\n\n\n**Input & output**\n\n- Model Input : [CLS] line 1 [SEP] line 2 [SEP] line 3 [SEP] line 4 [SEP] line 5 [SEP]\n- Average pooling on the last features + dense + tanh + dense\n\n\n**Training**\n- bs=32, max_len=200\n- lr: 4e-5, apart for some CodeBERTa exps (tried 2e-5 and 5e-5)\n- 5 epochs for codebert, 4 for 3 or 4 were enough for the other smaller models\n\n**Post-processing**\n- Noticed that there were duplicates in the training data, and codes that were present twice had different labels\n- Gave a 0.5 score to duplicated texts\n\n##### [7th solution](https://www.kaggle.com/competitions/wherethereiscodethereisbug/discussion/328731)\n\n\n**Injected Special Token**\n\n**Many Models Ensemble**\n\n> Note: always select a single fold.\n\n- codebert-base(only 3line)\n- roberta-base(all sentences)\n- electra-base-discriminator(all sentences, epoch3)\n- destilbert-base-uncased(all sentences, epoch3)\n- codeberta-small-v1(all sentences, epoch 3)\n- codebert_base(all sentences, epoch7)\n- codebert-base(all sentences, epoch3)\n- codebert-base-mlm(all sentences, epoch3)\n- codebert-base-mlm(all sentences, epoch3)\n- muppet-roberta-base(all sentences, epoch4)\n- graphcodebert-base(all sentences, epoch3)\n\n### NLC2CMD - NuerIPS 2020\n\n> Task: Translate english to bash syntax\n\n\n##### 1st place\n\n[github](https://github.com/magnumresearchgroup/Magnum-NLC2CMD) | [paper](https://www.dre.vanderbilt.edu/~schmidt/PDF/A_Transformer_based_Approach_for_TranslatingNatural_Language_to_Bash_Commands.pdf)\n\n**Transformer + Architecture Improvements**\n\n**Custom Tokenizer** \n\n- Found by hyperparam search that they can shrink the vocabulary size by 90% and it even improves the results \n\n- **Heuristic improved beam search**\n\n\n##### 2nd place\n\n[github](https://github.com/nokia/nlc2cmd-submission-hubris)\n\n**[Custom HF Trainer](https://github.com/nokia/nlc2cmd-submission-hubris/blob/main/src/trainer.py)**\n\n**[Heavy preprocessing](https://github.com/nokia/nlc2cmd-submission-hubris/blob/main/notebooks/data_preprocess.ipynb)**\n\n> why didnt they just summarize thisssssss???? @#$@#%\n\n**[Custom diverse_beam_search](https://github.com/nokia/nlc2cmd-submission-hubris/blob/main/src/diverse_beam_search.py)**\n\n\n\n##### 3rd place\n\n[github](https://github.com/JetBrains/nlc2cmd) | [paper](https://github.com/JetBrains/nlc2cmd/blob/master/report.pdf)\n\n**Augmentations**\n\nBacktranslated (only the english prompt) using fairseq pretrained models, used only predictions with BLEU higher then a threshold (to avoid training their model with the mistakes of the fairseq model)\n\n**Data Generation**\n\nGenerated more data from the manpage: Commands were created by concatenating cmd name & some random flags, \nChanged some common pipline endings that were \"too easy\" such as \"wc\" & \"how many\"\n\n**Preprocessing**\n\n- **English prompt:** Removed stopwords using nltk and stemmed using Snowballstemmer\n- **Bash cmd:** changed all arguments to \"ARG\"\n\n**Postprocessing**\n\n- **Used two models, one for \n\n\n##### 4th place\n\n[github](https://github.com/Samsung/NLC2CMD)\n\n\n> Again, reversing their code\n\n**Fairseq Model** \n\n- can by seed by their trainer [script](https://github.com/Samsung/NLC2CMD/blob/main/train/3.train_template_predictor.sh)\n\n**Two models**\n\n- One for detecting templates \n- Other for detecting arguments\n\n\n##### 6th place\n\n[github](https://github.com/DNGros/clai) | [demo](https://clai-home.mybluemix.net/) | [paper](https://arxiv.org/abs/2002.00762)\n\n##### 9th place\n\n[github](https://github.com/greenmonn/nlc2cmd-competition) | [paper](https://ibm.box.com/s/3nb9oypu2iuiq3ow0m849o44ycd8jdbw)\n\n**Tree RNN Tensorflow Model**\n\n- Right [here](https://github.com/greenmonn/nlc2cmd-competition/blob/dfb28149deee16043b604dda4ea8a45ce4413938/nl2bash/encoder_decoder/seq2tree/tree_rnn_cell.py)\n\n- They also have many variations in the code with multiple heads and so on. They probably didn't delete the past experiments they tried.\n\n\n\n### CodRep-2019\n\n> Task: predict formatting errors in source code\n\n##### [2nd place](https://gist.github.com/kyrcha/4d4ebf960051ccfc8764d1f0f7ca6a05)\n\n**Preprocessing**\n\n- **Manually defined** [22 regex](https://gist.github.com/kyrcha/4d4ebf960051ccfc8764d1f0f7ca6a05) expressions for automaticlly detect common source code errors.\n\n- **Used the regex to clean manually a high quality dataset by hand** then trained a generative language model only on those samples.\n\n[this is a really cool idea..]\n\n- **Added common meaningful words** to the tokenizer such as \"True\" \"False\" \"if\"..\n\n- Optimized n-grams size (7 was optimal) so the distribution of tokens for \"broken/not broken\" would by as close as possible to 50%/50% then they used these token to train a balanced XGBoost classifier and an 1-class SVM for outlier detection.\n\n- **Ensemble:** prediction char by char (mean) with all models, then activate the regex on the \"most problematic\" parts of the text as post processing. \n\n\n### [CodRep-2018](https://github.com/KTH/CodRep-competition)\n\n> Task: replace lines in a given code: On a pair (code line, code file) predict for each line in \"code file\" the probability so that this is the line that needs to be replaced by \"code line\" [sort of \"similarity\" to code line]\n\n\n##### [1st place](https://github.com/KTH/CodRep-competition/issues/29)\n\n**Treated as a search/ranking** \n\n- Extracted features and then used \"learning to rank\" model. \n\n\n**Post-processing**\n- Ruled out the highest ranked prediction if applying the repair at that location yields a syntax error (they were allowed to do that? 😂)\n\nHere's a table that summarizes our results:\n\n##### [2nd place](https://github.com/KTH/CodRep-competition/issues/16)\n\n> This was a bit messy to follow since they didn't have a full writeup. This is what I managed to extract from conversation over the pull request.\n\n- **Had simple string similarity** version that turned out to be a really strong baseline\n\n- **Hand crafted features for a tree based model**: High-level AST of each line (basically the type of each code element, VARIABLE, CLASS, STRING, COMMENT, NUMBER, TOKEN, KEYWORD)\n\n- **Cool Trick** One idea: because code tends to be added to the end of the file, and new code tend to be more buggy than old code.\n\n##### [3rd place](https://github.com/jjhenkel/instauro)\n\n- **LambdaMART using RankLib**\n\n- **Used standard text similarity metrics**\n\n\n##### [4th place](https://github.com/agb94/coldbrew/)\n\n> No writeup, reversed the code\n\n- **Trained GP** on the features: The Sum of Matching Blocks' Size, Edit Distance, Lexical Similarity, Keyword Similarity\n\n\n##### [5th place](https://github.com/cesarsotovalero/CodRep-submission)\n\nUsed:\n\n- string matching\n\n- parse checking\n\n- decision rules\n\n- heuristics\n\n[related paper](https://link.springer.com/article/10.1007/s10664-017-9564-7)\n\n##### [6th place](https://github.com/egor-bogomolov/CodRep-competition/blob/master/Solutions/prefSufMatch.py)\n\n- Submitted a naive (handcrafted, hardcoded) baseline", "1818622": "Thanks for putting this out! Just one quick question, why should you only select a single fold? Isn't using multiple per model just like a big pasting ensemble for the model?", "1820868": "Thank you so much for looking into so many references. I think it will be a lot of study. Thank you. @thedevastator 👍👍" }, "source": "meta" }