\nInitial experimentals didn't beat Khoi's baseline:
\n
![\"inital-exp1-fix.png\"](\"https://i.postimg.cc/bwT7bSNq/inital-exp1-fix.png\")
41th Place Solution
\nFirst, thanks Kaggle and the competition hosts for a challenging competition Google AI4Code – Understand Code in Python Notebooks: Predict the relationship between code and comments. This summarizes the effort of our team and it was a great experience working with my teammates! This is a writeup of our collective team effort.
\nBased on Khôi's initial idea, we modify a little bit to get more global context information. We sample 45 cells with shorter code cell's length (10 tokens). If the notebook has less than 45 cells, we dynamically adjust the max length of code cell based on the number of code cell.
\nEnsemble from 2 training setting (our setting and Khoi Nguyen setting - 20 code cell with 23 tokens each) also give us a boost on CV and LB. (we don't remember the exact number).
\nSome of the results on local CV are detailed in the photos below:
\nInitial experimental
\nInitial experimentals didn't beat Khoi's baseline:
\n
Dynamic tokenizer based on number of code cells and FGM training help boost score on both Local CV and LB:
\n![\"inital-exp2-fix.png\"](\"https://i.postimg.cc/ZqsXskdg/inital-exp2-fix.png\")
Final & best experimental
\nThe best model on both Local CV and LB is Deberta-v3-base.
\n![\"final-exp-fix.png\"](\"https://i.postimg.cc/P527NG2x/final-exp-fix.png\")
Inference code: Inference Ensemble Final Submission
\nWe really want to test the ideas in the papers below, but we really don't have enough time and computational resources.
\nWe want to thanks the Kaggle staff team & Competition Host for coordinating this great competition!
\nThanks for sharing the solution from all the teams, especially the top teams, we really learned a lot from their solutions.
Finally, thanks for the hard work of my teammates @dunglt
\n41th Place Solution
\nFirst, thanks Kaggle and the competition hosts for a challenging competition Google AI4Code – Understand Code in Python Notebooks: Predict the relationship between code and comments. This summarizes the effort of our team and it was a great experience working with my teammates! This is a writeup of our collective team effort.
\nBased on Khôi's initial idea, we modify a little bit to get more global context information. We sample 45 cells with shorter code cell's length (10 tokens). If the notebook has less than 45 cells, we dynamically adjust the max length of code cell based on the number of code cell.
\nEnsemble from 2 training setting (our setting and Khoi Nguyen setting - 20 code cell with 23 tokens each) also give us a boost on CV and LB. (we don't remember the exact number).
\nSome of the results on local CV are detailed in the photos below:
\nInitial experimental
\nInitial experimentals didn't beat Khoi's baseline:
\n
Dynamic tokenizer based on number of code cells and FGM training help boost score on both Local CV and LB:
\n
Final & best experimental
\nThe best model on both Local CV and LB is Deberta-v3-base.
\n
Inference code: Inference Ensemble Final Submission
\nWe really want to test the ideas in the papers below, but we really don't have enough time and computational resources.
\nWe want to thanks the Kaggle staff team & Competition Host for coordinating this great competition!
\nThanks for sharing the solution from all the teams, especially the top teams, we really learned a lot from their solutions.
Finally, thanks for the hard work of my teammates @dunglt
\nAwesome keep it running
", "rawMarkdown": "Awesome keep it running", "votes": null } ], "comments": [ { "id": 2038055, "author_name": "freakmind", "author_url": "", "post_date": "11/21/2022 06:49:35", "content": "Awesome keep it running
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2036713": "**41th Place Solution**\n\nFirst, thanks Kaggle and the competition hosts for a challenging competition Google AI4Code – Understand Code in Python Notebooks: Predict the relationship between code and comments. This summarizes the effort of our team and it was a great experience working with my teammates! This is a writeup of our collective team effort.\n\n# **Overview**\n\nBased on Khôi's initial idea, we modify a little bit to get more global context information. We sample 45 cells with shorter code cell's length (10 tokens). If the notebook has less than 45 cells, we dynamically adjust the max length of code cell based on the number of code cell.\n\nEnsemble from 2 training setting (our setting and Khoi Nguyen setting - 20 code cell with 23 tokens each) also give us a boost on CV and LB. (we don't remember the exact number).\n\n- **Model**: Weighted Average of some Point-wise models such as: Distilbert-base-uncased, Deberta-v3-base and Codebert-base\n- **Data**: All training competition data, no external data. Cross Validation Split: 90/10\n- **Hardware**: Two A6000-48GB. The largest model training for 3-4 days, the lighting model like distilbert is 1-2 days.\n\nSome of the results on local CV are detailed in the photos below:\n\n**Initial experimental**\nInitial experimentals didn't beat Khoi's baseline:\n[](https://postimg.cc/DSSp3ZXN)\n\nDynamic tokenizer based on number of code cells and FGM training help boost score on both Local CV and LB:\n\n[](https://postimg.cc/JyJTt2Kc)\n\n**Final & best experimental**\nThe best model on both Local CV and LB is Deberta-v3-base.\n[](https://postimg.cc/JsHxdFwV)\n\n**Inference code:** [Inference Ensemble Final Submission](https://www.kaggle.com/code/dunglt/inference-pct-rank-model?scriptVersionId=101081766)\n\n# **What worked**\n## [High Impact] Idea\n- Dynamic tokenizer based on number of code cells: Local + 0.007 and LB + 0.005\n- Local tokenizer to boost training speed\n## [Medium Impact] Idea\n- MLM pretrained (15% probability random mask token) on all data except 1000 val example: Local and LB + 0.002 (not boost much and much time to pretrained)\n- FGM training: Boost Local and CV + 0.002 but 2x time training.\n\n# **What didn’t work**\n- Larger model: Deberta-v3-large, Deberta-v3-xlarge\n- AWP/Fast AWP training: not boost score on local CV and LB\n- EMA training\n- Custom head: \n - mean/max pooling\n - concat mean-max pooling\n - concat last 4 layers\n - Attention pooling\n - LSTM/ Bi-LSTM\n\n## Additional context (idea not have time to try):\n\nWe really want to test the ideas in the papers below, but we really don't have enough time and computational resources.\n\n- Deep Attentive Ranking Networks for Learning to Order Sentences - [paper](https://arxiv.org/abs/2001.00056)\t\t\t\t\n- Efficient Relational Sentence Ordering Network\t- [paper](https://ieeexplore.ieee.org/document/9444816)\t\t\t\t\t\t\t\t\n- Neural Sentence Ordering Based on Constraint Graphs - [paper](https://arxiv.org/abs/2101.11178)\n- Is Everything in Order? A Simple Way to Order Sentences - [paper](https://arxiv.org/pdf/2104.07064v2.pdf)\t\t\t\t\t\t\n- Sentence Ordering and Coherence Modeling using Recurrent Neural Networks - [paper](https://arxiv.org/abs/1611.02654)\t\n- Knowledge Distillation (KD)\n- Augmentation technique: Mixup, Random mask\n\n# Thanks and Acknowledgements:\n\nWe want to thanks the Kaggle staff team & Competition Host for coordinating this great competition! \nThanks for sharing the solution from all the teams, especially the top teams, we really learned a lot from their solutions.\n\nFinally, thanks for the hard work of my teammates @dunglt \n\n# Team Members:\t\t\t\n - Khanh Vu Duy [KhanhVD](https://www.kaggle.com/duykhanh99)\t\n - Dung Le [Dzung](https://www.kaggle.com/dunglt)", "2038055": "Awesome keep it running" }, "source": "existing_discussion", "complete": true, "competition_id": 35887, "competition_slug": "AI4Code", "team_id": 8611091, "solution_url": "/competitions/AI4Code/writeups/dl-41th-place-solution" }