{
  "id": 343730,
  "title": "Public LB 14th Solution",
  "url": "/competitions/AI4Code/discussion/343730",
  "author_name": "Alireza",
  "post_date": "2022-08-12T10:07:10.122000",
  "votes": 24,
  "comment_count": 8,
  "views": 0,
  "content": "<p>First and foremost, I want to thank Kaggle and the Kaggle community for providing us with the chance to learn via experience and collaboration. This was my first time competing in a Kaggle competition, and I'm quite pleased because I learned a lot about various model Archs (such as transformers, Pointer Networks, and so on) and NLP in general. </p>\n<p>We have three approaches for ranking problems (correct me if I'm wrong!): point-wise, pair-wise, and list-wise. The absence of global context is a concern with point-wise, however, some participants attempted to fix it by attaching a subset of code cells to each markdown cell. The pair-wise strategy appears to be highly strong, but it comes at the cost of a long run-time. Finally, list-wise attempts to rank cells using a mix of global and local information.</p>\n<p>I picked the list-wise method since it was faster than the pair-wise method and seemed more natural to me. </p>\n<p>The following image depicts the model architecture:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F289290%2F2b83aa98253260dd4bceb7035dd91b99%2FAI4Code_arch.drawio.png?generation=1660293627287930&amp;alt=media\" alt=\"\"></p>\n<ol>\n<li>Using <code>sentence-transformers/all-mpnet-base-v2</code>, compute cell features separately: I choose the first and last 64 words of each cell's text after preprocessing it to get a feel of how the cell begins and finishes. This is then sent into the sentence transformer model, which produces a 768d feature vector.</li>\n<li>I then append the following 8 features to each vector:<ul>\n<li>is_code: 1 if cell is code else 0</li>\n<li>is_markdown: 1 if cell is markdown else 0</li>\n<li>code_rank: rank of the code cell if cell is code else 0</li>\n<li>toc_rank: rank of the markdown cell in the table of contents if cell is markdown and ToC exists else 0</li>\n<li>numberd_rank: rank of the md cell if it has numbered title (like 1.1, 1.2) else 0</li>\n<li>has_heading_1: 1 if markdown cell has heading 1 else 0</li>\n<li>has_heading_2: 1 if markdown cell has heading 2 else 0</li>\n<li>has_heading_3: 1 if markdown cell has heading 3 else 0</li></ul></li>\n<li>Then I use an 8-head, 10-layer encoder-only transformer to compute the final features for each cell.</li>\n<li>Finally, I have two outputs for the code cells and four outputs for the markdown cells:<ul>\n<li>rank: this is the ranking output that I use in the inference. I apply a sigmoid on it.</li>\n<li>code_rank: rank of the code cell among other code cells.</li>\n<li>markdown_rank: rank of the markdown cell among other md cells</li>\n<li>toc_rank: rank of the md cell in the toc </li>\n<li>numbered_rank: rank of the md cell among other numbered md cells</li></ul></li>\n</ol>\n<h2>Loss function:</h2>\n<p>L1Loss is used, however instead of using it on the [0, 1] range, I use it on the [0, #cells] range. To do this, I first apply sigmoid, then multiply by the number of cells and use the result as the predicted output. This is due to the fact that longer notebooks contribute more to the scoring function, thus we should pay more attention to their flaws.</p>\n<h2>training time:</h2>\n<p>Training the notebook transformer is pretty fast. One epoch of training with a batch size of 10 notebooks and 17k steps takes less than 30 mins on a P100. </p>\n<h2>Inference time:</h2>\n<p>The entire inference procedure, including my preprocessing, takes around two hours. However, the model itself takes less than an hour to complete.</p>\n<h2>Preprocessing</h2>\n<ol>\n<li>Fix encodings</li>\n<li>Fix HTML in markdown cells</li>\n<li>Detect ToC rankings</li>\n<li>Detect numbered rankings</li>\n<li>Detect language</li>\n<li>Translation of non-English languages using MarianMT</li>\n</ol>\n<h2>Challenges</h2>\n<p>My main challenge was that I couldn't train my entire model end2end on all the notebooks since my GPU (Colab P100 or V100) couldn't handle notebooks larger than 20 cells 😄 </p>\n<h2>External datasets</h2>\n<p>I crawled around 50k additional notebooks from Kaggle, but it seems they didn't overlap much with the Private test set cause it only improved my score by 0.01!</p>",
  "messages": [
    {
      "id": 1895677,
      "postDate": "2022-08-12T10:07:10.123Z",
      "content": "<p>First and foremost, I want to thank Kaggle and the Kaggle community for providing us with the chance to learn via experience and collaboration. This was my first time competing in a Kaggle competition, and I'm quite pleased because I learned a lot about various model Archs (such as transformers, Pointer Networks, and so on) and NLP in general. </p>\n<p>We have three approaches for ranking problems (correct me if I'm wrong!): point-wise, pair-wise, and list-wise. The absence of global context is a concern with point-wise, however, some participants attempted to fix it by attaching a subset of code cells to each markdown cell. The pair-wise strategy appears to be highly strong, but it comes at the cost of a long run-time. Finally, list-wise attempts to rank cells using a mix of global and local information.</p>\n<p>I picked the list-wise method since it was faster than the pair-wise method and seemed more natural to me. </p>\n<p>The following image depicts the model architecture:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F289290%2F2b83aa98253260dd4bceb7035dd91b99%2FAI4Code_arch.drawio.png?generation=1660293627287930&amp;alt=media\" alt=\"\"></p>\n<ol>\n<li>Using <code>sentence-transformers/all-mpnet-base-v2</code>, compute cell features separately: I choose the first and last 64 words of each cell's text after preprocessing it to get a feel of how the cell begins and finishes. This is then sent into the sentence transformer model, which produces a 768d feature vector.</li>\n<li>I then append the following 8 features to each vector:<ul>\n<li>is_code: 1 if cell is code else 0</li>\n<li>is_markdown: 1 if cell is markdown else 0</li>\n<li>code_rank: rank of the code cell if cell is code else 0</li>\n<li>toc_rank: rank of the markdown cell in the table of contents if cell is markdown and ToC exists else 0</li>\n<li>numberd_rank: rank of the md cell if it has numbered title (like 1.1, 1.2) else 0</li>\n<li>has_heading_1: 1 if markdown cell has heading 1 else 0</li>\n<li>has_heading_2: 1 if markdown cell has heading 2 else 0</li>\n<li>has_heading_3: 1 if markdown cell has heading 3 else 0</li></ul></li>\n<li>Then I use an 8-head, 10-layer encoder-only transformer to compute the final features for each cell.</li>\n<li>Finally, I have two outputs for the code cells and four outputs for the markdown cells:<ul>\n<li>rank: this is the ranking output that I use in the inference. I apply a sigmoid on it.</li>\n<li>code_rank: rank of the code cell among other code cells.</li>\n<li>markdown_rank: rank of the markdown cell among other md cells</li>\n<li>toc_rank: rank of the md cell in the toc </li>\n<li>numbered_rank: rank of the md cell among other numbered md cells</li></ul></li>\n</ol>\n<h2>Loss function:</h2>\n<p>L1Loss is used, however instead of using it on the [0, 1] range, I use it on the [0, #cells] range. To do this, I first apply sigmoid, then multiply by the number of cells and use the result as the predicted output. This is due to the fact that longer notebooks contribute more to the scoring function, thus we should pay more attention to their flaws.</p>\n<h2>training time:</h2>\n<p>Training the notebook transformer is pretty fast. One epoch of training with a batch size of 10 notebooks and 17k steps takes less than 30 mins on a P100. </p>\n<h2>Inference time:</h2>\n<p>The entire inference procedure, including my preprocessing, takes around two hours. However, the model itself takes less than an hour to complete.</p>\n<h2>Preprocessing</h2>\n<ol>\n<li>Fix encodings</li>\n<li>Fix HTML in markdown cells</li>\n<li>Detect ToC rankings</li>\n<li>Detect numbered rankings</li>\n<li>Detect language</li>\n<li>Translation of non-English languages using MarianMT</li>\n</ol>\n<h2>Challenges</h2>\n<p>My main challenge was that I couldn't train my entire model end2end on all the notebooks since my GPU (Colab P100 or V100) couldn't handle notebooks larger than 20 cells 😄 </p>\n<h2>External datasets</h2>\n<p>I crawled around 50k additional notebooks from Kaggle, but it seems they didn't overlap much with the Private test set cause it only improved my score by 0.01!</p>",
      "rawMarkdown": "First and foremost, I want to thank Kaggle and the Kaggle community for providing us with the chance to learn via experience and collaboration. This was my first time competing in a Kaggle competition, and I'm quite pleased because I learned a lot about various model Archs (such as transformers, Pointer Networks, and so on) and NLP in general. \n\nWe have three approaches for ranking problems (correct me if I'm wrong!): point-wise, pair-wise, and list-wise. The absence of global context is a concern with point-wise, however, some participants attempted to fix it by attaching a subset of code cells to each markdown cell. The pair-wise strategy appears to be highly strong, but it comes at the cost of a long run-time. Finally, list-wise attempts to rank cells using a mix of global and local information.\n\nI picked the list-wise method since it was faster than the pair-wise method and seemed more natural to me. \n\nThe following image depicts the model architecture:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F289290%2F2b83aa98253260dd4bceb7035dd91b99%2FAI4Code_arch.drawio.png?generation=1660293627287930&alt=media)\n\n\n1. Using `sentence-transformers/all-mpnet-base-v2`, compute cell features separately: I choose the first and last 64 words of each cell's text after preprocessing it to get a feel of how the cell begins and finishes. This is then sent into the sentence transformer model, which produces a 768d feature vector.\n2. I then append the following 8 features to each vector:\n   - is_code: 1 if cell is code else 0\n   - is_markdown: 1 if cell is markdown else 0\n   - code_rank: rank of the code cell if cell is code else 0\n   - toc_rank: rank of the markdown cell in the table of contents if cell is markdown and ToC exists else 0\n   - numberd_rank: rank of the md cell if it has numbered title (like 1.1, 1.2) else 0\n   - has_heading_1: 1 if markdown cell has heading 1 else 0\n   - has_heading_2: 1 if markdown cell has heading 2 else 0\n   - has_heading_3: 1 if markdown cell has heading 3 else 0\n3. Then I use an 8-head, 10-layer encoder-only transformer to compute the final features for each cell.\n4. Finally, I have two outputs for the code cells and four outputs for the markdown cells:\n   - rank: this is the ranking output that I use in the inference. I apply a sigmoid on it.\n   - code_rank: rank of the code cell among other code cells.\n   - markdown_rank: rank of the markdown cell among other md cells\n   - toc_rank: rank of the md cell in the toc \n   - numbered_rank: rank of the md cell among other numbered md cells\n\n##  Loss function: \nL1Loss is used, however instead of using it on the [0, 1] range, I use it on the [0, #cells] range. To do this, I first apply sigmoid, then multiply by the number of cells and use the result as the predicted output. This is due to the fact that longer notebooks contribute more to the scoring function, thus we should pay more attention to their flaws.\n\n## training time:\nTraining the notebook transformer is pretty fast. One epoch of training with a batch size of 10 notebooks and 17k steps takes less than 30 mins on a P100. \n\n## Inference time:\nThe entire inference procedure, including my preprocessing, takes around two hours. However, the model itself takes less than an hour to complete.\n\n## Preprocessing\n1. Fix encodings\n2. Fix HTML in markdown cells\n3. Detect ToC rankings\n4. Detect numbered rankings\n5. Detect language\n6. Translation of non-English languages using MarianMT\n\n## Challenges\nMy main challenge was that I couldn't train my entire model end2end on all the notebooks since my GPU (Colab P100 or V100) couldn't handle notebooks larger than 20 cells 😄 \n\n## External datasets\nI crawled around 50k additional notebooks from Kaggle, but it seems they didn't overlap much with the Private test set cause it only improved my score by 0.01!\n\n",
      "votes": 24
    },
    {
      "id": 1906438,
      "postDate": "2022-08-19T22:05:49.140Z",
      "content": "<p>Thanks for sharing your solution, It is a really simple idea and yet so powerful! <br>\nGreat stuff! </p>\n<p>And congrats btw! </p>\n<p>The Devastator.</p>",
      "rawMarkdown": "Thanks for sharing your solution, It is a really simple idea and yet so powerful! \nGreat stuff! \n\nAnd congrats btw! \n\n\n\nThe Devastator.\n",
      "votes": 1
    },
    {
      "id": 1895942,
      "postDate": "2022-08-12T13:00:55.887Z",
      "content": "<p>Congratulation <a href=\"https://www.kaggle.com/alireza9\" target=\"_blank\">@alireza9</a> for your elegent list-wise solution and your medal in this competition.<br>\nI'm really impressed that you successfully reached a decent score with such a limited access to GPUs </p>\n<p>I'm just beeing curious here : did you retrain mpnet-base-v2 or did you just freeze it ? </p>\n<p>Thanks a lot for sharing your work</p>",
      "rawMarkdown": "Congratulation @alireza9 for your elegent list-wise solution and your medal in this competition.\nI'm really impressed that you successfully reached a decent score with such a limited access to GPUs \n\nI'm just beeing curious here : did you retrain mpnet-base-v2 or did you just freeze it ? \n\nThanks a lot for sharing your work",
      "votes": 1,
      "replies": [
        {
          "id": 1895965,
          "postDate": "2022-08-12T13:16:33.923Z",
          "content": "<p>Thanks! I trained it independently on 10% of the training data using the same architecture, except for the notebook transformer I used a 1-layer 1-head encoder transformer with only the rank loss.</p>\n<p>The notebook transformer was then deleted, and the base sentence transformer model was utilized as my feature extractor. Following that, I did not train the base model.</p>",
          "rawMarkdown": "Thanks! I trained it independently on 10% of the training data using the same architecture, except for the notebook transformer I used a 1-layer 1-head encoder transformer with only the rank loss.\n\nThe notebook transformer was then deleted, and the base sentence transformer model was utilized as my feature extractor. Following that, I did not train the base model.",
          "votes": 2
        }
      ]
    },
    {
      "id": 1896108,
      "postDate": "2022-08-12T15:22:11.910Z",
      "content": "<p>Congrats for a strong finish! Elegant model. </p>\n<blockquote>\n  <p>30 mins per epoch on a P100.</p>\n</blockquote>\n<p>With a 90+ score, pound for pound, this is probably the strongest model in my view… </p>",
      "rawMarkdown": "Congrats for a strong finish! Elegant model. \n\n> 30 mins per epoch on a P100.\n\nWith a 90+ score, pound for pound, this is probably the strongest model in my view... ",
      "votes": 2,
      "replies": [
        {
          "id": 1896330,
          "postDate": "2022-08-12T18:51:49.437Z",
          "content": "<p>Thanks! The short training time indeed allowed me to iterate on my ideas much more quickly.</p>",
          "rawMarkdown": "Thanks! The short training time indeed allowed me to iterate on my ideas much more quickly.",
          "votes": 3
        }
      ]
    },
    {
      "id": 1906559,
      "postDate": "2022-08-20T01:54:27.683Z",
      "content": "<p>Thanks for sharing the solution! I'm wondering whether you had used the same dense layers for markdown and code cells?</p>",
      "rawMarkdown": "Thanks for sharing the solution! I'm wondering whether you had used the same dense layers for markdown and code cells?",
      "replies": [
        {
          "id": 1906900,
          "postDate": "2022-08-20T10:14:30.290Z",
          "content": "<p>Yeah, the dense layer is shared. </p>",
          "rawMarkdown": "Yeah, the dense layer is shared. "
        }
      ]
    },
    {
      "id": 1918887,
      "postDate": "2022-08-30T00:31:32.063Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1906438,
      "author_name": "The Devastator",
      "author_url": "",
      "post_date": "2022-08-19T22:05:49.140000",
      "content": "<p>Thanks for sharing your solution, It is a really simple idea and yet so powerful! <br>\nGreat stuff! </p>\n<p>And congrats btw! </p>\n<p>The Devastator.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1895942,
      "author_name": "Wassim",
      "author_url": "",
      "post_date": "2022-08-12T13:00:55.887000",
      "content": "<p>Congratulation <a href=\"https://www.kaggle.com/alireza9\" target=\"_blank\">@alireza9</a> for your elegent list-wise solution and your medal in this competition.<br>\nI'm really impressed that you successfully reached a decent score with such a limited access to GPUs </p>\n<p>I'm just beeing curious here : did you retrain mpnet-base-v2 or did you just freeze it ? </p>\n<p>Thanks a lot for sharing your work</p>",
      "votes": 1,
      "replies": [
        {
          "id": 1895965,
          "author_name": "Alireza",
          "author_url": "",
          "post_date": "2022-08-12T13:16:33.923000",
          "content": "<p>Thanks! I trained it independently on 10% of the training data using the same architecture, except for the notebook transformer I used a 1-layer 1-head encoder transformer with only the rank loss.</p>\n<p>The notebook transformer was then deleted, and the base sentence transformer model was utilized as my feature extractor. Following that, I did not train the base model.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1896108,
      "author_name": "Allohvk",
      "author_url": "",
      "post_date": "2022-08-12T15:22:11.910000",
      "content": "<p>Congrats for a strong finish! Elegant model. </p>\n<blockquote>\n  <p>30 mins per epoch on a P100.</p>\n</blockquote>\n<p>With a 90+ score, pound for pound, this is probably the strongest model in my view… </p>",
      "votes": 2,
      "replies": [
        {
          "id": 1896330,
          "author_name": "Alireza",
          "author_url": "",
          "post_date": "2022-08-12T18:51:49.437000",
          "content": "<p>Thanks! The short training time indeed allowed me to iterate on my ideas much more quickly.</p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 1906559,
      "author_name": "Cabbage972",
      "author_url": "",
      "post_date": "2022-08-20T01:54:27.683000",
      "content": "<p>Thanks for sharing the solution! I'm wondering whether you had used the same dense layers for markdown and code cells?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1906900,
          "author_name": "Alireza",
          "author_url": "",
          "post_date": "2022-08-20T10:14:30.290000",
          "content": "<p>Yeah, the dense layer is shared. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1918887,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-08-30T00:31:32.063000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1895677": "First and foremost, I want to thank Kaggle and the Kaggle community for providing us with the chance to learn via experience and collaboration. This was my first time competing in a Kaggle competition, and I'm quite pleased because I learned a lot about various model Archs (such as transformers, Pointer Networks, and so on) and NLP in general. \n\nWe have three approaches for ranking problems (correct me if I'm wrong!): point-wise, pair-wise, and list-wise. The absence of global context is a concern with point-wise, however, some participants attempted to fix it by attaching a subset of code cells to each markdown cell. The pair-wise strategy appears to be highly strong, but it comes at the cost of a long run-time. Finally, list-wise attempts to rank cells using a mix of global and local information.\n\nI picked the list-wise method since it was faster than the pair-wise method and seemed more natural to me. \n\nThe following image depicts the model architecture:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F289290%2F2b83aa98253260dd4bceb7035dd91b99%2FAI4Code_arch.drawio.png?generation=1660293627287930&alt=media)\n\n\n1. Using `sentence-transformers/all-mpnet-base-v2`, compute cell features separately: I choose the first and last 64 words of each cell's text after preprocessing it to get a feel of how the cell begins and finishes. This is then sent into the sentence transformer model, which produces a 768d feature vector.\n2. I then append the following 8 features to each vector:\n   - is_code: 1 if cell is code else 0\n   - is_markdown: 1 if cell is markdown else 0\n   - code_rank: rank of the code cell if cell is code else 0\n   - toc_rank: rank of the markdown cell in the table of contents if cell is markdown and ToC exists else 0\n   - numberd_rank: rank of the md cell if it has numbered title (like 1.1, 1.2) else 0\n   - has_heading_1: 1 if markdown cell has heading 1 else 0\n   - has_heading_2: 1 if markdown cell has heading 2 else 0\n   - has_heading_3: 1 if markdown cell has heading 3 else 0\n3. Then I use an 8-head, 10-layer encoder-only transformer to compute the final features for each cell.\n4. Finally, I have two outputs for the code cells and four outputs for the markdown cells:\n   - rank: this is the ranking output that I use in the inference. I apply a sigmoid on it.\n   - code_rank: rank of the code cell among other code cells.\n   - markdown_rank: rank of the markdown cell among other md cells\n   - toc_rank: rank of the md cell in the toc \n   - numbered_rank: rank of the md cell among other numbered md cells\n\n##  Loss function: \nL1Loss is used, however instead of using it on the [0, 1] range, I use it on the [0, #cells] range. To do this, I first apply sigmoid, then multiply by the number of cells and use the result as the predicted output. This is due to the fact that longer notebooks contribute more to the scoring function, thus we should pay more attention to their flaws.\n\n## training time:\nTraining the notebook transformer is pretty fast. One epoch of training with a batch size of 10 notebooks and 17k steps takes less than 30 mins on a P100. \n\n## Inference time:\nThe entire inference procedure, including my preprocessing, takes around two hours. However, the model itself takes less than an hour to complete.\n\n## Preprocessing\n1. Fix encodings\n2. Fix HTML in markdown cells\n3. Detect ToC rankings\n4. Detect numbered rankings\n5. Detect language\n6. Translation of non-English languages using MarianMT\n\n## Challenges\nMy main challenge was that I couldn't train my entire model end2end on all the notebooks since my GPU (Colab P100 or V100) couldn't handle notebooks larger than 20 cells 😄 \n\n## External datasets\nI crawled around 50k additional notebooks from Kaggle, but it seems they didn't overlap much with the Private test set cause it only improved my score by 0.01!\n\n",
    "1906438": "Thanks for sharing your solution, It is a really simple idea and yet so powerful! \nGreat stuff! \n\nAnd congrats btw! \n\n\n\nThe Devastator.\n",
    "1895942": "Congratulation @alireza9 for your elegent list-wise solution and your medal in this competition.\nI'm really impressed that you successfully reached a decent score with such a limited access to GPUs \n\nI'm just beeing curious here : did you retrain mpnet-base-v2 or did you just freeze it ? \n\nThanks a lot for sharing your work",
    "1896108": "Congrats for a strong finish! Elegant model. \n\n> 30 mins per epoch on a P100.\n\nWith a 90+ score, pound for pound, this is probably the strongest model in my view... ",
    "1906559": "Thanks for sharing the solution! I'm wondering whether you had used the same dense layers for markdown and code cells?",
    "1918887": ""
  }
}