{
  "id": 339390,
  "title": "Network size and hardware.",
  "url": "/competitions/AI4Code/discussion/339390",
  "author_name": "",
  "post_date": "2022-07-24T15:05:05.777474600Z",
  "votes": 6,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Im new to kaggle and would like to know the scale of your networks (just the number of parameters without sensitive details), and the hardware you use, it would be a bonus if you include training time as well, i hope that this will give me a perspective into what to expect when competing.</p>\n<p>Added to that if you could give any advice, such as the time you dedicate to working on this plus the style of your research, how do you come to a decision on a specific model, do you read a lot of papers ? and anything of the sort that you think is of importance.</p>\n<p>Thanks.</p>",
  "messages": [
    {
      "id": "1869188",
      "postDate": "07/24/2022 15:05:05",
      "content": "<p>Im new to kaggle and would like to know the scale of your networks (just the number of parameters without sensitive details), and the hardware you use, it would be a bonus if you include training time as well, i hope that this will give me a perspective into what to expect when competing.</p>\n<p>Added to that if you could give any advice, such as the time you dedicate to working on this plus the style of your research, how do you come to a decision on a specific model, do you read a lot of papers ? and anything of the sort that you think is of importance.</p>\n<p>Thanks.</p>",
      "rawMarkdown": "Im new to kaggle and would like to know the scale of your networks (just the number of parameters without sensitive details), and the hardware you use, it would be a bonus if you include training time as well, i hope that this will give me a perspective into what to expect when competing.\n\nAdded to that if you could give any advice, such as the time you dedicate to working on this plus the style of your research, how do you come to a decision on a specific model, do you read a lot of papers ? and anything of the sort that you think is of importance.\n\nThanks.",
      "votes": null
    },
    {
      "id": "1872191",
      "postDate": "07/26/2022 17:57:36",
      "content": "<p>The comp is at a sensitive stage now in its last 2 weeks and any hints (from the top teams) would immediately be caught on so most people may refrain from talking. Most of these would anyway be answered post the deadline. I could train only a single GraphCodeBert model. However I did experiment with many other models esp the bigger ones like Longformer, BigBird etc but none successfully. By success I mean being able to train the model on a bulk of data (at least 60%) under 4 hours per epoch on a Colab V100. (these are my limitations). <br>\nI spend a few hours on the weekends on this comp coding, though I try to read the discussions more frequently. Model choice has become like a hyperparam now with Huggingface so with minimal changes most of the models can be tried out. GCB and DeBERTa were promising but again I run into hardware challenges with Deberta. With my limitations, so far I got the best out of GCB. I did experiment with a variety of attention heads though again none worked better than the default one. More than reading papers, reading solutions of winners of previous NLp competitions will help. All the best on your Kaggle journey</p>",
      "rawMarkdown": "The comp is at a sensitive stage now in its last 2 weeks and any hints (from the top teams) would immediately be caught on so most people may refrain from talking. Most of these would anyway be answered post the deadline. I could train only a single GraphCodeBert model. However I did experiment with many other models esp the bigger ones like Longformer, BigBird etc but none successfully. By success I mean being able to train the model on a bulk of data (at least 60%) under 4 hours per epoch on a Colab V100. (these are my limitations). \nI spend a few hours on the weekends on this comp coding, though I try to read the discussions more frequently. Model choice has become like a hyperparam now with Huggingface so with minimal changes most of the models can be tried out. GCB and DeBERTa were promising but again I run into hardware challenges with Deberta. With my limitations, so far I got the best out of GCB. I did experiment with a variety of attention heads though again none worked better than the default one. More than reading papers, reading solutions of winners of previous NLp competitions will help. All the best on your Kaggle journey",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1872191,
      "author_name": "allohvk",
      "author_url": "",
      "post_date": "07/26/2022 17:57:36",
      "content": "<p>The comp is at a sensitive stage now in its last 2 weeks and any hints (from the top teams) would immediately be caught on so most people may refrain from talking. Most of these would anyway be answered post the deadline. I could train only a single GraphCodeBert model. However I did experiment with many other models esp the bigger ones like Longformer, BigBird etc but none successfully. By success I mean being able to train the model on a bulk of data (at least 60%) under 4 hours per epoch on a Colab V100. (these are my limitations). <br>\nI spend a few hours on the weekends on this comp coding, though I try to read the discussions more frequently. Model choice has become like a hyperparam now with Huggingface so with minimal changes most of the models can be tried out. GCB and DeBERTa were promising but again I run into hardware challenges with Deberta. With my limitations, so far I got the best out of GCB. I did experiment with a variety of attention heads though again none worked better than the default one. More than reading papers, reading solutions of winners of previous NLp competitions will help. All the best on your Kaggle journey</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1869188": "Im new to kaggle and would like to know the scale of your networks (just the number of parameters without sensitive details), and the hardware you use, it would be a bonus if you include training time as well, i hope that this will give me a perspective into what to expect when competing.\n\nAdded to that if you could give any advice, such as the time you dedicate to working on this plus the style of your research, how do you come to a decision on a specific model, do you read a lot of papers ? and anything of the sort that you think is of importance.\n\nThanks.",
    "1872191": "The comp is at a sensitive stage now in its last 2 weeks and any hints (from the top teams) would immediately be caught on so most people may refrain from talking. Most of these would anyway be answered post the deadline. I could train only a single GraphCodeBert model. However I did experiment with many other models esp the bigger ones like Longformer, BigBird etc but none successfully. By success I mean being able to train the model on a bulk of data (at least 60%) under 4 hours per epoch on a Colab V100. (these are my limitations). \nI spend a few hours on the weekends on this comp coding, though I try to read the discussions more frequently. Model choice has become like a hyperparam now with Huggingface so with minimal changes most of the models can be tried out. GCB and DeBERTa were promising but again I run into hardware challenges with Deberta. With my limitations, so far I got the best out of GCB. I did experiment with a variety of attention heads though again none worked better than the default one. More than reading papers, reading solutions of winners of previous NLp competitions will help. All the best on your Kaggle journey"
  },
  "source": "meta"
}