{
  "id": 102617,
  "title": "GNN optimizers and schedulers",
  "url": "/competitions/champs-scalar-coupling/discussion/102617",
  "author_name": "",
  "post_date": "2019-08-03T09:11:28.781064700Z",
  "votes": 8,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Since I am new to this whole GNN business, I am looking for suitable <strong>optimizers</strong> (SGD, ADAM, RMSProp) and learning rate <strong>schedulers</strong> (decay, reduce on plateau, cyclic, and so on). \nFor now, I am using ADAM and no scheduler but planning to experiment with different things. \nMaybe these parameters aren't that important for this dataset?\nAny help/hint (from papers, videos...) is appreciated. \nThanks. :) </p>",
  "messages": [
    {
      "id": "591177",
      "postDate": "08/03/2019 09:11:28",
      "content": "<p>Since I am new to this whole GNN business, I am looking for suitable <strong>optimizers</strong> (SGD, ADAM, RMSProp) and learning rate <strong>schedulers</strong> (decay, reduce on plateau, cyclic, and so on). \nFor now, I am using ADAM and no scheduler but planning to experiment with different things. \nMaybe these parameters aren't that important for this dataset?\nAny help/hint (from papers, videos...) is appreciated. \nThanks. :) </p>",
      "rawMarkdown": "Since I am new to this whole GNN business, I am looking for suitable **optimizers** (SGD, ADAM, RMSProp) and learning rate **schedulers** (decay, reduce on plateau, cyclic, and so on). \nFor now, I am using ADAM and no scheduler but planning to experiment with different things. \nMaybe these parameters aren't that important for this dataset?\nAny help/hint (from papers, videos...) is appreciated. \nThanks. :)",
      "votes": null
    },
    {
      "id": "591181",
      "postDate": "08/03/2019 09:23:48",
      "content": "<p>By the way, I am planning to apply the tips from this great blog post: <a href=\"http://karpathy.github.io/2019/04/25/recipe/\">http://karpathy.github.io/2019/04/25/recipe/</a>. \nMight be interesting to some of you.</p>",
      "rawMarkdown": "By the way, I am planning to apply the tips from this great blog post: http://karpathy.github.io/2019/04/25/recipe/. \nMight be interesting to some of you.",
      "votes": null
    },
    {
      "id": "591435",
      "postDate": "08/03/2019 17:21:09",
      "content": "<p>As further details: </p>\n\n<ul>\n<li>run a model with a fixed LR of 0.001 (for a long time tough) and got around -1.55 on local validation</li>\n<li>running a new model with an initial 0.1 LR and a reduce on plateau scheduler. Will update with the score once the model finishes training. </li>\n</ul>",
      "rawMarkdown": "As further details: \n\n- run a model with a fixed LR of 0.001 (for a long time tough) and got around -1.55 on local validation\n- running a new model with an initial 0.1 LR and a reduce on plateau scheduler. Will update with the score once the model finishes training.",
      "votes": null
    },
    {
      "id": "591526",
      "postDate": "08/03/2019 21:27:13",
      "content": "<p>What kind of NN do you use? If it's not a secret. :)</p>",
      "rawMarkdown": "What kind of NN do you use? If it's not a secret. :)",
      "votes": null
    },
    {
      "id": "591760",
      "postDate": "08/04/2019 08:08:12",
      "content": "<p>Not a secret of course. :)\nIt is an MPNN from this <a href=\"https://arxiv.org/pdf/1704.01212.pdf\">paper</a>. \nI am using the starter kit by <a href=\"/hengck23\">@hengck23</a>  and working on tweaking the newtorks' hyperparameters and the learning parameters (optimizer and scheduler). \nI might also use a temporal layer (LSTM to start with) for aggregating the passed messages instead of a simple sum. However, I am afraid that my single GPU won't be enough to run it on time :D \nFinally, check this great <a href=\"https://www.kaggle.com/fnands/1-mpnn\">kernel</a> if you want to check an implementation. <br>\nLet me know if you need more details.</p>",
      "rawMarkdown": "Not a secret of course. :)\nIt is an MPNN from this [paper](https://arxiv.org/pdf/1704.01212.pdf). \nI am using the starter kit by @hengck23  and working on tweaking the newtorks' hyperparameters and the learning parameters (optimizer and scheduler). \nI might also use a temporal layer (LSTM to start with) for aggregating the passed messages instead of a simple sum. However, I am afraid that my single GPU won't be enough to run it on time :D \nFinally, check this great [kernel](https://www.kaggle.com/fnands/1-mpnn) if you want to check an implementation.  \nLet me know if you need more details.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 591181,
      "author_name": "yassinealouini",
      "author_url": "",
      "post_date": "08/03/2019 09:23:48",
      "content": "<p>By the way, I am planning to apply the tips from this great blog post: <a href=\"http://karpathy.github.io/2019/04/25/recipe/\">http://karpathy.github.io/2019/04/25/recipe/</a>. \nMight be interesting to some of you.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 591435,
      "author_name": "yassinealouini",
      "author_url": "",
      "post_date": "08/03/2019 17:21:09",
      "content": "<p>As further details: </p>\n\n<ul>\n<li>run a model with a fixed LR of 0.001 (for a long time tough) and got around -1.55 on local validation</li>\n<li>running a new model with an initial 0.1 LR and a reduce on plateau scheduler. Will update with the score once the model finishes training. </li>\n</ul>",
      "votes": null,
      "replies": [
        {
          "id": 591526,
          "author_name": "sergeyzlobin",
          "author_url": "",
          "post_date": "08/03/2019 21:27:13",
          "content": "<p>What kind of NN do you use? If it's not a secret. :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 591760,
          "author_name": "yassinealouini",
          "author_url": "",
          "post_date": "08/04/2019 08:08:12",
          "content": "<p>Not a secret of course. :)\nIt is an MPNN from this <a href=\"https://arxiv.org/pdf/1704.01212.pdf\">paper</a>. \nI am using the starter kit by <a href=\"/hengck23\">@hengck23</a>  and working on tweaking the newtorks' hyperparameters and the learning parameters (optimizer and scheduler). \nI might also use a temporal layer (LSTM to start with) for aggregating the passed messages instead of a simple sum. However, I am afraid that my single GPU won't be enough to run it on time :D \nFinally, check this great <a href=\"https://www.kaggle.com/fnands/1-mpnn\">kernel</a> if you want to check an implementation. <br>\nLet me know if you need more details.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "591177": "Since I am new to this whole GNN business, I am looking for suitable **optimizers** (SGD, ADAM, RMSProp) and learning rate **schedulers** (decay, reduce on plateau, cyclic, and so on). \nFor now, I am using ADAM and no scheduler but planning to experiment with different things. \nMaybe these parameters aren't that important for this dataset?\nAny help/hint (from papers, videos...) is appreciated. \nThanks. :)",
    "591181": "By the way, I am planning to apply the tips from this great blog post: http://karpathy.github.io/2019/04/25/recipe/. \nMight be interesting to some of you.",
    "591435": "As further details: \n\n- run a model with a fixed LR of 0.001 (for a long time tough) and got around -1.55 on local validation\n- running a new model with an initial 0.1 LR and a reduce on plateau scheduler. Will update with the score once the model finishes training.",
    "591526": "What kind of NN do you use? If it's not a secret. :)",
    "591760": "Not a secret of course. :)\nIt is an MPNN from this [paper](https://arxiv.org/pdf/1704.01212.pdf). \nI am using the starter kit by @hengck23  and working on tweaking the newtorks' hyperparameters and the learning parameters (optimizer and scheduler). \nI might also use a temporal layer (LSTM to start with) for aggregating the passed messages instead of a simple sum. However, I am afraid that my single GPU won't be enough to run it on time :D \nFinally, check this great [kernel](https://www.kaggle.com/fnands/1-mpnn) if you want to check an implementation.  \nLet me know if you need more details."
  },
  "source": "meta"
}