{
  "id": 341796,
  "title": "AdamW in Ranger21 optimizer",
  "url": "/competitions/AI4Code/discussion/341796",
  "author_name": "",
  "post_date": "2022-08-04T10:22:25.632557500Z",
  "votes": 6,
  "comment_count": 1,
  "views": 0,
  "content": "<p>AdamW is a well-known optimizer when finetuning NLP models.<br>\nThe new Ranger21 has AdamW in the updated version of the classic Ranger optimizer, also with other great tools as well as built-in scheduler.<br>\nDon’t know if it works better here and in NLP finetuning but it is for sure an optimizer for further testing 😊</p>\n<pre><code>Ranger21 - integrating the latest deep learning components into a single optimizer\nA rewrite of the Ranger deep learning optimizer to integrate newer optimization ideas and, in particular:\n\nuses the AdamW optimizer as its core (or, optionally, MadGrad)\nAdaptive gradient clipping\nGradient centralization\nPositive-Negative momentum\nNorm loss\nStable weight decay\nLinear learning rate warm-up\nExplore-exploit learning rate schedule\nLookahead\nSoftplus transformation\nGradient Normalization\n</code></pre>\n<p><a href=\"https://github.com/lessw2020/Ranger21\" target=\"_blank\">https://github.com/lessw2020/Ranger21</a><br>\n<a href=\"https://arxiv.org/abs/2106.13731\" target=\"_blank\">https://arxiv.org/abs/2106.13731</a></p>",
  "messages": [
    {
      "id": "1884149",
      "postDate": "08/04/2022 10:22:25",
      "content": "<p>AdamW is a well-known optimizer when finetuning NLP models.<br>\nThe new Ranger21 has AdamW in the updated version of the classic Ranger optimizer, also with other great tools as well as built-in scheduler.<br>\nDon’t know if it works better here and in NLP finetuning but it is for sure an optimizer for further testing 😊</p>\n<pre><code>Ranger21 - integrating the latest deep learning components into a single optimizer\nA rewrite of the Ranger deep learning optimizer to integrate newer optimization ideas and, in particular:\n\nuses the AdamW optimizer as its core (or, optionally, MadGrad)\nAdaptive gradient clipping\nGradient centralization\nPositive-Negative momentum\nNorm loss\nStable weight decay\nLinear learning rate warm-up\nExplore-exploit learning rate schedule\nLookahead\nSoftplus transformation\nGradient Normalization\n</code></pre>\n<p><a href=\"https://github.com/lessw2020/Ranger21\" target=\"_blank\">https://github.com/lessw2020/Ranger21</a><br>\n<a href=\"https://arxiv.org/abs/2106.13731\" target=\"_blank\">https://arxiv.org/abs/2106.13731</a></p>",
      "rawMarkdown": "AdamW is a well-known optimizer when finetuning NLP models.\nThe new Ranger21 has AdamW in the updated version of the classic Ranger optimizer, also with other great tools as well as built-in scheduler.\nDon’t know if it works better here and in NLP finetuning but it is for sure an optimizer for further testing 😊\n\n```\nRanger21 - integrating the latest deep learning components into a single optimizer\nA rewrite of the Ranger deep learning optimizer to integrate newer optimization ideas and, in particular:\n\nuses the AdamW optimizer as its core (or, optionally, MadGrad)\nAdaptive gradient clipping\nGradient centralization\nPositive-Negative momentum\nNorm loss\nStable weight decay\nLinear learning rate warm-up\nExplore-exploit learning rate schedule\nLookahead\nSoftplus transformation\nGradient Normalization\n```\n\nhttps://github.com/lessw2020/Ranger21\nhttps://arxiv.org/abs/2106.13731",
      "votes": null
    },
    {
      "id": "1884193",
      "postDate": "08/04/2022 10:42:07",
      "content": "<p>Very well curated! Thanks for explaining AdamW and Ranger21 in detail..</p>",
      "rawMarkdown": "Very well curated! Thanks for explaining AdamW and Ranger21 in detail..",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1884193,
      "author_name": "ravi20076",
      "author_url": "",
      "post_date": "08/04/2022 10:42:07",
      "content": "<p>Very well curated! Thanks for explaining AdamW and Ranger21 in detail..</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1884149": "AdamW is a well-known optimizer when finetuning NLP models.\nThe new Ranger21 has AdamW in the updated version of the classic Ranger optimizer, also with other great tools as well as built-in scheduler.\nDon’t know if it works better here and in NLP finetuning but it is for sure an optimizer for further testing 😊\n\n```\nRanger21 - integrating the latest deep learning components into a single optimizer\nA rewrite of the Ranger deep learning optimizer to integrate newer optimization ideas and, in particular:\n\nuses the AdamW optimizer as its core (or, optionally, MadGrad)\nAdaptive gradient clipping\nGradient centralization\nPositive-Negative momentum\nNorm loss\nStable weight decay\nLinear learning rate warm-up\nExplore-exploit learning rate schedule\nLookahead\nSoftplus transformation\nGradient Normalization\n```\n\nhttps://github.com/lessw2020/Ranger21\nhttps://arxiv.org/abs/2106.13731",
    "1884193": "Very well curated! Thanks for explaining AdamW and Ranger21 in detail.."
  },
  "source": "meta"
}