![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1424766%2F18ccb3e171e0a4ad7b81b3f7025b74a0%2Fpizza.gif?generation=1566379541213407&alt=media\")
New optimizer - RAdam\n
\n\nhttps://medium.com/@lessw/new-state-of-the-art-ai-optimizer-rectified-adam-radam-5d854730807b
\n\nhttps://arxiv.org/abs/1908.03265v1
\n\nhttps://github.com/LiyuanLucasLiu/RAdam/blob/master/cifar_imagenet/utils/radam.py
\n\nKeras implementaion: https://github.com/CyberZHG/keras-radam
", "messages": [ { "id": "600615", "postDate": "08/16/2019 11:13:17", "content": "New optimizer - RAdam\n
\n\nhttps://medium.com/@lessw/new-state-of-the-art-ai-optimizer-rectified-adam-radam-5d854730807b
\n\nhttps://arxiv.org/abs/1908.03265v1
\n\nhttps://github.com/LiyuanLucasLiu/RAdam/blob/master/cifar_imagenet/utils/radam.py
\n\nKeras implementaion: https://github.com/CyberZHG/keras-radam
", "rawMarkdown": "New optimizer - RAdam\n\n\nhttps://medium.com/@lessw/new-state-of-the-art-ai-optimizer-rectified-adam-radam-5d854730807b\n\nhttps://arxiv.org/abs/1908.03265v1\n\nhttps://github.com/LiyuanLucasLiu/RAdam/blob/master/cifar_imagenet/utils/radam.py\n\nKeras implementaion: https://github.com/CyberZHG/keras-radam", "votes": null }, { "id": "600619", "postDate": "08/16/2019 11:18:38", "content": "Thanks for sharing. There is a keras implementration here https://pypi.org/project/keras-radam/
", "rawMarkdown": "Thanks for sharing. There is a keras implementration here https://pypi.org/project/keras-radam/", "votes": null }, { "id": "600643", "postDate": "08/16/2019 11:57:34", "content": "Thanks. Did you try it yourself?
", "rawMarkdown": "Thanks. Did you try it yourself?", "votes": null }, { "id": "600646", "postDate": "08/16/2019 12:03:15", "content": "Thank you!\nIs there a TF implementation?
", "rawMarkdown": "Thank you!\nIs there a TF implementation?", "votes": null }, { "id": "600654", "postDate": "08/16/2019 12:18:48", "content": "Thanks for sharing. Will try to implement this and looking forward to seeing your kernel.
", "rawMarkdown": "Thanks for sharing. Will try to implement this and looking forward to seeing your kernel.", "votes": null }, { "id": "600698", "postDate": "08/16/2019 13:16:36", "content": "I saw only keras and pytorch implementations
", "rawMarkdown": "I saw only keras and pytorch implementations", "votes": null }, { "id": "600699", "postDate": "08/16/2019 13:17:28", "content": "Maybe after competition ends or in some other competition :)
", "rawMarkdown": "Maybe after competition ends or in some other competition :)", "votes": null }, { "id": "600700", "postDate": "08/16/2019 13:18:10", "content": "Yes, I tried.\nIt gave some results ;)
", "rawMarkdown": "Yes, I tried.\nIt gave some results ;)", "votes": null }, { "id": "600752", "postDate": "08/16/2019 14:14:00", "content": "Thanks for sharing @artgor . Did you realize this result in your works ? 💯
", "rawMarkdown": "Thanks for sharing @artgor . Did you realize this result in your works ? 💯", "votes": null }, { "id": "600755", "postDate": "08/16/2019 14:18:50", "content": "Yes, I have already tried it.\nIt gave some results ;)
", "rawMarkdown": "Yes, I have already tried it.\nIt gave some results ;)", "votes": null }, { "id": "600834", "postDate": "08/16/2019 16:09:15", "content": "Thank you for you information. This remained me that we need to keep watching and trying SOTA papers until the deadline.
", "rawMarkdown": "Thank you for you information. This remained me that we need to keep watching and trying SOTA papers until the deadline.", "votes": null }, { "id": "600881", "postDate": "08/16/2019 17:31:39", "content": "Thanks for Sharing...!! @artgor
", "rawMarkdown": "Thanks for Sharing...!! @artgor", "votes": null }, { "id": "601039", "postDate": "08/16/2019 23:31:44", "content": "radams implementation in keras : https://pypi.org/project/keras-rectified-adam/
\n\ngithub : https://github.com/CyberZHG/keras-radam
", "rawMarkdown": "radams implementation in keras : https://pypi.org/project/keras-rectified-adam/\n\ngithub : https://github.com/CyberZHG/keras-radam", "votes": null }, { "id": "601117", "postDate": "08/17/2019 03:30:19", "content": "what changed? i want to know more.
", "rawMarkdown": "what changed? i want to know more.", "votes": null }, { "id": "601124", "postDate": "08/17/2019 03:57:22", "content": "It is insensitive to the learning rate, unlike to its predecessors aka SGD and ADAM
", "rawMarkdown": "It is insensitive to the learning rate, unlike to its predecessors aka SGD and ADAM", "votes": null }, { "id": "601126", "postDate": "08/17/2019 03:59:07", "content": "Pl add this link to the main thread.
", "rawMarkdown": "Pl add this link to the main thread.", "votes": null }, { "id": "601215", "postDate": "08/17/2019 07:49:52", "content": "What happens at epoch 80? Are they using RAdam + a learning rate scheduler (looks like a LR drop)? if so isn't this against the main promise of RAdam?
", "rawMarkdown": "What happens at epoch 80? Are they using RAdam + a learning rate scheduler (looks like a LR drop)? if so isn't this against the main promise of RAdam?", "votes": null }, { "id": "601429", "postDate": "08/17/2019 15:42:58", "content": "The last one paper's title tells me : A new optimizer beat Adam which named LookAhead will dominate the world . I don't trust media anymore , unless their code in github over 1000 stars .
Thanks for sharing ! Did anyone try this for APTOS?
", "rawMarkdown": "Thanks for sharing ! Did anyone try this for APTOS?", "votes": null }, { "id": "602819", "postDate": "08/19/2019 14:24:09", "content": "Certainly! It's great!
\n\nLink to kernel:\nhttps://www.kaggle.com/carlolepelaars/efficientnetb5-with-keras-aptos-2019?scriptVersionId=19115324
", "rawMarkdown": "Certainly! It's great!\n\nLink to kernel:\nhttps://www.kaggle.com/carlolepelaars/efficientnetb5-with-keras-aptos-2019?scriptVersionId=19115324", "votes": null }, { "id": "602918", "postDate": "08/19/2019 16:43:13", "content": "thanks for sharing \nI m beginner and it upgraded my skills
", "rawMarkdown": "thanks for sharing \nI m beginner and it upgraded my skills", "votes": null }, { "id": "603508", "postDate": "08/20/2019 11:37:05", "content": "Thanks, it looks promising !!
", "rawMarkdown": "Thanks, it looks promising !!", "votes": null }, { "id": "604339", "postDate": "08/21/2019 09:26:09", "content": "Using RAdam
\n\n
😂
", "rawMarkdown": "😂", "votes": null }, { "id": "604474", "postDate": "08/21/2019 12:40:14", "content": "Thanks for sharing!! @artgor
", "rawMarkdown": "Thanks for sharing!! @artgor", "votes": null }, { "id": "604740", "postDate": "08/21/2019 18:13:46", "content": "@christofhenkel why is it distasteful I'm seriously curious?
", "rawMarkdown": "christofhenkel why is it distasteful I'm seriously curious?", "votes": null }, { "id": "605189", "postDate": "08/22/2019 06:48:11", "content": "Try it ;)
", "rawMarkdown": "Try it ;)", "votes": null }, { "id": "605348", "postDate": "08/22/2019 09:27:27", "content": "@cpmpml @christofhenkel any vague figure to share on how much you'd lose using RAdam instead of Adam? (I've finally managed to make my GCN kind of work ~-1.9 and I'm using RAdam, just to know what I can expect by switching back to Adam)
", "rawMarkdown": "cpmpml @christofhenkel any vague figure to share on how much you'd lose using RAdam instead of Adam? (I've finally managed to make my GCN kind of work ~-1.9 and I'm using RAdam, just to know what I can expect by switching back to Adam)", "votes": null }, { "id": "605363", "postDate": "08/22/2019 09:54:09", "content": "\n\n\njust to know what I can expect by switching back to Adam)
\n
There is only one way to know: do it.
\n\nNot being mean, the difference depends on what optimizer you use, what NN architecture you use, what loss you use, what learning rate you use, etc.
", "rawMarkdown": "> just to know what I can expect by switching back to Adam)\n\nThere is only one way to know: do it.\n\nNot being mean, the difference depends on what optimizer you use, what NN architecture you use, what loss you use, what learning rate you use, etc.", "votes": null }, { "id": "605411", "postDate": "08/22/2019 11:06:33", "content": "I knew that you'd answer this, you could have shared one specific experiment of yours, at least I tried! :)\n(I'll share the difference if I do make the test)
", "rawMarkdown": "I knew that you'd answer this, you could have shared one specific experiment of yours, at least I tried! :)\n(I'll share the difference if I do make the test)", "votes": null }, { "id": "611562", "postDate": "08/29/2019 10:20:19", "content": "Now that the comp is over can anyone share whether they had success with RAdam?
\n\nI tried it only quickly with one run and it gave me worse results. I read somewhere that increasing the learning rate (to over x100) for RAdam helps but didn't have time to try it.
", "rawMarkdown": "Now that the comp is over can anyone share whether they had success with RAdam? \n\nI tried it only quickly with one run and it gave me worse results. I read somewhere that increasing the learning rate (to over x100) for RAdam helps but didn't have time to try it.", "votes": null }, { "id": "611583", "postDate": "08/29/2019 10:29:13", "content": "Haven't tried 😂
", "rawMarkdown": "Haven't tried 😂", "votes": null }, { "id": "611663", "postDate": "08/29/2019 11:26:28", "content": "RAdam didn't work at all.
\n\nRanger worked in one of my experiments, but we didn't have enough time/resources to use it.
", "rawMarkdown": "RAdam didn't work at all.\n\nRanger worked in one of my experiments, but we didn't have enough time/resources to use it.", "votes": null }, { "id": "611686", "postDate": "08/29/2019 11:40:07", "content": "And let me add the reason why I did not try. In this competition design of architecture was way more important than training details like optimizer, lr schedule, batch size etc. So I spent my time on things that have a potential high impact on score, and since Adam worked pretty well I just didn't think a different optimizer would have much impact
", "rawMarkdown": "And let me add the reason why I did not try. In this competition design of architecture was way more important than training details like optimizer, lr schedule, batch size etc. So I spent my time on things that have a potential high impact on score, and since Adam worked pretty well I just didn't think a different optimizer would have much impact", "votes": null }, { "id": "611763", "postDate": "08/29/2019 12:23:47", "content": "Makes sense. I just hoped it might speed the training up as experimenting with architectures was very time-consumung.
", "rawMarkdown": "Makes sense. I just hoped it might speed the training up as experimenting with architectures was very time-consumung.", "votes": null }, { "id": "611789", "postDate": "08/29/2019 12:45:41", "content": "thanks for sharing, I'm going to give it try over the weekend.
", "rawMarkdown": "thanks for sharing, I'm going to give it try over the weekend.", "votes": null }, { "id": "670271", "postDate": "11/11/2019 08:48:14", "content": "i tried to implement it, but when i run my model, the loss doesn't change from the first epoch till the last.
\n\nI put it in like that in my code:
\n\nmodel.compile(loss='mean_squared_error', optimizer=RAdam())
\n\ni also did the \"warmup\"-process, but still nothing changed.
\n\nI also tried the example:\nimport keras\nimport numpy as np\nfrom keras_radam import RAdam
\n\nmodel = keras.models.Sequential()\nmodel.add(keras.layers.Dense(input_shape=(17,), units=3))\nmodel.compile(RAdam(), loss='mse')
\n\nx = np.random.standard_normal((4096 * 30, 17))\nw = np.random.standard_normal((17, 3))\ny = np.dot(x, w)
\n\nmodel.fit(x, y, epochs=5)
\n\nAt each epoche i get loss: 24.5919.
\n\nAnyone got an idea?
", "rawMarkdown": "i tried to implement it, but when i run my model, the loss doesn't change from the first epoch till the last.\n\nI put it in like that in my code:\n\nmodel.compile(loss='mean_squared_error', optimizer=RAdam())\n\ni also did the \"warmup\"-process, but still nothing changed.\n\nI also tried the example:\nimport keras\nimport numpy as np\nfrom keras_radam import RAdam\n\n# Build toy model with RAdam optimizer\nmodel = keras.models.Sequential()\nmodel.add(keras.layers.Dense(input_shape=(17,), units=3))\nmodel.compile(RAdam(), loss='mse')\n\n# Generate toy data\nx = np.random.standard_normal((4096 * 30, 17))\nw = np.random.standard_normal((17, 3))\ny = np.dot(x, w)\n\n# Fit\nmodel.fit(x, y, epochs=5)\n\nAt each epoche i get loss: 24.5919.\n\nAnyone got an idea?", "votes": null }, { "id": "684116", "postDate": "11/29/2019 08:59:45", "content": "pytorch implementation: RAdam + lookahead : https://github.com/nachiket273/lookahead_pytorch
", "rawMarkdown": "pytorch implementation: RAdam + lookahead : https://github.com/nachiket273/lookahead_pytorch", "votes": null } ], "comments": [ { "id": 600619, "author_name": "cpmpml", "author_url": "", "post_date": "08/16/2019 11:18:38", "content": "Thanks for sharing. There is a keras implementration here https://pypi.org/project/keras-radam/
", "votes": null, "replies": [ { "id": 601126, "author_name": "pankaj1234", "author_url": "", "post_date": "08/17/2019 03:59:07", "content": "Pl add this link to the main thread.
", "votes": null, "replies": [] } ] }, { "id": 600643, "author_name": "aerdem4", "author_url": "", "post_date": "08/16/2019 11:57:34", "content": "Thanks. Did you try it yourself?
", "votes": null, "replies": [ { "id": 600700, "author_name": "artgor", "author_url": "", "post_date": "08/16/2019 13:18:10", "content": "Yes, I tried.\nIt gave some results ;)
", "votes": null, "replies": [] }, { "id": 600834, "author_name": "toshik", "author_url": "", "post_date": "08/16/2019 16:09:15", "content": "Thank you for you information. This remained me that we need to keep watching and trying SOTA papers until the deadline.
", "votes": null, "replies": [] } ] }, { "id": 600646, "author_name": "stanislavblinov", "author_url": "", "post_date": "08/16/2019 12:03:15", "content": "Thank you!\nIs there a TF implementation?
", "votes": null, "replies": [ { "id": 600698, "author_name": "artgor", "author_url": "", "post_date": "08/16/2019 13:16:36", "content": "I saw only keras and pytorch implementations
", "votes": null, "replies": [] } ] }, { "id": 600654, "author_name": "shwetagoyal4", "author_url": "", "post_date": "08/16/2019 12:18:48", "content": "Thanks for sharing. Will try to implement this and looking forward to seeing your kernel.
", "votes": null, "replies": [ { "id": 600699, "author_name": "artgor", "author_url": "", "post_date": "08/16/2019 13:17:28", "content": "Maybe after competition ends or in some other competition :)
", "votes": null, "replies": [] } ] }, { "id": 600752, "author_name": "utkukubilay", "author_url": "", "post_date": "08/16/2019 14:14:00", "content": "Thanks for sharing @artgor . Did you realize this result in your works ? 💯
", "votes": null, "replies": [ { "id": 600755, "author_name": "artgor", "author_url": "", "post_date": "08/16/2019 14:18:50", "content": "Yes, I have already tried it.\nIt gave some results ;)
", "votes": null, "replies": [] } ] }, { "id": 600881, "author_name": "veeralakrishna", "author_url": "", "post_date": "08/16/2019 17:31:39", "content": "Thanks for Sharing...!! @artgor
", "votes": null, "replies": [] }, { "id": 601039, "author_name": "mobassir", "author_url": "", "post_date": "08/16/2019 23:31:44", "content": "radams implementation in keras : https://pypi.org/project/keras-rectified-adam/
\n\ngithub : https://github.com/CyberZHG/keras-radam
", "votes": null, "replies": [] }, { "id": 601117, "author_name": "topchaebol", "author_url": "", "post_date": "08/17/2019 03:30:19", "content": "what changed? i want to know more.
", "votes": null, "replies": [ { "id": 601124, "author_name": "pankaj1234", "author_url": "", "post_date": "08/17/2019 03:57:22", "content": "It is insensitive to the learning rate, unlike to its predecessors aka SGD and ADAM
", "votes": null, "replies": [] } ] }, { "id": 601215, "author_name": "optimo", "author_url": "", "post_date": "08/17/2019 07:49:52", "content": "What happens at epoch 80? Are they using RAdam + a learning rate scheduler (looks like a LR drop)? if so isn't this against the main promise of RAdam?
", "votes": null, "replies": [] }, { "id": 601429, "author_name": "xujingzhao", "author_url": "", "post_date": "08/17/2019 15:42:58", "content": "The last one paper's title tells me : A new optimizer beat Adam which named LookAhead will dominate the world . I don't trust media anymore , unless their code in github over 1000 stars .
Thanks for sharing ! Did anyone try this for APTOS?
", "votes": null, "replies": [ { "id": 602819, "author_name": "carlolepelaars", "author_url": "", "post_date": "08/19/2019 14:24:09", "content": "Certainly! It's great!
\n\nLink to kernel:\nhttps://www.kaggle.com/carlolepelaars/efficientnetb5-with-keras-aptos-2019?scriptVersionId=19115324
", "votes": null, "replies": [] } ] }, { "id": 602918, "author_name": "niteshfre", "author_url": "", "post_date": "08/19/2019 16:43:13", "content": "thanks for sharing \nI m beginner and it upgraded my skills
", "votes": null, "replies": [] }, { "id": 603508, "author_name": "nhlr21", "author_url": "", "post_date": "08/20/2019 11:37:05", "content": "Thanks, it looks promising !!
", "votes": null, "replies": [] }, { "id": 604339, "author_name": "christofhenkel", "author_url": "", "post_date": "08/21/2019 09:26:09", "content": "Using RAdam
\n\n
😂
", "votes": null, "replies": [] }, { "id": 604740, "author_name": "michelml", "author_url": "", "post_date": "08/21/2019 18:13:46", "content": "@christofhenkel why is it distasteful I'm seriously curious?
", "votes": null, "replies": [] }, { "id": 605189, "author_name": "cpmpml", "author_url": "", "post_date": "08/22/2019 06:48:11", "content": "Try it ;)
", "votes": null, "replies": [] }, { "id": 605348, "author_name": "optimo", "author_url": "", "post_date": "08/22/2019 09:27:27", "content": "@cpmpml @christofhenkel any vague figure to share on how much you'd lose using RAdam instead of Adam? (I've finally managed to make my GCN kind of work ~-1.9 and I'm using RAdam, just to know what I can expect by switching back to Adam)
", "votes": null, "replies": [] }, { "id": 605363, "author_name": "cpmpml", "author_url": "", "post_date": "08/22/2019 09:54:09", "content": "\n\n\njust to know what I can expect by switching back to Adam)
\n
There is only one way to know: do it.
\n\nNot being mean, the difference depends on what optimizer you use, what NN architecture you use, what loss you use, what learning rate you use, etc.
", "votes": null, "replies": [] }, { "id": 605411, "author_name": "optimo", "author_url": "", "post_date": "08/22/2019 11:06:33", "content": "I knew that you'd answer this, you could have shared one specific experiment of yours, at least I tried! :)\n(I'll share the difference if I do make the test)
", "votes": null, "replies": [] } ] }, { "id": 604474, "author_name": "dhruvk090", "author_url": "", "post_date": "08/21/2019 12:40:14", "content": "Thanks for sharing!! @artgor
", "votes": null, "replies": [] }, { "id": 611562, "author_name": "taniaj", "author_url": "", "post_date": "08/29/2019 10:20:19", "content": "Now that the comp is over can anyone share whether they had success with RAdam?
\n\nI tried it only quickly with one run and it gave me worse results. I read somewhere that increasing the learning rate (to over x100) for RAdam helps but didn't have time to try it.
", "votes": null, "replies": [ { "id": 611583, "author_name": "christofhenkel", "author_url": "", "post_date": "08/29/2019 10:29:13", "content": "Haven't tried 😂
", "votes": null, "replies": [] }, { "id": 611663, "author_name": "artgor", "author_url": "", "post_date": "08/29/2019 11:26:28", "content": "RAdam didn't work at all.
\n\nRanger worked in one of my experiments, but we didn't have enough time/resources to use it.
", "votes": null, "replies": [] }, { "id": 611686, "author_name": "christofhenkel", "author_url": "", "post_date": "08/29/2019 11:40:07", "content": "And let me add the reason why I did not try. In this competition design of architecture was way more important than training details like optimizer, lr schedule, batch size etc. So I spent my time on things that have a potential high impact on score, and since Adam worked pretty well I just didn't think a different optimizer would have much impact
", "votes": null, "replies": [] }, { "id": 611763, "author_name": "taniaj", "author_url": "", "post_date": "08/29/2019 12:23:47", "content": "Makes sense. I just hoped it might speed the training up as experimenting with architectures was very time-consumung.
", "votes": null, "replies": [] } ] }, { "id": 611789, "author_name": "nachiket273", "author_url": "", "post_date": "08/29/2019 12:45:41", "content": "thanks for sharing, I'm going to give it try over the weekend.
", "votes": null, "replies": [ { "id": 684116, "author_name": "nachiket273", "author_url": "", "post_date": "11/29/2019 08:59:45", "content": "pytorch implementation: RAdam + lookahead : https://github.com/nachiket273/lookahead_pytorch
", "votes": null, "replies": [] } ] }, { "id": 670271, "author_name": "moeawahid", "author_url": "", "post_date": "11/11/2019 08:48:14", "content": "i tried to implement it, but when i run my model, the loss doesn't change from the first epoch till the last.
\n\nI put it in like that in my code:
\n\nmodel.compile(loss='mean_squared_error', optimizer=RAdam())
\n\ni also did the \"warmup\"-process, but still nothing changed.
\n\nI also tried the example:\nimport keras\nimport numpy as np\nfrom keras_radam import RAdam
\n\nmodel = keras.models.Sequential()\nmodel.add(keras.layers.Dense(input_shape=(17,), units=3))\nmodel.compile(RAdam(), loss='mse')
\n\nx = np.random.standard_normal((4096 * 30, 17))\nw = np.random.standard_normal((17, 3))\ny = np.dot(x, w)
\n\nmodel.fit(x, y, epochs=5)
\n\nAt each epoche i get loss: 24.5919.
\n\nAnyone got an idea?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "600615": "New optimizer - RAdam\n\n\nhttps://medium.com/@lessw/new-state-of-the-art-ai-optimizer-rectified-adam-radam-5d854730807b\n\nhttps://arxiv.org/abs/1908.03265v1\n\nhttps://github.com/LiyuanLucasLiu/RAdam/blob/master/cifar_imagenet/utils/radam.py\n\nKeras implementaion: https://github.com/CyberZHG/keras-radam", "600619": "Thanks for sharing. There is a keras implementration here https://pypi.org/project/keras-radam/", "600643": "Thanks. Did you try it yourself?", "600646": "Thank you!\nIs there a TF implementation?", "600654": "Thanks for sharing. Will try to implement this and looking forward to seeing your kernel.", "600698": "I saw only keras and pytorch implementations", "600699": "Maybe after competition ends or in some other competition :)", "600700": "Yes, I tried.\nIt gave some results ;)", "600752": "Thanks for sharing @artgor . Did you realize this result in your works ? 💯", "600755": "Yes, I have already tried it.\nIt gave some results ;)", "600834": "Thank you for you information. This remained me that we need to keep watching and trying SOTA papers until the deadline.", "600881": "Thanks for Sharing...!! @artgor", "601039": "radams implementation in keras : https://pypi.org/project/keras-rectified-adam/\n\ngithub : https://github.com/CyberZHG/keras-radam", "601117": "what changed? i want to know more.", "601124": "It is insensitive to the learning rate, unlike to its predecessors aka SGD and ADAM", "601126": "Pl add this link to the main thread.", "601215": "What happens at epoch 80? Are they using RAdam + a learning rate scheduler (looks like a LR drop)? if so isn't this against the main promise of RAdam?", "601429": "The last one paper's title tells me : `A new optimizer beat Adam which named LookAhead will dominate the world ` . I don't trust media anymore , unless their code in github over 1000 stars .", "601879": "Thanks for sharing ! Did anyone try this for APTOS?", "602819": "Certainly! It's great!\n\nLink to kernel:\nhttps://www.kaggle.com/carlolepelaars/efficientnetb5-with-keras-aptos-2019?scriptVersionId=19115324", "602918": "thanks for sharing \nI m beginner and it upgraded my skills", "603508": "Thanks, it looks promising !!", "604339": "Using RAdam\n\n", "604350": "😂", "604474": "Thanks for sharing!! @artgor", "604740": "christofhenkel why is it distasteful I'm seriously curious?", "605189": "Try it ;)", "605348": "cpmpml @christofhenkel any vague figure to share on how much you'd lose using RAdam instead of Adam? (I've finally managed to make my GCN kind of work ~-1.9 and I'm using RAdam, just to know what I can expect by switching back to Adam)", "605363": "> just to know what I can expect by switching back to Adam)\n\nThere is only one way to know: do it.\n\nNot being mean, the difference depends on what optimizer you use, what NN architecture you use, what loss you use, what learning rate you use, etc.", "605411": "I knew that you'd answer this, you could have shared one specific experiment of yours, at least I tried! :)\n(I'll share the difference if I do make the test)", "611562": "Now that the comp is over can anyone share whether they had success with RAdam? \n\nI tried it only quickly with one run and it gave me worse results. I read somewhere that increasing the learning rate (to over x100) for RAdam helps but didn't have time to try it.", "611583": "Haven't tried 😂", "611663": "RAdam didn't work at all.\n\nRanger worked in one of my experiments, but we didn't have enough time/resources to use it.", "611686": "And let me add the reason why I did not try. In this competition design of architecture was way more important than training details like optimizer, lr schedule, batch size etc. So I spent my time on things that have a potential high impact on score, and since Adam worked pretty well I just didn't think a different optimizer would have much impact", "611763": "Makes sense. I just hoped it might speed the training up as experimenting with architectures was very time-consumung.", "611789": "thanks for sharing, I'm going to give it try over the weekend.", "670271": "i tried to implement it, but when i run my model, the loss doesn't change from the first epoch till the last.\n\nI put it in like that in my code:\n\nmodel.compile(loss='mean_squared_error', optimizer=RAdam())\n\ni also did the \"warmup\"-process, but still nothing changed.\n\nI also tried the example:\nimport keras\nimport numpy as np\nfrom keras_radam import RAdam\n\n# Build toy model with RAdam optimizer\nmodel = keras.models.Sequential()\nmodel.add(keras.layers.Dense(input_shape=(17,), units=3))\nmodel.compile(RAdam(), loss='mse')\n\n# Generate toy data\nx = np.random.standard_normal((4096 * 30, 17))\nw = np.random.standard_normal((17, 3))\ny = np.dot(x, w)\n\n# Fit\nmodel.fit(x, y, epochs=5)\n\nAt each epoche i get loss: 24.5919.\n\nAnyone got an idea?", "684116": "pytorch implementation: RAdam + lookahead : https://github.com/nachiket273/lookahead_pytorch" }, "source": "meta" }