Outlier insensitive loss ! such loss functions are destined to be highly non-convex.The high-level story is that we can't hope to achieve robust recovery by such simple methods of alternating loss functions: there's no free lunch here.
\n\nMore info:\n- https://math.stackexchange.com/questions/2283165/proper-loss-function-for-this-robust-regression-problem\n- https://arxiv.org/pdf/1702.07709.pdf
", "messages": [ { "id": "477875", "postDate": "02/25/2019 12:06:47", "content": "Outlier insensitive loss ! such loss functions are destined to be highly non-convex.The high-level story is that we can't hope to achieve robust recovery by such simple methods of alternating loss functions: there's no free lunch here.
\n\nMore info:\n- https://math.stackexchange.com/questions/2283165/proper-loss-function-for-this-robust-regression-problem\n- https://arxiv.org/pdf/1702.07709.pdf
", "rawMarkdown": "Outlier insensitive loss ! such loss functions are destined to be highly non-convex.The high-level story is that we can't hope to achieve robust recovery by such simple methods of alternating loss functions: there's no free lunch here. \n\nMore info:\n- https://math.stackexchange.com/questions/2283165/proper-loss-function-for-this-robust-regression-problem\n- https://arxiv.org/pdf/1702.07709.pdf", "votes": null }, { "id": "478205", "postDate": "02/25/2019 22:05:13", "content": "did you manage to implement it lightgbm ?
", "rawMarkdown": "did you manage to implement it lightgbm ?", "votes": null }, { "id": "478230", "postDate": "02/25/2019 22:51:35", "content": "I am sure some people applied custom loss on lightgbm in this competition , at least tried to do. But if we could implement proper loss function for this robust regression problem the best RMSE in LB should be 1.5xxx, but as we see it never happened. This problem has a long history in mathematics if you search, and required different framework based on \"Convex Optimization\" , and please note , this is just my high level hypothesis.
", "rawMarkdown": "I am sure some people applied custom loss on lightgbm in this competition , at least tried to do. But if we could implement proper loss function for this robust regression problem the best RMSE in LB should be 1.5xxx, but as we see it never happened. This problem has a long history in mathematics if you search, and required different framework based on \"Convex Optimization\" , and please note , this is just my high level hypothesis.", "votes": null }, { "id": "478240", "postDate": "02/25/2019 23:23:13", "content": "Sorry, I really don't follow your logic ...
\n\nHow could an outlier insensitive loss function able to drive RMSE evaluation metric to be 1.5xxx? That is the case when the RMSE evaluation metric can magically become insensitive of outliers.
\n\nWith an outlier insensitive loss function, the model won't have a chance to realize for some card_id the ideal outputs are large negative numbers. Thus it is guaranteed to do poorly on the 1000+ outliers in the test set with an RMSE in the 3.xx region. I haven't tested it myself, but I think Huber loss is likely to do worse than RMSE loss here.
\n\nI guess you were trying to suggest that this competition should use an outlier insensitive evaluation metric?
", "rawMarkdown": "Sorry, I really don't follow your logic ...\n\nHow could an outlier insensitive loss function able to drive RMSE evaluation metric to be 1.5xxx? That is the case when the RMSE evaluation metric can magically become insensitive of outliers.\n\nWith an outlier insensitive loss function, the model won't have a chance to realize for some card_id the ideal outputs are large negative numbers. Thus it is guaranteed to do poorly on the 1000+ outliers in the test set with an RMSE in the 3.xx region. I haven't tested it myself, but I think Huber loss is likely to do worse than RMSE loss here. \n\nI guess you were trying to suggest that this competition should use an outlier insensitive evaluation metric?", "votes": null }, { "id": "478261", "postDate": "02/26/2019 00:19:23", "content": "Thanks @Ren, \nYour approach about insensitivity of RMSE to outliers is correct , but the degree of insensitivity depends on data, if you change outliers values to -10 for example rmse could be a proper metric. But these oultliers never would like to be trained because objective function is not proper. Also if you test huber(as objective for example in lightgbm) there is no difference.
", "rawMarkdown": "Thanks @Ren, \nYour approach about insensitivity of RMSE to outliers is correct , but the degree of insensitivity depends on data, if you change outliers values to -10 for example rmse could be a proper metric. But these oultliers never would like to be trained because objective function is not proper. Also if you test huber(as objective for example in lightgbm) there is no difference.", "votes": null }, { "id": "478265", "postDate": "02/26/2019 00:41:41", "content": "Hey @Möbius
\n\nBased on your reply, I sense more that you are not differentiate the different role between loss function and evalutation metric.
", "rawMarkdown": "Hey [@Möbius](https://www.kaggle.com/arashnic)\n\nBased on your reply, I sense more that you are not differentiate the different role between loss function and evalutation metric.", "votes": null }, { "id": "478270", "postDate": "02/26/2019 00:49:05", "content": "I tried couple of custom loss function as mentioned here : https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/78470#464014, but they didn’t help.
", "rawMarkdown": "I tried couple of custom loss function as mentioned here : https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/78470#464014, but they didn’t help.", "votes": null }, { "id": "478274", "postDate": "02/26/2019 00:55:29", "content": "Dear @Ren , Your opinion is yours, but this link helps you to understand my 'Topic ' of discussion all the topic is about loss function. Thanks for your comments\nhttps://stats.stackexchange.com/questions/379264/loss-function-and-evaluation-metric
", "rawMarkdown": "Dear @Ren , Your opinion is yours, but this link helps you to understand my 'Topic ' of discussion all the topic is about loss function. Thanks for your comments\nhttps://stats.stackexchange.com/questions/379264/loss-function-and-evaluation-metric", "votes": null }, { "id": "478290", "postDate": "02/26/2019 01:24:51", "content": "Your loss function is even more sensitive to outliers then l2. I guess, unless the test set is proved to have significantly more outliers, I don't think it would help.
", "rawMarkdown": "Your loss function is even more sensitive to outliers then l2. I guess, unless the test set is proved to have significantly more outliers, I don't think it would help.", "votes": null }, { "id": "478291", "postDate": "02/26/2019 01:34:10", "content": "What is the purpose of the link? You want me to read it up to learn the difference between loss function and evaluation function? Then just say it. Explicit is better than implicit. I don't want to guess.
", "rawMarkdown": "What is the purpose of the link? You want me to read it up to learn the difference between loss function and evaluation function? Then just say it. Explicit is better than implicit. I don't want to guess.", "votes": null } ], "comments": [ { "id": 478205, "author_name": "mks2192", "author_url": "", "post_date": "02/25/2019 22:05:13", "content": "did you manage to implement it lightgbm ?
", "votes": null, "replies": [ { "id": 478230, "author_name": "arashnic", "author_url": "", "post_date": "02/25/2019 22:51:35", "content": "I am sure some people applied custom loss on lightgbm in this competition , at least tried to do. But if we could implement proper loss function for this robust regression problem the best RMSE in LB should be 1.5xxx, but as we see it never happened. This problem has a long history in mathematics if you search, and required different framework based on \"Convex Optimization\" , and please note , this is just my high level hypothesis.
", "votes": null, "replies": [] }, { "id": 478240, "author_name": "ryanzhang", "author_url": "", "post_date": "02/25/2019 23:23:13", "content": "Sorry, I really don't follow your logic ...
\n\nHow could an outlier insensitive loss function able to drive RMSE evaluation metric to be 1.5xxx? That is the case when the RMSE evaluation metric can magically become insensitive of outliers.
\n\nWith an outlier insensitive loss function, the model won't have a chance to realize for some card_id the ideal outputs are large negative numbers. Thus it is guaranteed to do poorly on the 1000+ outliers in the test set with an RMSE in the 3.xx region. I haven't tested it myself, but I think Huber loss is likely to do worse than RMSE loss here.
\n\nI guess you were trying to suggest that this competition should use an outlier insensitive evaluation metric?
", "votes": null, "replies": [] }, { "id": 478261, "author_name": "arashnic", "author_url": "", "post_date": "02/26/2019 00:19:23", "content": "Thanks @Ren, \nYour approach about insensitivity of RMSE to outliers is correct , but the degree of insensitivity depends on data, if you change outliers values to -10 for example rmse could be a proper metric. But these oultliers never would like to be trained because objective function is not proper. Also if you test huber(as objective for example in lightgbm) there is no difference.
", "votes": null, "replies": [] }, { "id": 478265, "author_name": "ryanzhang", "author_url": "", "post_date": "02/26/2019 00:41:41", "content": "Hey @Möbius
\n\nBased on your reply, I sense more that you are not differentiate the different role between loss function and evalutation metric.
", "votes": null, "replies": [] }, { "id": 478270, "author_name": "harshit92", "author_url": "", "post_date": "02/26/2019 00:49:05", "content": "I tried couple of custom loss function as mentioned here : https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/78470#464014, but they didn’t help.
", "votes": null, "replies": [] }, { "id": 478274, "author_name": "arashnic", "author_url": "", "post_date": "02/26/2019 00:55:29", "content": "Dear @Ren , Your opinion is yours, but this link helps you to understand my 'Topic ' of discussion all the topic is about loss function. Thanks for your comments\nhttps://stats.stackexchange.com/questions/379264/loss-function-and-evaluation-metric
", "votes": null, "replies": [] }, { "id": 478290, "author_name": "ryanzhang", "author_url": "", "post_date": "02/26/2019 01:24:51", "content": "Your loss function is even more sensitive to outliers then l2. I guess, unless the test set is proved to have significantly more outliers, I don't think it would help.
", "votes": null, "replies": [] }, { "id": 478291, "author_name": "ryanzhang", "author_url": "", "post_date": "02/26/2019 01:34:10", "content": "What is the purpose of the link? You want me to read it up to learn the difference between loss function and evaluation function? Then just say it. Explicit is better than implicit. I don't want to guess.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "477875": "Outlier insensitive loss ! such loss functions are destined to be highly non-convex.The high-level story is that we can't hope to achieve robust recovery by such simple methods of alternating loss functions: there's no free lunch here. \n\nMore info:\n- https://math.stackexchange.com/questions/2283165/proper-loss-function-for-this-robust-regression-problem\n- https://arxiv.org/pdf/1702.07709.pdf", "478205": "did you manage to implement it lightgbm ?", "478230": "I am sure some people applied custom loss on lightgbm in this competition , at least tried to do. But if we could implement proper loss function for this robust regression problem the best RMSE in LB should be 1.5xxx, but as we see it never happened. This problem has a long history in mathematics if you search, and required different framework based on \"Convex Optimization\" , and please note , this is just my high level hypothesis.", "478240": "Sorry, I really don't follow your logic ...\n\nHow could an outlier insensitive loss function able to drive RMSE evaluation metric to be 1.5xxx? That is the case when the RMSE evaluation metric can magically become insensitive of outliers.\n\nWith an outlier insensitive loss function, the model won't have a chance to realize for some card_id the ideal outputs are large negative numbers. Thus it is guaranteed to do poorly on the 1000+ outliers in the test set with an RMSE in the 3.xx region. I haven't tested it myself, but I think Huber loss is likely to do worse than RMSE loss here. \n\nI guess you were trying to suggest that this competition should use an outlier insensitive evaluation metric?", "478261": "Thanks @Ren, \nYour approach about insensitivity of RMSE to outliers is correct , but the degree of insensitivity depends on data, if you change outliers values to -10 for example rmse could be a proper metric. But these oultliers never would like to be trained because objective function is not proper. Also if you test huber(as objective for example in lightgbm) there is no difference.", "478265": "Hey [@Möbius](https://www.kaggle.com/arashnic)\n\nBased on your reply, I sense more that you are not differentiate the different role between loss function and evalutation metric.", "478270": "I tried couple of custom loss function as mentioned here : https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/78470#464014, but they didn’t help.", "478274": "Dear @Ren , Your opinion is yours, but this link helps you to understand my 'Topic ' of discussion all the topic is about loss function. Thanks for your comments\nhttps://stats.stackexchange.com/questions/379264/loss-function-and-evaluation-metric", "478290": "Your loss function is even more sensitive to outliers then l2. I guess, unless the test set is proved to have significantly more outliers, I don't think it would help.", "478291": "What is the purpose of the link? You want me to read it up to learn the difference between loss function and evaluation function? Then just say it. Explicit is better than implicit. I don't want to guess." }, "source": "meta" }