Do we have some strategies to handle outliers when doing regression, if the outliers is important to us?
", "messages": [ { "id": 460680, "postDate": "2019-01-24T08:31:48.220Z", "content": "Do we have some strategies to handle outliers when doing regression, if the outliers is important to us?
", "rawMarkdown": "Do we have some strategies to handle outliers when doing regression, if the outliers is important to us?", "votes": 18 }, { "id": 460843, "postDate": "2019-01-24T14:28:43.567Z", "content": "If you want to fit more closely to outliers: Change the metric you use for minimizing the sum of the residuals. That is, instead of minimizing the sum of the squared residuals, maybe minimize for the sum of the residuals to the power of 5.
If you want to fit less closely to outliers: Remove the outliers from your sample, or change the the optimization from minimizing the sum of the squared residuals, to minimizing the sum of the L1 residuals maybe. Or minimizing the sum of the Square Root Residuals
I think you can also set weights for each point if these don't work. So, you can specify all non-outlier points have a weight of 1, but the outlier points have a weight of 10.
", "rawMarkdown": "1. If you want to fit more closely to outliers: Change the metric you use for minimizing the sum of the residuals. That is, instead of minimizing the sum of the squared residuals, maybe minimize for the sum of the residuals to the power of 5. \n\n2. If you want to fit less closely to outliers: Remove the outliers from your sample, or change the the optimization from minimizing the sum of the squared residuals, to minimizing the sum of the L1 residuals maybe. Or minimizing the sum of the Square Root Residuals\n\nI think you can also set weights for each point if these don't work. So, you can specify all non-outlier points have a weight of 1, but the outlier points have a weight of 10.", "votes": 16, "replies": [ { "id": 461008, "postDate": "2019-01-25T01:25:11.040Z", "content": "Thanks, very helpful! I'v tried weight for outlier and non-outlier just as you said.
", "rawMarkdown": "Thanks, very helpful! I'v tried weight for outlier and non-outlier just as you said.", "votes": 1 }, { "id": 461238, "postDate": "2019-01-25T15:42:27.517Z", "rawMarkdown": "", "isDeleted": true }, { "id": 461535, "postDate": "2019-01-26T09:31:25.850Z", "content": "@jiazhuang May I know what worked and did not work for you so far with handling outliers?
", "rawMarkdown": "@jiazhuang May I know what worked and did not work for you so far with handling outliers?" }, { "id": 461573, "postDate": "2019-01-26T12:17:03.163Z", "content": "@CoreyLevinson I tried to use the loss function of minimizing for the sum of the residuals to the power of 5 as follows:
\n\ndef custom_asymmetric_train(y_true, y_pred):\n residual = (y_true - y_pred).astype(\"float\")\n grad = -5*residual**4\n hess = 20.0*residual**3\n return grad, hess\n\ndef custom_asymmetric_valid(y_true, y_pred):\n residual = (y_true - y_pred).astype(\"float\")\n loss = residual**5\n return \"custom_asymmetric_eval\", np.mean(loss), False\n\nmodel = LGBMClassifier(boosting_type=\"gbdt\", objective=custom_asymmetric_train）\n\nmodel.fit( X_train, y_train, eval_set=eval_set, eval_metric=custom_asymmetric_valid）\nHowever, the loss does not change for each CV. Any help on this?
", "rawMarkdown": "@CoreyLevinson I tried to use the loss function of minimizing for the sum of the residuals to the power of 5 as follows:\n\n def custom_asymmetric_train(y_true, y_pred):\n residual = (y_true - y_pred).astype(\"float\")\n grad = -5*residual**4\n hess = 20.0*residual**3\n return grad, hess\n\n def custom_asymmetric_valid(y_true, y_pred):\n residual = (y_true - y_pred).astype(\"float\")\n loss = residual**5\n return \"custom_asymmetric_eval\", np.mean(loss), False\n\n model = LGBMClassifier(boosting_type=\"gbdt\", objective=custom_asymmetric_train）\n\n model.fit( X_train, y_train, eval_set=eval_set, eval_metric=custom_asymmetric_valid）\n\nHowever, the loss does not change for each CV. Any help on this?", "votes": 4 }, { "id": 461637, "postDate": "2019-01-26T16:05:50.050Z", "content": "custom_asymmetric_train it is not expecting you to already negate it.Try this, I know in previous competition my custom stuff did not work unless I specified parameters to 'None':
\n\nimport lightgbm as lgb
\n\nparams = {\n 'boosting_type': 'gbdt',\n 'objective': 'None',\n 'metric': 'None',\n}\n\nmodel = lgb.train(params,\n lgb_train, # a LGBM Dataset\n num_boost_round=10,\n fobj=custom_asymmetric_train,\n feval=custom_asymmetric_valid,\n valid_sets=eval_set # Also should be a LGBM Dataset I think\n)\n@CoreyLevinson Thanks! Let me try it.
", "rawMarkdown": "@CoreyLevinson Thanks! Let me try it." }, { "id": 463359, "postDate": "2019-01-29T21:21:43.247Z", "rawMarkdown": "", "isDeleted": true }, { "id": 463844, "postDate": "2019-01-30T17:52:43.940Z", "content": "@xiaojiu1414, @CoreyLevinson\nI tried to use the function above but it throws exception
\n\nTypeError: unsupported operand type(s) for -: 'float' and 'Dataset'
\n\nPrint type() shows that the first parameter (y_true) is a numpy array while the second (y_pred) is a lightGBM dataset.\nAny idea how to convert it to numpy? I found .get_data() function but it does not work either.
", "rawMarkdown": "@xiaojiu1414, @CoreyLevinson\nI tried to use the function above but it throws exception \n\nTypeError: unsupported operand type(s) for -: 'float' and 'Dataset'\n\nPrint type() shows that the first parameter (y_true) is a numpy array while the second (y_pred) is a lightGBM dataset.\nAny idea how to convert it to numpy? I found .get_data() function but it does not work either.", "votes": 1 }, { "id": 464013, "postDate": "2019-01-31T04:01:17.770Z", "content": "That is happening because you are using lightgbm instead of lightgbm scikt learn API
", "rawMarkdown": "That is happening because you are using lightgbm instead of lightgbm scikt learn API", "votes": 2 }, { "id": 464014, "postDate": "2019-01-31T04:02:55.673Z", "content": "You can use the following format:\ndef custom_asymmetric_train(preds, train_data):
\n\ny_true = train_data.get_label()\nresidual = (y_true - preds).astype(\"float\")\ngrad = -4*residual**3\nhess = 12*residual**2\nreturn grad, hess\n`def custom_asymmetric_valid(preds, train_data):
\n\ny_true = train_data.get_label()\nresidual = (y_true - preds).astype(\"float\")\nloss =residual**4\nreturn \"custom_asymmetric_eval\", np.mean(loss), False\nHarshit, much appreciated. It works now.
", "rawMarkdown": "Harshit, much appreciated. It works now. ", "votes": 1 } ] }, { "id": 460768, "postDate": "2019-01-24T11:40:27.340Z", "content": "You can try measuring how much of your regression model is being influenced by that outlier. I wouldn't recommend just erasing it (some people do that)
", "rawMarkdown": "You can try measuring how much of your regression model is being influenced by that outlier. I wouldn't recommend just erasing it (some people do that)", "votes": 1, "replies": [ { "id": 461010, "postDate": "2019-01-25T01:26:27.193Z", "content": "yes, we can't just simply remove it.
", "rawMarkdown": "yes, we can't just simply remove it." } ] }, { "id": 462323, "postDate": "2019-01-28T05:32:51.300Z", "content": "Hi,\n Did you get this solved? I tried power of 5, but the LGB never improved since the first round. Could you share your solution? Thank you!
", "rawMarkdown": "Hi,\n Did you get this solved? I tried power of 5, but the LGB never improved since the first round. Could you share your solution? Thank you!", "replies": [ { "id": 463213, "postDate": "2019-01-29T16:15:56.033Z", "content": "I would try power of 4 or power of 6. It should be even number.
", "rawMarkdown": "I would try power of 4 or power of 6. It should be even number. ", "votes": 1 } ] }, { "id": 460877, "postDate": "2019-01-24T15:32:14.463Z", "content": "I did a split onto two models\n1. LGB model Trained with a binary goal (outlier or not).\n2. LGB Is trained on the dataset without outliers (target<=-33) with all parameters set to the same values.\nResults are joined with. If the model 1 says it should be an outlier, then the predicted score set to -33. Otherwise the score is taken from model 2.\nBut it gave me very bad final score.\nMay be I had to optimize the first model better. Now it predicts correctly only 2-3 outliers out of hundreds we have.
", "rawMarkdown": "I did a split onto two models\n1. LGB model Trained with a binary goal (outlier or not).\n2. LGB Is trained on the dataset without outliers (target<=-33) with all parameters set to the same values.\nResults are joined with. If the model 1 says it should be an outlier, then the predicted score set to -33. Otherwise the score is taken from model 2.\nBut it gave me very bad final score.\nMay be I had to optimize the first model better. Now it predicts correctly only 2-3 outliers out of hundreds we have.", "replies": [ { "id": 461012, "postDate": "2019-01-25T01:29:11.047Z", "content": "yes, outlier classification is highly imbalanced. It's hard to predict them accurately. But It's still a good try.
", "rawMarkdown": "yes, outlier classification is highly imbalanced. It's hard to predict them accurately. But It's still a good try." }, { "id": 461141, "postDate": "2019-01-25T10:44:55.973Z", "content": "@Costa.Medvedev\nUse three model instead. Model with outliers , model without outliers, and model for outliers prediction. Check this kernel :https://www.kaggle.com/waitingli/combining-your-model-with-a-model-without-outlier
", "rawMarkdown": "@Costa.Medvedev\nUse three model instead. Model with outliers , model without outliers, and model for outliers prediction. Check this kernel :https://www.kaggle.com/waitingli/combining-your-model-with-a-model-without-outlier", "votes": 2 }, { "id": 461537, "postDate": "2019-01-26T09:38:57.687Z", "content": "@salamatburj I tried that, but cannot understand why it works. It is using a more general model for the (predicted) outliers than for the non-outliers, meaning that the general model is overly affected by the outlier values for predicting most of the (non-outlying) data. But why would a gbm behave this way? Thanks.
", "rawMarkdown": "@salamatburj I tried that, but cannot understand why it works. It is using a more general model for the (predicted) outliers than for the non-outliers, meaning that the general model is overly affected by the outlier values for predicting most of the (non-outlying) data. But why would a gbm behave this way? Thanks." }, { "id": 461866, "postDate": "2019-01-27T07:43:45Z", "content": "@GideonTeo Yes, outliers are affecting rmse error a lot. I think the key is the outliers prediction. You might have different features which are more important to outliers and less for regression. Outliers prediction provides extra insight to the data.
", "rawMarkdown": "@GideonTeo Yes, outliers are affecting rmse error a lot. I think the key is the outliers prediction. You might have different features which are more important to outliers and less for regression. Outliers prediction provides extra insight to the data. ", "votes": 1 }, { "id": 463296, "postDate": "2019-01-29T18:56:33.733Z", "content": "@salamatburj 3 is a good number )))\nI will try.
", "rawMarkdown": "@salamatburj 3 is a good number )))\nI will try." } ] }, { "id": 461407, "postDate": "2019-01-25T23:37:08.283Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 460843, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2019-01-24T14:28:43.567000", "content": "If you want to fit more closely to outliers: Change the metric you use for minimizing the sum of the residuals. That is, instead of minimizing the sum of the squared residuals, maybe minimize for the sum of the residuals to the power of 5.
If you want to fit less closely to outliers: Remove the outliers from your sample, or change the the optimization from minimizing the sum of the squared residuals, to minimizing the sum of the L1 residuals maybe. Or minimizing the sum of the Square Root Residuals
I think you can also set weights for each point if these don't work. So, you can specify all non-outlier points have a weight of 1, but the outlier points have a weight of 10.
", "votes": 16, "replies": [ { "id": 461008, "author_name": "Zhuang Jia", "author_url": "", "post_date": "2019-01-25T01:25:11.040000", "content": "Thanks, very helpful! I'v tried weight for outlier and non-outlier just as you said.
", "votes": 1, "replies": [] }, { "id": 461238, "author_name": "", "author_url": "", "post_date": "2019-01-25T15:42:27.517000", "content": "", "votes": 0, "replies": [] }, { "id": 461535, "author_name": "HyperSeedOptimized", "author_url": "", "post_date": "2019-01-26T09:31:25.850000", "content": "@jiazhuang May I know what worked and did not work for you so far with handling outliers?
", "votes": 0, "replies": [] }, { "id": 461573, "author_name": "DeepInvolution", "author_url": "", "post_date": "2019-01-26T12:17:03.163000", "content": "@CoreyLevinson I tried to use the loss function of minimizing for the sum of the residuals to the power of 5 as follows:
\n\ndef custom_asymmetric_train(y_true, y_pred):\n residual = (y_true - y_pred).astype(\"float\")\n grad = -5*residual**4\n hess = 20.0*residual**3\n return grad, hess\n\ndef custom_asymmetric_valid(y_true, y_pred):\n residual = (y_true - y_pred).astype(\"float\")\n loss = residual**5\n return \"custom_asymmetric_eval\", np.mean(loss), False\n\nmodel = LGBMClassifier(boosting_type=\"gbdt\", objective=custom_asymmetric_train）\n\nmodel.fit( X_train, y_train, eval_set=eval_set, eval_metric=custom_asymmetric_valid）\nHowever, the loss does not change for each CV. Any help on this?
", "votes": 4, "replies": [] }, { "id": 461637, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2019-01-26T16:05:50.050000", "content": "custom_asymmetric_train it is not expecting you to already negate it.Try this, I know in previous competition my custom stuff did not work unless I specified parameters to 'None':
\n\nimport lightgbm as lgb
\n\nparams = {\n 'boosting_type': 'gbdt',\n 'objective': 'None',\n 'metric': 'None',\n}\n\nmodel = lgb.train(params,\n lgb_train, # a LGBM Dataset\n num_boost_round=10,\n fobj=custom_asymmetric_train,\n feval=custom_asymmetric_valid,\n valid_sets=eval_set # Also should be a LGBM Dataset I think\n)\n@CoreyLevinson Thanks! Let me try it.
", "votes": 0, "replies": [] }, { "id": 463359, "author_name": "", "author_url": "", "post_date": "2019-01-29T21:21:43.247000", "content": "", "votes": 0, "replies": [] }, { "id": 463844, "author_name": "Costa.Medvedev", "author_url": "", "post_date": "2019-01-30T17:52:43.940000", "content": "@xiaojiu1414, @CoreyLevinson\nI tried to use the function above but it throws exception
\n\nTypeError: unsupported operand type(s) for -: 'float' and 'Dataset'
\n\nPrint type() shows that the first parameter (y_true) is a numpy array while the second (y_pred) is a lightGBM dataset.\nAny idea how to convert it to numpy? I found .get_data() function but it does not work either.
", "votes": 1, "replies": [] }, { "id": 464013, "author_name": "HarshitMehta", "author_url": "", "post_date": "2019-01-31T04:01:17.770000", "content": "That is happening because you are using lightgbm instead of lightgbm scikt learn API
", "votes": 2, "replies": [] }, { "id": 464014, "author_name": "HarshitMehta", "author_url": "", "post_date": "2019-01-31T04:02:55.673000", "content": "You can use the following format:\ndef custom_asymmetric_train(preds, train_data):
\n\ny_true = train_data.get_label()\nresidual = (y_true - preds).astype(\"float\")\ngrad = -4*residual**3\nhess = 12*residual**2\nreturn grad, hess\n`def custom_asymmetric_valid(preds, train_data):
\n\ny_true = train_data.get_label()\nresidual = (y_true - preds).astype(\"float\")\nloss =residual**4\nreturn \"custom_asymmetric_eval\", np.mean(loss), False\nHarshit, much appreciated. It works now.
", "votes": 1, "replies": [] } ] }, { "id": 460768, "author_name": "rafael chaves", "author_url": "", "post_date": "2019-01-24T11:40:27.340000", "content": "You can try measuring how much of your regression model is being influenced by that outlier. I wouldn't recommend just erasing it (some people do that)
", "votes": 1, "replies": [ { "id": 461010, "author_name": "Zhuang Jia", "author_url": "", "post_date": "2019-01-25T01:26:27.193000", "content": "yes, we can't just simply remove it.
", "votes": 0, "replies": [] } ] }, { "id": 462323, "author_name": "", "author_url": "", "post_date": "2019-01-28T05:32:51.300000", "content": "Hi,\n Did you get this solved? I tried power of 5, but the LGB never improved since the first round. Could you share your solution? Thank you!
", "votes": 0, "replies": [ { "id": 463213, "author_name": "Martin", "author_url": "", "post_date": "2019-01-29T16:15:56.033000", "content": "I would try power of 4 or power of 6. It should be even number.
", "votes": 1, "replies": [] } ] }, { "id": 460877, "author_name": "Costa.Medvedev", "author_url": "", "post_date": "2019-01-24T15:32:14.463000", "content": "I did a split onto two models\n1. LGB model Trained with a binary goal (outlier or not).\n2. LGB Is trained on the dataset without outliers (target<=-33) with all parameters set to the same values.\nResults are joined with. If the model 1 says it should be an outlier, then the predicted score set to -33. Otherwise the score is taken from model 2.\nBut it gave me very bad final score.\nMay be I had to optimize the first model better. Now it predicts correctly only 2-3 outliers out of hundreds we have.
", "votes": 0, "replies": [ { "id": 461012, "author_name": "Zhuang Jia", "author_url": "", "post_date": "2019-01-25T01:29:11.047000", "content": "yes, outlier classification is highly imbalanced. It's hard to predict them accurately. But It's still a good try.
", "votes": 0, "replies": [] }, { "id": 461141, "author_name": "Salamat", "author_url": "", "post_date": "2019-01-25T10:44:55.973000", "content": "@Costa.Medvedev\nUse three model instead. Model with outliers , model without outliers, and model for outliers prediction. Check this kernel :https://www.kaggle.com/waitingli/combining-your-model-with-a-model-without-outlier
", "votes": 2, "replies": [] }, { "id": 461537, "author_name": "HyperSeedOptimized", "author_url": "", "post_date": "2019-01-26T09:38:57.687000", "content": "@salamatburj I tried that, but cannot understand why it works. It is using a more general model for the (predicted) outliers than for the non-outliers, meaning that the general model is overly affected by the outlier values for predicting most of the (non-outlying) data. But why would a gbm behave this way? Thanks.
", "votes": 0, "replies": [] }, { "id": 461866, "author_name": "Salamat", "author_url": "", "post_date": "2019-01-27T07:43:45", "content": "@GideonTeo Yes, outliers are affecting rmse error a lot. I think the key is the outliers prediction. You might have different features which are more important to outliers and less for regression. Outliers prediction provides extra insight to the data.
", "votes": 1, "replies": [] }, { "id": 463296, "author_name": "Costa.Medvedev", "author_url": "", "post_date": "2019-01-29T18:56:33.733000", "content": "@salamatburj 3 is a good number )))\nI will try.
", "votes": 0, "replies": [] } ] }, { "id": 461407, "author_name": "", "author_url": "", "post_date": "2019-01-25T23:37:08.283000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "460680": "Do we have some strategies to handle outliers when doing regression, if the outliers is important to us?", "460843": "1. If you want to fit more closely to outliers: Change the metric you use for minimizing the sum of the residuals. That is, instead of minimizing the sum of the squared residuals, maybe minimize for the sum of the residuals to the power of 5. \n\n2. If you want to fit less closely to outliers: Remove the outliers from your sample, or change the the optimization from minimizing the sum of the squared residuals, to minimizing the sum of the L1 residuals maybe. Or minimizing the sum of the Square Root Residuals\n\nI think you can also set weights for each point if these don't work. So, you can specify all non-outlier points have a weight of 1, but the outlier points have a weight of 10.", "460768": "You can try measuring how much of your regression model is being influenced by that outlier. I wouldn't recommend just erasing it (some people do that)", "462323": "Hi,\n Did you get this solved? I tried power of 5, but the LGB never improved since the first round. Could you share your solution? Thank you!", "460877": "I did a split onto two models\n1. LGB model Trained with a binary goal (outlier or not).\n2. LGB Is trained on the dataset without outliers (target<=-33) with all parameters set to the same values.\nResults are joined with. If the model 1 says it should be an outlier, then the predicted score set to -33. Otherwise the score is taken from model 2.\nBut it gave me very bad final score.\nMay be I had to optimize the first model better. Now it predicts correctly only 2-3 outliers out of hundreds we have.", "461407": "" } }