{
  "id": 21585,
  "title": "Linear Regression with Skewed Data",
  "url": "/competitions/grupo-bimbo-inventory-demand/discussion/21585",
  "author_name": "",
  "post_date": "2016-06-11T00:59:50.077Z",
  "votes": 1,
  "comment_count": 10,
  "views": 2505,
  "content": "<p>Hi,</p>\n\n<p>A graph of the output variable shows a skewed distribution to the right (histogram is attached).</p>\n\n<p>Is a linear regression appropriate for skewed data like this?  If so, should there be any modifications to the output variable?</p>\n\n<p>The reason I ask this is because a simple linear regression using the sales channel variable ('Canal_ID') predicts values closer to the average output value of 7.2.  However, using a naive prediction of 3.9 or numbers closer to the median of 3 is getting better scores on the leaderboard than the linear regression. </p>\n\n<p>Any insight would be appreciated.</p>",
  "messages": [
    {
      "id": "123315",
      "postDate": "06/11/2016 00:59:50",
      "content": "<p>Hi,</p>\n\n<p>A graph of the output variable shows a skewed distribution to the right (histogram is attached).</p>\n\n<p>Is a linear regression appropriate for skewed data like this?  If so, should there be any modifications to the output variable?</p>\n\n<p>The reason I ask this is because a simple linear regression using the sales channel variable ('Canal_ID') predicts values closer to the average output value of 7.2.  However, using a naive prediction of 3.9 or numbers closer to the median of 3 is getting better scores on the leaderboard than the linear regression. </p>\n\n<p>Any insight would be appreciated.</p>",
      "rawMarkdown": "Hi,\r\n\r\nA graph of the output variable shows a skewed distribution to the right (histogram is attached).\r\n\r\nIs a linear regression appropriate for skewed data like this?  If so, should there be any modifications to the output variable?\r\n\r\nThe reason I ask this is because a simple linear regression using the sales channel variable ('Canal_ID') predicts values closer to the average output value of 7.2.  However, using a naive prediction of 3.9 or numbers closer to the median of 3 is getting better scores on the leaderboard than the linear regression. \r\n\r\nAny insight would be appreciated.",
      "votes": null
    },
    {
      "id": "123317",
      "postDate": "06/11/2016 01:30:16",
      "content": "<p>The response isn't necessarily skewed, demand just can't be less than zero. Try mean-centering the data and then plotting a histogram. </p>\n\n<p>For your linear regression, are you regressing the output on Canal_ID as a factor (dummy variables for each Channel ID) or Canal_ID as numeric? Only the former would make sense. </p>",
      "rawMarkdown": "The response isn't necessarily skewed, demand just can't be less than zero. Try mean-centering the data and then plotting a histogram. \r\n\r\nFor your linear regression, are you regressing the output on Canal_ID as a factor (dummy variables for each Channel ID) or Canal_ID as numeric? Only the former would make sense.",
      "votes": null
    },
    {
      "id": "123319",
      "postDate": "06/11/2016 01:38:08",
      "content": "<p>Hi Troy,</p>\n\n<p>I'm not sure I understand your response.  I think the data is a pretty classic skew.  If you look at the graph I attached, there is a tail to the right of the peak or, in other words, it's not a symmetric distribution.  Also, the median is 3 and lower than the mean of 7.2 - which is typically the case with a right skew.</p>\n\n<p>Yes, I am regressing the output on Canal_ID as a factor using dummy variables for each Channel ID, but thanks for pointing that out.</p>",
      "rawMarkdown": "Hi Troy,\r\n\r\nI'm not sure I understand your response.  I think the data is a pretty classic skew.  If you look at the graph I attached, there is a tail to the right of the peak or, in other words, it's not a symmetric distribution.  Also, the median is 3 and lower than the mean of 7.2 - which is typically the case with a right skew.\r\n\r\nYes, I am regressing the output on Canal_ID as a factor using dummy variables for each Channel ID, but thanks for pointing that out.",
      "votes": null
    },
    {
      "id": "123320",
      "postDate": "06/11/2016 01:50:31",
      "content": "<p>Hi Troy,</p>\n\n<p>Sorry, I think I get more of what you are saying now.  It could be a normal distribution if demand could be less than zero.  In other words, we can't really see the 'left half' of the distribution.</p>\n\n<p>Ok, how do you 'mean center'?  If the mean is 7.2, then subtract 7.2 from each output value?  Won't that just reduce the linear regression prediction by 7.2?</p>",
      "rawMarkdown": "Hi Troy,\r\n\r\nSorry, I think I get more of what you are saying now.  It could be a normal distribution if demand could be less than zero.  In other words, we can't really see the 'left half' of the distribution.\r\n\r\nOk, how do you 'mean center'?  If the mean is 7.2, then subtract 7.2 from each output value?  Won't that just reduce the linear regression prediction by 7.2?",
      "votes": null
    },
    {
      "id": "123332",
      "postDate": "06/11/2016 04:52:49",
      "content": "<p>Correct. Also, if you are concerned about normality, you could also try a qq plot. </p>\n\n<p>What is more important is that you regression residuals are normally distributed. </p>\n\n<p>Linear regression isn't technically appropriate when the response is bounded (by 0 in this case),\nalthough you can often still get away with it as long as residuals are gaussian. </p>",
      "rawMarkdown": "Correct. Also, if you are concerned about normality, you could also try a qq plot. \r\n\r\nWhat is more important is that you regression residuals are normally distributed. \r\n\r\nLinear regression isn't technically appropriate when the response is bounded (by 0 in this case),\r\nalthough you can often still get away with it as long as residuals are gaussian.",
      "votes": null
    },
    {
      "id": "124395",
      "postDate": "06/18/2016 02:17:45",
      "content": "<p>Hi Troy,</p>\n\n<p>Sorry for the delayed response.  You are right, I forgot that it's more important that the residuals are normally distributed.  And that's a good point about linear regression not technically appropriate here.</p>\n\n<p>I've taken a pause on the regression.  To me, there are a lot of categorical variables and so i'm not sure a regression would be workable and accurate.  For example, there are 540 sales depots, 1897 routes, 89417 customers and 290 products.</p>\n\n<p>If I take this up again, I will try your suggestion of doing a qq plot.</p>\n\n<p>Thanks</p>",
      "rawMarkdown": "Hi Troy,\r\n\r\nSorry for the delayed response.  You are right, I forgot that it's more important that the residuals are normally distributed.  And that's a good point about linear regression not technically appropriate here.\r\n\r\nI've taken a pause on the regression.  To me, there are a lot of categorical variables and so i'm not sure a regression would be workable and accurate.  For example, there are 540 sales depots, 1897 routes, 89417 customers and 290 products.\r\n\r\nIf I take this up again, I will try your suggestion of doing a qq plot.\r\n\r\nThanks",
      "votes": null
    },
    {
      "id": "124399",
      "postDate": "06/18/2016 03:12:53",
      "content": "<p>You could try to transform the variable you want to predict like using the function f(x) = log(x+1) (natural logarithm) to fit the metric then retransform it using g(y) = exp(y) - 1</p>\n\n<p>It should help a lot</p>",
      "rawMarkdown": "You could try to transform the variable you want to predict like using the function f(x) = log(x+1) (natural logarithm) to fit the metric then retransform it using g(y) = exp(y) - 1\r\n\r\nIt should help a lot",
      "votes": null
    },
    {
      "id": "124453",
      "postDate": "06/18/2016 18:17:39",
      "content": "<p>Thank you, Jonathan.  Yes, that's a good idea.  I did that earlier this week and it improved the LB score by 0.00586.  Thanks for the idea.</p>\n\n<p>I'm trying to figure out if there is any machine learning methods that can be used.  So far, I have mainly just been leveraging client and product info averages.</p>",
      "rawMarkdown": "Thank you, Jonathan.  Yes, that's a good idea.  I did that earlier this week and it improved the LB score by 0.00586.  Thanks for the idea.\r\n\r\nI'm trying to figure out if there is any machine learning methods that can be used.  So far, I have mainly just been leveraging client and product info averages.",
      "votes": null
    },
    {
      "id": "124584",
      "postDate": "06/20/2016 04:54:02",
      "content": "<p>Mike, Do you find any other machine learning approaches ?</p>",
      "rawMarkdown": "Mike, Do you find any other machine learning approaches ?",
      "votes": null
    },
    {
      "id": "124604",
      "postDate": "06/20/2016 10:40:31",
      "content": "<p>[quote=Mike Bak;123315]</p>\n\n<p>Hi,</p>\n\n<p>A graph of the output variable shows a skewed distribution to the right (histogram is attached).</p>\n\n<p>[/quote]</p>\n\n<p>Did you think about pre-processing the label using a Box-Cox transformation (since label &gt; 0) or a Yeo-Johnson transformation (practically the same)? The label would get bound to [0, 1] and the skew would get wiped (skewness = 0, excess kurtosis = 0). Then, you would just have to handle properly what to do for values out of bound.</p>\n\n<p>Also, you can try robust methods of linear regressions, although they might be computationally intensive. And if you are using a sparse matrix with all one-hot encoded features, you can try a zero-inflated -type regression.</p>\n\n<p>[quote=Mike Bak;124395]</p>\n\n<p>If I take this up again, I will try your suggestion of doing a qq plot.</p>\n\n<p>[/quote]</p>\n\n<p>QQ plot on a 10% sample of the data:</p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4444/qqplot.JPG\" alt=\"enter image description here\" title></p>\n\n<p>If applying a linear regression, there is an extremely high odd of not having normal distributed residuals. You should find heteroscedasticity in these residuals. You can use White's test and/or Breusch-Pagan's test to look for heteroscedasticity in the residuals.</p>",
      "rawMarkdown": "[quote=Mike Bak;123315]\r\n\r\nHi,\r\n\r\nA graph of the output variable shows a skewed distribution to the right (histogram is attached).\r\n\r\n[/quote]\r\n\r\nDid you think about pre-processing the label using a Box-Cox transformation (since label > 0) or a Yeo-Johnson transformation (practically the same)? The label would get bound to [0, 1] and the skew would get wiped (skewness = 0, excess kurtosis = 0). Then, you would just have to handle properly what to do for values out of bound.\r\n\r\nAlso, you can try robust methods of linear regressions, although they might be computationally intensive. And if you are using a sparse matrix with all one-hot encoded features, you can try a zero-inflated -type regression.\r\n\r\n[quote=Mike Bak;124395]\r\n\r\nIf I take this up again, I will try your suggestion of doing a qq plot.\r\n\r\n[/quote]\r\n\r\nQQ plot on a 10% sample of the data:\r\n\r\n![enter image description here][1]\r\n\r\nIf applying a linear regression, there is an extremely high odd of not having normal distributed residuals. You should find heteroscedasticity in these residuals. You can use White's test and/or Breusch-Pagan's test to look for heteroscedasticity in the residuals.\r\n\r\n  [1]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4444/qqplot.JPG",
      "votes": null
    },
    {
      "id": "124875",
      "postDate": "06/22/2016 23:03:26",
      "content": "<p>@Avinash_Rai:  I am going to try gbm</p>",
      "rawMarkdown": "Avinash_Rai:  I am going to try gbm",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 123317,
      "author_name": "captcalculator",
      "author_url": "",
      "post_date": "06/11/2016 01:30:16",
      "content": "<p>The response isn't necessarily skewed, demand just can't be less than zero. Try mean-centering the data and then plotting a histogram. </p>\n\n<p>For your linear regression, are you regressing the output on Canal_ID as a factor (dummy variables for each Channel ID) or Canal_ID as numeric? Only the former would make sense. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 123319,
      "author_name": "mikebak",
      "author_url": "",
      "post_date": "06/11/2016 01:38:08",
      "content": "<p>Hi Troy,</p>\n\n<p>I'm not sure I understand your response.  I think the data is a pretty classic skew.  If you look at the graph I attached, there is a tail to the right of the peak or, in other words, it's not a symmetric distribution.  Also, the median is 3 and lower than the mean of 7.2 - which is typically the case with a right skew.</p>\n\n<p>Yes, I am regressing the output on Canal_ID as a factor using dummy variables for each Channel ID, but thanks for pointing that out.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 123320,
      "author_name": "mikebak",
      "author_url": "",
      "post_date": "06/11/2016 01:50:31",
      "content": "<p>Hi Troy,</p>\n\n<p>Sorry, I think I get more of what you are saying now.  It could be a normal distribution if demand could be less than zero.  In other words, we can't really see the 'left half' of the distribution.</p>\n\n<p>Ok, how do you 'mean center'?  If the mean is 7.2, then subtract 7.2 from each output value?  Won't that just reduce the linear regression prediction by 7.2?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 123332,
      "author_name": "captcalculator",
      "author_url": "",
      "post_date": "06/11/2016 04:52:49",
      "content": "<p>Correct. Also, if you are concerned about normality, you could also try a qq plot. </p>\n\n<p>What is more important is that you regression residuals are normally distributed. </p>\n\n<p>Linear regression isn't technically appropriate when the response is bounded (by 0 in this case),\nalthough you can often still get away with it as long as residuals are gaussian. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124395,
      "author_name": "mikebak",
      "author_url": "",
      "post_date": "06/18/2016 02:17:45",
      "content": "<p>Hi Troy,</p>\n\n<p>Sorry for the delayed response.  You are right, I forgot that it's more important that the residuals are normally distributed.  And that's a good point about linear regression not technically appropriate here.</p>\n\n<p>I've taken a pause on the regression.  To me, there are a lot of categorical variables and so i'm not sure a regression would be workable and accurate.  For example, there are 540 sales depots, 1897 routes, 89417 customers and 290 products.</p>\n\n<p>If I take this up again, I will try your suggestion of doing a qq plot.</p>\n\n<p>Thanks</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124399,
      "author_name": "jonathanslomka",
      "author_url": "",
      "post_date": "06/18/2016 03:12:53",
      "content": "<p>You could try to transform the variable you want to predict like using the function f(x) = log(x+1) (natural logarithm) to fit the metric then retransform it using g(y) = exp(y) - 1</p>\n\n<p>It should help a lot</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124453,
      "author_name": "mikebak",
      "author_url": "",
      "post_date": "06/18/2016 18:17:39",
      "content": "<p>Thank you, Jonathan.  Yes, that's a good idea.  I did that earlier this week and it improved the LB score by 0.00586.  Thanks for the idea.</p>\n\n<p>I'm trying to figure out if there is any machine learning methods that can be used.  So far, I have mainly just been leveraging client and product info averages.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124584,
      "author_name": "avinashrai",
      "author_url": "",
      "post_date": "06/20/2016 04:54:02",
      "content": "<p>Mike, Do you find any other machine learning approaches ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124604,
      "author_name": "laurae2",
      "author_url": "",
      "post_date": "06/20/2016 10:40:31",
      "content": "<p>[quote=Mike Bak;123315]</p>\n\n<p>Hi,</p>\n\n<p>A graph of the output variable shows a skewed distribution to the right (histogram is attached).</p>\n\n<p>[/quote]</p>\n\n<p>Did you think about pre-processing the label using a Box-Cox transformation (since label &gt; 0) or a Yeo-Johnson transformation (practically the same)? The label would get bound to [0, 1] and the skew would get wiped (skewness = 0, excess kurtosis = 0). Then, you would just have to handle properly what to do for values out of bound.</p>\n\n<p>Also, you can try robust methods of linear regressions, although they might be computationally intensive. And if you are using a sparse matrix with all one-hot encoded features, you can try a zero-inflated -type regression.</p>\n\n<p>[quote=Mike Bak;124395]</p>\n\n<p>If I take this up again, I will try your suggestion of doing a qq plot.</p>\n\n<p>[/quote]</p>\n\n<p>QQ plot on a 10% sample of the data:</p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4444/qqplot.JPG\" alt=\"enter image description here\" title></p>\n\n<p>If applying a linear regression, there is an extremely high odd of not having normal distributed residuals. You should find heteroscedasticity in these residuals. You can use White's test and/or Breusch-Pagan's test to look for heteroscedasticity in the residuals.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124875,
      "author_name": "mikebak",
      "author_url": "",
      "post_date": "06/22/2016 23:03:26",
      "content": "<p>@Avinash_Rai:  I am going to try gbm</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "123315": "Hi,\r\n\r\nA graph of the output variable shows a skewed distribution to the right (histogram is attached).\r\n\r\nIs a linear regression appropriate for skewed data like this?  If so, should there be any modifications to the output variable?\r\n\r\nThe reason I ask this is because a simple linear regression using the sales channel variable ('Canal_ID') predicts values closer to the average output value of 7.2.  However, using a naive prediction of 3.9 or numbers closer to the median of 3 is getting better scores on the leaderboard than the linear regression. \r\n\r\nAny insight would be appreciated.",
    "123317": "The response isn't necessarily skewed, demand just can't be less than zero. Try mean-centering the data and then plotting a histogram. \r\n\r\nFor your linear regression, are you regressing the output on Canal_ID as a factor (dummy variables for each Channel ID) or Canal_ID as numeric? Only the former would make sense.",
    "123319": "Hi Troy,\r\n\r\nI'm not sure I understand your response.  I think the data is a pretty classic skew.  If you look at the graph I attached, there is a tail to the right of the peak or, in other words, it's not a symmetric distribution.  Also, the median is 3 and lower than the mean of 7.2 - which is typically the case with a right skew.\r\n\r\nYes, I am regressing the output on Canal_ID as a factor using dummy variables for each Channel ID, but thanks for pointing that out.",
    "123320": "Hi Troy,\r\n\r\nSorry, I think I get more of what you are saying now.  It could be a normal distribution if demand could be less than zero.  In other words, we can't really see the 'left half' of the distribution.\r\n\r\nOk, how do you 'mean center'?  If the mean is 7.2, then subtract 7.2 from each output value?  Won't that just reduce the linear regression prediction by 7.2?",
    "123332": "Correct. Also, if you are concerned about normality, you could also try a qq plot. \r\n\r\nWhat is more important is that you regression residuals are normally distributed. \r\n\r\nLinear regression isn't technically appropriate when the response is bounded (by 0 in this case),\r\nalthough you can often still get away with it as long as residuals are gaussian.",
    "124395": "Hi Troy,\r\n\r\nSorry for the delayed response.  You are right, I forgot that it's more important that the residuals are normally distributed.  And that's a good point about linear regression not technically appropriate here.\r\n\r\nI've taken a pause on the regression.  To me, there are a lot of categorical variables and so i'm not sure a regression would be workable and accurate.  For example, there are 540 sales depots, 1897 routes, 89417 customers and 290 products.\r\n\r\nIf I take this up again, I will try your suggestion of doing a qq plot.\r\n\r\nThanks",
    "124399": "You could try to transform the variable you want to predict like using the function f(x) = log(x+1) (natural logarithm) to fit the metric then retransform it using g(y) = exp(y) - 1\r\n\r\nIt should help a lot",
    "124453": "Thank you, Jonathan.  Yes, that's a good idea.  I did that earlier this week and it improved the LB score by 0.00586.  Thanks for the idea.\r\n\r\nI'm trying to figure out if there is any machine learning methods that can be used.  So far, I have mainly just been leveraging client and product info averages.",
    "124584": "Mike, Do you find any other machine learning approaches ?",
    "124604": "[quote=Mike Bak;123315]\r\n\r\nHi,\r\n\r\nA graph of the output variable shows a skewed distribution to the right (histogram is attached).\r\n\r\n[/quote]\r\n\r\nDid you think about pre-processing the label using a Box-Cox transformation (since label > 0) or a Yeo-Johnson transformation (practically the same)? The label would get bound to [0, 1] and the skew would get wiped (skewness = 0, excess kurtosis = 0). Then, you would just have to handle properly what to do for values out of bound.\r\n\r\nAlso, you can try robust methods of linear regressions, although they might be computationally intensive. And if you are using a sparse matrix with all one-hot encoded features, you can try a zero-inflated -type regression.\r\n\r\n[quote=Mike Bak;124395]\r\n\r\nIf I take this up again, I will try your suggestion of doing a qq plot.\r\n\r\n[/quote]\r\n\r\nQQ plot on a 10% sample of the data:\r\n\r\n![enter image description here][1]\r\n\r\nIf applying a linear regression, there is an extremely high odd of not having normal distributed residuals. You should find heteroscedasticity in these residuals. You can use White's test and/or Breusch-Pagan's test to look for heteroscedasticity in the residuals.\r\n\r\n  [1]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4444/qqplot.JPG",
    "124875": "Avinash_Rai:  I am going to try gbm"
  },
  "source": "meta"
}