{
  "id": 113604,
  "title": "How to deal with highly skewed dependent variable. Hope this helps!",
  "url": "/competitions/ashrae-energy-prediction/discussion/113604",
  "author_name": "Venkat Myneni",
  "post_date": "2019-10-20T18:37:57.459000",
  "votes": 6,
  "comment_count": 0,
  "views": 0,
  "content": "<p><em>What I have observed while referring through few research papers on “energy efficiency prediction models using retrofit equipment data” is GPM (Gaussian Process regression), SVM with RBF(Radial Basis Function) and ANN (Feed Forward &amp; Recurrent Neural Networks) models outperform the results compared to other algorithms.</em></p>\n\n<p><em>As a part of Regression model building, I would like to share here my experience on dealing with highly skewed dependent variable (It can be a continuous (or) discrete in nature).\nAs you all know that dependent variable can be found either positively skewed or negatively skewed in real time (seldom we see a dependent variable distributed normally unless we transform).</em></p>\n\n<p><strong>What if you find the given dependent variable is continuous and it is highly skewed?</strong></p>\n\n<p><strong>Approach-1:</strong> In case you wanted to try with Linear regression model initially to know the model benchmark, then you are highly recommended to transform the given dependent variable before proceeding to the model building and make sure that transformed dependent variable is close to Gaussian/Normal distribution as its one of the important assumption for linear regression models.\nMost of the times log transformation helps to bring positively skewed distribution close to the normal distribution. anti-log can be used if the given dependent variable is negatively distributed</p>\n\n<p><strong>Approach-2:</strong> If you do not want to try with approach-1 then you can pass distribution type as gamma (statistical estimation expectation: Mean (DV) =Variance (DV)^2) in your ML models if the dependent variable is positively skewed</p>\n\n<p><strong>Similarly, what if you find the given dependent variable is discrete and it is highly skewed?</strong></p>\n\n<p><strong>Approach -1:</strong> It is same as “Approach-1” recommended for continuous dependent variable</p>\n\n<p><strong>Approach-2:</strong> Poisson distribution method is highly recommended for discrete dependent variable, in fact Poisson has nothing to do with skewness. So, you can pass distribution type as Poisson (statistical estimation expectation: Mean (DV) =Variance (DV)) in your ML models if the dependent variable is positively/negatively skewed.</p>\n\n<p><strong>Note 1:</strong> If you want to try with both the distributions (Gamma and Poison) in your Machine Learning models (this distribution is called \"Tweedie\" distribution) you can try giving distribution as Tweedie.</p>\n\n<p><strong>Note 2:</strong> Ideally gamma distribution would be used for the positively skewed dependent variable and Poisson distribution would be used for discrete dependent variable (Poisson has nothing to do with skewness)</p>\n\n<hr>\n\n<p><em><strong>Please Up-vote if its helpful!</strong></em></p>\n\n<hr>\n\n<p><em><strong>Thanks!</strong></em></p>",
  "messages": [
    {
      "id": 653638,
      "postDate": "2019-10-20T18:37:57.460Z",
      "content": "<p><em>What I have observed while referring through few research papers on “energy efficiency prediction models using retrofit equipment data” is GPM (Gaussian Process regression), SVM with RBF(Radial Basis Function) and ANN (Feed Forward &amp; Recurrent Neural Networks) models outperform the results compared to other algorithms.</em></p>\n\n<p><em>As a part of Regression model building, I would like to share here my experience on dealing with highly skewed dependent variable (It can be a continuous (or) discrete in nature).\nAs you all know that dependent variable can be found either positively skewed or negatively skewed in real time (seldom we see a dependent variable distributed normally unless we transform).</em></p>\n\n<p><strong>What if you find the given dependent variable is continuous and it is highly skewed?</strong></p>\n\n<p><strong>Approach-1:</strong> In case you wanted to try with Linear regression model initially to know the model benchmark, then you are highly recommended to transform the given dependent variable before proceeding to the model building and make sure that transformed dependent variable is close to Gaussian/Normal distribution as its one of the important assumption for linear regression models.\nMost of the times log transformation helps to bring positively skewed distribution close to the normal distribution. anti-log can be used if the given dependent variable is negatively distributed</p>\n\n<p><strong>Approach-2:</strong> If you do not want to try with approach-1 then you can pass distribution type as gamma (statistical estimation expectation: Mean (DV) =Variance (DV)^2) in your ML models if the dependent variable is positively skewed</p>\n\n<p><strong>Similarly, what if you find the given dependent variable is discrete and it is highly skewed?</strong></p>\n\n<p><strong>Approach -1:</strong> It is same as “Approach-1” recommended for continuous dependent variable</p>\n\n<p><strong>Approach-2:</strong> Poisson distribution method is highly recommended for discrete dependent variable, in fact Poisson has nothing to do with skewness. So, you can pass distribution type as Poisson (statistical estimation expectation: Mean (DV) =Variance (DV)) in your ML models if the dependent variable is positively/negatively skewed.</p>\n\n<p><strong>Note 1:</strong> If you want to try with both the distributions (Gamma and Poison) in your Machine Learning models (this distribution is called \"Tweedie\" distribution) you can try giving distribution as Tweedie.</p>\n\n<p><strong>Note 2:</strong> Ideally gamma distribution would be used for the positively skewed dependent variable and Poisson distribution would be used for discrete dependent variable (Poisson has nothing to do with skewness)</p>\n\n<hr>\n\n<p><em><strong>Please Up-vote if its helpful!</strong></em></p>\n\n<hr>\n\n<p><em><strong>Thanks!</strong></em></p>",
      "rawMarkdown": "*What I have observed while referring through few research papers on “energy efficiency prediction models using retrofit equipment data” is GPM (Gaussian Process regression), SVM with RBF(Radial Basis Function) and ANN (Feed Forward &amp; Recurrent Neural Networks) models outperform the results compared to other algorithms.*\n\n*As a part of Regression model building, I would like to share here my experience on dealing with highly skewed dependent variable (It can be a continuous (or) discrete in nature).\nAs you all know that dependent variable can be found either positively skewed or negatively skewed in real time (seldom we see a dependent variable distributed normally unless we transform).*\n\n**What if you find the given dependent variable is continuous and it is highly skewed?**\n\n**Approach-1:** In case you wanted to try with Linear regression model initially to know the model benchmark, then you are highly recommended to transform the given dependent variable before proceeding to the model building and make sure that transformed dependent variable is close to Gaussian/Normal distribution as its one of the important assumption for linear regression models.\nMost of the times log transformation helps to bring positively skewed distribution close to the normal distribution. anti-log can be used if the given dependent variable is negatively distributed\n\n**Approach-2:** If you do not want to try with approach-1 then you can pass distribution type as gamma (statistical estimation expectation: Mean (DV) =Variance (DV)^2) in your ML models if the dependent variable is positively skewed\n\n\n**Similarly, what if you find the given dependent variable is discrete and it is highly skewed?**\n\n**Approach -1:** It is same as “Approach-1” recommended for continuous dependent variable\n\n**Approach-2:** Poisson distribution method is highly recommended for discrete dependent variable, in fact Poisson has nothing to do with skewness. So, you can pass distribution type as Poisson (statistical estimation expectation: Mean (DV) =Variance (DV)) in your ML models if the dependent variable is positively/negatively skewed.\n\n**Note 1:** If you want to try with both the distributions (Gamma and Poison) in your Machine Learning models (this distribution is called \"Tweedie\" distribution) you can try giving distribution as Tweedie.\n\n**Note 2:** Ideally gamma distribution would be used for the positively skewed dependent variable and Poisson distribution would be used for discrete dependent variable (Poisson has nothing to do with skewness)\n\n\n-------------------------------------------------------------------------------------------------------------------\n\n***Please Up-vote if its helpful!***\n\n\n-------------------------------------------------------------------------------------------------------------------\n\n***Thanks!***",
      "votes": 6
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "653638": "*What I have observed while referring through few research papers on “energy efficiency prediction models using retrofit equipment data” is GPM (Gaussian Process regression), SVM with RBF(Radial Basis Function) and ANN (Feed Forward &amp; Recurrent Neural Networks) models outperform the results compared to other algorithms.*\n\n*As a part of Regression model building, I would like to share here my experience on dealing with highly skewed dependent variable (It can be a continuous (or) discrete in nature).\nAs you all know that dependent variable can be found either positively skewed or negatively skewed in real time (seldom we see a dependent variable distributed normally unless we transform).*\n\n**What if you find the given dependent variable is continuous and it is highly skewed?**\n\n**Approach-1:** In case you wanted to try with Linear regression model initially to know the model benchmark, then you are highly recommended to transform the given dependent variable before proceeding to the model building and make sure that transformed dependent variable is close to Gaussian/Normal distribution as its one of the important assumption for linear regression models.\nMost of the times log transformation helps to bring positively skewed distribution close to the normal distribution. anti-log can be used if the given dependent variable is negatively distributed\n\n**Approach-2:** If you do not want to try with approach-1 then you can pass distribution type as gamma (statistical estimation expectation: Mean (DV) =Variance (DV)^2) in your ML models if the dependent variable is positively skewed\n\n\n**Similarly, what if you find the given dependent variable is discrete and it is highly skewed?**\n\n**Approach -1:** It is same as “Approach-1” recommended for continuous dependent variable\n\n**Approach-2:** Poisson distribution method is highly recommended for discrete dependent variable, in fact Poisson has nothing to do with skewness. So, you can pass distribution type as Poisson (statistical estimation expectation: Mean (DV) =Variance (DV)) in your ML models if the dependent variable is positively/negatively skewed.\n\n**Note 1:** If you want to try with both the distributions (Gamma and Poison) in your Machine Learning models (this distribution is called \"Tweedie\" distribution) you can try giving distribution as Tweedie.\n\n**Note 2:** Ideally gamma distribution would be used for the positively skewed dependent variable and Poisson distribution would be used for discrete dependent variable (Poisson has nothing to do with skewness)\n\n\n-------------------------------------------------------------------------------------------------------------------\n\n***Please Up-vote if its helpful!***\n\n\n-------------------------------------------------------------------------------------------------------------------\n\n***Thanks!***"
  }
}