{ "id": 113856, "title": "weather_train set EDA", "url": "/competitions/ashrae-energy-prediction/discussion/113856", "author_name": "", "post_date": "2019-10-22T15:54:36.328592300Z", "votes": 3, "comment_count": 3, "views": 0, "content": "
    \n

  1. Pearson's r : Pearson's correlation coefficient is the covariance of the two variables divided by the product of their standard deviations. The form of the definition involves a \"product moment\", that is, the mean (the first moment about the origin) of the product of the mean-adjusted random variables; hence the modifier product-moment in the name. \n\"\"

    2. \n

  2. Spearman's rho : In statistics, Spearman's rank correlation coefficient or Spearman's rho, named after Charles Spearman and often denoted by the Greek letter ρ (rho) or as\nr_{s}, is a nonparametric measure of rank correlation (statistical dependence between the rankings of two variables). It assesses how well the relationship between two variables can be described using a monotonic function.\nThe Spearman correlation between two variables is equal to the Pearson correlation between the rank values of those two variables; while Pearson's correlation assesses linear relationships, Spearman's correlation assesses monotonic relationships (whether linear or not). If there are no repeated data values, a perfect Spearman correlation of +1 or −1 occurs when each of the variables is a perfect monotone function of the other.\nIntuitively, the Spearman correlation between two variables will be high when observations have a similar (or identical for a correlation of 1) rank (i.e. relative position label of the observations within the variable: 1st, 2nd, 3rd, etc.) between the two variables, and low when observations have a dissimilar (or fully opposed for a correlation of −1) rank between the two variables.\nSpearman's coefficient is appropriate for both continuous and discrete ordinal variables. Both Spearman's ρ and Kendall's τ can be formulated as special cases of a more general correlation coefficient.\n\"\"

    3. \n

  3. Kendall's Tau : In statistics, the Kendall rank correlation coefficient, commonly referred to as Kendall's tau coefficient (after the Greek letter τ), is a statistic used to measure the ordinal association between two measured quantities. A tau test is a non-parametric hypothesis test for statistical dependence based on the tau coefficient.\nIt is a measure of rank correlation: the similarity of the orderings of the data when ranked by each of the quantities. It is named after Maurice Kendall, who developed it in 1938, though Gustav Fechner had proposed a similar measure in the context of time series in 1897.\nIntuitively, the Kendall correlation between two variables will be high when observations have a similar (or identical for a correlation of 1) rank (i.e. relative position label of the observations within the variable: 1st, 2nd, 3rd, etc.) between the two variables, and low when observations have a dissimilar (or fully different for a correlation of −1) rank between the two variables. Both Kendall's τ and Spearman's ρ can be formulated as special cases of a more general correlation coefficient. \"\"

    4. \n

  4. PhiK is a new and practical correlation coefficient based on several refinements to Pearson's hypothesis test of independence of two variables. The combined features of PhiK form an advantage over existing coefficients. First, it works consistently between categorical, ordinal and interval variables. Second, it captures non-linear dependency. Third, it reverts to the Pearson correlation coefficient in case of a bi-variate normal input distribution. These are useful features when studying the correlation matrix of variables with mixed types.\nThe presented algorithms are easy to use and available through this public Python library: the correlation analyzer package. Emphasis is paid to the proper evaluation of statistical significance of correlations and to the interpretation of variable relationships in a contingency table, in particular in case of low statistics samples.\nFor example, the Phi_K correlation analyzer package has been used to study surveys, insurance claims, correlograms, etc. \n\"\"

    5. \n
\n\n

Missing Value Analysis\n1. Count of missing values by features\n\"\"

\n\n
    \n

  1. Missing Value Matrix\n\"\"

    2. \n

  2. Dendogram of features\n\"\"

    3. \n
", "messages": [ { "id": "655023", "postDate": "10/22/2019 15:54:36", "content": "
    \n

  1. Pearson's r : Pearson's correlation coefficient is the covariance of the two variables divided by the product of their standard deviations. The form of the definition involves a \"product moment\", that is, the mean (the first moment about the origin) of the product of the mean-adjusted random variables; hence the modifier product-moment in the name. \n\"\"

    2. \n

  2. Spearman's rho : In statistics, Spearman's rank correlation coefficient or Spearman's rho, named after Charles Spearman and often denoted by the Greek letter ρ (rho) or as\nr_{s}, is a nonparametric measure of rank correlation (statistical dependence between the rankings of two variables). It assesses how well the relationship between two variables can be described using a monotonic function.\nThe Spearman correlation between two variables is equal to the Pearson correlation between the rank values of those two variables; while Pearson's correlation assesses linear relationships, Spearman's correlation assesses monotonic relationships (whether linear or not). If there are no repeated data values, a perfect Spearman correlation of +1 or −1 occurs when each of the variables is a perfect monotone function of the other.\nIntuitively, the Spearman correlation between two variables will be high when observations have a similar (or identical for a correlation of 1) rank (i.e. relative position label of the observations within the variable: 1st, 2nd, 3rd, etc.) between the two variables, and low when observations have a dissimilar (or fully opposed for a correlation of −1) rank between the two variables.\nSpearman's coefficient is appropriate for both continuous and discrete ordinal variables. Both Spearman's ρ and Kendall's τ can be formulated as special cases of a more general correlation coefficient.\n\"\"

    3. \n

  3. Kendall's Tau : In statistics, the Kendall rank correlation coefficient, commonly referred to as Kendall's tau coefficient (after the Greek letter τ), is a statistic used to measure the ordinal association between two measured quantities. A tau test is a non-parametric hypothesis test for statistical dependence based on the tau coefficient.\nIt is a measure of rank correlation: the similarity of the orderings of the data when ranked by each of the quantities. It is named after Maurice Kendall, who developed it in 1938, though Gustav Fechner had proposed a similar measure in the context of time series in 1897.\nIntuitively, the Kendall correlation between two variables will be high when observations have a similar (or identical for a correlation of 1) rank (i.e. relative position label of the observations within the variable: 1st, 2nd, 3rd, etc.) between the two variables, and low when observations have a dissimilar (or fully different for a correlation of −1) rank between the two variables. Both Kendall's τ and Spearman's ρ can be formulated as special cases of a more general correlation coefficient. \"\"

    4. \n

  4. PhiK is a new and practical correlation coefficient based on several refinements to Pearson's hypothesis test of independence of two variables. The combined features of PhiK form an advantage over existing coefficients. First, it works consistently between categorical, ordinal and interval variables. Second, it captures non-linear dependency. Third, it reverts to the Pearson correlation coefficient in case of a bi-variate normal input distribution. These are useful features when studying the correlation matrix of variables with mixed types.\nThe presented algorithms are easy to use and available through this public Python library: the correlation analyzer package. Emphasis is paid to the proper evaluation of statistical significance of correlations and to the interpretation of variable relationships in a contingency table, in particular in case of low statistics samples.\nFor example, the Phi_K correlation analyzer package has been used to study surveys, insurance claims, correlograms, etc. \n\"\"

    5. \n
\n\n

Missing Value Analysis\n1. Count of missing values by features\n\"\"

\n\n
    \n

  1. Missing Value Matrix\n\"\"

    2. \n

  2. Dendogram of features\n\"\"

    3. \n
", "rawMarkdown": "1. Pearson's r : Pearson's correlation coefficient is the covariance of the two variables divided by the product of their standard deviations. The form of the definition involves a \"product moment\", that is, the mean (the first moment about the origin) of the product of the mean-adjusted random variables; hence the modifier product-moment in the name. \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2F86278b0ad8a84e0dcce82017ef6f7dd1%2FScreen%20Shot%202019-10-22%20at%204.48.40%20PM.png?generation=1571759350103588&alt=media)\n\n2. Spearman's rho : In statistics, Spearman's rank correlation coefficient or Spearman's rho, named after Charles Spearman and often denoted by the Greek letter ρ (rho) or as\nr_{s}, is a nonparametric measure of rank correlation (statistical dependence between the rankings of two variables). It assesses how well the relationship between two variables can be described using a monotonic function.\nThe Spearman correlation between two variables is equal to the Pearson correlation between the rank values of those two variables; while Pearson's correlation assesses linear relationships, Spearman's correlation assesses monotonic relationships (whether linear or not). If there are no repeated data values, a perfect Spearman correlation of +1 or −1 occurs when each of the variables is a perfect monotone function of the other.\nIntuitively, the Spearman correlation between two variables will be high when observations have a similar (or identical for a correlation of 1) rank (i.e. relative position label of the observations within the variable: 1st, 2nd, 3rd, etc.) between the two variables, and low when observations have a dissimilar (or fully opposed for a correlation of −1) rank between the two variables.\nSpearman's coefficient is appropriate for both continuous and discrete ordinal variables. Both Spearman's ρ and Kendall's τ can be formulated as special cases of a more general correlation coefficient.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2F90a4f6b15983b3881bf606c32c4ffa50%2FScreen%20Shot%202019-10-22%20at%204.50.02%20PM.png?generation=1571759428659907&alt=media)\n\n3. Kendall's Tau : In statistics, the Kendall rank correlation coefficient, commonly referred to as Kendall's tau coefficient (after the Greek letter τ), is a statistic used to measure the ordinal association between two measured quantities. A tau test is a non-parametric hypothesis test for statistical dependence based on the tau coefficient.\nIt is a measure of rank correlation: the similarity of the orderings of the data when ranked by each of the quantities. It is named after Maurice Kendall, who developed it in 1938, though Gustav Fechner had proposed a similar measure in the context of time series in 1897.\nIntuitively, the Kendall correlation between two variables will be high when observations have a similar (or identical for a correlation of 1) rank (i.e. relative position label of the observations within the variable: 1st, 2nd, 3rd, etc.) between the two variables, and low when observations have a dissimilar (or fully different for a correlation of −1) rank between the two variables. Both Kendall's τ and Spearman's ρ can be formulated as special cases of a more general correlation coefficient. ![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2F8a02b6ff2c0d4c71e88885dd84d00c61%2FScreen%20Shot%202019-10-22%20at%204.51.11%20PM.png?generation=1571759502218985&alt=media)\n\n4. PhiK is a new and practical correlation coefficient based on several refinements to Pearson's hypothesis test of independence of two variables. The combined features of PhiK form an advantage over existing coefficients. First, it works consistently between categorical, ordinal and interval variables. Second, it captures non-linear dependency. Third, it reverts to the Pearson correlation coefficient in case of a bi-variate normal input distribution. These are useful features when studying the correlation matrix of variables with mixed types.\nThe presented algorithms are easy to use and available through this public Python library: the correlation analyzer package. Emphasis is paid to the proper evaluation of statistical significance of correlations and to the interpretation of variable relationships in a contingency table, in particular in case of low statistics samples.\nFor example, the Phi_K correlation analyzer package has been used to study surveys, insurance claims, correlograms, etc. \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2Fd1e2f169699ae7031292b447233c3f5f%2FScreen%20Shot%202019-10-22%20at%204.51.44%20PM.png?generation=1571759546699707&alt=media)\n\n**Missing Value Analysis**\n1. Count of missing values by features\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2Ff5643cfd2248a6137cb594220cc7ccfb%2FScreen%20Shot%202019-10-22%20at%204.58.07%20PM.png?generation=1571759916187209&alt=media)\n\n2. Missing Value Matrix\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2Fe1db67c66288f7ced14cfffab60bb588%2FScreen%20Shot%202019-10-22%20at%204.59.20%20PM.png?generation=1571759997011437&alt=media)\n\n3. Dendogram of features\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2F46b940462581d0fa3a22947b0edf5d25%2FScreen%20Shot%202019-10-22%20at%205.00.22%20PM.png?generation=1571760056113383&alt=media)", "votes": null }, { "id": "655123", "postDate": "10/22/2019 17:42:15", "content": "

@cyberia This is very insightful. Thanks for sharing good job! :)

", "rawMarkdown": "cyberia This is very insightful. Thanks for sharing good job! :)", "votes": null }, { "id": "655582", "postDate": "10/23/2019 08:09:01", "content": "

Nice!\nhow's your dendrogram generated?

", "rawMarkdown": "Nice!\nhow's your dendrogram generated?", "votes": null }, { "id": "663893", "postDate": "11/02/2019 20:11:57", "content": "

I am using pandas_profiler to generate this entire report including dendogram.

", "rawMarkdown": "I am using `pandas_profiler` to generate this entire report including dendogram.", "votes": null } ], "comments": [ { "id": 655123, "author_name": "themonologue", "author_url": "", "post_date": "10/22/2019 17:42:15", "content": "

@cyberia This is very insightful. Thanks for sharing good job! :)

", "votes": null, "replies": [] }, { "id": 655582, "author_name": "dhacker", "author_url": "", "post_date": "10/23/2019 08:09:01", "content": "

Nice!\nhow's your dendrogram generated?

", "votes": null, "replies": [ { "id": 663893, "author_name": "cyberia", "author_url": "", "post_date": "11/02/2019 20:11:57", "content": "

I am using pandas_profiler to generate this entire report including dendogram.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "655023": "1. Pearson's r : Pearson's correlation coefficient is the covariance of the two variables divided by the product of their standard deviations. The form of the definition involves a \"product moment\", that is, the mean (the first moment about the origin) of the product of the mean-adjusted random variables; hence the modifier product-moment in the name. \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2F86278b0ad8a84e0dcce82017ef6f7dd1%2FScreen%20Shot%202019-10-22%20at%204.48.40%20PM.png?generation=1571759350103588&alt=media)\n\n2. Spearman's rho : In statistics, Spearman's rank correlation coefficient or Spearman's rho, named after Charles Spearman and often denoted by the Greek letter ρ (rho) or as\nr_{s}, is a nonparametric measure of rank correlation (statistical dependence between the rankings of two variables). It assesses how well the relationship between two variables can be described using a monotonic function.\nThe Spearman correlation between two variables is equal to the Pearson correlation between the rank values of those two variables; while Pearson's correlation assesses linear relationships, Spearman's correlation assesses monotonic relationships (whether linear or not). If there are no repeated data values, a perfect Spearman correlation of +1 or −1 occurs when each of the variables is a perfect monotone function of the other.\nIntuitively, the Spearman correlation between two variables will be high when observations have a similar (or identical for a correlation of 1) rank (i.e. relative position label of the observations within the variable: 1st, 2nd, 3rd, etc.) between the two variables, and low when observations have a dissimilar (or fully opposed for a correlation of −1) rank between the two variables.\nSpearman's coefficient is appropriate for both continuous and discrete ordinal variables. Both Spearman's ρ and Kendall's τ can be formulated as special cases of a more general correlation coefficient.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2F90a4f6b15983b3881bf606c32c4ffa50%2FScreen%20Shot%202019-10-22%20at%204.50.02%20PM.png?generation=1571759428659907&alt=media)\n\n3. Kendall's Tau : In statistics, the Kendall rank correlation coefficient, commonly referred to as Kendall's tau coefficient (after the Greek letter τ), is a statistic used to measure the ordinal association between two measured quantities. A tau test is a non-parametric hypothesis test for statistical dependence based on the tau coefficient.\nIt is a measure of rank correlation: the similarity of the orderings of the data when ranked by each of the quantities. It is named after Maurice Kendall, who developed it in 1938, though Gustav Fechner had proposed a similar measure in the context of time series in 1897.\nIntuitively, the Kendall correlation between two variables will be high when observations have a similar (or identical for a correlation of 1) rank (i.e. relative position label of the observations within the variable: 1st, 2nd, 3rd, etc.) between the two variables, and low when observations have a dissimilar (or fully different for a correlation of −1) rank between the two variables. Both Kendall's τ and Spearman's ρ can be formulated as special cases of a more general correlation coefficient. ![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2F8a02b6ff2c0d4c71e88885dd84d00c61%2FScreen%20Shot%202019-10-22%20at%204.51.11%20PM.png?generation=1571759502218985&alt=media)\n\n4. PhiK is a new and practical correlation coefficient based on several refinements to Pearson's hypothesis test of independence of two variables. The combined features of PhiK form an advantage over existing coefficients. First, it works consistently between categorical, ordinal and interval variables. Second, it captures non-linear dependency. Third, it reverts to the Pearson correlation coefficient in case of a bi-variate normal input distribution. These are useful features when studying the correlation matrix of variables with mixed types.\nThe presented algorithms are easy to use and available through this public Python library: the correlation analyzer package. Emphasis is paid to the proper evaluation of statistical significance of correlations and to the interpretation of variable relationships in a contingency table, in particular in case of low statistics samples.\nFor example, the Phi_K correlation analyzer package has been used to study surveys, insurance claims, correlograms, etc. \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2Fd1e2f169699ae7031292b447233c3f5f%2FScreen%20Shot%202019-10-22%20at%204.51.44%20PM.png?generation=1571759546699707&alt=media)\n\n**Missing Value Analysis**\n1. Count of missing values by features\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2Ff5643cfd2248a6137cb594220cc7ccfb%2FScreen%20Shot%202019-10-22%20at%204.58.07%20PM.png?generation=1571759916187209&alt=media)\n\n2. Missing Value Matrix\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2Fe1db67c66288f7ced14cfffab60bb588%2FScreen%20Shot%202019-10-22%20at%204.59.20%20PM.png?generation=1571759997011437&alt=media)\n\n3. Dendogram of features\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F551520%2F46b940462581d0fa3a22947b0edf5d25%2FScreen%20Shot%202019-10-22%20at%205.00.22%20PM.png?generation=1571760056113383&alt=media)", "655123": "cyberia This is very insightful. Thanks for sharing good job! :)", "655582": "Nice!\nhow's your dendrogram generated?", "663893": "I am using `pandas_profiler` to generate this entire report including dendogram." }, "source": "meta" }