Hi Kagglers: \nI tried to visualize the data using PCA.\nI use 35,000 rows of the numerical data. NaN data is replaced with “-2”.\nThe results look quite surprising. It seems there are 3 clusters. I wonder why it looks like that. Has anyone got a similar result?
\n\nAnother question is it beneficial to use unsupervised learning at first to group the data into 3 clusters and then train a classifier for each cluster? Or it is not necessary because the algorithms like AdaBoost can automatically handle this kind of situation.
", "messages": [ { "id": "132839", "postDate": "08/29/2016 17:32:45", "content": "Hi Kagglers: \nI tried to visualize the data using PCA.\nI use 35,000 rows of the numerical data. NaN data is replaced with “-2”.\nThe results look quite surprising. It seems there are 3 clusters. I wonder why it looks like that. Has anyone got a similar result?
\n\nAnother question is it beneficial to use unsupervised learning at first to group the data into 3 clusters and then train a classifier for each cluster? Or it is not necessary because the algorithms like AdaBoost can automatically handle this kind of situation.
", "rawMarkdown": "Hi Kagglers: \r\nI tried to visualize the data using PCA.\r\nI use 35,000 rows of the numerical data. NaN data is replaced with “-2”.\r\nThe results look quite surprising. It seems there are 3 clusters. I wonder why it looks like that. Has anyone got a similar result?\r\n\r\nAnother question is it beneficial to use unsupervised learning at first to group the data into 3 clusters and then train a classifier for each cluster? Or it is not necessary because the algorithms like AdaBoost can automatically handle this kind of situation.", "votes": null }, { "id": "132864", "postDate": "08/30/2016 01:28:11", "content": "What was your motivation for replacing the NaN data with -2? Wouldn't that change the underlying per-feature distribution and the data's overall covariance? I haven't played with the data too much, but I suspect that you're capturing some underlying process about how the missing data is distributed (e.g: there may be three "types" of missing data situations) and it's messing up your PCA results. Perhaps try replacing the NaN values with the mean value of each feature and see how it changes.
\n\nFor your second point, that might be worthwhile if you can identify different classes of data that behave wildly different from one another and seem to have different processes that determine whether they pass or fail. However, the pass/fail cases don't look separable at all from your PCA results.
", "rawMarkdown": "What was your motivation for replacing the NaN data with -2? Wouldn't that change the underlying per-feature distribution and the data's overall covariance? I haven't played with the data too much, but I suspect that you're capturing some underlying process about how the missing data is distributed (e.g: there may be three \"types\" of missing data situations) and it's messing up your PCA results. Perhaps try replacing the NaN values with the mean value of each feature and see how it changes.\r\n\r\nFor your second point, that might be worthwhile if you can identify different classes of data that behave wildly different from one another and seem to have different processes that determine whether they pass or fail. However, the pass/fail cases don't look separable at all from your PCA results.", "votes": null }, { "id": "132886", "postDate": "08/30/2016 08:13:46", "content": "What is the explained variance ratio when using two components?
", "rawMarkdown": "What is the explained variance ratio when using two components?", "votes": null }, { "id": "132898", "postDate": "08/30/2016 10:07:19", "content": "[quote=Dalaska;132839]
\n\nI use 35,000 rows of the numerical data. NaN data is replaced with “-2”.\nThe results look quite surprising. It seems there are 3 clusters. I wonder why it looks like that. Has anyone got a similar result?
\n\n[/quote]
\n\nIf you are using R, use EOF in sinkr package (devtools:::install_github("marchtaylor/sinkr")) to use PCA with missing values.
Replacing missing values by a specific value beforehand (like you did) will bias PCA towards a specific structure, which is the sparse structure of the observations (and not the structure of the features/observations) when sparsity is high (like in the numeric train dataset), and when the distance of the replacement value to the other values is high.
", "rawMarkdown": "[quote=Dalaska;132839]\r\n\r\nI use 35,000 rows of the numerical data. **NaN data is replaced with “-2”.**\r\nThe results look quite surprising. It seems there are 3 clusters. I wonder why it looks like that. Has anyone got a similar result?\r\n\r\n[/quote]\r\n\r\nIf you are using R, use EOF in sinkr package (`devtools:::install_github(\"marchtaylor/sinkr\")`) to use PCA with missing values.\r\n\r\nReplacing missing values by a specific value beforehand (like you did) will bias PCA towards a specific structure, which is the sparse structure of the observations (and not the structure of the features/observations) when sparsity is high (like in the numeric train dataset), and when the distance of the replacement value to the other values is high.", "votes": null }, { "id": "132941", "postDate": "08/30/2016 14:06:03", "content": "Hi Laurae:\nThat make sense. The sparsity of the dataset is very high.
\n\nMy original thought its. I assume the data is not missing at random. For example, some part go through extra process so that has more features than others. So I assign a very different number to them as if they are from a different category. The original data seems to range from -1 to 1. So I choose -2, hope it can be handled by algorithms like random forest.
\n\nBut since the sparsity of dataset is very high, replace them with a different number will definitely change the variance, so maybe not a good idea for PCA.
\n\nThanks!
", "rawMarkdown": "Hi Laurae:\r\nThat make sense. The sparsity of the dataset is very high. \r\n\r\nMy original thought its. I assume the data is not missing at random. For example, some part go through extra process so that has more features than others. So I assign a very different number to them as if they are from a different category. The original data seems to range from -1 to 1. So I choose -2, hope it can be handled by algorithms like random forest.\r\n\r\nBut since the sparsity of dataset is very high, replace them with a different number will definitely change the variance, so maybe not a good idea for PCA.\r\n\r\nThanks!", "votes": null }, { "id": "133108", "postDate": "08/31/2016 09:17:49", "content": "Take a look at the production map here:
\nhttps://www.kaggle.com/c/bosch-production-line-performance/forums/t/23042/production-map
\n I'm guessing that's why you have such strong groupings
Thanks John! \nWow, cool. So there are actually thousands of different parts. 3 clusters come from 3 product family. \nIn this case, I think replacing the NaN number with a different value like "-2" actually make sense than replace it with the average, because the data is not missing randomly. It adds additional information to reflect the product family.
", "rawMarkdown": "Thanks John! \r\nWow, cool. So there are actually thousands of different parts. 3 clusters come from 3 product family. \r\nIn this case, I think replacing the NaN number with a different value like \"-2\" actually make sense than replace it with the average, because the data is not missing randomly. It adds additional information to reflect the product family.", "votes": null }, { "id": "133136", "postDate": "08/31/2016 14:25:16", "content": "If you want to capture the missingness then replacing with -2 makes sense, however its a weird mixture of capturing the variation and the misssingness. More sense would make to replace NAs with a higher number such as -+1000 and dont scale in the PCA, as then the variation of the nonmissing values is marginal and you capture only the missingness pattern!
", "rawMarkdown": "If you want to capture the missingness then replacing with -2 makes sense, however its a weird mixture of capturing the variation and the misssingness. More sense would make to replace NAs with a higher number such as -+1000 and dont scale in the PCA, as then the variation of the nonmissing values is marginal and you capture only the missingness pattern!", "votes": null }, { "id": "133147", "postDate": "08/31/2016 15:32:53", "content": "[quote=Malte Nalenz;133136]
\n\nIf you want to capture the missingness then replacing with -2 makes sense, however its a weird mixture of capturing the variation and the misssingness. More sense would make to replace NAs with a higher number such as -+1000 and dont scale in the PCA, as then the variation of the nonmissing values is marginal and you capture only the missingness pattern!
\n\n[/quote]\nMaybe just replacing missing/non-missing value with binary values wouldn't do the same ?
", "rawMarkdown": "[quote=Malte Nalenz;133136]\r\n\r\nIf you want to capture the missingness then replacing with -2 makes sense, however its a weird mixture of capturing the variation and the misssingness. More sense would make to replace NAs with a higher number such as -+1000 and dont scale in the PCA, as then the variation of the nonmissing values is marginal and you capture only the missingness pattern! \r\n\r\n\r\n[/quote]\r\nMaybe just replacing missing/non-missing value with binary values wouldn't do the same ?", "votes": null }, { "id": "133154", "postDate": "08/31/2016 16:29:40", "content": "[quote=SylvainBrouillat;133147]\nMaybe just replacing missing/non-missing value with binary values wouldn't do the same ?
\n\n[/quote]
\n\nYes that would do the same and is the way to go i think, while using smaller values leads to something else.
", "rawMarkdown": "[quote=SylvainBrouillat;133147]\r\nMaybe just replacing missing/non-missing value with binary values wouldn't do the same ?\r\n\r\n[/quote]\r\n\r\nYes that would do the same and is the way to go i think, while using smaller values leads to something else.", "votes": null }, { "id": "133323", "postDate": "09/01/2016 14:07:05", "content": "[quote=Dalaska;133121]
\n\nThanks John! \nWow, cool. So there are actually thousands of different parts. 3 clusters come from 3 product family. \nIn this case, I think replacing the NaN number with a different value like "-2" actually make sense than replace it with the average, because the data is not missing randomly. It adds additional information to reflect the product family.
\n\n[/quote]
\n\nCouldn't you just predict the missing values (either with the mean or Randomforest or something) and just assign dummy variables to those values - and exclude the dummy variable component from your PCA analysis?
", "rawMarkdown": "[quote=Dalaska;133121]\r\n\r\nThanks John! \r\nWow, cool. So there are actually thousands of different parts. 3 clusters come from 3 product family. \r\nIn this case, I think replacing the NaN number with a different value like \"-2\" actually make sense than replace it with the average, because the data is not missing randomly. It adds additional information to reflect the product family.\r\n\r\n\r\n\r\n[/quote]\r\n\r\nCouldn't you just predict the missing values (either with the mean or Randomforest or something) and just assign dummy variables to those values - and exclude the dummy variable component from your PCA analysis?", "votes": null } ], "comments": [ { "id": 132864, "author_name": "flynnm", "author_url": "", "post_date": "08/30/2016 01:28:11", "content": "What was your motivation for replacing the NaN data with -2? Wouldn't that change the underlying per-feature distribution and the data's overall covariance? I haven't played with the data too much, but I suspect that you're capturing some underlying process about how the missing data is distributed (e.g: there may be three "types" of missing data situations) and it's messing up your PCA results. Perhaps try replacing the NaN values with the mean value of each feature and see how it changes.
\n\nFor your second point, that might be worthwhile if you can identify different classes of data that behave wildly different from one another and seem to have different processes that determine whether they pass or fail. However, the pass/fail cases don't look separable at all from your PCA results.
", "votes": null, "replies": [] }, { "id": 132886, "author_name": "odjansse", "author_url": "", "post_date": "08/30/2016 08:13:46", "content": "What is the explained variance ratio when using two components?
", "votes": null, "replies": [] }, { "id": 132898, "author_name": "laurae2", "author_url": "", "post_date": "08/30/2016 10:07:19", "content": "[quote=Dalaska;132839]
\n\nI use 35,000 rows of the numerical data. NaN data is replaced with “-2”.\nThe results look quite surprising. It seems there are 3 clusters. I wonder why it looks like that. Has anyone got a similar result?
\n\n[/quote]
\n\nIf you are using R, use EOF in sinkr package (devtools:::install_github("marchtaylor/sinkr")) to use PCA with missing values.
Replacing missing values by a specific value beforehand (like you did) will bias PCA towards a specific structure, which is the sparse structure of the observations (and not the structure of the features/observations) when sparsity is high (like in the numeric train dataset), and when the distance of the replacement value to the other values is high.
", "votes": null, "replies": [] }, { "id": 132941, "author_name": "dariusalaska", "author_url": "", "post_date": "08/30/2016 14:06:03", "content": "Hi Laurae:\nThat make sense. The sparsity of the dataset is very high.
\n\nMy original thought its. I assume the data is not missing at random. For example, some part go through extra process so that has more features than others. So I assign a very different number to them as if they are from a different category. The original data seems to range from -1 to 1. So I choose -2, hope it can be handled by algorithms like random forest.
\n\nBut since the sparsity of dataset is very high, replace them with a different number will definitely change the variance, so maybe not a good idea for PCA.
\n\nThanks!
", "votes": null, "replies": [] }, { "id": 133108, "author_name": "johnsavageireland", "author_url": "", "post_date": "08/31/2016 09:17:49", "content": "Take a look at the production map here:
\nhttps://www.kaggle.com/c/bosch-production-line-performance/forums/t/23042/production-map
\n I'm guessing that's why you have such strong groupings
Thanks John! \nWow, cool. So there are actually thousands of different parts. 3 clusters come from 3 product family. \nIn this case, I think replacing the NaN number with a different value like "-2" actually make sense than replace it with the average, because the data is not missing randomly. It adds additional information to reflect the product family.
", "votes": null, "replies": [] }, { "id": 133136, "author_name": "malten", "author_url": "", "post_date": "08/31/2016 14:25:16", "content": "If you want to capture the missingness then replacing with -2 makes sense, however its a weird mixture of capturing the variation and the misssingness. More sense would make to replace NAs with a higher number such as -+1000 and dont scale in the PCA, as then the variation of the nonmissing values is marginal and you capture only the missingness pattern!
", "votes": null, "replies": [] }, { "id": 133147, "author_name": "sbrouil", "author_url": "", "post_date": "08/31/2016 15:32:53", "content": "[quote=Malte Nalenz;133136]
\n\nIf you want to capture the missingness then replacing with -2 makes sense, however its a weird mixture of capturing the variation and the misssingness. More sense would make to replace NAs with a higher number such as -+1000 and dont scale in the PCA, as then the variation of the nonmissing values is marginal and you capture only the missingness pattern!
\n\n[/quote]\nMaybe just replacing missing/non-missing value with binary values wouldn't do the same ?
", "votes": null, "replies": [] }, { "id": 133154, "author_name": "malten", "author_url": "", "post_date": "08/31/2016 16:29:40", "content": "[quote=SylvainBrouillat;133147]\nMaybe just replacing missing/non-missing value with binary values wouldn't do the same ?
\n\n[/quote]
\n\nYes that would do the same and is the way to go i think, while using smaller values leads to something else.
", "votes": null, "replies": [] }, { "id": 133323, "author_name": "ehbick01", "author_url": "", "post_date": "09/01/2016 14:07:05", "content": "[quote=Dalaska;133121]
\n\nThanks John! \nWow, cool. So there are actually thousands of different parts. 3 clusters come from 3 product family. \nIn this case, I think replacing the NaN number with a different value like "-2" actually make sense than replace it with the average, because the data is not missing randomly. It adds additional information to reflect the product family.
\n\n[/quote]
\n\nCouldn't you just predict the missing values (either with the mean or Randomforest or something) and just assign dummy variables to those values - and exclude the dummy variable component from your PCA analysis?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "132839": "Hi Kagglers: \r\nI tried to visualize the data using PCA.\r\nI use 35,000 rows of the numerical data. NaN data is replaced with “-2”.\r\nThe results look quite surprising. It seems there are 3 clusters. I wonder why it looks like that. Has anyone got a similar result?\r\n\r\nAnother question is it beneficial to use unsupervised learning at first to group the data into 3 clusters and then train a classifier for each cluster? Or it is not necessary because the algorithms like AdaBoost can automatically handle this kind of situation.", "132864": "What was your motivation for replacing the NaN data with -2? Wouldn't that change the underlying per-feature distribution and the data's overall covariance? I haven't played with the data too much, but I suspect that you're capturing some underlying process about how the missing data is distributed (e.g: there may be three \"types\" of missing data situations) and it's messing up your PCA results. Perhaps try replacing the NaN values with the mean value of each feature and see how it changes.\r\n\r\nFor your second point, that might be worthwhile if you can identify different classes of data that behave wildly different from one another and seem to have different processes that determine whether they pass or fail. However, the pass/fail cases don't look separable at all from your PCA results.", "132886": "What is the explained variance ratio when using two components?", "132898": "[quote=Dalaska;132839]\r\n\r\nI use 35,000 rows of the numerical data. **NaN data is replaced with “-2”.**\r\nThe results look quite surprising. It seems there are 3 clusters. I wonder why it looks like that. Has anyone got a similar result?\r\n\r\n[/quote]\r\n\r\nIf you are using R, use EOF in sinkr package (`devtools:::install_github(\"marchtaylor/sinkr\")`) to use PCA with missing values.\r\n\r\nReplacing missing values by a specific value beforehand (like you did) will bias PCA towards a specific structure, which is the sparse structure of the observations (and not the structure of the features/observations) when sparsity is high (like in the numeric train dataset), and when the distance of the replacement value to the other values is high.", "132941": "Hi Laurae:\r\nThat make sense. The sparsity of the dataset is very high. \r\n\r\nMy original thought its. I assume the data is not missing at random. For example, some part go through extra process so that has more features than others. So I assign a very different number to them as if they are from a different category. The original data seems to range from -1 to 1. So I choose -2, hope it can be handled by algorithms like random forest.\r\n\r\nBut since the sparsity of dataset is very high, replace them with a different number will definitely change the variance, so maybe not a good idea for PCA.\r\n\r\nThanks!", "133108": "Take a look at the production map here: \r\nhttps://www.kaggle.com/c/bosch-production-line-performance/forums/t/23042/production-map \r\n I'm guessing that's why you have such strong groupings", "133121": "Thanks John! \r\nWow, cool. So there are actually thousands of different parts. 3 clusters come from 3 product family. \r\nIn this case, I think replacing the NaN number with a different value like \"-2\" actually make sense than replace it with the average, because the data is not missing randomly. It adds additional information to reflect the product family.", "133136": "If you want to capture the missingness then replacing with -2 makes sense, however its a weird mixture of capturing the variation and the misssingness. More sense would make to replace NAs with a higher number such as -+1000 and dont scale in the PCA, as then the variation of the nonmissing values is marginal and you capture only the missingness pattern!", "133147": "[quote=Malte Nalenz;133136]\r\n\r\nIf you want to capture the missingness then replacing with -2 makes sense, however its a weird mixture of capturing the variation and the misssingness. More sense would make to replace NAs with a higher number such as -+1000 and dont scale in the PCA, as then the variation of the nonmissing values is marginal and you capture only the missingness pattern! \r\n\r\n\r\n[/quote]\r\nMaybe just replacing missing/non-missing value with binary values wouldn't do the same ?", "133154": "[quote=SylvainBrouillat;133147]\r\nMaybe just replacing missing/non-missing value with binary values wouldn't do the same ?\r\n\r\n[/quote]\r\n\r\nYes that would do the same and is the way to go i think, while using smaller values leads to something else.", "133323": "[quote=Dalaska;133121]\r\n\r\nThanks John! \r\nWow, cool. So there are actually thousands of different parts. 3 clusters come from 3 product family. \r\nIn this case, I think replacing the NaN number with a different value like \"-2\" actually make sense than replace it with the average, because the data is not missing randomly. It adds additional information to reflect the product family.\r\n\r\n\r\n\r\n[/quote]\r\n\r\nCouldn't you just predict the missing values (either with the mean or Randomforest or something) and just assign dummy variables to those values - and exclude the dummy variable component from your PCA analysis?" }, "source": "meta" }