When inspecting the data, I found that there are 44 features which never take any value different from the empty string. Is this really the case, or am I confused? The features in question are:
\n\nL0_S15_F396\nL0_S15_F399\nL0_S15_F402\nL0_S15_F405\nL0_S15_F408\nL0_S15_F411\nL0_S15_F414\nL0_S15_F417\nL0_S15_F420\nL1_S24_F1157\nL1_S24_F1159\nL1_S24_F1160\nL1_S24_F1561\nL1_S24_F1563\nL1_S24_F1564\nL1_S24_F1577\nL1_S24_F1580\nL1_S25_F2407\nL1_S25_F2410\nL1_S25_F2413\nL1_S25_F2416\nL1_S25_F2419\nL1_S25_F2422\nL1_S25_F2425\nL1_S25_F2428\nL3_S36_F3927\nL3_S36_F3929\nL3_S36_F3931\nL3_S36_F3933\nL3_S36_F3935\nL3_S36_F3937\nL3_S36_F3939\nL3_S36_F3941\nL3_S39_F3965\nL3_S39_F3967\nL3_S39_F3969\nL3_S39_F3971\nL3_S39_F3973\nL3_S39_F3975\nL3_S39_F3977\nL3_S39_F3979\nL3_S46_F4134\nL3_S46_F4136\nL3_S46_F4137
", "messages": [ { "id": "131780", "postDate": "08/19/2016 15:54:18", "content": "When inspecting the data, I found that there are 44 features which never take any value different from the empty string. Is this really the case, or am I confused? The features in question are:
\n\nL0_S15_F396\nL0_S15_F399\nL0_S15_F402\nL0_S15_F405\nL0_S15_F408\nL0_S15_F411\nL0_S15_F414\nL0_S15_F417\nL0_S15_F420\nL1_S24_F1157\nL1_S24_F1159\nL1_S24_F1160\nL1_S24_F1561\nL1_S24_F1563\nL1_S24_F1564\nL1_S24_F1577\nL1_S24_F1580\nL1_S25_F2407\nL1_S25_F2410\nL1_S25_F2413\nL1_S25_F2416\nL1_S25_F2419\nL1_S25_F2422\nL1_S25_F2425\nL1_S25_F2428\nL3_S36_F3927\nL3_S36_F3929\nL3_S36_F3931\nL3_S36_F3933\nL3_S36_F3935\nL3_S36_F3937\nL3_S36_F3939\nL3_S36_F3941\nL3_S39_F3965\nL3_S39_F3967\nL3_S39_F3969\nL3_S39_F3971\nL3_S39_F3973\nL3_S39_F3975\nL3_S39_F3977\nL3_S39_F3979\nL3_S46_F4134\nL3_S46_F4136\nL3_S46_F4137
", "rawMarkdown": "When inspecting the data, I found that there are 44 features which never take any value different from the empty string. Is this really the case, or am I confused? The features in question are:\r\n\r\nL0_S15_F396\r\nL0_S15_F399\r\nL0_S15_F402\r\nL0_S15_F405\r\nL0_S15_F408\r\nL0_S15_F411\r\nL0_S15_F414\r\nL0_S15_F417\r\nL0_S15_F420\r\nL1_S24_F1157\r\nL1_S24_F1159\r\nL1_S24_F1160\r\nL1_S24_F1561\r\nL1_S24_F1563\r\nL1_S24_F1564\r\nL1_S24_F1577\r\nL1_S24_F1580\r\nL1_S25_F2407\r\nL1_S25_F2410\r\nL1_S25_F2413\r\nL1_S25_F2416\r\nL1_S25_F2419\r\nL1_S25_F2422\r\nL1_S25_F2425\r\nL1_S25_F2428\r\nL3_S36_F3927\r\nL3_S36_F3929\r\nL3_S36_F3931\r\nL3_S36_F3933\r\nL3_S36_F3935\r\nL3_S36_F3937\r\nL3_S36_F3939\r\nL3_S36_F3941\r\nL3_S39_F3965\r\nL3_S39_F3967\r\nL3_S39_F3969\r\nL3_S39_F3971\r\nL3_S39_F3973\r\nL3_S39_F3975\r\nL3_S39_F3977\r\nL3_S39_F3979\r\nL3_S46_F4134\r\nL3_S46_F4136\r\nL3_S46_F4137", "votes": null }, { "id": "131816", "postDate": "08/19/2016 21:29:43", "content": "I discovered a mistake in my script, which made it only consider the test dataset. After fixing this mistake it seems now that only eight columns are totally empty: L1_S24_F1157, L1_S24_F1159, \nL1_S24_F1160, \nL1_S24_F1561, \nL1_S24_F1563, \nL1_S24_F1564, \nL3_S46_F4136, \nL3_S46_F4137. My apologies for the mistake.
", "rawMarkdown": "I discovered a mistake in my script, which made it only consider the test dataset. After fixing this mistake it seems now that only eight columns are totally empty: L1_S24_F1157, L1_S24_F1159, \r\nL1_S24_F1160, \r\nL1_S24_F1561, \r\nL1_S24_F1563, \r\nL1_S24_F1564, \r\nL3_S46_F4136, \r\nL3_S46_F4137. My apologies for the mistake.", "votes": null }, { "id": "131842", "postDate": "08/20/2016 04:29:00", "content": "You should only consider the train dataset.
", "rawMarkdown": "You should only consider the train dataset.", "votes": null }, { "id": "131951", "postDate": "08/20/2016 21:59:04", "content": "It wouldn't surprise me if there are certain categorical features that never show a positive. (I'm assuming a value in a categorical column represents a certain "non-normal" attribute of the component, e.g., "rough surface", etc.)
", "rawMarkdown": "It wouldn't surprise me if there are certain categorical features that never show a positive. (I'm assuming a value in a categorical column represents a certain \"non-normal\" attribute of the component, e.g., \"rough surface\", etc.)", "votes": null }, { "id": "132139", "postDate": "08/22/2016 21:20:01", "content": "[quote=Allenbj;131842]
\n\nYou should only consider the train dataset.
\n\n[/quote]
\n\nNot entirely true. If a variable has zero variance in the test set, you might want to reconsider its worth for training your model.
", "rawMarkdown": "[quote=Allenbj;131842]\r\n\r\nYou should only consider the train dataset.\r\n\r\n[/quote]\r\n\r\nNot entirely true. If a variable has zero variance in the test set, you might want to reconsider its worth for training your model.", "votes": null }, { "id": "132140", "postDate": "08/22/2016 21:59:36", "content": "I would phrase it differently: if a variable is constant in the training set, it doesn't particularly matter how it behaves in the test set - you can't learn from it, as there's nothing to establish a relationship with the target.
", "rawMarkdown": "I would phrase it differently: if a variable is constant in the training set, it doesn't particularly matter how it behaves in the test set - you can't learn from it, as there's nothing to establish a relationship with the target.", "votes": null }, { "id": "132142", "postDate": "08/22/2016 22:11:58", "content": "[quote=Konrad Banachewicz;132140]
\n\nI would phrase it differently: if a variable is constant in the training set, it doesn't particularly matter how it behaves in the test set - you can't learn from it, as there's nothing to establish a relationship with the target.
\n\n[/quote]
\n\nOf course what you say is true and the more important thing to check in the situation at hand. But my quote was targeting the word 'only' in Allenbj's statement. If we try to reduce the size of the dataset, it's worth considering to remove variables that have zero-variance in the test set from training entirely. At least in the Kaggle world, where the test set remains constant and we don't compromise on the future, should the test set change.
", "rawMarkdown": "[quote=Konrad Banachewicz;132140]\r\n\r\nI would phrase it differently: if a variable is constant in the training set, it doesn't particularly matter how it behaves in the test set - you can't learn from it, as there's nothing to establish a relationship with the target.\r\n\r\n[/quote]\r\n\r\nOf course what you say is true and the more important thing to check in the situation at hand. But my quote was targeting the word 'only' in Allenbj's statement. If we try to reduce the size of the dataset, it's worth considering to remove variables that have zero-variance in the test set from training entirely. At least in the Kaggle world, where the test set remains constant and we don't compromise on the future, should the test set change.", "votes": null } ], "comments": [ { "id": 131816, "author_name": "jochenvoss", "author_url": "", "post_date": "08/19/2016 21:29:43", "content": "I discovered a mistake in my script, which made it only consider the test dataset. After fixing this mistake it seems now that only eight columns are totally empty: L1_S24_F1157, L1_S24_F1159, \nL1_S24_F1160, \nL1_S24_F1561, \nL1_S24_F1563, \nL1_S24_F1564, \nL3_S46_F4136, \nL3_S46_F4137. My apologies for the mistake.
", "votes": null, "replies": [] }, { "id": 131842, "author_name": "", "author_url": "", "post_date": "08/20/2016 04:29:00", "content": "You should only consider the train dataset.
", "votes": null, "replies": [] }, { "id": 131951, "author_name": "inversion", "author_url": "", "post_date": "08/20/2016 21:59:04", "content": "It wouldn't surprise me if there are certain categorical features that never show a positive. (I'm assuming a value in a categorical column represents a certain "non-normal" attribute of the component, e.g., "rough surface", etc.)
", "votes": null, "replies": [] }, { "id": 132139, "author_name": "tobycheese", "author_url": "", "post_date": "08/22/2016 21:20:01", "content": "[quote=Allenbj;131842]
\n\nYou should only consider the train dataset.
\n\n[/quote]
\n\nNot entirely true. If a variable has zero variance in the test set, you might want to reconsider its worth for training your model.
", "votes": null, "replies": [] }, { "id": 132140, "author_name": "konradb", "author_url": "", "post_date": "08/22/2016 21:59:36", "content": "I would phrase it differently: if a variable is constant in the training set, it doesn't particularly matter how it behaves in the test set - you can't learn from it, as there's nothing to establish a relationship with the target.
", "votes": null, "replies": [] }, { "id": 132142, "author_name": "tobycheese", "author_url": "", "post_date": "08/22/2016 22:11:58", "content": "[quote=Konrad Banachewicz;132140]
\n\nI would phrase it differently: if a variable is constant in the training set, it doesn't particularly matter how it behaves in the test set - you can't learn from it, as there's nothing to establish a relationship with the target.
\n\n[/quote]
\n\nOf course what you say is true and the more important thing to check in the situation at hand. But my quote was targeting the word 'only' in Allenbj's statement. If we try to reduce the size of the dataset, it's worth considering to remove variables that have zero-variance in the test set from training entirely. At least in the Kaggle world, where the test set remains constant and we don't compromise on the future, should the test set change.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "131780": "When inspecting the data, I found that there are 44 features which never take any value different from the empty string. Is this really the case, or am I confused? The features in question are:\r\n\r\nL0_S15_F396\r\nL0_S15_F399\r\nL0_S15_F402\r\nL0_S15_F405\r\nL0_S15_F408\r\nL0_S15_F411\r\nL0_S15_F414\r\nL0_S15_F417\r\nL0_S15_F420\r\nL1_S24_F1157\r\nL1_S24_F1159\r\nL1_S24_F1160\r\nL1_S24_F1561\r\nL1_S24_F1563\r\nL1_S24_F1564\r\nL1_S24_F1577\r\nL1_S24_F1580\r\nL1_S25_F2407\r\nL1_S25_F2410\r\nL1_S25_F2413\r\nL1_S25_F2416\r\nL1_S25_F2419\r\nL1_S25_F2422\r\nL1_S25_F2425\r\nL1_S25_F2428\r\nL3_S36_F3927\r\nL3_S36_F3929\r\nL3_S36_F3931\r\nL3_S36_F3933\r\nL3_S36_F3935\r\nL3_S36_F3937\r\nL3_S36_F3939\r\nL3_S36_F3941\r\nL3_S39_F3965\r\nL3_S39_F3967\r\nL3_S39_F3969\r\nL3_S39_F3971\r\nL3_S39_F3973\r\nL3_S39_F3975\r\nL3_S39_F3977\r\nL3_S39_F3979\r\nL3_S46_F4134\r\nL3_S46_F4136\r\nL3_S46_F4137", "131816": "I discovered a mistake in my script, which made it only consider the test dataset. After fixing this mistake it seems now that only eight columns are totally empty: L1_S24_F1157, L1_S24_F1159, \r\nL1_S24_F1160, \r\nL1_S24_F1561, \r\nL1_S24_F1563, \r\nL1_S24_F1564, \r\nL3_S46_F4136, \r\nL3_S46_F4137. My apologies for the mistake.", "131842": "You should only consider the train dataset.", "131951": "It wouldn't surprise me if there are certain categorical features that never show a positive. (I'm assuming a value in a categorical column represents a certain \"non-normal\" attribute of the component, e.g., \"rough surface\", etc.)", "132139": "[quote=Allenbj;131842]\r\n\r\nYou should only consider the train dataset.\r\n\r\n[/quote]\r\n\r\nNot entirely true. If a variable has zero variance in the test set, you might want to reconsider its worth for training your model.", "132140": "I would phrase it differently: if a variable is constant in the training set, it doesn't particularly matter how it behaves in the test set - you can't learn from it, as there's nothing to establish a relationship with the target.", "132142": "[quote=Konrad Banachewicz;132140]\r\n\r\nI would phrase it differently: if a variable is constant in the training set, it doesn't particularly matter how it behaves in the test set - you can't learn from it, as there's nothing to establish a relationship with the target.\r\n\r\n[/quote]\r\n\r\nOf course what you say is true and the more important thing to check in the situation at hand. But my quote was targeting the word 'only' in Allenbj's statement. If we try to reduce the size of the dataset, it's worth considering to remove variables that have zero-variance in the test set from training entirely. At least in the Kaggle world, where the test set remains constant and we don't compromise on the future, should the test set change." }, "source": "meta" }