Hi,\nI have just entered this competition and I have seen that the data is not real, it has been generated. But does this data have a meaning ? \nIn the test.csv and the train.csv files, we have three features, with numerical values but without any explanations in the overview !\nSo should I just drop these tree features and only use the other files ? Or does these three features have a meaning that I need to create from the other files ?\nI don't really like the idea to feed unknown data to a machine learning algorithm and wait for a random result...
", "messages": [ { "id": "428420", "postDate": "11/27/2018 08:51:31", "content": "Hi,\nI have just entered this competition and I have seen that the data is not real, it has been generated. But does this data have a meaning ? \nIn the test.csv and the train.csv files, we have three features, with numerical values but without any explanations in the overview !\nSo should I just drop these tree features and only use the other files ? Or does these three features have a meaning that I need to create from the other files ?\nI don't really like the idea to feed unknown data to a machine learning algorithm and wait for a random result...
", "rawMarkdown": "Hi,\nI have just entered this competition and I have seen that the data is not real, it has been generated. But does this data have a meaning ? \nIn the test.csv and the train.csv files, we have three features, with numerical values but without any explanations in the overview !\nSo should I just drop these tree features and only use the other files ? Or does these three features have a meaning that I need to create from the other files ?\nI don't really like the idea to feed unknown data to a machine learning algorithm and wait for a random result...", "votes": null }, { "id": "428426", "postDate": "11/27/2018 08:56:52", "content": "There are plenty of ways to understand unknown data. Look for correlation with the target variable, see if you can bring in another variable using those etc. Don't drop them.
", "rawMarkdown": "There are plenty of ways to understand unknown data. Look for correlation with the target variable, see if you can bring in another variable using those etc. Don't drop them.", "votes": null }, { "id": "428434", "postDate": "11/27/2018 09:06:53", "content": "Thanks, I will try this approach. I will try to give them a meaning !
", "rawMarkdown": "Thanks, I will try this approach. I will try to give them a meaning !", "votes": null }, { "id": "428717", "postDate": "11/27/2018 18:47:52", "content": "Dealing with (partially) anonymized data is a regular thing in Kaggle competitions, usually for privacy reasons - but it's puzzling here given the explicit info that the data is simulated.
", "rawMarkdown": "Dealing with (partially) anonymized data is a regular thing in Kaggle competitions, usually for privacy reasons - but it's puzzling here given the explicit info that the data is simulated.", "votes": null }, { "id": "428772", "postDate": "11/27/2018 21:30:58", "content": "I guess the anonymized attributes are some highly specific or derived values which are confidential for business reasons.
", "rawMarkdown": "I guess the anonymized attributes are some highly specific or derived values which are confidential for business reasons.", "votes": null }, { "id": "428875", "postDate": "11/28/2018 02:28:52", "content": "If you are concerned about how to handle anonymized data you can take a look at the Santander Value Prediction challenge https://www.kaggle.com/c/santander-value-prediction-challenge also you can refer to this video for getting more insights on anonymized data https://www.coursera.org/lecture/competitive-data-science/exploring-anonymized-data-qJHOb.
", "rawMarkdown": "If you are concerned about how to handle anonymized data you can take a look at the Santander Value Prediction challenge https://www.kaggle.com/c/santander-value-prediction-challenge also you can refer to this video for getting more insights on anonymized data https://www.coursera.org/lecture/competitive-data-science/exploring-anonymized-data-qJHOb.", "votes": null }, { "id": "430688", "postDate": "11/30/2018 20:57:22", "content": "Pierre-Nicolas, I feel your pain. From the business perspective, it is not good. Anyway:
\n\nTrainData <- read.csv(\"H:\\NewFiles\\Kaggle competitions\\Elo Merchant Category Recommendation\\Data-Elo\\train.csv\")
\n\n\n", "rawMarkdown": "Pierre-Nicolas, I feel your pain. From the business perspective, it is not good. Anyway:\n\nTrainData <- read.csv(\"H:\\\\NewFiles\\\\Kaggle competitions\\\\Elo Merchant Category Recommendation\\\\Data-Elo\\\\train.csv\")\n\n> cor.test(TrainData$target, TrainData$feature_1, method=c(\"pearson\", \"kendall\", \"spearman\"))\n-0.01425105 \n\n> cor.test(TrainData$target, TrainData$feature_2, method=c(\"pearson\", \"kendall\", \"spearman\"))\n-0.006241765\n\n> cor.test(TrainData$target, TrainData$feature_3, method=c(\"pearson\", \"kendall\", \"spearman\"))\n-0.008125281", "votes": null } ], "comments": [ { "id": 428426, "author_name": "mcmuralishclint", "author_url": "", "post_date": "11/27/2018 08:56:52", "content": "cor.test(TrainData$target, TrainData$feature_1, method=c(\"pearson\", \"kendall\", \"spearman\"))\n -0.01425105
\n \ncor.test(TrainData$target, TrainData$feature_2, method=c(\"pearson\", \"kendall\", \"spearman\"))\n -0.006241765
\n \ncor.test(TrainData$target, TrainData$feature_3, method=c(\"pearson\", \"kendall\", \"spearman\"))\n -0.008125281
\n
There are plenty of ways to understand unknown data. Look for correlation with the target variable, see if you can bring in another variable using those etc. Don't drop them.
", "votes": null, "replies": [ { "id": 428434, "author_name": "pierrenicolaspiquin", "author_url": "", "post_date": "11/27/2018 09:06:53", "content": "Thanks, I will try this approach. I will try to give them a meaning !
", "votes": null, "replies": [] }, { "id": 430688, "author_name": "machado", "author_url": "", "post_date": "11/30/2018 20:57:22", "content": "Pierre-Nicolas, I feel your pain. From the business perspective, it is not good. Anyway:
\n\nTrainData <- read.csv(\"H:\\NewFiles\\Kaggle competitions\\Elo Merchant Category Recommendation\\Data-Elo\\train.csv\")
\n\n\n", "votes": null, "replies": [] } ] }, { "id": 428717, "author_name": "konradb", "author_url": "", "post_date": "11/27/2018 18:47:52", "content": "cor.test(TrainData$target, TrainData$feature_1, method=c(\"pearson\", \"kendall\", \"spearman\"))\n -0.01425105
\n \ncor.test(TrainData$target, TrainData$feature_2, method=c(\"pearson\", \"kendall\", \"spearman\"))\n -0.006241765
\n \ncor.test(TrainData$target, TrainData$feature_3, method=c(\"pearson\", \"kendall\", \"spearman\"))\n -0.008125281
\n
Dealing with (partially) anonymized data is a regular thing in Kaggle competitions, usually for privacy reasons - but it's puzzling here given the explicit info that the data is simulated.
", "votes": null, "replies": [] }, { "id": 428772, "author_name": "gykovacs", "author_url": "", "post_date": "11/27/2018 21:30:58", "content": "I guess the anonymized attributes are some highly specific or derived values which are confidential for business reasons.
", "votes": null, "replies": [] }, { "id": 428875, "author_name": "ishaan45", "author_url": "", "post_date": "11/28/2018 02:28:52", "content": "If you are concerned about how to handle anonymized data you can take a look at the Santander Value Prediction challenge https://www.kaggle.com/c/santander-value-prediction-challenge also you can refer to this video for getting more insights on anonymized data https://www.coursera.org/lecture/competitive-data-science/exploring-anonymized-data-qJHOb.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "428420": "Hi,\nI have just entered this competition and I have seen that the data is not real, it has been generated. But does this data have a meaning ? \nIn the test.csv and the train.csv files, we have three features, with numerical values but without any explanations in the overview !\nSo should I just drop these tree features and only use the other files ? Or does these three features have a meaning that I need to create from the other files ?\nI don't really like the idea to feed unknown data to a machine learning algorithm and wait for a random result...", "428426": "There are plenty of ways to understand unknown data. Look for correlation with the target variable, see if you can bring in another variable using those etc. Don't drop them.", "428434": "Thanks, I will try this approach. I will try to give them a meaning !", "428717": "Dealing with (partially) anonymized data is a regular thing in Kaggle competitions, usually for privacy reasons - but it's puzzling here given the explicit info that the data is simulated.", "428772": "I guess the anonymized attributes are some highly specific or derived values which are confidential for business reasons.", "428875": "If you are concerned about how to handle anonymized data you can take a look at the Santander Value Prediction challenge https://www.kaggle.com/c/santander-value-prediction-challenge also you can refer to this video for getting more insights on anonymized data https://www.coursera.org/lecture/competitive-data-science/exploring-anonymized-data-qJHOb.", "430688": "Pierre-Nicolas, I feel your pain. From the business perspective, it is not good. Anyway:\n\nTrainData <- read.csv(\"H:\\\\NewFiles\\\\Kaggle competitions\\\\Elo Merchant Category Recommendation\\\\Data-Elo\\\\train.csv\")\n\n> cor.test(TrainData$target, TrainData$feature_1, method=c(\"pearson\", \"kendall\", \"spearman\"))\n-0.01425105 \n\n> cor.test(TrainData$target, TrainData$feature_2, method=c(\"pearson\", \"kendall\", \"spearman\"))\n-0.006241765\n\n> cor.test(TrainData$target, TrainData$feature_3, method=c(\"pearson\", \"kendall\", \"spearman\"))\n-0.008125281" }, "source": "meta" }