{ "id": 73425, "title": "Some Concerns", "url": "/competitions/elo-merchant-category-recommendation/discussion/73425", "author_name": "", "post_date": "2018-12-03T03:45:54.679344400Z", "votes": 103, "comment_count": 23, "views": 0, "content": "
    \n
  1. As noted in several different kernels, for example here:\nhttps://www.kaggle.com/fabiendaniel/elo-world One of the most\nimportant features to most models is the amount of elapsed time. If\nI was organizing the competition I would find this a little alarming\nbecause despite the creation of 1000+ features from the original\ndata, a simple sequential count is more useful than the rest of the\ndata provided to competitors.
    2. \n
  2. As noted in this thread, more than HALF of the RMSE is due to about\n2000 outliers with loyalty scores of ~ -33. If you try to replace\nthese outliers with the median, retrain the model and submit, you'll\nfind your leaderboard score is WORSE than if you keep the outliers\nat their original values. This tells you that outliers are included\nin the test set. Furthermore, given the magnitude of the impact of\noutliers on the RMSE score, the entire focus of this conmpetition\n(should be) predicting those outliers as accurately as possible and\nlargely ignoring the other 99% of data points. This should also\nconcern the competition organizers because unless they care strictly\nabout those outliers, it's highly unlikely the winning model will\nprovide anything useful.
    3. \n
  3. As further evidence of the above concern, the RMSE on the leader\nboard is ~3.69. Since RMSE is in the original units of the target we\ncan get an idea of how good these predictions are. ~98% of the\nobservations have a target between -1 and 1. Therefore the best\nscore has an error 150% larger than the range of 98% of the data\npoints. This is terrible ( at no fault of the kagglers).
    4. \n
  4. Unless the data quality is fixed, it's unlikely leaderboard scores\nwill improve much from here. You can only aggregate and stack and\nblend so much with data that has very little correlation to the\nmetric being forecasted. It's Data science not magic.
    5. \n
  5. I noted the welcome thread had many Kaggle First Timers. The myriad\nissues with Data in this competition will be very discouraging and\nconfusing. Therefore, in my opinion, this is NOT a good competition\nfor first timers. They won't learn much about competitions, only how\nto deal with poor data, and may in fact draw incorrect or misleading\nconclusions based on this experience alone.
    6. \n
", "messages": [ { "id": "431877", "postDate": "12/03/2018 03:45:54", "content": "
    \n
  1. As noted in several different kernels, for example here:\nhttps://www.kaggle.com/fabiendaniel/elo-world One of the most\nimportant features to most models is the amount of elapsed time. If\nI was organizing the competition I would find this a little alarming\nbecause despite the creation of 1000+ features from the original\ndata, a simple sequential count is more useful than the rest of the\ndata provided to competitors.
    2. \n
  2. As noted in this thread, more than HALF of the RMSE is due to about\n2000 outliers with loyalty scores of ~ -33. If you try to replace\nthese outliers with the median, retrain the model and submit, you'll\nfind your leaderboard score is WORSE than if you keep the outliers\nat their original values. This tells you that outliers are included\nin the test set. Furthermore, given the magnitude of the impact of\noutliers on the RMSE score, the entire focus of this conmpetition\n(should be) predicting those outliers as accurately as possible and\nlargely ignoring the other 99% of data points. This should also\nconcern the competition organizers because unless they care strictly\nabout those outliers, it's highly unlikely the winning model will\nprovide anything useful.
    3. \n
  3. As further evidence of the above concern, the RMSE on the leader\nboard is ~3.69. Since RMSE is in the original units of the target we\ncan get an idea of how good these predictions are. ~98% of the\nobservations have a target between -1 and 1. Therefore the best\nscore has an error 150% larger than the range of 98% of the data\npoints. This is terrible ( at no fault of the kagglers).
    4. \n
  4. Unless the data quality is fixed, it's unlikely leaderboard scores\nwill improve much from here. You can only aggregate and stack and\nblend so much with data that has very little correlation to the\nmetric being forecasted. It's Data science not magic.
    5. \n
  5. I noted the welcome thread had many Kaggle First Timers. The myriad\nissues with Data in this competition will be very discouraging and\nconfusing. Therefore, in my opinion, this is NOT a good competition\nfor first timers. They won't learn much about competitions, only how\nto deal with poor data, and may in fact draw incorrect or misleading\nconclusions based on this experience alone.
    6. \n
", "rawMarkdown": "1. As noted in several different kernels, for example here:\n https://www.kaggle.com/fabiendaniel/elo-world One of the most\n important features to most models is the amount of elapsed time. If\n I was organizing the competition I would find this a little alarming\n because despite the creation of 1000+ features from the original\n data, a simple sequential count is more useful than the rest of the\n data provided to competitors.\n 2. As noted in this thread, more than HALF of the RMSE is due to about\n 2000 outliers with loyalty scores of ~ -33. If you try to replace\n these outliers with the median, retrain the model and submit, you'll\n find your leaderboard score is WORSE than if you keep the outliers\n at their original values. This tells you that outliers are included\n in the test set. Furthermore, given the magnitude of the impact of\n outliers on the RMSE score, the entire focus of this conmpetition\n (should be) predicting those outliers as accurately as possible and\n largely ignoring the other 99% of data points. This should also\n concern the competition organizers because unless they care strictly\n about those outliers, it's highly unlikely the winning model will\n provide anything useful.\n 3. As further evidence of the above concern, the RMSE on the leader\n board is ~3.69. Since RMSE is in the original units of the target we\n can get an idea of how good these predictions are. ~98% of the\n observations have a target between -1 and 1. Therefore the best\n score has an error 150% larger than the range of 98% of the data\n points. This is terrible ( at no fault of the kagglers).\n 4. Unless the data quality is fixed, it's unlikely leaderboard scores\n will improve much from here. You can only aggregate and stack and\n blend so much with data that has very little correlation to the\n metric being forecasted. It's Data science not magic.\n 5. I noted the welcome thread had many Kaggle First Timers. The myriad\n issues with Data in this competition will be very discouraging and\n confusing. Therefore, in my opinion, this is NOT a good competition\n for first timers. They won't learn much about competitions, only how\n to deal with poor data, and may in fact draw incorrect or misleading\n conclusions based on this experience alone.", "votes": null }, { "id": "431912", "postDate": "12/03/2018 04:48:51", "content": "

I largely agree on your analysis.

", "rawMarkdown": "I largely agree on your analysis.", "votes": null }, { "id": "432062", "postDate": "12/03/2018 10:47:33", "content": "

Did you try multiplicating the \"outliers\" with a small number to reduce outlier strength?

", "rawMarkdown": "Did you try multiplicating the \"outliers\" with a small number to reduce outlier strength?", "votes": null }, { "id": "432146", "postDate": "12/03/2018 13:13:22", "content": "

I have one other concern, which makes sense with your analysis. The RMSE of our predictions being high correlates to a low R2 score. That makes snese. Once I removed a few features to increase the R2 score, it increased the RMSE, which should not happen. There are too many outliers in the testing dataset, and that will make it difficult to make predictions.

", "rawMarkdown": "I have one other concern, which makes sense with your analysis. The RMSE of our predictions being high correlates to a low R2 score. That makes snese. Once I removed a few features to increase the R2 score, it increased the RMSE, which should not happen. There are too many outliers in the testing dataset, and that will make it difficult to make predictions.", "votes": null }, { "id": "432210", "postDate": "12/03/2018 15:00:37", "content": "

I tried one quick and simple technique that didn't improve results.\nI have a couple of other ideas to try, but again, with very little predictive power contained in any of the variables, you can only go so far with nearly random data.

", "rawMarkdown": "I tried one quick and simple technique that didn't improve results.\nI have a couple of other ideas to try, but again, with very little predictive power contained in any of the variables, you can only go so far with nearly random data.", "votes": null }, { "id": "432681", "postDate": "12/04/2018 07:17:51", "content": "

As I have tried to find the correlation between any two featrues, nothing,yes nothing can be found. I'm done

", "rawMarkdown": "As I have tried to find the correlation between any two featrues, nothing,yes nothing can be found. I'm done", "votes": null }, { "id": "433103", "postDate": "12/04/2018 17:50:03", "content": "

Thank you whamp, great points. I'm giving up this competition after reading some discussions here. Even the title of this competition is wrong, it is not a Catagory Recommendation, but a Loyalty Score prediction.

", "rawMarkdown": "Thank you whamp, great points. I'm giving up this competition after reading some discussions here. Even the title of this competition is wrong, it is not a Catagory Recommendation, but a Loyalty Score prediction.", "votes": null }, { "id": "433122", "postDate": "12/04/2018 18:12:48", "content": "

@Firat, Does multiplying the outliers with a small number work? Assuming there is sufficient outliers in the test set.

", "rawMarkdown": "Firat, Does multiplying the outliers with a small number work? Assuming there is sufficient outliers in the test set.", "votes": null }, { "id": "433165", "postDate": "12/04/2018 19:28:31", "content": "

This is a very useful analysis. I've tried several ways of dealing wit the target values < -30, and none of it seems to help.

", "rawMarkdown": "This is a very useful analysis. I've tried several ways of dealing wit the target values < -30, and none of it seems to help.", "votes": null }, { "id": "437381", "postDate": "12/11/2018 19:54:36", "content": "

Hi whamp, thank you for the post. Small note: You're writing:\n~98% of the observations have a target between -1 and 1.

\n\n

but it's actually 57%. Or do I miss something?

\n\n
\n
\n
\n

len(train[train.target.abs() < 1]) / len(train)\n 0.5703234497342967

\n
\n
\n
", "rawMarkdown": "Hi whamp, thank you for the post. Small note: You're writing:\n~98% of the observations have a target between -1 and 1.\n\nbut it's actually 57%. Or do I miss something?\n\n>>> len(train[train.target.abs() < 1]) / len(train)\n0.5703234497342967", "votes": null }, { "id": "437417", "postDate": "12/11/2018 21:39:27", "content": "

The outliers seem to be purposely truncated in the loyalty formula. This creates huge noise in the data.\nThe host could've provided us with more detailed information of the loyalty score formula~~\nBut they choose not to~~ :(

", "rawMarkdown": "The outliers seem to be purposely truncated in the loyalty formula. This creates huge noise in the data.\nThe host could've provided us with more detailed information of the loyalty score formula~~\nBut they choose not to~~ :(", "votes": null }, { "id": "437442", "postDate": "12/11/2018 23:59:10", "content": "

thanks for the comment, I am a first timer and I was indeed getting confused. I guess I'll go look somewhere else for now :)\nLet me know if you guys have any recommendations on competitions to take on for first timers!

", "rawMarkdown": "thanks for the comment, I am a first timer and I was indeed getting confused. I guess I'll go look somewhere else for now :)\nLet me know if you guys have any recommendations on competitions to take on for first timers!", "votes": null }, { "id": "437444", "postDate": "12/12/2018 00:03:15", "content": "

I think without outliers we can get ~ 1.7XX RMSE or so.
\nThis shows test data includes outliers as in train.

\n\n

OK. My proposals are,

\n\n
    \n
  1. Exclude outliers from this dataset
    2. \n
  2. Change RMSE to MAE or RMSLE
    3. \n
\n\n

Option 1 would be best, but Option 2 would be less effort.
\nHope this competition will change and go well.

", "rawMarkdown": "I think without outliers we can get `~ 1.7XX` RMSE or so. \nThis shows test data includes outliers as in train. \n \nOK. My proposals are, \n \n1. Exclude outliers from this dataset \n2. Change RMSE to MAE or RMSLE\n\nOption 1 would be best, but Option 2 would be less effort. \nHope this competition will change and go well.", "votes": null }, { "id": "437593", "postDate": "12/12/2018 06:51:28", "content": "

Outliers could be one of the main purposes of this competition.\nImagine if those represent fraud or credit default etc. they are important. \nBut the thing is, without telling us how loyalty is computed beforehand, we are wasting \nefforts in cracking the code instead of modeling the prediction.

", "rawMarkdown": "Outliers could be one of the main purposes of this competition.\nImagine if those represent fraud or credit default etc. they are important. \nBut the thing is, without telling us how loyalty is computed beforehand, we are wasting \nefforts in cracking the code instead of modeling the prediction.", "votes": null }, { "id": "437874", "postDate": "12/12/2018 17:13:18", "content": "

Thanks for this comment. My initial approach was to simply remove the outliers. It would be useful to know more information about the target (i.e. how is loyalty calculated) so that we can make more informed decisions about what to do with these values (and what they represent)!

", "rawMarkdown": "Thanks for this comment. My initial approach was to simply remove the outliers. It would be useful to know more information about the target (i.e. how is loyalty calculated) so that we can make more informed decisions about what to do with these values (and what they represent)!", "votes": null }, { "id": "437982", "postDate": "12/12/2018 22:15:41", "content": "

The 3 starter competitions are always good places to... well.. start.. \n* https://www.kaggle.com/c/titanic\n* https://www.kaggle.com/c/digit-recognizer\n* https://www.kaggle.com/c/house-prices-advanced-regression-techniques

\n\n

The advantage of those is that they'll be sure to have a whole bunch of EDAs and basic competition starter entries, and you can build on those quite easily.

", "rawMarkdown": "The 3 starter competitions are always good places to... well.. start.. \n* https://www.kaggle.com/c/titanic\n* https://www.kaggle.com/c/digit-recognizer\n* https://www.kaggle.com/c/house-prices-advanced-regression-techniques\n\nThe advantage of those is that they'll be sure to have a whole bunch of EDAs and basic competition starter entries, and you can build on those quite easily.", "votes": null }, { "id": "438072", "postDate": "12/13/2018 03:51:40", "content": "

Could this partially explain why my validation RMSE is hardly correlated with my test RMSE? If test performance is hugely weighted on predicting outliers, then that makes it even more difficult to have a validation set that is representative of the test set.

", "rawMarkdown": "Could this partially explain why my validation RMSE is hardly correlated with my test RMSE? If test performance is hugely weighted on predicting outliers, then that makes it even more difficult to have a validation set that is representative of the test set.", "votes": null }, { "id": "439414", "postDate": "12/15/2018 12:04:48", "content": "

Makes more sense now, Thanks!

", "rawMarkdown": "Makes more sense now, Thanks!", "votes": null }, { "id": "440339", "postDate": "12/17/2018 12:21:34", "content": "

Great topic! I had omitted the outliers first... but we maybe need to consider the outliers for making predictions.

", "rawMarkdown": "Great topic! I had omitted the outliers first... but we maybe need to consider the outliers for making predictions.", "votes": null }, { "id": "446072", "postDate": "12/27/2018 13:05:05", "content": "

Good points! \nThese outliers are really confusing ...

", "rawMarkdown": "Good points! \nThese outliers are really confusing ...", "votes": null }, { "id": "446155", "postDate": "12/27/2018 15:52:58", "content": "

Why don't you try changing this into a classification problem? Just cut the targets into x pieces and classify - you can the just optimize on x1*w1+x2*w2 ... etc.

\n\n

I have to say I am pretty concerned with this data as it does feel a bit hokey!

", "rawMarkdown": "Why don't you try changing this into a classification problem? Just cut the targets into x pieces and classify - you can the just optimize on x1*w1+x2*w2 ... etc.\n\nI have to say I am pretty concerned with this data as it does feel a bit hokey!", "votes": null }, { "id": "448635", "postDate": "01/01/2019 18:13:11", "content": "

A script for finding Outliers - may overfit!

", "rawMarkdown": "A script for finding Outliers - may overfit!", "votes": null }, { "id": "456380", "postDate": "01/15/2019 17:48:26", "content": "

Thank you for your post. As a first timer, I will re-evaluate other competitions to enter instead.

", "rawMarkdown": "Thank you for your post. As a first timer, I will re-evaluate other competitions to enter instead.", "votes": null }, { "id": "456648", "postDate": "01/16/2019 08:37:15", "content": "

Good point, thanks for sharing. It looks like add some sequential count does not help a lot for me in this competition. But this point is quite meaningful.

", "rawMarkdown": "Good point, thanks for sharing. It looks like add some sequential count does not help a lot for me in this competition. But this point is quite meaningful.", "votes": null } ], "comments": [ { "id": 431912, "author_name": "thequants", "author_url": "", "post_date": "12/03/2018 04:48:51", "content": "

I largely agree on your analysis.

", "votes": null, "replies": [] }, { "id": 432062, "author_name": "frtgnn", "author_url": "", "post_date": "12/03/2018 10:47:33", "content": "

Did you try multiplicating the \"outliers\" with a small number to reduce outlier strength?

", "votes": null, "replies": [ { "id": 432210, "author_name": "hampsonw", "author_url": "", "post_date": "12/03/2018 15:00:37", "content": "

I tried one quick and simple technique that didn't improve results.\nI have a couple of other ideas to try, but again, with very little predictive power contained in any of the variables, you can only go so far with nearly random data.

", "votes": null, "replies": [] }, { "id": 433122, "author_name": "harshit92", "author_url": "", "post_date": "12/04/2018 18:12:48", "content": "

@Firat, Does multiplying the outliers with a small number work? Assuming there is sufficient outliers in the test set.

", "votes": null, "replies": [] } ] }, { "id": 432146, "author_name": "eaturner", "author_url": "", "post_date": "12/03/2018 13:13:22", "content": "

I have one other concern, which makes sense with your analysis. The RMSE of our predictions being high correlates to a low R2 score. That makes snese. Once I removed a few features to increase the R2 score, it increased the RMSE, which should not happen. There are too many outliers in the testing dataset, and that will make it difficult to make predictions.

", "votes": null, "replies": [] }, { "id": 432681, "author_name": "zhch2016", "author_url": "", "post_date": "12/04/2018 07:17:51", "content": "

As I have tried to find the correlation between any two featrues, nothing,yes nothing can be found. I'm done

", "votes": null, "replies": [] }, { "id": 433103, "author_name": "braquino", "author_url": "", "post_date": "12/04/2018 17:50:03", "content": "

Thank you whamp, great points. I'm giving up this competition after reading some discussions here. Even the title of this competition is wrong, it is not a Catagory Recommendation, but a Loyalty Score prediction.

", "votes": null, "replies": [] }, { "id": 433165, "author_name": "delayedkarma", "author_url": "", "post_date": "12/04/2018 19:28:31", "content": "

This is a very useful analysis. I've tried several ways of dealing wit the target values < -30, and none of it seems to help.

", "votes": null, "replies": [] }, { "id": 437381, "author_name": "dustinthewind", "author_url": "", "post_date": "12/11/2018 19:54:36", "content": "

Hi whamp, thank you for the post. Small note: You're writing:\n~98% of the observations have a target between -1 and 1.

\n\n

but it's actually 57%. Or do I miss something?

\n\n
\n
\n
\n

len(train[train.target.abs() < 1]) / len(train)\n 0.5703234497342967

\n
\n
\n
", "votes": null, "replies": [] }, { "id": 437417, "author_name": "tclf90", "author_url": "", "post_date": "12/11/2018 21:39:27", "content": "

The outliers seem to be purposely truncated in the loyalty formula. This creates huge noise in the data.\nThe host could've provided us with more detailed information of the loyalty score formula~~\nBut they choose not to~~ :(

", "votes": null, "replies": [] }, { "id": 437442, "author_name": "timotheer", "author_url": "", "post_date": "12/11/2018 23:59:10", "content": "

thanks for the comment, I am a first timer and I was indeed getting confused. I guess I'll go look somewhere else for now :)\nLet me know if you guys have any recommendations on competitions to take on for first timers!

", "votes": null, "replies": [ { "id": 437982, "author_name": "delayedkarma", "author_url": "", "post_date": "12/12/2018 22:15:41", "content": "

The 3 starter competitions are always good places to... well.. start.. \n* https://www.kaggle.com/c/titanic\n* https://www.kaggle.com/c/digit-recognizer\n* https://www.kaggle.com/c/house-prices-advanced-regression-techniques

\n\n

The advantage of those is that they'll be sure to have a whole bunch of EDAs and basic competition starter entries, and you can build on those quite easily.

", "votes": null, "replies": [] } ] }, { "id": 437444, "author_name": "maxwell110", "author_url": "", "post_date": "12/12/2018 00:03:15", "content": "

I think without outliers we can get ~ 1.7XX RMSE or so.
\nThis shows test data includes outliers as in train.

\n\n

OK. My proposals are,

\n\n
    \n
  1. Exclude outliers from this dataset
    2. \n
  2. Change RMSE to MAE or RMSLE
    3. \n
\n\n

Option 1 would be best, but Option 2 would be less effort.
\nHope this competition will change and go well.

", "votes": null, "replies": [ { "id": 437593, "author_name": "tclf90", "author_url": "", "post_date": "12/12/2018 06:51:28", "content": "

Outliers could be one of the main purposes of this competition.\nImagine if those represent fraud or credit default etc. they are important. \nBut the thing is, without telling us how loyalty is computed beforehand, we are wasting \nefforts in cracking the code instead of modeling the prediction.

", "votes": null, "replies": [] } ] }, { "id": 437874, "author_name": "sathesan", "author_url": "", "post_date": "12/12/2018 17:13:18", "content": "

Thanks for this comment. My initial approach was to simply remove the outliers. It would be useful to know more information about the target (i.e. how is loyalty calculated) so that we can make more informed decisions about what to do with these values (and what they represent)!

", "votes": null, "replies": [] }, { "id": 438072, "author_name": "enzoamp", "author_url": "", "post_date": "12/13/2018 03:51:40", "content": "

Could this partially explain why my validation RMSE is hardly correlated with my test RMSE? If test performance is hugely weighted on predicting outliers, then that makes it even more difficult to have a validation set that is representative of the test set.

", "votes": null, "replies": [] }, { "id": 439414, "author_name": "doronhar", "author_url": "", "post_date": "12/15/2018 12:04:48", "content": "

Makes more sense now, Thanks!

", "votes": null, "replies": [] }, { "id": 440339, "author_name": "sshimizu2", "author_url": "", "post_date": "12/17/2018 12:21:34", "content": "

Great topic! I had omitted the outliers first... but we maybe need to consider the outliers for making predictions.

", "votes": null, "replies": [] }, { "id": 446072, "author_name": "bezul2", "author_url": "", "post_date": "12/27/2018 13:05:05", "content": "

Good points! \nThese outliers are really confusing ...

", "votes": null, "replies": [] }, { "id": 446155, "author_name": "scirpus", "author_url": "", "post_date": "12/27/2018 15:52:58", "content": "

Why don't you try changing this into a classification problem? Just cut the targets into x pieces and classify - you can the just optimize on x1*w1+x2*w2 ... etc.

\n\n

I have to say I am pretty concerned with this data as it does feel a bit hokey!

", "votes": null, "replies": [] }, { "id": 448635, "author_name": "scirpus", "author_url": "", "post_date": "01/01/2019 18:13:11", "content": "

A script for finding Outliers - may overfit!

", "votes": null, "replies": [] }, { "id": 456380, "author_name": "jwong4494", "author_url": "", "post_date": "01/15/2019 17:48:26", "content": "

Thank you for your post. As a first timer, I will re-evaluate other competitions to enter instead.

", "votes": null, "replies": [] }, { "id": 456648, "author_name": "longyin2", "author_url": "", "post_date": "01/16/2019 08:37:15", "content": "

Good point, thanks for sharing. It looks like add some sequential count does not help a lot for me in this competition. But this point is quite meaningful.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "431877": "1. As noted in several different kernels, for example here:\n https://www.kaggle.com/fabiendaniel/elo-world One of the most\n important features to most models is the amount of elapsed time. If\n I was organizing the competition I would find this a little alarming\n because despite the creation of 1000+ features from the original\n data, a simple sequential count is more useful than the rest of the\n data provided to competitors.\n 2. As noted in this thread, more than HALF of the RMSE is due to about\n 2000 outliers with loyalty scores of ~ -33. If you try to replace\n these outliers with the median, retrain the model and submit, you'll\n find your leaderboard score is WORSE than if you keep the outliers\n at their original values. This tells you that outliers are included\n in the test set. Furthermore, given the magnitude of the impact of\n outliers on the RMSE score, the entire focus of this conmpetition\n (should be) predicting those outliers as accurately as possible and\n largely ignoring the other 99% of data points. This should also\n concern the competition organizers because unless they care strictly\n about those outliers, it's highly unlikely the winning model will\n provide anything useful.\n 3. As further evidence of the above concern, the RMSE on the leader\n board is ~3.69. Since RMSE is in the original units of the target we\n can get an idea of how good these predictions are. ~98% of the\n observations have a target between -1 and 1. Therefore the best\n score has an error 150% larger than the range of 98% of the data\n points. This is terrible ( at no fault of the kagglers).\n 4. Unless the data quality is fixed, it's unlikely leaderboard scores\n will improve much from here. You can only aggregate and stack and\n blend so much with data that has very little correlation to the\n metric being forecasted. It's Data science not magic.\n 5. I noted the welcome thread had many Kaggle First Timers. The myriad\n issues with Data in this competition will be very discouraging and\n confusing. Therefore, in my opinion, this is NOT a good competition\n for first timers. They won't learn much about competitions, only how\n to deal with poor data, and may in fact draw incorrect or misleading\n conclusions based on this experience alone.", "431912": "I largely agree on your analysis.", "432062": "Did you try multiplicating the \"outliers\" with a small number to reduce outlier strength?", "432146": "I have one other concern, which makes sense with your analysis. The RMSE of our predictions being high correlates to a low R2 score. That makes snese. Once I removed a few features to increase the R2 score, it increased the RMSE, which should not happen. There are too many outliers in the testing dataset, and that will make it difficult to make predictions.", "432210": "I tried one quick and simple technique that didn't improve results.\nI have a couple of other ideas to try, but again, with very little predictive power contained in any of the variables, you can only go so far with nearly random data.", "432681": "As I have tried to find the correlation between any two featrues, nothing,yes nothing can be found. I'm done", "433103": "Thank you whamp, great points. I'm giving up this competition after reading some discussions here. Even the title of this competition is wrong, it is not a Catagory Recommendation, but a Loyalty Score prediction.", "433122": "Firat, Does multiplying the outliers with a small number work? Assuming there is sufficient outliers in the test set.", "433165": "This is a very useful analysis. I've tried several ways of dealing wit the target values < -30, and none of it seems to help.", "437381": "Hi whamp, thank you for the post. Small note: You're writing:\n~98% of the observations have a target between -1 and 1.\n\nbut it's actually 57%. Or do I miss something?\n\n>>> len(train[train.target.abs() < 1]) / len(train)\n0.5703234497342967", "437417": "The outliers seem to be purposely truncated in the loyalty formula. This creates huge noise in the data.\nThe host could've provided us with more detailed information of the loyalty score formula~~\nBut they choose not to~~ :(", "437442": "thanks for the comment, I am a first timer and I was indeed getting confused. I guess I'll go look somewhere else for now :)\nLet me know if you guys have any recommendations on competitions to take on for first timers!", "437444": "I think without outliers we can get `~ 1.7XX` RMSE or so. \nThis shows test data includes outliers as in train. \n \nOK. My proposals are, \n \n1. Exclude outliers from this dataset \n2. Change RMSE to MAE or RMSLE\n\nOption 1 would be best, but Option 2 would be less effort. \nHope this competition will change and go well.", "437593": "Outliers could be one of the main purposes of this competition.\nImagine if those represent fraud or credit default etc. they are important. \nBut the thing is, without telling us how loyalty is computed beforehand, we are wasting \nefforts in cracking the code instead of modeling the prediction.", "437874": "Thanks for this comment. My initial approach was to simply remove the outliers. It would be useful to know more information about the target (i.e. how is loyalty calculated) so that we can make more informed decisions about what to do with these values (and what they represent)!", "437982": "The 3 starter competitions are always good places to... well.. start.. \n* https://www.kaggle.com/c/titanic\n* https://www.kaggle.com/c/digit-recognizer\n* https://www.kaggle.com/c/house-prices-advanced-regression-techniques\n\nThe advantage of those is that they'll be sure to have a whole bunch of EDAs and basic competition starter entries, and you can build on those quite easily.", "438072": "Could this partially explain why my validation RMSE is hardly correlated with my test RMSE? If test performance is hugely weighted on predicting outliers, then that makes it even more difficult to have a validation set that is representative of the test set.", "439414": "Makes more sense now, Thanks!", "440339": "Great topic! I had omitted the outliers first... but we maybe need to consider the outliers for making predictions.", "446072": "Good points! \nThese outliers are really confusing ...", "446155": "Why don't you try changing this into a classification problem? Just cut the targets into x pieces and classify - you can the just optimize on x1*w1+x2*w2 ... etc.\n\nI have to say I am pretty concerned with this data as it does feel a bit hokey!", "448635": "A script for finding Outliers - may overfit!", "456380": "Thank you for your post. As a first timer, I will re-evaluate other competitions to enter instead.", "456648": "Good point, thanks for sharing. It looks like add some sequential count does not help a lot for me in this competition. But this point is quite meaningful." }, "source": "meta" }