My numeric CVs match up with the Public LB. My categorical CVs don't match up at all. Can the categorical features be used solo? Do they have to be combined with dates?
", "messages": [ { "id": "136051", "postDate": "09/19/2016 18:43:57", "content": "My numeric CVs match up with the Public LB. My categorical CVs don't match up at all. Can the categorical features be used solo? Do they have to be combined with dates?
", "rawMarkdown": "My numeric CVs match up with the Public LB. My categorical CVs don't match up at all. Can the categorical features be used solo? Do they have to be combined with dates?", "votes": null }, { "id": "136078", "postDate": "09/19/2016 23:05:42", "content": "You can do whatever you want with the categorical features. Currently I'm not using them at all. It is fairly clear to me that most of the information is in train_numeric.csv.
\n\nIt is not that unusual for Kaggle contests to contain some mostly useless (or even totally useless) data. For instance, in Walmart, I only used the main file of store-sales-by-date, called train.csv. I did not use the features.csv or stores.csv files at all.
", "rawMarkdown": "You can do whatever you want with the categorical features. Currently I'm not using them at all. It is fairly clear to me that most of the information is in train_numeric.csv. \r\n\r\nIt is not that unusual for Kaggle contests to contain some mostly useless (or even totally useless) data. For instance, in [Walmart][1], I only used the main file of store-sales-by-date, called train.csv. I did not use the features.csv or stores.csv files at all.\r\n\r\n\r\n [1]: https://www.kaggle.com/c/walmart-recruiting-store-sales-forecasting", "votes": null }, { "id": "136132", "postDate": "09/20/2016 05:12:05", "content": "@ David, may I ask whether you used train_date file, or you achieved .28+ score using only numerical data?
", "rawMarkdown": "David, may I ask whether you used train_date file, or you achieved .28+ score using only numerical data?", "votes": null }, { "id": "136199", "postDate": "09/20/2016 14:18:15", "content": "@David, Thanks for your reply. I understand numeric is more useful than categorical. my numeric lazy CV scores are about .26 and my public score is .22. When I train on categorical or numeric and categorical combined I get a positive CV score everytime on validation (MCC 0.05 AUC 0.68) but my LB score is always 0 or negative. The reason I'm frustrated is my CV scores make it look like the categorical is not completely useless but I've hit a brick wall. (Probably some bug of my own fault I should stop whining about :) )
", "rawMarkdown": "David, Thanks for your reply. I understand numeric is more useful than categorical. my numeric lazy CV scores are about .26 and my public score is .22. When I train on categorical or numeric and categorical combined I get a positive CV score everytime on validation (MCC 0.05 AUC 0.68) but my LB score is always 0 or negative. The reason I'm frustrated is my CV scores make it look like the categorical is not completely useless but I've hit a brick wall. (Probably some bug of my own fault I should stop whining about :) )", "votes": null }, { "id": "136205", "postDate": "09/20/2016 14:38:06", "content": "@Bryce, something is wrong with either your cross-validation or MCC optimization. My best 5-fold CV for numerical data is 0.23 and public score is always 0.01 or 0.02 higher. Your results with categorical data are also strange: using only categorical data I achieved easily 0.12 on CV (haven't tried LB yet). My guess is that you messed up somehow the row order of target/numerical/categorical during concatenation. I will try to upload my prediction on categorical data in 20-30 hours and I'll post the result.
", "rawMarkdown": "Bryce, something is wrong with either your cross-validation or MCC optimization. My best 5-fold CV for numerical data is 0.23 and public score is always 0.01 or 0.02 higher. Your results with categorical data are also strange: using only categorical data I achieved easily 0.12 on CV (haven't tried LB yet). My guess is that you messed up somehow the row order of target/numerical/categorical during concatenation. I will try to upload my prediction on categorical data in 20-30 hours and I'll post the result.", "votes": null }, { "id": "136206", "postDate": "09/20/2016 15:22:29", "content": "@Denis Vlasov can you tell me if you are using OHE for the categorical features ? or Just factorizing it ? I am running into memory issues if I use OHE .
", "rawMarkdown": "Denis Vlasov can you tell me if you are using OHE for the categorical features ? or Just factorizing it ? I am running into memory issues if I use OHE .", "votes": null }, { "id": "136212", "postDate": "09/20/2016 16:00:55", "content": "@denis Thanks so much. Just wanted to know I wasn't crazy. Currently only my xgb stopping is based on out of sample. feature selection and MCC are a bit leaky so thats why my LB is a bit lower than my CV. I'll get it fixed by the time I move to boosting.\n@zohair OHE might score better but this is way easier: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23290/turn-categorical-into-numeric/133692\nTree methods don't need OHE as much as neural methods.
", "rawMarkdown": "denis Thanks so much. Just wanted to know I wasn't crazy. Currently only my xgb stopping is based on out of sample. feature selection and MCC are a bit leaky so thats why my LB is a bit lower than my CV. I'll get it fixed by the time I move to boosting.\r\n@zohair OHE might score better but this is way easier: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23290/turn-categorical-into-numeric/133692\r\nTree methods don't need OHE as much as neural methods.", "votes": null }, { "id": "136231", "postDate": "09/20/2016 18:48:47", "content": "@ZohairZahid, I had the same memory issues. For now I decided to go with factorizing. I will try PCA on feature batches or feature hashing later.
\n\n@Bryce I recommend you to test different thresholds for MCC on hold-out. It's tricky.
", "rawMarkdown": "ZohairZahid, I had the same memory issues. For now I decided to go with factorizing. I will try PCA on feature batches or feature hashing later.\r\n\r\n@Bryce I recommend you to test different thresholds for MCC on hold-out. It's tricky.", "votes": null }, { "id": "136352", "postDate": "09/21/2016 14:47:14", "content": "Hi,\nAre you guys using Xgboost?
", "rawMarkdown": "Hi,\r\nAre you guys using Xgboost?", "votes": null }, { "id": "136414", "postDate": "09/21/2016 20:34:25", "content": "@zohair I'll stick with xgboost till I get into much bigger ensembles.
", "rawMarkdown": "zohair I'll stick with xgboost till I get into much bigger ensembles.", "votes": null }, { "id": "136505", "postDate": "09/22/2016 14:34:00", "content": "got it finally. when I switched form numeric to categorical I accidentally treated NAs differently in train than in test.
", "rawMarkdown": "got it finally. when I switched form numeric to categorical I accidentally treated NAs differently in train than in test.", "votes": null }, { "id": "136616", "postDate": "09/23/2016 12:43:23", "content": "@Bryce Did it improve the overall score ?
", "rawMarkdown": "Bryce Did it improve the overall score ?", "votes": null }, { "id": "136623", "postDate": "09/23/2016 14:01:20", "content": "Yes, categorical and date data improve the score. Its kind of frustrating though because xgboost focuses on the stronger numeric features so throwing them all together only helps a little bit.
", "rawMarkdown": "Yes, categorical and date data improve the score. Its kind of frustrating though because xgboost focuses on the stronger numeric features so throwing them all together only helps a little bit.", "votes": null } ], "comments": [ { "id": 136078, "author_name": "davidthaler", "author_url": "", "post_date": "09/19/2016 23:05:42", "content": "You can do whatever you want with the categorical features. Currently I'm not using them at all. It is fairly clear to me that most of the information is in train_numeric.csv.
\n\nIt is not that unusual for Kaggle contests to contain some mostly useless (or even totally useless) data. For instance, in Walmart, I only used the main file of store-sales-by-date, called train.csv. I did not use the features.csv or stores.csv files at all.
", "votes": null, "replies": [] }, { "id": 136132, "author_name": "curlyb", "author_url": "", "post_date": "09/20/2016 05:12:05", "content": "@ David, may I ask whether you used train_date file, or you achieved .28+ score using only numerical data?
", "votes": null, "replies": [] }, { "id": 136199, "author_name": "bschumacher", "author_url": "", "post_date": "09/20/2016 14:18:15", "content": "@David, Thanks for your reply. I understand numeric is more useful than categorical. my numeric lazy CV scores are about .26 and my public score is .22. When I train on categorical or numeric and categorical combined I get a positive CV score everytime on validation (MCC 0.05 AUC 0.68) but my LB score is always 0 or negative. The reason I'm frustrated is my CV scores make it look like the categorical is not completely useless but I've hit a brick wall. (Probably some bug of my own fault I should stop whining about :) )
", "votes": null, "replies": [] }, { "id": 136205, "author_name": "curlyb", "author_url": "", "post_date": "09/20/2016 14:38:06", "content": "@Bryce, something is wrong with either your cross-validation or MCC optimization. My best 5-fold CV for numerical data is 0.23 and public score is always 0.01 or 0.02 higher. Your results with categorical data are also strange: using only categorical data I achieved easily 0.12 on CV (haven't tried LB yet). My guess is that you messed up somehow the row order of target/numerical/categorical during concatenation. I will try to upload my prediction on categorical data in 20-30 hours and I'll post the result.
", "votes": null, "replies": [] }, { "id": 136206, "author_name": "zohairzohair4", "author_url": "", "post_date": "09/20/2016 15:22:29", "content": "@Denis Vlasov can you tell me if you are using OHE for the categorical features ? or Just factorizing it ? I am running into memory issues if I use OHE .
", "votes": null, "replies": [] }, { "id": 136212, "author_name": "bschumacher", "author_url": "", "post_date": "09/20/2016 16:00:55", "content": "@denis Thanks so much. Just wanted to know I wasn't crazy. Currently only my xgb stopping is based on out of sample. feature selection and MCC are a bit leaky so thats why my LB is a bit lower than my CV. I'll get it fixed by the time I move to boosting.\n@zohair OHE might score better but this is way easier: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23290/turn-categorical-into-numeric/133692\nTree methods don't need OHE as much as neural methods.
", "votes": null, "replies": [] }, { "id": 136231, "author_name": "curlyb", "author_url": "", "post_date": "09/20/2016 18:48:47", "content": "@ZohairZahid, I had the same memory issues. For now I decided to go with factorizing. I will try PCA on feature batches or feature hashing later.
\n\n@Bryce I recommend you to test different thresholds for MCC on hold-out. It's tricky.
", "votes": null, "replies": [] }, { "id": 136352, "author_name": "zohairzohair4", "author_url": "", "post_date": "09/21/2016 14:47:14", "content": "Hi,\nAre you guys using Xgboost?
", "votes": null, "replies": [] }, { "id": 136414, "author_name": "bschumacher", "author_url": "", "post_date": "09/21/2016 20:34:25", "content": "@zohair I'll stick with xgboost till I get into much bigger ensembles.
", "votes": null, "replies": [] }, { "id": 136505, "author_name": "bschumacher", "author_url": "", "post_date": "09/22/2016 14:34:00", "content": "got it finally. when I switched form numeric to categorical I accidentally treated NAs differently in train than in test.
", "votes": null, "replies": [] }, { "id": 136616, "author_name": "zohairzohair4", "author_url": "", "post_date": "09/23/2016 12:43:23", "content": "@Bryce Did it improve the overall score ?
", "votes": null, "replies": [] }, { "id": 136623, "author_name": "bschumacher", "author_url": "", "post_date": "09/23/2016 14:01:20", "content": "Yes, categorical and date data improve the score. Its kind of frustrating though because xgboost focuses on the stronger numeric features so throwing them all together only helps a little bit.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "136051": "My numeric CVs match up with the Public LB. My categorical CVs don't match up at all. Can the categorical features be used solo? Do they have to be combined with dates?", "136078": "You can do whatever you want with the categorical features. Currently I'm not using them at all. It is fairly clear to me that most of the information is in train_numeric.csv. \r\n\r\nIt is not that unusual for Kaggle contests to contain some mostly useless (or even totally useless) data. For instance, in [Walmart][1], I only used the main file of store-sales-by-date, called train.csv. I did not use the features.csv or stores.csv files at all.\r\n\r\n\r\n [1]: https://www.kaggle.com/c/walmart-recruiting-store-sales-forecasting", "136132": "David, may I ask whether you used train_date file, or you achieved .28+ score using only numerical data?", "136199": "David, Thanks for your reply. I understand numeric is more useful than categorical. my numeric lazy CV scores are about .26 and my public score is .22. When I train on categorical or numeric and categorical combined I get a positive CV score everytime on validation (MCC 0.05 AUC 0.68) but my LB score is always 0 or negative. The reason I'm frustrated is my CV scores make it look like the categorical is not completely useless but I've hit a brick wall. (Probably some bug of my own fault I should stop whining about :) )", "136205": "Bryce, something is wrong with either your cross-validation or MCC optimization. My best 5-fold CV for numerical data is 0.23 and public score is always 0.01 or 0.02 higher. Your results with categorical data are also strange: using only categorical data I achieved easily 0.12 on CV (haven't tried LB yet). My guess is that you messed up somehow the row order of target/numerical/categorical during concatenation. I will try to upload my prediction on categorical data in 20-30 hours and I'll post the result.", "136206": "Denis Vlasov can you tell me if you are using OHE for the categorical features ? or Just factorizing it ? I am running into memory issues if I use OHE .", "136212": "denis Thanks so much. Just wanted to know I wasn't crazy. Currently only my xgb stopping is based on out of sample. feature selection and MCC are a bit leaky so thats why my LB is a bit lower than my CV. I'll get it fixed by the time I move to boosting.\r\n@zohair OHE might score better but this is way easier: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23290/turn-categorical-into-numeric/133692\r\nTree methods don't need OHE as much as neural methods.", "136231": "ZohairZahid, I had the same memory issues. For now I decided to go with factorizing. I will try PCA on feature batches or feature hashing later.\r\n\r\n@Bryce I recommend you to test different thresholds for MCC on hold-out. It's tricky.", "136352": "Hi,\r\nAre you guys using Xgboost?", "136414": "zohair I'll stick with xgboost till I get into much bigger ensembles.", "136505": "got it finally. when I switched form numeric to categorical I accidentally treated NAs differently in train than in test.", "136616": "Bryce Did it improve the overall score ?", "136623": "Yes, categorical and date data improve the score. Its kind of frustrating though because xgboost focuses on the stronger numeric features so throwing them all together only helps a little bit." }, "source": "meta" }