it's obvious that there are 103 negative examples and 1911 positive examples in the trainning data,there are so many technics to classify the imbalanaced data at a low error-rate and in a simple but effective way,the sampling technique will be a good choice,so the question:will you under-sampling the majority class or over-sampling the minority class?
", "messages": [ { "id": "77266", "postDate": "05/06/2015 07:29:06", "content": "it's obvious that there are 103 negative examples and 1911 positive examples in the trainning data,there are so many technics to classify the imbalanaced data at a low error-rate and in a simple but effective way,the sampling technique will be a good choice,so the question:will you under-sampling the majority class or over-sampling the minority class?
", "rawMarkdown": "", "votes": null }, { "id": "77285", "postDate": "05/06/2015 09:16:42", "content": "I guess with question like that you do not expect to get the answer by the middle of June.
", "rawMarkdown": "", "votes": null }, { "id": "77288", "postDate": "05/06/2015 09:29:00", "content": "a general note that may cut down on the time to find something useful is that many techniques for dealing with imbalanced data only care about increasing performance on the minority class. To do this they end up doing worse on the majority class. In real world applications this can be very useful, but on Kaggle you care about performance on the majority because it will change your score more.
", "rawMarkdown": "", "votes": null }, { "id": "77490", "postDate": "05/07/2015 16:29:38", "content": "Undersampling the majority class would be a bad idea because there seem to be few examples in the first place. So if you want to fix class balance I'd oversample. Don't take my word for it though. Try both techniques, cross-validate (don't forget to shuffle), and check which performs best?
\nThere are other ways to take class balance into account. Some sklearn models allow you to set class_weight = 'auto'.
\nEither way, I wouldn't spend too much time on this. Considering the ROC AUC metric, I don't think it's going to matter much. Be sure to read up on how it works. The metric only cares about the correct order of predicted output probabilities. If you're evaluating 3 model outputs: 0.1 0.9 0.5, you will get the same score as 0.01 0.09 0.05, and even the same as 100 10000 1000 (try it!).
\nROC AUC cares the same about both classes, which is why it's a favorite for basically any binary classification problem.
", "rawMarkdown": "", "votes": null }, { "id": "77495", "postDate": "05/07/2015 17:02:22", "content": "[quote=Jonaz;77490]
\nROC AUC cares the same about both classes, which is why it's a favorite for basically any binary classification problem.
\n[/quote]
\nI realized after I posted in the case of ROC AUC we do care about both. Because improving one of the minority classes will have it ranked above many more of the majority. I am not quite sure how a degradation to majority performance will interact with an improvement to the minority class with AUC. In most cases(most metrics) my previous comment stands that you will worsen your score by trading off performance between classes as described. Here it may make no or only a little difference. As @Jonaz said try things out.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 77285, "author_name": "mpekalski", "author_url": "", "post_date": "05/06/2015 09:16:42", "content": "I guess with question like that you do not expect to get the answer by the middle of June.
", "votes": null, "replies": [] }, { "id": 77288, "author_name": "devinanzelmo", "author_url": "", "post_date": "05/06/2015 09:29:00", "content": "a general note that may cut down on the time to find something useful is that many techniques for dealing with imbalanced data only care about increasing performance on the minority class. To do this they end up doing worse on the majority class. In real world applications this can be very useful, but on Kaggle you care about performance on the majority because it will change your score more.
", "votes": null, "replies": [] }, { "id": 77490, "author_name": "", "author_url": "", "post_date": "05/07/2015 16:29:38", "content": "Undersampling the majority class would be a bad idea because there seem to be few examples in the first place. So if you want to fix class balance I'd oversample. Don't take my word for it though. Try both techniques, cross-validate (don't forget to shuffle), and check which performs best?
\nThere are other ways to take class balance into account. Some sklearn models allow you to set class_weight = 'auto'.
\nEither way, I wouldn't spend too much time on this. Considering the ROC AUC metric, I don't think it's going to matter much. Be sure to read up on how it works. The metric only cares about the correct order of predicted output probabilities. If you're evaluating 3 model outputs: 0.1 0.9 0.5, you will get the same score as 0.01 0.09 0.05, and even the same as 100 10000 1000 (try it!).
\nROC AUC cares the same about both classes, which is why it's a favorite for basically any binary classification problem.
", "votes": null, "replies": [] }, { "id": 77495, "author_name": "devinanzelmo", "author_url": "", "post_date": "05/07/2015 17:02:22", "content": "[quote=Jonaz;77490]
\nROC AUC cares the same about both classes, which is why it's a favorite for basically any binary classification problem.
\n[/quote]
\nI realized after I posted in the case of ROC AUC we do care about both. Because improving one of the minority classes will have it ranked above many more of the majority. I am not quite sure how a degradation to majority performance will interact with an improvement to the minority class with AUC. In most cases(most metrics) my previous comment stands that you will worsen your score by trading off performance between classes as described. Here it may make no or only a little difference. As @Jonaz said try things out.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "77266": "", "77285": "", "77288": "", "77490": "", "77495": "" }, "source": "meta" }