Hey fellow Kagglers, how are you?
\nFirst, Happy 2015!
\nI would like to ask your opinion on summarizing and generally looking at the data before running an ML algorithm.
\nI recently watched the lectures from the amazing "Learning From Data" course by Caltech, and a comment that the professor made stuck with me. It goes along the lines of "do not look at the data before feeding it to the algorithm".
\nIn my experience, in and out of Kaggle, looking at the data, without contaminating it (data snooping), has some benefits.
\nFor example, at Kaggle, separating a training and a validation set, and then summarizing, exploring only the training set, to test after on the validation set, and then use the information to train on everything and make a submission.
\nIf on a real ML case, separate the test set, lock it somewhere, and then doing the above "Kaggle approach" with the rest of the data (cross validation when possible).
\nDo you find that just looking at the training set to understand the features, plotting histograms, etc is bad? What about using the input only, or using the outputs too?
\nThank you!
", "messages": [ { "id": "62149", "postDate": "12/31/2014 21:23:19", "content": "Hey fellow Kagglers, how are you?
\nFirst, Happy 2015!
\nI would like to ask your opinion on summarizing and generally looking at the data before running an ML algorithm.
\nI recently watched the lectures from the amazing "Learning From Data" course by Caltech, and a comment that the professor made stuck with me. It goes along the lines of "do not look at the data before feeding it to the algorithm".
\nIn my experience, in and out of Kaggle, looking at the data, without contaminating it (data snooping), has some benefits.
\nFor example, at Kaggle, separating a training and a validation set, and then summarizing, exploring only the training set, to test after on the validation set, and then use the information to train on everything and make a submission.
\nIf on a real ML case, separate the test set, lock it somewhere, and then doing the above "Kaggle approach" with the rest of the data (cross validation when possible).
\nDo you find that just looking at the training set to understand the features, plotting histograms, etc is bad? What about using the input only, or using the outputs too?
\nThank you!
", "rawMarkdown": "", "votes": null }, { "id": "62161", "postDate": "01/01/2015 02:03:38", "content": "I think it depends of the dataset. In general, if you are more concerned with feature engineering and minimizing model bias than with overfitting, than snooping at the data is probably more helpful than harmful.
", "rawMarkdown": "", "votes": null }, { "id": "62162", "postDate": "01/01/2015 02:15:39", "content": "It also depends on whether you are interested in a "pure" challenge (i.e., simulating what you might face in real life) or in winning a competition.
\nTaking the time to understand the data can go a long way in improving your score.
\nFor example, you can discover and remove outliers. You may find a subset of the data that is better fit with different model parameters. etc.
", "rawMarkdown": "", "votes": null }, { "id": "62163", "postDate": "01/01/2015 02:44:42", "content": "I think they are referring to getting a baseline for the data that you start with to get a general idea of the error before you start trying to do anything crazy with feature engineering. It would be naive to never look at it, but to start with you'll need to have something to compare your new ideas to.
\n\nAlso, you could spend a lot of time analyzing the data before realizing that what you have isn't going to lead to any useful outcomes. You could probably argue that you'll have to do some analysis as well to convince others that the data isn't useful, but knowing that there could be this limitation is helpful to start out with. From there you can start to think about where else you'll need to pull data from just to solve the problem, if it's possible.
\n\nThis doesn't pertain to kaggle as I'd assume that all of the competitions will have a somewhat "solvable" dataset, but elsewhere knowing to focus your efforts in another direction can save a lot of time.
", "rawMarkdown": "", "votes": null }, { "id": "62164", "postDate": "01/01/2015 03:08:59", "content": "Two days before the end of the "Acquire Valued Shoppers" challenge, I was about ~500 on the leader board (out of 950 competitors).
\nLooking at the data, it was clear that certain loyalty card numbers had completely unrealistic amount of purchases.
\nJust by splitting those out and training on them separately, I went from ~500 to 34th.
\nI really wish I would have had that insight early on in the competition!
", "rawMarkdown": "", "votes": null }, { "id": "62171", "postDate": "01/01/2015 13:02:36", "content": "@dkay
\nYes, they're probably talking about the baseline score.
\n@inversion
\nThis is what I am talking about, thanks for sharing.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 62161, "author_name": "danielyoo", "author_url": "", "post_date": "01/01/2015 02:03:38", "content": "I think it depends of the dataset. In general, if you are more concerned with feature engineering and minimizing model bias than with overfitting, than snooping at the data is probably more helpful than harmful.
", "votes": null, "replies": [] }, { "id": 62162, "author_name": "inversion", "author_url": "", "post_date": "01/01/2015 02:15:39", "content": "It also depends on whether you are interested in a "pure" challenge (i.e., simulating what you might face in real life) or in winning a competition.
\nTaking the time to understand the data can go a long way in improving your score.
\nFor example, you can discover and remove outliers. You may find a subset of the data that is better fit with different model parameters. etc.
", "votes": null, "replies": [] }, { "id": 62163, "author_name": "dkaylor", "author_url": "", "post_date": "01/01/2015 02:44:42", "content": "I think they are referring to getting a baseline for the data that you start with to get a general idea of the error before you start trying to do anything crazy with feature engineering. It would be naive to never look at it, but to start with you'll need to have something to compare your new ideas to.
\n\nAlso, you could spend a lot of time analyzing the data before realizing that what you have isn't going to lead to any useful outcomes. You could probably argue that you'll have to do some analysis as well to convince others that the data isn't useful, but knowing that there could be this limitation is helpful to start out with. From there you can start to think about where else you'll need to pull data from just to solve the problem, if it's possible.
\n\nThis doesn't pertain to kaggle as I'd assume that all of the competitions will have a somewhat "solvable" dataset, but elsewhere knowing to focus your efforts in another direction can save a lot of time.
", "votes": null, "replies": [] }, { "id": 62164, "author_name": "inversion", "author_url": "", "post_date": "01/01/2015 03:08:59", "content": "Two days before the end of the "Acquire Valued Shoppers" challenge, I was about ~500 on the leader board (out of 950 competitors).
\nLooking at the data, it was clear that certain loyalty card numbers had completely unrealistic amount of purchases.
\nJust by splitting those out and training on them separately, I went from ~500 to 34th.
\nI really wish I would have had that insight early on in the competition!
", "votes": null, "replies": [] }, { "id": 62171, "author_name": "mariofilho", "author_url": "", "post_date": "01/01/2015 13:02:36", "content": "@dkay
\nYes, they're probably talking about the baseline score.
\n@inversion
\nThis is what I am talking about, thanks for sharing.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "62149": "", "62161": "", "62162": "", "62163": "", "62164": "", "62171": "" }, "source": "meta" }