{ "id": 1321, "title": "kinds of sampling techniques", "url": "/competitions/WhatDoYouKnow/discussion/1321", "author_name": "", "post_date": "2012-01-31T14:05:59.893Z", "votes": null, "comment_count": 1, "views": 1780, "content": "

I was wondering what are the different sampling techniques that are being used for validation set?

\r\n


\r\nWe have valid_training.csv - and to create a validation file from this, I am doing the following:

\r\n

Take the last question that was answered by each user and put it in validation set. Keep the remaining in training.

\r\n


\r\nWhat are the other methods that users have found effective?

\r\n

(of course for testing, we can use valid_test.csv)

", "messages": [ { "id": "8366", "postDate": "01/31/2012 14:05:59", "content": "

I was wondering what are the different sampling techniques that are being used for validation set?

\r\n


\r\nWe have valid_training.csv - and to create a validation file from this, I am doing the following:

\r\n

Take the last question that was answered by each user and put it in validation set. Keep the remaining in training.

\r\n


\r\nWhat are the other methods that users have found effective?

\r\n

(of course for testing, we can use valid_test.csv)

", "rawMarkdown": "", "votes": null }, { "id": "8410", "postDate": "02/02/2012 18:33:53", "content": "

You may not want to use that method for creating a validation set -- by including all users, you include users who have not answered very many questions. (And, if you read the description of how the test set is created, you will note that those users would\r\n not be included in the test set.) You might want to use a method more like that used to generate the valid_training and valid_test files (described at http://www.kaggle.com/c/WhatDoYouKnow/data and in the thread http://www.kaggle.com/c/WhatDoYouKnow/forums/t/1083/purpose-of-valid-test-and-valid-training-files)\r\n You may also want to look at YetiMan's comment on generating validation sets:
\r\n
\r\n[quote=YetiMan;8385]
\r\n
\r\nMy first train/validation pair is the one provided for the contest.
\r\n
\r\nThe second is similar, but I take the next to the last question for each user with >=5 questions (and add valid_test back in).
\r\n
\r\nThe third goes back in time one more question.
\r\n
\r\nTo be honest I'm not sure I needed to use all three sets for that particular model. Cross-validation didn't offer much improvement over the single set result.
\r\n
\r\n[/quote]

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 8410, "author_name": "thomaslotze", "author_url": "", "post_date": "02/02/2012 18:33:53", "content": "

You may not want to use that method for creating a validation set -- by including all users, you include users who have not answered very many questions. (And, if you read the description of how the test set is created, you will note that those users would\r\n not be included in the test set.) You might want to use a method more like that used to generate the valid_training and valid_test files (described at http://www.kaggle.com/c/WhatDoYouKnow/data and in the thread http://www.kaggle.com/c/WhatDoYouKnow/forums/t/1083/purpose-of-valid-test-and-valid-training-files)\r\n You may also want to look at YetiMan's comment on generating validation sets:
\r\n
\r\n[quote=YetiMan;8385]
\r\n
\r\nMy first train/validation pair is the one provided for the contest.
\r\n
\r\nThe second is similar, but I take the next to the last question for each user with >=5 questions (and add valid_test back in).
\r\n
\r\nThe third goes back in time one more question.
\r\n
\r\nTo be honest I'm not sure I needed to use all three sets for that particular model. Cross-validation didn't offer much improvement over the single set result.
\r\n
\r\n[/quote]

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "8366": "", "8410": "" }, "source": "meta" }