I couldn't find the answer to this question on the data page or in the forums: what is the purpose of the valid_training.csv and valid_test.csv files?
", "messages": [ { "id": "6715", "postDate": "11/22/2011 22:33:23", "content": "I couldn't find the answer to this question on the data page or in the forums: what is the purpose of the valid_training.csv and valid_test.csv files?
", "rawMarkdown": "", "votes": null }, { "id": "6719", "postDate": "11/22/2011 23:11:15", "content": "The purpose is to create a validation training and test set for participants to cross-validate different algorithms, without having to generate their own validation set. You can train each algorithm on the valid_training.csv file, have it predict for valid_test.csv,\r\n and compute the binomial deviance (since you have the outcomes for valid_test.csv) -- then you can estimate what that algorithm's performance would be on the actual test set (and choose an algorithm based on its performance on valid_test.csv). It's definitely\r\n not necessary (and there are likely better ways to generate a validation set), but I thought it might be useful for competitors who wanted a quick start on validation. Does that help?
", "rawMarkdown": "", "votes": null }, { "id": "6723", "postDate": "11/22/2011 23:43:56", "content": "How were valid_training.csv valid_test.csv created?
", "rawMarkdown": "", "votes": null }, { "id": "6724", "postDate": "11/22/2011 23:47:19", "content": "Thanks so much, Thomas, that explains it.
", "rawMarkdown": "", "votes": null }, { "id": "6726", "postDate": "11/23/2011 00:02:37", "content": "Chris -- the valid_test.csv is created by taking the last answer in the training set for every user in the test set; valid_training.csv is then all of the previous responses by those users, plus responses from users who aren't in the test set (it's described\r\n briefly at the bottom of http://www.kaggle.com/c/WhatDoYouKnow/Data)
", "rawMarkdown": "", "votes": null }, { "id": "8081", "postDate": "01/13/2012 01:04:34", "content": "I'm still confused about training.csv and valid_training.csv.
\r\nFrom the data description page, training.csv is generated as follows.
\r\n"Any later answers by this user [in the test set] are removed, and any earlier answers are included in the training set. All answers from users not in the test set are also used for the training set"
And from your reply:
\r\n"valid_training.csv is then all of the previous responses by those users [in the test set?], plus responses from users who aren't in the test set".
So, what's the difference between them?
\r\nAlso, to my understanding, test.csv, training.csv and valid_test.csv are generated as follows. Am I correct?
\r\nLet N(u) = no. of questions answered by user u.
\r\nLet Q(i,u) = the i-th question answered by user u.
test.csv and training.csv:
\r\nfor each user u
\r\n---if N(u) >= 6
\r\n------choose M=M(u) in {6,7,...,N(u)} at random
\r\n------insert into the test set the response to Q(M,u)
\r\n------insert into the training set the responses to
\r\n---------Q(1,u),...,Q(M-1,u)
\r\n---else
\r\n------insert into the training set the responses to
\r\n---------Q(1,u),...,Q(N(u),u)
valid_test.csv:
\r\nfor each user u in the test set
\r\n---insert into the test set the response to Q(M(u)-1, u)
By the way, as to the "6th question" (or i-th question, in general) answered by a user, what does "6th" mean? The 6th answered question when the questions are sorted in questionid, or the 6th answered question when the questions are sorted by "answeredat"?
", "rawMarkdown": "", "votes": null }, { "id": "8082", "postDate": "01/13/2012 01:15:55", "content": "You're correct on the test/training generation. (ith question is the ith question, when ordered by answered_at). For valid_test and valid_training, this is a particition on the training set, using an algorithm like the following:
\r\nvalid_training.csv, valid_test.csv:
\r\nfor each user u
if u is not in the test set
\r\n---insert into valid_training the responses to Q(1,u)...Q(N(u), u)
\r\nif u is in the test set
\r\n---insert into valid_training the responses to Q(1,u)...Q(M(u)-2, u)
---insert into valid_test the response to Q(M(u)-1, u)
\r\n
Does not seem like the training/test files were created the way they are on the databoard
\r\nThe databoard says: "The data used in this competition is a sample of Grockit students (from the past three years) answering questions to prepare for the sat, gmat, or act. The test/training split is derived by finding users who answered at least 6 questions,\r\n taking one of their answers (uniformly random, from their 6th question to their last), and inserting it into the test set. Any later answers by this user are removed, and any earlier answers are included in the training set"
\r\n\r\n
Then how come, 1/5th of the students in the training set have answered only 1 question?
\r\n
\r\n
rkirana: if a user in the sample has not answered at least 6 questions, all their questions are still inserted into the training set (but they are not included in the test set)
", "rawMarkdown": "", "votes": null }, { "id": "8153", "postDate": "01/18/2012 12:11:30", "content": "\r\n
valid_training.csv, valid_test.csv:
for each user u
if u is not in the test set\r\n---insert into valid_training the responses to Q(1,u)...Q(N(u), u)\r\nif u is in the test set\r\n
---insert into valid_training the responses to Q(1,u)...Q(M(u)-2, u)
---insert into valid_test the response to Q(M(u)-1, u)\r\n
mohit: that is correct, and intended
", "rawMarkdown": "", "votes": null }, { "id": "8167", "postDate": "01/19/2012 03:13:30", "content": "the validate set is not very reliable. my model does very well on valid_test. but binomial deviance is very high on real test set.
", "rawMarkdown": "", "votes": null }, { "id": "8168", "postDate": "01/19/2012 04:12:19", "content": "re: Runxiao
\r\nJust to state the common explanation:
\r\nIf you've custom tuned many parameters to get a very good score on the valid_test, then it is quite likely that you will not generalize as well to the real test set.
\r\nBy that I mean choosing how many training epochs for a neural network to use on the validtrain by repeatedly testing against validtest will give you an incorrect number to use against the leaderboard.
", "rawMarkdown": "", "votes": null }, { "id": "8169", "postDate": "01/19/2012 04:24:28", "content": "Runxiao,
\r\nIt seems reliable to me, please check this \r\nthread.
\r\nHow much difference between your validation CBD and the leaderboard CBD are you getting?
", "rawMarkdown": "", "votes": null }, { "id": "8260", "postDate": "01/25/2012 18:15:33", "content": "for submissions: are we allowed to train our model on both valid_training.csv and training.csv?
", "rawMarkdown": "", "votes": null }, { "id": "8265", "postDate": "01/26/2012 01:03:21", "content": "mathso, valid_training is just an abridged version of the training file so it shouldn't help.
\r\nSee the response by Thomas Lotze to mohit above.
", "rawMarkdown": "", "votes": null }, { "id": "8269", "postDate": "01/26/2012 09:03:32", "content": "@matso : How would you use the training data as is? The valid training is a nice ready made proxy.
\r\n@Runxiao : I submitted my first hasty entry an hour ago and also calculated a cross validated sample from valid test and valid train. Are you using log to base e and not 10?
The purpose is to create a validation training and test set for participants to cross-validate different algorithms, without having to generate their own validation set. You can train each algorithm on the valid_training.csv file, have it predict for valid_test.csv,\r\n and compute the binomial deviance (since you have the outcomes for valid_test.csv) -- then you can estimate what that algorithm's performance would be on the actual test set (and choose an algorithm based on its performance on valid_test.csv). It's definitely\r\n not necessary (and there are likely better ways to generate a validation set), but I thought it might be useful for competitors who wanted a quick start on validation. Does that help?
", "votes": null, "replies": [] }, { "id": 6723, "author_name": "cdubois", "author_url": "", "post_date": "11/22/2011 23:43:56", "content": "How were valid_training.csv valid_test.csv created?
", "votes": null, "replies": [] }, { "id": 6724, "author_name": "robertlachlan", "author_url": "", "post_date": "11/22/2011 23:47:19", "content": "Thanks so much, Thomas, that explains it.
", "votes": null, "replies": [] }, { "id": 6726, "author_name": "thomaslotze", "author_url": "", "post_date": "11/23/2011 00:02:37", "content": "Chris -- the valid_test.csv is created by taking the last answer in the training set for every user in the test set; valid_training.csv is then all of the previous responses by those users, plus responses from users who aren't in the test set (it's described\r\n briefly at the bottom of http://www.kaggle.com/c/WhatDoYouKnow/Data)
", "votes": null, "replies": [] }, { "id": 8081, "author_name": "thesuffocated", "author_url": "", "post_date": "01/13/2012 01:04:34", "content": "I'm still confused about training.csv and valid_training.csv.
\r\nFrom the data description page, training.csv is generated as follows.
\r\n"Any later answers by this user [in the test set] are removed, and any earlier answers are included in the training set. All answers from users not in the test set are also used for the training set"
And from your reply:
\r\n"valid_training.csv is then all of the previous responses by those users [in the test set?], plus responses from users who aren't in the test set".
So, what's the difference between them?
\r\nAlso, to my understanding, test.csv, training.csv and valid_test.csv are generated as follows. Am I correct?
\r\nLet N(u) = no. of questions answered by user u.
\r\nLet Q(i,u) = the i-th question answered by user u.
test.csv and training.csv:
\r\nfor each user u
\r\n---if N(u) >= 6
\r\n------choose M=M(u) in {6,7,...,N(u)} at random
\r\n------insert into the test set the response to Q(M,u)
\r\n------insert into the training set the responses to
\r\n---------Q(1,u),...,Q(M-1,u)
\r\n---else
\r\n------insert into the training set the responses to
\r\n---------Q(1,u),...,Q(N(u),u)
valid_test.csv:
\r\nfor each user u in the test set
\r\n---insert into the test set the response to Q(M(u)-1, u)
By the way, as to the "6th question" (or i-th question, in general) answered by a user, what does "6th" mean? The 6th answered question when the questions are sorted in questionid, or the 6th answered question when the questions are sorted by "answeredat"?
", "votes": null, "replies": [] }, { "id": 8082, "author_name": "thomaslotze", "author_url": "", "post_date": "01/13/2012 01:15:55", "content": "You're correct on the test/training generation. (ith question is the ith question, when ordered by answered_at). For valid_test and valid_training, this is a particition on the training set, using an algorithm like the following:
\r\nvalid_training.csv, valid_test.csv:
\r\nfor each user u
if u is not in the test set
\r\n---insert into valid_training the responses to Q(1,u)...Q(N(u), u)
\r\nif u is in the test set
\r\n---insert into valid_training the responses to Q(1,u)...Q(M(u)-2, u)
---insert into valid_test the response to Q(M(u)-1, u)
\r\n
Does not seem like the training/test files were created the way they are on the databoard
\r\nThe databoard says: "The data used in this competition is a sample of Grockit students (from the past three years) answering questions to prepare for the sat, gmat, or act. The test/training split is derived by finding users who answered at least 6 questions,\r\n taking one of their answers (uniformly random, from their 6th question to their last), and inserting it into the test set. Any later answers by this user are removed, and any earlier answers are included in the training set"
\r\n\r\n
Then how come, 1/5th of the students in the training set have answered only 1 question?
\r\n
\r\n
rkirana: if a user in the sample has not answered at least 6 questions, all their questions are still inserted into the training set (but they are not included in the test set)
", "votes": null, "replies": [] }, { "id": 8153, "author_name": "mohit29765", "author_url": "", "post_date": "01/18/2012 12:11:30", "content": "\r\n
valid_training.csv, valid_test.csv:
for each user u
if u is not in the test set\r\n---insert into valid_training the responses to Q(1,u)...Q(N(u), u)\r\nif u is in the test set\r\n
---insert into valid_training the responses to Q(1,u)...Q(M(u)-2, u)
---insert into valid_test the response to Q(M(u)-1, u)\r\n
mohit: that is correct, and intended
", "votes": null, "replies": [] }, { "id": 8167, "author_name": "runxiao", "author_url": "", "post_date": "01/19/2012 03:13:30", "content": "the validate set is not very reliable. my model does very well on valid_test. but binomial deviance is very high on real test set.
", "votes": null, "replies": [] }, { "id": 8168, "author_name": "biasvariance", "author_url": "", "post_date": "01/19/2012 04:12:19", "content": "re: Runxiao
\r\nJust to state the common explanation:
\r\nIf you've custom tuned many parameters to get a very good score on the valid_test, then it is quite likely that you will not generalize as well to the real test set.
\r\nBy that I mean choosing how many training epochs for a neural network to use on the validtrain by repeatedly testing against validtest will give you an incorrect number to use against the leaderboard.
", "votes": null, "replies": [] }, { "id": 8169, "author_name": "jpetterson", "author_url": "", "post_date": "01/19/2012 04:24:28", "content": "Runxiao,
\r\nIt seems reliable to me, please check this \r\nthread.
\r\nHow much difference between your validation CBD and the leaderboard CBD are you getting?
", "votes": null, "replies": [] }, { "id": 8260, "author_name": "mathso", "author_url": "", "post_date": "01/25/2012 18:15:33", "content": "for submissions: are we allowed to train our model on both valid_training.csv and training.csv?
", "votes": null, "replies": [] }, { "id": 8265, "author_name": "jstreet", "author_url": "", "post_date": "01/26/2012 01:03:21", "content": "mathso, valid_training is just an abridged version of the training file so it shouldn't help.
\r\nSee the response by Thomas Lotze to mohit above.
", "votes": null, "replies": [] }, { "id": 8269, "author_name": "pidtis", "author_url": "", "post_date": "01/26/2012 09:03:32", "content": "@matso : How would you use the training data as is? The valid training is a nice ready made proxy.
\r\n@Runxiao : I submitted my first hasty entry an hour ago and also calculated a cross validated sample from valid test and valid train. Are you using log to base e and not 10?