Let's say I divided my train set into train and validation. Then I fit a classifier on train, validate on validation set. So what's next ? Do you, guys, use this classifier as is, or you retrain it on the whole train set (train + val) and use this one as your final classifier ?
\nI use about 15% of data as validation, but in this task with highly unbalanced classes 15% contains a lot of important information I assume. After I fit a CNN on 85% data how should I fit the whole dataset ? Considering my experience I need to lower the regularization, but still have no rule of thumb how to do it.
\nProbably another solution is to divide train set in another way, fit same classifier and blend it with the one which was fit on the first split.
\nIt will be great if someone shared his strategy about that.
", "messages": [ { "id": "64791", "postDate": "02/23/2015 22:58:32", "content": "Let's say I divided my train set into train and validation. Then I fit a classifier on train, validate on validation set. So what's next ? Do you, guys, use this classifier as is, or you retrain it on the whole train set (train + val) and use this one as your final classifier ?
\nI use about 15% of data as validation, but in this task with highly unbalanced classes 15% contains a lot of important information I assume. After I fit a CNN on 85% data how should I fit the whole dataset ? Considering my experience I need to lower the regularization, but still have no rule of thumb how to do it.
\nProbably another solution is to divide train set in another way, fit same classifier and blend it with the one which was fit on the first split.
\nIt will be great if someone shared his strategy about that.
", "rawMarkdown": "", "votes": null }, { "id": "64827", "postDate": "02/24/2015 22:54:12", "content": "I am wondering the same thing, I guess we can just run the model on the whole dataset with the same setting, e.g. learning rate, iterations ... or maybe change learning rate or iterations a little bit?
", "rawMarkdown": "", "votes": null }, { "id": "64985", "postDate": "02/26/2015 19:37:46", "content": "Here is my approach, which I think is pretty standard.
\nTraining set = 60%
Valid set = 20 %
Test set = 20%
Step 1: Training on the training set and get a valid score.
Choose the model with the best valid score.
With this model predict the test set to have a score.
Step 2: Training on the full data set : Train + Valid + Test
\nStep 3: Make predictions and submit. Usually score on the leaderboard is slightly better than score obtained at Step 1
\n\nI don't change the parameters because the point of Step 1 is to choose the best parameters using cross validation. If I change them afterward it makes the whole process useless and I could start at Step 3 as well.
\n\nIn my opinion your claim that you should lower the regularization is very model dependent. Not all models will perform the same when given more data. If your model have a high bias, more data won't do any good.
\nOne useful trick might be to train your model with different size of dataset to see how it react to more data. This is often useful just to see if your model needs more data or not.
\n\nHere are some advices from Andrew Ng Machine learning course. The section What should you try next? is particularly interesting.
", "rawMarkdown": "", "votes": null }, { "id": "65011", "postDate": "02/27/2015 07:30:12", "content": "Hello, intesting question.
Although I'm not actively competing here this was my strategy from galaxy zoo and Criteo.
Based on this article from Alex Krizhevsky.
\nhttps://code.google.com/p/cuda-convnet/wiki/Methodology
\nTrain with holout set until no improvement on holdout loss.
Remember the loss you are at with the trainset.
Then continue training on all data.. until you reach roughly the same loss as you were on the trainset before..
Graph
It takes a leap of faith and perhaps it's "old" knowledge..
thanks guys for the discussion, really helpful!
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 64827, "author_name": "skylibrary", "author_url": "", "post_date": "02/24/2015 22:54:12", "content": "I am wondering the same thing, I guess we can just run the model on the whole dataset with the same setting, e.g. learning rate, iterations ... or maybe change learning rate or iterations a little bit?
", "votes": null, "replies": [] }, { "id": 64985, "author_name": "alexandrekowalczyk", "author_url": "", "post_date": "02/26/2015 19:37:46", "content": "Here is my approach, which I think is pretty standard.
\nTraining set = 60%
Valid set = 20 %
Test set = 20%
Step 1: Training on the training set and get a valid score.
Choose the model with the best valid score.
With this model predict the test set to have a score.
Step 2: Training on the full data set : Train + Valid + Test
\nStep 3: Make predictions and submit. Usually score on the leaderboard is slightly better than score obtained at Step 1
\n\nI don't change the parameters because the point of Step 1 is to choose the best parameters using cross validation. If I change them afterward it makes the whole process useless and I could start at Step 3 as well.
\n\nIn my opinion your claim that you should lower the regularization is very model dependent. Not all models will perform the same when given more data. If your model have a high bias, more data won't do any good.
\nOne useful trick might be to train your model with different size of dataset to see how it react to more data. This is often useful just to see if your model needs more data or not.
\n\nHere are some advices from Andrew Ng Machine learning course. The section What should you try next? is particularly interesting.
", "votes": null, "replies": [] }, { "id": 65011, "author_name": "juliandewit", "author_url": "", "post_date": "02/27/2015 07:30:12", "content": "Hello, intesting question.
Although I'm not actively competing here this was my strategy from galaxy zoo and Criteo.
Based on this article from Alex Krizhevsky.
\nhttps://code.google.com/p/cuda-convnet/wiki/Methodology
\nTrain with holout set until no improvement on holdout loss.
Remember the loss you are at with the trainset.
Then continue training on all data.. until you reach roughly the same loss as you were on the trainset before..
Graph
It takes a leap of faith and perhaps it's "old" knowledge..
thanks guys for the discussion, really helpful!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "64791": "", "64827": "", "64985": "", "65011": "", "65060": "" }, "source": "meta" }