{ "id": 164090, "title": "Clarification about CV", "url": "/competitions/global-wheat-detection/discussion/164090", "author_name": "", "post_date": "2020-07-04T16:47:16.424416900Z", "votes": 1, "comment_count": 13, "views": 0, "content": "

As there was a new update in competition few days back everyone started emphasizing on the importance of CV... Just to be clear from the training pipeling of yolov5 by @orkatz2 it splits the data into train and val... Does val means CV?? \nAnd will the performances on this validation set have a \"HUGE\" impact on scores??\nThank you for answering this noob question in advance

", "messages": [ { "id": "915337", "postDate": "07/04/2020 16:47:16", "content": "

As there was a new update in competition few days back everyone started emphasizing on the importance of CV... Just to be clear from the training pipeling of yolov5 by @orkatz2 it splits the data into train and val... Does val means CV?? \nAnd will the performances on this validation set have a \"HUGE\" impact on scores??\nThank you for answering this noob question in advance

", "rawMarkdown": "As there was a new update in competition few days back everyone started emphasizing on the importance of CV... Just to be clear from the training pipeling of yolov5 by @orkatz2 it splits the data into `train` and `val`... Does `val` means CV?? \nAnd will the performances on this validation set have a \"HUGE\" impact on scores??\nThank you for answering this noob question in advance", "votes": null }, { "id": "915372", "postDate": "07/04/2020 17:18:14", "content": "

CV means cross-validation. This page explains in more detail but briefly, CV means that you use all the data for training and all the data for validation but not at the same time. To achieve this it means that you have to train more than once and each time you validate on the portion of data that was not used for training. This is usually considered more reliable than validation using only part of the data. However, it can take much longer. If you use 5-fold cross-validation (which is common on Kaggle) it can take approximately 5 times as long as just training once on most of the training data and using the rest for validation.

", "rawMarkdown": "CV means cross-validation. This [page](https://machinelearningmastery.com/k-fold-cross-validation/#:~:text=Cross%2Dvalidation%20is%20a%20resampling,k%2Dfold%20cross%2Dvalidation.) explains in more detail but briefly, CV means that you use all the data for training and all the data for validation but not at the same time. To achieve this it means that you have to train more than once and each time you validate on the portion of data that was not used for training. This is usually considered more reliable than validation using only part of the data. However, it can take much longer. If you use 5-fold cross-validation (which is common on Kaggle) it can take approximately 5 times as long as just training once on most of the training data and using the rest for validation.", "votes": null }, { "id": "915393", "postDate": "07/04/2020 17:38:20", "content": "

Yeah it does take way too long. I think picking a good training and validation set and doing pseudo-labeling will still give better results than doing CV at least for this comp.

", "rawMarkdown": "Yeah it does take way too long. I think picking a good training and validation set and doing pseudo-labeling will still give better results than doing CV at least for this comp.", "votes": null }, { "id": "915402", "postDate": "07/04/2020 17:46:07", "content": "

Thank you @datahobbit and @jonykarki for your valuable insights...

", "rawMarkdown": "Thank you @datahobbit and @jonykarki for your valuable insights...", "votes": null }, { "id": "915404", "postDate": "07/04/2020 17:47:39", "content": "

@jonykarki , \nThough it may take a lot of time I guess it is worth trying as we have a month left for the competition to end and CV will play a strong role in determining the Private LB score...

", "rawMarkdown": "jonykarki , \nThough it may take a lot of time I guess it is worth trying as we have a month left for the competition to end and CV will play a strong role in determining the Private LB score...", "votes": null }, { "id": "915407", "postDate": "07/04/2020 17:51:00", "content": "

Oh yeah, for sure. It could be worth trying. Also, you say you're a beginner, but you're fifth in the competition man. you're pretty good. 😄

", "rawMarkdown": "Oh yeah, for sure. It could be worth trying. Also, you say you're a beginner, but you're fifth in the competition man. you're pretty good. 😄", "votes": null }, { "id": "915415", "postDate": "07/04/2020 17:59:32", "content": "

The reason for being fifth is because of my team mates \nAlso me being fifth doesn't mean I know everything right.. learning new things is a daily process ;)

", "rawMarkdown": "The reason for being fifth is because of my team mates \nAlso me being fifth doesn't mean I know everything right.. learning new things is a daily process ;)", "votes": null }, { "id": "915418", "postDate": "07/04/2020 18:06:02", "content": "

You're absolutely right. 💯

", "rawMarkdown": "You're absolutely right. 💯", "votes": null }, { "id": "916477", "postDate": "07/05/2020 17:34:33", "content": "

I am using OOF (out of fold prediction) to optimize the threshold value.\nAlso CV is not necessary for the final model. As we are more focused on the pseudo labeling and its working pretty well.

", "rawMarkdown": "I am using OOF (out of fold prediction) to optimize the threshold value.\nAlso CV is not necessary for the final model. As we are more focused on the pseudo labeling and its working pretty well.", "votes": null }, { "id": "916509", "postDate": "07/05/2020 17:55:05", "content": "

As mentioned, cross-validation is a good reflection of the private leaderboard. In the past, I've used just train / validation but then been bitten on the private leaderboard. But, in real-life, CV is very important. Not doing proper CV can cost a business time, money, and clients if a model is not carefully built and tested.

\n\n

Once the CV is solid and your model is good, you can train on all the data however.

", "rawMarkdown": "As mentioned, cross-validation is a good reflection of the private leaderboard. In the past, I've used just train / validation but then been bitten on the private leaderboard. But, in real-life, CV is very important. Not doing proper CV can cost a business time, money, and clients if a model is not carefully built and tested.\n\nOnce the CV is solid and your model is good, you can train on all the data however.", "votes": null }, { "id": "916525", "postDate": "07/05/2020 18:14:12", "content": "
\n

Once the CV is solid and your model is good, you can train on all the data however.

\n
\n\n

I'm not sure I agree with that. A validation set should almost always be used in order to select the best weights and prevent over/under-fitting.

", "rawMarkdown": "> Once the CV is solid and your model is good, you can train on all the data however.\n\nI'm not sure I agree with that. A validation set should almost always be used in order to select the best weights and prevent over/under-fitting.", "votes": null }, { "id": "916546", "postDate": "07/05/2020 18:32:24", "content": "

Thank you for your thoughts...

", "rawMarkdown": "Thank you for your thoughts...", "votes": null }, { "id": "916549", "postDate": "07/05/2020 18:34:18", "content": "

Thank you @rifat963 for sharing your knowledge I will work on it...

", "rawMarkdown": "Thank you @rifat963 for sharing your knowledge I will work on it...", "votes": null }, { "id": "917956", "postDate": "07/06/2020 20:40:24", "content": "

Sorry, yes I misspoke. I mostly meant that you need not run a CV. You COULD train on all the data but a small validation set is a good idea for the reasons you stated. Of course, if the data set is small, you may need to balance the need for training the data without a validation set to one with the set. You also need to be sure your validation set is a representation of the data itself too of course.

", "rawMarkdown": "Sorry, yes I misspoke. I mostly meant that you need not run a CV. You COULD train on all the data but a small validation set is a good idea for the reasons you stated. Of course, if the data set is small, you may need to balance the need for training the data without a validation set to one with the set. You also need to be sure your validation set is a representation of the data itself too of course.", "votes": null } ], "comments": [ { "id": 915372, "author_name": "datahobbit", "author_url": "", "post_date": "07/04/2020 17:18:14", "content": "

CV means cross-validation. This page explains in more detail but briefly, CV means that you use all the data for training and all the data for validation but not at the same time. To achieve this it means that you have to train more than once and each time you validate on the portion of data that was not used for training. This is usually considered more reliable than validation using only part of the data. However, it can take much longer. If you use 5-fold cross-validation (which is common on Kaggle) it can take approximately 5 times as long as just training once on most of the training data and using the rest for validation.

", "votes": null, "replies": [ { "id": 915393, "author_name": "jonykarki", "author_url": "", "post_date": "07/04/2020 17:38:20", "content": "

Yeah it does take way too long. I think picking a good training and validation set and doing pseudo-labeling will still give better results than doing CV at least for this comp.

", "votes": null, "replies": [] }, { "id": 915402, "author_name": "nitindatta", "author_url": "", "post_date": "07/04/2020 17:46:07", "content": "

Thank you @datahobbit and @jonykarki for your valuable insights...

", "votes": null, "replies": [] }, { "id": 915404, "author_name": "nitindatta", "author_url": "", "post_date": "07/04/2020 17:47:39", "content": "

@jonykarki , \nThough it may take a lot of time I guess it is worth trying as we have a month left for the competition to end and CV will play a strong role in determining the Private LB score...

", "votes": null, "replies": [] }, { "id": 915407, "author_name": "jonykarki", "author_url": "", "post_date": "07/04/2020 17:51:00", "content": "

Oh yeah, for sure. It could be worth trying. Also, you say you're a beginner, but you're fifth in the competition man. you're pretty good. 😄

", "votes": null, "replies": [] }, { "id": 915415, "author_name": "nitindatta", "author_url": "", "post_date": "07/04/2020 17:59:32", "content": "

The reason for being fifth is because of my team mates \nAlso me being fifth doesn't mean I know everything right.. learning new things is a daily process ;)

", "votes": null, "replies": [] }, { "id": 915418, "author_name": "jonykarki", "author_url": "", "post_date": "07/04/2020 18:06:02", "content": "

You're absolutely right. 💯

", "votes": null, "replies": [] } ] }, { "id": 916477, "author_name": "rifat963", "author_url": "", "post_date": "07/05/2020 17:34:33", "content": "

I am using OOF (out of fold prediction) to optimize the threshold value.\nAlso CV is not necessary for the final model. As we are more focused on the pseudo labeling and its working pretty well.

", "votes": null, "replies": [ { "id": 916549, "author_name": "nitindatta", "author_url": "", "post_date": "07/05/2020 18:34:18", "content": "

Thank you @rifat963 for sharing your knowledge I will work on it...

", "votes": null, "replies": [] } ] }, { "id": 916509, "author_name": "cbryant", "author_url": "", "post_date": "07/05/2020 17:55:05", "content": "

As mentioned, cross-validation is a good reflection of the private leaderboard. In the past, I've used just train / validation but then been bitten on the private leaderboard. But, in real-life, CV is very important. Not doing proper CV can cost a business time, money, and clients if a model is not carefully built and tested.

\n\n

Once the CV is solid and your model is good, you can train on all the data however.

", "votes": null, "replies": [ { "id": 916525, "author_name": "matthewmasters", "author_url": "", "post_date": "07/05/2020 18:14:12", "content": "
\n

Once the CV is solid and your model is good, you can train on all the data however.

\n
\n\n

I'm not sure I agree with that. A validation set should almost always be used in order to select the best weights and prevent over/under-fitting.

", "votes": null, "replies": [] }, { "id": 916546, "author_name": "nitindatta", "author_url": "", "post_date": "07/05/2020 18:32:24", "content": "

Thank you for your thoughts...

", "votes": null, "replies": [] }, { "id": 917956, "author_name": "cbryant", "author_url": "", "post_date": "07/06/2020 20:40:24", "content": "

Sorry, yes I misspoke. I mostly meant that you need not run a CV. You COULD train on all the data but a small validation set is a good idea for the reasons you stated. Of course, if the data set is small, you may need to balance the need for training the data without a validation set to one with the set. You also need to be sure your validation set is a representation of the data itself too of course.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "915337": "As there was a new update in competition few days back everyone started emphasizing on the importance of CV... Just to be clear from the training pipeling of yolov5 by @orkatz2 it splits the data into `train` and `val`... Does `val` means CV?? \nAnd will the performances on this validation set have a \"HUGE\" impact on scores??\nThank you for answering this noob question in advance", "915372": "CV means cross-validation. This [page](https://machinelearningmastery.com/k-fold-cross-validation/#:~:text=Cross%2Dvalidation%20is%20a%20resampling,k%2Dfold%20cross%2Dvalidation.) explains in more detail but briefly, CV means that you use all the data for training and all the data for validation but not at the same time. To achieve this it means that you have to train more than once and each time you validate on the portion of data that was not used for training. This is usually considered more reliable than validation using only part of the data. However, it can take much longer. If you use 5-fold cross-validation (which is common on Kaggle) it can take approximately 5 times as long as just training once on most of the training data and using the rest for validation.", "915393": "Yeah it does take way too long. I think picking a good training and validation set and doing pseudo-labeling will still give better results than doing CV at least for this comp.", "915402": "Thank you @datahobbit and @jonykarki for your valuable insights...", "915404": "jonykarki , \nThough it may take a lot of time I guess it is worth trying as we have a month left for the competition to end and CV will play a strong role in determining the Private LB score...", "915407": "Oh yeah, for sure. It could be worth trying. Also, you say you're a beginner, but you're fifth in the competition man. you're pretty good. 😄", "915415": "The reason for being fifth is because of my team mates \nAlso me being fifth doesn't mean I know everything right.. learning new things is a daily process ;)", "915418": "You're absolutely right. 💯", "916477": "I am using OOF (out of fold prediction) to optimize the threshold value.\nAlso CV is not necessary for the final model. As we are more focused on the pseudo labeling and its working pretty well.", "916509": "As mentioned, cross-validation is a good reflection of the private leaderboard. In the past, I've used just train / validation but then been bitten on the private leaderboard. But, in real-life, CV is very important. Not doing proper CV can cost a business time, money, and clients if a model is not carefully built and tested.\n\nOnce the CV is solid and your model is good, you can train on all the data however.", "916525": "> Once the CV is solid and your model is good, you can train on all the data however.\n\nI'm not sure I agree with that. A validation set should almost always be used in order to select the best weights and prevent over/under-fitting.", "916546": "Thank you for your thoughts...", "916549": "Thank you @rifat963 for sharing your knowledge I will work on it...", "917956": "Sorry, yes I misspoke. I mostly meant that you need not run a CV. You COULD train on all the data but a small validation set is a good idea for the reasons you stated. Of course, if the data set is small, you may need to balance the need for training the data without a validation set to one with the set. You also need to be sure your validation set is a representation of the data itself too of course." }, "source": "meta" }