I am really curious why most public notebook is not doing any cross validation and only making one model with some (maybe) tuned hyper parameter, are they doing the CV in the background and publishing the notebook with best hyper parameter?
", "messages": [ { "id": "2455017", "postDate": "09/25/2023 08:57:14", "content": "I am really curious why most public notebook is not doing any cross validation and only making one model with some (maybe) tuned hyper parameter, are they doing the CV in the background and publishing the notebook with best hyper parameter?
", "rawMarkdown": "I am really curious why most public notebook is not doing any cross validation and only making one model with some (maybe) tuned hyper parameter, are they doing the CV in the background and publishing the notebook with best hyper parameter?", "votes": null }, { "id": "2455024", "postDate": "09/25/2023 09:05:54", "content": "Possibly the use of public notebook is only for MVP, i.e. the lowest hanging fruit that we can develop it further. Implementing CV might take some time that violate this goal.
\nEither way, it's good for anyone to start a public notebook with innovations like implementing CV or other methods. It will attract more medals though.
", "rawMarkdown": "Possibly the use of public notebook is only for MVP, i.e. the lowest hanging fruit that we can develop it further. Implementing CV might take some time that violate this goal.\n\nEither way, it's good for anyone to start a public notebook with innovations like implementing CV or other methods. It will attract more medals though.", "votes": null }, { "id": "2455046", "postDate": "09/25/2023 09:22:32", "content": "wow, yeah, that's true, and think that answered my other question of why they are all using the basic random forest as base model. Thanks!
", "rawMarkdown": "wow, yeah, that's true, and think that answered my other question of why they are all using the basic random forest as base model. Thanks!", "votes": null }, { "id": "2455501", "postDate": "09/25/2023 14:05:08", "content": "Using a CV blows the notebook out of memory and creates issues, hence a no-CV approach is chosen for now @yuchen2066
", "rawMarkdown": "Using a CV blows the notebook out of memory and creates issues, hence a no-CV approach is chosen for now @yuchen2066", "votes": null }, { "id": "2457305", "postDate": "09/26/2023 19:18:54", "content": "True, I tried CV, and it occasionally also will do notebook timeout for submission
", "rawMarkdown": "True, I tried CV, and it occasionally also will do **notebook timeout** for submission", "votes": null }, { "id": "2457783", "postDate": "09/27/2023 06:53:50", "content": "Dear @yuchen2066
\nCV is designed for a local assessment of ones performance. When submitting the notebook one should turn off the local assessment beforehand as it simply becomes a waste of CPU and memory.
\nAll the best,
\ncarl
Dear Carl,
\nThank you for your answer. However, I always have the thought of using cross-validation to train multiple models and then averaging their predictions could potentially make the prediction more accurate or robust. Is this correct? and are we not using CV for prediction in this competition to save memory and time, because of the size of test file?
\nThank you!
\nYuchen.
", "rawMarkdown": "Dear Carl,\n\nThank you for your answer. However, I always have the thought of using cross-validation to train multiple models and then averaging their predictions could potentially make the prediction more accurate or robust. Is this correct? and are we not using CV for prediction in this competition to save memory and time, because of the size of test file?\n\nThank you!\n\nYuchen.", "votes": null }, { "id": "2458012", "postDate": "09/27/2023 09:26:26", "content": "Dear @yuchen2066
\nIndeed one could use the OOF predictions from multiple models obtained via a CV loop to then undertake stacking. Given how computationally intense this is I would suggest saving such a model beforehand, and then load the model into a dedicated \"prediction\" notebook (β¦or for members of the deep learning fraternity, an \"inference\" notebook π)
\nAll the best,
\ncarl
Hahaha π, Thanks for confirming. For now, I will stick with just one model. Interestingly, a LightGBM model with 3-fold cross-validation takes only 10 minutes to train, but it seems to require over 9 hours for inference on the test set. By the way, does increasing the \"n-estimators\" parameter in LightGBM significantly slow down the inference process? I suspect this might be the reason for the extended inference time.
", "rawMarkdown": "Hahaha π, Thanks for confirming. For now, I will stick with just one model. Interestingly, a LightGBM model with 3-fold cross-validation takes only 10 minutes to train, but it seems to require over 9 hours for inference on the test set. By the way, does increasing the \"n-estimators\" parameter in LightGBM significantly slow down the inference process? I suspect this might be the reason for the extended inference time.", "votes": null } ], "comments": [ { "id": 2455024, "author_name": "rimbax", "author_url": "", "post_date": "09/25/2023 09:05:54", "content": "Possibly the use of public notebook is only for MVP, i.e. the lowest hanging fruit that we can develop it further. Implementing CV might take some time that violate this goal.
\nEither way, it's good for anyone to start a public notebook with innovations like implementing CV or other methods. It will attract more medals though.
", "votes": null, "replies": [ { "id": 2455046, "author_name": "yuchen2066", "author_url": "", "post_date": "09/25/2023 09:22:32", "content": "wow, yeah, that's true, and think that answered my other question of why they are all using the basic random forest as base model. Thanks!
", "votes": null, "replies": [] } ] }, { "id": 2455501, "author_name": "ravi20076", "author_url": "", "post_date": "09/25/2023 14:05:08", "content": "Using a CV blows the notebook out of memory and creates issues, hence a no-CV approach is chosen for now @yuchen2066
", "votes": null, "replies": [ { "id": 2457305, "author_name": "yuchen2066", "author_url": "", "post_date": "09/26/2023 19:18:54", "content": "True, I tried CV, and it occasionally also will do notebook timeout for submission
", "votes": null, "replies": [ { "id": 2457783, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "09/27/2023 06:53:50", "content": "Dear @yuchen2066
\nCV is designed for a local assessment of ones performance. When submitting the notebook one should turn off the local assessment beforehand as it simply becomes a waste of CPU and memory.
\nAll the best,
\ncarl
Dear Carl,
\nThank you for your answer. However, I always have the thought of using cross-validation to train multiple models and then averaging their predictions could potentially make the prediction more accurate or robust. Is this correct? and are we not using CV for prediction in this competition to save memory and time, because of the size of test file?
\nThank you!
\nYuchen.
", "votes": null, "replies": [ { "id": 2458012, "author_name": "carlmcbrideellis", "author_url": "", "post_date": "09/27/2023 09:26:26", "content": "Dear @yuchen2066
\nIndeed one could use the OOF predictions from multiple models obtained via a CV loop to then undertake stacking. Given how computationally intense this is I would suggest saving such a model beforehand, and then load the model into a dedicated \"prediction\" notebook (β¦or for members of the deep learning fraternity, an \"inference\" notebook π)
\nAll the best,
\ncarl
Hahaha π, Thanks for confirming. For now, I will stick with just one model. Interestingly, a LightGBM model with 3-fold cross-validation takes only 10 minutes to train, but it seems to require over 9 hours for inference on the test set. By the way, does increasing the \"n-estimators\" parameter in LightGBM significantly slow down the inference process? I suspect this might be the reason for the extended inference time.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2455017": "I am really curious why most public notebook is not doing any cross validation and only making one model with some (maybe) tuned hyper parameter, are they doing the CV in the background and publishing the notebook with best hyper parameter?", "2455024": "Possibly the use of public notebook is only for MVP, i.e. the lowest hanging fruit that we can develop it further. Implementing CV might take some time that violate this goal.\n\nEither way, it's good for anyone to start a public notebook with innovations like implementing CV or other methods. It will attract more medals though.", "2455046": "wow, yeah, that's true, and think that answered my other question of why they are all using the basic random forest as base model. Thanks!", "2455501": "Using a CV blows the notebook out of memory and creates issues, hence a no-CV approach is chosen for now @yuchen2066", "2457305": "True, I tried CV, and it occasionally also will do **notebook timeout** for submission", "2457783": "Dear @yuchen2066 \n\nCV is designed for a local assessment of ones performance. When submitting the notebook one should turn off the local assessment beforehand as it simply becomes a waste of CPU and memory.\n\nAll the best,\ncarl", "2457999": "Dear Carl,\n\nThank you for your answer. However, I always have the thought of using cross-validation to train multiple models and then averaging their predictions could potentially make the prediction more accurate or robust. Is this correct? and are we not using CV for prediction in this competition to save memory and time, because of the size of test file?\n\nThank you!\n\nYuchen.", "2458012": "Dear @yuchen2066 \n\nIndeed one could use the OOF predictions from multiple models obtained via a CV loop to then undertake stacking. Given how computationally intense this is I would suggest saving such a model beforehand, and then load the model into a dedicated \"prediction\" notebook (...or for members of the deep learning fraternity, an \"inference\" notebook π)\n\nAll the best,\ncarl", "2458033": "Hahaha π, Thanks for confirming. For now, I will stick with just one model. Interestingly, a LightGBM model with 3-fold cross-validation takes only 10 minutes to train, but it seems to require over 9 hours for inference on the test set. By the way, does increasing the \"n-estimators\" parameter in LightGBM significantly slow down the inference process? I suspect this might be the reason for the extended inference time." }, "source": "meta" }