When training our model, we save the trained model after every epoch. Eventually, we select the model to use for test prediction manually out of all the models that were saved during training. Are we allowed to select the model to use in stage 2 manually in the same way after re-training the model, or does the number of epochs to train the model for need to be determined automatically?
", "messages": [ { "id": "311105", "postDate": "04/09/2018 11:50:04", "content": "When training our model, we save the trained model after every epoch. Eventually, we select the model to use for test prediction manually out of all the models that were saved during training. Are we allowed to select the model to use in stage 2 manually in the same way after re-training the model, or does the number of epochs to train the model for need to be determined automatically?
", "rawMarkdown": "When training our model, we save the trained model after every epoch. Eventually, we select the model to use for test prediction manually out of all the models that were saved during training. Are we allowed to select the model to use in stage 2 manually in the same way after re-training the model, or does the number of epochs to train the model for need to be determined automatically?", "votes": null }, { "id": "311180", "postDate": "04/09/2018 14:53:28", "content": "I'm sure this is not allowed, solution must not have any manual steps.
", "rawMarkdown": "I'm sure this is not allowed, solution must not have any manual steps.", "votes": null }, { "id": "311244", "postDate": "04/09/2018 17:19:13", "content": "https://www.kaggle.com/two-stage-faq
\n\n\n\n\nWhat happens if I want to change something in my code in the second stage?
\n \nWe expect you may need to make some \"non scientific\" alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.
\n
Number of epochs counts as a hyperparameter, I think, and manual changes are not automated tuning, so no.
", "rawMarkdown": "https://www.kaggle.com/two-stage-faq\n\n>**What happens if I want to change something in my code in the second stage?**\n\n>We expect you may need to make some \"non scientific\" alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.\n\nNumber of epochs counts as a hyperparameter, I think, and manual changes are not automated tuning, so no.", "votes": null }, { "id": "311299", "postDate": "04/09/2018 19:36:11", "content": "From the information that I found so far, the rules appear to be ambiguous about that, and may actually suggest that manual determination of when to stop training can be acceptable. As far as I understand, we are allowed to make more than 2 submissions in stage 2 then select the best 2 submissions for final scoring. Is my understanding correct? If so, selecting the best 2 submissions is a manual step, so I do not think we can generalize to the point of saying that no manual steps are allowed. In fact, the same effect of selecting the number of epochs manually can largely be achieved by running training multiple times for the same number of epochs and selecting the best result according to a non-automated criterion.
\n\nIs there a definitive statement of what is and is not allowed in the second stage of the competition that can resolve this ambiguity? All the material that I found so far try to define this by means of an example (e.g. paths vs. hyperparameters), but I am not sure how to accurately generalize from these examples.
", "rawMarkdown": "From the information that I found so far, the rules appear to be ambiguous about that, and may actually suggest that manual determination of when to stop training can be acceptable. As far as I understand, we are allowed to make more than 2 submissions in stage 2 then select the best 2 submissions for final scoring. Is my understanding correct? If so, selecting the best 2 submissions is a manual step, so I do not think we can generalize to the point of saying that no manual steps are allowed. In fact, the same effect of selecting the number of epochs manually can largely be achieved by running training multiple times for the same number of epochs and selecting the best result according to a non-automated criterion.\n\nIs there a definitive statement of what is and is not allowed in the second stage of the competition that can resolve this ambiguity? All the material that I found so far try to define this by means of an example (e.g. paths vs. hyperparameters), but I am not sure how to accurately generalize from these examples.", "votes": null }, { "id": "311300", "postDate": "04/09/2018 19:39:57", "content": "I am not sure if the number of epochs counts as a hyperparameter for the purpose of that FAQ answer.
", "rawMarkdown": "I am not sure if the number of epochs counts as a hyperparameter for the purpose of that FAQ answer.", "votes": null }, { "id": "311311", "postDate": "04/09/2018 20:09:51", "content": "You should select your 2 submissions (edit: not submissions on LB, but submissions you will make) before the model upload and describe which submissions you will make (for example, provide exact command for training and prediction for both submissions). I think the principle is very clear, I understand it this way: ideally, there should be one command you run without any thinking and you get two submissions. Or you provide a README that is essentially equivalent to this (run this command with this arguments, then run that command with that arguments).
", "rawMarkdown": "You should select your 2 submissions (edit: not submissions on LB, but submissions you will make) before the model upload and describe which submissions you will make (for example, provide exact command for training and prediction for both submissions). I think the principle is very clear, I understand it this way: ideally, there should be one command you run without any thinking and you get two submissions. Or you provide a README that is essentially equivalent to this (run this command with this arguments, then run that command with that arguments).", "votes": null }, { "id": "311322", "postDate": "04/09/2018 20:31:55", "content": "Does anyone know how many submissions we are allowed to make during stage 2? Is it more than 2?
\n\nOn one hand, if everything is automated then in theory there should be no need for more than 2 submissions (one per model). On the other hand, it is conceivable that someone with a great model could make a submission that would get a really low score because of a file formatting issue or something else that has nothing to do with training. In this case extra submissions would definitely help.
", "rawMarkdown": "Does anyone know how many submissions we are allowed to make during stage 2? Is it more than 2?\n\nOn one hand, if everything is automated then in theory there should be no need for more than 2 submissions (one per model). On the other hand, it is conceivable that someone with a great model could make a submission that would get a really low score because of a file formatting issue or something else that has nothing to do with training. In this case extra submissions would definitely help.", "votes": null } ], "comments": [ { "id": 311180, "author_name": "lopuhin", "author_url": "", "post_date": "04/09/2018 14:53:28", "content": "I'm sure this is not allowed, solution must not have any manual steps.
", "votes": null, "replies": [ { "id": 311299, "author_name": "mahmouda", "author_url": "", "post_date": "04/09/2018 19:36:11", "content": "From the information that I found so far, the rules appear to be ambiguous about that, and may actually suggest that manual determination of when to stop training can be acceptable. As far as I understand, we are allowed to make more than 2 submissions in stage 2 then select the best 2 submissions for final scoring. Is my understanding correct? If so, selecting the best 2 submissions is a manual step, so I do not think we can generalize to the point of saying that no manual steps are allowed. In fact, the same effect of selecting the number of epochs manually can largely be achieved by running training multiple times for the same number of epochs and selecting the best result according to a non-automated criterion.
\n\nIs there a definitive statement of what is and is not allowed in the second stage of the competition that can resolve this ambiguity? All the material that I found so far try to define this by means of an example (e.g. paths vs. hyperparameters), but I am not sure how to accurately generalize from these examples.
", "votes": null, "replies": [] }, { "id": 311311, "author_name": "lopuhin", "author_url": "", "post_date": "04/09/2018 20:09:51", "content": "You should select your 2 submissions (edit: not submissions on LB, but submissions you will make) before the model upload and describe which submissions you will make (for example, provide exact command for training and prediction for both submissions). I think the principle is very clear, I understand it this way: ideally, there should be one command you run without any thinking and you get two submissions. Or you provide a README that is essentially equivalent to this (run this command with this arguments, then run that command with that arguments).
", "votes": null, "replies": [] } ] }, { "id": 311244, "author_name": "pointyointment", "author_url": "", "post_date": "04/09/2018 17:19:13", "content": "https://www.kaggle.com/two-stage-faq
\n\n\n\n\nWhat happens if I want to change something in my code in the second stage?
\n \nWe expect you may need to make some \"non scientific\" alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.
\n
Number of epochs counts as a hyperparameter, I think, and manual changes are not automated tuning, so no.
", "votes": null, "replies": [ { "id": 311300, "author_name": "mahmouda", "author_url": "", "post_date": "04/09/2018 19:39:57", "content": "I am not sure if the number of epochs counts as a hyperparameter for the purpose of that FAQ answer.
", "votes": null, "replies": [] } ] }, { "id": 311322, "author_name": "skriventsov", "author_url": "", "post_date": "04/09/2018 20:31:55", "content": "Does anyone know how many submissions we are allowed to make during stage 2? Is it more than 2?
\n\nOn one hand, if everything is automated then in theory there should be no need for more than 2 submissions (one per model). On the other hand, it is conceivable that someone with a great model could make a submission that would get a really low score because of a file formatting issue or something else that has nothing to do with training. In this case extra submissions would definitely help.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "311105": "When training our model, we save the trained model after every epoch. Eventually, we select the model to use for test prediction manually out of all the models that were saved during training. Are we allowed to select the model to use in stage 2 manually in the same way after re-training the model, or does the number of epochs to train the model for need to be determined automatically?", "311180": "I'm sure this is not allowed, solution must not have any manual steps.", "311244": "https://www.kaggle.com/two-stage-faq\n\n>**What happens if I want to change something in my code in the second stage?**\n\n>We expect you may need to make some \"non scientific\" alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.\n\nNumber of epochs counts as a hyperparameter, I think, and manual changes are not automated tuning, so no.", "311299": "From the information that I found so far, the rules appear to be ambiguous about that, and may actually suggest that manual determination of when to stop training can be acceptable. As far as I understand, we are allowed to make more than 2 submissions in stage 2 then select the best 2 submissions for final scoring. Is my understanding correct? If so, selecting the best 2 submissions is a manual step, so I do not think we can generalize to the point of saying that no manual steps are allowed. In fact, the same effect of selecting the number of epochs manually can largely be achieved by running training multiple times for the same number of epochs and selecting the best result according to a non-automated criterion.\n\nIs there a definitive statement of what is and is not allowed in the second stage of the competition that can resolve this ambiguity? All the material that I found so far try to define this by means of an example (e.g. paths vs. hyperparameters), but I am not sure how to accurately generalize from these examples.", "311300": "I am not sure if the number of epochs counts as a hyperparameter for the purpose of that FAQ answer.", "311311": "You should select your 2 submissions (edit: not submissions on LB, but submissions you will make) before the model upload and describe which submissions you will make (for example, provide exact command for training and prediction for both submissions). I think the principle is very clear, I understand it this way: ideally, there should be one command you run without any thinking and you get two submissions. Or you provide a README that is essentially equivalent to this (run this command with this arguments, then run that command with that arguments).", "311322": "Does anyone know how many submissions we are allowed to make during stage 2? Is it more than 2?\n\nOn one hand, if everything is automated then in theory there should be no need for more than 2 submissions (one per model). On the other hand, it is conceivable that someone with a great model could make a submission that would get a really low score because of a file formatting issue or something else that has nothing to do with training. In this case extra submissions would definitely help." }, "source": "meta" }