Again, maybe a silly question.
\n\nThe rules say that we must upload our model before stage1 ends. Does it mean we have to use that model for stage 2, or can we further train/improve our model with stage 2 train data?
", "messages": [ { "id": "291023", "postDate": "03/05/2018 13:31:53", "content": "Again, maybe a silly question.
\n\nThe rules say that we must upload our model before stage1 ends. Does it mean we have to use that model for stage 2, or can we further train/improve our model with stage 2 train data?
", "rawMarkdown": "Again, maybe a silly question.\n\nThe rules say that we must upload our model before stage1 ends. Does it mean we have to use that model for stage 2, or can we further train/improve our model with stage 2 train data?", "votes": null }, { "id": "291050", "postDate": "03/05/2018 14:19:26", "content": "April 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline - it is not clear what is uploaded? csv? model weights in docker?
", "rawMarkdown": "April 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline - it is not clear what is uploaded? csv? model weights in docker?", "votes": null }, { "id": "291072", "postDate": "03/05/2018 15:24:19", "content": "According to what we've learned from ncfm (fishy) competition:
\n\nYou have to freeze your pipeline and do a blind-shot on stage twos' data. If you want to be prize eligible, your have to verify that your uploaded pipeline produces the prize-submission. Of course, if you know you are not going to win a prize, you can upload bullshit and retrain on the data to get a good (undeserved!) rank.
\n\nPlease note: If you do not upload a stage2 submission, you will be removed from the leaderboard. For example, if you are on rank 300 out of 2000 in stage1 and only the first 300 submit a stage2 submission and you stay on 300, you will be ranked last and get less points. Hence your stage1 ranking effort was for nothing. Because of this, in ncfm they decide to keep the stage1 submitter in the ranking, after the stage2 submitter.
", "rawMarkdown": "According to what we've learned from ncfm (fishy) competition:\n\n 1. You can't retrain your model in stage 2. Of course, fixes to handle and read the new data may be OK.\n 2. You must upload your fully pipeline (code, weights, models, etc.) by April 11: More -> Team -> Upload\n 3. If you don't want to upload your whole pipeline (because of size), zip your file and compute its sha256 hash. Upload that hash.\n\nYou have to freeze your pipeline and do a blind-shot on stage twos' data. If you want to be prize eligible, your have to verify that your uploaded pipeline produces the prize-submission. Of course, if you know you are not going to win a prize, you can upload bullshit and retrain on the data to get a good (undeserved!) rank.\n\nPlease note: If you do not upload a stage2 submission, you will be removed from the leaderboard. For example, if you are on rank 300 out of 2000 in stage1 and only the first 300 submit a stage2 submission and you stay on 300, you will be ranked last and get less points. Hence your stage1 ranking effort was for nothing. Because of this, in ncfm they decide to keep the stage1 submitter in the ranking, after the stage2 submitter.", "votes": null }, { "id": "291083", "postDate": "03/05/2018 15:46:23", "content": "Thanks, that was my understanding too, but it is not what the rules say. Clarification from Kaggle would be most welcome. My strategy going forward depends on the possibility to retrain or not in stage2.
\n\nI just checked the rules for ncfm, they were not explicit either. What made it explicit was a last minute post (2 days before stage1 ends) which probably took many people by surprise if I judge by the number of down votes.
\n\nIt is because I don't want to have a similar surprise that I am asking the question now.
", "rawMarkdown": "Thanks, that was my understanding too, but it is not what the rules say. Clarification from Kaggle would be most welcome. My strategy going forward depends on the possibility to retrain or not in stage2.\n\nI just checked the rules for ncfm, they were not explicit either. What made it explicit was a [last minute post][1] (2 days before stage1 ends) which probably took many people by surprise if I judge by the number of down votes.\n\nIt is because I don't want to have a similar surprise that I am asking the question now.\n\n\n [1]: https://www.kaggle.com/c/the-nature-conservancy-fisheries-monitoring/discussion/31103", "votes": null }, { "id": "291086", "postDate": "03/05/2018 15:52:56", "content": "Retrain on what? There is no stage2 train data just a test set. Or do you want to train on stage1 test data by using the published test labels or unsupervised on stage2 test?
", "rawMarkdown": "Retrain on what? There is no stage2 train data just a test set. Or do you want to train on stage1 test data by using the published test labels or unsupervised on stage2 test?", "votes": null }, { "id": "291091", "postDate": "03/05/2018 16:01:52", "content": "masks for stage1 test will be disclosed.
", "rawMarkdown": "masks for stage1 test will be disclosed.", "votes": null }, { "id": "291118", "postDate": "03/05/2018 16:45:57", "content": "As far as I understand, we can't use stage1 test labels to re-train our models, the models must be final at the upload stage. I was sure it was answered somewhere but I can't find the source, the best I found is this https://www.kaggle.com/c/data-science-bowl-2018/discussion/48136#282212
\n\nI hope someone from kaggle clarifies this or points to a place where it is clarified.
", "rawMarkdown": "As far as I understand, we can't use stage1 test labels to re-train our models, the models must be final at the upload stage. I was sure it was answered *somewhere* but I can't find the source, the best I found is this https://www.kaggle.com/c/data-science-bowl-2018/discussion/48136#282212\n\nI hope someone from kaggle clarifies this or points to a place where it is clarified.", "votes": null }, { "id": "291139", "postDate": "03/05/2018 17:31:42", "content": "Again, this is my understanding too, but it is NOT part of the rules.
", "rawMarkdown": "Again, this is my understanding too, but it is NOT part of the rules.", "votes": null }, { "id": "291219", "postDate": "03/05/2018 20:19:14", "content": "well, I have bit different understanding - you can do anything, as long as it's automated (i.e. you don't introduce new code)
\n\ni.e. you can retrain on stage 1 test labels, if all you need to do is run for example a shell script (which you already had)
\n\nyou definitely can't change anything in the code (unless it's done automatically - like maybe a dynamic NN based on cross-validation etc.)
\n\nbut definitely would love to hear an official answer as well, for example if I would make the NN architecture based on command line parameters, would that be OK? based on past I would say yes, since it's not a change in code, and Kaggle can reproduce the same result when you tell them the exact command line you used etc.
", "rawMarkdown": "well, I have bit different understanding - you can do anything, as long as it's automated (i.e. you don't introduce new code)\n\ni.e. you can retrain on stage 1 test labels, if all you need to do is run for example a shell script (which you already had)\n\nyou definitely can't change anything in the code (unless it's done automatically - like maybe a dynamic NN based on cross-validation etc.)\n\nbut definitely would love to hear an official answer as well, for example if I would make the NN architecture based on command line parameters, would that be OK? based on past I would say yes, since it's not a change in code, and Kaggle can reproduce the same result when you tell them the exact command line you used etc.", "votes": null }, { "id": "291260", "postDate": "03/05/2018 21:31:47", "content": "I like it better! We need to know.
", "rawMarkdown": "I like it better! We need to know.", "votes": null }, { "id": "294971", "postDate": "03/12/2018 22:49:14", "content": "I would not expect custom command line parameters to be allowed, you can't change hyperparameters and you can fit a lot of them into command line
", "rawMarkdown": "I would not expect custom command line parameters to be allowed, you can't change hyperparameters and you can fit a lot of them into command line", "votes": null }, { "id": "301042", "postDate": "03/22/2018 07:53:22", "content": "Hopefully we can rely on this to be fully applicable to the competition:
\n\n", "rawMarkdown": "Hopefully we can rely on this to be fully applicable to the competition:\n\n [Two-Stage Competition FAQ][1]\n\n\n [1]: https://www.kaggle.com/two-stage-faq \"https://www.kaggle.com/two-stage-faq\"", "votes": null }, { "id": "302228", "postDate": "03/23/2018 20:17:56", "content": "very helpful
", "rawMarkdown": "very helpful", "votes": null } ], "comments": [ { "id": 291050, "author_name": "maksimovka", "author_url": "", "post_date": "03/05/2018 14:19:26", "content": "April 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline - it is not clear what is uploaded? csv? model weights in docker?
", "votes": null, "replies": [] }, { "id": 291072, "author_name": "firolino", "author_url": "", "post_date": "03/05/2018 15:24:19", "content": "According to what we've learned from ncfm (fishy) competition:
\n\nYou have to freeze your pipeline and do a blind-shot on stage twos' data. If you want to be prize eligible, your have to verify that your uploaded pipeline produces the prize-submission. Of course, if you know you are not going to win a prize, you can upload bullshit and retrain on the data to get a good (undeserved!) rank.
\n\nPlease note: If you do not upload a stage2 submission, you will be removed from the leaderboard. For example, if you are on rank 300 out of 2000 in stage1 and only the first 300 submit a stage2 submission and you stay on 300, you will be ranked last and get less points. Hence your stage1 ranking effort was for nothing. Because of this, in ncfm they decide to keep the stage1 submitter in the ranking, after the stage2 submitter.
", "votes": null, "replies": [ { "id": 291083, "author_name": "cpmpml", "author_url": "", "post_date": "03/05/2018 15:46:23", "content": "Thanks, that was my understanding too, but it is not what the rules say. Clarification from Kaggle would be most welcome. My strategy going forward depends on the possibility to retrain or not in stage2.
\n\nI just checked the rules for ncfm, they were not explicit either. What made it explicit was a last minute post (2 days before stage1 ends) which probably took many people by surprise if I judge by the number of down votes.
\n\nIt is because I don't want to have a similar surprise that I am asking the question now.
", "votes": null, "replies": [] }, { "id": 291086, "author_name": "firolino", "author_url": "", "post_date": "03/05/2018 15:52:56", "content": "Retrain on what? There is no stage2 train data just a test set. Or do you want to train on stage1 test data by using the published test labels or unsupervised on stage2 test?
", "votes": null, "replies": [] }, { "id": 291091, "author_name": "cpmpml", "author_url": "", "post_date": "03/05/2018 16:01:52", "content": "masks for stage1 test will be disclosed.
", "votes": null, "replies": [] }, { "id": 291118, "author_name": "lopuhin", "author_url": "", "post_date": "03/05/2018 16:45:57", "content": "As far as I understand, we can't use stage1 test labels to re-train our models, the models must be final at the upload stage. I was sure it was answered somewhere but I can't find the source, the best I found is this https://www.kaggle.com/c/data-science-bowl-2018/discussion/48136#282212
\n\nI hope someone from kaggle clarifies this or points to a place where it is clarified.
", "votes": null, "replies": [] }, { "id": 291139, "author_name": "cpmpml", "author_url": "", "post_date": "03/05/2018 17:31:42", "content": "Again, this is my understanding too, but it is NOT part of the rules.
", "votes": null, "replies": [] } ] }, { "id": 291219, "author_name": "steelrose", "author_url": "", "post_date": "03/05/2018 20:19:14", "content": "well, I have bit different understanding - you can do anything, as long as it's automated (i.e. you don't introduce new code)
\n\ni.e. you can retrain on stage 1 test labels, if all you need to do is run for example a shell script (which you already had)
\n\nyou definitely can't change anything in the code (unless it's done automatically - like maybe a dynamic NN based on cross-validation etc.)
\n\nbut definitely would love to hear an official answer as well, for example if I would make the NN architecture based on command line parameters, would that be OK? based on past I would say yes, since it's not a change in code, and Kaggle can reproduce the same result when you tell them the exact command line you used etc.
", "votes": null, "replies": [ { "id": 291260, "author_name": "cpmpml", "author_url": "", "post_date": "03/05/2018 21:31:47", "content": "I like it better! We need to know.
", "votes": null, "replies": [] }, { "id": 294971, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "03/12/2018 22:49:14", "content": "I would not expect custom command line parameters to be allowed, you can't change hyperparameters and you can fit a lot of them into command line
", "votes": null, "replies": [] } ] }, { "id": 301042, "author_name": "dmytropoplavskiy", "author_url": "", "post_date": "03/22/2018 07:53:22", "content": "Hopefully we can rely on this to be fully applicable to the competition:
\n\n", "votes": null, "replies": [ { "id": 302228, "author_name": "wayne999", "author_url": "", "post_date": "03/23/2018 20:17:56", "content": "very helpful
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "291023": "Again, maybe a silly question.\n\nThe rules say that we must upload our model before stage1 ends. Does it mean we have to use that model for stage 2, or can we further train/improve our model with stage 2 train data?", "291050": "April 11, 2018 - Stage one deadline and stage two data release. Your model must be finalized and uploaded to Kaggle by this deadline - it is not clear what is uploaded? csv? model weights in docker?", "291072": "According to what we've learned from ncfm (fishy) competition:\n\n 1. You can't retrain your model in stage 2. Of course, fixes to handle and read the new data may be OK.\n 2. You must upload your fully pipeline (code, weights, models, etc.) by April 11: More -> Team -> Upload\n 3. If you don't want to upload your whole pipeline (because of size), zip your file and compute its sha256 hash. Upload that hash.\n\nYou have to freeze your pipeline and do a blind-shot on stage twos' data. If you want to be prize eligible, your have to verify that your uploaded pipeline produces the prize-submission. Of course, if you know you are not going to win a prize, you can upload bullshit and retrain on the data to get a good (undeserved!) rank.\n\nPlease note: If you do not upload a stage2 submission, you will be removed from the leaderboard. For example, if you are on rank 300 out of 2000 in stage1 and only the first 300 submit a stage2 submission and you stay on 300, you will be ranked last and get less points. Hence your stage1 ranking effort was for nothing. Because of this, in ncfm they decide to keep the stage1 submitter in the ranking, after the stage2 submitter.", "291083": "Thanks, that was my understanding too, but it is not what the rules say. Clarification from Kaggle would be most welcome. My strategy going forward depends on the possibility to retrain or not in stage2.\n\nI just checked the rules for ncfm, they were not explicit either. What made it explicit was a [last minute post][1] (2 days before stage1 ends) which probably took many people by surprise if I judge by the number of down votes.\n\nIt is because I don't want to have a similar surprise that I am asking the question now.\n\n\n [1]: https://www.kaggle.com/c/the-nature-conservancy-fisheries-monitoring/discussion/31103", "291086": "Retrain on what? There is no stage2 train data just a test set. Or do you want to train on stage1 test data by using the published test labels or unsupervised on stage2 test?", "291091": "masks for stage1 test will be disclosed.", "291118": "As far as I understand, we can't use stage1 test labels to re-train our models, the models must be final at the upload stage. I was sure it was answered *somewhere* but I can't find the source, the best I found is this https://www.kaggle.com/c/data-science-bowl-2018/discussion/48136#282212\n\nI hope someone from kaggle clarifies this or points to a place where it is clarified.", "291139": "Again, this is my understanding too, but it is NOT part of the rules.", "291219": "well, I have bit different understanding - you can do anything, as long as it's automated (i.e. you don't introduce new code)\n\ni.e. you can retrain on stage 1 test labels, if all you need to do is run for example a shell script (which you already had)\n\nyou definitely can't change anything in the code (unless it's done automatically - like maybe a dynamic NN based on cross-validation etc.)\n\nbut definitely would love to hear an official answer as well, for example if I would make the NN architecture based on command line parameters, would that be OK? based on past I would say yes, since it's not a change in code, and Kaggle can reproduce the same result when you tell them the exact command line you used etc.", "291260": "I like it better! We need to know.", "294971": "I would not expect custom command line parameters to be allowed, you can't change hyperparameters and you can fit a lot of them into command line", "301042": "Hopefully we can rely on this to be fully applicable to the competition:\n\n [Two-Stage Competition FAQ][1]\n\n\n [1]: https://www.kaggle.com/two-stage-faq \"https://www.kaggle.com/two-stage-faq\"", "302228": "very helpful" }, "source": "meta" }