{
  "id": 5458,
  "title": "Challenge ended",
  "url": "/competitions/cause-effect-pairs/discussion/5458",
  "author_name": "",
  "post_date": "2013-08-20T05:55:53.543Z",
  "votes": null,
  "comment_count": 23,
  "views": 7057,
  "content": "<p>The challenge ended today. Thank you all for your participation. I will contact you individually if there is a problem with your submitted software.</p>\n<p>The rest of the schedule is:</p>\n<p>Friday, August 23, 2013: Release of test data decryption key.</p>\n<p>Friday, August 30, 2013: Deadline for submitting results on test data.</p>\n<p>Friday, September 6, 2013: Deadline for the winners to make their source code publicly available.</p>\n<p>Wednesday October 9, 2013:&nbsp;JMLR proceedings paper and abstract submission deadline.</p>\n<p>Wednesday October 23, 2013:&nbsp;Paper notification of acceptance.</p>\n<p>December 9 or 10, 2013:&nbsp;NIPS 2013, Tahoe, Nevada, USA. Workshop to be announced.</p>",
  "messages": [
    {
      "id": "29010",
      "postDate": "08/20/2013 05:55:53",
      "content": "<p>The challenge ended today. Thank you all for your participation. I will contact you individually if there is a problem with your submitted software.</p>\n<p>The rest of the schedule is:</p>\n<p>Friday, August 23, 2013: Release of test data decryption key.</p>\n<p>Friday, August 30, 2013: Deadline for submitting results on test data.</p>\n<p>Friday, September 6, 2013: Deadline for the winners to make their source code publicly available.</p>\n<p>Wednesday October 9, 2013:&nbsp;JMLR proceedings paper and abstract submission deadline.</p>\n<p>Wednesday October 23, 2013:&nbsp;Paper notification of acceptance.</p>\n<p>December 9 or 10, 2013:&nbsp;NIPS 2013, Tahoe, Nevada, USA. Workshop to be announced.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29022",
      "postDate": "08/20/2013 09:00:05",
      "content": "<p>I think the quality of paper is correlated with the ranking onboard ?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29031",
      "postDate": "08/20/2013 10:02:42",
      "content": "<p>What is the purpose of the test data submission task&nbsp;next week? I was under the impression that the leaderboard score was based on a subset of our submission and that the final ranking would be done today on the full evaluation of selected 3 best submissions?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29032",
      "postDate": "08/20/2013 10:16:35",
      "content": "<p>Will the Valid labels be released?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29033",
      "postDate": "08/20/2013 10:19:32",
      "content": "<p>if the valid labels are released, are we allowed to retrain the model on the new data?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29039",
      "postDate": "08/20/2013 11:52:26",
      "content": "<p>[quote=Sitmo;29031]</p>\n<p>What is the purpose of the test data submission task&nbsp;next week? I was under the impression that the leaderboard score was based on a subset of our submission and that the final ranking would be done today on the full evaluation of selected 3 best submissions?</p>\n<p>[/quote]</p>\n<p>&nbsp;</p>\n<p>No, it's a 2 stage competition. You will now have to make submissions on the test set using your model&nbsp; (if you uploaded one) I assume we retrain using the exact model when we have the valid labels</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29042",
      "postDate": "08/20/2013 12:00:02",
      "content": "<p>[quote=Domcastro;29039]</p>\n<p>[quote=Sitmo;29031]</p>\n<p>What is the purpose of the test data submission task&nbsp;next week? I was under the impression that the leaderboard score was based on a subset of our submission and that the final ranking would be done today on the full evaluation of selected 3 best submissions?</p>\n<p>[/quote]</p>\n<p>&nbsp;</p>\n<p>No, it's a 2 stage competition. You will now have to make submissions on the test set using your model&nbsp; (if you uploaded one) I assume we retrain using the exact model when we have the valid labels</p>\n<p>[/quote]</p>\n<p>I see, ...so it's *not* finished?</p>\n<p>But why did I have to select 3 submissions last night before the deadline? What will happen with those?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29043",
      "postDate": "08/20/2013 12:08:28",
      "content": "<p>Sitmo: My guess is the 3 selections you mention are just an unused holdover from the &quot;standard&quot; Kaggle competition format, and won't have any effect in this one.</p>\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29044",
      "postDate": "08/20/2013 12:13:43",
      "content": "<p>Domcastro: My expectation is that we will not have any opportunity to retrain the models, because our training code was not a required element of the model uploads. .&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29045",
      "postDate": "08/20/2013 12:23:04",
      "content": "<p>I just found&nbsp;the new post done yesterday&nbsp; where she asks us to upload (up to 3) &nbsp;pre-trained models which need to be able to work on new data. The rules feel like a moving target that doesn't stop moving.</p>\n<p>I also remember that&nbsp; it was stated that submitting models was only needed if you wanted a money- price, and that &nbsp;it would be independent of the final ranking in the Kaggle competition.</p>\n<p>It looks like us working yesterday till &nbsp;the very last minute on making a good final submission (and selecting a top 3) was completely useless!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29046",
      "postDate": "08/20/2013 12:42:47",
      "content": "<p>My guess :</p>\n<p>In order to claim for the prize, you have to upload your feature extraction code and the training code, make them publicitly available after the competition, and achieves Top 3 within those who do the same.</p>\n<p>In ranking, you only need to do the best using the test datasets use whatever the method you are using.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29047",
      "postDate": "08/20/2013 12:58:43",
      "content": "<p>liubenyuan: that sounds logical. If that's correct, then the only thing that puzzles me is yesterdays deadline. For ranking I can tweak my model up untill the &quot;test data submission&quot;-deadline. Yesterday's deadline is then probably the deadline for the models of which&nbsp;&nbsp;the top 3 performance when run on the test data&nbsp; will win a price.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29049",
      "postDate": "08/20/2013 13:02:51",
      "content": "<p>if you submit a model and go up the leaderboard once the test set is available, they will verify it with the model you submitted. If you dont hamper the positions being held by top contenders (after the test set releases), you will keep your rank and thus the kaggle points without any prize. However, according to the Kaggle rules, your model can be challenged by any other competitor in that case (as this is a two stage competition).&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29050",
      "postDate": "08/20/2013 13:10:53",
      "content": "<p>Thanks Abhishek, that's clear too, excellent. &nbsp;We were tweaking our model (and didn't upload it) up to the last minute because we were focusing on the forecast submission. We were under the impression that for ranking we didn't need to worry about software uploads (only if we wanted to be eligable for a price).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29055",
      "postDate": "08/20/2013 13:55:10",
      "content": "<p>Isabelle</p>\n<p>It would great if you can clarify those issues, because they still seem confusing!</p>\n<p>1. To get Kaggle ranking on this competition do I need to submit result on test data, assuming I have no hope of getting the prize? Or was my ranking on validation data my final ranking on this competition?</p>\n<p>2. Again assuming I'm not getting the prize anyways can I continue to tweak the models for final submission on test data?</p>\n<p>3. How will percentile rank on this combination (top 10% , top 25%, etc.) be affected if only a certain percentage of participants submit results on test data? In other words I may be in 10% now, but if only 50 teams submit test data results, how will percentiles be computed?</p>\n<p>Thanks!</p>\n<p>Amit</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29058",
      "postDate": "08/20/2013 14:07:25",
      "content": "<p>Let me talk to Kaggle and get back to you on this.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29061",
      "postDate": "08/20/2013 14:45:40",
      "content": "<p>@Dirk Gently:<br>While Isabelle is looking for the definitive answer, you will likely want to submit something, and based on history of competitions like this one, your final ranking will depend solely on the final submission.</p>\n<p>&nbsp;</p>\n<p>Many competitions this year have been two-phase, and I think the most recent where we can see the structure is the KDD Cup. http://www.kaggle.com/c/kdd-cup-2013-author-paper-identification-challenge/leaderboard</p>\n<p><br>Notice everybody tied for 74th with a 0.0 score. Those are the people who did not upload a final submission. You can see from a sample profile that is the ranking that everyone gets:<br>http://www.kaggle.com/users/95303/example-account</p>\n<p>The GE Flight competition caused a bit of a stir because a skewed submission managed to score worse than those who did not submit. The response indicated Kaggle was looking into ranking those who did not submit by their public leaderboard ranking:<br>http://www.gequest.com/c/flight/forums/t/4185/sanity-check-in-two-steps-competitions/22301#post22301</p>\n<p>But based on the KDD Cup leaderboard it seems that has not been rolled out yet. So it would seem in your best interest, even if you have no chance at the top (like myself), to go ahead and make 1 (or 3?) final submissions when the key is unlocked later this week.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29063",
      "postDate": "08/20/2013 15:06:47",
      "content": "<p>I agree with mlandry,</p>\n<p>In KDD-cup,</p>\n<ul>\n<li>All participants were required to submit predictions on the final test dataset using their submitted model.</li>\n<li>The validation answers were released and the participants were allowed to retrain their model on validation set. Things like developing new features or adjusting parameters, were, of course, not allowed.</li>\n</ul>\n<p>I guess this competition should proceed similarly.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29077",
      "postDate": "08/20/2013 17:10:38",
      "content": "<p>I did not sent my model because of: &quot;The top ranking participants should start uploading their code.&quot;, and it is obvious, that my result is not at the top. So, will I get nothing at the end??? It is unfair...</p>\n<p>Can I submit my model now, to get scores?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29088",
      "postDate": "08/20/2013 18:07:36",
      "content": "<p>sbachish: this is my understanding: I think we are mostly fine: &nbsp;in the &nbsp;&quot;Code submission v.s. final ranking thread&quot; it is explained that</p>\n<p style=\"padding-left: 30px\">&quot;...you do not need to submit code if you don't not want to claim a prize and it is OK to submit final result on test data without submitting code. Verifications will be carried out only to the extent that they might affect the ranking of the three top ranking participants and the order on the leaderboard will prevail unless the ranking of the three top ranking participants is affected, in which case, the scores obtained by software verification will count.&quot;</p>\n<p>&nbsp;</p>\n<p>So the only thing it seems we need to do next is <strong>make another prediction for &nbsp;the upcoming test set next week</strong>. The challenge has not ended but is more like in some sort of limbo state.&nbsp;</p>\n<p>If you wanted to take part in the top 3 price competition then you should have mailed up to 3 models to Isabelle yesterday like she communcated yesterday (and which I failed to read because I thought the rules of the competition would not get all sorts of addendums)&nbsp;</p>\n<p>Yesterdays ranking is apparently irrelevant for any purpose, and the top 3 submissions are not used either. (and the SUP1 dataset if wrong too, and there was also a huge bug in the scoring function, and the deadline was moved half way through the competition etc etc etc)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29180",
      "postDate": "08/21/2013 18:36:52",
      "content": "<p>The final ranking of participants will be done using predictions made by the participants on&nbsp;<strong>test data.</strong> The test data decryption key will be released August 23, 2013. The participants will have to return results on test data before August 30, 2013.&nbsp;Participants not submitting code&nbsp;<strong>can&nbsp;</strong>be part of this.&nbsp;The ranking on test data is what will count for Kaggle points.</p>\n<p>The verification requirements will concern the top 3 ranking participants on test data who want to claim their prize and will follow the rule: &quot;Winning solutions need to be made available under a popular OSI-approved license in order to be eligible for recognition and prize money.&quot; The deadline for this is September 6, 2013. The organizers will verify that the code runs properly and can reproduce results. To that end, the organizers are helping the top ranking participants who already submitted their code to prepare for this by providing them feed-back on code submitted already.</p>\n<p>If you are not one of the prize winners, your solution will not be verified, but still count for Kaggle points.&nbsp;I will let Kaggle support answer about the percentiles.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29181",
      "postDate": "08/21/2013 18:44:31",
      "content": "<p>[quote=Domcastro;29032]</p>\n<p>Will the Valid labels be released?</p>\n<p>[/quote]</p>\n<p>No</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29319",
      "postDate": "08/23/2013 17:06:09",
      "content": "<p>[quote=Isabelle;29181]</p>\n<p>[quote=Domcastro;29032]</p>\n<p>Will the Valid labels be released?</p>\n<p>[/quote]</p>\n<p>No</p>\n<p>[/quote]</p>\n<p>&nbsp;</p>\n<p>Hi,&nbsp;Isabelle</p>\n<p>Will the labels of Valid and Test be released **after** the competition ? And moreover, will the labels indicate that the x-y pair is synthetic or real be released ?&nbsp;</p>\n<p>Thanks !</p>\n<p>Liu</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29333",
      "postDate": "08/23/2013 18:27:25",
      "content": "<p>[quote=liubenyuan;29319]&nbsp;</p>\n<p>Hi,&nbsp;Isabelle</p>\n<p>Will the labels of Valid and Test be released **after** the competition ? And moreover, will the labels indicate that the x-y pair is synthetic or real be released ?&nbsp;</p>\n<p>Thanks !</p>\n<p>Liu</p>\n<p>[/quote]</p>\n<p>At some point we might. We will keep you informed.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 29022,
      "author_name": "liubenyuan",
      "author_url": "",
      "post_date": "08/20/2013 09:00:05",
      "content": "<p>I think the quality of paper is correlated with the ranking onboard ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29031,
      "author_name": "sitmocom",
      "author_url": "",
      "post_date": "08/20/2013 10:02:42",
      "content": "<p>What is the purpose of the test data submission task&nbsp;next week? I was under the impression that the leaderboard score was based on a subset of our submission and that the final ranking would be done today on the full evaluation of selected 3 best submissions?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29032,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "08/20/2013 10:16:35",
      "content": "<p>Will the Valid labels be released?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29033,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "08/20/2013 10:19:32",
      "content": "<p>if the valid labels are released, are we allowed to retrain the model on the new data?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29039,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "08/20/2013 11:52:26",
      "content": "<p>[quote=Sitmo;29031]</p>\n<p>What is the purpose of the test data submission task&nbsp;next week? I was under the impression that the leaderboard score was based on a subset of our submission and that the final ranking would be done today on the full evaluation of selected 3 best submissions?</p>\n<p>[/quote]</p>\n<p>&nbsp;</p>\n<p>No, it's a 2 stage competition. You will now have to make submissions on the test set using your model&nbsp; (if you uploaded one) I assume we retrain using the exact model when we have the valid labels</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29042,
      "author_name": "sitmocom",
      "author_url": "",
      "post_date": "08/20/2013 12:00:02",
      "content": "<p>[quote=Domcastro;29039]</p>\n<p>[quote=Sitmo;29031]</p>\n<p>What is the purpose of the test data submission task&nbsp;next week? I was under the impression that the leaderboard score was based on a subset of our submission and that the final ranking would be done today on the full evaluation of selected 3 best submissions?</p>\n<p>[/quote]</p>\n<p>&nbsp;</p>\n<p>No, it's a 2 stage competition. You will now have to make submissions on the test set using your model&nbsp; (if you uploaded one) I assume we retrain using the exact model when we have the valid labels</p>\n<p>[/quote]</p>\n<p>I see, ...so it's *not* finished?</p>\n<p>But why did I have to select 3 submissions last night before the deadline? What will happen with those?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29043,
      "author_name": "bcragin",
      "author_url": "",
      "post_date": "08/20/2013 12:08:28",
      "content": "<p>Sitmo: My guess is the 3 selections you mention are just an unused holdover from the &quot;standard&quot; Kaggle competition format, and won't have any effect in this one.</p>\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29044,
      "author_name": "bcragin",
      "author_url": "",
      "post_date": "08/20/2013 12:13:43",
      "content": "<p>Domcastro: My expectation is that we will not have any opportunity to retrain the models, because our training code was not a required element of the model uploads. .&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29045,
      "author_name": "sitmocom",
      "author_url": "",
      "post_date": "08/20/2013 12:23:04",
      "content": "<p>I just found&nbsp;the new post done yesterday&nbsp; where she asks us to upload (up to 3) &nbsp;pre-trained models which need to be able to work on new data. The rules feel like a moving target that doesn't stop moving.</p>\n<p>I also remember that&nbsp; it was stated that submitting models was only needed if you wanted a money- price, and that &nbsp;it would be independent of the final ranking in the Kaggle competition.</p>\n<p>It looks like us working yesterday till &nbsp;the very last minute on making a good final submission (and selecting a top 3) was completely useless!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29046,
      "author_name": "liubenyuan",
      "author_url": "",
      "post_date": "08/20/2013 12:42:47",
      "content": "<p>My guess :</p>\n<p>In order to claim for the prize, you have to upload your feature extraction code and the training code, make them publicitly available after the competition, and achieves Top 3 within those who do the same.</p>\n<p>In ranking, you only need to do the best using the test datasets use whatever the method you are using.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29047,
      "author_name": "sitmocom",
      "author_url": "",
      "post_date": "08/20/2013 12:58:43",
      "content": "<p>liubenyuan: that sounds logical. If that's correct, then the only thing that puzzles me is yesterdays deadline. For ranking I can tweak my model up untill the &quot;test data submission&quot;-deadline. Yesterday's deadline is then probably the deadline for the models of which&nbsp;&nbsp;the top 3 performance when run on the test data&nbsp; will win a price.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29049,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "08/20/2013 13:02:51",
      "content": "<p>if you submit a model and go up the leaderboard once the test set is available, they will verify it with the model you submitted. If you dont hamper the positions being held by top contenders (after the test set releases), you will keep your rank and thus the kaggle points without any prize. However, according to the Kaggle rules, your model can be challenged by any other competitor in that case (as this is a two stage competition).&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29050,
      "author_name": "sitmocom",
      "author_url": "",
      "post_date": "08/20/2013 13:10:53",
      "content": "<p>Thanks Abhishek, that's clear too, excellent. &nbsp;We were tweaking our model (and didn't upload it) up to the last minute because we were focusing on the forecast submission. We were under the impression that for ranking we didn't need to worry about software uploads (only if we wanted to be eligable for a price).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29055,
      "author_name": "amitjuneja",
      "author_url": "",
      "post_date": "08/20/2013 13:55:10",
      "content": "<p>Isabelle</p>\n<p>It would great if you can clarify those issues, because they still seem confusing!</p>\n<p>1. To get Kaggle ranking on this competition do I need to submit result on test data, assuming I have no hope of getting the prize? Or was my ranking on validation data my final ranking on this competition?</p>\n<p>2. Again assuming I'm not getting the prize anyways can I continue to tweak the models for final submission on test data?</p>\n<p>3. How will percentile rank on this combination (top 10% , top 25%, etc.) be affected if only a certain percentage of participants submit results on test data? In other words I may be in 10% now, but if only 50 teams submit test data results, how will percentiles be computed?</p>\n<p>Thanks!</p>\n<p>Amit</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29058,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "08/20/2013 14:07:25",
      "content": "<p>Let me talk to Kaggle and get back to you on this.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29061,
      "author_name": "mlandry",
      "author_url": "",
      "post_date": "08/20/2013 14:45:40",
      "content": "<p>@Dirk Gently:<br>While Isabelle is looking for the definitive answer, you will likely want to submit something, and based on history of competitions like this one, your final ranking will depend solely on the final submission.</p>\n<p>&nbsp;</p>\n<p>Many competitions this year have been two-phase, and I think the most recent where we can see the structure is the KDD Cup. http://www.kaggle.com/c/kdd-cup-2013-author-paper-identification-challenge/leaderboard</p>\n<p><br>Notice everybody tied for 74th with a 0.0 score. Those are the people who did not upload a final submission. You can see from a sample profile that is the ranking that everyone gets:<br>http://www.kaggle.com/users/95303/example-account</p>\n<p>The GE Flight competition caused a bit of a stir because a skewed submission managed to score worse than those who did not submit. The response indicated Kaggle was looking into ranking those who did not submit by their public leaderboard ranking:<br>http://www.gequest.com/c/flight/forums/t/4185/sanity-check-in-two-steps-competitions/22301#post22301</p>\n<p>But based on the KDD Cup leaderboard it seems that has not been rolled out yet. So it would seem in your best interest, even if you have no chance at the top (like myself), to go ahead and make 1 (or 3?) final submissions when the key is unlocked later this week.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29063,
      "author_name": "sjuvekar",
      "author_url": "",
      "post_date": "08/20/2013 15:06:47",
      "content": "<p>I agree with mlandry,</p>\n<p>In KDD-cup,</p>\n<ul>\n<li>All participants were required to submit predictions on the final test dataset using their submitted model.</li>\n<li>The validation answers were released and the participants were allowed to retrain their model on validation set. Things like developing new features or adjusting parameters, were, of course, not allowed.</li>\n</ul>\n<p>I guess this competition should proceed similarly.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29077,
      "author_name": "sbachish",
      "author_url": "",
      "post_date": "08/20/2013 17:10:38",
      "content": "<p>I did not sent my model because of: &quot;The top ranking participants should start uploading their code.&quot;, and it is obvious, that my result is not at the top. So, will I get nothing at the end??? It is unfair...</p>\n<p>Can I submit my model now, to get scores?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29088,
      "author_name": "sitmocom",
      "author_url": "",
      "post_date": "08/20/2013 18:07:36",
      "content": "<p>sbachish: this is my understanding: I think we are mostly fine: &nbsp;in the &nbsp;&quot;Code submission v.s. final ranking thread&quot; it is explained that</p>\n<p style=\"padding-left: 30px\">&quot;...you do not need to submit code if you don't not want to claim a prize and it is OK to submit final result on test data without submitting code. Verifications will be carried out only to the extent that they might affect the ranking of the three top ranking participants and the order on the leaderboard will prevail unless the ranking of the three top ranking participants is affected, in which case, the scores obtained by software verification will count.&quot;</p>\n<p>&nbsp;</p>\n<p>So the only thing it seems we need to do next is <strong>make another prediction for &nbsp;the upcoming test set next week</strong>. The challenge has not ended but is more like in some sort of limbo state.&nbsp;</p>\n<p>If you wanted to take part in the top 3 price competition then you should have mailed up to 3 models to Isabelle yesterday like she communcated yesterday (and which I failed to read because I thought the rules of the competition would not get all sorts of addendums)&nbsp;</p>\n<p>Yesterdays ranking is apparently irrelevant for any purpose, and the top 3 submissions are not used either. (and the SUP1 dataset if wrong too, and there was also a huge bug in the scoring function, and the deadline was moved half way through the competition etc etc etc)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29180,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "08/21/2013 18:36:52",
      "content": "<p>The final ranking of participants will be done using predictions made by the participants on&nbsp;<strong>test data.</strong> The test data decryption key will be released August 23, 2013. The participants will have to return results on test data before August 30, 2013.&nbsp;Participants not submitting code&nbsp;<strong>can&nbsp;</strong>be part of this.&nbsp;The ranking on test data is what will count for Kaggle points.</p>\n<p>The verification requirements will concern the top 3 ranking participants on test data who want to claim their prize and will follow the rule: &quot;Winning solutions need to be made available under a popular OSI-approved license in order to be eligible for recognition and prize money.&quot; The deadline for this is September 6, 2013. The organizers will verify that the code runs properly and can reproduce results. To that end, the organizers are helping the top ranking participants who already submitted their code to prepare for this by providing them feed-back on code submitted already.</p>\n<p>If you are not one of the prize winners, your solution will not be verified, but still count for Kaggle points.&nbsp;I will let Kaggle support answer about the percentiles.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29181,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "08/21/2013 18:44:31",
      "content": "<p>[quote=Domcastro;29032]</p>\n<p>Will the Valid labels be released?</p>\n<p>[/quote]</p>\n<p>No</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29319,
      "author_name": "liubenyuan",
      "author_url": "",
      "post_date": "08/23/2013 17:06:09",
      "content": "<p>[quote=Isabelle;29181]</p>\n<p>[quote=Domcastro;29032]</p>\n<p>Will the Valid labels be released?</p>\n<p>[/quote]</p>\n<p>No</p>\n<p>[/quote]</p>\n<p>&nbsp;</p>\n<p>Hi,&nbsp;Isabelle</p>\n<p>Will the labels of Valid and Test be released **after** the competition ? And moreover, will the labels indicate that the x-y pair is synthetic or real be released ?&nbsp;</p>\n<p>Thanks !</p>\n<p>Liu</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 29333,
      "author_name": "iguyon",
      "author_url": "",
      "post_date": "08/23/2013 18:27:25",
      "content": "<p>[quote=liubenyuan;29319]&nbsp;</p>\n<p>Hi,&nbsp;Isabelle</p>\n<p>Will the labels of Valid and Test be released **after** the competition ? And moreover, will the labels indicate that the x-y pair is synthetic or real be released ?&nbsp;</p>\n<p>Thanks !</p>\n<p>Liu</p>\n<p>[/quote]</p>\n<p>At some point we might. We will keep you informed.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "29010": "",
    "29022": "",
    "29031": "",
    "29032": "",
    "29033": "",
    "29039": "",
    "29042": "",
    "29043": "",
    "29044": "",
    "29045": "",
    "29046": "",
    "29047": "",
    "29049": "",
    "29050": "",
    "29055": "",
    "29058": "",
    "29061": "",
    "29063": "",
    "29077": "",
    "29088": "",
    "29180": "",
    "29181": "",
    "29319": "",
    "29333": ""
  },
  "source": "meta"
}