{
  "id": 122584,
  "title": "Choice of final submissions based on public LB score",
  "url": "/competitions/ashrae-energy-prediction/discussion/122584",
  "author_name": "",
  "post_date": "2019-12-21T06:47:03.381330500Z",
  "votes": 9,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Now the host is spending a ton of time scoring all submissions for all participants. It seems like the major reason the host is doing this is because some participants didn't pick their final submissions, so the public LB needs to be re-calculated in order for the host to pick the submissions based on the \"new\" public LB without leaked data.</p>\n\n<p>If so, then this is not fair for other participants.</p>\n\n<p>Because none of us those know what data are actually leaked data. If we have a chance to re-pick the best submissions based on the new public LB, a lot of us would do so.  Because we have no idea what building meters are actually optimized when hyperparameters are tuned. If the final submission is not chosen before competition ends, the system will pick a submission which fits the actual test data better, this would be a huge benefit.</p>\n\n<p>This is based on the assumption that the \"new\" public LB has no leaked data. But if the new public LB contains a higher percentage of leaked data then the previous one -</p>\n\n<p>then it would be really bad for those who didn't pick their submission before the game ends. Because the system will pick the submission which is biased towards the leaked data.</p>\n\n<p>Whichever the case is, it will be unfair. So there is no need for the host to re-score everything. Just pick the submissions based on the old public LB (if the user didn't pick it manually), and then generate the new score on private LB picked by the user, which shouldn't take too long. I hope this is making sense?</p>",
  "messages": [
    {
      "id": "699950",
      "postDate": "12/21/2019 06:47:03",
      "content": "<p>Now the host is spending a ton of time scoring all submissions for all participants. It seems like the major reason the host is doing this is because some participants didn't pick their final submissions, so the public LB needs to be re-calculated in order for the host to pick the submissions based on the \"new\" public LB without leaked data.</p>\n\n<p>If so, then this is not fair for other participants.</p>\n\n<p>Because none of us those know what data are actually leaked data. If we have a chance to re-pick the best submissions based on the new public LB, a lot of us would do so.  Because we have no idea what building meters are actually optimized when hyperparameters are tuned. If the final submission is not chosen before competition ends, the system will pick a submission which fits the actual test data better, this would be a huge benefit.</p>\n\n<p>This is based on the assumption that the \"new\" public LB has no leaked data. But if the new public LB contains a higher percentage of leaked data then the previous one -</p>\n\n<p>then it would be really bad for those who didn't pick their submission before the game ends. Because the system will pick the submission which is biased towards the leaked data.</p>\n\n<p>Whichever the case is, it will be unfair. So there is no need for the host to re-score everything. Just pick the submissions based on the old public LB (if the user didn't pick it manually), and then generate the new score on private LB picked by the user, which shouldn't take too long. I hope this is making sense?</p>",
      "rawMarkdown": "Now the host is spending a ton of time scoring all submissions for all participants. It seems like the major reason the host is doing this is because some participants didn't pick their final submissions, so the public LB needs to be re-calculated in order for the host to pick the submissions based on the \"new\" public LB without leaked data.\n\nIf so, then this is not fair for other participants.\n\nBecause none of us those know what data are actually leaked data. If we have a chance to re-pick the best submissions based on the new public LB, a lot of us would do so.  Because we have no idea what building meters are actually optimized when hyperparameters are tuned. If the final submission is not chosen before competition ends, the system will pick a submission which fits the actual test data better, this would be a huge benefit.\n\nThis is based on the assumption that the \"new\" public LB has no leaked data. But if the new public LB contains a higher percentage of leaked data then the previous one -\n\nthen it would be really bad for those who didn't pick their submission before the game ends. Because the system will pick the submission which is biased towards the leaked data.\n\nWhichever the case is, it will be unfair. So there is no need for the host to re-score everything. Just pick the submissions based on the old public LB (if the user didn't pick it manually), and then generate the new score on private LB picked by the user, which shouldn't take too long. I hope this is making sense?",
      "votes": null
    },
    {
      "id": "699965",
      "postDate": "12/21/2019 07:37:07",
      "content": "<p>I agree with you but where is it mentioned that the rescoring is done to select best two submissions?\nI think the rescoring of all submissions is done only for the new private LB for display and analysis and the best two submissions will be based on the original public LB itself.</p>",
      "rawMarkdown": "I agree with you but where is it mentioned that the rescoring is done to select best two submissions?\nI think the rescoring of all submissions is done only for the new private LB for display and analysis and the best two submissions will be based on the original public LB itself.",
      "votes": null
    },
    {
      "id": "699974",
      "postDate": "12/21/2019 07:51:14",
      "content": "<p>Hi Vopani - I do hope like you said it's scored based on the original public LB. But after reading some forum posts I feel like some reminder to the host and other participants may help.</p>",
      "rawMarkdown": "Hi Vopani - I do hope like you said it's scored based on the original public LB. But after reading some forum posts I feel like some reminder to the host and other participants may help.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 699965,
      "author_name": "rohanrao",
      "author_url": "",
      "post_date": "12/21/2019 07:37:07",
      "content": "<p>I agree with you but where is it mentioned that the rescoring is done to select best two submissions?\nI think the rescoring of all submissions is done only for the new private LB for display and analysis and the best two submissions will be based on the original public LB itself.</p>",
      "votes": null,
      "replies": [
        {
          "id": 699974,
          "author_name": "shiutangli",
          "author_url": "",
          "post_date": "12/21/2019 07:51:14",
          "content": "<p>Hi Vopani - I do hope like you said it's scored based on the original public LB. But after reading some forum posts I feel like some reminder to the host and other participants may help.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "699950": "Now the host is spending a ton of time scoring all submissions for all participants. It seems like the major reason the host is doing this is because some participants didn't pick their final submissions, so the public LB needs to be re-calculated in order for the host to pick the submissions based on the \"new\" public LB without leaked data.\n\nIf so, then this is not fair for other participants.\n\nBecause none of us those know what data are actually leaked data. If we have a chance to re-pick the best submissions based on the new public LB, a lot of us would do so.  Because we have no idea what building meters are actually optimized when hyperparameters are tuned. If the final submission is not chosen before competition ends, the system will pick a submission which fits the actual test data better, this would be a huge benefit.\n\nThis is based on the assumption that the \"new\" public LB has no leaked data. But if the new public LB contains a higher percentage of leaked data then the previous one -\n\nthen it would be really bad for those who didn't pick their submission before the game ends. Because the system will pick the submission which is biased towards the leaked data.\n\nWhichever the case is, it will be unfair. So there is no need for the host to re-score everything. Just pick the submissions based on the old public LB (if the user didn't pick it manually), and then generate the new score on private LB picked by the user, which shouldn't take too long. I hope this is making sense?",
    "699965": "I agree with you but where is it mentioned that the rescoring is done to select best two submissions?\nI think the rescoring of all submissions is done only for the new private LB for display and analysis and the best two submissions will be based on the original public LB itself.",
    "699974": "Hi Vopani - I do hope like you said it's scored based on the original public LB. But after reading some forum posts I feel like some reminder to the host and other participants may help."
  },
  "source": "meta"
}