{
  "id": 15596,
  "title": "How many real data points in the test set?",
  "url": "/competitions/flavours-of-physics/discussion/15596",
  "author_name": "",
  "post_date": "2015-07-28T15:05:17.540Z",
  "votes": 1,
  "comment_count": 2,
  "views": 611,
  "content": "<p>As a I understand, the test set is a mix of test data and data from the control channel. What are the respective proportions?</p>\n\n<p>This information would be very useful to evaluate the reliability of the public leaderboard score. The variance of the weighted ROC AUC depends on how many actual data points it was computed on. I'm seeing significantly lower scores on the LB compared to local cross-validation, so I'm curious.</p>",
  "messages": [
    {
      "id": "87310",
      "postDate": "07/28/2015 15:05:17",
      "content": "<p>As a I understand, the test set is a mix of test data and data from the control channel. What are the respective proportions?</p>\n\n<p>This information would be very useful to evaluate the reliability of the public leaderboard score. The variance of the weighted ROC AUC depends on how many actual data points it was computed on. I'm seeing significantly lower scores on the LB compared to local cross-validation, so I'm curious.</p>",
      "rawMarkdown": "As a I understand, the test set is a mix of test data and data from the control channel. What are the respective proportions?\r\n\r\nThis information would be very useful to evaluate the reliability of the public leaderboard score. The variance of the weighted ROC AUC depends on how many actual data points it was computed on. I'm seeing significantly lower scores on the LB compared to local cross-validation, so I'm curious.",
      "votes": null
    },
    {
      "id": "87394",
      "postDate": "07/29/2015 09:01:38",
      "content": "<p>Hey,</p>\n\n<p>We tested the stability between LB and PB. The variance of different models trained with different hyperparameters and subsampling is about 10^-4.</p>\n\n<p>Cheers,\nMarcin</p>",
      "rawMarkdown": "Hey,\r\n\r\nWe tested the stability between LB and PB. The variance of different models trained with different hyperparameters and subsampling is about 10^-4.\r\n\r\nCheers,\r\nMarcin",
      "votes": null
    },
    {
      "id": "87710",
      "postDate": "07/31/2015 03:30:06",
      "content": "<p>My LB scores are consistently 5 * 10e-4 lower than CV scores, with 1 * 10e-4 variance in the difference. Which is to say that my CV scores are around 0.9900 and my LB scores are around 0.9850, with a number of different NNs (no ensembles).</p>",
      "rawMarkdown": "My LB scores are consistently 5 * 10e-4 lower than CV scores, with 1 * 10e-4 variance in the difference. Which is to say that my CV scores are around 0.9900 and my LB scores are around 0.9850, with a number of different NNs (no ensembles).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 87394,
      "author_name": "albercik1905",
      "author_url": "",
      "post_date": "07/29/2015 09:01:38",
      "content": "<p>Hey,</p>\n\n<p>We tested the stability between LB and PB. The variance of different models trained with different hyperparameters and subsampling is about 10^-4.</p>\n\n<p>Cheers,\nMarcin</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 87710,
      "author_name": "fchollet",
      "author_url": "",
      "post_date": "07/31/2015 03:30:06",
      "content": "<p>My LB scores are consistently 5 * 10e-4 lower than CV scores, with 1 * 10e-4 variance in the difference. Which is to say that my CV scores are around 0.9900 and my LB scores are around 0.9850, with a number of different NNs (no ensembles).</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "87310": "As a I understand, the test set is a mix of test data and data from the control channel. What are the respective proportions?\r\n\r\nThis information would be very useful to evaluate the reliability of the public leaderboard score. The variance of the weighted ROC AUC depends on how many actual data points it was computed on. I'm seeing significantly lower scores on the LB compared to local cross-validation, so I'm curious.",
    "87394": "Hey,\r\n\r\nWe tested the stability between LB and PB. The variance of different models trained with different hyperparameters and subsampling is about 10^-4.\r\n\r\nCheers,\r\nMarcin",
    "87710": "My LB scores are consistently 5 * 10e-4 lower than CV scores, with 1 * 10e-4 variance in the difference. Which is to say that my CV scores are around 0.9900 and my LB scores are around 0.9850, with a number of different NNs (no ensembles)."
  },
  "source": "meta"
}