{
  "id": 336774,
  "title": "Some LB Probing Results",
  "url": "/competitions/AI4Code/discussion/336774",
  "author_name": "",
  "post_date": "2022-07-12T22:38:03.596252700Z",
  "votes": 6,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I am getting a pretty big gap between my CV and LB so I decided to run some LB probes. I found that the test notebooks have a lot more notebooks with high cell counts. Depending on your approach, this might be a positive or a negative thing. I wonder what is going to the distribution in the private test data? Assuming time consistency, it's probably closer to test data than to training data.</p>\n<p>The number of Jupyter Notebooks with more than <strong>128</strong> cells:<br>\n<strong>At least</strong> 16% in test data and 3.84% in training data. </p>\n<p>The number of Jupyter Notebooks with more than <strong>64</strong> cells:<br>\n<strong>At least</strong> 30% in test data and 19.5% in training data. </p>\n<p>The number of Jupyter Notebooks with more than <strong>32</strong> cells:<br>\n<strong>At least</strong> 40% in test data and 52.1% in training data. </p>\n<p>Note that these are lower bounds.</p>",
  "messages": [
    {
      "id": "1853468",
      "postDate": "07/12/2022 22:38:03",
      "content": "<p>I am getting a pretty big gap between my CV and LB so I decided to run some LB probes. I found that the test notebooks have a lot more notebooks with high cell counts. Depending on your approach, this might be a positive or a negative thing. I wonder what is going to the distribution in the private test data? Assuming time consistency, it's probably closer to test data than to training data.</p>\n<p>The number of Jupyter Notebooks with more than <strong>128</strong> cells:<br>\n<strong>At least</strong> 16% in test data and 3.84% in training data. </p>\n<p>The number of Jupyter Notebooks with more than <strong>64</strong> cells:<br>\n<strong>At least</strong> 30% in test data and 19.5% in training data. </p>\n<p>The number of Jupyter Notebooks with more than <strong>32</strong> cells:<br>\n<strong>At least</strong> 40% in test data and 52.1% in training data. </p>\n<p>Note that these are lower bounds.</p>",
      "rawMarkdown": "I am getting a pretty big gap between my CV and LB so I decided to run some LB probes. I found that the test notebooks have a lot more notebooks with high cell counts. Depending on your approach, this might be a positive or a negative thing. I wonder what is going to the distribution in the private test data? Assuming time consistency, it's probably closer to test data than to training data.\n\nThe number of Jupyter Notebooks with more than __128__ cells:\n__At least__ 16% in test data and 3.84% in training data. \n\nThe number of Jupyter Notebooks with more than __64__ cells:\n__At least__ 30% in test data and 19.5% in training data. \n\nThe number of Jupyter Notebooks with more than __32__ cells:\n__At least__ 40% in test data and 52.1% in training data. \n\nNote that these are lower bounds.",
      "votes": null
    },
    {
      "id": "1855868",
      "postDate": "07/15/2022 01:53:04",
      "content": "<p>Thanks for sharing your findings!<br>\nInteresting! </p>",
      "rawMarkdown": "Thanks for sharing your findings!\nInteresting!",
      "votes": null
    },
    {
      "id": "1864095",
      "postDate": "07/20/2022 18:15:03",
      "content": "<p>I don't know how it can help, because the private leaderboard will be based on very new data, so this LB probing is useless.</p>",
      "rawMarkdown": "I don't know how it can help, because the private leaderboard will be based on very new data, so this LB probing is useless.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1855868,
      "author_name": "thedevastator",
      "author_url": "",
      "post_date": "07/15/2022 01:53:04",
      "content": "<p>Thanks for sharing your findings!<br>\nInteresting! </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1864095,
      "author_name": "vad13irt",
      "author_url": "",
      "post_date": "07/20/2022 18:15:03",
      "content": "<p>I don't know how it can help, because the private leaderboard will be based on very new data, so this LB probing is useless.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1853468": "I am getting a pretty big gap between my CV and LB so I decided to run some LB probes. I found that the test notebooks have a lot more notebooks with high cell counts. Depending on your approach, this might be a positive or a negative thing. I wonder what is going to the distribution in the private test data? Assuming time consistency, it's probably closer to test data than to training data.\n\nThe number of Jupyter Notebooks with more than __128__ cells:\n__At least__ 16% in test data and 3.84% in training data. \n\nThe number of Jupyter Notebooks with more than __64__ cells:\n__At least__ 30% in test data and 19.5% in training data. \n\nThe number of Jupyter Notebooks with more than __32__ cells:\n__At least__ 40% in test data and 52.1% in training data. \n\nNote that these are lower bounds.",
    "1855868": "Thanks for sharing your findings!\nInteresting!",
    "1864095": "I don't know how it can help, because the private leaderboard will be based on very new data, so this LB probing is useless."
  },
  "source": "meta"
}