{
  "id": 132152,
  "title": "Best way to make sure that your score is not due to leak.",
  "url": "/competitions/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/132152",
  "author_name": "",
  "post_date": "2020-02-24T14:41:37.625253400Z",
  "votes": 3,
  "comment_count": 3,
  "views": 0,
  "content": "<p>As pointed out in one of the posts,if your score is less than 0.46 it should be probably due to some leak and this won't hold in the final test set. So what are the methods to prevent this leak?\nI tried removing the data from 2015 onwards from the training set. Are there any other methods?</p>",
  "messages": [
    {
      "id": "755186",
      "postDate": "02/24/2020 14:41:37",
      "content": "<p>As pointed out in one of the posts,if your score is less than 0.46 it should be probably due to some leak and this won't hold in the final test set. So what are the methods to prevent this leak?\nI tried removing the data from 2015 onwards from the training set. Are there any other methods?</p>",
      "rawMarkdown": "As pointed out in one of the posts,if your score is less than 0.46 it should be probably due to some leak and this won't hold in the final test set. So what are the methods to prevent this leak?\nI tried removing the data from 2015 onwards from the training set. Are there any other methods?",
      "votes": null
    },
    {
      "id": "755255",
      "postDate": "02/24/2020 15:49:19",
      "content": "<p>There is obviously some data that leaks the outcome of the game, e.g. using stats from a game to predict its winner (shooting %, reb's, ast's or even the score itself if you really screw up)</p>",
      "rawMarkdown": "There is obviously some data that leaks the outcome of the game, e.g. using stats from a game to predict its winner (shooting %, reb's, ast's or even the score itself if you really screw up)",
      "votes": null
    },
    {
      "id": "755304",
      "postDate": "02/24/2020 16:47:38",
      "content": "<p>.46 would be an average score between 2015-2019. It is very possible to get scores below these numbers for one or maybe a few years but - up to this point - I have not been able to get an average score below .46. \nIf you want to prevent data leaks:\n-Don't use data from KenPom's website as it includes tournament info from the previous seasons. \n-Use only data from prior seasons to model and then predict on current season i.e. use data from 1985-2018 seasons to model and predict on 2019. Think of 1985-2018 as your train and test data with 2019 as your validation. \n-Try using a secondary benchmark like model accuracy. If your model can predict the outcomes of the tournament games &gt; 75% from 2015-2019, you should be suspicious of your results especially if your logloss is below .46. </p>",
      "rawMarkdown": ".46 would be an average score between 2015-2019. It is very possible to get scores below these numbers for one or maybe a few years but - up to this point - I have not been able to get an average score below .46. \nIf you want to prevent data leaks:\n-Don't use data from KenPom's website as it includes tournament info from the previous seasons. \n-Use only data from prior seasons to model and then predict on current season i.e. use data from 1985-2018 seasons to model and predict on 2019. Think of 1985-2018 as your train and test data with 2019 as your validation. \n-Try using a secondary benchmark like model accuracy. If your model can predict the outcomes of the tournament games &gt; 75% from 2015-2019, you should be suspicious of your results especially if your logloss is below .46.",
      "votes": null
    },
    {
      "id": "755337",
      "postDate": "02/24/2020 17:16:16",
      "content": "<p>Thanks for your suggestions, <a href=\"/btesch\">@btesch</a> </p>",
      "rawMarkdown": "Thanks for your suggestions, @btesch",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 755255,
      "author_name": "henriksjokvist",
      "author_url": "",
      "post_date": "02/24/2020 15:49:19",
      "content": "<p>There is obviously some data that leaks the outcome of the game, e.g. using stats from a game to predict its winner (shooting %, reb's, ast's or even the score itself if you really screw up)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 755304,
      "author_name": "btesch",
      "author_url": "",
      "post_date": "02/24/2020 16:47:38",
      "content": "<p>.46 would be an average score between 2015-2019. It is very possible to get scores below these numbers for one or maybe a few years but - up to this point - I have not been able to get an average score below .46. \nIf you want to prevent data leaks:\n-Don't use data from KenPom's website as it includes tournament info from the previous seasons. \n-Use only data from prior seasons to model and then predict on current season i.e. use data from 1985-2018 seasons to model and predict on 2019. Think of 1985-2018 as your train and test data with 2019 as your validation. \n-Try using a secondary benchmark like model accuracy. If your model can predict the outcomes of the tournament games &gt; 75% from 2015-2019, you should be suspicious of your results especially if your logloss is below .46. </p>",
      "votes": null,
      "replies": [
        {
          "id": 755337,
          "author_name": "shahules",
          "author_url": "",
          "post_date": "02/24/2020 17:16:16",
          "content": "<p>Thanks for your suggestions, <a href=\"/btesch\">@btesch</a> </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "755186": "As pointed out in one of the posts,if your score is less than 0.46 it should be probably due to some leak and this won't hold in the final test set. So what are the methods to prevent this leak?\nI tried removing the data from 2015 onwards from the training set. Are there any other methods?",
    "755255": "There is obviously some data that leaks the outcome of the game, e.g. using stats from a game to predict its winner (shooting %, reb's, ast's or even the score itself if you really screw up)",
    "755304": ".46 would be an average score between 2015-2019. It is very possible to get scores below these numbers for one or maybe a few years but - up to this point - I have not been able to get an average score below .46. \nIf you want to prevent data leaks:\n-Don't use data from KenPom's website as it includes tournament info from the previous seasons. \n-Use only data from prior seasons to model and then predict on current season i.e. use data from 1985-2018 seasons to model and predict on 2019. Think of 1985-2018 as your train and test data with 2019 as your validation. \n-Try using a secondary benchmark like model accuracy. If your model can predict the outcomes of the tournament games &gt; 75% from 2015-2019, you should be suspicious of your results especially if your logloss is below .46.",
    "755337": "Thanks for your suggestions, @btesch"
  },
  "source": "meta"
}