{
  "id": 314744,
  "title": "What's a good log loss and data-leakage",
  "url": "/competitions/football-match-probability-prediction/discussion/314744",
  "author_name": "",
  "post_date": "2022-03-24T08:56:08.324075800Z",
  "votes": 8,
  "comment_count": 14,
  "views": 0,
  "content": "<p>The competition loss is the log-loss of a three classes classification model. Since the data are past football matches and since we decided to provide Sportsmonk ids, <strong>data-leakage</strong> is easy, but obviously against the rules.</p>\n<p>It is easy to detect using the log-loss of this competition so here are some guidelines on what a good log-loss is:</p>\n<ul>\n<li>log-loss &lt; 0.98: it is genuinely good</li>\n<li>log-loss &lt; 0.90: you start to be an oracle</li>\n<li>log-loss &lt; 0.50: you are back from the futures with the Almanac</li>\n</ul>\n<p>For this competition, <strong>lightly</strong> beating the bookmaker's loss would be a great achievement. However, if you find any sign of leakage within the dataset let us know.</p>\n<p><strong>About the date leak</strong>: we were aware of it because we deal with time series. But we had to make a choice for the competition to be able to use past data.  <strong>That's not a problem</strong> if you do your CV carefully and do not use the test set in your learning model. </p>\n<p>Bear in mind:</p>\n<ul>\n<li><strong>Solution using the leak will be removed and not eligible for prizes as the private log-loss will show it</strong></li>\n<li>Both the train and test sets are time-ordered so CV should be done carefully.</li>\n<li>As a result, If you are using the test to predict the train you are effectively back from the future with the Almanac.</li>\n<li>Do not use the test set in your training procedure, which makes no sense.</li>\n<li>This leak is the same thing as going on the web and getting the results of the matches, then matching the test.</li>\n<li>Beating the bookmaker's log-loss is probably not achievable.</li>\n</ul>\n<p>Have fun &amp; good luck.</p>",
  "messages": [
    {
      "id": "1733411",
      "postDate": "03/24/2022 08:56:08",
      "content": "<p>The competition loss is the log-loss of a three classes classification model. Since the data are past football matches and since we decided to provide Sportsmonk ids, <strong>data-leakage</strong> is easy, but obviously against the rules.</p>\n<p>It is easy to detect using the log-loss of this competition so here are some guidelines on what a good log-loss is:</p>\n<ul>\n<li>log-loss &lt; 0.98: it is genuinely good</li>\n<li>log-loss &lt; 0.90: you start to be an oracle</li>\n<li>log-loss &lt; 0.50: you are back from the futures with the Almanac</li>\n</ul>\n<p>For this competition, <strong>lightly</strong> beating the bookmaker's loss would be a great achievement. However, if you find any sign of leakage within the dataset let us know.</p>\n<p><strong>About the date leak</strong>: we were aware of it because we deal with time series. But we had to make a choice for the competition to be able to use past data.  <strong>That's not a problem</strong> if you do your CV carefully and do not use the test set in your learning model. </p>\n<p>Bear in mind:</p>\n<ul>\n<li><strong>Solution using the leak will be removed and not eligible for prizes as the private log-loss will show it</strong></li>\n<li>Both the train and test sets are time-ordered so CV should be done carefully.</li>\n<li>As a result, If you are using the test to predict the train you are effectively back from the future with the Almanac.</li>\n<li>Do not use the test set in your training procedure, which makes no sense.</li>\n<li>This leak is the same thing as going on the web and getting the results of the matches, then matching the test.</li>\n<li>Beating the bookmaker's log-loss is probably not achievable.</li>\n</ul>\n<p>Have fun &amp; good luck.</p>",
      "rawMarkdown": "The competition loss is the log-loss of a three classes classification model. Since the data are past football matches and since we decided to provide Sportsmonk ids, **data-leakage** is easy, but obviously against the rules.\n\nIt is easy to detect using the log-loss of this competition so here are some guidelines on what a good log-loss is:\n- log-loss < 0.98: it is genuinely good\n- log-loss < 0.90: you start to be an oracle\n- log-loss < 0.50: you are back from the futures with the Almanac\n\nFor this competition, **lightly** beating the bookmaker's loss would be a great achievement. However, if you find any sign of leakage within the dataset let us know.\n\n**About the date leak**: we were aware of it because we deal with time series. But we had to make a choice for the competition to be able to use past data.  **That's not a problem** if you do your CV carefully and do not use the test set in your learning model. \n\nBear in mind:\n- **Solution using the leak will be removed and not eligible for prizes as the private log-loss will show it**\n- Both the train and test sets are time-ordered so CV should be done carefully.\n- As a result, If you are using the test to predict the train you are effectively back from the future with the Almanac.\n- Do not use the test set in your training procedure, which makes no sense.\n- This leak is the same thing as going on the web and getting the results of the matches, then matching the test.\n- Beating the bookmaker's log-loss is probably not achievable.\n\nHave fun & good luck.",
      "votes": null
    },
    {
      "id": "1733477",
      "postDate": "03/24/2022 10:06:17",
      "content": "<p>I assume from a glance at the top of the current leaderboard that leakage has occurred. Perhaps there are a couple of ways this might happen?</p>\n<ol>\n<li><p>This might come from matching the game currently being predicted to items in the historical data for a future game. That is, a game could appear both as a test item and as an historical game with a known outcome elsewhere in the either the training or test file. I haven't checked the data for instances of this. It is possible to design a model to exploit this, but also conceivable that a smart model might learn to do this without the user's specific intent.</p></li>\n<li><p>Someone could have used external knowledge or data to remember or look up the result of a game.</p></li>\n</ol>",
      "rawMarkdown": "I assume from a glance at the top of the current leaderboard that leakage has occurred. Perhaps there are a couple of ways this might happen?\n\n1. This might come from matching the game currently being predicted to items in the historical data for a future game. That is, a game could appear both as a test item and as an historical game with a known outcome elsewhere in the either the training or test file. I haven't checked the data for instances of this. It is possible to design a model to exploit this, but also conceivable that a smart model might learn to do this without the user's specific intent.\n\n2. Someone could have used external knowledge or data to remember or look up the result of a game.",
      "votes": null
    },
    {
      "id": "1733478",
      "postDate": "03/24/2022 10:09:19",
      "content": "<p>There is a leakage. It worked for me because I am not back from the future. Should I just post it open to everybody? </p>",
      "rawMarkdown": "There is a leakage. It worked for me because I am not back from the future. Should I just post it open to everybody?",
      "votes": null
    },
    {
      "id": "1733489",
      "postDate": "03/24/2022 10:30:55",
      "content": "<p>If there is one can you share it with us.</p>",
      "rawMarkdown": "If there is one can you share it with us.",
      "votes": null
    },
    {
      "id": "1733512",
      "postDate": "03/24/2022 11:01:20",
      "content": "<p>It might also be due to the fact that historical features are also actual matches played and because the dataset is time-ordered you can find them. But that should not be a problem if the model is not built to do so.</p>",
      "rawMarkdown": "It might also be due to the fact that historical features are also actual matches played and because the dataset is time-ordered you can find them. But that should not be a problem if the model is not built to do so.",
      "votes": null
    },
    {
      "id": "1733534",
      "postDate": "03/24/2022 11:26:02",
      "content": "<p>It is this the leak I found. But in my opinion is a problem because we are supposed to feed the model with results that we are supposed to predict too. The historical results are the correspond to results we have to predict. It makes no sense. In my opinion, a smaller test set with only the last match of a team to predict would have made a lot more sense</p>",
      "rawMarkdown": "It is this the leak I found. But in my opinion is a problem because we are supposed to feed the model with results that we are supposed to predict too. The historical results are the correspond to results we have to predict. It makes no sense. In my opinion, a smaller test set with only the last match of a team to predict would have made a lot more sense",
      "votes": null
    },
    {
      "id": "1733549",
      "postDate": "03/24/2022 11:49:52",
      "content": "<p>Normally with time series we should have used a Kaggle env and running for new matches coming up this year. But it is not available for the community competition. </p>\n<p>Without an env, it is the only way to provide you historical team results for you to use in the test set. A smaller test set is not a solution as that will make too much variance between your train and test loss. </p>\n<p>In any case, solutions will be reviewed and we think models can be built without exploiting the leak on purpose as is it basically based on joining dates.</p>\n<p>For now, we do not need to update the data.</p>",
      "rawMarkdown": "Normally with time series we should have used a Kaggle env and running for new matches coming up this year. But it is not available for the community competition. \n\nWithout an env, it is the only way to provide you historical team results for you to use in the test set. A smaller test set is not a solution as that will make too much variance between your train and test loss. \n\nIn any case, solutions will be reviewed and we think models can be built without exploiting the leak on purpose as is it basically based on joining dates.\n\nFor now, we do not need to update the data.",
      "votes": null
    },
    {
      "id": "1733746",
      "postDate": "03/24/2022 15:14:30",
      "content": "<p>Hello Octosport Team,</p>\n<p>Would you mind sharing what log loss you get using this dataset and your methods? The bookmakers' benchmark is good, but I believe the bookmakers use more information than is available in the competition dataset.</p>\n<p>Thanks in advance!</p>",
      "rawMarkdown": "Hello Octosport Team,\n\nWould you mind sharing what log loss you get using this dataset and your methods? The bookmakers' benchmark is good, but I believe the bookmakers use more information than is available in the competition dataset.\n\nThanks in advance!",
      "votes": null
    },
    {
      "id": "1734011",
      "postDate": "03/24/2022 21:48:53",
      "content": "<p>We actually do not use this dataset directly but you are right. We think the best models will be in the range [0.98-0.99]. Beating the bookmakers is really hard (if not impossible here) and not fair because their log-loss is based on closing odds while we should compare to opening odds, but we do not have them.</p>",
      "rawMarkdown": "We actually do not use this dataset directly but you are right. We think the best models will be in the range [0.98-0.99]. Beating the bookmakers is really hard (if not impossible here) and not fair because their log-loss is based on closing odds while we should compare to opening odds, but we do not have them.",
      "votes": null
    },
    {
      "id": "1734585",
      "postDate": "03/25/2022 13:23:34",
      "content": "<p>Some people may use the leak just to get a gold in the competition and screw up the score of others that do not use the leak.</p>",
      "rawMarkdown": "Some people may use the leak just to get a gold in the competition and screw up the score of others that do not use the leak.",
      "votes": null
    },
    {
      "id": "1734846",
      "postDate": "03/25/2022 16:34:44",
      "content": "<p>Community competition does not award medals. Plus people using the leak do not see the private loss which contains data that can not be leaked, so they will have a low score (and/or their submission removed)</p>",
      "rawMarkdown": "Community competition does not award medals. Plus people using the leak do not see the private loss which contains data that can not be leaked, so they will have a low score (and/or their submission removed)",
      "votes": null
    },
    {
      "id": "1737986",
      "postDate": "03/29/2022 00:13:41",
      "content": "<p>i got 0.96 score in public lb  , i mistakenly used internet(automl) and there is no use of test data for the shake of increasing accuracy. I removed that notebook and now i am going to use new notebook without using internet(rules)</p>",
      "rawMarkdown": "i got 0.96 score in public lb  , i mistakenly used internet(automl) and there is no use of test data for the shake of increasing accuracy. I removed that notebook and now i am going to use new notebook without using internet(rules)",
      "votes": null
    },
    {
      "id": "1738664",
      "postDate": "03/29/2022 12:49:39",
      "content": "<p>That was the other way of tricking the competition. But you will need to deselect your submission to reflect it on the leaderboard. Hopefully, everyone will do to clean the leaderboard. The \"true\" ranking starts below the bookmaker benchmark.</p>",
      "rawMarkdown": "That was the other way of tricking the competition. But you will need to deselect your submission to reflect it on the leaderboard. Hopefully, everyone will do to clean the leaderboard. The \"true\" ranking starts below the bookmaker benchmark.",
      "votes": null
    },
    {
      "id": "1744789",
      "postDate": "04/04/2022 10:13:48",
      "content": "<blockquote>\n  <p>their log-loss is based on closing odds while we should compare to opening odds</p>\n</blockquote>\n<p>In my opinon both opening and closing odds contain interesting information. The opening odds tell us what the bookmakers think (before testing the market). The closing odds tell us what the market thinks (modulo various assumptions about the efficiency of the market, distorting effects of isolated large bets and hedges against them etc.).</p>",
      "rawMarkdown": ">  their log-loss is based on closing odds while we should compare to opening odds\n\nIn my opinon both opening and closing odds contain interesting information. The opening odds tell us what the bookmakers think (before testing the market). The closing odds tell us what the market thinks (modulo various assumptions about the efficiency of the market, distorting effects of isolated large bets and hedges against them etc.).",
      "votes": null
    },
    {
      "id": "1744823",
      "postDate": "04/04/2022 10:56:45",
      "content": "<p>Indeed but in this dataset, you make the prediction one week forward while odds predictions are based on closing odds. So, they include news, lineups, weather, and so on. And research shows that closing odds are the most predictive.</p>",
      "rawMarkdown": "Indeed but in this dataset, you make the prediction one week forward while odds predictions are based on closing odds. So, they include news, lineups, weather, and so on. And research shows that closing odds are the most predictive.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1733477,
      "author_name": "jbomitchell",
      "author_url": "",
      "post_date": "03/24/2022 10:06:17",
      "content": "<p>I assume from a glance at the top of the current leaderboard that leakage has occurred. Perhaps there are a couple of ways this might happen?</p>\n<ol>\n<li><p>This might come from matching the game currently being predicted to items in the historical data for a future game. That is, a game could appear both as a test item and as an historical game with a known outcome elsewhere in the either the training or test file. I haven't checked the data for instances of this. It is possible to design a model to exploit this, but also conceivable that a smart model might learn to do this without the user's specific intent.</p></li>\n<li><p>Someone could have used external knowledge or data to remember or look up the result of a game.</p></li>\n</ol>",
      "votes": null,
      "replies": [
        {
          "id": 1733512,
          "author_name": "octosportio",
          "author_url": "",
          "post_date": "03/24/2022 11:01:20",
          "content": "<p>It might also be due to the fact that historical features are also actual matches played and because the dataset is time-ordered you can find them. But that should not be a problem if the model is not built to do so.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1733534,
          "author_name": "joanponsv",
          "author_url": "",
          "post_date": "03/24/2022 11:26:02",
          "content": "<p>It is this the leak I found. But in my opinion is a problem because we are supposed to feed the model with results that we are supposed to predict too. The historical results are the correspond to results we have to predict. It makes no sense. In my opinion, a smaller test set with only the last match of a team to predict would have made a lot more sense</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1733549,
          "author_name": "octosportio",
          "author_url": "",
          "post_date": "03/24/2022 11:49:52",
          "content": "<p>Normally with time series we should have used a Kaggle env and running for new matches coming up this year. But it is not available for the community competition. </p>\n<p>Without an env, it is the only way to provide you historical team results for you to use in the test set. A smaller test set is not a solution as that will make too much variance between your train and test loss. </p>\n<p>In any case, solutions will be reviewed and we think models can be built without exploiting the leak on purpose as is it basically based on joining dates.</p>\n<p>For now, we do not need to update the data.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1733478,
      "author_name": "joanponsv",
      "author_url": "",
      "post_date": "03/24/2022 10:09:19",
      "content": "<p>There is a leakage. It worked for me because I am not back from the future. Should I just post it open to everybody? </p>",
      "votes": null,
      "replies": [
        {
          "id": 1733489,
          "author_name": "octosportio",
          "author_url": "",
          "post_date": "03/24/2022 10:30:55",
          "content": "<p>If there is one can you share it with us.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1733746,
      "author_name": "paulofelipe",
      "author_url": "",
      "post_date": "03/24/2022 15:14:30",
      "content": "<p>Hello Octosport Team,</p>\n<p>Would you mind sharing what log loss you get using this dataset and your methods? The bookmakers' benchmark is good, but I believe the bookmakers use more information than is available in the competition dataset.</p>\n<p>Thanks in advance!</p>",
      "votes": null,
      "replies": [
        {
          "id": 1734011,
          "author_name": "octosportio",
          "author_url": "",
          "post_date": "03/24/2022 21:48:53",
          "content": "<p>We actually do not use this dataset directly but you are right. We think the best models will be in the range [0.98-0.99]. Beating the bookmakers is really hard (if not impossible here) and not fair because their log-loss is based on closing odds while we should compare to opening odds, but we do not have them.</p>",
          "votes": null,
          "replies": [
            {
              "id": 1744789,
              "author_name": "jbomitchell",
              "author_url": "",
              "post_date": "04/04/2022 10:13:48",
              "content": "<blockquote>\n  <p>their log-loss is based on closing odds while we should compare to opening odds</p>\n</blockquote>\n<p>In my opinon both opening and closing odds contain interesting information. The opening odds tell us what the bookmakers think (before testing the market). The closing odds tell us what the market thinks (modulo various assumptions about the efficiency of the market, distorting effects of isolated large bets and hedges against them etc.).</p>",
              "votes": null,
              "replies": []
            }
          ]
        },
        {
          "id": 1744823,
          "author_name": "octosportio",
          "author_url": "",
          "post_date": "04/04/2022 10:56:45",
          "content": "<p>Indeed but in this dataset, you make the prediction one week forward while odds predictions are based on closing odds. So, they include news, lineups, weather, and so on. And research shows that closing odds are the most predictive.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1734585,
      "author_name": "seraquevence",
      "author_url": "",
      "post_date": "03/25/2022 13:23:34",
      "content": "<p>Some people may use the leak just to get a gold in the competition and screw up the score of others that do not use the leak.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1734846,
          "author_name": "octosportio",
          "author_url": "",
          "post_date": "03/25/2022 16:34:44",
          "content": "<p>Community competition does not award medals. Plus people using the leak do not see the private loss which contains data that can not be leaked, so they will have a low score (and/or their submission removed)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1737986,
      "author_name": "himanshunayal",
      "author_url": "",
      "post_date": "03/29/2022 00:13:41",
      "content": "<p>i got 0.96 score in public lb  , i mistakenly used internet(automl) and there is no use of test data for the shake of increasing accuracy. I removed that notebook and now i am going to use new notebook without using internet(rules)</p>",
      "votes": null,
      "replies": [
        {
          "id": 1738664,
          "author_name": "octosportio",
          "author_url": "",
          "post_date": "03/29/2022 12:49:39",
          "content": "<p>That was the other way of tricking the competition. But you will need to deselect your submission to reflect it on the leaderboard. Hopefully, everyone will do to clean the leaderboard. The \"true\" ranking starts below the bookmaker benchmark.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1733411": "The competition loss is the log-loss of a three classes classification model. Since the data are past football matches and since we decided to provide Sportsmonk ids, **data-leakage** is easy, but obviously against the rules.\n\nIt is easy to detect using the log-loss of this competition so here are some guidelines on what a good log-loss is:\n- log-loss < 0.98: it is genuinely good\n- log-loss < 0.90: you start to be an oracle\n- log-loss < 0.50: you are back from the futures with the Almanac\n\nFor this competition, **lightly** beating the bookmaker's loss would be a great achievement. However, if you find any sign of leakage within the dataset let us know.\n\n**About the date leak**: we were aware of it because we deal with time series. But we had to make a choice for the competition to be able to use past data.  **That's not a problem** if you do your CV carefully and do not use the test set in your learning model. \n\nBear in mind:\n- **Solution using the leak will be removed and not eligible for prizes as the private log-loss will show it**\n- Both the train and test sets are time-ordered so CV should be done carefully.\n- As a result, If you are using the test to predict the train you are effectively back from the future with the Almanac.\n- Do not use the test set in your training procedure, which makes no sense.\n- This leak is the same thing as going on the web and getting the results of the matches, then matching the test.\n- Beating the bookmaker's log-loss is probably not achievable.\n\nHave fun & good luck.",
    "1733477": "I assume from a glance at the top of the current leaderboard that leakage has occurred. Perhaps there are a couple of ways this might happen?\n\n1. This might come from matching the game currently being predicted to items in the historical data for a future game. That is, a game could appear both as a test item and as an historical game with a known outcome elsewhere in the either the training or test file. I haven't checked the data for instances of this. It is possible to design a model to exploit this, but also conceivable that a smart model might learn to do this without the user's specific intent.\n\n2. Someone could have used external knowledge or data to remember or look up the result of a game.",
    "1733478": "There is a leakage. It worked for me because I am not back from the future. Should I just post it open to everybody?",
    "1733489": "If there is one can you share it with us.",
    "1733512": "It might also be due to the fact that historical features are also actual matches played and because the dataset is time-ordered you can find them. But that should not be a problem if the model is not built to do so.",
    "1733534": "It is this the leak I found. But in my opinion is a problem because we are supposed to feed the model with results that we are supposed to predict too. The historical results are the correspond to results we have to predict. It makes no sense. In my opinion, a smaller test set with only the last match of a team to predict would have made a lot more sense",
    "1733549": "Normally with time series we should have used a Kaggle env and running for new matches coming up this year. But it is not available for the community competition. \n\nWithout an env, it is the only way to provide you historical team results for you to use in the test set. A smaller test set is not a solution as that will make too much variance between your train and test loss. \n\nIn any case, solutions will be reviewed and we think models can be built without exploiting the leak on purpose as is it basically based on joining dates.\n\nFor now, we do not need to update the data.",
    "1733746": "Hello Octosport Team,\n\nWould you mind sharing what log loss you get using this dataset and your methods? The bookmakers' benchmark is good, but I believe the bookmakers use more information than is available in the competition dataset.\n\nThanks in advance!",
    "1734011": "We actually do not use this dataset directly but you are right. We think the best models will be in the range [0.98-0.99]. Beating the bookmakers is really hard (if not impossible here) and not fair because their log-loss is based on closing odds while we should compare to opening odds, but we do not have them.",
    "1734585": "Some people may use the leak just to get a gold in the competition and screw up the score of others that do not use the leak.",
    "1734846": "Community competition does not award medals. Plus people using the leak do not see the private loss which contains data that can not be leaked, so they will have a low score (and/or their submission removed)",
    "1737986": "i got 0.96 score in public lb  , i mistakenly used internet(automl) and there is no use of test data for the shake of increasing accuracy. I removed that notebook and now i am going to use new notebook without using internet(rules)",
    "1738664": "That was the other way of tricking the competition. But you will need to deselect your submission to reflect it on the leaderboard. Hopefully, everyone will do to clean the leaderboard. The \"true\" ranking starts below the bookmaker benchmark.",
    "1744789": ">  their log-loss is based on closing odds while we should compare to opening odds\n\nIn my opinon both opening and closing odds contain interesting information. The opening odds tell us what the bookmakers think (before testing the market). The closing odds tell us what the market thinks (modulo various assumptions about the efficiency of the market, distorting effects of isolated large bets and hedges against them etc.).",
    "1744823": "Indeed but in this dataset, you make the prediction one week forward while odds predictions are based on closing odds. So, they include news, lineups, weather, and so on. And research shows that closing odds are the most predictive."
  },
  "source": "meta"
}