{
  "id": 133118,
  "title": "A better evaluation function?",
  "url": "/competitions/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/discussion/133118",
  "author_name": "Michael Griffiths",
  "post_date": "2020-02-29T21:13:53.740000",
  "votes": 0,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hello,</p>\n\n<p>Firstly, I'm new to this particular competition and so I'm sure this subject has been thoroughly hashed out and the horses have left the barn, but I wanted to express my concerns on the current evaluation function.</p>\n\n<p>If I predict a probability of a win is .65 and 65% of the games that I've made this prediction for actually won, then this is a \"win\". And if this is true for every prediction, that is, 35% of the games that I predicted a probability of .35 actually won, 62% of the games that I predicted a probability of .62 actually won, etc. then my job of a predictionista is done.</p>\n\n<p>I'm not sure that the current logloss score really captures the quality of the predictions being made per the above \"definition\" so I'm proposing a new function. \nIt basically calculates the standard deviation of the prediction made versus the actual history of the games in this prediction \"bucket\".</p>\n\n<p>```\nfor each prediction {\n    // 0&lt;=predict&lt;=1, gets converted to 0&lt;=bucket&lt;=10</p>\n\n<pre><code>int bucket = (int) ((predict*10) + .49);\n\nif( is_actual_win ) \n    cntWin[bucket]++\ncntTotal[bucket]++\n</code></pre>\n\n<p>}</p>\n\n<p>sum = 0;\nfor each bucket {\n    double diff = bucket - 10*cntWin[bucket]/cntTotal[bucket]\n    sum += diff * diff\n}\nnewScore = Math.sqrt(sum/11)</p>\n\n<p>```</p>\n\n<p>If I were to build a \"perfect\" dataset and compute the logloss score, what would\n it be?  Not 0, it's in fact .6608. With the proposed evaluation function\nthe closer the score is to 0, the better your prediction is.</p>\n\n<p>And yes I do understand the merits of the logloss approach, but in the real world,\na typical prediction of a basketball game is something like .54 or .66, not 0 or 1. \nand so we should compare actual historic percentages to predictions.</p>\n\n<p>Thank you for your time, Mike</p>",
  "messages": [
    {
      "id": 760089,
      "postDate": "2020-02-29T21:13:53.740Z",
      "content": "<p>Hello,</p>\n\n<p>Firstly, I'm new to this particular competition and so I'm sure this subject has been thoroughly hashed out and the horses have left the barn, but I wanted to express my concerns on the current evaluation function.</p>\n\n<p>If I predict a probability of a win is .65 and 65% of the games that I've made this prediction for actually won, then this is a \"win\". And if this is true for every prediction, that is, 35% of the games that I predicted a probability of .35 actually won, 62% of the games that I predicted a probability of .62 actually won, etc. then my job of a predictionista is done.</p>\n\n<p>I'm not sure that the current logloss score really captures the quality of the predictions being made per the above \"definition\" so I'm proposing a new function. \nIt basically calculates the standard deviation of the prediction made versus the actual history of the games in this prediction \"bucket\".</p>\n\n<p>```\nfor each prediction {\n    // 0&lt;=predict&lt;=1, gets converted to 0&lt;=bucket&lt;=10</p>\n\n<pre><code>int bucket = (int) ((predict*10) + .49);\n\nif( is_actual_win ) \n    cntWin[bucket]++\ncntTotal[bucket]++\n</code></pre>\n\n<p>}</p>\n\n<p>sum = 0;\nfor each bucket {\n    double diff = bucket - 10*cntWin[bucket]/cntTotal[bucket]\n    sum += diff * diff\n}\nnewScore = Math.sqrt(sum/11)</p>\n\n<p>```</p>\n\n<p>If I were to build a \"perfect\" dataset and compute the logloss score, what would\n it be?  Not 0, it's in fact .6608. With the proposed evaluation function\nthe closer the score is to 0, the better your prediction is.</p>\n\n<p>And yes I do understand the merits of the logloss approach, but in the real world,\na typical prediction of a basketball game is something like .54 or .66, not 0 or 1. \nand so we should compare actual historic percentages to predictions.</p>\n\n<p>Thank you for your time, Mike</p>",
      "rawMarkdown": "\nHello,\n\nFirstly, I'm new to this particular competition and so I'm sure this subject has been thoroughly hashed out and the horses have left the barn, but I wanted to express my concerns on the current evaluation function.\n\nIf I predict a probability of a win is .65 and 65% of the games that I've made this prediction for actually won, then this is a \"win\". And if this is true for every prediction, that is, 35% of the games that I predicted a probability of .35 actually won, 62% of the games that I predicted a probability of .62 actually won, etc. then my job of a predictionista is done.\n\n\nI'm not sure that the current logloss score really captures the quality of the predictions being made per the above \"definition\" so I'm proposing a new function. \nIt basically calculates the standard deviation of the prediction made versus the actual history of the games in this prediction \"bucket\".\n\n\n```\nfor each prediction {\n\t// 0&lt;=predict&lt;=1, gets converted to 0&lt;=bucket&lt;=10\n\n\tint bucket = (int) ((predict*10) + .49);\n\n\tif( is_actual_win ) \n\t\tcntWin[bucket]++\n\tcntTotal[bucket]++\n}\n\nsum = 0;\nfor each bucket {\n\tdouble diff = bucket - 10*cntWin[bucket]/cntTotal[bucket]\n\tsum += diff * diff\n}\nnewScore = Math.sqrt(sum/11)\n\t\n```\n\nIf I were to build a \"perfect\" dataset and compute the logloss score, what would\n it be?  Not 0, it's in fact .6608. With the proposed evaluation function\nthe closer the score is to 0, the better your prediction is.\n\nAnd yes I do understand the merits of the logloss approach, but in the real world,\na typical prediction of a basketball game is something like .54 or .66, not 0 or 1. \nand so we should compare actual historic percentages to predictions.\n\nThank you for your time, Mike\n\n\n \n"
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "760089": "\nHello,\n\nFirstly, I'm new to this particular competition and so I'm sure this subject has been thoroughly hashed out and the horses have left the barn, but I wanted to express my concerns on the current evaluation function.\n\nIf I predict a probability of a win is .65 and 65% of the games that I've made this prediction for actually won, then this is a \"win\". And if this is true for every prediction, that is, 35% of the games that I predicted a probability of .35 actually won, 62% of the games that I predicted a probability of .62 actually won, etc. then my job of a predictionista is done.\n\n\nI'm not sure that the current logloss score really captures the quality of the predictions being made per the above \"definition\" so I'm proposing a new function. \nIt basically calculates the standard deviation of the prediction made versus the actual history of the games in this prediction \"bucket\".\n\n\n```\nfor each prediction {\n\t// 0&lt;=predict&lt;=1, gets converted to 0&lt;=bucket&lt;=10\n\n\tint bucket = (int) ((predict*10) + .49);\n\n\tif( is_actual_win ) \n\t\tcntWin[bucket]++\n\tcntTotal[bucket]++\n}\n\nsum = 0;\nfor each bucket {\n\tdouble diff = bucket - 10*cntWin[bucket]/cntTotal[bucket]\n\tsum += diff * diff\n}\nnewScore = Math.sqrt(sum/11)\n\t\n```\n\nIf I were to build a \"perfect\" dataset and compute the logloss score, what would\n it be?  Not 0, it's in fact .6608. With the proposed evaluation function\nthe closer the score is to 0, the better your prediction is.\n\nAnd yes I do understand the merits of the logloss approach, but in the real world,\na typical prediction of a basketball game is something like .54 or .66, not 0 or 1. \nand so we should compare actual historic percentages to predictions.\n\nThank you for your time, Mike\n\n\n \n"
  }
}