{
  "id": 24035,
  "title": "Performing worse than 0.5 benchmark?",
  "url": "/competitions/dogs-vs-cats-redux-kernels-edition/discussion/24035",
  "author_name": "",
  "post_date": "2016-10-02T17:03:28.367Z",
  "votes": null,
  "comment_count": 11,
  "views": 823,
  "content": "<p>According to the auto grader, my solution performs worse than the 0.5 benchmark. However, upon manual examination of the first 50 or so pictures, my submission seems to be mostly correct. Given the way the score is computed on the leaderboard (loss rather than percent correct), it's actually very hard to tell how many images the grader thinks I got correct. Can someone post a solution csv so I can see what I'm doing wrong?</p>\n\n<p>Here's the top of mine, with incorrect at ids 1,17,29</p>\n\n<pre><code>id,label\n1,0\n2,1\n3,1\n4,1\n5,0\n6,0\n7,0\n8,0\n9,0\n10,0\n11,0\n12,1\n13,0\n14,0\n15,0\n16,0\n17,0\n18,1\n19,0\n20,0\n21,1\n22,0\n23,1\n24,1\n25,0\n26,1\n27,1\n28,0\n29,1\n30,1\n31,1\n32,1\n33,1\n34,0\n35,0\n36,0\n37,0\n38,0\n39,1\n40,0\n41,1\n42,1\n43,1\n44,1\n45,0\n46,1\n47,0\n48,1\n49,1\n50,0\n</code></pre>",
  "messages": [
    {
      "id": "137537",
      "postDate": "10/02/2016 17:03:28",
      "content": "<p>According to the auto grader, my solution performs worse than the 0.5 benchmark. However, upon manual examination of the first 50 or so pictures, my submission seems to be mostly correct. Given the way the score is computed on the leaderboard (loss rather than percent correct), it's actually very hard to tell how many images the grader thinks I got correct. Can someone post a solution csv so I can see what I'm doing wrong?</p>\n\n<p>Here's the top of mine, with incorrect at ids 1,17,29</p>\n\n<pre><code>id,label\n1,0\n2,1\n3,1\n4,1\n5,0\n6,0\n7,0\n8,0\n9,0\n10,0\n11,0\n12,1\n13,0\n14,0\n15,0\n16,0\n17,0\n18,1\n19,0\n20,0\n21,1\n22,0\n23,1\n24,1\n25,0\n26,1\n27,1\n28,0\n29,1\n30,1\n31,1\n32,1\n33,1\n34,0\n35,0\n36,0\n37,0\n38,0\n39,1\n40,0\n41,1\n42,1\n43,1\n44,1\n45,0\n46,1\n47,0\n48,1\n49,1\n50,0\n</code></pre>",
      "rawMarkdown": "According to the auto grader, my solution performs worse than the 0.5 benchmark. However, upon manual examination of the first 50 or so pictures, my submission seems to be mostly correct. Given the way the score is computed on the leaderboard (loss rather than percent correct), it's actually very hard to tell how many images the grader thinks I got correct. Can someone post a solution csv so I can see what I'm doing wrong?\r\n\r\nHere's the top of mine, with incorrect at ids 1,17,29\r\n\r\n    id,label\r\n    1,0\r\n    2,1\r\n    3,1\r\n    4,1\r\n    5,0\r\n    6,0\r\n    7,0\r\n    8,0\r\n    9,0\r\n    10,0\r\n    11,0\r\n    12,1\r\n    13,0\r\n    14,0\r\n    15,0\r\n    16,0\r\n    17,0\r\n    18,1\r\n    19,0\r\n    20,0\r\n    21,1\r\n    22,0\r\n    23,1\r\n    24,1\r\n    25,0\r\n    26,1\r\n    27,1\r\n    28,0\r\n    29,1\r\n    30,1\r\n    31,1\r\n    32,1\r\n    33,1\r\n    34,0\r\n    35,0\r\n    36,0\r\n    37,0\r\n    38,0\r\n    39,1\r\n    40,0\r\n    41,1\r\n    42,1\r\n    43,1\r\n    44,1\r\n    45,0\r\n    46,1\r\n    47,0\r\n    48,1\r\n    49,1\r\n    50,0",
      "votes": null
    },
    {
      "id": "137538",
      "postDate": "10/02/2016 17:08:10",
      "content": "<p>Oh I see. If my predictions are binary classes (0,1), the logloss function results in some undefined values.</p>",
      "rawMarkdown": "Oh I see. If my predictions are binary classes (0,1), the logloss function results in some undefined values.",
      "votes": null
    },
    {
      "id": "137807",
      "postDate": "10/05/2016 06:03:43",
      "content": "<p>As it has been stated in the Get Data section &quot;For each image in the test set, you should predict a probability that the image is a dog (1 = dog, 0 = cat).&quot;, you should return the probability value but neither 0 or 1 (e.g. 0.0012, 0.892, etc) </p>",
      "rawMarkdown": "As it has been stated in the Get Data section \"For each image in the test set, you should predict a probability that the image is a dog (1 = dog, 0 = cat).\", you should return the probability value but neither 0 or 1 (e.g. 0.0012, 0.892, etc)",
      "votes": null
    },
    {
      "id": "137868",
      "postDate": "10/05/2016 18:51:24",
      "content": "<p>I must have missed this section (and actually I still can't find it). Where do you see &quot;but neither 0 or 1&quot;?</p>",
      "rawMarkdown": "I must have missed this section (and actually I still can't find it). Where do you see \"but neither 0 or 1\"?",
      "votes": null
    },
    {
      "id": "137917",
      "postDate": "10/06/2016 01:12:28",
      "content": "<p>[quote=Ben KTByte;137868]</p>\n\n<p>I must have missed this section (and actually I still can't find it). Where do you see &quot;but neither 0 or 1&quot;?</p>\n\n<p>[/quote]</p>\n\n<p>Please check the description below the download data section <a href=\"https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/data\">https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/data</a></p>",
      "rawMarkdown": "[quote=Ben KTByte;137868]\r\n\r\nI must have missed this section (and actually I still can't find it). Where do you see \"but neither 0 or 1\"?\r\n\r\n[/quote]\r\n\r\nPlease check the description below the download data section https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/data",
      "votes": null
    },
    {
      "id": "138039",
      "postDate": "10/06/2016 18:56:37",
      "content": "<p>Yeah I don't see the word &quot;neither&quot; anywhere on that page? I do see probability, but implicit in probability is it could be 0 or 1</p>",
      "rawMarkdown": "Yeah I don't see the word \"neither\" anywhere on that page? I do see probability, but implicit in probability is it could be 0 or 1",
      "votes": null
    },
    {
      "id": "139739",
      "postDate": "10/16/2016 07:14:06",
      "content": "<p>Hi to all, </p>\n\n<p>I have the same issue.</p>\n\n<p>My submision file contains '0' for cat and '1' for dog and I get a ridiculous 32% as a result, though I've been checking and points some misclassifications but not 32%.</p>\n\n<p>I try with the probability and not the &quot;binary value'. </p>",
      "rawMarkdown": "Hi to all, \r\n\r\nI have the same issue.\r\n\r\nMy submision file contains '0' for cat and '1' for dog and I get a ridiculous 32% as a result, though I've been checking and points some misclassifications but not 32%.\r\n\r\nI try with the probability and not the \"binary value'.",
      "votes": null
    },
    {
      "id": "139742",
      "postDate": "10/16/2016 07:34:28",
      "content": "<p>Going back to my last comment : sending a probability (0.999, 0.995...) ok !</p>\n\n<p>But what means this probality without the result  : Cat vs Dog, 0 vs 1, or whatever you want.</p>\n\n<p>Sending a probability without the &quot;associated result&quot; sounds very strange to me. </p>",
      "rawMarkdown": "Going back to my last comment : sending a probability (0.999, 0.995...) ok !\r\n\r\nBut what means this probality without the result  : Cat vs Dog, 0 vs 1, or whatever you want.\r\n\r\nSending a probability without the \"associated result\" sounds very strange to me.",
      "votes": null
    },
    {
      "id": "139751",
      "postDate": "10/16/2016 09:51:03",
      "content": "<p>My last tries are worst and worst !</p>\n\n<p>I really don't understand the submit format Kaggle is expecting...</p>\n\n<p>I think I will give up though my score should be in the Top 10.</p>\n\n<p>Disappointing...</p>",
      "rawMarkdown": "My last tries are worst and worst !\r\n\r\nI really don't understand the submit format Kaggle is expecting...\r\n\r\nI think I will give up though my score should be in the Top 10.\r\n\r\nDisappointing...",
      "votes": null
    },
    {
      "id": "139753",
      "postDate": "10/16/2016 10:09:51",
      "content": "<p>[quote=Patrick Fichou;139742]</p>\n\n<p>Sending a probability without the &quot;associated result&quot; sounds very strange to me. </p>\n\n<p>[/quote]</p>\n\n<p>But it is how it works. Log loss punishes probabilities which are overconfident and wrong. If you predict 0 and it is in fact 1, you are punished larger than predicting 0.5 for that 1. This competition is not about getting maximum accuracy, but about reasonable probability predictions. You can have 99%+ accuracy and have a log loss higher than someone with about 90%+ accuracy, if your predictions are not reasonable. Here, having 0 and 1 for everything is not reasonable predictions unless you know you have 100% accuracy out of the box.</p>\n\n<p>Then, to reproduce the &quot;associated result&quot; it is not hard (it is useless to reproduce the associated result for this competition): the closer you are to 0 the higher probability it is a cat. The closer you are to 1, the higher probability it is a dog: <a href=\"https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/details/evaluation\">https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/details/evaluation</a></p>\n\n<ul>\n<li>0.5 would be 50% probability of cat, 50% probability of dog. (unless you use non-linear probability thresholds)</li>\n<li>1 would be 0% cat, 100% dog.</li>\n<li>0 would be 100% cat, 0% dog.</li>\n</ul>",
      "rawMarkdown": "[quote=Patrick Fichou;139742]\r\n\r\nSending a probability without the \"associated result\" sounds very strange to me. \r\n\r\n[/quote]\r\n\r\nBut it is how it works. Log loss punishes probabilities which are overconfident and wrong. If you predict 0 and it is in fact 1, you are punished larger than predicting 0.5 for that 1. This competition is not about getting maximum accuracy, but about reasonable probability predictions. You can have 99%+ accuracy and have a log loss higher than someone with about 90%+ accuracy, if your predictions are not reasonable. Here, having 0 and 1 for everything is not reasonable predictions unless you know you have 100% accuracy out of the box.\r\n\r\nThen, to reproduce the \"associated result\" it is not hard (it is useless to reproduce the associated result for this competition): the closer you are to 0 the higher probability it is a cat. The closer you are to 1, the higher probability it is a dog: https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/details/evaluation\r\n\r\n* 0.5 would be 50% probability of cat, 50% probability of dog. (unless you use non-linear probability thresholds)\r\n* 1 would be 0% cat, 100% dog.\r\n* 0 would be 100% cat, 0% dog.",
      "votes": null
    },
    {
      "id": "139755",
      "postDate": "10/16/2016 10:37:17",
      "content": "<p>Thank you very much !!! (Merci beaucoup)</p>\n\n<p>I will try again according to your answer...</p>\n\n<p>Not clear in the description of the submission, much with your post !</p>\n\n<p>Patrick</p>",
      "rawMarkdown": "Thank you very much !!! (Merci beaucoup)\r\n\r\nI will try again according to your answer...\r\n\r\nNot clear in the description of the submission, much with your post !\r\n\r\nPatrick",
      "votes": null
    },
    {
      "id": "154661",
      "postDate": "01/07/2017 14:02:28",
      "content": "<p>As mentioned in <a href=\"https://github.com/fastai/courses/blob/master/deeplearning1/nbs/dogs_cats_redux.ipynb\">this tutorial</a>, log loss punishes predictions that are far off, so I would replace all ones with .95 and all zeros with .05 just for safe.</p>",
      "rawMarkdown": "As mentioned in [this tutorial][1], log loss punishes predictions that are far off, so I would replace all ones with .95 and all zeros with .05 just for safe.\r\n\r\n\r\n  [1]: https://github.com/fastai/courses/blob/master/deeplearning1/nbs/dogs_cats_redux.ipynb",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 137538,
      "author_name": "benkapparate",
      "author_url": "",
      "post_date": "10/02/2016 17:08:10",
      "content": "<p>Oh I see. If my predictions are binary classes (0,1), the logloss function results in some undefined values.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137807,
      "author_name": "sakares",
      "author_url": "",
      "post_date": "10/05/2016 06:03:43",
      "content": "<p>As it has been stated in the Get Data section &quot;For each image in the test set, you should predict a probability that the image is a dog (1 = dog, 0 = cat).&quot;, you should return the probability value but neither 0 or 1 (e.g. 0.0012, 0.892, etc) </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137868,
      "author_name": "benkapparate",
      "author_url": "",
      "post_date": "10/05/2016 18:51:24",
      "content": "<p>I must have missed this section (and actually I still can't find it). Where do you see &quot;but neither 0 or 1&quot;?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137917,
      "author_name": "sakares",
      "author_url": "",
      "post_date": "10/06/2016 01:12:28",
      "content": "<p>[quote=Ben KTByte;137868]</p>\n\n<p>I must have missed this section (and actually I still can't find it). Where do you see &quot;but neither 0 or 1&quot;?</p>\n\n<p>[/quote]</p>\n\n<p>Please check the description below the download data section <a href=\"https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/data\">https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/data</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 138039,
      "author_name": "benkapparate",
      "author_url": "",
      "post_date": "10/06/2016 18:56:37",
      "content": "<p>Yeah I don't see the word &quot;neither&quot; anywhere on that page? I do see probability, but implicit in probability is it could be 0 or 1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 139739,
      "author_name": "pfichou",
      "author_url": "",
      "post_date": "10/16/2016 07:14:06",
      "content": "<p>Hi to all, </p>\n\n<p>I have the same issue.</p>\n\n<p>My submision file contains '0' for cat and '1' for dog and I get a ridiculous 32% as a result, though I've been checking and points some misclassifications but not 32%.</p>\n\n<p>I try with the probability and not the &quot;binary value'. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 139742,
      "author_name": "pfichou",
      "author_url": "",
      "post_date": "10/16/2016 07:34:28",
      "content": "<p>Going back to my last comment : sending a probability (0.999, 0.995...) ok !</p>\n\n<p>But what means this probality without the result  : Cat vs Dog, 0 vs 1, or whatever you want.</p>\n\n<p>Sending a probability without the &quot;associated result&quot; sounds very strange to me. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 139751,
      "author_name": "pfichou",
      "author_url": "",
      "post_date": "10/16/2016 09:51:03",
      "content": "<p>My last tries are worst and worst !</p>\n\n<p>I really don't understand the submit format Kaggle is expecting...</p>\n\n<p>I think I will give up though my score should be in the Top 10.</p>\n\n<p>Disappointing...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 139753,
      "author_name": "laurae2",
      "author_url": "",
      "post_date": "10/16/2016 10:09:51",
      "content": "<p>[quote=Patrick Fichou;139742]</p>\n\n<p>Sending a probability without the &quot;associated result&quot; sounds very strange to me. </p>\n\n<p>[/quote]</p>\n\n<p>But it is how it works. Log loss punishes probabilities which are overconfident and wrong. If you predict 0 and it is in fact 1, you are punished larger than predicting 0.5 for that 1. This competition is not about getting maximum accuracy, but about reasonable probability predictions. You can have 99%+ accuracy and have a log loss higher than someone with about 90%+ accuracy, if your predictions are not reasonable. Here, having 0 and 1 for everything is not reasonable predictions unless you know you have 100% accuracy out of the box.</p>\n\n<p>Then, to reproduce the &quot;associated result&quot; it is not hard (it is useless to reproduce the associated result for this competition): the closer you are to 0 the higher probability it is a cat. The closer you are to 1, the higher probability it is a dog: <a href=\"https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/details/evaluation\">https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/details/evaluation</a></p>\n\n<ul>\n<li>0.5 would be 50% probability of cat, 50% probability of dog. (unless you use non-linear probability thresholds)</li>\n<li>1 would be 0% cat, 100% dog.</li>\n<li>0 would be 100% cat, 0% dog.</li>\n</ul>",
      "votes": null,
      "replies": []
    },
    {
      "id": 139755,
      "author_name": "pfichou",
      "author_url": "",
      "post_date": "10/16/2016 10:37:17",
      "content": "<p>Thank you very much !!! (Merci beaucoup)</p>\n\n<p>I will try again according to your answer...</p>\n\n<p>Not clear in the description of the submission, much with your post !</p>\n\n<p>Patrick</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 154661,
      "author_name": "howardhuo",
      "author_url": "",
      "post_date": "01/07/2017 14:02:28",
      "content": "<p>As mentioned in <a href=\"https://github.com/fastai/courses/blob/master/deeplearning1/nbs/dogs_cats_redux.ipynb\">this tutorial</a>, log loss punishes predictions that are far off, so I would replace all ones with .95 and all zeros with .05 just for safe.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "137537": "According to the auto grader, my solution performs worse than the 0.5 benchmark. However, upon manual examination of the first 50 or so pictures, my submission seems to be mostly correct. Given the way the score is computed on the leaderboard (loss rather than percent correct), it's actually very hard to tell how many images the grader thinks I got correct. Can someone post a solution csv so I can see what I'm doing wrong?\r\n\r\nHere's the top of mine, with incorrect at ids 1,17,29\r\n\r\n    id,label\r\n    1,0\r\n    2,1\r\n    3,1\r\n    4,1\r\n    5,0\r\n    6,0\r\n    7,0\r\n    8,0\r\n    9,0\r\n    10,0\r\n    11,0\r\n    12,1\r\n    13,0\r\n    14,0\r\n    15,0\r\n    16,0\r\n    17,0\r\n    18,1\r\n    19,0\r\n    20,0\r\n    21,1\r\n    22,0\r\n    23,1\r\n    24,1\r\n    25,0\r\n    26,1\r\n    27,1\r\n    28,0\r\n    29,1\r\n    30,1\r\n    31,1\r\n    32,1\r\n    33,1\r\n    34,0\r\n    35,0\r\n    36,0\r\n    37,0\r\n    38,0\r\n    39,1\r\n    40,0\r\n    41,1\r\n    42,1\r\n    43,1\r\n    44,1\r\n    45,0\r\n    46,1\r\n    47,0\r\n    48,1\r\n    49,1\r\n    50,0",
    "137538": "Oh I see. If my predictions are binary classes (0,1), the logloss function results in some undefined values.",
    "137807": "As it has been stated in the Get Data section \"For each image in the test set, you should predict a probability that the image is a dog (1 = dog, 0 = cat).\", you should return the probability value but neither 0 or 1 (e.g. 0.0012, 0.892, etc)",
    "137868": "I must have missed this section (and actually I still can't find it). Where do you see \"but neither 0 or 1\"?",
    "137917": "[quote=Ben KTByte;137868]\r\n\r\nI must have missed this section (and actually I still can't find it). Where do you see \"but neither 0 or 1\"?\r\n\r\n[/quote]\r\n\r\nPlease check the description below the download data section https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/data",
    "138039": "Yeah I don't see the word \"neither\" anywhere on that page? I do see probability, but implicit in probability is it could be 0 or 1",
    "139739": "Hi to all, \r\n\r\nI have the same issue.\r\n\r\nMy submision file contains '0' for cat and '1' for dog and I get a ridiculous 32% as a result, though I've been checking and points some misclassifications but not 32%.\r\n\r\nI try with the probability and not the \"binary value'.",
    "139742": "Going back to my last comment : sending a probability (0.999, 0.995...) ok !\r\n\r\nBut what means this probality without the result  : Cat vs Dog, 0 vs 1, or whatever you want.\r\n\r\nSending a probability without the \"associated result\" sounds very strange to me.",
    "139751": "My last tries are worst and worst !\r\n\r\nI really don't understand the submit format Kaggle is expecting...\r\n\r\nI think I will give up though my score should be in the Top 10.\r\n\r\nDisappointing...",
    "139753": "[quote=Patrick Fichou;139742]\r\n\r\nSending a probability without the \"associated result\" sounds very strange to me. \r\n\r\n[/quote]\r\n\r\nBut it is how it works. Log loss punishes probabilities which are overconfident and wrong. If you predict 0 and it is in fact 1, you are punished larger than predicting 0.5 for that 1. This competition is not about getting maximum accuracy, but about reasonable probability predictions. You can have 99%+ accuracy and have a log loss higher than someone with about 90%+ accuracy, if your predictions are not reasonable. Here, having 0 and 1 for everything is not reasonable predictions unless you know you have 100% accuracy out of the box.\r\n\r\nThen, to reproduce the \"associated result\" it is not hard (it is useless to reproduce the associated result for this competition): the closer you are to 0 the higher probability it is a cat. The closer you are to 1, the higher probability it is a dog: https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition/details/evaluation\r\n\r\n* 0.5 would be 50% probability of cat, 50% probability of dog. (unless you use non-linear probability thresholds)\r\n* 1 would be 0% cat, 100% dog.\r\n* 0 would be 100% cat, 0% dog.",
    "139755": "Thank you very much !!! (Merci beaucoup)\r\n\r\nI will try again according to your answer...\r\n\r\nNot clear in the description of the submission, much with your post !\r\n\r\nPatrick",
    "154661": "As mentioned in [this tutorial][1], log loss punishes predictions that are far off, so I would replace all ones with .95 and all zeros with .05 just for safe.\r\n\r\n\r\n  [1]: https://github.com/fastai/courses/blob/master/deeplearning1/nbs/dogs_cats_redux.ipynb"
  },
  "source": "meta"
}