{
  "id": 24784,
  "title": "What's your prediction's failure rate?",
  "url": "/competitions/bosch-production-line-performance/discussion/24784",
  "author_name": "",
  "post_date": "2016-10-26T17:46:35.753Z",
  "votes": null,
  "comment_count": 1,
  "views": 268,
  "content": "<p>I checked most of my prediction. I found there were only about 2000 fails, even before I use Faron's feature. That gives me a bad feeling. 2000 fails out of 1.2 mil is 0.16%, which is about 30% of 0.5% failure rate of the training data. The LB is using 30% test data, and I am doing okay with 2000 fails. Is this a sign that I am going to suck in PB?!!! :0</p>",
  "messages": [
    {
      "id": "141367",
      "postDate": "10/26/2016 17:46:35",
      "content": "<p>I checked most of my prediction. I found there were only about 2000 fails, even before I use Faron's feature. That gives me a bad feeling. 2000 fails out of 1.2 mil is 0.16%, which is about 30% of 0.5% failure rate of the training data. The LB is using 30% test data, and I am doing okay with 2000 fails. Is this a sign that I am going to suck in PB?!!! :0</p>",
      "rawMarkdown": "I checked most of my prediction. I found there were only about 2000 fails, even before I use Faron's feature. That gives me a bad feeling. 2000 fails out of 1.2 mil is 0.16%, which is about 30% of 0.5% failure rate of the training data. The LB is using 30% test data, and I am doing okay with 2000 fails. Is this a sign that I am going to suck in PB?!!! :0",
      "votes": null
    },
    {
      "id": "141371",
      "postDate": "10/26/2016 18:13:50",
      "content": "<p>The evaluation metric is very sensitive to the number of false positives/ false negatives in your prediction. You identified about 2000 fails by (explicitly or implicitly) choosing a threshold of a probability over which you consider a product as being positive. Most probably, this means that if you had a lower threshold and identified, say, 2500 positives, you would get so many additional false positives that your score would decrease, even if in the process you identified more true positives.</p>\n\n<p>It has been discussed again in the forum: the general strategy would be to find as many cases as possible for which you are quite certain they are positive. Most probably their number will be substantially lower than 0.5% of the test set. </p>",
      "rawMarkdown": "The evaluation metric is very sensitive to the number of false positives/ false negatives in your prediction. You identified about 2000 fails by (explicitly or implicitly) choosing a threshold of a probability over which you consider a product as being positive. Most probably, this means that if you had a lower threshold and identified, say, 2500 positives, you would get so many additional false positives that your score would decrease, even if in the process you identified more true positives.\r\n\r\nIt has been discussed again in the forum: the general strategy would be to find as many cases as possible for which you are quite certain they are positive. Most probably their number will be substantially lower than 0.5% of the test set.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 141371,
      "author_name": "panosc",
      "author_url": "",
      "post_date": "10/26/2016 18:13:50",
      "content": "<p>The evaluation metric is very sensitive to the number of false positives/ false negatives in your prediction. You identified about 2000 fails by (explicitly or implicitly) choosing a threshold of a probability over which you consider a product as being positive. Most probably, this means that if you had a lower threshold and identified, say, 2500 positives, you would get so many additional false positives that your score would decrease, even if in the process you identified more true positives.</p>\n\n<p>It has been discussed again in the forum: the general strategy would be to find as many cases as possible for which you are quite certain they are positive. Most probably their number will be substantially lower than 0.5% of the test set. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "141367": "I checked most of my prediction. I found there were only about 2000 fails, even before I use Faron's feature. That gives me a bad feeling. 2000 fails out of 1.2 mil is 0.16%, which is about 30% of 0.5% failure rate of the training data. The LB is using 30% test data, and I am doing okay with 2000 fails. Is this a sign that I am going to suck in PB?!!! :0",
    "141371": "The evaluation metric is very sensitive to the number of false positives/ false negatives in your prediction. You identified about 2000 fails by (explicitly or implicitly) choosing a threshold of a probability over which you consider a product as being positive. Most probably, this means that if you had a lower threshold and identified, say, 2500 positives, you would get so many additional false positives that your score would decrease, even if in the process you identified more true positives.\r\n\r\nIt has been discussed again in the forum: the general strategy would be to find as many cases as possible for which you are quite certain they are positive. Most probably their number will be substantially lower than 0.5% of the test set."
  },
  "source": "meta"
}