{ "id": 24864, "title": "How to decide the best threshold?", "url": "/competitions/bosch-production-line-performance/discussion/24864", "author_name": "", "post_date": "2016-10-28T12:05:12.067Z", "votes": 1, "comment_count": 9, "views": 960, "content": "

I don't have any idea about it, and I will use the average threshold of my best CV and LB scores.

", "messages": [ { "id": "141788", "postDate": "10/28/2016 12:05:12", "content": "

I don't have any idea about it, and I will use the average threshold of my best CV and LB scores.

", "rawMarkdown": "I don't have any idea about it, and I will use the average threshold of my best CV and LB scores.", "votes": null }, { "id": "141792", "postDate": "10/28/2016 12:43:26", "content": "

Or use the threshold that gives best: CV score + 0.3 * LB score

", "rawMarkdown": "Or use the threshold that gives best: CV score + 0.3 * LB score", "votes": null }, { "id": "141823", "postDate": "10/28/2016 15:26:57", "content": "

@BreakfastPirate : Is there any intuition behind generating such a formula of threshold?

", "rawMarkdown": "BreakfastPirate : Is there any intuition behind generating such a formula of threshold?", "votes": null }, { "id": "141826", "postDate": "10/28/2016 15:53:27", "content": "

The public LB is based on 30% of the test data. The test data is the same size as the training data. So "CV score + 0.3 * LB score" is a weighted average of the two scores based on data size.

", "rawMarkdown": "The public LB is based on 30% of the test data. The test data is the same size as the training data. So \"CV score + 0.3 * LB score\" is a weighted average of the two scores based on data size.", "votes": null }, { "id": "141828", "postDate": "10/28/2016 16:06:34", "content": "

What is CV?

", "rawMarkdown": "What is CV?", "votes": null }, { "id": "141830", "postDate": "10/28/2016 16:09:25", "content": "

Cross Validation https://en.wikipedia.org/wiki/Cross-validation_(statistics)

", "rawMarkdown": "Cross Validation https://en.wikipedia.org/wiki/Cross-validation_(statistics)", "votes": null }, { "id": "141836", "postDate": "10/28/2016 16:31:01", "content": "

@BreakfastPirate : Then shouldn't it be (0.7* best CV score threshold + 0.3*best Public LB Score threshold)

", "rawMarkdown": "BreakfastPirate : Then shouldn't it be (0.7* best CV score threshold + 0.3*best Public LB Score threshold)", "votes": null }, { "id": "141838", "postDate": "10/28/2016 16:41:03", "content": "
    \n
  1. Pick a threshold
    2. \n
  2. Find the CV score for that threshold
    3. \n
  3. Find the LB score for that same threshold
    4. \n
  4. Calculate 1.0 * CV score for that threshold + 0.3 * LB score for that threshold.
    5. \n
  5. Repeat steps #1 to #4 for other thresholds
    6. \n
  6. Choose the threshold that yields the maximum weighted average (or weighted sum in this case since I didn't divide the answer by 1.3)
    7. \n
", "rawMarkdown": "1. Pick a threshold\r\n2. Find the CV score for that threshold\r\n3. Find the LB score for that same threshold\r\n4. Calculate 1.0 * CV score for that threshold + 0.3 * LB score for that threshold.\r\n5. Repeat steps #1 to #4 for other thresholds\r\n6. Choose the threshold that yields the maximum weighted average (or weighted sum in this case since I didn't divide the answer by 1.3)", "votes": null }, { "id": "141839", "postDate": "10/28/2016 16:43:05", "content": "

[quote=Abhilash Awasthi;141836]

\n\n

@BreakfastPirate : Then shouldn't it be (0.7* best CV score threshold + 0.3*best Public LB Score threshold)

\n\n

[/quote]

\n\n

If you are going to use that approach, I'd recommend: (1.0* best CV score threshold + 0.3*best Public LB Score threshold) / (1.0 + 0.3)

", "rawMarkdown": "[quote=Abhilash Awasthi;141836]\r\n\r\n@BreakfastPirate : Then shouldn't it be (0.7* best CV score threshold + 0.3*best Public LB Score threshold)\r\n\r\n[/quote]\r\n\r\nIf you are going to use that approach, I'd recommend: (1.0* best CV score threshold + 0.3*best Public LB Score threshold) / (1.0 + 0.3)", "votes": null }, { "id": "142152", "postDate": "10/31/2016 09:33:58", "content": "

So far I used something like what is in @joconnor notebook: https://www.kaggle.com/joconnor/bosch-production-line-performance/python-xgboost-starter-0-209-public-mcc

\n\n

I did not use any info from public LB as I have been burned in the past by overfitting to it.

\n\n

I may change my mind here when my current idea will prove not to work ;)

", "rawMarkdown": "So far I used something like what is in @joconnor notebook: https://www.kaggle.com/joconnor/bosch-production-line-performance/python-xgboost-starter-0-209-public-mcc\r\n\r\nI did not use any info from public LB as I have been burned in the past by overfitting to it.\r\n\r\nI may change my mind here when my current idea will prove not to work ;)", "votes": null } ], "comments": [ { "id": 141792, "author_name": "breakfastpirate", "author_url": "", "post_date": "10/28/2016 12:43:26", "content": "

Or use the threshold that gives best: CV score + 0.3 * LB score

", "votes": null, "replies": [] }, { "id": 141823, "author_name": "abhilashawasthi", "author_url": "", "post_date": "10/28/2016 15:26:57", "content": "

@BreakfastPirate : Is there any intuition behind generating such a formula of threshold?

", "votes": null, "replies": [] }, { "id": 141826, "author_name": "breakfastpirate", "author_url": "", "post_date": "10/28/2016 15:53:27", "content": "

The public LB is based on 30% of the test data. The test data is the same size as the training data. So "CV score + 0.3 * LB score" is a weighted average of the two scores based on data size.

", "votes": null, "replies": [] }, { "id": 141828, "author_name": "tekkrr", "author_url": "", "post_date": "10/28/2016 16:06:34", "content": "

What is CV?

", "votes": null, "replies": [] }, { "id": 141830, "author_name": "breakfastpirate", "author_url": "", "post_date": "10/28/2016 16:09:25", "content": "

Cross Validation https://en.wikipedia.org/wiki/Cross-validation_(statistics)

", "votes": null, "replies": [] }, { "id": 141836, "author_name": "abhilashawasthi", "author_url": "", "post_date": "10/28/2016 16:31:01", "content": "

@BreakfastPirate : Then shouldn't it be (0.7* best CV score threshold + 0.3*best Public LB Score threshold)

", "votes": null, "replies": [] }, { "id": 141838, "author_name": "breakfastpirate", "author_url": "", "post_date": "10/28/2016 16:41:03", "content": "
    \n
  1. Pick a threshold
    2. \n
  2. Find the CV score for that threshold
    3. \n
  3. Find the LB score for that same threshold
    4. \n
  4. Calculate 1.0 * CV score for that threshold + 0.3 * LB score for that threshold.
    5. \n
  5. Repeat steps #1 to #4 for other thresholds
    6. \n
  6. Choose the threshold that yields the maximum weighted average (or weighted sum in this case since I didn't divide the answer by 1.3)
    7. \n
", "votes": null, "replies": [] }, { "id": 141839, "author_name": "breakfastpirate", "author_url": "", "post_date": "10/28/2016 16:43:05", "content": "

[quote=Abhilash Awasthi;141836]

\n\n

@BreakfastPirate : Then shouldn't it be (0.7* best CV score threshold + 0.3*best Public LB Score threshold)

\n\n

[/quote]

\n\n

If you are going to use that approach, I'd recommend: (1.0* best CV score threshold + 0.3*best Public LB Score threshold) / (1.0 + 0.3)

", "votes": null, "replies": [] }, { "id": 142152, "author_name": "cpmpml", "author_url": "", "post_date": "10/31/2016 09:33:58", "content": "

So far I used something like what is in @joconnor notebook: https://www.kaggle.com/joconnor/bosch-production-line-performance/python-xgboost-starter-0-209-public-mcc

\n\n

I did not use any info from public LB as I have been burned in the past by overfitting to it.

\n\n

I may change my mind here when my current idea will prove not to work ;)

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "141788": "I don't have any idea about it, and I will use the average threshold of my best CV and LB scores.", "141792": "Or use the threshold that gives best: CV score + 0.3 * LB score", "141823": "BreakfastPirate : Is there any intuition behind generating such a formula of threshold?", "141826": "The public LB is based on 30% of the test data. The test data is the same size as the training data. So \"CV score + 0.3 * LB score\" is a weighted average of the two scores based on data size.", "141828": "What is CV?", "141830": "Cross Validation https://en.wikipedia.org/wiki/Cross-validation_(statistics)", "141836": "BreakfastPirate : Then shouldn't it be (0.7* best CV score threshold + 0.3*best Public LB Score threshold)", "141838": "1. Pick a threshold\r\n2. Find the CV score for that threshold\r\n3. Find the LB score for that same threshold\r\n4. Calculate 1.0 * CV score for that threshold + 0.3 * LB score for that threshold.\r\n5. Repeat steps #1 to #4 for other thresholds\r\n6. Choose the threshold that yields the maximum weighted average (or weighted sum in this case since I didn't divide the answer by 1.3)", "141839": "[quote=Abhilash Awasthi;141836]\r\n\r\n@BreakfastPirate : Then shouldn't it be (0.7* best CV score threshold + 0.3*best Public LB Score threshold)\r\n\r\n[/quote]\r\n\r\nIf you are going to use that approach, I'd recommend: (1.0* best CV score threshold + 0.3*best Public LB Score threshold) / (1.0 + 0.3)", "142152": "So far I used something like what is in @joconnor notebook: https://www.kaggle.com/joconnor/bosch-production-line-performance/python-xgboost-starter-0-209-public-mcc\r\n\r\nI did not use any info from public LB as I have been burned in the past by overfitting to it.\r\n\r\nI may change my mind here when my current idea will prove not to work ;)" }, "source": "meta" }