{ "id": 8346, "title": "Area Under Curve question", "url": "/competitions/acquire-valued-shoppers-challenge/discussion/8346", "author_name": "", "post_date": "2014-05-31T21:20:09.160Z", "votes": null, "comment_count": 3, "views": 1646, "content": "

I recently started working on this challenge, and I have a question that I hope someone can help me out with.

\n

My first submission was using a very simple model (not even using the transaction data) which unsurprisingly scored rather poorly (0.50959).

\n

As my next step, I decided to implement a cross-validation step, by splitting the training data into 6 parts, alternately training the model on 5 of them, and then scoring it on the remaining part. This gives 6 sets of predictions, which generate 6 ROC curves.

\n

My cross-validation and ROC code is adapted from http://scikit-learn.org/stable/auto_examples/plot_roc_crossval.html#example-plot-roc-crossval-py.

\n

The funny thing is, when using this code, the AUC numbers I get back are significantly higher (0.67 +/- 0.02) than the score my submission got, and even higher than the top of the current leaderboard. I can think of a few possible explanations why this could be happening, but none of them seem plausible to me:

\n
    \n
  1. scikit-learn and Kaggle are calculating the ROC and/or AUC differently. Seems unlikely, because it's not that complicated of a measure.
    2. \n
  2. I'm overfitting my model. Seems very unlikely since I'm using a simple classifier on the training data, without tweaking parameters, and without using the transaction data.
    3. \n
  3. The disparity comes from the difference between the training data and the test data. This is possible, since the two datasets are from two nonintersecting (but neighboring) time intervals. However, such a large change seems unlikely to me. On the other hand, it would explain something else I've been wondering about: it strikes me that the leading scores are still quite low, effectively only 25% better than pure chance (0.62 / 0.50). I don't have much experience in machine learning, but I would intuitively think that with the amount of data available in this challenge, that it should be possible to score substantially higher, assuming of course the data is relevant.
    4. \n
\n

I guess what I'm asking is, has anyone else done the same exercise of calculating AUC on part of the training data, and seen a similar gap? Or, is there a much simpler explanation that I'm missing?

", "messages": [ { "id": "47412", "postDate": "05/31/2014 21:20:09", "content": "

I recently started working on this challenge, and I have a question that I hope someone can help me out with.

\n

My first submission was using a very simple model (not even using the transaction data) which unsurprisingly scored rather poorly (0.50959).

\n

As my next step, I decided to implement a cross-validation step, by splitting the training data into 6 parts, alternately training the model on 5 of them, and then scoring it on the remaining part. This gives 6 sets of predictions, which generate 6 ROC curves.

\n

My cross-validation and ROC code is adapted from http://scikit-learn.org/stable/auto_examples/plot_roc_crossval.html#example-plot-roc-crossval-py.

\n

The funny thing is, when using this code, the AUC numbers I get back are significantly higher (0.67 +/- 0.02) than the score my submission got, and even higher than the top of the current leaderboard. I can think of a few possible explanations why this could be happening, but none of them seem plausible to me:

\n
    \n
  1. scikit-learn and Kaggle are calculating the ROC and/or AUC differently. Seems unlikely, because it's not that complicated of a measure.
    2. \n
  2. I'm overfitting my model. Seems very unlikely since I'm using a simple classifier on the training data, without tweaking parameters, and without using the transaction data.
    3. \n
  3. The disparity comes from the difference between the training data and the test data. This is possible, since the two datasets are from two nonintersecting (but neighboring) time intervals. However, such a large change seems unlikely to me. On the other hand, it would explain something else I've been wondering about: it strikes me that the leading scores are still quite low, effectively only 25% better than pure chance (0.62 / 0.50). I don't have much experience in machine learning, but I would intuitively think that with the amount of data available in this challenge, that it should be possible to score substantially higher, assuming of course the data is relevant.
    4. \n
\n

I guess what I'm asking is, has anyone else done the same exercise of calculating AUC on part of the training data, and seen a similar gap? Or, is there a much simpler explanation that I'm missing?

", "rawMarkdown": "", "votes": null }, { "id": "47414", "postDate": "05/31/2014 21:26:01", "content": "

[quote=marks;47412]

\n
    \n
  1. The disparity comes from the difference between the training data and the test data. This is possible, since the two datasets are from two nonintersecting (but neighboring) time intervals. 
    2. \n
\n

[/quote]

\n

I think it is this. See also: http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/7671/evaluation/

", "rawMarkdown": "", "votes": null }, { "id": "47415", "postDate": "05/31/2014 21:29:17", "content": "

I've done previous competitions, and scoring is often an issue. So, most likely 3.

\n

Real-world data is messy and hard to predict. Marketers would kill for a 25% better than chance estimate. Remember, Netflix offered a $1m prize for a 10% improvement in movie ratings.

\n

But I'm just getting started today also, so take it with a grain of salt.

", "rawMarkdown": "", "votes": null }, { "id": "47418", "postDate": "05/31/2014 21:56:21", "content": "

Triskelion, thanks for the link, very helpful (and relieving) to read :-)

\n

inversion: what you say makes sense as well, I guess I was tripped up by the high AUC scores in the example I linked to, which is of course a pedagogical example..

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 47414, "author_name": "triskelion", "author_url": "", "post_date": "05/31/2014 21:26:01", "content": "

[quote=marks;47412]

\n
    \n
  1. The disparity comes from the difference between the training data and the test data. This is possible, since the two datasets are from two nonintersecting (but neighboring) time intervals. 
    2. \n
\n

[/quote]

\n

I think it is this. See also: http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/7671/evaluation/

", "votes": null, "replies": [] }, { "id": 47415, "author_name": "inversion", "author_url": "", "post_date": "05/31/2014 21:29:17", "content": "

I've done previous competitions, and scoring is often an issue. So, most likely 3.

\n

Real-world data is messy and hard to predict. Marketers would kill for a 25% better than chance estimate. Remember, Netflix offered a $1m prize for a 10% improvement in movie ratings.

\n

But I'm just getting started today also, so take it with a grain of salt.

", "votes": null, "replies": [] }, { "id": 47418, "author_name": "marks1", "author_url": "", "post_date": "05/31/2014 21:56:21", "content": "

Triskelion, thanks for the link, very helpful (and relieving) to read :-)

\n

inversion: what you say makes sense as well, I guess I was tripped up by the high AUC scores in the example I linked to, which is of course a pedagogical example..

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "47412": "", "47414": "", "47415": "", "47418": "" }, "source": "meta" }