{ "id": 7845, "title": "Cross Validation fails to evaluate the model", "url": "/competitions/acquire-valued-shoppers-challenge/discussion/7845", "author_name": "", "post_date": "2014-04-24T03:56:34.597Z", "votes": null, "comment_count": 9, "views": 3836, "content": "

Hi All,

\n

I am wondering how you guys evaluate the model performance? I test a few models with features discussed so far on this forum (has bought category within 30 days of the offer purchase). Random Forest has the highest 10-fold cross validation score, but its score on the test set is the lowest among others such as logistic regression and gbm. 

\n

Anyone experiencing the same? hopefully no so I just need to double check my code. 

\n

I understand the training and the testing sets have some fundamental difference. Just wondering what's the best strategy to evaluate model performance in this case?

\n

Best Regards,

\n

Y

", "messages": [ { "id": "42893", "postDate": "04/24/2014 03:56:34", "content": "

Hi All,

\n

I am wondering how you guys evaluate the model performance? I test a few models with features discussed so far on this forum (has bought category within 30 days of the offer purchase). Random Forest has the highest 10-fold cross validation score, but its score on the test set is the lowest among others such as logistic regression and gbm. 

\n

Anyone experiencing the same? hopefully no so I just need to double check my code. 

\n

I understand the training and the testing sets have some fundamental difference. Just wondering what's the best strategy to evaluate model performance in this case?

\n

Best Regards,

\n

Y

", "rawMarkdown": "", "votes": null }, { "id": "42942", "postDate": "04/24/2014 15:12:18", "content": "

I think others see this as well. The test and train sets are from different time periods. You might try adopting a cross-validation strategy that reflects this, rather than just randomly sampling.

", "rawMarkdown": "", "votes": null }, { "id": "42984", "postDate": "04/25/2014 04:15:43", "content": "

[quote=neggert;42942]

\n

I think others see this as well. The test and train sets are from different time periods. You might try adopting a cross-validation strategy that reflects this, rather than just randomly sampling.

\n

[/quote]

\n\n

Thanks neggert. I will think of a better validation method.

", "rawMarkdown": "", "votes": null }, { "id": "43093", "postDate": "04/26/2014 18:56:26", "content": "

One method that seems to work slightly better is a time-based split, using March for training and April for testing.

\n\n

However, the model that's best for April may not turn out to be the best for May/June/July.

\n\n

Proper cross-validation is tough in this competition.

", "rawMarkdown": "", "votes": null }, { "id": "43124", "postDate": "04/27/2014 07:07:26", "content": "

Hey Zach, can you quantify slightly better? I haven't tried a time-based split yet.

\n

What I have tried so far is to order the train set with respect to offers and then do CV without shuffling. Basically I was trying to simulate the fact that most the offers in the test set are not present in the train set.

\n

In the end it brought my CV scores down from +- 0.72 to 0.628, which in the end became a 0.58 in the ladder board. Which is quite better, however I can't say if improvements in this kind of CV will necessarily translate to ladder board improvement.

", "rawMarkdown": "", "votes": null }, { "id": "43211", "postDate": "04/28/2014 12:37:12", "content": "

Basically, I made a training set with days from March, and a test set with days from April.  It worked all right, but not awesome.

", "rawMarkdown": "", "votes": null }, { "id": "43253", "postDate": "04/29/2014 00:42:02", "content": "

@fernando nogueira: I noticed similar AUC numbers for Random CV, first vs second month CV and Kaggle leaderboard scores.

\n

Also, the public leaderboard score is calculated on only 20% of the test set, so I expect the final result to be quite different...

", "rawMarkdown": "", "votes": null }, { "id": "48703", "postDate": "06/05/2014 10:51:24", "content": "

This thread is hibernating :)

\n\n

Did anyone get any break through for better CV testing.

", "rawMarkdown": "", "votes": null }, { "id": "48712", "postDate": "06/05/2014 13:05:30", "content": "

not really a break-through, but one possible explanation for some of the leakage is that there are several discount offers that are given to the same product.

\n

this basic exploratory code:

\n\n\n

import pandas as pd

\n

offers = pd.read_csv("offers.csv")
trainHistory = pd.read_csv("trainHistory.csv")
testHistory = pd.read_csv("testHistory.csv")

\n

train = pd.merge(trainHistory, offers, on = 'offer', how='left',sort=False)
test = pd.merge(testHistory , offers, on = 'offer', how='left',sort=False)

train['productID'] = train['category'].astype(str) + '_' + train['company'].astype(str) + '_' + train['brand'].astype(str)
test['productID'] = test['category'].astype(str) + '_' + test['company'].astype(str) + '_' + test['brand'].astype(str)

train['brandID'] = train['company'].astype(str) + '_' + train['brand'].astype(str)
test['brandID'] = test['company'].astype(str) + '_' + test['brand'].astype(str)

print 'train-test split:'
for col in ( 'offer', 'chain', 'market', 'id', 'category', 'productID', 'brandID'):

\n


intersection = sum( train[col].drop_duplicates().isin( test[col].drop_duplicates()) )
trainOnly = len( train[col].drop_duplicates()) - intersection
testOnly = len( test[col].drop_duplicates() ) - intersection
print " [train only, both train and test, test only] : [" + str(trainOnly) + ',' + str(intersection) + ',' + str(testOnly) + '] for ' + str(col)

\n\n

provides the following output:

\n\n

train-test split:
[train only, both train and test, test only] : [8,16,13] for offer
[train only, both train and test, test only] : [3,127,4] for chain
[train only, both train and test, test only] : [0,34,0] for market
[train only, both train and test, test only] : [160057,0,151484] for id
[train only, both train and test, test only] : [5,8,7] for category
[train only, both train and test, test only] : [6,8,7] for productID
[train only, both train and test, test only] : [5,7,7] for brandID

", "rawMarkdown": "", "votes": null }, { "id": "49116", "postDate": "06/13/2014 17:43:16", "content": "

Hi Selfish Gene, can you provide some description please.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 42942, "author_name": "neggert", "author_url": "", "post_date": "04/24/2014 15:12:18", "content": "

I think others see this as well. The test and train sets are from different time periods. You might try adopting a cross-validation strategy that reflects this, rather than just randomly sampling.

", "votes": null, "replies": [] }, { "id": 42984, "author_name": "yzhao0527", "author_url": "", "post_date": "04/25/2014 04:15:43", "content": "

[quote=neggert;42942]

\n

I think others see this as well. The test and train sets are from different time periods. You might try adopting a cross-validation strategy that reflects this, rather than just randomly sampling.

\n

[/quote]

\n\n

Thanks neggert. I will think of a better validation method.

", "votes": null, "replies": [] }, { "id": 43093, "author_name": "zachmayer", "author_url": "", "post_date": "04/26/2014 18:56:26", "content": "

One method that seems to work slightly better is a time-based split, using March for training and April for testing.

\n\n

However, the model that's best for April may not turn out to be the best for May/June/July.

\n\n

Proper cross-validation is tough in this competition.

", "votes": null, "replies": [] }, { "id": 43124, "author_name": "fnogueira", "author_url": "", "post_date": "04/27/2014 07:07:26", "content": "

Hey Zach, can you quantify slightly better? I haven't tried a time-based split yet.

\n

What I have tried so far is to order the train set with respect to offers and then do CV without shuffling. Basically I was trying to simulate the fact that most the offers in the test set are not present in the train set.

\n

In the end it brought my CV scores down from +- 0.72 to 0.628, which in the end became a 0.58 in the ladder board. Which is quite better, however I can't say if improvements in this kind of CV will necessarily translate to ladder board improvement.

", "votes": null, "replies": [] }, { "id": 43211, "author_name": "zachmayer", "author_url": "", "post_date": "04/28/2014 12:37:12", "content": "

Basically, I made a training set with days from March, and a test set with days from April.  It worked all right, but not awesome.

", "votes": null, "replies": [] }, { "id": 43253, "author_name": "npetitclerc", "author_url": "", "post_date": "04/29/2014 00:42:02", "content": "

@fernando nogueira: I noticed similar AUC numbers for Random CV, first vs second month CV and Kaggle leaderboard scores.

\n

Also, the public leaderboard score is calculated on only 20% of the test set, so I expect the final result to be quite different...

", "votes": null, "replies": [] }, { "id": 48703, "author_name": "chitrasen", "author_url": "", "post_date": "06/05/2014 10:51:24", "content": "

This thread is hibernating :)

\n\n

Did anyone get any break through for better CV testing.

", "votes": null, "replies": [] }, { "id": 48712, "author_name": "selfishgene", "author_url": "", "post_date": "06/05/2014 13:05:30", "content": "

not really a break-through, but one possible explanation for some of the leakage is that there are several discount offers that are given to the same product.

\n

this basic exploratory code:

\n\n\n

import pandas as pd

\n

offers = pd.read_csv("offers.csv")
trainHistory = pd.read_csv("trainHistory.csv")
testHistory = pd.read_csv("testHistory.csv")

\n

train = pd.merge(trainHistory, offers, on = 'offer', how='left',sort=False)
test = pd.merge(testHistory , offers, on = 'offer', how='left',sort=False)

train['productID'] = train['category'].astype(str) + '_' + train['company'].astype(str) + '_' + train['brand'].astype(str)
test['productID'] = test['category'].astype(str) + '_' + test['company'].astype(str) + '_' + test['brand'].astype(str)

train['brandID'] = train['company'].astype(str) + '_' + train['brand'].astype(str)
test['brandID'] = test['company'].astype(str) + '_' + test['brand'].astype(str)

print 'train-test split:'
for col in ( 'offer', 'chain', 'market', 'id', 'category', 'productID', 'brandID'):

\n


intersection = sum( train[col].drop_duplicates().isin( test[col].drop_duplicates()) )
trainOnly = len( train[col].drop_duplicates()) - intersection
testOnly = len( test[col].drop_duplicates() ) - intersection
print " [train only, both train and test, test only] : [" + str(trainOnly) + ',' + str(intersection) + ',' + str(testOnly) + '] for ' + str(col)

\n\n

provides the following output:

\n\n

train-test split:
[train only, both train and test, test only] : [8,16,13] for offer
[train only, both train and test, test only] : [3,127,4] for chain
[train only, both train and test, test only] : [0,34,0] for market
[train only, both train and test, test only] : [160057,0,151484] for id
[train only, both train and test, test only] : [5,8,7] for category
[train only, both train and test, test only] : [6,8,7] for productID
[train only, both train and test, test only] : [5,7,7] for brandID

", "votes": null, "replies": [] }, { "id": 49116, "author_name": "anshul", "author_url": "", "post_date": "06/13/2014 17:43:16", "content": "

Hi Selfish Gene, can you provide some description please.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "42893": "", "42942": "", "42984": "", "43093": "", "43124": "", "43211": "", "43253": "", "48703": "", "48712": "", "49116": "" }, "source": "meta" }