{ "id": 2433, "title": "How the training and test sets were constructed.", "url": "/competitions/MerckActivity/discussion/2433", "author_name": "", "post_date": "2012-08-24T14:53:35.373Z", "votes": 4, "comment_count": 4, "views": 3234, "content": "

 

\r\n
For each activity, the training/test set split is done by dates of testing.  That is, the training set consists of compounds assayed by a certain date, and the test set consists of\r\n compounds tested after that date. Therefore it is expected that the distribution of descriptors will not necessarily be the same between the training and test sets.\r\n
\r\n
 
\r\n
We find "time-split" validation is a much more realistic simulation of true prospective prediction in terms of R2. We find construction of test sets by random sampling gives R2's\r\n that are much too optimistic. That is what makes these data sets more of a challenge.
\r\n
 
\r\n
I will upload a cartoon to illustrate.
\r\n
 
\r\n
 
", "messages": [ { "id": "13404", "postDate": "08/24/2012 14:53:35", "content": "

 

\r\n
For each activity, the training/test set split is done by dates of testing.  That is, the training set consists of compounds assayed by a certain date, and the test set consists of\r\n compounds tested after that date. Therefore it is expected that the distribution of descriptors will not necessarily be the same between the training and test sets.\r\n
\r\n
 
\r\n
We find "time-split" validation is a much more realistic simulation of true prospective prediction in terms of R2. We find construction of test sets by random sampling gives R2's\r\n that are much too optimistic. That is what makes these data sets more of a challenge.
\r\n
 
\r\n
I will upload a cartoon to illustrate.
\r\n
 
\r\n
 
", "rawMarkdown": "", "votes": null }, { "id": "13427", "postDate": "08/24/2012 20:56:34", "content": "

Thanks Bob. That's a well done infographic. I understand why you split the data this way. It might be worth a section in the competition information so people don't miss this point in the future.

\r\n

I find this refreshing in general, but I'd add a couple caveats:

\r\n
    \r\n
  1. \r\n

    It would be possible to carefully use leaderboard feedback to improve your submissions. This will be hampered by the 15 sets however.

    \r\n
  2. \r\n

    It's still only quasi realistic since we know what molecules you did test after the split date. To simulate your internal modelings, you'd have to provide test data for all the molecules that could have been tested instead of just those you actually tested.

    \r\n
", "rawMarkdown": "", "votes": null }, { "id": "13428", "postDate": "08/24/2012 21:01:30", "content": "

Great suggestion, Shea! I've added that to the Data page of the competition information.

", "rawMarkdown": "", "votes": null }, { "id": "13433", "postDate": "08/24/2012 23:15:43", "content": "

Even a blind (ape) can see ...

\r\n

 

\r\n

Blind Ape: (at http://www.kaggle.com/c/MerckActivity/forums/t/2380/duplicated-variables-and-molecules?limit=all)

\r\n

How the test molecules are selected? randomly or test set are an specific molecules type that the promoter has interest in them?

\r\n

The test set generation must be detailed in each challenge profile.

\r\n

Shea Parkes answer:

\r\n

Is it really that hard to figure out if the test and training sets are random samples from the same population? I realize it's fairly high dimensional data, but you can reduce those dimensions and make visual comparisons.

\r\n

I don't recall reading anything that says they have to confirm how they made the test set.

\r\n

Furthermore, I don't think I'd trust anything that was said since there are many cooks in the kitchen (Kaggle & Merck at least). It's not that hard to run some tests yourself.

\r\n

Shea Parkes (After Competition Admin post):

\r\n

Thanks Bob. That's a well done infographic. I understand why you split the data this way. It might be worth a section in the competition information so people don't miss this point in the future.\r\n(Oh, that was my suggestion yesterday). Shea's opinion was totally different !?

\r\n

Competion Admin (jcnhvnhck) gives the cookie to Shea:

\r\n

Great suggestion, Shea! I've added that to the Data page of the competition information.

\r\n

 

", "rawMarkdown": "", "votes": null }, { "id": "13802", "postDate": "09/02/2012 16:05:13", "content": "

Wouldn't it make sense to publish the date values of individual training samples, or at least the order they were processed in?

\r\n

At least we would be able to grasp the trends in the change over time from the training set and NOT from the leaderboard feedback.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 13427, "author_name": "biasvariance", "author_url": "", "post_date": "08/24/2012 20:56:34", "content": "

Thanks Bob. That's a well done infographic. I understand why you split the data this way. It might be worth a section in the competition information so people don't miss this point in the future.

\r\n

I find this refreshing in general, but I'd add a couple caveats:

\r\n
    \r\n
  1. \r\n

    It would be possible to carefully use leaderboard feedback to improve your submissions. This will be hampered by the 15 sets however.

    \r\n
  2. \r\n

    It's still only quasi realistic since we know what molecules you did test after the split date. To simulate your internal modelings, you'd have to provide test data for all the molecules that could have been tested instead of just those you actually tested.

    \r\n
", "votes": null, "replies": [] }, { "id": 13428, "author_name": "joycenv", "author_url": "", "post_date": "08/24/2012 21:01:30", "content": "

Great suggestion, Shea! I've added that to the Data page of the competition information.

", "votes": null, "replies": [] }, { "id": 13433, "author_name": "blindape", "author_url": "", "post_date": "08/24/2012 23:15:43", "content": "

Even a blind (ape) can see ...

\r\n

 

\r\n

Blind Ape: (at http://www.kaggle.com/c/MerckActivity/forums/t/2380/duplicated-variables-and-molecules?limit=all)

\r\n

How the test molecules are selected? randomly or test set are an specific molecules type that the promoter has interest in them?

\r\n

The test set generation must be detailed in each challenge profile.

\r\n

Shea Parkes answer:

\r\n

Is it really that hard to figure out if the test and training sets are random samples from the same population? I realize it's fairly high dimensional data, but you can reduce those dimensions and make visual comparisons.

\r\n

I don't recall reading anything that says they have to confirm how they made the test set.

\r\n

Furthermore, I don't think I'd trust anything that was said since there are many cooks in the kitchen (Kaggle & Merck at least). It's not that hard to run some tests yourself.

\r\n

Shea Parkes (After Competition Admin post):

\r\n

Thanks Bob. That's a well done infographic. I understand why you split the data this way. It might be worth a section in the competition information so people don't miss this point in the future.\r\n(Oh, that was my suggestion yesterday). Shea's opinion was totally different !?

\r\n

Competion Admin (jcnhvnhck) gives the cookie to Shea:

\r\n

Great suggestion, Shea! I've added that to the Data page of the competition information.

\r\n

 

", "votes": null, "replies": [] }, { "id": 13802, "author_name": "anaconda", "author_url": "", "post_date": "09/02/2012 16:05:13", "content": "

Wouldn't it make sense to publish the date values of individual training samples, or at least the order they were processed in?

\r\n

At least we would be able to grasp the trends in the change over time from the training set and NOT from the leaderboard feedback.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "13404": "", "13427": "", "13428": "", "13433": "", "13802": "" }, "source": "meta" }