I am a little confused by the following statement in the description of the data:
\r\nColumn 2: Activity. Note that these are raw activity values and different data sets can have activity measured in different units.
\r\nDoes this mean we are studying different types of activity? If so, I would fit 15 separate models. The benchmark model fits one model to all of them.
\r\nIf these are not separate types of activity, could we please have the units so we can convert them all to the same units?
\r\nThanks.
", "messages": [ { "id": "13308", "postDate": "08/22/2012 03:15:40", "content": "I am a little confused by the following statement in the description of the data:
\r\nColumn 2: Activity. Note that these are raw activity values and different data sets can have activity measured in different units.
\r\nDoes this mean we are studying different types of activity? If so, I would fit 15 separate models. The benchmark model fits one model to all of them.
\r\nIf these are not separate types of activity, could we please have the units so we can convert them all to the same units?
\r\nThanks.
", "rawMarkdown": "", "votes": null }, { "id": "13311", "postDate": "08/22/2012 04:01:34", "content": "Each data set has its own activity measure. Some measures are used for multiple data sets, but the point is that activity values across data sets are not necessarily comparable. No, we will not be releasing the units of the measures, and they are not necessarily\r\n convertible from one to the other anyway.
\r\nI explicitly pointed this out on the data page so that people would not be surprised when Activity was not reported as binary 0 or 1, or as numbers only between 0 and 1.
\r\nAlso, the benchmark uses the same approach on all different data sets, but they are unique models.
", "rawMarkdown": "", "votes": null }, { "id": "13317", "postDate": "08/22/2012 10:40:41", "content": "[quote=jcnhvnhck;13311]
\r\nEach data set has its own activity measure. Some measures are used for multiple data sets, but the point is that activity values across data sets are not necessarily comparable. No, we will not be releasing the units of the measures, and they are not necessarily\r\n convertible from one to the other anyway.
\r\nI explicitly pointed this out on the data page so that people would not be surprised when Activity was not reported as binary 0 or 1, or as numbers only between 0 and 1.
\r\nAlso, the benchmark uses the same approach on all different data sets, but they are unique models.
\r\n[/quote]
\r\n\r\n
Really this is an "15 in 1" challenge, so perhaps 2 submits / day isn't enough if we want evaluate marginal progress in each data set. I think 5 submits / day would be reasonably.
", "rawMarkdown": "", "votes": null }, { "id": "13330", "postDate": "08/22/2012 13:07:27", "content": "I think 5 submits per day would be just enough to comfortably overfit the leaderboard.
\r\nIn all reality though, what the heck would you do with so many submissions? Don't you know your own cross-validation score before you submit? I suppose if the test set isn't sampled form the same population then you can squeeze a little info out of the feedback,\r\n but it would be a bit difficult with this mess of 15 sets.
", "rawMarkdown": "", "votes": null }, { "id": "13827", "postDate": "09/03/2012 07:57:50", "content": "Can we use different approaches per set? (i.e. random forest on one and a linear model on another). I observed some approaches perform better on some and worse on others.
", "rawMarkdown": "", "votes": null }, { "id": "13840", "postDate": "09/03/2012 18:17:00", "content": "I can't imagine they'd care if you used different approaches on different sets.
\r\nI also can't imagine you couldn't do better by using a customized combination of randomForest and linear modeling on all of them instead of just picking one or the other.
", "rawMarkdown": "", "votes": null }, { "id": "14235", "postDate": "09/11/2012 19:46:41", "content": "random forest would take a lot of time to train given the high dimensionality.
\r\nit does not work as well when you apply PCA to it vs. otherwise
Each data set has its own activity measure. Some measures are used for multiple data sets, but the point is that activity values across data sets are not necessarily comparable. No, we will not be releasing the units of the measures, and they are not necessarily\r\n convertible from one to the other anyway.
\r\nI explicitly pointed this out on the data page so that people would not be surprised when Activity was not reported as binary 0 or 1, or as numbers only between 0 and 1.
\r\nAlso, the benchmark uses the same approach on all different data sets, but they are unique models.
", "votes": null, "replies": [] }, { "id": 13317, "author_name": "blindape", "author_url": "", "post_date": "08/22/2012 10:40:41", "content": "[quote=jcnhvnhck;13311]
\r\nEach data set has its own activity measure. Some measures are used for multiple data sets, but the point is that activity values across data sets are not necessarily comparable. No, we will not be releasing the units of the measures, and they are not necessarily\r\n convertible from one to the other anyway.
\r\nI explicitly pointed this out on the data page so that people would not be surprised when Activity was not reported as binary 0 or 1, or as numbers only between 0 and 1.
\r\nAlso, the benchmark uses the same approach on all different data sets, but they are unique models.
\r\n[/quote]
\r\n\r\n
Really this is an "15 in 1" challenge, so perhaps 2 submits / day isn't enough if we want evaluate marginal progress in each data set. I think 5 submits / day would be reasonably.
", "votes": null, "replies": [] }, { "id": 13330, "author_name": "biasvariance", "author_url": "", "post_date": "08/22/2012 13:07:27", "content": "I think 5 submits per day would be just enough to comfortably overfit the leaderboard.
\r\nIn all reality though, what the heck would you do with so many submissions? Don't you know your own cross-validation score before you submit? I suppose if the test set isn't sampled form the same population then you can squeeze a little info out of the feedback,\r\n but it would be a bit difficult with this mess of 15 sets.
", "votes": null, "replies": [] }, { "id": 13827, "author_name": "julianvillaruz0", "author_url": "", "post_date": "09/03/2012 07:57:50", "content": "Can we use different approaches per set? (i.e. random forest on one and a linear model on another). I observed some approaches perform better on some and worse on others.
", "votes": null, "replies": [] }, { "id": 13840, "author_name": "biasvariance", "author_url": "", "post_date": "09/03/2012 18:17:00", "content": "I can't imagine they'd care if you used different approaches on different sets.
\r\nI also can't imagine you couldn't do better by using a customized combination of randomForest and linear modeling on all of them instead of just picking one or the other.
", "votes": null, "replies": [] }, { "id": 14235, "author_name": "rkirana", "author_url": "", "post_date": "09/11/2012 19:46:41", "content": "random forest would take a lot of time to train given the high dimensionality.
\r\nit does not work as well when you apply PCA to it vs. otherwise