I have a quick question after looking through Shea Parkes' visualization package (which is awesome, btw!)
\r\nIs it permissible for us to exploit knowledge of the test set distribution to train models specifically for that set, or are we expected to model only using knowledge that can be inferred from the training set? I can make a case either way, so instead of\r\n guessing, I'd appreciate if someone can clarify.
", "messages": [ { "id": 15388, "postDate": "2012-10-10T12:00:15.263Z", "content": "[quote=Marcin Pionnier;15382]
\r\nI think it depends on the area. It is not clear for me for this competition. However in case of building predictive model for example in marketing campaign or churn - test data are available. In case of estimating risk of potential claim for insurance policy\r\n for new client - not available.
\r\n[/quote]
\r\nThere is some confusion with the "Test Data" terminology here.
\r\nTest data is any future data on which the predictive model is to be applied.
\r\nTo spell out in concrete terms:
\r\nLets say Training Sample spans 01JAN2011 - 31DEC2011. Model trained using this data to be applied & tested on data sampled between 01JAN2012 - 31MAR2012. Here after data from 01JAN2012-31MAR2012 will be refered to as the TEST data.
\r\nThe competition started on 01MAY2012. So we have access to training data & an out of time Test Data.
\r\nJust because test data & training data are made available at the same time does not mean that one can use any information from the Test data to incorporate into ones model.
\r\nThe organisers should also share some blame for this fiasco. If time dimension is important to your problem then you should consider providing timestamps or any integer valued field which can be used to chronologically order the training dataset, so that\r\n users can build there own validation schemes. For ex: if the training data spans 12-months, a user could use the 1st 9 months to train & the remaining 3 months to validate/tune the model. This would eliminate the need/urge to incorporate test data information.
\r\nMarketting campains are not analogous to this competition since one usually builds a bespoke model for marketting a particular product, so we cannot use it as a case to defend using test data here.
", "votes": 2 }, { "id": 15368, "postDate": "2012-10-09T21:37:45.667Z", "content": "[quote=jcnhvnhck;15367]
\r\nGiven how far we are into the competition, it would be unfair to change the rules at this point. So, no, you will not be disqualified or penalized if you used test set distributions since it was not explicitly disallowed at the beginning of the competition.
\r\n[/quote]
\r\nThanks. And just to put it out there: I'd welcome the chance to compete in a followup competition that has a different set of goals as well.
", "votes": 1 }, { "id": 15362, "postDate": "2012-10-09T18:18:14.153Z", "content": "Edit: The above post make most of this void.
\r\n\r\n
The feature descriptions for this competition are masked so I am not entirely sure of how much information of test molecules are typically known. But if I make my own assumptions...
\r\nMerck and most pharma companies test thousand of molecules for activities. The point here would be to improve the success rate of finding molecules with a certain activation. That being said, they already have the features for each molecule prior to testing\r\n for activity. The distribution of test molecules would be known and optimizing models to train on similar molecules would make sense.
\r\nJudging by the temporal grouping present in the train and test data, there are classes of molecules that are tested for activity. Which is some of the challenge of this competition, the shifts between the train and test is due to a new class of molecule\r\n being testing and we need to predict how it will react.
\r\nIn many real world applications this approach would not make sense. You would need enough test points to determine the distribution of the test data set. Training a model will take time so there would not be instant results for the new test data. Pharma\r\n molecule testing would not be limited by these needs. There is more of a financial incentive to waiting and training a model to predict which molecules to test than testing them all.
\r\nAs Shea alluded to above, we have used the distribution of the test features to train model optimized for the test data, but we have not used feedback from the leader board to influence our modeling decisions. Activity 3 for example, the train and test molecules\r\n are fairly different. Having 15 activities help mask some of the gaming that can be result from using leader board results. If you follow the HHP competition to compete at the top it appears you have to include some information derived from the leader board\r\n into your model. Maybe this will end up being part of the key to winning this competition, which will produce models are that are not useful for Merck in real application. I hope this is not the case, but considering that finik was behaving badly and that\r\n there are 15 activities. There could be more slave accounts in the depths of the player ranking or just dummy submissions that were testing outcomes of individual activities to optimize the model.
", "votes": 1 }, { "id": 15436, "postDate": "2012-10-11T17:06:59.010Z", "content": "agree with Shea on this - instead of saynig 0.47 or 0.46 or 0.45, just 0.5 or 0.4 would be good
\r\nThat way we can avoid the problems mentioned.
\r\nkaggle folks: Point to note - Shea is 'spot on' here and we need to implement this real soon for future competitions
Subsequent evaluation is also problematic. Classic example is the Impermium competition. The subsequent dataset was totally unrepresentative of the initial training set rendering many efforts useless. It was not that models were not generalizable but just\r\n that it was totally different from train. Folks jumped from the second decile to first and vice - versa.
\r\nUnless care is taken, subsequent evaluation methods are also useless
\r\n
[quote=Shea Parkes;15363]
\r\n[quote=jcnhvnhck;15361]
\r\nHi Everyone,
\r\nI checked with the competition sponsor and there is a strong preference for not using the test set distribution in creating the models since the distribution information for new molecules will not necessarily be known in practice.
\r\n[/quote]
\r\nIt's a bit late to redo our efforts. I would prefer this sort of issue be a "rule" instead of "preference" however. I don't think we're likely to finish in the money, but I would be disappointed if this invalidated our efforts.
\r\nWere this rule to be in effect in future competitions, and you are willing to enforce it during evaluation, I will respect Kaggle's wishes.
[/quote]
" }, { "id": 15428, "postDate": "2012-10-11T14:28:58.627Z", "content": "I agree the leaderboard drives competition which drives the innovation; I couldn't see this working without it. I just think you'd get the same benefit with less detailed feedback. Even just one less significant digit would be reasonable to me.
\r\nI am more curious than usually how the private leaderboard flop will fall on this competition. We've still got the advantage of less submissions for now.
" }, { "id": 15426, "postDate": "2012-10-11T14:12:51.753Z", "content": "same here - I am 9th without using leaderboard data.
\r\nShea: I think the greatest thing about kaggle is the leaderboard. That is what gets us data scientists excited to participate. Without a leaderboard we would not see such improvements in scores
" }, { "id": 15391, "postDate": "2012-10-10T12:56:13.720Z", "content": "I understand Merck wouldn't likely know the exact distribution of their test data; I never implied they would. Neil gave an argument that they would have some general ideas above. However, this is a contest. To win, you pretty much exploit any source of\r\n advantage allowable within the rules; I feel no shame in what I've done. I sympathize that Merck might not get exactly what they want, but I mostly hope Kaggle has learned from this and will adapt future contests. I find leaderboard probing distasteful, but\r\n I will do it if it is allowed and will net me an advantage. I don't mean dummy accounts, I mean changing my future submissions based on the public score of my prior submissions. I would also like to mention observational studies as an example again; that is\r\n a time where you know you have disparate distributions and need to do some manner of balancing. I do realize this is not an observational study, but this sort of problem space does exist outside of contrived contests.
" }, { "id": 15386, "postDate": "2012-10-10T11:44:09.367Z", "content": "Seeing that you cannot possibly address every possible loophole (exploited intentionally or otherwise) in the rules, Is it time to have an Honour Code/Code of Conduct for Kagglers?
" }, { "id": 15383, "postDate": "2012-10-10T08:34:52.977Z", "content": "@Marcin
\r\nYou are right according to the situations it depends but in Merck case I don't think it's possible.
\r\nThanks
\r\nThakur
" }, { "id": 15382, "postDate": "2012-10-10T08:05:32.777Z", "content": "I think it depends on the area. It is not clear for me for this competition. However in case of building predictive model for example in marketing campaign or churn - test data are available. In case of estimating risk of potential claim for insurance policy\r\n for new client - not available.
" }, { "id": 15380, "postDate": "2012-10-10T06:02:44.233Z", "content": "\r\nThis seems to be lot of confusion, becoz whenever someone make predictive models it's almost an assumption that you don't know anything about future data. So I am not making use of distribution of test data. This is a competition so u have the test\r\n data, In real scenario you can't have access to future data. This seems to be really wrong, can any of the contender who is making use of distribution of test data can clarify how Merck is going to obtain the information about distribution of future data.
\r\n\r\n
Thanks
\r\nThakur
" }, { "id": 15367, "postDate": "2012-10-09T21:32:59.107Z", "content": "[quote=Edge2;15366]
\r\nAny chance we can get a more definitive answer? Will we be disqualified or penalized in some way for using the test set distributions in this competition?
\r\n[/quote]
\r\nGiven how far we are into the competition, it would be unfair to change the rules at this point. So, no, you will not be disqualified or penalized if you used test set distributions since it was not explicitly disallowed at the beginning of the competition.
" }, { "id": 15366, "postDate": "2012-10-09T21:25:23.850Z", "content": "Any chance we can get a more definitive answer? Will we be disqualified or penalized in some way for using the test set distributions in this competition?
" }, { "id": 15363, "postDate": "2012-10-09T18:50:17.217Z", "content": "[quote=jcnhvnhck;15361]
\r\nHi Everyone,
\r\nI checked with the competition sponsor and there is a strong preference for not using the test set distribution in creating the models since the distribution information for new molecules will not necessarily be known in practice.\r\n
\r\n[/quote]
\r\nIt's a bit late to redo our efforts. I would prefer this sort of issue be a "rule" instead of "preference" however. I don't think we're likely to finish in the money, but I would be disappointed if this invalidated our efforts.Hi Everyone,
\r\nI checked with the competition sponsor and there is a strong preference for not using the test set distribution in creating the models since the distribution information for new molecules will not necessarily be known in practice.\r\n
" }, { "id": 15339, "postDate": "2012-10-09T12:39:36.113Z", "content": "When working on an observational study you definitely know the distribution of the "test" data; the testing data are the subjects who selected treatment. I'm not talking about predictive modeling in a forecast sense; this is much more of an explanatory sense.
\r\nI am in agreement that prior informal policies are not a good way to conduct a contest. Could an admin please confirm or deny the ability to utilize the distribution of the testing data for imputation or otherwise alter the modeling process?
" }, { "id": 15337, "postDate": "2012-10-09T09:58:03.783Z", "content": "[quote=Shea Parkes;15316]
\r\nRe: Crystal Ball
\r\nThe most common real life case would be doing the analysis of an observational study. You know the distribution of the test data and need to understand how to eliminate as much selection bias as you can. (And then defend arguments about how much is left.)
\r\n[/quote]
\r\nI agree that the data we analyse in real life is of observational nature. However, I disagree that one knows the distribution of the test data (or data coming through in production environment) apriori.
\r\nWhat one does know in practice comes from the sample data that has been collected and assumes that the test data & future data are stationary and follow similar distribution to one's sample data. If there are temporal variations in one's problem, then one\r\n would try to capture such variation through variables or build some kind of adaptive model.
\r\nPeeking ahead into the test data and modifying the fit of the model is something I avoid based on my practical experience( & I do see some do this and ending-up with overfitted model - which works well on validation sample but fails when applied on out of\r\n time sample i.e., a data sample obtained a few months after the model has been developed). Only exception to this would be semi-supervised learning & I know ahead that this type of model would be acceptable in practice.
\r\nJust because prior contests have allowed using test data in our training does not make the practice correct(slavery, sati, segregation were all allowed in their times but they were not the right thing to do then & they are not the right thing to do now).\r\n So, on this principle I decline to use test data in whatsoever form to tune my model - even if it is deterimental to my ranking.
" }, { "id": 15319, "postDate": "2012-10-09T02:38:52.143Z", "content": "Great, thanks for your input.
" }, { "id": 15316, "postDate": "2012-10-09T00:59:59.850Z", "content": "Re: Crystal Ball
\r\nThe most common real life case would be doing the analysis of an observational study. You know the distribution of the test data and need to understand how to eliminate as much selection bias as you can. (And then defend arguments about how much is left.)
" }, { "id": 15315, "postDate": "2012-10-09T00:54:23.367Z", "content": "In prior contests, they have fully allowed the test data to influence the training of the model. The classic case is for imputation of missing values; they've always stated you could include the testing data in your imputation models.
\r\nI just assumed that would extend to distorting the training data to match the testing data. Other forum posters have hinted at subsampling. I obviously tipped my hat to our methods; but a recent blog post on kaggle mentioned the same methodology. I'm a firm\r\n believer of "It's all been done."
\r\nI particularly enjoy the 15 activity aspect of this contest. Otherwise you'd have to use leaderboard feedback to win, and that's just distasteful to me. Covariate shift / propensity weighting has real life applications. Leaderboard probing is just contrived\r\n nonsense to me. (Although it did appear a few pupper accounts were trying to probe this leaderboard one activity at a time.) I'm personally in favor of heavier rounding in leaderboard scores. Maybe only 3 digits for this one. All that fourth one is good\r\n for is trouble.
\r\nAnd I'm glad you enjoyed the visualizations. I'm contemplating doing another entry based on the difference between tuning a model univariately and tuning it as part of an ensemble.
" }, { "id": 15310, "postDate": "2012-10-08T22:30:55.127Z", "content": "Yeah, either way makes sense to me. If you need the most accurate measure of activity for a newly characterized 10k molecules, it makes sense to train a model for that task using the best possible information. In this case, you aren't predicting with information\r\n that doesn't exist yet. On the other hand, if you want to put a model on the shelf that predicts the activity of molecules that haven't yet been characterized, then using the distribution doesn't make sense.\r\n
" }, { "id": 15309, "postDate": "2012-10-08T22:13:55.117Z", "content": "I think a function of (training data, test data without answers) is still a valid and usable model, although it may be computationally expensive and limit prediction to a large batches.
\r\nI don't really use the test data... perhaps using it is a key to good results.
" }, { "id": 15308, "postDate": "2012-10-08T21:58:11.457Z", "content": "[quote=Edge2;15304]
\r\nI have a quick question after looking through Shea Parkes' visualization package (which is awesome, btw!)
\r\nIs it permissible for us to exploit knowledge of the test set distribution to train models specifically for that set, or are we expected to model only using knowledge that can be inferred from the training set? I can make a case either way, so instead of\r\n guessing, I'd appreciate if someone can clarify.
\r\n[/quote]
\r\n\r\n
Are we not defeating the purpose of having a test set if we are going to exploit the distribution of test set?
\r\nThink about how it would work in practice, unless you are some sort of time lord who can peer into the future and see test set distribution (while at it might as well sneak a peek at the target values as well).
" }, { "id": 15304, "postDate": "2012-10-08T20:58:15.607Z", "content": "I have a quick question after looking through Shea Parkes' visualization package (which is awesome, btw!)
\r\nIs it permissible for us to exploit knowledge of the test set distribution to train models specifically for that set, or are we expected to model only using knowledge that can be inferred from the training set? I can make a case either way, so instead of\r\n guessing, I'd appreciate if someone can clarify.
" }, { "id": 15408, "postDate": "2012-10-10T19:44:15.087Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 15388, "author_name": "Sashikanth Dareddy", "author_url": "", "post_date": "2012-10-10T12:00:15.263000", "content": "[quote=Marcin Pionnier;15382]
\r\nI think it depends on the area. It is not clear for me for this competition. However in case of building predictive model for example in marketing campaign or churn - test data are available. In case of estimating risk of potential claim for insurance policy\r\n for new client - not available.
\r\n[/quote]
\r\nThere is some confusion with the "Test Data" terminology here.
\r\nTest data is any future data on which the predictive model is to be applied.
\r\nTo spell out in concrete terms:
\r\nLets say Training Sample spans 01JAN2011 - 31DEC2011. Model trained using this data to be applied & tested on data sampled between 01JAN2012 - 31MAR2012. Here after data from 01JAN2012-31MAR2012 will be refered to as the TEST data.
\r\nThe competition started on 01MAY2012. So we have access to training data & an out of time Test Data.
\r\nJust because test data & training data are made available at the same time does not mean that one can use any information from the Test data to incorporate into ones model.
\r\nThe organisers should also share some blame for this fiasco. If time dimension is important to your problem then you should consider providing timestamps or any integer valued field which can be used to chronologically order the training dataset, so that\r\n users can build there own validation schemes. For ex: if the training data spans 12-months, a user could use the 1st 9 months to train & the remaining 3 months to validate/tune the model. This would eliminate the need/urge to incorporate test data information.
\r\nMarketting campains are not analogous to this competition since one usually builds a bespoke model for marketting a particular product, so we cannot use it as a case to defend using test data here.
", "votes": 2, "replies": [] }, { "id": 15368, "author_name": "Edge2", "author_url": "", "post_date": "2012-10-09T21:37:45.667000", "content": "[quote=jcnhvnhck;15367]
\r\nGiven how far we are into the competition, it would be unfair to change the rules at this point. So, no, you will not be disqualified or penalized if you used test set distributions since it was not explicitly disallowed at the beginning of the competition.
\r\n[/quote]
\r\nThanks. And just to put it out there: I'd welcome the chance to compete in a followup competition that has a different set of goals as well.
", "votes": 1, "replies": [] }, { "id": 15362, "author_name": "Neil Schneider, FSA", "author_url": "", "post_date": "2012-10-09T18:18:14.153000", "content": "Edit: The above post make most of this void.
\r\n\r\n
The feature descriptions for this competition are masked so I am not entirely sure of how much information of test molecules are typically known. But if I make my own assumptions...
\r\nMerck and most pharma companies test thousand of molecules for activities. The point here would be to improve the success rate of finding molecules with a certain activation. That being said, they already have the features for each molecule prior to testing\r\n for activity. The distribution of test molecules would be known and optimizing models to train on similar molecules would make sense.
\r\nJudging by the temporal grouping present in the train and test data, there are classes of molecules that are tested for activity. Which is some of the challenge of this competition, the shifts between the train and test is due to a new class of molecule\r\n being testing and we need to predict how it will react.
\r\nIn many real world applications this approach would not make sense. You would need enough test points to determine the distribution of the test data set. Training a model will take time so there would not be instant results for the new test data. Pharma\r\n molecule testing would not be limited by these needs. There is more of a financial incentive to waiting and training a model to predict which molecules to test than testing them all.
\r\nAs Shea alluded to above, we have used the distribution of the test features to train model optimized for the test data, but we have not used feedback from the leader board to influence our modeling decisions. Activity 3 for example, the train and test molecules\r\n are fairly different. Having 15 activities help mask some of the gaming that can be result from using leader board results. If you follow the HHP competition to compete at the top it appears you have to include some information derived from the leader board\r\n into your model. Maybe this will end up being part of the key to winning this competition, which will produce models are that are not useful for Merck in real application. I hope this is not the case, but considering that finik was behaving badly and that\r\n there are 15 activities. There could be more slave accounts in the depths of the player ranking or just dummy submissions that were testing outcomes of individual activities to optimize the model.
", "votes": 1, "replies": [] }, { "id": 15436, "author_name": "Dr Kiran R", "author_url": "", "post_date": "2012-10-11T17:06:59.010000", "content": "agree with Shea on this - instead of saynig 0.47 or 0.46 or 0.45, just 0.5 or 0.4 would be good
\r\nThat way we can avoid the problems mentioned.
\r\nkaggle folks: Point to note - Shea is 'spot on' here and we need to implement this real soon for future competitions
Subsequent evaluation is also problematic. Classic example is the Impermium competition. The subsequent dataset was totally unrepresentative of the initial training set rendering many efforts useless. It was not that models were not generalizable but just\r\n that it was totally different from train. Folks jumped from the second decile to first and vice - versa.
\r\nUnless care is taken, subsequent evaluation methods are also useless
\r\n
[quote=Shea Parkes;15363]
\r\n[quote=jcnhvnhck;15361]
\r\nHi Everyone,
\r\nI checked with the competition sponsor and there is a strong preference for not using the test set distribution in creating the models since the distribution information for new molecules will not necessarily be known in practice.
\r\n[/quote]
\r\nIt's a bit late to redo our efforts. I would prefer this sort of issue be a "rule" instead of "preference" however. I don't think we're likely to finish in the money, but I would be disappointed if this invalidated our efforts.
\r\nWere this rule to be in effect in future competitions, and you are willing to enforce it during evaluation, I will respect Kaggle's wishes.
[/quote]
", "votes": 0, "replies": [] }, { "id": 15428, "author_name": "Shea Parkes", "author_url": "", "post_date": "2012-10-11T14:28:58.627000", "content": "I agree the leaderboard drives competition which drives the innovation; I couldn't see this working without it. I just think you'd get the same benefit with less detailed feedback. Even just one less significant digit would be reasonable to me.
\r\nI am more curious than usually how the private leaderboard flop will fall on this competition. We've still got the advantage of less submissions for now.
", "votes": 0, "replies": [] }, { "id": 15426, "author_name": "Dr Kiran R", "author_url": "", "post_date": "2012-10-11T14:12:51.753000", "content": "same here - I am 9th without using leaderboard data.
\r\nShea: I think the greatest thing about kaggle is the leaderboard. That is what gets us data scientists excited to participate. Without a leaderboard we would not see such improvements in scores
", "votes": 0, "replies": [] }, { "id": 15391, "author_name": "Shea Parkes", "author_url": "", "post_date": "2012-10-10T12:56:13.720000", "content": "I understand Merck wouldn't likely know the exact distribution of their test data; I never implied they would. Neil gave an argument that they would have some general ideas above. However, this is a contest. To win, you pretty much exploit any source of\r\n advantage allowable within the rules; I feel no shame in what I've done. I sympathize that Merck might not get exactly what they want, but I mostly hope Kaggle has learned from this and will adapt future contests. I find leaderboard probing distasteful, but\r\n I will do it if it is allowed and will net me an advantage. I don't mean dummy accounts, I mean changing my future submissions based on the public score of my prior submissions. I would also like to mention observational studies as an example again; that is\r\n a time where you know you have disparate distributions and need to do some manner of balancing. I do realize this is not an observational study, but this sort of problem space does exist outside of contrived contests.
", "votes": 0, "replies": [] }, { "id": 15386, "author_name": "Sashikanth Dareddy", "author_url": "", "post_date": "2012-10-10T11:44:09.367000", "content": "Seeing that you cannot possibly address every possible loophole (exploited intentionally or otherwise) in the rules, Is it time to have an Honour Code/Code of Conduct for Kagglers?
", "votes": 0, "replies": [] }, { "id": 15383, "author_name": "DataGeek", "author_url": "", "post_date": "2012-10-10T08:34:52.977000", "content": "@Marcin
\r\nYou are right according to the situations it depends but in Merck case I don't think it's possible.
\r\nThanks
\r\nThakur
", "votes": 0, "replies": [] }, { "id": 15382, "author_name": "Marcin Pionnier", "author_url": "", "post_date": "2012-10-10T08:05:32.777000", "content": "I think it depends on the area. It is not clear for me for this competition. However in case of building predictive model for example in marketing campaign or churn - test data are available. In case of estimating risk of potential claim for insurance policy\r\n for new client - not available.
", "votes": 0, "replies": [] }, { "id": 15380, "author_name": "DataGeek", "author_url": "", "post_date": "2012-10-10T06:02:44.233000", "content": "\r\nThis seems to be lot of confusion, becoz whenever someone make predictive models it's almost an assumption that you don't know anything about future data. So I am not making use of distribution of test data. This is a competition so u have the test\r\n data, In real scenario you can't have access to future data. This seems to be really wrong, can any of the contender who is making use of distribution of test data can clarify how Merck is going to obtain the information about distribution of future data.
\r\n\r\n
Thanks
\r\nThakur
", "votes": 0, "replies": [] }, { "id": 15367, "author_name": "joycenv", "author_url": "", "post_date": "2012-10-09T21:32:59.107000", "content": "[quote=Edge2;15366]
\r\nAny chance we can get a more definitive answer? Will we be disqualified or penalized in some way for using the test set distributions in this competition?
\r\n[/quote]
\r\nGiven how far we are into the competition, it would be unfair to change the rules at this point. So, no, you will not be disqualified or penalized if you used test set distributions since it was not explicitly disallowed at the beginning of the competition.
", "votes": 0, "replies": [] }, { "id": 15366, "author_name": "Edge2", "author_url": "", "post_date": "2012-10-09T21:25:23.850000", "content": "Any chance we can get a more definitive answer? Will we be disqualified or penalized in some way for using the test set distributions in this competition?
", "votes": 0, "replies": [] }, { "id": 15363, "author_name": "Shea Parkes", "author_url": "", "post_date": "2012-10-09T18:50:17.217000", "content": "[quote=jcnhvnhck;15361]
\r\nHi Everyone,
\r\nI checked with the competition sponsor and there is a strong preference for not using the test set distribution in creating the models since the distribution information for new molecules will not necessarily be known in practice.\r\n
\r\n[/quote]
\r\nIt's a bit late to redo our efforts. I would prefer this sort of issue be a "rule" instead of "preference" however. I don't think we're likely to finish in the money, but I would be disappointed if this invalidated our efforts.Hi Everyone,
\r\nI checked with the competition sponsor and there is a strong preference for not using the test set distribution in creating the models since the distribution information for new molecules will not necessarily be known in practice.\r\n
", "votes": 0, "replies": [] }, { "id": 15339, "author_name": "Shea Parkes", "author_url": "", "post_date": "2012-10-09T12:39:36.113000", "content": "When working on an observational study you definitely know the distribution of the "test" data; the testing data are the subjects who selected treatment. I'm not talking about predictive modeling in a forecast sense; this is much more of an explanatory sense.
\r\nI am in agreement that prior informal policies are not a good way to conduct a contest. Could an admin please confirm or deny the ability to utilize the distribution of the testing data for imputation or otherwise alter the modeling process?
", "votes": 0, "replies": [] }, { "id": 15337, "author_name": "Sashikanth Dareddy", "author_url": "", "post_date": "2012-10-09T09:58:03.783000", "content": "[quote=Shea Parkes;15316]
\r\nRe: Crystal Ball
\r\nThe most common real life case would be doing the analysis of an observational study. You know the distribution of the test data and need to understand how to eliminate as much selection bias as you can. (And then defend arguments about how much is left.)
\r\n[/quote]
\r\nI agree that the data we analyse in real life is of observational nature. However, I disagree that one knows the distribution of the test data (or data coming through in production environment) apriori.
\r\nWhat one does know in practice comes from the sample data that has been collected and assumes that the test data & future data are stationary and follow similar distribution to one's sample data. If there are temporal variations in one's problem, then one\r\n would try to capture such variation through variables or build some kind of adaptive model.
\r\nPeeking ahead into the test data and modifying the fit of the model is something I avoid based on my practical experience( & I do see some do this and ending-up with overfitted model - which works well on validation sample but fails when applied on out of\r\n time sample i.e., a data sample obtained a few months after the model has been developed). Only exception to this would be semi-supervised learning & I know ahead that this type of model would be acceptable in practice.
\r\nJust because prior contests have allowed using test data in our training does not make the practice correct(slavery, sati, segregation were all allowed in their times but they were not the right thing to do then & they are not the right thing to do now).\r\n So, on this principle I decline to use test data in whatsoever form to tune my model - even if it is deterimental to my ranking.
", "votes": 0, "replies": [] }, { "id": 15319, "author_name": "Edge2", "author_url": "", "post_date": "2012-10-09T02:38:52.143000", "content": "Great, thanks for your input.
", "votes": 0, "replies": [] }, { "id": 15316, "author_name": "Shea Parkes", "author_url": "", "post_date": "2012-10-09T00:59:59.850000", "content": "Re: Crystal Ball
\r\nThe most common real life case would be doing the analysis of an observational study. You know the distribution of the test data and need to understand how to eliminate as much selection bias as you can. (And then defend arguments about how much is left.)
", "votes": 0, "replies": [] }, { "id": 15315, "author_name": "Shea Parkes", "author_url": "", "post_date": "2012-10-09T00:54:23.367000", "content": "In prior contests, they have fully allowed the test data to influence the training of the model. The classic case is for imputation of missing values; they've always stated you could include the testing data in your imputation models.
\r\nI just assumed that would extend to distorting the training data to match the testing data. Other forum posters have hinted at subsampling. I obviously tipped my hat to our methods; but a recent blog post on kaggle mentioned the same methodology. I'm a firm\r\n believer of "It's all been done."
\r\nI particularly enjoy the 15 activity aspect of this contest. Otherwise you'd have to use leaderboard feedback to win, and that's just distasteful to me. Covariate shift / propensity weighting has real life applications. Leaderboard probing is just contrived\r\n nonsense to me. (Although it did appear a few pupper accounts were trying to probe this leaderboard one activity at a time.) I'm personally in favor of heavier rounding in leaderboard scores. Maybe only 3 digits for this one. All that fourth one is good\r\n for is trouble.
\r\nAnd I'm glad you enjoyed the visualizations. I'm contemplating doing another entry based on the difference between tuning a model univariately and tuning it as part of an ensemble.
", "votes": 0, "replies": [] }, { "id": 15310, "author_name": "Edge2", "author_url": "", "post_date": "2012-10-08T22:30:55.127000", "content": "Yeah, either way makes sense to me. If you need the most accurate measure of activity for a newly characterized 10k molecules, it makes sense to train a model for that task using the best possible information. In this case, you aren't predicting with information\r\n that doesn't exist yet. On the other hand, if you want to put a model on the shelf that predicts the activity of molecules that haven't yet been characterized, then using the distribution doesn't make sense.\r\n
", "votes": 0, "replies": [] }, { "id": 15309, "author_name": "pzaw", "author_url": "", "post_date": "2012-10-08T22:13:55.117000", "content": "I think a function of (training data, test data without answers) is still a valid and usable model, although it may be computationally expensive and limit prediction to a large batches.
\r\nI don't really use the test data... perhaps using it is a key to good results.
", "votes": 0, "replies": [] }, { "id": 15308, "author_name": "Sashikanth Dareddy", "author_url": "", "post_date": "2012-10-08T21:58:11.457000", "content": "[quote=Edge2;15304]
\r\nI have a quick question after looking through Shea Parkes' visualization package (which is awesome, btw!)
\r\nIs it permissible for us to exploit knowledge of the test set distribution to train models specifically for that set, or are we expected to model only using knowledge that can be inferred from the training set? I can make a case either way, so instead of\r\n guessing, I'd appreciate if someone can clarify.
\r\n[/quote]
\r\n\r\n
Are we not defeating the purpose of having a test set if we are going to exploit the distribution of test set?
\r\nThink about how it would work in practice, unless you are some sort of time lord who can peer into the future and see test set distribution (while at it might as well sneak a peek at the target values as well).
", "votes": 0, "replies": [] }, { "id": 15408, "author_name": "", "author_url": "", "post_date": "2012-10-10T19:44:15.087000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "15388": "", "15368": "", "15362": "", "15436": "", "15433": "", "15428": "", "15426": "", "15391": "", "15386": "", "15383": "", "15382": "", "15380": "", "15367": "", "15366": "", "15363": "", "15361": "", "15339": "", "15337": "", "15319": "", "15316": "", "15315": "", "15310": "", "15309": "", "15308": "", "15304": "", "15408": "" } }