We are providing two \r\nnew training sets of artificial data to address the problem of artifacts that were pointed out by one participant:
\r\nFor SUP1data, human performance (as evaluated by me on ~250 examples of pairs) is approximately 0.7.
\r\nWe intend to compose the final test data on which the participants will be evaluated with data generated similarly as SUP2data and real data examples.
\r\nIMPORTANT: We remind you that you are not limited to training data we provide. You may use your own data instead or in addition to data provided to develop your methods.
\r\nThank you for your participation and good continuation of your work!
", "messages": [ { "id": "24579", "postDate": "05/21/2013 05:38:43", "content": "We are providing two \r\nnew training sets of artificial data to address the problem of artifacts that were pointed out by one participant:
\r\nFor SUP1data, human performance (as evaluated by me on ~250 examples of pairs) is approximately 0.7.
\r\nWe intend to compose the final test data on which the participants will be evaluated with data generated similarly as SUP2data and real data examples.
\r\nIMPORTANT: We remind you that you are not limited to training data we provide. You may use your own data instead or in addition to data provided to develop your methods.
\r\nThank you for your participation and good continuation of your work!
", "rawMarkdown": "", "votes": null }, { "id": "24585", "postDate": "05/21/2013 09:12:06", "content": "mmmm so what's the actual plan then? Is there a point in giving more artificial data? I'm very annoyed about the amount of time I've wasted on this competition
\r\n\r\nDo we get rid of the old training data? You haven't explained yourself very well
\r\n\r\n\r\nSounds like you are giving more artifical data in training and only have real data in testing. lol!!! The validation set is supposed to be a reflection of the test set so people can estimate their final score. Now, I've 88 on validation and might end up\r\n bottom!!!
", "rawMarkdown": "", "votes": null }, { "id": "24586", "postDate": "05/21/2013 09:28:48", "content": "Also, you're pretty much choosing who will win the competition : You've gone through peoples validation predictions, found people with good real predictions and now finding a way to make them win
", "rawMarkdown": "", "votes": null }, { "id": "24591", "postDate": "05/21/2013 14:03:39", "content": "Hi Isabelle, Domcastro!
\r\n| \r\n Data set \r\n | \r\n\r\n Error CV (1000 examples) \r\n | \r\n\r\n Error LeaderBoard \r\n | \r\n
| \r\n SUP2 \r\n | \r\n\r\n 0.632189 \r\n | \r\n\r\n --------------------------- \r\n | \r\n
| \r\n SUP1 \r\n | \r\n\r\n 0.630215 \r\n | \r\n\r\n --------------------------- \r\n | \r\n
| \r\n CEdata_train_pairs \r\n | \r\n\r\n 0.858964 \r\n | \r\n\r\n 0.83902 \r\n | \r\n
The performance drop is very much!
\r\n\r\nIsabelle wrote:
\r\nDomcastro wrote:
\r\nAll the best!
", "rawMarkdown": "", "votes": null }, { "id": "24622", "postDate": "05/21/2013 19:45:02", "content": "I understand that this change in the data generates frustration because of a performance drop. I am sorry that the problem with the data was not discovered sooner, but it is a good thing that it was discovered and I believe that the new version addresses\r\n the problem in a satisfactory way. It removes undesirable bias and it does not favor methods that I know of in any particular way. In fact, I was so worried about the drop in performance in the baseline methods that performed best previously that I generated\r\n SUP1data and classified examples visually to make sure I did not destroy essential information needed to carry out the task. While preparing the data, I had no way to know how it might affect the methods of the participants.
\r\nCompared to the original data, the data generating process is identical. What changed is only the normalization and quantization, which appear to be playing a critical role and have side effects on feature extraction. I think that there is much to be learned\r\n from this and I am optimistic that you will quickly recover from the performance drop and extract new meaningful features. I am particularly impressed by the fact that Alexandro has similar performances on SUP1data and SUP2data because SUP2data is very challenging\r\n to the human eye.
\r\n", "rawMarkdown": "", "votes": null }, { "id": "24623", "postDate": "05/21/2013 19:58:35", "content": "For real data, you may want to use the data posted at: http://webdav.tuebingen.mpg.de/cause-effect/
\r\nThose examples (and others) were incorporated in the training and validation data originally released.
", "rawMarkdown": "", "votes": null }, { "id": "24624", "postDate": "05/21/2013 20:13:41", "content": "you haven't released new validation data so have you just given us the answers?
", "rawMarkdown": "", "votes": null }, { "id": "24646", "postDate": "05/22/2013 00:42:02", "content": "In my opinion this new data set release is in the best interest of the aims of the competition and hence should be supported. The winning algorithms will be of little practical use if they heavily exploit features that are purely a result from the data creation/generation\r\n process and not inherent in the data itself.
\r\nI agree that this is discouraging to those who have already invested significant effort in tuning models and engineering features to achieve good leaderboard scores.
\r\nThere is still plenty of time left to tune models on the latest data set.
\r\n@Isabelle: If this test data set change is official it would be good to see another public leaderboard data set released so that the public leaderboard reflects the private leaderboard more closely.
", "rawMarkdown": "", "votes": null }, { "id": "24650", "postDate": "05/22/2013 07:14:44", "content": "Moody comment
", "rawMarkdown": "", "votes": null }, { "id": "24651", "postDate": "05/22/2013 07:18:48", "content": "And what is the point of rebuilding the models when in a couple of weeks, the data will change again to suit her favourite validation predictions? I doubt this will be the last data change. We now have some answers to a validation set, a validation set based\r\n on discarded training data, and as yet a test set that does not exist but will exist based on validation predictions.
", "rawMarkdown": "", "votes": null }, { "id": "24652", "postDate": "05/22/2013 07:24:16", "content": "\r\n
Hi Domcastro!
\r\n\r\nYou have as much degraded performance? As I have 35%!!!
\r\nIf not a secret, please tell us.
\r\n\r\nAll the best!
\r\n", "rawMarkdown": "", "votes": null }, { "id": "24653", "postDate": "05/22/2013 07:31:20", "content": "Hi - I haven't bothered to rebuild them. I'm very busy which is why I put so much effort in at the beginning. Also, I don't want to put new effort in because it's likely that the data / validation / *everything* will change again. I'm going to wait a few\r\n weeks and then give it a go.
\r\n\r\nThough I think I will go down because my model didn't deal with negative numbers! so I need to rethink the whole thing
", "rawMarkdown": "", "votes": null }, { "id": "24654", "postDate": "05/22/2013 07:35:17", "content": "Maybe you're right.
\r\n", "rawMarkdown": "", "votes": null }, { "id": "24655", "postDate": "05/22/2013 07:37:53", "content": "35% is a huge drop! I ran my preprocessing tool but it crashed with the negatives so I need to start again but it's not a good time for me to write "fun" tools
", "rawMarkdown": "", "votes": null }, { "id": "24656", "postDate": "05/22/2013 07:51:05", "content": "Yes! I understand that.
\r\nI ran into a large deviation (10,000), the transformation of the space ceases to work!
I don't understand. You mean that initial data are no longer valids and each one needs to shift to these data you provide to train his model(s)?
", "rawMarkdown": "", "votes": null }, { "id": "24671", "postDate": "05/22/2013 15:37:12", "content": "[quote=Alexander Larko;24591]
\r\n[/quote]
\r\nNo, but we has a \r\ncall for data in track 1 that gave the opportunity to everyone to contribute data and have other participant's algorithms tested against those data. However, we received very few submissions. Let me know if you are still interested in making such submissions\r\n and we may open a second call for data.
", "rawMarkdown": "", "votes": null }, { "id": "24674", "postDate": "05/22/2013 15:50:37", "content": "[quote=Herimanitra;24667]
\r\nI don't understand. You mean that initial data are no longer valids and each one needs to shift to these data you provide to train his model(s)?
\r\n[/quote]
\r\nWe are proposing a new way of normalizing/quantizing the artificial data and providing examples of data using this new method. If noone reveals a flaw in this new method, the test data will incorporate data prepared in the same way as SUP2data, together\r\n with some real data. Everyone is free to use data from any source for training, including the data we provided, but the original training data suffers from biases and it may be not advisable to use it for training. To avoid making multiple changes, we are\r\n presently keeping the original validation data. We indicated one source of real data http://webdav.tuebingen.mpg.de/cause-effect/ that you may use for practice purposes. None of these pairs will make it in the final test data.
", "rawMarkdown": "", "votes": null }, { "id": "24675", "postDate": "05/22/2013 15:58:48", "content": "And what about tests set? (public/private)?
", "rawMarkdown": "", "votes": null }, { "id": "24676", "postDate": "05/22/2013 16:06:26", "content": "[quote=Herimanitra;24675]
\r\nAnd what about tests set? (public/private)?
\r\n[/quote]
\r\nThe results on the validation set are shown on the public leaderboard. The final test set (which has not been released yet) will be used to produce the results of the final ranking that will show on the private leaderboard.
", "rawMarkdown": "", "votes": null }, { "id": "24680", "postDate": "05/22/2013 17:54:15", "content": "As a further clarification on the fairness of the evaluation:
\r\nThe protocol is that:
\r\n1) We will release the final test data (for which the participants will have to submit the results used for the final ranking shown on the private leaderboard) encrypted BEFORE the participants will have to submit their code.
\r\n2) The participants will submit their code BEFORE we release the final test data decryption key.
\r\nThis ensures that the final test data and the methods of the participants are produced independently of each other.
\r\n", "rawMarkdown": "", "votes": null }, { "id": "24682", "postDate": "05/22/2013 22:00:49", "content": "[quote=Isabelle;24676]
\r\nThe results on the validation set are shown on the public leaderboard. The final test set (which has not been released yet) will be used to produce the results of the final ranking that will show on the private leaderboard.
\r\n[/quote]
\r\n\r\nIsabelle,
\r\n\r\n1. The public validation leaderboard is now useless - it will not be a reflection of the score one would obtain on the final test set
\r\n2. In the rules, it say when the test set is released, the answers to the validation set will be given
\r\nHowever, the validation set is in the old format and therefore the "artificial data artefacts" will be present. Therefore is there any point in using the validation set for predicting the test set?
\r\n3. You need to make a decision on the final test set and should really stop looking at peoples validation predictions - this is introduing bias into the competition
\r\nI appreciate the data changes are best for your field, but I don't think you're appreciating data analysis protocols - the validation set is vital to data miners but the one that is provided is totally different from the new training sets and is totally\r\n different from the test set. This is data analysis sacrilege!
\r\n", "rawMarkdown": "", "votes": null }, { "id": "24700", "postDate": "05/23/2013 03:39:40", "content": "Dear Isabelle, you're right!
\r\nI spent a quick analysis of the data.
\r\nDecline in performance of my method is not associated with the normalization of the data. There is not a lot of performance\r\n degradation of about 2%, the rest is related to the reduction of the artifacts in the source data.
\r\nHave you checked the performance of the best methods on a data set SUP2, could you tell the test results, for the orientation?
\r\n\r\nAll the best.
", "rawMarkdown": "", "votes": null }, { "id": "24703", "postDate": "05/23/2013 04:51:10", "content": "\r\n
Like I have written previously, this competition is a mess. Kaggle Admin should take control of it. Where are you Kaggle Admin?
\r\n", "rawMarkdown": "", "votes": null }, { "id": "24710", "postDate": "05/23/2013 09:02:13", "content": "
I agree with Dom Castro: it is a data mining sacrilege to change the training and test data sets, but leave the public leaderboard validation set with the old "artifacts".
\r\n\r\nThe most consistent solution in my opinion would be to release a part of the real test set as public validation set. This will keep the competition on firm data mining ground,
\r\nand leave the blind guessing part away.
\r\n\r\nThis is much more desirable than simply saying that any further changes should be avoided and leaving the data sets in a half baked state.
", "rawMarkdown": "", "votes": null }, { "id": "24711", "postDate": "05/23/2013 11:45:06", "content": "Hello Everyone!
\r\nDear colleagues, somebody check the performance of his method of modeling, on a data set SUP2?
\r\nError of my method is 0.63 ...the size of the test data - 1000.
\r\nShare your experience!
All the best.
", "rawMarkdown": "", "votes": null }, { "id": "24722", "postDate": "05/23/2013 18:02:07", "content": "I am investigating with Kaggle what can be done to change the validation set. If this cannot be done, at the time of release of the encrypted test set, a "development test set" of a similar composition will be released for self evaluation.
", "rawMarkdown": "", "votes": null }, { "id": "24745", "postDate": "05/24/2013 00:25:22", "content": "Hi Alex,
\r\nMy CV error on the Sup2 data is 0.73 also a significant drop from 0.87 but not as bad as your drop. Note that I have only run the new data on 1 of my models which is no longer likely to be the best one.
", "rawMarkdown": "", "votes": null }, { "id": "24810", "postDate": "05/25/2013 16:39:48", "content": "I am totally confused. Is the new dataset the "real one"? If so, are the validation tests useless and the leaderboard irrelevant ?
\r\nIf that's the case, is it possible to finalise the training set ? Should we re-train everything with the new data, and if yes, since the test is going to be based on SUP2, is it the case that only SUP2 is relevant ?
", "rawMarkdown": "", "votes": null }, { "id": "24815", "postDate": "05/25/2013 17:46:21", "content": "What further confuses me is that the final test set will have data from SUP2 and real casuality examples.
\r\nDid the first train set have also both real and synthetic examples?
\r\nIf that is so will an accurate training set be achieved by combining SUP2 with the set of real casuality data from the old train set?
\r\n\r\nAnother question: is all this confusion for new/old data only abouth the quantization and normalization of data?
\r\nWill normalizing of the old train data from the beginning to a mean 0 and std 10000 make the data consistent with Isabelle's future test set?
\r\nKnowing this can be potentially an easy way to have clean data, even if the leaderboard does not reflect it.
\r\n\r\nbest regards
\r\n\r\n\r\n\r\n\r\n", "rawMarkdown": "", "votes": null }, { "id": "24821", "postDate": "05/25/2013 20:49:54", "content": "I've spent some time reading the forums, my understanding of the situation is as follows:
\r\n\r\na) There is no standardised training set. Training data can come from everywhere, as long one thinks the data is relevant.
\r\nb) There is no validation data set. Leaderboard scores are at best misleading.
\r\nc) There is no uniform format for the training data.
\r\nd) The (final) test set is not ready yet.
\r\n\r\nObviously this is not an ideal situation to be in. IMHO the real problem is that this competition tries to be two things at the same time. A "Machine Learning" challenge and a platform for solving a real scientific problem. I don't think these two goals\r\n can be totally reconciled (I can elaborate further on this if anyone cares).
\r\nNow my two cents on what I think is a fair way to address the issues above:
\r\n\r\nA) Prize goes out to whoever wins this competition based on a standardised training/validation/testing set combination. Doesn't matter if there are artifacts or not, the important thing here is that these sets drawn from the same distribution.
\r\nB) Since there is a "scientific" aspect to this challenge, the organisers can have an internal ranking based on which algorithms score better at what data. These ranking can then be used to augment the quality of the workshop papers (e.g. by having the\r\n results in the papers) or for future reference/publications.
\r\n\r\nThanks,
\r\nFirfi
\r\n\r\n\r\n\r\n\r\n
\r\n\r\n", "rawMarkdown": "", "votes": null }, { "id": "24823", "postDate": "05/25/2013 21:58:28", "content": "
IMHO, we should work only with real data. The artificial data should be available in case someone wants to use it to train the model. But I think the test set and the validation set should be composed only by real data.
\r\nIsabelle posted this a couple of weeks ago:
\r\n"1) Should the final evaluation include only real data or both artificial and real data? The participants were favorable to include both because there is as much risk to introduce artifacts in real data as there is in artificial data while collecting and preparing the data."\r\n
\r\n
I think that the risk of introducing artifacts in data is something that can happen in any Kaggle competition.
\r\nAlso many scientific theories have been valid until someone has found a sample that contradicts them. But that is the way science works...many times you can't tell if your sample represents the total population.
\r\nI understand that introducing artificial data has some benefits, but we could be also introducing noise in the data that avoids us to discover some features present in real data.
\r\nTherefore, as I said in a previous post, I think that we should know which training samples are real data and which are artificial data. And then make the evaluations using only real data. Or at least keeping two separate scores: one for real data, and another\r\n for artificial or mixed data.
", "rawMarkdown": "", "votes": null }, { "id": "24830", "postDate": "05/26/2013 18:28:29", "content": "Dear all
\r\nI also think that artificial data is of very limited interest at least as test data (as in most tasks in machine learning / Kaggle).
\r\n1) Artificial data is too simple to model the complexity of real data, specifically of real noise (shape, variable variance, etc). At the same time artificial data can be made too complex to be considered a 'normal' real case. For example I can start generating\r\n X and then its square root with a random sign x -> +/-sqrt(x), you can even map a non Gaussian distribution to a Gaussian distribution with a function like this or a similar one.
\r\n2) Artificial data can not be used to validate how good are the simplifying or sufficient assumptions for causally orienting two variables or the approximate formulations because we somehow choose a priori if we are going to generate the artificial samples\r\n following one of this simple models (ANM, PNL, IGCI, ...)
\r\nTherefore I subscribe to Aupathletic opinion that "we should know which training samples are real data and which are artificial data. And then make the evaluations using only real data. Or at least keeping two separate scores: one for real data, and another\r\n for artificial or mixed data."
", "rawMarkdown": "", "votes": null }, { "id": "25066", "postDate": "06/02/2013 20:34:31", "content": "Most real data will have artifacts also, due to experimental design. In the altitude example, causation can be a tricky thing. As you rise in altitude, pressure and temperature drop. So there would be correlation between temp and pressure with causation\r\n from an increase in altitude. While temp and pressure will have a number of other influential factors (error term in this model), altitude will not. In the experimental design, temp and pressure values will be measured every 500m or every t time interval as\r\n a weather ballon accends. This pattern of measurement will provide an artifact flagging which variable the designer thought was the cause. In truth, it is all more complex. Altitude does not effect pressure, gravity does and altitude is just a piece to that\r\n equation. Then pressure also has an effect on temperature. So where measurements from a weather ballon may predict that pressure and temp are correlated but cause by something else, data from a laboratory experiment on PVRT( if I recall my physics correctly)\r\n would show there is causation.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 24585, "author_name": "domcastro", "author_url": "", "post_date": "05/21/2013 09:12:06", "content": "mmmm so what's the actual plan then? Is there a point in giving more artificial data? I'm very annoyed about the amount of time I've wasted on this competition
\r\n\r\nDo we get rid of the old training data? You haven't explained yourself very well
\r\n\r\n\r\nSounds like you are giving more artifical data in training and only have real data in testing. lol!!! The validation set is supposed to be a reflection of the test set so people can estimate their final score. Now, I've 88 on validation and might end up\r\n bottom!!!
", "votes": null, "replies": [] }, { "id": 24586, "author_name": "domcastro", "author_url": "", "post_date": "05/21/2013 09:28:48", "content": "Also, you're pretty much choosing who will win the competition : You've gone through peoples validation predictions, found people with good real predictions and now finding a way to make them win
", "votes": null, "replies": [] }, { "id": 24591, "author_name": "alexxanderlarko", "author_url": "", "post_date": "05/21/2013 14:03:39", "content": "Hi Isabelle, Domcastro!
\r\n| \r\n Data set \r\n | \r\n\r\n Error CV (1000 examples) \r\n | \r\n\r\n Error LeaderBoard \r\n | \r\n
| \r\n SUP2 \r\n | \r\n\r\n 0.632189 \r\n | \r\n\r\n --------------------------- \r\n | \r\n
| \r\n SUP1 \r\n | \r\n\r\n 0.630215 \r\n | \r\n\r\n --------------------------- \r\n | \r\n
| \r\n CEdata_train_pairs \r\n | \r\n\r\n 0.858964 \r\n | \r\n\r\n 0.83902 \r\n | \r\n
The performance drop is very much!
\r\n\r\nIsabelle wrote:
\r\nDomcastro wrote:
\r\nAll the best!
", "votes": null, "replies": [] }, { "id": 24622, "author_name": "iguyon", "author_url": "", "post_date": "05/21/2013 19:45:02", "content": "I understand that this change in the data generates frustration because of a performance drop. I am sorry that the problem with the data was not discovered sooner, but it is a good thing that it was discovered and I believe that the new version addresses\r\n the problem in a satisfactory way. It removes undesirable bias and it does not favor methods that I know of in any particular way. In fact, I was so worried about the drop in performance in the baseline methods that performed best previously that I generated\r\n SUP1data and classified examples visually to make sure I did not destroy essential information needed to carry out the task. While preparing the data, I had no way to know how it might affect the methods of the participants.
\r\nCompared to the original data, the data generating process is identical. What changed is only the normalization and quantization, which appear to be playing a critical role and have side effects on feature extraction. I think that there is much to be learned\r\n from this and I am optimistic that you will quickly recover from the performance drop and extract new meaningful features. I am particularly impressed by the fact that Alexandro has similar performances on SUP1data and SUP2data because SUP2data is very challenging\r\n to the human eye.
\r\n", "votes": null, "replies": [] }, { "id": 24623, "author_name": "iguyon", "author_url": "", "post_date": "05/21/2013 19:58:35", "content": "For real data, you may want to use the data posted at: http://webdav.tuebingen.mpg.de/cause-effect/
\r\nThose examples (and others) were incorporated in the training and validation data originally released.
", "votes": null, "replies": [] }, { "id": 24624, "author_name": "domcastro", "author_url": "", "post_date": "05/21/2013 20:13:41", "content": "you haven't released new validation data so have you just given us the answers?
", "votes": null, "replies": [] }, { "id": 24646, "author_name": "thomasatkins", "author_url": "", "post_date": "05/22/2013 00:42:02", "content": "In my opinion this new data set release is in the best interest of the aims of the competition and hence should be supported. The winning algorithms will be of little practical use if they heavily exploit features that are purely a result from the data creation/generation\r\n process and not inherent in the data itself.
\r\nI agree that this is discouraging to those who have already invested significant effort in tuning models and engineering features to achieve good leaderboard scores.
\r\nThere is still plenty of time left to tune models on the latest data set.
\r\n@Isabelle: If this test data set change is official it would be good to see another public leaderboard data set released so that the public leaderboard reflects the private leaderboard more closely.
", "votes": null, "replies": [] }, { "id": 24650, "author_name": "domcastro", "author_url": "", "post_date": "05/22/2013 07:14:44", "content": "Moody comment
", "votes": null, "replies": [] }, { "id": 24651, "author_name": "domcastro", "author_url": "", "post_date": "05/22/2013 07:18:48", "content": "And what is the point of rebuilding the models when in a couple of weeks, the data will change again to suit her favourite validation predictions? I doubt this will be the last data change. We now have some answers to a validation set, a validation set based\r\n on discarded training data, and as yet a test set that does not exist but will exist based on validation predictions.
", "votes": null, "replies": [] }, { "id": 24652, "author_name": "alexxanderlarko", "author_url": "", "post_date": "05/22/2013 07:24:16", "content": "\r\n
Hi Domcastro!
\r\n\r\nYou have as much degraded performance? As I have 35%!!!
\r\nIf not a secret, please tell us.
\r\n\r\nAll the best!
\r\n", "votes": null, "replies": [] }, { "id": 24653, "author_name": "domcastro", "author_url": "", "post_date": "05/22/2013 07:31:20", "content": "Hi - I haven't bothered to rebuild them. I'm very busy which is why I put so much effort in at the beginning. Also, I don't want to put new effort in because it's likely that the data / validation / *everything* will change again. I'm going to wait a few\r\n weeks and then give it a go.
\r\n\r\nThough I think I will go down because my model didn't deal with negative numbers! so I need to rethink the whole thing
", "votes": null, "replies": [] }, { "id": 24654, "author_name": "alexxanderlarko", "author_url": "", "post_date": "05/22/2013 07:35:17", "content": "Maybe you're right.
\r\n", "votes": null, "replies": [] }, { "id": 24655, "author_name": "domcastro", "author_url": "", "post_date": "05/22/2013 07:37:53", "content": "35% is a huge drop! I ran my preprocessing tool but it crashed with the negatives so I need to start again but it's not a good time for me to write "fun" tools
", "votes": null, "replies": [] }, { "id": 24656, "author_name": "alexxanderlarko", "author_url": "", "post_date": "05/22/2013 07:51:05", "content": "Yes! I understand that.
\r\nI ran into a large deviation (10,000), the transformation of the space ceases to work!
I don't understand. You mean that initial data are no longer valids and each one needs to shift to these data you provide to train his model(s)?
", "votes": null, "replies": [] }, { "id": 24671, "author_name": "iguyon", "author_url": "", "post_date": "05/22/2013 15:37:12", "content": "[quote=Alexander Larko;24591]
\r\n[/quote]
\r\nNo, but we has a \r\ncall for data in track 1 that gave the opportunity to everyone to contribute data and have other participant's algorithms tested against those data. However, we received very few submissions. Let me know if you are still interested in making such submissions\r\n and we may open a second call for data.
", "votes": null, "replies": [] }, { "id": 24674, "author_name": "iguyon", "author_url": "", "post_date": "05/22/2013 15:50:37", "content": "[quote=Herimanitra;24667]
\r\nI don't understand. You mean that initial data are no longer valids and each one needs to shift to these data you provide to train his model(s)?
\r\n[/quote]
\r\nWe are proposing a new way of normalizing/quantizing the artificial data and providing examples of data using this new method. If noone reveals a flaw in this new method, the test data will incorporate data prepared in the same way as SUP2data, together\r\n with some real data. Everyone is free to use data from any source for training, including the data we provided, but the original training data suffers from biases and it may be not advisable to use it for training. To avoid making multiple changes, we are\r\n presently keeping the original validation data. We indicated one source of real data http://webdav.tuebingen.mpg.de/cause-effect/ that you may use for practice purposes. None of these pairs will make it in the final test data.
", "votes": null, "replies": [] }, { "id": 24675, "author_name": "herimanitra", "author_url": "", "post_date": "05/22/2013 15:58:48", "content": "And what about tests set? (public/private)?
", "votes": null, "replies": [] }, { "id": 24676, "author_name": "iguyon", "author_url": "", "post_date": "05/22/2013 16:06:26", "content": "[quote=Herimanitra;24675]
\r\nAnd what about tests set? (public/private)?
\r\n[/quote]
\r\nThe results on the validation set are shown on the public leaderboard. The final test set (which has not been released yet) will be used to produce the results of the final ranking that will show on the private leaderboard.
", "votes": null, "replies": [] }, { "id": 24680, "author_name": "iguyon", "author_url": "", "post_date": "05/22/2013 17:54:15", "content": "As a further clarification on the fairness of the evaluation:
\r\nThe protocol is that:
\r\n1) We will release the final test data (for which the participants will have to submit the results used for the final ranking shown on the private leaderboard) encrypted BEFORE the participants will have to submit their code.
\r\n2) The participants will submit their code BEFORE we release the final test data decryption key.
\r\nThis ensures that the final test data and the methods of the participants are produced independently of each other.
\r\n", "votes": null, "replies": [] }, { "id": 24682, "author_name": "domcastro", "author_url": "", "post_date": "05/22/2013 22:00:49", "content": "[quote=Isabelle;24676]
\r\nThe results on the validation set are shown on the public leaderboard. The final test set (which has not been released yet) will be used to produce the results of the final ranking that will show on the private leaderboard.
\r\n[/quote]
\r\n\r\nIsabelle,
\r\n\r\n1. The public validation leaderboard is now useless - it will not be a reflection of the score one would obtain on the final test set
\r\n2. In the rules, it say when the test set is released, the answers to the validation set will be given
\r\nHowever, the validation set is in the old format and therefore the "artificial data artefacts" will be present. Therefore is there any point in using the validation set for predicting the test set?
\r\n3. You need to make a decision on the final test set and should really stop looking at peoples validation predictions - this is introduing bias into the competition
\r\nI appreciate the data changes are best for your field, but I don't think you're appreciating data analysis protocols - the validation set is vital to data miners but the one that is provided is totally different from the new training sets and is totally\r\n different from the test set. This is data analysis sacrilege!
\r\n", "votes": null, "replies": [] }, { "id": 24700, "author_name": "alexxanderlarko", "author_url": "", "post_date": "05/23/2013 03:39:40", "content": "Dear Isabelle, you're right!
\r\nI spent a quick analysis of the data.
\r\nDecline in performance of my method is not associated with the normalization of the data. There is not a lot of performance\r\n degradation of about 2%, the rest is related to the reduction of the artifacts in the source data.
\r\nHave you checked the performance of the best methods on a data set SUP2, could you tell the test results, for the orientation?
\r\n\r\nAll the best.
", "votes": null, "replies": [] }, { "id": 24703, "author_name": "jc3626439", "author_url": "", "post_date": "05/23/2013 04:51:10", "content": "\r\n
Like I have written previously, this competition is a mess. Kaggle Admin should take control of it. Where are you Kaggle Admin?
\r\n", "votes": null, "replies": [] }, { "id": 24710, "author_name": "dieselboy", "author_url": "", "post_date": "05/23/2013 09:02:13", "content": "
I agree with Dom Castro: it is a data mining sacrilege to change the training and test data sets, but leave the public leaderboard validation set with the old "artifacts".
\r\n\r\nThe most consistent solution in my opinion would be to release a part of the real test set as public validation set. This will keep the competition on firm data mining ground,
\r\nand leave the blind guessing part away.
\r\n\r\nThis is much more desirable than simply saying that any further changes should be avoided and leaving the data sets in a half baked state.
", "votes": null, "replies": [] }, { "id": 24711, "author_name": "alexxanderlarko", "author_url": "", "post_date": "05/23/2013 11:45:06", "content": "Hello Everyone!
\r\nDear colleagues, somebody check the performance of his method of modeling, on a data set SUP2?
\r\nError of my method is 0.63 ...the size of the test data - 1000.
\r\nShare your experience!
All the best.
", "votes": null, "replies": [] }, { "id": 24722, "author_name": "iguyon", "author_url": "", "post_date": "05/23/2013 18:02:07", "content": "I am investigating with Kaggle what can be done to change the validation set. If this cannot be done, at the time of release of the encrypted test set, a "development test set" of a similar composition will be released for self evaluation.
", "votes": null, "replies": [] }, { "id": 24745, "author_name": "thomasatkins", "author_url": "", "post_date": "05/24/2013 00:25:22", "content": "Hi Alex,
\r\nMy CV error on the Sup2 data is 0.73 also a significant drop from 0.87 but not as bad as your drop. Note that I have only run the new data on 1 of my models which is no longer likely to be the best one.
", "votes": null, "replies": [] }, { "id": 24810, "author_name": "firfirikos", "author_url": "", "post_date": "05/25/2013 16:39:48", "content": "I am totally confused. Is the new dataset the "real one"? If so, are the validation tests useless and the leaderboard irrelevant ?
\r\nIf that's the case, is it possible to finalise the training set ? Should we re-train everything with the new data, and if yes, since the test is going to be based on SUP2, is it the case that only SUP2 is relevant ?
", "votes": null, "replies": [] }, { "id": 24815, "author_name": "dieselboy", "author_url": "", "post_date": "05/25/2013 17:46:21", "content": "What further confuses me is that the final test set will have data from SUP2 and real casuality examples.
\r\nDid the first train set have also both real and synthetic examples?
\r\nIf that is so will an accurate training set be achieved by combining SUP2 with the set of real casuality data from the old train set?
\r\n\r\nAnother question: is all this confusion for new/old data only abouth the quantization and normalization of data?
\r\nWill normalizing of the old train data from the beginning to a mean 0 and std 10000 make the data consistent with Isabelle's future test set?
\r\nKnowing this can be potentially an easy way to have clean data, even if the leaderboard does not reflect it.
\r\n\r\nbest regards
\r\n\r\n\r\n\r\n\r\n", "votes": null, "replies": [] }, { "id": 24821, "author_name": "firfirikos", "author_url": "", "post_date": "05/25/2013 20:49:54", "content": "I've spent some time reading the forums, my understanding of the situation is as follows:
\r\n\r\na) There is no standardised training set. Training data can come from everywhere, as long one thinks the data is relevant.
\r\nb) There is no validation data set. Leaderboard scores are at best misleading.
\r\nc) There is no uniform format for the training data.
\r\nd) The (final) test set is not ready yet.
\r\n\r\nObviously this is not an ideal situation to be in. IMHO the real problem is that this competition tries to be two things at the same time. A "Machine Learning" challenge and a platform for solving a real scientific problem. I don't think these two goals\r\n can be totally reconciled (I can elaborate further on this if anyone cares).
\r\nNow my two cents on what I think is a fair way to address the issues above:
\r\n\r\nA) Prize goes out to whoever wins this competition based on a standardised training/validation/testing set combination. Doesn't matter if there are artifacts or not, the important thing here is that these sets drawn from the same distribution.
\r\nB) Since there is a "scientific" aspect to this challenge, the organisers can have an internal ranking based on which algorithms score better at what data. These ranking can then be used to augment the quality of the workshop papers (e.g. by having the\r\n results in the papers) or for future reference/publications.
\r\n\r\nThanks,
\r\nFirfi
\r\n\r\n\r\n\r\n\r\n
\r\n\r\n", "votes": null, "replies": [] }, { "id": 24823, "author_name": "", "author_url": "", "post_date": "05/25/2013 21:58:28", "content": "
IMHO, we should work only with real data. The artificial data should be available in case someone wants to use it to train the model. But I think the test set and the validation set should be composed only by real data.
\r\nIsabelle posted this a couple of weeks ago:
\r\n"1) Should the final evaluation include only real data or both artificial and real data? The participants were favorable to include both because there is as much risk to introduce artifacts in real data as there is in artificial data while collecting and preparing the data."\r\n
\r\n
I think that the risk of introducing artifacts in data is something that can happen in any Kaggle competition.
\r\nAlso many scientific theories have been valid until someone has found a sample that contradicts them. But that is the way science works...many times you can't tell if your sample represents the total population.
\r\nI understand that introducing artificial data has some benefits, but we could be also introducing noise in the data that avoids us to discover some features present in real data.
\r\nTherefore, as I said in a previous post, I think that we should know which training samples are real data and which are artificial data. And then make the evaluations using only real data. Or at least keeping two separate scores: one for real data, and another\r\n for artificial or mixed data.
", "votes": null, "replies": [] }, { "id": 24830, "author_name": "jarfo1", "author_url": "", "post_date": "05/26/2013 18:28:29", "content": "Dear all
\r\nI also think that artificial data is of very limited interest at least as test data (as in most tasks in machine learning / Kaggle).
\r\n1) Artificial data is too simple to model the complexity of real data, specifically of real noise (shape, variable variance, etc). At the same time artificial data can be made too complex to be considered a 'normal' real case. For example I can start generating\r\n X and then its square root with a random sign x -> +/-sqrt(x), you can even map a non Gaussian distribution to a Gaussian distribution with a function like this or a similar one.
\r\n2) Artificial data can not be used to validate how good are the simplifying or sufficient assumptions for causally orienting two variables or the approximate formulations because we somehow choose a priori if we are going to generate the artificial samples\r\n following one of this simple models (ANM, PNL, IGCI, ...)
\r\nTherefore I subscribe to Aupathletic opinion that "we should know which training samples are real data and which are artificial data. And then make the evaluations using only real data. Or at least keeping two separate scores: one for real data, and another\r\n for artificial or mixed data."
", "votes": null, "replies": [] }, { "id": 25066, "author_name": "nschneider", "author_url": "", "post_date": "06/02/2013 20:34:31", "content": "Most real data will have artifacts also, due to experimental design. In the altitude example, causation can be a tricky thing. As you rise in altitude, pressure and temperature drop. So there would be correlation between temp and pressure with causation\r\n from an increase in altitude. While temp and pressure will have a number of other influential factors (error term in this model), altitude will not. In the experimental design, temp and pressure values will be measured every 500m or every t time interval as\r\n a weather ballon accends. This pattern of measurement will provide an artifact flagging which variable the designer thought was the cause. In truth, it is all more complex. Altitude does not effect pressure, gravity does and altitude is just a piece to that\r\n equation. Then pressure also has an effect on temperature. So where measurements from a weather ballon may predict that pressure and temp are correlated but cause by something else, data from a laboratory experiment on PVRT( if I recall my physics correctly)\r\n would show there is causation.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "24579": "", "24585": "", "24586": "", "24591": "", "24622": "", "24623": "", "24624": "", "24646": "", "24650": "", "24651": "", "24652": "", "24653": "", "24654": "", "24655": "", "24656": "", "24667": "", "24671": "", "24674": "", "24675": "", "24676": "", "24680": "", "24682": "", "24700": "", "24703": "", "24710": "", "24711": "", "24722": "", "24745": "", "24810": "", "24815": "", "24821": "", "24823": "", "24830": "", "25066": "" }, "source": "meta" }