I'm a bit put out that the test data has been changed half way through this competition. I assume the valid test set was a random sample taken from the data - but now the rest of the data will be changed which means that the valid set is not a reflection\r\n of the test set anymore. Considering you only have one blind shot at the test data - this is very frustrating especially when you have to write a 6 page academic paper just to take part!
", "messages": [ { "id": "23979", "postDate": "05/07/2013 09:51:20", "content": "I'm a bit put out that the test data has been changed half way through this competition. I assume the valid test set was a random sample taken from the data - but now the rest of the data will be changed which means that the valid set is not a reflection\r\n of the test set anymore. Considering you only have one blind shot at the test data - this is very frustrating especially when you have to write a 6 page academic paper just to take part!
", "rawMarkdown": "", "votes": null }, { "id": "23980", "postDate": "05/07/2013 09:59:50", "content": "Also, how did you find out about the "artefacts"? By looking at people's valid predictions?
\r\n\r\n
I think you will need to change the train / valid and test data if you are now going to muck about with the test data
", "rawMarkdown": "", "votes": null }, { "id": "24014", "postDate": "05/08/2013 15:22:30", "content": "We are trying to make the evaluation as fair as possible AND get valid results on the cause-effect pairs problem. You would not want to win for the wrong reason. If you propose a method that is efficient to solve the cause-effect pairs problem, this could\r\n have an enormous impact. But if it only catches on some data artefact, what good would that be?
\r\nOur analysis of the results on validation data reveals that results are good both on artificial and on real data, so this is a reassuing indication that the proposed methods are actually addressing the real problem. However, some participants\r\n have a larger difference between the results on real data and artificial data than others and we think we found why this could be: one participant pointed out to us that there is an imbalance in the distribution of values of the variables in the various classes\r\n that could be exploited to get better results. We are very grateful that he disclosed this to us. We invite you and all the other participants to scrutinize the data and report any other possible such problems so we can address all the problems\r\n reported (which may include re-issuing new versions of the data if necessary). For now, we recommend\r\nnot to rely on features such as the number of unique values a variable takes or the maximum value.\r\nUsing the number of unique values relative to the total number of values should be OK.
\r\nRegarding publishing: this is an academic challenge and publishing is an important part of it. You are working on solving a new and important scientific problem. Part of solving that problem is not only to get good results, but also to analyze the method\r\n and explain why it works. Remember that if you win you will have to publicly publish the source code and your method will be under a lot of scrutiny, not only by us, but by people all over the world who are interested in using your method.
", "rawMarkdown": "", "votes": null }, { "id": "24017", "postDate": "05/08/2013 16:15:01", "content": "Will you disqualify someone with a high score because it just happens their algorithm picks up the artificial data? Will you disqualify someone who follows the benchmark and uses unique values? I understand the need to publish but in another competition\r\n I only had to publish after I had been told I had won. Here, you have to write the paper before with the risk you may be disqualified because of artefacts you weren't aware of!
", "rawMarkdown": "", "votes": null }, { "id": "24024", "postDate": "05/08/2013 17:16:38", "content": "No, we do not want to disqualify people because of problems with the data. But we do not want to discover problems with the data after the challenge is over and give the winner a Pyrrhic victory.
", "rawMarkdown": "", "votes": null }, { "id": "24029", "postDate": "05/08/2013 18:29:05", "content": "You could write another paper on the perils of creating artificial data. Domain knowledge has now increased as you now know how not to create artificial data. So it's all good. Science is incremental. People should publish failures more often as it will\r\n reduce duplication of effort (and failure!)
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 23980, "author_name": "domcastro", "author_url": "", "post_date": "05/07/2013 09:59:50", "content": "Also, how did you find out about the "artefacts"? By looking at people's valid predictions?
\r\n\r\n
I think you will need to change the train / valid and test data if you are now going to muck about with the test data
", "votes": null, "replies": [] }, { "id": 24014, "author_name": "iguyon", "author_url": "", "post_date": "05/08/2013 15:22:30", "content": "We are trying to make the evaluation as fair as possible AND get valid results on the cause-effect pairs problem. You would not want to win for the wrong reason. If you propose a method that is efficient to solve the cause-effect pairs problem, this could\r\n have an enormous impact. But if it only catches on some data artefact, what good would that be?
\r\nOur analysis of the results on validation data reveals that results are good both on artificial and on real data, so this is a reassuing indication that the proposed methods are actually addressing the real problem. However, some participants\r\n have a larger difference between the results on real data and artificial data than others and we think we found why this could be: one participant pointed out to us that there is an imbalance in the distribution of values of the variables in the various classes\r\n that could be exploited to get better results. We are very grateful that he disclosed this to us. We invite you and all the other participants to scrutinize the data and report any other possible such problems so we can address all the problems\r\n reported (which may include re-issuing new versions of the data if necessary). For now, we recommend\r\nnot to rely on features such as the number of unique values a variable takes or the maximum value.\r\nUsing the number of unique values relative to the total number of values should be OK.
\r\nRegarding publishing: this is an academic challenge and publishing is an important part of it. You are working on solving a new and important scientific problem. Part of solving that problem is not only to get good results, but also to analyze the method\r\n and explain why it works. Remember that if you win you will have to publicly publish the source code and your method will be under a lot of scrutiny, not only by us, but by people all over the world who are interested in using your method.
", "votes": null, "replies": [] }, { "id": 24017, "author_name": "domcastro", "author_url": "", "post_date": "05/08/2013 16:15:01", "content": "Will you disqualify someone with a high score because it just happens their algorithm picks up the artificial data? Will you disqualify someone who follows the benchmark and uses unique values? I understand the need to publish but in another competition\r\n I only had to publish after I had been told I had won. Here, you have to write the paper before with the risk you may be disqualified because of artefacts you weren't aware of!
", "votes": null, "replies": [] }, { "id": 24024, "author_name": "iguyon", "author_url": "", "post_date": "05/08/2013 17:16:38", "content": "No, we do not want to disqualify people because of problems with the data. But we do not want to discover problems with the data after the challenge is over and give the winner a Pyrrhic victory.
", "votes": null, "replies": [] }, { "id": 24029, "author_name": "domcastro", "author_url": "", "post_date": "05/08/2013 18:29:05", "content": "You could write another paper on the perils of creating artificial data. Domain knowledge has now increased as you now know how not to create artificial data. So it's all good. Science is incremental. People should publish failures more often as it will\r\n reduce duplication of effort (and failure!)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "23979": "", "23980": "", "24014": "", "24017": "", "24024": "", "24029": "" }, "source": "meta" }