{ "id": 6794, "title": "Data constraints and competition reboot", "url": "/competitions/galaxy-zoo-the-galaxy-challenge/discussion/6794", "author_name": "", "post_date": "2014-01-10T01:57:35.683Z", "votes": 5, "comment_count": 40, "views": 12106, "content": "

Hi everyone,

\n

Many thanks to everyone who has participated in our galaxy challenge so far. We've been excited by the interest and by the intelligence of the questions, and judging from your scores, we're very optimistic that you've come up with some excellent solutions. 

\n

Over the first couple of weeks of the contest, several participants (notably @sedielem and others) found that some of the data was not behaving in the way expected from the Galaxy Zoo decision tree. The administrators and scientists have been looking into this in detail over the last week or so, and have confirmed that it's indeed a genuine error on our part. 

\n

The cause is that a fraction of the original classifications for which the total number of votes didn't completely carry through each step of the decision tree. The cause isn't completely known; possibly recording of incomplete classifications, or the method by which we removed duplicate classifications. The effect on the data that you received, though, meant that for many of the lower nodes in the decision tree (the values of which are expressed as normalized, cumulative fractions), it was possible to record a zero when the value should have been higher than that.

\n

This is early enough in the competition, and affected enough galaxies in the table, that we've decided to reboot the leaderboard and current competition state. The reason for this is that it we haven't just fixed the solutions, but we've inserted new images into the competition from the larger Galaxy Zoo dataset. The sizes and parameters for all data and solution files remain the same; we have the expectation that competitors can rerun their code and likely get similar RMSE values to what they had before. 

\n

The administrators and I apologize for having to do this, and not catching the errors when the data were originally posted. However, it's critical that we get solutions that will be of the highest possible use for science, and we believe that fixing the dataset will make that happen. We've added an additional two weeks to the competition deadline, and hope that everyone who submitted a first solution will do so again. Please post on the forum if you have more questions, and happy hunting.

", "messages": [ { "id": "37191", "postDate": "01/10/2014 01:57:35", "content": "

Hi everyone,

\n

Many thanks to everyone who has participated in our galaxy challenge so far. We've been excited by the interest and by the intelligence of the questions, and judging from your scores, we're very optimistic that you've come up with some excellent solutions. 

\n

Over the first couple of weeks of the contest, several participants (notably @sedielem and others) found that some of the data was not behaving in the way expected from the Galaxy Zoo decision tree. The administrators and scientists have been looking into this in detail over the last week or so, and have confirmed that it's indeed a genuine error on our part. 

\n

The cause is that a fraction of the original classifications for which the total number of votes didn't completely carry through each step of the decision tree. The cause isn't completely known; possibly recording of incomplete classifications, or the method by which we removed duplicate classifications. The effect on the data that you received, though, meant that for many of the lower nodes in the decision tree (the values of which are expressed as normalized, cumulative fractions), it was possible to record a zero when the value should have been higher than that.

\n

This is early enough in the competition, and affected enough galaxies in the table, that we've decided to reboot the leaderboard and current competition state. The reason for this is that it we haven't just fixed the solutions, but we've inserted new images into the competition from the larger Galaxy Zoo dataset. The sizes and parameters for all data and solution files remain the same; we have the expectation that competitors can rerun their code and likely get similar RMSE values to what they had before. 

\n

The administrators and I apologize for having to do this, and not catching the errors when the data were originally posted. However, it's critical that we get solutions that will be of the highest possible use for science, and we believe that fixing the dataset will make that happen. We've added an additional two weeks to the competition deadline, and hope that everyone who submitted a first solution will do so again. Please post on the forum if you have more questions, and happy hunting.

", "rawMarkdown": "", "votes": null }, { "id": "37202", "postDate": "01/10/2014 07:34:02", "content": "

I was anticipating new labels, but new images and a reset of the leaderboard? I won't be submitting again but I wish you all the best and hope you get a great result :)

", "rawMarkdown": "", "votes": null }, { "id": "37204", "postDate": "01/10/2014 08:30:31", "content": "

Nice. I think this is an appropriate decision and hope the dataset is now cleaner :)

\n

Thanks for your reactivity.

", "rawMarkdown": "", "votes": null }, { "id": "37210", "postDate": "01/10/2014 10:36:41", "content": "

Thanks for fixing the data! I didn't expect the images to be replaced as well, but I'm glad this was addressed.

\n

I'm assuming it is now prohibited to use any of the old training data, because it now counts as "external data"? The 'Rules' page is a bit ambiguous:

\n

External data is not allowed without explicit, public consent (via the competition forums). Participants agree to make no attempt to use additional data sources or data sources not provided on these competition pages.  

\n

Technically, the old data was provided on these competition pages, so then it would be okay :)

", "rawMarkdown": "", "votes": null }, { "id": "37211", "postDate": "01/10/2014 10:42:52", "content": "

It is a very frustrating decision, I wasted hours...((

", "rawMarkdown": "", "votes": null }, { "id": "37227", "postDate": "01/10/2014 13:55:55", "content": "

So what was wrong with the original data? It is not completely clear to me.

\n

Was it that the original solutions_training file (size 70948 x 37) contained some wrong figures ?

\n

Thanks for the clarification.

\n

 

\n

GL

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "37231", "postDate": "01/10/2014 15:03:03", "content": "

The original solutions files had data that didn't all obey the constraints given in the decision tree (eg, that Class 10.1 + 10.2 + 10.3 = Class 4.1). We didn't have an immediate fix for the solution files, but did have the option to replace them with galaxies that DO obey the constraints. It affected enough of the galaxies that we chose to replace the full set rather than eliminate the affected ones. 

\n

Again, our sincere apologies to anyone who submitted a solution initially and had it removed. We hope most of you are still interested in the project and the prize money, and will submit solutions again. 

", "rawMarkdown": "", "votes": null }, { "id": "37234", "postDate": "01/10/2014 16:24:17", "content": "

[quote=Kyle Willett;37231]

\n

The original solutions files had data that didn't all obey the constraints given in the decision tree (eg, that Class 10.1 + 10.2 + 10.3 = Class 4.1). [/quote]

\n

Earlier you wrote:

\n

[quote]

\n

The cause is that a fraction of the original classifications for which
the total number of votes didn't completely carry through each step of
the decision tree. The cause isn't completely known; possibly recording
of incomplete classifications, or the method by which we removed
duplicate classifications.[/quote]

\n

Does this mean that the Galaxy Zoo 2 catalog contains erroneous classification data too?

", "rawMarkdown": "", "votes": null }, { "id": "37236", "postDate": "01/10/2014 16:51:04", "content": "

Yes, although I'm not sure "erroneous" is the correct description (potentially incomplete?). There's a small fraction of the classifications in which the total number of subsequent votes doesn't equal the sum above it. That being said, they're only ever off by ~+/- 1 vote, which doesn't have a significant effect on the vote fractions if there are sufficient counts. Our data and debiasing process did take this into account, and it's one of the reasons we emphasize that using the catalog intelligently should incorporate BOTH vote fractions and total numbers. 

", "rawMarkdown": "", "votes": null }, { "id": "37240", "postDate": "01/10/2014 18:10:02", "content": "", "rawMarkdown": "", "votes": null }, { "id": "37242", "postDate": "01/10/2014 18:11:38", "content": "

I dont know what's the fuss is all about when the same algorithm you used previously can give you similar results. Isnt it so? or am i missing something here?

", "rawMarkdown": "", "votes": null }, { "id": "37243", "postDate": "01/10/2014 18:25:08", "content": "

 

\n

[quote=Abhishek;37242]

\n

I dont know what's the fuss is all about when the same algorithm you used previously can give you similar results. Isnt it so? or am i missing something here?

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "37244", "postDate": "01/10/2014 18:31:01", "content": "

I see how this can be frustrating to some, especially those who spent days training deep networks and fine tuning parameters. I think the largest impact, in practical terms, will be on the number of participants and, points being a function of that number, ranking. I don't think many of the one timers who submitted the benchmark file will do so again. That means that a score that could have put you in the top 25% with the old leaderboard will likely no longer do so. But, it is what it is. Kaggle is a strange world. In theory we all want to participate to (learn and) win, but then reality kicks in and many of us work hard to get in the top 10, top 10% and top 25%. In my case, I won't be using deep learning and a more realistic goal would be to get in the top 25% with a .12 type of score. Having lost so many submission will likely disincentivize the former top 25%-ers to re-enter the competition. But the top of the leaderboard is what really matters to the sponsor. And again, it is what it is.

", "rawMarkdown": "", "votes": null }, { "id": "37246", "postDate": "01/10/2014 18:36:01", "content": "

[quote=Black Magic;37243]Dude,

\n

Downloading the data and running the same script again is an issue. I suppose there is some value for time - of course, for those that don't value their own time, there is no fuss[/quote]I don't follow this line of reasoning. If you are seriously participating in this competition (i.e. doing more than just submitting the benchmark), surely the time you're investing is orders of magnitude larger than the time it takes to download the new data and re-run your existing code? It is a bit of a nuisance, but mistakes happen. The time we lose on this doesn't weigh up to having correct training data, as far as I'm concerned.

", "rawMarkdown": "", "votes": null }, { "id": "37247", "postDate": "01/10/2014 18:44:51", "content": "", "rawMarkdown": "", "votes": null }, { "id": "37248", "postDate": "01/10/2014 18:48:01", "content": "", "rawMarkdown": "", "votes": null }, { "id": "37249", "postDate": "01/10/2014 18:49:55", "content": "

[quote=Black Magic;37247]I was at around 0.13 or 0.14 on leaderboard with a good linear model and had spent last 5 days training a deep learning network. So it is a waste of time for me

\n

Of course I will train the network again but time lost is time lost.

\n

I don't understand your line of reasoning either - if the data had been right in the first place, we need not have gone through this[/quote]

\n

I don't dispute any of this - if the data had been right the first time, that would definitely have been better. But it wasn't, because mistakes happen, and in that case I'd rather they are dealt with.

\n

And once again, I don't dispute that we've all lost some valuable time on this either. I just don't see why it's such a big deal since we're investing a much larger amount of time by participating in the first place :)

", "rawMarkdown": "", "votes": null }, { "id": "37250", "postDate": "01/10/2014 18:58:21", "content": "

yes, agree - getting the best data in place is the right thing to do. Kaggle has done the right thing.

\n

Some of us who spent time trying a memory CPU intensive activity for days will be unhappy.
Net-net: I agree that what has happened is the best thing - it is important to have right data and fix at source

\n

[quote=sedielem;37249]

\n

[quote=Black Magic;37247]I was at around 0.13 or 0.14 on leaderboard with a good linear model and had spent last 5 days training a deep learning network. So it is a waste of time for me

\n

Of course I will train the network again but time lost is time lost.

\n

I don't understand your line of reasoning either - if the data had been right in the first place, we need not have gone through this[/quote]

\n

I don't dispute any of this - if the data had been right the first time, that would definitely have been better. But it wasn't, because mistakes happen, and in that case I'd rather they are dealt with.

\n

And once again, I don't dispute that we've all lost some valuable time on this either. I just don't see why it's such a big deal since we're investing a much larger amount of time by participating in the first place :)

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "37251", "postDate": "01/10/2014 19:04:06", "content": "

Thanks for your understanding, everyone - I absolutely acknowledge that it's frustrating to have spent your CPU time, bandwidth, and own time/resources training models that aren't currently on the leaderboard. I think @BlackMagic puts it well, though - Kaggle/Galaxy Zoo badly wish this hadn't happened, but for our scientific goals, it's critically important that we have your models being fitted to the correct data. 

\n

I'm also interested in hearing about your experiences - more than 50% of the new training set of images did appear in the old set, and the way in which we sampled/selected the galaxies was identical. So while some the individual galaxies are different, the overall sample is extremely similar. Have you found that the fine-tuning parameters you've been using need significant adjustment?

", "rawMarkdown": "", "votes": null }, { "id": "37256", "postDate": "01/10/2014 21:02:51", "content": "

Black Magic - I've literally just installed a new graphics card in my PC so that I can learn deep learning on the gpu. I plan to use pylearn2 and theano. Is this what you use? I'm going to play with the dogs and cats competition first then will move to this one. You said that your net has been running for 5 days - is this usual for deep learning? I'm excited about learning it and am now spending the rest of the evening installing the software. Any hints or tips or anything you wish you knew before you started?

", "rawMarkdown": "", "votes": null }, { "id": "37264", "postDate": "01/11/2014 00:20:25", "content": "

[quote=Black Magic;37250]

\n

yes, agree - getting the best data in place is the right thing to do. Kaggle has done the right thing.

\n

Some of us who spent time trying a memory CPU intensive activity for days will be unhappy.
Net-net: I agree that what has happened is the best thing - it is important to have right data and fix at source

\n

[/quote]

\n

It took me about one day to train a model on personal computer without GPU. Computing resources and time will limit the performance in addition to the model itself. I can spend less than 10 hours on this competition per week although my computer can spend more.

\n

Since it's several months before deadline there will be enough time for all guys.

", "rawMarkdown": "", "votes": null }, { "id": "37278", "postDate": "01/11/2014 10:59:11", "content": "

Given your remark that only about 50% of the old train set is in the new train set, and the fact that a good 90% of the old train set seemed to have correct labels (abiding by the constraints, at least), I'd like to reiterate this question :)

\n

I'm assuming the answer will be no, since this wouldn't exactly be fair to people who didn't get a chance to download the old data. But a definitive answer would be great nevertheless!

\n

[quote=sedielem;37210]I'm assuming it is now prohibited to use any of the old training data, because it now counts as "external data"? The 'Rules' page is a bit ambiguous:

\n

External data is not allowed without explicit, public consent (via the competition forums). Participants agree to make no attempt to use additional data sources or data sources not provided on these competition pages.  

\n

Technically, the old data was provided on these competition pages, so then it would be okay :)[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "37309", "postDate": "01/12/2014 12:49:25", "content": "", "rawMarkdown": "", "votes": null }, { "id": "37324", "postDate": "01/12/2014 19:43:21", "content": "

[quote=Domcastro;37256]

\n

Black Magic - I've literally just installed a new graphics card in my PC so that I can learn deep learning on the gpu. I plan to use pylearn2 and theano. Is this what you use? I'm going to play with the dogs and cats competition first then will move to this one. You said that your net has been running for 5 days - is this usual for deep learning? I'm excited about learning it and am now spending the rest of the evening installing the software. Any hints or tips or anything you wish you knew before you started?

\n

[/quote]

\n

 

\n

My basic model is a deep autoencoder network trained using Theano with a GTX 660 GPU.  It takes about 1 hour to train which gets me a leaderboard score of 1.4779.  So you can certainly achieve reasonable results with deep learning without training for 5 days - having said that, I am still only 25th on the leaderboard! 

", "rawMarkdown": "", "votes": null }, { "id": "37744", "postDate": "01/22/2014 18:19:45", "content": "

[quote=sedielem;37278]

\n

Given your remark that only about 50% of the old train set is in the new train set, and the fact that a good 90% of the old train set seemed to have correct labels (abiding by the constraints, at least), I'd like to reiterate this question :)

\n

I'm assuming the answer will be no, since this wouldn't exactly be fair to people who didn't get a chance to download the old data. But a definitive answer would be great nevertheless!

\n

[quote=sedielem;37210]I'm assuming it is now prohibited to use any of the old training data, because it now counts as "external data"? The 'Rules' page is a bit ambiguous:

\n

External data is not allowed without explicit, public consent (via the competition forums). Participants agree to make no attempt to use additional data sources or data sources not provided on these competition pages.  

\n

Technically, the old data was provided on these competition pages, so then it would be okay :)[/quote][/quote]

\n

Was this - excellent - question ever answered? If so, what was the answer?

", "rawMarkdown": "", "votes": null }, { "id": "37773", "postDate": "01/23/2014 00:38:21", "content": "

In the sense that any insights or techniques developed were on the original data, participants may use it. However, given that some of the input constraints were incorrect (which is why it was updated and replaced), I very much doubt training your model on it would yield better results.  

", "rawMarkdown": "", "votes": null }, { "id": "37785", "postDate": "01/23/2014 10:26:44", "content": "

My question was more about whether we can include it as training data for our (current) models. The script I wrote to check the constraints indicated that only a good 3000 datapoints from the old training set were affected; the rest seemed to have correct labels (or "correct enough", anyway). Since the overlap with the current dataset is ~50%, that leaves plenty of valuable datapoints that could be used as additional training data.

\n

I don't want to risk being disqualified, so if using this data as training data is not allowed, it would be great if one of the organisers could confirm this explicitly. But if it is allowed, then I am definitely going to give it a try, as I suspect this could improve results. More data is better :)

\n

That said, in the spirit of the competition I hope it will be explicitly disallowed, as it would be unfair to newer participants, unless the old dataset is made publicly available again.

", "rawMarkdown": "", "votes": null }, { "id": "37808", "postDate": "01/23/2014 16:44:29", "content": "

I agree with the last point - all competitors should have equal access to data. Given that, the earlier training data are explicitly disallowed; solutions must be based on only the current sets of data that are posted. I apologize if this was not expressly clear before. 

", "rawMarkdown": "", "votes": null }, { "id": "37809", "postDate": "01/23/2014 16:50:23", "content": "

What about the test data, against which the submissions will be tested? Does the test data contain "a fraction of the original classifications for which the total number of votes didn't completely carry through each step of the decision tree" (as you described it earlier)? Or has the test data also been checked, and any such removed?

", "rawMarkdown": "", "votes": null }, { "id": "37810", "postDate": "01/23/2014 16:57:04", "content": "

Both the test data and training data should obey the constraints in the decision tree 100%. 

", "rawMarkdown": "", "votes": null }, { "id": "37835", "postDate": "01/23/2014 21:50:33", "content": "

Strange, unless I am mistaken, for several objects, the sum of the probabilities to end the process is greater than 1 !

\n

For example, for galaxy #100078, the sum of the classes 1.3 (end from 1) + 6.2 (end from 6) and 8.n (7 ways to end from 8) = 0.0680590 + 0.6796020 + 0.0961194 + 0.0961194 + 0.1281592 + 0 + 0 + 0 + 0 = 1.0680590 > 1.0. What am I missing ?

\n

There are also 3 nodes where the sum of probabilities going in is different than the sum of probabilities going out (nodes 5, 6 and 10).

\n

 

\n

Am I wrong ? Hopefully, P(me = "wrong") < 1.0 !

", "rawMarkdown": "", "votes": null }, { "id": "38499", "postDate": "02/08/2014 12:24:16", "content": "

[quote=Kyle Willett;37191]

\n

We've added an additional two weeks to the competition deadline

\n

[/quote]

\n

Did you update Timeline page? It says the competition ends on March 21, while the bar at the top of the page indicates April 4 as the finish date. What is the correct date?

\n

Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "38504", "postDate": "02/08/2014 15:53:49", "content": "

[quote=Maxim Milakov;38499]

\n

Did you update Timeline page? It says the competition ends on March 21, while the bar at the top of the page indicates April 4 as the finish date. What is the correct date?

\n

Thanks!

\n

[/quote]

\n

Done, thanks for the pointer.

", "rawMarkdown": "", "votes": null }, { "id": "38737", "postDate": "02/13/2014 19:10:21", "content": "

Hi, I'm new to this and I've been enjoying the galaxy data. I only now found out about the new data! 

\n

 

\n

As I've been reading through this forum to learn more about the change, I found the following statement from Kyle: "Our data and debiasing process did take this into account, and it's one of the reasons we emphasize that using the catalog intelligently should incorporate BOTH vote fractions and total numbers. "

\n

 

\n

Where can I find this advice about how to use the catalog intelligently? I looked through everything I can think of, but I can't find a place where this is described. Sorry, I'm sure it's just because I'm new at this. I find the catalog really weird because it's sort of probabilities and sort of not. So I've been trying to understand it better, and any advice would help.

\n

 

\n

Thanks for your help

", "rawMarkdown": "", "votes": null }, { "id": "38740", "postDate": "02/13/2014 19:41:53", "content": "

To be more specific: How do I know what the total numbers of votes are? I know what the fractions are, that's what's in the solutions file. But how do I know how many votes went into each fraction? This is also why I'm not sure how to implement the evaluation metric, rmse, which specifies that N includes number of responses. How do I get that information?

\n

 

\n

Thanks for your patience with a first-time-kaggler 

\n

 

\n

s

", "rawMarkdown": "", "votes": null }, { "id": "38820", "postDate": "02/14/2014 18:55:26", "content": "

The data you've been given for the Kaggle competition doesn't include the number of votes specifically. We stressed that it's helpful for astronomers, but wanted to focus the ML in this case on a narrower set of constraints and slightly less complicated data set. So you only will have access to the normalized vote fractions for the competition.

\n

 

\n

I can tell you that the range in total votes for each galaxy is not very big - the median was 42.6 +- 5.8. In addition, none of the scientific analysis indicated significant differences in the vote fractions as a function of the total number of votes. 

", "rawMarkdown": "", "votes": null }, { "id": "38823", "postDate": "02/14/2014 19:26:29", "content": "

Dear Kyle.

\n

Thanks for clarifying. So data from all galaxies should be equally weighted for this project.

\n

Am I correct then in understanding that all values in the solution set are weighted equally when calculating the RMSE? Regardless of whether the column represents a high-level or low-level category?

\n

Best,

\n

Shani

", "rawMarkdown": "", "votes": null }, { "id": "38834", "postDate": "02/14/2014 23:08:03", "content": "

The values will be weighted equally, but the nature of an RMSE solution means that columns for low-level categories will be less important for the global solution (since the values are lower). Getting the high-level categories correct should result in a more accurate solution, although you will need both in order to have one that is competitive on the leaderboard. 

", "rawMarkdown": "", "votes": null }, { "id": "39046", "postDate": "02/18/2014 16:27:10", "content": "

Hi Kyle,

\n

Thanks for your response.

\n

I don't see why that follows. Why should error be proportional to value? For example, if I have an algorithm that does very well with high-level classification, but is bad with the lower levels, I would expect a higher error for the lower levels.

\n

To be concrete in a cartoon way: Say there were 2 levels. Level 1 was important and Level 2 was less important. The true value for Level 1 was .998. The true value for Level 2 was .03. If my algorithm predicts 1 for Level 1 and 0 for Level 2, I will be getting the more important answer correct, but my RMSE will be dominated by the error for level 2 (.03>>.002)

\n

I think the error has to be weighted in some way in order to make the less important categories have less impact on the error calculation.

\n

What do you think?

\n

Shani

", "rawMarkdown": "", "votes": null }, { "id": "39055", "postDate": "02/18/2014 18:13:42", "content": "

I think he means that the range for the values in the lower level categories are generally smaller than the higher level categories, so your RMSE should be less for these categories.

\n

For example - the higher level category could be anywhere in [0,1], but the lower level ones may in [0, 0.05], giving a wider prediction possibility for the higher level categories and thus a higher RMSE.

", "rawMarkdown": "", "votes": null }, { "id": "39060", "postDate": "02/18/2014 19:06:36", "content": "

Exactly.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 37202, "author_name": "ryankiros", "author_url": "", "post_date": "01/10/2014 07:34:02", "content": "

I was anticipating new labels, but new images and a reset of the leaderboard? I won't be submitting again but I wish you all the best and hope you get a great result :)

", "votes": null, "replies": [] }, { "id": 37204, "author_name": "alexandrekowalczyk", "author_url": "", "post_date": "01/10/2014 08:30:31", "content": "

Nice. I think this is an appropriate decision and hope the dataset is now cleaner :)

\n

Thanks for your reactivity.

", "votes": null, "replies": [] }, { "id": 37210, "author_name": "sedielem", "author_url": "", "post_date": "01/10/2014 10:36:41", "content": "

Thanks for fixing the data! I didn't expect the images to be replaced as well, but I'm glad this was addressed.

\n

I'm assuming it is now prohibited to use any of the old training data, because it now counts as "external data"? The 'Rules' page is a bit ambiguous:

\n

External data is not allowed without explicit, public consent (via the competition forums). Participants agree to make no attempt to use additional data sources or data sources not provided on these competition pages.  

\n

Technically, the old data was provided on these competition pages, so then it would be okay :)

", "votes": null, "replies": [] }, { "id": 37211, "author_name": "sbachish", "author_url": "", "post_date": "01/10/2014 10:42:52", "content": "

It is a very frustrating decision, I wasted hours...((

", "votes": null, "replies": [] }, { "id": 37227, "author_name": "gregoirel", "author_url": "", "post_date": "01/10/2014 13:55:55", "content": "

So what was wrong with the original data? It is not completely clear to me.

\n

Was it that the original solutions_training file (size 70948 x 37) contained some wrong figures ?

\n

Thanks for the clarification.

\n

 

\n

GL

\n

 

", "votes": null, "replies": [] }, { "id": 37231, "author_name": "willettk", "author_url": "", "post_date": "01/10/2014 15:03:03", "content": "

The original solutions files had data that didn't all obey the constraints given in the decision tree (eg, that Class 10.1 + 10.2 + 10.3 = Class 4.1). We didn't have an immediate fix for the solution files, but did have the option to replace them with galaxies that DO obey the constraints. It affected enough of the galaxies that we chose to replace the full set rather than eliminate the affected ones. 

\n

Again, our sincere apologies to anyone who submitted a solution initially and had it removed. We hope most of you are still interested in the project and the prize money, and will submit solutions again. 

", "votes": null, "replies": [] }, { "id": 37234, "author_name": "jeantate", "author_url": "", "post_date": "01/10/2014 16:24:17", "content": "

[quote=Kyle Willett;37231]

\n

The original solutions files had data that didn't all obey the constraints given in the decision tree (eg, that Class 10.1 + 10.2 + 10.3 = Class 4.1). [/quote]

\n

Earlier you wrote:

\n

[quote]

\n

The cause is that a fraction of the original classifications for which
the total number of votes didn't completely carry through each step of
the decision tree. The cause isn't completely known; possibly recording
of incomplete classifications, or the method by which we removed
duplicate classifications.[/quote]

\n

Does this mean that the Galaxy Zoo 2 catalog contains erroneous classification data too?

", "votes": null, "replies": [] }, { "id": 37236, "author_name": "willettk", "author_url": "", "post_date": "01/10/2014 16:51:04", "content": "

Yes, although I'm not sure "erroneous" is the correct description (potentially incomplete?). There's a small fraction of the classifications in which the total number of subsequent votes doesn't equal the sum above it. That being said, they're only ever off by ~+/- 1 vote, which doesn't have a significant effect on the vote fractions if there are sufficient counts. Our data and debiasing process did take this into account, and it's one of the reasons we emphasize that using the catalog intelligently should incorporate BOTH vote fractions and total numbers. 

", "votes": null, "replies": [] }, { "id": 37240, "author_name": "rkirana", "author_url": "", "post_date": "01/10/2014 18:10:02", "content": "", "votes": null, "replies": [] }, { "id": 37242, "author_name": "abhishek", "author_url": "", "post_date": "01/10/2014 18:11:38", "content": "

I dont know what's the fuss is all about when the same algorithm you used previously can give you similar results. Isnt it so? or am i missing something here?

", "votes": null, "replies": [] }, { "id": 37243, "author_name": "rkirana", "author_url": "", "post_date": "01/10/2014 18:25:08", "content": "

 

\n

[quote=Abhishek;37242]

\n

I dont know what's the fuss is all about when the same algorithm you used previously can give you similar results. Isnt it so? or am i missing something here?

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 37244, "author_name": "adjgiulio", "author_url": "", "post_date": "01/10/2014 18:31:01", "content": "

I see how this can be frustrating to some, especially those who spent days training deep networks and fine tuning parameters. I think the largest impact, in practical terms, will be on the number of participants and, points being a function of that number, ranking. I don't think many of the one timers who submitted the benchmark file will do so again. That means that a score that could have put you in the top 25% with the old leaderboard will likely no longer do so. But, it is what it is. Kaggle is a strange world. In theory we all want to participate to (learn and) win, but then reality kicks in and many of us work hard to get in the top 10, top 10% and top 25%. In my case, I won't be using deep learning and a more realistic goal would be to get in the top 25% with a .12 type of score. Having lost so many submission will likely disincentivize the former top 25%-ers to re-enter the competition. But the top of the leaderboard is what really matters to the sponsor. And again, it is what it is.

", "votes": null, "replies": [] }, { "id": 37246, "author_name": "sedielem", "author_url": "", "post_date": "01/10/2014 18:36:01", "content": "

[quote=Black Magic;37243]Dude,

\n

Downloading the data and running the same script again is an issue. I suppose there is some value for time - of course, for those that don't value their own time, there is no fuss[/quote]I don't follow this line of reasoning. If you are seriously participating in this competition (i.e. doing more than just submitting the benchmark), surely the time you're investing is orders of magnitude larger than the time it takes to download the new data and re-run your existing code? It is a bit of a nuisance, but mistakes happen. The time we lose on this doesn't weigh up to having correct training data, as far as I'm concerned.

", "votes": null, "replies": [] }, { "id": 37247, "author_name": "rkirana", "author_url": "", "post_date": "01/10/2014 18:44:51", "content": "", "votes": null, "replies": [] }, { "id": 37248, "author_name": "rkirana", "author_url": "", "post_date": "01/10/2014 18:48:01", "content": "", "votes": null, "replies": [] }, { "id": 37249, "author_name": "sedielem", "author_url": "", "post_date": "01/10/2014 18:49:55", "content": "

[quote=Black Magic;37247]I was at around 0.13 or 0.14 on leaderboard with a good linear model and had spent last 5 days training a deep learning network. So it is a waste of time for me

\n

Of course I will train the network again but time lost is time lost.

\n

I don't understand your line of reasoning either - if the data had been right in the first place, we need not have gone through this[/quote]

\n

I don't dispute any of this - if the data had been right the first time, that would definitely have been better. But it wasn't, because mistakes happen, and in that case I'd rather they are dealt with.

\n

And once again, I don't dispute that we've all lost some valuable time on this either. I just don't see why it's such a big deal since we're investing a much larger amount of time by participating in the first place :)

", "votes": null, "replies": [] }, { "id": 37250, "author_name": "rkirana", "author_url": "", "post_date": "01/10/2014 18:58:21", "content": "

yes, agree - getting the best data in place is the right thing to do. Kaggle has done the right thing.

\n

Some of us who spent time trying a memory CPU intensive activity for days will be unhappy.
Net-net: I agree that what has happened is the best thing - it is important to have right data and fix at source

\n

[quote=sedielem;37249]

\n

[quote=Black Magic;37247]I was at around 0.13 or 0.14 on leaderboard with a good linear model and had spent last 5 days training a deep learning network. So it is a waste of time for me

\n

Of course I will train the network again but time lost is time lost.

\n

I don't understand your line of reasoning either - if the data had been right in the first place, we need not have gone through this[/quote]

\n

I don't dispute any of this - if the data had been right the first time, that would definitely have been better. But it wasn't, because mistakes happen, and in that case I'd rather they are dealt with.

\n

And once again, I don't dispute that we've all lost some valuable time on this either. I just don't see why it's such a big deal since we're investing a much larger amount of time by participating in the first place :)

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 37251, "author_name": "willettk", "author_url": "", "post_date": "01/10/2014 19:04:06", "content": "

Thanks for your understanding, everyone - I absolutely acknowledge that it's frustrating to have spent your CPU time, bandwidth, and own time/resources training models that aren't currently on the leaderboard. I think @BlackMagic puts it well, though - Kaggle/Galaxy Zoo badly wish this hadn't happened, but for our scientific goals, it's critically important that we have your models being fitted to the correct data. 

\n

I'm also interested in hearing about your experiences - more than 50% of the new training set of images did appear in the old set, and the way in which we sampled/selected the galaxies was identical. So while some the individual galaxies are different, the overall sample is extremely similar. Have you found that the fine-tuning parameters you've been using need significant adjustment?

", "votes": null, "replies": [] }, { "id": 37256, "author_name": "domcastro", "author_url": "", "post_date": "01/10/2014 21:02:51", "content": "

Black Magic - I've literally just installed a new graphics card in my PC so that I can learn deep learning on the gpu. I plan to use pylearn2 and theano. Is this what you use? I'm going to play with the dogs and cats competition first then will move to this one. You said that your net has been running for 5 days - is this usual for deep learning? I'm excited about learning it and am now spending the rest of the evening installing the software. Any hints or tips or anything you wish you knew before you started?

", "votes": null, "replies": [] }, { "id": 37264, "author_name": "waynezhang", "author_url": "", "post_date": "01/11/2014 00:20:25", "content": "

[quote=Black Magic;37250]

\n

yes, agree - getting the best data in place is the right thing to do. Kaggle has done the right thing.

\n

Some of us who spent time trying a memory CPU intensive activity for days will be unhappy.
Net-net: I agree that what has happened is the best thing - it is important to have right data and fix at source

\n

[/quote]

\n

It took me about one day to train a model on personal computer without GPU. Computing resources and time will limit the performance in addition to the model itself. I can spend less than 10 hours on this competition per week although my computer can spend more.

\n

Since it's several months before deadline there will be enough time for all guys.

", "votes": null, "replies": [] }, { "id": 37278, "author_name": "sedielem", "author_url": "", "post_date": "01/11/2014 10:59:11", "content": "

Given your remark that only about 50% of the old train set is in the new train set, and the fact that a good 90% of the old train set seemed to have correct labels (abiding by the constraints, at least), I'd like to reiterate this question :)

\n

I'm assuming the answer will be no, since this wouldn't exactly be fair to people who didn't get a chance to download the old data. But a definitive answer would be great nevertheless!

\n

[quote=sedielem;37210]I'm assuming it is now prohibited to use any of the old training data, because it now counts as "external data"? The 'Rules' page is a bit ambiguous:

\n

External data is not allowed without explicit, public consent (via the competition forums). Participants agree to make no attempt to use additional data sources or data sources not provided on these competition pages.  

\n

Technically, the old data was provided on these competition pages, so then it would be okay :)[/quote]

", "votes": null, "replies": [] }, { "id": 37309, "author_name": "sandeepritu", "author_url": "", "post_date": "01/12/2014 12:49:25", "content": "", "votes": null, "replies": [] }, { "id": 37324, "author_name": "senecaur", "author_url": "", "post_date": "01/12/2014 19:43:21", "content": "

[quote=Domcastro;37256]

\n

Black Magic - I've literally just installed a new graphics card in my PC so that I can learn deep learning on the gpu. I plan to use pylearn2 and theano. Is this what you use? I'm going to play with the dogs and cats competition first then will move to this one. You said that your net has been running for 5 days - is this usual for deep learning? I'm excited about learning it and am now spending the rest of the evening installing the software. Any hints or tips or anything you wish you knew before you started?

\n

[/quote]

\n

 

\n

My basic model is a deep autoencoder network trained using Theano with a GTX 660 GPU.  It takes about 1 hour to train which gets me a leaderboard score of 1.4779.  So you can certainly achieve reasonable results with deep learning without training for 5 days - having said that, I am still only 25th on the leaderboard! 

", "votes": null, "replies": [] }, { "id": 37744, "author_name": "jeantate", "author_url": "", "post_date": "01/22/2014 18:19:45", "content": "

[quote=sedielem;37278]

\n

Given your remark that only about 50% of the old train set is in the new train set, and the fact that a good 90% of the old train set seemed to have correct labels (abiding by the constraints, at least), I'd like to reiterate this question :)

\n

I'm assuming the answer will be no, since this wouldn't exactly be fair to people who didn't get a chance to download the old data. But a definitive answer would be great nevertheless!

\n

[quote=sedielem;37210]I'm assuming it is now prohibited to use any of the old training data, because it now counts as "external data"? The 'Rules' page is a bit ambiguous:

\n

External data is not allowed without explicit, public consent (via the competition forums). Participants agree to make no attempt to use additional data sources or data sources not provided on these competition pages.  

\n

Technically, the old data was provided on these competition pages, so then it would be okay :)[/quote][/quote]

\n

Was this - excellent - question ever answered? If so, what was the answer?

", "votes": null, "replies": [] }, { "id": 37773, "author_name": "willettk", "author_url": "", "post_date": "01/23/2014 00:38:21", "content": "

In the sense that any insights or techniques developed were on the original data, participants may use it. However, given that some of the input constraints were incorrect (which is why it was updated and replaced), I very much doubt training your model on it would yield better results.  

", "votes": null, "replies": [] }, { "id": 37785, "author_name": "sedielem", "author_url": "", "post_date": "01/23/2014 10:26:44", "content": "

My question was more about whether we can include it as training data for our (current) models. The script I wrote to check the constraints indicated that only a good 3000 datapoints from the old training set were affected; the rest seemed to have correct labels (or "correct enough", anyway). Since the overlap with the current dataset is ~50%, that leaves plenty of valuable datapoints that could be used as additional training data.

\n

I don't want to risk being disqualified, so if using this data as training data is not allowed, it would be great if one of the organisers could confirm this explicitly. But if it is allowed, then I am definitely going to give it a try, as I suspect this could improve results. More data is better :)

\n

That said, in the spirit of the competition I hope it will be explicitly disallowed, as it would be unfair to newer participants, unless the old dataset is made publicly available again.

", "votes": null, "replies": [] }, { "id": 37808, "author_name": "willettk", "author_url": "", "post_date": "01/23/2014 16:44:29", "content": "

I agree with the last point - all competitors should have equal access to data. Given that, the earlier training data are explicitly disallowed; solutions must be based on only the current sets of data that are posted. I apologize if this was not expressly clear before. 

", "votes": null, "replies": [] }, { "id": 37809, "author_name": "jeantate", "author_url": "", "post_date": "01/23/2014 16:50:23", "content": "

What about the test data, against which the submissions will be tested? Does the test data contain "a fraction of the original classifications for which the total number of votes didn't completely carry through each step of the decision tree" (as you described it earlier)? Or has the test data also been checked, and any such removed?

", "votes": null, "replies": [] }, { "id": 37810, "author_name": "willettk", "author_url": "", "post_date": "01/23/2014 16:57:04", "content": "

Both the test data and training data should obey the constraints in the decision tree 100%. 

", "votes": null, "replies": [] }, { "id": 37835, "author_name": "", "author_url": "", "post_date": "01/23/2014 21:50:33", "content": "

Strange, unless I am mistaken, for several objects, the sum of the probabilities to end the process is greater than 1 !

\n

For example, for galaxy #100078, the sum of the classes 1.3 (end from 1) + 6.2 (end from 6) and 8.n (7 ways to end from 8) = 0.0680590 + 0.6796020 + 0.0961194 + 0.0961194 + 0.1281592 + 0 + 0 + 0 + 0 = 1.0680590 > 1.0. What am I missing ?

\n

There are also 3 nodes where the sum of probabilities going in is different than the sum of probabilities going out (nodes 5, 6 and 10).

\n

 

\n

Am I wrong ? Hopefully, P(me = "wrong") < 1.0 !

", "votes": null, "replies": [] }, { "id": 38499, "author_name": "milakov", "author_url": "", "post_date": "02/08/2014 12:24:16", "content": "

[quote=Kyle Willett;37191]

\n

We've added an additional two weeks to the competition deadline

\n

[/quote]

\n

Did you update Timeline page? It says the competition ends on March 21, while the bar at the top of the page indicates April 4 as the finish date. What is the correct date?

\n

Thanks!

", "votes": null, "replies": [] }, { "id": 38504, "author_name": "joycenv", "author_url": "", "post_date": "02/08/2014 15:53:49", "content": "

[quote=Maxim Milakov;38499]

\n

Did you update Timeline page? It says the competition ends on March 21, while the bar at the top of the page indicates April 4 as the finish date. What is the correct date?

\n

Thanks!

\n

[/quote]

\n

Done, thanks for the pointer.

", "votes": null, "replies": [] }, { "id": 38737, "author_name": "ssinger", "author_url": "", "post_date": "02/13/2014 19:10:21", "content": "

Hi, I'm new to this and I've been enjoying the galaxy data. I only now found out about the new data! 

\n

 

\n

As I've been reading through this forum to learn more about the change, I found the following statement from Kyle: "Our data and debiasing process did take this into account, and it's one of the reasons we emphasize that using the catalog intelligently should incorporate BOTH vote fractions and total numbers. "

\n

 

\n

Where can I find this advice about how to use the catalog intelligently? I looked through everything I can think of, but I can't find a place where this is described. Sorry, I'm sure it's just because I'm new at this. I find the catalog really weird because it's sort of probabilities and sort of not. So I've been trying to understand it better, and any advice would help.

\n

 

\n

Thanks for your help

", "votes": null, "replies": [] }, { "id": 38740, "author_name": "ssinger", "author_url": "", "post_date": "02/13/2014 19:41:53", "content": "

To be more specific: How do I know what the total numbers of votes are? I know what the fractions are, that's what's in the solutions file. But how do I know how many votes went into each fraction? This is also why I'm not sure how to implement the evaluation metric, rmse, which specifies that N includes number of responses. How do I get that information?

\n

 

\n

Thanks for your patience with a first-time-kaggler 

\n

 

\n

s

", "votes": null, "replies": [] }, { "id": 38820, "author_name": "willettk", "author_url": "", "post_date": "02/14/2014 18:55:26", "content": "

The data you've been given for the Kaggle competition doesn't include the number of votes specifically. We stressed that it's helpful for astronomers, but wanted to focus the ML in this case on a narrower set of constraints and slightly less complicated data set. So you only will have access to the normalized vote fractions for the competition.

\n

 

\n

I can tell you that the range in total votes for each galaxy is not very big - the median was 42.6 +- 5.8. In addition, none of the scientific analysis indicated significant differences in the vote fractions as a function of the total number of votes. 

", "votes": null, "replies": [] }, { "id": 38823, "author_name": "ssinger", "author_url": "", "post_date": "02/14/2014 19:26:29", "content": "

Dear Kyle.

\n

Thanks for clarifying. So data from all galaxies should be equally weighted for this project.

\n

Am I correct then in understanding that all values in the solution set are weighted equally when calculating the RMSE? Regardless of whether the column represents a high-level or low-level category?

\n

Best,

\n

Shani

", "votes": null, "replies": [] }, { "id": 38834, "author_name": "willettk", "author_url": "", "post_date": "02/14/2014 23:08:03", "content": "

The values will be weighted equally, but the nature of an RMSE solution means that columns for low-level categories will be less important for the global solution (since the values are lower). Getting the high-level categories correct should result in a more accurate solution, although you will need both in order to have one that is competitive on the leaderboard. 

", "votes": null, "replies": [] }, { "id": 39046, "author_name": "ssinger", "author_url": "", "post_date": "02/18/2014 16:27:10", "content": "

Hi Kyle,

\n

Thanks for your response.

\n

I don't see why that follows. Why should error be proportional to value? For example, if I have an algorithm that does very well with high-level classification, but is bad with the lower levels, I would expect a higher error for the lower levels.

\n

To be concrete in a cartoon way: Say there were 2 levels. Level 1 was important and Level 2 was less important. The true value for Level 1 was .998. The true value for Level 2 was .03. If my algorithm predicts 1 for Level 1 and 0 for Level 2, I will be getting the more important answer correct, but my RMSE will be dominated by the error for level 2 (.03>>.002)

\n

I think the error has to be weighted in some way in order to make the less important categories have less impact on the error calculation.

\n

What do you think?

\n

Shani

", "votes": null, "replies": [] }, { "id": 39055, "author_name": "dkaylor", "author_url": "", "post_date": "02/18/2014 18:13:42", "content": "

I think he means that the range for the values in the lower level categories are generally smaller than the higher level categories, so your RMSE should be less for these categories.

\n

For example - the higher level category could be anywhere in [0,1], but the lower level ones may in [0, 0.05], giving a wider prediction possibility for the higher level categories and thus a higher RMSE.

", "votes": null, "replies": [] }, { "id": 39060, "author_name": "willettk", "author_url": "", "post_date": "02/18/2014 19:06:36", "content": "

Exactly.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "37191": "", "37202": "", "37204": "", "37210": "", "37211": "", "37227": "", "37231": "", "37234": "", "37236": "", "37240": "", "37242": "", "37243": "", "37244": "", "37246": "", "37247": "", "37248": "", "37249": "", "37250": "", "37251": "", "37256": "", "37264": "", "37278": "", "37309": "", "37324": "", "37744": "", "37773": "", "37785": "", "37808": "", "37809": "", "37810": "", "37835": "", "38499": "", "38504": "", "38737": "", "38740": "", "38820": "", "38823": "", "38834": "", "39046": "", "39055": "", "39060": "" }, "source": "meta" }