I was writing code to compute the RMSE, and was testing it by running on the central pixel benchmark CSV. After much head scratching as to why my code insisted on finding an empty intersection between the GalaxyIDs of the solutions CSV and the benchmark CSV, I finally started looking at the raw data and... it looks like the three benchmark files are all run against a set of GalaxyIDs that are no longer in the current solutions CSV.
\n\n
Is this correct? Will we get updated files - it's hard to verify I'm calculating the error correctly without at least one sample file with known error to check against.
\n
Thank you,
Shayne Hodge
I was writing code to compute the RMSE, and was testing it by running on the central pixel benchmark CSV. After much head scratching as to why my code insisted on finding an empty intersection between the GalaxyIDs of the solutions CSV and the benchmark CSV, I finally started looking at the raw data and... it looks like the three benchmark files are all run against a set of GalaxyIDs that are no longer in the current solutions CSV.
\n\n
Is this correct? Will we get updated files - it's hard to verify I'm calculating the error correctly without at least one sample file with known error to check against.
\n
Thank you,
Shayne Hodge
Hi Shayne,
\nI double checked that the benchmarks are consistent with the test set. If the labels for GalaxyIds did not exactly match up, the scorer would return an error. Are you sure you are working with the latest versions of the benchmarks? Could you re-download them and check?
", "rawMarkdown": "", "votes": null }, { "id": "37644", "postDate": "01/20/2014 17:11:19", "content": "I just downloaded the central_pixel csv and the training...rev1.csv, opened in Excel, and sorted smallest-to-largest on Galaxy ID to be safe. Can't insert screenshots, here's the first 10 of each copy/pasted:
\n\n
Training:
\n| GalaxyID | \n
| 100008 | \n
| 100023 | \n
| 100053 | \n
| 100078 | \n
| 100090 | \n
| 100122 | \n
| 100123 | \n
| 100128 | \n
| 100134 | \n
Central:
| GalaxyID | \n
| 100018 | \n
| 100037 | \n
| 100042 | \n
| 100052 | \n
| 100056 | \n
| 100058 | \n
| 100062 | \n
| 100065 | \n
| 100071 | \n
| 100076 | \n
?
\nThanks,
\n
Shayne
\n
", "rawMarkdown": "", "votes": null }, { "id": "37660", "postDate": "01/21/2014 04:05:07", "content": "
The central pixel benchmark is for the test set, not the training set. The test set and training set do not (and should not) have overlapping galaxies. Your submission file should give your predictions for the probabilities of test set galaxies.
\nThis benchmark did not require "training" as it is based on the central pixel's color. You can apply this benchmark to the training set and see how well you do since you have the answers for the training set.
", "rawMarkdown": "", "votes": null }, { "id": "37661", "postDate": "01/21/2014 04:51:13", "content": "[quote=joycenv;37660]
\nThe central pixel benchmark is for the test set, not the training set. The test set and training set do not (and should not) have overlapping galaxies. [/quote]
\nOops (and duh!). This thread shall now stand in perpetuity about the dangers of not paying enough attention and making extremely silly mistakes.
\nThanks for the help,
\nShayne
\n", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 37642, "author_name": "joycenv", "author_url": "", "post_date": "01/20/2014 15:35:50", "content": "
Hi Shayne,
\nI double checked that the benchmarks are consistent with the test set. If the labels for GalaxyIds did not exactly match up, the scorer would return an error. Are you sure you are working with the latest versions of the benchmarks? Could you re-download them and check?
", "votes": null, "replies": [] }, { "id": 37644, "author_name": "schodge", "author_url": "", "post_date": "01/20/2014 17:11:19", "content": "I just downloaded the central_pixel csv and the training...rev1.csv, opened in Excel, and sorted smallest-to-largest on Galaxy ID to be safe. Can't insert screenshots, here's the first 10 of each copy/pasted:
\n\n
Training:
\n| GalaxyID | \n
| 100008 | \n
| 100023 | \n
| 100053 | \n
| 100078 | \n
| 100090 | \n
| 100122 | \n
| 100123 | \n
| 100128 | \n
| 100134 | \n
Central:
| GalaxyID | \n
| 100018 | \n
| 100037 | \n
| 100042 | \n
| 100052 | \n
| 100056 | \n
| 100058 | \n
| 100062 | \n
| 100065 | \n
| 100071 | \n
| 100076 | \n
?
\nThanks,
\n
Shayne
\n
", "votes": null, "replies": [] }, { "id": 37660, "author_name": "joycenv", "author_url": "", "post_date": "01/21/2014 04:05:07", "content": "
The central pixel benchmark is for the test set, not the training set. The test set and training set do not (and should not) have overlapping galaxies. Your submission file should give your predictions for the probabilities of test set galaxies.
\nThis benchmark did not require "training" as it is based on the central pixel's color. You can apply this benchmark to the training set and see how well you do since you have the answers for the training set.
", "votes": null, "replies": [] }, { "id": 37661, "author_name": "schodge", "author_url": "", "post_date": "01/21/2014 04:51:13", "content": "[quote=joycenv;37660]
\nThe central pixel benchmark is for the test set, not the training set. The test set and training set do not (and should not) have overlapping galaxies. [/quote]
\nOops (and duh!). This thread shall now stand in perpetuity about the dangers of not paying enough attention and making extremely silly mistakes.
\nThanks for the help,
\nShayne
\n", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "37617": "", "37642": "", "37644": "", "37660": "", "37661": "" }, "source": "meta" }