Interesting competition. Thank you!
\r\nLet me briefly describe the approach / findings of the team Supernova. I believe it is quite different from what I have read here so far.
\r\nWe had the public score 0.72590 and the consistent private score of 0.72645.
\r\nOmitting the first few weeks of the competition when we tried approaches similar to those already mentioned, we then (finally) managed to compile and run lenstool and directed our efforts in this direction. Our current approach could be named "lenstool wrapper"\r\n or "lenstool ensemble." It can be seen as a random forest with (randomized) lenstool predictions as base learners. Steps of our algorithm for a fixed sky:
\r\nGood point, it would make more sense to "weight" the angular error term based on the halo mass.
\r\nLet me clarify my argument: Let's say you have accurate estimates of 1Halo skies, and a fixed solution of 2Halo and 3Halo skies. Now, shifting your 1Halo sky predictions farther from ground truth (and leaving everything else untouched) should not improve\r\n the "quality" of your solution. If this noticeably improves your score, then the score metric is inappropriately defined. In other words, I believe that intentional degradation of your solution (in terms of the distance from the ground truth) shouldn't lead\r\n to score improvements.
\r\nReal application of our techniques would be for a separate discussion, as in reality you don't know the number of halos.
", "votes": 1, "replies": [] }, { "id": 18050, "author_name": "Iain", "author_url": "", "post_date": "2012-12-17T18:08:53.893000", "content": "I don't buy the argument "Shifting the predictions (and thus making them less accurate) shouldn't improve the overall quality of your solution". That statement assumes that the Dark Worlds metric is wrong (prefers less accurate answers, rather than defining\r\n accuracy), so can't be used as an argument against the metric. The halo positions are uncertain:\r\nany point prediction throws away information, and different error measures will suggest different point estimates. The measure that you started with (maybe a mean-square-error cost function) may not have good properties for some purposes, and shifting\r\n the points from that could be appropriate.
\r\nHaving said that, I am also not convinced about the cost function: for example, the position of a halo with negligible mass can really matter. However, it's difficult to know what should be done without knowing how the results will be used downstream. In\r\n the real application it may be possible to do better by not summarizing all of the information in a sky into any single point estimate of its halos.
", "votes": 0, "replies": [] }, { "id": 18044, "author_name": "Anaconda", "author_url": "", "post_date": "2012-12-17T17:23:16.083000", "content": "I believe that final shifting of halos helped us to reduce the second error term, yes. At the same time, we were increasing the very same angular component by putting other halos to the center. So overall, we may just have been "fixing the mess" we had created\r\n :-)
\r\nI am still personally not convinced about the usefullness of the angular component, though. Shifting the predictions (and thus making them less accurate) shouldn't improve the overall quality of your solution.
", "votes": 0, "replies": [] }, { "id": 18037, "author_name": "Iain", "author_url": "", "post_date": "2012-12-17T16:51:22.203000", "content": "Interesting, thank you. I'm guessing that a big part of your success was explicitly tuning for the Dark World's metric (in combination with all of your clever ensembling/wrapping). A lot of the methods that other teams have described didn't seem to consider\r\n the weird way in which the predictions would be judged.
", "votes": 0, "replies": [] } ], "comment_count": 4, "messages": [ { "id": 18028, "postDate": "2012-12-17T16:12:38.493Z", "content": "Interesting competition. Thank you!
\r\nLet me briefly describe the approach / findings of the team Supernova. I believe it is quite different from what I have read here so far.
\r\nWe had the public score 0.72590 and the consistent private score of 0.72645.
\r\nOmitting the first few weeks of the competition when we tried approaches similar to those already mentioned, we then (finally) managed to compile and run lenstool and directed our efforts in this direction. Our current approach could be named "lenstool wrapper"\r\n or "lenstool ensemble." It can be seen as a random forest with (randomized) lenstool predictions as base learners. Steps of our algorithm for a fixed sky:
\r\nGood point, it would make more sense to "weight" the angular error term based on the halo mass.
\r\nLet me clarify my argument: Let's say you have accurate estimates of 1Halo skies, and a fixed solution of 2Halo and 3Halo skies. Now, shifting your 1Halo sky predictions farther from ground truth (and leaving everything else untouched) should not improve\r\n the "quality" of your solution. If this noticeably improves your score, then the score metric is inappropriately defined. In other words, I believe that intentional degradation of your solution (in terms of the distance from the ground truth) shouldn't lead\r\n to score improvements.
\r\nReal application of our techniques would be for a separate discussion, as in reality you don't know the number of halos.
", "votes": 1 }, { "id": 18050, "postDate": "2012-12-17T18:08:53.893Z", "content": "I don't buy the argument "Shifting the predictions (and thus making them less accurate) shouldn't improve the overall quality of your solution". That statement assumes that the Dark Worlds metric is wrong (prefers less accurate answers, rather than defining\r\n accuracy), so can't be used as an argument against the metric. The halo positions are uncertain:\r\nany point prediction throws away information, and different error measures will suggest different point estimates. The measure that you started with (maybe a mean-square-error cost function) may not have good properties for some purposes, and shifting\r\n the points from that could be appropriate.
\r\nHaving said that, I am also not convinced about the cost function: for example, the position of a halo with negligible mass can really matter. However, it's difficult to know what should be done without knowing how the results will be used downstream. In\r\n the real application it may be possible to do better by not summarizing all of the information in a sky into any single point estimate of its halos.
" }, { "id": 18044, "postDate": "2012-12-17T17:23:16.083Z", "content": "I believe that final shifting of halos helped us to reduce the second error term, yes. At the same time, we were increasing the very same angular component by putting other halos to the center. So overall, we may just have been "fixing the mess" we had created\r\n :-)
\r\nI am still personally not convinced about the usefullness of the angular component, though. Shifting the predictions (and thus making them less accurate) shouldn't improve the overall quality of your solution.
" }, { "id": 18037, "postDate": "2012-12-17T16:51:22.203Z", "content": "Interesting, thank you. I'm guessing that a big part of your success was explicitly tuning for the Dark World's metric (in combination with all of your clever ensembling/wrapping). A lot of the methods that other teams have described didn't seem to consider\r\n the weird way in which the predictions would be judged.
" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T18:58:02Z", "competition_id": 3084, "competition_slug": "DarkWorlds", "team_id": 25583, "solution_url": "/competitions/DarkWorlds/writeups/supernova-approach-of-team-supernova-lenstool-wrap" }