{ "id": 3367, "title": "Solutions", "content": "

Hi, guys,

\r\n

Congratulations to the winners!

\r\n

This competition was very fun. I suggest to describe our algorithms in this thread.

\r\n

My algorithm is the following.

\r\n

1. I have created the train grid with number of bins 3600 for each sky from the training set. For each center of the bin I have calculated the set of features as weighted sum of tangential ellipticites, average of tangential ellipticities, standard deviation\r\n of tangential ellipticites, the same for cross component of ellipticities.

\r\n

2. I have defined the target variable according to the distance to the halos. If the distance from the center of the bin to one of the halo is less than 500, then target variable equals to 1, otherwise 0.

\r\n

3. I have trained simple gbm with 2000 trees for this training table. For prediction I have created another grid with 28224 bins for each sky. As a result I have predicted map of probabilities for every test sky. When I look at the visualization of these\r\n maps, I found out that in many cases this method gives high probability for the regions of halos. Of course, it gives very high accuracy for the first halo, but for the second and the third halos in many cases it highlight the regions on the skies.

\r\n

4. To find these regions automatically I ran the search by small squares over map and choose the square with the maximal average probability inside the square (the center of this square is the center of predicted halo). After I found the first halo, I have\r\n removed 500-radius neighbourhood around this predicted halo and repeat the same search for the 2nd and 3rd halos.

\r\n

As a result this method gave me average distance 740 and metric 0.82 over the trainins skies. The disadvantage of my algorithm is that it is very difficult to control the angle in the metric. My best result of private leaderboard is 0.77, but unfortunately\r\n I have not chosen it as one of the last final submissions. I suppose that the huge part of my result is the angle part of metric.

\r\n

The interesting thing is that my best submission was created by manual prediction. It means that my small square search over maps is not perfect and it is possible to improve it by applying more sophisticated algorithms to the probabilities maps.

\r\n

I am interested if anybody used the similar approach?

", "comments": [ { "id": 18194, "author_name": "Tim Salimans", "author_url": "", "post_date": "2012-12-19T14:50:37.540000", "content": "

A summary of my solution is up at http://timsalimans.com/observing-dark-worlds/

\r\n

Also in case any fellow dutchies read this: I will be speaking on this competition (as well as others) at the\r\n\r\nData Science NL meetup in Utrecht January 24: do sign up!

\r\n

 

", "votes": 10, "replies": [] }, { "id": 18228, "author_name": "Iain", "author_url": "", "post_date": "2012-12-19T21:21:18.687000", "content": "

@jostheim I see what you mean about maximum likelihood, but if that's the goal I think you'd be better with an optimizer than a posterior sampler. Although I agree that MCMC could be a useful part of a heuristic optimizer, such as simulated annealing, suitable\r\n for nasty multimodal problems.

\r\n

We may have to agree to disagree about appropriate use of priors. The best points to report aren't necessarily sensible explanations of the data themselves. For example, the reported points could be hedging between two plausible explanations, but in themselves\r\n be nonsense. Trying to fudge the prior to make these hedged predictions typical of the posterior seems like a difficult game. If making predictions that don't explain the data seems icky, I agree: I'm not a fan of point estimates! I'd rather propagate more\r\n information about the posterior distribution to the next stage of analysis, or graphically display what we do and don't know.

\r\n

(BTW when I said cost function, I meant the competition metric, which I think of as the cost function for this challenge.)

\r\n

I'm sorry my writeup skimmed on precise details. I will be providing code and a more technical description for the organizers, and will put it on the web too. My model gave every halo its own {x, y, mass, extra core noise, r_0}, which meant my 3 halo skies\r\n had 15 variables that I sampled over. Tim was write to clamp some of those if he worked out that he could. In a real modelling problem one would learn their distributions (build a hierarchical prior), and it seems Tim learned (rightly or wrong) some delta\r\n functions for r_0.

", "votes": 3, "replies": [] }, { "id": 18870, "author_name": "Anil Thomas", "author_url": "", "post_date": "2013-01-01T19:08:05.327000", "content": "

[quote=Anil Thomas;18225]

\r\n

Reposting this to the more appropriate thread:

\r\n

http://www.kaggle.com/c/DarkWorlds/forums/t/3261/possible-data-problem/17988#post17988

\r\n

This algorithm gets a score of 0.73142 for a total running time of 90 seconds. If there is sufficient interest, I can clean up the code and post it. I probably won't get to it until after the holidays, though.

\r\n

[/quote]

\r\n

Here's the code:

\r\n
/*\r\n  For a description of what this program does, check\r\n  the forum posts at http://www.kaggle.com/c/DarkWorlds\r\n \r\n  To use this program, copy it to Test_Skies/odw.cpp and\r\n  run the following command:\r\n \r\n  make "CPPFLAGS=-O3" odw && ./odw > results.csv\r\n*/\r\n\r\n#include <stdio.h>
#include <assert.h>
#include <math.h>\r\n\r\n// The model used has two parameters. Optimum values were\r\n// found using cross validation on the training data.\r\nconst double ALPHA  = 0.22;\r\nconst double BETA   = 125;\r\n\r\ndouble sqr(double val)\r\n{\r\n    return (val * val);\r\n}\r\n\r\nclass Point\r\n{\r\npublic:\r\n    Point() : x(0), y(0)\r\n    {\r\n    }\r\n\r\n    void print()\r\n    {\r\n        printf(",%.2f,%.2f", x, y);\r\n    }\r\n\r\n    double x, y;\r\n};\r\n\r\nclass Galaxy\r\n{\r\npublic:\r\n    void read(FILE* file)\r\n    {\r\n        int id;\r\n        int result = fscanf(file, "Galaxy%d,%lf,%lf,%lf,%lf\\n",\r\n                            &id, &x, &y, &e1, &e2);  \r\n        assert(result == 5);\r\n    }\r\n\r\n    double distance(Point& p)\r\n    {\r\n        return sqrt(sqr(p.x - x) + sqr(p.y - y)); \r\n    }\r\n\r\n\r\n    double  x, y, e1, e2;\r\n};\r\n\r\nclass Sky\r\n{\r\npublic:\r\n    Sky(int id)\r\n    {\r\n        char    buffer[1024];\r\n        sprintf(buffer, "Test_Sky%d.csv", id+1);\r\n        FILE*   file = fopen(buffer, "r");\r\n        assert(file != 0);\r\n        int result  = fscanf(file, "%s\\n", buffer);\r\n        assert(result == 1);\r\n\r\n        for (galaxyCount = 0; feof(file) == 0; galaxyCount++) {\r\n            assert(galaxyCount < sizeof(galaxies) / sizeof(galaxies[0]));\r\n            galaxies[galaxyCount].read(file);\r\n        }\r\n\r\n        haloCount = (id < 40) ? 1 : ((id < 80) ? 2 : 3); \r\n        fclose(file);\r\n    }\r\n\r\n    void findHalos(Point* halos)\r\n    {\r\n        for (unsigned int k = 0; k < haloCount; k++) {\r\n            if (k > 0) {\r\n                wipeHalo(halos[k-1]);\r\n            }\r\n\r\n            findNextHalo(halos[k]);\r\n        }\r\n    }\r\n\r\nprivate:\r\n    double getSignalStrength(Point& p)\r\n    {\r\n        double  sum = 0;\r\n        double  weightSum = 0;\r\n\r\n        for (unsigned int i = 0; i < galaxyCount; i++) {\r\n            Galaxy& g = galaxies[i];\r\n            double  r = g.distance(p);\r\n            if (r >= 4200) {\r\n                continue;\r\n            }\r\n\r\n            double  weight = 4200 - r;\r\n            double  phi = atan2(g.y - p.y, g.x - p.x);\r\n            sum -= weight * (g.e1 * cos(2 * phi) + g.e2 * sin(2 * phi)); \r\n            weightSum += weight;\r\n        }\r\n\r\n        // Return the weighted average of tangential\r\n        // ellipticity observed from the given point.\r\n        return (sum / weightSum);\r\n    }\r\n\r\n    void wipeHalo(Point& h)\r\n    {\r\n        // Estimate the mass of the halo to be a multiple\r\n        // of the signal strength seen at its center.\r\n        double  mass = getSignalStrength(h) * BETA;\r\n        for (unsigned int i = 0; i < galaxyCount; i++) {\r\n            Galaxy& g = galaxies[i];\r\n            double  phi = atan2(g.y - h.y, g.x - h.x);\r\n            double  r = g.distance(h);\r\n            // Model the gravitational pull according to the halo's estimated\r\n            // mass and the distance from the center of the halo. \r\n            double  force = mass * exp(-pow(r, ALPHA));\r\n            g.e1 += force * cos(2 * phi);\r\n            g.e2 += force * sin(2 * phi);\r\n        }\r\n    }\r\n\r\n    void findNextHalo(Point& h)\r\n    {\r\n        Point   ref;\r\n        int     sizes[] = {105, 21, 3, 1};\r\n        int     curr = 4200;\r\n\r\n        // Divide the sky into square tiles and check the signal strength\r\n        // at the tile's midpoint. Pick the tile with the strongest\r\n        // signal, divide it further and iterate.\r\n        for (unsigned int iter = 0; iter < 4; iter++) { \r\n            double  max = -1;\r\n            int     count = curr / sizes[iter];  \r\n            curr = sizes[iter];\r\n\r\n            for (int row = 0; row < count; row++) {\r\n                for (int col = 0; col < count; col++) {\r\n                    Point   p;\r\n                    p.x = ref.x + col * curr + curr / 2;\r\n                    p.y = ref.y + row * curr + curr / 2;\r\n                    double  sig = getSignalStrength(p);\r\n                    if (sig > max) { \r\n                        max = sig; \r\n                        h = p;\r\n                    }\r\n                }\r\n            }\r\n\r\n            ref.x = h.x - curr / 2;\r\n            ref.y = h.y - curr / 2;\r\n        }\r\n    }\r\n\r\n    Galaxy          galaxies[800];\r\n    unsigned int    galaxyCount;\r\n    unsigned int    haloCount;\r\n};\r\n\r\nint main()\r\n{\r\n    for (int j = 0; j < 120; j++) {\r\n        Point   halos[3];\r\n        Sky     sky(j);\r\n\r\n        sky.findHalos(halos);\r\n        printf("Sky%d", j+1);\r\n        for (int k = 0; k < 3; k++) {\r\n            halos[k].print();\r\n        }\r\n\r\n        printf("\\n");\r\n    }\r\n\r\n    return 0;\r\n}\r\n
\r\n

The algorithm to get rid of a halo is a bit different from what I actually used during the contest. The new version is simpler, but works better. The program must be run from the leaf directory where the Test_Sky*.csv files are. Redirect the output to a\r\n file in order to generate a submission that can be sent to Kaggle.

", "votes": 4, "replies": [] }, { "id": 18200, "author_name": "Iain", "author_url": "", "post_date": "2012-12-19T15:38:29.120000", "content": "

Hot on the heals of Tim's writeup, here's mine, with an almost identical method: http://homepages.inf.ed.ac.uk/imurray2/pub/12kaggle_dark/

", "votes": 4, "replies": [] }, { "id": 19077, "author_name": "Quantum Leap", "author_url": "", "post_date": "2013-01-08T01:35:21.057000", "content": "

Like many others, I used an optimizer to find the best fit of a model using the Maximum Likelihood measure, combined with halo subtraction. This gave me scores around 0.8, both on the training data and private leaderboard data.

\r\n

To find each halo position I used a model with 2 parameters (center strength and radius) plus some contraints loosely tying the values of the two parameters.

\r\n

Once I found the position of the strongest remaining halo and before subtraction, I refined the shape of the halo with a model with four parameters (the strength vs. distance graph was a cubic polynomial when seen on a log-log plot) and without changing\r\n the coordinates anymore to avoid overfitting.

\r\n

This competion was lots of fun and the 2D geometry allowed for nice visualizations. Apart from the inconsistent (?) public leaderboard scores, one of the main problems I encountered was avoiding overfitting to the noise, I tried building synthetic skies\r\n where I defined the halo model and added similar gaussian noise to the ellipticities in the same measure as in the training data, and for models with multiple parameters it was common in multi-halo skies for the optimization to converge to wrong solutions\r\n that had a better MLE measure than the correct solution (that's why in the end I limited the model to 1-2 parameters per halo).

\r\n

EDIT: added some pictures and more details here: \r\nhttp://recursivegoose.blogspot.com/2013/02/views-from-dark-worlds.html

", "votes": 1, "replies": [] }, { "id": 18235, "author_name": "jostheim", "author_url": "", "post_date": "2012-12-19T22:30:43.210000", "content": "

@Iain-

\r\n

There definitely are other ways to search for the maximum.  In my previous posts you'll see I talk about trying to do expectations across my MCMC samples but the results being worse so dropping them.  So really I used MCMC initially expecting to do expectations,\r\n and then found they weren't very good, but did find that my maximums were actually quite good (which makes me think I had a bug).  I like MCMC as a way of searching and sampling b/c it does a nice job of sampling the space and getting out of local minima.\r\n  Still I probably would have gone with a genetic algorithm if I hadn't thought I was going to use the samples.  So I am in general agreement that MCMC is probably best for sampling the posterior, but I would say don't discount it as an easy to code (metropolis\r\n hastings unoptimized is like 20 lines of Python for me) way of searching for a maximum either.

\r\n

As for the prior's, I think you are probably right on that.  I was just thinking about a methodology where one could jointly search for the best model while minimizing the competition metric.  After thinking it over I am guessing that if you have a good\r\n functional form for your model that such a joint thing should be the same as applying the cost function to the samples.  Anyway it was really just a half-formed thought.

", "votes": 1, "replies": [] }, { "id": 19432, "author_name": "vgoklani", "author_url": "", "post_date": "2013-01-18T03:46:16.633000", "content": "

Hi Iain,

\r\n

I worked through your solution in detail and rewrote it as an ipython notebook: just click on the link below to open it.\r\n

\r\n

http://bit.ly/103OS2k

\r\n

I followed the same procedure as you, which was to use MCMC to optimize the log posterior (although I used Metropolis instead of slice sampling). I also constrained the prior by making careful observations of the training data.

\r\n

I didn't quite follow the logic in your (hacked) optimizer, and I was hoping either you or Tim could give a more thorough explanation of how you optimized your samples from the Markov Chain.\r\n

\r\n

I found that the predictions from the 2-halo samples (without optimization) were very noisy, and my results were just plain terrible. Were your results also this bad (you can see the graph in my notebook), or perhaps there is something wrong with my implementation.

\r\n

Best,

\r\n

Vishal

", "votes": 2, "replies": [] }, { "id": 19538, "author_name": "Iain", "author_url": "", "post_date": "2013-01-20T18:04:18.917000", "content": "

@Vishal Hi!

\r\n

I'm afraid I don't have time to work through what you're doing: my semester has just started and I have a lot on. I've already attempted to explain what I did in a couple of text files in the source tar ball (in addition to the kaggle blog post). I'm sorry\r\n if they're not sufficiently clear, but I'm afraid you'd have to be more specific in your questions.

\r\n

There are certainly halos with their posterior distribution spanning most of the sky.

", "votes": 0, "replies": [] }, { "id": 19325, "author_name": "Tim Salimans", "author_url": "", "post_date": "2013-01-15T08:52:02.857000", "content": "

The likelihood is very similar to the likelihood of a standard linear regression model, so I suggest you have a look at maximum likelihood / Bayesian approaches to linear regression first. My naming of variables ('x','y') is consistent with e.g. wikipedia's\r\n treatment of linear regression.

", "votes": 0, "replies": [] }, { "id": 19323, "author_name": "absolutelyNoWarranty", "author_url": "", "post_date": "2013-01-15T08:38:20.143000", "content": "

Hey everyone I'm trying hard to understand your solutions.
\r\nCan somebody help me to understand how the likelihood is calculated? In Tim's code, likelihoodOfPositions.m there is

\r\n
\r\n

 % target variable
\r\n    y_res = y - small_mass*sum(x(:,1:n_halo ~= i),2);
\r\n    x = x(:,i);
\r\n
\r\n    % sufficient statistics for the marginal likelihood
\r\n    yy = y_res'*y_res;
\r\n    xx = x'*x;
\r\n    xy = x'*y_res;
\r\n    mlm = xy/xx;
\r\n    
\r\n    % calculate log marginal likelihood
\r\n    logscale = -0.5*prec*(xx*mlm^2-2*mlm*xy);
\r\n    log_probs(i) = -0.5*prec*yy + logscale + log(myQuad(-0.5*prec*xx,prec*xy,logscale,lb_log_big_mass,ub_log_big_mass));

\r\n
\r\n

 

\r\n

(And Iain's code has a superficially similar section?)

\r\n

Thanks.

", "votes": 0, "replies": [] }, { "id": 18923, "author_name": "Anil Thomas", "author_url": "", "post_date": "2013-01-03T04:20:53.537000", "content": "

[quote=Galileo;18911]

\r\n

Hi Anil,

\r\n

Would you mind posting your code for calculating Alpha and Beta via cross-validation. I'm curious how you did this in C.

\r\n

Thanks!

\r\n

[/quote]

\r\n

I tuned the parameters manually. No doubt they can be optimized further. At the time, it didn't look like the test set and the training set had similar properties, so I wasn't motivated to spend a lot of time tuning. Another potential improvement would be\r\n to get rid of the speed optimization inside findNextHalo() and compute the signal strength for all 4200x4200 pixels. It will run much slower, but the results may be better.

", "votes": 0, "replies": [] }, { "id": 18911, "author_name": "Galileo", "author_url": "", "post_date": "2013-01-02T21:03:54.883000", "content": "

Hi Anil,

\r\n

Would you mind posting your code for calculating Alpha and Beta via cross-validation. I'm curious how you did this in C.

\r\n

Thanks!

", "votes": 0, "replies": [] }, { "id": 18408, "author_name": "deadbeatdawg", "author_url": "", "post_date": "2012-12-22T04:19:07.533000", "content": "

@ Anil

\r\n

Yes, there's interest. I'm curious to see the alternative to MCMC.

", "votes": 0, "replies": [] }, { "id": 18250, "author_name": "Tim Salimans", "author_url": "", "post_date": "2012-12-20T09:04:01.123000", "content": "

OK, that makes sense. I also tried a specification where the prior had some probability at zero mass and found that it helped mixing a lot, although the training score deteriorated slightly. After my initial poor public scores I actually went through the\r\n trouble of coding up an annealed importance sampler, since mixing seemed to be more of a problem on the test data compared to the training data, but in the end this did not improve prediction accuracy as measured by the Dark Worlds metric.

", "votes": 0, "replies": [] }, { "id": 18237, "author_name": "Iain", "author_url": "", "post_date": "2012-12-19T22:35:16.497000", "content": "

@Tim yes you understand correctly. It's quite possible I just wasn't careful enough. I just quickly checked that my performance wasn't hurt compared to initializing at the truth, which isn't the same as proper mixing. And I did try lots of random samples\r\n to initialize.

\r\n

However, I did see some label switching in my samples, and quite a lot of moving around. The extra flexibility in my model might have helped with that: my r_0 and core noise parameter moved around a lot, and I allowed my mass to go down to zero. So, for\r\n some settings, some of the halos wouldn't care where they were and would move very freely. (Statistically some of this flexibility might have been a bad idea.)

", "votes": 0, "replies": [] }, { "id": 18230, "author_name": "Tim Salimans", "author_url": "", "post_date": "2012-12-19T22:06:32.807000", "content": "

Do I understand correctly that you used univariate slice sampling to update each of these 15 variables in turn? If so, I'm surprised you got it to converge so well. I found that I really needed to move the x and y coordinate of each halo jointly, and I also\r\n integrated out the mass when calculating the M-H acceptance probability.

", "votes": 0, "replies": [] }, { "id": 18225, "author_name": "Anil Thomas", "author_url": "", "post_date": "2012-12-19T19:56:50.743000", "content": "

Reposting this to the more appropriate thread:

\r\n

http://www.kaggle.com/c/DarkWorlds/forums/t/3261/possible-data-problem/17988#post17988

\r\n

This algorithm gets a score of 0.73142 for a total running time of 90 seconds. If there is sufficient interest, I can clean up the code and post it. I probably won't get to it until after the holidays, though.

", "votes": 0, "replies": [] }, { "id": 18217, "author_name": "TomC", "author_url": "", "post_date": "2012-12-19T18:36:49.730000", "content": "

Thanks everyone for posting their methods! I'm brand new to machine learning / data science and I did not do very well in this competition, so I'm excited to try replicating your methods to learn some new skills. Tim and Lain, I'm eagerly awaiting your code\r\n :) Thanks again.

", "votes": 0, "replies": [] }, { "id": 18215, "author_name": "jostheim", "author_url": "", "post_date": "2012-12-19T17:59:47.387000", "content": "

@Iain

\r\n

I used MCMC in my thesis (wow 10 years ago) to map the posterior space for a model I was fitting.  If you map the space well enough then you will find the max posterior probability, and if you are using an uninformed prior I believe that will be the maximum\r\n likelihood (it has been a long time since I thought about this stuff though).  For reference: http://en.wikipedia.org/wiki/Maximum_likelihood, and "A maximum likelihood estimator coincides with the most\r\n probable Bayesian estimator given a uniform prior\r\n distribution on the parameters."   So I believe I was correct in my original post but didn't give all the relevant information, I found the maximum posterior point, but\r\n since my prior was uniform it was also the maximum likelihood.

\r\n

That particular tweet must talk about a particular problem with a particular paper that I have no knowledge of and won't speak to.  I will say that when I was doing astronomy 10 years ago, no one I knew used Bayesian techinques or MCMC, so even trying to\r\n use MCMC seems like a step up.

\r\n

In terms of a cost function (if you look back I actually only was talking about the competition metric, not the cost function), what I was attempting to say is that if my prior knowledge tells me that certain models should be selected over others, in this\r\n case that models which minimize the competition metric should be favored, and that if I have the ability to compute this as a prior then I could evaluate it as part of the P(M) terms in Bayes Rule.  Such a prior would say "My prior knowledge tells me to favor\r\n models which will win the competition", which to me seems like a good thing if I want to win the competition.   I understand how you used the cost function and about delaying the decision, I was simply considering another potential methodology.  What is and\r\n isn't a prior seems pretty subjective here, I am saying that the competition metric was given and therefore is prior knowledge, but I see your point.

\r\n

I ran my MCMC for a long time, but I didn't lock down my parameters the way you guys did, I let them vary much more widely, which gave me 12 parameters for the 3 halo model, which is a rather large space and required quite a while to settle in.  Obviously\r\n I could have done better by locking down the parameters more based on you and Tim's results, but ce'st la vie.

\r\n

 

", "votes": 0, "replies": [] }, { "id": 18211, "author_name": "Iain", "author_url": "", "post_date": "2012-12-19T17:19:44.527000", "content": "

@jostheim optimizing the expected competition metric, rather than using some other point estimate based on samples, made a meaningful difference to score in my Training set simulations, I seem to recall 0.03 to 0.04ish.

\r\n

I don't think it was actually necessary to run the Markov chains for long, or to use clever moves. My first submission used 1000 sweeps of slice sampling per sky (after picking an initialization from 4000 random draws), and would also have got me 2nd place\r\n (as would 6/7 of my submissions: maybe Tim did something systematically better, possibly in his more careful choice of priors, or possibly his optimization which I did in a really dumb way).

\r\n

MCMC isn't meant for finding maximum likelihood solutions. Astrophysicists on twitter say so:\r\nhttps://twitter.com/dfm/status/280850550436818944

\r\n

A cost function isn't used to set a prior, it's something you use to make decisions after making inference. Priors are set based on your beliefs about a system, not the decisions you're going to make later.

", "votes": 0, "replies": [] }, { "id": 18206, "author_name": "jostheim", "author_url": "", "post_date": "2012-12-19T16:39:10.597000", "content": "

@Tim and @Iain-

\r\n

I used a very similar methodology, it seems like in the end it probably came down to how long we ran our MCMC, what f(r) was used, what sort of priors we used and how we setup the random walks. Oh, and luck!

\r\n

One thing I didn't think of was using the actual metric of the competition as a post facto filter on the MCMC samples (if that is what you guys did, I am not 100% sure).  I just stuck with my maximum likelihood estimates that came out of my MCMC runs.  I\r\n now wonder if I couldn't have pulled in the competition metric as a prior...

\r\n

Thanks for the writeups!

", "votes": 0, "replies": [] }, { "id": 18203, "author_name": "Dmitry Efimov", "author_url": "", "post_date": "2012-12-19T16:18:12.657000", "content": "

Hi, Tim and Iain,

\r\n

Thank you very much for your writeups! I have learned a lot from this competition. And I am really impressed how bayesian methods can work!

\r\n

Special thank to lain for the link to the tutorial on MCMC!

\r\n

Dmitry.

", "votes": 0, "replies": [] }, { "id": 18197, "author_name": "Frans Slothouber", "author_url": "", "post_date": "2012-12-19T15:29:00.013000", "content": "

Thanks for the write-up. And I'll be there!

", "votes": 0, "replies": [] }, { "id": 18036, "author_name": "Edge2", "author_url": "", "post_date": "2012-12-17T16:50:41.133000", "content": "

[quote=Jose H. Solorzano;18018]

\r\n\r\n

[/quote]

\r\n

Maybe the organizers will comment, but a working assumption was that the cleansed data they posted after the competition start removed all galaxies with non-physical descriptions, and these only existed very near a strong halo. This is as opposed to, say,\r\n bracketing the galaxy descriptions, or some other approach to guaranteeing physical descriptions. I think this explains the observed density profiles in light of what is publically known.

", "votes": 0, "replies": [] }, { "id": 18022, "author_name": "Bruce Cragin", "author_url": "", "post_date": "2012-12-17T15:17:34.510000", "content": "

Jose, I agree with your comment about the image shifting. Although we did not use it in any of our predictions (as it had been stated that the idea of the competition was to see what could be done with shear alone) we did look at KDE estimates of the galaxy\r\n density early on, and the (anti)correlation between that and tangential ellipticity seemed clear.

", "votes": 0, "replies": [] }, { "id": 18018, "author_name": "José H. Solórzano", "author_url": "", "post_date": "2012-12-17T14:51:53.443000", "content": "

Thanks to the organizers for one of the most interesting Kaggle competitions I've seen.

\r\n

My best private score was 0.80 (out of 4 submissions, 1.2 public score). I used genetic programming for search, sampling x, y, mass, and other parameters over an assumed prior distribution, and then refining more granularly if it was the first halo.\r\n

\r\n

After removing the first halo, it was pretty easy for the other 2 predictions to be biased toward the center of mass. This hurts the angular component of the evaluation substantially. I never found a good way to deal with this, and maybe the "removal" approach\r\n is not the right one.

\r\n

Some findings:

\r\n", "votes": 0, "replies": [] }, { "id": 18006, "author_name": "Squark", "author_url": "", "post_date": "2012-12-17T10:32:41.120000", "content": "

My approach was to do a detailed analysis of the training set to look at salient features, by histogramming and profiling the training set and the truth information. The reference point allowed to solve the masses of halos 2 and 3 relative to the halo 1\r\n mass. These were almost always below 20%, which is pretty low, and peaking around minuscule 4-8%. Halo 3 was similar in mass to halo 2 so for most purposes these could be grouped together in the analysis. This suggested that one should first aim at finding\r\n halo 1 as accurately as possible and then remove it's effect from the sky to look for the other halos.

\r\n

Profiling the halo1 e_tan vs r revealed a very clear 1/r fall some distance from the halo center, but quite some variation within around 500 pixels from the center. This is what one would expect from physics: for spherical mass distributions one can ignore\r\n the internal details outside the halo and just replace it with a point source, while the contributions to spherical shells further out add up to zero. It wasn't quite clear what the shape inside the halo should be, except that one would expect it to fall to\r\n exactly 0 at r=0 (because with finite density and infitesimal volume one gets zero mass). I got a good fit with a simple m/( (r/R_0) + (R_0/r)), with chi2/NDF~1 for the average distribution over 300 skies. I did two profiles: one relative to true halo for\r\n log-likelihood calculation, another around the nearest candidate halo to account for smearing in center position when subtracting the halo1 contribution from the sky.

\r\n

Profiling halos2,3 revealed a much smaller e_tan vs r that was peaking closer to r=0. This I guess is natural, if one assumes a fairly universal dark matter density: halos of smaller mass should also be smaller. These were fitted both before and after subtracting\r\n the contribution of halo1, and I submitted both. I expected the latter to do better, but I got a much better score with the former, possibly because it lead to somewhat more defensive predictions with the matching background profile.

\r\n

For the background I estimated RMS of about 0.22 by histogramming the e_tan for all galaxies in all skies. This was a perfect Gaussian distribution, ideal for log-likelihood fitting. I also checked the background e_tan vs r relative to the (2100,2100), and\r\n before subtracting the halo1 this was going up at high r due to the contribution of halo1, so I plugged that fit also into the log-likelihood formula. After subtracting halo1 this became flat as should be, so to good approximation one could ignore the bias\r\n from halos 2,3. However, the submission with background profile from before subtracting halo1 got better score, presumably because it discounted the information at high r which can be more noisy.

\r\n

Checking the predictions against training truth showed that one can catch most halo1s with delta_r<100, and <delta_r>~177, while getting halos 2, 3 was a real pain and often a game of luck. Just trying to not do awfully for 2,3 would be enough, but I didnt'\r\n have time to check the efficiency of finding halos 2,3 as a function of m2,3 / m1 in order to optimize for the cases that could plausible be found. The angular component befuddled me so I didn't study it much.

\r\n

My final algo was to

\r\n

1) do a 26x26 log-likelihood grid search for halo1 with 'true' halo1 profile

\r\n

2) zoom into square with highest likelihood and do another 26x26 grid search within this bin

\r\n

3) estimate mass of halo1 using log-likelihood with 'smeared' halo1 profile, adding (m-m_avg) as an extra constraint to keep it close to average mass over 300 skies

\r\n

4) subtract the contribution of halo1 from the sky using the 'smeared' halo1 profile

\r\n

5) repeat above for halos 1, 2 using separate halo2,3 profile for log-likelihood

\r\n

I got my last calculation finished within 10 minutes from the competition end, so made it quite a race (I only entered 5 days ago after reading about Kaggle on New Scientist). My last version wasn't improving that much, having a training score of 0.8335\r\n with average distance of 714.4 pixels, so I figured I'd use my last spare slot and also submit my second to last attempt that had a better average distance of 677.9, but larger angular vector for score of 0.8462. After all, so many people commented on the\r\n forums on how much scatter the angular component adds. That turned out to have been the best tip of the competition, because (mostly by luck, I assume) this set scored substantially better on the public set, bringing me some 100 places up and well below Lenstool.\r\n Better yet, this same set scored a whopping 0.78504 on the private set. Made my day (or night, as it was) :)

\r\n

Now just really eagerly waiting to see the final scores. I saw my badge briefly flicker with top-10% marker, but waiting for the final lists to appear.

", "votes": 0, "replies": [] }, { "id": 18005, "author_name": "Ivan Guz", "author_url": "", "post_date": "2012-12-17T10:11:46.273000", "content": "

This competition was fun! Public and Private scores turned out not to be highly correlated. I calculated simple correlation coefficient between private and public scores and it was only 0.77 =) That means that it was luck who choose winners to this challenge\r\n =)

\r\n

My best submitted solution gave me 1.05344 - public 0.74608 - private
\r\nMy best submitted and selected solution gave me 0.77057 - public 0.78783 - private

\r\n

My idea was to fit halo models sequentially and then subtract effect of the previous halo when I was fitting next halo. No machine learning techniques were used, only brute force optimization of model parameters on net and estimation of the overall effect\r\n on the training skies.

", "votes": 0, "replies": [] }, { "id": 18004, "author_name": "Frans Slothouber", "author_url": "", "post_date": "2012-12-17T09:53:47.847000", "content": "

We (Team Paheli) tried a similar approach and a bunch of others.  We are working on a write-up about our experiences.

", "votes": 0, "replies": [] }, { "id": 18003, "author_name": "", "author_url": "", "post_date": "2012-12-17T09:48:49.240000", "content": "", "votes": 0, "replies": [] }, { "id": 18001, "author_name": "", "author_url": "", "post_date": "2012-12-17T09:32:55.560000", "content": "", "votes": 0, "replies": [] }, { "id": 17999, "author_name": "", "author_url": "", "post_date": "2012-12-17T09:18:11.317000", "content": "", "votes": 0, "replies": [] }, { "id": 17997, "author_name": "", "author_url": "", "post_date": "2012-12-17T09:10:54.020000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 33, "messages": [ { "id": 18194, "postDate": "2012-12-19T14:50:37.540Z", "content": "

A summary of my solution is up at http://timsalimans.com/observing-dark-worlds/

\r\n

Also in case any fellow dutchies read this: I will be speaking on this competition (as well as others) at the\r\n\r\nData Science NL meetup in Utrecht January 24: do sign up!

\r\n

 

", "votes": 10 }, { "id": 18228, "postDate": "2012-12-19T21:21:18.687Z", "content": "

@jostheim I see what you mean about maximum likelihood, but if that's the goal I think you'd be better with an optimizer than a posterior sampler. Although I agree that MCMC could be a useful part of a heuristic optimizer, such as simulated annealing, suitable\r\n for nasty multimodal problems.

\r\n

We may have to agree to disagree about appropriate use of priors. The best points to report aren't necessarily sensible explanations of the data themselves. For example, the reported points could be hedging between two plausible explanations, but in themselves\r\n be nonsense. Trying to fudge the prior to make these hedged predictions typical of the posterior seems like a difficult game. If making predictions that don't explain the data seems icky, I agree: I'm not a fan of point estimates! I'd rather propagate more\r\n information about the posterior distribution to the next stage of analysis, or graphically display what we do and don't know.

\r\n

(BTW when I said cost function, I meant the competition metric, which I think of as the cost function for this challenge.)

\r\n

I'm sorry my writeup skimmed on precise details. I will be providing code and a more technical description for the organizers, and will put it on the web too. My model gave every halo its own {x, y, mass, extra core noise, r_0}, which meant my 3 halo skies\r\n had 15 variables that I sampled over. Tim was write to clamp some of those if he worked out that he could. In a real modelling problem one would learn their distributions (build a hierarchical prior), and it seems Tim learned (rightly or wrong) some delta\r\n functions for r_0.

", "votes": 3 }, { "id": 18870, "postDate": "2013-01-01T19:08:05.327Z", "content": "

[quote=Anil Thomas;18225]

\r\n

Reposting this to the more appropriate thread:

\r\n

http://www.kaggle.com/c/DarkWorlds/forums/t/3261/possible-data-problem/17988#post17988

\r\n

This algorithm gets a score of 0.73142 for a total running time of 90 seconds. If there is sufficient interest, I can clean up the code and post it. I probably won't get to it until after the holidays, though.

\r\n

[/quote]

\r\n

Here's the code:

\r\n
/*\r\n  For a description of what this program does, check\r\n  the forum posts at http://www.kaggle.com/c/DarkWorlds\r\n \r\n  To use this program, copy it to Test_Skies/odw.cpp and\r\n  run the following command:\r\n \r\n  make "CPPFLAGS=-O3" odw && ./odw > results.csv\r\n*/\r\n\r\n#include <stdio.h>
#include <assert.h>
#include <math.h>\r\n\r\n// The model used has two parameters. Optimum values were\r\n// found using cross validation on the training data.\r\nconst double ALPHA  = 0.22;\r\nconst double BETA   = 125;\r\n\r\ndouble sqr(double val)\r\n{\r\n    return (val * val);\r\n}\r\n\r\nclass Point\r\n{\r\npublic:\r\n    Point() : x(0), y(0)\r\n    {\r\n    }\r\n\r\n    void print()\r\n    {\r\n        printf(",%.2f,%.2f", x, y);\r\n    }\r\n\r\n    double x, y;\r\n};\r\n\r\nclass Galaxy\r\n{\r\npublic:\r\n    void read(FILE* file)\r\n    {\r\n        int id;\r\n        int result = fscanf(file, "Galaxy%d,%lf,%lf,%lf,%lf\\n",\r\n                            &id, &x, &y, &e1, &e2);  \r\n        assert(result == 5);\r\n    }\r\n\r\n    double distance(Point& p)\r\n    {\r\n        return sqrt(sqr(p.x - x) + sqr(p.y - y)); \r\n    }\r\n\r\n\r\n    double  x, y, e1, e2;\r\n};\r\n\r\nclass Sky\r\n{\r\npublic:\r\n    Sky(int id)\r\n    {\r\n        char    buffer[1024];\r\n        sprintf(buffer, "Test_Sky%d.csv", id+1);\r\n        FILE*   file = fopen(buffer, "r");\r\n        assert(file != 0);\r\n        int result  = fscanf(file, "%s\\n", buffer);\r\n        assert(result == 1);\r\n\r\n        for (galaxyCount = 0; feof(file) == 0; galaxyCount++) {\r\n            assert(galaxyCount < sizeof(galaxies) / sizeof(galaxies[0]));\r\n            galaxies[galaxyCount].read(file);\r\n        }\r\n\r\n        haloCount = (id < 40) ? 1 : ((id < 80) ? 2 : 3); \r\n        fclose(file);\r\n    }\r\n\r\n    void findHalos(Point* halos)\r\n    {\r\n        for (unsigned int k = 0; k < haloCount; k++) {\r\n            if (k > 0) {\r\n                wipeHalo(halos[k-1]);\r\n            }\r\n\r\n            findNextHalo(halos[k]);\r\n        }\r\n    }\r\n\r\nprivate:\r\n    double getSignalStrength(Point& p)\r\n    {\r\n        double  sum = 0;\r\n        double  weightSum = 0;\r\n\r\n        for (unsigned int i = 0; i < galaxyCount; i++) {\r\n            Galaxy& g = galaxies[i];\r\n            double  r = g.distance(p);\r\n            if (r >= 4200) {\r\n                continue;\r\n            }\r\n\r\n            double  weight = 4200 - r;\r\n            double  phi = atan2(g.y - p.y, g.x - p.x);\r\n            sum -= weight * (g.e1 * cos(2 * phi) + g.e2 * sin(2 * phi)); \r\n            weightSum += weight;\r\n        }\r\n\r\n        // Return the weighted average of tangential\r\n        // ellipticity observed from the given point.\r\n        return (sum / weightSum);\r\n    }\r\n\r\n    void wipeHalo(Point& h)\r\n    {\r\n        // Estimate the mass of the halo to be a multiple\r\n        // of the signal strength seen at its center.\r\n        double  mass = getSignalStrength(h) * BETA;\r\n        for (unsigned int i = 0; i < galaxyCount; i++) {\r\n            Galaxy& g = galaxies[i];\r\n            double  phi = atan2(g.y - h.y, g.x - h.x);\r\n            double  r = g.distance(h);\r\n            // Model the gravitational pull according to the halo's estimated\r\n            // mass and the distance from the center of the halo. \r\n            double  force = mass * exp(-pow(r, ALPHA));\r\n            g.e1 += force * cos(2 * phi);\r\n            g.e2 += force * sin(2 * phi);\r\n        }\r\n    }\r\n\r\n    void findNextHalo(Point& h)\r\n    {\r\n        Point   ref;\r\n        int     sizes[] = {105, 21, 3, 1};\r\n        int     curr = 4200;\r\n\r\n        // Divide the sky into square tiles and check the signal strength\r\n        // at the tile's midpoint. Pick the tile with the strongest\r\n        // signal, divide it further and iterate.\r\n        for (unsigned int iter = 0; iter < 4; iter++) { \r\n            double  max = -1;\r\n            int     count = curr / sizes[iter];  \r\n            curr = sizes[iter];\r\n\r\n            for (int row = 0; row < count; row++) {\r\n                for (int col = 0; col < count; col++) {\r\n                    Point   p;\r\n                    p.x = ref.x + col * curr + curr / 2;\r\n                    p.y = ref.y + row * curr + curr / 2;\r\n                    double  sig = getSignalStrength(p);\r\n                    if (sig > max) { \r\n                        max = sig; \r\n                        h = p;\r\n                    }\r\n                }\r\n            }\r\n\r\n            ref.x = h.x - curr / 2;\r\n            ref.y = h.y - curr / 2;\r\n        }\r\n    }\r\n\r\n    Galaxy          galaxies[800];\r\n    unsigned int    galaxyCount;\r\n    unsigned int    haloCount;\r\n};\r\n\r\nint main()\r\n{\r\n    for (int j = 0; j < 120; j++) {\r\n        Point   halos[3];\r\n        Sky     sky(j);\r\n\r\n        sky.findHalos(halos);\r\n        printf("Sky%d", j+1);\r\n        for (int k = 0; k < 3; k++) {\r\n            halos[k].print();\r\n        }\r\n\r\n        printf("\\n");\r\n    }\r\n\r\n    return 0;\r\n}\r\n
\r\n

The algorithm to get rid of a halo is a bit different from what I actually used during the contest. The new version is simpler, but works better. The program must be run from the leaf directory where the Test_Sky*.csv files are. Redirect the output to a\r\n file in order to generate a submission that can be sent to Kaggle.

", "votes": 4 }, { "id": 18200, "postDate": "2012-12-19T15:38:29.120Z", "content": "

Hot on the heals of Tim's writeup, here's mine, with an almost identical method: http://homepages.inf.ed.ac.uk/imurray2/pub/12kaggle_dark/

", "votes": 4 }, { "id": 19077, "postDate": "2013-01-08T01:35:21.057Z", "content": "

Like many others, I used an optimizer to find the best fit of a model using the Maximum Likelihood measure, combined with halo subtraction. This gave me scores around 0.8, both on the training data and private leaderboard data.

\r\n

To find each halo position I used a model with 2 parameters (center strength and radius) plus some contraints loosely tying the values of the two parameters.

\r\n

Once I found the position of the strongest remaining halo and before subtraction, I refined the shape of the halo with a model with four parameters (the strength vs. distance graph was a cubic polynomial when seen on a log-log plot) and without changing\r\n the coordinates anymore to avoid overfitting.

\r\n

This competion was lots of fun and the 2D geometry allowed for nice visualizations. Apart from the inconsistent (?) public leaderboard scores, one of the main problems I encountered was avoiding overfitting to the noise, I tried building synthetic skies\r\n where I defined the halo model and added similar gaussian noise to the ellipticities in the same measure as in the training data, and for models with multiple parameters it was common in multi-halo skies for the optimization to converge to wrong solutions\r\n that had a better MLE measure than the correct solution (that's why in the end I limited the model to 1-2 parameters per halo).

\r\n

EDIT: added some pictures and more details here: \r\nhttp://recursivegoose.blogspot.com/2013/02/views-from-dark-worlds.html

", "votes": 1 }, { "id": 18235, "postDate": "2012-12-19T22:30:43.210Z", "content": "

@Iain-

\r\n

There definitely are other ways to search for the maximum.  In my previous posts you'll see I talk about trying to do expectations across my MCMC samples but the results being worse so dropping them.  So really I used MCMC initially expecting to do expectations,\r\n and then found they weren't very good, but did find that my maximums were actually quite good (which makes me think I had a bug).  I like MCMC as a way of searching and sampling b/c it does a nice job of sampling the space and getting out of local minima.\r\n  Still I probably would have gone with a genetic algorithm if I hadn't thought I was going to use the samples.  So I am in general agreement that MCMC is probably best for sampling the posterior, but I would say don't discount it as an easy to code (metropolis\r\n hastings unoptimized is like 20 lines of Python for me) way of searching for a maximum either.

\r\n

As for the prior's, I think you are probably right on that.  I was just thinking about a methodology where one could jointly search for the best model while minimizing the competition metric.  After thinking it over I am guessing that if you have a good\r\n functional form for your model that such a joint thing should be the same as applying the cost function to the samples.  Anyway it was really just a half-formed thought.

", "votes": 1 }, { "id": 19432, "postDate": "2013-01-18T03:46:16.633Z", "content": "

Hi Iain,

\r\n

I worked through your solution in detail and rewrote it as an ipython notebook: just click on the link below to open it.\r\n

\r\n

http://bit.ly/103OS2k

\r\n

I followed the same procedure as you, which was to use MCMC to optimize the log posterior (although I used Metropolis instead of slice sampling). I also constrained the prior by making careful observations of the training data.

\r\n

I didn't quite follow the logic in your (hacked) optimizer, and I was hoping either you or Tim could give a more thorough explanation of how you optimized your samples from the Markov Chain.\r\n

\r\n

I found that the predictions from the 2-halo samples (without optimization) were very noisy, and my results were just plain terrible. Were your results also this bad (you can see the graph in my notebook), or perhaps there is something wrong with my implementation.

\r\n

Best,

\r\n

Vishal

", "votes": 2 }, { "id": 17994, "postDate": "2012-12-17T08:41:28.277Z", "content": "

Hi, guys,

\r\n

Congratulations to the winners!

\r\n

This competition was very fun. I suggest to describe our algorithms in this thread.

\r\n

My algorithm is the following.

\r\n

1. I have created the train grid with number of bins 3600 for each sky from the training set. For each center of the bin I have calculated the set of features as weighted sum of tangential ellipticites, average of tangential ellipticities, standard deviation\r\n of tangential ellipticites, the same for cross component of ellipticities.

\r\n

2. I have defined the target variable according to the distance to the halos. If the distance from the center of the bin to one of the halo is less than 500, then target variable equals to 1, otherwise 0.

\r\n

3. I have trained simple gbm with 2000 trees for this training table. For prediction I have created another grid with 28224 bins for each sky. As a result I have predicted map of probabilities for every test sky. When I look at the visualization of these\r\n maps, I found out that in many cases this method gives high probability for the regions of halos. Of course, it gives very high accuracy for the first halo, but for the second and the third halos in many cases it highlight the regions on the skies.

\r\n

4. To find these regions automatically I ran the search by small squares over map and choose the square with the maximal average probability inside the square (the center of this square is the center of predicted halo). After I found the first halo, I have\r\n removed 500-radius neighbourhood around this predicted halo and repeat the same search for the 2nd and 3rd halos.

\r\n

As a result this method gave me average distance 740 and metric 0.82 over the trainins skies. The disadvantage of my algorithm is that it is very difficult to control the angle in the metric. My best result of private leaderboard is 0.77, but unfortunately\r\n I have not chosen it as one of the last final submissions. I suppose that the huge part of my result is the angle part of metric.

\r\n

The interesting thing is that my best submission was created by manual prediction. It means that my small square search over maps is not perfect and it is possible to improve it by applying more sophisticated algorithms to the probabilities maps.

\r\n

I am interested if anybody used the similar approach?

", "votes": 2 }, { "id": 19538, "postDate": "2013-01-20T18:04:18.917Z", "content": "

@Vishal Hi!

\r\n

I'm afraid I don't have time to work through what you're doing: my semester has just started and I have a lot on. I've already attempted to explain what I did in a couple of text files in the source tar ball (in addition to the kaggle blog post). I'm sorry\r\n if they're not sufficiently clear, but I'm afraid you'd have to be more specific in your questions.

\r\n

There are certainly halos with their posterior distribution spanning most of the sky.

" }, { "id": 19325, "postDate": "2013-01-15T08:52:02.857Z", "content": "

The likelihood is very similar to the likelihood of a standard linear regression model, so I suggest you have a look at maximum likelihood / Bayesian approaches to linear regression first. My naming of variables ('x','y') is consistent with e.g. wikipedia's\r\n treatment of linear regression.

" }, { "id": 19323, "postDate": "2013-01-15T08:38:20.143Z", "content": "

Hey everyone I'm trying hard to understand your solutions.
\r\nCan somebody help me to understand how the likelihood is calculated? In Tim's code, likelihoodOfPositions.m there is

\r\n
\r\n

 % target variable
\r\n    y_res = y - small_mass*sum(x(:,1:n_halo ~= i),2);
\r\n    x = x(:,i);
\r\n
\r\n    % sufficient statistics for the marginal likelihood
\r\n    yy = y_res'*y_res;
\r\n    xx = x'*x;
\r\n    xy = x'*y_res;
\r\n    mlm = xy/xx;
\r\n    
\r\n    % calculate log marginal likelihood
\r\n    logscale = -0.5*prec*(xx*mlm^2-2*mlm*xy);
\r\n    log_probs(i) = -0.5*prec*yy + logscale + log(myQuad(-0.5*prec*xx,prec*xy,logscale,lb_log_big_mass,ub_log_big_mass));

\r\n
\r\n

 

\r\n

(And Iain's code has a superficially similar section?)

\r\n

Thanks.

" }, { "id": 18923, "postDate": "2013-01-03T04:20:53.537Z", "content": "

[quote=Galileo;18911]

\r\n

Hi Anil,

\r\n

Would you mind posting your code for calculating Alpha and Beta via cross-validation. I'm curious how you did this in C.

\r\n

Thanks!

\r\n

[/quote]

\r\n

I tuned the parameters manually. No doubt they can be optimized further. At the time, it didn't look like the test set and the training set had similar properties, so I wasn't motivated to spend a lot of time tuning. Another potential improvement would be\r\n to get rid of the speed optimization inside findNextHalo() and compute the signal strength for all 4200x4200 pixels. It will run much slower, but the results may be better.

" }, { "id": 18911, "postDate": "2013-01-02T21:03:54.883Z", "content": "

Hi Anil,

\r\n

Would you mind posting your code for calculating Alpha and Beta via cross-validation. I'm curious how you did this in C.

\r\n

Thanks!

" }, { "id": 18408, "postDate": "2012-12-22T04:19:07.533Z", "content": "

@ Anil

\r\n

Yes, there's interest. I'm curious to see the alternative to MCMC.

" }, { "id": 18250, "postDate": "2012-12-20T09:04:01.123Z", "content": "

OK, that makes sense. I also tried a specification where the prior had some probability at zero mass and found that it helped mixing a lot, although the training score deteriorated slightly. After my initial poor public scores I actually went through the\r\n trouble of coding up an annealed importance sampler, since mixing seemed to be more of a problem on the test data compared to the training data, but in the end this did not improve prediction accuracy as measured by the Dark Worlds metric.

" }, { "id": 18237, "postDate": "2012-12-19T22:35:16.497Z", "content": "

@Tim yes you understand correctly. It's quite possible I just wasn't careful enough. I just quickly checked that my performance wasn't hurt compared to initializing at the truth, which isn't the same as proper mixing. And I did try lots of random samples\r\n to initialize.

\r\n

However, I did see some label switching in my samples, and quite a lot of moving around. The extra flexibility in my model might have helped with that: my r_0 and core noise parameter moved around a lot, and I allowed my mass to go down to zero. So, for\r\n some settings, some of the halos wouldn't care where they were and would move very freely. (Statistically some of this flexibility might have been a bad idea.)

" }, { "id": 18230, "postDate": "2012-12-19T22:06:32.807Z", "content": "

Do I understand correctly that you used univariate slice sampling to update each of these 15 variables in turn? If so, I'm surprised you got it to converge so well. I found that I really needed to move the x and y coordinate of each halo jointly, and I also\r\n integrated out the mass when calculating the M-H acceptance probability.

" }, { "id": 18225, "postDate": "2012-12-19T19:56:50.743Z", "content": "

Reposting this to the more appropriate thread:

\r\n

http://www.kaggle.com/c/DarkWorlds/forums/t/3261/possible-data-problem/17988#post17988

\r\n

This algorithm gets a score of 0.73142 for a total running time of 90 seconds. If there is sufficient interest, I can clean up the code and post it. I probably won't get to it until after the holidays, though.

" }, { "id": 18217, "postDate": "2012-12-19T18:36:49.730Z", "content": "

Thanks everyone for posting their methods! I'm brand new to machine learning / data science and I did not do very well in this competition, so I'm excited to try replicating your methods to learn some new skills. Tim and Lain, I'm eagerly awaiting your code\r\n :) Thanks again.

" }, { "id": 18215, "postDate": "2012-12-19T17:59:47.387Z", "content": "

@Iain

\r\n

I used MCMC in my thesis (wow 10 years ago) to map the posterior space for a model I was fitting.  If you map the space well enough then you will find the max posterior probability, and if you are using an uninformed prior I believe that will be the maximum\r\n likelihood (it has been a long time since I thought about this stuff though).  For reference: http://en.wikipedia.org/wiki/Maximum_likelihood, and "A maximum likelihood estimator coincides with the most\r\n probable Bayesian estimator given a uniform prior\r\n distribution on the parameters."   So I believe I was correct in my original post but didn't give all the relevant information, I found the maximum posterior point, but\r\n since my prior was uniform it was also the maximum likelihood.

\r\n

That particular tweet must talk about a particular problem with a particular paper that I have no knowledge of and won't speak to.  I will say that when I was doing astronomy 10 years ago, no one I knew used Bayesian techinques or MCMC, so even trying to\r\n use MCMC seems like a step up.

\r\n

In terms of a cost function (if you look back I actually only was talking about the competition metric, not the cost function), what I was attempting to say is that if my prior knowledge tells me that certain models should be selected over others, in this\r\n case that models which minimize the competition metric should be favored, and that if I have the ability to compute this as a prior then I could evaluate it as part of the P(M) terms in Bayes Rule.  Such a prior would say "My prior knowledge tells me to favor\r\n models which will win the competition", which to me seems like a good thing if I want to win the competition.   I understand how you used the cost function and about delaying the decision, I was simply considering another potential methodology.  What is and\r\n isn't a prior seems pretty subjective here, I am saying that the competition metric was given and therefore is prior knowledge, but I see your point.

\r\n

I ran my MCMC for a long time, but I didn't lock down my parameters the way you guys did, I let them vary much more widely, which gave me 12 parameters for the 3 halo model, which is a rather large space and required quite a while to settle in.  Obviously\r\n I could have done better by locking down the parameters more based on you and Tim's results, but ce'st la vie.

\r\n

 

" }, { "id": 18211, "postDate": "2012-12-19T17:19:44.527Z", "content": "

@jostheim optimizing the expected competition metric, rather than using some other point estimate based on samples, made a meaningful difference to score in my Training set simulations, I seem to recall 0.03 to 0.04ish.

\r\n

I don't think it was actually necessary to run the Markov chains for long, or to use clever moves. My first submission used 1000 sweeps of slice sampling per sky (after picking an initialization from 4000 random draws), and would also have got me 2nd place\r\n (as would 6/7 of my submissions: maybe Tim did something systematically better, possibly in his more careful choice of priors, or possibly his optimization which I did in a really dumb way).

\r\n

MCMC isn't meant for finding maximum likelihood solutions. Astrophysicists on twitter say so:\r\nhttps://twitter.com/dfm/status/280850550436818944

\r\n

A cost function isn't used to set a prior, it's something you use to make decisions after making inference. Priors are set based on your beliefs about a system, not the decisions you're going to make later.

" }, { "id": 18206, "postDate": "2012-12-19T16:39:10.597Z", "content": "

@Tim and @Iain-

\r\n

I used a very similar methodology, it seems like in the end it probably came down to how long we ran our MCMC, what f(r) was used, what sort of priors we used and how we setup the random walks. Oh, and luck!

\r\n

One thing I didn't think of was using the actual metric of the competition as a post facto filter on the MCMC samples (if that is what you guys did, I am not 100% sure).  I just stuck with my maximum likelihood estimates that came out of my MCMC runs.  I\r\n now wonder if I couldn't have pulled in the competition metric as a prior...

\r\n

Thanks for the writeups!

" }, { "id": 18203, "postDate": "2012-12-19T16:18:12.657Z", "content": "

Hi, Tim and Iain,

\r\n

Thank you very much for your writeups! I have learned a lot from this competition. And I am really impressed how bayesian methods can work!

\r\n

Special thank to lain for the link to the tutorial on MCMC!

\r\n

Dmitry.

" }, { "id": 18197, "postDate": "2012-12-19T15:29:00.013Z", "content": "

Thanks for the write-up. And I'll be there!

" }, { "id": 18036, "postDate": "2012-12-17T16:50:41.133Z", "content": "

[quote=Jose H. Solorzano;18018]

\r\n\r\n

[/quote]

\r\n

Maybe the organizers will comment, but a working assumption was that the cleansed data they posted after the competition start removed all galaxies with non-physical descriptions, and these only existed very near a strong halo. This is as opposed to, say,\r\n bracketing the galaxy descriptions, or some other approach to guaranteeing physical descriptions. I think this explains the observed density profiles in light of what is publically known.

" }, { "id": 18022, "postDate": "2012-12-17T15:17:34.510Z", "content": "

Jose, I agree with your comment about the image shifting. Although we did not use it in any of our predictions (as it had been stated that the idea of the competition was to see what could be done with shear alone) we did look at KDE estimates of the galaxy\r\n density early on, and the (anti)correlation between that and tangential ellipticity seemed clear.

" }, { "id": 18018, "postDate": "2012-12-17T14:51:53.443Z", "content": "

Thanks to the organizers for one of the most interesting Kaggle competitions I've seen.

\r\n

My best private score was 0.80 (out of 4 submissions, 1.2 public score). I used genetic programming for search, sampling x, y, mass, and other parameters over an assumed prior distribution, and then refining more granularly if it was the first halo.\r\n

\r\n

After removing the first halo, it was pretty easy for the other 2 predictions to be biased toward the center of mass. This hurts the angular component of the evaluation substantially. I never found a good way to deal with this, and maybe the "removal" approach\r\n is not the right one.

\r\n

Some findings:

\r\n" }, { "id": 18006, "postDate": "2012-12-17T10:32:41.120Z", "content": "

My approach was to do a detailed analysis of the training set to look at salient features, by histogramming and profiling the training set and the truth information. The reference point allowed to solve the masses of halos 2 and 3 relative to the halo 1\r\n mass. These were almost always below 20%, which is pretty low, and peaking around minuscule 4-8%. Halo 3 was similar in mass to halo 2 so for most purposes these could be grouped together in the analysis. This suggested that one should first aim at finding\r\n halo 1 as accurately as possible and then remove it's effect from the sky to look for the other halos.

\r\n

Profiling the halo1 e_tan vs r revealed a very clear 1/r fall some distance from the halo center, but quite some variation within around 500 pixels from the center. This is what one would expect from physics: for spherical mass distributions one can ignore\r\n the internal details outside the halo and just replace it with a point source, while the contributions to spherical shells further out add up to zero. It wasn't quite clear what the shape inside the halo should be, except that one would expect it to fall to\r\n exactly 0 at r=0 (because with finite density and infitesimal volume one gets zero mass). I got a good fit with a simple m/( (r/R_0) + (R_0/r)), with chi2/NDF~1 for the average distribution over 300 skies. I did two profiles: one relative to true halo for\r\n log-likelihood calculation, another around the nearest candidate halo to account for smearing in center position when subtracting the halo1 contribution from the sky.

\r\n

Profiling halos2,3 revealed a much smaller e_tan vs r that was peaking closer to r=0. This I guess is natural, if one assumes a fairly universal dark matter density: halos of smaller mass should also be smaller. These were fitted both before and after subtracting\r\n the contribution of halo1, and I submitted both. I expected the latter to do better, but I got a much better score with the former, possibly because it lead to somewhat more defensive predictions with the matching background profile.

\r\n

For the background I estimated RMS of about 0.22 by histogramming the e_tan for all galaxies in all skies. This was a perfect Gaussian distribution, ideal for log-likelihood fitting. I also checked the background e_tan vs r relative to the (2100,2100), and\r\n before subtracting the halo1 this was going up at high r due to the contribution of halo1, so I plugged that fit also into the log-likelihood formula. After subtracting halo1 this became flat as should be, so to good approximation one could ignore the bias\r\n from halos 2,3. However, the submission with background profile from before subtracting halo1 got better score, presumably because it discounted the information at high r which can be more noisy.

\r\n

Checking the predictions against training truth showed that one can catch most halo1s with delta_r<100, and <delta_r>~177, while getting halos 2, 3 was a real pain and often a game of luck. Just trying to not do awfully for 2,3 would be enough, but I didnt'\r\n have time to check the efficiency of finding halos 2,3 as a function of m2,3 / m1 in order to optimize for the cases that could plausible be found. The angular component befuddled me so I didn't study it much.

\r\n

My final algo was to

\r\n

1) do a 26x26 log-likelihood grid search for halo1 with 'true' halo1 profile

\r\n

2) zoom into square with highest likelihood and do another 26x26 grid search within this bin

\r\n

3) estimate mass of halo1 using log-likelihood with 'smeared' halo1 profile, adding (m-m_avg) as an extra constraint to keep it close to average mass over 300 skies

\r\n

4) subtract the contribution of halo1 from the sky using the 'smeared' halo1 profile

\r\n

5) repeat above for halos 1, 2 using separate halo2,3 profile for log-likelihood

\r\n

I got my last calculation finished within 10 minutes from the competition end, so made it quite a race (I only entered 5 days ago after reading about Kaggle on New Scientist). My last version wasn't improving that much, having a training score of 0.8335\r\n with average distance of 714.4 pixels, so I figured I'd use my last spare slot and also submit my second to last attempt that had a better average distance of 677.9, but larger angular vector for score of 0.8462. After all, so many people commented on the\r\n forums on how much scatter the angular component adds. That turned out to have been the best tip of the competition, because (mostly by luck, I assume) this set scored substantially better on the public set, bringing me some 100 places up and well below Lenstool.\r\n Better yet, this same set scored a whopping 0.78504 on the private set. Made my day (or night, as it was) :)

\r\n

Now just really eagerly waiting to see the final scores. I saw my badge briefly flicker with top-10% marker, but waiting for the final lists to appear.

" }, { "id": 18005, "postDate": "2012-12-17T10:11:46.273Z", "content": "

This competition was fun! Public and Private scores turned out not to be highly correlated. I calculated simple correlation coefficient between private and public scores and it was only 0.77 =) That means that it was luck who choose winners to this challenge\r\n =)

\r\n

My best submitted solution gave me 1.05344 - public 0.74608 - private
\r\nMy best submitted and selected solution gave me 0.77057 - public 0.78783 - private

\r\n

My idea was to fit halo models sequentially and then subtract effect of the previous halo when I was fitting next halo. No machine learning techniques were used, only brute force optimization of model parameters on net and estimation of the overall effect\r\n on the training skies.

" }, { "id": 18004, "postDate": "2012-12-17T09:53:47.847Z", "content": "

We (Team Paheli) tried a similar approach and a bunch of others.  We are working on a write-up about our experiences.

" }, { "id": 18003, "postDate": "2012-12-17T09:48:49.240Z", "content": "

I've tried a somewhat similar approach without much success - performance is poorer than the simple maximum likelihood grid search.

\r\n

My implementation, if I remember correctly, involves gridding each sky as 20x20 and using std dev e1, std dev e2, max likelihood and max signal as features, normalizing the features w.r.t. to each sky, shuffling them and training using a random forest regressor.\r\n My targets are created as (if halo in grid, target = 1, else 0) and adding that to (4200 / distance of grid center from halo) so grids affected by multiple halos would have much greater target values. I sorted all the grids by descending probability and used\r\n their centers as halo centers.

" }, { "id": 18001, "postDate": "2012-12-17T09:32:55.560Z", "content": "

I have generated many "hand-made" solutions.
\r\nThe best solution gave me 1.03510 (public), 0.81091 (private).

" }, { "id": 17999, "postDate": "2012-12-17T09:18:11.317Z", "content": "

hi dmitry,

\r\n

thanks for sharing the method. i used (a simple form) of compressive sampling. i have just realized that my second submission was the best one using this method. then, like most of the participants, the leaderboard misguided me :( i didn't even consider\r\n to choose it as the final submission.

" }, { "id": 17997, "postDate": "2012-12-17T09:10:54.020Z", "content": "

Thanks, Dmitry, the LeaderBoard in this particular instance was clearly misleading, and I shall be writing about my own experience in a few hours time. Right now I have to go skiing..

" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T18:58:11Z", "competition_id": 3084, "competition_slug": "DarkWorlds", "team_id": 26686, "solution_url": "/competitions/DarkWorlds/writeups/dmitry-efimov-solutions" }