![\"Affect](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/3084/media/heic0814f.jpg\" "\\"Affect")
The $20,000 USD prize will be split between the top three finishers for the Observing Dark Worlds challenge as follows:
\r\nTo be eligible for prize money, you must reveal your model code and documentation under\r\nGPLv3 and agree to your First Name, First Initial of Last (Family) Name, City and Country being listed on the website.
\r\nBut wait, there's more ...
\r\nDo you want to put your hard-earned scientific know-how to work using real-world data sets in a first-class research environment? Itching to test your skills against a new challenge?
\r\nWinton Capital Management are looking for motivated, creative scientists to join their cutting edge research teams in Oxford, London, Zurich and Hong Kong. Winton is a world leader in applying advanced scientific methods to today's finance problems in a\r\n collaborative -- dare we say even academic -- environment that rewards innovation and robust research discoveries.
\r\nResearch activities at Winton include analyzing historical data for trends and correlations, applying proprietary pattern recognition and signal processing techniques to financial data, and of course, developing novel tests and tools to unlock the hidden\r\n worlds of markets and their prediction.
\r\nWinton will review the top entries and offer interviews to the creators of those submissions which are exceptional. Send your resumes to\r\nWintonRecruiting@kaggle.com for consideration.
\r\nPlease\r\n note: You must compete as an individual in recruiting competitions. You may only use the data provided to make your predictions. Winton will review the resumes and code of the top participants before deciding whether to offer an interview.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "rules", "content": "\r\n\t\r\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\r\n\r\n Sharing code or data outside of teams is not permitted.\r\n This includes making code or data made available to all participants, such as on the forums.\r\n
\r\n\r\n Team mergers are allowed and can be performed by the team leader. In order to merge, the combined team must have a total submission count less than or equal to the maximum allowed as of the merge date. The maximum allowed is the number of submissions per day multiplied by the number of days the competition has been running.
\r\n
The maximum size of a team is 8 participants.
\r\nYou are allowed to make your team anonymous, which hides participant details.
\r\nYou may submit a maximum of 2 entries per day.
\r\nYou may select up to 5 final submissions for judging.
\r\nStart Date: 10/12/2012 4:00 PM UTC
\r\nMerger Deadline: None
\r\nEntry Deadline: None
\r\nEnd Date: 12/16/2012 11:59 PM UTC
\r\n\t\r\nThis competition is both a Recruiting competition and a \r\nResearch competition.
\r\nAs a recruiting competition, participants are allowed to compete anonymously, and are encouraged to compete as individuals if interested in employment at Winton. (Please send your resumes to WintonRecruiting@kaggle.com for\r\n consideration.) Other participants are welcome to compete in teams.
\r\nAs a research competition, winning participants are required to provide model documentation and code under\r\nGPLv3 in a form such that these methods can be developed to work on real data. All participants are encouraged to provide code and documentation of their methods. Winners will be announced on\r\n the Winners page displaying at least their First Name, First Initial of Last (Family) Name, City, and Country (as required by US Sweepstakes law).
\r\nSharing of code and exchanging of ideas is allowed on the forums.
\r\nThe data sets used in this competition may be used for other purposes, including class projects and other research endeavors, given proper attribution: David Harvey and Thomas Kitching, Observing Dark Worlds Competition, https://www.kaggle.com/c/DarkWorlds. The\r\n challenge organisers should be notified (via the Competition Forum or Kaggle Support) if any article is written using results or data from the Observing Dark Worlds challenge.
\r\nAn article intended for publication in a scientific journal will be written summarising the results from this competition. By submitting an entry you agree that a discussion of the method may appear in this article.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "data-description", "content": "Training Data
\r\nThe training data consists of 300 simulated skies similar to the final panel in Figure 2 of the\r\ndescription page. Each sky contains between 300 and 740 galaxies. Each galaxy will have an x and y position ranging from 0 to 4200 (units are pixels), and a measure of ellipticity: e1 and e2 (see\r\n\r\nAn Introduction to Ellipticity).
\r\nTraining galaxy data is provided in a series of 300 files, one file for each Sky (e.g, Training_Sky27.csv or Training_Sky123.csv). These files have 4 columns:
\r\nTest Data
\r\nThe test data is in a similar format to the training data. There are 120 simulated skies (see final panel in Figure 2 of the\r\n\r\ndescription). Each sky contains 300 to 740 galaxies. Each galaxy will have an x and y position ranging from 0 to 4200, e1 and e2 values (totalling 4 columns per galaxy in the sky).
\r\nIn each sky there are either 1, 2 or 3 dark matter halos. The halo counts in each sky are provided in the file\r\nTest_haloCounts.csv.
\r\nThe challenge is to predict the center of each dark matter halo in each test sky based on the galaxy information provided.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Description", "content": "There is more to the Universe than meets the eye. Out in the cosmos exists a form of matter that outnumbers the stuff we can see by almost 7 to 1, and we don't know what it is. What we do know is that it does not emit\r\n or absorb light, so we call it Dark Matter.
\r\nSuch a vast amount of aggregated matter does not go unnoticed. In fact we observe that this stuff aggregates and forms massive structures called Dark Matter Halos.
\r\nAlthough dark, it warps and bends spacetime such that any light from a background galaxy which passes close to the\r\nDark Matter will have its path altered and changed. This bending causes the galaxy to appear as an ellipse in the sky.
\r\n\r\n
Figure 1: Dark Matter bending the light from a background galaxy. In extreme cases the galaxy here is seen as the two arcs surrounding it. (Credit: NASA, ESA, and Johan Richard (Caltech, USA))
\r\nSince there are many galaxies behind a \r\nDark Matter halo, their shapes will correlate with its position.
\r\n![\"Distortion](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/3084/media/reorderdarkmatter.png\" "\\"Distortion")
Figure 2: The effect of Dark Matter on the sky
\r\n\r\n
What’s The Problem?
\r\nDetecting these Dark Matter halos is hard, but possible using this data. If we can accurately estimate the positions of these halos, we can then understand the function they play in the Universe. There are various methods to attack the problem (we\r\n have given you some examples), however we have not been able to reach the level of precision required to understand exactly where this Dark Matter is for all\r\nDark Matter halos.
\r\nWe challenge YOU to detect the most elusive, mysterious and yet most abundant matter in all existence.
\r\n
\r\n![\"Dark](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/3084/media/a1689_opt.png\" "\\"Dark")
Figure 3: Dark Matter in Action. If you look closely at this real world example, you can see the warped and elliptical galaxies. (Credit NASA; ESA; L. Bradley (Johns Hopkins University); R. Bouwens (University of California, Santa Cruz); H.\r\n Ford (Johns Hopkins University); and G. Illingworth (University of California, Santa Cruz)
\r\nChallenge Organisers: David Harvey (Astrophysics PhD Student, Institute for Astronomy, University of Edinburgh), Dr. Tom Kitching (Royal Society Post Doctorial Fellow, Institute for Astronomy, University of Edinburgh)
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Evaluation", "content": "The challenge is to predict the centers (x and y coordinates) of the dark matter halos in the test set of 120 skies.
\r\nWe have provided DarkWorldsMetric.py that you can use to calculate the metric of your estimates to the training data. All you need are the python packages numpy, math, csv, c, getopt, sys and argparse and to run it type
\r\npython DarkWorldsMetric.py PATH_TO_PREDICTIONS PATH_TO_Training_halos.csv
\r\nand it shold output the metric. Here is how it works...
\r\nIn order to test your algorithm for Accuracy and \r\nPrecision there will be two parts to the metric. The first part will be how close you get to the true position of the halo, and the second will be to make sure there is no preferred direction to your estimates (positional bias).
\r\n$$m=F/1000+G$$
\r\nWhere F will be the average radial distance from the user estimate to the true position of the halo. In the case of more than one halo we will match up your estimates to the true estimates in the configuration that gives the best average radial distance\r\n in that particular sky so you will get the best score possible.
\r\nG is calculated by firstly finding the angle of the predicted position with respect to the centre of the true halo position, where the angle is measured from the estimated position to the line connecting the true halo position and a given reference point\r\n as shown in the figure. Then each angle is converted into a vector and then the average vector is calculated. If the algorithm has no preferred direction this average vector should be 0. We calculate the vector using the following equation.
\r\n$$G=\\sqrt{\\left(\\frac{1}{N}\\sum_{i=1}^N\\cos(\\phi_i)\\right)^2+\\left(\\frac{1}{N}\\sum_{j=1}^N\\sin(\\phi_j)\\right)^2} $$
\r\n![\"How](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/3084/media/evalmeta.png\")
We will estimate the vector your distribution of phi over all the halos in all the simulations.
\r\nThe reference point is provided in the Training_halos.csv file for you to evaluate your predictions locally.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Submission Instructions", "content": "The challenge is the predict the center of each dark matter halo in the provided 120 test skies. Each test sky can have either 1, 2, or 3 dark matter halos. x- and y-coordinate values are in terms of pixels.
\r\nEach row of the submission file must have 7 columns in comma separated values (CSV) format:
\r\nWinton is a data science company operating at the forefront of finance. We build automated trading systems by analysing large volumes of historical time series data covering everything from currency\r\n exchange rates to weather patterns.
\r\nWe are a research-driven business home to over 100 scientists and engineers working collaboratively in a dynamic, productive research environment. Our research centres are located in the UK (London and Oxford), Switzerland (Zurich) and Hong Kong.
\r\nOur focus is on a robust, scientific approach to investment management and this has allowed our company to grow into one of the most successful alternative investment firms in the world. Despite coming from a wide range of scientific disciplines (Physics,\r\n Statistics, Computer Science, Engineering etc), our scientists all share the creative drive and technical know-how to seek patterns and signals in data that can be used to develop novel algorithms for predicting movements in financial markets.
\r\nClick here to get the data and get started!
\r\nBest of luck with the competition!
\r\nWinton
Ps To hear from some of the scientists who already work at Winton check out our video at\r\nwww.WintonLovesData.com.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "An Introduction To Ellipticity", "content": "What do e1 and e2 represent in the data?
\r\ne1 and e2 are just the two components of ellipiticity. e1 describes how elongated the galaxy is in the x and y direction, while e2 describes how elongated the galaxy is along a 45 degree angle.
\r\n ![\"Figure](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/3084/media/g1g2a.png\" "\\"Figure")
Figure 4: The two components of ellipticity. As shown above, a galaxy whose ellipticity is just positive e1 is horizontal, and a galaxy whose ellipticity is just negtive e1 is vertical. A positive e2 is stretched along the 45 degree angle and a negative\r\n e2 is stretched along the 135 degree angle.
\r\nIn the case that galaxies are circular, their major axis becomes aligned with the dark matter (see Figure 2 on the description page). This is because the force exerted by the dark matter halo on the galaxy\r\n is tangential.
\r\nThe ellipiticity of a galaxy at a position (x,y) tangential to a point (x',y') is
\r\n$$ e_{tangential} = -(e_1\\cos(2\\phi) +e_2\\sin(2\\phi)) $$
\r\nwhere the angle of the galaxy with respect to the dark matter centre is given by
\r\n$$ \\phi = \\arctan\\left(\\frac{y-y'}{x-x'}\\right) $$
\r\nMore can be found from the Mapping Dark Matter competition.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Getting Started With Code", "content": "Check out Dave Harvey's \r\nObserving Dark Worlds Tutorial on No Free Hunch.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Winners", "content": "A big congratulations to the competition's winners!
\r\n