In this contest, competitors will design systems to learn about two modalities of data: images and text. The provided training data is Louis von Ahn's Small ESP Game Dataset, containing images and word tags for these images. Competitors should train their system to associate images to sets of word tags. At test time, the system is presented with two possible sets of word tags for an image, and must determine which is the correct set of word tags.
\r\nBecause this task is very easy for humans to do, we will not provide the final test inputs until one week before the contest closes. Preliminary winners will need to release their winning code and demonstrate that they did not manually label the test set. We reserve the right to disqualify entries that may involve any manual labeling of the test set.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Evaluation", "content": "The test examples come in triplets: an image, and two annotations.The test images are located in public_test_images.tgz.
\r\nThe file public_test_options.tgz contains two files per test image: image_id.option_0.desc and image_id.option_1.desc. The models should predict which of the options fits the image better. If option 0 fits better, the prediction for image_id should be 0. If option 1 fits better, the prediction for image_id should be 1.
\r\nSubmissions are scored based on their accuracy at predicting which of the two descriptions fits the image best.
\r\npublic_test_images.tgz will be used to make the leaderboard while the contest is active. Because this task is very easy for humans to do, we will not provide the final test inputs until one week before the contest closes. After we provide the final test inputs, teams wishing to be scored for the final competition should run their classifier on the new data and upload a new .csv file before the competition closes. This procedure is intended to discourage manual labeling of test data.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "rules", "content": "\r\n\t\r\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\r\n\r\n Privately sharing code or data outside of teams is not permitted.\r\n It's okay to share code if made available to all participants on the forums.\r\n
\r\nKaggle and the competition host have the right to publicly disseminate any entries or models.
\r\n\r\n Winning solutions need to be made available under\r\n a popular OSI-approved license\r\n in order to be eligible for recognition and prize money.\r\n
\r\n\r\n Prizes will be awarded after the winners have posted their solutions to the competition forum.\r\n Winners must post or link to their solutions within seven (7) days of being notified of their winning status.\r\n
\r\n\r\n Team mergers are allowed and can be performed by the team leader. In order to merge, the combined team must have a total submission count less than or equal to the maximum allowed as of the merge date. The maximum allowed is the number of submissions per day multiplied by the number of days the competition has been running.
\r\n
There is no maximum team size.
\r\nYou may submit a maximum of 2 entries per day.
\r\nYou may select up to 5 final submissions for judging.
\r\n\r\n Because the data can be manually decoded and thus an entry can be created by hand, scoring data will be withheld until you have locked down a model.\r\n
\r\n\r\n You must upload all kernels and model files alongside documentation in a single zip file\r\n in order for your entry to be eligible. You can do this at the time of submission, or append\r\n it to your submission at a later time (before the competition end).\r\n
\r\nStart Date: 4/12/2013 11:10 PM UTC
\r\nMerger Deadline: None
\r\nEntry Deadline: None
\r\nEnd Date: 5/24/2013 11:59 PM UTC
\r\n\t\r\nYou must release your code under a popular OSI-approved license in order to be eligible for prize money.The organization will not claim any intellectual property from the winning team.
\r\nPeople affiliated with Google and with Université de Montréal may enter the contest, but no one who participated in the creation of the dataset may do so. People affiliated with Google are not eligible for prize money. People involved in the creation of the dataset are forbidden to give any advice regarding the contest to any of the competitors.
\r\nYou may use additional training data, but please restrict yourself to publicly available datasets. Do not manually label the test data and train your classifier on it.
\r\nYou do not need to use a representation learning algorithm. Algorithms that use hand-designed features such as HOG, SIFT, etc. are perfectly acceptable, as are algorithms that augment the dataset with synthetic transformations. However, your final system must be able to autonomously classify the test data without a human in the loop.
\r\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "data-description", "content": "
The test examples come in triplets: an image, and two annotations. The test images are located in public_test_images.tgz.
\r\nThe file public_test_options.tgz contains two files per test image: image_id.option_0.desc and image_id.option_1.desc. The models should predict which of the options fits the image better. If option 0 fits better, the prediction forimage_id should be 0. If option 1 fits better, the prediction for image_id should be 1.
\r\nThis test data was manually labeled by Ian Goodfellow, and designed to resemble the Small ESP Game dataset created by Luis von Ahn.
\r\nYou are therefore recommended to train your system on the Small ESP Game dataset, though you are allowed to use any publicly available training set. Please download the Small ESP Game dataset from this contest website, to avoid putting undue strain on Louis von Ahn's personal site.
\r\nThe ESP Game was an online game that awarded players points if they could label an image with the same word as another unknown player logged in from a different location. The fun game format encouraged a large number of people to participate in a data crowdsourcing effort. Keep in mind that the game format influences the kind of labels that were provided.
\r\nThe test data has some statistical differences from the ESP Game dataset. Most notably, test images are 300 pixels long on the larger dimension, while the training data comes in a variety of sizes. It is probably important to design your learning system to handle multiple scales of images.
\r\nThe ESP Game Dataset consists of 100,000 labeled images. The test data consists of 1000 images, split into public test and private test. The ESP Game Dataset has a single correct description for each image, while the test dataset provides a correct and an incorrect description for each image. The incorrect description is always the correct description of one other test image.
\r\nAn example script for preparing a submission is provided as part of the pylearn2 package:
\r\nhttp://github.com/lisa-lab/pylearn2
\r\nThe relevant script is at pylearn2/scripts/icml_2013_wrepl/multimodal
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Timeline", "content": "Post-Competition Wrapup
\r\nChallenges in Representation Learning ICML 2013 Workshop
\r\nThis contest is associated with the ICML 2013 Workshop on Representation Learning.
\r\nFirst prize: $350 and an invitation to speak at the workshop.
\r\nSecond prize: $150
\r\nWe encourage submission of papers to the workshop. Scientifically interesting papers will be accepted regardless of performance in the contest.
\r\nThe funding for the prizes was generously provided by Google, Inc.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false } ] }