There's recently been a lot of work done in unsupervised feature learning for classification, with great advances made by approaches such as deep belief nets, graphical models, and transfer learning. Meanwhile, there are a ton of older methods that also\n work well, including matrix factorization, random projections, and clustering methods. The purpose of this competition is to find out which of these methods work the best, on relatively large-scale high dimensional learning tasks.
\nThe Short Version
\nIn this task, you'll do the following:
\nWe have public data to be used for the leaderboard evaluations, and a separate private data set used for final evaluation through a special submission process. We have scripts to simplify the training and evaluation; how you do the feature transformation\n is up to you.
\n\n
The Long Version
\nThe task that we're evaluating on is a binary-class classification problem, drawn from web classification. (The data has been cleaned heavily and anonymized.) The data itself is sparse, high dimensional data with about a million features. A few features\n have non-zero values in many or even all examples; most features have non-zero values in very few examples.
\nYour task is to transform the data from a high dimensional space to a lower dimensional space of at most 100 features. The goal is to make this new feature space so rich and informative that it allows a new classifier to be trained with the best possible\n predictive performance. Any method of producing a condensed representation is fair game: deep learning graphical models, transfer learning, supervised learning, semi-supervised learning, matrix factorization, random projection, clustering, feature selection,\n or anything else you can invent.
\nIn addition to a small amount of labeled data, you will also be given a large amount of unlabeled data. Both the labeled and the unlabeled data can be used to learn good ways to transform the feature space.
\nThe final evaluation will be done by using your method to transform training and test sets whose labels are not known. These transformed data sets will be sent to tne organizers, who will apply the hidden labels and use these new data sets to train and\n test a standard supervised classifier. The data set that produces the best classification performance on test data (using AUC as the evaluation measure) will be declared the winner. We also provide versions of our evaluation scripts to be used on public\n versions of our private evaluation data sets, and these results can be used to update the leaderboard. Please see the \"Evaluation\" page for full details.
\nAlthough there is a modest cash prize, the main goal of this competition is to encourage research and share ideas. The results of this competition will be included in a paper submitted to the\n2011 NIPS workshop on deep learning and unsupervised feature learning. Contestants will be acknowledged by name in this paper for noteworthy performance, including results that do especially\n well or which are especially interesting.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Evaluation", "content": "The purpose of this task is to learn features that enable improved classification performance, so the evaluation is set up to test how well a classifier learns from your 100 features. The evaluation metric is AUC of a standard linear classifier trained\n on a labeled data set that uses only your features, and applied on a test set using your features.
\n\n
Preparing a Submission for Evaluation
\nTo prepare a submission file, please follow these steps. Note that we have provided a script runLeaderboardEval.pl that automates steps 3, 4, and 5 assuming you have downloaded and compiled libsvm.
\n1. Learn a way to transform the data from its sparse representation with a million features to a dense representation of at most 100 features. You may use any or all of the unlabeled and labeled data that has been made available to help learn this representation.
\n2. Use this representation to transform the data in the files public_train_data.svmlight.dat and public_test_data.svmlight.dat to your new feature space.
\n3. Train a Linear SVM with C=1.0 on your transformed training data, using the labels in public_train.labels.dat as ground truth for training. You can use\nlibsvm, \nSVM-light, or other similar packages.
\n4. Use your linear model to predict the class label for each example in your transformed test data.
\n5. Prepare a submission file in CSV format. The first value in each line should be the predicted value for that example using your learned model in step 4. The next 100 values in that line should be your 100 feature values. Each submission\n file should have 50,000 rows (one per test example, in order), and 101 comma-seaparated values. You can use the provided verifyFormat.pl script to check the format of your submission file.
\n6. Submit! You may make up to 2 submissions per day.
\n\n
Leaderboard Evaluations and Final Evaluation
\nThe leaderboard evaluation is done by computing AUC using 30% of the test data, and is to be used only for informational purposes.
\nThe final evaluation is done by computing AUC on the remaining 70% of the test data.
\nNote that because the evaluation is intended to evaluate the performance of a linear classifier trained on your features, we reserve the right to replicate the results of top-performers by performing 10-fold cross validation on the submitted transformed\n data, using libsvm with a linear SVM with C=1.0. This will allow us to confirm that a simple linear model can indeed be trained on the feature transformation provided. Submissions that cannot be shown to generate similar cross-validation results with the\n specified linear modeling scheme will be ruled ineligible.
\n\n
Describing Your Methods
\nBecause this is a research oriented competition, we ask all participants to fully describe their methods so that others may learn about the techniques used and the results can be used to make meaningful comparisons.
\nThe description should include some level of technical detail about your approaches. It should also include a rough wall-clock estimate of computation time needed to learn your feature representation, the time needed to apply your feature representation,\n and a description of the computation platform that was used. Finally, the full names of each contributor to your team should be included, along with contact info for payment should your team be the winner.
\nPlease send the description of your methods to semisupervisedfeatures@gmail.com no later than 11:59 PM (UTC) on Monday, Oct. 17th. Only teams that have submited informative descriptions will be eligible for the cash prize or for acknowledgement in the formal\n writeup of these results.
\nWe also encourage participants to share methods and ideas freely on the forum.
\n\n
Leaderboard Evaluation Script
\nWe have provided a script that can help to prepare submission files. See the \"Data\" page for instructions and to download.
\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Rules", "content": "\r\n\t\r\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\r\n\r\n Privately sharing code or data outside of teams is not permitted.\r\n It's okay to share code if made available to all participants on the forums.\r\n
\r\n\r\n Team mergers are allowed and can be performed by the team leader. In order to merge, the combined team must have a total submission count less than or equal to the maximum allowed as of the merge date. The maximum allowed is the number of submissions per day multiplied by the number of days the competition has been running.
\r\n
There is no maximum team size.
\r\nYou may submit a maximum of 2 entries per day.
\r\nYou may select up to 4 final submissions for judging.
\r\nStart Date: 9/24/2011 4:01 AM UTC
\r\nMerger Deadline: None
\r\nEntry Deadline: None
\r\nEnd Date: 10/17/2011 11:59 PM UTC
\r\n\t\r\nThe main rule to keep in mind is that this is primarily a research-oriented competition. Please consider the spirit of this event to be more collaborative than competitive.
\r\nWe do expect contestants to be on their honor when posting files to the leaderboard -- please only use a linear SVM with C=1 to produce predicted results from your transformed data, so that the leaderboard results are meaninful. This will be enforced at\r\n final evaluation time when we verify results via cross-validation.
\r\nIn all cases, the decisions of the organizers will be final.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "SubmissionInstructions", "content": "See the \"Evaluation\" page for detailed submission instructions both for leaderboard evaluations and the final evaluation. Note that the leaderboard evaluations are only for informational purposes; the winner will be determined solely on the basis of the\n final evaluation.
\nTip: due to the relatively large file size of the submissions, keep in mind that Kaggle supports compressed submissions. See\nthis link for more details
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Prizes", "content": "The winning team will be awarded $500 USD. The winning team will be determined by best AUC on the final submission, as detailed on the \"Evaluation\" page. Only submissions complying with all submission requirements are eligible for the prize.
\nOther teams with strong results or particularly interesting results will be acknowledged in the formal writeup of this competition. Our goal in the writeup is to give an overview of a diverse set of reasonable approaches.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false } ] }