![\"\"](\"http://i.imgur.com/F255rFT.png\")
The format of the exam is a mock peer-reviewed conference. You will develop an algorithm, prepare a professional paper, submit an anonymized version to the EasyChair conference system, and peer-review the work from other groups. There are three deliverables:
\r\nThe UK government has just passed an exciting new surveillance bill that allows the government to spy on any citizen for any reason! These new legislative powers record and harvest images from every camera in the world without a warrant. However, government agents do not have the data science skills or coding chops to make sense of this large-scale imagery. The only natural solution: outsourcing the problem to Startups Like Yours! What a delicious opportunity for data scientists to score some sweet public funding! You and your team of two to three Cornell Tech students are surely on the path to fame and fortune!
\r\nThe specifications are simple: Our agents need a way to search for relevant images given a natural language query. For instance, if an agent types \"dog jumping to catch frisbee,\" your system will rank-order the most relevant images from a large database.
\r\n\"This system will be important,\" they tell you! \"We can use it to find propaganda! And contraband! And public enemies! All to protect our country and our children! And root out terrible terrorists!\"
\r\nIgnoring the obvious ethical implications, here are some clear specifications:
\r\nUse all of your skills, tools, and experience. It is OK to use libraries like numpy, scikit-learn, pandas, etc., as long as you cite them. Use cross-validation on training set to debug your algorithm. Submit your results to the Kaggle leaderboard and send your complete writeup to EasyChair. The data you use --- and the way you use the data --- is completely up to you.
\r\nThe best teams (of two to three Cornell Tech students, recall) might use visualization techniques for debugging (e.g., show top images retrieved by your algorithm and see whether they make sense or not), preprocessing, a nice way to compare tags and descriptions, leveraging visual features and combining them with tags and descriptions, supervised and/or unsupervised learning to best understand how to best take advantage of each data source available to them.
\r\nThe best reports will be professionally written. It begins with an Abstract. Then it follows an Introduction section, including motivations of your method, a detailed description of each step in your pipeline / framework you used for this task. It will also contain a detailed Experiment section and a Results section. If time permits, a Background / Related Work section can help to point out the relevant work in the image retrieval literature. Remember to include your team name on Kaggle in your report so we know who you are.
\r\nThe best reports use a professional style adopted by a major academic conference. NIPS is a good choice. Teams can download a template from Section 4 Paper Format of the NIPS Author Guidelines. We recommend you to use the LaTeX template, but this is not a requirement. If you're more familiar with Word, use Word.
\r\nThe best peer reviews will be detailed and thorough, pointing out potential areas of improvement as well as complimenting their peers' strengths.
\r\nThe best teams understand how to divide the precious time and energy among all members of the group. They might split into roles, preventing programming students from messing up the writing and ensuring writing gurus don't screw up the code. Close contact is always maintained within top-performing teams. When hardship happens within the best teams, blame is not assigned---it is overcome.
\r\nMay the best team win. God help us all.
\r\n-- Agent Belongie
\r\nYou are strongly encouraged to work in a group of two students, but no more than three students.
\r\nGroups may not collaborate with other groups. You cannot cite other teams' work. This includes sharing data, intermediate and final results, writing, or discussing your detailed method with other groups. You may not view reports from other teams until after the peer-review phase begins!
\r\nThe \"Peer Review\" phase will be double-blind. This means authors will not know who the reviewers are, and the reviewers should not know who the authors are. This protects both sides.
\r\nTo maintain anonymity, don't include team member names in the actual paper that you submit to EasyChair!
\r\nHowever, do indicate all team member names as authors on the EasyChair web page when you submit your paper! This will be hidden from your reviewers.
\r\n
See the following resources for tips on how to write a professional review:
\r\nEvery student should have an email in their Cornell inbox containing a link to sign into EasyChair as a program committee member.
\r\nTo submit the assignment, the team leader should log into EasyChair and switch to the Author role using the \"Change Role\" menu option on the top right:
\r\n![\"\"](\"http://i.imgur.com/omXGAg0.png\")
From there, the author can start a new submission using the menu on the top left. Be sure to put team members on the web page rather than on the paper PDF itself (see \"How to anonymize your paper\" above). You can come back and change your submission any time before the deadline.
\r\n![\"\"](\"http://i.imgur.com/pYFO4dK.png\")
The evaluation metric is Mean Average Precision at 20 (MAP@20) on the test set. A good explanation of the metric can be found here.
\r\nIn our case, there is only one true corresponding image. So for a description, suppose your algorithm find the corresponding image in top 20 of your returned list and its ranked i-th, MAP@20 gives a score of
\r\n$$score = \\frac{20+1-i}{20}$$
\r\nFor example, if its ranked 1st, you get 1; if its 11th, you get 0.5; if its 20th, you get 0.05. If the corresponding image is not in your top 20 list, you get 0 score. The overall score is calculated as the average score on the test set.
\r\nWe divide the test set into two splits: public split and private split. Results on the Leaderboard are evaluated on public split and the private split is used for grading.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "rules", "content": "\r\n\t\r\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\r\n\r\n Privately sharing code or data outside of teams is not permitted.\r\n It's okay to share code if made available to all participants on the forums.\r\n
\r\n\r\n Team mergers are allowed and can be performed by the team leader. In order to merge, the combined team must have a total submission count less than or equal to the maximum allowed as of the merge date. The maximum allowed is the number of submissions per day multiplied by the number of days the competition has been running.
\r\n
The maximum size of a team is 3 participants.
\r\nYou are allowed to make your team anonymous, which hides participant details.
\r\nYou may submit a maximum of 5 entries per day.
\r\nYou may select up to 3 final submissions for judging.
\r\nStart Date: 11/29/2016 3:52 PM UTC
\r\nMerger Deadline: None
\r\nEntry Deadline: None
\r\nEnd Date: 12/7/2016 4:59 AM UTC
\r\n\t\r\nYou are strongly encouraged to work in a group of two students, but no more than three students.
\r\nGroups may not collaborate with other groups. You cannot cite other teams' work. This includes sharing data, intermediate and final results, writing, or discussing your detailed method with other groups. You may not view reports from other teams until after the peer-review phase begins!
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false } ] }