About The Exam
The format of the exam is a mock peer-reviewed conference. You will develop an algorithm, prepare a professional paper, submit an anonymized version to the EasyChair conference system, and peer-review the work from other groups. There are three deliverables:
Phase 1: Develop your algorithm and write your writeup
Develop your algorithm and submit your results to the Kaggle leaderboard.
Deadline: Tuesday, Dec 6, 11:59 PM EST.
Write your paper submission and
[submit to EasyChair](https://easychair.org/conferences/?conf=iccfeprc2016)
using your Program Committee account in the Author role.
Deadline: Tuesday, Dec 6, 11:59 PM EST
. There should be only one submission from each team.
Phase 2: Peer-review other groups' work!
Write reviews and submit via EasyChair using your Program Committee account.
Deadline: Thursday, Dec 8, 11:59 PM EST
. (Will open after Tuesday)
Everybody
on each team must log in and complete their peer reviews. (They are not "per-team")
The Challenge: Build a large-scale image search engine!
The UK government has just passed an exciting new surveillance bill that allows the government to spy on any citizen for any reason! These new legislative powers record and harvest images from every camera in the world without a warrant. However, government agents do not have the data science skills or coding chops to make sense of this large-scale imagery. The only natural solution:
outsourcing the problem to Startups Like Yours!
What a delicious opportunity for data scientists to score some sweet public funding! You and your team of
two to three Cornell Tech students
are surely on the path to fame and fortune!
The specifications are simple: Our agents need a way to
search for relevant images
given a natural language query. For instance, if an agent types "dog jumping to catch frisbee," your system will
rank-order the most relevant images
from a large database.
"This system will be important," they tell you! "We can use it to find propaganda! And contraband! And public enemies! All to protect our country and our children! And root out
terrible terrorists
!"
Ignoring the obvious ethical implications, here are some clear specifications:
During training,
you have a dataset of 10,000 samples. Each sample has the following data available for learning:
A 224x224 JPG image.
A list of tags indicating objects appeared in the image.
Feature vectors extracted using
[ResNet](https://arxiv.org/abs/1512.03385)
, a state-of-the-art Deep-learned CNN (You don't have to train or run ResNet -- we are providing the features for you). See
[here](http://ethereon.github.io/netscope/#/gist/b21e2aae116dc1ac7b50)
for the illustration of the ResNet-101 architecture. The features are extracted from
pool5
and
fc1000
layer.
A five-sentence description, used to train your search engine.
During testing,
your system matches a
single five-sentence description
against a pool of 2,000
candidate samples
from the test set. Each sample has:
A 224x224 JPEG image.
A list of tags for that image.
ResNet feature vectors for that image.
Output
: For each description, your system must
rank-score
each testing image with the likelihood of that image matches the given sentence. Your system then returns the name of the top 20 relevant images, delimited by space. See "sample_submission.csv" on
[the data page](https://inclass.kaggle.com/c/cornell-cs5785-2016-fall-final/data)
for more details on the output format.
Evaluation metric
: There are 2,000 descriptions, and for each description, you must compare against the entire 2,000-image test set. That is, rank-order test images for each test description. We will use
MAP@20
as the evaluation metric. If the corresponding image of a description is among your algorithm's 20 highest scoring images, this metric gives you a certain score based on the ranking of the corresponding image. Please refer to
[the evaluation page](https://inclass.kaggle.com/c/cornell-cs5785-2016-fall-final/details/evaluation)
for more details.
Use all of your skills, tools, and experience. It is OK to use libraries like numpy, scikit-learn, pandas, etc., as long as you cite them. Use cross-validation on training set to debug your algorithm. Submit your results to the Kaggle leaderboard and send your complete writeup to EasyChair. The data you use --- and the way you use the data --- is completely up to you.
The best teams (of
two to three Cornell Tech students
, recall) might use visualization techniques for debugging (e.g., show top images retrieved by your algorithm and see whether they make sense or not), preprocessing, a nice way to compare tags and descriptions, leveraging visual features and combining them with tags and descriptions, supervised and/or unsupervised learning to best understand how to best take advantage of each data source available to them.
The best reports will be professionally written. It begins with an
Abstract
. Then it follows an
Introduction
section, including motivations of your method, a detailed description of each step in your pipeline / framework you used for this task. It will also contain a detailed
Experiment
section and a
Results
section. If time permits, a
Background / Related Work
section can help to point out the relevant work in the image retrieval literature. Remember to
include your team name on Kaggle in your report
so we know who you are.
The best reports use a professional style adopted by a major academic conference. NIPS is a good choice. Teams can download a template from Section 4 Paper Format of the
[NIPS Author Guidelines](https://nips.cc/Conferences/2015/PaperInformation/AuthorSubmissionInstructions)
. We recommend you to use the LaTeX template, but this is not a requirement. If you're more familiar with Word, use Word.
The best peer reviews will be detailed and thorough, pointing out potential areas of improvement as well as complimenting their peers' strengths.
The best teams understand how to divide the precious time and energy among all members of the group. They might split into roles, preventing programming students from messing up the writing and ensuring writing gurus don't screw up the code. Close contact is always maintained within top-performing teams. When hardship happens within the best teams, blame is not assigned---it is overcome.
May the best team win. God help us all.
-- Agent Belongie
Rules
You are strongly encouraged to work in a group of two students, but no more than three students.
Groups may not collaborate with other groups. You cannot cite other teams' work. This includes sharing data, intermediate and final results, writing, or discussing your detailed method with other groups. You
may not
view reports from other teams until
after
the peer-review phase begins!
How to anonymize your paper
The "Peer Review" phase will be double-blind. This means authors will not know who the reviewers are, and the reviewers should not know who the authors are. This protects both sides.
To maintain anonymity,
don't
include team member names in the actual paper that you submit to EasyChair!
However,
do
indicate all team member names as authors on the EasyChair web page when you submit your paper! This will be hidden from your reviewers.
![](http://i.imgur.com/F255rFT.png)
Peer Review Tips
See the following resources for tips on how to write a professional review:
Matthew Might’s notes on
[how to peer review](http://matt.might.net/articles/how-to-peer-review/)
and
[Peer Fortress: The Scientific Battlefield](http://matt.might.net/articles/peer-fortress/)
(the last one provides examples NOT to follow)
[CVPR 2016 Reviewer’s Guidelines](http://www.pamitc.org/cvpr16/reviewer_guidelines.php)
.
[So You're a Program Committee Member Now: On Excellence in Reviews and Meta-Reviews and Championing Submitted Work That Has Merit](http://mobilehci.acm.org/2015/download/ExcellenceInReviewsforHCICommunity.pdf)
How to submit your paper to EasyChair
Every student should have an email in their Cornell inbox containing a link to sign into EasyChair as a program committee member.
To submit the assignment, the team leader should log into EasyChair and switch to the Author role using the "Change Role" menu option on the top right:
![](http://i.imgur.com/omXGAg0.png)
From there, the author can start a new submission using the menu on the top left. Be sure to put team members on the web page rather than on the paper PDF itself (see "How to anonymize your paper" above). You can come back and change your submission any time before the deadline.
![](http://i.imgur.com/pYFO4dK.png)
Score Breakdown
30% on Kaggle performance (score on the private test set).
30% on report: is the proposed method clearly described? is it professionally written? does it include enough detail for a professional (ie. skilled graduate student) to re-implement the results? A good report should at least contain a brief introduction of what you did, a detailed description of your method, an experimental evaluation part that shows the experimental results and analysis of your results.
20% on method: does it make sense? is it suited to the task? does it make use of the data in appropriate ways?
10% on peer reviews from other classmates.
10% on the quality of peer review: is it well-written and thoughtful? does it provide insight to the authors?