This competition has completed, congratulations to the preliminary winners and the other participants!
\r\nThe William and Flora Hewlett Foundation (Hewlett Foundation) is sponsoring the Automated Student Assessment Prize (ASAP) in hopes of discovering new tools to support schools and teachers. The competition aspires to solve the problem of the high cost and the slow turnaround of hand scoring thousands of written responses in standardized tests. As a result many schools exclude written responses in favor of multiple-choice questions, which are less able to assess students’ critical reasoning and writing skills. ASAP has been designed to help determine whether computerized systems are capable of grading written content accurately for schools and teachers to adopt those solutions. ASAP aspires to inform key decision makers, who are already considering adopting these systems, by delivering a fair, impartial and open series of trials to test current capabilities and to drive greater awareness when outcomes warrant further consideration.
\r\nCritical reasoning is one of a suite of skills that experts believe students must be taught to succeed in the new century. The Hewlett Foundation makes grants to educators and nonprofit organizations in support of these skills, which it calls “deeper learning.” They include the mastery of core academic content, critical reasoning and problem solving, working collaboratively, communicating effectively, and learning how to learn independently. With ASAP, Hewlett is appealing to data scientists to help solve an important problem in the field of educational testing.
\r\nHewlett is sponsoring the following prizes as part of Phase Two:
\r\n$50,000: 1st place
$25,000: 2nd place
$15,000: 3rd place
$ 7,500: 4th place
$ 2,500: 5th place
In May of this year, $100,000 in prizes was rewarded for ASAP, Phase One, and we have launched ASAP, Phase Two, with the same intentions. During Phase One, we focused on systems to support the grading of student written essays. This time, we’re offering a similar competition, only focused on short answer responses. We welcome you to learn more about our previous phase at www.kaggle.com/c/asap-aes
\r\nDuring Phase Two, you are provided access to graded short answer responses and their corresponding prompts, so that you can build, train and test your scoring engines against a wide field of competitors. Your success depends upon how closely you can align your scores to those of human expert graders. While we believe that a pool of $100,000 in potential financial incentives are important, we also intend to secure and distribute your solutions to the public, in hopes of elevating the field of automated assessment through your contributions. We want you to induce a breakthrough that is both personally satisfying and game-changing for improving public education.
\r\nWe have already learned that automated assessment systems can yield fast, effective and affordable solutions that would allow states to introduce new testing tools capable of assessing deeper measures of learning. We believe that you can help us pave the way towards better student assessment.
\r\nASAP is designed to achieve the following goals:
\r\nThe Phase Two graded content is selected according to specific characteristics. On average, each answer is approximately 50 words in length. Some are more dependent upon source materials than others, and the answers cover a broad range of disciplines (from English Language Arts to Science). The range of answer types is provided so that we can better understand the strengths of your solution. It is our intent to showcase quality and reliability, based on how well you can align with expert human graders for each response.
\r\nYou will be provided with training data for each prompt. Most training sets will consist of about 1,800 responses that have been randomly selected from a sample of approximately 3,000. The number of training data may vary. The data will contain ASCII formatted text for each response followed by two hand scores. The first score is the final score and the one that you are trying to predict. The second score was used to determine reliability of the first score. The second score did not in any way influence the first (final) score. You are provided with both scores, so that you may evaluate the reliability of the hand scoring. Further instruction and clarification regarding the data is available on the DATA tab.
\r\nFollowing a period of 2.5 months to train your scoring engine, you will be provided with test data that will contain approximately 6,000 new responses (600 per data set), randomly selected for blind evaluation. However, you will notice that the score columns will be blank. You will be asked to supply, based on your engine's predictions for each response, your score for each response and to submit your new scored data set on this site.
\r\nAs part of the ZIP file that you will submit with your predictive scores, you will be asked to submit a technical METHODS PAPER. We would like to understand your specific approach to developing your scorig engine, along with any known limitations. Basically, you will have the opportunity to present your scoring engine to the world, so that others may build upon it. Your technical METHODS PAPER will not be used to determine any prize rewards, but it is a required component of your final submission.
\r\nAlso, please note that it is our intention to continue staging other follow-on ASAP phases in the months ahead. We have started with graded essays (Phase 1), and we are now focusing on short answers (Phase 2); we are developing plans for a third phase, and we’re planning to launch a phase to demonstrate efficacy of systems capable of offering formative feedback as part of classroom applications:
\r\nIn every instance, we seek to drive innovation for new solutions to student assessment, to support teachers in evaluating critical reasoning skills. We hope that you will enjoy this process. May the best model win!
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Background", "content": "One of the key roadblocks to teaching and evaluating critical thinking and analytical skills is the expense associated with scoring tests to measure those abilities. For example, tests that require “constructed responses” (i.e., written answers) are useful tools, but they typically are hand scored, commanding considerable time and expense from public agencies. So, because of those costs, standardized examinations have increasingly been limited to using “bubble tests” that deny us opportunities to challenge our students with more sophisticated measures of ability. Recent developments in innovative software to evaluate student written responses and other response types are promising. And, states are showing increasing interest in them.
\r\nDemonstrations will be based on scoring actual student responses that were graded by experts, to determine whether or not competing solutions can deliver the same accuracy and reliability. The first prize focused on “long-form constructed response” (i.e., essays), but we are now focusing on “short-form constructed response” (i.e., short answers), which for some have proven even more difficult. We will launch other ASAP phases in the months ahead using other forms of graded student content. Hewlett intends to drive innovation in this sector, at a time when state departments of education are working towards adopting new and more sophisticated student assessments.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Evaluation", "content": "The code to calculate Quadratic Weighted Kappa, along with test cases, can be found at this github repo.
\r\nScore predictions are evaluated based on objective criteria, and specifically using the quadratic weighted kappa error metric, which measures the agreement between two raters. This metric typically varies from 0 (only random agreement between raters) to 1 (complete agreement between raters). In the event that there is less agreement between the raters than expected by chance, this metric may go below 0. The quadratic weighted kappa is calculated between the automated scores for the responses and the resolved score for human raters on each set of responses. The mean of the quadratic weighted kappa is then taken across all sets of responses.
\r\nA set of essay responses E has N possible ratings, 1,2,…,N, and two raters, Rater A and Rater B. Each essay response e is characterized by a tuple (ea,eb), which corresponds to its scores by Rater A (resolved human score) and Rater B (automated score). The quadratic weighted kappa is calculated as follows. First, and N-by-N histogram matrix O is constructed over the essay ratings, such that Oi,j corresponds to the number of essays that received a rating i by Rater A and a rating j by Rater B.
\r\nAn N-by-N matrix of weights, w, is calculated based on the difference between raters’ scores:
\r\n$$w_{i,j} = \\frac{\\left(i-j\\right)^2}{\\left(N-1\\right)^2}$$
\r\nAn N-by-N histogram matrix of expected ratings, E, is calculated, assuming that there is no correlation between rating scores. This is calculated as the outer product between each rater’s histogram vector of ratings, normalized such that E and O have the same sum.
\r\nFrom these three matrices, the quadratic weighted kappa is calculated:
\r\n$$\\kappa=1-\\frac{\\sum_{i,j}w_{i,j}O_{i,j}}{\\sum_{i,j}w_{i,j}E_{i,j}}$$
\r\nThe Fisher Transformation is approximately a variance-stabilizing transformation and is defined:
\r\n$$z = \\frac{1}{2} \\ln \\frac{1+\\kappa}{1-\\kappa}$$
\r\nSince this transformation approaches infinity as kappa approaches 1, the maximum kappa value is capped at 0.999. Next the mean of the transformed kappa values is calculated in the z-space. Finally, the reverse transformation is applied to get the average kappa value:
\r\n$$\\kappa = \\frac{e^{2z}-1}{e^{2z}+1}$$
\r\nIf you have questions regarding the evaluation criteria, please refer to the help page.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Submission Instructions", "content": "Score predictions for all essay sets are submitted to Kaggle in a single comma-separated value (CSV) file. The submission file contains 2 columns:
\r\nSample submission files for the public leaderboard set is on the data page, a sample submission for the private leaderboard set will be released with that set.
\r\nTo create this CSV file, you may copy and paste your score predictions into the essay_score column, or submit a text file containing a single score prediction on each line and nothing else. In both cases, the order of the score predictions should be the same as it is specified in the example submission file.
\r\nOver the course of the model training period (June 25, 2012 - August 29, 2012), you may submit predictions on the validation set. In order to be eligible for prizes, you must make at least one submission for the validation set that uses the same model you will use for the test set.
\r\nModel Submission
\r\n\r\n
Prizes are available for top performers, comprised of a total cash pool of $100,000 to the five winners, which will be divided as follows:
\r\n1st PLACE: $50,000.00
2nd PLACE: $25,000.00
3rd PLACE: $15,000.00
4th PLACE: $ 7,500.00
5th PLACE: $ 2,500.00
You are welcome to ask any questions you have regarding this competition, the data, and the Kaggle platform in the competition forum.
\r\nPlease contact us here for any private enquiries.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "data-description", "content": "\r\nFor this competition, there are ten data sets. Each of the data sets was generated from a single prompt. Selected respones have an average length of 50 words per response. Some of the essays are dependent upon source information and others are not. All responses were written by students primarily in Grade 10. All responses were hand graded and were double-scored. Each of the eight data sets has its own unique characteristics. The variability is intended to test the limits of your scoring engine's capabilities.
\r\nThe training data is provided in a tab-separated value (TSV) file containing the following columns:
\r\nThe private leaderboard set will not be released until August 30, 2012. The public leaderboard and private leaderboard files each have the following columns:
\r\nIn addition, a Microsoft Word 2010 Readme file describes each essay set. The Readme file contains the prompt that the essays in the data file were generated from. If applicable, the Readme file also includes the source information for essays that required students to read and respond to an excerpt.
\r\nThe timeline for this competition is based on the release of graded student essays in various sets, along with the deadlines for you to submit your models and predicted essay scores. We are offering eight essay sets, provided by seven states. Each essay set was randomly split into a training, validation, and test set.
\r\nHere is a brief description of each data set (please refer to the data page for more information):
\r\nHere is the timeline:
\r\nIn order to be eligible for prizes, you are required to submit the complete model you will use to score the test set prior to the release of the test set.
\r\nAll releases will be provided at 10am PST (6pm UTC); the deadlines are at midnight UTC on each scheduled date.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Team", "content": "The Automated Student Assessment Prize (ASAP) intends to solve the longstanding problem of high cost and low turnaround of current testing. The goal is to shift testing away from standardized bubble tests to tests that evaluate critical thinking, problem solving and other 21st century skills.
The William and Flora Hewlett Foundation is the sponsor of the competition. The foundation appreciates the cooperation of the two testing consortia, PARCC and SBAC, in the development of this demonstration and competition.
ASAP was designed and constructed by a prize partnership between Open Education Solutions (OpenEd), The Common Pool, and Kaggle. Click here for biographical sketches of the team.
You cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\r\n\r\n Privately sharing code or data outside of teams is not permitted.\r\n It's okay to share code if made available to all participants on the forums.\r\n
\r\n\r\n Team mergers are allowed and can be performed by the team leader. In order to merge, the combined team must have a total submission count less than or equal to the maximum allowed as of the merge date. The maximum allowed is the number of submissions per day multiplied by the number of days the competition has been running.
\r\n
There is no maximum team size.
\r\nYou may submit a maximum of 2 entries per day.
\r\nYou may select up to 1 final submissions for judging.
\r\n\r\n Because the data can be manually decoded and thus an entry can be created by hand, scoring data will be withheld until you have locked down a model.\r\n
\r\n\r\n You must upload all kernels and model files alongside documentation in a single zip file\r\n in order for your entry to be eligible. You can do this at the time of submission, or append\r\n it to your submission at a later time (before the competition end).\r\n
\r\nStart Date: 6/25/2012 5:09 PM UTC
\r\nMerger Deadline: None
\r\nEntry Deadline: None
\r\nEnd Date: 9/5/2012 11:59 PM UTC
\r\n\t\r\nWe would like to congratulate the preliminary prize winners for this competition! Additionally we would like to recognize teams Measurement Inc, Educational Testing Servces, and Stefan Henß & SirGuessALot, who came in 1st, 5th, and 6th on the leaderboard. These teams elected not to open source their models and methodologies, so the monetary rewards will go to the subsequent teams on the leaderboard.
\r\n\r\n