The William and Flora Hewlett Foundation (Hewlett) is sponsoring the Automated Student Assessment Prize (ASAP). Hewlett is appealing to data scientists and machine learning specialists to help solve an important social problem. We need fast, effective and affordable solutions for automated grading of student-written essays.
\r\nHewlett is sponsoring the following prizes:
\r\nYou are provided access to hand scored essays, so that you can build, train and test scoring engines against a wide field of competitors. Your success depends upon how closely you can deliver scores to those of human expert graders. While we believe that these financial incentives are important, we also intend to introduce top performers both to leading vendors in the industry and/or an established base of interested buyers. Hewlett is opening the field of automated student assessment to you. We want to induce a breakthrough that is both personally satisfying and game-changing for improving public education.
\r\nToday, state departments of education are developing new forms of testing and grading methods, to assess the new common core standards. In this environment the need for more sophisticated and affordable options is vital. For example, we know that essays are an important expression of academic achievement, but they are expensive and time consuming for states to grade them by hand. So, we are frequently limited to multiple-choice standardized tests. We believe that automated scoring systems can yield fast, effective and affordable solutions that would allow states to introduce essays and other sophisticated testing tools. We believe that you can help us pave the way towards a breakthrough. ASAP is designed to achieve the following goals:
\r\nThe graded essays are selected according to specific data characteristics. On average, each essay is approximately 150 to 550 words in length. Some are more dependent upon source materials than others. This range of essay type is provided so that we can better understand the strengths of your solution. It is our intent to showcase quality and reliability, based on how well you can match expert human graders for each essay.
\r\nYou will be provided with training data for each essay prompt. The number of training essays does vary. For example, the lowest amount of training data is 1,190 essays, randomly selected from a total of 1,982. The data will contain ASCII formatted text for each essay followed by one or more human scores, and (where necessary) a final resolved human score. Where it is relevant, you are provided with more than one human score, so that you may evaluate the reliability of the human scorers, but - keep in mind - that you will be predicting to the resolved score. Also, please note that most essays are scored using a holistic scoring rubric. However, one data set uses a trait scoring rubric. The variability is intended to test the limits of your scoring engine’s capabilities.
\r\nFollowing a period of 3 months to build and/or train your engine, you will be provided with test data that will contain new essays, randomly selected for blind evaluation. However, you will notice that the rater and resolved score columns will be blank. You will be asked to supply, based on your engine's predictions for each essay, your score in the resolved score column and then submit your new data set on this site.
\r\nAs part of the file that you will submit with your predictive scores, you will be asked to submit additional information. We would like to understand both the time and capital that you’ve spent developing your engine, the profile of your team (or you as an individual if you are working alone) and the projected cost to implement your solution on a larger scale, along with any known limitations. Basically, you will have the opportunity to present your case for who you are, why your model is commercially viable and to what extent you can use your model to satisfy the interests of potential buyers. This other information will not be used to determine any prize rewards, and it is optional. But, if you provide it, it will be used to evaluate whether or not your model should be presented to state departments of education and others who stand to benefit from your work.
\r\nAlso, please note that it is our intention to stage other follow-on ASAP phases in the months ahead. We are starting with graded essays and will follow with new data:
\r\nIn every instance, we seek to drive innovation for new solutions to automated student assessment. We hope that you will enjoy this process. May the best model win!
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Background", "content": "One of the key roadblocks to advancing school-based curricula focused on critical thinking and analytical skills is the expense associated with scoring tests to measure those abilities. For example, tests that require essays and other constructed responses are useful tools, but they typically are hand scored, commanding considerable time and expense from public agencies. So, because of those costs, standardized examinations have increasingly been limited to using “bubble tests” that deny us opportunities to challenge our students with more sophisticated measures of ability. Recent developments in automating assessment of student essays and other response types are promising. And, states are showing increasing interest in them. To meet their needs, The William and Flora Hewlett Foundation (Hewlett) is sponsoring a series of competitions through the Automated Student Assessment Prize (ASAP), in which new participants will compete to demonstrate their capabilities for automating the grading process.
\r\nDemonstrations will be based on scoring actual student essays that were graded by experts, to determine whether or not competing solutions can deliver the same accuracy and reliability. The first prize focuses on “long-form constructed response” (essays), but it will be followed by other prizes in the months ahead using other forms of graded student content. Hewlett intends to drive innovation in this sector, at a time when state departments of education are working towards adopting new and more sophisticated student assessments. Hewlett believes that breakthroughs will lead to adoption of these models at a time when the need for solutions is critical to improving education. Hewlett believes that there is – more than ever before – vital interest among a large base of commercial interest. We are asking for your participation, both to improve the conditions that are inhibiting public education and to present viable and effective business solutions among a strongly motivated base of investors.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Evaluation", "content": "Essay score predictions are evaluated using objective criteria.
\r\nSpecifically, your performance will be evaluated with the quadratic weighted kappa error metric, which measures the agreement between two raters. This metric typically varies from 0 (only random agreement between raters) to 1 (complete agreement between raters). In the event that there is less agreement between the raters than expected by chance, this metric may go below 0. The quadratic weighted kappa is calculated between the automated scores for the essays and the resolved score for human raters on each set of essays. The mean of the quadratic weighted kappa is then taken across all sets of essays. This mean is calculated after applying the Fisher Transformation to the kappa values.
\r\nA set of essay responses E has N possible ratings, 1,2,…,N, and two raters, Rater A and Rater B. Each essay response e is characterized by a tuple (ea,eb), which corresponds to its scores by Rater A (resolved human score) and Rater B (automated score). The quadratic weighted kappa is calculated as follows. First, and N-by-N histogram matrix O is constructed over the essay ratings, such that Oi,j corresponds to the number of essays that received a rating i by Rater A and a rating j by Rater B.
\r\nAn N-by-N matrix of weights, w, is calculated based on the difference between raters’ scores:
\r\n$$w_{i,j} = \\frac{\\left(i-j\\right)^2}{\\left(N-1\\right)^2}$$
\r\nAn N-by-N histogram matrix of expected ratings, E, is calculated, assuming that there is no correlation between rating scores. This is calculated as the outer product between each rater’s histogram vector of ratings, normalized such that E and O have the same sum.
\r\nFrom these three matrices, the quadratic weighted kappa is calculated:
\r\n$$\\kappa=1-\\frac{\\sum_{i,j}w_{i,j}O_{i,j}}{\\sum_{i,j}w_{i,j}E_{i,j}}$$
\r\nThe Fisher Transformation is approximately a variance-stabilizing transformation and is defined:
\r\n$$z = \\frac{1}{2} \\ln \\frac{1+\\kappa}{1-\\kappa}$$
\r\nSince this transformation approaches infinity as kappa approaches 1, the maximum kappa value is capped at 0.999. Next the mean of the transformed kappa values is calculated in the z-space. For Essay Set #2, which has scores in two different domains, each transformed kappa is weighted by 0.5. This means that each dataset has an equally weighted contribution to the final score. Finally, the reverse transformation is applied to get the average kappa value:
\r\n$$\\kappa = \\frac{e^{2z}-1}{e^{2z}+1}$$
\r\nIf you have questions regarding the evaluation criteria, please refer to the help page.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "rules", "content": "\r\n\t\r\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\r\n\r\n Privately sharing code or data outside of teams is not permitted.\r\n It's okay to share code if made available to all participants on the forums.\r\n
\r\n\r\n Team mergers are allowed and can be performed by the team leader. In order to merge, the combined team must have a total submission count less than or equal to the maximum allowed as of the merge date. The maximum allowed is the number of submissions per day multiplied by the number of days the competition has been running.
\r\n
There is no maximum team size.
\r\nYou may submit a maximum of 2 entries per day.
\r\nYou may select up to 1 final submissions for judging.
\r\nStart Date: 2/10/2012 12:00 AM UTC
\r\nMerger Deadline: None
\r\nEntry Deadline: None
\r\nEnd Date: 4/30/2012 11:59 PM UTC
\r\n\t\r\nScore predictions for all essay sets are submitted to Kaggle in a single comma-separated value (CSV) file. The submission file contains 5 columns:
\r\nSample submission files for the validation and test sets will be released along with their corresponding data sets.
\r\nTo create this CSV file, you may copy and paste your score predictions into the predicted_score column, or submit a text file containing a single score prediction on each line and nothing else. In both cases, the order of the score predictions should be the same as it is specified in the example submission file.
\r\nOver the course of the model training period (February 10, 2012 - April 22, 2012), you may submit predictions on the validation set. In order to be eligible for prizes, you must make at least one submission for the validation set that uses the same model you will use for the test set.
\r\nModel Submission
\r\nThe timeline for this competition is based on the release of graded student essays in various sets, along with the deadlines for you to submit your models and predicted essay scores. We are offering eight essay sets, provided by seven states. Each essay set was randomly split into a training, validation, and test set.
\r\nHere is a brief description of each data set (please refer to the data page for more information):
\r\nHere is the timeline:
\r\nIn order to be eligible for prizes, you are required to submit the complete model you will use to score the test set prior to the release of the test set.
\r\nAll releases will be provided at 10am PST (6pm UTC); the deadlines are at midnight UTC on each scheduled date.
\r\n*If these sets are finalized prior to February 10, 2012, we reserve the right to release them at an earlier date.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Help", "content": "You are welcome to ask any questions you have regarding this competition, the data, and the Kaggle platform in the competition forum.
\r\nPlease contact us here for any private enquiries.
\r\n", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "data-description", "content": "
Code for evaluation metric and benchmarks
\r\nFor this competition, there are eight essay sets. Each of the sets of essays was generated from a single prompt. Selected essays range from an average length of 150 to 550 words per response. Some of the essays are dependent upon source information and others are not. All responses were written by students ranging in grade levels from Grade 7 to Grade 10. All essays were hand graded and were double-scored. Each of the eight data sets has its own unique characteristics. The variability is intended to test the limits of your scoring engine's capabilities.
\r\nThe training data is provided in three formats: a tab-separated value (TSV) file, a Microsoft Excel 2010 spreadsheet, and a Microsoft Excel 2003 spreadsheet. The current release of the training data contains essay sets 1-6. Sets 7-8 will be released on February 10, 2012. Each of these files contains 28 columns:
\r\n\r\n
\r\n
\r\n
\r\n
\r\n
Anonymization
\r\nWe have made an effort to remove personally identifying information from the essays using the Named Entity Recognizer (NER) from the Stanford Natural Language Processing group and a variety of other approaches. The relevant entities are identified in the text and then replaced with a string such as \"@PERSON1.\"
\r\nThe entitities identified by NER are: \"PERSON\", \"ORGANIZATION\", \"LOCATION\", \"DATE\", \"TIME\", \"MONEY\", \"PERCENT\"
\r\nOther replacements made: \"MONTH\" (any month name not tagged as a date by the NER), \"EMAIL\" (anything that looks like an e-mail address), \"NUM\" (word containing digits or non-alphanumeric symbols), and \"CAPS\" (any capitalized word that doesn't begin a sentence, except in essays where more than 20% of the characters are capitalized letters), \"DR\" (any word following \"Dr.\" with or without the period, with any capitalization, that doesn't fall into any of the above), \"CITY\" and \"STATE\" (various cities and states).
\r\nHere are some hypothetical examples of replacements made:
\r\nThe Automated Student Assessment Prize (ASAP) intends to solve the longstanding problem of high cost and low turnaround of current testing. The goal is to shift testing away from standardized bubble tests to tests that evaluate critical thinking, problem\r\n solving and other 21st century skills.
\r\n
\r\nThe William and Flora Hewlett Foundation is the sponsor of the competition. The foundation appreciates the cooperation of the two testing consortia, PARCC and SBAC, in the development of this demonstration and competition.
\r\n
\r\nASAP was designed and constructed by a prize partnership between Open Education Solutions (OpenEd), The Common Pool, and Kaggle.\r\nClick here for biographical sketches of the team.
\r\n
\r\n