In this competition, we're predicting the reading ease of excerpts from literature. We've provided excerpts from several time periods and a wide range of reading ease scores. Note that the test set includes a slightly larger proportion of modern texts (the type of texts we want to generalize to) than the training set.

**Also note that while licensing information is provided for the public test set (because the associated excerpts are available for display / use), the hidden private test set includes only blank license / legal information.**

## Files

*   **train.csv** - the training set
*   **test.csv** - the test set
*   **sample_submission.csv** - a sample submission file in the correct format

## Columns

*   `id` - unique ID for excerpt
*   `url_legal` - URL of source - this is blank in the test set.
*   `license` - license of source material - this is blank in the test set.
*   `excerpt` - text to predict reading ease of
*   `target` - reading ease
*   `standard_error` - measure of spread of scores among multiple raters for each excerpt. Not included for test data.

## Update

This dataset, the **CLEAR Corpus**, has now been released in full. You may obtain it from either of the following locations:

- [commonlit.org](https://blog.commonlit.org/introducing-the-clear-corpus-an-open-dataset-to-advance-research-28ff8cfea84a)
- [github.com](https://github.com/scrosseye/CLEAR-Corpus)

The full corpus contains an expanded set of fields as well as six readability predictions on each except resulting from this competition.

You may read more about the CLEAR Corpus from the following publications:

- Crossley, S. A., Heintz, A., Choi, J., Batchelor, J., Karimi, M., & Malatinszky, A. (in press). A large-scaled corpus for assessing text readability. *Behavior Research Methods.* [[link]](https://link.springer.com/article/10.3758/s13428-022-01802-x)
- Crossley, S. A., Heintz, A., Choi, J., Batchelor, J., & Karimi, M. (2021). The CommonLit Ease of Readability (CLEAR) Corpus. Proceedings of the 14th International Conference on Educational Data Mining (EDM). Paris, France. [[link]](https://educationaldatamining.org/EDM2021/virtual/static/pdf/EDM21_paper_35.pdf)