The dataset consists of two json files: *train.json* and *test.json*. These files are different from the original release of the CoNNL-2003 since we have already processed and tokenized them for you. Hence, the documents are represented as a list of strings. The dataset is **not** split into separate training and development/validation sets. You will need to do this yourself as needed using train set.

The train file contain the following 4 fields (each is a nested list): 

  - **'text'** - actual input tokens

  - **'NER'** - the entity tag (LOC/MISC/O/ORG/PER)

  - **'POS'** - the parts of speech tag (will be handy for feature engineering of the MEMM model)

  - **'index'** - index of the token in the dataset

The test data only has 'text', 'POS' and 'index' fields. You will need to submit your prediction of 'NER' tag to the Kaggle.