{"metadata":{"kernelspec":{"display_name":"Python 3.9 (pytorch)","language":"python","name":"pytorch"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.16"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 📗 Goodreads: RoBERTa with Trainer API (Torch)","metadata":{}},{"cell_type":"markdown","source":"## Use Book Reviews to Predict Ratings","metadata":{}},{"cell_type":"markdown","source":"<div align=\"center\">\n    <img src=\"https://raw.githubusercontent.com/justinsiowqi/-Goodreads-RoBERTa-with-Trainer-API-Torch-/main/BERT.png\" alt=\"Sesame Street\" style=\"width: 500px;\"> \n</div>\n<div align=\"center\">\n  © Sesame Street (1969 TV Series)\n</div>","metadata":{}},{"cell_type":"markdown","source":"Welcome back to Part 2 of the series! \n\nIn the [previous notebook](https://www.kaggle.com/code/justinsiow/goodreads-distilbert-automodel-tensorflow), we used the HuggingFace transformers library to predict book ratings. Specifically, we created a **DistilBERT** AutoTokenizer and AutoModel and achieved an accuracy of **0.56808**. \n\nIn this notebook, we will take advantage of two other HuggingFace libraries (**datasets** and **evaluate**) to process the data and evaluate the model's performance. Instead of using the DistilBERT model, we will use a bigger model called **RoBERTa** and train it using the **Trainer API**. By increasing the amount of training data twofold, we managed to get a better accuracy of **0.59792**.","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"### <font color='000000'>Table of contents<font><a class='anchor' id='top'></a>\n\n1. [Introduction](#section-one)  \n    \n2. [Get Data](#section-two)\n    \n3. [Prepare Data](#section-three)\n    \n4. [Build Tokenizer](#section-four)\n    \n5. [Build & Train Model](#section-five) \n    \n6. [Test Model](#section-six)\n\n7. [Conclusion](#section-seven)","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-one\"></a>\n## 1. Introduction\n\nThe goal of this notebook is to **classify text** from books reviews and predict how many stars readers will rate that book. We'll begin by loading and processing the text using the **datasets** library, followed by tokenizing the text so that our model can understand. Then, we'll use the **transformers** library to create a **R**obustly **o**ptimized **BERT**-Pretraining **a**pproach (**RoBERTa**) model and train it using the Trainer API. In order to gauge how well our model has performed, we'll compute the model's accuracy using the **evaluate** library. Finally, we'll predict the rating of the test set and wrap everything up into a csv. \n\n**How to use this notebook**: This notebook uses PyTorch. If you'd prefer a simpler implementation using Tensorflow, check out [this notebook](https://www.kaggle.com/code/justinsiow/goodreads-distilbert-automodel-tensorflow) instead. Both notebooks come with guides that will help you better understand how the code works. \nLast but not least, have fun playing around with different models and hyperparamters!","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-two\"></a>\n## 2. Get Data\n\n- Download dependencies.\n- Define the checkpoint for AutoTokenizer and AutoModel.\n- Load training and test dataset using HuggingFace datasets.","metadata":{}},{"cell_type":"code","source":"# Dependencies\n# If on kaggle/Colab, uncomment and run this cell. If on terminal, remove exclamation marks\n\n# ! pip install datasets\n# ! pip install evaluate\n# ! pip install transformers","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Import libraries\n\nimport evaluate\nimport numpy as np\nimport pandas as pd\nimport torch\n\nfrom datasets import load_dataset, Features, Value, ClassLabel, load_metric, Dataset\nfrom transformers import AutoTokenizer, TrainingArguments, Trainer, AutoModelForSequenceClassification, AdamW","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### A Guide to AutoTokenizer and AutoModel:\n\n**AutoTokenizer** and **AutoModel** makes it super simple for you to retrieve the model you want to use. All you have to do is instantiate it using from_pretrained() and pass in the model/checkpoint. We set **use_fast=True** so that we can load the faster version of the tokenizer. There are 6 ratings in total (0 to 5), hence the model has to take an argument where num_labels=6.\n\nIf you want to try out different models, simply head over to [HuggingFace Models](https://huggingface.co/models) and replace 'roberta-base' in the checkpoint (yes, it's that simple!)","metadata":{}},{"cell_type":"code","source":"# Define the checkpoint for AutoModel and Autotokenizer\n\ncheckpoint = 'roberta-base'\ntokenizer = AutoTokenizer.from_pretrained(checkpoint, use_fast=True)\nmodel = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=6)\n\n# Activate Mac M1 GPU. If on Windows/Colab/Kaggle, replace all the mps with cuda\nif torch.backends.mps.is_available():\n    mps_device = torch.device('mps')\n    model.to(mps_device)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### A Guide to HuggingFace Datasets Part 1:\n\nLoading a dataset using HuggingFace **datasets** is simple. All you need to do is call **load_dataset()**. Next, you have to state the type of file that you're loading (eg. csv, json or txt). After that, you just need to attach the file path. HuggingFace datasets will assume that every file path is a training dataset, that's why we have to add 'train' and 'test' for HuggingFace datasets to differentiate.\n\nHuggingFace datasets is similar to Pandas dataframe but with lesser functionality. We will be using some functions later on. First, let's convert the rating column into the class label using **class_encode_column()**. This tells our model that we want to predict the ratings later on. Next, we'll call **rename_columns()** on rating and review_text to labels and text respectively because our model expects it later.","metadata":{}},{"cell_type":"code","source":"# Load the datasets\n\nraw_train_dataset = load_dataset('csv', data_files={\n    'train': '/kaggle/input/goodreads-books-reviews-290312/goodreads_train.csv'}\n)\n\nraw_test_dataset = load_dataset('csv', data_files={\n    'test': '/kaggle/input/goodreads-books-reviews-290312/goodreads_test.csv'}\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Convert rating column into class label\nraw_train_dataset['train'] = raw_train_dataset['train'].class_encode_column('rating')\n\n# Rename rating and review_text columns\nraw_train_dataset['train'] = raw_train_dataset['train'].rename_columns({'rating': 'labels', 'review_text': 'text'})","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Traning dataset has 11 features and 900,000 rows\n\ntrain_ds = raw_train_dataset['train']\ntrain_ds","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-three\"></a>\n## 3. Prepare Data\n\n- Remove books reviews that have:\n    - Negative number of votes.\n    - Negative number of comments.\n    - NaN values.\n    - Less than 30 words.\n- Take a random sample of the training dataset.\n- Split and encode the target variable.","metadata":{}},{"cell_type":"markdown","source":"### A Guide to HuggingFace Datasets Part 2:\n\nThere are two other functions we will use from the HuggingFace datasets library. First, we'll use the **filter()** function to remove book reviews where 1) n_votes is negative, 2) n_comments is negative, 3) NaN values exist and 4) number of words is less than 30. This helps us narrow down the text so that our model can learn better.\n\nNext, we'll use the **remove_columns()** function to remove all the columns except for the labels and text columns. The last thing to do is to randomly shuffle and sample 20% of the training dataset. The reason why we won't use the full dataset is because it'll take much longer to tokenize and train later on.","metadata":{}},{"cell_type":"code","source":"# Remove reviews where number of votes or number of comments is negative\ntrain_ds = train_ds.filter(lambda x: x['n_votes'] >= 0 and x['n_comments'] >= 0)\n\n# Remove NaN values\ntrain_ds = train_ds.filter(lambda x: x['text'] is not None)\n\n# Remove rows that have less than 30 words\ntrain_ds = train_ds.filter(lambda example: len(example['text'].split()) >= 30)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Drop the unnecessary columns from the training and test dataset\n\ntrain_columns_to_delete = ['user_id', 'book_id', 'review_id', 'date_added', 'date_updated', 'read_at', \\\n                           'started_at', 'n_votes', 'n_comments']\n\ntrain_ds = train_ds.remove_columns(train_columns_to_delete)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Take a random sample of the training dataset\n\ntrain_ds = train_ds.shuffle(seed=28).select(range(int(len(train_ds) * 0.2)))\nnum_rows = train_ds.num_rows\n\ntrain_ds","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split the training into 90% training and 10% validation\n\ndataset = train_ds.train_test_split(test_size=0.1, seed=28)\n\ntrain_dataset = dataset['train']\nval_dataset = dataset['test']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-four\"></a>\n## 4. Build Tokenizer\n\n- Create a function to tokenize text from the training and test dataset.\n- Set the maximum length to 128 words. For reviews with more than or less than 128 words, pad and truncate the text.","metadata":{}},{"cell_type":"code","source":"# Define a function to tokenize text from the training and test dataset. In order to re-use it later, we need to\n# create an if-else statement since the test dataset has no labels. \n\nlabels = ClassLabel(names=['0', '1', '2', '3', '4', '5'])\n\ndef tokenize_function(example):\n    tokens = tokenizer(example['text'], truncation=True, padding=True, max_length=128)\n    if 'labels' in example:\n        tokens['labels'] = labels.str2int(example['labels'])\n    else:\n        pass\n    return tokens\n\ntokenized_train = train_dataset.map(tokenize_function, batched=True)\ntokenized_val = val_dataset.map(tokenize_function, batched=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Remove the text column and set format to PyTorch\n\ntokenized_train = tokenized_train.remove_columns(['text'])\ntokenized_val = tokenized_val.remove_columns(['text'])\n\ntokenized_train.set_format('torch')\ntokenized_val.set_format('torch')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-five\"></a>\n## 5. Build & Train Model\n\n- Create a function to compute metrics and evaluate the model's performance.\n- Cutomize the training process via TrainingArguments.\n- Instantiate the trainer class and train the model.","metadata":{}},{"cell_type":"markdown","source":"### A Guide to Trainer API:\n\nThe HuggingFace **Trainer API** handles the training loop (and all its details) for you. There's just four simple steps to make it work: 1) define the model, 2) create a function to evaluate the model, 3) define training arguments and 4) define trainer. \n\nStep 1 has already been done right on top of this notebook. The next step is to create a function to compute the metrics and **evaluate our model's performance**. Let's load the accuracy metric from the HuggingFace Evaluate library. We need np.argmax() to transform the logits to something we can use to compare with the labels. The compute() method will basically calculate the metrics for us. \n\nIn Step 3, we'll define a **TrainingArguments** class that will contain all the hyperparameters we will use for training and evaluation. The model is evaluated and saved at the end of each epoch. Once training is complete, the best model will be loaded at the end. Finally, we'll define the trainer class and pass in the model, training arguments, tokenized data, tokenizer and the compute_metrics function. To start the training process, we just need to call **trainer.train()**.","metadata":{}},{"cell_type":"code","source":"# Define a function to evaluate the model's performance\n\ndef compute_metrics(eval_pred):\n    metric = evaluate.load('accuracy')\n    logits, labels = eval_pred\n    val_predictions = np.argmax(logits, axis=-1)\n    return metric.compute(predictions=val_predictions, references=labels)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Set the hyperparameters for the Trainer API. Feel free to tweak this\n\ntraining_args = TrainingArguments(\n    output_dir=f'{checkpoint}-{num_rows}-samples',\n    evaluation_strategy = 'epoch',\n    save_strategy = 'epoch',\n    per_device_train_batch_size=4,\n    per_device_eval_batch_size=4,\n    learning_rate=1e-5,\n    num_train_epochs=2,\n    weight_decay=0.01,\n    load_best_model_at_end=True\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use the Trainer API to train the RoBERTa model\n\ntrainer = Trainer(\n    model=model, \n    args=training_args, \n    train_dataset=tokenized_train,\n    eval_dataset=tokenized_val,\n    tokenizer=tokenizer,\n    compute_metrics=compute_metrics\n)\n\ntrainer.train()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-six\"></a>\n## 6. Test Model\n\n- Tokenize the review text from test set.\n- Feed the tokens into the RoBERTa model and use it to predict the book ratings.\n- Create a new CSV file for submission.","metadata":{}},{"cell_type":"code","source":"# Tokenize the test dataset. \n# All the steps here are exactly the same as the above, except we don't delete the review_id column\n\ntest_columns_to_delete = ['user_id', 'book_id', 'date_added', 'date_updated', 'read_at', 'started_at', \\\n                          'n_votes', 'n_comments']\n\ntest_dataset = raw_test_dataset['test'].rename_columns({'review_text': 'text'})\ntest_dataset = test_dataset.remove_columns(test_columns_to_delete)\n\ntokenized_test = test_dataset.map(tokenize_function, batched=True)\n\ntokenized_test = tokenized_test.remove_columns(['text'])\ntokenized_test.set_format('torch')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use the Trainer API to predict the tokenized test dataset\n\ntest_predictions = trainer.predict(tokenized_test)\npreds = np.argmax(test_predictions.predictions, axis=-1)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a new CSV file for submission\n\nmy_submission = pd.DataFrame({\n    'review_id': tokenized_test['review_id'],\n    'rating': preds\n})\n\nmy_submission.to_csv('submission.csv', index=False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-seven\"></a>\n## 7. Conclusion\n\nAnd that's a wrap! \n\nIn this notebook, we leveraged the power of the HuggingFace **datasets**, **transformers** and **evaluate** libraries. Specifically, we used the **Trainer API** to fine-tune a pre-trained **RoBERTa** model to predict book ratings from Goodreads. We achieved an accuracy of 59.792% which is pretty decent!\n\nThank you for sticking around till the end! If you've found this notebook helpful, please upvote it :)","metadata":{}},{"cell_type":"markdown","source":"### References:\n\n- [Processing the Data](https://huggingface.co/course/chapter3/2?fw=pt)\n- [Fine-Tuning a Model with Trainer API or Keras](https://huggingface.co/course/chapter3/3?fw=pt)\n- [Loading Datasets](https://huggingface.co/docs/datasets/loading)\n- [Text Classification on the IMDB Dataset](https://huggingface.co/docs/transformers/tasks/sequence_classification)","metadata":{}}]}