{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 📕 Goodreads: DistilBERT AutoModel (Tensorflow)","metadata":{}},{"cell_type":"markdown","source":"## Use Book Reviews to Predict Ratings","metadata":{}},{"cell_type":"markdown","source":"<div align=\"center\">\n    <img src=\"https://github.com/justinsiowqi/-Goodreads-DistilBERT-AutoModel-Tensorflow-/blob/main/Sesame%20Street.gif?raw=true\" alt=\"Sesame Street\" style=\"width: 500px;\"> \n</div>\n<div align=\"center\">\n  © Sesame Street (1969 TV Series)\n</div>","metadata":{}},{"cell_type":"markdown","source":"In this notebook, we will create a **DistilBERT model** that is able to take book reviews, understand them and use them to predict book ratings. The **AutoTokenizer** and **AutoModel** classes from the **HuggingFace library** will come in handy here! \n\nAlso, since we're working with two huge datasets, we'll need to make certain adjustments to speed up the processing. This can be achieved through **Datatable**, **random sampling** and **fast tokenizers**. \n\nLet's dive in!","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"### <font color='000000'>Table of contents<font><a class='anchor' id='top'></a>\n\n1. [Introduction](#section-one)  \n    \n2. [Get Data](#section-two)\n    \n3. [Prepare Data](#section-three)\n    \n4. [Build Tokenizer](#section-four)\n    \n5. [Build & Train DistilBERT](#section-five) \n    \n6. [Test Model](#section-six)\n\n7. [Conclusion](#section-seven)","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-one\"></a>\n## 1. Introduction\n\nIn this notebook, we'll explore a type of **Natural Language Processing** (NLP) called **text classification**. Text classification is a process of adding labels to text. Specifically, we want to create a model that can understand book reviews and predict whether readers gave it one star, two stars... or five stars. In order to do so, we need two key components. First, we'll use a  **tokenizer** to convert words into tokens so that the model can understand. Secondly, the **classification model** will take the tokens and learn its context.\n\nSounds like a lot right? Thankfully, we can make use of the **HuggingFace library** to tokenize the text and train the model. In fact, these two parts can be created in 7 lines of code! We used the **DistilBERT** (distilled version of BERT) and got an accuracy of 0.57. If you'd like to see how to implement the BERT model using Pytorch, stay tuned for the next notebook [here](https://www.kaggle.com/justinsiow/code).","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-two\"></a>\n## 2. Get Data\n\n- Download dependencies\n- Load training and test dataset using datatable","metadata":{}},{"cell_type":"code","source":"# Dependencies\n# If on kaggle/Colab, uncomment and run this cell. If on terminal, remove exclamation marks\n\n# ! pip install datatable\n# ! pip install transformers\n# ! pip install tensorflow","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Import libraries\n\nimport datatable as dt\nimport pandas as pd\nimport pickle\nimport numpy as np\nimport tensorflow as tf\n\nfrom sklearn.preprocessing import LabelEncoder\nfrom tensorflow.keras.optimizers import Adam\nfrom transformers import AutoTokenizer, TFAutoModelForSequenceClassification","metadata":{"id":"2cuSDnG938mi","outputId":"6639cc77-b302-4386-b082-803eb39891f6"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### A Guide to DataTable:\n\nDatatable was created to process **humongous** amounts of data. You can think of it as a much faster version of pandas with lesser functionality. Using Datatable is really simple. Instead of the read_csv() function, we'll use the **fread()** function. Then, we use **.to_pandas()** to convert into a pandas dataframe. That's all! In the next section, we'll make use of all the functions we're familiar with in pandas.","metadata":{}},{"cell_type":"code","source":"# Load the training and test dataset\n\ntrain_df = dt.fread('/kaggle/input/goodreads-books-reviews-290312/goodreads_train.csv').to_pandas()\ntest_df = dt.fread('/kaggle/input/goodreads-books-reviews-290312/goodreads_test.csv').to_pandas()","metadata":{"id":"fvcdgxcj40Hk"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Take a look at the first 5 rows of the training set\n\ntrain_df.head()","metadata":{"id":"UaVGbEiL6Vku","outputId":"39e0781c-f7c2-41c9-977f-81c62d08a774"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{"id":"ny2_7bo2eXXM"}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-three\"></a>\n## 3. Prepare Data\n\n- Remove books reviews where:\n    - Number of votes are negative.\n    - Number of comments are negative.\n    - There are duplicates.\n- Take a random sample of the training dataset.\n- Split and encode the target variable.","metadata":{}},{"cell_type":"code","source":"# Remove reviews where number of votes or number of comments are negative\n\ntrain_df = train_df[train_df['n_votes'] >= 0]\n\ntrain_df = train_df[train_df['n_comments'] >= 0]","metadata":{"id":"FnT16S5L6-Qy"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Remove duplicate reviews\n\ntrain_df.drop_duplicates(subset=['review_text'], inplace=True)","metadata":{"id":"-P_urFh97E3V"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Take a random sample of the training dataset\n\ntrain_df = train_df.sample(int(len(train_df) * 0.1), random_state=28)","metadata":{"id":"ZdBs85Gp7GfW"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Drop the unnecessary columns from the training dataset. Reset index will remove index.\n\ntrain_columns_to_delete = ['user_id', 'book_id', 'review_id', 'date_added', 'date_updated', 'read_at', \\\n                     'started_at', 'n_votes', 'n_comments']\n\ntrain_df = train_df.drop(train_columns_to_delete, axis=1).reset_index(drop=True)","metadata":{"id":"mlyavLJB7IFZ"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split the target variable\n\nX_train = train_df.drop('rating', axis=1)\ny_train = train_df['rating']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Encode the target variable in the training dataset\n\nlabel_encoder = LabelEncoder()\ny_train = label_encoder.fit_transform(y_train)","metadata":{"id":"pCb9Rd3o7LAn"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Take a look at the X_train. There are 89,020 rows\n\nX_train","metadata":{"id":"QCdqADoO7McB","outputId":"b3142bf8-5d15-42ee-fde3-871bab4d8e25"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{"id":"WXfiOE0idjBL"}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-four\"></a>\n## 4. Build Tokenizer\n\n- Use the fast version of the DistilBERT AutoTokenizer.\n- Set the maximum length to 128 words. For reviews with more than or less than 128 words, pad and truncate the text.","metadata":{}},{"cell_type":"markdown","source":"### A Guide to AutoTokenizer:\n\nA tokenizer **splits text into tokens** so that the model can understand. We can start by calling **AutoTokenizer.from_pretrained()** which basically loads the **vocab** from a pretrained tokenizer. We will set **use_fast=True** which will load faster version of the model (Rust-based). \n\nNext, we'll call tokenizer on the review text from X_train. We will set return_tensors to numpy and the **max length to 128 words**. If the sentence has more than 128 words, we need to **truncate** the text (cut down the number of words). On the other hand, if the text has less than 128 words, we need to add **padding** (add zeros to make the sentence longer).\n\nThe tokenizer will return a BatchEncoding object. We'll need to convert it to a dictionary. Also, we'll create a variable called labels which is a numpy array of y_train.","metadata":{}},{"cell_type":"code","source":"# DistilBERT AutoTokenizer with a maximum length of 128 words. Truncation and padding is used as well.\n\ntokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased', use_fast=True)\ntokenized_data = tokenizer(list(X_train['review_text']), return_tensors=\"np\", max_length=128, truncation=True, padding=True)\n\n# Convert tokenized data to a dictionary and y_train to a numpy array\ntokenized_data = dict(tokenized_data)\nlabels = np.array(y_train) ","metadata":{"id":"ZNqF_f1aa-f3"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-five\"></a>\n## 5. Build & Train DistilBERT\n\n- Use the DistilBERT AutoModel with 6 labels.\n- Use the Adam optimizer. HuggingFace will define the loss function for us.\n- Save the model.","metadata":{}},{"cell_type":"markdown","source":"### A Guide to AutoModel:\n\nA **pretrained model** saves you a lot of time and effort compared to training the model from scratch. First, let's call **TFAutoModelForSequenceClassification.from_pretrained()** and pass in the same model as the tokenizer above. The number of labels will be 6 since the ratings are from 0 to 5. \n\nWe'll use the **Adam optimizer** with a very low learning rate. We **don't have to specify a loss function**, HuggingFace will automatically do that for us! Finally, we'll call .fit() and pass in the tokenized data and labels.","metadata":{}},{"cell_type":"code","source":"# DistilBERT AutoModel with 6 labels. Optimizer is Adam and loss is automatically set.\n\n# Load and compile our model\nmodel = TFAutoModelForSequenceClassification.from_pretrained(\"distilbert-base-uncased\", num_labels=6)\n\n# Lower learning rates are often better for fine-tuning transformers\nmodel.compile(optimizer=Adam(3e-5))\n\nmodel.fit(tokenized_data, labels)","metadata":{"id":"ybsDFqsKb5-4","outputId":"f5b9642d-8f4c-4e39-8389-8ca5a3b2d3e3"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Save model\n\nmodel.save_pretrained('./model/clf')\nwith open('./model/info.pkl', 'wb') as f:\n    pickle.dump(('distilbert-base-uncased', 128), f)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-six\"></a>\n## 6. Test Model\n\n- Tokenize the review text from test set.\n- Feed the tokens into the DistilBERT model and use it to predict the book ratings.\n- Create a new CSV file for submission.","metadata":{}},{"cell_type":"code","source":"# Drop the unnecessary columns from the test dataset. Reset index will remove index.\n\ntest_columns_to_delete = ['user_id', 'book_id', 'date_added', 'date_updated', 'read_at', \\\n                     'started_at', 'n_votes', 'n_comments']\n\ntest_df = test_df.drop(test_columns_to_delete, axis=1).reset_index(drop=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now we need to tokenize the test dataset and then use the model to predict\n\ntokenized_test = tokenizer(list(test_df['review_text']), return_tensors=\"np\", max_length=128, truncation=True, padding=True)\ntokenized_test = dict(tokenized_test)\n\npreds = model.predict(tokenized_test)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a new CSV file that includes the 'review_id' and predicted 'rating'\n\nfor n in range(len(test_df)):\n    logit = preds.logits[n]\n    results[n] = int(np.argmax(logit))\n\nmy_submission = pd.DataFrame({\n    \"review_id\": test_df[\"review_id\"],\n    \"rating\": results.astype(int)\n})\n\nmy_submission.to_csv('submission.csv', index=False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-seven\"></a>\n## 7. Conclusion\n\nIn this notebook, we used the **DistilBERT** model to predict book ratings. We started by loading the datasets using **Datatable**. Next, we preprocessed the data by removing irrelevant text and taking a random sample. Then, the **AutoTokenizer** function converts text to tokens and **AutoModel** calls the pretrained model for us to train. Finally, we used the predict() function to get our submission file.\n\nThanks for looking through this notebook. Feel free to check out my second and third attempts at the Goodreads Books Reviews competition. Also, do drop an upvote if this has helped you in any way :)","metadata":{}},{"cell_type":"markdown","source":"### References:\n\n- [An Overview of Python's Datatable Package](https://towardsdatascience.com/an-overview-of-pythons-datatable-package-5d3a97394ee9)\n- [Preprocess](https://huggingface.co/docs/transformers/preprocessing)\n- [Fine-tune a Pretrained Model](https://huggingface.co/docs/transformers/training)","metadata":{}}]}