{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Goodreads Books Reviews prediction\n\nHello,\n\nIn this notebook, I aim to provide predictions for the Goodreads Book Reviews competition. The notebook includes several steps such as exploratory data analysis (EDA), text cleaning, modeling with a Neural Network, and model analysis. Additionally, I have included some comments here and there or in the code.\n\nFeel free to comment, criticize  or upvote this notebook.","metadata":{}},{"cell_type":"markdown","source":"### Getting the training data","metadata":{}},{"cell_type":"code","source":"# Import pandas\nimport pandas as pd\n\n# Import training data\ndf_train = pd.read_csv('/kaggle/input/goodreads-books-reviews-290312/goodreads_train.csv')\n\n# Print few random entries\ndf_train.sample(5)","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:40:45.710459Z","iopub.execute_input":"2023-02-24T14:40:45.711207Z","iopub.status.idle":"2023-02-24T14:40:59.882050Z","shell.execute_reply.started":"2023-02-24T14:40:45.711095Z","shell.execute_reply":"2023-02-24T14:40:59.881109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### EDA","metadata":{}},{"cell_type":"code","source":"# Checking the number of rows and columns\nprint(\"The DataFrame has \" + str(df_train.shape[0]) + \" samples and \" + str(df_train.shape[1]) + \" columns\")","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:40:59.884023Z","iopub.execute_input":"2023-02-24T14:40:59.885126Z","iopub.status.idle":"2023-02-24T14:40:59.892799Z","shell.execute_reply.started":"2023-02-24T14:40:59.885064Z","shell.execute_reply":"2023-02-24T14:40:59.891945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check duplicates\nprint(\"Duplicate entries in the dataset: \" + str(df_train.duplicated().sum()))","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:40:59.894135Z","iopub.execute_input":"2023-02-24T14:40:59.894568Z","iopub.status.idle":"2023-02-24T14:41:04.468680Z","shell.execute_reply.started":"2023-02-24T14:40:59.894511Z","shell.execute_reply":"2023-02-24T14:41:04.467642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check null values and data type of each column\ndf_train.info()","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:41:04.470228Z","iopub.execute_input":"2023-02-24T14:41:04.470602Z","iopub.status.idle":"2023-02-24T14:41:04.885469Z","shell.execute_reply.started":"2023-02-24T14:41:04.470563Z","shell.execute_reply":"2023-02-24T14:41:04.884188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n%matplotlib inline\n%config InlineBackend.figure_format = 'retina' \n\n# Check target balance\ndf_train['rating'].value_counts().sort_index().plot.bar(figsize=(10,5))\nplt.title('Rating distribution in the training set', fontweight=\"bold\")\nplt.xlabel('Target')\nplt.ylabel('Number of reviews')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:41:04.888588Z","iopub.execute_input":"2023-02-24T14:41:04.889951Z","iopub.status.idle":"2023-02-24T14:41:05.293482Z","shell.execute_reply.started":"2023-02-24T14:41:04.889900Z","shell.execute_reply":"2023-02-24T14:41:05.292271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This bar plot is pretty self-explanatory, with most of the ratings having 4 or 5 stars.\n\nNow, I will focus on the text data, hence drop the columns that I won't need anymore in this project.","metadata":{}},{"cell_type":"code","source":"# Drop few columns\ndf_train = df_train.drop(columns=['user_id','book_id','date_added','date_updated',\n                                  'read_at','started_at','n_votes','n_comments'])","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:41:05.295105Z","iopub.execute_input":"2023-02-24T14:41:05.295357Z","iopub.status.idle":"2023-02-24T14:41:05.456341Z","shell.execute_reply.started":"2023-02-24T14:41:05.295330Z","shell.execute_reply":"2023-02-24T14:41:05.455472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"At this point we can write a function to clean the raw text and make it more appropriate for the modeling part.","metadata":{}},{"cell_type":"code","source":"import re\nfrom nltk.corpus import stopwords\n\n# Create a stopwords set adding some personal 'words'\nstopwords_english = set(stopwords.words('english'))\nmy_stopwords = set([\"http\", \"'s\", \"n't\", \"'m\", \"'re\", \"'ve\"])\nstopwords_english.update(my_stopwords)\n\ndef preprocess_review(text):\n    # Convert to lower case\n    text = text.lower()\n\n    # Remove numbers\n    text = re.sub(r'\\d+', '', text)\n\n    # Remove punctuation\n    text = re.sub(r'[^\\w\\s]', '', text)\n\n    # Split text into tokens\n    tokens = text.split()\n\n    # Filter tokens\n    clean_tokens = [tok for tok in tokens if tok not in stopwords_english and len(tok) > 1]\n\n    # Join tokens into a string\n    clean_text = ' '.join(clean_tokens)\n\n    return clean_text","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:41:05.457906Z","iopub.execute_input":"2023-02-24T14:41:05.458165Z","iopub.status.idle":"2023-02-24T14:41:06.132638Z","shell.execute_reply.started":"2023-02-24T14:41:05.458136Z","shell.execute_reply":"2023-02-24T14:41:06.131261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are multiple ways to clean text, including stemming and lemmatization. Although these techniques can enhance the outcome, they also slow down the function considerably, especially for larger datasets like this one. For instance, my previous function using these techniques took more than an hour to train the dataset, whereas the new solution takes a little over a minute. Therefore, I opted for a faster function even though it may result in slightly less accuracy. \n\nLet's check it out:","metadata":{}},{"cell_type":"code","source":"# Get one review as sample\nsample = df_train['review_text'][5]\n\nprint('ORIGINAL REVIEW:   ' + sample + '\\n')\nprint('---------------------------------------------------------------------------- \\n')\nprint('WITH PROCESSING:    ' + preprocess_review(sample))","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:41:06.134896Z","iopub.execute_input":"2023-02-24T14:41:06.135421Z","iopub.status.idle":"2023-02-24T14:41:06.144328Z","shell.execute_reply.started":"2023-02-24T14:41:06.135354Z","shell.execute_reply":"2023-02-24T14:41:06.142994Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Not bad, should work well enough. Let's apply the function to all the corpus:","metadata":{}},{"cell_type":"code","source":"%%time\n\n# Preprocess training data adding new column\ndf_train[\"clean_text\"] = df_train[\"review_text\"].apply(preprocess_review)","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:41:06.146463Z","iopub.execute_input":"2023-02-24T14:41:06.146911Z","iopub.status.idle":"2023-02-24T14:42:31.093118Z","shell.execute_reply.started":"2023-02-24T14:41:06.146857Z","shell.execute_reply":"2023-02-24T14:42:31.092069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's perform some additional EDA with the cleaned text","metadata":{}},{"cell_type":"code","source":"# Get 50 most frequent words in the training set\nfrequent_words = pd.Series(' '.join(df_train.clean_text).split()).value_counts()[:50]\nfrequent_words.plot.bar(figsize=(15,4))\nplt.title('50 most frequent terms in the training set', fontweight=\"bold\")\nplt.xlabel('Tokens')\nplt.ylabel('Frequency')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:42:31.094737Z","iopub.execute_input":"2023-02-24T14:42:31.095206Z","iopub.status.idle":"2023-02-24T14:43:07.185498Z","shell.execute_reply.started":"2023-02-24T14:42:31.095156Z","shell.execute_reply":"2023-02-24T14:43:07.184333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# I found that the variable 'frequent_words' takes up about 6 Gb of RAM! Better clean it up!\nimport gc\n\ndel frequent_words\n\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:43:07.187023Z","iopub.execute_input":"2023-02-24T14:43:07.187339Z","iopub.status.idle":"2023-02-24T14:43:10.269148Z","shell.execute_reply.started":"2023-02-24T14:43:07.187302Z","shell.execute_reply":"2023-02-24T14:43:10.267193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the number of tokens in cleaned reviews\nplt.figure(figsize=(15,4))\nplt.hist(df_train['clean_text'].apply(lambda x:len(x.split())), bins=50, edgecolor='white')\nplt.xlabel('Number of tokens', fontsize=12)\nplt.ylabel('Number of reviews',fontsize=12)\nplt.title('Distribution of token counts in reviews', fontsize=14)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:43:10.272102Z","iopub.execute_input":"2023-02-24T14:43:10.273869Z","iopub.status.idle":"2023-02-24T14:43:17.007469Z","shell.execute_reply.started":"2023-02-24T14:43:10.273748Z","shell.execute_reply":"2023-02-24T14:43:17.006270Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Filter rows with less than 250 tokens\nless_than_250 = df_train['clean_text'].apply(lambda x: len(x.split())) < 250  # Check by getting True and False values\n\n# Print percentage\nprint(f\"{(sum(less_than_250) / len(df_train) * 100):.2f}% of rows have less than 250 tokens.\")","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:43:17.009507Z","iopub.execute_input":"2023-02-24T14:43:17.009968Z","iopub.status.idle":"2023-02-24T14:43:22.490452Z","shell.execute_reply.started":"2023-02-24T14:43:17.009913Z","shell.execute_reply":"2023-02-24T14:43:22.489316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"What does this mean? As visible in the token distribution graph, most reviews are not very long, although a few reach a length greater than 1500 tokens.\n\nLet's get our final processed training set:","metadata":{}},{"cell_type":"code","source":"# Remove original text column\ndf_train = df_train.drop(columns=['review_text'])\n\n# Get few entries to check operation\ndf_train.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:43:22.495649Z","iopub.execute_input":"2023-02-24T14:43:22.496257Z","iopub.status.idle":"2023-02-24T14:43:22.989067Z","shell.execute_reply.started":"2023-02-24T14:43:22.496210Z","shell.execute_reply":"2023-02-24T14:43:22.987871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Test Set\n\nImporting and processing the text data as did in the training set","metadata":{}},{"cell_type":"code","source":"# Import training data\ndf_test = pd.read_csv('/kaggle/input/goodreads-books-reviews-290312/goodreads_test.csv')\n\n# Print a random row\ndf_test.sample()","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:43:22.990793Z","iopub.execute_input":"2023-02-24T14:43:22.991630Z","iopub.status.idle":"2023-02-24T14:43:43.257320Z","shell.execute_reply.started":"2023-02-24T14:43:22.991568Z","shell.execute_reply":"2023-02-24T14:43:43.255951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\n# Remove unceccessary columns\ndf_test = df_test.drop(columns=['user_id','book_id','date_added','date_updated',\n                                  'read_at','started_at','n_votes','n_comments'])\n\n# Preprocess training data adding new column\ndf_test[\"clean_text\"] = df_test[\"review_text\"].apply(preprocess_review)\n\n# Remove original text column\ndf_test = df_test.drop(columns=['review_text'])\n\n# Get few entries to check cleaning operation\ndf_test.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:43:43.258742Z","iopub.execute_input":"2023-02-24T14:43:43.259075Z","iopub.status.idle":"2023-02-24T14:44:28.001663Z","shell.execute_reply.started":"2023-02-24T14:43:43.259038Z","shell.execute_reply":"2023-02-24T14:44:28.000522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Split the data","metadata":{}},{"cell_type":"code","source":"# Create X, y arrays\nfrom sklearn.model_selection import train_test_split\n\n# Training data\nX_train = df_train[\"clean_text\"].values\ny_train = df_train[\"rating\"].values\n\n# Split into train/test sets\nX_tr, X_va, y_tr, y_va = train_test_split(X_train, y_train, test_size=0.2, random_state=0)\n\n# Test data\nX_te = df_test[\"clean_text\"].values\n\nprint(\"Training data:\", X_tr.shape, y_tr.shape)\nprint(\"Validation data:\", X_va.shape, y_va.shape)\nprint(\"Test data:\", X_te.shape)","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:44:28.003557Z","iopub.execute_input":"2023-02-24T14:44:28.003907Z","iopub.status.idle":"2023-02-24T14:44:28.208281Z","shell.execute_reply.started":"2023-02-24T14:44:28.003851Z","shell.execute_reply":"2023-02-24T14:44:28.207578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model Part\n\nGiven the dataset's size and my preference for using a Neural Network model, it's certainly worthwhile to employ an accelerator. Below are the default setup settings for enabling the TPU (Tensor Processing Unit).","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport tensorflow as tf\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    \n    strategy = tf.distribute.experimental.TPUStrategy\nexcept ValueError:\n    strategy = tf.distribute.get_strategy() \n    print('Number of replicas:', strategy.num_replicas_in_sync) ","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:44:28.209804Z","iopub.execute_input":"2023-02-24T14:44:28.210359Z","iopub.status.idle":"2023-02-24T14:44:38.579116Z","shell.execute_reply.started":"2023-02-24T14:44:28.210314Z","shell.execute_reply":"2023-02-24T14:44:38.577957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() # TPU detection\nexcept ValueError:\n    tpu = None\n    gpus = tf.config.experimental.list_logical_devices(\"GPU\")\n    \nif tpu:\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu,) \n    print('Running on TPU ', tpu.cluster_spec().as_dict()['worker'])\nelif len(gpus) > 1:\n    strategy = tf.distribute.MirroredStrategy([gpu.name for gpu in gpus])\n    print('Running on multiple GPUs ', [gpu.name for gpu in gpus])\nelif len(gpus) == 1:\n    strategy = tf.distribute.get_strategy() \n    print('Running on single GPU ', gpus[0].name)\nelse:\n    strategy = tf.distribute.get_strategy() \n    print('Running on CPU')\nprint(\"Number of accelerators: \", strategy.num_replicas_in_sync)","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:44:38.581028Z","iopub.execute_input":"2023-02-24T14:44:38.581359Z","iopub.status.idle":"2023-02-24T14:44:43.172990Z","shell.execute_reply.started":"2023-02-24T14:44:38.581319Z","shell.execute_reply":"2023-02-24T14:44:43.170469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, this code uses the Keras module from TensorFlow to preprocess text data. Specifically, it uses the 'Tokenizer' function to convert the text into a sequence of integers, and the 'pad_sequences' function to ensure that each sequence has the same length.","metadata":{}},{"cell_type":"code","source":"%%time\n\nfrom tensorflow.keras.preprocessing.text import Tokenizer\nfrom tensorflow.keras.preprocessing.sequence import pad_sequences\n\n# Tokenize and pad the sequences\ntokenizer = Tokenizer(num_words=20000)\ntokenizer.fit_on_texts(X_tr)\n\nmax_seq_length = 250\n\nX_tr_seq = tokenizer.texts_to_sequences(X_tr)\nX_tr_seq = pad_sequences(X_tr_seq, maxlen=max_seq_length)\n\nX_va_seq = tokenizer.texts_to_sequences(X_va)\nX_va_seq = pad_sequences(X_va_seq, maxlen=max_seq_length)\n\nX_te_seq = tokenizer.texts_to_sequences(X_te)\nX_te_seq = pad_sequences(X_te_seq, maxlen=max_seq_length)","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:44:43.176992Z","iopub.execute_input":"2023-02-24T14:44:43.177669Z","iopub.status.idle":"2023-02-24T14:48:02.450633Z","shell.execute_reply.started":"2023-02-24T14:44:43.177609Z","shell.execute_reply":"2023-02-24T14:48:02.450019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's build the model. There are various approaches available, but I decided to experiment with a combination of several different layers, including a convolutional layer and a bidirectional LTSM layers. Probably these ones may be responsible for slowing down the training time more than the other layers. Anyway, let's evaluate it afterwards:","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Embedding, Conv1D, MaxPooling1D, LSTM, Bidirectional, Dense, BatchNormalization, Dropout\nfrom tensorflow.keras.optimizers import Adam\n\nwith strategy.scope():\n    model = Sequential()\n    model.add(Embedding(input_dim=20000, output_dim=128, input_length=250))\n\n    model.add(Conv1D(filters=64, kernel_size=3, activation='relu'))\n    model.add(MaxPooling1D(pool_size=2))\n\n    model.add(Bidirectional(LSTM(units=64, return_sequences=True)))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.2))\n\n    model.add(Bidirectional(LSTM(units=128)))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.2))\n\n    model.add(Dense(units=32, activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.2))\n\n    model.add(Dense(units=6, activation='softmax'))\n\n    model.compile(optimizer=Adam(learning_rate = 1e-4),\n                  loss = 'sparse_categorical_crossentropy',\n                  metrics=['acc']\n                 )\n\nmodel.summary()\n","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:48:02.452012Z","iopub.execute_input":"2023-02-24T14:48:02.452271Z","iopub.status.idle":"2023-02-24T14:48:05.100642Z","shell.execute_reply.started":"2023-02-24T14:48:02.452240Z","shell.execute_reply":"2023-02-24T14:48:05.099682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.callbacks import EarlyStopping\n\n# Creates 'EarlyStopping' callback\nearlystopping_cb = EarlyStopping(patience=4, restore_best_weights=True)","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:48:05.102893Z","iopub.execute_input":"2023-02-24T14:48:05.103688Z","iopub.status.idle":"2023-02-24T14:48:05.108955Z","shell.execute_reply.started":"2023-02-24T14:48:05.103643Z","shell.execute_reply":"2023-02-24T14:48:05.108102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nhistory = model.fit(X_tr_seq,\n                    y_tr,\n                    validation_data=(X_va_seq, y_va),\n                    callbacks=[earlystopping_cb],\n                    batch_size=64,\n                    epochs=15,\n                    verbose=1,\n                   )","metadata":{"execution":{"iopub.status.busy":"2023-02-24T14:48:05.110646Z","iopub.execute_input":"2023-02-24T14:48:05.110920Z","iopub.status.idle":"2023-02-24T15:20:05.835738Z","shell.execute_reply.started":"2023-02-24T14:48:05.110889Z","shell.execute_reply":"2023-02-24T15:20:05.834185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Indeed, not as fast as others NN configuration out there. Moving on, it's always a good practice to evaluate the model by seeing the training/validation curves as done below:","metadata":{}},{"cell_type":"code","source":"import numpy as np\n\n# Create two plots: one for the loss value, one for the accuracy\nfig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2, figsize=(20, 6))\n\nplt.suptitle('Model training/validation curves', size=15)\n\n# Plot loss values\nax1.plot(history.history[\"loss\"], label=\"train loss\")\nax1.plot(history.history[\"val_loss\"], label=\"val loss\")\nax1.set_title(\n    \"Validation loss {:.3f} (mean last 3)\".format(\n        np.mean(history.history[\"val_loss\"][-3:])  # last three values\n    )\n)\nax1.set_xlabel(\"epoch\")\nax1.set_ylabel(\"loss value\")\nax1.legend()\n\n# Plot accuracy values\nax2.plot(history.history[\"acc\"], label=\"train acc\")\nax2.plot(history.history[\"val_acc\"], label=\"val acc\")\nax2.set_title(\n    \"Validation accuracy {:.3f} (mean last 3)\".format(\n        np.mean(history.history[\"val_acc\"][-3:])  # last three values\n    )\n)\nax2.set_xlabel(\"epoch\")\nax2.set_ylabel(\"accuracy\")\nax2.legend()\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-24T15:20:05.837625Z","iopub.execute_input":"2023-02-24T15:20:05.837989Z","iopub.status.idle":"2023-02-24T15:20:06.716496Z","shell.execute_reply.started":"2023-02-24T15:20:05.837949Z","shell.execute_reply":"2023-02-24T15:20:06.715589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we see that only after 2 epoch the curves begin to diverge, a sign of overfitting. We can get the results on the validation set and plot the confusion matrices to further see the performance of the model","metadata":{}},{"cell_type":"code","source":"test_loss, test_acc = model.evaluate(X_va_seq,y_va)\n\nprint('Validation loss:', test_loss)\nprint('Validation accuracy:', test_acc)","metadata":{"execution":{"iopub.status.busy":"2023-02-24T15:20:06.717888Z","iopub.execute_input":"2023-02-24T15:20:06.718163Z","iopub.status.idle":"2023-02-24T15:21:30.171116Z","shell.execute_reply.started":"2023-02-24T15:20:06.718131Z","shell.execute_reply":"2023-02-24T15:21:30.170176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compute validation set predictions\npred = [np.argmax(i) for i in model.predict(X_va_seq)]","metadata":{"execution":{"iopub.status.busy":"2023-02-24T15:21:30.172282Z","iopub.execute_input":"2023-02-24T15:21:30.174011Z","iopub.status.idle":"2023-02-24T15:21:52.384177Z","shell.execute_reply.started":"2023-02-24T15:21:30.173965Z","shell.execute_reply":"2023-02-24T15:21:52.382937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n\n# Compute confusion matrix for raw counts\ncm_raw = confusion_matrix(y_va, pred)\n\n# Compute confusion matrix for normalized ratios\ncm_norm = confusion_matrix(y_va, pred, normalize='true')\n\n# Create confusion matrix plots\ndisp_raw = ConfusionMatrixDisplay(confusion_matrix=cm_raw)\ndisp_norm = ConfusionMatrixDisplay(confusion_matrix=cm_norm)\n\n# Plot confusion matrices side by side\nfig, axs = plt.subplots(1, 2, figsize=(20, 8), gridspec_kw={\"width_ratios\": [10, 10], \"wspace\": 0.1}, subplot_kw={\"aspect\":\"equal\"})\nplt.suptitle('Validation set results', size = 15)\ndisp_raw.plot(ax=axs[0])\naxs[0].set_title('Confusion Matrix (counts)')\ndisp_norm.plot(ax=axs[1])\naxs[1].set_title('Confusion Matrix (ratios)')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-24T15:21:52.386688Z","iopub.execute_input":"2023-02-24T15:21:52.387534Z","iopub.status.idle":"2023-02-24T15:21:54.115774Z","shell.execute_reply.started":"2023-02-24T15:21:52.387488Z","shell.execute_reply":"2023-02-24T15:21:54.114708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Based on the confusion matrices in the validation set, we can see that the model performed better with 4 and 5-star ratings. This is not surprising given that these ratings had the largest number of samples available for the training.\n\nConcerning the errors, overall the model tends to predict ratings that are close to each other, for instance often mistakenly predicting a rating of 2 for a rating of 3, or vice versa.","metadata":{}},{"cell_type":"markdown","source":"### Make predictions","metadata":{}},{"cell_type":"code","source":"# Compute test set predictions\npredictions = [np.argmax(i) for i in model.predict(X_te_seq)]\n    \n# Create a new DataFrame to merge review ids and the model predictions\nsubmission = pd.DataFrame({'review_id': df_test.review_id, 'rating': predictions})\n\n# Check few random entries\nsubmission.sample(10)","metadata":{"execution":{"iopub.status.busy":"2023-02-24T15:21:54.117292Z","iopub.execute_input":"2023-02-24T15:21:54.117615Z","iopub.status.idle":"2023-02-24T15:23:20.634457Z","shell.execute_reply.started":"2023-02-24T15:21:54.117577Z","shell.execute_reply":"2023-02-24T15:23:20.633448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check target balance\nsubmission['rating'].value_counts().sort_index().plot.bar(figsize=(10,5))\nplt.title('Rating distribution of the predictions in the test set', fontweight=\"bold\")\nplt.xlabel('Ratings')\nplt.ylabel('Number of reviews')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-02-24T15:23:20.637986Z","iopub.execute_input":"2023-02-24T15:23:20.638930Z","iopub.status.idle":"2023-02-24T15:23:21.023547Z","shell.execute_reply.started":"2023-02-24T15:23:20.638867Z","shell.execute_reply":"2023-02-24T15:23:21.022488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The target distribution in the test set is similar to the one in training set. ","metadata":{}},{"cell_type":"code","source":"# Export predictions\nsubmission.to_csv(\"submission.csv\", index=None)","metadata":{"execution":{"iopub.status.busy":"2023-02-24T15:23:21.024919Z","iopub.execute_input":"2023-02-24T15:23:21.025204Z","iopub.status.idle":"2023-02-24T15:23:21.840523Z","shell.execute_reply.started":"2023-02-24T15:23:21.025170Z","shell.execute_reply":"2023-02-24T15:23:21.839509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Final score of this notebook: ~0.55","metadata":{}}]}