` tokens. We also see that a large swath of the resulting tensor is just 0s for padding.\n\nLastly, we can use the `.get_vocabulary()` method on the vectorization layer to look up a particular index in the dictionary.","metadata":{}},{"cell_type":"markdown","source":"# Simple Model\n\nLet's start with a simple model that just uses an Embedding layer, followed by an LSTM, followed by a single-neuron Dense layer since this is a binary-classification problem.\n\n\n Embedding Layer\n It's not enough to just convert the input strings into integer tokens. Passing these integer tokens straight into an LSTM would not have enough complexity to learn any patterns. We need to also use an Embedding layer to turn the integer indexes into dense vectors that contain word representation. These word representations are learned based on the training data and allow a richer representation for words than a simple number could ever contain.\n
\n\nNote: If you enable `train_simple_model`, the following code will run and this simple model will be trained and the loss curves plotted. I've disabled it in the main version of this notebook as it wasn't the best performing.\n","metadata":{}},{"cell_type":"code","source":"# Create basic model\n\ndef get_model():\n \n model = keras.Sequential([\n text_vec_layer,\n # To support masking, you need to use VOCAB_SIZE+1 because that takes a token!\n keras.layers.Embedding(VOCAB_SIZE + 1, EMBEDDING_SIZE, mask_zero=True, ), # input dim., output dim., turn on masking\n keras.layers.LSTM(128), # Try simple LSTM first\n keras.layers.Dense(1, activation='sigmoid') # Binary classification problem\n ])\n \n model.compile(loss='binary_crossentropy',\n optimizer='nadam',\n metrics=['accuracy'])\n \n return model\n\nif train_simple_model:\n model = get_model()\n model.summary()","metadata":{"execution":{"iopub.status.busy":"2023-09-18T19:40:37.197238Z","iopub.execute_input":"2023-09-18T19:40:37.197528Z","iopub.status.idle":"2023-09-18T19:40:37.203901Z","shell.execute_reply.started":"2023-09-18T19:40:37.197502Z","shell.execute_reply":"2023-09-18T19:40:37.202864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Try to train\nif train_simple_model:\n epochs = 25\n history = model.fit(train_ds,\n validation_data=val_ds,\n epochs=epochs,\n verbose=1)\n\n# This simple model will only reach about an 75% val accuracy, as it overfits to the training set","metadata":{"execution":{"iopub.status.busy":"2023-09-18T19:40:37.206471Z","iopub.execute_input":"2023-09-18T19:40:37.207129Z","iopub.status.idle":"2023-09-18T19:40:37.215010Z","shell.execute_reply.started":"2023-09-18T19:40:37.207090Z","shell.execute_reply":"2023-09-18T19:40:37.214064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot the loss and accuracy curves for training and validation \nif train_simple_model:\n fig, ax = plt.subplots(2,1)\n ax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\n ax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\n legend = ax[0].legend(loc='best', shadow=True)\n\n ax[1].plot(history.history['accuracy'], color='b', label=\"Training accuracy\")\n ax[1].plot(history.history['val_accuracy'], color='r',label=\"Validation accuracy\")\n legend = ax[1].legend(loc='best', shadow=True)","metadata":{"execution":{"iopub.status.busy":"2023-09-18T19:40:37.216095Z","iopub.execute_input":"2023-09-18T19:40:37.216360Z","iopub.status.idle":"2023-09-18T19:40:37.229585Z","shell.execute_reply.started":"2023-09-18T19:40:37.216338Z","shell.execute_reply":"2023-09-18T19:40:37.228825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Simple Model overfits to the training data and is unable to achieve better than about 75% on the validation accuracy.","metadata":{}},{"cell_type":"markdown","source":"# Model with some Regularization techniques\n\nThe previous Simple Model is powerful enough to overfit to 97% accuracy on the training set, but unable to achieve better than about 75% validation accuracy, meaning we are overfitting.\n\nLet's try to add some regularizion techniques such as **dropout** to the model so that we can perform better generalization. Here we have added dropout directly to each of the LSTMs as well as more dropout before going into the Dense layer.\n\n\n Bi-Directional Layer\n I also added a bi-directional layer to the model since a classifier has access to the whole sentence before making a determination. Oftentimes a bi-directional layer can improve performance for sentiment analysis (like essentially this case). For example, if the LSTM sees the word \"Apple\" in the sentence, it can use context from the remaining parts of the sentence to know if we're talking about the tech company, or the fruit. However, this didn't add any benefit, and on hindsight it's not shocking that this didn't help greatly. I mean, we're able to overfit the training data already, which shows that with padding masking enabled, the single-directional LSTM was plenty powerful enough.\n
","metadata":{}},{"cell_type":"code","source":"# The model is clearly overfitting. Let's try some regularization techniques.\n\n# But also, it makes sense that a classifier should have access to the whole sentence\n# before it has to start forming its guesses, so let's make this bi-directional as well.\ndef get_model_reg():\n \n model = keras.Sequential([\n text_vec_layer,\n # To support masking, you need to use VOCAB_SIZE+1 because that takes a token!\n keras.layers.Embedding(VOCAB_SIZE + 1, EMBEDDING_SIZE, mask_zero=True, ), # input dim., output dim., turn on masking\n # Make bi-directional, and add dropout too\n keras.layers.Bidirectional(keras.layers.LSTM(128, dropout=0.3)), # Try simple LSTM first\n keras.layers.Dropout(0.4),\n keras.layers.Dense(1, activation='sigmoid') # Binary classification problem\n ])\n \n model.compile(loss='binary_crossentropy',\n optimizer='nadam',\n metrics=['accuracy'])\n \n return model\n\nif train_regularized_simple_model:\n model = get_model_reg()\n model.summary()","metadata":{"execution":{"iopub.status.busy":"2023-09-18T19:40:37.230751Z","iopub.execute_input":"2023-09-18T19:40:37.231606Z","iopub.status.idle":"2023-09-18T19:40:37.241594Z","shell.execute_reply.started":"2023-09-18T19:40:37.231568Z","shell.execute_reply":"2023-09-18T19:40:37.240713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Try to train\nif train_regularized_simple_model:\n epochs = 25\n history = model.fit(train_ds,\n validation_data=val_ds,\n epochs=epochs,\n verbose=1)","metadata":{"execution":{"iopub.status.busy":"2023-09-18T19:40:37.242765Z","iopub.execute_input":"2023-09-18T19:40:37.243043Z","iopub.status.idle":"2023-09-18T19:40:37.251639Z","shell.execute_reply.started":"2023-09-18T19:40:37.243019Z","shell.execute_reply":"2023-09-18T19:40:37.250861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if train_regularized_simple_model:\n # Plot the loss and accuracy curves for training and validation \n fig, ax = plt.subplots(2,1)\n ax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\n ax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\n legend = ax[0].legend(loc='best', shadow=True)\n\n ax[1].plot(history.history['accuracy'], color='b', label=\"Training accuracy\")\n ax[1].plot(history.history['val_accuracy'], color='r',label=\"Validation accuracy\")\n legend = ax[1].legend(loc='best', shadow=True)","metadata":{"execution":{"iopub.status.busy":"2023-09-18T19:40:37.252678Z","iopub.execute_input":"2023-09-18T19:40:37.253045Z","iopub.status.idle":"2023-09-18T19:40:37.265714Z","shell.execute_reply.started":"2023-09-18T19:40:37.253009Z","shell.execute_reply":"2023-09-18T19:40:37.264850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Sadly, this model doesn't perform much better. It does end up performing about 1% better on the validation set, so it's slightly more regularized, but still not good enough. \n\n\n Stemming or Stopwords Removal?\n At this point I stopped to think some more about if we should be doing more text preprocessing before training these models. In the end, I didn't think stemming would provide much benefit for the work it would take, and it's possible that removing stopwords can even break sentiment analysis (a higher-level class for this problem) by removing negating words like \"not\" or words that change subject/object assignment, such as removing \"to the.\"\n
\n\nIn the end, I decided that I might end up using a pretrained language model, which would already embed lots of contextual meaning into the words. For example, \"right\" could mean \"right foot\" or \"correct\" or a \"right of passage.\" So, if we end up fine-tuning a model on the full text, then more powerful models like BERT or Google's Universal Sentence Encoder will have been trained by keeping stop words in the corpus, and so removing them would create a data mismatch.","metadata":{}},{"cell_type":"markdown","source":"# Universal Sentence Encoder (Transformer based)\n\nLet's test that hypothesis. Let's see if a pretrained language model, which already has a lot of contextual meanings baked into the word embeddings, can do better. \n\n\n A transformer is a much more powerful model, and we're already overfitting, but the hope is that the extra word context can make it better at generalizing.\n
\n\nWe'll use the Universal Sentence Encoder, by Google, a transformer-based architecture based on [this paper](https://arxiv.org/abs/1803.11175) by Cer et al., 2018. TensorFlow Hub makes it easy to download and use it directly as a layer in our model. Note that for this model, we can also fine tune it (not true for all models). If we couldn't fine tune it, we could still use it and only train our following dense layers. If you enable the following code, you'll see that there are about 256M parameters to fine-tune!","metadata":{}},{"cell_type":"code","source":"# Let's try a more powerful model using a pretrained language model. Already overfitting but want to see.\n# This model is based on the Universal Sentence Encoder, a model introduced in 2018 by Google and based \n# on the transformer architecture.\ndef get_USE_model():\n\n import os\n import tensorflow_hub as hub\n\n os.environ[\"TFHUB_CACHE_DIR\"] = \"my_tfhub_cache\" # Cache the download since it's 1 GB. Doesn't help us too much since this is a VM, but still best practice!\n\n model = keras.Sequential([\n hub.KerasLayer(\"https://tfhub.dev/google/universal-sentence-encoder/4\", # model to download, and documentation\n trainable=True, # ability to fine-tune it\n dtype=tf.string,\n input_shape=[]),\n keras.layers.Dense(64, activation='relu'),\n keras.layers.Dense(1, activation='sigmoid')\n ])\n \n model.compile(loss='binary_crossentropy',\n optimizer='nadam',\n metrics=['accuracy']) # Same as all the previous models\n \n return model\n\n\nif train_USE_model:\n model = get_USE_model()\n model.summary()","metadata":{"execution":{"iopub.status.busy":"2023-09-18T19:40:37.266834Z","iopub.execute_input":"2023-09-18T19:40:37.267287Z","iopub.status.idle":"2023-09-18T19:40:37.277960Z","shell.execute_reply.started":"2023-09-18T19:40:37.267240Z","shell.execute_reply":"2023-09-18T19:40:37.276357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Try to train\nif train_USE_model:\n epochs = 25\n history = model.fit(train_ds,\n validation_data=val_ds,\n epochs=epochs,\n verbose=1)","metadata":{"execution":{"iopub.status.busy":"2023-09-18T19:40:37.279780Z","iopub.execute_input":"2023-09-18T19:40:37.280787Z","iopub.status.idle":"2023-09-18T19:40:37.287024Z","shell.execute_reply.started":"2023-09-18T19:40:37.280756Z","shell.execute_reply":"2023-09-18T19:40:37.286128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if train_USE_model:\n # Plot the loss and accuracy curves for training and validation \n fig, ax = plt.subplots(2,1)\n ax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\n ax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\n legend = ax[0].legend(loc='best', shadow=True)\n\n ax[1].plot(history.history['accuracy'], color='b', label=\"Training accuracy\")\n ax[1].plot(history.history['val_accuracy'], color='r',label=\"Validation accuracy\")\n legend = ax[1].legend(loc='best', shadow=True)","metadata":{"execution":{"iopub.status.busy":"2023-09-18T19:40:37.288138Z","iopub.execute_input":"2023-09-18T19:40:37.288789Z","iopub.status.idle":"2023-09-18T19:40:37.303776Z","shell.execute_reply.started":"2023-09-18T19:40:37.288760Z","shell.execute_reply":"2023-09-18T19:40:37.302592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Unfortunately, the Universal Sentence Encoder just overfits even more and cannot generalize any better, despite having pre-trained word embeddings.","metadata":{}},{"cell_type":"markdown","source":"# Global Average Pooling Model\n\nWe've seen that our simple models are overfitting, and a more powerful Transformer-based model is overfitting as well.\n\nCan we find a model that is a little bit *less* powerful so that it doesn't overfit, but is still able to learn?\n\nAs it turns out, Global Average Pooling is another popular technique for classification problems. How it works is it returns a fixed-length output vector for each example by averaging over the feature dimension. For example, if the dimensionality of the data going *into* that layer were `(batch, sequence, embedding)`, like `(None, 70, 128)` in our case, then the tensors coming *out of* the Global Average Pooling layers will just be `(batch, embedding)`, or in our case, `(None, 128)`. All of the embeddings for all 70 tokens in the input string have been averaged together! For a classification problem like sentiment analysis, it's kindof like treating it as a bag-of-words model.","metadata":{}},{"cell_type":"code","source":"# Still overfitting. Let's try a Global Average Pooling model, another popular architecture\n# for classification problems. Averaging out all the word embeddings, similar to a bag of words model.\ndef get_model_avgpool():\n \n model = keras.Sequential([\n text_vec_layer,\n # To support masking, you need to use VOCAB_SIZE+1 because that takes a token!\n keras.layers.Embedding(VOCAB_SIZE + 1, EMBEDDING_SIZE, mask_zero=True, ), # input dim., output dim., turn on masking\n keras.layers.Dropout(0.25),\n keras.layers.GlobalAveragePooling1D(),\n keras.layers.Dropout(0.25),\n keras.layers.Dense(1, activation='sigmoid') # Binary classification problem\n ])\n \n model.compile(loss='binary_crossentropy',\n optimizer='nadam',\n metrics=['accuracy'])\n \n return model\n\nif train_average_pooling_model:\n model = get_model_avgpool()\n model.summary()","metadata":{"execution":{"iopub.status.busy":"2023-09-18T19:41:16.184106Z","iopub.execute_input":"2023-09-18T19:41:16.184498Z","iopub.status.idle":"2023-09-18T19:41:16.374762Z","shell.execute_reply.started":"2023-09-18T19:41:16.184466Z","shell.execute_reply":"2023-09-18T19:41:16.373653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Try to train\nif train_average_pooling_model:\n epochs = 25\n history = model.fit(train_ds,\n validation_data=val_ds,\n epochs=epochs,\n verbose=1)","metadata":{"execution":{"iopub.status.busy":"2023-09-18T19:47:04.868405Z","iopub.execute_input":"2023-09-18T19:47:04.868837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if train_average_pooling_model:\n # Plot the loss and accuracy curves for training and validation \n fig, ax = plt.subplots(2,1)\n ax[0].plot(history.history['loss'], color='b', label=\"Training loss\")\n ax[0].plot(history.history['val_loss'], color='r', label=\"validation loss\",axes =ax[0])\n legend = ax[0].legend(loc='best', shadow=True)\n\n ax[1].plot(history.history['accuracy'], color='b', label=\"Training accuracy\")\n ax[1].plot(history.history['val_accuracy'], color='r',label=\"Validation accuracy\")\n legend = ax[1].legend(loc='best', shadow=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This model using global average pooling generalizes much better than the other models, reaching about 81.5% classification accuracy on the validation set.\n\nβ But remember earlier when we said we had an unbalanced dataset and that therefore we couldn't trust accuracy alone?\n\nThis is where **F1 Scores** come into play. Let's talk about that now.","metadata":{}},{"cell_type":"markdown","source":"---\n\n# Precision and Recall\n\nBefore we can talk about F1 Scores, we need to understand two other terms, **Precision** and **Recall**.\n\n### A Thought Experiment\n\nIn order to gain an intuitive understanding of these, let's escape the world of Machine Learning for a second and pretend for the moment that you are writing the firmware for a biometric safe. Your code will determine if the safe opens based on the user scanning a fingerprint on a sensor on the front of the safe. We don't know what's in the safe β could be cash and diamonds, could be guns, could be keys to a crypto wallet. But it's your job to decide, *what threshold should you set to determine if a user is a match or not?* After all, even if it's the correct person, a scan will usually not get every single measured minutiae on the fingerprint to match. So you will need to say, if we get 25% match, is that good enough? Or should it be 75%? 95 or 99%? π€·ββοΈ\n\n\n The Precision-Recall Tradeoff\n As you start to play with the threshold number for your biometric safe, you will quickly realize a tradeoff occurs between false positives and false negatives. For example, let's say your safe is a handgun safe. If you set a 10% threshold, then when the correct gun owner scans their finger, they will definitely get in. But now it's more likely that the wrong person could gain access to the safe, for example, children in the house. So you decide to set a super high threshold, say 99%. Now the children are far less likely to gain access, but you increase the chance that the gun owner can't open the safe during a home invasion due to their sweaty fingers.. also not a good situation!\n
\n\nThis fundamental tradeoff is known as the Precision-Recall Tradeoff. Whereas the **Precision, P** is a way of judging the accuracy of the true positives predicted (what percent of predicted positives were actually positive?), the **Recall, R** is a measure of sensitivity (what percentage of actual positives in the set were correctly detected?). \n\nBack to our biometric safe example, setting a high threshold of 99% is an example of having high Precision (of all the fingerprints that opened the safe, nearly all of them will be true positives) at the cost of having low Recall (many valid fingerprints will be wrongly rejected). \n\n# F1 Scores\n\nTo compare classifiers against each other, it's helpful to have a single metric instead of two. For this reason, the **F1 Score** was invented to combine these two metrics in a way that makes sense. The F1 Score is just the [harmonic mean](https://en.wikipedia.org/wiki/Harmonic_mean) of the two numbers, P and R.\n\nEven within using the F1 Score, there are a few different ways to do the calculation. There is both the **Macro F1 Score** and the **Micro F1 Score**. What's the difference? The Macro score computes the F1 Score for each class individually, and then takes the arithmetic mean of those. For example, it would mean computing an F1 Score on \"news\" alone, and then on \"clickbait\" alone, and then summing those F1 Scores and dividing by 2 (since in this case we only have two classes). On the other hand, the Micro score is computed by aggregating all the contributions (e.g. true positives, false positives, false negatives) across all the classes. In other words, the Macro score gives equal weights to each class, where the Micro scores gives equal weight to each sample.\n\n\n So which should we use? π€ Micro or Macro?\n
\n\nWell, in our case, the choice was made for us. The original competition decided to use the Micro F1 Score as their metric. **By using the same metric, I can compare my results to those on the leaderboard and make sure I'm in the ballpark.**\n\nBut say you weren't constrained by that.. then you pick a metric according to your objective for that particular problem. If you care about overall accuracy and no classes are more important than others, then you should select Macro, as it weighs classes equally. However, if a particular class is rare but very important, you would probably select Micro, as it weighs observations equally. This makes Micro also the natural choice when you have an imbalanced dataset, otherwise the score will be weighted unfairly towards whichever class had more samples. Also if the class labels are perfectly balanced, then both Micro and Macro averages will be identical.\n\nI believe it's for these reasons that the competition chose Micro F1-Score as its metric, given the imbalanced classes.\n\nLet's actually calculate the F1 Scores now. We'll let SciKit-Learn handle the actual computation for us, but first we'll need to get our predictions into the right format.","metadata":{}},{"cell_type":"code","source":"# The main metric we're actually competing on is Micro-averaged F1 score, so let's calculate that.\n# SciKit-Learn has it built in.\nfrom sklearn.metrics import f1_score\n\n# model.predict() for this dataset will return a (n x 1) array of predictions for the dataset\n# These are probabilities of which class they belong to. We can just convert them to 0 or 1\n# by snapping classes that are larger than 0.5 to a 1.\n\ny_hat = (model.predict(val_ds) > 0.5).astype(\"int32\") # Snap predictions to binary class labels\n\nprint(\"y_hat:\")\nprint(y_hat)\nprint(f\"y_hat.shape: {y_hat.shape}\")\n\nprint(\"\\n\\n15 Headlines, True labels, and Predictions: \")\nfor i, (title, label) in enumerate(val_ds.unbatch()):\n print(f\"idx: {i} title: {title.numpy()} label: {label.numpy()} pred: {y_hat[i, 0]}\")\n if i > 15:\n break\n\nprint(\"\\n\\n10 first headlines with accurate predictions:\")\ncnt = 0\nfor i, (title, label) in enumerate(val_ds.unbatch()):\n if label.numpy() == y_hat[i, 0]:\n print(f\"idx: {i} title: {title.numpy()} label: {label.numpy()} pred: {y_hat[i, 0]}\")\n cnt += 1\n if cnt > 10:\n break\n\nprint(\"\\n\\n10 first headlines with WRONG predictions:\")\ncnt = 0\nfor i, (title, label) in enumerate(val_ds.unbatch()):\n if label.numpy() != y_hat[i, 0]:\n print(f\"idx: {i} title: {title.numpy()} label: {label.numpy()} pred: {y_hat[i, 0]}\")\n cnt += 1\n if cnt > 10:\n break\n\nprint(\"\\n\\n10 first News:\")\ncnt = 0\nfor i, (title, label) in enumerate(val_ds.unbatch()):\n if label.numpy() == 0:\n print(f\"idx: {i} title: {title.numpy()} label: {label.numpy()} pred: {y_hat[i, 0]}\")\n cnt += 1\n if cnt > 10:\n break\n\nprint(\"\\n\\n10 first Clickbaits:\")\ncnt = 0\nfor i, (title, label) in enumerate(val_ds.unbatch()):\n if label.numpy() == 1:\n print(f\"idx: {i} title: {title.numpy()} label: {label.numpy()} pred: {y_hat[i, 0]}\")\n cnt += 1\n if cnt > 10:\n break\n\n\n\nprint(\"\\n\\nBuilding y_true (ground truth)..\")\ny_true = np.zeros((y_hat.shape)) # Start by filling a zeroed array, of size (2011, 1)\n\n# Iterate over all the labels and fill y_true to match same shape as y_pred\n# Need to unbatch the samples to do one at a time\nfor i, (_, label) in enumerate(val_ds.unbatch()):\n y_true[i, 0] = label.numpy()\n \nprint(f\"y_true.shape: {y_true.shape}\")\n\n# And Finally!...\n# Calculate F1 score using Scikit-Learn\nf1_micro = f1_score(y_true, y_hat, average='micro')\nprint(f\"\\n\\nMicro-F1 Score: {f1_micro}\")","metadata":{"execution":{"iopub.status.busy":"2023-09-09T00:06:29.475028Z","iopub.execute_input":"2023-09-09T00:06:29.475746Z","iopub.status.idle":"2023-09-09T00:06:31.684643Z","shell.execute_reply.started":"2023-09-09T00:06:29.475710Z","shell.execute_reply":"2023-09-09T00:06:31.683647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n\n# Conclusion\n\nIf we run through the various models, we get these resulting Micro F1-Scores:\n\n| Model | Micro F1-Score |\n|-------|----------------|\n| LSTM w/Regularization | 0.7737 |\n| Universal Sentence Encoder | 0.7841 |\n| **Global Average Pooling** | **0.8125** |","metadata":{}},{"cell_type":"markdown","source":"The best-performing model uses Global Average Pooling, with a Micro F1-Score of 0.8125. This is actually pretty good, given that the winning entry for the competition scored a 0.8639.\n\n## Label Consistency Limits Accuracy\n\nYou might be wondering, why is it hard to do much better on this problem? \n\nI think the answer lies in the actual labels in the dataset. I think there are many inconsistencies in the labeling. Perhaps, for example, the dataset was labeled by many different people, all with varying opinions of what makes clickbait, clickbait? If that were the case, it will be very hard to build an accurate classifier if the opinions of data labelers are somewhat inconsistent, given the subjective nature of this problem. This isn't as hard-and-fast as labeling all bicycles in an image, for instance. So, given that \"clickbait\" doesn't even have a strict definition, this is probably about the best results one could get.\n\nFor example, \"12 Oven-Baked Foil Packet Dinners To Try\" was labeled as \"news\" even though I'd probably personally label any listicles as \"clickbait.\" Then again, listicles don't *have* to be clickbait, so long as they deliver the information that was promised in the headline.\n\n## Contrary Examples\n\nLet's take a look at some labeled examples that I disagree with (and again, this is now just *my* opinion!) to make my point:\n\nThe Top Beaches In The World, According To National Geographic\nThis is labeled as clickbait in the data. It might not exactly be newsworthy information, but I bet that article does exactly what it says on the tin.\n\nThis Is How Differently Priced Spanx Can Actually Make Your Body Look\nThis is labeled as news in the dataset. I'd say this is classic clickbait.\n\nNational Photographic Portrait Prize finalists share the stories behind the pictures\nThis sounds to me like a descriptive headline with no curiosity gap whatsoever, yet it's been labeled as clickbait. Perhaps the labelers aren't familiar with National Geographic?\n\nDude Gets Blazed With His Grandma for the First Time, Ends Up Talking About Porn\nApparently the labelers think a guy talking about porn while high with his grandma is news. I'd say clickbait. **However**, even though I doubt the newsworthiness of this one, I could understand not labeling it as clickbait, since the article could be exactly about what it says in the headline. \n\nI think this just further illustrates the subjectiveness and ambiguity of such a classifier. If the labelers are not able to label the data consistently, that is going to severely limit the accuracy of any machine learning model.\n\nGood luck using this model to build your *BaitBlock* Chrome extension. Might want to get a different dataset first though!\n\n## One last thing..\n\n\n If you've enjoyed following along with me or have learned something new, β please consider UPVOTING this notebook! πΌ It helps others discover my notebook and encourages me to spend my time writing more of these. Thank you! π\n
","metadata":{}},{"cell_type":"markdown","source":"π‘ **You might also enjoy these other notebooks of mine as well:**\n\n**Natural Language Processing**\n* [Generate Amazon Book Reviews with Transformers](https://www.kaggle.com/code/quackaddict7/generate-amazon-book-reviews-with-transformers/) \n* [Write Your Own CliffNotes (Book Text Summarizer)](https://www.kaggle.com/code/quackaddict7/write-your-own-cliffnotes-book-text-summarizer/) \n* [Answering Questions from Product Reviews](https://www.kaggle.com/code/quackaddict7/answering-questions-from-product-reviews/) \n* [Ingredient Standardization via Machine Translation](https://www.kaggle.com/code/quackaddict7/ingredient-standardization-via-machine-translation/) \n* [What Kaggle WON'T Tell You About Your Notebooks](https://www.kaggle.com/code/quackaddict7/what-kaggle-won-t-tell-you-about-your-notebooks/) \n\n**Computer Vision**\n* [Enhance, enhance, enhance! (image upscaling)](https://www.kaggle.com/code/quackaddict7/enhance-enhance-enhance-image-upscaling) \n* [Detecting Vehicles in Traffic (Object Detection)](https://www.kaggle.com/code/quackaddict7/object-detection-detecting-vehicles-in-traffic/) \n* [Legions of Lesions (Detecting Skin Cancer with Computer Vision)](https://www.kaggle.com/code/quackaddict7/legions-of-lesions-detecting-skin-cancer-with-cv/) \n* [Creating Synthetic Wildfire Images with Unreal Engine (Blog Post)](https://joelwigton.com/synthetic-data-for-machine-learning-with-unreal-engine) \n* [How I scored Top 5% on MNIST (without cheating!)](https://www.kaggle.com/code/quackaddict7/how-i-scored-top-5-on-mnist-without-cheating/) \n\n**Machine Learning**\n* [Detecting Android Malware from App Permissions](https://www.kaggle.com/code/quackaddict7/detecting-android-malware-from-app-permissions/) \n","metadata":{}}]}