{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Introdution\n\nOnline content publishers often use catchy headlines for their\narticles in order to attract users to their websites. These headlines, popularly known as clickbaits, exploit a user’s curiosity gap and lure them to click on links that often disappoint them. Existing methods for automatically detecting clickbaits rely on heavy feature engineering and domain knowledge. Here, my goal is to classification of clickbait news and non-clicbait news. I have fine-tune SOTA XLNet model for classification. I have used the amazing Transformers library by Hugging Face with pytorch. \n\n#### Dataset\n\nThe train1.csv collected from Abhijnan Chakraborty, Bhargavi Paranjape, Sourya Kakarla, and Niloy Ganguly. \"Stop Clickbait: Detecting and Preventing Clickbaits in Online News Media”. In Proceedings of the 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), San Fransisco, US, August 2016.[GitHub](https://github.com/bhargaviparanjape/clickbait/tree/master/dataset)\n\nIt has two columns first one contains headlines and the second one has numerical labels of clickbait in which 1 represents that it is clickbait and 0 represents that it is non-clickbait headline. The dataset contains total 32000 rows of which 50% are clickbait and other 50% are non-clickbait.\n\nThe train2.csv collected from [Clickbait news detection dataset](https://www.kaggle.com/c/clickbait-news-detection/data) from the Kaggle InClass Prediction Competition. The dataset contains title and text  of the news and label. \n\n#### XLNet Model\nXLNet is the latest and greatest model to emerge from the booming field of Natural Language Processing (NLP). The [XLNet paper](https://arxiv.org/pdf/1906.08237.pdf) combines recent advances in NLP with innovative choices in how the language modelling problem is approached. When trained on a very large NLP corpus, the model achieves state-of-the-art performance for the standard NLP tasks that comprise the GLUE benchmark. To learn more about the model see the [link.](https://towardsdatascience.com/what-is-xlnet-and-why-it-outperforms-bert-8d8fce710335) \n\n#### XLNet Fine-Tuning With PyTorch¶\nI'll use XLNet with the huggingface PyTorch library to quickly and efficiently fine-tune a model to get near state of the art performance in sentence classification. More broadly, I describe the practical application of transfer learning in NLP to create high performance models with minimal effort on a range of NLP tasks. Specifically, we will take the pre-trained BERT model, add an untrained layer of neurons on the end, and train the new model for our classification task.\n\n#### Reference\n- [zihangdai/xlnet](https://github.com/zihangdai/xlnet),\n- [Hugging Face](https://huggingface.co/transformers/model_doc/bert.html),\n- [papers with code](https://paperswithcode.com/task/clickbait-detection)\n- [Training Sentiment Model Using BERT-By Abhishek Thakur](https://www.youtube.com/watch?v=hinZO--TEk4&t=449s)\n- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/abs/1810.04805),\n- [The Illustrated BERT, ELMo, and co.](https://jalammar.github.io/illustrated-bert/),\n- [BERT Fine-Tuning Tutorial with PyTorch](https://mccormickml.com/2019/07/22/BERT-fine-tuning/),\n- [How to Fine-Tune BERT for Text Classification?](https://arxiv.org/pdf/1905.05583.pdf)\\,\n- [Huggingface Transformers](https://huggingface.co/transformers/),\n- [BERT Explained: State of the art language model for NLP](https://towardsdatascience.com/bert-explained-state-of-the-art-language-model-for-nlp-f8b21a9b6270)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Importing Packages","execution_count":null},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"%%time\n# https://www.kaggle.com/tunguz/melanoma-tsne-and-umap-embeddings-with-rapids/?\n\n# INSTALL RAPIDS OFFLINE (FROM KAGGLE DATASET). TAKES 1 MINUTE :-)\nimport sys\n!cp ../input/rapids/rapids.0.13.0 /opt/conda/envs/rapids.tar.gz\n!cd /opt/conda/envs/ && tar -xzvf rapids.tar.gz\nsys.path = [\"/opt/conda/envs/rapids/lib\"] + [\"/opt/conda/envs/rapids/lib/python3.6\"] + [\"/opt/conda/envs/rapids/lib/python3.6/site-packages\"] + sys.path\n!cp /opt/conda/envs/rapids/lib/libxgboost.so /opt/conda/lib/","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport time\nimport random\nimport numpy as np\nimport pandas as pd\nfrom contextlib import contextmanager\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\n# Thanks to @andradaolteanu for color_nude\n# Set Color Palettes for the notebook\ncolors_nude = ['#e0798c','#65365a']\nsns.palplot(sns.color_palette(colors_nude))\n# Set Style\nsns.set_style(\"whitegrid\")\nsns.despine(left=True, bottom=True)\n\nimport transformers\nimport torch\nimport torch.nn as nn\nfrom tqdm import tqdm\nfrom sklearn import model_selection\nfrom sklearn import preprocessing\nfrom sklearn import metrics\nfrom transformers import AdamW\nfrom transformers import get_linear_schedule_with_warmup\nimport warnings\nimport warnings as wrn\nwrn.filterwarnings('ignore', category = DeprecationWarning) \nwrn.filterwarnings('ignore', category = FutureWarning) \nwrn.filterwarnings('ignore', category = UserWarning) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Configuration","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"class config:\n    # maximum possible sequence length is 512.\n    MAX_LEN = 256\n    \n    # refer to the table here https://github.com/google-research/bert to adjust batch size to seq length\n    TRAIN_BATCH_SIZE = 16  \n    VALID_BATCH_SIZE = 8\n    SEED = 2020\n    \n    # number of epochs\n    EPOCHS = 3\n    # learning rate to tune xlnet\n    LRATE = 2e-5 \n    \n    # wait for several backward steps, then one optimization step, this allows to use larger batch size\n    # well explained here https://bit.ly/2MCMHUX\n    ACCUM_STEPS = 2\n    \n    # warmup helps to tackle instability in the initial phase of training with large learning rates. \n    # During warmup, learning rate is gradually increased from 0 to LRATE.\n    # WARMUP is a proportion of total weight updates for which warmup is done. By default, it's linear warmup\n    WARMUP = 5e-2\n    \n    # File Paths\n    MODEL_PATH = \"model.bin\"         \n    TRAINING_FILE_1 = \"../input/news-clickbait-dataset/train1.csv\"\n    TRAINING_FILE_2 = \"../input/news-clickbait-dataset/train2.csv\"\n    \n    # Loading xlnet tokenizer\n    print('Loading XLNet tokenizer...')\n    TOKENIZER = transformers.XLNetTokenizer.from_pretrained(\n        \"xlnet-base-cased\", \n        do_lower_case=True\n    )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper Functions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# nice way to report running times\n@contextmanager\ndef timer(name):\n    t0 = time.time()\n    yield\n    print(f'[{name}] done in {time.time() - t0:.0f} s')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# make results fully reproducible\ndef seed_everything(seed=123):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reading and Preparing the data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reading dataset\ntrain1 = pd.read_csv(config.TRAINING_FILE_1)\ndisplay(train1.sample(n=8))\nprint('shape of data', train1.shape)\nprint(' ')\nprint(train1['clickbait'].value_counts())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reading dataset\ntrain2 = pd.read_csv(config.TRAINING_FILE_2)\ndisplay(train2.sample(n=8))\nprint('shape of data', train2.shape)\nprint(' ')\nprint(train2['label'].value_counts())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# rename the column clickbait to label and title to headline\ntrain1.rename(columns={'clickbait': 'label'}, inplace=True)\ntrain2.rename(columns={'title': 'headline'}, inplace=True)\n\n# coverting categorical label to numerical.\ntrain2.label = train2.label.apply(lambda x: 1 if x == \"clickbait\" else 0)\n\n# concating train1 and train2 dataset\ntrain_df = pd.concat([train1, train2]).reset_index(drop=True)\n\n# Data Dresription\nprint('Shape of the final dataset', train_df.shape)\nprint(' ')\nprint('Missing values')\nprint(train_df.isnull().sum())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# example of news and clickbait\nprint('===== NON-CLICKBAIT =====')\nwith pd.option_context('display.max_colwidth', 100):\n    print(train_df[train_df['label']== 0]['headline'].sample(5))\nprint('\\n')\n\nprint('===== CLICKBAIT =====')\nwith pd.option_context('display.max_colwidth', 100):\n    print(train_df[train_df['label']== 1]['headline'].sample(5))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"total = len(train_df)\nplt.figure(dpi=100)\n\ng = sns.countplot(x= 'label', data=train_df, palette=colors_nude)\ng.set_title(\"TARGET DISTRIBUTION\", fontsize = 15)\ng.set_xlabel(\"Target Vaues\", fontsize = 10)\ng.set_ylabel(\"Count\", fontsize = 10)\nsizes=[] # Get highest values in y\nfor p in g.patches:\n    height = p.get_height()\n    sizes.append(height)\n    g.text(p.get_x()+p.get_width()/2.,\n            height + 3,\n            '{:1.2f}%'.format(height/total*100),\n            ha=\"center\", fontsize=14) \ng.set_ylim(0, max(sizes) * 1.15) # set y limit based on highest heights\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot sentence lenght\nfig,(ax1,ax2)= plt.subplots(ncols=2, figsize=(20, 8), dpi=100)\nnum=train_df[train_df[\"label\"]== 0]['headline'].str.split().map(lambda x: len(x))\nax1.hist(num,bins = 80,color=colors_nude[0])\nax1.set_title('Non-clickbaits')\nax1.set_ylabel('Lenght of headline')\n\nnum=train_df[train_df[\"label\"]== 1]['headline'].str.split().map(lambda x: len(x))\nax2.hist(num,bins = 80, color=colors_nude[1])\nax2.set_title('clickbaits')\nax2.set_xlabel('Lenght of headline')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Thanks to https://www.kaggle.com/httpwwwfszyc/bert-in-keras-taming\n# Let check who many line is longer than MAX_LEN \ndef convert_lines(example, max_seq_length, tokenizer):\n    max_seq_length -= 2\n    all_tokens = []\n    longer = 0\n    for text in tqdm(example):\n        tokens_a = tokenizer.tokenize(text)\n        if len(tokens_a) > max_seq_length:\n            tokens_a = tokens_a[:max_seq_length]\n            longer += 1\n        one_token = tokenizer.convert_tokens_to_ids([\"[CLS]\"] + tokens_a + [\"[SEP]\"]) + [0] * (max_seq_length - len(tokens_a))\n        all_tokens.append(one_token)\n    print(f\"There are only {longer} lines longer than {max_seq_length}\")\n\n    \nconvert_lines(train_df[\"headline\"], config.MAX_LEN, config.TOKENIZER)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**We can see in graph that most of the text lenght is less the 100  and only 65 line is longer than 256, so we can set our max length between 64 to 256.**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 6. tSNE Embeddings\nt-SNE stands for t-distributed stochastic neighbor embedding. It is a technique for dimensionality reduction that is best suited for the visualization of high dimensional data-set. t-SNE is a randomized algorithm, i;e every time we run the algorithm it returns slightly different results on the same data-set. To control this we set a random state with some arbitrary value. Random state is used here to seed the cost function of the algorithm.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"tokenized = train_df[\"headline\"].apply((lambda x: config.TOKENIZER.encode(x, max_length = config.MAX_LEN, add_special_tokens=True)))\nmax_len = 0\nfor i in tokenized.values:\n    if len(i) > max_len:\n        max_len = len(i)\n\npadded = np.array([i + [0]*(max_len-len(i)) for i in tokenized.values])\nnp.array(padded).shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from cuml.manifold import TSNE\nfrom cuml.decomposition import PCA\ntime_start = time.time()\npca = PCA(n_components=2)\npca_2D = pca.fit_transform(padded.astype(np.float32))\nprint(pca.explained_variance_ratio_)\nprint(' ')\nprint('PCA done! Time elapsed: {} seconds'.format(time.time()-time_start))\n\npca_2D_one  = pca_2D[:,0]\npca_2D_two = pca_2D[:,1]\nplt.figure(figsize=(16,10))\nsns.scatterplot(\n    x=pca_2D_one, y=pca_2D_two,\n    hue=train_df['label'].values,\n    palette=sns.color_palette(colors_nude),\n    legend=\"full\",\n    alpha=0.7\n)\nplt.xlabel('Principal Component 1', fontsize = 15)\nplt.ylabel('Principal Component 2', fontsize = 15)\nplt.title('2 component PCA', fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**By using the attribute explained_varianceratio, you can see that the first principal component contains 9.03% of the variance and the second principal component contains 6.75% of the variance. Together, the two components contain 9.03d% of the information. From graph we can say that text embeddings data not clearly have some clustering structure.**\n\n**Let's now take a look at what TSNE can discern.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"time_start = time.time()\ntsne = TSNE(n_components=2)\ntsne_2D = tsne.fit_transform(padded)\nprint('t-SNE done! Time elapsed: {} seconds'.format(time.time()-time_start))\ntsne_2D_one  = tsne_2D[:,0]\ntsne_2D_two = tsne_2D[:,1]\nplt.figure(figsize=(16,10))\nsns.scatterplot(\n    x=tsne_2D_one, y=tsne_2D_two,\n    hue=train_df['label'].values,\n    palette=sns.color_palette(colors_nude),\n    legend=\"full\",\n    alpha=0.7\n)\nplt.xlabel('t-SNE 1', fontsize = 15)\nplt.ylabel('t-SNE 2', fontsize = 15)\nplt.title('2 component t-SNE' ,fontsize = 20)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**This is already a significant improvement over the PCA visualisation we used earlier. We see that the text embeddings data definitley has some clustering structure. At first glance it doesn't seem that it's easy to clearly separate the target cases, partly becasue they comprise less than 2% of all the points. However, it seems like the most of them are concentrated in the center areas.**","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class XLNETDataset:\n    '''\n    We are required to give it a number of pieces of information which seem redundant, \n    or like they could easily be inferred from the data without us explicity providing it.\n    This class prepare the dataset or input format to XLNet modeling.\n    '''\n    def __init__(self, text, target):\n        self.text = text\n        self.target = target\n        self.tokenizer = config.TOKENIZER\n        self.max_len = config.MAX_LEN\n\n    def __len__(self):\n        return len(self.text)\n\n    def __getitem__(self, item):\n        text = str(self.text[item])\n        text = \" \".join(text.split())\n\n        inputs = self.tokenizer.encode_plus(\n            text,                            # sentence to encode\n            None,\n            add_special_tokens=True,         # add '[CLS]' and '[SEP]'\n            max_length=self.max_len,         # pad & truncate all sentences.\n            pad_to_max_length=True,\n        )\n        \n         # Map the tokens to thier word ids, mask and attention_mask\n        ids = inputs[\"input_ids\"]\n        mask = inputs[\"attention_mask\"]\n        token_type_ids = inputs[\"token_type_ids\"]\n        \n         # Return pytorch tensors\n        return {\n            \"ids\": torch.tensor(ids, dtype=torch.long),\n            \"mask\": torch.tensor(mask, dtype=torch.long),\n            \"token_type_ids\": torch.tensor(token_type_ids, dtype=torch.long),\n            \"targets\": torch.tensor(self.target[item], dtype=torch.float),\n        }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nclass XLNETBaseCased(nn.Module):\n        '''\n        We'll be using XLNet Model. This class we feed the bert input data, the entire pre-trained \n        XLNet model and the additional untrained classification layer is trained on our specific task.\n        '''\n        def __init__(self):\n            super(XLNETBaseCased, self).__init__()\n            self.xlnet = transformers.XLNetModel.from_pretrained(\"xlnet-base-cased\")\n            self.xlnet_drop = nn.Dropout(0.25)    # define the dropout\n            self.out = nn.Linear(768, 1)         # fully connected linear layer\n\n        def forward(self, ids, mask, token_type_ids):\n            # Perform a forward pass. Feeding the inputs in the XLNet model\n            last_hidden_state = self.xlnet(\n                ids, attention_mask=mask, token_type_ids=token_type_ids\n            )\n            # last hidden layer \n            last_hidden_state = last_hidden_state[0]\n            # pool the outputs into a mean vector\n            mean_last_hidden_state = torch.mean(last_hidden_state, 1)\n            # performing dropout on  output vector\n            xo = self.xlnet_drop(mean_last_hidden_state)\n            # performing fully connected linear layer on output vector\n            output = self.out(xo)                        \n            return output","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\nimport datetime\nt0 = time.time() \ndef format_time(elapsed):\n    '''\n    Takes a time in seconds and returns a string hh:mm:ss\n    '''\n    # Round to the nearest second.\n    elapsed_rounded = int(round((elapsed)))\n    \n    # Format as hh:mm:ss\n    return str(datetime.timedelta(seconds=elapsed_rounded))\n\n# define the binary_cross_entropy loss function\ndef loss_fn(outputs, targets):\n    return nn.BCEWithLogitsLoss()(outputs, targets.view(-1, 1))\n\n# define the trainging function\ndef train_fn(data_loader, model, optimizer, device, scheduler):\n    print(\"\")\n    print('Training...')\n    # Set the model to training mode\n    model.train()\n    \n    # trange is a tqdm wrapper around the normal python range\n    for bi, d in enumerate(data_loader):\n        # Unpack training batch from our dataloader.\n        ids = d[\"ids\"]\n        mask = d[\"mask\"]\n        token_type_ids = d[\"token_type_ids\"]\n        targets = d[\"targets\"]\n        \n        # copy each tensor to the GPU\n        ids = ids.to(device, dtype=torch.long)\n        mask = mask.to(device, dtype=torch.long)\n        token_type_ids = token_type_ids.to(device, dtype=torch.long)\n        targets = targets.to(device, dtype=torch.float)\n        \n        # clear any previously calculated gradients\n        optimizer.zero_grad()\n        \n        # outputs prior to activation.\n        outputs = model(ids=ids, mask=mask, token_type_ids=token_type_ids)\n\n        loss = loss_fn(outputs, targets)    # Perform a loss funtion\n        if bi % 200 == 0 and not bi == 0:\n            # Calculate elapsed time in minutes.\n            elapsed = format_time(time.time() - t0)\n            \n            # Report progress.\n            print('  Batch {:>5,}  Training Loss  {:>5,}.    Elapsed: {:}.'.format(bi, loss, elapsed))\n            \n        loss.backward()                     # Perform a backward pass to calculate the gradients\n        if (bi+1) % config.ACCUM_STEPS == 0:  # Wait for several backward steps\n            optimizer.step()                  # Update parameters\n            scheduler.step()                  # Update the learning rate\n        \n\n        \n# define the validation function        \ndef eval_fn(data_loader, model, device):\n    print(\"\")\n    print(\"Running Validation...\")\n    model.eval()            # Set the model to training mode\n    fin_targets = []        # target variable\n    fin_outputs = []        # ouput variable\n    \n    # Tell pytorch not to bother with constructing the compute graph during\n    # the forward pass, since this is only needed for backprop (training).\n    with torch.no_grad():\n        # trange is a tqdm wrapper around the normal python range\n        for bi, d in enumerate(data_loader):\n            # Unpack validation batch from our dataloader.\n            ids = d[\"ids\"]\n            token_type_ids = d[\"token_type_ids\"]\n            mask = d[\"mask\"]\n            targets = d[\"targets\"]\n            \n            # copy each tensor to the GPU\n            ids = ids.to(device, dtype=torch.long)\n            token_type_ids = token_type_ids.to(device, dtype=torch.long)\n            mask = mask.to(device, dtype=torch.long)\n            targets = targets.to(device, dtype=torch.float)\n            \n            # outputs prior to activation.\n            outputs = model(ids=ids, mask=mask, token_type_ids=token_type_ids)\n            \n            loss = loss_fn(outputs, targets)        # Perform a loss funtion\n            if bi % 100 == 0 and not bi == 0:\n                # Calculate elapsed time in minutes.\n                elapsed = format_time(time.time() - t0)\n            \n                # Report progress.\n                print('  Batch {:>5,}  Valid Loss  {:>5,}.    Elapsed: {:}.'.format(bi, loss, elapsed))\n            \n            fin_targets.extend(targets.cpu().detach().numpy().tolist())\n            fin_outputs.extend(torch.sigmoid(outputs).cpu().detach().numpy().tolist())\n    return fin_outputs, fin_targets","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Loading XLNet Model...')\ndevice = torch.device(\"cuda\")     # define the device\nmodel = XLNETBaseCased()         # define the model\nmodel.to(device)                # copy the model to the gpu\n\n# Get all of the model's parameters as a list of tuples.\nparams = list(model.named_parameters())\n\nprint('The BERT model has {:} different named parameters.\\n'.format(len(params)))\n\nprint('==== Embedding Layer ====\\n')\n\nfor p in params[0:5]:\n    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))\n\nprint('\\n==== First Transformer ====\\n')\n\nfor p in params[5:21]:\n    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))\n\nprint('\\n==== Output Layer ====\\n')\n\nfor p in params[-4:]:\n    print(\"{:<55} {:>12}\".format(p[0], str(tuple(p[1].size()))))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# split up our traing set to use 90% for training and 10% for validation\ndf_train, df_valid = model_selection.train_test_split(\n            train_df, test_size=0.1, random_state=101, stratify=train_df.label.values\n        )\n        \ndf_train = df_train.reset_index(drop=True)\ndf_valid = df_valid.reset_index(drop=True)\n\nwith timer('Training and Validation'):\n    \n    def run():\n        seed_everything(config.SEED)\n        \n        # pass the sentence and target from training dataset into class\n        train_dataset = XLNETDataset(\n            text=df_train.headline.values, target=df_train.label.values\n        )\n        \n        # Combine the training inputs into a TensorDataset.\n        train_data_loader = torch.utils.data.DataLoader(\n            train_dataset, batch_size=config.TRAIN_BATCH_SIZE, num_workers=4\n        )\n        \n        # pass the sentence and target from training dataset into class\n        valid_dataset = XLNETDataset(\n            text=df_valid.headline.values, target=df_valid.label.values\n        )\n        \n        # Combine the training inputs into a TensorDataset.\n        valid_data_loader = torch.utils.data.DataLoader(\n            valid_dataset, batch_size=config.VALID_BATCH_SIZE, num_workers=1\n        )\n        \n        # Prepare optimizer and schedule (linear warmup and decay)\n        param_optimizer = list(model.named_parameters())\n        no_decay = [\"bias\", \"LayerNorm.bias\", \"LayerNorm.weight\"]\n        optimizer_parameters = [\n            {\n                \"params\": [\n                    p for n, p in param_optimizer if not any(nd in n for nd in no_decay)\n                ],\n                \"weight_decay\": 0.01,\n            },\n            {\n                \"params\": [\n                    p for n, p in param_optimizer if any(nd in n for nd in no_decay)\n                ],\n                \"weight_decay\": 0.0,\n            },\n        ]\n        \n         # Create the numer of training steps, optimizer and scheduler\n        num_train_steps = int(config.EPOCHS * len(train_dataset) / config.TRAIN_BATCH_SIZE / config.ACCUM_STEPS)\n        optimizer = AdamW(optimizer_parameters, lr= config.LRATE)\n        scheduler = get_linear_schedule_with_warmup(\n            optimizer, num_warmup_steps=config.WARMUP, num_training_steps=num_train_steps\n        )\n        \n        # running the loop for every epochs\n        # Tracking variables \n        best_f1_score = 0\n        for epoch in range(config.EPOCHS):\n            print('======== Epoch {:} / {:} ========'.format(epoch + 1, config.EPOCHS))\n            # passing training and validation funtion\n            train_fn(train_data_loader, model, optimizer, device, scheduler)\n            outputs, targets = eval_fn(valid_data_loader, model, device)\n            outputs = np.array(outputs) >= 0.5\n        \n            # Calculate the validation accuracy for test sentences\n            f1_score = metrics.f1_score(targets, outputs)\n            \n            # Report the final accuracy for this validation run.\n            print(\"\")\n            print(f\"Validation F1 Score = {f1_score}\")\n                        \n            # saving the model\n            if f1_score > best_f1_score:\n                torch.save(model.state_dict(), config.MODEL_PATH)\n                best_f1_score = f1_score\n\n\n    if __name__ == \"__main__\":\n        run()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Setting up Device and Loading Models\nDEVICE = 'cuda'\nMODEL = XLNETBaseCased()\nMODEL.load_state_dict(torch.load(\"./model.bin\"))\nMODEL.to(DEVICE)\nMODEL.eval()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# pass the sentence and target from training dataset into class\nvalid_dataset = XLNETDataset(\n    text=df_valid.headline.values, target=df_valid.label.values\n)\n\n# Combine the training inputs into a TensorDataset.\nvalid_data_loader = torch.utils.data.DataLoader(\n    valid_dataset, batch_size=config.VALID_BATCH_SIZE, num_workers=1\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Tell pytorch not to bother with constructing the compute graph during\n# the forward pass, since this is only needed for backprop (training).\nwith torch.no_grad():\n    fin_outputs = []\n    for bi, d in tqdm(enumerate(valid_data_loader)):\n        ids = d[\"ids\"]\n        token_type_ids = d[\"token_type_ids\"]\n        mask = d[\"mask\"]\n\n        ids = ids.to(device, dtype=torch.long)\n        token_type_ids = token_type_ids.to(device, dtype=torch.long)\n        mask = mask.to(device, dtype=torch.long)\n\n        outputs = model(\n            ids=ids,\n            mask=mask,\n            token_type_ids=token_type_ids\n        )\n        outputs_np = outputs.cpu().detach().numpy().tolist()\n        fin_outputs.extend(outputs_np)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# prediction\ndf_valid['prediction'] = (np.array(fin_outputs))\ndf_valid['prediction'] = (df_valid['prediction'] > 0.5).astype(np.uint)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Classification Report\nfrom sklearn.metrics import classification_report\nprint(classification_report(df_valid.label.values, df_valid.prediction.values))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# creating a function will take sentence and make the prediction\n# It will return the probalility of the clickbait detection\ndef sentence_prediction(sentence):\n    tokenizer = config.TOKENIZER\n    max_len = config.MAX_LEN\n    title = str(sentence)\n    title = \" \".join(title.split())\n\n    inputs = tokenizer.encode_plus(\n        title,\n        None,\n        add_special_tokens=True,\n        max_length=max_len,\n        pad_to_max_length=True,\n    )\n\n    ids = inputs[\"input_ids\"]\n    mask = inputs[\"attention_mask\"]\n    token_type_ids = inputs['token_type_ids']\n\n    ids = torch.tensor(ids, dtype=torch.long).unsqueeze(0)\n    mask = torch.tensor(mask, dtype=torch.long).unsqueeze(0)\n    token_type_ids = torch.tensor(token_type_ids, dtype=torch.long).unsqueeze(0)\n\n    ids = ids.to(DEVICE, dtype=torch.long)\n    mask = mask.to(DEVICE, dtype=torch.long)\n    token_type_ids = token_type_ids.to(DEVICE, dtype=torch.long)\n\n    outputs = MODEL(ids=ids, mask=mask, token_type_ids=token_type_ids)\n\n    outputs = torch.sigmoid(outputs).cpu().detach().numpy()\n    return outputs[0][0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Probalility of the Clickbait ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sentence_prediction(\"Man tries to hug a wild lion! You won't belive what happens next\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sentence_prediction(\"Britain says the drug has been approved for use in the state-run health service, and export restrictions have been introduced\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sentence_prediction(\"Earn 50,000 dollar with this sample trick\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}