{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# # This Python 3 environment comes with many helpful analytics libraries installed\n# # It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# # For example, here's several helpful packages to load\n\n# import numpy as np # linear algebra\n# import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# # Input data files are available in the read-only \"../input/\" directory\n# # For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n# import os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# # You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# # You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-06-11T03:51:37.944582Z","iopub.execute_input":"2021-06-11T03:51:37.944924Z","iopub.status.idle":"2021-06-11T03:51:37.949463Z","shell.execute_reply.started":"2021-06-11T03:51:37.944850Z","shell.execute_reply":"2021-06-11T03:51:37.948425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This notebook is similar to the Bert for Token Classification [Training](https://www.kaggle.com/thanish/bert-for-token-classification-training)  and [Inference](http://www.kaggle.com/thanish/bert-for-token-classification-inference?scriptVersionId=63525213). Do check them out. \n\nThe main difference you will find in this notebook is how Roberta tokenizer differs from Bert tokenizer. There is a slight modification required during tokenizing and also when we extract the predicted words from the token. \n\n##### Let's talk a little bit about the difference in tokenization for Bert and Roberta\nIn Bert when the sentences are tokenized the words are not prefixed by any special character when it's a single word. However when it's a compund word, then the word is split into mulitple chunk of sub-words were every sub-word except for the 1st sub-word is prefixed with \"##\". In Bert tokenizer ## is notation that the current subword should be combined with the previous sub-word to make it a single word\n\nIn Roberta during tokenization, the first word in the sentence is not prefixed with any special character. However every word other than the 1st word in the sentence is prefixed with \"Ġ\", also to note when a single word is split into multiple sub-words then the first sub word is prefixed with \"Ġ\" and the rest of the subwords are not prefixed with any special character. In Roberta tokenizer, subwords without \"Ġ\" is a notation that the current subword should be combined with the previous sub-word to make it a single word.\n\nThe reason for the difference is Bert uses WordPiece tokenizer and Roberta uses Byte Pair Encoding(BPE) tokenizer. You can read more about it in this excellent [blog](https://blog.floydhub.com/tokenization-nlp/) how these 2 are different. ","metadata":{}},{"cell_type":"markdown","source":"##### Let check out with an example the tokenization between Bert and Roberta to understand better.\n\n```\nfrom transformers import AutoTokenizer\n\nbert_base_path = '../input/huggingface-bert/bert-base-uncased'\nroberta_base_path = '../input/huggingface-roberta/roberta-base'\n\nbert_tokenizer = AutoTokenizer.from_pretrained(bert_base_path , do_lower_case=True)\nroberta_tokenizer = AutoTokenizer.from_pretrained(roberta_base_path , do_lower_case=True)\n\n##########\n\nexample_sentence = 'shipboard cetacean surveys within the pacific islands region'\nexample_label = 'cetacean surveys'\n\n##########\n\nprint(\"Bert tokenized sentence:\",bert_tokenizer.tokenize(example_sentence))\nprint(\"Bert tokenized label:\",bert_tokenizer.tokenize(example_label))\n\nprint(\"\")\nprint(\"Roberta  tokenized sentence:\", roberta_tokenizer.tokenize(example_sentence))\nprint(\"Roberta tokenized sentence:\",roberta_tokenizer.tokenize(example_label))\n\n##########\n```\n\n```\nBert tokenized sentence: ['ship', '##board', 'ce', '##ta', '##cea', '##n', 'surveys', 'within', 'the', 'pacific', 'islands', 'region']\nBert tokenized label: ['ce', '##ta', '##cea', '##n', 'surveys']\n\nRoberta  tokenized sentence: ['ship', 'board', 'Ġc', 'et', 'ace', 'an', 'Ġsurveys', 'Ġwithin', 'Ġthe', 'Ġpac', 'ific', 'Ġislands', 'Ġregion']\nRoberta tokenized sentence: ['c', 'et', 'ace', 'an', 'Ġsurveys']\n```\n\nFrom the above example, we can understand If we try to do a direct string to string match between the tokenized sentence and the tokenized label, we will be able to get the match for bert_tokenizer but the same may not work for Roberta_tokenier. \n\nThis is for the very simple reason that the labels are not always at the starting of the sentence. So when we tokenize using Roberta the first word in the label is not prefixed by \"Ġ\" and the first word of the label in the tokenized sentence is always prefixed with \"Ġ\". \n\nOne simple trick to get past is to add a \" \" before the label and the sentence to bypass this. \n\nLet's look at an example below\n\n```\nprint(roberta_tokenizer.tokenize(\" \" + example_sentence))\nprint(roberta_tokenizer.tokenize(\" \" + example_label))\n```\n```\n['Ġship', 'board', 'Ġc', 'et', 'ace', 'an', 'Ġsurveys', 'Ġwithin', 'Ġthe', 'Ġpac', 'ific', 'Ġislands', 'Ġregion']\n['Ġc', 'et', 'ace', 'an', 'Ġsurveys']\n```\n\nFor this reason, you will find in the rest of the notebook, whenever I'm tokenizing I will prefix the text with \" \".\n\nThe rest of the content will be similar to the Notebooks:\nBert for Token Classification [Training](https://www.kaggle.com/thanish/bert-for-token-classification-training)  and [Inference](http://www.kaggle.com/thanish/bert-for-token-classification-inference?scriptVersionId=63525213)","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nimport json\nimport re\nfrom nltk.tokenize import sent_tokenize \nfrom transformers import AutoTokenizer\nfrom torch.utils.data import DataLoader\nimport torch\nimport torch.nn as nn\nimport transformers\nfrom tqdm import tqdm\nimport glob\n\nimport random\n\nfrom sklearn.model_selection import train_test_split\n\nimport datetime \n\nimport warnings\nwarnings.filterwarnings('ignore')","metadata":{"execution":{"iopub.status.busy":"2021-06-11T03:51:38.053222Z","iopub.execute_input":"2021-06-11T03:51:38.053505Z","iopub.status.idle":"2021-06-11T03:51:40.960501Z","shell.execute_reply.started":"2021-06-11T03:51:38.053469Z","shell.execute_reply":"2021-06-11T03:51:40.959448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Config","metadata":{}},{"cell_type":"code","source":"platform = 'Kaggle'\nmodel_name = 'model_roberta_base.bin'\n\nif platform == 'Kaggle':\n    roberta_path = '../input/huggingface-roberta/roberta-base'\n    train_path = '/kaggle/input/coleridgeinitiative-show-us-the-data/train/'\n    test_path = '/kaggle/input/coleridgeinitiative-show-us-the-data/test/*'\n    model_path = '../input/coleridgemodels/'+ model_name\n    \nconfig = {'MAX_LEN':128,\n          'tokenizer': AutoTokenizer.from_pretrained(roberta_path , do_lower_case=True),\n          'batch_size':5,\n          'Epoch': 2,\n          'train_path':train_path,\n          'test_path':test_path, \n          'device': 'cuda' if torch.cuda.is_available() else 'cpu',\n          'model_path':model_path,\n          'model_name':model_name\n         }","metadata":{"execution":{"iopub.status.busy":"2021-06-11T03:51:40.962209Z","iopub.execute_input":"2021-06-11T03:51:40.962572Z","iopub.status.idle":"2021-06-11T03:51:41.203772Z","shell.execute_reply.started":"2021-06-11T03:51:40.962532Z","shell.execute_reply":"2021-06-11T03:51:41.202884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reading the train csv","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(\"../input/coleridgeinitiative-show-us-the-data/train.csv\")\ntrain","metadata":{"execution":{"iopub.status.busy":"2021-06-11T03:51:41.205601Z","iopub.execute_input":"2021-06-11T03:51:41.206177Z","iopub.status.idle":"2021-06-11T03:51:41.358806Z","shell.execute_reply.started":"2021-06-11T03:51:41.206139Z","shell.execute_reply":"2021-06-11T03:51:41.357330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Combining the labels together","metadata":{}},{"cell_type":"code","source":"train_df = train.groupby(['Id']).agg(label_count = ('cleaned_label', 'count'),\n                                     label = ('cleaned_label', '|'.join)).reset_index()\ntrain_df","metadata":{"execution":{"iopub.status.busy":"2021-06-11T03:51:41.367500Z","iopub.execute_input":"2021-06-11T03:51:41.367789Z","iopub.status.idle":"2021-06-11T03:51:41.536442Z","shell.execute_reply.started":"2021-06-11T03:51:41.367761Z","shell.execute_reply":"2021-06-11T03:51:41.535724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reading all the json train files","metadata":{}},{"cell_type":"code","source":"def read_all_json(df, path):\n    '''\n    This function reads all the json input files and return a dictionary containing the id as the key and all the contents of the json as values\n    '''\n    text_data = {}\n    for i, rec_id in tqdm(enumerate(df.Id), total = len(df.Id)):\n        location = f'{path}{rec_id}.json'\n\n        with open(location, 'r') as f:\n            text_data[rec_id] = json.load(f)\n        \n    print(\"All files read\")\n    end = datetime.datetime.now()\n    \n    return text_data","metadata":{"execution":{"iopub.status.busy":"2021-06-11T03:51:41.538626Z","iopub.execute_input":"2021-06-11T03:51:41.539175Z","iopub.status.idle":"2021-06-11T03:51:41.545166Z","shell.execute_reply.started":"2021-06-11T03:51:41.539136Z","shell.execute_reply":"2021-06-11T03:51:41.544226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%time \ndata_dict = read_all_json(df=train_df, path=config['train_path'])","metadata":{"execution":{"iopub.status.busy":"2021-06-11T03:51:41.547457Z","iopub.execute_input":"2021-06-11T03:51:41.547732Z","iopub.status.idle":"2021-06-11T03:52:40.480319Z","shell.execute_reply.started":"2021-06-11T03:51:41.547699Z","shell.execute_reply":"2021-06-11T03:52:40.477789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def clean_text(txt):\n    return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower())","metadata":{"execution":{"iopub.status.busy":"2021-06-11T03:52:40.481661Z","iopub.execute_input":"2021-06-11T03:52:40.482013Z","iopub.status.idle":"2021-06-11T03:52:40.486766Z","shell.execute_reply.started":"2021-06-11T03:52:40.481976Z","shell.execute_reply":"2021-06-11T03:52:40.485578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def data_joining(data_dict_id):\n    '''\n    This function is to join all the text data from different sections in the json to a single text file. \n    '''\n    data_length = len(data_dict_id)\n\n    #     temp = [clean_text(data_dict_id[i]['text']) for i in range(data_length)]\n    temp = [data_dict_id[i]['text'] for i in range(data_length)]\n    temp = '. '.join(temp)\n    \n    return temp","metadata":{"execution":{"iopub.status.busy":"2021-06-11T03:52:40.490137Z","iopub.execute_input":"2021-06-11T03:52:40.490717Z","iopub.status.idle":"2021-06-11T03:52:40.496889Z","shell.execute_reply.started":"2021-06-11T03:52:40.490655Z","shell.execute_reply":"2021-06-11T03:52:40.496092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def make_shorter_sentence(sentence):\n    sent_tokenized = sent_tokenize(sentence)\n    \n    max_length = config['MAX_LEN']\n    overlap = 20\n    \n    final_sentences = []\n    \n    for tokenized_sent in sent_tokenized:\n        sent_tokenized_clean = clean_text(tokenized_sent)\n        sent_tokenized_clean = sent_tokenized_clean.replace('.','').rstrip() \n        \n        tok_sent = sent_tokenized_clean.split(\" \")\n        \n        if len(tok_sent)<max_length:\n            final_sentences.append(sent_tokenized_clean)\n        else :\n#             print(\"Making shorter sentences\")\n            start = 0\n            end = len(tok_sent)\n            \n            for i in range(start, end, max_length-overlap):\n                temp = tok_sent[i: (i + max_length)]\n                final_sentences.append(\" \".join(i for i in temp))\n\n    return final_sentences","metadata":{"execution":{"iopub.status.busy":"2021-06-11T03:52:40.500089Z","iopub.execute_input":"2021-06-11T03:52:40.500373Z","iopub.status.idle":"2021-06-11T03:52:40.509590Z","shell.execute_reply.started":"2021-06-11T03:52:40.500337Z","shell.execute_reply":"2021-06-11T03:52:40.508751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def form_labels(sentence, labels_list):\n    '''\n    This function labels the training data \n    '''\n    matched_kwords = []\n    matched_token = []\n    un_matched_kwords = []\n    label = []\n\n    # Since there are many sentences which are more than 512. Let's make the max length of all\n    # the sentences be 64\n    tokens = make_shorter_sentence(sentence)\n    \n    for tok in tokens:    \n        tok_split = config['tokenizer'].tokenize(\" \" + tok)\n        \n        z = np.array(['O'] * len(tok_split)) # Create final label == len(tokens) of each sentence\n        matched_keywords = 0 # Initially no kword matched    \n\n        for kword in labels_list:\n            kword_split = config['tokenizer'].tokenize(\" \" + kword)\n            for i in range(len(tok_split)):\n                if tok_split[i: (i + len(kword_split))] == kword_split:\n                    matched_keywords += 1\n#                     print(\"matched keyword with token:\", tok_split[i: (i+len(kword_split))] )\n#                     print(tok_split)\n\n                    if (len(kword_split) == 1):\n                        z[i] = 'B'\n                    else:\n                        z[i] = 'B'\n                        z[(i+1) : (i+ len(kword_split))]= 'B'\n\n                    if matched_keywords >1:\n                        label[-1] = (z.tolist())\n                        matched_token[-1] = tok\n                        matched_kwords[-1].append(kword)\n                    else:\n                        label.append(z.tolist())\n                        matched_token.append(tok)\n                        matched_kwords.append([kword])\n                    #print(label[-1])\n                    #print(\"\")\n    #                 break\n                else:\n                    un_matched_kwords.append(tok)\n                \n    return matched_token, matched_kwords, label, un_matched_kwords\n","metadata":{"execution":{"iopub.status.busy":"2021-06-11T03:52:40.511003Z","iopub.execute_input":"2021-06-11T03:52:40.511415Z","iopub.status.idle":"2021-06-11T03:52:40.522785Z","shell.execute_reply.started":"2021-06-11T03:52:40.511380Z","shell.execute_reply":"2021-06-11T03:52:40.521874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def labelling(dataset, data_dict):\n    \n    Id_list_ = []\n    sentences_ = []\n    key_ = []\n    labels_ = []\n    un_mat = []\n    un_matched_reviews = 0\n\n    for i, Id in tqdm(enumerate(dataset.Id), total=len(dataset.Id)):\n#         print(Id)\n        \n        sentence = data_joining(data_dict[Id])\n        labels = train_df.label[train_df.Id == Id].tolist()[0].split(\"|\")\n\n        s, k, l, un_matched = form_labels(sentence=sentence, labels_list = labels)\n\n        if len(s) == 0:\n            un_matched_reviews += 1\n            un_mat.append(un_matched)\n        else: \n            sentences_.append(s)\n            key_.append(k)\n            labels_.append(l)\n            Id_list_.append([Id]*len(l))\n\n#         if (i%100) == 0:\n#             print(f\"Completed {i}/{train_df.Id.shape[0]}\")\n\n    print(\"Total unmatched keywords:\", un_matched_reviews)\n    sentences = [item for sublist in sentences_ for item in sublist]\n    final_labels = [item for sublist in labels_ for item in sublist]\n    keywords = [item for sublist in key_ for item in sublist]\n    Id_list = [item for sublist in Id_list_ for item in sublist]\n    \n    return sentences, final_labels, keywords, Id_list\n","metadata":{"execution":{"iopub.status.busy":"2021-06-11T03:52:40.524045Z","iopub.execute_input":"2021-06-11T03:52:40.524369Z","iopub.status.idle":"2021-06-11T03:52:40.535354Z","shell.execute_reply.started":"2021-06-11T03:52:40.524334Z","shell.execute_reply":"2021-06-11T03:52:40.534415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_sentences, train_labels, train_keywords, train_Id_list = labelling(dataset = train_df, data_dict=data_dict)\n# valid_sentences, valid_labels, valid_keywords, valid_Id_list = labelling(dataset = DF_valid)\n\nprint(\"\")\nprint(f\" train sentences: {len(train_sentences)}, train label: {len(train_labels)}, train keywords: {len(train_keywords)}, train_id list: {len(train_Id_list)}\")","metadata":{"execution":{"iopub.status.busy":"2021-06-11T03:52:40.536584Z","iopub.execute_input":"2021-06-11T03:52:40.536963Z","iopub.status.idle":"2021-06-11T04:19:45.381080Z","shell.execute_reply.started":"2021-06-11T03:52:40.536927Z","shell.execute_reply":"2021-06-11T04:19:45.380227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create DataFrame to remove the duplicates","metadata":{}},{"cell_type":"code","source":"unique_df = pd.DataFrame({'id':train_Id_list, \n                          'train_sentences': train_sentences, \n                          'kword': train_keywords, \n                          'label':train_labels})\nunique_df.label = unique_df.label.astype('str')\nunique_df.kword = unique_df.kword.astype('str')\nunique_df['sent_len'] = unique_df.train_sentences.apply(lambda x : len(x.split(\" \")))\nunique_df.head(60)","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:19:45.382573Z","iopub.execute_input":"2021-06-11T04:19:45.383158Z","iopub.status.idle":"2021-06-11T04:19:45.892796Z","shell.execute_reply.started":"2021-06-11T04:19:45.383115Z","shell.execute_reply":"2021-06-11T04:19:45.891978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Take unique of the dataset","metadata":{}},{"cell_type":"code","source":"unique_df = unique_df.drop_duplicates()\nunique_df.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:19:45.893989Z","iopub.execute_input":"2021-06-11T04:19:45.894317Z","iopub.status.idle":"2021-06-11T04:19:45.990645Z","shell.execute_reply.started":"2021-06-11T04:19:45.894281Z","shell.execute_reply":"2021-06-11T04:19:45.989874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Taking a sample of the dataset\nLet's take only a small sample of the data (20%) to train a model for experimentation","metadata":{}},{"cell_type":"code","source":"unique_df = unique_df.sample(int(unique_df.shape[0]*0.05)).reset_index(drop=True)\nunique_df.shape\nunique_df","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:19:45.992501Z","iopub.execute_input":"2021-06-11T04:19:45.993134Z","iopub.status.idle":"2021-06-11T04:19:46.012771Z","shell.execute_reply.started":"2021-06-11T04:19:45.993095Z","shell.execute_reply":"2021-06-11T04:19:46.011856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train and validation split","metadata":{}},{"cell_type":"code","source":"np.random.seed(100)\ntrain_df, valid_df = train_test_split(unique_df, test_size=0.2)\n\ntrain_df = train_df.reset_index(drop=True)\nvalid_df = valid_df.reset_index(drop=True)\n\nprint(train_df.shape, valid_df.shape)","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:19:46.014121Z","iopub.execute_input":"2021-06-11T04:19:46.014481Z","iopub.status.idle":"2021-06-11T04:19:46.023085Z","shell.execute_reply.started":"2021-06-11T04:19:46.014443Z","shell.execute_reply":"2021-06-11T04:19:46.022237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Converting the DataFrame back to list","metadata":{}},{"cell_type":"code","source":"tags_2_idx = {'O': 0 , 'B': 1, 'P': 2} # 'P' means padding. \n\ndef dataset_2_list(df):\n    id_list = df.id.values.tolist()\n    sentences_list = df.train_sentences.values.tolist()\n    keywords_list = df.kword.apply(lambda x : eval(x)).values.tolist()\n    \n    labels_list = df.label.apply(lambda x : eval(x)).values.tolist()    \n    labels_list = [list(map(tags_2_idx.get, lab)) for lab in labels_list]\n    \n    return id_list, sentences_list, keywords_list, labels_list\n\nfinal_train_id_list, final_train_sentences, final_train_keywords, final_train_labels = dataset_2_list(df=train_df)\nfinal_valid_id_list, final_valid_sentences, final_valid_keywords, final_valid_labels = dataset_2_list(df=valid_df)","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:19:46.024427Z","iopub.execute_input":"2021-06-11T04:19:46.024941Z","iopub.status.idle":"2021-06-11T04:19:46.174597Z","shell.execute_reply.started":"2021-06-11T04:19:46.024881Z","shell.execute_reply":"2021-06-11T04:19:46.173756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Forming the input","metadata":{}},{"cell_type":"code","source":"class form_input():\n    def __init__(self, ID, sentence, kword, label, data_type='test'):\n        self.id = ID\n        self.sentence = sentence\n        self.kword = kword\n        self.label = label\n        self.max_length = config['MAX_LEN']\n        self.tokenizer = config['tokenizer']\n        self.data_type = data_type\n    \n    def __len__(self):\n        return len(self.sentence)\n    \n    def __getitem__(self, item):\n        toks = config['tokenizer'].tokenize(\" \" + self.sentence[item])\n        label = self.label[item]\n\n        if len(toks)>self.max_length:\n            toks = toks[:self.max_length]\n            label = label[:self.max_length]\n                \n        ########################################\n        # Forming the inputs\n        ids = config['tokenizer'].convert_tokens_to_ids(toks)\n        tok_type_id = [0] * len(ids)\n        att_mask = [1] * len(ids)\n        \n        # Padding\n        pad_len = self.max_length - len(ids)        \n        ids = ids + [2] * pad_len\n        tok_type_id = tok_type_id + [0] * pad_len\n        att_mask = att_mask + [0] * pad_len\n        \n        ########################################            \n        # Forming the label\n        if self.data_type !='test':\n            label = label + [2]*pad_len\n        else:\n            label = 1\n        \n        ########################################\n                \n        return {'pub_id': self.id[item],\n                #'item': item,\n                #'sentence': self.sentence[item],\n                #'kword' : self.kword[item],\n                'ids': torch.tensor(ids, dtype = torch.long),\n                'tok_type_id': torch.tensor(tok_type_id, dtype = torch.long),\n                'att_mask': torch.tensor(att_mask, dtype = torch.long),\n                'target': torch.tensor(label, dtype = torch.long)\n               }\n            ","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:19:46.175917Z","iopub.execute_input":"2021-06-11T04:19:46.176241Z","iopub.status.idle":"2021-06-11T04:19:46.188645Z","shell.execute_reply.started":"2021-06-11T04:19:46.176207Z","shell.execute_reply":"2021-06-11T04:19:46.187861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Define the dataloader","metadata":{}},{"cell_type":"code","source":"train_prod_input = form_input(ID=final_train_id_list, \n                              sentence=final_train_sentences, \n                              kword=final_train_keywords, \n                              label=final_train_labels, \n                              data_type='train')\n\nvalid_prod_input = form_input(ID=final_valid_id_list, \n                              sentence=final_valid_sentences, \n                              kword=final_valid_keywords, \n                              label=final_valid_labels, \n                              data_type='valid')\n\ntrain_prod_input_data_loader = DataLoader(train_prod_input, \n                                          batch_size= config['batch_size'], \n                                          shuffle=True)\n\nvalid_prod_input_data_loader = DataLoader(valid_prod_input, \n                                          batch_size= config['batch_size'], \n                                          shuffle=True)","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:19:46.191489Z","iopub.execute_input":"2021-06-11T04:19:46.192110Z","iopub.status.idle":"2021-06-11T04:19:46.198730Z","shell.execute_reply.started":"2021-06-11T04:19:46.192069Z","shell.execute_reply":"2021-06-11T04:19:46.197865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Checking the output\n# for ind in range(8057):\n#     print(ind)\n#     train_prod_input[ind], valid_prod_input[ind]","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:19:46.200106Z","iopub.execute_input":"2021-06-11T04:19:46.200481Z","iopub.status.idle":"2021-06-11T04:19:46.211453Z","shell.execute_reply.started":"2021-06-11T04:19:46.200416Z","shell.execute_reply":"2021-06-11T04:19:46.210603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def setting_seed(seed_no=100):\n    random.seed(seed_no)\n    np.random.seed(seed_no)\n    torch.manual_seed(seed_no)\n    torch.cuda.manual_seed_all(seed_no)    \n","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:19:46.212827Z","iopub.execute_input":"2021-06-11T04:19:46.213301Z","iopub.status.idle":"2021-06-11T04:19:46.220287Z","shell.execute_reply.started":"2021-06-11T04:19:46.213266Z","shell.execute_reply":"2021-06-11T04:19:46.219448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_fn(data_loader, model, optimizer):\n    '''\n    Function to train the model\n    '''\n    setting_seed(seed_no=100)\n    train_loss = 0\n    for index, dataset in enumerate(tqdm(data_loader, total = len(data_loader))):\n        batch_input_ids = dataset['ids'].to(config['device'], dtype = torch.long)\n        batch_att_mask = dataset['att_mask'].to(config['device'], dtype = torch.long)\n        batch_tok_type_id = dataset['tok_type_id'].to(config['device'], dtype = torch.long)\n        batch_target = dataset['target'].to(config['device'], dtype = torch.long)\n                \n        output = model(batch_input_ids, \n                       token_type_ids=None,\n                       attention_mask=batch_att_mask,\n                       labels=batch_target)\n        \n        step_loss = output[0]\n        prediction = output[1]\n        \n        step_loss.sum().backward()\n        optimizer.step()        \n        train_loss += step_loss\n        optimizer.zero_grad()\n        \n    return train_loss.sum()","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:19:46.221546Z","iopub.execute_input":"2021-06-11T04:19:46.222108Z","iopub.status.idle":"2021-06-11T04:19:46.230943Z","shell.execute_reply.started":"2021-06-11T04:19:46.222071Z","shell.execute_reply":"2021-06-11T04:19:46.229862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def eval_fn(data_loader, model):\n    '''\n    Functiont to evaluate the model on each epoch. \n    We can also use Jaccard metric to see the performance on each epoch.\n    '''\n    setting_seed(seed_no=100)\n    model.eval()\n    \n    eval_loss = 0\n    predictions = np.array([], dtype = np.int64).reshape(0, config['MAX_LEN'])\n    true_labels = np.array([], dtype = np.int64).reshape(0, config['MAX_LEN'])\n    \n    with torch.no_grad():\n        for index, dataset in enumerate(tqdm(data_loader, total = len(data_loader))):\n            batch_input_ids = dataset['ids'].to(config['device'], dtype = torch.long)\n            batch_att_mask = dataset['att_mask'].to(config['device'], dtype = torch.long)\n            batch_tok_type_id = dataset['tok_type_id'].to(config['device'], dtype = torch.long)\n            batch_target = dataset['target'].to(config['device'], dtype = torch.long)\n\n            output = model(batch_input_ids, \n                           token_type_ids=None,\n                           attention_mask=batch_att_mask,\n                           labels=batch_target)\n\n            step_loss = output[0]\n            eval_prediction = output[1]\n\n            eval_loss += step_loss\n            \n            eval_prediction = np.argmax(eval_prediction.detach().to('cpu').numpy(), axis = 2)\n            actual = batch_target.to('cpu').numpy()\n            \n            predictions = np.concatenate((predictions, eval_prediction), axis = 0)\n            true_labels = np.concatenate((true_labels, actual), axis = 0)\n            \n    return eval_loss.sum(), predictions, true_labels","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:19:46.234828Z","iopub.execute_input":"2021-06-11T04:19:46.235088Z","iopub.status.idle":"2021-06-11T04:19:46.245059Z","shell.execute_reply.started":"2021-06-11T04:19:46.235063Z","shell.execute_reply":"2021-06-11T04:19:46.244006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_engine(epoch, train_data, valid_data):\n    setting_seed(seed_no=100)\n    model = transformers.RobertaForTokenClassification.from_pretrained('roberta-base',  num_labels = len(tags_2_idx))\n    model = nn.DataParallel(model)\n    model = model.to(config['device'])\n    \n    params = model.parameters()\n    optimizer = torch.optim.Adam(params, lr= 3e-5)\n    \n    best_eval_loss = 1000000\n    for i in range(epoch):\n        train_loss = train_fn(data_loader = train_data, \n                              model=model, \n                              optimizer=optimizer)\n        eval_loss, eval_predictions, true_labels = eval_fn(data_loader = valid_data, \n                                                           model=model)\n        \n        print(f\"Epoch {i} , Train loss: {train_loss}, Eval loss: {eval_loss}\")\n\n        if eval_loss < best_eval_loss:\n            best_eval_loss = eval_loss           \n            \n            print(\"Saving the model\")\n            torch.save(model.state_dict(), config['model_name'])\n            \n    return model, eval_predictions, true_labels ","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:19:46.246782Z","iopub.execute_input":"2021-06-11T04:19:46.247107Z","iopub.status.idle":"2021-06-11T04:19:46.257284Z","shell.execute_reply.started":"2021-06-11T04:19:46.247070Z","shell.execute_reply":"2021-06-11T04:19:46.256466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model, val_predictions, val_true_labels = train_engine(epoch=config['Epoch'],\n                                                       train_data=train_prod_input_data_loader, \n                                                       valid_data=valid_prod_input_data_loader)","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:19:46.258301Z","iopub.execute_input":"2021-06-11T04:19:46.258795Z","iopub.status.idle":"2021-06-11T04:21:45.541636Z","shell.execute_reply.started":"2021-06-11T04:19:46.258754Z","shell.execute_reply":"2021-06-11T04:21:45.540782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reading the test dataset","metadata":{}},{"cell_type":"code","source":"def read_test_json(test_data_folder):\n    '''\n    This function reads all the json input files and return a dictionary containing the id as the key\n    and all the contents of the json as values\n    '''\n\n    test_text_data = {}\n    total_files = len(glob.glob(test_data_folder))\n    \n    for i, test_json_loc in enumerate(glob.glob(test_data_folder)):\n        filename = test_json_loc.split(\"/\")[-1][:-5]\n\n        with open(test_json_loc, 'r') as f:\n            test_text_data[filename] = json.load(f)\n\n    print(\"All files read\")\n    return test_text_data","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:21:45.543227Z","iopub.execute_input":"2021-06-11T04:21:45.543845Z","iopub.status.idle":"2021-06-11T04:21:45.549938Z","shell.execute_reply.started":"2021-06-11T04:21:45.543805Z","shell.execute_reply":"2021-06-11T04:21:45.549166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data_dict = read_test_json(test_data_folder=config['test_path'])","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:21:45.551103Z","iopub.execute_input":"2021-06-11T04:21:45.551459Z","iopub.status.idle":"2021-06-11T04:21:45.593606Z","shell.execute_reply.started":"2021-06-11T04:21:45.551417Z","shell.execute_reply":"2021-06-11T04:21:45.592852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Prediction\ndef prediction_fn(tokenized_sub_sentence):\n\n    tkns = tokenized_sub_sentence\n    indexed_tokens = config['tokenizer'].convert_tokens_to_ids(tkns)\n    segments_ids = [0] * len(indexed_tokens)\n\n    tokens_tensor = torch.tensor([indexed_tokens]).to(config['device'])\n    segments_tensors = torch.tensor([segments_ids]).to(config['device'])\n    \n    setting_seed(seed_no=100)\n    model.eval()\n    with torch.no_grad():\n        logit = model(tokens_tensor, \n                      token_type_ids=None,\n                      attention_mask=segments_tensors)\n\n        logit_new = logit[0].argmax(2).detach().cpu().numpy().tolist()\n        prediction = logit_new[0]\n\n#         print(tkns)\n#         print(logit_new)\n#         print(prediction)\n        \n        kword = ''\n        kword_list = []\n\n        for k, j in enumerate(prediction):\n            if (len(prediction)>1):\n\n                if (j!=0) & (k==0):\n                    #if it's the first word in the first position\n                    #print('At begin first word')\n                    begin = tkns[k]\n                    kword = begin\n\n                elif (j!=0) & (k>=1) & (prediction[k-1]==0):\n                    #begin word is in the middle of the sentence\n                    begin = tkns[k]\n                    previous = tkns[k-1]\n\n                    if not begin.startswith('Ġ'):\n                        kword = previous + begin\n                    else:\n                        kword = begin\n\n                    if k == (len(prediction) - 1):\n                        #print('begin and end word is the last word of the sentence')\n                        kword_list.append(kword.rstrip().lstrip().replace('Ġ', ''))\n\n                elif (j!=0) & (k>=1) & (prediction[k-1]!=0):\n                    # intermediate word of the same keyword\n                    inter = tkns[k]\n\n                    if not inter.startswith('Ġ'):\n                        kword = kword + \"\" + inter\n                    else:\n                        kword = kword + \" \" + inter\n\n\n                    if k == (len(prediction) - 1):\n                        #print('begin and end')\n                        kword_list.append(kword.rstrip().lstrip().replace('Ġ', ''))\n\n                elif (j==0) & (k>=1) & (prediction[k-1] !=0):\n                    # End of a keywords but not end of sentence.\n                    kword_list.append(kword.rstrip().lstrip().replace('Ġ', ''))\n                    kword = ''\n                    inter = ''\n            else:\n                if (j!=0):\n                    begin = tkns[k]\n                    kword = begin\n                    kword_list.append(kword.rstrip().lstrip().replace('Ġ', ''))\n#         print(kword_list)\n#         print(\"\")\n    return kword_list\n","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:21:45.594890Z","iopub.execute_input":"2021-06-11T04:21:45.595224Z","iopub.status.idle":"2021-06-11T04:21:45.609573Z","shell.execute_reply.started":"2021-06-11T04:21:45.595188Z","shell.execute_reply":"2021-06-11T04:21:45.608647Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def long_sent_split(text):\n    sent_split = text.split(\" \")\n\n    start = 0\n    end = len(sent_split)\n    max_length = 64\n\n    final_sent_split = []\n    for i in range(start, end, max_length):\n        temp = sent_split[i: (i + max_length)]\n        final_sent_split.append(\" \".join(i for i in temp))\n    return final_sent_split","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:21:45.610998Z","iopub.execute_input":"2021-06-11T04:21:45.611437Z","iopub.status.idle":"2021-06-11T04:21:45.623602Z","shell.execute_reply.started":"2021-06-11T04:21:45.611402Z","shell.execute_reply":"2021-06-11T04:21:45.622782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_predictions(data_dict):\n    \n    results = {}\n\n    for i, Id in enumerate(data_dict.keys()):\n        current_id_predictions = []\n    \n        print(Id)\n        sentences = data_joining(data_dict[Id])\n        sentence_tokens = sent_tokenize(sentences)\n        \n        for sub_sentence in sentence_tokens:\n            cleaned_sub_sentence = clean_text(sub_sentence)\n        \n            # Tokenize the sentence\n            tokenized_sub_sentence = config['tokenizer'].tokenize(\" \" + cleaned_sub_sentence)\n            \n            if len(tokenized_sub_sentence) == 0:\n                # If the tokenized sentence are empty\n                sub_sentence_prediction_kword_list = []\n                \n            elif len(tokenized_sub_sentence) <= 512:\n                # If the tokenized sentence are less than 512\n                sub_sentence_prediction_kword_list = prediction_fn(tokenized_sub_sentence)\n\n            else:\n                # If the tokenized sentence are >512 which is long sentences\n                long_sent_kword_list = []\n                \n                tokenized_sub_sentence_tok_split = long_sent_split(text = tokenized_sub_sentence)\n                for i, sent_tok in enumerate(tokenized_sub_sentence_tok_split):\n                    if len(sent) != 0:\n                        kword_list = prediction_fn(sent_tok)\n                        long_sent_kword_list.append(kword_list)\n                flat_long_sent_kword = [item for sublist in long_sent_kword_list for item in sublist]\n                sub_sentence_prediction_kword_list = flat_long_sent_kword\n                            \n            if len(sub_sentence_prediction_kword_list) !=0:\n                current_id_predictions = current_id_predictions + sub_sentence_prediction_kword_list\n\n        results[Id] = list(set(current_id_predictions))\n                \n    print(\"All predictions completed\")\n    \n    return results","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:21:45.624924Z","iopub.execute_input":"2021-06-11T04:21:45.625346Z","iopub.status.idle":"2021-06-11T04:21:45.635184Z","shell.execute_reply.started":"2021-06-11T04:21:45.625312Z","shell.execute_reply":"2021-06-11T04:21:45.634019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nresults = get_predictions(data_dict = test_data_dict)","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:21:45.636720Z","iopub.execute_input":"2021-06-11T04:21:45.637095Z","iopub.status.idle":"2021-06-11T04:22:13.489955Z","shell.execute_reply.started":"2021-06-11T04:21:45.637061Z","shell.execute_reply":"2021-06-11T04:22:13.489051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub_df = pd.DataFrame({'Id': list(results.keys()),\n                       'PredictionString': list(results.values())})\nsub_df.PredictionString = sub_df.PredictionString.apply(lambda x : \"|\".join(x))\nsub_df","metadata":{"execution":{"iopub.status.busy":"2021-06-11T04:22:13.491168Z","iopub.execute_input":"2021-06-11T04:22:13.491662Z","iopub.status.idle":"2021-06-11T04:22:13.503916Z","shell.execute_reply.started":"2021-06-11T04:22:13.491621Z","shell.execute_reply":"2021-06-11T04:22:13.503105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# --------------------- Consider upvoting if you like it:) ---------------------","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}