{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

Coleridge: Complete EDA in ONE Notebook

\n\n
\n\n ","metadata":{}},{"cell_type":"markdown","source":"The [Coleridge Initiative](https://coleridgeinitiative.org/) is a not-for-profit organization, originally established at New York University, that is working with governments to ensure that data are more effectively used for public decision-making. We achieve this goal by working with government agencies to create value for the taxpayer from the careful use of data, by building new technologies to enable secure access to and sharing of confidential microdata, and by training agency staff to acquire modern data skills.","metadata":{}},{"cell_type":"markdown","source":"
\n

Contents

","metadata":{}},{"cell_type":"markdown","source":"1. [Competition Overview](#competition-overview)\n2. [Preliminaries](#preliminaries)\n3. [Load Datasets](#load-datasets)\n4. [Tabular Exploration](#tabular-exploration)\n5. [Publication Title Exploration](#publication-title-exploration)\n6. [Dataset Title Exploration](#dataset-title-exploration)\n7. [Dataset Label Exploration](#dataset-label-exploration)\n8. [Model](#model)","metadata":{}},{"cell_type":"markdown","source":"
\n

If you find this notebook useful, do give me an upvote, it helps to keep up my motivation.
This notebook will be updated frequently so keep checking for furthur developments.

","metadata":{}},{"cell_type":"markdown","source":"#### If you would have any doubts or need guidance in Data Science you can reach out to me on [LinkedIn](https://www.linkedin.com/in/ishandutta0098)","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Competition Overview

","metadata":{}},{"cell_type":"markdown","source":"### Description\n\nIn this competition, you'll use natural language processing (NLP) to automate the discovery of how scientific data are referenced in publications. Utilizing the full text of scientific publications from numerous research areas gathered from CHORUS publisher members and other sources, you'll identify data sets that the publications' authors used in their work.\n\nIf successful, you'll help support evidence in government data. Automated NLP approaches will enable government agencies and researchers to quickly find the information they need. The approach will be used to develop data usage scorecards to better enable agencies to show how their data are used and bring down a critical barrier to the access and use of public data.","metadata":{}},{"cell_type":"markdown","source":"### Evaluation Criteria\nThe objective of the competition is to identify the mention of datasets within scientific publications. Your predictions will be short excerpts from the publications that appear to note a dataset.\n\nSubmissions are evaluated on a **Jaccard-based FBeta score** between predicted texts and ground truth texts, with `Beta = 0.5` (a `micro F0.5` score). Multiple predictions are delineated with a pipe (|) character in the submission file.\n\nThe following is Python code for calculating the Jaccard score for a single prediction string against a single ground truth string. Note that the overall score for a sample uses Jaccard to compare multiple ground truth and prediction strings that are pipe-delimited - this code does not handle that process or the final `micro F-beta` calculation.","metadata":{}},{"cell_type":"markdown","source":"### Code Requirements\n\n \n\nSubmissions to this competition must be made through Notebooks. In order for the \"Submit\" button to be active after a commit, the following conditions must be met:\n\n- **CPU Notebook** <= 9 hours run-time\n- **GPU Notebook** <= 9 hours run-time\n- **Internet** access disabled\n- Freely & publicly available **external data** is allowed, including pre-trained models\n- **Submission file** must be named submission.csv","metadata":{}},{"cell_type":"markdown","source":"### Data Description\nThe objective of the competition is to identify the mention of datasets within scientific publications. Your predictions will be short excerpts from the **publications** that appear to note a dataset. Predictions that more accurately match the precise words used to identify the dataset within the publication will score higher. Predictions should be cleaned using the `clean_text` function from the [Evaluation page](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation) to ensure proper matching.\n\nPublications are provided in **JSON format**, broken up into sections with section titles.\n\nThe goal in this competition is not just to match known dataset strings but to generalize to datasets that have never been seen before using **NLP** and statistical techniques. A percentage of the public test set publications are drawn from the training set - not all datasets have been identified in train, so these unidentified datasets have been used as a portion of the public test labels. These should serve as guides for the difficult task of labeling the private test set.\n\nNote that the hidden test set has roughly ~8000 publications, many times the size of the public test set. Plan your compute time accordingly.","metadata":{}},{"cell_type":"markdown","source":"### Files\n- **train** - the full text of the training set's publications in JSON format, broken into sections with section titles\n- **test** - the full text of the test set's publications in JSON format, broken into sections with section titles\n- **train.csv** - labels and metadata for the training set\n- **sample_submission.csv** - a sample submission file in the correct format\n\n### Columns\n- `id` - publication id - note that there are multiple rows for some training documents, indicating multiple mentioned datasets\n- `pub_title` - title of the publication (a small number of publications have the same title)\n- `dataset_title` - the title of the dataset that is mentioned within the publication\n- `dataset_label` - a portion of the text that indicates the dataset\n- `cleaned_label` - the dataset_label, as passed through the `clean_text` function from the [Evaluation page](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/overview/evaluation)","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Weights and Biases (W&B)

","metadata":{}},{"cell_type":"markdown","source":"
","metadata":{}},{"cell_type":"markdown","source":"**Weights & Biases** is the machine learning platform for developers to build better models faster. \n\nYou can use W&B's lightweight, interoperable tools to \n- quickly track experiments, \n- version and iterate on datasets, \n- evaluate model performance, \n- reproduce models, \n- visualize results and spot regressions, \n- and share findings with colleagues. \n\nSet up W&B in 5 minutes, then quickly iterate on your machine learning pipeline with the confidence that your datasets and models are tracked and versioned in a reliable system of record.\n\nIn this notebook I will use Weights and Biases's amazing features to perform wonderful visualizations and logging seamlessly. ","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Preliminaries

","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\n# Data Visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set_style(\"darkgrid\")\n\nimport cv2\nfrom wordcloud import WordCloud, STOPWORDS\n\n#Text Color\nfrom termcolor import colored\n\n#Data Preprocessing\nfrom sklearn.preprocessing import StandardScaler, LabelEncoder\nfrom sklearn.model_selection import train_test_split, GridSearchCV, RandomizedSearchCV\n\n#NLP\nfrom sklearn.feature_extraction.text import CountVectorizer\n\n#WordCloud\nfrom wordcloud import WordCloud, STOPWORDS\n\n#Text Processing\nimport nltk\nnltk.download('popular')\n\n#Language Detection\n!pip install langdetect\nimport langdetect\n\n#Sentiment\nfrom textblob import TextBlob\n\n#ner\nimport spacy\n\n#Vectorizer\nfrom sklearn import feature_extraction, manifold\n\n#Word Embedding\nimport gensim.downloader as gensim_api\n\n#Topic Modeling\nimport gensim\n\n# HTML\nfrom IPython.core.display import HTML\n\nimport re\nimport os\nimport gc\nimport glob\nimport json\nimport time\nimport torch\nimport random\nimport datetime\nimport tokenizers\nimport numpy as np\nimport transformers\nimport pandas as pd\nimport torch.nn as nn\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nfrom tokenizers import *\nfrom transformers import *\nfrom functools import partial\nfrom tqdm.notebook import tqdm\nfrom torch.nn import functional as F\nfrom torch.utils.data import Dataset, DataLoader\nfrom sklearn.model_selection import GroupKFold","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-12T08:47:49.523488Z","iopub.execute_input":"2021-12-12T08:47:49.523906Z","iopub.status.idle":"2021-12-12T08:48:06.662469Z","shell.execute_reply.started":"2021-12-12T08:47:49.523817Z","shell.execute_reply":"2021-12-12T08:48:06.661168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Load Datasets

","metadata":{}},{"cell_type":"markdown","source":"We begin with loading the dataset to work upon. Here we have one `.csv` file which has been loaded,","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-12-12T08:48:06.664856Z","iopub.execute_input":"2021-12-12T08:48:06.665312Z","iopub.status.idle":"2021-12-12T08:48:06.847614Z","shell.execute_reply.started":"2021-12-12T08:48:06.665267Z","shell.execute_reply":"2021-12-12T08:48:06.846354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Tabular Exploration

","metadata":{}},{"cell_type":"markdown","source":"### Dataset Head and Info","metadata":{}},{"cell_type":"markdown","source":"Let us quickly have a brief look upon the dataset. Hope to not encounter any NULL values!","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:06.849473Z","iopub.execute_input":"2021-12-12T08:48:06.849905Z","iopub.status.idle":"2021-12-12T08:48:06.882372Z","shell.execute_reply.started":"2021-12-12T08:48:06.849859Z","shell.execute_reply":"2021-12-12T08:48:06.880999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.info()","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:06.884026Z","iopub.execute_input":"2021-12-12T08:48:06.884472Z","iopub.status.idle":"2021-12-12T08:48:06.912866Z","shell.execute_reply.started":"2021-12-12T08:48:06.884410Z","shell.execute_reply":"2021-12-12T08:48:06.911579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Aha! There are no NULL values, so we are good to go.","metadata":{}},{"cell_type":"markdown","source":"### Dataset Size","metadata":{}},{"cell_type":"code","source":"print(f\"Training Dataset Shape: {colored(train.shape, 'yellow')}\")","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:06.917764Z","iopub.execute_input":"2021-12-12T08:48:06.918218Z","iopub.status.idle":"2021-12-12T08:48:06.924774Z","shell.execute_reply.started":"2021-12-12T08:48:06.918170Z","shell.execute_reply":"2021-12-12T08:48:06.923342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Column-wise Unique Values","metadata":{}},{"cell_type":"code","source":"for col in train.columns:\n print(col + \":\" + colored(str(len(train[col].unique())), 'yellow'))","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:06.927985Z","iopub.execute_input":"2021-12-12T08:48:06.928383Z","iopub.status.idle":"2021-12-12T08:48:06.958444Z","shell.execute_reply.started":"2021-12-12T08:48:06.928347Z","shell.execute_reply":"2021-12-12T08:48:06.956922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- The Training Dataset has **19,661** samples but it looks like there are only **14,316** unique IDs in the dataset, meaning that some publications include a multitude of datasets. \n- Also, notice that the `pub_title` unique count is slightly smaller than the `Id` unique counts. This points to the precense of several occurences of having 2 separate publications, each with a unique ID, but sharing the exact same title.\n- Also, there are a total of **45 unique** `dataset_title` and **130** `unique dataset_label`. Meaning that a single dataset could have multible labels throughout different publications.","metadata":{}},{"cell_type":"markdown","source":"From here onwards for the next part, we shall have a little deeper look at the dataset. We move ahead trying to understand each column one by one.","metadata":{}},{"cell_type":"markdown","source":"\n
\n

Publication Title Exploration

","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"train['pub_title']","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:06.960340Z","iopub.execute_input":"2021-12-12T08:48:06.960855Z","iopub.status.idle":"2021-12-12T08:48:06.972122Z","shell.execute_reply.started":"2021-12-12T08:48:06.960804Z","shell.execute_reply":"2021-12-12T08:48:06.970655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Wordcloud","metadata":{}},{"cell_type":"markdown","source":"Word Cloud is a data visualization technique used for representing text data in which the size of each word indicates its frequency or importance. Significant textual data points can be highlighted using a word cloud. Word clouds are widely used for analyzing data from social network websites.","metadata":{}},{"cell_type":"markdown","source":"We will write a simple and intuitive function `plot_wordcloud` that will help us plot wordclouds with ease. ","metadata":{}},{"cell_type":"code","source":"def plot_wordcloud(column, title):\n \n \"\"\"\n Function to Plot Wordcloud of given dataframe column.\n \n params: column(string): The Column of the DataFrame for plotting.\n title(string) : The Title of the Wordcloud.\n \"\"\"\n # Define stopwords\n stopwords = set(STOPWORDS) \n \n # Define the Wordcloud \n wordcloud = WordCloud(width = 800, \n height = 800,\n background_color ='black',\n min_font_size = 10,\n stopwords = stopwords).generate(' '.join(train[column])) \n\n # Plot the WordCloud image \n plt.figure(figsize = (8, 8), facecolor = None) \n plt.imshow(wordcloud) \n plt.axis(\"off\") \n plt.tight_layout(pad = 0) \n plt.title('Wordcloud: ' + title, fontsize = 20)\n\n plt.show() ","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:06.974144Z","iopub.execute_input":"2021-12-12T08:48:06.974542Z","iopub.status.idle":"2021-12-12T08:48:06.984569Z","shell.execute_reply.started":"2021-12-12T08:48:06.974506Z","shell.execute_reply":"2021-12-12T08:48:06.983225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_wordcloud(column = 'pub_title', title = 'Publication Title')","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:06.986264Z","iopub.execute_input":"2021-12-12T08:48:06.986717Z","iopub.status.idle":"2021-12-12T08:48:10.211976Z","shell.execute_reply.started":"2021-12-12T08:48:06.986672Z","shell.execute_reply":"2021-12-12T08:48:10.209875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Basic NLP","metadata":{}},{"cell_type":"markdown","source":"To understand the column better we need to clean it first. For this purpose we write another intuitive function `preprocess_text`.","metadata":{}},{"cell_type":"code","source":"def preprocess_text(text, flg_stemm=False, flg_lemm=True):\n \n \"\"\"\n Function to Preprocess the Text by removing extra spaces and stopwords.\n It also performs Stemming or Lemmatization.\n \n params: text(string) : The Text which has to be cleaned.\n flg_stemm(bool): Flag for Stemming\n flg_lemm(bool) : Flag for Lemmatization\n \n returns: text(string) : Returns the Cleaned Text\n \"\"\"\n\n lst_stopwords = nltk.corpus.stopwords.words(\"english\")\n \n ## clean (convert to lowercase and remove punctuations and characters and then strip)\n text = re.sub(r'[^\\w\\s]', '', str(text).lower().strip())\n \n ## Tokenize (convert from string to list)\n lst_text = text.split()\n ## remove Stopwords\n if lst_stopwords is not None:\n lst_text = [word for word in lst_text if word not in \n lst_stopwords]\n \n ## Stemming (remove -ing, -ly, ...)\n if flg_stemm == True:\n ps = nltk.stem.porter.PorterStemmer()\n lst_text = [ps.stem(word) for word in lst_text]\n \n ## Lemmatisation (convert the word into root word)\n if flg_lemm == True:\n lem = nltk.stem.wordnet.WordNetLemmatizer() \n lst_text = [lem.lemmatize(word) for word in lst_text]\n \n ## back to string from list\n text = \" \".join(lst_text)\n return text","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:10.213249Z","iopub.execute_input":"2021-12-12T08:48:10.213557Z","iopub.status.idle":"2021-12-12T08:48:10.222078Z","shell.execute_reply.started":"2021-12-12T08:48:10.213530Z","shell.execute_reply":"2021-12-12T08:48:10.221133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that we have our function, let's clean the column and calculate some stats!","metadata":{}},{"cell_type":"code","source":"#Clean Publication Title\ntrain[\"clean_pub_title\"] = train[\"pub_title\"].apply(lambda x: preprocess_text(x, flg_stemm=False, flg_lemm=True, ))\n\n#Length\ntrain['clean_pub_title_len'] = train['clean_pub_title'].apply(lambda x: len(x))\n\n#Word Count\ntrain['clean_pub_title_word_count'] =train[\"clean_pub_title\"].apply(lambda x: len(str(x).split(\" \")))\n\n#Character Count\ntrain['clean_pub_title_char_count'] = train[\"clean_pub_title\"].apply(lambda x: sum(len(word) for word in str(x).split(\" \")))\n\n#Average Word Length\ntrain['clean_pub_title_avg_word_length'] = train['clean_pub_title_char_count'] / train['clean_pub_title_word_count']","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:10.223120Z","iopub.execute_input":"2021-12-12T08:48:10.223581Z","iopub.status.idle":"2021-12-12T08:48:16.932990Z","shell.execute_reply.started":"2021-12-12T08:48:10.223538Z","shell.execute_reply":"2021-12-12T08:48:16.932049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Distribution Plots","metadata":{}},{"cell_type":"markdown","source":"A good way to understand the distribution of our calculated stats is by plotting a distribution plot. We again write a function `plot_distribution` which will save us a lot of time.","metadata":{}},{"cell_type":"code","source":"def plot_distribution(x, title):\n \n \"\"\"\n Function to obtain the distribution plot of given data.\n \n params: x(string) : Name of the Column for the Plot.\n title(string) : Title of the Plot\n \"\"\"\n sns.displot(train, x = x, kind=\"kde\", bw_adjust=2)\n\n plt.title(title, fontsize = 15)\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:16.934088Z","iopub.execute_input":"2021-12-12T08:48:16.934546Z","iopub.status.idle":"2021-12-12T08:48:16.940317Z","shell.execute_reply.started":"2021-12-12T08:48:16.934513Z","shell.execute_reply":"2021-12-12T08:48:16.939369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Instead of calling the function 4 times, we imporovise and make a list of tuples. These tuples of the form (column, title) contain the inputs of the function `plot_distribution`.","metadata":{}},{"cell_type":"code","source":"# List of Tuples for Plot Distribution\n\npub_title_list = [(\"clean_pub_title_len\", \"Publication Title: Length Distribution\"),\n (\"clean_pub_title_word_count\", \"Publication Title: Word Count Distribution\"),\n (\"clean_pub_title_char_count\", \"Publication Title: Character Count Distribution\"),\n (\"clean_pub_title_avg_word_length\", \"Publication Title: Average Word Length Distribution\")]","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:16.942051Z","iopub.execute_input":"2021-12-12T08:48:16.942336Z","iopub.status.idle":"2021-12-12T08:48:16.957963Z","shell.execute_reply.started":"2021-12-12T08:48:16.942308Z","shell.execute_reply":"2021-12-12T08:48:16.957133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for column, title in pub_title_list:\n plot_distribution(x = column, title = title) ","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:16.959339Z","iopub.execute_input":"2021-12-12T08:48:16.959923Z","iopub.status.idle":"2021-12-12T08:48:19.018305Z","shell.execute_reply.started":"2021-12-12T08:48:16.959817Z","shell.execute_reply":"2021-12-12T08:48:19.017132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Dataset Title Exploration

","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"train['dataset_title']","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:19.019498Z","iopub.execute_input":"2021-12-12T08:48:19.019770Z","iopub.status.idle":"2021-12-12T08:48:19.028736Z","shell.execute_reply.started":"2021-12-12T08:48:19.019744Z","shell.execute_reply":"2021-12-12T08:48:19.027490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Wordcloud","metadata":{}},{"cell_type":"code","source":"plot_wordcloud(column = 'dataset_title', title = 'Dataset Title')","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:19.030397Z","iopub.execute_input":"2021-12-12T08:48:19.030826Z","iopub.status.idle":"2021-12-12T08:48:20.538250Z","shell.execute_reply.started":"2021-12-12T08:48:19.030785Z","shell.execute_reply":"2021-12-12T08:48:20.536971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Basic NLP","metadata":{}},{"cell_type":"code","source":"#Clean Dataset Title\ntrain[\"clean_dataset_title\"] = train[\"dataset_title\"].apply(lambda x: preprocess_text(x, flg_stemm=False, flg_lemm=True, ))\n\n#Length\ntrain['clean_dataset_title_len'] = train['clean_dataset_title'].apply(lambda x: len(x))\n\n#Word Count\ntrain['clean_dataset_title_word_count'] =train[\"clean_dataset_title\"].apply(lambda x: len(str(x).split(\" \")))\n\n#Character Count\ntrain['clean_dataset_title_char_count'] = train[\"clean_dataset_title\"].apply(lambda x: sum(len(word) for word in str(x).split(\" \")))\n\n#Average Word Length\ntrain['clean_dataset_title_avg_word_length'] = train['clean_dataset_title_char_count'] / train['clean_dataset_title_word_count']","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:20.539810Z","iopub.execute_input":"2021-12-12T08:48:20.540211Z","iopub.status.idle":"2021-12-12T08:48:23.977359Z","shell.execute_reply.started":"2021-12-12T08:48:20.540161Z","shell.execute_reply":"2021-12-12T08:48:23.976557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Distribution Plots","metadata":{}},{"cell_type":"code","source":"# List of Tuples for Plot Distribution\n\ndataset_title_list = [(\"clean_dataset_title_len\", \"Dataset Title: Length Distribution\"),\n (\"clean_dataset_title_word_count\", \"Dataset Title: Word Count Distribution\"),\n (\"clean_dataset_title_char_count\", \"Dataset Title: Character Count Distribution\"),\n (\"clean_dataset_title_avg_word_length\", \"Dataset Title: Average Word Length Distribution\")]","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:23.978461Z","iopub.execute_input":"2021-12-12T08:48:23.978890Z","iopub.status.idle":"2021-12-12T08:48:23.983655Z","shell.execute_reply.started":"2021-12-12T08:48:23.978860Z","shell.execute_reply":"2021-12-12T08:48:23.982329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for column, title in dataset_title_list:\n plot_distribution(x = column, title = title)","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:23.984934Z","iopub.execute_input":"2021-12-12T08:48:23.985235Z","iopub.status.idle":"2021-12-12T08:48:26.027652Z","shell.execute_reply.started":"2021-12-12T08:48:23.985207Z","shell.execute_reply":"2021-12-12T08:48:26.025920Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Dataset Label Exploration

","metadata":{}},{"cell_type":"code","source":"train['dataset_label']","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:26.029647Z","iopub.execute_input":"2021-12-12T08:48:26.030343Z","iopub.status.idle":"2021-12-12T08:48:26.041280Z","shell.execute_reply.started":"2021-12-12T08:48:26.030289Z","shell.execute_reply":"2021-12-12T08:48:26.039603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Wordcloud","metadata":{}},{"cell_type":"code","source":"plot_wordcloud(column = 'dataset_label', title = 'Dataset Label')","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:26.043474Z","iopub.execute_input":"2021-12-12T08:48:26.043794Z","iopub.status.idle":"2021-12-12T08:48:28.200410Z","shell.execute_reply.started":"2021-12-12T08:48:26.043765Z","shell.execute_reply":"2021-12-12T08:48:28.199442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Basic NLP","metadata":{}},{"cell_type":"code","source":"#Clean Dataset Label\ntrain[\"clean_dataset_label\"] = train[\"dataset_label\"].apply(lambda x: preprocess_text(x, flg_stemm=False, flg_lemm=True, ))\n\n#Length\ntrain['clean_dataset_label_len'] = train['clean_dataset_label'].apply(lambda x: len(x))\n\n#Word Count\ntrain['clean_dataset_label_word_count'] =train[\"clean_dataset_label\"].apply(lambda x: len(str(x).split(\" \")))\n\n#Character Count\ntrain['clean_dataset_label_char_count'] = train[\"clean_dataset_label\"].apply(lambda x: sum(len(word) for word in str(x).split(\" \")))\n\n#Average Word Length\ntrain['clean_dataset_label_avg_word_length'] = train['clean_dataset_label_char_count'] / train['clean_dataset_label_word_count']","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:28.201908Z","iopub.execute_input":"2021-12-12T08:48:28.202465Z","iopub.status.idle":"2021-12-12T08:48:31.483324Z","shell.execute_reply.started":"2021-12-12T08:48:28.202408Z","shell.execute_reply":"2021-12-12T08:48:31.482527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Distribution Plots","metadata":{}},{"cell_type":"code","source":"# List of Tuples for Plot Distribution\n\ndataset_label_list = [(\"clean_dataset_label_len\", \"Dataset Label: Length Distribution\"),\n (\"clean_dataset_label_word_count\", \"Dataset Label: Word Count Distribution\"),\n (\"clean_dataset_label_char_count\", \"Dataset Label: Character Count Distribution\"),\n (\"clean_dataset_label_avg_word_length\", \"Dataset Label: Average Word Length Distribution\")]","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:31.484520Z","iopub.execute_input":"2021-12-12T08:48:31.485006Z","iopub.status.idle":"2021-12-12T08:48:31.489365Z","shell.execute_reply.started":"2021-12-12T08:48:31.484961Z","shell.execute_reply":"2021-12-12T08:48:31.487897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for column, title in dataset_label_list:\n plot_distribution(x = column, title = title)","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:48:31.490645Z","iopub.execute_input":"2021-12-12T08:48:31.490930Z","iopub.status.idle":"2021-12-12T08:48:33.425168Z","shell.execute_reply.started":"2021-12-12T08:48:31.490893Z","shell.execute_reply":"2021-12-12T08:48:33.423919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n
\n

Model

","metadata":{}},{"cell_type":"markdown","source":"## Imports","metadata":{}},{"cell_type":"code","source":"import wandb\nwandb.login()","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:13:02.693726Z","iopub.execute_input":"2021-12-12T09:13:02.694059Z","iopub.status.idle":"2021-12-12T09:13:34.243742Z","shell.execute_reply.started":"2021-12-12T09:13:02.694027Z","shell.execute_reply":"2021-12-12T09:13:34.242759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Params","metadata":{}},{"cell_type":"code","source":"SEED = 2020\n\nDATA_PATH = \"../input/coleridgeinitiative-show-us-the-data/\"\nDATA_PATH_TRAIN = DATA_PATH + 'train/'\nDATA_PATH_TEST = DATA_PATH + 'test/'\n\nNUM_WORKERS = 4\n\nVOCABS = {\n \"bert-base-uncased\": \"../input/vocabs/bert-base-uncased-vocab.txt\",\n}\n\nMODEL_PATHS = {\n 'bert-base-uncased': '../input/bertconfigs/uncased_L-12_H-768_A-12/uncased_L-12_H-768_A-12/',\n 'bert-large-uncased-whole-word-masking-finetuned-squad': '../input/bertconfigs/wwm_uncased_L-24_H-1024_A-16/wwm_uncased_L-24_H-1024_A-16/',\n 'albert-large-v2': '../input/albert-configs/albert-large-v2/albert-large-v2/',\n 'albert-base-v2': '../input/albert-configs/albert-base-v2/albert-base-v2/',\n 'distilbert': '../input/albert-configs/distilbert/distilbert/',\n}","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:56:31.817496Z","iopub.execute_input":"2021-12-12T08:56:31.818066Z","iopub.status.idle":"2021-12-12T08:56:31.824689Z","shell.execute_reply.started":"2021-12-12T08:56:31.818015Z","shell.execute_reply":"2021-12-12T08:56:31.824009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# wandb config\nWANDB_CONFIG = {\n 'competition': 'Coleridge', \n '_wandb_kernel': 'neuracort'\n }","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:13:38.118224Z","iopub.execute_input":"2021-12-12T09:13:38.118564Z","iopub.status.idle":"2021-12-12T09:13:38.122730Z","shell.execute_reply.started":"2021-12-12T09:13:38.118534Z","shell.execute_reply":"2021-12-12T09:13:38.121884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Make Data","metadata":{}},{"cell_type":"code","source":"def load_text(id_, root=\"\"):\n with open(os.path.join(root, id_ + \".json\")) as f:\n text = json.load(f)\n return text\n\n\ndef clean_text(txt):\n return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()\n\n\ndef create_data():\n new_df = []\n\n for idx in tqdm(range(len(df))):\n article = load_text(df['Id'][idx], DATA_PATH_TRAIN)\n id_, pub_title, dataset_title, dataset_label, cleaned_label = df.iloc[idx]\n \n \n for i, section in enumerate(article):\n text = section['text']\n title = section['section_title']\n \n cleaned_text = clean_text(section['text'])\n \n found = cleaned_label in cleaned_text\n \n dic = {\n \"id\": [id_], \n \"section_id\": [i],\n \"pub_title\": [pub_title], \n \"dataset_title\": [dataset_title], \n \"dataset_label\": [dataset_label], \n \"cleaned_label\": [cleaned_label],\n \"text\": [text],\n \"cleaned_text\": [cleaned_text],\n \"label_found\": [found],\n }\n new_df.append(pd.DataFrame.from_dict(dic))\n \n return pd.concat(new_df).reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2021-12-12T08:56:31.826656Z","iopub.execute_input":"2021-12-12T08:56:31.827431Z","iopub.status.idle":"2021-12-12T08:56:31.837916Z","shell.execute_reply.started":"2021-12-12T08:56:31.827392Z","shell.execute_reply":"2021-12-12T08:56:31.837207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(DATA_PATH + 'train.csv')\n\nnew_df = create_data()","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:00:03.800282Z","iopub.execute_input":"2021-12-12T09:00:03.800609Z","iopub.status.idle":"2021-12-12T09:11:46.232123Z","shell.execute_reply.started":"2021-12-12T09:00:03.800577Z","shell.execute_reply":"2021-12-12T09:11:46.231209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = new_df[new_df['label_found']].reset_index(drop=True)\n\ndf['length'] = df['cleaned_text'].apply(lambda x: len(x.split()))\ndf = df[df['length'] < 3000] # remove too long texts\n\ndf.to_csv(\"df_train.csv\", index=False) # saving, just in case","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:11:46.242153Z","iopub.execute_input":"2021-12-12T09:11:46.242497Z","iopub.status.idle":"2021-12-12T09:11:59.670685Z","shell.execute_reply.started":"2021-12-12T09:11:46.242459Z","shell.execute_reply":"2021-12-12T09:11:59.669878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Utils","metadata":{}},{"cell_type":"code","source":"def seed_everything(seed):\n \"\"\"\n Seeds basic parameters for reproductibility of results.\n\n Args:\n seed (int): Number of the seed.\n \"\"\"\n\n random.seed(seed)\n os.environ[\"PYTHONHASHSEED\"] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = False\n\n\ndef save_model_weights(model, filename, verbose=1, cp_folder=\"\"):\n \"\"\"\n Saves the weights of a PyTorch model.\n\n Args:\n model (torch model): Model to save the weights of.\n filename (str): Name of the checkpoint.\n verbose (int, optional): Whether to display infos. Defaults to 1.\n cp_folder (str, optional): Folder to save to. Defaults to \"\".\n \"\"\"\n\n if verbose:\n print(f\"\\n -> Saving weights to {os.path.join(cp_folder, filename)}\\n\")\n torch.save(model.state_dict(), os.path.join(cp_folder, filename))\n\n\ndef count_parameters(model, all=False):\n \"\"\"\n Count the parameters of a model.\n\n Args:\n model (torch model): Model to count the parameters of.\n all (bool, optional): Whether to count not trainable parameters. Defaults to False.\n\n Returns:\n int: Number of parameters.\n \"\"\"\n\n if all:\n return sum(p.numel() for p in model.parameters())\n else:\n return sum(p.numel() for p in model.parameters() if p.requires_grad)","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:12:34.723169Z","iopub.execute_input":"2021-12-12T09:12:34.723564Z","iopub.status.idle":"2021-12-12T09:12:34.742590Z","shell.execute_reply.started":"2021-12-12T09:12:34.723526Z","shell.execute_reply":"2021-12-12T09:12:34.741713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data","metadata":{}},{"cell_type":"code","source":"class EncodedText:\n def __init__(self, ids, offsets):\n self.ids = ids\n self.offsets = offsets\n\n\ndef create_tokenizer_and_tokens(config):\n if \"roberta\" in config.selected_model:\n raise NotImplementedError\n \n elif \"albert\" in config.selected_model:\n raise NotImplementedError\n \n else:\n tokenizer = BertWordPieceTokenizer(\n MODEL_PATHS[config.selected_model] + 'vocab.txt',\n lowercase=config.lowercase,\n )\n\n tokens = {\n 'cls': tokenizer.token_to_id('[CLS]'),\n 'sep': tokenizer.token_to_id('[SEP]'),\n 'pad': tokenizer.token_to_id('[PAD]'),\n }\n \n return tokenizer, tokens\n\ndef locate_label_string(text, label):\n \"\"\"\n Finds the label in the text\n \"\"\"\n len_label = len(label) - 1\n\n candidates_idx = [i for i, e in enumerate(text) if e == label[1]]\n for idx in candidates_idx:\n if \" \" + text[idx: idx + len_label] == label:\n idx_start = idx\n idx_end = idx + len_label\n break\n\n assert (\n text[idx_start:idx_end] == label[1:]\n ), f'\"{text[idx_start: idx_end]}\" instead of \"{label}\" in \"{text}\"'\n\n char_targets = np.zeros(len(text))\n char_targets[idx_start:idx_end] = 1\n\n return idx_start, idx_end, char_targets\n\n\ndef locate_label_tokens(offsets, char_targets):\n \"\"\"\n Finds the tokens corresponding to the found labels\n \"\"\"\n target_idx = []\n for idx, (offset1, offset2) in enumerate(offsets):\n if sum(char_targets[offset1:offset2]) > 0:\n target_idx.append(idx)\n\n if not len(target_idx):\n for idx, (offset1, offset2) in enumerate(offsets):\n if sum(char_targets[offset1:offset2]) > 0:\n target_idx.append(idx)\n\n return target_idx[0], target_idx[-1]\n\ndef process_data(\n text,\n label,\n tokenizer,\n tokens,\n max_len=100,\n model_name=\"bert\",\n):\n \"\"\"\n Prepares the data for the question answering task.\n Adapted from Abishek's work on the Tweet Sentiment extraction competition, \n check his work for more details !\n \"\"\"\n target_start, target_end = 0, 0\n text = \" \" + \" \".join(str(text).split())\n label = \" \" + \" \".join(str(label).split())\n\n if label != \" \":\n idx_start, idx_end, char_targets = locate_label_string(\n text, label\n )\n\n tokenized = tokenizer.encode(text)\n input_ids_text = tokenized.ids[1:-1]\n\n # print(input_ids_text, len(input_ids_text))\n\n offsets = tokenized.offsets[1:-1]\n\n if label != \" \":\n target_start, target_end = locate_label_tokens(offsets, char_targets)\n\n if target_end >= max_len - 2: # target is too far in the sentence, we crop its beginning.\n n_tok_to_crop = target_start - max_len // 2\n new_str_start = offsets[n_tok_to_crop][0]\n\n input_ids_text = input_ids_text[n_tok_to_crop:]\n\n offsets = [tuple(t) for t in np.array(offsets[n_tok_to_crop:]) - new_str_start]\n text = text[new_str_start:]\n\n target_start -= n_tok_to_crop\n target_end -= n_tok_to_crop\n\n input_ids = (\n [tokens[\"cls\"]]\n + input_ids_text[:max_len - 2]\n + [tokens[\"sep\"]]\n )\n\n if \"roberta\" in model_name:\n token_type_ids = [0] * len(input_ids)\n else:\n token_type_ids = [1] * len(input_ids)\n\n text_offsets = [(0, 0)] + offsets[:max_len - 2] + [(0, 0)]\n\n target_start += 1\n target_end += 1\n\n # target_end = min(target_end, max_len - 1)\n\n assert len(input_ids) == len(token_type_ids) and len(input_ids) == len(text_offsets), (len(input_ids), len(text_offsets)) # noqa\n\n padding_length = max_len - len(input_ids)\n if padding_length > 0:\n input_ids = input_ids + ([tokens[\"pad\"]] * padding_length)\n token_type_ids = token_type_ids + ([0] * padding_length)\n text_offsets = text_offsets + ([(0, 0)] * padding_length)\n\n return {\n \"ids\": input_ids,\n \"token_type_ids\": token_type_ids,\n \"targets_start\": target_start,\n \"targets_end\": target_end,\n \"text\": text,\n \"label\": label,\n \"offsets\": text_offsets,\n }","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:12:37.143862Z","iopub.execute_input":"2021-12-12T09:12:37.144240Z","iopub.status.idle":"2021-12-12T09:12:37.167080Z","shell.execute_reply.started":"2021-12-12T09:12:37.144208Z","shell.execute_reply":"2021-12-12T09:12:37.166062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dataset","metadata":{}},{"cell_type":"code","source":"class SectionDataset(Dataset):\n def __init__(\n self,\n df,\n tokenizer,\n tokens,\n max_len=512,\n model_name=\"bert\",\n ):\n self.tokens = tokens\n self.tokenizer = tokenizer\n self.max_len = max_len\n self.model_name = model_name\n\n self.texts = df[\"cleaned_text\"].values\n self.labels = df[\"cleaned_label\"].values\n\n def __len__(self):\n return len(self.texts)\n\n def __getitem__(self, idx):\n data = process_data(\n self.texts[idx],\n self.labels[idx],\n self.tokenizer,\n self.tokens,\n max_len=self.max_len,\n model_name=self.model_name,\n )\n\n return {\n \"ids\": torch.tensor(data[\"ids\"], dtype=torch.long),\n \"token_type_ids\": torch.tensor(data[\"token_type_ids\"], dtype=torch.long),\n \"target_start\": torch.tensor(data[\"targets_start\"], dtype=torch.long),\n \"target_end\": torch.tensor(data[\"targets_end\"], dtype=torch.long),\n \"text\": data[\"text\"],\n \"label\": data[\"label\"],\n \"offsets\": torch.tensor(data[\"offsets\"], dtype=torch.long),\n }","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:12:41.163364Z","iopub.execute_input":"2021-12-12T09:12:41.163724Z","iopub.status.idle":"2021-12-12T09:12:41.172846Z","shell.execute_reply.started":"2021-12-12T09:12:41.163690Z","shell.execute_reply":"2021-12-12T09:12:41.171850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model","metadata":{}},{"cell_type":"code","source":"TRANSFORMERS = {\n \"roberta-base\": (RobertaModel, \"roberta-base\"),\n 'albert-base-v2': (AlbertModel, 'albert-base-v2'),\n 'albert-large-v2': (AlbertModel, 'albert-large-v2'),\n 'albert-xlarge-v2': (AlbertModel, 'albert-xlarge-v2'),\n 'albert-xxlarge-v2': (AlbertModel, 'albert-xxlarge-v2'),\n \"bert-base-uncased\": (BertModel, \"bert-base-uncased\"),\n \"bert-base-cased\": (BertModel, \"bert-base-cased\"),\n \"bert-large-uncased-whole-word-masking\": (BertModel, \"bert-large-uncased-whole-word-masking\"),\n \"distilbert-base-uncased-distilled-squad\": (\n DistilBertModel,\n \"distilbert-base-uncased-distilled-squad\"\n )\n}\n\n\nclass QATransformer(nn.Module):\n def __init__(self, model):\n super().__init__()\n self.name = model\n\n self.pad_idx = 1 if \"roberta\" in self.name else 0\n\n model_class, pretrained_weights = TRANSFORMERS[model]\n\n self.transformer = model_class.from_pretrained(\n pretrained_weights, output_hidden_states=True\n )\n\n self.nb_features = self.transformer.pooler.dense.out_features\n\n self.logits = nn.Sequential(\n nn.Linear(self.nb_features, self.nb_features),\n nn.Tanh(),\n nn.Linear(self.nb_features, 2),\n )\n\n def forward(self, tokens, token_type_ids):\n \"\"\"\n Usual torch forward function\n\n Arguments:\n tokens {torch tensor} -- Sentence tokens\n token_type_ids {torch tensor} -- Sentence tokens ids\n \"\"\"\n\n hidden_states = self.transformer(\n tokens,\n attention_mask=(tokens != self.pad_idx).long(),\n token_type_ids=token_type_ids,\n )[-1]\n\n features = hidden_states[-1]\n logits = self.logits(features)\n\n start_logits, end_logits = logits[:, :, 0], logits[:, :, 1]\n\n return start_logits, end_logits","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:12:44.500147Z","iopub.execute_input":"2021-12-12T09:12:44.500473Z","iopub.status.idle":"2021-12-12T09:12:44.510455Z","shell.execute_reply.started":"2021-12-12T09:12:44.500441Z","shell.execute_reply":"2021-12-12T09:12:44.509575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Metric","metadata":{}},{"cell_type":"code","source":"def jaccard_similarity(s1, s2): # could be wrong, see CPMP's thread\n l1 = s1.split(\" \")\n l2 = s2.split(\" \")\n intersection = len(list(set(l1).intersection(l2)))\n union = (len(l1) + len(l2)) - intersection\n return float(intersection) / union\n\n\ndef compute_score(y_true, y_pred, beta=0.5):\n \"\"\"\n From https://www.kaggle.com/tungmphung/coleridge-initiative-local-score-computation/\n \"\"\"\n TP, FP, FN = 0, 0, 0\n\n for truth, pred in zip(y_true, y_pred):\n true_datasets = truth.split('|')\n # Predicted strings for each publication are sorted alphabetically\n # and processed in that order.\n pred_datasets = sorted(pred.split('|'))\n\n for true_dataset in true_datasets:\n if len(pred_datasets):\n match_scores = [jaccard_similarity(true_dataset, pred_dataset)\n for pred_dataset in pred_datasets]\n # The prediction with the highest score for a given ground truth\n # is matched with that ground truth.\n match_index = np.argmax(match_scores)\n\n if match_scores[match_index] >= 0.5:\n # Any matched predictions where the Jaccard score meets or\n # exceeds the threshold of 0.5 are counted as true positives (TP),\n TP += 1\n else:\n # the remainder as false positives (FP).\n FP += 1\n\n del(pred_datasets[match_index])\n else:\n # Any ground truths with no nearest predictions are counted as\n # false negatives (FN).\n FN += 1\n # Any unmatched predictions are counted as false positives (FP).\n FP += len(pred_datasets)\n\n precision = TP / (TP + FP)\n recall = TP / (TP + FN)\n f_score = (1 + beta**2)*(precision*recall)/((beta**2)*precision + recall)\n\n return f_score","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:12:46.461402Z","iopub.execute_input":"2021-12-12T09:12:46.461743Z","iopub.status.idle":"2021-12-12T09:12:46.471144Z","shell.execute_reply.started":"2021-12-12T09:12:46.461711Z","shell.execute_reply":"2021-12-12T09:12:46.470078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_string_from_idx(text, idx_start, idx_end, offsets):\n \"\"\"\n Uses the offsets to retrieve the predicted string based on the start and end indices\n \"\"\"\n if idx_end < idx_start:\n idx_end = idx_start\n\n predicted_string = \"\"\n for i in range(idx_start, idx_end + 1):\n predicted_string += text[offsets[i][0]: offsets[i][1]]\n if i + 1 < len(offsets) and offsets[i][1] < offsets[i + 1][0]:\n predicted_string += \" \"\n\n return predicted_string\n\ndef get_pred_from_logits(data, start_logits, end_logits, from_proba=False):\n if not from_proba:\n start_logits = torch.softmax(start_logits, dim=1).cpu().detach().numpy()\n end_logits = torch.softmax(end_logits, dim=1).cpu().detach().numpy()\n\n offsets = data[\"offsets\"].cpu().numpy()\n\n preds = []\n for i in range(len(start_logits)):\n start_idx = np.argmax(start_logits[i])\n end_idx = np.argmax(end_logits[i])\n preds.append(get_string_from_idx(data[\"text\"][i], start_idx, end_idx, offsets[i]))\n\n return preds","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:12:47.120202Z","iopub.execute_input":"2021-12-12T09:12:47.120539Z","iopub.status.idle":"2021-12-12T09:12:47.129805Z","shell.execute_reply.started":"2021-12-12T09:12:47.120508Z","shell.execute_reply":"2021-12-12T09:12:47.128964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def ce_loss(\n pred, truth, smoothing=False, trg_pad_idx=-1, eps=0.1\n):\n \"\"\"\n Computes the cross entropy loss with label smoothing\n\n Args:\n pred (torch tensor): Prediction\n truth (torch tensor): Target\n smoothing (bool, optional): Whether to use smoothing. Defaults to False.\n trg_pad_idx (int, optional): Indices to ignore in the loss. Defaults to -1.\n eps (float, optional): Smoothing coefficient. Defaults to 0.1.\n\n Returns:\n [type]: [description]\n \"\"\"\n truth = truth.contiguous().view(-1)\n\n one_hot = torch.zeros_like(pred).scatter(1, truth.view(-1, 1), 1)\n\n if smoothing:\n n_class = pred.size(1)\n one_hot = one_hot * (1 - eps) + (1 - one_hot) * eps / (n_class - 1)\n\n loss = -one_hot * F.log_softmax(pred, dim=1)\n\n if trg_pad_idx >= 0:\n loss = loss.sum(dim=1)\n non_pad_mask = truth.ne(trg_pad_idx)\n loss = loss.masked_select(non_pad_mask)\n\n return loss.sum()\n\n\ndef qa_loss_fn(start_logits, end_logits, start_positions, end_positions, config):\n \"\"\"\n Loss function for the question answering task.\n It is the sum of the cross entropy for the start and end logits\n\n Args:\n start_logits (torch tensor): Start logits\n end_logits (torch tensor): End logits\n start_positions (torch tensor): Start ground truth\n end_positions (torch tensor): End ground truth\n config (dict): Dictionary of parameters for the CE Loss.\n\n Returns:\n torch tensor: Loss value\n \"\"\"\n bs = start_logits.size(0)\n\n start_loss = ce_loss(\n start_logits,\n start_positions,\n smoothing=config[\"smoothing\"],\n eps=config[\"eps\"],\n )\n\n end_loss = ce_loss(\n end_logits,\n end_positions,\n smoothing=config[\"smoothing\"],\n eps=config[\"eps\"],\n )\n\n total_loss = start_loss + end_loss\n\n return total_loss / bs","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:12:47.373190Z","iopub.execute_input":"2021-12-12T09:12:47.373508Z","iopub.status.idle":"2021-12-12T09:12:47.383419Z","shell.execute_reply.started":"2021-12-12T09:12:47.373475Z","shell.execute_reply":"2021-12-12T09:12:47.382476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def trim_tensors(tokens, input_ids, model_name='bert', min_len=10):\n \"\"\"\n Trim tensors so that within a batch, padding is shortened.\n This speeds up training for RNNs and Transformers\n\n Arguments:\n tokens {torch tensor} -- Text tokens\n\n Keyword Arguments:\n min_len {int} -- Minimum length to trim to (default: {10})\n\n Returns:\n torch tensor -- trimmed tokens\n \"\"\"\n pad_token = 1 if \"roberta\" in model_name else 0\n max_len = max(torch.max(torch.sum((tokens != pad_token), 1)), min_len)\n return tokens[:, :max_len], input_ids[:, :max_len]","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:12:47.735779Z","iopub.execute_input":"2021-12-12T09:12:47.736148Z","iopub.status.idle":"2021-12-12T09:12:47.741890Z","shell.execute_reply.started":"2021-12-12T09:12:47.736095Z","shell.execute_reply":"2021-12-12T09:12:47.740814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Fit","metadata":{}},{"cell_type":"code","source":"def fit(\n model,\n train_dataset,\n val_dataset,\n loss_config,\n epochs=5,\n batch_size=8,\n weight_decay=0,\n warmup_prop=0.0,\n lr=5e-4,\n):\n\n train_loader = DataLoader(\n train_dataset, batch_size=batch_size, num_workers=NUM_WORKERS, shuffle=True,\n )\n val_loader = DataLoader(\n val_dataset, batch_size=batch_size, shuffle=False, num_workers=NUM_WORKERS\n )\n\n opt_params = []\n no_decay = [\"bias\", \"LayerNorm.bias\", \"LayerNorm.weight\"]\n for n, p in model.named_parameters():\n wd = 0 if any(nd in n for nd in no_decay) else weight_decay\n opt_params.append(\n {\"params\": [p], \"weight_decay\": wd, \"lr\": lr}\n )\n\n optimizer = AdamW(opt_params, lr=lr, betas=(0.5, 0.999))\n\n n_steps = epochs * len(train_loader)\n num_warmup_steps = int(warmup_prop * n_steps)\n scheduler = get_linear_schedule_with_warmup(\n optimizer, num_warmup_steps, n_steps\n )\n\n # Initialize W&B\n run = wandb.init(project='Coleridge-BERT', config= WANDB_CONFIG)\n \n total_steps = 0\n for epoch in range(epochs):\n model.train()\n start_time = time.time()\n\n optimizer.zero_grad()\n avg_loss = 0\n\n for step, data in enumerate(train_loader):\n total_steps += 1\n\n ids, token_type_ids = trim_tensors(\n data[\"ids\"], data[\"token_type_ids\"], model.name\n )\n\n start_logits, end_logits = model(ids.cuda(), token_type_ids.cuda())\n\n loss = qa_loss_fn(\n start_logits,\n end_logits,\n data[\"target_start\"].cuda(),\n data[\"target_end\"].cuda(),\n config=loss_config,\n )\n \n wandb.log({'loss': loss})\n\n avg_loss += loss.item() / len(train_loader)\n loss.backward()\n\n nn.utils.clip_grad_norm_(model.parameters(), 10.0)\n\n optimizer.step()\n scheduler.step()\n model.zero_grad()\n \n # Close W&B run\n wandb.finish()\n\n model.eval()\n avg_val_loss = 0.\n preds, truths = [], []\n\n with torch.no_grad():\n\n for data in val_loader:\n ids, token_type_ids = trim_tensors(\n data[\"ids\"], data[\"token_type_ids\"], model.name\n )\n\n start_logits, end_logits = model(ids.cuda(), token_type_ids.cuda())\n\n loss = qa_loss_fn(\n start_logits.detach(),\n end_logits.detach(),\n data[\"target_start\"].cuda(),\n data[\"target_end\"].cuda(),\n config=loss_config,\n )\n\n avg_val_loss += loss.item() / len(val_loader)\n\n preds += get_pred_from_logits(data, start_logits, end_logits)\n truths += data['label']\n\n score = compute_score(truths, preds)\n\n dt = time.time() - start_time\n lr = scheduler.get_last_lr()[0]\n print(f\"Epoch {epoch + 1}/{epochs} \\t lr={lr:.1e} \\t t={dt:.0f}s \\t\", end=\"\")\n print(\n f\"loss={avg_loss:.3f} \\t val_loss={avg_val_loss:.3f} \\t val_score={score:.4f}\"\n )\n\n del loss, data, avg_val_loss, avg_loss, train_loader, val_loader\n torch.cuda.empty_cache()\n gc.collect()\n\n return preds","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:12:48.885934Z","iopub.execute_input":"2021-12-12T09:12:48.886409Z","iopub.status.idle":"2021-12-12T09:12:48.904204Z","shell.execute_reply.started":"2021-12-12T09:12:48.886373Z","shell.execute_reply":"2021-12-12T09:12:48.903153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def k_fold(config, df, save=True):\n tokenizer, tokens = create_tokenizer_and_tokens(config)\n \n time = re.sub(' ', '_', str(datetime.datetime.now())[:16])\n score = 0\n \n gkf = GroupKFold(n_splits=config.k)\n folds = list(gkf.split(X=df, groups=df['dataset_title']))\n \n pred_oof = [''] * len(df)\n \n for fold, (train_idx, val_idx) in enumerate(folds):\n if fold in config.selected_folds:\n print(f\"\\n------------- Fold {fold + 1} / {len(folds)} -------------\\n\")\n seed_everything(config.seed + fold)\n\n model = QATransformer(config.selected_model).cuda()\n model.zero_grad()\n\n train_dataset = SectionDataset(\n df.iloc[train_idx], \n tokenizer, \n tokens, \n max_len=config.max_len, \n model_name=config.selected_model\n )\n\n val_dataset = SectionDataset(\n df.iloc[val_idx], \n tokenizer, \n tokens,\n max_len=config.max_len, \n model_name=config.selected_model\n )\n\n n_parameters = count_parameters(model)\n\n print(f\" -> {len(train_dataset)} training texts\")\n print(f\" -> {len(val_dataset)} validation texts\")\n print(f\" -> {n_parameters} trainable parameters\\n\") \n\n preds = fit(\n model, \n train_dataset, \n val_dataset, \n config.loss_config,\n epochs=config.epochs, \n batch_size=config.batch_size, \n weight_decay=config.weight_decay, \n lr=config.lr, \n warmup_prop=config.warmup_prop,\n )\n\n for i, idx in enumerate(val_idx):\n pred_oof[idx] = preds[i]\n\n if save:\n save_model_weights(model, f'{config.selected_model}_{fold + 1}.pt', cp_folder=\"\")\n\n del model, train_dataset, val_dataset\n torch.cuda.empty_cache()\n gc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:12:49.686179Z","iopub.execute_input":"2021-12-12T09:12:49.686506Z","iopub.status.idle":"2021-12-12T09:12:49.697247Z","shell.execute_reply.started":"2021-12-12T09:12:49.686474Z","shell.execute_reply":"2021-12-12T09:12:49.696334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Config:\n # General\n k = 5\n seed = 2021\n selected_folds = [0]\n\n # Texts\n max_len = 256\n \n # Architecture\n selected_model = \"bert-base-uncased\"\n lowercase = True\n \n # Loss function\n loss_config = {\n \"smoothing\": False,\n \"eps\": 0.1,\n }\n \n # Training\n batch_size = 16\n batch_size_val = batch_size * 2\n weight_decay = 1.\n \n epochs = 1\n lr = 5e-5\n warmup_prop = 0.1","metadata":{"execution":{"iopub.status.busy":"2021-12-12T09:12:50.259405Z","iopub.execute_input":"2021-12-12T09:12:50.260097Z","iopub.status.idle":"2021-12-12T09:12:50.267252Z","shell.execute_reply.started":"2021-12-12T09:12:50.260031Z","shell.execute_reply":"2021-12-12T09:12:50.266136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(\"df_train.csv\")\n\nk_fold(\n Config,\n df,\n save=True,\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[WandB Project Page](https://wandb.ai/ishandutta/Coleridge-BERT?workspace=user-ishandutta)","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}