{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This notebook fine-tunes Google's BigBird model for the Coleridge Intiaitive Show US the data competition.  \n\nIt was difficult to get it to run even a single epoch within Kaggle's 9hr timout limit.  \n\nIn order to do that I separated out the data preparation, here: https://www.kaggle.com/danieldorosz/show-us-the-data-bigbird-dataprep  \nand the inference, here: https://www.kaggle.com/danieldorosz/show-us-the-data-bigbird-inference  \n\nA chunk of the logic is farmed-out to a coleridge-helpers utility script.   \n\nThe main intuition behind this effort was that I wanted to include as much context as possible in my training examples. \nAlso that I wanted to keep related context together. We have a ready-provided demarkation of context expressed as \nsections in the training data. So what I did was create contextual 'snippets' as my training examples. Each snippet \ncontains one or more sections such that my training examples get as close as possible to BigBird's maximum of 4096\ntokens, without breaking up any sections. If a single section is longer the training example limit, I break it up \nat the last period prior to the limit.   \n\nThe code is very much a rough-and-ready first draft, please don't judge me ;-) There is much to be improved for which \nI didn't have time. This mainly serves as a baseline to assess the score I could expect from this kind of approach.  \n\nI ran it a couple of times using the last checkpoint from the first (timed-out) run as input to the next.","metadata":{}},{"cell_type":"markdown","source":"# Imports & Preamble","metadata":{}},{"cell_type":"code","source":"!pip install -qU --no-warn-conflicts transformers --no-index --find-links=file:///kaggle/input/coleridge-packages\n!pip install -qU --no-warn-conflicts tokenizers --no-index --find-links=file:///kaggle/input/coleridge-packages\n!pip install -qU --no-warn-conflicts datasets --no-index --find-links=file:///kaggle/input/coleridge-packages\n!pip install -qU --no-warn-conflicts fsspec --no-index --find-links=file:///kaggle/input/coleridge-packages\n!pip install -qU --no-warn-conflicts seqeval --no-index --find-links=file:///kaggle/input/coleridge-packages\n    \n# need to set wandb off otherwise we get errors using this kernel offline\n!wandb off","metadata":{"execution":{"iopub.status.busy":"2021-07-01T16:57:36.858848Z","iopub.execute_input":"2021-07-01T16:57:36.859228Z","iopub.status.idle":"2021-07-01T16:57:51.470873Z","shell.execute_reply.started":"2021-07-01T16:57:36.859148Z","shell.execute_reply":"2021-07-01T16:57:51.469954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np \nfrom transformers import (\n    BigBirdForTokenClassification,\n    BigBirdTokenizerFast,\n    BigBirdConfig,\n    TrainingArguments, \n    Trainer,\n    DataCollatorForTokenClassification,\n)\nfrom datasets import (\n    Dataset,\n    load_metric,\n)","metadata":{"execution":{"iopub.status.busy":"2021-07-01T16:57:51.474531Z","iopub.execute_input":"2021-07-01T16:57:51.474786Z","iopub.status.idle":"2021-07-01T16:57:58.951155Z","shell.execute_reply.started":"2021-07-01T16:57:51.474758Z","shell.execute_reply":"2021-07-01T16:57:58.950372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Training Data","metadata":{}},{"cell_type":"code","source":"# tokenized_dataset = Dataset.from_json(\"../input/coleridgetaggedsnippets/tokenized_dataset_reduced.json\")\n# tokenized_dataset = tokenized_dataset.shuffle(seed=42)\n\ntokenized_dataset = Dataset.from_json(\"../input/show-us-the-data-bigbird-dataprep/tokenized_dataset.json\")","metadata":{"execution":{"iopub.status.busy":"2021-07-01T16:58:18.116245Z","iopub.execute_input":"2021-07-01T16:58:18.116621Z","iopub.status.idle":"2021-07-01T16:59:30.340926Z","shell.execute_reply.started":"2021-07-01T16:58:18.11659Z","shell.execute_reply":"2021-07-01T16:59:30.340004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Instantiate Pretrained BigBird Model & Tokenizer","metadata":{}},{"cell_type":"code","source":"# BigBird roberta-base\nmodel_class, tokenizer_class, pretrained_weights = (BigBirdForTokenClassification, BigBirdTokenizerFast, '../input/huggingfacebigbirdrobertabase')\n\ntokenizer = tokenizer_class.from_pretrained(pretrained_weights)\n\ndata_collator = DataCollatorForTokenClassification(tokenizer)\n\nlabel_list = [\"O\", \"B\", \"I\"]\nlabel2id = {label : id for id, label in enumerate(label_list)}\nid2label = {id : label for label, id in label2id.items()}\n\ndef get_pretrained_model(checkpoint=pretrained_weights):\n    config = BigBirdConfig(attention_type=\"block_sparse\", gradient_checkpointing=True, num_labels=3, id2label=id2label, label2id=label2id)\n    return model_class.from_pretrained(checkpoint, config=config)","metadata":{"execution":{"iopub.status.busy":"2021-07-01T17:02:15.445343Z","iopub.execute_input":"2021-07-01T17:02:15.445665Z","iopub.status.idle":"2021-07-01T17:02:15.709726Z","shell.execute_reply.started":"2021-07-01T17:02:15.445637Z","shell.execute_reply":"2021-07-01T17:02:15.708871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Fine Tuning","metadata":{}},{"cell_type":"markdown","source":"## Metrics\n\nThe metrics will only reflect training accuracy as we are using the traning set for evaluation.\nThis could be improved by using kfold validation.","metadata":{}},{"cell_type":"code","source":"# copy my_seqeval.py to the working directory because the input directory is non-writable\n!cp ../input/coleridge-packages/seqeval_script.py ./","metadata":{"execution":{"iopub.status.busy":"2021-07-01T17:02:21.502669Z","iopub.execute_input":"2021-07-01T17:02:21.505187Z","iopub.status.idle":"2021-07-01T17:02:22.204091Z","shell.execute_reply.started":"2021-07-01T17:02:21.505139Z","shell.execute_reply":"2021-07-01T17:02:22.202905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# TODO: This could be improved to use f0.5 score (used on LB evaluation) instead of f1\n\nmetric = load_metric(\"seqeval_script.py\")\n\ndef compute_metrics(p):\n    predictions, labels = p\n    predictions = np.argmax(predictions, axis=2)\n\n    # Remove ignored index (special tokens)\n    true_predictions = [\n        [label_list[p] for (p, l) in zip(prediction, label) if l != -100]\n        for prediction, label in zip(predictions, labels)\n    ]\n    true_labels = [\n        [label_list[l] for (p, l) in zip(prediction, label) if l != -100]\n        for prediction, label in zip(predictions, labels)\n    ]\n\n    results = metric.compute(predictions=true_predictions, references=true_labels)\n    return {\n        \"precision\": results[\"overall_precision\"],\n        \"recall\": results[\"overall_recall\"],\n        \"f1\": results[\"overall_f1\"],\n        \"accuracy\": results[\"overall_accuracy\"],\n    }","metadata":{"execution":{"iopub.status.busy":"2021-07-01T17:02:24.896128Z","iopub.execute_input":"2021-07-01T17:02:24.896496Z","iopub.status.idle":"2021-07-01T17:02:24.932627Z","shell.execute_reply.started":"2021-07-01T17:02:24.896461Z","shell.execute_reply":"2021-07-01T17:02:24.931769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Run Training","metadata":{}},{"cell_type":"code","source":"batch_size = 4\n\nargs = TrainingArguments(\n    \"model_checkpoints\",\n    evaluation_strategy = \"epoch\",\n    per_device_train_batch_size=batch_size,\n    per_device_eval_batch_size=batch_size,\n    num_train_epochs=1,\n    save_total_limit=1,\n    save_strategy=\"steps\",\n    save_steps=1000,\n)\n\ntrainer = Trainer(\n    get_pretrained_model(\"../input/coleridgemodelcheckpoint\"),\n    args,\n    train_dataset=tokenized_dataset,\n    eval_dataset=tokenized_dataset,\n    data_collator=data_collator,\n    tokenizer=tokenizer,\n    compute_metrics=compute_metrics\n)\n\ntrainer.train()","metadata":{"execution":{"iopub.status.busy":"2021-07-01T17:02:28.572569Z","iopub.execute_input":"2021-07-01T17:02:28.572915Z","iopub.status.idle":"2021-07-01T17:07:24.221713Z","shell.execute_reply.started":"2021-07-01T17:02:28.572884Z","shell.execute_reply":"2021-07-01T17:07:24.218458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trainer.save_model(\"/saved_model\")","metadata":{"execution":{"iopub.status.busy":"2021-07-01T16:57:59.315601Z","iopub.status.idle":"2021-07-01T16:57:59.316219Z"},"trusted":true},"execution_count":null,"outputs":[]}]}