{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Coleridge - Huggingface Question Answering\n\nThis is not exactly what the competition metric is asking for, but is an interesting experiment nonetheless.\n\nI've taken the Huggingface Question Answering pre-trained model, and asked it to predict which dataset is referenced (as opposed to the text mentioning it)."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport glob\nimport numpy as np\nimport pandas as pd\nimport re\nimport simplejson\nimport torch\nfrom joblib import Parallel, delayed\nfrom typing import *\nfrom transformers import pipeline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thanks to [@Nobu](https://www.kaggle.com/enukuro) for figuring out how to get question answering working in offline mode with this dataset\n- https://www.kaggle.com/enukuro/huggingface-distilbertbasecaseddistilledsquad"},{"metadata":{"trusted":true},"cell_type":"code","source":"### Online Mode\n# from transformers import pipeline\n# question_answering = pipeline(\"question-answering\", device=device)  # cache_dir=\"/kaggle/working/transformers\")\n\n\n### Offline Mode\n\nfrom transformers import QuestionAnsweringPipeline, DistilBertTokenizerFast, TFDistilBertForQuestionAnswering, DistilBertConfig\n\ntokenizer = DistilBertTokenizerFast(\n    vocab_file='../input/huggingface-distilbertbasecaseddistilledsquad/distilbert-base-cased-distilled-squad_vocab.txt', \n    tokenizer_file='../input/huggingface-distilbertbasecaseddistilledsquad/distilbert-base-cased-distilled-squad_tokenizer.json', \n    do_lower_case=False\n)\nconfig = DistilBertConfig.from_pretrained(\n    '../input/huggingface-distilbertbasecaseddistilledsquad/distilbert-base-cased-distilled-squad_config.json'\n)\nmodel = TFDistilBertForQuestionAnswering.from_pretrained(\n    '../input/huggingface-distilbertbasecaseddistilledsquad/distilbert-base-cased-distilled-squad.h5', \n    config=config\n)\nquestion_answering = QuestionAnsweringPipeline(\n    model=model, \n    tokenizer=tokenizer,\n    device=(0 if torch.cuda.is_available() else -1)\n) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Example Usage\n\nExample taken from: https://towardsdatascience.com/question-answering-with-pretrained-transformers-using-pytorch-c3e7a44b4012"},{"metadata":{"trusted":true},"cell_type":"code","source":"context = \"\"\"\nMachine learning (ML) is the study of computer algorithms that improve automatically through experience. \nIt is seen as a part of artificial intelligence.\nMachine learning algorithms build a model based on sample data, known as \"training data\", \nin order to make predictions or decisions without being explicitly programmed to do so. \nMachine learning algorithms are used in a wide variety of applications, \nsuch as email filtering and computer vision, \nwhere it is difficult or unfeasible to develop conventional algorithms to perform the needed tasks.\n\"\"\"\nquestion = \"What are machine learning models based on?\"\n\nresult = question_answering(question=question, context=context)\nprint(\"Answer:\", result['answer'])\nprint(\"Score: \", result['score'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prepare Dataset\n\nCode reused from: https://www.kaggle.com/jamesmcguigan/coleridge-string-literals/"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')\ntrain_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndef clean_text(text: str) -> str:               return re.sub('[^A-Za-z0-9]+', ' ', str(text).lower()).strip()\ndef clean_texts(texts: List[str]) -> List[str]: return [ clean_text(text) for text in texts ] \n\ndef read_json(index: str, test_train) -> Dict:\n    filename = f\"../input/coleridgeinitiative-show-us-the-data/{test_train}/{index}.json\"\n    with open(filename) as f:\n        json = simplejson.load(f)\n    return json\n        \ndef json2text(index: str, test_train) -> str:\n    json  = read_json(index, test_train)\n    texts = [\n        row[\"section_title\"] + \" \" + row[\"text\"] \n        for row in json\n    ]\n    # texts = clean_texts(texts)\n    text  = \" \".join(texts)\n    return text\n\ndef filename_to_index(filename):\n    return re.sub(\"^.*/|\\.[^.]+$\", '', filename)\n\ndef glob_to_indices(globpath):\n    return list(map(filename_to_index, glob.glob(globpath)))\n       \n# Inspired by: https://www.kaggle.com/hamditarek/merge-multiple-json-files-to-a-dataframe\ndef dataset_df(test_train=\"test\"):\n    indices = glob_to_indices(f\"../input/coleridgeinitiative-show-us-the-data/{test_train}/*.json\")    \n    texts   = Parallel(-1)( \n        delayed(json2text)(index, test_train)\n        for index in indices  \n    )\n    df = pd.DataFrame([\n        { \"id\": index, \"text\": text }\n        for index, text in zip(indices, texts)\n    ])\n    df.to_csv(f\"{test_train}.json.csv\", index=False)\n    return df\n\ntrain_data = dataset_df(\"train\")\ntest_data  = dataset_df(\"test\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Question Answering\n\nLets try out a variety of different question formats"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/coleridgeinitiative-show-us-the-data/train.csv\")\n\ndef answer_questions(question, df, count=0):\n    submission_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv', index_col=0)\n    for n, (_, row) in enumerate(df.iterrows()):\n        context  = row[\"text\"]\n        result   = question_answering(question=question, context=context)\n        expected = train_df[ train_df[\"Id\"] == row[\"id\"] ]\n        datasets = \"; \".join(expected['dataset_label']) if len(expected) else \"\"  # Predict the internal text \n        \n        submission_df['PredictionString'] = result['answer']\n        submission_df.to_csv(\"submission.csv\")\n        \n        print(f\"{row['id']} | {result['score']:.3f}\")\n        if len(datasets): \n            print('answer:       ', result['answer'])\n            print('dataset_label:', set(expected['dataset_label']))\n            print('dataset_title:', set(expected['dataset_title']))\n            print('pub_title:    ', set(expected['pub_title']) )\n            print()\n        if count and count <= n: break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"answer_questions(\"Which was said about the study dataset?\", train_data, 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"answer_questions(\"Which dataset is referenced?\", train_data, 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"answer_questions(\"Which study dataset is referenced?\", train_data, 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"answer_questions(\"What is referenced?\", train_data, 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"answer_questions(\"What papers are referenced?\", train_data, 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"answer_questions(\"Which study is referenced?\", train_data, 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"answer_questions(\"Which study, program, data or database?\", train_data, 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"answer_questions(\"What did you say about the study?\", train_data, 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"answer_questions(\"Identify the mention of datasets?\", train_data, 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"answer_questions(\"What was said about the study, program, data or database?\", train_data, 10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"answer_questions(\"What was said about the study program data?\", train_data, 10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"answer_questions(\"What was said about the study program data?\", test_data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Unsolved Problems\n\n- SOLVED: How do I get `os.environ[\"TRANSFORMERS_CACHE\"]` to work in offline mode? (Thanks: [@Nobu](https://www.kaggle.com/enukuro))\n- Does anybody have any advise for how these pretrained models could be fine-tuned on the competition dataset?"},{"metadata":{},"cell_type":"markdown","source":"# Conclusion\n\nThis might not be exactly what the competition is asking for, but the results are intresting none the less.\n\nIf you learnt something from this notebook, or want to fork it, then please leave an upvote. Thank you."},{"metadata":{},"cell_type":"markdown","source":"# Further Reading\n\n\nThe original `String Literals` notebook \n- https://www.kaggle.com/jamesmcguigan/coleridge-string-literals/"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}