{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Description\nThis notebook is part of our 27th place solution which is based on simple regex and some heuristics. We just started the competition 10 days before deadline. So, we are happy about where we stand. We tried to train some models. But none worked better than heuristics based solution. Plus, we didn't find any proper validation strategy. With Heuristics based solution, we can just use all train data as validation set and there is less leakage in public lb scores. We further evaluated our solution on RichContext competition data (https://coleridgeinitiative.org/richcontext/richcontextcompetition/). This gave us better idea on what may or may not work on unseen test. In this notebook, We just tried to predict very confident datasets to mantain high precision and sufficient recall. We were little worried about low recall on rich-context competition dataset, so, in final submission, we just took union of predictions of string matching with heuristics based solution. Our best solution (a variant of this notebook) scored above 0.4 on private lb but we didn't have enough reasons to select it as our final submission. \n\n### Challenges in proper validation\n\n1. Train data was weakly labelled\n2. Public test set consists only of 12% of total test set\n3. RichContext competition had slightly longer dataset names. They mention dates, places, etc which are not there in train set.\n\n### Statistics\n1. Heuristics based solution 0.391 (CV) 0.499 (Public) 0.397 (Private)\n2. String Matching 0.575 (Public) 0.105 (Private)\n3. Heuristics + String Matching (Selected Submission) 0.559 (Public) 0.395 (Private)","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nimport json\nimport re\nfrom tqdm.auto import tqdm\nimport nltk.data\nfrom IPython.display import display\njaccard_threshold = 0.5\ntokenizer = nltk.data.load('tokenizers/punkt/english.pickle')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv',index_col='Id')\ntemp_1 = [x.lower() for x in train_df['dataset_label'].unique()]\ntemp_2 = [x.lower() for x in train_df['dataset_title'].unique()]\ntemp_3 = [x.lower() for x in train_df['cleaned_label'].unique()]\nexisting_labels = set(temp_1 + temp_2 + temp_3)","metadata":{"execution":{"iopub.status.busy":"2021-06-17T19:51:37.744322Z","iopub.execute_input":"2021-06-17T19:51:37.744695Z","iopub.status.idle":"2021-06-17T19:51:37.832776Z","shell.execute_reply.started":"2021-06-17T19:51:37.744657Z","shell.execute_reply":"2021-06-17T19:51:37.831939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"IS_SUBMIT = len(os.listdir('../input/coleridgeinitiative-show-us-the-data/test/'))>4\n\n# IS_SUBMIT = True\nprint(IS_SUBMIT)\n\nif IS_SUBMIT:\n    test_root = '../input/coleridgeinitiative-show-us-the-data/test/'\nelse:\n    test_root = '../input/coleridgeinitiative-show-us-the-data/train/'\n#     test_root = '../input/show-us-the-data-sample-test/test/'","metadata":{"execution":{"iopub.status.busy":"2021-06-17T19:51:38.694028Z","iopub.execute_input":"2021-06-17T19:51:38.694598Z","iopub.status.idle":"2021-06-17T19:51:38.701807Z","shell.execute_reply.started":"2021-06-17T19:51:38.694549Z","shell.execute_reply":"2021-06-17T19:51:38.70067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission = pd.DataFrame(os.listdir(test_root),columns=['Id'])\nsample_submission['Id'] = sample_submission['Id'].apply(lambda x:x.replace('.json',''))\nsample_submission['PredictionString'] = None\nsample_submission = sample_submission.set_index(\"Id\").sort_index()\nprint(sample_submission.shape)","metadata":{"execution":{"iopub.status.busy":"2021-06-17T19:51:40.631194Z","iopub.execute_input":"2021-06-17T19:51:40.631754Z","iopub.status.idle":"2021-06-17T19:51:40.676561Z","shell.execute_reply.started":"2021-06-17T19:51:40.631709Z","shell.execute_reply":"2021-06-17T19:51:40.675478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_text(id_):\n    with open(f'{test_root}{id_}.json') as f:\n        text_set = json.load(f)\n        text_set = {entry['section_title']:entry['text'] for entry in text_set}\n    return text_set\n\ndef split_sentences(id_,debug=False):\n    \"\"\"\n    Split text into sentences\n    \"\"\"\n    all_sentences = []\n    text_set = read_text(id_)\n    for key in text_set:\n        text = text_set[key]\n        sentences = pd.DataFrame(tokenizer.tokenize(text),columns=['sentence'])\n        sentences['section'] = key\n        sentences['section_line'] = sentences.index\n        all_sentences.append(sentences)\n    all_sentences = pd.concat(all_sentences).reset_index(drop=True)\n    all_sentences['line'] = all_sentences.index\n    all_sentences['id'] = id_\n    return all_sentences","metadata":{"execution":{"iopub.status.busy":"2021-06-17T19:51:41.401167Z","iopub.execute_input":"2021-06-17T19:51:41.401602Z","iopub.status.idle":"2021-06-17T19:51:41.410627Z","shell.execute_reply.started":"2021-06-17T19:51:41.401569Z","shell.execute_reply":"2021-06-17T19:51:41.40958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')","metadata":{"execution":{"iopub.status.busy":"2021-06-17T19:51:43.395096Z","iopub.execute_input":"2021-06-17T19:51:43.395545Z","iopub.status.idle":"2021-06-17T19:51:43.49086Z","shell.execute_reply.started":"2021-06-17T19:51:43.395506Z","shell.execute_reply":"2021-06-17T19:51:43.48962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def clean_text_pre(txt):\n    return re.sub('[^A-Za-z0-9-]+', ' ', str(txt))\n\ndef clean_text_post(txt):\n    return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower())","metadata":{"execution":{"iopub.status.busy":"2021-06-17T19:51:44.914337Z","iopub.execute_input":"2021-06-17T19:51:44.914674Z","iopub.status.idle":"2021-06-17T19:51:44.919118Z","shell.execute_reply.started":"2021-06-17T19:51:44.914647Z","shell.execute_reply":"2021-06-17T19:51:44.918435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset_keywords = ['study', 'dataset', 'model','survey','data','adni','codes', 'genome', 'program','assessment','database','census','initiative','sequences'\n                    'gauge','system','stewardship','surge']\n\ndataset_keywords = ['study', 'dataset','survey','database','sequences','census','sequence', 'data', 'data set','poll']\n# dataset_keywords = ['study', 'dataset','survey','data']\n\ndataset_keywords = list(map(lambda x:x.capitalize(),dataset_keywords))\n\nregex_pattern = '\\w*{}\\w*'.format(\"|\".join(dataset_keywords))\nregex_pattern = re.compile(regex_pattern)\n\nregex_pattern","metadata":{"execution":{"iopub.status.busy":"2021-06-17T19:51:46.865144Z","iopub.execute_input":"2021-06-17T19:51:46.865665Z","iopub.status.idle":"2021-06-17T19:51:46.873819Z","shell.execute_reply.started":"2021-06-17T19:51:46.865621Z","shell.execute_reply":"2021-06-17T19:51:46.872811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#### remove >.5 jaccard matches from predicitons\ndef jaccard_similarity(s1, s2):\n    l1 = s1.split(\" \")\n    l2 = s2.split(\" \")    \n    intersection = len(list(set(l1).intersection(l2)))\n    union = (len(l1) + len(l2)) - intersection\n    return float(intersection) / union","metadata":{"execution":{"iopub.status.busy":"2021-06-17T19:51:48.615281Z","iopub.execute_input":"2021-06-17T19:51:48.615821Z","iopub.status.idle":"2021-06-17T19:51:48.620678Z","shell.execute_reply.started":"2021-06-17T19:51:48.615787Z","shell.execute_reply":"2021-06-17T19:51:48.619946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def preprocess_sentence(sentence):\n#     for word in dataset_keywords:\n#         sentence = sentence.replace(\" \"+word.lower()+\" \",\" \"+word+\" \").strip()\n    sentence = sentence[0].lower()+sentence[1:]\n    return sentence\n\ndef find_keyword(df,col):\n    df['keyword_present'] = 0\n    for word in dataset_keywords:\n        df['keyword_present'] += (df[col].apply(lambda x:x.find(word.capitalize()))>0).astype(int)\n    df['keyword_present'] = df['keyword_present']>0\n    \ndef is_highlighted(word):\n    return any(x.isupper() for x in word)\n\ndef extract_dataset_names(sentence):\n    all_datasets = []\n    for m in regex_pattern.finditer(sentence):\n        start_pos = m.start()\n        end_pos = m.start()+len(m.group())\n        #To Do -  Use regex to split\n        pretext = clean_text_pre(sentence[:end_pos].split(',')[-1]).strip().split()\n        posttext = clean_text_pre(sentence[end_pos:].split(',')[0]).strip().split()\n\n        limit = 1\n        dataset_start_idx = -1\n        for i,x in enumerate(reversed(pretext)):\n            if not is_highlighted(x):\n                limit -= 1\n            if is_highlighted(x):\n                dataset_start_idx = -1*(i+1)\n            if limit == -1:\n                break\n\n        limit = 1     \n        dataset_end_idx = 0\n        for i,x in enumerate(posttext):\n            if not is_highlighted(x):\n                limit -= 1\n            if is_highlighted(x):\n                dataset_end_idx = (i+1)\n            if limit == -1:\n                break\n\n        dataset_name = pretext[dataset_start_idx:] + posttext[:dataset_end_idx]\n        dataset_name = \" \".join(dataset_name)\n        all_datasets.append(dataset_name)\n    return all_datasets\n\ndef filter_duplicates(dataset_names):\n    dataset_names = list(sorted(dataset_names,key = lambda x:-1*len(x)))\n    filtered_dataset_names = []\n    for dataset in dataset_names:\n        is_duplicate = False\n        for reference in filtered_dataset_names:\n            if jaccard_similarity(dataset,reference)>=0.9:\n                is_duplicate=True\n        if not is_duplicate:\n            filtered_dataset_names.append(dataset)\n    return filtered_dataset_names\n\ndef filter_by_length(dataset_names):\n    return [x for x in dataset_names if len(x.split())>3 and len(x.split())<=10]\n\ndef filter_by_count(dataset_names):\n    if len(dataset_names)>10:\n        return []\n#         return dataset_names[:10]\n    else:\n        return dataset_names\n    \ndef clean_label(dataset_names):\n    return [clean_text_post(x) for x in dataset_names]\n\ndef remove_existing_labels(dataset_names):\n    return [x for x in dataset_names if x not in existing_labels]","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:24:56.419858Z","iopub.execute_input":"2021-06-17T20:24:56.420218Z","iopub.status.idle":"2021-06-17T20:24:56.438574Z","shell.execute_reply.started":"2021-06-17T20:24:56.420188Z","shell.execute_reply":"2021-06-17T20:24:56.437415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_abbreviations(text):\n    cleaned = clean_text_pre(text)\n    words = cleaned.split()\n    abbreviations = [word for word in words if sum(x.isupper() for x in word)>=3]\n    return abbreviations\n\ndef get_abbreviation_details(row):\n    abbreviations = {}\n    cleaned = clean_text_pre(row.sentence)\n    for abbreviation in row.abbreviations:\n        start_pos = cleaned.find(abbreviation)\n        end_pos = start_pos+len(abbreviation)\n        MAX_LENGTH = len(abbreviation)+2\n        left = cleaned[:start_pos].split()[-1*MAX_LENGTH:]\n        right = cleaned[end_pos:].split()[:MAX_LENGTH]\n        if sum(is_highlighted(x) for x in left)>2:\n            first_letters = [x[0] for x in left]\n            pos = -1\n            for i in range(len(first_letters)-1,-1,-1):\n                x = first_letters[i]\n                if x.lower()==abbreviation[pos].lower():\n                    if pos==-1:\n                        end_word = i\n                    pos-=1\n                if pos<-1*len(abbreviation):\n                    start_word = i\n                    abbreviations[abbreviation] = \" \".join(left[start_word:end_word+1])\n                    break\n        elif sum(is_highlighted(x) for x in right)>2:\n            first_letters = [x[0] for x in right]\n            pos = 0\n            for i,x in enumerate(first_letters):\n                if x.lower()==abbreviation[pos].lower():\n                    if pos==0:\n                        start_word = i\n                    pos+=1\n                if pos==len(abbreviation):\n                    end_word = i\n                    abbreviations[abbreviation] = \" \".join(right[start_word:end_word+1])\n                    break\n    return [f\"{x}::{abbreviations[x]}\" for x in abbreviations]","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:24:56.871559Z","iopub.execute_input":"2021-06-17T20:24:56.872226Z","iopub.status.idle":"2021-06-17T20:24:56.884851Z","shell.execute_reply.started":"2021-06-17T20:24:56.872101Z","shell.execute_reply":"2021-06-17T20:24:56.883748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SIMULATE = test_root == '../input/coleridgeinitiative-show-us-the-data/train/'\nSIMULATE","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:24:57.175381Z","iopub.execute_input":"2021-06-17T20:24:57.175709Z","iopub.status.idle":"2021-06-17T20:24:57.180955Z","shell.execute_reply.started":"2021-06-17T20:24:57.175681Z","shell.execute_reply":"2021-06-17T20:24:57.180277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Get All the abbreviations pair from the text\n\nAbbreviations may be a sequence of capitalized word followed by a uppercased word. Example - Alzheimer's Disease Neuroimaging Initiative (ADNI) ","metadata":{}},{"cell_type":"code","source":"if SIMULATE:\n    all_abbreviations = pd.read_csv('../input/coleridge-abbreviations/abbreviations.csv')\nelse:\n    all_abbreviations = []\n    predictions = pd.DataFrame(columns = ['PredictionString'])\n    for id_,row in tqdm(sample_submission.iterrows()):\n        sentences = split_sentences(id_)\n        sentences['abbreviations'] = sentences.sentence.apply(get_abbreviations)\n        sentences['abbreviations'] = sentences.apply(get_abbreviation_details,axis=1)\n        abbreviations = np.concatenate(sentences.abbreviations.values)\n        abbreviations = {x.split('::')[0]:x.split('::')[1] for x in abbreviations}\n        abbreviations = pd.Series(abbreviations).reset_index()\n        abbreviations.columns = ['short_form','long_form']\n        abbreviations['Id'] = id_\n        all_abbreviations.append(abbreviations)\n    all_abbreviations = pd.concat(all_abbreviations).reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:24:57.541814Z","iopub.execute_input":"2021-06-17T20:24:57.542327Z","iopub.status.idle":"2021-06-17T20:24:57.622604Z","shell.execute_reply.started":"2021-06-17T20:24:57.542283Z","shell.execute_reply":"2021-06-17T20:24:57.62185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Filter dataset names from the shortform-longform list\n\nLogic - Dataset names should either end with any of dataset keywords or should contain (DATASET-KEYWORD + of/for/on) ","metadata":{}},{"cell_type":"code","source":"long_forms = all_abbreviations.long_form.apply(clean_text_post).drop_duplicates()\nshort_forms = all_abbreviations.short_form.apply(clean_text_post).drop_duplicates()\ndataset_short_forms = np.unique(all_abbreviations[all_abbreviations.long_form.str.lower().apply(lambda x: any(y.lower() in x for y in dataset_keywords))].short_form.values)\ndataset_short_forms = [x for x in dataset_short_forms if len(x)>3]\nregex_pattern_abbreviations = '\\w*{}\\w*'.format(\"|\".join(dataset_short_forms))\nregex_pattern_abbreviations = re.compile(regex_pattern_abbreviations)\nlong_forms.shape,short_forms.shape","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:24:57.882351Z","iopub.execute_input":"2021-06-17T20:24:57.882987Z","iopub.status.idle":"2021-06-17T20:24:58.451532Z","shell.execute_reply.started":"2021-06-17T20:24:57.882943Z","shell.execute_reply":"2021-06-17T20:24:58.450621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset_keywords_lowered = [x.lower() for x in dataset_keywords]\ndataset_keywords_cleaned = [clean_text_post(x) for x in dataset_keywords]\ndataset_keywords,dataset_keywords_lowered,dataset_keywords_cleaned","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:24:58.455039Z","iopub.execute_input":"2021-06-17T20:24:58.455435Z","iopub.status.idle":"2021-06-17T20:24:58.461203Z","shell.execute_reply.started":"2021-06-17T20:24:58.455407Z","shell.execute_reply":"2021-06-17T20:24:58.460594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def filter_false_positives_from_longforms(dataset_names):\n    filtered = []\n    for dataset in dataset_names:\n        dataset_cleaned = clean_text_post(dataset)\n        if any(x+\" of\" in dataset_cleaned for x in dataset_keywords_cleaned):\n            filtered.append(dataset)\n        elif any(x+\" from\" in dataset_cleaned for x in dataset_keywords_cleaned):\n            filtered.append(dataset)\n        elif any(x+\" on\" in dataset_cleaned for x in dataset_keywords_cleaned):\n            filtered.append(dataset)\n        elif any(dataset_cleaned.endswith(x) for x in dataset_keywords_cleaned):\n            filtered.append(dataset)\n    return filtered","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:24:58.642642Z","iopub.execute_input":"2021-06-17T20:24:58.643102Z","iopub.status.idle":"2021-06-17T20:24:58.649236Z","shell.execute_reply.started":"2021-06-17T20:24:58.643072Z","shell.execute_reply":"2021-06-17T20:24:58.648541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"long_forms_filtered = filter_false_positives_from_longforms(long_forms)","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:24:59.47567Z","iopub.execute_input":"2021-06-17T20:24:59.476201Z","iopub.status.idle":"2021-06-17T20:24:59.707415Z","shell.execute_reply.started":"2021-06-17T20:24:59.476167Z","shell.execute_reply":"2021-06-17T20:24:59.706531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def filter_false_positives(dataset_names):\n    filtered = []\n    for dataset in dataset_names:\n        dataset_cleaned = clean_text_post(dataset)\n        if any(x+\" of\" in dataset_cleaned for x in dataset_keywords_cleaned):\n            filtered.append(dataset)\n        elif any(x+\" from\" in dataset_cleaned for x in dataset_keywords_cleaned):\n            filtered.append(dataset)\n        elif any(x+\" on\" in dataset_cleaned for x in dataset_keywords_cleaned):\n            filtered.append(dataset)\n    return filtered\n\ndef filter_by_abbreviations(dataset_names):\n    filtered = []\n    for dataset1 in dataset_names:\n        accepted = False\n        for dataset2 in long_forms_filtered:\n            if jaccard_similarity(dataset1,dataset2)>=0.7:\n                accepted = True\n        if accepted:\n            filtered.append(dataset1)\n    return filtered","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:25:00.10953Z","iopub.execute_input":"2021-06-17T20:25:00.109867Z","iopub.status.idle":"2021-06-17T20:25:00.118053Z","shell.execute_reply.started":"2021-06-17T20:25:00.109839Z","shell.execute_reply":"2021-06-17T20:25:00.117271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Heuristics Explained\nDataset names should be either ending with dataset keywords or consists of\n1. Extract all dataset candidates using some heuristics (Using regex find all keywords- look left and right to get sequence of capitalized words)\n2. filter_by_length - Remove dataset with no of words  <3 or >10\n3. Filter by abbreviation - filter dataset whose jaccard with longform list is > 0.7\n4. If no dataset was found in previous step - filter FP from datasets (in 2) by using logic similar to that used for abbreviations (DATASET-KEYWORD + of/for/on) \n5. If no dataset was found in previous step and no of dataset (in 2) < 3 - Use the list from (2)\n6. If no dataset was found in previous step and no of dataset (in 2) >= 3 -Return no predictions","metadata":{}},{"cell_type":"code","source":"predictions = pd.DataFrame(index=sample_submission.index,columns = ['PredictionString'])\nfor id_,row in tqdm(sample_submission.iterrows()):\n    if SIMULATE:\n        sentences = pd.read_csv(f'../input/coleridge-sud-sentences/sentences/tmp/sentences/{id_}.csv')\n    else:\n        sentences = split_sentences(id_)\n    sentences['sentence'] = sentences.sentence.apply(preprocess_sentence)\n    sentences['predicted_datasets'] = sentences.sentence.apply(extract_dataset_names)\n    sentences['predicted_datasets'] = sentences.predicted_datasets.apply(filter_by_length)\n    predicted_datasets_orig = np.unique(np.concatenate(sentences.predicted_datasets.values))\n    predicted_datasets = [clean_text_post(x) for x in predicted_datasets_orig]\n    predicted_datasets = filter_by_abbreviations(predicted_datasets)\n    if len(predicted_datasets)==0:\n        predicted_datasets = filter_false_positives(predicted_datasets_orig)\n        predicted_datasets = [clean_text_post(x) for x in predicted_datasets]\n    if len(predicted_datasets)==0 and len(predicted_datasets_orig)<3:\n        predicted_datasets = [clean_text_post(x) for x in predicted_datasets_orig]\n    predicted_datasets = filter_duplicates(predicted_datasets)\n    predicted_datasets = \"|\".join(predicted_datasets)\n    if predicted_datasets!=\"\":\n        predictions.loc[id_,'PredictionString'] = predicted_datasets","metadata":{"execution":{"iopub.status.busy":"2021-06-23T01:04:46.08892Z","iopub.execute_input":"2021-06-23T01:04:46.089594Z","iopub.status.idle":"2021-06-23T01:04:46.188093Z","shell.execute_reply.started":"2021-06-23T01:04:46.089447Z","shell.execute_reply":"2021-06-23T01:04:46.185991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"assert sample_submission.shape[0]==predictions.shape[0]\nassert (sample_submission.index.values==predictions.index.values).sum()==sample_submission.shape[0]\nsubmission = predictions.reset_index()\nsubmission.columns = ['Id','PredictionString']\nsubmission.to_csv('submission.csv',index=False)\nprint(submission.shape)\nsubmission.sample(2)","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:38:58.485108Z","iopub.execute_input":"2021-06-17T20:38:58.485603Z","iopub.status.idle":"2021-06-17T20:38:58.565006Z","shell.execute_reply.started":"2021-06-17T20:38:58.485574Z","shell.execute_reply":"2021-06-17T20:38:58.56428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.PredictionString.value_counts().head(1)","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:38:59.725946Z","iopub.execute_input":"2021-06-17T20:38:59.726459Z","iopub.status.idle":"2021-06-17T20:38:59.739063Z","shell.execute_reply.started":"2021-06-17T20:38:59.726427Z","shell.execute_reply":"2021-06-17T20:38:59.73836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission = pd.read_csv('submission.csv')\nsubmission.PredictionString.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:39:02.310277Z","iopub.execute_input":"2021-06-17T20:39:02.310803Z","iopub.status.idle":"2021-06-17T20:39:02.340455Z","shell.execute_reply.started":"2021-06-17T20:39:02.310774Z","shell.execute_reply":"2021-06-17T20:39:02.339756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Metric Implementation","metadata":{}},{"cell_type":"code","source":"def calculate_tp_fp_fn(pred_row):\n    \n    if pd.isnull(pred_row.cleaned_label):\n        true = []\n    else:\n        true = pred_row.cleaned_label.split('|')\n        \n    if pd.isnull(pred_row.PredictionString):\n        predicted = []\n    else:\n        predicted = pred_row.PredictionString.split('|')\n        \n    true = [clean_text_post(x) for x in true]\n    predicted = [clean_text_post(x) for x in predicted]\n    scores = pd.DataFrame(columns=['true','predicted','score'])\n    i = 0\n    for j,sample1 in enumerate(true):\n        for k,sample2 in enumerate(predicted):\n            scores.loc[i] = [j,k,jaccard_similarity(sample1,sample2)]\n            i += 1\n    scores = scores[scores.score>0.5].sort_values('score',ascending=False).reset_index(drop=True)\n    true_done = {}\n    predicted_done = {}\n    tp = 0\n    for i,row in scores.iterrows():\n        if (row.true not in true_done) and (row.predicted not in predicted_done):\n            tp += 1\n            true_done[row.true] = True\n            predicted_done[row.predicted] = True\n    fp = len(predicted) - tp\n    fn = len(true) - tp\n    pred_row['tp'] = tp\n    pred_row['fp'] = fp\n    pred_row['fn'] = fn\n    return pred_row\n\ndef F_beta(precision,recall,beta=0.5):\n    epsilon = 0.0000001\n    return (1+beta**2)*precision*recall/(((beta**2)*precision)+recall+epsilon)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if SIMULATE:\n    train_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv',index_col='Id')\n    all_labels = set(train_df.cleaned_label.unique())\n    ground_truth = train_df.reset_index().groupby('Id').cleaned_label.apply(lambda x: '|'.join(x)).reset_index()\n    print(ground_truth.shape)\n    predictions_df = pd.merge(ground_truth,submission,on='Id',how='inner')\n    tqdm.pandas()\n    scores = predictions_df.progress_apply(calculate_tp_fp_fn,axis=1)\n    scores.to_csv('scores.csv')\n    tp,fp,fn = scores[['tp','fp','fn']].sum().values\n    precision = tp/(tp+fp)\n    recall = tp/(tp+fn)\n    Fbeta05 = F_beta(precision,recall,beta=0.5)\n    print(\"tp =\",tp)\n    print(\"fp =\",fp)\n    print(\"fn =\",fn)\n    print(\"Precision =\",precision)\n    print(\"Recall =\",recall)\n    print(\"F Beta at 0.5 =\",Fbeta05)","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:39:13.586384Z","iopub.execute_input":"2021-06-17T20:39:13.586895Z","iopub.status.idle":"2021-06-17T20:41:04.0058Z","shell.execute_reply.started":"2021-06-17T20:39:13.586856Z","shell.execute_reply":"2021-06-17T20:41:04.004897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## False Positives","metadata":{}},{"cell_type":"code","source":"if SIMULATE:\n    print(scores[scores.fp>1].PredictionString.sample(50).values)","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:41:31.776801Z","iopub.execute_input":"2021-06-17T20:41:31.777167Z","iopub.status.idle":"2021-06-17T20:41:31.785629Z","shell.execute_reply.started":"2021-06-17T20:41:31.777139Z","shell.execute_reply":"2021-06-17T20:41:31.784514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## False Negatives","metadata":{}},{"cell_type":"code","source":"if SIMULATE:\n    print(scores[scores.fn>1].cleaned_label.drop_duplicates().sample(50).values)","metadata":{"execution":{"iopub.status.busy":"2021-06-17T20:41:34.997151Z","iopub.execute_input":"2021-06-17T20:41:34.997525Z","iopub.status.idle":"2021-06-17T20:41:35.007742Z","shell.execute_reply.started":"2021-06-17T20:41:34.997494Z","shell.execute_reply":"2021-06-17T20:41:35.006723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}