{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Coleridge Initiative - Show US the Data\n### Discover how data is used for the public good\n\nThis competition challenges data scientists to show how publicly funded data are used to serve science and society. Evidence through data is critical if government is to address the many threats facing society, including; pandemics, climate change, Alzheimer’s disease, child hunger, increasing food production, maintaining biodiversity, and addressing many other challenges. Yet much of the information about data necessary to inform evidence and science is locked inside publications.\n\n","metadata":{}},{"cell_type":"markdown","source":"The objective of the competition is to identify the mention of datasets within scientific publications. Your predictions will be short excerpts from the publications that appear to note a dataset. Predictions that more accurately match the precise words used to identify the dataset within the publication will score higher. Predictions should be cleaned using the clean_text function from the Evaluation page to ensure proper matching.\n\nPublications are provided in JSON format, broken up into sections with section titles.\n\nThe goal in this competition is not just to match known dataset strings but to generalize to datasets that have never been seen before using NLP and statistical techniques. A percentage of the public test set publications are drawn from the training set - not all datasets have been identified in train, so these unidentified datasets have been used as a portion of the public test labels. These should serve as guides for the difficult task of labeling the private test set.\n\nNote that the hidden test set has roughly ~8000 publications, many times the size of the public test set. Plan \nyour compute time accordingly.\n\n\n#### This competition to classify the written about dataset_label in whole of text content of an academic paper. ¶","metadata":{}},{"cell_type":"markdown","source":"## 1. Import Required Libraries","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore', category=DeprecationWarning)\nwarnings.filterwarnings('ignore', category=FutureWarning)\n\nimport os\nimport re\nimport json\nimport glob\nfrom collections import defaultdict\nfrom textblob import TextBlob\nfrom functools import partial\n\nimport pandas as pd\nimport numpy as np\n\nimport matplotlib.pyplot as plt\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport seaborn as sns\n\nimport nltk\nimport spacy\nnlp = spacy.load('en_core_web_lg', disable=['parser', 'ner'])\nnlp.max_length = 4000000\nfrom nltk.probability import FreqDist\nfrom wordcloud import WordCloud, STOPWORDS\n\nfrom tqdm.autonotebook import tqdm\nimport string\n\n%matplotlib inline\n\nos.listdir('../input/coleridgeinitiative-show-us-the-data')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-05-26T08:21:58.787443Z","iopub.execute_input":"2021-05-26T08:21:58.788214Z","iopub.status.idle":"2021-05-26T08:22:07.925417Z","shell.execute_reply.started":"2021-05-26T08:21:58.788148Z","shell.execute_reply":"2021-05-26T08:22:07.924425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import random\nrandom_seed = 42\ndef seed_all(seed=random_seed):\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    random.seed(seed)\nseed_all()","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:07.926846Z","iopub.execute_input":"2021-05-26T08:22:07.927114Z","iopub.status.idle":"2021-05-26T08:22:07.934797Z","shell.execute_reply.started":"2021-05-26T08:22:07.927087Z","shell.execute_reply":"2021-05-26T08:22:07.934011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Explore the data ","metadata":{}},{"cell_type":"markdown","source":"## Data\n\n#### Files\n1. train - the full text of the training set's publications in JSON format, broken into sections with section titles\n2. test - the full text of the test set's publications in JSON format, broken into sections with section titles\n3. train.csv - labels and metadata for the training set\n4. sample_submission.csv - a sample submission file in the correct format\n\n#### Columns\n1. id - publication id - note that there are multiple rows for some training documents, indicating multiple mentioned datasets\n2. pub_title - title of the publication (a small number of publications have the same title)\n3. dataset_title - the title of the dataset that is mentioned within the publication\n4. dataset_label - a portion of the text that indicates the dataset\n5. cleaned_label - the dataset_label, as passed through the clean_text function from the Evaluation page","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(\"../input/coleridgeinitiative-show-us-the-data/train.csv\")\nsample = pd.read_csv(\"../input/coleridgeinitiative-show-us-the-data/sample_submission.csv\")\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:07.936897Z","iopub.execute_input":"2021-05-26T08:22:07.937206Z","iopub.status.idle":"2021-05-26T08:22:08.306665Z","shell.execute_reply.started":"2021-05-26T08:22:07.937176Z","shell.execute_reply":"2021-05-26T08:22:08.305731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.nunique()","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:08.308342Z","iopub.execute_input":"2021-05-26T08:22:08.308626Z","iopub.status.idle":"2021-05-26T08:22:08.346655Z","shell.execute_reply.started":"2021-05-26T08:22:08.3086Z","shell.execute_reply":"2021-05-26T08:22:08.345766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:08.347942Z","iopub.execute_input":"2021-05-26T08:22:08.348218Z","iopub.status.idle":"2021-05-26T08:22:08.353209Z","shell.execute_reply.started":"2021-05-26T08:22:08.348192Z","shell.execute_reply":"2021-05-26T08:22:08.352334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.describe().T","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:08.354699Z","iopub.execute_input":"2021-05-26T08:22:08.355033Z","iopub.status.idle":"2021-05-26T08:22:08.43298Z","shell.execute_reply.started":"2021-05-26T08:22:08.355003Z","shell.execute_reply":"2021-05-26T08:22:08.431976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"1. **Id 14316**: The id of an academic paper. The train folder contains this id + \".json\" file. This json file will be the full text of the treatise.\n\n2. **pub_title 14271**: The title of the publication of the academic paper.\n\n3. **dataset_title 45**: The title of the dataset mentioned in the publication. \n\n4. **dataset_label 130**: Predict this. Part of the text that indicates the dataset. (The name used by the author of the treatise. It seems that there are more number of dataset_title because there are missing items)\n    \n5. **cleaned_label 130 pieces**: As shown in the evaluation item, dataset_label is nicely formatted with lowercase letters. The submission should be in this form \n\n#### There are 19661 lines in all, but there are many duplicates. (we will explore further in following section)","metadata":{}},{"cell_type":"code","source":"sample","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:08.435661Z","iopub.execute_input":"2021-05-26T08:22:08.436099Z","iopub.status.idle":"2021-05-26T08:22:08.445058Z","shell.execute_reply.started":"2021-05-26T08:22:08.436067Z","shell.execute_reply":"2021-05-26T08:22:08.444107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample.describe().T","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:08.448039Z","iopub.execute_input":"2021-05-26T08:22:08.448687Z","iopub.status.idle":"2021-05-26T08:22:08.474143Z","shell.execute_reply.started":"2021-05-26T08:22:08.448637Z","shell.execute_reply":"2021-05-26T08:22:08.47306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"id + \".json\" file in the Id: test folder. This json file will be the full text of the treatise. \n\n**PreditionString**: As we will see later, we will list the dataset_label here. If you think there are more than one, connect them with \"|\".","metadata":{}},{"cell_type":"markdown","source":"### View the train Data ","metadata":{}},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:08.476226Z","iopub.execute_input":"2021-05-26T08:22:08.476594Z","iopub.status.idle":"2021-05-26T08:22:08.498149Z","shell.execute_reply.started":"2021-05-26T08:22:08.476555Z","shell.execute_reply":"2021-05-26T08:22:08.496839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_path = \"../input/coleridgeinitiative-show-us-the-data/train\"\ntest_path = \"../input/coleridgeinitiative-show-us-the-data/test\"\n","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:08.500367Z","iopub.execute_input":"2021-05-26T08:22:08.501111Z","iopub.status.idle":"2021-05-26T08:22:08.50974Z","shell.execute_reply.started":"2021-05-26T08:22:08.501008Z","shell.execute_reply":"2021-05-26T08:22:08.508891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_train_path = [os.path.join(train_path,s) + \".json\" for s in train[\"Id\"]]\nall_test_path = [os.path.join(test_path,s) + \".json\" for s in sample[\"Id\"]]","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:08.511045Z","iopub.execute_input":"2021-05-26T08:22:08.511401Z","iopub.status.idle":"2021-05-26T08:22:08.570897Z","shell.execute_reply.started":"2021-05-26T08:22:08.511372Z","shell.execute_reply":"2021-05-26T08:22:08.569847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"json_path = all_train_path[0]\njson_path","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:08.572338Z","iopub.execute_input":"2021-05-26T08:22:08.572755Z","iopub.status.idle":"2021-05-26T08:22:08.579838Z","shell.execute_reply.started":"2021-05-26T08:22:08.572712Z","shell.execute_reply":"2021-05-26T08:22:08.578912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with open(json_path, 'r') as f:\n        json_decode = json.load(f)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:08.58126Z","iopub.execute_input":"2021-05-26T08:22:08.581834Z","iopub.status.idle":"2021-05-26T08:22:08.606782Z","shell.execute_reply.started":"2021-05-26T08:22:08.581775Z","shell.execute_reply":"2021-05-26T08:22:08.606009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"json_decode[:1]","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:08.607988Z","iopub.execute_input":"2021-05-26T08:22:08.608551Z","iopub.status.idle":"2021-05-26T08:22:08.615153Z","shell.execute_reply.started":"2021-05-26T08:22:08.608509Z","shell.execute_reply":"2021-05-26T08:22:08.614086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### After review of the above data closely, it looks like that section title and text are alternately dictionary type. we can create a dataframe","metadata":{}},{"cell_type":"code","source":"jsontest = pd.DataFrame(json_decode)\njsontest","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:08.616499Z","iopub.execute_input":"2021-05-26T08:22:08.616883Z","iopub.status.idle":"2021-05-26T08:22:08.934594Z","shell.execute_reply.started":"2021-05-26T08:22:08.616846Z","shell.execute_reply":"2021-05-26T08:22:08.93334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Look at the above file it contains section_title and its contents written in text. It's easy to get an image if you imagine an academic paper. \n\n### Let us attach the full text of the title and text as follows ","metadata":{}},{"cell_type":"code","source":"texts = \"\"\n\nfor a in jsontest.values:\n    texts += a[0] +\" \"+ a[1] + \" \"","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:08.936132Z","iopub.execute_input":"2021-05-26T08:22:08.936459Z","iopub.status.idle":"2021-05-26T08:22:08.941245Z","shell.execute_reply.started":"2021-05-26T08:22:08.936431Z","shell.execute_reply":"2021-05-26T08:22:08.940171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"texts[:300]","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:08.942374Z","iopub.execute_input":"2021-05-26T08:22:08.94279Z","iopub.status.idle":"2021-05-26T08:22:08.954978Z","shell.execute_reply.started":"2021-05-26T08:22:08.942752Z","shell.execute_reply":"2021-05-26T08:22:08.953962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"json_path = all_train_path[0]\nwith open(json_path, 'r') as f:\n        json_decode = json.load(f)\n\njsontest = pd.DataFrame(json_decode)\n\ntexts = \"\"\n\nfor a in jsontest.values:\n    texts += a[0] +\" \"+ a[1] +\" \"","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:13.315588Z","iopub.execute_input":"2021-05-26T08:22:13.315964Z","iopub.status.idle":"2021-05-26T08:22:13.323715Z","shell.execute_reply.started":"2021-05-26T08:22:13.31593Z","shell.execute_reply":"2021-05-26T08:22:13.322764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let us execute this for all of train dataset with a for statement) ","metadata":{}},{"cell_type":"code","source":"%%time\n\nalltexts = []\n\nfor json_path in tqdm(all_train_path):\n\n    with open(json_path, 'r') as f:\n            json_decode = json.load(f)\n    jsontest = pd.DataFrame(json_decode)\n\n    texts = \"\"\n\n    for a in jsontest.values:\n        texts += a[0] +\" \"+ a[1] + \" \"\n        \n    alltexts.append(texts)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:22:16.905902Z","iopub.execute_input":"2021-05-26T08:22:16.906434Z","iopub.status.idle":"2021-05-26T08:24:00.413013Z","shell.execute_reply.started":"2021-05-26T08:22:16.906402Z","shell.execute_reply":"2021-05-26T08:24:00.411944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"text\"] = alltexts\ntrain","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:24:15.591138Z","iopub.execute_input":"2021-05-26T08:24:15.59148Z","iopub.status.idle":"2021-05-26T08:24:15.618667Z","shell.execute_reply.started":"2021-05-26T08:24:15.591452Z","shell.execute_reply":"2021-05-26T08:24:15.617774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Idgroup = train.groupby(\"Id\")[\"dataset_label\"].count().reset_index()\nIdgroup.columns = [\"Id\",\"count\"]\nIdgroup = Idgroup.sort_values(\"count\").reset_index(drop=True)\nIdgroup","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:25:13.656568Z","iopub.execute_input":"2021-05-26T08:25:13.656967Z","iopub.status.idle":"2021-05-26T08:25:13.702966Z","shell.execute_reply.started":"2021-05-26T08:25:13.656935Z","shell.execute_reply":"2021-05-26T08:25:13.701945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Most IDs ","metadata":{}},{"cell_type":"code","source":"mostId = train[train[\"Id\"] == Idgroup[\"Id\"].iloc[-1]]\nmostId.head()","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:25:26.492237Z","iopub.execute_input":"2021-05-26T08:25:26.492793Z","iopub.status.idle":"2021-05-26T08:25:26.585739Z","shell.execute_reply.started":"2021-05-26T08:25:26.492742Z","shell.execute_reply":"2021-05-26T08:25:26.584729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Look at the above data same ID has different dataset_title and dataset_label.\n### There are multiple dataset_labels in the PredictionString of submission, they are connected by \"|\", so in this case, let us try creating them","metadata":{}},{"cell_type":"code","source":"mostIdlist = mostId[\"cleaned_label\"].to_list()\nmostIdlist\n","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:30:49.550153Z","iopub.execute_input":"2021-05-26T08:30:49.550545Z","iopub.status.idle":"2021-05-26T08:30:49.557523Z","shell.execute_reply.started":"2021-05-26T08:30:49.550512Z","shell.execute_reply":"2021-05-26T08:30:49.556458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mostIdlist = (\"|\").join(mostIdlist)\nmostIdlist","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:30:59.088447Z","iopub.execute_input":"2021-05-26T08:30:59.088884Z","iopub.status.idle":"2021-05-26T08:30:59.094781Z","shell.execute_reply.started":"2021-05-26T08:30:59.088848Z","shell.execute_reply":"2021-05-26T08:30:59.093934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Look at the above, very long, but it is described in PredictionString like this and submitted. Let us see how to handle this","metadata":{}},{"cell_type":"code","source":"sample","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:34:51.509047Z","iopub.execute_input":"2021-05-26T08:34:51.509443Z","iopub.status.idle":"2021-05-26T08:34:51.519233Z","shell.execute_reply.started":"2021-05-26T08:34:51.509411Z","shell.execute_reply":"2021-05-26T08:34:51.518211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### There may be multiple papers in the same publication, or there may be several lines in the same paper. ¶","metadata":{}},{"cell_type":"markdown","source":"#### 1 Hypothesis: If the whole sentence contains words that appear in the dataset label and cleaned dataset label, check that these are part of train data are included in the full text.\n#### Looks like  some test data is not included in this. ","metadata":{}},{"cell_type":"markdown","source":"#### dataset_label is defined in evaluation. It becomes cleaned_label when it passes through the following function ...","metadata":{}},{"cell_type":"code","source":"def clean_text(txt):\n    return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:43:04.488195Z","iopub.execute_input":"2021-05-26T08:43:04.488578Z","iopub.status.idle":"2021-05-26T08:43:04.493057Z","shell.execute_reply.started":"2021-05-26T08:43:04.488546Z","shell.execute_reply":"2021-05-26T08:43:04.492094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"check = []\nfor a in range(len(train)):\n    \n    if clean_text(train[\"dataset_label\"].iloc[a]) == train[\"cleaned_label\"].iloc[a]:\n        check.append(1)\n    else:\n        check.append(0)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:43:12.154838Z","iopub.execute_input":"2021-05-26T08:43:12.155348Z","iopub.status.idle":"2021-05-26T08:43:12.724894Z","shell.execute_reply.started":"2021-05-26T08:43:12.155302Z","shell.execute_reply":"2021-05-26T08:43:12.724136Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.sum(check)/len(train)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:43:15.636013Z","iopub.execute_input":"2021-05-26T08:43:15.636551Z","iopub.status.idle":"2021-05-26T08:43:15.646174Z","shell.execute_reply.started":"2021-05-26T08:43:15.636518Z","shell.execute_reply":"2021-05-26T08:43:15.645122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let me check it should ideally be 1","metadata":{}},{"cell_type":"code","source":"train[\"check\"] = check\ncheckdf = train[train[\"check\"]==0]\ncheckdf.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:43:49.864654Z","iopub.execute_input":"2021-05-26T08:43:49.865066Z","iopub.status.idle":"2021-05-26T08:43:49.895341Z","shell.execute_reply.started":"2021-05-26T08:43:49.865031Z","shell.execute_reply":"2021-05-26T08:43:49.89419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checkdf[\"dataset_label\"].unique()","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:44:01.535436Z","iopub.execute_input":"2021-05-26T08:44:01.535835Z","iopub.status.idle":"2021-05-26T08:44:01.543884Z","shell.execute_reply.started":"2021-05-26T08:44:01.535784Z","shell.execute_reply":"2021-05-26T08:44:01.542911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checkdf[\"cleaned_label\"].unique()","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:44:05.373617Z","iopub.execute_input":"2021-05-26T08:44:05.373978Z","iopub.status.idle":"2021-05-26T08:44:05.380685Z","shell.execute_reply.started":"2021-05-26T08:44:05.373948Z","shell.execute_reply":"2021-05-26T08:44:05.379655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clean_text(checkdf[\"dataset_label\"].iloc[2])","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:44:13.16344Z","iopub.execute_input":"2021-05-26T08:44:13.163788Z","iopub.status.idle":"2021-05-26T08:44:13.169689Z","shell.execute_reply.started":"2021-05-26T08:44:13.163756Z","shell.execute_reply":"2021-05-26T08:44:13.168718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cleanlabel = []\nfor a in tqdm(train[\"cleaned_label\"]):\n    if a[-1] == \" \":\n        cleanlabel.append(a[:-1])\n    else:\n        cleanlabel.append(a)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:44:16.969747Z","iopub.execute_input":"2021-05-26T08:44:16.970148Z","iopub.status.idle":"2021-05-26T08:44:17.034805Z","shell.execute_reply.started":"2021-05-26T08:44:16.970112Z","shell.execute_reply":"2021-05-26T08:44:17.034037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"check = []\nfor a in range(len(train)):\n    \n    if clean_text(train[\"dataset_label\"].iloc[a]) == cleanlabel[a]:\n        check.append(1)\n    else:\n        check.append(0)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:44:20.323753Z","iopub.execute_input":"2021-05-26T08:44:20.324318Z","iopub.status.idle":"2021-05-26T08:44:20.685014Z","shell.execute_reply.started":"2021-05-26T08:44:20.324269Z","shell.execute_reply":"2021-05-26T08:44:20.683918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.sum(check)/len(train)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:44:25.836741Z","iopub.execute_input":"2021-05-26T08:44:25.837118Z","iopub.status.idle":"2021-05-26T08:44:25.847216Z","shell.execute_reply.started":"2021-05-26T08:44:25.837086Z","shell.execute_reply":"2021-05-26T08:44:25.846085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now its correct and we will replace this ","metadata":{}},{"cell_type":"code","source":"train[\"cleaned_label\"]=cleanlabel","metadata":{"execution":{"iopub.status.busy":"2021-05-26T08:45:17.21652Z","iopub.execute_input":"2021-05-26T08:45:17.216912Z","iopub.status.idle":"2021-05-26T08:45:17.224085Z","shell.execute_reply.started":"2021-05-26T08:45:17.216877Z","shell.execute_reply":"2021-05-26T08:45:17.222824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"text\"] = [clean_text(s) for s in tqdm(train[\"text\"])]","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:06:40.348159Z","iopub.execute_input":"2021-05-26T09:06:40.348529Z","iopub.status.idle":"2021-05-26T09:08:11.589254Z","shell.execute_reply.started":"2021-05-26T09:06:40.348495Z","shell.execute_reply":"2021-05-26T09:08:11.588365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dslabel = [clean_text(s) for s in train[\"dataset_label\"].unique()]","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:08:11.591056Z","iopub.execute_input":"2021-05-26T09:08:11.591529Z","iopub.status.idle":"2021-05-26T09:08:11.598358Z","shell.execute_reply.started":"2021-05-26T09:08:11.591492Z","shell.execute_reply":"2021-05-26T09:08:11.597496Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"List of unique dataset_label","metadata":{}},{"cell_type":"code","source":"len(dslabel)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:06:29.559181Z","iopub.execute_input":"2021-05-26T09:06:29.559715Z","iopub.status.idle":"2021-05-26T09:06:29.565576Z","shell.execute_reply.started":"2021-05-26T09:06:29.559664Z","shell.execute_reply":"2021-05-26T09:06:29.564884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labeljudge = []\nall_labels = []\nlabel_len = []\n\nfor a in tqdm(train[\"text\"]):\n    labels = []\n    for b in dslabel:\n        if b in a:\n            labels.append(clean_text(b))\n            break\n    if len(labels)==0:\n        labeljudge.append(0)\n    else:\n        labeljudge.append(1)\n    \n    #all_labels.append(\"|\".join(labels))\n    #label_len.append(len(labels))","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:08:11.599652Z","iopub.execute_input":"2021-05-26T09:08:11.600102Z","iopub.status.idle":"2021-05-26T09:08:28.073157Z","shell.execute_reply.started":"2021-05-26T09:08:11.600062Z","shell.execute_reply":"2021-05-26T09:08:28.07236Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.sum(labeljudge)/len(train)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:08:59.778614Z","iopub.execute_input":"2021-05-26T09:08:59.779014Z","iopub.status.idle":"2021-05-26T09:08:59.789022Z","shell.execute_reply.started":"2021-05-26T09:08:59.778975Z","shell.execute_reply":"2021-05-26T09:08:59.788125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Test Data and Submission","metadata":{}},{"cell_type":"markdown","source":"### From the above results, if the test data includes exist_label, clean it as dataset_label and submit it. * A rule that connects multiple items with \"|\". ","metadata":{}},{"cell_type":"code","source":"sample","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:10:28.174674Z","iopub.execute_input":"2021-05-26T09:10:28.175115Z","iopub.status.idle":"2021-05-26T09:10:28.186525Z","shell.execute_reply.started":"2021-05-26T09:10:28.175076Z","shell.execute_reply":"2021-05-26T09:10:28.185427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"alltexts = []\n\nfor json_path in (all_test_path):\n\n    with open(json_path, 'r') as f:\n            json_decode = json.load(f)\n    jsontest = pd.DataFrame(json_decode)\n\n    texts = \"\"\n\n    for a in jsontest.values:\n        texts += a[0] + \" \" + a[1] + \" \"\n        \n    alltexts.append(texts)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:11:08.81022Z","iopub.execute_input":"2021-05-26T09:11:08.810583Z","iopub.status.idle":"2021-05-26T09:11:08.900241Z","shell.execute_reply.started":"2021-05-26T09:11:08.81055Z","shell.execute_reply":"2021-05-26T09:11:08.899089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample[\"text\"] = alltexts","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:11:32.048866Z","iopub.execute_input":"2021-05-26T09:11:32.049216Z","iopub.status.idle":"2021-05-26T09:11:32.054958Z","shell.execute_reply.started":"2021-05-26T09:11:32.049185Z","shell.execute_reply":"2021-05-26T09:11:32.053952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:11:47.562563Z","iopub.execute_input":"2021-05-26T09:11:47.563019Z","iopub.status.idle":"2021-05-26T09:11:47.579394Z","shell.execute_reply.started":"2021-05-26T09:11:47.562981Z","shell.execute_reply":"2021-05-26T09:11:47.578363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Clean the treatise","metadata":{}},{"cell_type":"code","source":"sample[\"text\"] = [clean_text(s) for s in tqdm(sample[\"text\"])]","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:12:45.69231Z","iopub.execute_input":"2021-05-26T09:12:45.692645Z","iopub.status.idle":"2021-05-26T09:12:45.77003Z","shell.execute_reply.started":"2021-05-26T09:12:45.692617Z","shell.execute_reply":"2021-05-26T09:12:45.768949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### If there is a word in dslabel, remove the word and merge it, add dataset_title to dslabel as it will increase the score. ¶","metadata":{}},{"cell_type":"code","source":"print(len(dslabel))\ndstitle = [clean_text(s) for s in train[\"dataset_title\"].unique()]\ndslabel = set(dslabel + dstitle) \nlen(dslabel)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:13:51.898983Z","iopub.execute_input":"2021-05-26T09:13:51.899366Z","iopub.status.idle":"2021-05-26T09:13:51.910628Z","shell.execute_reply.started":"2021-05-26T09:13:51.899334Z","shell.execute_reply":"2021-05-26T09:13:51.909443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labeljudge = []\nall_labels = []\nlabel_len = []\n\nfor a in tqdm(sample[\"text\"]):\n    labels = []\n    for b in dslabel:\n        if b in a:\n            labels.append(clean_text(b))\n            \n    if len(labels)==0:\n        labeljudge.append(0)\n    else:\n        labeljudge.append(1)\n    \n    all_labels.append(\"|\".join(labels))\n    label_len.append(len(labels))","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:14:04.439621Z","iopub.execute_input":"2021-05-26T09:14:04.440008Z","iopub.status.idle":"2021-05-26T09:14:04.505617Z","shell.execute_reply.started":"2021-05-26T09:14:04.439974Z","shell.execute_reply":"2021-05-26T09:14:04.504895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample[\"PredictionString\"] = all_labels\nsample","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:14:16.731638Z","iopub.execute_input":"2021-05-26T09:14:16.731999Z","iopub.status.idle":"2021-05-26T09:14:16.744884Z","shell.execute_reply.started":"2021-05-26T09:14:16.731969Z","shell.execute_reply":"2021-05-26T09:14:16.74384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample[\"PredictionString\"].iloc[3]","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:14:31.857371Z","iopub.execute_input":"2021-05-26T09:14:31.858033Z","iopub.status.idle":"2021-05-26T09:14:31.866359Z","shell.execute_reply.started":"2021-05-26T09:14:31.857983Z","shell.execute_reply":"2021-05-26T09:14:31.865218Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample = sample[[\"Id\",\"PredictionString\"]]","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:14:38.236326Z","iopub.execute_input":"2021-05-26T09:14:38.236666Z","iopub.status.idle":"2021-05-26T09:14:38.242308Z","shell.execute_reply.started":"2021-05-26T09:14:38.236636Z","shell.execute_reply":"2021-05-26T09:14:38.241155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample.to_csv(\"submission.csv\",index=False)","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:14:40.737665Z","iopub.execute_input":"2021-05-26T09:14:40.738027Z","iopub.status.idle":"2021-05-26T09:14:40.748976Z","shell.execute_reply.started":"2021-05-26T09:14:40.737997Z","shell.execute_reply":"2021-05-26T09:14:40.74792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample","metadata":{"execution":{"iopub.status.busy":"2021-05-26T09:14:42.700761Z","iopub.execute_input":"2021-05-26T09:14:42.70115Z","iopub.status.idle":"2021-05-26T09:14:42.711371Z","shell.execute_reply.started":"2021-05-26T09:14:42.701118Z","shell.execute_reply":"2021-05-26T09:14:42.710324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Credits to lot of Kagglers who have spent time in understanding the problem and solivng this\n\n# I am absolute beginner and learning from great stuff people have done, please comment and if you like upvote and share it further. I will definetely try to improve this further. \n\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}