{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Coleridgh Initiabive 日本語EDA\n## お役に立ちましたらupvoteお願いいたします !","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. 要約\n## 簡単に言えば、学術論文の全文章内容から、dataset_labelという何について書かれているかの分類分けをするコンペだと思います。","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nimport os\nimport random\nimport json\nfrom tqdm import tqdm","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import re\nimport glob\nfrom textblob import TextBlob\nfrom functools import partial\nimport string\nimport nltk\nimport spacy\nfrom nltk.probability import FreqDist\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"RANDOM_SEED = 42\ndef seed_everything(seed=RANDOM_SEED):\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    random.seed(seed)\nseed_everything()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. データの中身確認","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(\"../input/coleridgeinitiative-show-us-the-data/train.csv\")\ntrain.head(3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for col in train.columns:\n    print(col + \":\" + str(len(train[col].unique())))\n\nprint(\"all rows = \" + str(len(train)))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Id　14316個 : 学術論文のidです。trainフォルダには、このid + \".json\" ファイルがあります。このjsonファイルが論文の全文章になります。\n\n\n* pub_title 14271個 : 学術論文の出版物のタイトルです。\n\n\n* dataset_title 45個 : パブリケーション内で言及されているデータセットのタイトル。(※　製作者が命名したものらしいです。)\n\n\n* dataset_label 130個 : これを予測します。データセットを示すテキストの一部。（※　論文筆者が使っている名前。省略などあるため、dataset_titleより数が多いらしいです。)\n\n                                                                                        ※ 印はコメントいただきました。ありがとうございます！\n                                                                                        \n                                                                                        \n\n* cleaned_label 130個 : evaluation項目にあるように、dataset_labelを小文字とかきれいに整形したものです。submissionはこの形にする必要があります(関数通すだけです)\n\n* 全部の行は19661個あるのに、たいしてそれぞれ重複しているものがある。（3.2で考察します)","metadata":{}},{"cell_type":"code","source":"sample = pd.read_csv(\"../input/coleridgeinitiative-show-us-the-data/sample_submission.csv\")\nsample","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Id : testフォルダにこのid + \".json\"ファイルがあります。このjsonファイルが論文の全文章になります。\nPreditionString : 後程説明しますが、dataset_labelをここに記載します。複数あると思う場合は、\"|\"で連結します。","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3 trainデータの中身確認","metadata":{}},{"cell_type":"code","source":"train.head(3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.1 論文を見る","metadata":{}},{"cell_type":"code","source":"train_path = \"../input/coleridgeinitiative-show-us-the-data/train\"\ntest_path = \"../input/coleridgeinitiative-show-us-the-data/test\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_train_path = [os.path.join(train_path,s) + \".json\" for s in train[\"Id\"]]\nall_test_path = [os.path.join(test_path,s) + \".json\" for s in sample[\"Id\"]]\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.1.1 まずは１つだけ","metadata":{}},{"cell_type":"code","source":"train.head(3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"json_path = all_train_path[0]\njson_path","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with open(json_path, 'r') as f:\n        json_decode = json.load(f)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"json_decode[:3] # すごく長いので、表示は最初から3つに。","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 細かく見るとsection titleとtextが交互に辞書型になっているように見える.pandasに食わせてみる","metadata":{}},{"cell_type":"code","source":"jsontest = pd.DataFrame(json_decode)\njsontest","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### jsonファイルは、section_titleとその中身がtextで書かれていることがわかる。学術論文を想像すればイメージつきやすいですね。","metadata":{}},{"cell_type":"markdown","source":"## タイトルとテキストを全文くっつけます。(textだけくっつけたいとかは、以下を編集してください)","metadata":{}},{"cell_type":"code","source":"texts = \"\"\n\nfor a in jsontest.values:\n    texts += a[0] +\" \"+ a[1] + \" \"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"texts[:300]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"json_path = all_train_path[0]\nwith open(json_path, 'r') as f:\n        json_decode = json.load(f)\n\njsontest = pd.DataFrame(json_decode)\n\ntexts = \"\"\n\nfor a in jsontest.values:\n    texts += a[0] +\" \"+ a[1] +\" \"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## train-data全部に対して、それを行う。(↑の3.1.1というところからここまでをひとくくりにしてfor文で回すだけ)","metadata":{}},{"cell_type":"code","source":"%%time\n\nalltexts = []\n\nfor json_path in tqdm(all_train_path):\n\n    with open(json_path, 'r') as f:\n            json_decode = json.load(f)\n    jsontest = pd.DataFrame(json_decode)\n\n    texts = \"\"\n\n    for a in jsontest.values:\n        texts += a[0] +\" \"+ a[1] + \" \"\n        \n    alltexts.append(texts)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train[\"text\"] = alltexts\ntrain","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3.2 考察","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.2.1 同じid列について。2の結果から、全部で19661行あるのに、idのuniqueは、14316個。つまり、同じ論文でも行を分けて、複数存在している。","metadata":{}},{"cell_type":"code","source":"train.head(3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(train[\"Id\"].unique()))\nprint(len(train))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Idgroup = train.groupby(\"Id\")[\"dataset_label\"].count().reset_index()\nIdgroup.columns = [\"Id\",\"count\"]\nIdgroup = Idgroup.sort_values(\"count\").reset_index(drop=True)\nIdgroup","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 一番多いIdを見てみる","metadata":{}},{"cell_type":"code","source":"mostId = train[train[\"Id\"] == Idgroup[\"Id\"].iloc[-1]]\nmostId.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 同じ論文でも異なるdataset_title,dataset_labelを持っていることがわかる。","metadata":{}},{"cell_type":"markdown","source":"### submissionのPredictionStringには、複数のdataset_labelを持つときは、\"|\"でつなげるとあるので、この場合お試しで作成してみる。","metadata":{}},{"cell_type":"code","source":"mostIdlist = mostId[\"cleaned_label\"].to_list()\nmostIdlist","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mostIdlist = (\"|\").join(mostIdlist)\nmostIdlist","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### この場合、少し長いが、このようにPredictionStringに記載してsubmitする(後程お試しでsubmitできる形にしています)","metadata":{}},{"cell_type":"code","source":"sample","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 少し冗長になってしまいそうなので、割愛しますが、publicationのtitleの重複なども同様です。同じpublicationでも複数論文があったり、同じ論文の行が何行かあったりします。","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4.検証","metadata":{}},{"cell_type":"markdown","source":"### 4.1 仮説 : 全文章の中に、dataset label, cleaned dataset labelで出てくるワードが含まれていれば、\n### そのワードをsubmissionしてあげればよいのでは ?\n### それを検証するために、trainデータのこれらの項目が全文章に入っているかの検証をする\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"#### テストデータには、この中に含まれていないものもあるみたいです。このやり方が正しいわけではなく、検証です。\n#### でも、Shopeeコンペみたいに最後はこれとマージするのかなとは思います。","metadata":{}},{"cell_type":"code","source":"train.head(3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## v6 : dataset_labelは、以下の関数を通したらcleaned_labelになるようにevaluationで定義されているが・・・","metadata":{}},{"cell_type":"code","source":"# 最後はこの形に\ndef clean_text(txt):\n    return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"check = []\nfor a in range(len(train)):\n    \n    if clean_text(train[\"dataset_label\"].iloc[a]) == train[\"cleaned_label\"].iloc[a]:\n        check.append(1)\n    else:\n        check.append(0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.sum(check)/len(train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 1にならない・・・確認してみる","metadata":{}},{"cell_type":"code","source":"train[\"check\"] = check\ncheckdf = train[train[\"check\"]==0]\ncheckdf.head(3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checkdf[\"dataset_label\"].unique()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"checkdf[\"cleaned_label\"].unique()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clean_text(checkdf[\"dataset_label\"].iloc[0])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 最後のスペースが大本は消えていない。","metadata":{}},{"cell_type":"code","source":"cleanlabel = []\nfor a in tqdm(train[\"cleaned_label\"]):\n    if a[-1] == \" \":\n        cleanlabel.append(a[:-1])\n    else:\n        cleanlabel.append(a)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"check = []\nfor a in range(len(train)):\n    \n    if clean_text(train[\"dataset_label\"].iloc[a]) == cleanlabel[a]:\n        check.append(1)\n    else:\n        check.append(0)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.sum(check)/len(train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### きちんと1になったので、置換します。","metadata":{}},{"cell_type":"code","source":"train[\"cleaned_label\"]=cleanlabel","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### ↓は以下を踏まえて。","metadata":{}},{"cell_type":"code","source":"train.head(3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### train[\"text\"]をきれいに","metadata":{}},{"cell_type":"code","source":"train[\"text\"] = [clean_text(s) for s in tqdm(train[\"text\"])]\n    ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### dataset_labelのuniqueのリスト化","metadata":{}},{"cell_type":"code","source":"dslabel = [clean_text(s) for s in train[\"dataset_label\"].unique()]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(dslabel)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labeljudge = []\nall_labels = []\nlabel_len = []\n\nfor a in tqdm(train[\"text\"]):\n    labels = []\n    for b in dslabel:\n        if b in a:\n            labels.append(clean_text(b))\n            break\n    if len(labels)==0:\n        labeljudge.append(0)\n    else:\n        labeljudge.append(1)\n    \n    #all_labels.append(\"|\".join(labels))\n    #label_len.append(len(labels))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.sum(labeljudge)/len(train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 100%一致していることを確認。","metadata":{}},{"cell_type":"markdown","source":"# 4.2 testデータと提出","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 上記結果から、テストデータで、exist_labelを含んでいたらdataset_labelとしてcleaningして提出する。※複数該当するときは\"|\"で繋げるルール。","metadata":{}},{"cell_type":"code","source":"sample","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2.1 論文を読み込む","metadata":{}},{"cell_type":"code","source":"%%time\n\nalltexts = []\n\nfor json_path in tqdm(all_test_path):\n\n    with open(json_path, 'r') as f:\n            json_decode = json.load(f)\n    jsontest = pd.DataFrame(json_decode)\n\n    texts = \"\"\n\n    for a in jsontest.values:\n        texts += a[0] + \" \" + a[1] + \" \"\n        \n    alltexts.append(texts)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample[\"text\"] = alltexts","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2.2 論文をきれいにする","metadata":{}},{"cell_type":"code","source":"sample[\"text\"] = [clean_text(s) for s in tqdm(sample[\"text\"])]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2.3 dslabelにワードがあれば、その言葉を抜いて結合する","metadata":{}},{"cell_type":"markdown","source":"#### v11 少しだけスコアが上がるので、dataset_titleもdslabelに追加します。","metadata":{}},{"cell_type":"code","source":"print(len(dslabel))\ndstitle = [clean_text(s) for s in train[\"dataset_title\"].unique()]\ndslabel = set(dslabel + dstitle) # setを使うことで重複を削除してくれます。\nlen(dslabel)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labeljudge = []\nall_labels = []\nlabel_len = []\n\nfor a in tqdm(sample[\"text\"]):\n    labels = []\n    for b in dslabel:\n        if b in a:\n            labels.append(clean_text(b))\n            \n    if len(labels)==0:\n        labeljudge.append(0)\n    else:\n        labeljudge.append(1)\n    \n    all_labels.append(\"|\".join(labels))\n    label_len.append(len(labels))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample[\"PredictionString\"] = all_labels\nsample","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample[\"PredictionString\"].iloc[2]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample = sample[[\"Id\",\"PredictionString\"]]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample.to_csv(\"submission.csv\",index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ここまで見ていただいてありがとうございます。\n## 少しでもお役に立ちましたらupvote/followしてもらえると嬉しいです !","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}