{"cells":[{"metadata":{},"cell_type":"markdown","source":"Hey guys, just noticed that \n\n(I haven't found nothing on the [discussion](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/discussion) tab, if it's a \"repost\", sorry for that!)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# As per the \"evaluation\" page:\nimport re\ndef jaccard_similarity(s1, s2):\n    l1 = s1.split(\" \")\n    l2 = s2.split(\" \")    \n    intersection = len(list(set(l1).intersection(l2)))\n    union = (len(l1) + len(l2)) - intersection\n    return float(intersection) / union\n\ndef clean_text(txt):\n    return re.sub('[^A-Za-z0-9]+', ' ', str(txt).lower()).strip()    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\ntrain = pd.read_csv(\"/kaggle/input/coleridgeinitiative-show-us-the-data/train.csv\")\ntrain.head(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"According to the [data](https://www.kaggle.com/c/coleridgeinitiative-show-us-the-data/data) page, the `cleaned_label` column is `dataset_label` as passed through the clean_text function from the Evaluation page ..."},{"metadata":{"trusted":true},"cell_type":"code","source":"(train['dataset_label'].apply(clean_text) == train.cleaned_label).all()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Wait, what? Isn't that strange?\n\nLet's investigate further ..."},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train['dataset_label'].apply(clean_text) != train.cleaned_label].head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['dataset_label'].apply(clean_text).iloc[436]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.cleaned_label.iloc[436]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Mhh, almost the same, but not _exactly_ the same:"},{"metadata":{"trusted":true},"cell_type":"code","source":"jaccard_similarity(train['dataset_label'].apply(clean_text).iloc[436], train.cleaned_label.iloc[436])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Be aware! And good luck with the competition! 🏆"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}