{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Basic text cleaning and string matching notebook\nThis notebook is a version of the below two notebooks. I tried not to copy too much from them because I wanted to work through it on my own, but the concpet is basically the same.\n\n* https://www.kaggle.com/manabendrarout/tabular-data-preparation-basic-eda-and-baseline\n* https://www.kaggle.com/josephassaker/coleridge-initiative-eda-na-ve-submission\n\nI did this because I just wanted to do a basic exercise to work python/pandas text manipulation and text matching so that I can get a handle on the data a little bit, and set up the problem."},{"metadata":{"trusted":true},"cell_type":"code","source":"# You don't need much! Not even numpy.\nimport pandas as pd\nimport regex as re\nimport glob","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/train.csv')\ntest_files_path = '../input/coleridgeinitiative-show-us-the-data/test'\ntrain_df.head()\n\n# Note that there's no need the training json files for this simple notebook.\n#train_files_path = '../input/coleridgeinitiative-show-us-the-data/train'\n\n# Also don't need the submission CSV as it's generated from the test data. \n# This is a different approach from the notebooks linked at the top, I think a bit more realistic\n#sample_sub = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Get the names of unique datasets\nRunning the training data list of datasets through a .unique and tokenizing them through the gven text cleaning function."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get a list of unique publicaitons from the training set and put them into a new df\ndf_unique_data_sets = pd.DataFrame(train_df['dataset_title'].unique().tolist(), columns = ['data_sets']) \n\n# Run the publications through the official text cleaning function .\n# I converted the function to a lambda because I think it's a little nicer \n# and I also wanted some practice working with lambdas.\ndf_unique_data_sets['datasets_cleaned'] = df_unique_data_sets.apply(\n   lambda txt: re.sub('[^A-Za-z0-9]+', ' ', str(txt['data_sets']).lower()).strip(), axis =1)\n\ndf_unique_data_sets.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Get all text from publications\nGet the list of json files using Glob, loop through all files and pull out and put them in a pandas DF\nThen run the article text through the clenaing function. This is so that it'll have a better chance of matching the dataset text later in the notebook."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Get a list of all json files in the test data set\ntest_files = glob.glob(\"../input/coleridgeinitiative-show-us-the-data/test/*.json\")\n\ndf_test_publications = pd.DataFrame()\nfor test_file in test_files: \n    file_data = pd.read_json(test_file) #read the JSON from the test files\n    \n    # Pull out an parse each line of test json file name into pub_id column\n    file_data.insert(0,'pub_id', test_file.split('/')[-1].split('.')[0]) \n    \n    #concat the pub id's with JSON pulled above\n    df_test_publications = pd.concat([df_test_publications, file_data])\n\ndf_test_publications.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Run all text in the training DF through the text cleaning function\ndf_test_publications['text_cleaned'] = df_test_publications.apply(\n    lambda txt: re.sub('[^A-Za-z0-9]+', ' ', str(txt['text']).lower()).strip(), axis =1)\n\ndf_test_publications.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Match dataset titles with text\nThe block here is doing an Excel vlookup style string matching function. The loop goes through each publication and looks for matches to datasets titles."},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings # suppressing some warnings with str.contains\nwarnings.filterwarnings(\"ignore\", 'This pattern has match groups')\n\n\n# this loop goes through all unique dataset titles and looks for them in the publications\n# Uses versions of both the datasets and publications that have been put through the tokenizer\ndf_sub = pd.DataFrame() #blank df to append to at the end of the loop\n\nfor i in range (len(df_unique_data_sets.index)):\n\n#make a temp df with only matched datasets to publications\n    dfx = df_test_publications[df_test_publications['text_cleaned'].str.contains( df_unique_data_sets['datasets_cleaned'][i])]\n\n# The below produces a \"set with copy\" warning, \n# but it's fine as dfx is just temporary and only used for appending to df_sub.\n    dfx['PredictionString'] = df_unique_data_sets['datasets_cleaned'][i] # new column with the pub name\n    df_sub = df_sub.append(dfx[['pub_id', 'PredictionString']]) \n\ndf_sub.rename(columns={\"pub_id\": \"Id\"}, inplace = True)\ndf_sub.sort_values('PredictionString', inplace = True) #contest requires alphabetical order\ndf_sub.head(20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Format predictions for submission\nAbove we have all preictions, but the contest wants each line in the submission to only have one id and all the datasets seperated by a bar.\n\nAlso in the resulting DF above there is a lot of redundancy that we can easily strip out."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Last step is to put the prediction strings together\nunique_publications = df_sub['Id'].unique().tolist()\n\nid_pred_dict={}\nfor n in unique_publications:\n    dfxx = df_sub[df_sub['Id']== n] # filter on Id each pass of the loop\n    unique_preds = dfxx['PredictionString'].unique().tolist() # pull out a unique list from the filtered DF\n    strung = '|'.join(unique_preds) #make a string with bar concatonation\n    id_pred_dict[n] = strung\n\ndf_sub1 = pd.DataFrame(id_pred_dict.items(), columns=['id', 'PredictionString'])\ndf_sub1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# and finally, put it in a CSV for submission\ndf_sub1.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}