{"cells":[{"metadata":{"_uuid":"d10a159e6f453af360736473257bc33ffe2fca28","_cell_guid":"08d2c5da-7cfa-4159-9a6a-dedbc12df63d"},"cell_type":"markdown","source":"<center>\n<h1> Danbury AI June 2018: Workshop Part 5</h1>\n<h2>Modeling Online Toxicity with LSTMs</h2>\n</center>\n\nWelcome to this interactive laboratory for exploring the application of LSTM's to modeling toxicity in online dialog.\n\nOutline\n\n1. Libraries and Environment Setup\n2. Preprocessing \n3. Modeling \n4. Visualizing and Interpreting Results\n\nResources and Sources \n* https://www.kaggle.com/sbongo/for-beginners-tackling-toxic-using-keras\n* http://www.deeplearningbook.org/contents/rnn.html\n\n**NOTE**: In order for the interactive componnents of this kernel to function, you must either fork this kernel or download it to your local machine which has the required environment and dependencies. The simplest method is to fork the kernel here on kaggle. "},{"metadata":{"_uuid":"28b620d44b79cf82c9c647787dc611b559d6e206","_cell_guid":"13c3d987-96da-4490-a68c-fe02df960789"},"cell_type":"markdown","source":"**$\\large\\text{1. Libraries and Environment Setup}$**"},{"metadata":{"_uuid":"af2dd8674eaa1ac4902be9b503b602a072853c87","_cell_guid":"e7ed960e-487b-43a4-832e-039a145a051a","trusted":true},"cell_type":"code","source":"import sys, os, re, csv, codecs\nimport numpy as np \nimport pandas as pd\n\nfrom keras.preprocessing.text import Tokenizer\nfrom keras.preprocessing.sequence import pad_sequences\nfrom keras.layers import Dense, Input, LSTM, Embedding, Dropout, Activation\nfrom keras.layers import Bidirectional, GlobalMaxPool1D\nfrom keras.models import Model\nfrom keras import initializers, regularizers, constraints, optimizers, layers\n\nfrom ipywidgets import interact,interact_manual\nfrom IPython.display import display\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\nprint(\"Datasets: {0}\".format(os.listdir(\"../input\")))","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"557fd8aa0476065d1a4fb5394fe3b8b6c83acdfc","_cell_guid":"8688a93d-b687-4e61-801d-d7b8df0eb985"},"cell_type":"markdown","source":"$\\large\\text{2. Preprocessing }$"},{"metadata":{"_uuid":"edbacd93566b632779d746139a93811459729832","_cell_guid":"3bc07324-34ad-44e9-adee-95506283fbc7"},"cell_type":"markdown","source":"Load our training and test data using pandas into dataframe variables amicably named. \n\n*Note:* If you are using this kernel as a fork on kaggle, you will not need to change the training and test paths. If you have a local version of this kernel, you must download the training data and change the paths below to your local copy of the data. "},{"metadata":{"_uuid":"a8398d543217375e5fc613b8938dbc68237f683b","_cell_guid":"838b6f23-515b-43ee-9dc2-d78d3926a983","trusted":true,"collapsed":true},"cell_type":"code","source":"train = pd.read_csv('../input/jigsaw-toxic-comment-classification-challenge/train.csv')\ntest = pd.read_csv('../input/jigsaw-toxic-comment-classification-challenge/test.csv')","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"f19d1ffb2f240040ef5b6cc0b9fcaed0fafa902f","_cell_guid":"7c55fdba-8f4f-4600-89b2-294225a5dc45"},"cell_type":"markdown","source":"Basic data information."},{"metadata":{"_uuid":"3c7fbf442414841934cb5aef64ecffe06783fa94","_cell_guid":"db257b69-2dc9-4ea2-a123-0bada0413b04","trusted":true,"collapsed":true},"cell_type":"code","source":"print(\"Shape of the training data: {0}.\".format(train.shape) )\nprint(\"Shape of the test data: {0}.\".format(test.shape))\n\n# View some of the training data. \ntrain.head()","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"aecd38fed13d580b0095af97b368245385dbec46","_cell_guid":"1256ded2-68ea-41bc-9e0f-d889772dfd41"},"cell_type":"markdown","source":"Interactive data visualization"},{"metadata":{"_uuid":"235aa14a94b225b466fe5fe63d15e0d9eaac6d2e","_cell_guid":"a492730f-03df-43a7-80e1-9f67f7588d39","collapsed":true,"trusted":true},"cell_type":"code","source":"# Cashed filter computations for the visualization below. \ncashedFilters = {}","execution_count":6,"outputs":[]},{"metadata":{"_uuid":"f29721b9ae4e51c364bf629f5adbc591ec8ae73e","_cell_guid":"8c2244e5-20da-4888-8bdc-9ad7b8e85f8e","trusted":true,"collapsed":true},"cell_type":"code","source":"# An interactive widget for exploring the data. \ndef visRow(row):\n    print( row[\"comment_text\"] )\n    print (\"{0} - Toxic( {1} ) - Severe Toxic ( {2} ) - Obscene( {3} ) - Threat ( {4} ) - Insult ( {5} ) - Identity Hate ( {6} )\"\n           .format(row[\"id\"],row[\"toxic\"],row[\"severe_toxic\"],\n                   row[\"obscene\"],row[\"threat\"],row[\"insult\"],row[\"identity_hate\"]))\n    \ndef categoryVis(toxic,severeToxic, obscene, threat, insult, identHate,idx=0):\n    try:\n        visRow(cashedFilters[toxic,severeToxic, obscene, threat, insult, identHate].iloc[idx])\n        \n    except KeyError:\n        print(\"Computing filter...\")\n        filterRes = train[train.apply(lambda x: x[\"toxic\"] == toxic and x[\"severe_toxic\"] == severeToxic and x[\"obscene\"] == obscene \n                                  and x[\"threat\"] == threat and x[\"insult\"] == insult and x[\"identity_hate\"] == identHate , axis=1)]\n        cashedFilters[toxic,severeToxic, obscene, threat, insult, identHate] =  filterRes\n        print(\"Done.\\n---------\")\n        \n        visRow(filterRes.iloc[idx])\n    \ninteract(categoryVis,toxic=False,severeToxic=False, obscene=False, threat=False, insult=False, identHate=False,idx=(0,100))","execution_count":7,"outputs":[]},{"metadata":{"_uuid":"5d735a9e914c1ac335cd6fda90ff0232880c9316","_cell_guid":"49635851-a048-4531-a406-1d6a9467d13a"},"cell_type":"markdown","source":"Split training data into data and label vectors. "},{"metadata":{"_uuid":"161178fa493b42cb490c46c95b24b4fa6e9a74ac","_cell_guid":"b6bbfbae-13be-437e-aed3-387d2b81c858","collapsed":true,"trusted":true},"cell_type":"code","source":"list_classes = [\"toxic\", \"severe_toxic\", \"obscene\", \"threat\", \"insult\", \"identity_hate\"]\n\ny_train = train[list_classes].values\nlist_sentences_train = train[\"comment_text\"]\n\nlist_sentences_test = test[\"comment_text\"]","execution_count":8,"outputs":[]},{"metadata":{"_uuid":"37a7c9cd4861d60c49b00175cecee5e9348d84b7","_cell_guid":"5415cd73-f98a-457d-9145-96df8dd52b13"},"cell_type":"markdown","source":"Tokenize the words. "},{"metadata":{"_uuid":"14beb2fdbf899d9dc0a60419c33a0b67a0efaa40","_cell_guid":"5f108649-c5bf-4274-b3cd-21ee01d205b2","collapsed":true,"trusted":true},"cell_type":"code","source":"max_features = 20000\ntokenizer = Tokenizer(num_words=max_features)\ntokenizer.fit_on_texts(list(list_sentences_train))\nlist_tokenized_train = tokenizer.texts_to_sequences(list_sentences_train)\nlist_tokenized_test = tokenizer.texts_to_sequences(list_sentences_test)","execution_count":9,"outputs":[]},{"metadata":{"_uuid":"3f6dc2b2cc367de06c17b64f2557565422457e9a","_cell_guid":"9e5dd3dd-1897-49a0-9ab2-efcca3c7dffc"},"cell_type":"markdown","source":"The following shows the result of the tokenizing process. Our text comments are now sequences of token ids."},{"metadata":{"_uuid":"9cdd492417501a78a9b4067415460e5fa89bef47","_cell_guid":"27bbf2ce-4e6f-4e77-b285-953ab8963a55","trusted":true,"collapsed":true},"cell_type":"code","source":"print(list_tokenized_train[:1])","execution_count":10,"outputs":[]},{"metadata":{"_uuid":"4fc13cb1e6a9623c16f59f3166fa2bdc45984d1c","_cell_guid":"e71695b7-5998-4f0f-82cc-cc72e6ffa1a4"},"cell_type":"markdown","source":"Padd our sequences so we can have a fixed training input size. "},{"metadata":{"_uuid":"72561aa8a8f9c1cf52d65ea40342e8b7a12a9a68","_cell_guid":"c2311d49-2988-4f9e-ab27-d0f6e5d3bc54","collapsed":true,"trusted":true},"cell_type":"code","source":"maxlen = 200\nX_t = pad_sequences(list_tokenized_train, maxlen=maxlen)\nX_te = pad_sequences(list_tokenized_test, maxlen=maxlen)","execution_count":11,"outputs":[]},{"metadata":{"_uuid":"eef31d6b11dadb62cf6eacdfacd5cb800ed8476b","_cell_guid":"1091da02-67bd-42bf-a67a-fcaee7375e37"},"cell_type":"markdown","source":"$\\large\\text{3. Modeling }$"},{"metadata":{"_uuid":"4e4d94f8e9c32875dc0b99e84293e2975443219c","_cell_guid":"970b9a5d-2204-45e7-84cb-5180e915df95","collapsed":true,"trusted":true},"cell_type":"code","source":"embed_size = 128\n\ninp = Input(shape=(maxlen, )) #maxlen=200 as defined earlier\nx = Embedding(max_features, embed_size)(inp)\nx = LSTM(60, return_sequences=True,name='lstm_layer')(x)\nx = GlobalMaxPool1D()(x)\nx = Dropout(0.1)(x)\nx = Dense(50, activation=\"relu\")(x)\nx = Dropout(0.1)(x)\nx = Dense(6, activation=\"sigmoid\")(x)\n\nmodel = Model(inputs=inp, outputs=x)\nmodel.compile(loss='binary_crossentropy',\n                  optimizer='adam',\n                  metrics=['accuracy'])","execution_count":13,"outputs":[]},{"metadata":{"_uuid":"ec1a497f11b17f991b539f7c6d186efc63ffec7e","trusted":true,"collapsed":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"13325b973ec7c67d258b6915ce5034cbc4b15247","_cell_guid":"556dbd0a-eec2-4562-aa31-6d42e145f8c4","trusted":true,"collapsed":true},"cell_type":"code","source":"batch_size = 32\nepochs = 2\nmodel.fit(X_t,y_train, batch_size=batch_size, epochs=epochs, validation_split=0.1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8d995bb1a6b101447a79a279be045e6fe1f67541","_cell_guid":"51cd2cbf-7721-471b-aac6-fb7b2f082424","collapsed":true,"trusted":true},"cell_type":"code","source":"model.save('lstm_toxic.h5')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bf765a418a44f5292e525da13bb800a430bfb071","_cell_guid":"c3f28a4f-97fc-4f5d-a63f-0a7dc3f7972e"},"cell_type":"markdown","source":"$\\large\\text{4. Visualizing and Interpreting Results }$"},{"metadata":{"_uuid":"90a78f1e7545700f1ced5d8cf56874cecac5ba89","_cell_guid":"7de9ce1f-c7f0-434d-9fc6-c32b1b96ad6b","collapsed":true,"trusted":true},"cell_type":"code","source":"def testModel(text):\n    textSeq = tokenizer.texts_to_sequences([text])\n    t = pad_sequences(textSeq, maxlen=maxlen)\n    prediction = model.predict(t)[0]\n    print(\"Toxic( {0:f} ) - Severe Toxic ( {1:f} ) - Obscene( {2:f} ) - Threat ( {3:f} ) - Insult ( {4:f} ) - Identity Hate ( {5:f} )\"\n          .format(prediction[0],prediction[1],prediction[2],prediction[3],prediction[3],prediction[4],prediction[5]))\ninteract(testModel,text=\"Testing 1 2 3\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7fdaf7718089f8a0de9bf7791a6af880e8b0b711","_cell_guid":"b268a783-e12e-4cf7-93b3-624443f6f344","collapsed":true,"trusted":true},"cell_type":"code","source":"testPred = model.predict(X_te)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"29e0a0fe1463ab1f724ee393bf47cfe7050e093f","_cell_guid":"516f92e9-4bbd-4f25-964d-71a0dee4ebdb","collapsed":true,"trusted":true},"cell_type":"code","source":"[\"toxic\", \"severe_toxic\", \"obscene\", \"threat\", \"insult\", \"identity_hate\"]\ndata_to_submit = pd.DataFrame.from_items([\n    ('id',test[\"id\"]),\n    ('toxic',testPred[:,0]),\n    ('severe_toxic',testPred[:,1]),\n    ('obscene',testPred[:,2]),\n    ('threat',testPred[:,3]),\n    ('insult',testPred[:,4]),\n    ('identity_hate',testPred[:,5])\n])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6ace399ec88b34b4e56ad9c3728df22e9e9e249b","_cell_guid":"2ac09c6d-73c8-4ef2-b9d8-1fcea35d2ff9","collapsed":true,"trusted":true},"cell_type":"code","source":"data_to_submit.to_csv('csv_to_submit.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ffc0f3e2258a5ffb78ca1d5a1f7a8ffcc593cd51","_cell_guid":"32c4062c-1a69-46f8-bcca-33d5b7b4120f","collapsed":true,"trusted":true},"cell_type":"code","source":"data_to_submit.head()","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.5","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}