{"cells":[{"metadata":{},"cell_type":"markdown","source":"# <center> Clickbait news detection with ULMFiT. Starter\n\nHere we mostly follow the training scheme described by Jeremy Howard in [fast.ai Lesson 4](https://course.fast.ai/videos/?lesson=4): taking a pretrained language model, fine-tuning it with unlabeled data, then fine-tuning classification head for our particular task.\n\nThis is just a starter. At each step, I also mention how you can do better."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"from tqdm import tqdm_notebook\nimport torch\nimport fastai\nfrom fastai.text import *\nfastai.__version__","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Preprocessing\nHere we write all news texts from train, validation and text files into `unlabeled_news.csv` - to train a language model.\n\nThen, we write texts and labels into `train_28k.csv` and texts only into `test_5k.csv`.\n\n**How to do better:** go for that 80k unlabeled set as well."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv').fillna(' ')\nvalid = pd.read_csv('../input/valid.csv').fillna(' ')\ntest = pd.read_csv('../input/test.csv').fillna(' ')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.concat([train['text'], valid['text'], test['text']]).to_csv(\n    'unlabeled_news.csv', index=None, header=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.concat([train[['text', 'label']],valid[['text', 'label']]]).to_csv(\n    'train_28k.csv', index=None, header=True)\ntest[['text']].to_csv('test_5k.csv', index=None, header=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"folder = '.'\nunlabeled_file = 'unlabeled_news.csv'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Reading unlabeled data to train ULMFiT language model"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ndata_lm = TextLMDataBunch.from_csv(folder, unlabeled_file, text_cols='text')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# LM training \n\nHere we resort to the training scheme described by Jeremy Howard, [fast.ai](https://course.fast.ai/):\n - finding good initial learning rate\n - training for one epoch\n - unfreezing and more training\n\n**How to do better:** train for 10-15 epochs after unfreezing"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nlearn = language_model_learner(data_lm, drop_mult=0.3, arch=AWD_LSTM)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nlearn.lr_find(start_lr = slice(10e-7, 10e-5), end_lr=slice(0.1, 10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.recorder.plot(skip_end=10, suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"best_lm_lr = learn.recorder.min_grad_lr\nbest_lm_lr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nlearn.fit_one_cycle(1, best_lm_lr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.unfreeze()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nlearn.fit_one_cycle(1, best_lm_lr)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generating some text\n\nIt's always interesting to see whether a LM is able to generate nice text. With LM training improvement (in terms of loss), at some point you'll notice some nice improvement in quality of the generated text.\n\nOne sample generated with my better-trained LM:\n\n> 'An italian man was found dead in his yard due to heat conditions on Sunday night , his spokeswoman said . The office manager of the Ultra retired man ’s office told buzzfeed News there being no sign of comfort . The man at his 911 home told guy , he had been in contact with his car ’s owner before asleep and then immediately responded to starting fire . The man named Guy made a news video at PARKING Station in which the Mississippi State Police shot multiple people with Tim Shepherd to get their son alive , Mark Morris , a family friend dangling near his wife ’s house , said . The teen told police he was winning inclusion in general . Police dragged him into the house — where the officer had been yards away — during his die - hard bid at a nearby snow salon . The family voted in favor of Appreciative and arrested more than three months later : They tried to detained him . He and his family stopped , per the station , all the way up . “'\n\nNo much sense, but at least some structure :) And now with GPT-2 we see that quantitative improvements can also lead to qualital improvements."},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.predict('An italian man was found dead in his yard due to', n_words=200)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.save_encoder('clickbait_news_enc')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training classification head\n\nHere again we follow Jeremy Howard. \n\n**How to do better:** hyperparam tuning (though it's extremely annoying with such a heavy model), more epochs after unfreezing, check for some live examples of ULMFiT training, different learning rates for different layers etc."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_file, test_file = 'train_28k.csv', 'test_5k.csv'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_clas = TextClasDataBunch.from_csv(path=folder, \n                                        csv_name=train_file,\n                                        test=test_file,\n                                        vocab=data_lm.train_ds.vocab, \n                                        bs=64,\n                                        text_cols='text', \n                                        label_cols='label')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_clas.save('ulmfit_data_clas_clickbait_news')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn_clas = text_classifier_learner(data_clas, drop_mult=0.3, arch=AWD_LSTM)\nlearn_clas.load_encoder('clickbait_news_enc')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn_clas.lr_find(start_lr=slice(10e-7, 10e-5), end_lr=slice(0.1, 10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn_clas.recorder.plot(skip_end=10, suggestion=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"best_clf_lr = learn_clas.recorder.min_grad_lr\nbest_clf_lr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn_clas.fit_one_cycle(1, best_clf_lr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn_clas.freeze_to(-2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn_clas.fit_one_cycle(1, best_clf_lr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn_clas.unfreeze()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn_clas.fit_one_cycle(1, best_clf_lr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn_clas.show_results()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predictions for the test set\n\nThanks to [Noisefield](https://www.kaggle.com/mamamot) for his comments on how to do it efficiently."},{"metadata":{"trusted":true},"cell_type":"code","source":"data_clas.add_test(test[\"text\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_preds, _ = learn_clas.get_preds(DatasetType.Test, \n                                     ordered=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Forming a submission file"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_pred_df = pd.DataFrame(test_preds.data.cpu().numpy(),\n                            columns=['clickbait', 'news', 'other'])\nulmfit_preds = pd.Series(np.argmax(test_pred_df.values, axis=1),\n                        name='label').map({0: 'clickbait', 1: 'news', 2: 'other'})\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ulmfit_preds.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ulmfit_preds.to_csv('ulmfit_predictions.csv', index_label='id', header=True)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}