{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# A fastai implementation for the CommonLit Readability Contest","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"In this knowledge contest, we are trying to predict the readability for a passage of text.","metadata":{}},{"cell_type":"markdown","source":"**Version History**\n* *Version 1* - Getting the notebook setup, importing basic modules, reading the dataset, building a simple model, evaluate results, submission using pretrained language model only'\n* *Version 2* - Internet off :)\n* *Version 3* - Integrating [my own dataset](https://www.kaggle.com/abee82/fastai-wikitext-wt103-pretrained-model) to pull in the WikiText-103 pretained model for `fastai`\n* *Version 4* - Fixed submision file\n* *Version 5* - Added fine tuned model submission","metadata":{}},{"cell_type":"markdown","source":"## Background\n\n@TODO: update\n\n### For beginners forking this notebook\n\nI highly suggest for anyone interested in solving problems like this to look into the course by Sylvain Gugger and Jeremy Howard. They spent a lot of time building `fastai` into a great library for accessing the latest deep learning techniques.\n\n[Practical Deep Learning for Coders](https://course.fast.ai/)","metadata":{}},{"cell_type":"markdown","source":"# Setup","metadata":{}},{"cell_type":"code","source":"%reload_ext autoreload\n%autoreload 2\n%matplotlib inline","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%time\nfrom fastai.text.all import *","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"INPUT_DIR = Path(\"../input/commonlitreadabilityprize\")\nINPUT_DIR.ls()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"WORKING_DIR = Path(\"./\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv(INPUT_DIR / 'train.csv')\ntest_df = pd.read_csv(INPUT_DIR / 'test.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Making pretrained weights work without needing to find the default filename\n# if not os.path.exists('/root/.cache/torch/hub/checkpoints/'):\n#         os.makedirs('/root/.cache/torch/hub/checkpoints/')\n# !cp '../input/resnet50/resnet50.pth' '/root/.cache/torch/hub/checkpoints/resnet50-19c8e357.pth'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!mkdir -p ~/.fastai/models/\n\n!cp -R ../input/fastai-wikitext-wt103-pretrained-model/* ~/.fastai/models/","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Utilize out of the box language model from `fastai`","metadata":{}},{"cell_type":"code","source":"dls_class = DataBlock(blocks=(TextBlock.from_df('excerpt', seq_len=256), RegressionBlock),\n                      get_x=ColReader('text'),\n                      get_y=ColReader('target'),\n                      splitter=RandomSplitter())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dls = dls_class.dataloaders(train_df, bs=64)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dls.show_batch()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=rmse)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lrs = learn.lr_find()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lrs","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cbs = [\n#     ShowGraphCallback(),\n#     GradientAccumulation(),\n    MixedPrecision(),\n    SaveModelCallback(monitor='_rmse', comp=np.less, min_delta=0.01),\n#     ReduceLROnPlateau(monitor='rmse', comp=np.less, min_delta=0.001, patience=2),\n#     MixUp(0.4),\n    EarlyStoppingCallback(monitor='_rmse', comp=np.less, min_delta=0.01, patience=3),\n#     GradientClip(0.1),\n      ]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.fine_tune(20, lrs.lr_min/2, cbs=cbs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.show_results()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.save('pretrained_LM')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Pretrained LM Submission","metadata":{}},{"cell_type":"code","source":"SAMPLE_SUBMISSION = pd.read_csv(INPUT_DIR / 'sample_submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SAMPLE_SUBMISSION.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_dl = learn.dls.test_dl(test_df.excerpt)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds = learn.get_preds(dl=test_dl, with_decoded=True)\npreds = preds[2].tolist()\npreds = [x for l in preds for x in l]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SAMPLE_SUBMISSION['target'] = preds","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SAMPLE_SUBMISSION.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SAMPLE_SUBMISSION.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Custom Language Model","metadata":{}},{"cell_type":"code","source":"train_df.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dls_lm = TextDataLoaders.from_df(train_df, text_col='excerpt', is_lm=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dls_lm.show_batch(max_n=3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn = language_model_learner(dls_lm, AWD_LSTM, metrics=[accuracy, Perplexity()], wd=0.1).to_fp16()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lrs = learn.lr_find()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lrs","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cbs = [\n#     ShowGraphCallback(),\n#     GradientAccumulation(),\n#     MixedPrecision(),\n    SaveModelCallback(monitor='valid_loss', comp=np.less, min_delta=0.01),\n#     ReduceLROnPlateau(monitor='rmse', comp=np.less, min_delta=0.001, patience=2),\n#     MixUp(0.4),\n    EarlyStoppingCallback(monitor='valid_loss', comp=np.less, min_delta=0.01, patience=3),\n#     GradientClip(0.1),\n      ]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.fine_tune(20, lrs.lr_min, cbs=cbs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.save_encoder('finetuned_lm')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dls = dls_class.dataloaders(train_df, bs=64, text_vocab=dls_lm.vocab)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn = text_classifier_learner(dls, AWD_LSTM, drop_mult=0.5, metrics=rmse)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lrs = learn.lr_find()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lrs","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cbs = [\n    MixedPrecision(),\n    SaveModelCallback(monitor='valid_loss', comp=np.less, min_delta=0.01),\n    EarlyStoppingCallback(monitor='valid_loss', comp=np.less, min_delta=0.01, patience=3),\n      ]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn = learn.load_encoder('finetuned_lm')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.fit_one_cycle(1, lrs.lr_min, cbs=cbs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.freeze_to(-2)\nlearn.fit_one_cycle(1, slice(lrs.lr_min/2/2.5**4, lrs.lr_min/2), cbs=cbs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.freeze_to(-3)\nlearn.fit_one_cycle(1, slice(lrs.lr_min/2/2/2.5**4, lrs.lr_min/2/2), cbs=cbs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn.unfreeze()\nlearn.fit_one_cycle(20, slice(lrs.lr_min/2/2/2/2.5**4, lrs.lr_min/2/2.2), cbs=cbs)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Fine tuned LM submission","metadata":{}},{"cell_type":"code","source":"SAMPLE_SUBMISSION = pd.read_csv(INPUT_DIR / 'sample_submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SAMPLE_SUBMISSION.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_dl = learn.dls.test_dl(test_df.excerpt)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds = learn.get_preds(dl=test_dl, with_decoded=True)\npreds = preds[2].tolist()\npreds = [x for l in preds for x in l]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\npreds","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SAMPLE_SUBMISSION['target'] = preds","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SAMPLE_SUBMISSION.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nSAMPLE_SUBMISSION.to_csv('submission.csv', index=False)","metadata":{},"execution_count":null,"outputs":[]}]}