{"cells":[{"metadata":{},"cell_type":"markdown","source":"Copied from [this Kernel](https://www.kaggle.com/kashnitsky/eda-logit-tf-idf-starter).\n\nTf-Idf + Logistic regression is a very nice baseline for many tasks. Here we'll briefly explore a dataset with news classified as \"news\" (normal), \"clickbait\", and others. Then we'll build a simple baseline based on Tf-Idf representations of news titles and texts. Further we'll compare heavier approaches (like ULMFiT) with this baseline."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.model_selection import cross_val_score, StratifiedKFold\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom scipy.sparse import hstack\nfrom matplotlib import pyplot as plt\nimport seaborn as sns\nimport eli5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/train.csv', index_col='id').fillna(' ')\nvalid = pd.read_csv('../input/valid.csv', index_col='id').fillna(' ')\ntest = pd.read_csv('../input/test.csv', index_col='id').fillna(' ')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll be validating with train + validation files."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_val = pd.concat([train, valid])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(train_val['label']);\nplt.title('Train+val: Target distribution');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that test titles are a bit shorter."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.subplots(1, 2)\nplt.subplot(1, 2, 1)\ntrain_val['text'].apply(lambda x: len(x.split())).plot(kind='hist');\nplt.yscale('log');\nplt.title('Text length in words');\nplt.subplot(1, 2, 2)\ntrain_val['title'].apply(lambda x: len(x.split())).plot(kind='hist');\nplt.yscale('log');\nplt.title('Title length in words');","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.subplots(1, 2)\nplt.subplot(1, 2, 1)\ntest['text'].apply(lambda x: len(x.split())).plot(kind='hist');\nplt.yscale('log');\nplt.title('Text length in words');\nplt.subplot(1, 2, 2)\ntest['title'].apply(lambda x: len(x.split())).plot(kind='hist');\nplt.yscale('log');\nplt.title('Title length in words');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As an entertainment, we can build a wordcloud for news titles. However, no useful insights from such a picture."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from wordcloud import WordCloud, STOPWORDS\n\n# Thanks : https://www.kaggle.com/aashita/word-clouds-of-various-shapes ##\ndef plot_wordcloud(text, mask=None, max_words=200, max_font_size=100, figure_size=(24.0,16.0), \n                   title = None, title_size=40, image_color=False):\n    stopwords = set(STOPWORDS)\n    more_stopwords = {'one', 'br', 'Po', 'th', 'sayi', 'fo', 'Unknown'}\n    stopwords = stopwords.union(more_stopwords)\n\n    wordcloud = WordCloud(background_color='black',\n                    stopwords = stopwords,\n                    max_words = max_words,\n                    max_font_size = max_font_size, \n                    random_state = 42,\n                    width=800, \n                    height=400,\n                    mask = mask)\n    wordcloud.generate(str(text))\n    \n    plt.figure(figsize=figure_size)\n    if image_color:\n        image_colors = ImageColorGenerator(mask);\n        plt.imshow(wordcloud.recolor(color_func=image_colors), interpolation=\"bilinear\");\n        plt.title(title, fontdict={'size': title_size,  \n                                  'verticalalignment': 'bottom'})\n    else:\n        plt.imshow(wordcloud);\n        plt.title(title, fontdict={'size': title_size, 'color': 'black', \n                                  'verticalalignment': 'bottom'})\n    plt.axis('off');\n    plt.tight_layout()  \n    \nplot_wordcloud(train_val[\"title\"], title=\"Word Cloud of Titles\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We'll be using 2 Tf-Idf vectorizers - for titles and texts separetely."},{"metadata":{"trusted":true},"cell_type":"code","source":"title_transformer = TfidfVectorizer(stop_words='english', ngram_range=(1, 3), lowercase=True, max_features=50000)\ntext_transformer = TfidfVectorizer(stop_words='english', ngram_range=(1, 2), lowercase=True, max_features=150000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nX_train_text = text_transformer.fit_transform(train_val['text'])\nX_test_text = text_transformer.transform(test['text'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nX_train_title = title_transformer.fit_transform(train_val['title'])\nX_test_title = title_transformer.transform(test['title'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After concatenation, we get 200k sparse features to represent news titles and texts."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = hstack([X_train_text, X_train_title])\nX_test = hstack([X_test_text, X_test_title])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train.shape, X_test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As for model, let's simply pick logistic regression.****"},{"metadata":{"trusted":true},"cell_type":"code","source":"logit = LogisticRegression(C=5e1, solver='lbfgs', multi_class='multinomial',\n                          random_state=17, n_jobs=4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Cross-validation**"},{"metadata":{"trusted":true},"cell_type":"code","source":"skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=17)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ncv_results = cross_val_score(logit, X_train, train_val['label'], cv=skf, scoring='f1_micro')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cv_results, cv_results.mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's nice to see that cross-validation is more or less stable across folds. Let's train the model on train + val."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nlogit.fit(X_train, train_val['label'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Trying to interpret model weights with ELI5 - look reasonable.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"eli5.show_weights(estimator=logit, \n                  feature_names= list(text_transformer.get_feature_names()) + \n                                 list(title_transformer.get_feature_names()),\n                 top=(50, 5))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Preparing submission.**"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_preds = logit.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame(test_preds, columns=['label']).head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.DataFrame(test_preds, columns=['label']).to_csv('logit_tf_idf_starter_submission.csv',\n                                                  index_label='id')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**What you can try next**\n\n - tune hyperparams, those of `TfIdfVectorizers` as well\n - add Word2Vec/GloVE/Fasttext embeddings, at least for titles\n - switch to ULMFiT and other heavy stuff\n "},{"metadata":{"trusted":true},"cell_type":"code","source":"test_preds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}