{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction \n`V1.0.0` `2021-12-04`\n### Who am I\nJust a fellow Kaggle learner. I was creating this Notebook as practice and thought it could be useful to some others \n### Who is this for\nThis Notebook is for people that learn from examples. Forget the boring lectures and follow along for some fun/instructive time :)\n### What can I learn here\nYou learn all the basics needed to create a rudimentary sentiment analyzer using NLTK library and reddit API. I go over a multitude of steps with explanations. Hopefully with these building blocks,you can go ahead and build much more complex models.\n\n### Things to remember\n+ Please Upvote/Like the Notebook so other people can learn from it\n+ Feel free to give any recommendations/changes. \n+ I will be continuously updating the notebook. Look forward to many more upcoming changes in the future.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"!pip install praw","metadata":{"execution":{"iopub.status.busy":"2021-12-05T05:07:12.616213Z","iopub.execute_input":"2021-12-05T05:07:12.616486Z","iopub.status.idle":"2021-12-05T05:07:23.765600Z","shell.execute_reply.started":"2021-12-05T05:07:12.616457Z","shell.execute_reply":"2021-12-05T05:07:23.764922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Imports\nFirst let us start by importing the relevant libraries that we need.","metadata":{}},{"cell_type":"code","source":"!pip install praw\nimport praw                  # Reddit API \nimport nltk                  # Natural Language Tool Kit\nimport pandas as pd          # DataFrames\nfrom pprint import pprint    # Printing\nfrom nltk.sentiment.vader import SentimentIntensityAnalyzer as SIA   # Natural Language Tool Kit - Sentiment Analyzer Class","metadata":{"execution":{"iopub.status.busy":"2021-12-05T05:43:53.820654Z","iopub.execute_input":"2021-12-05T05:43:53.820968Z","iopub.status.idle":"2021-12-05T05:44:02.339123Z","shell.execute_reply.started":"2021-12-05T05:43:53.820932Z","shell.execute_reply":"2021-12-05T05:44:02.338412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Reddit API Instance\nYou will have to create your own oAuth token to create an instance of the Reddit class. You can find more information on how to obtain the `client_id` and `client_secret` on [link](https://praw.readthedocs.io/en/stable/getting_started/authentication.html).","metadata":{}},{"cell_type":"code","source":"# Here you can replace the 'your_user_here' by your reddit account name\nuser = \"Web Scraper v1.0 by /u/your_user_here\"\n\n# Create Reddit Instance\nreddit = praw.Reddit(client_id=\"pycRk7W4euOOk0P-6OErvQ\",\n                     client_secret=\"YmR9iG_8hDNTz8yVlH_qwnayNpCuIw\",\n                     user_agent=user)","metadata":{"execution":{"iopub.status.busy":"2021-12-05T05:44:02.340665Z","iopub.execute_input":"2021-12-05T05:44:02.340923Z","iopub.status.idle":"2021-12-05T05:44:02.598506Z","shell.execute_reply.started":"2021-12-05T05:44:02.340892Z","shell.execute_reply":"2021-12-05T05:44:02.597677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Parsing Reddit Posts\nIn this section, we parse through all the hot posts under the CryptoCurrency subredit. We only keep posts that have more than 500 upvotes.","metadata":{}},{"cell_type":"code","source":"# Initialize a set to store unique headlines\nheadlines = set()\n\n# Iterate through every HOT posts of the CryptoCurrency subreddit\nfor submission in reddit.subreddit(\"CryptoCurrency\").hot(limit=None):\n    # If upvotes higher than 500, store the headline of the post\n    if submission.score >= 500:\n        headlines.add(submission.title)\n\n# Print headlines \nprint(*headlines, sep='\\n')\n\n# Create DataFreame from set of headlines\nheadlines_df = pd.DataFrame(headlines)\nprint(headlines_df.head(5))","metadata":{"execution":{"iopub.status.busy":"2021-12-05T05:44:02.599761Z","iopub.execute_input":"2021-12-05T05:44:02.599980Z","iopub.status.idle":"2021-12-05T05:45:04.975939Z","shell.execute_reply.started":"2021-12-05T05:44:02.599952Z","shell.execute_reply":"2021-12-05T05:45:04.975062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Sentiment Analyzer Instance\nHere, we create the instance of the `SIA()` class from the natural language tool kit (`nltk`) library. This allows us to analyze each post and get a sentiment rating.","metadata":{}},{"cell_type":"code","source":"# Download the vader_lexicon to be able to create an appropriate sentiment analyze with SIA() class\nnltk.download(\"vader_lexicon\")\nsia = SIA()\n\n# Init results list to store headline and its score as a dict\nresults = []","metadata":{"execution":{"iopub.status.busy":"2021-12-05T05:45:04.977735Z","iopub.execute_input":"2021-12-05T05:45:04.977958Z","iopub.status.idle":"2021-12-05T05:45:05.257865Z","shell.execute_reply.started":"2021-12-05T05:45:04.977932Z","shell.execute_reply":"2021-12-05T05:45:05.256869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Analyze sentiment\nWe analyze the sentiment of each headline using the `ntlk` library and the `SIA()` class. We store it in a list called results to then be able to create a dataframe.","metadata":{}},{"cell_type":"code","source":"# Iterate through list of headliens and obtain sentiment score with nltk library\nfor headline in headlines:\n    polarity_score = sia.polarity_scores(headline)\n    polarity_score['headline'] = headline\n    results.append(polarity_score)\n\n# Use pretty print library to print the list of dicts\npprint(results[:], width=100)","metadata":{"execution":{"iopub.status.busy":"2021-12-05T05:45:05.259170Z","iopub.execute_input":"2021-12-05T05:45:05.259387Z","iopub.status.idle":"2021-12-05T05:45:05.317845Z","shell.execute_reply.started":"2021-12-05T05:45:05.259349Z","shell.execute_reply":"2021-12-05T05:45:05.316954Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create DataFrame from Records\nWe have recorded each headline and its respective scores as a dictionary in a list named results. Results containts multiple dicts representing each headline/post. We use the `from_records()` method to create a DataFrame from a list of dicts.","metadata":{}},{"cell_type":"code","source":"# Create DataFrame from list of dicts\nresults_df = pd.DataFrame.from_records(results)\nprint(results_df.head())","metadata":{"execution":{"iopub.status.busy":"2021-12-05T05:45:05.319139Z","iopub.execute_input":"2021-12-05T05:45:05.320710Z","iopub.status.idle":"2021-12-05T05:45:05.331936Z","shell.execute_reply.started":"2021-12-05T05:45:05.320659Z","shell.execute_reply":"2021-12-05T05:45:05.331399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Post-Process Results\nHere we create a new column `label` which will rather contain a 1 if the sentiment is postive or a -1 if the sentiment is negative. We do not need to keep all the information that the nltk library produces. We are just interested to know if the post is positive or negative.","metadata":{}},{"cell_type":"code","source":"# Create new column label and fill all values with 0\nresults_df['label'] = 0\n\n# Label 1 would be positive\nresults_df.loc[results_df['compound'] > 0.2, 'label'] = 1\n\n# Label -1 would be negative\nresults_df.loc[results_df['compound'] < 0.2, 'label'] = -1\n\n# Display DataFrame head for quick visualization\nresults_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-12-05T05:45:05.332811Z","iopub.execute_input":"2021-12-05T05:45:05.333472Z","iopub.status.idle":"2021-12-05T05:45:05.359205Z","shell.execute_reply.started":"2021-12-05T05:45:05.333429Z","shell.execute_reply":"2021-12-05T05:45:05.358267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we filter the DataFrame to only contain the headline and the label. We then proceed to save the results in a csv file that I can use in the future. For example, I might use these results as additional feature to a time-series prediction algorithm for CryptoCurrency. \n\n**Needless to say, if I succeed on making it... This will be the last time you see me on Kaggle :)**","metadata":{}},{"cell_type":"code","source":"# Create new filtered DataFrame which only containts columns headline and label\nresults_filtered_df = results_df[['headline', 'label']]\n\n# Save to a csv file\nresults_filtered_df.to_csv('reddit_sentiment_analysis.csv', encoding='utf-8', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-12-05T05:46:07.004701Z","iopub.execute_input":"2021-12-05T05:46:07.004978Z","iopub.status.idle":"2021-12-05T05:46:07.012315Z","shell.execute_reply.started":"2021-12-05T05:46:07.004948Z","shell.execute_reply":"2021-12-05T05:46:07.011688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Overview of Results\nThis allows us to have a quick overview of how much percentage of posts were positive and negative. We use `pandas` `value_counts()` method which gives us the amount of pos. and neg. values in the `label` column. We added the `normalize` argument and multiplied the output by 100 to obtain the percentage amount of pos. and neg. headlines.","metadata":{}},{"cell_type":"code","source":"# Obtain percentage of how many headlines/posts were postive and negative\nresults_filtered_df.label.value_counts(normalize=True) * 100","metadata":{"execution":{"iopub.status.busy":"2021-12-05T05:45:05.373121Z","iopub.execute_input":"2021-12-05T05:45:05.373902Z","iopub.status.idle":"2021-12-05T05:45:05.385517Z","shell.execute_reply.started":"2021-12-05T05:45:05.373866Z","shell.execute_reply":"2021-12-05T05:45:05.384914Z"},"trusted":true},"execution_count":null,"outputs":[]}]}