{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 💻 Data Science Best Practices with pandas\n---\n# 📝 Introduction to the TED Talks dataset\n\n> This is a kernel follows the talk from `PyCon 2019` of `Kevin Markham`.\n> - 📍 PyCon: [Full Conference](https://www.youtube.com/watch?v=ZjrUmNq41Eo&t=3778s)\n> - 📍 Youtube channel: [Data School](https://www.youtube.com/user/dataschool)\n---\n\n# 🔍 Exploratory Data Analysis\n> Exploratory Data Analysis is all about answering a specific question. In this notebook we will try to answer the following questions:\n> 1. Which talks provoke the most online discussion?\n> 2. What were the \"best\" events in TED history to attend?\n> 3. Which occupations deliver the funniest TED talks on average?","metadata":{}},{"cell_type":"code","source":"!pip install -q hvplot","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-02-15T22:39:18.576558Z","iopub.execute_input":"2023-02-15T22:39:18.577380Z","iopub.status.idle":"2023-02-15T22:39:34.985889Z","shell.execute_reply.started":"2023-02-15T22:39:18.577279Z","shell.execute_reply":"2023-02-15T22:39:34.984592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport hvplot.pandas","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:34.988352Z","iopub.execute_input":"2023-02-15T22:39:34.988722Z","iopub.status.idle":"2023-02-15T22:39:38.678926Z","shell.execute_reply.started":"2023-02-15T22:39:34.988688Z","shell.execute_reply":"2023-02-15T22:39:38.677608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ted = pd.read_csv(\"/kaggle/input/ted-talks/ted_main.csv\")\nted.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:38.681072Z","iopub.execute_input":"2023-02-15T22:39:38.682036Z","iopub.status.idle":"2023-02-15T22:39:38.941107Z","shell.execute_reply.started":"2023-02-15T22:39:38.681972Z","shell.execute_reply":"2023-02-15T22:39:38.939904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# rows, columns\nted.shape","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:38.944730Z","iopub.execute_input":"2023-02-15T22:39:38.945546Z","iopub.status.idle":"2023-02-15T22:39:38.952104Z","shell.execute_reply.started":"2023-02-15T22:39:38.945505Z","shell.execute_reply":"2023-02-15T22:39:38.950942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# object columns are usually strings, but can also be arbitrary Python objects (lists, dictionaries)\nted.dtypes","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:38.953832Z","iopub.execute_input":"2023-02-15T22:39:38.954212Z","iopub.status.idle":"2023-02-15T22:39:38.965179Z","shell.execute_reply.started":"2023-02-15T22:39:38.954178Z","shell.execute_reply":"2023-02-15T22:39:38.964007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ted.info()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:38.966617Z","iopub.execute_input":"2023-02-15T22:39:38.966969Z","iopub.status.idle":"2023-02-15T22:39:39.003226Z","shell.execute_reply.started":"2023-02-15T22:39:38.966939Z","shell.execute_reply":"2023-02-15T22:39:39.001852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# count the number of missing values in each column\nted.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:39.004685Z","iopub.execute_input":"2023-02-15T22:39:39.005142Z","iopub.status.idle":"2023-02-15T22:39:39.017831Z","shell.execute_reply.started":"2023-02-15T22:39:39.005096Z","shell.execute_reply":"2023-02-15T22:39:39.016651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ❌ Missing Values\n> In `machine learning`, we need to handle `missing values`. There are many types of missing values:\n> - `Standard Missing Values`: These are missing values that Pandas can detect.\n> - `Non-Standard Missing Values`: Sometimes it might be the case where there’s missing values that have different formats.\n> - `Unexpected Missing Values`: For example, if our feature is expected to be a string, but there’s a numeric type, then technically this is also a missing value.\n\n---\n\n> It’s important to understand these different types of missing data from a `statistics point of view`. The type of missing data will influence how you deal with filling in the missing values.\n> - Sometimes you’ll simply want to delete those rows, other times you’ll replace them.\n> - A very common way to replace missing values is using a median (for objects) or mean (for numerique values). \n\n---\n\n> But those are weak appraoch, some times we need `domain knowledge` about the data and `statical study` to fill the missing values.","metadata":{}},{"cell_type":"code","source":"# fill the miising values. In the general case if the missing values type is numerique\n# we fill it with mean values. if its an object we fill missing values with mode.\nted['speaker_occupation'] = ted.speaker_occupation.fillna(ted.speaker_occupation.mode()[0])","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:39.019393Z","iopub.execute_input":"2023-02-15T22:39:39.020375Z","iopub.status.idle":"2023-02-15T22:39:39.027871Z","shell.execute_reply.started":"2023-02-15T22:39:39.020342Z","shell.execute_reply":"2023-02-15T22:39:39.027020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ted.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:39.029348Z","iopub.execute_input":"2023-02-15T22:39:39.029933Z","iopub.status.idle":"2023-02-15T22:39:39.048161Z","shell.execute_reply.started":"2023-02-15T22:39:39.029901Z","shell.execute_reply":"2023-02-15T22:39:39.046968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ted.describe()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:39.052013Z","iopub.execute_input":"2023-02-15T22:39:39.052454Z","iopub.status.idle":"2023-02-15T22:39:39.091169Z","shell.execute_reply.started":"2023-02-15T22:39:39.052416Z","shell.execute_reply":"2023-02-15T22:39:39.089767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 📢 Which talks provoke the most online discussion?","metadata":{}},{"cell_type":"code","source":"ted.hvplot.hist(subplots=True, height=250, width=250, shared_axes=False, value_label='Rate').cols(3)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:39.093189Z","iopub.execute_input":"2023-02-15T22:39:39.094113Z","iopub.status.idle":"2023-02-15T22:39:40.109302Z","shell.execute_reply.started":"2023-02-15T22:39:39.094061Z","shell.execute_reply":"2023-02-15T22:39:40.108280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(12, 10))\nsns.heatmap(ted.corr(), annot=True)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:40.110498Z","iopub.execute_input":"2023-02-15T22:39:40.111463Z","iopub.status.idle":"2023-02-15T22:39:40.784958Z","shell.execute_reply.started":"2023-02-15T22:39:40.111428Z","shell.execute_reply":"2023-02-15T22:39:40.783733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the heatmap, the number of views correlates well with language and comments.","metadata":{}},{"cell_type":"code","source":"ted.hvplot.hist(y='languages', height=350, width=450)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:40.787007Z","iopub.execute_input":"2023-02-15T22:39:40.787704Z","iopub.status.idle":"2023-02-15T22:39:40.938324Z","shell.execute_reply.started":"2023-02-15T22:39:40.787656Z","shell.execute_reply":"2023-02-15T22:39:40.937405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# sort by the number of first-level comments, though this is biased in favor of older talks\nted.sort_values('comments').tail()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:40.940022Z","iopub.execute_input":"2023-02-15T22:39:40.940718Z","iopub.status.idle":"2023-02-15T22:39:40.962485Z","shell.execute_reply.started":"2023-02-15T22:39:40.940679Z","shell.execute_reply":"2023-02-15T22:39:40.961124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 📌 Limitations of this approach: \n> 1. sub comments (nested comments).\n> 2. how long its being online.\n\n## 📏 To correct this behavior, one solution is to normalise comments by views.","metadata":{}},{"cell_type":"code","source":"# creating a new column 'comments_per_view'\nted['comments_per_view'] = ted.comments / ted.views\n\n# interpretation: for every view of the same-sex marriage talk, there are 0.002 comments\nted.sort_values('comments_per_view').tail()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:40.964289Z","iopub.execute_input":"2023-02-15T22:39:40.964658Z","iopub.status.idle":"2023-02-15T22:39:40.991988Z","shell.execute_reply.started":"2023-02-15T22:39:40.964615Z","shell.execute_reply":"2023-02-15T22:39:40.990567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# make this more interpretable by inverting the calculation\nted['views_per_comment'] = ted.views / ted.comments\n\n# interpretation: 1 out of every 450 people leave a comment\nted.sort_values('views_per_comment').head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:40.993819Z","iopub.execute_input":"2023-02-15T22:39:40.995033Z","iopub.status.idle":"2023-02-15T22:39:41.022727Z","shell.execute_reply.started":"2023-02-15T22:39:40.994980Z","shell.execute_reply":"2023-02-15T22:39:41.021560Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ted.hvplot.line(y='views', height=350, width=550)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.024536Z","iopub.execute_input":"2023-02-15T22:39:41.024969Z","iopub.status.idle":"2023-02-15T22:39:41.168015Z","shell.execute_reply.started":"2023-02-15T22:39:41.024927Z","shell.execute_reply":"2023-02-15T22:39:41.166611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🧾 **Lessons:**\n\n> 1. Consider the limitations and biases of your data when analyzing it\n> 2. Make your results understandable","metadata":{}},{"cell_type":"markdown","source":"# 📊 Visualize the distribution of comments","metadata":{}},{"cell_type":"code","source":"# line plot is not appropriate here (use it to measure something over time)\nted.hvplot.hist(y='comments', bins=50, height=350, width=550)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.171571Z","iopub.execute_input":"2023-02-15T22:39:41.172141Z","iopub.status.idle":"2023-02-15T22:39:41.322975Z","shell.execute_reply.started":"2023-02-15T22:39:41.172083Z","shell.execute_reply":"2023-02-15T22:39:41.321800Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check how many observations we removed from the plot\nted[ted.comments >= 1000].shape","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.324441Z","iopub.execute_input":"2023-02-15T22:39:41.324879Z","iopub.status.idle":"2023-02-15T22:39:41.333408Z","shell.execute_reply.started":"2023-02-15T22:39:41.324848Z","shell.execute_reply":"2023-02-15T22:39:41.332250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After filtering the data we lose only a small amount of data. This process is called excluding outliers.","metadata":{}},{"cell_type":"code","source":"ted[ted.comments < 1000].hvplot.hist(y='comments', bins=50, height=350, width=550)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.334882Z","iopub.execute_input":"2023-02-15T22:39:41.335220Z","iopub.status.idle":"2023-02-15T22:39:41.484814Z","shell.execute_reply.started":"2023-02-15T22:39:41.335190Z","shell.execute_reply":"2023-02-15T22:39:41.483574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🧾 **Lessons:**\n\n> 1. Choose your plot type based on the question you are answering and the data type(s) you are working with\n> 2. Use pandas one-liners to iterate through plots quickly\n> 3. Try modifying the plot defaults\n> 4. Creating plots involves decision-making","metadata":{}},{"cell_type":"markdown","source":"# 📉 Plot the number of talks that took place each year","metadata":{}},{"cell_type":"code","source":"# event column does not always include the year\nted.event.sample(10)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.486276Z","iopub.execute_input":"2023-02-15T22:39:41.486625Z","iopub.status.idle":"2023-02-15T22:39:41.496371Z","shell.execute_reply.started":"2023-02-15T22:39:41.486592Z","shell.execute_reply":"2023-02-15T22:39:41.495473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can't rely on `event` Feature, because most of the events don't have a year.","metadata":{}},{"cell_type":"code","source":"# dataset documentation for film_date says \"Unix timestamp of the filming\"\nted.film_date.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.497699Z","iopub.execute_input":"2023-02-15T22:39:41.497992Z","iopub.status.idle":"2023-02-15T22:39:41.510367Z","shell.execute_reply.started":"2023-02-15T22:39:41.497966Z","shell.execute_reply":"2023-02-15T22:39:41.509289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# results don't look right\npd.to_datetime(ted.film_date).head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.511668Z","iopub.execute_input":"2023-02-15T22:39:41.512326Z","iopub.status.idle":"2023-02-15T22:39:41.527201Z","shell.execute_reply.started":"2023-02-15T22:39:41.512281Z","shell.execute_reply":"2023-02-15T22:39:41.526014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# now the results look right\npd.to_datetime(ted.film_date, unit='s').head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.528563Z","iopub.execute_input":"2023-02-15T22:39:41.528867Z","iopub.status.idle":"2023-02-15T22:39:41.539908Z","shell.execute_reply.started":"2023-02-15T22:39:41.528840Z","shell.execute_reply":"2023-02-15T22:39:41.539003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ted['film_datetime'] = pd.to_datetime(ted.film_date, unit='s')\n\n# verify that event name matches film_datetime for a random sample\nted[['event', 'film_datetime']].sample(5)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.541063Z","iopub.execute_input":"2023-02-15T22:39:41.541410Z","iopub.status.idle":"2023-02-15T22:39:41.556884Z","shell.execute_reply.started":"2023-02-15T22:39:41.541380Z","shell.execute_reply":"2023-02-15T22:39:41.555631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# new column uses the datetime data type (this was an automatic conversion)\nted.dtypes","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.558104Z","iopub.execute_input":"2023-02-15T22:39:41.558493Z","iopub.status.idle":"2023-02-15T22:39:41.567768Z","shell.execute_reply.started":"2023-02-15T22:39:41.558460Z","shell.execute_reply":"2023-02-15T22:39:41.566550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# datetime columns have convenient attributes under the dt namespace\nted.film_datetime.dt.year.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.569641Z","iopub.execute_input":"2023-02-15T22:39:41.570888Z","iopub.status.idle":"2023-02-15T22:39:41.581154Z","shell.execute_reply.started":"2023-02-15T22:39:41.570841Z","shell.execute_reply":"2023-02-15T22:39:41.579873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# count the number of talks each year using value_counts()\nted.film_datetime.dt.year.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.592885Z","iopub.execute_input":"2023-02-15T22:39:41.593321Z","iopub.status.idle":"2023-02-15T22:39:41.602727Z","shell.execute_reply.started":"2023-02-15T22:39:41.593285Z","shell.execute_reply":"2023-02-15T22:39:41.601575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# points are plotted and connected in the order you give them to pandas\nted.film_datetime.dt.year.value_counts().hvplot.line(height=350, width=550)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.604370Z","iopub.execute_input":"2023-02-15T22:39:41.605668Z","iopub.status.idle":"2023-02-15T22:39:41.763084Z","shell.execute_reply.started":"2023-02-15T22:39:41.605613Z","shell.execute_reply":"2023-02-15T22:39:41.761889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# need to sort the index before plotting\nted.film_datetime.dt.year.value_counts().sort_index().hvplot.line(height=350, width=550)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.764513Z","iopub.execute_input":"2023-02-15T22:39:41.764866Z","iopub.status.idle":"2023-02-15T22:39:41.901580Z","shell.execute_reply.started":"2023-02-15T22:39:41.764834Z","shell.execute_reply":"2023-02-15T22:39:41.900661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# we only have partial data for 2017\nted.film_datetime.max()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.902559Z","iopub.execute_input":"2023-02-15T22:39:41.902892Z","iopub.status.idle":"2023-02-15T22:39:41.912210Z","shell.execute_reply.started":"2023-02-15T22:39:41.902861Z","shell.execute_reply":"2023-02-15T22:39:41.910713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🧾 **Lessons:**\n\n> 1. Read the documentation\n> 2. Use the datetime data type for dates and times\n> 3. Check your work as you go\n> 4. Consider excluding data if it might not be relevant","metadata":{}},{"cell_type":"markdown","source":"# 🥇 What were the \"best\" events in TED history to attend?","metadata":{}},{"cell_type":"code","source":"def clean_event(event):\n    years = list(range(1990, 2018, 1))\n    \n    for year in years:\n        if str(year) in event:\n            return f\"TED_{year}\"\n    else:\n        return event","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.913734Z","iopub.execute_input":"2023-02-15T22:39:41.914109Z","iopub.status.idle":"2023-02-15T22:39:41.921205Z","shell.execute_reply.started":"2023-02-15T22:39:41.914078Z","shell.execute_reply":"2023-02-15T22:39:41.919883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(ted.event.nunique())\nprint(ted.event.apply(clean_event).nunique())","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.922567Z","iopub.execute_input":"2023-02-15T22:39:41.922892Z","iopub.status.idle":"2023-02-15T22:39:41.952534Z","shell.execute_reply.started":"2023-02-15T22:39:41.922864Z","shell.execute_reply":"2023-02-15T22:39:41.951220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ted['event'] = ted.event.apply(clean_event)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.953958Z","iopub.execute_input":"2023-02-15T22:39:41.954792Z","iopub.status.idle":"2023-02-15T22:39:41.981505Z","shell.execute_reply.started":"2023-02-15T22:39:41.954746Z","shell.execute_reply":"2023-02-15T22:39:41.980279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# count the number of talks (great if you value variety, but they may not be great talks)\nted.event.value_counts().head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.982820Z","iopub.execute_input":"2023-02-15T22:39:41.983158Z","iopub.status.idle":"2023-02-15T22:39:41.993423Z","shell.execute_reply.started":"2023-02-15T22:39:41.983128Z","shell.execute_reply":"2023-02-15T22:39:41.992049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# use views as a proxy for \"quality of talk\"\nted.groupby('event').views.mean().head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:41.994730Z","iopub.execute_input":"2023-02-15T22:39:41.995167Z","iopub.status.idle":"2023-02-15T22:39:42.008550Z","shell.execute_reply.started":"2023-02-15T22:39:41.995133Z","shell.execute_reply":"2023-02-15T22:39:42.007321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# find the largest values, but we don't know how many talks are being averaged\nted.groupby('event').views.mean().sort_values().tail()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.009852Z","iopub.execute_input":"2023-02-15T22:39:42.010316Z","iopub.status.idle":"2023-02-15T22:39:42.022519Z","shell.execute_reply.started":"2023-02-15T22:39:42.010273Z","shell.execute_reply":"2023-02-15T22:39:42.021292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# show the number of talks along with the mean (events with the highest means had only 1 or 2 talks)\nted.groupby('event').views.agg(['count', 'mean']).sort_values('mean').tail()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.025155Z","iopub.execute_input":"2023-02-15T22:39:42.026099Z","iopub.status.idle":"2023-02-15T22:39:42.040632Z","shell.execute_reply.started":"2023-02-15T22:39:42.026053Z","shell.execute_reply":"2023-02-15T22:39:42.039293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# calculate the total views per event\nted.groupby('event').views.agg(['count', 'mean', 'sum']).sort_values('sum').tail()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.042309Z","iopub.execute_input":"2023-02-15T22:39:42.043072Z","iopub.status.idle":"2023-02-15T22:39:42.059469Z","shell.execute_reply.started":"2023-02-15T22:39:42.043037Z","shell.execute_reply":"2023-02-15T22:39:42.058099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ted.event.value_counts()[:20].hvplot.barh()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.061058Z","iopub.execute_input":"2023-02-15T22:39:42.061441Z","iopub.status.idle":"2023-02-15T22:39:42.211111Z","shell.execute_reply.started":"2023-02-15T22:39:42.061408Z","shell.execute_reply":"2023-02-15T22:39:42.209839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ted.groupby('event').views.mean().sort_values(ascending=False)[:20].hvplot.barh()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.212759Z","iopub.execute_input":"2023-02-15T22:39:42.213944Z","iopub.status.idle":"2023-02-15T22:39:42.362190Z","shell.execute_reply.started":"2023-02-15T22:39:42.213901Z","shell.execute_reply":"2023-02-15T22:39:42.361025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🧾 **Lessons:**\n\n> 1. Think creatively for how you can use the data you have to answer your question\n> 2. Watch out for small sample sizes","metadata":{}},{"cell_type":"markdown","source":"# 📤 Unpack the ratings data","metadata":{}},{"cell_type":"code","source":"# previously, users could tag talks on the TED website (funny, inspiring, confusing, etc.)\nted.ratings.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.363856Z","iopub.execute_input":"2023-02-15T22:39:42.364450Z","iopub.status.idle":"2023-02-15T22:39:42.373039Z","shell.execute_reply.started":"2023-02-15T22:39:42.364418Z","shell.execute_reply":"2023-02-15T22:39:42.371842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# two ways to examine the ratings data for the first talk\nted.loc[0, 'ratings']\nted.ratings[0]","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.374667Z","iopub.execute_input":"2023-02-15T22:39:42.375002Z","iopub.status.idle":"2023-02-15T22:39:42.387680Z","shell.execute_reply.started":"2023-02-15T22:39:42.374972Z","shell.execute_reply":"2023-02-15T22:39:42.386667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# this is a string not a list\ntype(ted.ratings[0])","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.388982Z","iopub.execute_input":"2023-02-15T22:39:42.389357Z","iopub.status.idle":"2023-02-15T22:39:42.400271Z","shell.execute_reply.started":"2023-02-15T22:39:42.389328Z","shell.execute_reply":"2023-02-15T22:39:42.399064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# convert this into something useful using Python's ast module (Abstract Syntax Tree)\nimport ast\n\n# literal_eval() allows you to evaluate a string containing a Python literal or container\nast.literal_eval('[1, 2, 3]')\n\n# if you have a string representation of something, you can retrieve what it actually represents\ntype(ast.literal_eval('[1, 2, 3]'))","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.401811Z","iopub.execute_input":"2023-02-15T22:39:42.402621Z","iopub.status.idle":"2023-02-15T22:39:42.411970Z","shell.execute_reply.started":"2023-02-15T22:39:42.402579Z","shell.execute_reply":"2023-02-15T22:39:42.410771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# unpack the ratings data for the first talk\nast.literal_eval(ted.ratings[0])","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.413466Z","iopub.execute_input":"2023-02-15T22:39:42.414038Z","iopub.status.idle":"2023-02-15T22:39:42.427002Z","shell.execute_reply.started":"2023-02-15T22:39:42.414006Z","shell.execute_reply":"2023-02-15T22:39:42.425749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# now we have a list (of dictionaries)\ntype(ast.literal_eval(ted.ratings[0]))","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.428490Z","iopub.execute_input":"2023-02-15T22:39:42.428897Z","iopub.status.idle":"2023-02-15T22:39:42.438517Z","shell.execute_reply.started":"2023-02-15T22:39:42.428862Z","shell.execute_reply":"2023-02-15T22:39:42.437667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# define a function to convert an element in the ratings Series from string to list\ndef str_to_list(ratings_str):\n    return ast.literal_eval(ratings_str)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.439631Z","iopub.execute_input":"2023-02-15T22:39:42.440774Z","iopub.status.idle":"2023-02-15T22:39:42.451013Z","shell.execute_reply.started":"2023-02-15T22:39:42.440714Z","shell.execute_reply":"2023-02-15T22:39:42.449723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# test the function\nstr_to_list(ted.ratings[0])","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.452683Z","iopub.execute_input":"2023-02-15T22:39:42.453560Z","iopub.status.idle":"2023-02-15T22:39:42.467281Z","shell.execute_reply.started":"2023-02-15T22:39:42.453523Z","shell.execute_reply":"2023-02-15T22:39:42.466277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Series apply method applies a function to every element in a Series and returns a Series\nted.ratings.apply(str_to_list).head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.469032Z","iopub.execute_input":"2023-02-15T22:39:42.469486Z","iopub.status.idle":"2023-02-15T22:39:42.992486Z","shell.execute_reply.started":"2023-02-15T22:39:42.469450Z","shell.execute_reply":"2023-02-15T22:39:42.991601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# lambda is a shorter alternative\nted.ratings.apply(lambda x: ast.literal_eval(x)).head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:42.993659Z","iopub.execute_input":"2023-02-15T22:39:42.994441Z","iopub.status.idle":"2023-02-15T22:39:43.514387Z","shell.execute_reply.started":"2023-02-15T22:39:42.994405Z","shell.execute_reply":"2023-02-15T22:39:43.513293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# an even shorter alternative is to apply the function directly (without lambda)\nted.ratings.apply(ast.literal_eval).head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:43.515872Z","iopub.execute_input":"2023-02-15T22:39:43.516252Z","iopub.status.idle":"2023-02-15T22:39:44.052804Z","shell.execute_reply.started":"2023-02-15T22:39:43.516201Z","shell.execute_reply":"2023-02-15T22:39:44.051674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ted['ratings_list'] = ted.ratings.apply(lambda x: ast.literal_eval(x))","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.054696Z","iopub.execute_input":"2023-02-15T22:39:44.055170Z","iopub.status.idle":"2023-02-15T22:39:44.568944Z","shell.execute_reply.started":"2023-02-15T22:39:44.055127Z","shell.execute_reply":"2023-02-15T22:39:44.567906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check that the new Series looks as expected\nted.ratings_list[0]","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.570392Z","iopub.execute_input":"2023-02-15T22:39:44.570722Z","iopub.status.idle":"2023-02-15T22:39:44.578642Z","shell.execute_reply.started":"2023-02-15T22:39:44.570693Z","shell.execute_reply":"2023-02-15T22:39:44.577819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# each element in the Series is a list\ntype(ted.ratings_list[0])","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.579831Z","iopub.execute_input":"2023-02-15T22:39:44.580156Z","iopub.status.idle":"2023-02-15T22:39:44.591072Z","shell.execute_reply.started":"2023-02-15T22:39:44.580126Z","shell.execute_reply":"2023-02-15T22:39:44.589664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# data type of the new Series is object\nted.ratings_list.dtype","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.592966Z","iopub.execute_input":"2023-02-15T22:39:44.593495Z","iopub.status.idle":"2023-02-15T22:39:44.604184Z","shell.execute_reply.started":"2023-02-15T22:39:44.593452Z","shell.execute_reply":"2023-02-15T22:39:44.603329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# object is not just for strings\nted.dtypes","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.605797Z","iopub.execute_input":"2023-02-15T22:39:44.606229Z","iopub.status.idle":"2023-02-15T22:39:44.618085Z","shell.execute_reply.started":"2023-02-15T22:39:44.606187Z","shell.execute_reply":"2023-02-15T22:39:44.616733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🧾 **Lessons:**\n\n> 1. Pay attention to data types in pandas\n> 2. Use apply any time it is necessary","metadata":{}},{"cell_type":"markdown","source":"# 🧮 Count the total number of ratings received by each talk\n\n**Bonus exercises:**\n\n> - for each talk, calculate the percentage of ratings that were negative\n> - for each talk, calculate the average number of ratings it received per day since it was published","metadata":{}},{"cell_type":"code","source":"ted.ratings_list[0]","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.620293Z","iopub.execute_input":"2023-02-15T22:39:44.621033Z","iopub.status.idle":"2023-02-15T22:39:44.632141Z","shell.execute_reply.started":"2023-02-15T22:39:44.620985Z","shell.execute_reply":"2023-02-15T22:39:44.631330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# start by building a simple function\ndef get_num_ratings(list_of_dicts):\n    return list_of_dicts[0]","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.634203Z","iopub.execute_input":"2023-02-15T22:39:44.635013Z","iopub.status.idle":"2023-02-15T22:39:44.641301Z","shell.execute_reply.started":"2023-02-15T22:39:44.634964Z","shell.execute_reply":"2023-02-15T22:39:44.640367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# pass it a list, and it returns the first element in the list, which is a dictionary\nget_num_ratings(ted.ratings_list[0])","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.642800Z","iopub.execute_input":"2023-02-15T22:39:44.643502Z","iopub.status.idle":"2023-02-15T22:39:44.658712Z","shell.execute_reply.started":"2023-02-15T22:39:44.643460Z","shell.execute_reply":"2023-02-15T22:39:44.657426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# modify the function to return the vote count\ndef get_num_ratings(list_of_dicts):\n    return list_of_dicts[0]['count']","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.660138Z","iopub.execute_input":"2023-02-15T22:39:44.660808Z","iopub.status.idle":"2023-02-15T22:39:44.668590Z","shell.execute_reply.started":"2023-02-15T22:39:44.660600Z","shell.execute_reply":"2023-02-15T22:39:44.667445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# pass it a list, and it returns a value from the first dictionary in the list\nget_num_ratings(ted.ratings_list[0])","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.669880Z","iopub.execute_input":"2023-02-15T22:39:44.670220Z","iopub.status.idle":"2023-02-15T22:39:44.680963Z","shell.execute_reply.started":"2023-02-15T22:39:44.670189Z","shell.execute_reply":"2023-02-15T22:39:44.680068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# modify the function to get the sum of count\ndef get_num_ratings(list_of_dicts):\n    num = 0\n    for d in list_of_dicts:\n        num = num + d['count']\n    return num","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.682512Z","iopub.execute_input":"2023-02-15T22:39:44.683415Z","iopub.status.idle":"2023-02-15T22:39:44.690130Z","shell.execute_reply.started":"2023-02-15T22:39:44.683378Z","shell.execute_reply":"2023-02-15T22:39:44.688922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# looks about right\nget_num_ratings(ted.ratings_list[0])","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.691607Z","iopub.execute_input":"2023-02-15T22:39:44.691992Z","iopub.status.idle":"2023-02-15T22:39:44.701884Z","shell.execute_reply.started":"2023-02-15T22:39:44.691958Z","shell.execute_reply":"2023-02-15T22:39:44.700828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check with another record\nted.ratings_list[1]","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.703127Z","iopub.execute_input":"2023-02-15T22:39:44.704097Z","iopub.status.idle":"2023-02-15T22:39:44.714997Z","shell.execute_reply.started":"2023-02-15T22:39:44.704063Z","shell.execute_reply":"2023-02-15T22:39:44.713990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# looks about right\nget_num_ratings(ted.ratings_list[1])","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.716213Z","iopub.execute_input":"2023-02-15T22:39:44.716974Z","iopub.status.idle":"2023-02-15T22:39:44.725468Z","shell.execute_reply.started":"2023-02-15T22:39:44.716930Z","shell.execute_reply":"2023-02-15T22:39:44.724573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# apply it to every element in the Series\nted.ratings_list.apply(get_num_ratings).head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.726930Z","iopub.execute_input":"2023-02-15T22:39:44.727533Z","iopub.status.idle":"2023-02-15T22:39:44.746699Z","shell.execute_reply.started":"2023-02-15T22:39:44.727501Z","shell.execute_reply":"2023-02-15T22:39:44.745740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# another alternative is to use a generator expression\nsum((d['count'] for d in ted.ratings_list[0]))","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.748095Z","iopub.execute_input":"2023-02-15T22:39:44.748672Z","iopub.status.idle":"2023-02-15T22:39:44.755030Z","shell.execute_reply.started":"2023-02-15T22:39:44.748638Z","shell.execute_reply":"2023-02-15T22:39:44.754110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# use lambda to apply this method\nted.ratings_list.apply(lambda x: sum((d['count'] for d in x))).head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.756362Z","iopub.execute_input":"2023-02-15T22:39:44.757489Z","iopub.status.idle":"2023-02-15T22:39:44.780169Z","shell.execute_reply.started":"2023-02-15T22:39:44.757454Z","shell.execute_reply":"2023-02-15T22:39:44.779011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# another alternative is to use pd.DataFrame()\npd.DataFrame(ted.ratings_list[0])['count'].sum()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.781323Z","iopub.execute_input":"2023-02-15T22:39:44.782199Z","iopub.status.idle":"2023-02-15T22:39:44.790017Z","shell.execute_reply.started":"2023-02-15T22:39:44.782167Z","shell.execute_reply":"2023-02-15T22:39:44.788964Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# use lambda to apply this method\nted.ratings_list.apply(lambda x: pd.DataFrame(x)['count'].sum()).head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:44.791355Z","iopub.execute_input":"2023-02-15T22:39:44.791903Z","iopub.status.idle":"2023-02-15T22:39:46.456224Z","shell.execute_reply.started":"2023-02-15T22:39:44.791872Z","shell.execute_reply":"2023-02-15T22:39:46.455401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ted['num_ratings'] = ted.ratings_list.apply(get_num_ratings)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.457424Z","iopub.execute_input":"2023-02-15T22:39:46.457912Z","iopub.status.idle":"2023-02-15T22:39:46.469623Z","shell.execute_reply.started":"2023-02-15T22:39:46.457882Z","shell.execute_reply":"2023-02-15T22:39:46.468567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# do one more check\nted.num_ratings.describe()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.471057Z","iopub.execute_input":"2023-02-15T22:39:46.471464Z","iopub.status.idle":"2023-02-15T22:39:46.487987Z","shell.execute_reply.started":"2023-02-15T22:39:46.471430Z","shell.execute_reply":"2023-02-15T22:39:46.486665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🧾 **Lessons:**\n\n> 1. Write your code in small chunks, and check your work as you go\n> 2. Lambda is best for simple functions","metadata":{}},{"cell_type":"markdown","source":"# 🤹 Which occupations deliver the funniest TED talks on average?\n\nBonus exercises:\n\n> - for each talk, calculate the most frequent rating\n> - for each talk, clean the occupation data so that there's only one occupation per talk","metadata":{}},{"cell_type":"markdown","source":"## ✔️ Step 1: Count the number of funny ratings","metadata":{}},{"cell_type":"code","source":"# \"Funny\" is not always the first dictionary in the list\nted.ratings_list.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.489460Z","iopub.execute_input":"2023-02-15T22:39:46.489830Z","iopub.status.idle":"2023-02-15T22:39:46.510757Z","shell.execute_reply.started":"2023-02-15T22:39:46.489798Z","shell.execute_reply":"2023-02-15T22:39:46.509540Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check ratings (not ratings_list) to see if \"Funny\" is always a rating type\nted.ratings.str.contains('Funny').value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.512185Z","iopub.execute_input":"2023-02-15T22:39:46.512585Z","iopub.status.idle":"2023-02-15T22:39:46.523610Z","shell.execute_reply.started":"2023-02-15T22:39:46.512552Z","shell.execute_reply":"2023-02-15T22:39:46.522487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# write a custom function\ndef get_funny_ratings(list_of_dicts):\n    for d in list_of_dicts:\n        if d['name'] == 'Funny':\n            return d['count']","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.525491Z","iopub.execute_input":"2023-02-15T22:39:46.526342Z","iopub.status.idle":"2023-02-15T22:39:46.532869Z","shell.execute_reply.started":"2023-02-15T22:39:46.526296Z","shell.execute_reply":"2023-02-15T22:39:46.531728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# examine a record in which \"Funny\" is not the first dictionary\nted.ratings_list[3]","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.534700Z","iopub.execute_input":"2023-02-15T22:39:46.535208Z","iopub.status.idle":"2023-02-15T22:39:46.550434Z","shell.execute_reply.started":"2023-02-15T22:39:46.535161Z","shell.execute_reply":"2023-02-15T22:39:46.549615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check that the function works\nget_funny_ratings(ted.ratings_list[3])","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.551724Z","iopub.execute_input":"2023-02-15T22:39:46.552200Z","iopub.status.idle":"2023-02-15T22:39:46.560796Z","shell.execute_reply.started":"2023-02-15T22:39:46.552169Z","shell.execute_reply":"2023-02-15T22:39:46.559981Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# apply it to every element in the Series\nted['funny_ratings'] = ted.ratings_list.apply(get_funny_ratings)\nted.funny_ratings.head()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.562016Z","iopub.execute_input":"2023-02-15T22:39:46.562524Z","iopub.status.idle":"2023-02-15T22:39:46.581781Z","shell.execute_reply.started":"2023-02-15T22:39:46.562494Z","shell.execute_reply":"2023-02-15T22:39:46.580751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# check for missing values\nted.funny_ratings.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.583301Z","iopub.execute_input":"2023-02-15T22:39:46.583619Z","iopub.status.idle":"2023-02-15T22:39:46.596610Z","shell.execute_reply.started":"2023-02-15T22:39:46.583591Z","shell.execute_reply":"2023-02-15T22:39:46.595588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ✔️ Step 2: Calculate the percentage of ratings that are funny","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ted['funny_rate'] = ted.funny_ratings / ted.num_ratings\n\n# \"gut check\" that this calculation makes sense by examining the occupations of the funniest talks\nted.sort_values('funny_rate').speaker_occupation.tail(20)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.598202Z","iopub.execute_input":"2023-02-15T22:39:46.598705Z","iopub.status.idle":"2023-02-15T22:39:46.612652Z","shell.execute_reply.started":"2023-02-15T22:39:46.598674Z","shell.execute_reply":"2023-02-15T22:39:46.611336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# examine the occupations of the least funny talks\nted.sort_values('funny_rate').speaker_occupation.head(20)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.625624Z","iopub.execute_input":"2023-02-15T22:39:46.626056Z","iopub.status.idle":"2023-02-15T22:39:46.638783Z","shell.execute_reply.started":"2023-02-15T22:39:46.626020Z","shell.execute_reply":"2023-02-15T22:39:46.637792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ✔️ Step 3: Analyze the funny rate by occupation","metadata":{}},{"cell_type":"code","source":"# calculate the mean funny rate for each occupation\nted.groupby('speaker_occupation').funny_rate.mean().sort_values().tail()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.640185Z","iopub.execute_input":"2023-02-15T22:39:46.640602Z","iopub.status.idle":"2023-02-15T22:39:46.651989Z","shell.execute_reply.started":"2023-02-15T22:39:46.640568Z","shell.execute_reply":"2023-02-15T22:39:46.650618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# however, most of the occupations have a sample size of 1\nted.speaker_occupation.describe()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.653250Z","iopub.execute_input":"2023-02-15T22:39:46.654142Z","iopub.status.idle":"2023-02-15T22:39:46.665756Z","shell.execute_reply.started":"2023-02-15T22:39:46.654105Z","shell.execute_reply":"2023-02-15T22:39:46.664453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ✔️ Step 4: Focus on occupations that are well-represented in the data","metadata":{}},{"cell_type":"code","source":"# count how many times each occupation appears\nted.speaker_occupation.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.667394Z","iopub.execute_input":"2023-02-15T22:39:46.667882Z","iopub.status.idle":"2023-02-15T22:39:46.677044Z","shell.execute_reply.started":"2023-02-15T22:39:46.667853Z","shell.execute_reply":"2023-02-15T22:39:46.675750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# value_counts() outputs a pandas Series, thus we can use pandas to manipulate the output\noccupation_counts = ted.speaker_occupation.value_counts()\ntype(occupation_counts)","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.678617Z","iopub.execute_input":"2023-02-15T22:39:46.679652Z","iopub.status.idle":"2023-02-15T22:39:46.687299Z","shell.execute_reply.started":"2023-02-15T22:39:46.679616Z","shell.execute_reply":"2023-02-15T22:39:46.686493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# show occupations which appear at least 10 times\noccupation_counts[occupation_counts >= 10].hvplot.barh()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.688518Z","iopub.execute_input":"2023-02-15T22:39:46.689007Z","iopub.status.idle":"2023-02-15T22:39:46.838325Z","shell.execute_reply.started":"2023-02-15T22:39:46.688977Z","shell.execute_reply":"2023-02-15T22:39:46.837125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# save the index of this Series\ntop_occupations = occupation_counts[occupation_counts >= 5].index\ntop_occupations","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.839978Z","iopub.execute_input":"2023-02-15T22:39:46.840452Z","iopub.status.idle":"2023-02-15T22:39:46.849840Z","shell.execute_reply.started":"2023-02-15T22:39:46.840406Z","shell.execute_reply":"2023-02-15T22:39:46.848451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## ✔️ Step 5: Re-analyze the funny rate by occupation (for top occupations only)","metadata":{}},{"cell_type":"code","source":"# filter DataFrame to include only those occupations\nted_top_occupations = ted[ted.speaker_occupation.isin(top_occupations)]\nted_top_occupations.shape","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.851347Z","iopub.execute_input":"2023-02-15T22:39:46.851728Z","iopub.status.idle":"2023-02-15T22:39:46.867525Z","shell.execute_reply.started":"2023-02-15T22:39:46.851696Z","shell.execute_reply":"2023-02-15T22:39:46.866308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# redo the previous groupby\nted_top_occupations.groupby('speaker_occupation').funny_rate.mean().sort_values()[:20].hvplot.barh()","metadata":{"execution":{"iopub.status.busy":"2023-02-15T22:39:46.869617Z","iopub.execute_input":"2023-02-15T22:39:46.870096Z","iopub.status.idle":"2023-02-15T22:39:47.020517Z","shell.execute_reply.started":"2023-02-15T22:39:46.870050Z","shell.execute_reply":"2023-02-15T22:39:47.019155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 🧾 **Lessons:**\n\n> 1. Check your assumptions about your data\n> 2. Check whether your results are reasonable\n> 3. Take advantage of the fact that pandas operations often output a DataFrame or a Series\n> 4. Watch out for small sample sizes\n> 5. Consider the impact of missing data\n> 6. Data scientists are hilarious","metadata":{}}]}