{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":3136,"databundleVersionId":26502,"sourceType":"competition"},{"sourceId":5407,"databundleVersionId":868283,"sourceType":"competition"},{"sourceId":14999,"databundleVersionId":653984,"sourceType":"competition"},{"sourceId":17000,"databundleVersionId":895292,"sourceType":"competition"},{"sourceId":4437,"sourceType":"datasetVersion","datasetId":2405}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 💻 Data Science Best Practices with pandas\n---\n# 📝 Introduction to the TED Talks dataset\n\n> This is a kernel follows the talk from `PyCon 2019` of `Kevin Markham`.\n> - 📍 PyCon: [Full Conference](https://www.youtube.com/watch?v=ZjrUmNq41Eo&t=3778s)\n> - 📍 Youtube channel: [Data School](https://www.youtube.com/user/dataschool)\n---\n\n# 🔍 Exploratory Data Analysis\n> Exploratory Data Analysis is all about answering a specific question. In this notebook we will try to answer the following questions:\n> 1. Which talks provoke the most online discussion?\n> 2. What were the \"best\" events in TED history to attend?\n> 3. Which occupations deliver the funniest TED talks on average?","metadata":{}},{"cell_type":"code","source":"!pip install -q hvplot","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2025-02-02T06:14:08.642809Z","iopub.execute_input":"2025-02-02T06:14:08.643151Z","iopub.status.idle":"2025-02-02T06:14:12.920442Z","shell.execute_reply.started":"2025-02-02T06:14:08.643125Z","shell.execute_reply":"2025-02-02T06:14:12.919149Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport hvplot.pandas","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:12.922301Z","iopub.execute_input":"2025-02-02T06:14:12.922621Z","iopub.status.idle":"2025-02-02T06:14:13.107700Z","shell.execute_reply.started":"2025-02-02T06:14:12.922594Z","shell.execute_reply":"2025-02-02T06:14:13.106621Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ted = pd.read_csv(\"/kaggle/input/ted-talks/ted_main.csv\")\nted.head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:13.109998Z","iopub.execute_input":"2025-02-02T06:14:13.110302Z","iopub.status.idle":"2025-02-02T06:14:13.236463Z","shell.execute_reply.started":"2025-02-02T06:14:13.110254Z","shell.execute_reply":"2025-02-02T06:14:13.235562Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# rows, columns\nted.shape","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:13.237775Z","iopub.execute_input":"2025-02-02T06:14:13.238074Z","iopub.status.idle":"2025-02-02T06:14:13.243256Z","shell.execute_reply.started":"2025-02-02T06:14:13.238050Z","shell.execute_reply":"2025-02-02T06:14:13.242437Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# object columns are usually strings, but can also be arbitrary Python objects (lists, dictionaries)\nted.dtypes","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:13.244226Z","iopub.execute_input":"2025-02-02T06:14:13.244521Z","iopub.status.idle":"2025-02-02T06:14:13.263787Z","shell.execute_reply.started":"2025-02-02T06:14:13.244496Z","shell.execute_reply":"2025-02-02T06:14:13.262755Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ted.info()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:13.264884Z","iopub.execute_input":"2025-02-02T06:14:13.265221Z","iopub.status.idle":"2025-02-02T06:14:13.291650Z","shell.execute_reply.started":"2025-02-02T06:14:13.265187Z","shell.execute_reply":"2025-02-02T06:14:13.290726Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# count the number of missing values in each column\nted.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:13.292880Z","iopub.execute_input":"2025-02-02T06:14:13.293235Z","iopub.status.idle":"2025-02-02T06:14:13.313882Z","shell.execute_reply.started":"2025-02-02T06:14:13.293192Z","shell.execute_reply":"2025-02-02T06:14:13.312781Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# ❌ Missing Values\n> In `machine learning`, we need to handle `missing values`. There are many types of missing values:\n> - `Standard Missing Values`: These are missing values that Pandas can detect.\n> - `Non-Standard Missing Values`: Sometimes it might be the case where there’s missing values that have different formats.\n> - `Unexpected Missing Values`: For example, if our feature is expected to be a string, but there’s a numeric type, then technically this is also a missing value.\n\n---\n\n> It’s important to understand these different types of missing data from a `statistics point of view`. The type of missing data will influence how you deal with filling in the missing values.\n> - Sometimes you’ll simply want to delete those rows, other times you’ll replace them.\n> - A very common way to replace missing values is using a median (for objects) or mean (for numerique values). \n\n---\n\n> But those are weak appraoch, some times we need `domain knowledge` about the data and `statical study` to fill the missing values.","metadata":{}},{"cell_type":"code","source":"# fill the miising values. In the general case if the missing values type is numerique\n# we fill it with mean values. if its an object we fill missing values with mode.\nted['speaker_occupation'] = ted.speaker_occupation.fillna(ted.speaker_occupation.mode()[0])","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:13.314927Z","iopub.execute_input":"2025-02-02T06:14:13.315255Z","iopub.status.idle":"2025-02-02T06:14:13.334154Z","shell.execute_reply.started":"2025-02-02T06:14:13.315229Z","shell.execute_reply":"2025-02-02T06:14:13.332840Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ted.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:13.338770Z","iopub.execute_input":"2025-02-02T06:14:13.339121Z","iopub.status.idle":"2025-02-02T06:14:13.361232Z","shell.execute_reply.started":"2025-02-02T06:14:13.339085Z","shell.execute_reply":"2025-02-02T06:14:13.359544Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ted.describe()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:13.363862Z","iopub.execute_input":"2025-02-02T06:14:13.364167Z","iopub.status.idle":"2025-02-02T06:14:13.402327Z","shell.execute_reply.started":"2025-02-02T06:14:13.364142Z","shell.execute_reply":"2025-02-02T06:14:13.401315Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 📢 Which talks provoke the most online discussion?","metadata":{}},{"cell_type":"code","source":"ted.hvplot.hist(subplots=True, height=250, width=250, shared_axes=False, value_label='Rate').cols(3)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:13.403452Z","iopub.execute_input":"2025-02-02T06:14:13.403797Z","iopub.status.idle":"2025-02-02T06:14:14.150731Z","shell.execute_reply.started":"2025-02-02T06:14:13.403766Z","shell.execute_reply":"2025-02-02T06:14:14.149691Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\n# Filter only numeric columns\nnumeric_columns = ted.select_dtypes(include=['number'])\n\n# Compute correlation matrix\ncorr_matrix = numeric_columns.corr()\n\n# Plot heatmap\nplt.figure(figsize=(12, 10))\nsns.heatmap(corr_matrix, annot=True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:14.151778Z","iopub.execute_input":"2025-02-02T06:14:14.152063Z","iopub.status.idle":"2025-02-02T06:14:14.552576Z","shell.execute_reply.started":"2025-02-02T06:14:14.152039Z","shell.execute_reply":"2025-02-02T06:14:14.551514Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"From the heatmap, the number of views correlates well with language and comments.","metadata":{}},{"cell_type":"code","source":"ted.hvplot.hist(y='languages', height=350, width=450)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:14.553684Z","iopub.execute_input":"2025-02-02T06:14:14.553976Z","iopub.status.idle":"2025-02-02T06:14:14.682879Z","shell.execute_reply.started":"2025-02-02T06:14:14.553951Z","shell.execute_reply":"2025-02-02T06:14:14.681834Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# sort by the number of first-level comments, though this is biased in favor of older talks\nted.sort_values('comments').tail()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:14.683895Z","iopub.execute_input":"2025-02-02T06:14:14.684172Z","iopub.status.idle":"2025-02-02T06:14:14.704445Z","shell.execute_reply.started":"2025-02-02T06:14:14.684148Z","shell.execute_reply":"2025-02-02T06:14:14.703381Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 📌 Limitations of this approach: \n> 1. sub comments (nested comments).\n> 2. how long its being online.\n\n## 📏 To correct this behavior, one solution is to normalise comments by views.","metadata":{}},{"cell_type":"code","source":"# creating a new column 'comments_per_view'\nted['comments_per_view'] = ted.comments / ted.views\n\n# interpretation: for every view of the same-sex marriage talk, there are 0.002 comments\nted.sort_values('comments_per_view').tail()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:14.705516Z","iopub.execute_input":"2025-02-02T06:14:14.705812Z","iopub.status.idle":"2025-02-02T06:14:14.727674Z","shell.execute_reply.started":"2025-02-02T06:14:14.705786Z","shell.execute_reply":"2025-02-02T06:14:14.726181Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# make this more interpretable by inverting the calculation\nted['views_per_comment'] = ted.views / ted.comments\n\n# interpretation: 1 out of every 450 people leave a comment\nted.sort_values('views_per_comment').head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:14.729221Z","iopub.execute_input":"2025-02-02T06:14:14.729613Z","iopub.status.idle":"2025-02-02T06:14:14.757204Z","shell.execute_reply.started":"2025-02-02T06:14:14.729578Z","shell.execute_reply":"2025-02-02T06:14:14.756253Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ted.hvplot.line(y='views', height=350, width=550)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:14.758316Z","iopub.execute_input":"2025-02-02T06:14:14.758664Z","iopub.status.idle":"2025-02-02T06:14:15.103039Z","shell.execute_reply.started":"2025-02-02T06:14:14.758634Z","shell.execute_reply":"2025-02-02T06:14:15.101952Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🧾 **Lessons:**\n\n> 1. Consider the limitations and biases of your data when analyzing it\n> 2. Make your results understandable","metadata":{}},{"cell_type":"markdown","source":"# 📊 Visualize the distribution of comments","metadata":{}},{"cell_type":"code","source":"# line plot is not appropriate here (use it to measure something over time)\nted.hvplot.hist(y='comments', bins=50, height=350, width=550)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.104236Z","iopub.execute_input":"2025-02-02T06:14:15.104671Z","iopub.status.idle":"2025-02-02T06:14:15.235065Z","shell.execute_reply.started":"2025-02-02T06:14:15.104633Z","shell.execute_reply":"2025-02-02T06:14:15.234047Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# check how many observations we removed from the plot\nted[ted.comments >= 1000].shape","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.236223Z","iopub.execute_input":"2025-02-02T06:14:15.236607Z","iopub.status.idle":"2025-02-02T06:14:15.243889Z","shell.execute_reply.started":"2025-02-02T06:14:15.236573Z","shell.execute_reply":"2025-02-02T06:14:15.242952Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"After filtering the data we lose only a small amount of data. This process is called excluding outliers.","metadata":{}},{"cell_type":"code","source":"ted[ted.comments < 1000].hvplot.hist(y='comments', bins=50, height=350, width=550)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.244968Z","iopub.execute_input":"2025-02-02T06:14:15.245249Z","iopub.status.idle":"2025-02-02T06:14:15.386034Z","shell.execute_reply.started":"2025-02-02T06:14:15.245211Z","shell.execute_reply":"2025-02-02T06:14:15.384922Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🧾 **Lessons:**\n\n> 1. Choose your plot type based on the question you are answering and the data type(s) you are working with\n> 2. Use pandas one-liners to iterate through plots quickly\n> 3. Try modifying the plot defaults\n> 4. Creating plots involves decision-making","metadata":{}},{"cell_type":"markdown","source":"# 📉 Plot the number of talks that took place each year","metadata":{}},{"cell_type":"code","source":"# event column does not always include the year\nted.event.sample(10)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.387029Z","iopub.execute_input":"2025-02-02T06:14:15.387325Z","iopub.status.idle":"2025-02-02T06:14:15.394762Z","shell.execute_reply.started":"2025-02-02T06:14:15.387290Z","shell.execute_reply":"2025-02-02T06:14:15.393874Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We can't rely on `event` Feature, because most of the events don't have a year.","metadata":{}},{"cell_type":"code","source":"# dataset documentation for film_date says \"Unix timestamp of the filming\"\nted.film_date.head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.395633Z","iopub.execute_input":"2025-02-02T06:14:15.395896Z","iopub.status.idle":"2025-02-02T06:14:15.417624Z","shell.execute_reply.started":"2025-02-02T06:14:15.395869Z","shell.execute_reply":"2025-02-02T06:14:15.416733Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# results don't look right\npd.to_datetime(ted.film_date).head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.418626Z","iopub.execute_input":"2025-02-02T06:14:15.418938Z","iopub.status.idle":"2025-02-02T06:14:15.437617Z","shell.execute_reply.started":"2025-02-02T06:14:15.418913Z","shell.execute_reply":"2025-02-02T06:14:15.436646Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# now the results look right\npd.to_datetime(ted.film_date, unit='s').head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.438641Z","iopub.execute_input":"2025-02-02T06:14:15.438960Z","iopub.status.idle":"2025-02-02T06:14:15.453650Z","shell.execute_reply.started":"2025-02-02T06:14:15.438936Z","shell.execute_reply":"2025-02-02T06:14:15.452740Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ted['film_datetime'] = pd.to_datetime(ted.film_date, unit='s')\n\n# verify that event name matches film_datetime for a random sample\nted[['event', 'film_datetime']].sample(5)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.454536Z","iopub.execute_input":"2025-02-02T06:14:15.454833Z","iopub.status.idle":"2025-02-02T06:14:15.473828Z","shell.execute_reply.started":"2025-02-02T06:14:15.454809Z","shell.execute_reply":"2025-02-02T06:14:15.472826Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# new column uses the datetime data type (this was an automatic conversion)\nted.dtypes","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.474705Z","iopub.execute_input":"2025-02-02T06:14:15.474963Z","iopub.status.idle":"2025-02-02T06:14:15.482187Z","shell.execute_reply.started":"2025-02-02T06:14:15.474942Z","shell.execute_reply":"2025-02-02T06:14:15.481232Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# datetime columns have convenient attributes under the dt namespace\nted.film_datetime.dt.year.head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.490044Z","iopub.execute_input":"2025-02-02T06:14:15.490391Z","iopub.status.idle":"2025-02-02T06:14:15.501100Z","shell.execute_reply.started":"2025-02-02T06:14:15.490339Z","shell.execute_reply":"2025-02-02T06:14:15.500111Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# count the number of talks each year using value_counts()\nted.film_datetime.dt.year.value_counts()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.504503Z","iopub.execute_input":"2025-02-02T06:14:15.504797Z","iopub.status.idle":"2025-02-02T06:14:15.518662Z","shell.execute_reply.started":"2025-02-02T06:14:15.504761Z","shell.execute_reply":"2025-02-02T06:14:15.517728Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# points are plotted and connected in the order you give them to pandas\nted.film_datetime.dt.year.value_counts().hvplot.line(height=350, width=550)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.519649Z","iopub.execute_input":"2025-02-02T06:14:15.519999Z","iopub.status.idle":"2025-02-02T06:14:15.643445Z","shell.execute_reply.started":"2025-02-02T06:14:15.519965Z","shell.execute_reply":"2025-02-02T06:14:15.642564Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# need to sort the index before plotting\nted.film_datetime.dt.year.value_counts().sort_index().hvplot.line(height=350, width=550)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.644062Z","iopub.execute_input":"2025-02-02T06:14:15.644329Z","iopub.status.idle":"2025-02-02T06:14:15.762939Z","shell.execute_reply.started":"2025-02-02T06:14:15.644299Z","shell.execute_reply":"2025-02-02T06:14:15.761788Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# we only have partial data for 2017\nted.film_datetime.max()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.763897Z","iopub.execute_input":"2025-02-02T06:14:15.764175Z","iopub.status.idle":"2025-02-02T06:14:15.770698Z","shell.execute_reply.started":"2025-02-02T06:14:15.764151Z","shell.execute_reply":"2025-02-02T06:14:15.769748Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🧾 **Lessons:**\n\n> 1. Read the documentation\n> 2. Use the datetime data type for dates and times\n> 3. Check your work as you go\n> 4. Consider excluding data if it might not be relevant","metadata":{}},{"cell_type":"markdown","source":"# 🥇 What were the \"best\" events in TED history to attend?","metadata":{}},{"cell_type":"code","source":"def clean_event(event):\n    years = list(range(1990, 2018, 1))\n    \n    for year in years:\n        if str(year) in event:\n            return f\"TED_{year}\"\n    else:\n        return event","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.771782Z","iopub.execute_input":"2025-02-02T06:14:15.772110Z","iopub.status.idle":"2025-02-02T06:14:15.784293Z","shell.execute_reply.started":"2025-02-02T06:14:15.772068Z","shell.execute_reply":"2025-02-02T06:14:15.783374Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(ted.event.nunique())\nprint(ted.event.apply(clean_event).nunique())","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.785181Z","iopub.execute_input":"2025-02-02T06:14:15.785536Z","iopub.status.idle":"2025-02-02T06:14:15.816623Z","shell.execute_reply.started":"2025-02-02T06:14:15.785511Z","shell.execute_reply":"2025-02-02T06:14:15.815363Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ted['event'] = ted.event.apply(clean_event)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.817648Z","iopub.execute_input":"2025-02-02T06:14:15.817999Z","iopub.status.idle":"2025-02-02T06:14:15.844329Z","shell.execute_reply.started":"2025-02-02T06:14:15.817973Z","shell.execute_reply":"2025-02-02T06:14:15.843177Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# count the number of talks (great if you value variety, but they may not be great talks)\nted.event.value_counts().head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.845544Z","iopub.execute_input":"2025-02-02T06:14:15.845914Z","iopub.status.idle":"2025-02-02T06:14:15.862824Z","shell.execute_reply.started":"2025-02-02T06:14:15.845881Z","shell.execute_reply":"2025-02-02T06:14:15.861837Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# use views as a proxy for \"quality of talk\"\nted.groupby('event').views.mean().head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.864178Z","iopub.execute_input":"2025-02-02T06:14:15.864575Z","iopub.status.idle":"2025-02-02T06:14:15.885142Z","shell.execute_reply.started":"2025-02-02T06:14:15.864539Z","shell.execute_reply":"2025-02-02T06:14:15.884089Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# find the largest values, but we don't know how many talks are being averaged\nted.groupby('event').views.mean().sort_values().tail()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.886177Z","iopub.execute_input":"2025-02-02T06:14:15.886465Z","iopub.status.idle":"2025-02-02T06:14:15.900902Z","shell.execute_reply.started":"2025-02-02T06:14:15.886441Z","shell.execute_reply":"2025-02-02T06:14:15.899874Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# show the number of talks along with the mean (events with the highest means had only 1 or 2 talks)\nted.groupby('event').views.agg(['count', 'mean']).sort_values('mean').tail()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.901682Z","iopub.execute_input":"2025-02-02T06:14:15.901972Z","iopub.status.idle":"2025-02-02T06:14:15.924477Z","shell.execute_reply.started":"2025-02-02T06:14:15.901947Z","shell.execute_reply":"2025-02-02T06:14:15.923255Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# calculate the total views per event\nted.groupby('event').views.agg(['count', 'mean', 'sum']).sort_values('sum').tail()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.925400Z","iopub.execute_input":"2025-02-02T06:14:15.925685Z","iopub.status.idle":"2025-02-02T06:14:15.938587Z","shell.execute_reply.started":"2025-02-02T06:14:15.925661Z","shell.execute_reply":"2025-02-02T06:14:15.937625Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ted.event.value_counts()[:20].hvplot.barh()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:15.939614Z","iopub.execute_input":"2025-02-02T06:14:15.939973Z","iopub.status.idle":"2025-02-02T06:14:16.079950Z","shell.execute_reply.started":"2025-02-02T06:14:15.939940Z","shell.execute_reply":"2025-02-02T06:14:16.078966Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ted.groupby('event').views.mean().sort_values(ascending=False)[:20].hvplot.barh()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:16.080959Z","iopub.execute_input":"2025-02-02T06:14:16.081287Z","iopub.status.idle":"2025-02-02T06:14:16.209574Z","shell.execute_reply.started":"2025-02-02T06:14:16.081235Z","shell.execute_reply":"2025-02-02T06:14:16.208588Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🧾 **Lessons:**\n\n> 1. Think creatively for how you can use the data you have to answer your question\n> 2. Watch out for small sample sizes","metadata":{}},{"cell_type":"markdown","source":"# 📤 Unpack the ratings data","metadata":{}},{"cell_type":"code","source":"# previously, users could tag talks on the TED website (funny, inspiring, confusing, etc.)\nted.ratings.head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:16.210450Z","iopub.execute_input":"2025-02-02T06:14:16.210682Z","iopub.status.idle":"2025-02-02T06:14:16.217588Z","shell.execute_reply.started":"2025-02-02T06:14:16.210664Z","shell.execute_reply":"2025-02-02T06:14:16.216712Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# two ways to examine the ratings data for the first talk\nted.loc[0, 'ratings']\nted.ratings[0]","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:16.218514Z","iopub.execute_input":"2025-02-02T06:14:16.218878Z","iopub.status.idle":"2025-02-02T06:14:16.236979Z","shell.execute_reply.started":"2025-02-02T06:14:16.218842Z","shell.execute_reply":"2025-02-02T06:14:16.236062Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# this is a string not a list\ntype(ted.ratings[0])","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:16.237921Z","iopub.execute_input":"2025-02-02T06:14:16.238311Z","iopub.status.idle":"2025-02-02T06:14:16.253345Z","shell.execute_reply.started":"2025-02-02T06:14:16.238252Z","shell.execute_reply":"2025-02-02T06:14:16.252289Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# convert this into something useful using Python's ast module (Abstract Syntax Tree)\nimport ast\n\n# literal_eval() allows you to evaluate a string containing a Python literal or container\nast.literal_eval('[1, 2, 3]')\n\n# if you have a string representation of something, you can retrieve what it actually represents\ntype(ast.literal_eval('[1, 2, 3]'))","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:16.254297Z","iopub.execute_input":"2025-02-02T06:14:16.254556Z","iopub.status.idle":"2025-02-02T06:14:16.270284Z","shell.execute_reply.started":"2025-02-02T06:14:16.254533Z","shell.execute_reply":"2025-02-02T06:14:16.269385Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# unpack the ratings data for the first talk\nast.literal_eval(ted.ratings[0])","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:16.271108Z","iopub.execute_input":"2025-02-02T06:14:16.271454Z","iopub.status.idle":"2025-02-02T06:14:16.289787Z","shell.execute_reply.started":"2025-02-02T06:14:16.271426Z","shell.execute_reply":"2025-02-02T06:14:16.288430Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# now we have a list (of dictionaries)\ntype(ast.literal_eval(ted.ratings[0]))","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:16.291089Z","iopub.execute_input":"2025-02-02T06:14:16.291474Z","iopub.status.idle":"2025-02-02T06:14:16.308378Z","shell.execute_reply.started":"2025-02-02T06:14:16.291437Z","shell.execute_reply":"2025-02-02T06:14:16.307301Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# define a function to convert an element in the ratings Series from string to list\ndef str_to_list(ratings_str):\n    return ast.literal_eval(ratings_str)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:16.309472Z","iopub.execute_input":"2025-02-02T06:14:16.309798Z","iopub.status.idle":"2025-02-02T06:14:16.323098Z","shell.execute_reply.started":"2025-02-02T06:14:16.309773Z","shell.execute_reply":"2025-02-02T06:14:16.321896Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# test the function\nstr_to_list(ted.ratings[0])","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:16.324093Z","iopub.execute_input":"2025-02-02T06:14:16.324481Z","iopub.status.idle":"2025-02-02T06:14:16.342491Z","shell.execute_reply.started":"2025-02-02T06:14:16.324437Z","shell.execute_reply":"2025-02-02T06:14:16.341586Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Series apply method applies a function to every element in a Series and returns a Series\nted.ratings.apply(str_to_list).head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:16.343417Z","iopub.execute_input":"2025-02-02T06:14:16.343756Z","iopub.status.idle":"2025-02-02T06:14:16.851888Z","shell.execute_reply.started":"2025-02-02T06:14:16.343731Z","shell.execute_reply":"2025-02-02T06:14:16.851035Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# lambda is a shorter alternative\nted.ratings.apply(lambda x: ast.literal_eval(x)).head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:16.852754Z","iopub.execute_input":"2025-02-02T06:14:16.852982Z","iopub.status.idle":"2025-02-02T06:14:17.346723Z","shell.execute_reply.started":"2025-02-02T06:14:16.852962Z","shell.execute_reply":"2025-02-02T06:14:17.345641Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# an even shorter alternative is to apply the function directly (without lambda)\nted.ratings.apply(ast.literal_eval).head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:17.347740Z","iopub.execute_input":"2025-02-02T06:14:17.348076Z","iopub.status.idle":"2025-02-02T06:14:17.843842Z","shell.execute_reply.started":"2025-02-02T06:14:17.348050Z","shell.execute_reply":"2025-02-02T06:14:17.842757Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ted['ratings_list'] = ted.ratings.apply(lambda x: ast.literal_eval(x))","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:17.844963Z","iopub.execute_input":"2025-02-02T06:14:17.845367Z","iopub.status.idle":"2025-02-02T06:14:18.334760Z","shell.execute_reply.started":"2025-02-02T06:14:17.845331Z","shell.execute_reply":"2025-02-02T06:14:18.333825Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# check that the new Series looks as expected\nted.ratings_list[0]","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.335703Z","iopub.execute_input":"2025-02-02T06:14:18.336050Z","iopub.status.idle":"2025-02-02T06:14:18.343304Z","shell.execute_reply.started":"2025-02-02T06:14:18.336017Z","shell.execute_reply":"2025-02-02T06:14:18.342029Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# each element in the Series is a list\ntype(ted.ratings_list[0])","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.344271Z","iopub.execute_input":"2025-02-02T06:14:18.344575Z","iopub.status.idle":"2025-02-02T06:14:18.359564Z","shell.execute_reply.started":"2025-02-02T06:14:18.344538Z","shell.execute_reply":"2025-02-02T06:14:18.358570Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# data type of the new Series is object\nted.ratings_list.dtype","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.360611Z","iopub.execute_input":"2025-02-02T06:14:18.360860Z","iopub.status.idle":"2025-02-02T06:14:18.376378Z","shell.execute_reply.started":"2025-02-02T06:14:18.360839Z","shell.execute_reply":"2025-02-02T06:14:18.375367Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# object is not just for strings\nted.dtypes","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.377355Z","iopub.execute_input":"2025-02-02T06:14:18.377664Z","iopub.status.idle":"2025-02-02T06:14:18.394609Z","shell.execute_reply.started":"2025-02-02T06:14:18.377639Z","shell.execute_reply":"2025-02-02T06:14:18.393622Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🧾 **Lessons:**\n\n> 1. Pay attention to data types in pandas\n> 2. Use apply any time it is necessary","metadata":{}},{"cell_type":"markdown","source":"# 🧮 Count the total number of ratings received by each talk\n\n**Bonus exercises:**\n\n> - for each talk, calculate the percentage of ratings that were negative\n> - for each talk, calculate the average number of ratings it received per day since it was published","metadata":{}},{"cell_type":"code","source":"ted.ratings_list[0]","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.395746Z","iopub.execute_input":"2025-02-02T06:14:18.396093Z","iopub.status.idle":"2025-02-02T06:14:18.412795Z","shell.execute_reply.started":"2025-02-02T06:14:18.396058Z","shell.execute_reply":"2025-02-02T06:14:18.411793Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# start by building a simple function\ndef get_num_ratings(list_of_dicts):\n    return list_of_dicts[0]","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.413795Z","iopub.execute_input":"2025-02-02T06:14:18.414018Z","iopub.status.idle":"2025-02-02T06:14:18.429033Z","shell.execute_reply.started":"2025-02-02T06:14:18.413999Z","shell.execute_reply":"2025-02-02T06:14:18.427997Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# pass it a list, and it returns the first element in the list, which is a dictionary\nget_num_ratings(ted.ratings_list[0])","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.430174Z","iopub.execute_input":"2025-02-02T06:14:18.430853Z","iopub.status.idle":"2025-02-02T06:14:18.450611Z","shell.execute_reply.started":"2025-02-02T06:14:18.430814Z","shell.execute_reply":"2025-02-02T06:14:18.449388Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# modify the function to return the vote count\ndef get_num_ratings(list_of_dicts):\n    return list_of_dicts[0]['count']","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.451688Z","iopub.execute_input":"2025-02-02T06:14:18.452072Z","iopub.status.idle":"2025-02-02T06:14:18.463779Z","shell.execute_reply.started":"2025-02-02T06:14:18.452045Z","shell.execute_reply":"2025-02-02T06:14:18.462780Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# pass it a list, and it returns a value from the first dictionary in the list\nget_num_ratings(ted.ratings_list[0])","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.464748Z","iopub.execute_input":"2025-02-02T06:14:18.465055Z","iopub.status.idle":"2025-02-02T06:14:18.481077Z","shell.execute_reply.started":"2025-02-02T06:14:18.465026Z","shell.execute_reply":"2025-02-02T06:14:18.480131Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# modify the function to get the sum of count\ndef get_num_ratings(list_of_dicts):\n    num = 0\n    for d in list_of_dicts:\n        num = num + d['count']\n    return num","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.481926Z","iopub.execute_input":"2025-02-02T06:14:18.482184Z","iopub.status.idle":"2025-02-02T06:14:18.496242Z","shell.execute_reply.started":"2025-02-02T06:14:18.482162Z","shell.execute_reply":"2025-02-02T06:14:18.495251Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# looks about right\nget_num_ratings(ted.ratings_list[0])","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.497234Z","iopub.execute_input":"2025-02-02T06:14:18.497552Z","iopub.status.idle":"2025-02-02T06:14:18.515218Z","shell.execute_reply.started":"2025-02-02T06:14:18.497529Z","shell.execute_reply":"2025-02-02T06:14:18.514363Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# check with another record\nted.ratings_list[1]","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.516320Z","iopub.execute_input":"2025-02-02T06:14:18.516691Z","iopub.status.idle":"2025-02-02T06:14:18.531286Z","shell.execute_reply.started":"2025-02-02T06:14:18.516655Z","shell.execute_reply":"2025-02-02T06:14:18.530421Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# looks about right\nget_num_ratings(ted.ratings_list[1])","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.532137Z","iopub.execute_input":"2025-02-02T06:14:18.532419Z","iopub.status.idle":"2025-02-02T06:14:18.548061Z","shell.execute_reply.started":"2025-02-02T06:14:18.532397Z","shell.execute_reply":"2025-02-02T06:14:18.546969Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# apply it to every element in the Series\nted.ratings_list.apply(get_num_ratings).head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.549572Z","iopub.execute_input":"2025-02-02T06:14:18.549964Z","iopub.status.idle":"2025-02-02T06:14:18.571004Z","shell.execute_reply.started":"2025-02-02T06:14:18.549925Z","shell.execute_reply":"2025-02-02T06:14:18.570085Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# another alternative is to use a generator expression\nsum((d['count'] for d in ted.ratings_list[0]))","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.572058Z","iopub.execute_input":"2025-02-02T06:14:18.572351Z","iopub.status.idle":"2025-02-02T06:14:18.588974Z","shell.execute_reply.started":"2025-02-02T06:14:18.572312Z","shell.execute_reply":"2025-02-02T06:14:18.588019Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# use lambda to apply this method\nted.ratings_list.apply(lambda x: sum((d['count'] for d in x))).head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.590160Z","iopub.execute_input":"2025-02-02T06:14:18.590616Z","iopub.status.idle":"2025-02-02T06:14:18.614447Z","shell.execute_reply.started":"2025-02-02T06:14:18.590580Z","shell.execute_reply":"2025-02-02T06:14:18.613568Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# another alternative is to use pd.DataFrame()\npd.DataFrame(ted.ratings_list[0])['count'].sum()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.615422Z","iopub.execute_input":"2025-02-02T06:14:18.615785Z","iopub.status.idle":"2025-02-02T06:14:18.631371Z","shell.execute_reply.started":"2025-02-02T06:14:18.615758Z","shell.execute_reply":"2025-02-02T06:14:18.630395Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# use lambda to apply this method\nted.ratings_list.apply(lambda x: pd.DataFrame(x)['count'].sum()).head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:18.632479Z","iopub.execute_input":"2025-02-02T06:14:18.632876Z","iopub.status.idle":"2025-02-02T06:14:19.800641Z","shell.execute_reply.started":"2025-02-02T06:14:18.632838Z","shell.execute_reply":"2025-02-02T06:14:19.799731Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ted['num_ratings'] = ted.ratings_list.apply(get_num_ratings)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:19.801570Z","iopub.execute_input":"2025-02-02T06:14:19.801827Z","iopub.status.idle":"2025-02-02T06:14:19.813129Z","shell.execute_reply.started":"2025-02-02T06:14:19.801805Z","shell.execute_reply":"2025-02-02T06:14:19.811993Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# do one more check\nted.num_ratings.describe()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:19.814458Z","iopub.execute_input":"2025-02-02T06:14:19.814752Z","iopub.status.idle":"2025-02-02T06:14:19.835674Z","shell.execute_reply.started":"2025-02-02T06:14:19.814727Z","shell.execute_reply":"2025-02-02T06:14:19.834605Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🧾 **Lessons:**\n\n> 1. Write your code in small chunks, and check your work as you go\n> 2. Lambda is best for simple functions","metadata":{}},{"cell_type":"markdown","source":"# 🤹 Which occupations deliver the funniest TED talks on average?\n\nBonus exercises:\n\n> - for each talk, calculate the most frequent rating\n> - for each talk, clean the occupation data so that there's only one occupation per talk","metadata":{}},{"cell_type":"markdown","source":"## ✔️ Step 1: Count the number of funny ratings","metadata":{}},{"cell_type":"code","source":"# \"Funny\" is not always the first dictionary in the list\nted.ratings_list.head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:19.836868Z","iopub.execute_input":"2025-02-02T06:14:19.837209Z","iopub.status.idle":"2025-02-02T06:14:19.859718Z","shell.execute_reply.started":"2025-02-02T06:14:19.837170Z","shell.execute_reply":"2025-02-02T06:14:19.858653Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# check ratings (not ratings_list) to see if \"Funny\" is always a rating type\nted.ratings.str.contains('Funny').value_counts()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:19.860947Z","iopub.execute_input":"2025-02-02T06:14:19.861346Z","iopub.status.idle":"2025-02-02T06:14:19.882143Z","shell.execute_reply.started":"2025-02-02T06:14:19.861309Z","shell.execute_reply":"2025-02-02T06:14:19.881215Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# write a custom function\ndef get_funny_ratings(list_of_dicts):\n    for d in list_of_dicts:\n        if d['name'] == 'Funny':\n            return d['count']","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:19.883211Z","iopub.execute_input":"2025-02-02T06:14:19.883562Z","iopub.status.idle":"2025-02-02T06:14:19.898918Z","shell.execute_reply.started":"2025-02-02T06:14:19.883536Z","shell.execute_reply":"2025-02-02T06:14:19.897902Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# examine a record in which \"Funny\" is not the first dictionary\nted.ratings_list[3]","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:19.899805Z","iopub.execute_input":"2025-02-02T06:14:19.900046Z","iopub.status.idle":"2025-02-02T06:14:19.917945Z","shell.execute_reply.started":"2025-02-02T06:14:19.900026Z","shell.execute_reply":"2025-02-02T06:14:19.916714Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# check that the function works\nget_funny_ratings(ted.ratings_list[3])","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:19.918894Z","iopub.execute_input":"2025-02-02T06:14:19.919186Z","iopub.status.idle":"2025-02-02T06:14:19.934809Z","shell.execute_reply.started":"2025-02-02T06:14:19.919160Z","shell.execute_reply":"2025-02-02T06:14:19.933838Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# apply it to every element in the Series\nted['funny_ratings'] = ted.ratings_list.apply(get_funny_ratings)\nted.funny_ratings.head()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:19.936049Z","iopub.execute_input":"2025-02-02T06:14:19.936394Z","iopub.status.idle":"2025-02-02T06:14:19.964486Z","shell.execute_reply.started":"2025-02-02T06:14:19.936367Z","shell.execute_reply":"2025-02-02T06:14:19.963378Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# check for missing values\nted.funny_ratings.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:19.965121Z","iopub.execute_input":"2025-02-02T06:14:19.965513Z","iopub.status.idle":"2025-02-02T06:14:19.988609Z","shell.execute_reply.started":"2025-02-02T06:14:19.965486Z","shell.execute_reply":"2025-02-02T06:14:19.987347Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ✔️ Step 2: Calculate the percentage of ratings that are funny","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"ted['funny_rate'] = ted.funny_ratings / ted.num_ratings\n\n# \"gut check\" that this calculation makes sense by examining the occupations of the funniest talks\nted.sort_values('funny_rate').speaker_occupation.tail(20)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:20.001739Z","iopub.execute_input":"2025-02-02T06:14:20.002061Z","iopub.status.idle":"2025-02-02T06:14:20.014916Z","shell.execute_reply.started":"2025-02-02T06:14:20.002034Z","shell.execute_reply":"2025-02-02T06:14:20.014176Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# examine the occupations of the least funny talks\nted.sort_values('funny_rate').speaker_occupation.head(20)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:20.016035Z","iopub.execute_input":"2025-02-02T06:14:20.016357Z","iopub.status.idle":"2025-02-02T06:14:20.034698Z","shell.execute_reply.started":"2025-02-02T06:14:20.016319Z","shell.execute_reply":"2025-02-02T06:14:20.033592Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ✔️ Step 3: Analyze the funny rate by occupation","metadata":{}},{"cell_type":"code","source":"# calculate the mean funny rate for each occupation\nted.groupby('speaker_occupation').funny_rate.mean().sort_values().tail()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:20.035789Z","iopub.execute_input":"2025-02-02T06:14:20.036129Z","iopub.status.idle":"2025-02-02T06:14:20.051190Z","shell.execute_reply.started":"2025-02-02T06:14:20.036098Z","shell.execute_reply":"2025-02-02T06:14:20.050159Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# however, most of the occupations have a sample size of 1\nted.speaker_occupation.describe()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:20.052253Z","iopub.execute_input":"2025-02-02T06:14:20.052594Z","iopub.status.idle":"2025-02-02T06:14:20.062119Z","shell.execute_reply.started":"2025-02-02T06:14:20.052568Z","shell.execute_reply":"2025-02-02T06:14:20.061109Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ✔️ Step 4: Focus on occupations that are well-represented in the data","metadata":{}},{"cell_type":"code","source":"# count how many times each occupation appears\nted.speaker_occupation.value_counts()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:20.063078Z","iopub.execute_input":"2025-02-02T06:14:20.063395Z","iopub.status.idle":"2025-02-02T06:14:20.080094Z","shell.execute_reply.started":"2025-02-02T06:14:20.063363Z","shell.execute_reply":"2025-02-02T06:14:20.078985Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# value_counts() outputs a pandas Series, thus we can use pandas to manipulate the output\noccupation_counts = ted.speaker_occupation.value_counts()\ntype(occupation_counts)","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:20.081041Z","iopub.execute_input":"2025-02-02T06:14:20.081479Z","iopub.status.idle":"2025-02-02T06:14:20.097761Z","shell.execute_reply.started":"2025-02-02T06:14:20.081453Z","shell.execute_reply":"2025-02-02T06:14:20.096687Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# show occupations which appear at least 10 times\noccupation_counts[occupation_counts >= 10].hvplot.barh()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:20.098828Z","iopub.execute_input":"2025-02-02T06:14:20.099195Z","iopub.status.idle":"2025-02-02T06:14:20.237494Z","shell.execute_reply.started":"2025-02-02T06:14:20.099148Z","shell.execute_reply":"2025-02-02T06:14:20.236392Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# save the index of this Series\ntop_occupations = occupation_counts[occupation_counts >= 5].index\ntop_occupations","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:20.238563Z","iopub.execute_input":"2025-02-02T06:14:20.238910Z","iopub.status.idle":"2025-02-02T06:14:20.245664Z","shell.execute_reply.started":"2025-02-02T06:14:20.238885Z","shell.execute_reply":"2025-02-02T06:14:20.244783Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ✔️ Step 5: Re-analyze the funny rate by occupation (for top occupations only)","metadata":{}},{"cell_type":"code","source":"# filter DataFrame to include only those occupations\nted_top_occupations = ted[ted.speaker_occupation.isin(top_occupations)]\nted_top_occupations.shape","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:20.246566Z","iopub.execute_input":"2025-02-02T06:14:20.246807Z","iopub.status.idle":"2025-02-02T06:14:20.268204Z","shell.execute_reply.started":"2025-02-02T06:14:20.246786Z","shell.execute_reply":"2025-02-02T06:14:20.267424Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# redo the previous groupby\nted_top_occupations.groupby('speaker_occupation').funny_rate.mean().sort_values()[:20].hvplot.barh()","metadata":{"execution":{"iopub.status.busy":"2025-02-02T06:14:20.269051Z","iopub.execute_input":"2025-02-02T06:14:20.269301Z","iopub.status.idle":"2025-02-02T06:14:20.407256Z","shell.execute_reply.started":"2025-02-02T06:14:20.269281Z","shell.execute_reply":"2025-02-02T06:14:20.406241Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🧾 **Lessons:**\n\n> 1. Check your assumptions about your data\n> 2. Check whether your results are reasonable\n> 3. Take advantage of the fact that pandas operations often output a DataFrame or a Series\n> 4. Watch out for small sample sizes\n> 5. Consider the impact of missing data\n> 6. Data scientists are hilarious","metadata":{}}]}