{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59156,"databundleVersionId":13874099,"sourceType":"competition"}],"dockerImageVersionId":31153,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🐁 Overview\nHi! I'm Coz, here to step you through a basic Exploratory Data Analysis (EDA) for the MABe Challenge competition! This is an important step in understanding the problem before beginning work building machine learning models.\n\nHere I'll take you through how to explore some of the data that's been provided and create useful visualisations. We'll go through each data file in turn and see what we can get out of it.\n\nRemember to leave an upvote and a comment if you like it!","metadata":{}},{"cell_type":"markdown","source":"# 🎯 Aims\n\nThe aim of this competition is to classify mouse behaviour from videos as accurately as possible. The challenge is that we have video files from many different lab setups with different tracking tools. We need to make the most of this training data to successfully classify the data.","metadata":{}},{"cell_type":"markdown","source":"# ⚙️ Setup \nFirst we need to import the necessary libraries, set the right settings and prepare the right tools for our EDA.","metadata":{}},{"cell_type":"code","source":"# Essential Imports\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\nimport os\nimport ipywidgets as widgets\nfrom IPython.display import display\nimport warnings\nimport json\n\n# Niche Imports\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T17:17:07.514579Z","iopub.execute_input":"2025-10-19T17:17:07.514909Z","iopub.status.idle":"2025-10-19T17:17:07.520367Z","shell.execute_reply.started":"2025-10-19T17:17:07.514886Z","shell.execute_reply":"2025-10-19T17:17:07.518797Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Settings\nsns.set_style(\"whitegrid\")\ncolor_pal = plt.rcParams[\"axes.prop_cycle\"].by_key()[\"color\"]\nwarnings.filterwarnings(\"ignore\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T16:15:12.687290Z","iopub.execute_input":"2025-10-19T16:15:12.688122Z","iopub.status.idle":"2025-10-19T16:15:12.704812Z","shell.execute_reply.started":"2025-10-19T16:15:12.688079Z","shell.execute_reply":"2025-10-19T16:15:12.703933Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Tools","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T16:15:12.706556Z","iopub.execute_input":"2025-10-19T16:15:12.706800Z","iopub.status.idle":"2025-10-19T16:15:12.727203Z","shell.execute_reply.started":"2025-10-19T16:15:12.706780Z","shell.execute_reply":"2025-10-19T16:15:12.725964Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 📂 Prepare files\nLet's look at what we've been given and prepare our files for use.\n\nThe description for each file is given on the competition data page: https://www.kaggle.com/competitions/MABe-mouse-behavior-detection/overview","metadata":{}},{"cell_type":"code","source":"# View files\n\ndef get_comp_files_and_dirs(input_dir):\n    file_list = []\n    dir_list = []\n    try:\n        for comp_dir in os.listdir(input_dir):\n            comp_path = '/'.join([input_dir, comp_dir])\n            print(f\"Competition Directory: {comp_path}\")\n            print(\"Contains:\")\n            with os.scandir(comp_path) as entries:\n                for entry in entries:\n                    if entry.is_file():\n                        print(f\"- (File) {entry.name}, Size: {entry.stat().st_size} bytes\")\n                        file_list.append(os.path.join(input_dir, comp_dir, entry))\n                    elif entry.is_dir():\n                        print(f\"- (Folder) {entry.name}\")\n                        dir_list.append(os.path.join(input_dir, comp_dir, entry))\n\n    except FileNotFoundError:\n        print(f\"The specified directory '{directory}' does not exist.\")\n    except PermissionError:\n        print(f\"Permission error accessing directory '{directory}'.\")\n    return file_list, dir_list\n    \ninput_dir = '/kaggle/input'\nfile_list, dir_list = get_comp_files_and_dirs(input_dir)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T16:15:12.728092Z","iopub.execute_input":"2025-10-19T16:15:12.728386Z","iopub.status.idle":"2025-10-19T16:15:12.749606Z","shell.execute_reply.started":"2025-10-19T16:15:12.728360Z","shell.execute_reply":"2025-10-19T16:15:12.748591Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Load Data\ncomp_path = '/kaggle/input/MABe-mouse-behavior-detection'\ntrain_df = pd.read_csv(os.path.join(comp_path, 'train.csv'))\ntest_df = pd.read_csv(os.path.join(comp_path, 'test.csv'))\nss_df = pd.read_csv(os.path.join(comp_path, 'sample_submission.csv'))\ndf_dict = {\n    \"Training data\": train_df,\n    \"Testing data\": test_df,\n    \"Sample submission\": ss_df,\n}\n\ntest_tracking_dir = os.path.join(comp_path, 'test_tracking')\ntrain_annotation_dir = os.path.join(comp_path, 'train_annotation')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T16:41:08.486531Z","iopub.execute_input":"2025-10-19T16:41:08.486820Z","iopub.status.idle":"2025-10-19T16:41:08.564460Z","shell.execute_reply.started":"2025-10-19T16:41:08.486798Z","shell.execute_reply":"2025-10-19T16:41:08.563717Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Example data\ntrain_annotation_example = pd.read_parquet('/kaggle/input/MABe-mouse-behavior-detection/train_annotation/AdaptableSnail/1212811043.parquet')\ntrain_tracking_example = pd.read_parquet(r'/kaggle/input/MABe-mouse-behavior-detection/train_tracking/AdaptableSnail/1212811043.parquet')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T16:15:12.844589Z","iopub.execute_input":"2025-10-19T16:15:12.844821Z","iopub.status.idle":"2025-10-19T16:15:13.029075Z","shell.execute_reply.started":"2025-10-19T16:15:12.844802Z","shell.execute_reply":"2025-10-19T16:15:13.028219Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 👀 First Look\nLet's take a quick look at what we're working with","metadata":{}},{"cell_type":"code","source":"display(ss_df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T16:15:13.030048Z","iopub.execute_input":"2025-10-19T16:15:13.030293Z","iopub.status.idle":"2025-10-19T16:15:13.039417Z","shell.execute_reply.started":"2025-10-19T16:15:13.030273Z","shell.execute_reply":"2025-10-19T16:15:13.038507Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"display(train_df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T16:15:13.040384Z","iopub.execute_input":"2025-10-19T16:15:13.040779Z","iopub.status.idle":"2025-10-19T16:15:13.076550Z","shell.execute_reply.started":"2025-10-19T16:15:13.040753Z","shell.execute_reply":"2025-10-19T16:15:13.075672Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"display(test_df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T16:15:13.077419Z","iopub.execute_input":"2025-10-19T16:15:13.077646Z","iopub.status.idle":"2025-10-19T16:15:13.093267Z","shell.execute_reply.started":"2025-10-19T16:15:13.077628Z","shell.execute_reply":"2025-10-19T16:15:13.092337Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(train_df.iloc[0])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T16:15:13.094071Z","iopub.execute_input":"2025-10-19T16:15:13.094354Z","iopub.status.idle":"2025-10-19T16:15:13.109746Z","shell.execute_reply.started":"2025-10-19T16:15:13.094330Z","shell.execute_reply":"2025-10-19T16:15:13.108937Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(f'Example Body parts tracked: \\n{train_df.iloc[0][\"body_parts_tracked\"]}')\nprint(f'Example Behaviours labelled: \\n{train_df.iloc[0][\"behaviors_labeled\"]}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T16:15:13.110804Z","iopub.execute_input":"2025-10-19T16:15:13.111123Z","iopub.status.idle":"2025-10-19T16:15:13.126706Z","shell.execute_reply.started":"2025-10-19T16:15:13.111100Z","shell.execute_reply":"2025-10-19T16:15:13.125764Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"display(train_annotation_example)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T16:15:13.127572Z","iopub.execute_input":"2025-10-19T16:15:13.127810Z","iopub.status.idle":"2025-10-19T16:15:13.149054Z","shell.execute_reply.started":"2025-10-19T16:15:13.127790Z","shell.execute_reply":"2025-10-19T16:15:13.148188Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"display(train_tracking_example)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T16:15:13.149925Z","iopub.execute_input":"2025-10-19T16:15:13.150342Z","iopub.status.idle":"2025-10-19T16:15:13.172052Z","shell.execute_reply.started":"2025-10-19T16:15:13.150314Z","shell.execute_reply":"2025-10-19T16:15:13.171179Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 🗺️ Data Exploration\nNow that we've got an idea of what we're working with, let's start building up some statistics to understand our data better.","metadata":{}},{"cell_type":"code","source":"from matplotlib.ticker import ScalarFormatter\n\n# Count unique video_ids per lab_id\nvideos_per_lab = train_df.groupby('lab_id')['video_id'].nunique().sort_values(ascending=False)\n\n# Create column plot with logarithmic y-axis\nplt.figure(figsize=(12, 6))\nplt.bar(range(len(videos_per_lab)), videos_per_lab.values, color=color_pal[0])\nplt.xlabel('Lab ID', fontsize=12)\nplt.ylabel('Number of Unique Videos (log scale)', fontsize=12)\nplt.title('Number of Unique Videos per Lab', fontsize=14, fontweight='bold')\nplt.xticks(range(len(videos_per_lab)), videos_per_lab.index, rotation=45, ha='right')\nplt.yscale('log')\n\n# Format y-axis to show full numbers instead of scientific notation\nax = plt.gca()\nax.yaxis.set_major_formatter(ScalarFormatter())\nax.yaxis.get_major_formatter().set_scientific(False)\n\n# Add gridlines with lighter color\nplt.grid(True, which='both', alpha=0.3, linestyle='-', linewidth=0.5)\n\nplt.tight_layout()\nplt.show()\n\nprint(f'\\nSummary Statistics:')\nprint(f'Total labs: {len(videos_per_lab)}')\nprint(f'Total videos: {videos_per_lab.sum()}')\nprint(f'Mean videos per lab: {videos_per_lab.mean():.1f}')\nprint(f'Median videos per lab: {videos_per_lab.median():.1f}')\nprint(f'Max videos (lab {videos_per_lab.idxmax()}): {videos_per_lab.max()}')\nprint(f'Min videos (lab {videos_per_lab.idxmin()}): {videos_per_lab.min()}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T17:17:23.780365Z","iopub.execute_input":"2025-10-19T17:17:23.780647Z","iopub.status.idle":"2025-10-19T17:17:24.510656Z","shell.execute_reply.started":"2025-10-19T17:17:23.780627Z","shell.execute_reply":"2025-10-19T17:17:24.509783Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 5. Heatmap: Action Types by Lab ID\n# Create a matrix showing which action types are used by each lab\n\n# Build a dataframe with lab_id and action types\nlab_action_data = []\n\nfor idx, row in train_df.iterrows():\n    lab_id = row['lab_id']\n    try:\n        behaviors_list = json.loads(row['behaviors_labeled'].replace(\"'\", '\"'))\n        # Extract action types from behaviors\n        for behavior in behaviors_list:\n            action = behavior.split(',')[-1]  # Get the action part\n            lab_action_data.append({'lab_id': lab_id, 'action': action})\n    except:\n        pass\n\n# Create dataframe and count occurrences\nlab_action_df = pd.DataFrame(lab_action_data)\naction_matrix = lab_action_df.groupby(['lab_id', 'action']).size().unstack(fill_value=0)\n\n# Sort columns by total occurrences (descending)\ncolumn_sums = action_matrix.sum(axis=0).sort_values(ascending=False)\naction_matrix = action_matrix[column_sums.index]\n\n# Sort rows by total occurrences (descending)\nrow_sums = action_matrix.sum(axis=1).sort_values(ascending=False)\naction_matrix = action_matrix.loc[row_sums.index]\n\n# Replace 0 with NaN to make them appear white/empty in the heatmap\naction_matrix_masked = action_matrix.replace(0, np.nan)\n\n# Create heatmap\nplt.figure(figsize=(14, 10))\nsns.heatmap(action_matrix_masked, cmap='viridis', annot=False, fmt='g',\n            cbar_kws={'label': 'Number of Occurrences'},\n            linewidths=0.5, linecolor='lightgray')\nplt.xlabel('Action Type', fontsize=12, fontweight='bold')\nplt.ylabel('Lab ID', fontsize=12, fontweight='bold')\nplt.title('Action Types by Lab ID', fontsize=14, fontweight='bold')\nplt.xticks(rotation=45, ha='right')\nplt.yticks(rotation=0)\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T17:18:08.524237Z","iopub.execute_input":"2025-10-19T17:18:08.524851Z","iopub.status.idle":"2025-10-19T17:18:09.748184Z","shell.execute_reply.started":"2025-10-19T17:18:08.524825Z","shell.execute_reply":"2025-10-19T17:18:09.747337Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 2. Heatmap: Body Parts Tracked by Lab ID\n# Create a matrix showing which body parts are tracked by each lab\n\n# Build a dataframe with lab_id and body parts\nlab_bodypart_data = []\n\nfor idx, row in train_df.iterrows():\n    lab_id = row['lab_id']\n    try:\n        body_parts_list = json.loads(row['body_parts_tracked'].replace(\"'\", '\"'))\n        # Extract each body part\n        for body_part in body_parts_list:\n            lab_bodypart_data.append({'lab_id': lab_id, 'body_part': body_part})\n    except:\n        pass\n\n# Create dataframe and get unique combinations (1 if tracked, 0 if not)\nlab_bodypart_df = pd.DataFrame(lab_bodypart_data)\nbodypart_matrix = lab_bodypart_df.groupby(['lab_id', 'body_part']).size().unstack(fill_value=0)\n\n# Convert to boolean (True if tracked, False if not)\nbodypart_matrix_bool = (bodypart_matrix > 0).astype(int)\n\n# Sort columns by total occurrences (descending)\ncolumn_sums = bodypart_matrix_bool.sum(axis=0).sort_values(ascending=False)\nbodypart_matrix_bool = bodypart_matrix_bool[column_sums.index]\n\n# Sort rows by total occurrences (descending)\nrow_sums = bodypart_matrix_bool.sum(axis=1).sort_values(ascending=False)\nbodypart_matrix_bool = bodypart_matrix_bool.loc[row_sums.index]\n\n# Create custom colormap for binary data (white = not tracked, blue = tracked)\nfrom matplotlib.colors import ListedColormap\ncolors = ['#f0f0f0', '#2E86AB']  # Light gray for 0, Blue for 1\ncmap = ListedColormap(colors)\n\n# Create heatmap\nplt.figure(figsize=(16, 10))\nsns.heatmap(bodypart_matrix_bool, cmap=cmap, annot=False, fmt='d',\n            cbar_kws={'label': 'Tracked', 'ticks': [0.25, 0.75]},\n            linewidths=0.5, linecolor='white', vmin=0, vmax=1)\n\n# Customize colorbar labels\ncolorbar = plt.gca().collections[0].colorbar\ncolorbar.set_ticklabels(['Not Tracked', 'Tracked'])\n\nplt.xlabel('Body Part', fontsize=12, fontweight='bold')\nplt.ylabel('Lab ID', fontsize=12, fontweight='bold')\nplt.title('Body Parts Tracked by Lab ID', fontsize=14, fontweight='bold')\nplt.xticks(rotation=45, ha='right')\nplt.yticks(rotation=0)\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T17:22:41.402710Z","iopub.execute_input":"2025-10-19T17:22:41.403092Z","iopub.status.idle":"2025-10-19T17:22:42.708837Z","shell.execute_reply.started":"2025-10-19T17:22:41.403071Z","shell.execute_reply":"2025-10-19T17:22:42.707943Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 1. Heatmap: Tracking Methods by Lab ID\n# Create a matrix showing which tracking methods are used by each lab\nimport json\n\n# Get unique tracking methods per lab\ntracking_by_lab = train_df.groupby('lab_id')['tracking_method'].apply(lambda x: x.unique().tolist()).to_dict()\n\n# Create a summary dataframe\ntracking_summary = train_df.groupby(['lab_id', 'tracking_method']).size().unstack(fill_value=0)\n\n# Convert to boolean (True if used, False if not)\ntracking_summary_bool = (tracking_summary > 0).astype(int)\n\n# Sort columns by total occurrences (descending)\ncolumn_sums = tracking_summary_bool.sum(axis=0).sort_values(ascending=False)\ntracking_summary_bool = tracking_summary_bool[column_sums.index]\n\n# Sort rows by total occurrences (descending)\nrow_sums = tracking_summary_bool.sum(axis=1).sort_values(ascending=False)\ntracking_summary_bool = tracking_summary_bool.loc[row_sums.index]\n\n# Create custom colormap for binary data (light gray = not used, green = used)\nfrom matplotlib.colors import ListedColormap\ncolors = ['#f0f0f0', '#27AE60']  # Light gray for 0, Green for 1\ncmap = ListedColormap(colors)\n\n# Create heatmap\nplt.figure(figsize=(12, 10))\nsns.heatmap(tracking_summary_bool, cmap=cmap, annot=False, fmt='d',\n            cbar_kws={'label': 'Used', 'ticks': [0.25, 0.75]},\n            linewidths=0.5, linecolor='white', vmin=0, vmax=1)\n\n# Customize colorbar labels\ncolorbar = plt.gca().collections[0].colorbar\ncolorbar.set_ticklabels(['Not Used', 'Used'])\n\nplt.xlabel('Tracking Method', fontsize=12, fontweight='bold')\nplt.ylabel('Lab ID', fontsize=12, fontweight='bold')\nplt.title('Tracking Methods by Lab ID', fontsize=14, fontweight='bold')\nplt.xticks(rotation=45, ha='right')\nplt.yticks(rotation=0)\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T17:27:48.903517Z","iopub.execute_input":"2025-10-19T17:27:48.903786Z","iopub.status.idle":"2025-10-19T17:27:49.339298Z","shell.execute_reply.started":"2025-10-19T17:27:48.903768Z","shell.execute_reply":"2025-10-19T17:27:49.338519Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 4. Heatmap: Mouse Strains by Lab ID\n# Create a matrix showing which mouse strains are used by each lab\n\n# Build a dataframe with lab_id and mouse strains\nlab_strain_data = []\n\nfor idx, row in train_df.iterrows():\n    lab_id = row['lab_id']\n    # Collect strains from all mice (mouse1 through mouse4)\n    for i in range(1, 5):\n        strain_col = f'mouse{i}_strain'\n        if strain_col in row and pd.notna(row[strain_col]) and row[strain_col] != '':\n            lab_strain_data.append({'lab_id': lab_id, 'strain': row[strain_col]})\n\n# Create dataframe and get unique combinations (1 if used, 0 if not)\nlab_strain_df = pd.DataFrame(lab_strain_data)\nstrain_matrix = lab_strain_df.groupby(['lab_id', 'strain']).size().unstack(fill_value=0)\n\n# Convert to boolean (True if used, False if not)\nstrain_matrix_bool = (strain_matrix > 0).astype(int)\n\n# Sort columns by total occurrences (descending)\ncolumn_sums = strain_matrix_bool.sum(axis=0).sort_values(ascending=False)\nstrain_matrix_bool = strain_matrix_bool[column_sums.index]\n\n# Sort rows by total occurrences (descending)\nrow_sums = strain_matrix_bool.sum(axis=1).sort_values(ascending=False)\nstrain_matrix_bool = strain_matrix_bool.loc[row_sums.index]\n\n# Create custom colormap for binary data (light gray = not used, orange = used)\nfrom matplotlib.colors import ListedColormap\ncolors = ['#f0f0f0', '#E67E22']  # Light gray for 0, Orange for 1\ncmap = ListedColormap(colors)\n\n# Create heatmap\nplt.figure(figsize=(14, 10))\nsns.heatmap(strain_matrix_bool, cmap=cmap, annot=False, fmt='d',\n            cbar_kws={'label': 'Used', 'ticks': [0.25, 0.75]},\n            linewidths=0.5, linecolor='white', vmin=0, vmax=1)\n\n# Customize colorbar labels\ncolorbar = plt.gca().collections[0].colorbar\ncolorbar.set_ticklabels(['Not Used', 'Used'])\n\nplt.xlabel('Mouse Strain', fontsize=12, fontweight='bold')\nplt.ylabel('Lab ID', fontsize=12, fontweight='bold')\nplt.title('Mouse Strains by Lab ID', fontsize=14, fontweight='bold')\nplt.xticks(rotation=45, ha='right')\nplt.yticks(rotation=0)\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-10-19T17:27:57.366473Z","iopub.execute_input":"2025-10-19T17:27:57.367346Z","iopub.status.idle":"2025-10-19T17:27:58.488283Z","shell.execute_reply.started":"2025-10-19T17:27:57.367311Z","shell.execute_reply":"2025-10-19T17:27:58.487415Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 🎉 Conclusion\nMore work to come, please upvote this notebook to help other people find it!","metadata":{}}]}