{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59156,"databundleVersionId":13719397,"sourceType":"competition"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n **🐭 Welcome to the MABe Challenge.**\n\n> A Visual Odyssey into Mouse Social BehaviorsEver wondered what secrets lie in the subtle movements of mice as they interact in their tiny worlds? The MABe Mouse Behavior Detection competition invites us to decode over 30 social and non-social behaviors from pose estimates, revolutionizing how we study animal behavior. This notebook is your guide to navigating the rich, complex dataset of mouse movements, offering a visually immersive and analytically deep Exploratory Data Analysis (EDA) to kickstart your journey.\n\nUnlike standard EDAs, this notebook goes beyond static plots. It brings mouse behaviors to life with interactive Plotly visualizations, animated trajectories, and intuitive heatmaps, making it easier to grasp the dataset’s nuances—variability across labs, sparse annotations, and diverse behaviors like sniffing, chasing, or grooming. Whether you're a data scientist, neuroscientist, or ethology enthusiast, this EDA equips you with actionable insights to build robust models that generalize across labs and setups.\n\nWhat’s Inside:\n\n****🔍  Metadata Mastery: Uncover patterns in labs, mouse traits, arena setups, and tracking methods with interactive scatter plots and histograms.****\n\n****🐾 Behavior Breakdown: Dive into behavior frequencies, durations, and lab-specific patterns via dynamic bar charts and heatmaps.****\n\n****🕹️ * Pose Visualizations: Watch mice move with an enhanced visualizer class, featuring animated trajectories and frame-by-frame behavior annotations.****\n\n****📈 * Advanced Insights: Explore inter-mouse distances, correlations, and statistical summaries to inspire feature engineering.****\n\n> Goal: Empower you to understand the dataset’s structure, spot challenges (like imbalanced behaviors or lab variability), and spark ideas for modeling success. This notebook is designed for clarity, reproducibility, and visual appeal, making complex data intuitive and engaging.\nIf this EDA sparks inspiration or saves you time, please consider an upvote ⭐—it fuels my motivation to share high-quality, community-driven solutions. Let’s dive into the world of mice and unlock the secrets of their social behaviors together!","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport warnings\nfrom sklearn.decomposition import PCA\nfrom sklearn.preprocessing import StandardScaler\nwarnings.filterwarnings('ignore')\n\n\nplt.style.use('ggplot')\nsns.set_palette(\"viridis\")\npd.set_option('display.max_columns', None)\n\n\ntrain_df = pd.read_csv('/kaggle/input/MABe-mouse-behavior-detection/train.csv')\n\n\nprint(\"Dataset Shape:\", train_df.shape)\nprint(\"\\nColumns Overview:\")\nprint(train_df.info())\nprint(\"\\nFirst Few Rows:\")\ndisplay(train_df.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-22T19:03:24.038319Z","iopub.execute_input":"2025-09-22T19:03:24.038652Z","iopub.status.idle":"2025-09-22T19:03:24.579591Z","shell.execute_reply.started":"2025-09-22T19:03:24.038628Z","shell.execute_reply":"2025-09-22T19:03:24.578242Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\n\nlab_distribution = train_df['lab_id'].value_counts()\n\n\nplt.figure(figsize=(10,6))\nlab_distribution.sort_values().plot(kind='barh', color='skyblue')\nplt.title(\"Data Distribution by Lab\")\nplt.xlabel(\"Count\")\nplt.ylabel(\"Lab ID\")\nplt.tight_layout()\nplt.show()\n\nthreshold = 0.02 \ncounts = lab_distribution / lab_distribution.sum()\nothers = counts[counts < threshold].sum()\ncounts = counts[counts >= threshold]\ncounts[\"Others\"] = others\n\nplt.figure(figsize=(8,8))\nplt.pie(counts, labels=counts.index, autopct='%1.1f%%', startangle=140)\nplt.title(\"Data Proportion by Lab\")\nplt.tight_layout()\nplt.show()\n\n\nplt.figure(figsize=(12,6))\ntrain_df.boxplot(column=\"video_duration_sec\", by=\"lab_id\", rot=45)\nplt.title(\"Video Duration Distribution by Laboratory\")\nplt.suptitle(\"\")\nplt.xlabel(\"Laboratory\")\nplt.ylabel(\"Duration (seconds)\")\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-22T18:49:05.987678Z","iopub.execute_input":"2025-09-22T18:49:05.988915Z","iopub.status.idle":"2025-09-22T18:49:06.866011Z","shell.execute_reply.started":"2025-09-22T18:49:05.988834Z","shell.execute_reply":"2025-09-22T18:49:06.865045Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\nmouse_columns = [col for col in train_df.columns if 'mouse' in col and any(x in col for x in ['strain', 'color', 'sex', 'age', 'condition'])]\nmouse_data = []\n\nfor i in range(1, 5):\n    mouse_cols = [col for col in mouse_columns if col.startswith(f'mouse{i}_')]\n    if mouse_cols:\n        for idx, row in train_df.iterrows():\n            if pd.notna(row[f'mouse{i}_strain']):\n                mouse_data.append({\n                    'video_id': row['video_id'],\n                    'lab_id': row['lab_id'],\n                    'mouse_id': f'mouse{i}',\n                    'strain': row[f'mouse{i}_strain'],\n                    'color': row[f'mouse{i}_color'],\n                    'sex': row[f'mouse{i}_sex'],\n                    'age': row[f'mouse{i}_age'],\n                    'condition': row[f'mouse{i}_condition']\n                })\n\nmouse_df = pd.DataFrame(mouse_data)\n\nstrain_counts = mouse_df['strain'].value_counts()\nsex_counts = mouse_df['sex'].value_counts()\ncolor_counts = mouse_df['color'].value_counts()\ncondition_counts = mouse_df['condition'].value_counts()\n\nfig, axs = plt.subplots(2, 2, figsize=(12, 10))\nstrain_counts.plot(kind='barh', ax=axs[0,0], color='skyblue')\naxs[0,0].set_title(\"Strain Distribution\")\naxs[0,0].set_xlabel(\"Count\")\naxs[0,0].set_ylabel(\"Strain\")\n\nsex_counts.plot(kind='barh', ax=axs[0,1], color='lightgreen')\naxs[0,1].set_title(\"Sex Distribution\")\naxs[0,1].set_xlabel(\"Count\")\naxs[0,1].set_ylabel(\"Sex\")\n\ncolor_counts.plot(kind='barh', ax=axs[1,0], color='salmon')\naxs[1,0].set_title(\"Color Distribution\")\naxs[1,0].set_xlabel(\"Count\")\naxs[1,0].set_ylabel(\"Color\")\n\ncondition_counts.plot(kind='barh', ax=axs[1,1], color='orchid')\naxs[1,1].set_title(\"Condition Distribution\")\naxs[1,1].set_xlabel(\"Count\")\naxs[1,1].set_ylabel(\"Condition\")\n\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-22T18:56:42.774414Z","iopub.execute_input":"2025-09-22T18:56:42.774764Z","iopub.status.idle":"2025-09-22T18:57:18.763127Z","shell.execute_reply.started":"2025-09-22T18:56:42.774742Z","shell.execute_reply":"2025-09-22T18:57:18.762158Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\ntrain_df['resolution'] = train_df['video_width_pix'].astype(str) + 'x' + train_df['video_height_pix'].astype(str)\nresolution_counts = train_df['resolution'].value_counts().head(10)\n\nfig, axs = plt.subplots(2, 2, figsize=(14, 10))\n\naxs[0,0].hist(train_df['frames_per_second'].dropna(), bins=20, color='skyblue', edgecolor='black')\naxs[0,0].set_title(\"FPS Distribution\")\naxs[0,0].set_xlabel(\"Frames per Second\")\naxs[0,0].set_ylabel(\"Count\")\n\naxs[0,1].hist(train_df['video_duration_sec'].dropna(), bins=20, color='lightgreen', edgecolor='black')\naxs[0,1].set_title(\"Duration Distribution\")\naxs[0,1].set_xlabel(\"Video Duration (sec)\")\naxs[0,1].set_ylabel(\"Count\")\n\naxs[1,0].bar(resolution_counts.index, resolution_counts.values, color='salmon', edgecolor='black')\naxs[1,0].set_title(\"Video Resolutions (Top 10)\")\naxs[1,0].set_xlabel(\"Resolution\")\naxs[1,0].set_ylabel(\"Count\")\naxs[1,0].tick_params(axis='x', rotation=45)\n\naxs[1,1].hist(train_df['pix_per_cm_approx'].dropna(), bins=20, color='orchid', edgecolor='black')\naxs[1,1].set_title(\"Pixels per cm Distribution\")\naxs[1,1].set_xlabel(\"Pixels per cm\")\naxs[1,1].set_ylabel(\"Count\")\n\nplt.tight_layout()\nplt.show()\n\ntechnical_params = train_df[['frames_per_second', 'video_duration_sec', 'pix_per_cm_approx', \n                            'video_width_pix', 'video_height_pix', 'arena_width_cm', 'arena_height_cm']].corr()\n\nplt.figure(figsize=(10,8))\nsns.heatmap(technical_params, annot=True, fmt=\".2f\", cmap=\"coolwarm\", square=True)\nplt.title(\"Correlation Matrix of Technical Parameters\")\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-22T18:59:07.159238Z","iopub.execute_input":"2025-09-22T18:59:07.160502Z","iopub.status.idle":"2025-09-22T18:59:08.910331Z","shell.execute_reply.started":"2025-09-22T18:59:07.160468Z","shell.execute_reply":"2025-09-22T18:59:08.909388Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\n\narena_counts = train_df['arena_type'].value_counts()\narena_shape_counts = train_df['arena_shape'].value_counts()\n\nfig, axs = plt.subplots(1, 2, figsize=(12, 5))\n\naxs[0].bar(arena_counts.index, arena_counts.values, color='skyblue', edgecolor='black')\naxs[0].set_title(\"Arena Types\")\naxs[0].set_xlabel(\"Type\")\naxs[0].set_ylabel(\"Count\")\naxs[0].tick_params(axis='x', rotation=45)\n\naxs[1].bar(arena_shape_counts.index, arena_shape_counts.values, color='salmon', edgecolor='black')\naxs[1].set_title(\"Arena Shapes\")\naxs[1].set_xlabel(\"Shape\")\naxs[1].set_ylabel(\"Count\")\naxs[1].tick_params(axis='x', rotation=45)\n\nplt.tight_layout()\nplt.show()\n\nplt.figure(figsize=(8,6))\nsns.scatterplot(data=train_df, x='arena_width_cm', y='arena_height_cm', hue='arena_shape', palette='Set2')\nplt.title(\"Arena Dimensions (cm)\")\nplt.xlabel(\"Width (cm)\")\nplt.ylabel(\"Height (cm)\")\nplt.legend(title='Arena Shape')\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-22T19:01:25.753869Z","iopub.execute_input":"2025-09-22T19:01:25.754752Z","iopub.status.idle":"2025-09-22T19:01:26.705163Z","shell.execute_reply.started":"2025-09-22T19:01:25.754716Z","shell.execute_reply":"2025-09-22T19:01:26.704000Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\ntracking_counts = train_df['tracking_method'].value_counts()\n\nplt.figure(figsize=(8,5))\nplt.bar(tracking_counts.index, tracking_counts.values, color='skyblue', edgecolor='black')\nplt.title(\"Tracking Methods Used\")\nplt.xlabel(\"Method\")\nplt.ylabel(\"Number of Videos\")\nplt.xticks(rotation=45)\nplt.tight_layout()\nplt.show()\n\nbody_parts = train_df['body_parts_tracked'].str.split(',', expand=True).stack().value_counts()\nbody_parts = body_parts[body_parts > 10]\n\nplt.figure(figsize=(10,5))\nplt.bar(body_parts.index, body_parts.values, color='salmon', edgecolor='black')\nplt.title(\"Tracked Body Parts (Most Common)\")\nplt.xlabel(\"Body Part\")\nplt.ylabel(\"Number of Occurrences\")\nplt.xticks(rotation=45)\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-22T19:01:31.286416Z","iopub.execute_input":"2025-09-22T19:01:31.287199Z","iopub.status.idle":"2025-09-22T19:01:31.911811Z","shell.execute_reply.started":"2025-09-22T19:01:31.287167Z","shell.execute_reply":"2025-09-22T19:01:31.910875Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"behaviors = train_df['behaviors_labeled'].str.split(',', expand=True).stack().str.strip().value_counts()\n\nplt.figure(figsize=(12,5))\nplt.bar(behaviors.index, behaviors.values, color='skyblue', edgecolor='black')\nplt.title(\"Annotated Behaviors in the Dataset\")\nplt.xlabel(\"Behavior\")\nplt.ylabel(\"Number of Occurrences\")\nplt.xticks(rotation=45)\nplt.tight_layout()\nplt.show()\n\nlab_behaviors = train_df.groupby('lab_id')['behaviors_labeled'] \\\n    .apply(lambda x: x.str.split(',').explode().str.strip().value_counts())\nlab_behaviors = lab_behaviors.reset_index()\nlab_behaviors.columns = ['lab_id', 'behavior', 'count']\n\ntop_behaviors = behaviors.head(10).index\nlab_behaviors_top = lab_behaviors[lab_behaviors['behavior'].isin(top_behaviors)]\n\nplt.figure(figsize=(12,6))\nsns.barplot(data=lab_behaviors_top, x='behavior', y='count', hue='lab_id', palette='Set2')\nplt.title(\"Top 10 Behaviors by Laboratory\")\nplt.xlabel(\"Behavior\")\nplt.ylabel(\"Number of Occurrences\")\nplt.xticks(rotation=45)\nplt.legend(title='Lab ID')\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-22T19:02:12.215128Z","iopub.execute_input":"2025-09-22T19:02:12.215531Z","iopub.status.idle":"2025-09-22T19:02:14.726453Z","shell.execute_reply.started":"2025-09-22T19:02:12.215504Z","shell.execute_reply":"2025-09-22T19:02:14.725400Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"lab_features = train_df.groupby('lab_id').agg({\n    'frames_per_second': 'mean',\n    'video_duration_sec': 'mean',\n    'pix_per_cm_approx': 'mean',\n    'video_width_pix': 'mean',\n    'video_height_pix': 'mean'\n}).round(2)\n\nplt.figure(figsize=(18,10))\nsns.heatmap(lab_features.T, annot=True, fmt=\".2f\", cmap=\"coolwarm\", cbar=True)\nplt.title(\"Average Technical Features by Laboratory\")\nplt.ylabel(\"Feature\")\nplt.xlabel(\"Lab ID\")\nplt.tight_layout()\nplt.show()\n\nnumeric_cols = ['frames_per_second', 'video_duration_sec', 'pix_per_cm_approx', \n                'video_width_pix', 'video_height_pix', 'arena_width_cm', 'arena_height_cm']\npca_data = train_df[numeric_cols].dropna()\n\nscaler = StandardScaler()\nscaled_data = scaler.fit_transform(pca_data)\n\npca = PCA(n_components=2)\nprincipal_components = pca.fit_transform(scaled_data)\n\npca_df = pd.DataFrame(data=principal_components, columns=['PC1', 'PC2'])\npca_df['lab_id'] = train_df.loc[pca_data.index, 'lab_id'].values\n\nplt.figure(figsize=(8,6))\nsns.scatterplot(data=pca_df, x='PC1', y='PC2', hue='lab_id', palette='Set2', s=60)\nplt.title(\"Principal Component Analysis of Technical Parameters\")\nplt.xlabel(\"PC1\")\nplt.ylabel(\"PC2\")\nplt.legend(title='Lab ID')\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-22T19:05:28.841725Z","iopub.execute_input":"2025-09-22T19:05:28.842098Z","iopub.status.idle":"2025-09-22T19:05:30.819682Z","shell.execute_reply.started":"2025-09-22T19:05:28.842076Z","shell.execute_reply":"2025-09-22T19:05:30.818707Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\n\npairplot_vars = ['frames_per_second', 'video_duration_sec', 'pix_per_cm_approx', 'arena_width_cm', 'arena_height_cm']\npairplot_data = train_df[pairplot_vars + ['lab_id']].dropna()\n\nsns.pairplot(pairplot_data, vars=pairplot_vars, hue='lab_id', corner=False, plot_kws={'alpha':0.6, 's':40})\nplt.suptitle(\"Relationships Between Technical Variables\", y=1.02)\nplt.show()\n\nsunburst_data = train_df.groupby(['lab_id', 'arena_shape', 'arena_type']).size().reset_index(name='count')\n\n\nsunburst_pivot = sunburst_data.pivot_table(index=['lab_id', 'arena_shape'], columns='arena_type', values='count', fill_value=0)\n\nsunburst_pivot.plot(kind='bar', stacked=True, figsize=(12,6), colormap='tab20')\nplt.title(\"Data Hierarchy: Lab > Arena Shape > Arena Type\")\nplt.xlabel(\"(Lab ID, Arena Shape)\")\nplt.ylabel(\"Count\")\nplt.xticks(rotation=45, ha='right')\nplt.legend(title='Arena Type', bbox_to_anchor=(1.05,1))\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-22T19:06:13.060687Z","iopub.execute_input":"2025-09-22T19:06:13.061106Z","iopub.status.idle":"2025-09-22T19:06:37.117389Z","shell.execute_reply.started":"2025-09-22T19:06:13.061078Z","shell.execute_reply":"2025-09-22T19:06:37.116163Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\ndef generate_statistical_summary(df):\n    summary = pd.DataFrame({\n        'Variable': df.columns,\n        'Type': df.dtypes.values,\n        'Missing Values': df.isnull().sum().values,\n        '% Missing Values': (df.isnull().sum() / len(df) * 100).round(2).values,\n        'Unique Values': [df[col].nunique() for col in df.columns]\n    })\n    \n \n    numeric_stats = df.describe().T\n    summary = summary.merge(numeric_stats, how='left', left_on='Variable', right_index=True)\n    \n    return summary\n\n\nstatistical_summary = generate_statistical_summary(train_df)\nstatistical_summary.to_csv('statistical_summary.csv', index=False)\n\nprint(\"Statistical Summary of Numerical Variables:\")\nprint(train_df.describe())\n\n\nmissing_data = train_df.isnull().sum().sort_values(ascending=False)\nmissing_percentage = (train_df.isnull().sum() / train_df.shape[0] * 100).sort_values(ascending=False)\nmissing_df = pd.DataFrame({'Missing Values': missing_data, 'Percentage': missing_percentage.round(2)})\nprint(\"\\nMissing Values Analysis:\")\ndisplay(missing_df[missing_df['Missing Values'] > 0])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-22T19:04:44.546439Z","iopub.execute_input":"2025-09-22T19:04:44.546748Z","iopub.status.idle":"2025-09-22T19:04:44.687474Z","shell.execute_reply.started":"2025-09-22T19:04:44.546725Z","shell.execute_reply":"2025-09-22T19:04:44.686468Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}