{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59156,"databundleVersionId":13874099,"sourceType":"competition"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Notebook 1: The Observer's Log - A Deep Dive into the Data (EDA)\n\n**Overall Goal:** Before writing a single line of model code, our objective is to build a strong intuition for the MABe dataset. We will explore the structure of the data, visualize the mouse movements, and identify the core challenges of this competition, such as class imbalance and data variability. This deep understanding will guide all of our future feature engineering and modeling decisions.\n\n---\n\n# **Step 1: Setup and Metadata Exploration**\n\n**Goal:** Based on the file structure, the primary data is stored in efficient `.parquet` files located in separate folders for tracking and annotations. The `train.csv` and `test.csv` files likely serve as metadata indexes, providing a list of all video IDs and potentially other high-level information.\n\nOur first step is to load these metadata files to understand the scope of the dataset (how many training/test videos are there?) and how we can use them to access the individual data files.\n\n**Action:** Please run the code block below to import libraries and inspect the head and info of the `train.csv` and `test.csv` metadata files.","metadata":{}},{"cell_type":"code","source":"# Import necessary libraries\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\nfrom tqdm.auto import tqdm\nfrom matplotlib.animation import FuncAnimation\nfrom IPython.display import HTML","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:57:55.609688Z","iopub.execute_input":"2025-09-23T07:57:55.610008Z","iopub.status.idle":"2025-09-23T07:57:55.628867Z","shell.execute_reply.started":"2025-09-23T07:57:55.609984Z","shell.execute_reply":"2025-09-23T07:57:55.627968Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Set some display options for pandas for better readability\npd.set_option('display.max_columns', 100)\nsns.set_style('whitegrid')\n\n# Define the path to your data\nDATA_PATH = '/kaggle/input/MABe-mouse-behavior-detection/'\n\n# Load the metadata files\nprint(\"Loading train.csv (metadata)...\")\ndf_train_meta = pd.read_csv(DATA_PATH + 'train.csv')\n\nprint(\"Loading test.csv (metadata)...\")\ndf_test_meta = pd.read_csv(DATA_PATH + 'test.csv')\n\nprint(\"\\n--- Train Metadata ---\")\nprint(f\"Shape: {df_train_meta.shape}\")\ndisplay(df_train_meta.head())\n\nprint(\"\\n--- Test Metadata ---\")\nprint(f\"Shape: {df_test_meta.shape}\")\ndisplay(df_test_meta.head())","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:16:06.185763Z","iopub.execute_input":"2025-09-23T07:16:06.186026Z","iopub.status.idle":"2025-09-23T07:16:06.328821Z","shell.execute_reply.started":"2025-09-23T07:16:06.186007Z","shell.execute_reply":"2025-09-23T07:16:06.328112Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 1\n\n*   **Metadata Confirmed:** `train.csv` and `test.csv` are indeed high-level metadata files, not the raw tracking data. They serve as an index for the entire dataset.\n*   **Dataset Scale:** The training set is substantial, with **8,790 videos**. This means our methods need to be efficient.\n*   **Code Competition Structure:** The public test set is tiny (just **1 video**). This is a classic sign of a code competition where our submitted notebook will be re-run on a much larger, hidden test set. This emphasizes the need for a **generalizable solution**, not one that is overfit to this single test video.\n*   **Rich Metadata:** We have a treasure trove of information for each video:\n    *   `lab_id`: Tells us which lab the data came from. This is a crucial feature for handling data variability.\n    *   `mouse_...`: Details about the strain, sex, age, and condition of up to four mice.\n    *   `video_...` / `arena_...`: Technical details about the recording setup (FPS, resolution, arena size).\n*   **File Path Keys:** The `lab_id` and `video_id` columns are the keys we need to construct the paths to the actual `.parquet` data files.","metadata":{}},{"cell_type":"markdown","source":"# Step 2: Loading and Inspecting a Single Data Sample\n\n**Goal:** Now that we understand the map (the metadata), it's time to explore the territory (the actual data). We will select the very first video from our training metadata and load its corresponding tracking and annotation files. This will reveal the low-level data structure we'll be working with for our models.\n\n**Action:** The code below will:\n1.  Select the first video from `df_train_meta`.\n2.  Construct the file paths for its tracking and annotation data.\n3.  Load the two `.parquet` files into new pandas DataFrames.\n4.  Display the first few rows, shape, and column information for both the tracking and annotation data.","metadata":{}},{"cell_type":"code","source":"# Select the first video from the metadata as our sample\nsample_video_meta = df_train_meta.iloc[0]\nsample_lab_id = sample_video_meta['lab_id']\nsample_video_id = sample_video_meta['video_id']\n\nprint(f\"Loading sample video...\\n  Lab ID: {sample_lab_id}\\n  Video ID: {sample_video_id}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:20:25.158858Z","iopub.execute_input":"2025-09-23T07:20:25.159169Z","iopub.status.idle":"2025-09-23T07:20:25.166286Z","shell.execute_reply.started":"2025-09-23T07:20:25.159144Z","shell.execute_reply":"2025-09-23T07:20:25.165293Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Construct the file paths using the lab and video IDs\ntracking_path = os.path.join(DATA_PATH, 'train_tracking', sample_lab_id, f'{sample_video_id}.parquet')\nannotation_path = os.path.join(DATA_PATH, 'train_annotation', sample_lab_id, f'{sample_video_id}.parquet')\n\n# Load the actual data from the parquet files\ndf_tracking_sample = pd.read_parquet(tracking_path)\ndf_annot_sample = pd.read_parquet(annotation_path)\n\nprint(\"\\n--- Sample Tracking Data ---\")\nprint(f\"Shape: {df_tracking_sample.shape}\")\nprint(\"Info:\")\ndf_tracking_sample.info()\nprint(\"\\nFirst 5 rows:\")\ndisplay(df_tracking_sample.head())\n\nprint(\"\\n\\n--- Sample Annotation Data ---\")\nprint(f\"Shape: {df_annot_sample.shape}\")\nprint(\"Info:\")\ndf_annot_sample.info()\nprint(\"\\nFirst 5 rows:\")\ndisplay(df_annot_sample.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:20:36.67738Z","iopub.execute_input":"2025-09-23T07:20:36.678162Z","iopub.status.idle":"2025-09-23T07:20:37.238859Z","shell.execute_reply.started":"2025-09-23T07:20:36.678129Z","shell.execute_reply":"2025-09-23T07:20:37.238022Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 2\n\n*   **\"Long\" Data Format:** The tracking data is in a \"long\" or \"tidy\" format. This means every single row represents just **one bodypart** for **one mouse** in **one frame**. While this is efficient for storage, it's not immediately useful for machine learning, where we typically want to see all the information for a single frame in one row.\n*   **The Reshaping Challenge:** Our next major task will be to \"pivot\" or reshape this data. We need to transform it into a \"wide\" format where each row represents a single `video_frame`, and the columns contain all the x and y coordinates for all mice (e.g., `mouse1_head_x`, `mouse1_head_y`, `mouse2_head_x`, etc.).\n*   **Annotation Structure:** The annotation data is very clear. It defines discrete behavioral events with a start and end frame. We can see some actions are individual (where `agent_id` == `target_id`, like \"rear\") and some are social (where they are different, like \"avoid\").\n*   **Data Granularity:** For a video that is ~615 seconds long at 30 FPS (from the metadata), we'd expect around 18,450 frames. The tracking data has `1,087,658` rows. Dividing this by the number of bodyparts and mice should give us the frame count, confirming the data's structure.","metadata":{}},{"cell_type":"markdown","source":"# Step 3: Reshaping the Data for Analysis (Pivoting)\n\n**Goal:** To transform our \"long\" tracking data into a \"wide\" format. This is the most critical data manipulation step in our EDA. A wide format (one row per frame) will allow us to easily calculate features, visualize entire scenes, and feed the data into a model later.\n\n**Action:** The code below will:\n1.  First, list the unique bodyparts being tracked in this video to see what we're working with.\n2.  Use the `pivot` function in pandas to reshape the data.\n3.  We will pivot the `x` and `y` coordinates separately and then merge them together to create one comprehensive DataFrame for the sample video.\n4.  Display the head of the new, wide DataFrame to see the result.","metadata":{}},{"cell_type":"code","source":"# 1. See what bodyparts are available\nunique_bodyparts = df_tracking_sample['bodypart'].unique()\nprint(f\"Unique bodyparts tracked: {unique_bodyparts}\\n\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:34:18.759006Z","iopub.execute_input":"2025-09-23T07:34:18.759354Z","iopub.status.idle":"2025-09-23T07:34:18.83585Z","shell.execute_reply.started":"2025-09-23T07:34:18.759328Z","shell.execute_reply":"2025-09-23T07:34:18.834868Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 2. Pivot the table to get a \"wide\" format\n# We want one row per video_frame, and columns for each mouse's bodypart's x and y coordinates.\n\nprint(\"Pivoting data from long to wide format...\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:34:20.699329Z","iopub.execute_input":"2025-09-23T07:34:20.699675Z","iopub.status.idle":"2025-09-23T07:34:20.704473Z","shell.execute_reply.started":"2025-09-23T07:34:20.699647Z","shell.execute_reply":"2025-09-23T07:34:20.703465Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create a pivot for the 'x' coordinates\npivot_x = df_tracking_sample.pivot(\n    index='video_frame', \n    columns=['mouse_id', 'bodypart'], \n    values='x'\n)\n# Rename columns for clarity, e.g., (1, 'nose') -> 'mouse1_nose_x'\npivot_x.columns = [f\"mouse{m}_{bp}_x\" for m, bp in pivot_x.columns]\n\n\n# Create a pivot for the 'y' coordinates\npivot_y = df_tracking_sample.pivot(\n    index='video_frame', \n    columns=['mouse_id', 'bodypart'], \n    values='y'\n)\n# Rename columns for clarity\npivot_y.columns = [f\"mouse{m}_{bp}_y\" for m, bp in pivot_y.columns]\n\n\n# 3. Merge the x and y pivots into a single DataFrame\ndf_wide_sample = pd.concat([pivot_x, pivot_y], axis=1)\n\n# Sort columns alphabetically for consistent order\ndf_wide_sample = df_wide_sample.sort_index(axis=1)\n\n\nprint(\"Pivoting complete.\\n\")\nprint(\"--- Reshaped Wide DataFrame ---\")\nprint(f\"Shape: {df_wide_sample.shape}\")\ndisplay(df_wide_sample.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:34:34.435293Z","iopub.execute_input":"2025-09-23T07:34:34.435643Z","iopub.status.idle":"2025-09-23T07:34:35.256068Z","shell.execute_reply.started":"2025-09-23T07:34:34.435616Z","shell.execute_reply":"2025-09-23T07:34:35.254944Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 3\n\n*   **Pivoting is Key:** We have successfully transformed the data from a long, stacked format into a wide, intuitive format. The shape `(18451, 142)` tells us we have 18,451 frames of data, and 142 feature columns (a mix of x and y coordinates for all tracked bodyparts on all mice).\n*   **Missing Data (`NaN`):** Notice the presence of `NaN` (Not a Number) values. This is completely normal in tracking data. It means the tracking algorithm (e.g., DeepLabCut) was not confident enough to assign a coordinate for that bodypart in that specific frame. This could be due to one mouse blocking another (occlusion) or fast movements causing motion blur. We will need to keep this in mind when engineering features.\n*   **Complexity of Bodyparts:** The list of unique bodyparts shows a mix of standard anatomical points (`nose`, `ear_left`, `tail_base`) and some experiment-specific ones (`headpiece_...`). For general-purpose features, we'll focus on the standard anatomical points first.","metadata":{}},{"cell_type":"markdown","source":"# Step 4: Visualizing the Data - A Static Snapshot\n\n**Goal:** Before we animate the mice, let's make sure we can plot a single frame correctly. This helps us understand the coordinate system and see the posture of all mice at one moment in time.\n\n**Action:** We will write a function that takes a single frame's data (one row from our wide DataFrame) and plots the keypoints for each mouse. We'll connect some keypoints with lines to form a simple \"skeleton\" for better visualization.\n1.  Define a list of standard, anatomical bodyparts we want to focus on.\n2.  Define the connections between these parts to draw skeletons.\n3.  Create the plotting function.\n4.  Use the function to plot frame `1000` of our sample video.","metadata":{}},{"cell_type":"code","source":"# 1. Define the core bodyparts we want to visualize\n# We will ignore the 'headpiece' parts for this general visualization\nANATOMICAL_BODYPARTS = [\n    'nose', 'ear_left', 'ear_right', 'neck', 'body_center', \n    'lateral_left', 'lateral_right', 'tail_base'\n]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:38:26.509681Z","iopub.execute_input":"2025-09-23T07:38:26.509968Z","iopub.status.idle":"2025-09-23T07:38:26.514522Z","shell.execute_reply.started":"2025-09-23T07:38:26.509947Z","shell.execute_reply":"2025-09-23T07:38:26.513591Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 2. Define connections to draw a simple skeleton\n# Each tuple represents a line from one bodypart to another\nSKELETON_CONNECTIONS = [\n    ('nose', 'ear_left'), ('nose', 'ear_right'), ('ear_left', 'ear_right'),\n    ('nose', 'neck'), ('neck', 'body_center'),\n    ('body_center', 'lateral_left'), ('body_center', 'lateral_right'),\n    ('body_center', 'tail_base')\n]\n\n# Define a color for each mouse for consistent plotting\nMOUSE_COLORS = {1: 'blue', 2: 'orange', 3: 'green', 4: 'red'}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:38:26.930519Z","iopub.execute_input":"2025-09-23T07:38:26.930793Z","iopub.status.idle":"2025-09-23T07:38:26.935757Z","shell.execute_reply.started":"2025-09-23T07:38:26.930775Z","shell.execute_reply":"2025-09-23T07:38:26.934656Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 3. Create the plotting function\ndef plot_frame(frame_data):\n    \"\"\"Plots the skeletons of all mice for a single frame of data.\"\"\"\n    \n    plt.figure(figsize=(8, 8))\n    \n    # Iterate through each mouse\n    for mouse_id in range(1, 5): # Assumes up to 4 mice\n        \n        # Check if data for this mouse exists in the frame\n        if f'mouse{mouse_id}_nose_x' not in frame_data or pd.isna(frame_data[f'mouse{mouse_id}_nose_x']):\n            continue # Skip if this mouse isn't tracked in this frame\n            \n        # Plot the keypoints (bodyparts)\n        for part in ANATOMICAL_BODYPARTS:\n            col_x = f'mouse{mouse_id}_{part}_x'\n            col_y = f'mouse{mouse_id}_{part}_y'\n            if col_x in frame_data and col_y in frame_data:\n                plt.scatter(frame_data[col_x], frame_data[col_y], color=MOUSE_COLORS[mouse_id], label=f'Mouse {mouse_id}' if part == 'nose' else \"\")\n\n        # Plot the skeleton connections\n        for part1, part2 in SKELETON_CONNECTIONS:\n            col1_x, col1_y = f'mouse{mouse_id}_{part1}_x', f'mouse{mouse_id}_{part1}_y'\n            col2_x, col2_y = f'mouse{mouse_id}_{part2}_x', f'mouse{mouse_id}_{part2}_y'\n\n            # Check if both points for the line exist\n            if all(c in frame_data for c in [col1_x, col1_y, col2_x, col2_y]) and \\\n               pd.notna(frame_data[col1_x]) and pd.notna(frame_data[col2_x]):\n                \n                plt.plot([frame_data[col1_x], frame_data[col2_x]], \n                         [frame_data[col1_y], frame_data[col2_y]], \n                         color=MOUSE_COLORS[mouse_id], alpha=0.7)\n\n    plt.title(f\"Mouse Positions at Frame {frame_data.name}\")\n    plt.xlabel(\"X-coordinate\")\n    plt.ylabel(\"Y-coordinate\")\n    \n    # Invert the y-axis because image coordinates (0,0) are usually at the top-left\n    plt.gca().invert_yaxis()\n    plt.legend()\n    plt.axis('equal') # Ensure aspect ratio is maintained\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:38:27.382119Z","iopub.execute_input":"2025-09-23T07:38:27.382452Z","iopub.status.idle":"2025-09-23T07:38:27.391344Z","shell.execute_reply.started":"2025-09-23T07:38:27.382426Z","shell.execute_reply":"2025-09-23T07:38:27.39061Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 4. Use the function to plot a specific frame\nFRAME_TO_PLOT = 1000\nplot_frame(df_wide_sample.loc[FRAME_TO_PLOT])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:38:31.555288Z","iopub.execute_input":"2025-09-23T07:38:31.555571Z","iopub.status.idle":"2025-09-23T07:38:32.169879Z","shell.execute_reply.started":"2025-09-23T07:38:31.55555Z","shell.execute_reply":"2025-09-23T07:38:32.16881Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 4\n\n*   **Visualization Success:** Our plotting function works perfectly! We can now take any single frame of data and instantly visualize the entire scene.\n*   **Relative Positions are Clear:** We can see that at frame 1000, Mouse 3 (green) and Mouse 1 (blue) are relatively close, while Mouse 4 (red) is further away. This ability to see spatial relationships is the foundation for understanding social behavior.\n*   **Missing Mice Handled:** Notice that Mouse 2 is not plotted. Our code correctly handled the missing data for this mouse at this frame, which is essential for creating a robust visualization tool.\n*   **Static is Not Enough:** A single frame shows posture, but behavior is defined by **movement through time**. To truly understand what's happening, we need to see the sequence of these frames.","metadata":{}},{"cell_type":"markdown","source":"# Step 5: The Dynamic View - Animating a Behavior\n\n**Goal:** This is the most intuitive part of our EDA. We will create an animation—a mini-movie—of the mice. This will allow us to see how their positions and postures change over time, giving us a true feel for their behavior.\n\n**Action:**\n1.  We will pick the first labeled behavior from our `df_annot_sample` DataFrame.\n2.  We'll extract the `start_frame` and `stop_frame` for that behavior.\n3.  We will create an animation of the mouse movements during that specific time window.\n4.  We will display the animation directly in the notebook.\n\nThis will be our first look at a specific, labeled action as it actually happened.","metadata":{}},{"cell_type":"code","source":"# 1. Pick a behavior to animate from our annotation sample\nbehavior_to_animate = df_annot_sample.iloc[0]\nstart_frame = behavior_to_animate['start_frame']\nstop_frame = behavior_to_animate['stop_frame']\naction = behavior_to_animate['action']\nagent = behavior_to_animate['agent_id']\n\nprint(f\"Preparing to animate behavior: '{action}' by Mouse {agent}\")\nprint(f\"Frame range: {start_frame} to {stop_frame}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:46:46.755778Z","iopub.execute_input":"2025-09-23T07:46:46.75605Z","iopub.status.idle":"2025-09-23T07:46:46.762418Z","shell.execute_reply.started":"2025-09-23T07:46:46.756032Z","shell.execute_reply":"2025-09-23T07:46:46.761379Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Add a small buffer before and after to see the context\nANIM_START = max(0, start_frame - 20)\nANIM_STOP = stop_frame + 20\n\n# Slice our wide dataframe to get only the frames we need for the animation\nanim_df = df_wide_sample.loc[ANIM_START:ANIM_STOP]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:47:26.688081Z","iopub.execute_input":"2025-09-23T07:47:26.688423Z","iopub.status.idle":"2025-09-23T07:47:26.693483Z","shell.execute_reply.started":"2025-09-23T07:47:26.688395Z","shell.execute_reply":"2025-09-23T07:47:26.692663Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# --- Animation Setup ---\n\n# Set up the figure and axis\nfig, ax = plt.subplots(figsize=(8, 8))\n\n# Determine axis limits from the entire animation sequence to prevent jittering\nx_min, x_max = anim_df.filter(like='_x').min().min(), anim_df.filter(like='_x').max().max()\ny_min, y_max = anim_df.filter(like='_y').min().min(), anim_df.filter(like='_y').max().max()\npadding = 50 # Add some padding to the plot\nax.set_xlim(x_min - padding, x_max + padding)\nax.set_ylim(y_min - padding, y_max + padding)\n\n\n# The function that will draw each frame of the animation\ndef update(frame_num):\n    ax.clear() # Clear the previous frame\n    \n    # Get the data for the current frame\n    frame_data = anim_df.iloc[frame_num]\n    current_real_frame = anim_df.index[frame_num]\n    \n    # Plot each mouse for the current frame\n    for mouse_id in range(1, 5):\n        if f'mouse{mouse_id}_nose_x' not in frame_data or pd.isna(frame_data[f'mouse{mouse_id}_nose_x']):\n            continue\n\n        # Plot keypoints\n        for part in ANATOMICAL_BODYPARTS:\n            col_x, col_y = f'mouse{mouse_id}_{part}_x', f'mouse{mouse_id}_{part}_y'\n            if col_x in frame_data and col_y in frame_data:\n                ax.scatter(frame_data[col_x], frame_data[col_y], color=MOUSE_COLORS[mouse_id])\n\n        # Plot skeleton\n        for part1, part2 in SKELETON_CONNECTIONS:\n            col1_x, col1_y = f'mouse{mouse_id}_{part1}_x', f'mouse{mouse_id}_{part1}_y'\n            col2_x, col2_y = f'mouse{mouse_id}_{part2}_x', f'mouse{mouse_id}_{part2}_y'\n            if all(c in frame_data for c in [col1_x, col1_y, col2_x, col2_y]) and \\\n               pd.notna(frame_data[col1_x]) and pd.notna(frame_data[col2_x]):\n                ax.plot([frame_data[col1_x], frame_data[col2_x]], [frame_data[col1_y], frame_data[col2_y]], color=MOUSE_COLORS[mouse_id], alpha=0.7)\n\n    # Set titles and labels for the frame\n    ax.set_title(f\"Behavior: '{action}' by Mouse {agent} | Frame: {current_real_frame}\")\n    ax.set_xlabel(\"X-coordinate\")\n    ax.set_ylabel(\"Y-coordinate\")\n    ax.set_xlim(x_min - padding, x_max + padding)\n    ax.set_ylim(y_min - padding, y_max + padding)\n    ax.invert_yaxis() # Invert y-axis for image coordinates\n    return ax,","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:47:27.971913Z","iopub.execute_input":"2025-09-23T07:47:27.972198Z","iopub.status.idle":"2025-09-23T07:47:28.195351Z","shell.execute_reply.started":"2025-09-23T07:47:27.972176Z","shell.execute_reply":"2025-09-23T07:47:28.194427Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create the animation\n# frames=len(anim_df) specifies how many times to call the update function\n# interval=50 is the delay between frames in milliseconds\nani = FuncAnimation(fig, update, frames=len(anim_df), interval=50, blit=False)\n\n# Display the animation in the notebook\n# This may take a little while to render\nHTML(ani.to_jshtml())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:47:35.719896Z","iopub.execute_input":"2025-09-23T07:47:35.720876Z","iopub.status.idle":"2025-09-23T07:48:31.385524Z","shell.execute_reply.started":"2025-09-23T07:47:35.720846Z","shell.execute_reply":"2025-09-23T07:48:31.384274Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 5\n\n*   **Behavior is Motion:** The animation makes it crystal clear that behaviors are not static poses but dynamic sequences of movements. For the \"rear\" behavior, you likely saw a mouse lift its upper body, stay in that position for a few frames, and then lower itself.\n*   **Context is Key:** By adding a buffer before and after the labeled event, we can see the transitions into and out of a behavior. This is critical information that a sequence model can learn.\n*   **The Power of Visualization:** We now have a powerful tool to debug our future models. If our model incorrectly labels a segment, we can create an animation of that segment to try and understand *why* it made a mistake. Was it a subtle movement? Was there an occlusion?\n*   **From Deep to Wide:** We have now performed a \"deep dive\" on a single video. The next step is to \"zoom out\" and analyze the characteristics of the entire training dataset to understand the big picture.","metadata":{}},{"cell_type":"markdown","source":"# Step 6: Dataset-Wide Analysis - Behavior Distribution & Duration\n\n**Goal:** To understand the overall properties of the behaviors we need to predict. We will now use the complete `annotations.csv` file (which we loaded as `df_annotations` in Step 1) to answer critical questions:\n1.  **What are all the different behaviors?**\n2.  **How often does each behavior occur (Class Balance)?** This is one of the most important questions. A heavy imbalance will significantly influence our model training and evaluation strategy.\n3.  **How long do behaviors typically last (Duration)?** Are some behaviors very short (a quick sniff) while others are very long (huddling)?\n\n**Action:** We will create plots to visualize the frequency and duration of every behavior across the entire training set.","metadata":{}},{"cell_type":"code","source":"# This cell is designed to be self-contained. It will build the full annotation\n# dataframe if it doesn't already exist in memory.\nif 'df_annotations_full' not in locals():\n    print(\"Building the full annotations dataframe by combining all individual annotation files...\")\n\n    all_annotations_list = []\n    for index, row in tqdm(df_train_meta.iterrows(), total=df_train_meta.shape[0]):\n        lab_id = row['lab_id']\n        video_id = row['video_id']\n        annotation_path = os.path.join(DATA_PATH, 'train_annotation', lab_id, f'{video_id}.parquet')\n        \n        if os.path.exists(annotation_path):\n            temp_df = pd.read_parquet(annotation_path)\n            temp_df['video_id'] = video_id\n            all_annotations_list.append(temp_df)\n\n    df_annotations_full = pd.concat(all_annotations_list, ignore_index=True)\n    print(f\"\\nSuccessfully created full annotation dataframe with shape: {df_annotations_full.shape}\")\nelse:\n    print(\"Full annotation dataframe already exists in memory. Proceeding with analysis.\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T08:08:46.536936Z","iopub.execute_input":"2025-09-23T08:08:46.537682Z","iopub.status.idle":"2025-09-23T08:08:46.544705Z","shell.execute_reply.started":"2025-09-23T08:08:46.537655Z","shell.execute_reply":"2025-09-23T08:08:46.543658Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# --- Behavior Frequency Analysis ---\nprint(\"\\n--- Behavior Frequency Analysis ---\")\n\nbehavior_counts = df_annotations_full['action'].value_counts()\nplt.figure(figsize=(12, 8))\nsns.barplot(x=behavior_counts.index, y=behavior_counts.values, palette='viridis')\nplt.title('Frequency of Each Behavior Across the Entire Training Set', fontsize=16)\nplt.xlabel('Behavior', fontsize=12)\nplt.ylabel('Count', fontsize=12)\nplt.xticks(rotation=45, ha='right')\nplt.tight_layout()\nplt.show()\n\n# --- Behavior Duration Analysis ---\nprint(\"\\n--- Behavior Duration Analysis ---\")\ndf_annotations_full['duration_frames'] = df_annotations_full['stop_frame'] - df_annotations_full['start_frame']\n\n# Let's see how many zero-duration events we have\nzero_duration_count = (df_annotations_full['duration_frames'] == 0).sum()\nprint(f\"Found {zero_duration_count} events with a duration of 0 frames.\")\n\nprint(\"\\nBasic statistics for behavior durations (in frames):\")\ndisplay(df_annotations_full['duration_frames'].describe())\n\n# Add 1 to duration before plotting on a log scale to handle zeros\nplt.figure(figsize=(12, 6))\nsns.histplot(df_annotations_full['duration_frames'] + 1, bins=100, log_scale=True)\nplt.title('Distribution of Behavior Durations (Log Scale, Duration+1)', fontsize=16)\nplt.xlabel('Duration (Frames) + 1 - Log Scale', fontsize=12)\nplt.ylabel('Frequency', fontsize=12)\nplt.show()\n\nplt.figure(figsize=(12, 10))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T08:09:04.585931Z","iopub.execute_input":"2025-09-23T08:09:04.586285Z","iopub.status.idle":"2025-09-23T08:09:05.998808Z","shell.execute_reply.started":"2025-09-23T08:09:04.586212Z","shell.execute_reply":"2025-09-23T08:09:05.997986Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\norder = df_annotations_full.groupby('action')['duration_frames'].median().sort_values(ascending=False).index\n# Use duration + 1 for the x-axis in the boxplot as well\nsns.boxplot(x=df_annotations_full['duration_frames'] + 1, y='action', data=df_annotations_full, order=order, palette='coolwarm')\nplt.title('Duration of Each Behavior Type', fontsize=16)\nplt.xlabel('Duration (Frames) + 1 - Log Scale', fontsize=12)\nplt.ylabel('Behavior', fontsize=12)\nplt.xscale('log')\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T08:09:07.636062Z","iopub.execute_input":"2025-09-23T08:09:07.636415Z","iopub.status.idle":"2025-09-23T08:09:08.96086Z","shell.execute_reply.started":"2025-09-23T08:09:07.636391Z","shell.execute_reply":"2025-09-23T08:09:08.960018Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 6\n\nThe dataset-wide analysis has revealed the most critical challenges of this competition.\n\n**From the Frequency Plot (Bar Chart):**\n\n*   **Extreme Class Imbalance:** This is the #1 challenge. The behavior `sniff` occurs nearly 40,000 times, while `ejaculate` and `biteobject` are at the far end, likely with only a few dozen occurrences.\n*   **Modeling Implication:** A standard model will become an expert at predicting `sniff` and `attack` but will completely ignore the rare classes because it can achieve high accuracy by just focusing on the majority. We **must** use special techniques to handle this, such as:\n    *   Using an appropriate evaluation metric that cares about rare classes (like the competition's F-Score variant).\n    *   Applying class weights during training to penalize the model more for misclassifying rare behaviors.\n    *   Using special sampling techniques (e.g., oversampling rare classes).\n\n**From the Duration Distribution Plot (Histogram):**\n\n*   **Bimodal Distribution:** The histogram isn't a simple bell curve. It has two \"humps\" or modes. There's a large peak for short-duration behaviors (around 10-30 frames) and another, wider peak for longer behaviors (around 30-200 frames).\n*   **Modeling Implication:** This suggests there isn't one \"typical\" behavior length. Our model needs to be flexible enough to recognize both very brief events and long, sustained actions. The sequence length we choose for our models (e.g., LSTMs, Transformers) will be an important hyperparameter.\n\n**From the Duration by Type Plot (Box Plot):**\n\n*   **Behaviors Have Characteristic Durations:** This plot is incredibly useful. We can clearly see that behaviors like `flinch` and `sniffface` are almost always very short (median duration is less than 10 frames). In contrast, behaviors like `rest`, `intromit`, and `ejaculate` are typically very long.\n*   **Feature Engineering Idea:** The duration of an action is itself a powerful predictive feature! While we don't know the duration in advance, our model might learn that if a certain type of interaction has been happening for 100 frames, it's more likely to be a `rest` than a `flinch`.\n*   **High Variance:** Notice the long tails and many outlier points (the black diamonds) for almost every behavior. This means that while `attack` has a *typical* duration, it can sometimes be very short or drag on for a very long time. Our model must be robust to this variability.","metadata":{}},{"cell_type":"markdown","source":"# Step 7: The Lab Effect - Analyzing Data Variability\n\n**Goal:** The competition description explicitly mentions the challenge of generalizing across data from different labs, which may use different equipment and tracking methods. Our final EDA step is to investigate this \"lab effect.\"\n\n**Action:** We will join our full annotation data with the original metadata to get the `lab_id` for each behavior. Then, we will create a plot to see if the distribution of behaviors is different from one lab to another. If it is, this confirms that using `lab_id` as a feature or for our cross-validation strategy will be crucial.","metadata":{}},{"cell_type":"code","source":"# First, ensure 'df_annotations_full' exists\nif 'df_annotations_full' not in locals():\n    print(\"Full annotation dataframe is not in memory. Please re-run the previous cell (Step 6).\")\nelse:\n    print(\"--- Lab Variability Analysis ---\")\n    \n    # We need to merge our annotations with the metadata to get the lab_id for each event\n    # We select only the 'video_id' and 'lab_id' from the metadata to keep the merge light\n    df_lab_info = df_train_meta[['video_id', 'lab_id']]\n    \n    # Perform a left merge to add 'lab_id' to each annotation\n    df_annotations_with_lab = pd.merge(df_annotations_full, df_lab_info, on='video_id', how='left')\n    \n    print(f\"Successfully merged lab info. New shape: {df_annotations_with_lab.shape}\")\n    display(df_annotations_with_lab.head())\n    \n    # Now, let's plot the behavior counts per lab\n    plt.figure(figsize=(15, 8))\n    \n    # We use crosstab to count occurrences of each action within each lab, then normalize\n    # to see the percentage/proportion, which is better for comparison\n    crosstab_norm = pd.crosstab(df_annotations_with_lab['lab_id'], \n                               df_annotations_with_lab['action'], \n                               normalize='index') # 'normalize=index' calculates percentages per lab\n    \n    sns.heatmap(crosstab_norm, cmap='viridis', annot=False) # 'annot=True' can be messy if too many classes\n    plt.title('Proportion of Behaviors by Lab', fontsize=16)\n    plt.xlabel('Behavior', fontsize=12)\n    plt.ylabel('Lab ID', fontsize=12)\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T08:20:48.658421Z","iopub.execute_input":"2025-09-23T08:20:48.659165Z","iopub.status.idle":"2025-09-23T08:20:49.484093Z","shell.execute_reply.started":"2025-09-23T08:20:48.65914Z","shell.execute_reply":"2025-09-23T08:20:49.483247Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 7\n\nThe heatmap provides definitive evidence of the \"lab effect\" and is perhaps the most important visualization for designing a winning strategy.\n\n*   **The 'Lab Effect' is Real and Severe:** The plot is not uniform at all. It's very \"blocky.\" This shows that the types of behaviors and how often they occur are drastically different from one lab to the next. The bright yellow squares indicate that for a specific lab, a single behavior can make up a huge proportion of all its labeled events.\n\n*   **Lab Specialization:** Look at the bright yellow square for lab `BoisterousParrot` under the behavior `sniffbody`. This means a massive percentage of all behaviors labeled in that lab's data are `sniffbody`. Similarly, `CRIM13`'s data seems to be overwhelmingly focused on the `run` behavior. These labs were likely designed to study these specific actions.\n\n*   **Rare Behaviors are Lab-Specific:** Some behaviors might only appear in data from one or two labs. If a model learns to recognize a rare behavior, it might accidentally learn features of that specific lab's camera setup (e.g., lighting, arena color) instead of the true features of the mouse behavior.\n\n*   **The Critical Takeaway - Validation Strategy:** This plot tells us that a simple random validation split is **the wrong approach** and will be misleading. If we randomly sprinkle data from all labs into our training and validation sets, our model will get an artificially high score because it learns the quirks of each lab. To build a model that truly generalizes, our validation strategy **must** simulate the challenge of seeing a new, unseen lab. The correct approach is to use **GroupKFold cross-validation**, with `lab_id` as the grouping variable. This ensures that all data from one lab is either in the training set or the validation set, but never both.","metadata":{}},{"cell_type":"markdown","source":"# Notebook 1 Conclusion: The Story of the Data\n\nThis concludes our deep-dive Exploratory Data Analysis. We have gone from raw, disconnected files to a deep, intuitive understanding of the MABe dataset. We didn't just look at the data; we visualized it, animated it, and uncovered its deepest challenges.\n\n### Our Key Findings and Action Plan:\n\n1.  **Data Structure:** The data is stored efficiently in a long format across thousands of Parquet files. Our first challenge was to create a robust pipeline to load and reshape this data into a usable \"wide\" format (one row per frame), which we have successfully done.\n\n2.  **Extreme Class Imbalance:** We discovered that some behaviors (like `sniff`) are thousands of times more common than others (like `ejaculate`).\n    *   **Action Plan:** We must use techniques like class weighting or special sampling methods and focus on metrics that value rare classes during modeling.\n\n3.  **Variable Behavior Durations:** Behaviors can last from a few frames to thousands.\n    *   **Action Plan:** This confirms that sequence-based models (LSTMs, Transformers) that can handle variable-length patterns will be essential.\n\n4.  **The Lab Generalization Problem:** The distribution of behaviors varies significantly between labs.\n    *   **Action Plan:** Our validation strategy must be built around `GroupKFold` using `lab_id` to ensure we are building a model that generalizes to unseen experimental setups.\n\nWe are now perfectly equipped to move on to the next stage. We understand the problem, we know the pitfalls, and we have a clear action plan.\n\n**Next Up: Notebook 2 - The First Hypothesis: A Simple Frame-by-Frame Baseline**","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Notebook 1 Extended: Advanced EDA - Uncovering Hidden Patterns\n\n**Goal:** Now that we understand the basic structure and challenges of the dataset, we need to dig deeper. This extended EDA will focus on:\n\n1. **Missing Data Patterns** - Understanding tracking quality and failure modes\n2. **Spatial Dynamics** - Mouse distances, velocities, and movement patterns\n3. **Temporal Patterns** - When behaviors occur within videos\n4. **Behavior Transitions** - What happens before and after each behavior\n5. **Multi-Mouse Interactions** - Social vs. individual behavior patterns\n6. **Tracking Quality by Lab** - Understanding data quality variations\n7. **Feature Correlation Analysis** - Which raw features are most informative\n\nThese insights will directly inform our feature engineering and model architecture choices.\n\n---\n\n# **Step 8: Missing Data Deep Dive**\n\n**Goal:** NaN values in tracking data represent tracking failures. Understanding *when* and *where* these failures occur is critical because:\n- They might correlate with specific behaviors (e.g., fast movements, occlusions)\n- Different labs might have different failure rates\n- We need strategies to handle them in our models\n\n**Action:** Analyze missing data patterns across bodyparts, labs, and behaviors.","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"# Calculate missing data statistics for our sample video\nprint(\"=== Missing Data Analysis for Sample Video ===\\n\")\n\n# Calculate percentage of missing values per column\nmissing_pct = (df_tracking_sample.isna().sum() / len(df_tracking_sample)) * 100\nmissing_by_bodypart = missing_pct.groupby(df_tracking_sample.columns.str.extract(r'(\\w+)_[xy]$')[0]).mean()\n\nprint(\"Missing data percentage by bodypart:\")\ndisplay(missing_by_bodypart.sort_values(ascending=False))\n\n# Visualize missing data pattern\nplt.figure(figsize=(14, 6))\nmissing_pct_wide = (df_wide_sample.isna().sum() / len(df_wide_sample)) * 100\nmissing_pct_wide.sort_values(ascending=False).head(20).plot(kind='barh', color='coral')\nplt.xlabel('Percentage Missing (%)', fontsize=12)\nplt.ylabel('Feature (Mouse_Bodypart_Coordinate)', fontsize=12)\nplt.title('Top 20 Features with Most Missing Data', fontsize=14)\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Analyze missing data patterns across ALL videos and labs","metadata":{}},{"cell_type":"code","source":"\nprint(\"=== Dataset-Wide Missing Data Analysis ===\\n\")\nprint(\"This will sample 100 random videos to analyze tracking quality...\\n\")\n\n# Sample videos for efficiency\nsample_size = min(100, len(df_train_meta))\nsampled_videos = df_train_meta.sample(n=sample_size, random_state=42)\n\nlab_missing_data = []\n\nfor idx, row in tqdm(sampled_videos.iterrows(), total=sample_size, desc=\"Analyzing videos\"):\n    lab_id = row['lab_id']\n    video_id = row['video_id']\n    tracking_path = os.path.join(DATA_PATH, 'train_tracking', lab_id, f'{video_id}.parquet')\n    \n    if os.path.exists(tracking_path):\n        df_track = pd.read_parquet(tracking_path)\n        missing_pct = (df_track[['x', 'y']].isna().sum().sum() / (len(df_track) * 2)) * 100\n        \n        lab_missing_data.append({\n            'lab_id': lab_id,\n            'video_id': video_id,\n            'missing_pct': missing_pct\n        })\n\ndf_missing = pd.DataFrame(lab_missing_data)\n\n# Plot missing data by lab\nplt.figure(figsize=(12, 6))\nsns.boxplot(data=df_missing, x='lab_id', y='missing_pct', palette='Set2')\nplt.xticks(rotation=45, ha='right')\nplt.ylabel('Missing Data Percentage (%)', fontsize=12)\nplt.xlabel('Lab ID', fontsize=12)\nplt.title('Tracking Quality (Missing Data %) by Lab', fontsize=14)\nplt.tight_layout()\nplt.show()\n\nprint(f\"\\nOverall missing data statistics:\")\nprint(df_missing['missing_pct'].describe())","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 8\n\n*   **Tracking Quality Varies by Bodypart:** Some bodyparts (like ears or tail tips) are consistently harder to track than others (like body center or nose). This is expected as smaller, faster-moving parts are more challenging.\n*   **Lab-Specific Tracking Quality:** Different labs have dramatically different tracking quality. Some labs have near-perfect tracking (<5% missing), while others have 20%+ missing data. This could be due to:\n    - Different camera quality or frame rates\n    - Different lighting conditions\n    - Different tracking algorithms (DeepLabCut vs. SLEAP, etc.)\n    - Different mouse strains (some fur colors are harder to track)\n*   **Modeling Implication:** We need robust imputation strategies. Simple approaches:\n    - Forward-fill or interpolate for short gaps\n    - Use only well-tracked bodyparts for initial models\n    - Create a \"tracking confidence\" feature that the model can learn to use\n\n---\n\n# **Step 9: Spatial Dynamics - Distances and Velocities**\n\n**Goal:** Social behaviors are fundamentally about spatial relationships. When mice \"sniff\" each other, they're close. When they \"avoid,\" they move apart. Let's engineer some basic spatial features and see if they correlate with behaviors.\n\n**Action:** Calculate inter-mouse distances, velocities, and accelerations for our sample video.","metadata":{}},{"cell_type":"code","source":"# Calculate distance between all pairs of mice for the sample video\nprint(\"=== Calculating Spatial Features ===\\n\")\n\ndef calculate_distance(x1, y1, x2, y2):\n    \"\"\"Calculate Euclidean distance between two points.\"\"\"\n    return np.sqrt((x2 - x1)**2 + (y2 - y1)**2)\n\ndef calculate_velocity(df, mouse_id, bodypart='nose', fps=30):\n    \"\"\"Calculate velocity for a specific mouse and bodypart.\"\"\"\n    x_col = f'mouse{mouse_id}_{bodypart}_x'\n    y_col = f'mouse{mouse_id}_{bodypart}_y'\n    \n    if x_col not in df.columns or y_col not in df.columns:\n        return None\n    \n    # Calculate displacement between consecutive frames\n    dx = df[x_col].diff()\n    dy = df[y_col].diff()\n    \n    # Calculate velocity (pixels per second)\n    velocity = np.sqrt(dx**2 + dy**2) * fps\n    \n    return velocity\n\n# Calculate distances between all mouse pairs (using nose positions)\nmouse_pairs = [(1, 2), (1, 3), (1, 4), (2, 3), (2, 4), (3, 4)]\n\nfor m1, m2 in mouse_pairs:\n    x1_col = f'mouse{m1}_nose_x'\n    y1_col = f'mouse{m1}_nose_y'\n    x2_col = f'mouse{m2}_nose_x'\n    y2_col = f'mouse{m2}_nose_y'\n    \n    if all(col in df_wide_sample.columns for col in [x1_col, y1_col, x2_col, y2_col]):\n        df_wide_sample[f'dist_mouse{m1}_mouse{m2}'] = calculate_distance(\n            df_wide_sample[x1_col], df_wide_sample[y1_col],\n            df_wide_sample[x2_col], df_wide_sample[y2_col]\n        )\n\n# Calculate velocities for each mouse\nfps = sample_video_meta['frames_per_second']\nfor mouse_id in range(1, 5):\n    vel = calculate_velocity(df_wide_sample, mouse_id, 'nose', fps)\n    if vel is not None:\n        df_wide_sample[f'mouse{mouse_id}_velocity'] = vel\n\nprint(\"Spatial features calculated successfully!\")\nprint(f\"\\nNew feature columns: {[col for col in df_wide_sample.columns if 'dist_' in col or 'velocity' in col]}\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Visualize distance patterns during different behaviors\nprint(\"=== Visualizing Spatial Patterns During Behaviors ===\\n\")\n\n# Get a few different behavior examples from our sample\nsample_behaviors = df_annot_sample.head(5)\n\nfig, axes = plt.subplots(len(sample_behaviors), 1, figsize=(14, 4*len(sample_behaviors)))\nif len(sample_behaviors) == 1:\n    axes = [axes]\n\nfor idx, (_, behavior) in enumerate(sample_behaviors.iterrows()):\n    start = behavior['start_frame']\n    stop = behavior['stop_frame']\n    action = behavior['action']\n    agent = behavior['agent_id']\n    target = behavior['target_id']\n    \n    # Add buffer\n    plot_start = max(0, start - 50)\n    plot_stop = min(len(df_wide_sample), stop + 50)\n    \n    # Get relevant distance column\n    dist_col = f'dist_mouse{agent}_mouse{target}'\n    \n    ax = axes[idx]\n    \n    if dist_col in df_wide_sample.columns:\n        # Plot distance over time\n        frames = range(plot_start, plot_stop)\n        distances = df_wide_sample.loc[plot_start:plot_stop-1, dist_col]\n        \n        ax.plot(frames, distances, label=f'Distance M{agent}-M{target}', linewidth=2)\n        \n        # Highlight the behavior period\n        ax.axvspan(start, stop, alpha=0.3, color='red', label=f'Behavior: {action}')\n        \n        ax.set_xlabel('Frame', fontsize=11)\n        ax.set_ylabel('Distance (pixels)', fontsize=11)\n        ax.set_title(f'Distance Pattern: Mouse {agent} → {action} → Mouse {target}', fontsize=12)\n        ax.legend()\n        ax.grid(True, alpha=0.3)\n    else:\n        ax.text(0.5, 0.5, f'Distance data not available for {action}', \n                ha='center', va='center', transform=ax.transAxes)\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 9\n\n*   **Distance Correlates with Behavior:** For social behaviors like \"sniff\" or \"mount,\" we can clearly see that the distance between the agent and target mouse decreases dramatically during the behavior and increases before/after. This is a strong signal!\n*   **Velocity Patterns:** Fast movements likely correlate with behaviors like \"chase\" or \"flee,\" while slow or zero velocity might indicate \"rest\" or \"huddle.\"\n*   **Feature Engineering Goldmine:** These spatial features are likely to be some of our most powerful predictors:\n    - Inter-mouse distances (nose-to-nose, nose-to-tail, etc.)\n    - Velocities and accelerations\n    - Relative angles and orientations\n    - Distance to arena walls\n*   **Next Level:** We could calculate even more sophisticated features like:\n    - Angular relationships (is one mouse behind another?)\n    - Relative velocities (are they moving toward or away from each other?)\n    - Body orientation alignment\n\n---\n\n# **Step 10: Temporal Patterns - When Do Behaviors Occur?**\n\n**Goal:** Do certain behaviors tend to happen at the beginning, middle, or end of videos? Understanding temporal patterns can help us:\n- Design better data augmentation strategies\n- Understand if there are \"warm-up\" or \"cool-down\" periods in experiments\n- Detect potential annotation artifacts\n\n**Action:** Analyze when each behavior type occurs relative to video length.","metadata":{}},{"cell_type":"code","source":"# Calculate relative timing of behaviors within videos\nprint(\"=== Temporal Pattern Analysis ===\\n\")\n\n# Merge annotations with metadata to get video durations\ndf_annot_temporal = pd.merge(\n    df_annotations_full, \n    df_train_meta[['video_id', 'video_duration_sec', 'frames_per_second']], \n    on='video_id', \n    how='left'\n)\n\n# Calculate the relative position of each behavior within its video (0 = start, 1 = end)\ndf_annot_temporal['total_frames'] = df_annot_temporal['video_duration_sec'] * df_annot_temporal['frames_per_second']\ndf_annot_temporal['behavior_midpoint'] = (df_annot_temporal['start_frame'] + df_annot_temporal['stop_frame']) / 2\ndf_annot_temporal['relative_position'] = df_annot_temporal['behavior_midpoint'] / df_annot_temporal['total_frames']\n\n# Ensure relative position is between 0 and 1\ndf_annot_temporal['relative_position'] = df_annot_temporal['relative_position'].clip(0, 1)\n\nprint(f\"Temporal features calculated for {len(df_annot_temporal)} behaviors\\n\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Visualize when different behaviors occur in videos\nplt.figure(figsize=(14, 10))\n\n# Get top 15 most common behaviors for readability\ntop_behaviors = df_annot_temporal['action'].value_counts().head(15).index\n\ndf_plot = df_annot_temporal[df_annot_temporal['action'].isin(top_behaviors)]\n\nsns.violinplot(\n    data=df_plot, \n    y='action', \n    x='relative_position',\n    order=top_behaviors,\n    palette='coolwarm',\n    inner='box'\n)\n\nplt.axvline(x=0.5, color='black', linestyle='--', alpha=0.5, label='Video Midpoint')\nplt.xlabel('Relative Position in Video (0=Start, 1=End)', fontsize=12)\nplt.ylabel('Behavior', fontsize=12)\nplt.title('When Do Behaviors Occur? (Distribution Across Video Timeline)', fontsize=14)\nplt.legend()\nplt.tight_layout()\nplt.show()\n\n# Statistical summary\nprint(\"\\nBehaviors occurring predominantly in first half of videos:\")\nearly_behaviors = df_annot_temporal.groupby('action')['relative_position'].median().sort_values().head(5)\nprint(early_behaviors)\n\nprint(\"\\nBehaviors occurring predominantly in second half of videos:\")\nlate_behaviors = df_annot_temporal.groupby('action')['relative_position'].median().sort_values(ascending=False).head(5)\nprint(late_behaviors)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 10\n\n*   **Most Behaviors Are Uniform:** The violin plots show that most behaviors occur relatively uniformly throughout videos, which is good - it suggests experiments were well-designed and annotations are consistent.\n*   **Potential Edge Effects:** If we see behaviors concentrated at the very start or end, it could indicate:\n    - Natural behavioral patterns (e.g., exploration at start, fatigue at end)\n    - Annotation artifacts (annotators might focus more on certain periods)\n    - Experimental protocol effects\n*   **Modeling Implication:** The relative timestamp could be a useful feature, especially combined with video-level metadata (e.g., time of day, experimental condition).\n\n---\n\n# **Step 11: Behavior Transitions - The Sequence Story**\n\n**Goal:** Behaviors don't occur in isolation. A \"sniff\" might lead to an \"attack,\" or a \"chase\" might end in \"rest.\" Understanding these transition patterns can:\n- Inform sequence model architecture\n- Help with data augmentation\n- Reveal biological insights about mouse behavior\n\n**Action:** Build a transition matrix showing which behaviors commonly follow which others.","metadata":{}},{"cell_type":"code","source":"# Analyze behavior transitions (what comes after what)\nprint(\"=== Behavior Transition Analysis ===\\n\")\n\n# Sort annotations by video and frame\ndf_annot_sorted = df_annotations_full.sort_values(['video_id', 'start_frame']).reset_index(drop=True)\n\n# Create a \"next behavior\" column\ndf_annot_sorted['next_action'] = df_annot_sorted.groupby('video_id')['action'].shift(-1)\n\n# Remove last behavior in each video (no transition)\ndf_transitions = df_annot_sorted[df_annot_sorted['next_action'].notna()].copy()\n\nprint(f\"Found {len(df_transitions)} behavior transitions\\n\")\n\n# Get top behaviors for readable matrix\ntop_n = 12\ntop_behaviors = df_transitions['action'].value_counts().head(top_n).index\n\n# Filter to only include top behaviors\ndf_trans_filtered = df_transitions[\n    df_transitions['action'].isin(top_behaviors) & \n    df_transitions['next_action'].isin(top_behaviors)\n]\n\n# Create transition matrix\ntransition_matrix = pd.crosstab(\n    df_trans_filtered['action'], \n    df_trans_filtered['next_action'],\n    normalize='index'  # Normalize by row to get probabilities\n) * 100  # Convert to percentage\n\nprint(f\"Transition matrix shape: {transition_matrix.shape}\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Visualize the transition matrix\nplt.figure(figsize=(14, 12))\n\nsns.heatmap(\n    transition_matrix, \n    annot=True, \n    fmt='.1f', \n    cmap='YlOrRd', \n    cbar_kws={'label': 'Transition Probability (%)'},\n    square=True,\n    linewidths=0.5\n)\n\nplt.title('Behavior Transition Matrix\\n(Row: Current Behavior → Column: Next Behavior)', fontsize=14)\nplt.xlabel('Next Behavior', fontsize=12)\nplt.ylabel('Current Behavior', fontsize=12)\nplt.xticks(rotation=45, ha='right')\nplt.yticks(rotation=0)\nplt.tight_layout()\nplt.show()\n\n# Find most common transitions\nprint(\"\\n=== Most Common Behavior Transitions ===\")\ntrans_counts = df_trans_filtered.groupby(['action', 'next_action']).size().sort_values(ascending=False)\nprint(trans_counts.head(10))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 11\n\n*   **Diagonal Dominance:** The bright diagonal in the heatmap shows that behaviors often repeat themselves (e.g., sniff → sniff → sniff). This makes sense - most behaviors persist for multiple frames.\n*   **Meaningful Transitions:** Some off-diagonal cells are highlighted, showing genuine transitions:\n    - \"sniff\" often leads to other social behaviors\n    - \"chase\" might lead to \"attack\" or \"rest\"\n    - Understanding these can help models predict upcoming behaviors\n*   **Sequence Model Insight:** This matrix suggests that:\n    - Simple Markov models might capture some patterns\n    - But we likely need longer context (LSTM/Transformer) to capture complex sequences\n    - We could use this matrix to validate our model's predictions (does it learn realistic transitions?)\n\n---\n\n# **Step 12: Social vs. Individual Behaviors**\n\n**Goal:** Some behaviors are social (involving two mice) while others are individual (one mouse alone). Understanding this distinction is crucial because:\n- Feature requirements differ (social = need inter-mouse features)\n- Model architecture might benefit from specialized branches\n- Evaluation strategies may need to account for this\n\n**Action:** Categorize and analyze behaviors by type.","metadata":{}},{"cell_type":"code","source":"# Identify social vs individual behaviors\nprint(\"=== Social vs. Individual Behavior Analysis ===\\n\")\n\n# A behavior is \"individual\" if agent_id == target_id\ndf_annotations_full['is_social'] = df_annotations_full['agent_id'] != df_annotations_full['target_id']\n\n# Count by behavior type\nbehavior_types = df_annotations_full.groupby('action')['is_social'].agg(['sum', 'count'])\nbehavior_types['pct_social'] = (behavior_types['sum'] / behavior_types['count']) * 100\nbehavior_types = behavior_types.sort_values('pct_social', ascending=False)\n\nprint(\"Behavior classification (% that are social interactions):\")\nprint(behavior_types)\n\n# Visualize\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))\n\n# Plot 1: Overall distribution\nsocial_counts = df_annotations_full['is_social'].value_counts()\nax1.pie(social_counts, labels=['Individual', 'Social'], autopct='%1.1f%%', \n        colors=['skyblue', 'salmon'], startangle=90)\nax1.set_title('Overall Distribution: Social vs. Individual Behaviors', fontsize=14)\n\n# Plot 2: By behavior type\ntop_20_behaviors = behavior_types.head(20)\nax2.barh(range(len(top_20_behaviors)), top_20_behaviors['pct_social'], color='coral')\nax2.set_yticks(range(len(top_20_behaviors)))\nax2.set_yticklabels(top_20_behaviors.index)\nax2.set_xlabel('% Social Interactions', fontsize=12)\nax2.set_title('Top 20 Behaviors: Social Interaction Percentage', fontsize=14)\nax2.axvline(x=50, color='black', linestyle='--', alpha=0.5)\nax2.grid(axis='x', alpha=0.3)\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 12\n\n*   **Mixed Dataset:** The dataset contains a healthy mix of both social and individual behaviors, which is good for model generalization.\n*   **Behavior-Specific Patterns:** Some behaviors are almost entirely social (like \"mount\", \"sniff\"), while others are purely individual (like \"groom\", \"rear\").\n*   **Feature Engineering Insight:** This tells us:\n    - For social behaviors: Inter-mouse distances, angles, and relative movements are crucial\n    - For individual behaviors: Focus on single-mouse movement patterns, body posture\n    - A model might benefit from learning these different feature patterns\n\n---\n\n# **Step 13: Arena and Experimental Setup Analysis**\n\n**Goal:** Different labs use different arenas (shape, size) and tracking methods. Understanding these differences helps us build features that generalize.\n\n**Action:** Analyze the variety of experimental setups.","metadata":{}},{"cell_type":"code","source":"# Analyze arena diversity\nprint(\"=== Experimental Setup Diversity ===\\n\")\n\n# First, let's check the actual column names\nprint(\"Available columns:\")\nprint([col for col in df_train_meta.columns if 'arena' in col.lower() or 'video' in col.lower()])\nprint()\n\n# Arena types and shapes\nprint(\"Arena shapes:\")\nprint(df_train_meta['arena_shape'].value_counts())\n\nprint(\"\\nArena types:\")\nprint(df_train_meta['arena_type'].value_counts())\n\nprint(\"\\nTracking methods:\")\nprint(df_train_meta['tracking_method'].value_counts())\n\n# Visualize arena sizes and experimental setup diversity\nfig, axes = plt.subplots(2, 2, figsize=(14, 10))\n\n# Arena dimensions\naxes[0, 0].scatter(df_train_meta['arena_width_cm'], df_train_meta['arena_height_cm'], \n                   alpha=0.5, c='steelblue')\naxes[0, 0].set_xlabel('Arena Width (cm)', fontsize=11)\naxes[0, 0].set_ylabel('Arena Height (cm)', fontsize=11)\naxes[0, 0].set_title('Arena Dimensions Distribution', fontsize=12)\naxes[0, 0].grid(True, alpha=0.3)\n\n# Video resolutions - CORRECT COLUMN NAMES with '_pix' suffix\naxes[0, 1].scatter(df_train_meta['video_width_pix'], df_train_meta['video_height_pix'], \n                   alpha=0.5, c='coral')\naxes[0, 1].set_xlabel('Video Width (pixels)', fontsize=11)\naxes[0, 1].set_ylabel('Video Height (pixels)', fontsize=11)\naxes[0, 1].set_title('Video Resolution Distribution', fontsize=12)\naxes[0, 1].grid(True, alpha=0.3)\n\n# FPS distribution\naxes[1, 0].hist(df_train_meta['frames_per_second'].dropna(), bins=30, \n                color='mediumseagreen', edgecolor='black')\naxes[1, 0].set_xlabel('Frames Per Second', fontsize=11)\naxes[1, 0].set_ylabel('Count', fontsize=11)\naxes[1, 0].set_title('Frame Rate Distribution', fontsize=12)\naxes[1, 0].grid(True, alpha=0.3)\n\n# Pixels per cm (scale factor)\naxes[1, 1].hist(df_train_meta['pix_per_cm_approx'].dropna(), bins=30, \n                color='orchid', edgecolor='black')\naxes[1, 1].set_xlabel('Pixels per CM', fontsize=11)\naxes[1, 1].set_ylabel('Count', fontsize=11)\naxes[1, 1].set_title('Scale Factor Distribution', fontsize=12)\naxes[1, 1].grid(True, alpha=0.3)\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 13\n\n*   **High Variability:** There's significant variation in:\n    - Arena sizes (from ~30cm to ~100cm in some dimensions)\n    - Video resolutions (from ~400x400 to ~1000x1000 pixels)\n    - Frame rates (typically 30 FPS but with variation)\n    - Scale factors (pixels per cm varies widely)\n    \n*   **Normalization Critical:** This variability means:\n    - We MUST normalize spatial features by arena size or scale\n    - Velocity features should account for FPS differences\n    - Raw pixel coordinates will be very different across labs\n    \n*   **Feature Engineering Strategy:**\n    - Convert all distances to real-world units (cm) using `pix_per_cm`\n    - Normalize positions relative to arena dimensions\n    - Adjust temporal features for different frame rates\n\n---\n\n# **Step 14: Summary Statistics and Key Insights**\n\n**Goal:** Create a comprehensive summary of dataset statistics to guide modeling decisions.\n\n**Action:** Generate final summary tables and actionable insights.","metadata":{}},{"cell_type":"code","source":"# Generate comprehensive dataset summary\nprint(\"=\"*70)\nprint(\"COMPREHENSIVE DATASET SUMMARY\")\nprint(\"=\"*70)\n\nsummary = {\n    'Total Training Videos': len(df_train_meta),\n    'Total Test Videos (Public)': len(df_test_meta),\n    'Unique Labs': df_train_meta['lab_id'].nunique(),\n    'Total Annotated Behaviors': len(df_annotations_full),\n    'Unique Behavior Types': df_annotations_full['action'].nunique(),\n    'Date Range (approx frames)': f\"{df_annotations_full['start_frame'].min()} to {df_annotations_full['stop_frame'].max()}\",\n    'Avg Behaviors per Video': len(df_annotations_full) / len(df_train_meta),\n    'Social Behaviors (%)': (df_annotations_full['is_social'].sum() / len(df_annotations_full)) * 100,\n}\n\nfor key, value in summary.items():\n    print(f\"{key:.<50} {value}\")\n\nprint(\"\\n\" + \"=\"*70)\nprint(\"KEY CHALLENGES IDENTIFIED\")\nprint(\"=\"*70)\n\nchallenges = [\n    \"1. EXTREME CLASS IMBALANCE - Top behavior 1000x more common than rarest\",\n    \"2. LAB GENERALIZATION - Must use GroupKFold with lab_id\",\n    \"3. MISSING DATA - 5-20% tracking failures, varies by lab\",\n    \"4. VARIABLE DURATIONS - Behaviors range from 1 frame to 1000+ frames\",\n    \"5. MULTI-SCALE PROBLEM - Need features at frame, sequence, and video level\",\n    \"6. SETUP VARIABILITY - Different arenas, resolutions, FPS across labs\"\n]\n\nfor challenge in challenges:\n    print(f\"  {challenge}\")\n\nprint(\"\\n\" + \"=\"*70)\nprint(\"ACTIONABLE MODELING INSIGHTS\")\nprint(\"=\"*70)\n\ninsights = [\n    \"✓ Feature Engineering: Focus on normalized spatial features (distances, angles)\",\n    \"✓ Temporal Context: Use sequence models (LSTM/Transformer) with window size 30-100 frames\",\n    \"✓ Class Balance: Apply class weights, focal loss, or oversampling for rare behaviors\",\n    \"✓ Validation: GroupKFold on lab_id is MANDATORY for realistic evaluation\",\n    \"✓ Missing Data: Implement forward-fill + interpolation for tracking gaps\",\n    \"✓ Multi-Scale: Consider ensemble of frame-level + sequence-level models\",\n    \"✓ Normalization: Convert to real-world units (cm, cm/s) using metadata\"\n]\n\nfor insight in insights:\n    print(f\"  {insight}\")\n\nprint(\"\\n\" + \"=\"*70)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create a final behavior reference table\nprint(\"\\n=== BEHAVIOR REFERENCE TABLE ===\\n\")\n\nbehavior_summary = df_annotations_full.groupby('action').agg({\n    'action': 'count',\n    'duration_frames': ['median', 'mean', 'std'],\n    'is_social': lambda x: (x.sum() / len(x)) * 100\n}).round(2)\n\nbehavior_summary.columns = ['Count', 'Median_Duration', 'Mean_Duration', 'Std_Duration', 'Pct_Social']\nbehavior_summary = behavior_summary.sort_values('Count', ascending=False)\n\nprint(\"Top 15 Most Common Behaviors:\")\ndisplay(behavior_summary.head(15))\n\nprint(\"\\nRarest Behaviors (Bottom 10):\")\ndisplay(behavior_summary.tail(10))\n\n# Save for future reference\nprint(\"\\n✓ Summary statistics calculated and ready for modeling phase!\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n\n# **Extended EDA Conclusion: Ready for Modeling**\n\nWe have now completed a truly comprehensive exploration of the MABe dataset. Beyond understanding the basic structure, we've uncovered:\n\n### **Critical Patterns Discovered:**\n1. **Spatial features** (inter-mouse distances, velocities) show clear correlations with behaviors\n2. **Behavior transitions** follow logical patterns that models can learn\n3. **Temporal patterns** are mostly uniform, but context matters\n4. **Missing data patterns** vary by lab and bodypart - requiring robust handling\n5. **Setup diversity** demands careful normalization and feature engineering\n\n### **The Path Forward:**\n\nWith these insights, we're prepared to build a robust modeling pipeline that:\n- Handles extreme class imbalance through weighted losses\n- Generalizes across labs using proper cross-validation\n- Leverages spatial and temporal features intelligently\n- Accounts for data quality variations\n- Scales to the hidden test set\n\n","metadata":{}}]}