{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59156,"databundleVersionId":13719397,"sourceType":"competition"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Notebook 1: The Observer's Log - A Deep Dive into the Data (EDA)\n\n**Overall Goal:** Before writing a single line of model code, our objective is to build a strong intuition for the MABe dataset. We will explore the structure of the data, visualize the mouse movements, and identify the core challenges of this competition, such as class imbalance and data variability. This deep understanding will guide all of our future feature engineering and modeling decisions.\n\n---\n\n# **Step 1: Setup and Metadata Exploration**\n\n**Goal:** Based on the file structure, the primary data is stored in efficient `.parquet` files located in separate folders for tracking and annotations. The `train.csv` and `test.csv` files likely serve as metadata indexes, providing a list of all video IDs and potentially other high-level information.\n\nOur first step is to load these metadata files to understand the scope of the dataset (how many training/test videos are there?) and how we can use them to access the individual data files.\n\n**Action:** Please run the code block below to import libraries and inspect the head and info of the `train.csv` and `test.csv` metadata files.","metadata":{}},{"cell_type":"code","source":"# Import necessary libraries\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\nfrom tqdm.auto import tqdm\nfrom matplotlib.animation import FuncAnimation\nfrom IPython.display import HTML","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:57:55.609688Z","iopub.execute_input":"2025-09-23T07:57:55.610008Z","iopub.status.idle":"2025-09-23T07:57:55.628867Z","shell.execute_reply.started":"2025-09-23T07:57:55.609984Z","shell.execute_reply":"2025-09-23T07:57:55.627968Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Set some display options for pandas for better readability\npd.set_option('display.max_columns', 100)\nsns.set_style('whitegrid')\n\n# Define the path to your data\nDATA_PATH = '/kaggle/input/MABe-mouse-behavior-detection/'\n\n# Load the metadata files\nprint(\"Loading train.csv (metadata)...\")\ndf_train_meta = pd.read_csv(DATA_PATH + 'train.csv')\n\nprint(\"Loading test.csv (metadata)...\")\ndf_test_meta = pd.read_csv(DATA_PATH + 'test.csv')\n\nprint(\"\\n--- Train Metadata ---\")\nprint(f\"Shape: {df_train_meta.shape}\")\ndisplay(df_train_meta.head())\n\nprint(\"\\n--- Test Metadata ---\")\nprint(f\"Shape: {df_test_meta.shape}\")\ndisplay(df_test_meta.head())","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:16:06.185763Z","iopub.execute_input":"2025-09-23T07:16:06.186026Z","iopub.status.idle":"2025-09-23T07:16:06.328821Z","shell.execute_reply.started":"2025-09-23T07:16:06.186007Z","shell.execute_reply":"2025-09-23T07:16:06.328112Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 1\n\n*   **Metadata Confirmed:** `train.csv` and `test.csv` are indeed high-level metadata files, not the raw tracking data. They serve as an index for the entire dataset.\n*   **Dataset Scale:** The training set is substantial, with **8,790 videos**. This means our methods need to be efficient.\n*   **Code Competition Structure:** The public test set is tiny (just **1 video**). This is a classic sign of a code competition where our submitted notebook will be re-run on a much larger, hidden test set. This emphasizes the need for a **generalizable solution**, not one that is overfit to this single test video.\n*   **Rich Metadata:** We have a treasure trove of information for each video:\n    *   `lab_id`: Tells us which lab the data came from. This is a crucial feature for handling data variability.\n    *   `mouse_...`: Details about the strain, sex, age, and condition of up to four mice.\n    *   `video_...` / `arena_...`: Technical details about the recording setup (FPS, resolution, arena size).\n*   **File Path Keys:** The `lab_id` and `video_id` columns are the keys we need to construct the paths to the actual `.parquet` data files.","metadata":{}},{"cell_type":"markdown","source":"# Step 2: Loading and Inspecting a Single Data Sample\n\n**Goal:** Now that we understand the map (the metadata), it's time to explore the territory (the actual data). We will select the very first video from our training metadata and load its corresponding tracking and annotation files. This will reveal the low-level data structure we'll be working with for our models.\n\n**Action:** The code below will:\n1.  Select the first video from `df_train_meta`.\n2.  Construct the file paths for its tracking and annotation data.\n3.  Load the two `.parquet` files into new pandas DataFrames.\n4.  Display the first few rows, shape, and column information for both the tracking and annotation data.","metadata":{}},{"cell_type":"code","source":"# Select the first video from the metadata as our sample\nsample_video_meta = df_train_meta.iloc[0]\nsample_lab_id = sample_video_meta['lab_id']\nsample_video_id = sample_video_meta['video_id']\n\nprint(f\"Loading sample video...\\n  Lab ID: {sample_lab_id}\\n  Video ID: {sample_video_id}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:20:25.158858Z","iopub.execute_input":"2025-09-23T07:20:25.159169Z","iopub.status.idle":"2025-09-23T07:20:25.166286Z","shell.execute_reply.started":"2025-09-23T07:20:25.159144Z","shell.execute_reply":"2025-09-23T07:20:25.165293Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Construct the file paths using the lab and video IDs\ntracking_path = os.path.join(DATA_PATH, 'train_tracking', sample_lab_id, f'{sample_video_id}.parquet')\nannotation_path = os.path.join(DATA_PATH, 'train_annotation', sample_lab_id, f'{sample_video_id}.parquet')\n\n# Load the actual data from the parquet files\ndf_tracking_sample = pd.read_parquet(tracking_path)\ndf_annot_sample = pd.read_parquet(annotation_path)\n\nprint(\"\\n--- Sample Tracking Data ---\")\nprint(f\"Shape: {df_tracking_sample.shape}\")\nprint(\"Info:\")\ndf_tracking_sample.info()\nprint(\"\\nFirst 5 rows:\")\ndisplay(df_tracking_sample.head())\n\nprint(\"\\n\\n--- Sample Annotation Data ---\")\nprint(f\"Shape: {df_annot_sample.shape}\")\nprint(\"Info:\")\ndf_annot_sample.info()\nprint(\"\\nFirst 5 rows:\")\ndisplay(df_annot_sample.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:20:36.677380Z","iopub.execute_input":"2025-09-23T07:20:36.678162Z","iopub.status.idle":"2025-09-23T07:20:37.238859Z","shell.execute_reply.started":"2025-09-23T07:20:36.678129Z","shell.execute_reply":"2025-09-23T07:20:37.238022Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 2\n\n*   **\"Long\" Data Format:** The tracking data is in a \"long\" or \"tidy\" format. This means every single row represents just **one bodypart** for **one mouse** in **one frame**. While this is efficient for storage, it's not immediately useful for machine learning, where we typically want to see all the information for a single frame in one row.\n*   **The Reshaping Challenge:** Our next major task will be to \"pivot\" or reshape this data. We need to transform it into a \"wide\" format where each row represents a single `video_frame`, and the columns contain all the x and y coordinates for all mice (e.g., `mouse1_head_x`, `mouse1_head_y`, `mouse2_head_x`, etc.).\n*   **Annotation Structure:** The annotation data is very clear. It defines discrete behavioral events with a start and end frame. We can see some actions are individual (where `agent_id` == `target_id`, like \"rear\") and some are social (where they are different, like \"avoid\").\n*   **Data Granularity:** For a video that is ~615 seconds long at 30 FPS (from the metadata), we'd expect around 18,450 frames. The tracking data has `1,087,658` rows. Dividing this by the number of bodyparts and mice should give us the frame count, confirming the data's structure.","metadata":{}},{"cell_type":"markdown","source":"# Step 3: Reshaping the Data for Analysis (Pivoting)\n\n**Goal:** To transform our \"long\" tracking data into a \"wide\" format. This is the most critical data manipulation step in our EDA. A wide format (one row per frame) will allow us to easily calculate features, visualize entire scenes, and feed the data into a model later.\n\n**Action:** The code below will:\n1.  First, list the unique bodyparts being tracked in this video to see what we're working with.\n2.  Use the `pivot` function in pandas to reshape the data.\n3.  We will pivot the `x` and `y` coordinates separately and then merge them together to create one comprehensive DataFrame for the sample video.\n4.  Display the head of the new, wide DataFrame to see the result.","metadata":{}},{"cell_type":"code","source":"# 1. See what bodyparts are available\nunique_bodyparts = df_tracking_sample['bodypart'].unique()\nprint(f\"Unique bodyparts tracked: {unique_bodyparts}\\n\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:34:18.759006Z","iopub.execute_input":"2025-09-23T07:34:18.759354Z","iopub.status.idle":"2025-09-23T07:34:18.835850Z","shell.execute_reply.started":"2025-09-23T07:34:18.759328Z","shell.execute_reply":"2025-09-23T07:34:18.834868Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 2. Pivot the table to get a \"wide\" format\n# We want one row per video_frame, and columns for each mouse's bodypart's x and y coordinates.\n\nprint(\"Pivoting data from long to wide format...\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:34:20.699329Z","iopub.execute_input":"2025-09-23T07:34:20.699675Z","iopub.status.idle":"2025-09-23T07:34:20.704473Z","shell.execute_reply.started":"2025-09-23T07:34:20.699647Z","shell.execute_reply":"2025-09-23T07:34:20.703465Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create a pivot for the 'x' coordinates\npivot_x = df_tracking_sample.pivot(\n    index='video_frame', \n    columns=['mouse_id', 'bodypart'], \n    values='x'\n)\n# Rename columns for clarity, e.g., (1, 'nose') -> 'mouse1_nose_x'\npivot_x.columns = [f\"mouse{m}_{bp}_x\" for m, bp in pivot_x.columns]\n\n\n# Create a pivot for the 'y' coordinates\npivot_y = df_tracking_sample.pivot(\n    index='video_frame', \n    columns=['mouse_id', 'bodypart'], \n    values='y'\n)\n# Rename columns for clarity\npivot_y.columns = [f\"mouse{m}_{bp}_y\" for m, bp in pivot_y.columns]\n\n\n# 3. Merge the x and y pivots into a single DataFrame\ndf_wide_sample = pd.concat([pivot_x, pivot_y], axis=1)\n\n# Sort columns alphabetically for consistent order\ndf_wide_sample = df_wide_sample.sort_index(axis=1)\n\n\nprint(\"Pivoting complete.\\n\")\nprint(\"--- Reshaped Wide DataFrame ---\")\nprint(f\"Shape: {df_wide_sample.shape}\")\ndisplay(df_wide_sample.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:34:34.435293Z","iopub.execute_input":"2025-09-23T07:34:34.435643Z","iopub.status.idle":"2025-09-23T07:34:35.256068Z","shell.execute_reply.started":"2025-09-23T07:34:34.435616Z","shell.execute_reply":"2025-09-23T07:34:35.254944Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 3\n\n*   **Pivoting is Key:** We have successfully transformed the data from a long, stacked format into a wide, intuitive format. The shape `(18451, 142)` tells us we have 18,451 frames of data, and 142 feature columns (a mix of x and y coordinates for all tracked bodyparts on all mice).\n*   **Missing Data (`NaN`):** Notice the presence of `NaN` (Not a Number) values. This is completely normal in tracking data. It means the tracking algorithm (e.g., DeepLabCut) was not confident enough to assign a coordinate for that bodypart in that specific frame. This could be due to one mouse blocking another (occlusion) or fast movements causing motion blur. We will need to keep this in mind when engineering features.\n*   **Complexity of Bodyparts:** The list of unique bodyparts shows a mix of standard anatomical points (`nose`, `ear_left`, `tail_base`) and some experiment-specific ones (`headpiece_...`). For general-purpose features, we'll focus on the standard anatomical points first.","metadata":{}},{"cell_type":"markdown","source":"# Step 4: Visualizing the Data - A Static Snapshot\n\n**Goal:** Before we animate the mice, let's make sure we can plot a single frame correctly. This helps us understand the coordinate system and see the posture of all mice at one moment in time.\n\n**Action:** We will write a function that takes a single frame's data (one row from our wide DataFrame) and plots the keypoints for each mouse. We'll connect some keypoints with lines to form a simple \"skeleton\" for better visualization.\n1.  Define a list of standard, anatomical bodyparts we want to focus on.\n2.  Define the connections between these parts to draw skeletons.\n3.  Create the plotting function.\n4.  Use the function to plot frame `1000` of our sample video.","metadata":{}},{"cell_type":"code","source":"# 1. Define the core bodyparts we want to visualize\n# We will ignore the 'headpiece' parts for this general visualization\nANATOMICAL_BODYPARTS = [\n    'nose', 'ear_left', 'ear_right', 'neck', 'body_center', \n    'lateral_left', 'lateral_right', 'tail_base'\n]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:38:26.509681Z","iopub.execute_input":"2025-09-23T07:38:26.509968Z","iopub.status.idle":"2025-09-23T07:38:26.514522Z","shell.execute_reply.started":"2025-09-23T07:38:26.509947Z","shell.execute_reply":"2025-09-23T07:38:26.513591Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 2. Define connections to draw a simple skeleton\n# Each tuple represents a line from one bodypart to another\nSKELETON_CONNECTIONS = [\n    ('nose', 'ear_left'), ('nose', 'ear_right'), ('ear_left', 'ear_right'),\n    ('nose', 'neck'), ('neck', 'body_center'),\n    ('body_center', 'lateral_left'), ('body_center', 'lateral_right'),\n    ('body_center', 'tail_base')\n]\n\n# Define a color for each mouse for consistent plotting\nMOUSE_COLORS = {1: 'blue', 2: 'orange', 3: 'green', 4: 'red'}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:38:26.930519Z","iopub.execute_input":"2025-09-23T07:38:26.930793Z","iopub.status.idle":"2025-09-23T07:38:26.935757Z","shell.execute_reply.started":"2025-09-23T07:38:26.930775Z","shell.execute_reply":"2025-09-23T07:38:26.934656Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 3. Create the plotting function\ndef plot_frame(frame_data):\n    \"\"\"Plots the skeletons of all mice for a single frame of data.\"\"\"\n    \n    plt.figure(figsize=(8, 8))\n    \n    # Iterate through each mouse\n    for mouse_id in range(1, 5): # Assumes up to 4 mice\n        \n        # Check if data for this mouse exists in the frame\n        if f'mouse{mouse_id}_nose_x' not in frame_data or pd.isna(frame_data[f'mouse{mouse_id}_nose_x']):\n            continue # Skip if this mouse isn't tracked in this frame\n            \n        # Plot the keypoints (bodyparts)\n        for part in ANATOMICAL_BODYPARTS:\n            col_x = f'mouse{mouse_id}_{part}_x'\n            col_y = f'mouse{mouse_id}_{part}_y'\n            if col_x in frame_data and col_y in frame_data:\n                plt.scatter(frame_data[col_x], frame_data[col_y], color=MOUSE_COLORS[mouse_id], label=f'Mouse {mouse_id}' if part == 'nose' else \"\")\n\n        # Plot the skeleton connections\n        for part1, part2 in SKELETON_CONNECTIONS:\n            col1_x, col1_y = f'mouse{mouse_id}_{part1}_x', f'mouse{mouse_id}_{part1}_y'\n            col2_x, col2_y = f'mouse{mouse_id}_{part2}_x', f'mouse{mouse_id}_{part2}_y'\n\n            # Check if both points for the line exist\n            if all(c in frame_data for c in [col1_x, col1_y, col2_x, col2_y]) and \\\n               pd.notna(frame_data[col1_x]) and pd.notna(frame_data[col2_x]):\n                \n                plt.plot([frame_data[col1_x], frame_data[col2_x]], \n                         [frame_data[col1_y], frame_data[col2_y]], \n                         color=MOUSE_COLORS[mouse_id], alpha=0.7)\n\n    plt.title(f\"Mouse Positions at Frame {frame_data.name}\")\n    plt.xlabel(\"X-coordinate\")\n    plt.ylabel(\"Y-coordinate\")\n    \n    # Invert the y-axis because image coordinates (0,0) are usually at the top-left\n    plt.gca().invert_yaxis()\n    plt.legend()\n    plt.axis('equal') # Ensure aspect ratio is maintained\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:38:27.382119Z","iopub.execute_input":"2025-09-23T07:38:27.382452Z","iopub.status.idle":"2025-09-23T07:38:27.391344Z","shell.execute_reply.started":"2025-09-23T07:38:27.382426Z","shell.execute_reply":"2025-09-23T07:38:27.390610Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 4. Use the function to plot a specific frame\nFRAME_TO_PLOT = 1000\nplot_frame(df_wide_sample.loc[FRAME_TO_PLOT])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:38:31.555288Z","iopub.execute_input":"2025-09-23T07:38:31.555571Z","iopub.status.idle":"2025-09-23T07:38:32.169879Z","shell.execute_reply.started":"2025-09-23T07:38:31.555550Z","shell.execute_reply":"2025-09-23T07:38:32.168810Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 4\n\n*   **Visualization Success:** Our plotting function works perfectly! We can now take any single frame of data and instantly visualize the entire scene.\n*   **Relative Positions are Clear:** We can see that at frame 1000, Mouse 3 (green) and Mouse 1 (blue) are relatively close, while Mouse 4 (red) is further away. This ability to see spatial relationships is the foundation for understanding social behavior.\n*   **Missing Mice Handled:** Notice that Mouse 2 is not plotted. Our code correctly handled the missing data for this mouse at this frame, which is essential for creating a robust visualization tool.\n*   **Static is Not Enough:** A single frame shows posture, but behavior is defined by **movement through time**. To truly understand what's happening, we need to see the sequence of these frames.","metadata":{}},{"cell_type":"markdown","source":"# Step 5: The Dynamic View - Animating a Behavior\n\n**Goal:** This is the most intuitive part of our EDA. We will create an animation—a mini-movie—of the mice. This will allow us to see how their positions and postures change over time, giving us a true feel for their behavior.\n\n**Action:**\n1.  We will pick the first labeled behavior from our `df_annot_sample` DataFrame.\n2.  We'll extract the `start_frame` and `stop_frame` for that behavior.\n3.  We will create an animation of the mouse movements during that specific time window.\n4.  We will display the animation directly in the notebook.\n\nThis will be our first look at a specific, labeled action as it actually happened.","metadata":{}},{"cell_type":"code","source":"# 1. Pick a behavior to animate from our annotation sample\nbehavior_to_animate = df_annot_sample.iloc[0]\nstart_frame = behavior_to_animate['start_frame']\nstop_frame = behavior_to_animate['stop_frame']\naction = behavior_to_animate['action']\nagent = behavior_to_animate['agent_id']\n\nprint(f\"Preparing to animate behavior: '{action}' by Mouse {agent}\")\nprint(f\"Frame range: {start_frame} to {stop_frame}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:46:46.755778Z","iopub.execute_input":"2025-09-23T07:46:46.756050Z","iopub.status.idle":"2025-09-23T07:46:46.762418Z","shell.execute_reply.started":"2025-09-23T07:46:46.756032Z","shell.execute_reply":"2025-09-23T07:46:46.761379Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Add a small buffer before and after to see the context\nANIM_START = max(0, start_frame - 20)\nANIM_STOP = stop_frame + 20\n\n# Slice our wide dataframe to get only the frames we need for the animation\nanim_df = df_wide_sample.loc[ANIM_START:ANIM_STOP]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:47:26.688081Z","iopub.execute_input":"2025-09-23T07:47:26.688423Z","iopub.status.idle":"2025-09-23T07:47:26.693483Z","shell.execute_reply.started":"2025-09-23T07:47:26.688395Z","shell.execute_reply":"2025-09-23T07:47:26.692663Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# --- Animation Setup ---\n\n# Set up the figure and axis\nfig, ax = plt.subplots(figsize=(8, 8))\n\n# Determine axis limits from the entire animation sequence to prevent jittering\nx_min, x_max = anim_df.filter(like='_x').min().min(), anim_df.filter(like='_x').max().max()\ny_min, y_max = anim_df.filter(like='_y').min().min(), anim_df.filter(like='_y').max().max()\npadding = 50 # Add some padding to the plot\nax.set_xlim(x_min - padding, x_max + padding)\nax.set_ylim(y_min - padding, y_max + padding)\n\n\n# The function that will draw each frame of the animation\ndef update(frame_num):\n    ax.clear() # Clear the previous frame\n    \n    # Get the data for the current frame\n    frame_data = anim_df.iloc[frame_num]\n    current_real_frame = anim_df.index[frame_num]\n    \n    # Plot each mouse for the current frame\n    for mouse_id in range(1, 5):\n        if f'mouse{mouse_id}_nose_x' not in frame_data or pd.isna(frame_data[f'mouse{mouse_id}_nose_x']):\n            continue\n\n        # Plot keypoints\n        for part in ANATOMICAL_BODYPARTS:\n            col_x, col_y = f'mouse{mouse_id}_{part}_x', f'mouse{mouse_id}_{part}_y'\n            if col_x in frame_data and col_y in frame_data:\n                ax.scatter(frame_data[col_x], frame_data[col_y], color=MOUSE_COLORS[mouse_id])\n\n        # Plot skeleton\n        for part1, part2 in SKELETON_CONNECTIONS:\n            col1_x, col1_y = f'mouse{mouse_id}_{part1}_x', f'mouse{mouse_id}_{part1}_y'\n            col2_x, col2_y = f'mouse{mouse_id}_{part2}_x', f'mouse{mouse_id}_{part2}_y'\n            if all(c in frame_data for c in [col1_x, col1_y, col2_x, col2_y]) and \\\n               pd.notna(frame_data[col1_x]) and pd.notna(frame_data[col2_x]):\n                ax.plot([frame_data[col1_x], frame_data[col2_x]], [frame_data[col1_y], frame_data[col2_y]], color=MOUSE_COLORS[mouse_id], alpha=0.7)\n\n    # Set titles and labels for the frame\n    ax.set_title(f\"Behavior: '{action}' by Mouse {agent} | Frame: {current_real_frame}\")\n    ax.set_xlabel(\"X-coordinate\")\n    ax.set_ylabel(\"Y-coordinate\")\n    ax.set_xlim(x_min - padding, x_max + padding)\n    ax.set_ylim(y_min - padding, y_max + padding)\n    ax.invert_yaxis() # Invert y-axis for image coordinates\n    return ax,","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:47:27.971913Z","iopub.execute_input":"2025-09-23T07:47:27.972198Z","iopub.status.idle":"2025-09-23T07:47:28.195351Z","shell.execute_reply.started":"2025-09-23T07:47:27.972176Z","shell.execute_reply":"2025-09-23T07:47:28.194427Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create the animation\n# frames=len(anim_df) specifies how many times to call the update function\n# interval=50 is the delay between frames in milliseconds\nani = FuncAnimation(fig, update, frames=len(anim_df), interval=50, blit=False)\n\n# Display the animation in the notebook\n# This may take a little while to render\nHTML(ani.to_jshtml())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T07:47:35.719896Z","iopub.execute_input":"2025-09-23T07:47:35.720876Z","iopub.status.idle":"2025-09-23T07:48:31.385524Z","shell.execute_reply.started":"2025-09-23T07:47:35.720846Z","shell.execute_reply":"2025-09-23T07:48:31.384274Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 5\n\n*   **Behavior is Motion:** The animation makes it crystal clear that behaviors are not static poses but dynamic sequences of movements. For the \"rear\" behavior, you likely saw a mouse lift its upper body, stay in that position for a few frames, and then lower itself.\n*   **Context is Key:** By adding a buffer before and after the labeled event, we can see the transitions into and out of a behavior. This is critical information that a sequence model can learn.\n*   **The Power of Visualization:** We now have a powerful tool to debug our future models. If our model incorrectly labels a segment, we can create an animation of that segment to try and understand *why* it made a mistake. Was it a subtle movement? Was there an occlusion?\n*   **From Deep to Wide:** We have now performed a \"deep dive\" on a single video. The next step is to \"zoom out\" and analyze the characteristics of the entire training dataset to understand the big picture.","metadata":{}},{"cell_type":"markdown","source":"# Step 6: Dataset-Wide Analysis - Behavior Distribution & Duration\n\n**Goal:** To understand the overall properties of the behaviors we need to predict. We will now use the complete `annotations.csv` file (which we loaded as `df_annotations` in Step 1) to answer critical questions:\n1.  **What are all the different behaviors?**\n2.  **How often does each behavior occur (Class Balance)?** This is one of the most important questions. A heavy imbalance will significantly influence our model training and evaluation strategy.\n3.  **How long do behaviors typically last (Duration)?** Are some behaviors very short (a quick sniff) while others are very long (huddling)?\n\n**Action:** We will create plots to visualize the frequency and duration of every behavior across the entire training set.","metadata":{}},{"cell_type":"code","source":"# This cell is designed to be self-contained. It will build the full annotation\n# dataframe if it doesn't already exist in memory.\nif 'df_annotations_full' not in locals():\n    print(\"Building the full annotations dataframe by combining all individual annotation files...\")\n\n    all_annotations_list = []\n    for index, row in tqdm(df_train_meta.iterrows(), total=df_train_meta.shape[0]):\n        lab_id = row['lab_id']\n        video_id = row['video_id']\n        annotation_path = os.path.join(DATA_PATH, 'train_annotation', lab_id, f'{video_id}.parquet')\n        \n        if os.path.exists(annotation_path):\n            temp_df = pd.read_parquet(annotation_path)\n            temp_df['video_id'] = video_id\n            all_annotations_list.append(temp_df)\n\n    df_annotations_full = pd.concat(all_annotations_list, ignore_index=True)\n    print(f\"\\nSuccessfully created full annotation dataframe with shape: {df_annotations_full.shape}\")\nelse:\n    print(\"Full annotation dataframe already exists in memory. Proceeding with analysis.\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T08:08:46.536936Z","iopub.execute_input":"2025-09-23T08:08:46.537682Z","iopub.status.idle":"2025-09-23T08:08:46.544705Z","shell.execute_reply.started":"2025-09-23T08:08:46.537655Z","shell.execute_reply":"2025-09-23T08:08:46.543658Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# --- Behavior Frequency Analysis ---\nprint(\"\\n--- Behavior Frequency Analysis ---\")\n\nbehavior_counts = df_annotations_full['action'].value_counts()\nplt.figure(figsize=(12, 8))\nsns.barplot(x=behavior_counts.index, y=behavior_counts.values, palette='viridis')\nplt.title('Frequency of Each Behavior Across the Entire Training Set', fontsize=16)\nplt.xlabel('Behavior', fontsize=12)\nplt.ylabel('Count', fontsize=12)\nplt.xticks(rotation=45, ha='right')\nplt.tight_layout()\nplt.show()\n\n# --- Behavior Duration Analysis ---\nprint(\"\\n--- Behavior Duration Analysis ---\")\ndf_annotations_full['duration_frames'] = df_annotations_full['stop_frame'] - df_annotations_full['start_frame']\n\n# Let's see how many zero-duration events we have\nzero_duration_count = (df_annotations_full['duration_frames'] == 0).sum()\nprint(f\"Found {zero_duration_count} events with a duration of 0 frames.\")\n\nprint(\"\\nBasic statistics for behavior durations (in frames):\")\ndisplay(df_annotations_full['duration_frames'].describe())\n\n# Add 1 to duration before plotting on a log scale to handle zeros\nplt.figure(figsize=(12, 6))\nsns.histplot(df_annotations_full['duration_frames'] + 1, bins=100, log_scale=True)\nplt.title('Distribution of Behavior Durations (Log Scale, Duration+1)', fontsize=16)\nplt.xlabel('Duration (Frames) + 1 - Log Scale', fontsize=12)\nplt.ylabel('Frequency', fontsize=12)\nplt.show()\n\nplt.figure(figsize=(12, 10))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T08:09:04.585931Z","iopub.execute_input":"2025-09-23T08:09:04.586285Z","iopub.status.idle":"2025-09-23T08:09:05.998808Z","shell.execute_reply.started":"2025-09-23T08:09:04.586212Z","shell.execute_reply":"2025-09-23T08:09:05.997986Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\norder = df_annotations_full.groupby('action')['duration_frames'].median().sort_values(ascending=False).index\n# Use duration + 1 for the x-axis in the boxplot as well\nsns.boxplot(x=df_annotations_full['duration_frames'] + 1, y='action', data=df_annotations_full, order=order, palette='coolwarm')\nplt.title('Duration of Each Behavior Type', fontsize=16)\nplt.xlabel('Duration (Frames) + 1 - Log Scale', fontsize=12)\nplt.ylabel('Behavior', fontsize=12)\nplt.xscale('log')\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T08:09:07.636062Z","iopub.execute_input":"2025-09-23T08:09:07.636415Z","iopub.status.idle":"2025-09-23T08:09:08.960860Z","shell.execute_reply.started":"2025-09-23T08:09:07.636391Z","shell.execute_reply":"2025-09-23T08:09:08.960018Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 6\n\nThe dataset-wide analysis has revealed the most critical challenges of this competition.\n\n**From the Frequency Plot (Bar Chart):**\n\n*   **Extreme Class Imbalance:** This is the #1 challenge. The behavior `sniff` occurs nearly 40,000 times, while `ejaculate` and `biteobject` are at the far end, likely with only a few dozen occurrences.\n*   **Modeling Implication:** A standard model will become an expert at predicting `sniff` and `attack` but will completely ignore the rare classes because it can achieve high accuracy by just focusing on the majority. We **must** use special techniques to handle this, such as:\n    *   Using an appropriate evaluation metric that cares about rare classes (like the competition's F-Score variant).\n    *   Applying class weights during training to penalize the model more for misclassifying rare behaviors.\n    *   Using special sampling techniques (e.g., oversampling rare classes).\n\n**From the Duration Distribution Plot (Histogram):**\n\n*   **Bimodal Distribution:** The histogram isn't a simple bell curve. It has two \"humps\" or modes. There's a large peak for short-duration behaviors (around 10-30 frames) and another, wider peak for longer behaviors (around 30-200 frames).\n*   **Modeling Implication:** This suggests there isn't one \"typical\" behavior length. Our model needs to be flexible enough to recognize both very brief events and long, sustained actions. The sequence length we choose for our models (e.g., LSTMs, Transformers) will be an important hyperparameter.\n\n**From the Duration by Type Plot (Box Plot):**\n\n*   **Behaviors Have Characteristic Durations:** This plot is incredibly useful. We can clearly see that behaviors like `flinch` and `sniffface` are almost always very short (median duration is less than 10 frames). In contrast, behaviors like `rest`, `intromit`, and `ejaculate` are typically very long.\n*   **Feature Engineering Idea:** The duration of an action is itself a powerful predictive feature! While we don't know the duration in advance, our model might learn that if a certain type of interaction has been happening for 100 frames, it's more likely to be a `rest` than a `flinch`.\n*   **High Variance:** Notice the long tails and many outlier points (the black diamonds) for almost every behavior. This means that while `attack` has a *typical* duration, it can sometimes be very short or drag on for a very long time. Our model must be robust to this variability.","metadata":{}},{"cell_type":"markdown","source":"# Step 7: The Lab Effect - Analyzing Data Variability\n\n**Goal:** The competition description explicitly mentions the challenge of generalizing across data from different labs, which may use different equipment and tracking methods. Our final EDA step is to investigate this \"lab effect.\"\n\n**Action:** We will join our full annotation data with the original metadata to get the `lab_id` for each behavior. Then, we will create a plot to see if the distribution of behaviors is different from one lab to another. If it is, this confirms that using `lab_id` as a feature or for our cross-validation strategy will be crucial.","metadata":{}},{"cell_type":"code","source":"# First, ensure 'df_annotations_full' exists\nif 'df_annotations_full' not in locals():\n    print(\"Full annotation dataframe is not in memory. Please re-run the previous cell (Step 6).\")\nelse:\n    print(\"--- Lab Variability Analysis ---\")\n    \n    # We need to merge our annotations with the metadata to get the lab_id for each event\n    # We select only the 'video_id' and 'lab_id' from the metadata to keep the merge light\n    df_lab_info = df_train_meta[['video_id', 'lab_id']]\n    \n    # Perform a left merge to add 'lab_id' to each annotation\n    df_annotations_with_lab = pd.merge(df_annotations_full, df_lab_info, on='video_id', how='left')\n    \n    print(f\"Successfully merged lab info. New shape: {df_annotations_with_lab.shape}\")\n    display(df_annotations_with_lab.head())\n    \n    # Now, let's plot the behavior counts per lab\n    plt.figure(figsize=(15, 8))\n    \n    # We use crosstab to count occurrences of each action within each lab, then normalize\n    # to see the percentage/proportion, which is better for comparison\n    crosstab_norm = pd.crosstab(df_annotations_with_lab['lab_id'], \n                               df_annotations_with_lab['action'], \n                               normalize='index') # 'normalize=index' calculates percentages per lab\n    \n    sns.heatmap(crosstab_norm, cmap='viridis', annot=False) # 'annot=True' can be messy if too many classes\n    plt.title('Proportion of Behaviors by Lab', fontsize=16)\n    plt.xlabel('Behavior', fontsize=12)\n    plt.ylabel('Lab ID', fontsize=12)\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-23T08:20:48.658421Z","iopub.execute_input":"2025-09-23T08:20:48.659165Z","iopub.status.idle":"2025-09-23T08:20:49.484093Z","shell.execute_reply.started":"2025-09-23T08:20:48.659140Z","shell.execute_reply":"2025-09-23T08:20:49.483247Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## What We Learned in Step 7\n\nThe heatmap provides definitive evidence of the \"lab effect\" and is perhaps the most important visualization for designing a winning strategy.\n\n*   **The 'Lab Effect' is Real and Severe:** The plot is not uniform at all. It's very \"blocky.\" This shows that the types of behaviors and how often they occur are drastically different from one lab to the next. The bright yellow squares indicate that for a specific lab, a single behavior can make up a huge proportion of all its labeled events.\n\n*   **Lab Specialization:** Look at the bright yellow square for lab `BoisterousParrot` under the behavior `sniffbody`. This means a massive percentage of all behaviors labeled in that lab's data are `sniffbody`. Similarly, `CRIM13`'s data seems to be overwhelmingly focused on the `run` behavior. These labs were likely designed to study these specific actions.\n\n*   **Rare Behaviors are Lab-Specific:** Some behaviors might only appear in data from one or two labs. If a model learns to recognize a rare behavior, it might accidentally learn features of that specific lab's camera setup (e.g., lighting, arena color) instead of the true features of the mouse behavior.\n\n*   **The Critical Takeaway - Validation Strategy:** This plot tells us that a simple random validation split is **the wrong approach** and will be misleading. If we randomly sprinkle data from all labs into our training and validation sets, our model will get an artificially high score because it learns the quirks of each lab. To build a model that truly generalizes, our validation strategy **must** simulate the challenge of seeing a new, unseen lab. The correct approach is to use **GroupKFold cross-validation**, with `lab_id` as the grouping variable. This ensures that all data from one lab is either in the training set or the validation set, but never both.","metadata":{}},{"cell_type":"markdown","source":"# Notebook 1 Conclusion: The Story of the Data\n\nThis concludes our deep-dive Exploratory Data Analysis. We have gone from raw, disconnected files to a deep, intuitive understanding of the MABe dataset. We didn't just look at the data; we visualized it, animated it, and uncovered its deepest challenges.\n\n### Our Key Findings and Action Plan:\n\n1.  **Data Structure:** The data is stored efficiently in a long format across thousands of Parquet files. Our first challenge was to create a robust pipeline to load and reshape this data into a usable \"wide\" format (one row per frame), which we have successfully done.\n\n2.  **Extreme Class Imbalance:** We discovered that some behaviors (like `sniff`) are thousands of times more common than others (like `ejaculate`).\n    *   **Action Plan:** We must use techniques like class weighting or special sampling methods and focus on metrics that value rare classes during modeling.\n\n3.  **Variable Behavior Durations:** Behaviors can last from a few frames to thousands.\n    *   **Action Plan:** This confirms that sequence-based models (LSTMs, Transformers) that can handle variable-length patterns will be essential.\n\n4.  **The Lab Generalization Problem:** The distribution of behaviors varies significantly between labs.\n    *   **Action Plan:** Our validation strategy must be built around `GroupKFold` using `lab_id` to ensure we are building a model that generalizes to unseen experimental setups.\n\nWe are now perfectly equipped to move on to the next stage. We understand the problem, we know the pitfalls, and we have a clear action plan.\n\n**Next Up: Notebook 2 - The First Hypothesis: A Simple Frame-by-Frame Baseline**","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}