{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59156,"databundleVersionId":13719397,"sourceType":"competition"}],"dockerImageVersionId":31089,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 📑 Findings and Interpretations\n\n---\n\n## Finding 1: Dataset Composition\n- **Observation:** 8,790 training videos and 1 test video. Majority (60.5%) from `MABe22_keypoints`, followed by `MABe22_movies` (29.6%).  \n- **Interpretation:** The dataset is heavily skewed towards a few sources, which may bias models to learn lab-specific behaviors rather than general patterns.\n\n---\n\n## Finding 2: Technical Diversity\n- **Observation:**  \n  - Frame rates range from 10–120 FPS (median 30).  \n  - Video durations vary widely (19–19,801s).  \n  - 89 unique resolution combinations, with no single dominant standard.  \n- **Interpretation:** High variability in video specs could cause inconsistencies in feature extraction. Normalization or resampling strategies are required.\n\n---\n\n## Finding 3: Arena Configurations\n- **Observation:**  \n  - Square arenas dominate (90.6%).  \n  - Minority setups include rectangular, split-rectangular, and circular arenas.  \n- **Interpretation:** Models may overfit to square setups. Special handling may be required for rare arena types to ensure generalization.\n\n---\n\n## Finding 4: Mouse Demographics\n- **Observation:**  \n  - 90.7% of videos have no mice metadata.  \n  - Most common strain: C57Bl/6J.  \n  - Sex distribution: overwhelmingly male (98.6%).  \n- **Interpretation:** Severe metadata imbalance limits demographic-driven analysis. Gender/strain-related behavior patterns may not be generalizable.\n\n---\n\n## Finding 5: Tracking Data Quality\n- **Observation:**  \n  - High-velocity anomalies present across all labs (e.g., CRIM13: 2806 events).  \n  - Test video (`AdaptableSnail`) tracking incomplete (82.2% completeness).  \n- **Interpretation:** Tracking errors may corrupt downstream behavior inference. Test set quality issues could introduce evaluation bias.\n\n---\n\n## Finding 6: Annotation Characteristics\n- **Observation:**  \n  - Each lab defines 3–7 behaviors (11 total unique).  \n  - CRIM13 annotations include 7 behaviors; others fewer (3 each).  \n  - Strong skew: `sniff` and variants dominate across labs.  \n- **Interpretation:** Behavior diversity is lab-specific. No universal annotation set exists, complicating cross-lab generalization.\n\n---\n\n## Finding 7: Annotation Durations\n- **Observation:**  \n  - Global median = 24 frames (~1s at 30 FPS).  \n  - Short behaviors (<10 frames): 23%.  \n  - Long behaviors (>52 frames): 24.7%.  \n- **Interpretation:** Class imbalance exists between fleeting vs. sustained behaviors. Models must handle temporal extremes carefully.\n\n---\n\n## Finding 8: Annotation Quality Issues\n- **Observation:**  \n  - Temporal overlaps: up to 80.2% in CRIM13.  \n  - Invalid frame ranges and ID mismatches.  \n- **Interpretation:** Annotation errors may mislead training. Preprocessing/cleaning needed to avoid noisy supervision.\n\n---\n\n## Finding 9: Metadata Quality Issues\n- **Observation:**  \n  - Extremely high missingness in `mouse_id` fields (90–100%).  \n  - `behaviors_labeled` missing in 90.2% of videos.  \n- **Interpretation:** Metadata cannot be relied upon as strong supervision signals. Models must work primarily from video + tracking data.\n\n---\n\n## Finding 10: Cross-Laboratory Patterns\n- **Observation:**  \n  - Public datasets: shorter videos, fewer resolution variants.  \n  - Anonymous labs: longer videos, more heterogeneous setups.  \n  - Resident-intruder setups common across 11 labs.  \n- **Interpretation:** Lab origin influences recording conditions and behavior types. Domain adaptation is crucial.\n\n---\n\n## Finding 11: Tracking Methodologies\n- **Observation:**  \n  - 90% tracked with custom HRNet.  \n  - Minority use MARS, DeepLabCut, SLEAP.  \n- **Interpretation:** Inconsistencies in tracking pipelines may produce systematic differences in data quality across labs.\n\n---\n\n## Finding 12: Strain Preferences\n- **Observation:**  \n  - C57Bl/6J is the dominant strain (used by 16 labs).  \n  - Others rarely represented (C57Bl/6N, CD-1, hybrid).  \n- **Interpretation:** Findings may overrepresent behavior typical of C57Bl/6J mice, limiting external validity across strains.\n\n---\n","metadata":{}},{"cell_type":"code","source":"#!/usr/bin/env python3\n\"\"\"\nMABe Challenge - Social Action Recognition in Mice\nComprehensive Exploratory Data Analysis\nPrepared by: NeurIPS Researcher, Anthropic Research Team\nTarget: Leadership Review (Demis Hassabis equivalent)\n\"\"\"\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport warnings\nimport os\nfrom pathlib import Path\nimport glob\nfrom collections import Counter, defaultdict\nfrom scipy import stats\nfrom scipy.spatial.distance import pdist, squareform\nimport json\nfrom datetime import datetime\n\n# Configuration\nwarnings.filterwarnings('ignore')\nplt.style.use('default')\nsns.set_palette(\"husl\")\nplt.rcParams['figure.figsize'] = (15, 8)\nplt.rcParams['font.size'] = 12\n\nclass MABeEDAAnalyzer:\n    \"\"\"Comprehensive EDA analyzer for MABe Challenge dataset\"\"\"\n    \n    def __init__(self):\n        self.train_df = None\n        self.test_df = None\n        self.sample_submission = None\n        self.tracking_data = {}\n        self.annotation_data = {}\n        self.analysis_results = {}\n        \n        print(\"=\"*80)\n        print(\"MABe Challenge - Mouse Social Behavior Recognition\")\n        print(\"Comprehensive Exploratory Data Analysis\")\n        print(\"=\"*80)\n        print(f\"Analysis initiated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\")\n        print()\n    \n    def load_data(self):\n        \"\"\"Load all available data files\"\"\"\n        print(\"📂 LOADING DATASET...\")\n        print(\"-\" * 40)\n        \n        try:\n            # Load metadata\n            if os.path.exists('/kaggle/input/MABe-mouse-behavior-detection/train.csv'):\n                self.train_df = pd.read_csv('/kaggle/input/MABe-mouse-behavior-detection/train.csv')\n                print(f\"✅ Train metadata loaded: {self.train_df.shape}\")\n            else:\n                print(\"⚠️  train.csv not found - generating representative data\")\n                self.train_df = self._generate_representative_metadata('train')\n            \n            if os.path.exists('/kaggle/input/MABe-mouse-behavior-detection/test.csv'):\n                self.test_df = pd.read_csv('/kaggle/input/MABe-mouse-behavior-detection/test.csv')\n                print(f\"✅ Test metadata loaded: {self.test_df.shape}\")\n            else:\n                print(\"⚠️  test.csv not found - generating representative data\")\n                self.test_df = self._generate_representative_metadata('test')\n            \n            if os.path.exists('sample_submission.csv'):\n                self.sample_submission = pd.read_csv('sample_submission.csv')\n                print(f\"✅ Sample submission loaded: {self.sample_submission.shape}\")\n            else:\n                self.sample_submission = self._generate_sample_submission()\n            \n            # Load tracking data samples\n            self._load_tracking_samples()\n            \n            # Load annotation data samples\n            self._load_annotation_samples()\n            \n        except Exception as e:\n            print(f\"❌ Error loading data: {e}\")\n            print(\"📝 Generating representative mock data for analysis...\")\n            self._generate_all_mock_data()\n        \n        print(\"\\n📊 DATA LOADING SUMMARY:\")\n        print(f\"   • Train videos: {len(self.train_df) if self.train_df is not None else 0}\")\n        print(f\"   • Test videos: {len(self.test_df) if self.test_df is not None else 0}\")\n        print(f\"   • Tracking files loaded: {len(self.tracking_data)}\")\n        print(f\"   • Annotation files loaded: {len(self.annotation_data)}\")\n        print()\n    \n    def _generate_representative_metadata(self, split='train'):\n        \"\"\"Generate realistic metadata for analysis\"\"\"\n        labs = ['CalMS21_task1', 'CalMS21_task2', 'CalMS21_supplemental', 'CRIM13', \n                'MABe22_keypoints', 'MABe22_movies', 'AdaptableSnail', 'BoisterousParrot', \n                'CautiousGiraffe', 'DeliriousFly', 'ElegantMink', 'GroovyShrew', \n                'InvincibleJellyfish', 'JovialSwallow', 'LyricalHare', 'NiftyGoldfinch', \n                'PleasantMeerkat', 'ReflectiveManatee', 'SparklingTapir', 'TranquilPanther', \n                'UppityFerret']\n        \n        np.random.seed(42 if split == 'train' else 84)\n        n_videos = 180 if split == 'train' else 45\n        \n        data = []\n        for i in range(n_videos):\n            lab = np.random.choice(labs, p=self._get_lab_probabilities())\n            \n            # Realistic technical parameters\n            fps = np.random.choice([15, 25, 30, 50, 60], p=[0.1, 0.2, 0.4, 0.2, 0.1])\n            duration = np.random.lognormal(np.log(300), 0.5)  # Log-normal distribution\n            pix_per_cm = np.random.uniform(2.5, 12.0)\n            \n            # Video dimensions correlated with era/lab\n            if 'CalMS21' in lab or 'CRIM13' in lab:\n                width, height = np.random.choice([(640, 480), (800, 600)], p=[0.7, 0.3])\n            else:\n                width, height = np.random.choice([(1024, 768), (1280, 960), (640, 480)], \n                                                p=[0.4, 0.4, 0.2])\n            \n            # Arena parameters\n            arena_shape = np.random.choice(['circle', 'rectangle', 'square'], p=[0.6, 0.3, 0.1])\n            arena_type = np.random.choice(['open_field', 'home_cage', 'social_box', 'tube_test'], \n                                        p=[0.5, 0.2, 0.2, 0.1])\n            \n            # Number of mice (realistic distribution)\n            n_mice = np.random.choice([1, 2, 3, 4], p=[0.15, 0.65, 0.15, 0.05])\n            \n            # Body parts tracked (lab-specific)\n            body_parts_sets = {\n                'minimal': ['nose', 'body_center', 'tail_base'],\n                'standard': ['nose', 'left_ear', 'right_ear', 'neck', 'body_center', \n                           'left_hip', 'right_hip', 'tail_base', 'tail_tip'],\n                'extended': ['nose', 'left_ear', 'right_ear', 'neck', 'body_center',\n                           'left_shoulder', 'right_shoulder', 'left_hip', 'right_hip',\n                           'tail_base', 'tail_mid', 'tail_tip', 'left_paw', 'right_paw']\n            }\n            \n            bp_type = np.random.choice(['minimal', 'standard', 'extended'], p=[0.2, 0.6, 0.2])\n            body_parts = body_parts_sets[bp_type]\n            \n            # Behaviors (lab and setup dependent)\n            all_behaviors = ['grooming', 'rearing', 'locomotion', 'chasing', 'following',\n                           'mounting', 'fighting', 'sniffing', 'huddling', 'circling',\n                           'digging', 'drinking', 'eating', 'freezing', 'jumping']\n            \n            if n_mice == 1:\n                behaviors = [b for b in all_behaviors if b in \n                           ['grooming', 'rearing', 'locomotion', 'digging', 'drinking', \n                            'eating', 'freezing', 'jumping']]\n            else:\n                behaviors = all_behaviors\n            \n            selected_behaviors = np.random.choice(behaviors, \n                                                np.random.randint(3, min(8, len(behaviors))),\n                                                replace=False)\n            \n            row = {\n                'lab_id': lab,\n                'video_id': f\"{lab}_{i:06d}\",\n                'frames_per_second': fps,\n                'video_duration_sec': duration,\n                'pix_per_cm_approx': pix_per_cm,\n                'video_width_pix': width,\n                'video_height_pix': height,\n                'arena_width_cm': np.random.uniform(15, 80),\n                'arena_height_cm': np.random.uniform(15, 80),\n                'arena_shape': arena_shape,\n                'arena_type': arena_type,\n                'body_parts_tracked': ', '.join(body_parts),\n                'behaviors_labeled': ', '.join(selected_behaviors),\n                'tracking_method': np.random.choice(['DeepLabCut', 'SLEAP', 'Custom', 'MARS'], \n                                                  p=[0.4, 0.3, 0.2, 0.1])\n            }\n            \n            # Mouse details\n            strains = ['C57BL6', 'BALB/c', 'CD1', '129', 'Swiss', 'FVB']\n            conditions = ['control', 'treatment', 'knockout', 'wildtype', 'stressed']\n            \n            for j in range(4):\n                prefix = f'mouse{j+1}_'\n                if j < n_mice:\n                    row[f'{prefix}strain'] = np.random.choice(strains)\n                    row[f'{prefix}color'] = np.random.choice(['black', 'white', 'brown', \n                                                            'gray', 'mixed'])\n                    row[f'{prefix}sex'] = np.random.choice(['M', 'F'])\n                    row[f'{prefix}id'] = f'{lab[:3]}_M{j+1:03d}'\n                    row[f'{prefix}age'] = np.random.randint(6, 25)  # weeks\n                    row[f'{prefix}condition'] = np.random.choice(conditions)\n                else:\n                    for attr in ['strain', 'color', 'sex', 'id', 'age', 'condition']:\n                        row[f'{prefix}{attr}'] = None\n            \n            data.append(row)\n        \n        return pd.DataFrame(data)\n    \n    def _get_lab_probabilities(self):\n        \"\"\"Realistic lab contribution probabilities\"\"\"\n        n_labs = 21\n        # Some labs contribute more data\n        probs = np.array([0.15, 0.12, 0.10, 0.08, 0.08] + [0.47/16]*16)  # First 5 major, rest smaller\n        return probs\n    \n    def _generate_sample_submission(self):\n        \"\"\"Generate realistic sample submission\"\"\"\n        behaviors = ['grooming', 'rearing', 'locomotion', 'chasing', 'following',\n                    'mounting', 'fighting', 'sniffing', 'huddling', 'circling']\n        \n        video_ids = self.test_df['video_id'].tolist() if self.test_df is not None else []\n        if not video_ids:\n            video_ids = [f\"test_video_{i:06d}\" for i in range(50)]\n        \n        n_predictions = 1200\n        \n        return pd.DataFrame({\n            'row_id': range(n_predictions),\n            'video_id': np.random.choice(video_ids, n_predictions),\n            'agent_id': np.random.randint(1, 5, n_predictions),\n            'target_id': np.random.randint(1, 5, n_predictions),\n            'action': np.random.choice(behaviors, n_predictions),\n            'start_frame': np.random.randint(0, 5000, n_predictions),\n            'stop_frame': np.random.randint(0, 5000, n_predictions)\n        })\n    \n    def _load_tracking_samples(self):\n        \"\"\"Load sample tracking data files\"\"\"\n        tracking_dirs = ['/kaggle/input/MABe-mouse-behavior-detection/train_tracking', '/kaggle/input/MABe-mouse-behavior-detection/test_tracking']\n        \n        for tracking_dir in tracking_dirs:\n            if os.path.exists(tracking_dir):\n                lab_dirs = [d for d in os.listdir(tracking_dir) \n                           if os.path.isdir(os.path.join(tracking_dir, d))]\n                \n                # Sample a few files for analysis\n                for lab in lab_dirs[:3]:  # Analyze first 3 labs\n                    lab_path = os.path.join(tracking_dir, lab)\n                    files = glob.glob(os.path.join(lab_path, \"*.parquet\"))\n                    \n                    if files:\n                        # Load first file as sample\n                        try:\n                            sample_file = files[0]\n                            df = pd.read_parquet(sample_file)\n                            self.tracking_data[f\"{tracking_dir}_{lab}\"] = df\n                            print(f\"  📊 Loaded tracking sample: {lab} - {df.shape}\")\n                        except Exception as e:\n                            print(f\"  ⚠️  Could not load {sample_file}: {e}\")\n        \n        # Generate mock tracking data if none found\n        if not self.tracking_data:\n            self._generate_mock_tracking_data()\n    \n    def _generate_mock_tracking_data(self):\n        \"\"\"Generate representative tracking data\"\"\"\n        print(\"  📝 Generating mock tracking data...\")\n        \n        labs = ['AdaptableSnail', 'BoisterousParrot', 'CautiousGiraffe']\n        body_parts = ['nose', 'left_ear', 'right_ear', 'neck', 'body_center', \n                     'left_hip', 'right_hip', 'tail_base', 'tail_tip']\n        \n        for lab in labs:\n            # Generate realistic mouse tracking data\n            n_frames = 3000\n            n_mice = 2\n            \n            data = []\n            for frame in range(n_frames):\n                for mouse_id in range(1, n_mice + 1):\n                    # Simulate mouse movement (random walk with realistic constraints)\n                    center_x = 320 + 100 * np.sin(frame * 0.01 + mouse_id)\n                    center_y = 240 + 80 * np.cos(frame * 0.01 + mouse_id)\n                    \n                    for bodypart in body_parts:\n                        # Add realistic noise and body part offsets\n                        offset_x = np.random.normal(0, 5)\n                        offset_y = np.random.normal(0, 5)\n                        \n                        if bodypart == 'nose':\n                            x = center_x + offset_x\n                            y = center_y - 15 + offset_y\n                        elif bodypart == 'body_center':\n                            x = center_x + offset_x\n                            y = center_y + offset_y\n                        elif bodypart == 'tail_base':\n                            x = center_x + offset_x\n                            y = center_y + 20 + offset_y\n                        else:\n                            x = center_x + np.random.normal(0, 8)\n                            y = center_y + np.random.normal(0, 8)\n                        \n                        # Add occasional tracking failures\n                        if np.random.random() < 0.02:  # 2% failure rate\n                            x, y = np.nan, np.nan\n                        \n                        data.append({\n                            'video_frame': frame,\n                            'mouse_id': mouse_id,\n                            'bodypart': bodypart,\n                            'x': x,\n                            'y': y\n                        })\n            \n            self.tracking_data[f\"train_tracking_{lab}\"] = pd.DataFrame(data)\n    \n    def _load_annotation_samples(self):\n        \"\"\"Load sample annotation data\"\"\"\n        if os.path.exists('/kaggle/input/MABe-mouse-behavior-detection/train_annotation'):\n            lab_dirs = [d for d in os.listdir('/kaggle/input/MABe-mouse-behavior-detection/train_annotation') \n                       if os.path.isdir(os.path.join('/kaggle/input/MABe-mouse-behavior-detection/train_annotation', d))]\n            \n            for lab in lab_dirs[:3]:  # Sample first 3 labs\n                lab_path = os.path.join('/kaggle/input/MABe-mouse-behavior-detection/train_annotation', lab)\n                files = glob.glob(os.path.join(lab_path, \"*.parquet\"))\n                \n                if files:\n                    try:\n                        # Combine all annotation files for this lab\n                        lab_annotations = []\n                        for file in files:\n                            df = pd.read_parquet(file)\n                            lab_annotations.append(df)\n                        \n                        if lab_annotations:\n                            combined = pd.concat(lab_annotations, ignore_index=True)\n                            self.annotation_data[lab] = combined\n                            print(f\"  📝 Loaded annotations: {lab} - {combined.shape}\")\n                    \n                    except Exception as e:\n                        print(f\"  ⚠️  Could not load annotations for {lab}: {e}\")\n        \n        # Generate mock annotation data if none found\n        if not self.annotation_data:\n            self._generate_mock_annotation_data()\n    \n    def _generate_mock_annotation_data(self):\n        \"\"\"Generate representative annotation data\"\"\"\n        print(\"  📝 Generating mock annotation data...\")\n        \n        labs = ['AdaptableSnail', 'BoisterousParrot', 'CautiousGiraffe']\n        behaviors = ['grooming', 'rearing', 'locomotion', 'chasing', 'following',\n                    'mounting', 'fighting', 'sniffing', 'huddling']\n        \n        for lab in labs:\n            data = []\n            n_annotations = np.random.randint(200, 800)\n            \n            for i in range(n_annotations):\n                behavior = np.random.choice(behaviors)\n                agent_id = np.random.randint(1, 3)\n                \n                # Self-directed vs social behaviors\n                if behavior in ['grooming', 'rearing', 'locomotion']:\n                    target_id = agent_id  # Self-directed\n                else:\n                    target_id = np.random.choice([j for j in [1, 2] if j != agent_id])\n                \n                start_frame = np.random.randint(0, 2500)\n                duration = np.random.lognormal(np.log(30), 0.8)  # Log-normal duration\n                stop_frame = int(start_frame + duration)\n                \n                data.append({\n                    'agent_id': agent_id,\n                    'target_id': target_id,\n                    'action': behavior,\n                    'start_frame': start_frame,\n                    'stop_frame': stop_frame\n                })\n            \n            self.annotation_data[lab] = pd.DataFrame(data)\n    \n    def _generate_all_mock_data(self):\n        \"\"\"Generate all mock data when files are not available\"\"\"\n        self.train_df = self._generate_representative_metadata('train')\n        self.test_df = self._generate_representative_metadata('test')\n        self.sample_submission = self._generate_sample_submission()\n        self._generate_mock_tracking_data()\n        self._generate_mock_annotation_data()\n    \n    def analyze_metadata_distribution(self):\n        \"\"\"Comprehensive metadata analysis\"\"\"\n        print(\"📈 METADATA DISTRIBUTION ANALYSIS\")\n        print(\"-\" * 40)\n        \n        results = {}\n        \n        # Laboratory distribution\n        lab_counts = self.train_df['lab_id'].value_counts()\n        results['lab_distribution'] = lab_counts.to_dict()\n        \n        print(\"🏢 Laboratory Distribution:\")\n        for lab, count in lab_counts.head(10).items():\n            print(f\"   {lab}: {count} videos ({count/len(self.train_df)*100:.1f}%)\")\n        \n        if len(lab_counts) > 10:\n            print(f\"   ... and {len(lab_counts)-10} other labs\")\n        print()\n        \n        # Technical specifications analysis\n        print(\"⚙️ Technical Specifications:\")\n        \n        # Frame rates\n        fps_stats = self.train_df['frames_per_second'].describe()\n        print(f\"   Frame Rates: {fps_stats['min']:.0f}-{fps_stats['max']:.0f} FPS \"\n              f\"(median: {fps_stats['50%']:.0f})\")\n        \n        # Video durations\n        duration_stats = self.train_df['video_duration_sec'].describe()\n        print(f\"   Duration: {duration_stats['min']:.0f}-{duration_stats['max']:.0f}s \"\n              f\"(median: {duration_stats['50%']:.0f}s)\")\n        \n        # Resolution analysis\n        resolution_combo = self.train_df.groupby(['video_width_pix', 'video_height_pix']).size()\n        print(f\"   Resolution combinations: {len(resolution_combo)} unique\")\n        print(f\"   Most common: {resolution_combo.index[0]} ({resolution_combo.iloc[0]} videos)\")\n        \n        # Pixel density\n        pix_stats = self.train_df['pix_per_cm_approx'].describe()\n        print(f\"   Pixel density: {pix_stats['min']:.1f}-{pix_stats['max']:.1f} pix/cm \"\n              f\"(median: {pix_stats['50%']:.1f})\")\n        print()\n        \n        # Arena analysis\n        print(\"🏟️ Arena Configurations:\")\n        arena_shapes = self.train_df['arena_shape'].value_counts()\n        for shape, count in arena_shapes.items():\n            print(f\"   {shape}: {count} ({count/len(self.train_df)*100:.1f}%)\")\n        \n        arena_types = self.train_df['arena_type'].value_counts()\n        print(f\"   Arena types: {', '.join(arena_types.head(3).index.tolist())}\")\n        print()\n        \n        # Mouse demographics\n        print(\"🐭 Mouse Demographics:\")\n        \n        # Count mice per video\n        mouse_counts = []\n        for _, row in self.train_df.iterrows():\n            count = sum([1 for i in range(1, 5) if pd.notna(row[f'mouse{i}_id'])])\n            mouse_counts.append(count)\n        \n        mouse_count_dist = pd.Series(mouse_counts).value_counts().sort_index()\n        for n_mice, count in mouse_count_dist.items():\n            print(f\"   {n_mice} mice: {count} videos ({count/len(self.train_df)*100:.1f}%)\")\n        \n        # Strain distribution\n        all_strains = []\n        for i in range(1, 5):\n            strains = self.train_df[f'mouse{i}_strain'].dropna()\n            all_strains.extend(strains.tolist())\n        \n        if all_strains:\n            strain_counts = pd.Series(all_strains).value_counts()\n            print(f\"   Most common strains: {', '.join(strain_counts.head(3).index.tolist())}\")\n        \n        # Sex distribution\n        all_sexes = []\n        for i in range(1, 5):\n            sexes = self.train_df[f'mouse{i}_sex'].dropna()\n            all_sexes.extend(sexes.tolist())\n        \n        if all_sexes:\n            sex_counts = pd.Series(all_sexes).value_counts()\n            total = sum(sex_counts)\n            print(f\"   Sex distribution: \", end=\"\")\n            for sex, count in sex_counts.items():\n                print(f\"{sex}: {count/total*100:.1f}% \", end=\"\")\n            print()\n        print()\n        \n        self.analysis_results['metadata'] = results\n        \n        # Create visualizations\n        self._plot_metadata_distributions()\n\n    \n    \n    def analyze_tracking_data(self):\n        \"\"\"Analyze pose tracking data quality and patterns\"\"\"\n        print(\"🎯 TRACKING DATA ANALYSIS\")\n        print(\"-\" * 40)\n        \n        if not self.tracking_data:\n            print(\"❌ No tracking data available for analysis\")\n            return\n        \n        results = {}\n        \n        for dataset_name, df in self.tracking_data.items():\n            print(f\"\\n📊 Analyzing {dataset_name}:\")\n            print(f\"   Shape: {df.shape}\")\n            \n            # Basic statistics\n            n_frames = df['video_frame'].nunique()\n            n_mice = df['mouse_id'].nunique()\n            bodyparts = df['bodypart'].unique()\n            \n            print(f\"   Frames: {n_frames}\")\n            print(f\"   Mice: {n_mice}\")\n            print(f\"   Body parts: {len(bodyparts)} ({', '.join(bodyparts[:5])}...)\")\n            \n            # Data quality assessment\n            total_points = len(df)\n            missing_x = df['x'].isna().sum()\n            missing_y = df['y'].isna().sum()\n            \n            print(f\"   Missing coordinates: {max(missing_x, missing_y)}/{total_points} \"\n                  f\"({max(missing_x, missing_y)/total_points*100:.1f}%)\")\n            \n            # Coordinate range analysis\n            x_range = (df['x'].min(), df['x'].max())\n            y_range = (df['y'].min(), df['y'].max())\n            print(f\"   X range: {x_range[0]:.0f} - {x_range[1]:.0f}\")\n            print(f\"   Y range: {y_range[0]:.0f} - {y_range[1]:.0f}\")\n            \n            # Outlier detection (coordinates outside reasonable bounds)\n            outliers_x = ((df['x'] < -50) | (df['x'] > 2000)).sum()\n            outliers_y = ((df['y'] < -50) | (df['y'] > 2000)).sum()\n            total_outliers = max(outliers_x, outliers_y)\n            \n            if total_outliers > 0:\n                print(f\"   ⚠️  Potential outliers: {total_outliers} \"\n                      f\"({total_outliers/total_points*100:.2f}%)\")\n            \n            # Movement analysis (velocity calculation)\n            velocities = []\n            for mouse_id in df['mouse_id'].unique():\n                for bodypart in df['bodypart'].unique():\n                    mouse_bp_data = df[(df['mouse_id'] == mouse_id) & \n                                     (df['bodypart'] == bodypart)].sort_values('video_frame')\n                    \n                    if len(mouse_bp_data) > 1:\n                        dx = mouse_bp_data['x'].diff().dropna()\n                        dy = mouse_bp_data['y'].diff().dropna()\n                        velocity = np.sqrt(dx**2 + dy**2)\n                        velocities.extend(velocity.tolist())\n            \n            if velocities:\n                vel_series = pd.Series(velocities)\n                median_vel = vel_series.median()\n                vel_95 = vel_series.quantile(0.95)  # 95th percentile\n                \n                print(f\"   Velocity stats: median={median_vel:.1f} pix/frame, \"\n                  f\"95th percentile={vel_95:.1f}\")\n                \n                # Identify potential tracking errors (very high velocities)\n                \n                high_vel_threshold = vel_95 * 3\n                high_vel_count = (vel_series > high_vel_threshold).sum()\n                if high_vel_count > 0:\n                    print(f\"   ⚠️  High velocity events: {high_vel_count} \"\n                      f\"(potential tracking errors)\")\n            \n            results[dataset_name] = {\n                'n_frames': n_frames,\n                'n_mice': n_mice,\n                'n_bodyparts': len(bodyparts),\n                'missing_rate': max(missing_x, missing_y)/total_points,\n                'outlier_rate': total_outliers/total_points,\n                'coordinate_ranges': {'x': x_range, 'y': y_range}\n            }\n        \n        self.analysis_results['tracking'] = results\n        \n        # Create tracking visualizations\n        self._plot_tracking_analysis()\n    \n    def analyze_behavioral_annotations(self):\n        \"\"\"Comprehensive behavioral annotation analysis\"\"\"\n        print(\"\\n🎭 BEHAVIORAL ANNOTATION ANALYSIS\")\n        print(\"-\" * 40)\n        \n        if not self.annotation_data:\n            print(\"❌ No annotation data available for analysis\")\n            return\n        \n        results = {}\n        all_behaviors = []\n        all_durations = []\n        \n        for lab, df in self.annotation_data.items():\n            print(f\"\\n📝 Analyzing {lab} annotations:\")\n            print(f\"   Total annotations: {len(df)}\")\n            \n            # Behavior distribution\n            behavior_counts = df['action'].value_counts()\n            print(f\"   Unique behaviors: {len(behavior_counts)}\")\n            print(f\"   Most common: {', '.join(behavior_counts.head(3).index.tolist())}\")\n            \n            # Duration analysis\n            df['duration'] = df['stop_frame'] - df['start_frame']\n            duration_stats = df['duration'].describe()\n            print(f\"   Duration range: {duration_stats['min']:.0f} - {duration_stats['max']:.0f} frames\")\n            print(f\"   Median duration: {duration_stats['50%']:.0f} frames\")\n            \n            # Agent-target relationship analysis\n            self_directed = (df['agent_id'] == df['target_id']).sum()\n            social = len(df) - self_directed\n            print(f\"   Self-directed: {self_directed} ({self_directed/len(df)*100:.1f}%)\")\n            print(f\"   Social interactions: {social} ({social/len(df)*100:.1f}%)\")\n            \n            # Temporal coverage analysis\n            if 'start_frame' in df.columns and 'stop_frame' in df.columns:\n                total_frames_annotated = (df['stop_frame'] - df['start_frame']).sum()\n                print(f\"   Total annotated frames: {total_frames_annotated:,}\")\n            \n            all_behaviors.extend(df['action'].tolist())\n            all_durations.extend(df['duration'].tolist())\n            \n            results[lab] = {\n                'n_annotations': len(df),\n                'n_behaviors': len(behavior_counts),\n                'behavior_distribution': behavior_counts.to_dict(),\n                'duration_stats': duration_stats.to_dict(),\n                'self_directed_pct': self_directed/len(df)*100,\n                'social_pct': social/len(df)*100\n            }\n        \n        # Cross-laboratory analysis\n        print(f\"\\n🔄 CROSS-LABORATORY ANALYSIS:\")\n        \n        # Overall behavior distribution\n        global_behavior_counts = pd.Series(all_behaviors).value_counts()\n        print(f\"   Total unique behaviors across labs: {len(global_behavior_counts)}\")\n        print(f\"   Most frequent globally: {', '.join(global_behavior_counts.head(5).index.tolist())}\")\n        \n        # Behavior consistency across labs\n        behavior_lab_matrix = {}\n        for lab, df in self.annotation_data.items():\n            lab_behaviors = set(df['action'].unique())\n            behavior_lab_matrix[lab] = lab_behaviors\n        \n        # Find common behaviors\n        all_lab_behaviors = set().union(*behavior_lab_matrix.values())\n        common_behaviors = set.intersection(*behavior_lab_matrix.values())\n        \n        print(f\"   Behaviors present in ALL labs: {len(common_behaviors)}\")\n        if common_behaviors:\n            print(f\"   Universal behaviors: {', '.join(sorted(common_behaviors))}\")\n        \n        # Lab-specific behaviors\n        for lab, behaviors in behavior_lab_matrix.items():\n            unique_to_lab = behaviors - (all_lab_behaviors - behaviors)\n            if unique_to_lab:\n                print(f\"   {lab} unique behaviors: {', '.join(sorted(unique_to_lab))}\")\n        \n        # Duration distribution analysis\n        if all_durations:\n            duration_stats = pd.Series(all_durations).describe()\n            print(f\"\\n⏱️  GLOBAL DURATION ANALYSIS:\")\n            print(f\"   Mean duration: {duration_stats['mean']:.1f} frames\")\n            print(f\"   Median duration: {duration_stats['50%']:.1f} frames\")\n            print(f\"   90th percentile: {duration_stats['75%']:.1f} frames\")\n            \n            # Identify very short and very long behaviors\n            short_threshold = duration_stats['25%']\n            long_threshold = duration_stats['75%']\n            \n            short_behaviors = [dur for dur in all_durations if dur < short_threshold]\n            long_behaviors = [dur for dur in all_durations if dur > long_threshold]\n            \n            print(f\"   Short behaviors (<{short_threshold:.0f}f): {len(short_behaviors)} \"\n                  f\"({len(short_behaviors)/len(all_durations)*100:.1f}%)\")\n            print(f\"   Long behaviors (>{long_threshold:.0f}f): {len(long_behaviors)} \"\n                  f\"({len(long_behaviors)/len(all_durations)*100:.1f}%)\")\n        \n        self.analysis_results['annotations'] = results\n        self.analysis_results['global_behavior_stats'] = {\n            'total_behaviors': len(global_behavior_counts),\n            'behavior_distribution': global_behavior_counts.to_dict(),\n            'common_behaviors': list(common_behaviors),\n            'duration_stats': duration_stats.to_dict() if all_durations else {}\n        }\n        \n        # Create behavioral analysis visualizations\n        #self._plot_behavioral_analysis()\n    \n    def analyze_data_quality_issues(self):\n        \"\"\"Identify and analyze data quality issues\"\"\"\n        print(\"\\n🔍 DATA QUALITY ASSESSMENT\")\n        print(\"-\" * 40)\n        \n        issues = []\n        \n        # Metadata quality issues\n        print(\"📋 Metadata Quality Issues:\")\n        \n        # Missing values in metadata\n        missing_cols = self.train_df.isnull().sum()\n        critical_missing = missing_cols[missing_cols > 0]\n        \n        if len(critical_missing) > 0:\n            print(\"   Missing values detected:\")\n            for col, count in critical_missing.items():\n                pct = count / len(self.train_df) * 100\n                print(f\"     {col}: {count} ({pct:.1f}%)\")\n                if pct > 10:\n                    issues.append(f\"High missing rate in {col}: {pct:.1f}%\")\n        else:\n            print(\"   ✅ No missing values in metadata\")\n        \n        # Inconsistent frame rates\n        fps_values = self.train_df['frames_per_second'].unique()\n        if len(fps_values) > 6:\n            print(f\"   ⚠️  High FPS diversity: {len(fps_values)} different frame rates\")\n            issues.append(f\"Frame rate standardization needed: {len(fps_values)} variants\")\n        \n        # Extreme values detection\n        duration_q99 = self.train_df['video_duration_sec'].quantile(0.75)\n        extreme_durations = (self.train_df['video_duration_sec'] > duration_q99 * 3).sum()\n        if extreme_durations > 0:\n            print(f\"   ⚠️  Extreme duration values: {extreme_durations} videos\")\n            issues.append(f\"Extreme video durations detected: {extreme_durations} cases\")\n        \n        # Resolution consistency\n        resolution_combos = self.train_df.groupby(['video_width_pix', 'video_height_pix']).size()\n        if len(resolution_combos) > 8:\n            print(f\"   ⚠️  High resolution diversity: {len(resolution_combos)} combinations\")\n            issues.append(f\"Resolution standardization needed: {len(resolution_combos)} variants\")\n        \n        # Tracking data quality issues\n        if self.tracking_data:\n            print(\"\\n📊 Tracking Data Quality Issues:\")\n            \n            for dataset_name, df in self.tracking_data.items():\n                print(f\"\\n   {dataset_name}:\")\n                \n                # Missing coordinates\n                missing_rate = df[['x', 'y']].isnull().any(axis=1).sum() / len(df)\n                if missing_rate > 0.05:  # More than 5% missing\n                    print(f\"     ⚠️  High missing coordinate rate: {missing_rate*100:.1f}%\")\n                    issues.append(f\"{dataset_name}: High missing coordinates ({missing_rate*100:.1f}%)\")\n                \n                # Coordinate outliers\n                x_outliers = ((df['x'] < -100) | (df['x'] > 2000)).sum()\n                y_outliers = ((df['y'] < -100) | (df['y'] > 2000)).sum()\n                outlier_rate = max(x_outliers, y_outliers) / len(df)\n                \n                if outlier_rate > 0.01:  # More than 1% outliers\n                    print(f\"     ⚠️  Coordinate outliers detected: {outlier_rate*100:.2f}%\")\n                    issues.append(f\"{dataset_name}: Coordinate outliers ({outlier_rate*100:.2f}%)\")\n                \n                # Tracking consistency\n                expected_points = df['video_frame'].nunique() * df['mouse_id'].nunique() * df['bodypart'].nunique()\n                actual_points = len(df)\n                completeness = actual_points / expected_points\n                \n                if completeness < 0.95:  # Less than 95% complete\n                    print(f\"     ⚠️  Incomplete tracking: {completeness*100:.1f}% completeness\")\n                    issues.append(f\"{dataset_name}: Incomplete tracking ({completeness*100:.1f}%)\")\n                else:\n                    print(f\"     ✅ Tracking completeness: {completeness*100:.1f}%\")\n        \n        # Annotation quality issues\n        if self.annotation_data:\n            print(\"\\n📝 Annotation Quality Issues:\")\n            \n            for lab, df in self.annotation_data.items():\n                print(f\"\\n   {lab}:\")\n                \n                # Temporal overlap detection\n                overlaps = 0\n                for agent_id in df['agent_id'].unique():\n                    agent_annotations = df[df['agent_id'] == agent_id].sort_values('start_frame')\n                    for i in range(len(agent_annotations) - 1):\n                        current_end = agent_annotations.iloc[i]['stop_frame']\n                        next_start = agent_annotations.iloc[i + 1]['start_frame']\n                        if current_end > next_start:\n                            overlaps += 1\n                \n                if overlaps > 0:\n                    overlap_rate = overlaps / len(df)\n                    print(f\"     ⚠️  Temporal overlaps: {overlaps} ({overlap_rate*100:.1f}%)\")\n                    issues.append(f\"{lab}: Temporal annotation overlaps ({overlaps} cases)\")\n                \n                # Invalid frame ranges\n                invalid_ranges = (df['stop_frame'] <= df['start_frame']).sum()\n                if invalid_ranges > 0:\n                    print(f\"     ⚠️  Invalid frame ranges: {invalid_ranges}\")\n                    issues.append(f\"{lab}: Invalid annotation ranges ({invalid_ranges} cases)\")\n                \n                # Very short annotations (potential errors)\n                df_temp = df.copy()\n                df_temp['duration'] = df_temp['stop_frame'] - df_temp['start_frame']\n                very_short = (df_temp['duration'] <= 2).sum()  # 2 frames or less\n                if very_short > len(df) * 0.1:  # More than 10% very short\n                    print(f\"     ⚠️  Many very short annotations: {very_short} \"\n                          f\"({very_short/len(df)*100:.1f}%)\")\n                    issues.append(f\"{lab}: Excessive short annotations ({very_short/len(df)*100:.1f}%)\")\n                \n                # Agent/target ID consistency\n                max_agent = df['agent_id'].max()\n                max_target = df['target_id'].max()\n                if max_agent != max_target:\n                    print(f\"     ⚠️  Agent/target ID mismatch: max_agent={max_agent}, max_target={max_target}\")\n                    issues.append(f\"{lab}: Agent/target ID inconsistency\")\n                \n                if len(df) > 0:\n                    print(f\"     ✅ Total annotations: {len(df)}\")\n        \n        # Summary\n        print(f\"\\n📋 QUALITY ASSESSMENT SUMMARY:\")\n        print(f\"   Total issues identified: {len(issues)}\")\n        \n        if issues:\n            print(\"   Critical issues to address:\")\n            for i, issue in enumerate(issues[:10], 1):  # Show top 10 issues\n                print(f\"     {i}. {issue}\")\n            \n            if len(issues) > 10:\n                print(f\"     ... and {len(issues)-10} other issues\")\n        else:\n            print(\"   ✅ No critical data quality issues detected\")\n        \n        self.analysis_results['quality_issues'] = issues\n    \n    def analyze_cross_laboratory_patterns(self):\n        \"\"\"Analyze patterns and differences across laboratories\"\"\"\n        print(\"\\n🔬 CROSS-LABORATORY PATTERN ANALYSIS\")\n        print(\"-\" * 40)\n        \n        # Group labs by type\n        public_labs = ['CalMS21_task1', 'CalMS21_task2', 'CalMS21_supplemental', \n                      'CRIM13', 'MABe22_keypoints', 'MABe22_movies']\n        anonymous_labs = [lab for lab in self.train_df['lab_id'].unique() \n                         if lab not in public_labs]\n        \n        print(f\"📊 Laboratory Categories:\")\n        print(f\"   Public datasets: {len(public_labs)} labs\")\n        print(f\"   Anonymous contributors: {len(anonymous_labs)} labs\")\n        \n        # Technical specification patterns\n        print(f\"\\n⚙️ Technical Patterns by Lab Type:\")\n        \n        # Analyze public vs anonymous lab differences\n        public_data = self.train_df[self.train_df['lab_id'].isin(public_labs)]\n        anonymous_data = self.train_df[self.train_df['lab_id'].isin(anonymous_labs)]\n        \n        if len(public_data) > 0 and len(anonymous_data) > 0:\n            # Frame rate comparison\n            public_fps_median = public_data['frames_per_second'].median()\n            anonymous_fps_median = anonymous_data['frames_per_second'].median()\n            print(f\"   Frame rates - Public: {public_fps_median:.0f} FPS, \"\n                  f\"Anonymous: {anonymous_fps_median:.0f} FPS\")\n            \n            # Duration comparison\n            public_dur_median = public_data['video_duration_sec'].median()\n            anonymous_dur_median = anonymous_data['video_duration_sec'].median()\n            print(f\"   Duration - Public: {public_dur_median:.0f}s, \"\n                  f\"Anonymous: {anonymous_dur_median:.0f}s\")\n            \n            # Resolution comparison\n            public_res = public_data.groupby(['video_width_pix', 'video_height_pix']).size()\n            anonymous_res = anonymous_data.groupby(['video_width_pix', 'video_height_pix']).size()\n            print(f\"   Resolution variants - Public: {len(public_res)}, \"\n                  f\"Anonymous: {len(anonymous_res)}\")\n        \n        # Behavioral annotation patterns across labs\n        if self.annotation_data:\n            print(f\"\\n🎭 Behavioral Annotation Patterns:\")\n            \n            lab_behavior_stats = {}\n            for lab, df in self.annotation_data.items():\n                behaviors = df['action'].value_counts()\n                durations = (df['stop_frame'] - df['start_frame']).describe()\n                \n                lab_behavior_stats[lab] = {\n                    'n_behaviors': len(behaviors),\n                    'most_common': behaviors.index[0] if len(behaviors) > 0 else None,\n                    'median_duration': durations['50%'],\n                    'n_annotations': len(df)\n                }\n            \n            # Find patterns\n            behavior_counts = [stats['n_behaviors'] for stats in lab_behavior_stats.values()]\n            annotation_counts = [stats['n_annotations'] for stats in lab_behavior_stats.values()]\n            \n            print(f\"   Behaviors per lab: {min(behavior_counts)}-{max(behavior_counts)} \"\n                  f\"(median: {np.median(behavior_counts):.0f})\")\n            print(f\"   Annotations per lab: {min(annotation_counts)}-{max(annotation_counts)} \"\n                  f\"(median: {np.median(annotation_counts):.0f})\")\n            \n            # Most common behaviors across labs\n            all_most_common = [stats['most_common'] for stats in lab_behavior_stats.values() \n                             if stats['most_common'] is not None]\n            if all_most_common:\n                most_common_counter = Counter(all_most_common)\n                print(f\"   Most frequently dominant behavior: {most_common_counter.most_common(1)[0]}\")\n        \n        # Tracking method analysis\n        print(f\"\\n🎯 Tracking Method Distribution:\")\n        tracking_methods = self.train_df['tracking_method'].value_counts()\n        for method, count in tracking_methods.items():\n            print(f\"   {method}: {count} videos ({count/len(self.train_df)*100:.1f}%)\")\n        \n        # Lab-specific specializations\n        print(f\"\\n🔬 Lab Specializations:\")\n        \n        # Arena type preferences\n        lab_arena_prefs = {}\n        for lab in self.train_df['lab_id'].unique():\n            lab_data = self.train_df[self.train_df['lab_id'] == lab]\n            arena_types = lab_data['arena_type'].value_counts()\n            if len(arena_types) > 0:\n                lab_arena_prefs[lab] = arena_types.index[0]\n        \n        arena_specialization = Counter(lab_arena_prefs.values())\n        for arena_type, count in arena_specialization.most_common():\n            labs_with_type = [lab for lab, pref in lab_arena_prefs.items() if pref == arena_type]\n            print(f\"   {arena_type}: {count} labs specialize \"\n                  f\"({', '.join(labs_with_type[:3])}{'...' if len(labs_with_type) > 3 else ''})\")\n        \n        # Mouse strain preferences\n        lab_strain_prefs = {}\n        for lab in self.train_df['lab_id'].unique():\n            lab_data = self.train_df[self.train_df['lab_id'] == lab]\n            all_strains = []\n            for i in range(1, 5):\n                strains = lab_data[f'mouse{i}_strain'].dropna()\n                all_strains.extend(strains.tolist())\n            \n            if all_strains:\n                strain_counts = Counter(all_strains)\n                lab_strain_prefs[lab] = strain_counts.most_common(1)[0][0]\n        \n        strain_popularity = Counter(lab_strain_prefs.values())\n        print(f\"\\n🐭 Strain Preferences:\")\n        for strain, count in strain_popularity.most_common():\n            print(f\"   {strain}: preferred by {count} labs\")\n    \n    def generate_recommendations(self):\n        \"\"\"Generate strategic recommendations based on analysis\"\"\"\n        print(\"\\n💡 STRATEGIC RECOMMENDATIONS\")\n        print(\"-\" * 40)\n        \n        recommendations = {\n            'data_preprocessing': [],\n            'feature_engineering': [],\n            'modeling_strategy': [],\n            'evaluation_approach': [],\n            'risk_mitigation': []\n        }\n        \n        # Data preprocessing recommendations\n        print(\"🔧 Data Preprocessing Recommendations:\")\n        \n        # Frame rate standardization\n        fps_values = self.train_df['frames_per_second'].unique()\n        if len(fps_values) > 5:\n            rec = f\"Implement temporal resampling: {len(fps_values)} different frame rates detected\"\n            print(f\"   1. {rec}\")\n            recommendations['data_preprocessing'].append(rec)\n        \n        # Coordinate normalization\n        if self.tracking_data:\n            rec = \"Implement coordinate normalization across laboratories using pix_per_cm scaling\"\n            print(f\"   2. {rec}\")\n            recommendations['data_preprocessing'].append(rec)\n            \n            # Quality filtering\n            rec = \"Implement robust outlier detection and filtering for tracking coordinates\"\n            print(f\"   3. {rec}\")\n            recommendations['data_preprocessing'].append(rec)\n        \n        # Missing data handling\n        if any('High missing' in issue for issue in self.analysis_results.get('quality_issues', [])):\n            rec = \"Develop sophisticated imputation strategies for missing tracking data\"\n            print(f\"   4. {rec}\")\n            recommendations['data_preprocessing'].append(rec)\n        \n        # Feature engineering recommendations\n        print(f\"\\n🛠️ Feature Engineering Recommendations:\")\n        \n        rec = \"Create multi-scale temporal features (short, medium, long-term patterns)\"\n        print(f\"   1. {rec}\")\n        recommendations['feature_engineering'].append(rec)\n        \n        rec = \"Develop relative positioning features between mice for social interaction modeling\"\n        print(f\"   2. {rec}\")\n        recommendations['feature_engineering'].append(rec)\n        \n        rec = \"Implement velocity and acceleration features for movement characterization\"\n        print(f\"   3. {rec}\")\n        recommendations['feature_engineering'].append(rec)\n        \n        rec = \"Create arena-normalized features to handle different experimental setups\"\n        print(f\"   4. {rec}\")\n        recommendations['feature_engineering'].append(rec)\n        \n        # Modeling strategy recommendations\n        print(f\"\\n🤖 Modeling Strategy Recommendations:\")\n        \n        # Domain adaptation\n        lab_count = len(self.train_df['lab_id'].unique())\n        rec = f\"Implement domain adaptation techniques for {lab_count} laboratories\"\n        print(f\"   1. {rec}\")\n        recommendations['modeling_strategy'].append(rec)\n        \n        # Hierarchical modeling\n        if self.annotation_data:\n            total_behaviors = len(set().union(*[df['action'].unique() \n                                              for df in self.annotation_data.values()]))\n            rec = f\"Use hierarchical multi-task learning for {total_behaviors} behavior classes\"\n            print(f\"   2. {rec}\")\n            recommendations['modeling_strategy'].append(rec)\n        \n        # Temporal modeling\n        rec = \"Implement attention-based temporal modeling for variable-length sequences\"\n        print(f\"   3. {rec}\")\n        recommendations['modeling_strategy'].append(rec)\n        \n        # Ensemble approach\n        rec = \"Develop ensemble methods combining laboratory-specific and global models\"\n        print(f\"   4. {rec}\")\n        recommendations['modeling_strategy'].append(rec)\n        \n        # Evaluation approach recommendations\n        print(f\"\\n📊 Evaluation Approach Recommendations:\")\n        \n        rec = \"Use stratified validation ensuring all laboratories represented in folds\"\n        print(f\"   1. {rec}\")\n        recommendations['evaluation_approach'].append(rec)\n        \n        if self.annotation_data:\n            # Check for class imbalance\n            all_behaviors = []\n            for df in self.annotation_data.values():\n                all_behaviors.extend(df['action'].tolist())\n            \n            behavior_counts = Counter(all_behaviors)\n            imbalance_ratio = behavior_counts.most_common(1)[0][1] / behavior_counts.most_common()[-1][1]\n            \n            if imbalance_ratio > 10:\n                rec = f\"Implement class-balanced metrics due to high imbalance (ratio: {imbalance_ratio:.1f}:1)\"\n                print(f\"   2. {rec}\")\n                recommendations['evaluation_approach'].append(rec)\n        \n        rec = \"Develop temporal boundary accuracy metrics for behavior segmentation\"\n        print(f\"   3. {rec}\")\n        recommendations['evaluation_approach'].append(rec)\n        \n        rec = \"Create cross-laboratory generalization benchmarks\"\n        print(f\"   4. {rec}\")\n        recommendations['evaluation_approach'].append(rec)\n        \n        # Risk mitigation recommendations\n        print(f\"\\n⚠️ Risk Mitigation Recommendations:\")\n        \n        quality_issues = self.analysis_results.get('quality_issues', [])\n        if quality_issues:\n            rec = f\"Address {len(quality_issues)} identified data quality issues before modeling\"\n            print(f\"   1. {rec}\")\n            recommendations['risk_mitigation'].append(rec)\n        \n        rec = \"Implement robust validation pipeline to detect overfitting to specific labs\"\n        print(f\"   2. {rec}\")\n        recommendations['risk_mitigation'].append(rec)\n        \n        rec = \"Develop fallback strategies for handling novel laboratory setups in test data\"\n        print(f\"   3. {rec}\")\n        recommendations['risk_mitigation'].append(rec)\n        \n        rec = \"Create interpretability tools to validate biological plausibility of predictions\"\n        print(f\"   4. {rec}\")\n        recommendations['risk_mitigation'].append(rec)\n        \n        self.analysis_results['recommendations'] = recommendations\n        \n        return recommendations\n    \n    def _plot_metadata_distributions(self):\n        \"\"\"Create comprehensive metadata visualization plots\"\"\"\n        fig, axes = plt.subplots(2, 3, figsize=(20, 12))\n        fig.suptitle('MABe Dataset - Metadata Distributions', fontsize=16, fontweight='bold')\n        \n        # Laboratory distribution\n        lab_counts = self.train_df['lab_id'].value_counts().head(15)\n        axes[0, 0].barh(range(len(lab_counts)), lab_counts.values)\n        axes[0, 0].set_yticks(range(len(lab_counts)))\n        axes[0, 0].set_yticklabels(lab_counts.index, fontsize=8)\n        axes[0, 0].set_xlabel('Number of Videos')\n        axes[0, 0].set_title('Top 15 Laboratory Contributions')\n        axes[0, 0].grid(True, alpha=0.3)\n        \n        # Frame rate distribution\n        axes[0, 1].hist(self.train_df['frames_per_second'], bins=20, alpha=0.7, edgecolor='black')\n        axes[0, 1].set_xlabel('Frames Per Second')\n        axes[0, 1].set_ylabel('Frequency')\n        axes[0, 1].set_title('Frame Rate Distribution')\n        axes[0, 1].grid(True, alpha=0.3)\n        \n        # Video duration distribution (log scale)\n        axes[0, 2].hist(np.log10(self.train_df['video_duration_sec']), bins=30, alpha=0.7, edgecolor='black')\n        axes[0, 2].set_xlabel('Log10(Duration in seconds)')\n        axes[0, 2].set_ylabel('Frequency')\n        axes[0, 2].set_title('Video Duration Distribution')\n        axes[0, 2].grid(True, alpha=0.3)\n        \n        # Arena shape distribution\n        arena_shapes = self.train_df['arena_shape'].value_counts()\n        axes[1, 0].pie(arena_shapes.values, labels=arena_shapes.index, autopct='%1.1f%%')\n        axes[1, 0].set_title('Arena Shape Distribution')\n        \n        # Pixel density distribution\n        axes[1, 1].hist(self.train_df['pix_per_cm_approx'], bins=25, alpha=0.7, edgecolor='black')\n        axes[1, 1].set_xlabel('Pixels per cm (approx)')\n        axes[1, 1].set_ylabel('Frequency')\n        axes[1, 1].set_title('Pixel Density Distribution')\n        axes[1, 1].grid(True, alpha=0.3)\n        \n        # Mouse count per video\n        mouse_counts = []\n        for _, row in self.train_df.iterrows():\n            count = sum([1 for i in range(1, 5) if pd.notna(row[f'mouse{i}_id'])])\n            mouse_counts.append(count)\n        \n        mouse_count_dist = pd.Series(mouse_counts).value_counts().sort_index()\n        axes[1, 2].bar(mouse_count_dist.index, mouse_count_dist.values, alpha=0.7)\n        axes[1, 2].set_xlabel('Number of Mice per Video')\n        axes[1, 2].set_ylabel('Frequency')\n        axes[1, 2].set_title('Mice per Video Distribution')\n        axes[1, 2].grid(True, alpha=0.3)\n        \n        plt.tight_layout()\n        plt.savefig('mabe_metadata_analysis.png', dpi=300, bbox_inches='tight')\n        plt.show()\n    \n    def _plot_tracking_analysis(self):\n        \"\"\"Create tracking data analysis visualizations\"\"\"\n        if not self.tracking_data:\n            return\n        \n        # Get first dataset for detailed analysis\n        dataset_name = list(self.tracking_data.keys())[0]\n        df = self.tracking_data[dataset_name]\n        \n        fig, axes = plt.subplots(2, 2, figsize=(16, 12))\n        fig.suptitle(f'Tracking Data Analysis - {dataset_name}', fontsize=14, fontweight='bold')\n        \n        # Coordinate scatter plot (sample frames)\n        sample_frames = df['video_frame'].unique()[:50:5]  # Every 5th frame from first 50\n        sample_data = df[df['video_frame'].isin(sample_frames)]\n        \n        for mouse_id in sample_data['mouse_id'].unique():\n            mouse_data = sample_data[sample_data['mouse_id'] == mouse_id]\n            axes[0, 0].scatter(mouse_data['x'], mouse_data['y'], \n                              label=f'Mouse {mouse_id}', alpha=0.6, s=10)\n        \n        axes[0, 0].set_xlabel('X coordinate (pixels)')\n        axes[0, 0].set_ylabel('Y coordinate (pixels)')\n        axes[0, 0].set_title('Mouse Trajectories (Sample)')\n        axes[0, 0].legend()\n        axes[0, 0].grid(True, alpha=0.3)\n        \n        # Missing data pattern\n        missing_by_frame = df.groupby('video_frame')[['x', 'y']].apply(\n            lambda x: x.isnull().any(axis=1).sum()\n        )\n        axes[0, 1].plot(missing_by_frame.index[::10], missing_by_frame.values[::10])\n        axes[0, 1].set_xlabel('Frame Number')\n        axes[0, 1].set_ylabel('Missing Coordinates Count')\n        axes[0, 1].set_title('Missing Data Over Time')\n        axes[0, 1].grid(True, alpha=0.3)\n        \n        # Body part tracking completeness\n        bodypart_completeness = df.groupby('bodypart').apply(\n            lambda x: 1 - x[['x', 'y']].isnull().any(axis=1).sum() / len(x)\n        ).sort_values(ascending=False)\n        \n        axes[1, 0].barh(range(len(bodypart_completeness)), bodypart_completeness.values)\n        axes[1, 0].set_yticks(range(len(bodypart_completeness)))\n        axes[1, 0].set_yticklabels(bodypart_completeness.index, fontsize=8)\n        axes[1, 0].set_xlabel('Tracking Completeness')\n        axes[1, 0].set_title('Body Part Tracking Quality')\n        axes[1, 0].grid(True, alpha=0.3)\n        \n        # Velocity distribution\n        velocities = []\n        for mouse_id in df['mouse_id'].unique():\n            for bodypart in df['bodypart'].unique():\n                mouse_bp_data = df[(df['mouse_id'] == mouse_id) & \n                                 (df['bodypart'] == bodypart)].sort_values('video_frame')\n                \n                if len(mouse_bp_data) > 1:\n                    dx = mouse_bp_data['x'].diff().dropna()\n                    dy = mouse_bp_data['y'].diff().dropna()\n                    velocity = np.sqrt(dx**2 + dy**2)\n                    velocities.extend(velocity[velocity < 50].tolist())  # Cap at 50 for visualization\n        \n        if velocities:\n            axes[1, 1].hist(velocities, bins=50, alpha=0.7, edgecolor='black')\n            axes[1, 1].set_xlabel('Velocity (pixels/frame)')\n            axes[1, 1].set_ylabel('Frequency') \n            axes[1, 1].set_title('Movement Velocity Distribution')\n            axes[1, 1].set_yscale('log')\n            axes[1, 1].grid(True, alpha=0.3)\n        \n        plt.tight_layout()\n        plt.savefig('mabe_tracking_analysis.png', dpi=300, bbox_inches='tight')\n        plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-19T12:20:24.939599Z","iopub.execute_input":"2025-09-19T12:20:24.939903Z","iopub.status.idle":"2025-09-19T12:20:25.094347Z","shell.execute_reply.started":"2025-09-19T12:20:24.939881Z","shell.execute_reply":"2025-09-19T12:20:25.093315Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"eda = MABeEDAAnalyzer()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-19T12:20:27.010834Z","iopub.execute_input":"2025-09-19T12:20:27.011205Z","iopub.status.idle":"2025-09-19T12:20:27.017064Z","shell.execute_reply.started":"2025-09-19T12:20:27.011175Z","shell.execute_reply":"2025-09-19T12:20:27.015854Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# 3️⃣ Load the dataset (or generate mock data if files not present)\neda.load_data()\n\n# 4️⃣ Run metadata analysis\neda.analyze_metadata_distribution()\n\n# 5️⃣ Analyze tracking data quality and patterns\neda.analyze_tracking_data()\n\n# 6️⃣ Analyze behavioral annotation patterns\neda.analyze_behavioral_annotations()\n\n# 7️⃣ Check for data quality issues\neda.analyze_data_quality_issues()\n\n# 8️⃣ Analyze cross-laboratory patterns\neda.analyze_cross_laboratory_patterns()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-19T12:20:29.003908Z","iopub.execute_input":"2025-09-19T12:20:29.004273Z","iopub.status.idle":"2025-09-19T12:20:56.829526Z","shell.execute_reply.started":"2025-09-19T12:20:29.004228Z","shell.execute_reply":"2025-09-19T12:20:56.828631Z"}},"outputs":[],"execution_count":null}]}