{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":102335,"databundleVersionId":12518947,"sourceType":"competition"},{"sourceId":12139340,"sourceType":"datasetVersion","datasetId":7645099},{"sourceId":12293285,"sourceType":"datasetVersion","datasetId":7748073},{"sourceId":12328761,"sourceType":"datasetVersion","datasetId":7771623},{"sourceId":12411879,"sourceType":"datasetVersion","datasetId":7827797},{"sourceId":12573306,"sourceType":"datasetVersion","datasetId":7932089},{"sourceId":12723935,"sourceType":"datasetVersion","datasetId":7869970},{"sourceId":240649816,"sourceType":"kernelVersion"},{"sourceId":245926974,"sourceType":"kernelVersion"},{"sourceId":246893721,"sourceType":"kernelVersion"},{"sourceId":248244433,"sourceType":"kernelVersion"},{"sourceId":251413288,"sourceType":"kernelVersion"},{"sourceId":470587,"sourceType":"modelInstanceVersion","isSourceIdPinned":false,"modelInstanceId":379625,"modelId":398856},{"sourceId":471764,"sourceType":"modelInstanceVersion","isSourceIdPinned":false,"modelInstanceId":380358,"modelId":400086}],"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true},"papermill":{"default_parameters":{},"duration":294.145223,"end_time":"2025-08-22T19:28:13.914229","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2025-08-22T19:23:19.769006","version":"2.6.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"bcaa258e-bf9a-44df-958a-d6307ab70e6d","cell_type":"markdown","source":"# Code Modification\n\nComprehensive Explanation of the CMI Behavior Detection Code\nThis is a sophisticated memory-optimized implementation for detecting human behavior using sensor data. Let me break down this complex code in detail:\n\n1. Overview and Architecture\nThis code implements a complete machine learning pipeline for classifying human behaviors (4 classes) using sensor data from accelerometers, gyroscopes, thermal sensors, and time-of-flight sensors. The key innovation is its comprehensive memory management system that allows it to handle large datasets efficiently.\n\n2. Detailed Component Breakdown\n2.1 Imports and Setup\nStandard libraries: os, gc, json, numpy, pandas, etc.\n\nMachine learning: scikit-learn for preprocessing and evaluation\n\nDeep learning: TensorFlow/Keras for model building\n\nMemory management: psutil for memory monitoring\n\nReproducibility: Fixed random seeds across all libraries\n\nVisualization: matplotlib and seaborn for plotting\n\n2.2 Memory Management System (MemoryManager class)\nThis is the core innovation of the implementation:\n\npython\nclass MemoryManager:\n    \"\"\"Comprehensive memory management and monitoring system\"\"\"\nKey Features:\n\nReal-time monitoring: Tracks process and system memory usage\n\nThreshold-based cleanup: Automatically triggers garbage collection when memory usage exceeds a threshold (default: 85%)\n\nMultiple cleanup strategies:\n\nClear TensorFlow session\n\nForce garbage collection\n\nClear matplotlib cache\n\nDecorator pattern: @memory_check_decorator wraps functions with memory monitoring\n\nContext manager: @contextmanager for scoped memory monitoring\n\nStatistics tracking: Monitors peak memory usage and cleanup count\n\n2.3 Configuration\nThe get_config() function provides optimized hyperparameters:\n\nReduced sequence length (64 instead of 128)\n\nIncreased stride (2) to reduce number of sequences\n\nSmaller batch size (16)\n\nFewer cross-validation folds (3)\n\nMemory threshold setting (80%)\n\n2.4 Memory-Optimized Data Loading\nChunked CSV Loading:\n\npython\ndef load_csv_data_chunked(file_path: str, chunk_size: int = 10000)\nReads large CSV files in manageable chunks\n\nPeriodically checks memory status during loading\n\nConcatenates chunks only at the end\n\nFeature Extraction:\n\npython\ndef extract_sensor_features_optimized(df: pd.DataFrame)\nProcesses sensor data in priority order (accelerometer > rotation > thermal > ToF)\n\nLimits the number of thermal (20) and ToF (30) sensors used\n\nProcesses data in chunks with periodic memory checks\n\nHandles NaN and infinite values\n\n2.5 Model Architecture\nThe build_memory_efficient_model() function creates a custom neural network:\n\nArchitecture Features:\n\nInput layer: Accepts sensor sequences\n\nGaussian noise: Regularization (0.005 noise factor)\n\nConvolutional blocks: For feature extraction\n\nResidual connections: Lightweight residual blocks for better gradient flow\n\nBidirectional GRU: For sequence modeling (more memory-efficient than LSTM)\n\nClassification head: Dense layers with dropout for final prediction\n\nMemory Optimization Techniques:\n\nSmaller filter sizes (32, 64 instead of 64, 128)\n\nStrategic use of dropout for regularization\n\nGRU instead of LSTM for recurrence (fewer parameters)\n\nL2 regularization to prevent overfitting\n\n2.6 Enhanced Callbacks\nPeakAccuracyTracker:\n\nTracks both training and validation accuracy\n\nSaves model at peak validation performance\n\nProvides detailed accuracy history and visualization\n\nMemoryMonitorCallback:\n\nMonitors memory during training\n\nTriggers cleanup when memory usage is high\n\nChecks memory at specified intervals\n\n2.7 Memory-Efficient Data Generator\npython\nclass MemoryEfficientDataGenerator(Sequence):\nInherits from Keras Sequence class for efficient loading\n\nImplements __getitem__ to load batches on demand\n\nUses explicit copying to manage memory\n\nPeriodic garbage collection at epoch end\n\n2.8 Training Pipeline\nSequence Preparation:\n\npython\ndef prepare_sequences_optimized(X, y, config)\nCreates sliding window sequences from time series data\n\nPre-allocates arrays to avoid fragmentation\n\nProcesses in chunks with memory monitoring\n\nTraining Function:\n\npython\ndef train_model_with_validation_optimized(X_train, y_train, X_val, y_val, config)\nComprehensive memory management throughout training\n\nUses all custom callbacks for monitoring\n\nClears intermediate variables to free memory\n\n2.9 Main Pipeline\nThe main_training_pipeline_optimized() function orchestrates the complete process:\n\nMemory initialization: Starts memory tracking\n\nData loading: With memory-managed chunk loading\n\nFeature extraction: Optimized sensor processing\n\nNormalization: Chunked standardization for large datasets\n\nData splitting: Stratified train-validation split\n\nModel training: With comprehensive memory management\n\nArtifact saving: Model, scaler, label encoder, and history\n\n2.10 Evaluation and Prediction\nModel Evaluation:\n\nComprehensive performance metrics\n\nTest set evaluation if available\n\nMemory-managed prediction in batches\n\nDetailed classification reports and confusion matrices\n\nPrediction Function:\n\npython\ndef predict_behavior_sequences(model, X_new, scaler, label_encoder, config)\nProcesses new data through the same pipeline\n\nMakes predictions in batches to manage memory\n\nProvides confidence scores and summary statistics\n\n2.11 Utility Functions\nArtifact Management:\n\nsave_training_artifacts(): Saves model, preprocessing objects, config, and history\n\nload_training_artifacts(): Loads previously saved artifacts for inference\n\n3. Memory Optimization Techniques\nThis implementation uses multiple sophisticated memory management strategies:\n\nChunked Processing: All data operations process in manageable chunks\n\nEager Cleanup: Immediate deletion of temporary variables and garbage collection\n\nSequential Loading: Data generators that load batches on demand\n\nModel Optimization: Architecturally efficient neural network design\n\nMonitoring and Intervention: Real-time memory usage tracking with automatic cleanup\n\nEfficient Data Types: Use of float32 instead of float64 where possible\n\nStrategic Caching: Clearing TensorFlow session and matplotlib cache\n\n4. Execution Flow\nWhen run as a script, the code:\n\nInitializes memory management\n\nRuns the complete training pipeline\n\nEvaluates model performance\n\nSaves all artifacts\n\nPerforms final cleanup and reports memory usage\n\n5. Key Innovations\nComprehensive Memory Awareness: Unlike typical ML code, this implementation is designed with memory constraints as a first-class concern.\n\nMulti-level Optimization: Implements optimization at the data loading, feature extraction, model architecture, and training process levels.\n\nSelf-monitoring: The code tracks its own memory usage and adapts its behavior accordingly.\n\nRobust Error Handling: Includes extensive exception handling for memory-related errors.\n\nReproducibility: Maintains detailed history and artifacts for reproducible results.\n\nThis implementation represents a production-ready approach to handling large sensor datasets on memory-constrained systems, making it suitable for deployment in resource-limited environments or for processing very large datasets that wouldn't fit in memory using conventional approaches.","metadata":{}},{"id":"368ea23a-5dc5-4c17-a4d4-09bd1b1f7cb9","cell_type":"code","source":"# ==============================================================================\n# CMI Detect Behavior with Sensor Data - Memory-Optimized Implementation\n# Enhanced with robust memory management and resource optimization\n# ==============================================================================\n\n# 1. Imports and Setup\n# ==============================================================================\n\nimport os\nimport gc\nimport json\nimport joblib\nimport numpy as np\nimport pandas as pd\nimport polars as pl\nimport random\nimport copy\nimport warnings\nimport psutil\nfrom pathlib import Path\nfrom typing import Dict, List, Tuple, Any\nfrom contextlib import contextmanager\n\nwarnings.filterwarnings(\"ignore\")\n\n# Standard libraries\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.model_selection import train_test_split, StratifiedGroupKFold\nfrom sklearn.preprocessing import StandardScaler, LabelEncoder\nfrom sklearn.utils.class_weight import compute_class_weight\nfrom sklearn.metrics import classification_report, confusion_matrix, accuracy_score\n\n# TensorFlow and Keras\nimport tensorflow as tf\nfrom tensorflow.keras.utils import Sequence, to_categorical, pad_sequences\nfrom tensorflow.keras.models import Model, load_model, Sequential\nfrom tensorflow.keras.layers import (\n    Input, Conv1D, BatchNormalization, Activation, add, MaxPooling1D, Dropout,\n    Bidirectional, LSTM, GlobalAveragePooling1D, Dense, Multiply, Reshape,\n    Lambda, Concatenate, GRU, GaussianNoise\n)\nfrom tensorflow.keras.regularizers import l2\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau\nfrom tensorflow.keras import backend as K\n\n# Set random seeds for reproducibility\nnp.random.seed(42)\ntf.random.set_seed(42)\nrandom.seed(42)\n\n# Set matplotlib style\nplt.style.use('seaborn-v0_8')\nsns.set_palette(\"husl\")\n\nprint(\"All libraries imported successfully!\")\n\n# ==============================================================================\n# 2. Memory Management System\n# ==============================================================================\n\nclass MemoryManager:\n    \"\"\"Comprehensive memory management and monitoring system\"\"\"\n    \n    def __init__(self, memory_threshold=0.85, verbose=True):\n        \"\"\"\n        Initialize memory manager\n        \n        Args:\n            memory_threshold: Memory usage threshold (0-1) to trigger cleanup\n            verbose: Print memory status\n        \"\"\"\n        self.memory_threshold = memory_threshold\n        self.verbose = verbose\n        self.peak_memory = 0\n        self.cleanup_count = 0\n        \n    def get_memory_usage(self):\n        \"\"\"Get current memory usage statistics\"\"\"\n        process = psutil.Process(os.getpid())\n        memory_info = process.memory_info()\n        virtual_memory = psutil.virtual_memory()\n        \n        memory_stats = {\n            'process_memory_mb': memory_info.rss / 1024 / 1024,\n            'process_memory_percent': process.memory_percent(),\n            'system_memory_percent': virtual_memory.percent / 100,\n            'available_memory_gb': virtual_memory.available / 1024 / 1024 / 1024\n        }\n        \n        # Update peak memory\n        if memory_stats['process_memory_mb'] > self.peak_memory:\n            self.peak_memory = memory_stats['process_memory_mb']\n        \n        return memory_stats\n    \n    def check_memory_status(self, context=\"\"):\n        \"\"\"Check and print memory status\"\"\"\n        stats = self.get_memory_usage()\n        \n        if self.verbose:\n            print(f\"[Memory {context}] Process: {stats['process_memory_mb']:.1f}MB \"\n                  f\"({stats['process_memory_percent']:.1f}%) | \"\n                  f\"System: {stats['system_memory_percent']:.1f}% | \"\n                  f\"Available: {stats['available_memory_gb']:.1f}GB\")\n        \n        return stats\n    \n    def force_cleanup(self, aggressive=False):\n        \"\"\"Aggressive memory cleanup\"\"\"\n        if self.verbose:\n            print(f\"[Memory Manager] Performing {'aggressive' if aggressive else 'standard'} cleanup...\")\n        \n        # Clear TensorFlow session\n        try:\n            K.clear_session()\n        except:\n            pass\n        \n        # Force garbage collection multiple times\n        for _ in range(3 if aggressive else 2):\n            gc.collect()\n        \n        # Clear matplotlib cache\n        try:\n            plt.close('all')\n        except:\n            pass\n        \n        self.cleanup_count += 1\n        \n        if self.verbose:\n            stats = self.get_memory_usage()\n            print(f\"[Memory Manager] Cleanup #{self.cleanup_count} completed. \"\n                  f\"Current usage: {stats['process_memory_mb']:.1f}MB\")\n    \n    def memory_check_decorator(self, func):\n        \"\"\"Decorator to monitor memory usage around function calls\"\"\"\n        def wrapper(*args, **kwargs):\n            if self.verbose:\n                print(f\"\\n[Memory Check] Starting {func.__name__}\")\n            \n            # Pre-execution memory check\n            pre_stats = self.check_memory_status(\"PRE\")\n            \n            # Check if cleanup needed\n            if pre_stats['system_memory_percent'] > self.memory_threshold:\n                self.force_cleanup()\n            \n            try:\n                # Execute function\n                result = func(*args, **kwargs)\n                \n                # Post-execution memory check\n                post_stats = self.check_memory_status(\"POST\")\n                \n                return result\n                \n            except MemoryError as e:\n                print(f\"[Memory Manager] Memory error in {func.__name__}: {e}\")\n                self.force_cleanup(aggressive=True)\n                raise\n            \n        return wrapper\n    \n    @contextmanager\n    def memory_context(self, context_name=\"\"):\n        \"\"\"Context manager for memory monitoring\"\"\"\n        print(f\"\\n[Memory Context] Entering {context_name}\")\n        pre_stats = self.check_memory_status(\"ENTER\")\n        \n        try:\n            yield self\n        finally:\n            post_stats = self.check_memory_status(\"EXIT\")\n            memory_diff = post_stats['process_memory_mb'] - pre_stats['process_memory_mb']\n            \n            if self.verbose:\n                print(f\"[Memory Context] {context_name} completed. \"\n                      f\"Memory change: {memory_diff:+.1f}MB\")\n            \n            # Cleanup if memory usage increased significantly\n            if memory_diff > 500:  # 500MB threshold\n                self.force_cleanup()\n\n# Global memory manager instance\nmemory_manager = MemoryManager(verbose=True)\n\n# ==============================================================================\n# 3. Enhanced Configuration with Memory Optimization\n# ==============================================================================\n\ndef get_config():\n    \"\"\"Configuration dictionary optimized for memory usage\"\"\"\n    config = {\n        'sr': 10,\n        'seq_len': 64,  # Reduced from 128 to save memory\n        'stride': 2,    # Increased stride to reduce sequences\n        'batch_size': 16,  # Reduced batch size\n        'epochs': 20,\n        'folds': 3,     # Reduced folds to save memory\n        'l2': 1e-4,\n        'optimizer': 'adam',\n        'patience': 7,  # Increased patience for better convergence\n        'verbose': 1,\n        'learning_rate': 1e-3,\n        'memory_threshold': 0.80,\n        'checkpoint_frequency': 5,  # Save model every 5 epochs\n        'early_stop_on_memory': True,\n    }\n    return config\n\n# ==============================================================================\n# 4. Memory-Optimized Data Loading\n# ==============================================================================\n\ndef load_csv_data_chunked(file_path: str, chunk_size: int = 10000) -> pd.DataFrame:\n    \"\"\"Load CSV data in chunks to manage memory\"\"\"\n    try:\n        if os.path.exists(file_path):\n            print(f\"Loading {file_path} in chunks...\")\n            \n            # Read in chunks and concatenate\n            chunks = []\n            total_rows = 0\n            \n            for chunk in pd.read_csv(file_path, chunksize=chunk_size):\n                chunks.append(chunk)\n                total_rows += len(chunk)\n                \n                # Memory check every 100k rows\n                if total_rows % 100000 == 0:\n                    memory_manager.check_memory_status(f\"Loaded {total_rows} rows\")\n            \n            df = pd.concat(chunks, ignore_index=True)\n            del chunks  # Free memory immediately\n            gc.collect()\n            \n            print(f\"Loaded {file_path}: {df.shape}\")\n            return df\n        else:\n            print(f\"File not found: {file_path}\")\n            return None\n    except Exception as e:\n        print(f\"Error loading {file_path}: {str(e)}\")\n        return None\n\n@memory_manager.memory_check_decorator\ndef load_all_data():\n    \"\"\"Load all available data files with memory management\"\"\"\n    data_paths = [\n        \"/kaggle/input/cmi-detect-behavior-with-sensor-data/\",\n        \"./data/\",\n        \"../input/cmi-detect-behavior-with-sensor-data/\"\n    ]\n    \n    data = {}\n    files_to_load = {\n        'train': 'train.csv',\n        'test': 'test.csv', \n        'train_demographics': 'train_demographics.csv',\n        'test_demographics': 'test_demographics.csv'\n    }\n    \n    for base_path in data_paths:\n        if os.path.exists(base_path):\n            print(f\"Found data directory: {base_path}\")\n            for key, filename in files_to_load.items():\n                full_path = os.path.join(base_path, filename)\n                \n                with memory_manager.memory_context(f\"Loading {filename}\"):\n                    df = load_csv_data_chunked(full_path)\n                    if df is not None:\n                        data[key] = df\n            break\n    \n    return data\n\ndef extract_sensor_features_optimized(df: pd.DataFrame) -> np.ndarray:\n    \"\"\"Extract sensor features with memory optimization\"\"\"\n    print(\"[Feature Extraction] Starting optimized feature extraction...\")\n    \n    # Define sensor columns in priority order\n    priority_sensors = ['acc_x', 'acc_y', 'acc_z', 'rot_w', 'rot_x', 'rot_y', 'rot_z']\n    thermal_cols = [col for col in df.columns if col.startswith('thm_')][:20]  # Limit thermal sensors\n    tof_cols = [col for col in df.columns if col.startswith('tof_')][:30]      # Limit ToF sensors\n    \n    sensor_cols = priority_sensors + thermal_cols + tof_cols\n    available_cols = [col for col in sensor_cols if col in df.columns]\n    \n    if not available_cols:\n        print(\"Warning: No sensor columns found!\")\n        return np.array([])\n    \n    print(f\"Using {len(available_cols)} sensor features: {available_cols[:10]}...\")\n    \n    # Process features in chunks to manage memory\n    chunk_size = 50000\n    feature_chunks = []\n    \n    for i in range(0, len(df), chunk_size):\n        chunk = df.iloc[i:i+chunk_size]\n        chunk_features = chunk[available_cols].values\n        chunk_features = np.nan_to_num(chunk_features, nan=0.0, posinf=0.0, neginf=0.0)\n        feature_chunks.append(chunk_features)\n        \n        if i % (chunk_size * 5) == 0:  # Memory check every 250k rows\n            memory_manager.check_memory_status(f\"Processed {min(i+chunk_size, len(df))} rows\")\n    \n    features = np.vstack(feature_chunks)\n    del feature_chunks, chunk_features  # Immediate cleanup\n    gc.collect()\n    \n    print(f\"[Feature Extraction] Completed. Shape: {features.shape}\")\n    return features\n\n# ==============================================================================\n# 5. Memory-Optimized Model Architecture\n# ==============================================================================\n\ndef build_memory_efficient_model(config: Dict, input_shape: Tuple[int, int]) -> Model:\n    \"\"\"Build memory-efficient model architecture\"\"\"\n    \n    def residual_block(x, filters, kernel_size=3):\n        \"\"\"Lightweight residual block\"\"\"\n        res = x\n        x = Conv1D(filters, kernel_size, padding='same', kernel_regularizer=l2(config['l2']))(x)\n        x = BatchNormalization()(x)\n        x = Activation('relu')(x)\n        x = Conv1D(filters, kernel_size, padding='same', kernel_regularizer=l2(config['l2']))(x)\n        x = BatchNormalization()(x)\n        \n        # Skip connection with dimension matching\n        if res.shape[-1] != filters:\n            res = Conv1D(filters, 1, padding='same')(res)\n        \n        x = add([res, x])\n        x = Activation('relu')(x)\n        return x\n\n    input_layer = Input(shape=input_shape, name='sensor_input')\n    \n    # Light noise for regularization\n    x = GaussianNoise(0.005)(input_layer)\n    \n    # Initial feature extraction\n    x = Conv1D(32, 5, padding='same', kernel_regularizer=l2(config['l2']))(x)\n    x = BatchNormalization()(x)\n    x = Activation('relu')(x)\n    x = MaxPooling1D(2)(x)\n    x = Dropout(0.2)(x)\n    \n    # Residual blocks with progressive complexity\n    x = residual_block(x, 32)\n    x = Dropout(0.2)(x)\n    x = residual_block(x, 64)\n    x = MaxPooling1D(2)(x)\n    x = Dropout(0.3)(x)\n    \n    # Recurrent processing\n    x = Bidirectional(GRU(32, return_sequences=False, kernel_regularizer=l2(config['l2'])))(x)\n    x = Dropout(0.4)(x)\n    \n    # Classification head\n    x = Dense(64, activation='relu', kernel_regularizer=l2(config['l2']))(x)\n    x = Dropout(0.5)(x)\n    x = Dense(32, activation='relu', kernel_regularizer=l2(config['l2']))(x)\n    x = Dropout(0.3)(x)\n    \n    # Output layer\n    output_layer = Dense(4, activation='softmax', name='behavior_output')(x)  # 4 classes\n    \n    model = Model(inputs=input_layer, outputs=output_layer)\n    \n    print(f\"[Model Architecture] Built memory-efficient model\")\n    print(f\"Total parameters: {model.count_params():,}\")\n    \n    return model\n\n# ==============================================================================\n# 6. Enhanced Model Callbacks with Peak Accuracy Tracking\n# ==============================================================================\n\nclass PeakAccuracyTracker(tf.keras.callbacks.Callback):\n    \"\"\"Custom callback to track peak accuracy and save best models\"\"\"\n    \n    def __init__(self, save_path='best_model_peak_acc.h5', verbose=1):\n        super().__init__()\n        self.save_path = save_path\n        self.verbose = verbose\n        self.peak_val_accuracy = 0.0\n        self.peak_train_accuracy = 0.0\n        self.peak_epoch = 0\n        self.accuracy_history = []\n        self.val_accuracy_history = []\n        \n    def on_epoch_end(self, epoch, logs=None):\n        current_val_acc = logs.get('val_accuracy', 0)\n        current_train_acc = logs.get('accuracy', 0)\n        \n        self.accuracy_history.append(current_train_acc)\n        self.val_accuracy_history.append(current_val_acc)\n        \n        # Check for new peak validation accuracy\n        if current_val_acc > self.peak_val_accuracy:\n            self.peak_val_accuracy = current_val_acc\n            self.peak_train_accuracy = current_train_acc\n            self.peak_epoch = epoch + 1\n            \n            # Save the model at peak performance\n            self.model.save(self.save_path)\n            \n            if self.verbose:\n                print(f\"\\n[Peak Accuracy] New peak validation accuracy: {self.peak_val_accuracy:.4f} \"\n                      f\"(train: {self.peak_train_accuracy:.4f}) at epoch {self.peak_epoch}\")\n                print(f\"[Peak Accuracy] Model saved to {self.save_path}\")\n    \n    def plot_accuracy_curves(self):\n        \"\"\"Plot accuracy curves with peak markers\"\"\"\n        plt.figure(figsize=(12, 8))\n        \n        plt.subplot(2, 2, 1)\n        epochs = range(1, len(self.accuracy_history) + 1)\n        plt.plot(epochs, self.accuracy_history, 'b-', label='Training Accuracy', linewidth=2)\n        plt.plot(epochs, self.val_accuracy_history, 'r-', label='Validation Accuracy', linewidth=2)\n        \n        # Mark peak accuracy\n        plt.scatter([self.peak_epoch], [self.peak_val_accuracy], \n                   color='red', s=100, zorder=5, label=f'Peak Val Acc: {self.peak_val_accuracy:.4f}')\n        plt.scatter([self.peak_epoch], [self.peak_train_accuracy], \n                   color='blue', s=100, zorder=5, label=f'Peak Train Acc: {self.peak_train_accuracy:.4f}')\n        \n        plt.title('Model Accuracy Over Time')\n        plt.xlabel('Epoch')\n        plt.ylabel('Accuracy')\n        plt.legend()\n        plt.grid(True, alpha=0.3)\n        \n        # Zoomed view around peak\n        plt.subplot(2, 2, 2)\n        zoom_start = max(0, self.peak_epoch - 10)\n        zoom_end = min(len(epochs), self.peak_epoch + 10)\n        zoom_epochs = epochs[zoom_start:zoom_end]\n        \n        if zoom_epochs:\n            plt.plot(zoom_epochs, self.accuracy_history[zoom_start:zoom_end], \n                    'b-', label='Training Accuracy', linewidth=2)\n            plt.plot(zoom_epochs, self.val_accuracy_history[zoom_start:zoom_end], \n                    'r-', label='Validation Accuracy', linewidth=2)\n            plt.scatter([self.peak_epoch], [self.peak_val_accuracy], \n                       color='red', s=100, zorder=5)\n            plt.title(f'Zoomed View Around Peak (Epoch {self.peak_epoch})')\n            plt.xlabel('Epoch')\n            plt.ylabel('Accuracy')\n            plt.legend()\n            plt.grid(True, alpha=0.3)\n        \n        # Accuracy improvement over time\n        plt.subplot(2, 2, 3)\n        val_acc_diff = np.diff(self.val_accuracy_history)\n        plt.plot(range(2, len(epochs) + 1), val_acc_diff, 'g-', label='Val Accuracy Change', linewidth=2)\n        plt.axhline(y=0, color='k', linestyle='--', alpha=0.5)\n        plt.title('Validation Accuracy Change per Epoch')\n        plt.xlabel('Epoch')\n        plt.ylabel('Accuracy Change')\n        plt.legend()\n        plt.grid(True, alpha=0.3)\n        \n        # Summary statistics\n        plt.subplot(2, 2, 4)\n        stats_text = f\"\"\"Peak Performance Summary:\n        \nPeak Validation Accuracy: {self.peak_val_accuracy:.4f}\nPeak Training Accuracy: {self.peak_train_accuracy:.4f}\nPeak Epoch: {self.peak_epoch}\n\nFinal Validation Accuracy: {self.val_accuracy_history[-1]:.4f}\nFinal Training Accuracy: {self.accuracy_history[-1]:.4f}\n\nAccuracy Range (Val): {min(self.val_accuracy_history):.4f} - {max(self.val_accuracy_history):.4f}\nAccuracy Range (Train): {min(self.accuracy_history):.4f} - {max(self.accuracy_history):.4f}\n        \"\"\"\n        \n        plt.text(0.1, 0.9, stats_text, transform=plt.gca().transAxes, \n                fontsize=10, verticalalignment='top', fontfamily='monospace',\n                bbox=dict(boxstyle='round', facecolor='lightblue', alpha=0.8))\n        plt.axis('off')\n        \n        plt.tight_layout()\n        plt.show()\n\nclass MemoryMonitorCallback(tf.keras.callbacks.Callback):\n    \"\"\"Monitor memory usage during training\"\"\"\n    \n    def __init__(self, memory_manager, check_frequency=5):\n        super().__init__()\n        self.memory_manager = memory_manager\n        self.check_frequency = check_frequency\n        \n    def on_epoch_end(self, epoch, logs=None):\n        if epoch % self.check_frequency == 0:\n            self.memory_manager.check_memory_status(f\"Epoch {epoch+1}\")\n            \n            # Force cleanup if memory usage is high\n            stats = self.memory_manager.get_memory_usage()\n            if stats['system_memory_percent'] > 0.85:\n                print(f\"[Memory Monitor] High memory usage detected, performing cleanup...\")\n                self.memory_manager.force_cleanup()\n\n# ==============================================================================\n# 7. Memory-Optimized Data Generator\n# ==============================================================================\n\nclass MemoryEfficientDataGenerator(Sequence):\n    \"\"\"Memory-efficient data generator with cleanup\"\"\"\n    \n    def __init__(self, X, y, config, shuffle=True, memory_manager=None):\n        self.X = X\n        self.y = y\n        self.config = config\n        self.shuffle = shuffle\n        self.memory_manager = memory_manager\n        self.indices = np.arange(len(X))\n        \n        if self.shuffle:\n            np.random.shuffle(self.indices)\n    \n    def __len__(self):\n        return int(np.ceil(len(self.X) / self.config['batch_size']))\n    \n    def __getitem__(self, idx):\n        start_idx = idx * self.config['batch_size']\n        end_idx = min((idx + 1) * self.config['batch_size'], len(self.X))\n        batch_indices = self.indices[start_idx:end_idx]\n        \n        X_batch = self.X[batch_indices].copy()  # Explicit copy to manage memory\n        y_batch = self.y[batch_indices].copy()\n        \n        return X_batch, y_batch\n    \n    def on_epoch_end(self):\n        if self.shuffle:\n            np.random.shuffle(self.indices)\n        \n        # Periodic cleanup\n        if hasattr(self, 'memory_manager') and self.memory_manager:\n            if np.random.random() < 0.1:  # 10% chance per epoch end\n                gc.collect()\n\n# ==============================================================================\n# 8. Memory-Optimized Training Pipeline\n# ==============================================================================\n\n@memory_manager.memory_check_decorator\ndef prepare_sequences_optimized(X, y, config):\n    \"\"\"Prepare sequences with memory optimization\"\"\"\n    print(f\"[Sequence Preparation] Creating sequences with length {config['seq_len']}, stride {config['stride']}\")\n    \n    seq_len = config['seq_len']\n    stride = config['stride']\n    \n    # Calculate number of sequences to pre-allocate arrays\n    num_sequences = (len(X) - seq_len) // stride + 1\n    print(f\"[Sequence Preparation] Will create {num_sequences:,} sequences\")\n    \n    # Pre-allocate arrays\n    sequences = np.zeros((num_sequences, seq_len, X.shape[1]), dtype=np.float32)\n    labels = np.zeros(num_sequences, dtype=np.int32)\n    \n    # Fill arrays in chunks\n    chunk_size = 1000\n    for i, start_idx in enumerate(range(0, len(X) - seq_len + 1, stride)):\n        sequences[i] = X[start_idx:start_idx + seq_len]\n        labels[i] = y[start_idx + seq_len - 1]\n        \n        # Periodic memory check\n        if i % (chunk_size * 10) == 0 and i > 0:\n            memory_manager.check_memory_status(f\"Sequences: {i}/{num_sequences}\")\n    \n    print(f\"[Sequence Preparation] Completed. Shape: {sequences.shape}\")\n    return sequences, labels\n\n@memory_manager.memory_check_decorator\ndef train_model_with_validation_optimized(X_train, y_train, X_val, y_val, config):\n    \"\"\"Memory-optimized training with peak accuracy tracking\"\"\"\n    \n    print(\"\\n\" + \"=\"*60)\n    print(\"STARTING MEMORY-OPTIMIZED TRAINING\")\n    print(\"=\"*60)\n    \n    with memory_manager.memory_context(\"Sequence Preparation\"):\n        # Prepare sequences\n        X_train_seq, y_train_seq = prepare_sequences_optimized(X_train, y_train, config)\n        X_val_seq, y_val_seq = prepare_sequences_optimized(X_val, y_val, config)\n    \n    # Clear original arrays to save memory\n    del X_train, y_train, X_val, y_val\n    gc.collect()\n    \n    print(f\"Training sequences: {X_train_seq.shape}\")\n    print(f\"Validation sequences: {X_val_seq.shape}\")\n    \n    with memory_manager.memory_context(\"Model Building\"):\n        # Build and compile model\n        input_shape = (X_train_seq.shape[1], X_train_seq.shape[2])\n        model = build_memory_efficient_model(config, input_shape)\n        \n        # Compile model\n        optimizer = Adam(learning_rate=config['learning_rate'])\n        model.compile(\n            optimizer=optimizer, \n            loss='sparse_categorical_crossentropy',  # Use sparse for memory efficiency\n            metrics=['accuracy']\n        )\n    \n    # Print model summary\n    model.summary()\n    \n    # Initialize peak accuracy tracker\n    peak_tracker = PeakAccuracyTracker(save_path='best_model_peak_accuracy.h5', verbose=1)\n    memory_monitor = MemoryMonitorCallback(memory_manager, check_frequency=3)\n    \n    # Enhanced callbacks\n    callbacks = [\n        peak_tracker,\n        memory_monitor,\n        EarlyStopping(\n            monitor='val_accuracy',\n            patience=config['patience'], \n            restore_best_weights=True, \n            verbose=1,\n            mode='max'\n        ),\n        ReduceLROnPlateau(\n            monitor='val_accuracy',\n            factor=0.5, \n            patience=4, \n            verbose=1,\n            mode='max',\n            min_lr=1e-6\n        ),\n        ModelCheckpoint(\n            'checkpoint_model.h5', \n            monitor='val_accuracy',\n            save_best_only=True, \n            verbose=1,\n            mode='max'\n        )\n    ]\n    \n    print(f\"\\n[Training] Starting training for {config['epochs']} epochs...\")\n    \n    try:\n        with memory_manager.memory_context(\"Model Training\"):\n            # Train model with memory monitoring\n            history = model.fit(\n                X_train_seq, y_train_seq,\n                validation_data=(X_val_seq, y_val_seq),\n                epochs=config['epochs'],\n                batch_size=config['batch_size'],\n                callbacks=callbacks,\n                verbose=config['verbose'],\n                shuffle=True\n            )\n        \n        print(f\"\\n[Training] Completed successfully!\")\n        print(f\"Peak validation accuracy: {peak_tracker.peak_val_accuracy:.4f} at epoch {peak_tracker.peak_epoch}\")\n        \n        # Plot training curves with peak accuracy\n        peak_tracker.plot_accuracy_curves()\n        \n        # Final memory status\n        memory_manager.check_memory_status(\"Training Complete\")\n        \n        return model, history, peak_tracker\n        \n    except Exception as e:\n        print(f\"[Training] Error occurred: {e}\")\n        memory_manager.force_cleanup(aggressive=True)\n        raise\n\n# ==============================================================================\n# 9. Enhanced Main Pipeline with Memory Management\n# ==============================================================================\n\ndef main_training_pipeline_optimized():\n    \"\"\"Complete memory-optimized training pipeline\"\"\"\n    \n    print(\"=\"*80)\n    print(\"CMI BEHAVIOR DETECTION - MEMORY-OPTIMIZED PIPELINE\")\n    print(\"=\"*80)\n    \n    # Initialize memory tracking\n    memory_manager.check_memory_status(\"Pipeline Start\")\n    \n    # Load configuration\n    config = get_config()\n    print(f\"Configuration: {config}\")\n    \n    # Load data with memory management\n    print(\"\\n1. Loading Data...\")\n    with memory_manager.memory_context(\"Data Loading\"):\n        data = load_all_data()\n    \n    if not data or 'train' not in data or data['train'] is None:\n        print(\"No training data found. Please ensure data files are available.\")\n        return None\n    \n    # Prepare training data\n    print(\"\\n3. Preparing Training Data...\")\n    train_df = data['train']\n    \n    with memory_manager.memory_context(\"Feature Extraction\"):\n        # Extract features\n        X = extract_sensor_features_optimized(train_df)\n        \n        if X.size == 0:\n            print(\"No sensor features found!\")\n            return None\n        \n        # Prepare labels\n        if 'behavior' in train_df.columns:\n            le = LabelEncoder()\n            y = le.fit_transform(train_df['behavior'])\n            class_names = list(le.classes_)\n            print(f\"Classes: {class_names}\")\n        else:\n            print(\"No behavior column found for labels!\")\n            return None\n    \n    # Clear train_df to save memory\n    del train_df\n    gc.collect()\n    \n    with memory_manager.memory_context(\"Data Normalization\"):\n        # Normalize features in chunks if dataset is large\n        if X.shape[0] > 100000:\n            print(\"[Normalization] Large dataset detected, using chunked normalization...\")\n            scaler = StandardScaler()\n            \n            # Fit scaler on subset\n            sample_size = min(50000, X.shape[0])\n            sample_idx = np.random.choice(X.shape[0], sample_size, replace=False)\n            scaler.fit(X[sample_idx])\n            \n            # Transform in chunks\n            chunk_size = 10000\n            X_scaled_chunks = []\n            \n            for i in range(0, X.shape[0], chunk_size):\n                chunk = X[i:i+chunk_size]\n                chunk_scaled = scaler.transform(chunk)\n                X_scaled_chunks.append(chunk_scaled)\n                \n                if i % (chunk_size * 5) == 0:\n                    memory_manager.check_memory_status(f\"Normalized {i+chunk_size}/{X.shape[0]}\")\n            \n            X_scaled = np.vstack(X_scaled_chunks)\n            del X_scaled_chunks, chunk, chunk_scaled\n            gc.collect()\n        else:\n            scaler = StandardScaler()\n            X_scaled = scaler.fit_transform(X)\n    \n    # Clear original X to\n#*********************************************************************************************\n# Clear original X to save memory\n    del X\n    gc.collect()\n    \n    print(f\"Features prepared: {X_scaled.shape}\")\n    print(f\"Labels prepared: {y.shape}\")\n    print(f\"Memory after preparation:\")\n    memory_manager.check_memory_status(\"Data Prepared\")\n    \n    # Train-validation split with stratification\n    print(\"\\n4. Splitting Data...\")\n    with memory_manager.memory_context(\"Data Splitting\"):\n        X_train, X_val, y_train, y_val = train_test_split(\n            X_scaled, y, \n            test_size=0.2, \n            random_state=42, \n            stratify=y\n        )\n        \n        print(f\"Train set: {X_train.shape}, {y_train.shape}\")\n        print(f\"Validation set: {X_val.shape}, {y_val.shape}\")\n        \n        # Clear full dataset from memory\n        del X_scaled, y\n        gc.collect()\n    \n    # Model training with memory optimization\n    print(\"\\n5. Training Model...\")\n    try:\n        model, history, peak_tracker = train_model_with_validation_optimized(\n            X_train, y_train, X_val, y_val, config\n        )\n        \n        print(f\"\\n6. Training Summary:\")\n        print(f\"Peak validation accuracy: {peak_tracker.peak_val_accuracy:.4f}\")\n        print(f\"Peak training accuracy: {peak_tracker.peak_train_accuracy:.4f}\")\n        print(f\"Peak achieved at epoch: {peak_tracker.peak_epoch}\")\n        \n        # Save final results\n        with memory_manager.memory_context(\"Results Saving\"):\n            # Save training history\n            history_dict = {\n                'accuracy': peak_tracker.accuracy_history,\n                'val_accuracy': peak_tracker.val_accuracy_history,\n                'peak_val_accuracy': peak_tracker.peak_val_accuracy,\n                'peak_train_accuracy': peak_tracker.peak_train_accuracy,\n                'peak_epoch': peak_tracker.peak_epoch\n            }\n            \n            try:\n                import json\n                with open('training_history.json', 'w') as f:\n                    json.dump(history_dict, f, indent=2)\n                print(\"Training history saved to training_history.json\")\n            except Exception as e:\n                print(f\"Could not save training history: {e}\")\n            \n            # Save scaler and label encoder\n            try:\n                joblib.dump(scaler, 'feature_scaler.pkl')\n                joblib.dump(le, 'label_encoder.pkl')\n                print(\"Scaler and label encoder saved\")\n            except Exception as e:\n                print(f\"Could not save preprocessing objects: {e}\")\n        \n        # Final memory report\n        print(f\"\\n7. Memory Usage Summary:\")\n        print(f\"Peak memory usage: {memory_manager.peak_memory:.1f}MB\")\n        print(f\"Total cleanups performed: {memory_manager.cleanup_count}\")\n        memory_manager.check_memory_status(\"Pipeline Complete\")\n        \n        return {\n            'model': model,\n            'history': history_dict,\n            'peak_tracker': peak_tracker,\n            'scaler': scaler,\n            'label_encoder': le,\n            'config': config,\n            'class_names': class_names\n        }\n        \n    except Exception as e:\n        print(f\"Training failed: {e}\")\n        memory_manager.force_cleanup(aggressive=True)\n        return None\n\n# ==============================================================================\n# 10. Evaluation and Prediction Functions\n# ==============================================================================\n\n@memory_manager.memory_check_decorator\ndef evaluate_model_performance(results_dict, X_test=None, y_test=None):\n    \"\"\"Comprehensive model evaluation with memory management\"\"\"\n    \n    if results_dict is None:\n        print(\"No results to evaluate\")\n        return\n    \n    model = results_dict['model']\n    peak_tracker = results_dict['peak_tracker']\n    class_names = results_dict['class_names']\n    \n    print(\"\\n\" + \"=\"*60)\n    print(\"MODEL PERFORMANCE EVALUATION\")\n    print(\"=\"*60)\n    \n    # Training performance summary\n    print(f\"\\n1. Training Performance Summary:\")\n    print(f\"   Peak Validation Accuracy: {peak_tracker.peak_val_accuracy:.4f}\")\n    print(f\"   Peak Training Accuracy: {peak_tracker.peak_train_accuracy:.4f}\")\n    print(f\"   Peak Epoch: {peak_tracker.peak_epoch}\")\n    print(f\"   Final Validation Accuracy: {peak_tracker.val_accuracy_history[-1]:.4f}\")\n    \n    # Performance analysis\n    val_acc_improvement = peak_tracker.peak_val_accuracy - peak_tracker.val_accuracy_history[0]\n    print(f\"   Validation Accuracy Improvement: {val_acc_improvement:.4f}\")\n    \n    # Overfitting analysis\n    accuracy_gap = peak_tracker.peak_train_accuracy - peak_tracker.peak_val_accuracy\n    print(f\"   Train-Val Accuracy Gap: {accuracy_gap:.4f}\")\n    \n    if accuracy_gap > 0.1:\n        print(\"     Warning: Possible overfitting detected\")\n    elif accuracy_gap < 0.05:\n        print(\"    Good generalization performance\")\n    else:\n        print(\"     Moderate generalization performance\")\n    \n    # Test set evaluation if provided\n    if X_test is not None and y_test is not None:\n        print(f\"\\n2. Test Set Evaluation:\")\n        \n        with memory_manager.memory_context(\"Test Evaluation\"):\n            # Prepare test sequences\n            config = results_dict['config']\n            X_test_seq, y_test_seq = prepare_sequences_optimized(X_test, y_test, config)\n            \n            # Predict in batches to manage memory\n            batch_size = config['batch_size'] * 2  # Larger batch for inference\n            test_predictions = []\n            test_true_labels = []\n            \n            for i in range(0, len(X_test_seq), batch_size):\n                batch_X = X_test_seq[i:i+batch_size]\n                batch_y = y_test_seq[i:i+batch_size]\n                \n                batch_pred = model.predict(batch_X, verbose=0)\n                batch_pred_classes = np.argmax(batch_pred, axis=1)\n                \n                test_predictions.extend(batch_pred_classes)\n                test_true_labels.extend(batch_y)\n                \n                if i % (batch_size * 5) == 0:\n                    memory_manager.check_memory_status(f\"Test prediction batch {i//batch_size + 1}\")\n            \n            # Calculate test accuracy\n            test_accuracy = accuracy_score(test_true_labels, test_predictions)\n            print(f\"   Test Accuracy: {test_accuracy:.4f}\")\n            \n            # Classification report\n            print(f\"\\n   Classification Report:\")\n            print(classification_report(test_true_labels, test_predictions, \n                                      target_names=class_names))\n            \n            # Confusion matrix\n            cm = confusion_matrix(test_true_labels, test_predictions)\n            \n            plt.figure(figsize=(10, 8))\n            sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', \n                       xticklabels=class_names, yticklabels=class_names)\n            plt.title('Confusion Matrix - Test Set')\n            plt.ylabel('True Label')\n            plt.xlabel('Predicted Label')\n            plt.show()\n    \n    # Model architecture summary\n    print(f\"\\n3. Model Architecture Summary:\")\n    print(f\"   Total Parameters: {model.count_params():,}\")\n    print(f\"   Model Size: {model.count_params() * 4 / 1024 / 1024:.2f} MB (float32)\")\n    \n    # Memory usage summary\n    print(f\"\\n4. Memory Usage Summary:\")\n    print(f\"   Peak Memory Usage: {memory_manager.peak_memory:.1f} MB\")\n    print(f\"   Total Cleanups: {memory_manager.cleanup_count}\")\n\n@memory_manager.memory_check_decorator\ndef predict_behavior_sequences(model, X_new, scaler, label_encoder, config, sequence_length=None):\n    \"\"\"Make predictions on new sensor data\"\"\"\n    \n    print(\"\\n\" + \"=\"*50)\n    print(\"BEHAVIOR PREDICTION\")\n    print(\"=\"*50)\n    \n    if sequence_length is None:\n        sequence_length = config['seq_len']\n    \n    # Normalize features\n    with memory_manager.memory_context(\"Feature Normalization\"):\n        X_scaled = scaler.transform(X_new)\n    \n    # Create sequences\n    with memory_manager.memory_context(\"Sequence Creation\"):\n        X_sequences, _ = prepare_sequences_optimized(\n            X_scaled, \n            np.zeros(len(X_scaled)),  # Dummy labels\n            config\n        )\n    \n    print(f\"Created {len(X_sequences)} prediction sequences\")\n    \n    # Predict in batches\n    predictions = []\n    confidence_scores = []\n    batch_size = config['batch_size'] * 2\n    \n    with memory_manager.memory_context(\"Prediction\"):\n        for i in range(0, len(X_sequences), batch_size):\n            batch_X = X_sequences[i:i+batch_size]\n            batch_pred = model.predict(batch_X, verbose=0)\n            \n            batch_classes = np.argmax(batch_pred, axis=1)\n            batch_confidence = np.max(batch_pred, axis=1)\n            \n            predictions.extend(batch_classes)\n            confidence_scores.extend(batch_confidence)\n            \n            if i % (batch_size * 5) == 0:\n                print(f\"Processed {i + len(batch_X)}/{len(X_sequences)} sequences\")\n    \n    # Convert predictions to labels\n    predicted_behaviors = label_encoder.inverse_transform(predictions)\n    \n    # Create results summary\n    results = {\n        'predictions': predicted_behaviors,\n        'prediction_classes': predictions,\n        'confidence_scores': confidence_scores,\n        'sequence_count': len(predictions)\n    }\n    \n    # Print summary\n    unique_behaviors, counts = np.unique(predicted_behaviors, return_counts=True)\n    print(f\"\\nPrediction Summary:\")\n    for behavior, count in zip(unique_behaviors, counts):\n        percentage = (count / len(predictions)) * 100\n        avg_confidence = np.mean([conf for i, conf in enumerate(confidence_scores) \n                                 if predicted_behaviors[i] == behavior])\n        print(f\"  {behavior}: {count} sequences ({percentage:.1f}%) - \"\n              f\"Avg confidence: {avg_confidence:.3f}\")\n    \n    return results\n\n# ==============================================================================\n# 11. Utility Functions\n# ==============================================================================\n\ndef save_training_artifacts(results_dict, save_dir='./training_artifacts/'):\n    \"\"\"Save all training artifacts for later use\"\"\"\n    \n    os.makedirs(save_dir, exist_ok=True)\n    \n    print(f\"\\nSaving training artifacts to {save_dir}...\")\n    \n    try:\n        # Save model\n        results_dict['model'].save(os.path.join(save_dir, 'final_model.h5'))\n        print(\" Model saved\")\n        \n        # Save preprocessing objects\n        joblib.dump(results_dict['scaler'], os.path.join(save_dir, 'scaler.pkl'))\n        joblib.dump(results_dict['label_encoder'], os.path.join(save_dir, 'label_encoder.pkl'))\n        print(\" Preprocessing objects saved\")\n        \n        # Save configuration\n        with open(os.path.join(save_dir, 'config.json'), 'w') as f:\n            json.dump(results_dict['config'], f, indent=2)\n        print(\" Configuration saved\")\n        \n        # Save training history\n        history_data = {\n            'accuracy': results_dict['history']['accuracy'],\n            'val_accuracy': results_dict['history']['val_accuracy'],\n            'peak_val_accuracy': results_dict['history']['peak_val_accuracy'],\n            'peak_train_accuracy': results_dict['history']['peak_train_accuracy'],\n            'peak_epoch': results_dict['history']['peak_epoch'],\n            'class_names': results_dict['class_names']\n        }\n        \n        with open(os.path.join(save_dir, 'training_history.json'), 'w') as f:\n            json.dump(history_data, f, indent=2)\n        print(\" Training history saved\")\n        \n        print(f\"All artifacts saved successfully to {save_dir}\")\n        \n    except Exception as e:\n        print(f\" Error saving artifacts: {e}\")\n\ndef load_training_artifacts(load_dir='./training_artifacts/'):\n    \"\"\"Load previously saved training artifacts\"\"\"\n    \n    print(f\"Loading training artifacts from {load_dir}...\")\n    \n    try:\n        # Load model\n        model = load_model(os.path.join(load_dir, 'final_model.h5'))\n        \n        # Load preprocessing objects\n        scaler = joblib.load(os.path.join(load_dir, 'scaler.pkl'))\n        label_encoder = joblib.load(os.path.join(load_dir, 'label_encoder.pkl'))\n        \n        # Load configuration\n        with open(os.path.join(load_dir, 'config.json'), 'r') as f:\n            config = json.load(f)\n        \n        # Load training history\n        with open(os.path.join(load_dir, 'training_history.json'), 'r') as f:\n            history_data = json.load(f)\n        \n        results = {\n            'model': model,\n            'scaler': scaler,\n            'label_encoder': label_encoder,\n            'config': config,\n            'history': history_data,\n            'class_names': history_data['class_names']\n        }\n        \n        print(\" All artifacts loaded successfully\")\n        return results\n        \n    except Exception as e:\n        print(f\" Error loading artifacts: {e}\")\n        return None\n\n# ==============================================================================\n# 12. Main Execution Block\n# ==============================================================================\n\nif __name__ == \"__main__\":\n    print(\"Starting CMI Behavior Detection Pipeline...\")\n    \n    try:\n        # Run the complete training pipeline\n        results = main_training_pipeline_optimized()\n        \n        if results is not None:\n            # Evaluate model performance\n            evaluate_model_performance(results)\n            \n            # Save training artifacts\n            save_training_artifacts(results)\n            \n            print(\"\\n\" + \"=\"*80)\n            print(\"PIPELINE COMPLETED SUCCESSFULLY!\")\n            print(\"=\"*80)\n            print(f\"Peak validation accuracy achieved: {results['peak_tracker'].peak_val_accuracy:.4f}\")\n            print(f\"Model and artifacts saved for future use.\")\n            \n            # Final cleanup\n            memory_manager.force_cleanup(aggressive=True)\n            \n        else:\n            print(\"Pipeline failed to complete.\")\n            \n    except Exception as e:\n        print(f\"Pipeline error: {e}\")\n        import traceback\n        traceback.print_exc()\n        memory_manager.force_cleanup(aggressive=True)\n    \n    finally:\n        print(f\"\\nFinal memory status:\")\n        memory_manager.check_memory_status(\"Pipeline End\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-09-02T20:48:14.223905Z","iopub.execute_input":"2025-09-02T20:48:14.224160Z","iopub.status.idle":"2025-09-02T22:06:00.268507Z","shell.execute_reply.started":"2025-09-02T20:48:14.224139Z","shell.execute_reply":"2025-09-02T22:06:00.267604Z"}},"outputs":[],"execution_count":null},{"id":"954604a4-0336-4f56-9c40-5ff753c5ccc5","cell_type":"markdown","source":"# Understanding the Results\n\nOverview\nThis part provides a detailed, panel-by-panel explanation of the four visual elements visible in the embedded image. The image contains 4 panels: 1) 'Model Accuracy Over Time' (top-left), 2) 'Zoomed View Around Peak (Epoch 19)' (top-right), 3) 'Validation Accuracy Change per Epoch' (bottom-left), and 4) 'Peak Performance Summary' (bottom-right). Each panel is analyzed for what it shows, what the axes mean, key numbers annotated on the figure, likely causes of the observed trends, and actionable interpretation or recommended next steps for model development.\n1. Panel: Model Accuracy Over Time (Top-left)\nLocation & Purpose: This top-left plot shows the progression of both training and validation accuracy across epochs (1–20). It is intended to give a high-level view of how the model's ability to correctly classify or predict (accuracy) changes as training proceeds.\nAxes and Scales: The x-axis is 'Epoch' (training iterations over the dataset). The y-axis is 'Accuracy' and ranges approximately from 0.57 to 0.645. Accuracy is expressed as a decimal fraction (for example, 0.64 means 64.0%).\nPlotted Curves & Markers: Two curves appear: a blue line for Training Accuracy and a red line for Validation Accuracy. The training accuracy rises steeply in early epochs and then gradually flattens out — a typical learning curve. The validation accuracy follows a similar upward trend but is slightly higher at the peak and has marker dots highlighting peak values.\nAnnotated Peak Values: The legend and annotations indicate 'Peak Val Acc: 0.6443' and 'Peak Train Acc: 0.6368'. A red marker near epoch 19 indicates the highest validation accuracy (0.6443). A blue marker near epoch 20 indicates the peak training accuracy (0.6368). These annotations were taken directly from the image.\nTrends and Interpretation: The model learns quickly during the first few epochs (big gains in accuracy), then experiences diminishing returns as epochs increase. Both training and validation rise and then roughly plateau, suggesting the model is converging. That the validation accuracy achieves a slightly higher peak than training accuracy is unusual but can happen due to fluctuations or when validation sampling produces a temporarily easier batch of data at that epoch.\nOverfitting / Underfitting Considerations: Because the validation accuracy does not sharply fall while training accuracy continues to increase, there is no dramatic overfitting visible in this plot. The curves are relatively close (difference on the order of ~0.007–0.008 at peak), which suggests the model is generalizing reasonably well. However, the small gap combined with the plateau suggests limited remaining gains from more training without other changes (architecture, data, regularization).\nActionable Recommendations: Given the slow improvements after epoch ~5–10, consider early stopping around the highest validation epoch (epoch 19) to avoid wasted compute. Also consider: increasing model capacity (if underfitting), adding or tuning regularization (dropout, weight decay) if you later observe divergence, or improving the dataset (more samples or augmentation) to push the validation ceiling higher.\n2. Panel: Zoomed View Around Peak (Epoch 19) (Top-right)\nLocation & Purpose: The top-right panel zooms into epochs roughly 10–20 to highlight the behavior near the best validation performance (annotated as epoch 19). This close-up makes it easier to see small fluctuations that the full-range plot hides.\nAxes and Scales: The x-axis is 'Epoch' (10 to 20 shown); the y-axis is a narrow accuracy window (about 0.636 to 0.6445), focused on the plateau region. This tight range magnifies small changes in metric.\nWhat the Zoom Reveals: The validation accuracy (red) exhibits modest fluctuation across the shown epochs, with a dip around epoch 14 followed by a notable rise to the peak near epoch 19. The training accuracy (blue) also shows small ups and downs but remains slightly lower overall in this zoom. The magnified scale makes these fluctuations look more pronounced than in the full-range view.\nInterpretation of the Fluctuations: Such small oscillations near the plateau are typical and can be caused by random sampling variation in the validation set, mini-batch stochasticity, or slight changes in model parameter updates. The dip at epoch 14 followed by recovery suggests the model briefly explored a parameter region with marginally worse validation generalization before moving to a better one.\nImplications: Because the fluctuations are small (hundreds to a few thousandths of accuracy), they are within normal noise. The presence of a clear peak at epoch 19 supports selecting that epoch for model checkpointing. It also suggests that using a larger validation set or repeated cross-validation could reduce noise and yield a more stable estimate of peak performance.\nActionable Recommendations: Use model checkpointing (save the weights from epoch 19) or implement rolling-validation averages and early stopping with a small patience (e.g., patience of 3–5 epochs) to capture genuine improvements and avoid reacting to noise.\n3. Panel: Validation Accuracy Change per Epoch (Bottom-left)\nLocation & Purpose: The bottom-left panel shows the per-epoch change in validation accuracy (Δ Val Accuracy) — essentially the first difference of the validation accuracy time series. This helps quantify how much the validation metric improves or worsens from one epoch to the next.\nAxes and Scales: The x-axis is 'Epoch' and the y-axis is 'Accuracy Change' with a small range centered near zero (approximately -0.001 to +0.006). Positive values indicate an increase in validation accuracy compared to the previous epoch; negative values indicate a drop.\nKey Observations: The largest positive change occurs very early (epoch 2–3), showing the biggest gains happen near the start of training. After that, the per-epoch gains diminish and often hover around zero, with occasional small positive spikes and occasional small negative dips. Toward the later epochs the deltas become tiny (near zero), indicating a plateau.\nInterpretation: The diminishing per-epoch changes show typical learning behavior: large early returns, then diminishing marginal gains as the model fits training data. Small negative changes are normal jitter; they do not necessarily imply model degradation unless they accumulate into a trend. The dotted horizontal line at y=0 serves as the reference for no change.\nImplications for Training: This plot is very useful to decide when further training yields negligible returns — when most deltas are near zero you can stop training. It also quantifies the improvement rate; if changes are mostly <0.001 late in training, it may be inefficient to continue for many more epochs.\nActionable Recommendations: Use the magnitude of delta to set early-stopping thresholds (e.g., if validation change per epoch < 0.0005 for N consecutive epochs, stop). Also consider learning-rate scheduling or restarts if you want to escape small plateaus and attempt further improvements.\n4. Panel: Peak Performance Summary (Bottom-right)\nLocation & Purpose: The bottom-right panel is a compact text box summarizing the key scalar performance numbers extracted from the run. It gives quick access to peak and final metrics and simple ranges.\nNumbers Quoted in the Box (as read from image):\n  • Peak Validation Accuracy: 0.6443\n  • Peak Training Accuracy: 0.6368\n  • Peak Epoch: 19\n  • Final Validation Accuracy: 0.6436\n  • Final Training Accuracy: 0.6371\n  • Accuracy Range (Val): 0.6280 - 0.6443\n  • Accuracy Range (Train): 0.5690 - 0.6378\nInterpretation of the Summary: The peak validation accuracy (0.6443) is slightly higher than the final validation (0.6436), meaning the best model state occurred at epoch 19 and the last saved state (epoch 20) slightly underperformed relative to the peak. Training values are close to validation values, suggesting reasonable generalization.\nActionable Use: The concise summary is ideal for quick reporting and for passing to automated scripts (e.g., to save the best checkpoint). The ranges show the variability across training: the validation range is narrow (≈0.0163 span), while the training range is wider (≈0.0688 span), reflecting early training rapid improvements in training accuracy.\nCaveats: All numeric values were transcribed visually from the embedded figure; if you require precise logged metrics for official reporting or for automated pipelines, extract them directly from the training logs or saved history arrays rather than from a plotted image.\nConsolidated Recommendations\n1. Save/checkpoint the model state at epoch 19 (the annotated peak).  \n2. Implement early stopping based on validation accuracy with a small patience (3–5 epochs) to avoid wasting compute.  \n3. If further gains are desired and plateau persists, try: (a) gathering more data or augmenting it, (b) tuning learning rate schedules or restarts, (c) modestly increasing model capacity, and (d) using ensembling or cross-validation to stabilize estimates.  \n4. Use a larger validation set or repeated cross-validation if fluctuations near the peak are a concern.  \n5. For reproducibility, extract numeric histories (training and validation accuracy arrays) from the training logs or callbacks rather than relying on the plotted figure.\n","metadata":{}},{"id":"0b526a4f-1f93-485a-9460-3e51a942ae9a","cell_type":"markdown","source":"","metadata":{}}]}