{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":30894,"databundleVersionId":3147400,"sourceType":"competition"},{"sourceId":9513072,"sourceType":"datasetVersion","datasetId":5758231},{"sourceId":197392195,"sourceType":"kernelVersion"},{"sourceId":198863234,"sourceType":"kernelVersion"}],"dockerImageVersionId":30761,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## 📊 Scalable Feature Engineering with Frypto\nWith **Frypto**, a Python package designed for financial time series, you can easily generate **42 advanced features** such as price spreads, volatility indicators, momentum metrics, and more. \n\nIn this notebook, we demonstrate:\n- How **Frypto** handles a **29 million-row cryptocurrency dataset** efficiently.\n- How to generate **42 engineered features** in **under 9 minutes**.\n- The impact of feature engineering on model performance.\n\n### Key Sections:\n1. **Environment Setup**: Install and import all necessary libraries.\n2. **Data Loading**: Load a large crypto dataset (29M rows) and preprocess it.\n3. **Feature Engineering**: Use **Frypto** to generate features for each asset.\n4. **Training Models**: Train models with and without Frypto features to compare performance.\n5. **Results & Analysis**: Evaluate and visualize the improvement in RMSE and feature importance.\n","metadata":{"jp-MarkdownHeadingCollapsed":true}},{"cell_type":"markdown","source":"# 1. Environment Setup","metadata":{"jp-MarkdownHeadingCollapsed":true}},{"cell_type":"markdown","source":"### Install frypto: feature engineering package [Github Repo](https://github.com/NotAbdelrahmanelsayed/frypto) \n","metadata":{"jp-MarkdownHeadingCollapsed":true}},{"cell_type":"code","source":"!pip install frypto ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-16T10:44:41.840335Z","iopub.execute_input":"2024-11-16T10:44:41.841489Z","iopub.status.idle":"2024-11-16T10:45:29.867986Z","shell.execute_reply.started":"2024-11-16T10:44:41.841424Z","shell.execute_reply":"2024-11-16T10:45:29.866785Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"_kg_hide-output":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Imports","metadata":{"jp-MarkdownHeadingCollapsed":true}},{"cell_type":"code","source":"import frypto\n\nimport warnings\nimport sys\nimport os\nimport gc\nimport pickle\nimport shutil\nfrom pathlib import Path\n\nfrom tqdm import tqdm\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom joblib import Parallel, delayed\nfrom lightgbm import LGBMRegressor\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import TimeSeriesSplit\nfrom sklearn.metrics import mean_squared_error\n\n# Ignore DeprecationWarnings and RuntimeWarnings\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\nwarnings.filterwarnings(\"ignore\", category=RuntimeWarning)\npd.set_option(\"future.no_silent_downcasting\", True)\n\n# Append 'src' directory to sys.path to import local modules\nsys.path.append('/kaggle/input/source')\n\n# Import local modules\nfrom utils import (\n    read_data, fill_timestamps, \n    optimize_dataframe, weighted_correlation, evaluate_predictions,\n    reduce_memory_usage, get_time_series_splits\n)\n","metadata":{"execution":{"iopub.status.busy":"2024-11-16T11:59:57.661186Z","iopub.execute_input":"2024-11-16T11:59:57.662375Z","iopub.status.idle":"2024-11-16T11:59:57.671450Z","shell.execute_reply.started":"2024-11-16T11:59:57.662315Z","shell.execute_reply":"2024-11-16T11:59:57.670239Z"},"trusted":true,"jupyter":{"source_hidden":true},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 2. Data Loading ","metadata":{"jp-MarkdownHeadingCollapsed":true}},{"cell_type":"markdown","source":"### Set up data paths","metadata":{"jp-MarkdownHeadingCollapsed":true}},{"cell_type":"code","source":"data_folder = Path('data')\noutputs_dir = Path('outputs')\noutputs_dir.mkdir(parents=True, exist_ok=True)\nmodel_dir = outputs_dir / 'trained_models'\nmodel_dir.mkdir(parents=True, exist_ok=True)\nresults_dir = Path('results')\nresults_dir.mkdir(parents=True, exist_ok=True)\nfeature_importance_dir = results_dir / 'feature_importance_per_asset'\nfeature_importance_dir.mkdir(parents=True, exist_ok=True)","metadata":{"execution":{"iopub.status.busy":"2024-11-16T10:47:14.040020Z","iopub.execute_input":"2024-11-16T10:47:14.040459Z","iopub.status.idle":"2024-11-16T10:47:14.049806Z","shell.execute_reply.started":"2024-11-16T10:47:14.040419Z","shell.execute_reply":"2024-11-16T10:47:14.048534Z"},"trusted":true,"jupyter":{"source_hidden":true},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Load Data","metadata":{"jp-MarkdownHeadingCollapsed":true}},{"cell_type":"code","source":"train_data_path = data_folder / '/kaggle/input/g-research-crypto-forecasting/train.csv'\nif train_data_path.exists():\n    crypto_df = reduce_memory_usage(pd.read_csv(train_data_path))\nelse:\n    # If 'train.csv' is not available, use 'sample.csv'\n    sample_data_path = data_folder / 'sample.csv'\n    if sample_data_path.exists():\n        crypto_df = reduce_memory_usage(pd.read_csv(sample_data_path))\n    else:\n        print(\"No suitable data file found. Please ensure 'train.csv' or 'sample.csv' is available in the 'data' folder.\")\n        raise FileNotFoundError(\"No data file found in 'data' directory.\")","metadata":{"execution":{"iopub.status.busy":"2024-11-16T10:47:20.115733Z","iopub.execute_input":"2024-11-16T10:47:20.116120Z","iopub.status.idle":"2024-11-16T10:48:39.548124Z","shell.execute_reply.started":"2024-11-16T10:47:20.116087Z","shell.execute_reply":"2024-11-16T10:48:39.546479Z"},"trusted":true,"jupyter":{"source_hidden":true},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Load asset details\nasset_details_path = data_folder / '/kaggle/input/g-research-crypto-forecasting/asset_details.csv'\nif asset_details_path.exists():\n    asset_details = pd.read_csv(asset_details_path)\nelse:\n    print(\"Asset details file 'asset_details.csv' not found in 'data' directory.\")\n    raise FileNotFoundError(\"Asset details file not found.\")","metadata":{"execution":{"iopub.status.busy":"2024-11-16T10:48:39.574124Z","iopub.execute_input":"2024-11-16T10:48:39.574529Z","iopub.status.idle":"2024-11-16T10:48:39.586381Z","shell.execute_reply.started":"2024-11-16T10:48:39.574490Z","shell.execute_reply":"2024-11-16T10:48:39.584817Z"},"trusted":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Fill missing timestamps and fill missing values using backward and forward fill.\n","metadata":{"jp-MarkdownHeadingCollapsed":true}},{"cell_type":"code","source":"preprocessed_assets = []\nunique_ids = crypto_df['Asset_ID'].unique()\nfor id in unique_ids:\n    asset_df = fill_timestamps(crypto_df, id)\n    asset_df.loc[:, 'Asset_ID'] = id\n    preprocessed_assets.append(asset_df)\n    \ndel crypto_df\ngc.collect()\n\npreprocessed_df = pd.concat(preprocessed_assets)\npreprocessed_df.reset_index(inplace=True)\npreprocessed_df = reduce_memory_usage(preprocessed_df)\nsorted_indices = np.argsort(preprocessed_df['timestamp'])\npreprocessed_df = preprocessed_df.iloc[sorted_indices]\n\n# Optional: Sample a fraction of the data for faster processing during development\n# preprocessed_df = preprocessed_df.sample(frac=0.1, random_state=50)","metadata":{"execution":{"iopub.status.busy":"2024-11-16T10:48:55.919313Z","iopub.execute_input":"2024-11-16T10:48:55.919931Z","iopub.status.idle":"2024-11-16T10:49:22.478295Z","shell.execute_reply.started":"2024-11-16T10:48:55.919882Z","shell.execute_reply":"2024-11-16T10:49:22.476497Z"},"trusted":true,"jupyter":{"source_hidden":true},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 3. Feature Engineering (Frypto)\n\n","metadata":{"jp-MarkdownHeadingCollapsed":true}},{"cell_type":"code","source":"%%time\n%load_ext memory_profiler\n%memit\nprint(f\"using frypto for Feature engineering {len(preprocessed_df)} row....\\n\")\nimport frypto\n\n\nchunk_size = 100_000  \nfinal_df = pd.DataFrame()\n\nfor asset_id in tqdm(unique_ids, desc=\"Processing Assets\"):\n    print(f'Feature engineering at asset_id {asset_id}')\n    asset_df = preprocessed_df[preprocessed_df['Asset_ID'] == asset_id]\n    \n    num_chunks = (len(asset_df) // chunk_size) + 1\n    for i in range(num_chunks):\n        start_idx = i * chunk_size\n        end_idx = start_idx + chunk_size\n        chunk_df = asset_df.iloc[start_idx:end_idx].copy()\n        feature_engineer = frypto.AllFeatures(chunk_df) # Frypto goes here\n        df = feature_engineer.compute() # Frypto goes here\n\n        # Append processed chunk to final DataFrame\n        final_df = pd.concat([final_df, df], ignore_index=True)\n\n        # Free memory\n        del chunk_df, feature_engineer\n        gc.collect()\nfinal_df = pd.concat([preprocessed_df, final_df], axis=\"columns\")\n# del preprocessed_df\nprint()\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-11-16T10:52:16.537571Z","iopub.execute_input":"2024-11-16T10:52:16.538100Z","iopub.status.idle":"2024-11-16T11:00:59.617328Z","shell.execute_reply.started":"2024-11-16T10:52:16.538049Z","shell.execute_reply":"2024-11-16T11:00:59.615261Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Apply memory optimization\nfinal_df = reduce_memory_usage(final_df)\ngc.collect()\nfinal_df.sample(5)","metadata":{"execution":{"iopub.status.busy":"2024-11-16T11:00:59.621178Z","iopub.execute_input":"2024-11-16T11:00:59.621778Z","iopub.status.idle":"2024-11-16T11:01:24.288549Z","shell.execute_reply.started":"2024-11-16T11:00:59.621714Z","shell.execute_reply":"2024-11-16T11:01:24.287327Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"final_df.columns","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-16T11:01:24.290188Z","iopub.execute_input":"2024-11-16T11:01:24.290686Z","iopub.status.idle":"2024-11-16T11:01:24.299281Z","shell.execute_reply.started":"2024-11-16T11:01:24.290632Z","shell.execute_reply":"2024-11-16T11:01:24.297953Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Taking samples from the dataset to train the models faster","metadata":{}},{"cell_type":"code","source":"df_with_features = final_df.sample(frac=0.1, random_state=50)\ndf_without_features = preprocessed_df.sample(frac=0.1, random_state=50)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-16T11:01:24.302320Z","iopub.execute_input":"2024-11-16T11:01:24.303495Z","iopub.status.idle":"2024-11-16T11:01:30.153375Z","shell.execute_reply.started":"2024-11-16T11:01:24.303436Z","shell.execute_reply":"2024-11-16T11:01:30.152027Z"},"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 4. Training Models","metadata":{"jp-MarkdownHeadingCollapsed":true}},{"cell_type":"markdown","source":"### Training function","metadata":{"jp-MarkdownHeadingCollapsed":true}},{"cell_type":"code","source":"EXCLUDE_COLUMNS = ['timestamp', 'Asset_ID', 'Date', 'Target', 'Volume', 'Count']\ndef train_models(df, dataset_name):\n    print(f\"\\nTraining models on dataset: {dataset_name}\")\n    \n    # Initialize variables for storing results\n    oof_predictions = []\n    oof_targets = []\n    feature_importances = pd.DataFrame()\n    \n    # Get the unique asset IDs\n    unique_ids = df['Asset_ID'].unique()\n    \n    for asset_id in unique_ids:\n        print(f\"\\nTraining model for Asset ID {asset_id}\")\n        \n        # Filter data for the asset\n        asset_data = df[df['Asset_ID'] == asset_id].copy()\n        asset_data.reset_index(drop=True, inplace=True)\n        \n        # Skip assets with insufficient data\n        if len(asset_data) < 500:\n            print(f\"Not enough data for Asset ID {asset_id}. Skipping.\")\n            continue\n        \n        # Define features and target\n        TARGET = 'Target'\n        FEATURES = [col for col in asset_data.columns if col not in EXCLUDE_COLUMNS]\n        \n        # Handle missing values and ensure numeric data\n        asset_data[FEATURES] = asset_data[FEATURES].apply(pd.to_numeric, errors='coerce')\n        asset_data[FEATURES] = asset_data[FEATURES].fillna(0)\n        asset_data[TARGET] = asset_data[TARGET].fillna(0)\n        \n        # Initialize OOF predictions\n        oof_pred = np.zeros(len(asset_data))\n        \n        # Time series split\n        tscv = TimeSeriesSplit(n_splits=5)\n        for fold, (train_idx, valid_idx) in enumerate(tscv.split(asset_data)):\n            print(f\"Fold {fold + 1}\")\n            \n            X_train, X_valid = asset_data.loc[train_idx, FEATURES], asset_data.loc[valid_idx, FEATURES]\n            y_train, y_valid = asset_data.loc[train_idx, TARGET], asset_data.loc[valid_idx, TARGET]\n\n            # Replace infinite values and fill NaNs with appropriate values in features\n            X_train.replace([np.inf, -np.inf], np.nan, inplace=True)\n            X_valid.replace([np.inf, -np.inf], np.nan, inplace=True)\n            X_train.fillna(0, inplace=True)\n            X_valid.fillna(0, inplace=True)\n    \n            # Replace infinite values and fill NaNs in target variable\n            y_train.replace([np.inf, -np.inf], np.nan, inplace=True)\n            y_valid.replace([np.inf, -np.inf], np.nan, inplace=True)\n            y_train.fillna(0, inplace=True)\n            y_valid.fillna(0, inplace=True)\n    \n            # Ensure that there are no more invalid values in the target\n            if np.isnan(y_train.values).any() or np.isinf(y_train.values).any():\n                print(\"There are still NaN or infinite values in y_train after cleaning.\")\n                continue\n            if np.isnan(y_valid.values).any() or np.isinf(y_valid.values).any():\n                print(\"There are still NaN or infinite values in y_valid after cleaning.\")\n                continue\n            \n            # Scale the data\n            scaler = StandardScaler()\n            X_train_scaled = scaler.fit_transform(X_train)\n            X_valid_scaled = scaler.transform(X_valid)\n            \n            # Initialize and train the model\n            model = LGBMRegressor(\n                objective='regression',\n                learning_rate=0.05,\n                num_leaves=31,\n                n_estimators=500,\n                random_state=42 + fold,\n                n_jobs=-1,\n                verbosity=1\n                \n            )\n            model.fit(\n                X_train_scaled, y_train,\n                eval_set=[(X_valid_scaled, y_valid)],\n                eval_metric='rmse',\n            )\n            \n            # Predict on validation set\n            y_pred_valid = model.predict(X_valid_scaled)\n            oof_pred[valid_idx] = y_pred_valid\n            \n            # Collect feature importances\n            fold_importance_df = pd.DataFrame()\n            fold_importance_df[\"feature\"] = FEATURES\n            fold_importance_df[\"importance\"] = model.feature_importances_\n            fold_importance_df[\"fold\"] = fold + 1\n            fold_importance_df[\"asset_id\"] = asset_id\n            feature_importances = pd.concat([feature_importances, fold_importance_df], axis=0)\n            \n            # Clean up\n            del X_train, X_valid, y_train, y_valid, X_train_scaled, X_valid_scaled\n            gc.collect()\n        \n        # Calculate OOF RMSE for the asset\n        asset_rmse = np.sqrt(mean_squared_error(asset_data[TARGET], oof_pred))\n        print(f\"Asset ID {asset_id} OOF RMSE: {asset_rmse:.6f}\")\n        \n        # Store predictions and targets\n        oof_predictions.extend(oof_pred)\n        oof_targets.extend(asset_data[TARGET])\n        \n        # Clean up\n        del asset_data, oof_pred\n        gc.collect()\n    \n    # Calculate overall OOF RMSE\n    overall_rmse = np.sqrt(mean_squared_error(oof_targets, oof_predictions))\n    print(f\"\\nOverall OOF RMSE for dataset '{dataset_name}': {overall_rmse:.6f}\")\n    \n    return oof_predictions, oof_targets, feature_importances\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-16T11:01:30.155507Z","iopub.execute_input":"2024-11-16T11:01:30.156100Z","iopub.status.idle":"2024-11-16T11:01:30.180689Z","shell.execute_reply.started":"2024-11-16T11:01:30.156047Z","shell.execute_reply":"2024-11-16T11:01:30.179483Z"},"jupyter":{"source_hidden":true},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Train the model on the dataset with and without frypto features","metadata":{"jp-MarkdownHeadingCollapsed":true}},{"cell_type":"code","source":"# Train model with engineered features\noof_preds_with_features, oof_targets_with_features, feature_importances_with = train_models(df_with_features, 'with_features')\n\n# Train model without engineered features\noof_preds_without_features, oof_targets_without_features, feature_importances_without = train_models(df_without_features, 'without_features')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-16T11:30:45.322360Z","iopub.execute_input":"2024-11-16T11:30:45.322871Z","iopub.status.idle":"2024-11-16T11:59:57.659175Z","shell.execute_reply.started":"2024-11-16T11:30:45.322817Z","shell.execute_reply":"2024-11-16T11:59:57.657564Z"},"_kg_hide-output":true,"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 5. Results & Analysis","metadata":{}},{"cell_type":"markdown","source":"### Aggreegate feature importance","metadata":{}},{"cell_type":"code","source":"def aggregate_feature_importances(feature_importances):\n    # Group by feature and calculate mean importance\n    aggregated_importance = feature_importances.groupby('feature')['importance'].mean().reset_index()\n    # Sort features by importance\n    aggregated_importance.sort_values(by='importance', ascending=False, inplace=True)\n    return aggregated_importance\n\n# For model with features\naggregated_importance_with = aggregate_feature_importances(feature_importances_with)\n\n# For model without features\naggregated_importance_without = aggregate_feature_importances(feature_importances_without)\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-16T11:59:57.674455Z","iopub.execute_input":"2024-11-16T11:59:57.674898Z","iopub.status.idle":"2024-11-16T11:59:57.699016Z","shell.execute_reply.started":"2024-11-16T11:59:57.674859Z","shell.execute_reply":"2024-11-16T11:59:57.697548Z"},"jupyter":{"source_hidden":true},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Plot Feature Importance","metadata":{}},{"cell_type":"code","source":"def plot_feature_importances(aggregated_importance, dataset_name, top_n=20):\n    plt.figure(figsize=(10, 8))\n    sns.barplot(\n        x='importance', \n        y='feature', \n        data=aggregated_importance.head(top_n),\n        palette='viridis'\n    )\n    plt.title(f'Top {top_n} Feature Importances for {dataset_name}')\n    plt.tight_layout()\n    plt.savefig(f'feature_importance_{dataset_name}.png')\n    plt.show()\n\n# Model with features\nplot_feature_importances(aggregated_importance_with, 'Model with frypto Features')\n\n# Model without features\nplot_feature_importances(aggregated_importance_without, 'Model without frypto Features')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-16T11:59:57.704860Z","iopub.execute_input":"2024-11-16T11:59:57.705334Z","iopub.status.idle":"2024-11-16T11:59:59.246708Z","shell.execute_reply.started":"2024-11-16T11:59:57.705281Z","shell.execute_reply":"2024-11-16T11:59:59.245365Z"},"_kg_hide-input":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Calculate RMSE for models with and without Frypto features","metadata":{}},{"cell_type":"code","source":"rmse_with_features = np.sqrt(mean_squared_error(oof_targets_with_features, oof_preds_with_features))\nrmse_without_features = np.sqrt(mean_squared_error(oof_targets_without_features, oof_preds_without_features))\n\nprint(f\"\\nRMSE with engineered features: {rmse_with_features:.6f}\")\nprint(f\"RMSE without engineered features: {rmse_without_features:.6f}\")\nprint(f\"RMSE Improvement: {rmse_without_features - rmse_with_features:.6f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2024-11-16T12:00:13.697919Z","iopub.execute_input":"2024-11-16T12:00:13.699176Z","iopub.status.idle":"2024-11-16T12:00:14.283668Z","shell.execute_reply.started":"2024-11-16T12:00:13.699123Z","shell.execute_reply":"2024-11-16T12:00:14.282323Z"},"jupyter":{"source_hidden":true},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 📈 Summary and Takeaways\n\n1. **Efficiency**: Frypto processed 29M rows and generated 42 features in under 9 minutes.\n2. **Scalability**: Chunked processing and memory optimization allowed us to handle a massive dataset efficiently.\n3. **Impact on Model Performance**:\n   - **With Frypto Features**: 69.573486\n   - **Without Features**: 84.883070\n   - **Improvement**: 15.309585\n\nThis demonstrates the power of advanced feature engineering in improving prediction accuracy for crypto datasets.datasets.\r\n","metadata":{"jp-MarkdownHeadingCollapsed":true}}]}