{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":84253,"databundleVersionId":9421888,"sourceType":"competition"}],"dockerImageVersionId":30761,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"raw","source":"","metadata":{}},{"cell_type":"raw","source":"# Introduction\nIn this notebook you all are given a glimpse of some data preprocessing techniques, a few outliers have been removed from the data i.e. data points that are significantly different from the other data points in a dataset. And the data is also scaled on a scale of -1 to 1 to this ensures uniformity in features importance as on an unscaled data some features might influence the prediction more than others. Further you will also see a feature selection algorithm being used in this notebook that select best features from the data for the prediction.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-08-24T09:41:45.341263Z","iopub.execute_input":"2024-08-24T09:41:45.341762Z","iopub.status.idle":"2024-08-24T09:41:47.341512Z","shell.execute_reply.started":"2024-08-24T09:41:45.341703Z","shell.execute_reply":"2024-08-24T09:41:47.340297Z"}}},{"cell_type":"markdown","source":"# Importing Libraries and Loading Data","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.feature_selection import RFE\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.linear_model import LinearRegression","metadata":{"execution":{"iopub.status.busy":"2024-08-25T15:40:58.163738Z","iopub.execute_input":"2024-08-25T15:40:58.164023Z","iopub.status.idle":"2024-08-25T15:40:59.776038Z","shell.execute_reply.started":"2024-08-25T15:40:58.163990Z","shell.execute_reply":"2024-08-25T15:40:59.775111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/first-steps-in-data-science-new/train (7).csv\") \ntest = pd.read_csv(\"/kaggle/input/first-steps-in-data-science-new/test (4).csv\")","metadata":{"execution":{"iopub.status.busy":"2024-08-25T15:40:59.779831Z","iopub.execute_input":"2024-08-25T15:40:59.780298Z","iopub.status.idle":"2024-08-25T15:41:03.134409Z","shell.execute_reply.started":"2024-08-25T15:40:59.780260Z","shell.execute_reply":"2024-08-25T15:41:03.133635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Preprocessing","metadata":{}},{"cell_type":"code","source":"train","metadata":{"execution":{"iopub.status.busy":"2024-08-25T15:41:10.546776Z","iopub.execute_input":"2024-08-25T15:41:10.547133Z","iopub.status.idle":"2024-08-25T15:41:10.592816Z","shell.execute_reply.started":"2024-08-25T15:41:10.547101Z","shell.execute_reply":"2024-08-25T15:41:10.591940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After loading the data and taking a peek at it you can see the first column is just index and that is a useless feature and might even effect our prediction so we will remove it from both train and test dataset.","metadata":{}},{"cell_type":"code","source":"train= train.drop(train.columns[0], axis=1)\ntest= test.drop(test.columns[0], axis=1)","metadata":{"execution":{"iopub.status.busy":"2024-08-25T07:06:24.265812Z","iopub.execute_input":"2024-08-25T07:06:24.266522Z","iopub.status.idle":"2024-08-25T07:06:24.295347Z","shell.execute_reply.started":"2024-08-25T07:06:24.266485Z","shell.execute_reply":"2024-08-25T07:06:24.294525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.array(train.isnull().sum())","metadata":{"execution":{"iopub.status.busy":"2024-08-25T15:41:14.469014Z","iopub.execute_input":"2024-08-25T15:41:14.469384Z","iopub.status.idle":"2024-08-25T15:41:14.481891Z","shell.execute_reply.started":"2024-08-25T15:41:14.469346Z","shell.execute_reply":"2024-08-25T15:41:14.480771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"any_duplicates = train.duplicated().any()\n\nif any_duplicates:\n    print(\"There are duplicates in the DataFrame.\")\nelse:\n    print(\"There are no duplicates in the DataFrame.\")","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:41:59.430947Z","iopub.execute_input":"2024-08-24T09:41:59.431415Z","iopub.status.idle":"2024-08-24T09:41:59.639263Z","shell.execute_reply.started":"2024-08-24T09:41:59.431373Z","shell.execute_reply":"2024-08-24T09:41:59.638045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We checked for null and duplicate values in the training set and there weren't any so that means our data is clean in that aspect. Now we will remove outliers from the data using Z score as our parameter, there are other ways to remove outliers you all are encouraged to check them out.","metadata":{}},{"cell_type":"markdown","source":"## Removing Outliers","metadata":{}},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:42:01.58389Z","iopub.execute_input":"2024-08-24T09:42:01.584419Z","iopub.status.idle":"2024-08-24T09:42:01.592438Z","shell.execute_reply.started":"2024-08-24T09:42:01.584372Z","shell.execute_reply":"2024-08-24T09:42:01.59103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\n\ndef remove_outliers_iqr(df):\n    # Copy the DataFrame to avoid changing the original data\n    df_clean = df.copy()\n    \n    # For each column in the DataFrame\n    for col in df.columns:\n      \n        \n        # Calculate Q1 (25th percentile) and Q3 (75th percentile)\n        Q1 = df[col].quantile(0.25)\n        Q3 = df[col].quantile(0.75)\n        \n        # Calculate IQR (Interquartile Range)\n        IQR = Q3 - Q1\n        \n        # Define bounds for outliers\n        lower_bound = Q1 - 1.5 * IQR\n        upper_bound = Q3 + 1.5 * IQR\n        \n        # Filter the DataFrame to exclude outliers\n        df_clean = df_clean[(df_clean[col] >= lower_bound) & (df_clean[col] <= upper_bound)]\n    \n    return df_clean\n\n# Usage\ntrain_no_outliers= remove_outliers_iqr(train)","metadata":{"execution":{"iopub.status.busy":"2024-08-25T04:29:51.589948Z","iopub.execute_input":"2024-08-25T04:29:51.590485Z","iopub.status.idle":"2024-08-25T04:29:52.346030Z","shell.execute_reply.started":"2024-08-25T04:29:51.590440Z","shell.execute_reply":"2024-08-25T04:29:52.344817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"z_scores = np.abs((train - train.mean()) / train.std())\n\n# Set a threshold\nthreshold = 4.5\n\n# Filter out rows with Z-score greater than the threshold\ntrain_no_outliers = train[(z_scores < threshold).all(axis=1)]\n","metadata":{"execution":{"iopub.status.busy":"2024-08-25T16:16:43.129371Z","iopub.execute_input":"2024-08-25T16:16:43.129763Z","iopub.status.idle":"2024-08-25T16:16:43.161784Z","shell.execute_reply.started":"2024-08-25T16:16:43.129727Z","shell.execute_reply":"2024-08-25T16:16:43.160992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_no_outliers.shape","metadata":{"execution":{"iopub.status.busy":"2024-08-25T16:16:44.025470Z","iopub.execute_input":"2024-08-25T16:16:44.025869Z","iopub.status.idle":"2024-08-25T16:16:44.032116Z","shell.execute_reply.started":"2024-08-25T16:16:44.025831Z","shell.execute_reply":"2024-08-25T16:16:44.030957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see our data got reduced from 7500 to 6819, maybe this is enough maybe this is not or maybe this was a bad idea? try different thigs and see for yourself. Let's move on to scaling the data, here we have used minmaxscaler that scales data on a scale of -1 to 1","metadata":{}},{"cell_type":"markdown","source":"## Scaling","metadata":{}},{"cell_type":"code","source":"train_no_outliers = train_no_outliers.reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2024-08-25T16:16:46.625061Z","iopub.execute_input":"2024-08-25T16:16:46.625434Z","iopub.status.idle":"2024-08-25T16:16:46.631647Z","shell.execute_reply.started":"2024-08-25T16:16:46.625388Z","shell.execute_reply":"2024-08-25T16:16:46.630805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = train_no_outliers.drop(columns=['target'])\ny = train_no_outliers['target']","metadata":{"execution":{"iopub.status.busy":"2024-08-25T16:16:49.057758Z","iopub.execute_input":"2024-08-25T16:16:49.058125Z","iopub.status.idle":"2024-08-25T16:16:49.066251Z","shell.execute_reply.started":"2024-08-25T16:16:49.058089Z","shell.execute_reply":"2024-08-25T16:16:49.065212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Initialize MinMaxScaler with the desired range (-1 to 1)\nscaler = MinMaxScaler(feature_range=(-1, 1))\n\n# Fit and transform the data\nX_final = pd.DataFrame(scaler.fit_transform(X), columns=X.columns)\ntest_final = pd.DataFrame(scaler.transform(test), columns = test.columns)","metadata":{"execution":{"iopub.status.busy":"2024-08-25T16:16:49.258604Z","iopub.execute_input":"2024-08-25T16:16:49.258930Z","iopub.status.idle":"2024-08-25T16:16:49.333166Z","shell.execute_reply.started":"2024-08-25T16:16:49.258897Z","shell.execute_reply":"2024-08-25T16:16:49.332335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install pytorch_tabnet\n","metadata":{"execution":{"iopub.status.busy":"2024-08-25T16:02:43.122008Z","iopub.execute_input":"2024-08-25T16:02:43.122370Z","iopub.status.idle":"2024-08-25T16:02:46.165280Z","shell.execute_reply.started":"2024-08-25T16:02:43.122334Z","shell.execute_reply":"2024-08-25T16:02:46.164107Z"},"jupyter":{"source_hidden":true,"outputs_hidden":true},"collapsed":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note: Any data transormation technique you apply to the train data remember to apply it to the test data aswell","metadata":{}},{"cell_type":"code","source":"# Import necessary libraries\nfrom pytorch_tabnet.tab_model import TabNetRegressor\nimport numpy as np\nfrom sklearn.metrics import mean_squared_error, r2_score\nfrom sklearn.model_selection import KFold\nfrom sklearn.preprocessing import StandardScaler\n\n# Initialize your data (already defined)\ntrain_data = X_final\ntrain_target = y\n\n# Initialize KFold cross-validation\nn_folds = 15\nkf = KFold(n_splits=n_folds, shuffle=True, random_state=42)\n\n# Standardize features for TabNet\nscaler = StandardScaler()\ntrain_data_scaled = scaler.fit_transform(train_data)\ntest_data_scaled = scaler.transform(test_final)\n\n# Convert target to NumPy array and reshape\ntrain_target = train_target.values.reshape(-1, 1)\n\n# Initialize lists to store metrics\nrmse_scores = []\nr2_scores = []\nweights = []\n\n# Array to store the predictions for the test set\ntest_preds = np.zeros((test_data_scaled.shape[0],))\n\n# Loop over each fold in the KFold cross-validation\nfor fold, (train_idx, val_idx) in enumerate(kf.split(train_data_scaled)):\n    # Split the data into training and validation sets for the current fold\n    X_train, y_train = train_data_scaled[train_idx], train_target[train_idx]\n    X_val, y_val = train_data_scaled[val_idx], train_target[val_idx]\n\n    # Initialize the TabNet Regressor\n    model = TabNetRegressor(verbose=0, seed=42,device_name='cuda')\n\n    # Fit the model on the training data\n    model.fit(\n        X_train, y_train,\n        eval_set=[(X_val, y_val)],\n        patience=10, \n        max_epochs=100,\n        eval_metric=['rmse']\n    )\n\n    # Predict the target for the validation set\n    val_preds = model.predict(X_val)\n\n    # Calculate the RMSE for the current fold\n    fold_rmse = mean_squared_error(y_val, val_preds, squared=False)\n    rmse_scores.append(fold_rmse)\n\n    # Calculate the R² for the current fold\n    fold_r2 = r2_score(y_val, val_preds)\n    r2_scores.append(fold_r2)\n\n    # Calculate the weight for the current fold (inverse of RMSE)\n    weights.append(1 / fold_rmse)\n\n    # Print the RMSE and R² for the current fold\n    print(f'Fold {fold+1} RMSE: {fold_rmse}, R²: {fold_r2}')\n\n    # Predict the target for the test data and update test_preds\n    test_preds += model.predict(test_data_scaled).ravel() * weights[fold]\n\n# Normalize the test predictions by dividing by the sum of the weights\ntest_preds /= sum(weights)\n\n# Calculate the mean RMSE and R² across all folds\nmean_rmse = np.mean(rmse_scores)\nmean_r2 = np.mean(r2_scores)\n\n# Print the final averaged metrics\nprint(f'Average RMSE: {mean_rmse}')\nprint(f'Average R²: {mean_r2}')\n","metadata":{"execution":{"iopub.status.busy":"2024-08-25T07:08:15.234741Z","iopub.execute_input":"2024-08-25T07:08:15.235138Z","iopub.status.idle":"2024-08-25T07:15:17.190775Z","shell.execute_reply.started":"2024-08-25T07:08:15.235101Z","shell.execute_reply":"2024-08-25T07:15:17.189781Z"},"jupyter":{"source_hidden":true,"outputs_hidden":true},"collapsed":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Selection and K-Fold Cross validation\n\nRFE (Recursive Feature Elimination) : Recursive Feature Elimination (RFE) is a feature selection method that helps you identify the most important features in a dataset by recursively training a model and removing the least important features. Here we have used Linear regression as that model which is known as the estimator and the final no. of features we have now is 150. We will train our model now with these 150 features.\n\nK-Fold CV: What happens here is that we divide our data in certain parts, say 15. Now We validate our model for each fold while training on other folds. I have used a basic Linear regression model here with no hyperparameter tuning or regularization techniques.\nThe final prediction is the average of all the predictions from different folds.","metadata":{}},{"cell_type":"code","source":"from sklearn import linear_model\nestimator = linear_model.Lasso(alpha=0.00001)\n","metadata":{"execution":{"iopub.status.busy":"2024-08-25T16:02:51.613533Z","iopub.execute_input":"2024-08-25T16:02:51.613939Z","iopub.status.idle":"2024-08-25T16:02:51.618931Z","shell.execute_reply.started":"2024-08-25T16:02:51.613900Z","shell.execute_reply":"2024-08-25T16:02:51.617995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"1.1592469593313177\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.linear_model import Lasso\nfrom sklearn.model_selection import GridSearchCV\n\n# Initialize the base model (Lasso Regression) for RFE\nlasso = Lasso()\n\n# Set up a range of alpha values to test\nalpha_values = {'alpha': [0.001, 0.01, 0.1, 1, 10, 100]}\n\n# Initialize GridSearchCV with Lasso and the alpha values\ngrid_search = GridSearchCV(lasso, alpha_values, cv=5, scoring='neg_mean_squared_error')\n\n# Fit the model with GridSearch to find the best alpha\ngrid_search.fit(rfe_train_data, train_target)\n\n# Best alpha found by cross-validation\nbest_alpha = grid_search.best_params_['alpha']\nprint(f'Best alpha: {best_alpha}')\n\n# Now use this alpha value for your model\nselector = RFE(Lasso(alpha=best_alpha), n_features_to_select=72, step=1)\n\n# Fit the RFE model on the training data\nselector.fit(train_data, train_target)\n\n# Follow the rest of your original code...\n","metadata":{"execution":{"iopub.status.busy":"2024-08-25T16:49:17.906650Z","iopub.execute_input":"2024-08-25T16:49:17.907020Z","iopub.status.idle":"2024-08-25T16:49:20.211126Z","shell.execute_reply.started":"2024-08-25T16:49:17.906983Z","shell.execute_reply":"2024-08-25T16:49:20.210208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"base_regressors = [\n    ('lr', estimator),\n    ('ls', Lasso(0.001))\n]\nmeta_regressor = MLPRegressor(hidden_layer_sizes=(400, 200, 100, 50), activation='relu', solver='adam', random_state=42)\n\n\nstacking_model = StackingRegressor(estimators=base_regressors, final_estimator=meta_regressor)","metadata":{"execution":{"iopub.status.busy":"2024-08-25T16:51:42.399328Z","iopub.execute_input":"2024-08-25T16:51:42.399787Z","iopub.status.idle":"2024-08-25T16:51:42.406398Z","shell.execute_reply.started":"2024-08-25T16:51:42.399739Z","shell.execute_reply":"2024-08-25T16:51:42.405412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.neural_network import MLPRegressor\nfrom sklearn.ensemble import StackingRegressor","metadata":{"execution":{"iopub.status.busy":"2024-08-25T16:51:40.372355Z","iopub.execute_input":"2024-08-25T16:51:40.372785Z","iopub.status.idle":"2024-08-25T16:51:40.457532Z","shell.execute_reply.started":"2024-08-25T16:51:40.372743Z","shell.execute_reply":"2024-08-25T16:51:40.456806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Initialize lists to store metrics\nfrom sklearn.metrics import mean_squared_error, r2_score  # Import the required metrics\nn_folds = 15\n# List to store the weights (based on RMSE) for each fold\nweights = []\n\n# Initialize KFold cross-validation\nkf = KFold(n_splits=n_folds, shuffle=True, random_state=42)\n\n# Assign your features (train data) and target (labels)\ntrain_data = X_final\ntrain_target = y\n\n# Initialize the base model (Linear Regression) for RFE\nestimator = LinearRegression()\n#estimator = linear_model.Lasso(alpha=0.001)\n# Initialize RFE (Recursive Feature Elimination) to select the top 150 features\nselector = RFE(estimator, n_features_to_select=72, step=1)\n\n# Fit the RFE model on the training data to select important features\nselector.fit(train_data, train_target)\n\n# Create a new training dataset with only the selected features\nrfe_train_data = train_data.iloc[:, selector.support_]\n\n# Apply the same feature selection to the test data\ntest_data = test_final.iloc[:, selector.support_]\n\n# Initialize an array to store the predictions for the test set\ntest_preds = np.zeros((test_data.shape[0],))\n\n\nrmse_scores = []\nr2_scores = []\n\n# Loop over each fold in the KFold cross-validation\nfor fold, (train_idx, val_idx) in enumerate(kf.split(train_data)):\n    # Split the data into training and validation sets for the current fold\n    X_train, y_train = rfe_train_data.iloc[train_idx], train_target[train_idx]\n    X_val, y_val = rfe_train_data.iloc[val_idx], train_target[val_idx]\n\n    # Initialize the model (Linear Regression) for the current fold\n    #model = Lasso(alpha=0.001)\n    model=stacking_model \n    # Train the model on the training data of the current fold\n    model.fit(X_train, y_train)\n\n    # Predict the target for the validation set\n    val_preds = model.predict(X_val)\n\n    # Calculate the RMSE for the current fold\n    fold_rmse = mean_squared_error(y_val, val_preds, squared=False)\n    rmse_scores.append(fold_rmse)\n\n    # Calculate the R² for the current fold\n    fold_r2 = r2_score(y_val, val_preds)\n    r2_scores.append(fold_r2)\n\n    # Calculate the weight for the current fold (inverse of RMSE)\n    weights.append(1 / fold_rmse)\n\n    # Print the RMSE and R² for the current fold\n    print(f'Fold {fold+1} RMSE: {fold_rmse}, R²: {fold_r2}')\n\n    # Predict the target for the test data and update test_preds\n    test_preds += model.predict(test_data) * weights[fold]\n\n# Normalize the test predictions by dividing by the sum of the weights\ntest_preds /= sum(weights)\n\n# Calculate the mean RMSE and R² across all folds\nmean_rmse = np.mean(rmse_scores)\nmean_r2 = np.mean(r2_scores)\n\n# Print the final averaged metrics\nprint(f'Average RMSE: {mean_rmse}')\nprint(f'Average R²: {mean_r2}')\n","metadata":{"execution":{"iopub.status.busy":"2024-08-25T17:19:40.943671Z","iopub.execute_input":"2024-08-25T17:19:40.944031Z","iopub.status.idle":"2024-08-25T17:21:25.155175Z","shell.execute_reply.started":"2024-08-25T17:19:40.943998Z","shell.execute_reply":"2024-08-25T17:21:25.153991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"stack:","metadata":{}},{"cell_type":"markdown","source":"1.983314793983133","metadata":{}},{"cell_type":"markdown","source":"Average RMSE: 1.9856860021001752(outlier 4.5)\n\n","metadata":{}},{"cell_type":"markdown","source":"1.9790831972371514(outlier3+72\n)","metadata":{}},{"cell_type":"markdown","source":"Average RMSE: 1.9788412386832899\naplha0.02","metadata":{}},{"cell_type":"markdown","source":"Average RMSE: 2.009902327769605\n1.9969842384302876(120)\n\n165->1.156998206761087","metadata":{}},{"cell_type":"markdown","source":"180->1.15748657065746\n","metadata":{}},{"cell_type":"code","source":"from sklearn.linear_model import Lasso, LinearRegression\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_error, r2_score\nimport numpy as np\n\n# Number of folds for cross-validation\nn_folds = 15\n\n# Initialize KFold cross-validation\nkf = KFold(n_splits=n_folds, shuffle=True, random_state=42)\n\n# Assign your features (train data) and target (labels)\ntrain_data = X_final.to_numpy()  # Convert to numpy array\ntrain_target = y.to_numpy()  # Convert to numpy array\n\n# Initialize lists to store the predictions and metrics\nstacked_train_preds = np.zeros((train_data.shape[0], 2))  # To store predictions from each model\ntest_data = test_final.to_numpy()  # Convert to numpy array\ntest_preds = np.zeros((test_data.shape[0], 2))\nweights = []\n\n# Initialize lists to store the final metrics\nrmse_scores = []\nr2_scores = []\n\n# Loop over each fold in the KFold cross-validation\nfor fold, (train_idx, val_idx) in enumerate(kf.split(train_data)):\n    # Split the data into training and validation sets for the current fold\n    X_train, y_train = train_data[train_idx], train_target[train_idx]\n    X_val, y_val = train_data[val_idx], train_target[val_idx]\n\n    # Initialize and train the Lasso model\n    lasso_model = Lasso(alpha=0.0001)\n    lasso_model.fit(X_train, y_train)\n\n    # Initialize and train the Linear Regression model\n    lr_model = LinearRegression()\n    lr_model.fit(X_train, y_train)\n\n    # Predict on the validation set with both models\n    val_preds_lasso = lasso_model.predict(X_val)\n    val_preds_lr = lr_model.predict(X_val)\n\n    # Store predictions from both models\n    stacked_train_preds[val_idx, 0] = val_preds_lasso\n    stacked_train_preds[val_idx, 1] = val_preds_lr\n\n    # Predict on the test data with both models and store weighted predictions\n    test_preds[:, 0] += lasso_model.predict(test_data) * (1 / n_folds)\n    test_preds[:, 1] += lr_model.predict(test_data) * (1 / n_folds)\n\n    # Combine the predictions for validation set (stacking)\n    combined_val_preds = 0.5 * val_preds_lasso + 0.5 * val_preds_lr\n\n    # Calculate the RMSE and R² for the current fold\n    fold_rmse = mean_squared_error(y_val, combined_val_preds, squared=False)\n    fold_r2 = r2_score(y_val, combined_val_preds)\n\n    # Store the metrics\n    rmse_scores.append(fold_rmse)\n    r2_scores.append(fold_r2)\n\n    # Print the RMSE and R² for the current fold\n    print(f'Fold {fold+1} RMSE: {fold_rmse}, R²: {fold_r2}')\n\n# Calculate the mean RMSE and R² across all folds\nmean_rmse = np.mean(rmse_scores)\nmean_r2 = np.mean(r2_scores)\n\n# Print the final averaged metrics\nprint(f'Average RMSE: {mean_rmse}')\nprint(f'Average R²: {mean_r2}')\n\n# Combine the predictions for the test set (stacking)\nfinal_test_preds = 0.5 * test_preds[:, 0] + 0.5 * test_preds[:, 1]\n\n# Print the final test predictions\nprint(f'Final Test Predictions Mean: {final_test_preds.mean()}')\n","metadata":{"execution":{"iopub.status.busy":"2024-08-25T04:26:33.651705Z","iopub.execute_input":"2024-08-25T04:26:33.652659Z","iopub.status.idle":"2024-08-25T04:26:36.794539Z","shell.execute_reply.started":"2024-08-25T04:26:33.652606Z","shell.execute_reply":"2024-08-25T04:26:36.793234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"1.1624764071537002","metadata":{}},{"cell_type":"markdown","source":"1.1594372520222074->original\n\n","metadata":{}},{"cell_type":"markdown","source":"# Generating Submission File and Conclusion","metadata":{}},{"cell_type":"code","source":"submission = pd.read_csv(\"/kaggle/input/first-steps-in-data-science-new/sample.csv\")\nsubmission['target'] = test_preds\nsubmission.to_csv(\"/kaggle/working/stacked.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2024-08-25T17:21:49.113998Z","iopub.execute_input":"2024-08-25T17:21:49.114746Z","iopub.status.idle":"2024-08-25T17:21:49.188838Z","shell.execute_reply.started":"2024-08-25T17:21:49.114703Z","shell.execute_reply":"2024-08-25T17:21:49.188087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You all saw how we implemented some data preprocessing technques and feature engineering methods in this notebook, but this is just the tip of iceberg, dive deep into the sea of data science and discover yourself some better techniques and methods that might or might not improve your score on leaderboard but will definitely improve your understanding of Machine Learning and Data Science.\n\nBest of Luck to you all!!","metadata":{}},{"cell_type":"code","source":"test_preds","metadata":{"execution":{"iopub.status.busy":"2024-08-25T17:13:08.257302Z","iopub.execute_input":"2024-08-25T17:13:08.257989Z","iopub.status.idle":"2024-08-25T17:13:08.264021Z","shell.execute_reply.started":"2024-08-25T17:13:08.257947Z","shell.execute_reply":"2024-08-25T17:13:08.263049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}