{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":84253,"databundleVersionId":9421888,"sourceType":"competition"}],"dockerImageVersionId":30761,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\nIn this notebook you all are given a glimpse of some data preprocessing techniques, a few outliers have been removed from the data i.e. data points that are significantly different from the other data points in a dataset. And the data is also scaled on a scale of -1 to 1 to this ensures uniformity in features importance as on an unscaled data some features might influence the prediction more than others. Further you will also see a feature selection algorithm being used in this notebook that select best features from the data for the prediction.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-08-24T09:41:45.341263Z","iopub.execute_input":"2024-08-24T09:41:45.341762Z","iopub.status.idle":"2024-08-24T09:41:47.341512Z","shell.execute_reply.started":"2024-08-24T09:41:45.341703Z","shell.execute_reply":"2024-08-24T09:41:47.340297Z"}}},{"cell_type":"markdown","source":"# Importing Libraries and Loading Data","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom sklearn.preprocessing import MinMaxScaler\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.feature_selection import RFE\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn.linear_model import LinearRegression","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/first-steps-in-data-science-new/train (7).csv\") \ntest = pd.read_csv(\"/kaggle/input/first-steps-in-data-science-new/test (4).csv\")","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:41:47.343446Z","iopub.execute_input":"2024-08-24T09:41:47.344042Z","iopub.status.idle":"2024-08-24T09:41:50.469730Z","shell.execute_reply.started":"2024-08-24T09:41:47.343998Z","shell.execute_reply":"2024-08-24T09:41:50.468395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Preprocessing","metadata":{}},{"cell_type":"code","source":"train","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:41:50.472216Z","iopub.execute_input":"2024-08-24T09:41:50.472645Z","iopub.status.idle":"2024-08-24T09:41:50.526471Z","shell.execute_reply.started":"2024-08-24T09:41:50.472603Z","shell.execute_reply":"2024-08-24T09:41:50.525096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After loading the data and taking a peek at it you can see the first column is just index and that is a useless feature and might even effect our prediction so we will remove it from both train and test dataset.","metadata":{}},{"cell_type":"code","source":"train= train.drop(train.columns[0], axis=1)\ntest= test.drop(test.columns[0], axis=1)","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:41:53.384732Z","iopub.execute_input":"2024-08-24T09:41:53.385154Z","iopub.status.idle":"2024-08-24T09:41:53.415582Z","shell.execute_reply.started":"2024-08-24T09:41:53.385116Z","shell.execute_reply":"2024-08-24T09:41:53.413817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.array(train.isnull().sum())","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:41:54.855667Z","iopub.execute_input":"2024-08-24T09:41:54.856145Z","iopub.status.idle":"2024-08-24T09:41:54.869088Z","shell.execute_reply.started":"2024-08-24T09:41:54.856102Z","shell.execute_reply":"2024-08-24T09:41:54.867477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"any_duplicates = train.duplicated().any()\n\nif any_duplicates:\n    print(\"There are duplicates in the DataFrame.\")\nelse:\n    print(\"There are no duplicates in the DataFrame.\")","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:41:59.430947Z","iopub.execute_input":"2024-08-24T09:41:59.431415Z","iopub.status.idle":"2024-08-24T09:41:59.639263Z","shell.execute_reply.started":"2024-08-24T09:41:59.431373Z","shell.execute_reply":"2024-08-24T09:41:59.638045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We checked for null and duplicate values in the training set and there weren't any so that means our data is clean in that aspect. Now we will remove outliers from the data using Z score as our parameter, there are other ways to remove outliers you all are encouraged to check them out.","metadata":{}},{"cell_type":"markdown","source":"## Removing Outliers","metadata":{}},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:42:01.583890Z","iopub.execute_input":"2024-08-24T09:42:01.584419Z","iopub.status.idle":"2024-08-24T09:42:01.592438Z","shell.execute_reply.started":"2024-08-24T09:42:01.584372Z","shell.execute_reply":"2024-08-24T09:42:01.591030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"z_scores = np.abs((train - train.mean()) / train.std())\n\n# Set a threshold\nthreshold = 3.5\n\n# Filter out rows with Z-score greater than the threshold\ntrain_no_outliers = train[(z_scores < threshold).all(axis=1)]\n","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:42:11.582981Z","iopub.execute_input":"2024-08-24T09:42:11.583498Z","iopub.status.idle":"2024-08-24T09:42:11.644060Z","shell.execute_reply.started":"2024-08-24T09:42:11.583452Z","shell.execute_reply":"2024-08-24T09:42:11.642500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_no_outliers.shape","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:42:13.467819Z","iopub.execute_input":"2024-08-24T09:42:13.468801Z","iopub.status.idle":"2024-08-24T09:42:13.477820Z","shell.execute_reply.started":"2024-08-24T09:42:13.468750Z","shell.execute_reply":"2024-08-24T09:42:13.476487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see our data got reduced from 7500 to 6819, maybe this is enough maybe this is not or maybe this was a bad idea? try different thigs and see for yourself. Let's move on to scaling the data, here we have used minmaxscaler that scales data on a scale of -1 to 1","metadata":{}},{"cell_type":"markdown","source":"## Scaling","metadata":{}},{"cell_type":"code","source":"train_no_outliers = train_no_outliers.reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:42:15.524421Z","iopub.execute_input":"2024-08-24T09:42:15.524944Z","iopub.status.idle":"2024-08-24T09:42:15.536043Z","shell.execute_reply.started":"2024-08-24T09:42:15.524900Z","shell.execute_reply":"2024-08-24T09:42:15.534769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = train_no_outliers.drop(columns=['target'])\ny = train_no_outliers['target']","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:42:17.218830Z","iopub.execute_input":"2024-08-24T09:42:17.219300Z","iopub.status.idle":"2024-08-24T09:42:17.227893Z","shell.execute_reply.started":"2024-08-24T09:42:17.219257Z","shell.execute_reply":"2024-08-24T09:42:17.226157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Initialize MinMaxScaler with the desired range (-1 to 1)\nscaler = MinMaxScaler(feature_range=(-1, 1))\n\n# Fit and transform the data\nX_final = pd.DataFrame(scaler.fit_transform(X), columns=X.columns)\ntest_final = pd.DataFrame(scaler.transform(test), columns = test.columns)","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:42:18.533879Z","iopub.execute_input":"2024-08-24T09:42:18.534322Z","iopub.status.idle":"2024-08-24T09:42:18.618763Z","shell.execute_reply.started":"2024-08-24T09:42:18.534279Z","shell.execute_reply":"2024-08-24T09:42:18.617397Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note: Any data transormation technique you apply to the train data remember to apply it to the test data aswell","metadata":{}},{"cell_type":"markdown","source":"# Feature Selection and K-Fold Cross validation\n\nRFE (Recursive Feature Elimination) : Recursive Feature Elimination (RFE) is a feature selection method that helps you identify the most important features in a dataset by recursively training a model and removing the least important features. Here we have used Linear regression as that model which is known as the estimator and the final no. of features we have now is 150. We will train our model now with these 150 features.\n\nK-Fold CV: What happens here is that we divide our data in certain parts, say 15. Now We validate our model for each fold while training on other folds. I have used a basic Linear regression model here with no hyperparameter tuning or regularization techniques.\nThe final prediction is the average of all the predictions from different folds.","metadata":{}},{"cell_type":"code","source":"# Number of folds for cross-validation\nn_folds = 15\n# List to store the weights (based on RMSE) for each fold\nweights = []\n\n# Initialize KFold cross-validation\nkf = KFold(n_splits=n_folds, shuffle=True, random_state=42)\n\n# Assign your features (train data) and target (labels)\ntrain_data = X_final\ntrain_target = y\n\n# Initialize the base model (Linear Regression) for RFE\nestimator = LinearRegression()\n\n# Initialize RFE (Recursive Feature Elimination) to select the top 150 features\nselector = RFE(estimator, n_features_to_select=150, step=1)\n\n# Fit the RFE model on the training data to select important features\nselector.fit(train_data, train_target)\n\n# Create a new training dataset with only the selected features\nrfe_train_data = train_data.iloc[:, selector.support_]\n\n# Apply the same feature selection to the test data\ntest_data = test_final.iloc[:, selector.support_]\n\n# Initialize an array to store the predictions for the test set\ntest_preds = np.zeros((test_data.shape[0],))\n\n# Loop over each fold in the KFold cross-validation\nfor fold, (train_idx, val_idx) in enumerate(kf.split(train_data)):\n    # Split the data into training and validation sets for the current fold\n    X_train, y_train = rfe_train_data.iloc[train_idx], train_target[train_idx]\n    X_val, y_val = rfe_train_data.iloc[val_idx], train_target[val_idx]\n\n    # Initialize the model (Linear Regression) for the current fold\n    model = LinearRegression()\n\n    # Train the model on the training data of the current fold\n    model.fit(X_train, y_train)\n\n    # Predict the target for the validation set\n    val_preds = model.predict(X_val)\n\n    # Calculate the RMSE for the current fold\n    fold_rmse = mean_squared_error(y_val, val_preds, squared=False)\n\n    # Calculate the weight for the current fold (inverse of RMSE)\n    weights.append(1 / fold_rmse)\n\n    # Print the RMSE for the current fold\n    print(f'Fold {fold+1} RMSE: {fold_rmse}')\n\n    # Predict the target for the test data and update test_preds\n    # Multiply the predictions by the weight for the current fold\n    test_preds += model.predict(test_data) * weights[fold]\n\n# Normalize the test predictions by dividing by the sum of the weights\ntest_preds /= sum(weights)\n\n# Print the final test predictions\nprint(test_preds)","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:42:26.943065Z","iopub.execute_input":"2024-08-24T09:42:26.943542Z","iopub.status.idle":"2024-08-24T09:42:35.013559Z","shell.execute_reply.started":"2024-08-24T09:42:26.943494Z","shell.execute_reply":"2024-08-24T09:42:35.012084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generating Submission File and Conclusion","metadata":{}},{"cell_type":"code","source":"submission = pd.read_csv(\"/kaggle/input/first-steps-in-data-science-new/sample.csv\")\nsubmission['target'] = test_preds\nsubmission.to_csv(\"/kaggle/working/submission.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2024-08-24T09:46:40.222004Z","iopub.execute_input":"2024-08-24T09:46:40.222529Z","iopub.status.idle":"2024-08-24T09:46:40.324278Z","shell.execute_reply.started":"2024-08-24T09:46:40.222484Z","shell.execute_reply":"2024-08-24T09:46:40.323148Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You all saw how we implemented some data preprocessing technques and feature engineering methods in this notebook, but this is just the tip of iceberg, dive deep into the sea of data science and discover yourself some better techniques and methods that might or might not improve your score on leaderboard but will definitely improve your understanding of Machine Learning and Data Science.\n\nBest of Luck to you all!!","metadata":{}}]}