{
  "id": 275164,
  "title": "1st place approach",
  "url": "/competitions/beyond-analysis/writeups/priyanshu-chaudhary-1st-place-approach",
  "author_name": "",
  "post_date": "2021-09-29T06:15:50.127921200Z",
  "votes": 2,
  "comment_count": 1,
  "views": 0,
  "content": "<p><strong>(THE FEATURE ENGINEERING GAME)</strong><br>\n<strong>KEY POINTS:</strong><br>\n<strong>THINGS WORKED:</strong><br>\n   <strong>FEATURE ENGINEERING</strong><br>\n     -   Grouping the dataframe on the basis of unique identifier decreases the standard deviation by 40 and this gave me the idea to train the model on grouped data since number of rows after grouping were still significantly large.<br>\n     -  Grouping would have definitely reduced the information so we needed to build some new features that contains the flucation in the values averages and minimum and maximum as well as standard deviation to help or model perform better.<br>\n       <strong>FEATURE ENGINERRING WAS DIVIDED INTO TWO CATEGORIES</strong><br>\n           <strong>BEFORE GROUPING</strong><br>\n      -          Feature engineering before grouping gamewas the magic worked for me.<br>\n    -            Before grouping features captured the fluctuations in the prices numbers so that while goruing this information provides good bost and info to model.<br>\n    -           Fractions and differences are some of the examples of the features before grouping.<br>\n           <strong>DURING GROUPING</strong></p>\n<ul>\n<li>Information of each of the columns made before grouping is taken with the help of using aggregated features. For Ex. mean,sum,min,max,std,percentile,median<ul>\n<li>Grouping gave me the boost of over 8 since standard deviation reduced significantly after grouping by UI.<br>\n<strong>PREPROCESSING</strong><br>\nCategorical features were label encoded<br>\nData is scaled with the help of quantile transformer that gave quite good boost in CV.<br>\nfor outlier handling also i used quantile trasformer as it reduces the effect of marignal outliers.<br>\n<strong>CV STRATEGY AND MODELLING</strong><br>\nCross Validation Strategy<br>\n   simple kfold strtegy with 10 folds.<br>\n   loss function rmse<br>\nModelling<br>\n   4 tree based models were used 1) Lightgbm. 2) catboost. 3) xgboost. 4) hyper gradient boost regressor.<br>\n   Denoising Auto Encoder + lgbm<br>\n<strong>KEY OBSERVATIONS</strong><br>\nIf fluctuations in the model wouldn;t have been recorded there will be huge date loss while grouping thats why fluctuations were recorded infact feature engineering columns came out to be the match winning strategy for me.<br>\nLightgbm was the best single model<br>\n5 different models were used for avergaing and reducing the rmse.<br>\nDenoising was not providing significant result on cv but in the end it came to be the match winner on private leaderboard.I used it for better diversity in models.<br>\n<strong>UNIQUE IDEAS</strong><br>\nKmeans clustering to form stratified kfolds (I was able to see equal results but in the end I was not able to add it into the main model due to time constraint).<br>\nOutlier removal- it could have boosted my score a little bit but there was equal equal chance of underfitting depend on the private data so I didn't take the risk since there were not that much outliers in the provided data.<br>\nDAE (denoising auto encoder) to denoise the data and then using lgbm on it improved my cv by .1(approx).<br>\nCategorical variables were not providing boost so I didn't focused on them still I feel there could have been a better improvement in the CV of Y1 matrix.<br>\n<strong>DENOISING AUTO ENCODER</strong><br>\nI made a neural nets of 3 hidden layers with swish activation and used linear activation as output for denoising the data this strategy proved really good to provide diversity in the modelling.<br>\nSwish activation is used over ReLU as swish activation had non zero derivative for values below zero.</li></ul></li>\n</ul>",
  "messages": [
    {
      "id": "1527826",
      "postDate": "09/29/2021 06:15:50",
      "content": "<p><strong>(THE FEATURE ENGINEERING GAME)</strong><br>\n<strong>KEY POINTS:</strong><br>\n<strong>THINGS WORKED:</strong><br>\n   <strong>FEATURE ENGINEERING</strong><br>\n     -   Grouping the dataframe on the basis of unique identifier decreases the standard deviation by 40 and this gave me the idea to train the model on grouped data since number of rows after grouping were still significantly large.<br>\n     -  Grouping would have definitely reduced the information so we needed to build some new features that contains the flucation in the values averages and minimum and maximum as well as standard deviation to help or model perform better.<br>\n       <strong>FEATURE ENGINERRING WAS DIVIDED INTO TWO CATEGORIES</strong><br>\n           <strong>BEFORE GROUPING</strong><br>\n      -          Feature engineering before grouping gamewas the magic worked for me.<br>\n    -            Before grouping features captured the fluctuations in the prices numbers so that while goruing this information provides good bost and info to model.<br>\n    -           Fractions and differences are some of the examples of the features before grouping.<br>\n           <strong>DURING GROUPING</strong></p>\n<ul>\n<li>Information of each of the columns made before grouping is taken with the help of using aggregated features. For Ex. mean,sum,min,max,std,percentile,median<ul>\n<li>Grouping gave me the boost of over 8 since standard deviation reduced significantly after grouping by UI.<br>\n<strong>PREPROCESSING</strong><br>\nCategorical features were label encoded<br>\nData is scaled with the help of quantile transformer that gave quite good boost in CV.<br>\nfor outlier handling also i used quantile trasformer as it reduces the effect of marignal outliers.<br>\n<strong>CV STRATEGY AND MODELLING</strong><br>\nCross Validation Strategy<br>\n   simple kfold strtegy with 10 folds.<br>\n   loss function rmse<br>\nModelling<br>\n   4 tree based models were used 1) Lightgbm. 2) catboost. 3) xgboost. 4) hyper gradient boost regressor.<br>\n   Denoising Auto Encoder + lgbm<br>\n<strong>KEY OBSERVATIONS</strong><br>\nIf fluctuations in the model wouldn;t have been recorded there will be huge date loss while grouping thats why fluctuations were recorded infact feature engineering columns came out to be the match winning strategy for me.<br>\nLightgbm was the best single model<br>\n5 different models were used for avergaing and reducing the rmse.<br>\nDenoising was not providing significant result on cv but in the end it came to be the match winner on private leaderboard.I used it for better diversity in models.<br>\n<strong>UNIQUE IDEAS</strong><br>\nKmeans clustering to form stratified kfolds (I was able to see equal results but in the end I was not able to add it into the main model due to time constraint).<br>\nOutlier removal- it could have boosted my score a little bit but there was equal equal chance of underfitting depend on the private data so I didn't take the risk since there were not that much outliers in the provided data.<br>\nDAE (denoising auto encoder) to denoise the data and then using lgbm on it improved my cv by .1(approx).<br>\nCategorical variables were not providing boost so I didn't focused on them still I feel there could have been a better improvement in the CV of Y1 matrix.<br>\n<strong>DENOISING AUTO ENCODER</strong><br>\nI made a neural nets of 3 hidden layers with swish activation and used linear activation as output for denoising the data this strategy proved really good to provide diversity in the modelling.<br>\nSwish activation is used over ReLU as swish activation had non zero derivative for values below zero.</li></ul></li>\n</ul>",
      "rawMarkdown": "**(THE FEATURE ENGINEERING GAME)**\n**KEY POINTS:**\n**THINGS WORKED:**\n\n    **FEATURE ENGINEERING**\n      -   Grouping the dataframe on the basis of unique identifier decreases the standard deviation by 40 and this gave me the idea to train the model on grouped data since number of rows after grouping were still significantly large.\n\n      -  Grouping would have definitely reduced the information so we needed to build some new features that contains the flucation in the values averages and minimum and maximum as well as standard deviation to help or model perform better.\n\n        **FEATURE ENGINERRING WAS DIVIDED INTO TWO CATEGORIES**\n            **BEFORE GROUPING**\n       -          Feature engineering before grouping gamewas the magic worked for me.\n     -            Before grouping features captured the fluctuations in the prices numbers so that while goruing this information provides good bost and info to model.\n     -           Fractions and differences are some of the examples of the features before grouping.\n            **DURING GROUPING**\n   -              Information of each of the columns made before grouping is taken with the help of using aggregated features. For Ex. mean,sum,min,max,std,percentile,median\n      -           Grouping gave me the boost of over 8 since standard deviation reduced significantly after grouping by UI.\n\n    **PREPROCESSING**\n        Categorical features were label encoded\n        Data is scaled with the help of quantile transformer that gave quite good boost in CV.\n        for outlier handling also i used quantile trasformer as it reduces the effect of marignal outliers.\n\n    **CV STRATEGY AND MODELLING**\n        Cross Validation Strategy\n            simple kfold strtegy with 10 folds.\n            loss function rmse\n        Modelling\n            4 tree based models were used 1) Lightgbm. 2) catboost. 3) xgboost. 4) hyper gradient boost regressor.\n            Denoising Auto Encoder + lgbm\n\n    **KEY OBSERVATIONS**\n        If fluctuations in the model wouldn;t have been recorded there will be huge date loss while grouping thats why fluctuations were recorded infact feature engineering columns came out to be the match winning strategy for me.\n        Lightgbm was the best single model\n        5 different models were used for avergaing and reducing the rmse.\n        Denoising was not providing significant result on cv but in the end it came to be the match winner on private leaderboard.I used it for better diversity in models.\n\n    **UNIQUE IDEAS**\n        Kmeans clustering to form stratified kfolds (I was able to see equal results but in the end I was not able to add it into the main model due to time constraint).\n        Outlier removal- it could have boosted my score a little bit but there was equal equal chance of underfitting depend on the private data so I didn't take the risk since there were not that much outliers in the provided data.\n        DAE (denoising auto encoder) to denoise the data and then using lgbm on it improved my cv by .1(approx).\n        Categorical variables were not providing boost so I didn't focused on them still I feel there could have been a better improvement in the CV of Y1 matrix.\n\n    **DENOISING AUTO ENCODER**\n        I made a neural nets of 3 hidden layers with swish activation and used linear activation as output for denoising the data this strategy proved really good to provide diversity in the modelling.\n        Swish activation is used over ReLU as swish activation had non zero derivative for values below zero.",
      "votes": null
    },
    {
      "id": "1527834",
      "postDate": "09/29/2021 06:25:17",
      "content": "<p>LINK to the code:<br>\n<a href=\"https://www.kaggle.com/chaudharypriyanshu/1st-place-solution/notebook\" target=\"_blank\">https://www.kaggle.com/chaudharypriyanshu/1st-place-solution/notebook</a></p>",
      "rawMarkdown": "LINK to the code:\nhttps://www.kaggle.com/chaudharypriyanshu/1st-place-solution/notebook",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1527834,
      "author_name": "chaudharypriyanshu",
      "author_url": "",
      "post_date": "09/29/2021 06:25:17",
      "content": "<p>LINK to the code:<br>\n<a href=\"https://www.kaggle.com/chaudharypriyanshu/1st-place-solution/notebook\" target=\"_blank\">https://www.kaggle.com/chaudharypriyanshu/1st-place-solution/notebook</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1527826": "**(THE FEATURE ENGINEERING GAME)**\n**KEY POINTS:**\n**THINGS WORKED:**\n\n    **FEATURE ENGINEERING**\n      -   Grouping the dataframe on the basis of unique identifier decreases the standard deviation by 40 and this gave me the idea to train the model on grouped data since number of rows after grouping were still significantly large.\n\n      -  Grouping would have definitely reduced the information so we needed to build some new features that contains the flucation in the values averages and minimum and maximum as well as standard deviation to help or model perform better.\n\n        **FEATURE ENGINERRING WAS DIVIDED INTO TWO CATEGORIES**\n            **BEFORE GROUPING**\n       -          Feature engineering before grouping gamewas the magic worked for me.\n     -            Before grouping features captured the fluctuations in the prices numbers so that while goruing this information provides good bost and info to model.\n     -           Fractions and differences are some of the examples of the features before grouping.\n            **DURING GROUPING**\n   -              Information of each of the columns made before grouping is taken with the help of using aggregated features. For Ex. mean,sum,min,max,std,percentile,median\n      -           Grouping gave me the boost of over 8 since standard deviation reduced significantly after grouping by UI.\n\n    **PREPROCESSING**\n        Categorical features were label encoded\n        Data is scaled with the help of quantile transformer that gave quite good boost in CV.\n        for outlier handling also i used quantile trasformer as it reduces the effect of marignal outliers.\n\n    **CV STRATEGY AND MODELLING**\n        Cross Validation Strategy\n            simple kfold strtegy with 10 folds.\n            loss function rmse\n        Modelling\n            4 tree based models were used 1) Lightgbm. 2) catboost. 3) xgboost. 4) hyper gradient boost regressor.\n            Denoising Auto Encoder + lgbm\n\n    **KEY OBSERVATIONS**\n        If fluctuations in the model wouldn;t have been recorded there will be huge date loss while grouping thats why fluctuations were recorded infact feature engineering columns came out to be the match winning strategy for me.\n        Lightgbm was the best single model\n        5 different models were used for avergaing and reducing the rmse.\n        Denoising was not providing significant result on cv but in the end it came to be the match winner on private leaderboard.I used it for better diversity in models.\n\n    **UNIQUE IDEAS**\n        Kmeans clustering to form stratified kfolds (I was able to see equal results but in the end I was not able to add it into the main model due to time constraint).\n        Outlier removal- it could have boosted my score a little bit but there was equal equal chance of underfitting depend on the private data so I didn't take the risk since there were not that much outliers in the provided data.\n        DAE (denoising auto encoder) to denoise the data and then using lgbm on it improved my cv by .1(approx).\n        Categorical variables were not providing boost so I didn't focused on them still I feel there could have been a better improvement in the CV of Y1 matrix.\n\n    **DENOISING AUTO ENCODER**\n        I made a neural nets of 3 hidden layers with swish activation and used linear activation as output for denoising the data this strategy proved really good to provide diversity in the modelling.\n        Swish activation is used over ReLU as swish activation had non zero derivative for values below zero.",
    "1527834": "LINK to the code:\nhttps://www.kaggle.com/chaudharypriyanshu/1st-place-solution/notebook"
  },
  "source": "existing_discussion",
  "complete": true,
  "competition_id": 30591,
  "competition_slug": "beyond-analysis",
  "team_id": 7349791,
  "solution_url": "/competitions/beyond-analysis/writeups/priyanshu-chaudhary-1st-place-approach"
}