{
  "id": 136020,
  "title": "methods of handling imbalance classification",
  "url": "/competitions/cat-in-the-dat-ii/discussion/136020",
  "author_name": "",
  "post_date": "2020-03-17T04:30:22.527501Z",
  "votes": 1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hi,Since Target class is highly imbalanced ,has anyone used strategies like sampling techniques or something like SMOTE ?\nDoes Boosting algorithms handle imbalance classes automatically?</p>",
  "messages": [
    {
      "id": "776007",
      "postDate": "03/17/2020 04:30:22",
      "content": "<p>Hi,Since Target class is highly imbalanced ,has anyone used strategies like sampling techniques or something like SMOTE ?\nDoes Boosting algorithms handle imbalance classes automatically?</p>",
      "rawMarkdown": "Hi,Since Target class is highly imbalanced ,has anyone used strategies like sampling techniques or something like SMOTE ?\nDoes Boosting algorithms handle imbalance classes automatically?",
      "votes": null
    },
    {
      "id": "782757",
      "postDate": "03/22/2020 16:09:06",
      "content": "<p>since this dataset has a lot of rows, under-sampling, i think, can provide acceptable quality. i'd use <a href=\"https://imbalanced-learn.readthedocs.io/en/stable/index.html\">imblearn's</a> RandomUnderSampler for this</p>",
      "rawMarkdown": "since this dataset has a lot of rows, under-sampling, i think, can provide acceptable quality. i'd use [imblearn's](https://imbalanced-learn.readthedocs.io/en/stable/index.html) RandomUnderSampler for this",
      "votes": null
    },
    {
      "id": "787172",
      "postDate": "03/26/2020 15:21:45",
      "content": "<p>You can use Under-Sampling or Over-Sampling  techniques based on your dataset.\nOver-Sampling introduces the problem of overfitting. So Under-Sampling is preferred.\nI have used SMOOTE but it gave me the problem of Overfitting.</p>\n\n<p>''' Code for SMOOTE '''\nfrom imblearn.over_sampling import SMOTE\nsmt= SMOTE()\nTrain,Trainlabel=smt.fit_sample(Train,Trainlabel)</p>\n\n<p>I also used another method. In which I copied the data where the target was 0 or 1 (depends upon your dataset) into a dataframe and then merged this dataframe with the original one.\nThis method gave me far better results.\n''' Code '''\nmeta_train= train[train.target == 1]\ntrain = pd.concat([train,meta_train],axis = 0)</p>",
      "rawMarkdown": "You can use Under-Sampling or Over-Sampling  techniques based on your dataset.\nOver-Sampling introduces the problem of overfitting. So Under-Sampling is preferred.\nI have used SMOOTE but it gave me the problem of Overfitting.\n\n''' Code for SMOOTE '''\nfrom imblearn.over_sampling import SMOTE\nsmt= SMOTE()\nTrain,Trainlabel=smt.fit_sample(Train,Trainlabel)\n\nI also used another method. In which I copied the data where the target was 0 or 1 (depends upon your dataset) into a dataframe and then merged this dataframe with the original one.\nThis method gave me far better results.\n''' Code '''\nmeta_train= train[train.target == 1]\ntrain = pd.concat([train,meta_train],axis = 0)",
      "votes": null
    },
    {
      "id": "789668",
      "postDate": "03/28/2020 21:38:13",
      "content": "<p>I wrote a ballancing function from scratch, you can find it in <a href=\"https://www.kaggle.com/catadanna/categorical-encoding-my-solution\">my notebook</a>.</p>",
      "rawMarkdown": "I wrote a ballancing function from scratch, you can find it in [my notebook](https://www.kaggle.com/catadanna/categorical-encoding-my-solution).",
      "votes": null
    },
    {
      "id": "790616",
      "postDate": "03/29/2020 18:17:08",
      "content": "<p>I Have used SMOTEENN which performs first oversampling then undersampling. It reduced my overfitting.</p>",
      "rawMarkdown": "I Have used SMOTEENN which performs first oversampling then undersampling. It reduced my overfitting.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 782757,
      "author_name": "fkdplc",
      "author_url": "",
      "post_date": "03/22/2020 16:09:06",
      "content": "<p>since this dataset has a lot of rows, under-sampling, i think, can provide acceptable quality. i'd use <a href=\"https://imbalanced-learn.readthedocs.io/en/stable/index.html\">imblearn's</a> RandomUnderSampler for this</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 787172,
      "author_name": "exxyler",
      "author_url": "",
      "post_date": "03/26/2020 15:21:45",
      "content": "<p>You can use Under-Sampling or Over-Sampling  techniques based on your dataset.\nOver-Sampling introduces the problem of overfitting. So Under-Sampling is preferred.\nI have used SMOOTE but it gave me the problem of Overfitting.</p>\n\n<p>''' Code for SMOOTE '''\nfrom imblearn.over_sampling import SMOTE\nsmt= SMOTE()\nTrain,Trainlabel=smt.fit_sample(Train,Trainlabel)</p>\n\n<p>I also used another method. In which I copied the data where the target was 0 or 1 (depends upon your dataset) into a dataframe and then merged this dataframe with the original one.\nThis method gave me far better results.\n''' Code '''\nmeta_train= train[train.target == 1]\ntrain = pd.concat([train,meta_train],axis = 0)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 789668,
      "author_name": "catadanna",
      "author_url": "",
      "post_date": "03/28/2020 21:38:13",
      "content": "<p>I wrote a ballancing function from scratch, you can find it in <a href=\"https://www.kaggle.com/catadanna/categorical-encoding-my-solution\">my notebook</a>.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 790616,
      "author_name": "keerthivikram",
      "author_url": "",
      "post_date": "03/29/2020 18:17:08",
      "content": "<p>I Have used SMOTEENN which performs first oversampling then undersampling. It reduced my overfitting.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "776007": "Hi,Since Target class is highly imbalanced ,has anyone used strategies like sampling techniques or something like SMOTE ?\nDoes Boosting algorithms handle imbalance classes automatically?",
    "782757": "since this dataset has a lot of rows, under-sampling, i think, can provide acceptable quality. i'd use [imblearn's](https://imbalanced-learn.readthedocs.io/en/stable/index.html) RandomUnderSampler for this",
    "787172": "You can use Under-Sampling or Over-Sampling  techniques based on your dataset.\nOver-Sampling introduces the problem of overfitting. So Under-Sampling is preferred.\nI have used SMOOTE but it gave me the problem of Overfitting.\n\n''' Code for SMOOTE '''\nfrom imblearn.over_sampling import SMOTE\nsmt= SMOTE()\nTrain,Trainlabel=smt.fit_sample(Train,Trainlabel)\n\nI also used another method. In which I copied the data where the target was 0 or 1 (depends upon your dataset) into a dataframe and then merged this dataframe with the original one.\nThis method gave me far better results.\n''' Code '''\nmeta_train= train[train.target == 1]\ntrain = pd.concat([train,meta_train],axis = 0)",
    "789668": "I wrote a ballancing function from scratch, you can find it in [my notebook](https://www.kaggle.com/catadanna/categorical-encoding-my-solution).",
    "790616": "I Have used SMOTEENN which performs first oversampling then undersampling. It reduced my overfitting."
  },
  "source": "meta"
}