{
  "id": 11374,
  "title": "Subsampling dataset with python",
  "url": "/competitions/avazu-ctr-prediction/discussion/11374",
  "author_name": "",
  "post_date": "2014-12-28T07:01:28.950Z",
  "votes": null,
  "comment_count": 3,
  "views": 2579,
  "content": "<p>I am new to python, and was wondering what the most efficient way to subsample the dataset is. &nbsp;Any help would be greatly appreciated. &nbsp;</p>",
  "messages": [
    {
      "id": "61974",
      "postDate": "12/28/2014 07:01:28",
      "content": "<p>I am new to python, and was wondering what the most efficient way to subsample the dataset is. &nbsp;Any help would be greatly appreciated. &nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62192",
      "postDate": "01/01/2015 22:27:09",
      "content": "<p>You can try tools provided by scikit-learn. Check out&nbsp;http://scikit-learn.org/stable/modules/cross_validation.html#k-fold</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62206",
      "postDate": "01/02/2015 09:50:28",
      "content": "<p>You can use this simple script I'&nbsp;ve written for that purpose.</p>\n<p>It runs pretty fast especially with&nbsp;pypy.&nbsp;</p>\n<p>In default it samples 1/40 of whole dataset but changing the holdout variable can sample more or less.</p>\n<p>You can even try leave some variables of by removing them from script or making them a comment with '#'.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "716046",
      "postDate": "01/11/2020 07:16:11",
      "content": "<p>Use train-test split with stratification, as seen here. </p>\n\n<p>from sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y,  test_size=0.25)</p>\n\n<p>Here you are dividing input data X, and targets y into training set(X_train,y_train) and test set(X_test and y_test) while preserving distribution in the input data.</p>",
      "rawMarkdown": "Use train-test split with stratification, as seen here. \n\nfrom sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y,  test_size=0.25)\n\nHere you are dividing input data X, and targets y into training set(X_train,y_train) and test set(X_test and y_test) while preserving distribution in the input data.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 62192,
      "author_name": "yankov",
      "author_url": "",
      "post_date": "01/01/2015 22:27:09",
      "content": "<p>You can try tools provided by scikit-learn. Check out&nbsp;http://scikit-learn.org/stable/modules/cross_validation.html#k-fold</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62206,
      "author_name": "ouranos",
      "author_url": "",
      "post_date": "01/02/2015 09:50:28",
      "content": "<p>You can use this simple script I'&nbsp;ve written for that purpose.</p>\n<p>It runs pretty fast especially with&nbsp;pypy.&nbsp;</p>\n<p>In default it samples 1/40 of whole dataset but changing the holdout variable can sample more or less.</p>\n<p>You can even try leave some variables of by removing them from script or making them a comment with '#'.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 716046,
      "author_name": "hassanamin",
      "author_url": "",
      "post_date": "01/11/2020 07:16:11",
      "content": "<p>Use train-test split with stratification, as seen here. </p>\n\n<p>from sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y,  test_size=0.25)</p>\n\n<p>Here you are dividing input data X, and targets y into training set(X_train,y_train) and test set(X_test and y_test) while preserving distribution in the input data.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "61974": "",
    "62192": "",
    "62206": "",
    "716046": "Use train-test split with stratification, as seen here. \n\nfrom sklearn.model_selection import train_test_split\nX_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y,  test_size=0.25)\n\nHere you are dividing input data X, and targets y into training set(X_train,y_train) and test set(X_test and y_test) while preserving distribution in the input data."
  },
  "source": "meta"
}