{
  "id": 22063,
  "title": "How to apply one-hot encoding to big files?",
  "url": "/competitions/grupo-bimbo-inventory-demand/discussion/22063",
  "author_name": "",
  "post_date": "2016-07-06T08:29:05.787Z",
  "votes": 1,
  "comment_count": 1,
  "views": 747,
  "content": "<p>I have 3 Gb train file and 350 Mb test file, 6 Gb RAM. That files seem to be not too big to hold them in the memory, but  I can't even append them (memory error occurs).</p>\n\n<p>My current aproach looks this way:</p>\n\n<pre><code> # reading files\n   dtypes = {'Semana' : 'int32',\n                          'Agencia_ID' :'int32',\n                          'Canal_ID' : 'int32',\n                          'Ruta_SAK' : 'int32',\n                          'Cliente-ID' : 'int32',\n                          'Producto_ID':'int32',\n                          'Venta_hoy':'float32',\n                          'Venta_uni_hoy': 'int32',\n                          'Dev_uni_proxima':'int32',\n                          'Dev_proxima':'float32',\n                          'Demanda_uni_equil':'int32'}\n\ntrain = pd.read_csv('train.csv', dtype  = dtypes, usecols=[&quot;Semana&quot;, &quot;Agencia_ID&quot;, &quot;Canal_ID&quot;, 'Ruta_SAK',  'Cliente_ID', 'Producto_ID','Demanda_uni_equil'])\ntrain = train.loc[train['Demanda_uni_equil'] &lt; 53,:]\n\ntest = pd.read_csv('test.csv', dtype  = dtypes) #, nrows=numsToRead)\nids = test['id']\ntest.drop(['id'], axis =1, inplace = True)\n\nshapeTrain = train.shape[0]\nshapeTest = test.shape[0]\n\ntrain = train.append(test) # raises memory error\n\n    #applying one-hot encoding\n    train = pd.concat([train, pd.get_dummies(train['Semana'],sparse=True)], axis=1, join_axes=[train.index])\n    train.drop([11,'Semana'],axis=1, inplace = True)\n\n    train = pd.concat([train, pd.get_dummies(train['Producto_ID'],sparse=True)], axis=1, join_axes=[train.index])\n    train.drop([123,'Producto_ID'],axis=1, inplace = True)\n\n\n    train = pd.concat([train, pd.get_dummies(train['Canal_ID'],sparse=True)], axis=1, join_axes=[train.index])\n    train.drop([11,'Canal_ID'],axis=1, inplace = True)\n\n    # separating back to train and test\n    test = train[shapeTrain:shapeTrain+shapeTest]\n    train = train[0:shapeTrain]\n\n    test['id'] = ids\n\n    train.to_csv('train_1.csv', index=False)\n    test.to_csv('test_1.csv', index=False)\n</code></pre>\n\n<p>I see 2 ways to solve:</p>\n\n<p>1) Somehow iterate through the files:\n    df_train = pd.read_csv('train.csv', chunksize=1500)\n    for chunk in df_train:\n        #apply one-hot encoding</p>\n\n<p>2) To decrease amount of columns by applying one-hot encoding only to train, and then to test, but for new categorical value setting all columns to 0.</p>\n\n<p>How can I solve the problem?</p>",
  "messages": [
    {
      "id": "126104",
      "postDate": "07/06/2016 08:29:05",
      "content": "<p>I have 3 Gb train file and 350 Mb test file, 6 Gb RAM. That files seem to be not too big to hold them in the memory, but  I can't even append them (memory error occurs).</p>\n\n<p>My current aproach looks this way:</p>\n\n<pre><code> # reading files\n   dtypes = {'Semana' : 'int32',\n                          'Agencia_ID' :'int32',\n                          'Canal_ID' : 'int32',\n                          'Ruta_SAK' : 'int32',\n                          'Cliente-ID' : 'int32',\n                          'Producto_ID':'int32',\n                          'Venta_hoy':'float32',\n                          'Venta_uni_hoy': 'int32',\n                          'Dev_uni_proxima':'int32',\n                          'Dev_proxima':'float32',\n                          'Demanda_uni_equil':'int32'}\n\ntrain = pd.read_csv('train.csv', dtype  = dtypes, usecols=[&quot;Semana&quot;, &quot;Agencia_ID&quot;, &quot;Canal_ID&quot;, 'Ruta_SAK',  'Cliente_ID', 'Producto_ID','Demanda_uni_equil'])\ntrain = train.loc[train['Demanda_uni_equil'] &lt; 53,:]\n\ntest = pd.read_csv('test.csv', dtype  = dtypes) #, nrows=numsToRead)\nids = test['id']\ntest.drop(['id'], axis =1, inplace = True)\n\nshapeTrain = train.shape[0]\nshapeTest = test.shape[0]\n\ntrain = train.append(test) # raises memory error\n\n    #applying one-hot encoding\n    train = pd.concat([train, pd.get_dummies(train['Semana'],sparse=True)], axis=1, join_axes=[train.index])\n    train.drop([11,'Semana'],axis=1, inplace = True)\n\n    train = pd.concat([train, pd.get_dummies(train['Producto_ID'],sparse=True)], axis=1, join_axes=[train.index])\n    train.drop([123,'Producto_ID'],axis=1, inplace = True)\n\n\n    train = pd.concat([train, pd.get_dummies(train['Canal_ID'],sparse=True)], axis=1, join_axes=[train.index])\n    train.drop([11,'Canal_ID'],axis=1, inplace = True)\n\n    # separating back to train and test\n    test = train[shapeTrain:shapeTrain+shapeTest]\n    train = train[0:shapeTrain]\n\n    test['id'] = ids\n\n    train.to_csv('train_1.csv', index=False)\n    test.to_csv('test_1.csv', index=False)\n</code></pre>\n\n<p>I see 2 ways to solve:</p>\n\n<p>1) Somehow iterate through the files:\n    df_train = pd.read_csv('train.csv', chunksize=1500)\n    for chunk in df_train:\n        #apply one-hot encoding</p>\n\n<p>2) To decrease amount of columns by applying one-hot encoding only to train, and then to test, but for new categorical value setting all columns to 0.</p>\n\n<p>How can I solve the problem?</p>",
      "rawMarkdown": "I have 3 Gb train file and 350 Mb test file, 6 Gb RAM. That files seem to be not too big to hold them in the memory, but  I can't even append them (memory error occurs).\r\n\r\nMy current aproach looks this way:\r\n\r\n   \r\n\r\n     # reading files\r\n       dtypes = {'Semana' : 'int32',\r\n                              'Agencia_ID' :'int32',\r\n                              'Canal_ID' : 'int32',\r\n                              'Ruta_SAK' : 'int32',\r\n                              'Cliente-ID' : 'int32',\r\n                              'Producto_ID':'int32',\r\n                              'Venta_hoy':'float32',\r\n                              'Venta_uni_hoy': 'int32',\r\n                              'Dev_uni_proxima':'int32',\r\n                              'Dev_proxima':'float32',\r\n                              'Demanda_uni_equil':'int32'}\r\n\r\n    train = pd.read_csv('train.csv', dtype  = dtypes, usecols=[\"Semana\", \"Agencia_ID\", \"Canal_ID\", 'Ruta_SAK',  'Cliente_ID', 'Producto_ID','Demanda_uni_equil'])\r\n    train = train.loc[train['Demanda_uni_equil'] < 53,:]\r\n\r\n    test = pd.read_csv('test.csv', dtype  = dtypes) #, nrows=numsToRead)\r\n    ids = test['id']\r\n    test.drop(['id'], axis =1, inplace = True)\r\n\r\n    shapeTrain = train.shape[0]\r\n    shapeTest = test.shape[0]\r\n    \r\n    train = train.append(test) # raises memory error\r\n    \r\n        #applying one-hot encoding\r\n        train = pd.concat([train, pd.get_dummies(train['Semana'],sparse=True)], axis=1, join_axes=[train.index])\r\n        train.drop([11,'Semana'],axis=1, inplace = True)\r\n        \r\n        train = pd.concat([train, pd.get_dummies(train['Producto_ID'],sparse=True)], axis=1, join_axes=[train.index])\r\n        train.drop([123,'Producto_ID'],axis=1, inplace = True)\r\n        \r\n        \r\n        train = pd.concat([train, pd.get_dummies(train['Canal_ID'],sparse=True)], axis=1, join_axes=[train.index])\r\n        train.drop([11,'Canal_ID'],axis=1, inplace = True)\r\n        \r\n        # separating back to train and test\r\n        test = train[shapeTrain:shapeTrain+shapeTest]\r\n        train = train[0:shapeTrain]\r\n        \r\n        test['id'] = ids\r\n        \r\n        train.to_csv('train_1.csv', index=False)\r\n        test.to_csv('test_1.csv', index=False)\r\n\r\n\r\n\r\nI see 2 ways to solve:\r\n\r\n1) Somehow iterate through the files:\r\n    df_train = pd.read_csv('train.csv', chunksize=1500)\r\n    for chunk in df_train:\r\n        #apply one-hot encoding\r\n\r\n2) To decrease amount of columns by applying one-hot encoding only to train, and then to test, but for new categorical value setting all columns to 0.\r\n\r\nHow can I solve the problem?",
      "votes": null
    },
    {
      "id": "126118",
      "postDate": "07/06/2016 13:09:48",
      "content": "<p>Use the hashing trick!  <a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html\">http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html</a></p>",
      "rawMarkdown": "Use the hashing trick!  http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 126118,
      "author_name": "zachmayer",
      "author_url": "",
      "post_date": "07/06/2016 13:09:48",
      "content": "<p>Use the hashing trick!  <a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html\">http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "126104": "I have 3 Gb train file and 350 Mb test file, 6 Gb RAM. That files seem to be not too big to hold them in the memory, but  I can't even append them (memory error occurs).\r\n\r\nMy current aproach looks this way:\r\n\r\n   \r\n\r\n     # reading files\r\n       dtypes = {'Semana' : 'int32',\r\n                              'Agencia_ID' :'int32',\r\n                              'Canal_ID' : 'int32',\r\n                              'Ruta_SAK' : 'int32',\r\n                              'Cliente-ID' : 'int32',\r\n                              'Producto_ID':'int32',\r\n                              'Venta_hoy':'float32',\r\n                              'Venta_uni_hoy': 'int32',\r\n                              'Dev_uni_proxima':'int32',\r\n                              'Dev_proxima':'float32',\r\n                              'Demanda_uni_equil':'int32'}\r\n\r\n    train = pd.read_csv('train.csv', dtype  = dtypes, usecols=[\"Semana\", \"Agencia_ID\", \"Canal_ID\", 'Ruta_SAK',  'Cliente_ID', 'Producto_ID','Demanda_uni_equil'])\r\n    train = train.loc[train['Demanda_uni_equil'] < 53,:]\r\n\r\n    test = pd.read_csv('test.csv', dtype  = dtypes) #, nrows=numsToRead)\r\n    ids = test['id']\r\n    test.drop(['id'], axis =1, inplace = True)\r\n\r\n    shapeTrain = train.shape[0]\r\n    shapeTest = test.shape[0]\r\n    \r\n    train = train.append(test) # raises memory error\r\n    \r\n        #applying one-hot encoding\r\n        train = pd.concat([train, pd.get_dummies(train['Semana'],sparse=True)], axis=1, join_axes=[train.index])\r\n        train.drop([11,'Semana'],axis=1, inplace = True)\r\n        \r\n        train = pd.concat([train, pd.get_dummies(train['Producto_ID'],sparse=True)], axis=1, join_axes=[train.index])\r\n        train.drop([123,'Producto_ID'],axis=1, inplace = True)\r\n        \r\n        \r\n        train = pd.concat([train, pd.get_dummies(train['Canal_ID'],sparse=True)], axis=1, join_axes=[train.index])\r\n        train.drop([11,'Canal_ID'],axis=1, inplace = True)\r\n        \r\n        # separating back to train and test\r\n        test = train[shapeTrain:shapeTrain+shapeTest]\r\n        train = train[0:shapeTrain]\r\n        \r\n        test['id'] = ids\r\n        \r\n        train.to_csv('train_1.csv', index=False)\r\n        test.to_csv('test_1.csv', index=False)\r\n\r\n\r\n\r\nI see 2 ways to solve:\r\n\r\n1) Somehow iterate through the files:\r\n    df_train = pd.read_csv('train.csv', chunksize=1500)\r\n    for chunk in df_train:\r\n        #apply one-hot encoding\r\n\r\n2) To decrease amount of columns by applying one-hot encoding only to train, and then to test, but for new categorical value setting all columns to 0.\r\n\r\nHow can I solve the problem?",
    "126118": "Use the hashing trick!  http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html"
  },
  "source": "meta"
}