{
  "id": 23290,
  "title": "Turn categorical into numeric",
  "url": "/competitions/bosch-production-line-performance/discussion/23290",
  "author_name": "",
  "post_date": "2016-09-02T13:23:45.983Z",
  "votes": 17,
  "comment_count": 22,
  "views": 3565,
  "content": "<p>The train_categorical file contains strings and NaNs.  This is not very friendly to pandas.  Here is a simple preprocessing command you can run in Linux or Mac OSX (and maybe cygwin on Windows) to turn all strings into numerical values:</p>\n\n<pre><code>sed -i .sav 's/T//g' train_categorical.csv \nsed -i .sav 's/T//g' test_categorical.csv \n</code></pre>\n\n<p>It saves the original files with a sav extension, and produces new files where all data is numeric.  NaNs aren't modified.</p>\n\n<p>This saves lots of memory compared to reading the original file, especially if you specify a dtype when reading.</p>",
  "messages": [
    {
      "id": "133475",
      "postDate": "09/02/2016 13:23:45",
      "content": "<p>The train_categorical file contains strings and NaNs.  This is not very friendly to pandas.  Here is a simple preprocessing command you can run in Linux or Mac OSX (and maybe cygwin on Windows) to turn all strings into numerical values:</p>\n\n<pre><code>sed -i .sav 's/T//g' train_categorical.csv \nsed -i .sav 's/T//g' test_categorical.csv \n</code></pre>\n\n<p>It saves the original files with a sav extension, and produces new files where all data is numeric.  NaNs aren't modified.</p>\n\n<p>This saves lots of memory compared to reading the original file, especially if you specify a dtype when reading.</p>",
      "rawMarkdown": "The train_categorical file contains strings and NaNs.  This is not very friendly to pandas.  Here is a simple preprocessing command you can run in Linux or Mac OSX (and maybe cygwin on Windows) to turn all strings into numerical values:\r\n\r\n    sed -i .sav 's/T//g' train_categorical.csv \r\n    sed -i .sav 's/T//g' test_categorical.csv \r\n\r\nIt saves the original files with a sav extension, and produces new files where all data is numeric.  NaNs aren't modified.\r\n\r\nThis saves lots of memory compared to reading the original file, especially if you specify a dtype when reading.",
      "votes": null
    },
    {
      "id": "133575",
      "postDate": "09/02/2016 19:42:47",
      "content": "<p>If you want a non-negative representation, to be more specific, you also have to remove the '-' character that appears in some features. </p>",
      "rawMarkdown": "If you want a non-negative representation, to be more specific, you also have to remove the '-' character that appears in some features.",
      "votes": null
    },
    {
      "id": "133692",
      "postDate": "09/03/2016 11:13:46",
      "content": "<p>Is it useful to get rid of negativity?</p>\n\n<p>Note that the following will not remove the negativity, it will remove all content ;)</p>\n\n<pre><code>sed -i .sav 's/T-//g' train_categorical.csv \nsed -i .sav 's/T-//g' test_categorical.csv \n</code></pre>",
      "rawMarkdown": "Is it useful to get rid of negativity?\r\n\r\nNote that the following will not remove the negativity, it will remove all content ;)\r\n\r\n    sed -i .sav 's/T-//g' train_categorical.csv \r\n    sed -i .sav 's/T-//g' test_categorical.csv",
      "votes": null
    },
    {
      "id": "133810",
      "postDate": "09/03/2016 22:02:41",
      "content": "<p>I think it might be useful to save some memory. I have to check, but I am quite sure that there are some huge numbers that may not fit in an integer for example.</p>\n\n<p>By default, python numpy/scipy.sparse uses float64 (double) which consumes 8 bytes, on the other hand you can save half of memory using float32 (float). I am doing this in order to save memory for all datasets.</p>",
      "rawMarkdown": "I think it might be useful to save some memory. I have to check, but I am quite sure that there are some huge numbers that may not fit in an integer for example.\r\n\r\nBy default, python numpy/scipy.sparse uses float64 (double) which consumes 8 bytes, on the other hand you can save half of memory using float32 (float). I am doing this in order to save memory for all datasets.",
      "votes": null
    },
    {
      "id": "133883",
      "postDate": "09/04/2016 09:08:21",
      "content": "<p>Right,, but...</p>\n\n<p>If you do the above followed by pandas.factorize, then actual numbers do not matter and you can use <code>int8</code> to encode categorical features if you wish to.  NaNs are encoded with -1 in this case.</p>",
      "rawMarkdown": "Right,, but...\r\n\r\nIf you do the above followed by pandas.factorize, then actual numbers do not matter and you can use `int8` to encode categorical features if you wish to.  NaNs are encoded with -1 in this case.",
      "votes": null
    },
    {
      "id": "133915",
      "postDate": "09/04/2016 13:29:08",
      "content": "<p>I see, it makes sense. I am not too familiar with pandas :)</p>",
      "rawMarkdown": "I see, it makes sense. I am not too familiar with pandas :)",
      "votes": null
    },
    {
      "id": "133959",
      "postDate": "09/04/2016 15:30:05",
      "content": "<p>No pb, I am learning about pandas too...  It is so rich and evolving so fast...</p>",
      "rawMarkdown": "No pb, I am learning about pandas too...  It is so rich and evolving so fast...",
      "votes": null
    },
    {
      "id": "134218",
      "postDate": "09/05/2016 21:11:48",
      "content": "<p>Thanks, this worked well.</p>\n\n<p>I had to use a slightly different syntax with MINGW in WIndows</p>\n\n<pre><code>sed -i.sav -e 's/T-//g' train_categorical.csv \nsed -i.sav -e 's/T-//g' test_categorical.csv \n</code></pre>\n\n<p>Notice, no space after the -i. I thought it wasn't doing anything until I realized it overwrote the original CSV.</p>",
      "rawMarkdown": "Thanks, this worked well.\r\n\r\nI had to use a slightly different syntax with MINGW in WIndows\r\n\r\n    sed -i.sav -e 's/T-//g' train_categorical.csv \r\n    sed -i.sav -e 's/T-//g' test_categorical.csv \r\n\r\nNotice, no space after the -i. I thought it wasn't doing anything until I realized it overwrote the original CSV.",
      "votes": null
    },
    {
      "id": "134358",
      "postDate": "09/06/2016 09:50:17",
      "content": "<p>Thanks Jeff, most cases are covered now.</p>",
      "rawMarkdown": "Thanks Jeff, most cases are covered now.",
      "votes": null
    },
    {
      "id": "134431",
      "postDate": "09/06/2016 15:37:09",
      "content": "<p>[quote=CPMP;133959]</p>\n\n<p>No pb, I am learning about pandas too...  It is so rich and evolving so fast...</p>\n\n<p>[/quote]</p>\n\n<p>How can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other</p>",
      "rawMarkdown": "[quote=CPMP;133959]\r\n\r\nNo pb, I am learning about pandas too...  It is so rich and evolving so fast...\r\n\r\n[/quote]\r\n\r\nHow can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other",
      "votes": null
    },
    {
      "id": "134432",
      "postDate": "09/06/2016 15:38:30",
      "content": "<p>I don't get your question.  Here we simply get rid of the 'T' character so that the categorical files only contain numerical values.  </p>",
      "rawMarkdown": "I don't get your question.  Here we simply get rid of the 'T' character so that the categorical files only contain numerical values.",
      "votes": null
    },
    {
      "id": "134436",
      "postDate": "09/06/2016 16:23:06",
      "content": "<p>[quote=ZohairZahid;134431]</p>\n\n<p>How can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other</p>\n\n<p>[/quote]</p>\n\n<p>If I understand your question correctly, in my experience I generally create a method that loads and &quot;pre-process&quot; the datasets, which I call during training or testing(predicting).</p>",
      "rawMarkdown": "[quote=ZohairZahid;134431]\r\n\r\nHow can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other\r\n\r\n[/quote]\r\n\r\nIf I understand your question correctly, in my experience I generally create a method that loads and \"pre-process\" the datasets, which I call during training or testing(predicting).",
      "votes": null
    },
    {
      "id": "134484",
      "postDate": "09/06/2016 20:49:49",
      "content": "<p>[quote=ZohairZahid;134431]</p>\n\n<p>[quote=CPMP;133959]</p>\n\n<p>No pb, I am learning about pandas too...  It is so rich and evolving so fast...</p>\n\n<p>[/quote]</p>\n\n<p>How can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other</p>\n\n<p>[/quote]</p>\n\n<p>I think Zohair means how do we handle the following...</p>\n\n<p>Train.columnx has the following values in it {'a','b','c'}.\nTest.columnx has the values {'a','c','d'}</p>\n\n<p>When you convert the values to numerical values in each file by itself train converts 'a' to 1, 'b' to 2,'c' to 3.  While test converts 'a' to 1, 'c' to 2, and 'd' to 3.  When you build a model on training data a model looks at c=3, but when predicting on the test set d=3.</p>\n\n<p>Usually you solve this by appending the train and test set together before label encoding so that a=1, b=2, c=3 and d=4 in both sets of data.</p>",
      "rawMarkdown": "[quote=ZohairZahid;134431]\r\n\r\n[quote=CPMP;133959]\r\n\r\nNo pb, I am learning about pandas too...  It is so rich and evolving so fast...\r\n\r\n[/quote]\r\n\r\nHow can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other\r\n\r\n[/quote]\r\n\r\nI think Zohair means how do we handle the following...\r\n\r\nTrain.columnx has the following values in it {'a','b','c'}.\r\nTest.columnx has the values {'a','c','d'}\r\n\r\nWhen you convert the values to numerical values in each file by itself train converts 'a' to 1, 'b' to 2,'c' to 3.  While test converts 'a' to 1, 'c' to 2, and 'd' to 3.  When you build a model on training data a model looks at c=3, but when predicting on the test set d=3.\r\n\r\nUsually you solve this by appending the train and test set together before label encoding so that a=1, b=2, c=3 and d=4 in both sets of data.",
      "votes": null
    },
    {
      "id": "134515",
      "postDate": "09/07/2016 01:52:38",
      "content": "<p>[quote=Michael Maguire;134484]</p>\n\n<p>[quote=ZohairZahid;134431]</p>\n\n<p>[quote=CPMP;133959]</p>\n\n<p>No pb, I am learning about pandas too...  It is so rich and evolving so fast...</p>\n\n<p>[/quote]</p>\n\n<p>How can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other</p>\n\n<p>[/quote]</p>\n\n<p>I think Zohair means how do we handle the following...</p>\n\n<p>Train.columnx has the following values in it {'a','b','c'}.\nTest.columnx has the values {'a','c','d'}</p>\n\n<p>When you convert the values to numerical values in each file by itself train converts 'a' to 1, 'b' to 2,'c' to 3.  While test converts 'a' to 1, 'c' to 2, and 'd' to 3.  When you build a model on training data a model looks at c=3, but when predicting on the test set d=3.</p>\n\n<p>Usually you solve this by appending the train and test set together before label encoding so that a=1, b=2, c=3 and d=4 in both sets of data.</p>\n\n<p>[/quote]</p>\n\n<p>Thanks Michael Maguire, agree</p>",
      "rawMarkdown": "[quote=Michael Maguire;134484]\r\n\r\n[quote=ZohairZahid;134431]\r\n\r\n[quote=CPMP;133959]\r\n\r\nNo pb, I am learning about pandas too...  It is so rich and evolving so fast...\r\n\r\n[/quote]\r\n\r\nHow can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other\r\n\r\n[/quote]\r\n\r\nI think Zohair means how do we handle the following...\r\n\r\nTrain.columnx has the following values in it {'a','b','c'}.\r\nTest.columnx has the values {'a','c','d'}\r\n\r\nWhen you convert the values to numerical values in each file by itself train converts 'a' to 1, 'b' to 2,'c' to 3.  While test converts 'a' to 1, 'c' to 2, and 'd' to 3.  When you build a model on training data a model looks at c=3, but when predicting on the test set d=3.\r\n\r\nUsually you solve this by appending the train and test set together before label encoding so that a=1, b=2, c=3 and d=4 in both sets of data.\r\n\r\n[/quote]\r\n\r\nThanks Michael Maguire, agree",
      "votes": null
    },
    {
      "id": "134548",
      "postDate": "09/07/2016 07:19:48",
      "content": "<p>[quote=fxarte;134515]</p>\n\n<p>[quote=Michael Maguire;134484]</p>\n\n<p>[quote=ZohairZahid;134431]</p>\n\n<p>[quote=CPMP;133959]</p>\n\n<p>No pb, I am learning about pandas too...  It is so rich and evolving so fast...</p>\n\n<p>[/quote]</p>\n\n<p>How can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other</p>\n\n<p>[/quote]</p>\n\n<p>I think Zohair means how do we handle the following...</p>\n\n<p>Train.columnx has the following values in it {'a','b','c'}.\nTest.columnx has the values {'a','c','d'}</p>\n\n<p>When you convert the values to numerical values in each file by itself train converts 'a' to 1, 'b' to 2,'c' to 3.  While test converts 'a' to 1, 'c' to 2, and 'd' to 3.  When you build a model on training data a model looks at c=3, but when predicting on the test set d=3.</p>\n\n<p>Usually you solve this by appending the train and test set together before label encoding so that a=1, b=2, c=3 and d=4 in both sets of data.</p>\n\n<p>[/quote]</p>\n\n<p>Thanks Michael Maguire, agree</p>\n\n<p>[/quote]</p>\n\n<p>Yes exactly Michael. Thanks</p>",
      "rawMarkdown": "[quote=fxarte;134515]\r\n\r\n[quote=Michael Maguire;134484]\r\n\r\n[quote=ZohairZahid;134431]\r\n\r\n[quote=CPMP;133959]\r\n\r\nNo pb, I am learning about pandas too...  It is so rich and evolving so fast...\r\n\r\n[/quote]\r\n\r\nHow can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other\r\n\r\n[/quote]\r\n\r\nI think Zohair means how do we handle the following...\r\n\r\nTrain.columnx has the following values in it {'a','b','c'}.\r\nTest.columnx has the values {'a','c','d'}\r\n\r\nWhen you convert the values to numerical values in each file by itself train converts 'a' to 1, 'b' to 2,'c' to 3.  While test converts 'a' to 1, 'c' to 2, and 'd' to 3.  When you build a model on training data a model looks at c=3, but when predicting on the test set d=3.\r\n\r\nUsually you solve this by appending the train and test set together before label encoding so that a=1, b=2, c=3 and d=4 in both sets of data.\r\n\r\n[/quote]\r\n\r\nThanks Michael Maguire, agree\r\n\r\n[/quote]\r\n\r\nYes exactly Michael. Thanks",
      "votes": null
    },
    {
      "id": "134561",
      "postDate": "09/07/2016 08:36:54",
      "content": "<p>Michael, thanks.  As said elsewhere, if you follow this with one hot encoding then you just sdrop the values appearing in test that do not appear in train.</p>\n\n<p>I this example Train.columnx has the following values in it {'a','b','c'}. Test.columnx has the values {'a','c','d'}</p>\n\n<p>You replace Train.columnx and Test.columnx with 3 binary features, one for each of {'a','b','c'}.  rows in test that have the value 'd' for columnx will have a value 0 for all three features.</p>",
      "rawMarkdown": "Michael, thanks.  As said elsewhere, if you follow this with one hot encoding then you just sdrop the values appearing in test that do not appear in train.\r\n\r\nI this example Train.columnx has the following values in it {'a','b','c'}. Test.columnx has the values {'a','c','d'}\r\n\r\nYou replace Train.columnx and Test.columnx with 3 binary features, one for each of {'a','b','c'}.  rows in test that have the value 'd' for columnx will have a value 0 for all three features.",
      "votes": null
    },
    {
      "id": "134605",
      "postDate": "09/07/2016 13:53:32",
      "content": "<p>[quote=CPMP;134561]</p>\n\n<p>Michael, thanks.  As said elsewhere, if you follow this with one hot encoding then you just sdrop the values appearing in test that do not appear in train.</p>\n\n<p>I this example Train.columnx has the following values in it {'a','b','c'}. Test.columnx has the values {'a','c','d'}</p>\n\n<p>You replace Train.columnx and Test.columnx with 3 binary features, one for each of {'a','b','c'}.  rows in test that have the value 'd' for columnx will have a value 0 for all three features.</p>\n\n<p>[/quote]\nSorry, the pandas factorize function was mentioned above and that label encodes the values.  One hot encoding would solve the problem.</p>",
      "rawMarkdown": "[quote=CPMP;134561]\r\n\r\nMichael, thanks.  As said elsewhere, if you follow this with one hot encoding then you just sdrop the values appearing in test that do not appear in train.\r\n\r\nI this example Train.columnx has the following values in it {'a','b','c'}. Test.columnx has the values {'a','c','d'}\r\n\r\nYou replace Train.columnx and Test.columnx with 3 binary features, one for each of {'a','b','c'}.  rows in test that have the value 'd' for columnx will have a value 0 for all three features.\r\n\r\n[/quote]\r\nSorry, the pandas factorize function was mentioned above and that label encodes the values.  One hot encoding would solve the problem.",
      "votes": null
    },
    {
      "id": "134733",
      "postDate": "09/08/2016 07:16:33",
      "content": "<p>Michael,</p>\n\n<p>I think we are in agreement.  If you use pandas factorize before doing ohe then you definitely need to concatenate train and test before calling factorize.</p>",
      "rawMarkdown": "Michael,\r\n\r\nI think we are in agreement.  If you use pandas factorize before doing ohe then you definitely need to concatenate train and test before calling factorize.",
      "votes": null
    },
    {
      "id": "134739",
      "postDate": "09/08/2016 08:06:42",
      "content": "<p>Hi CMPM, \nI tried to concatenate using pandas but apparently it seems that the data is too big for it . I get the following error.</p>\n\n<pre><code>File &quot;C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\tools\\merge.py&quot;, line 835, in concat\n    return op.get_result()\n  File &quot;C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\tools\\merge.py&quot;, line 1025, in get_result\n    concat_axis=self.axis, copy=self.copy)\n  File &quot;C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py&quot;, line 4474, in concatenate_block_managers\n    for placement, join_units in concat_plan]\n  File &quot;C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py&quot;, line 4567, in concatenate_join_units\n    empty_dtype, upcasted_na = get_empty_dtype_and_na(join_units)\n  File &quot;C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py&quot;, line 4528, in get_empty_dtype_and_na\n    if unit.is_null:\n  File &quot;pandas\\src\\properties.pyx&quot;, line 34, in pandas.lib.cache_readonly.__get__ (pandas\\lib.c:44357)\n  File &quot;C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py&quot;, line 4808, in is_null\n    for i in range(0, total_len, chunk_len):\nOverflowError: Python int too large to convert to C long\n</code></pre>\n\n<p>[quote=CPMP;134733]</p>\n\n<p>Michael,</p>\n\n<p>I think we are in agreement.  If you use pandas factorize before doing ohe then you definitely need to concatenate train and test before calling factorize.</p>\n\n<p>[/quote]</p>",
      "rawMarkdown": "Hi CMPM, \r\nI tried to concatenate using pandas but apparently it seems that the data is too big for it . I get the following error.\r\n\r\n    File \"C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\tools\\merge.py\", line 835, in concat\r\n        return op.get_result()\r\n      File \"C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\tools\\merge.py\", line 1025, in get_result\r\n        concat_axis=self.axis, copy=self.copy)\r\n      File \"C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py\", line 4474, in concatenate_block_managers\r\n        for placement, join_units in concat_plan]\r\n      File \"C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py\", line 4567, in concatenate_join_units\r\n        empty_dtype, upcasted_na = get_empty_dtype_and_na(join_units)\r\n      File \"C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py\", line 4528, in get_empty_dtype_and_na\r\n        if unit.is_null:\r\n      File \"pandas\\src\\properties.pyx\", line 34, in pandas.lib.cache_readonly.__get__ (pandas\\lib.c:44357)\r\n      File \"C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py\", line 4808, in is_null\r\n        for i in range(0, total_len, chunk_len):\r\n    OverflowError: Python int too large to convert to C long\r\n\r\n \r\n\r\n[quote=CPMP;134733]\r\n\r\nMichael,\r\n\r\nI think we are in agreement.  If you use pandas factorize before doing ohe then you definitely need to concatenate train and test before calling factorize.\r\n\r\n[/quote]",
      "votes": null
    },
    {
      "id": "134741",
      "postDate": "09/08/2016 08:14:45",
      "content": "<p>Zohair,</p>\n\n<p>I could load all data in memory using pandas sparse data frames.  Call to_sparse() from your data frames to get sparse versions of them, then you may be able to concatenate them.</p>\n\n<p>For this you need to use the latest pandas version (18.1 if I am not mistaken).  I see you use anaconda.  just type the following in an anaconda prompt on Windows, or in a shell window on linux or macosx:</p>\n\n<pre><code>conda update pandas\n</code></pre>\n\n<p>then restart your notebooks.</p>\n\n<p>Main issue with these sparse data frames is that code is young and has some limit.   I could not save them for instance.</p>",
      "rawMarkdown": "Zohair,\r\n\r\nI could load all data in memory using pandas sparse data frames.  Call to_sparse() from your data frames to get sparse versions of them, then you may be able to concatenate them.\r\n\r\nFor this you need to use the latest pandas version (18.1 if I am not mistaken).  I see you use anaconda.  just type the following in an anaconda prompt on Windows, or in a shell window on linux or macosx:\r\n\r\n    conda update pandas\r\n\r\nthen restart your notebooks.\r\n\r\nMain issue with these sparse data frames is that code is young and has some limit.   I could not save them for instance.",
      "votes": null
    },
    {
      "id": "134803",
      "postDate": "09/08/2016 14:51:24",
      "content": "<p>CMPM,\nI was able to concatenate but the thing is I think pandas factorize does not take sparse matrix as input so it needs to be converted to dense matrix again. Now that I am factorizing it, it's running for last 2 hours, no idea what state it is in. Any hints or any idea how much time it takes ?</p>\n\n<p>[quote=CPMP;134741]</p>\n\n<p>Zohair,</p>\n\n<p>I could load all data in memory using pandas sparse data frames.  Call to_sparse() from your data frames to get sparse versions of them, then you may be able to concatenate them.</p>\n\n<p>For this you need to use the latest pandas version (18.1 if I am not mistaken).  I see you use anaconda.  just type the following in an anaconda prompt on Windows, or in a shell window on linux or macosx:</p>\n\n<pre><code>conda update pandas\n</code></pre>\n\n<p>then restart your notebooks.</p>\n\n<p>Main issue with these sparse data frames is that code is young and has some limit.   I could not save them for instance.</p>\n\n<p>[/quote]</p>",
      "rawMarkdown": "CMPM,\r\nI was able to concatenate but the thing is I think pandas factorize does not take sparse matrix as input so it needs to be converted to dense matrix again. Now that I am factorizing it, it's running for last 2 hours, no idea what state it is in. Any hints or any idea how much time it takes ?\r\n\r\n[quote=CPMP;134741]\r\n\r\nZohair,\r\n\r\nI could load all data in memory using pandas sparse data frames.  Call to_sparse() from your data frames to get sparse versions of them, then you may be able to concatenate them.\r\n\r\nFor this you need to use the latest pandas version (18.1 if I am not mistaken).  I see you use anaconda.  just type the following in an anaconda prompt on Windows, or in a shell window on linux or macosx:\r\n\r\n    conda update pandas\r\n\r\nthen restart your notebooks.\r\n\r\nMain issue with these sparse data frames is that code is young and has some limit.   I could not save them for instance.\r\n\r\n[/quote]",
      "votes": null
    },
    {
      "id": "138071",
      "postDate": "10/07/2016 00:19:28",
      "content": "<p>[quote=JeffH;134218]</p>\n\n<p>Thanks, this worked well.</p>\n\n<p>I had to use a slightly different syntax with MINGW in WIndows</p>\n\n<pre><code>sed -i.sav -e 's/T-//g' train_categorical.csv \nsed -i.sav -e 's/T-//g' test_categorical.csv \n</code></pre>\n\n<p>Notice, no space after the -i. I thought it wasn't doing anything until I realized it overwrote the original CSV.</p>\n\n<p>[/quote]</p>\n\n<p>I'm using sed (GNU sed) 4.2.2 on Linux and had to use this method.</p>",
      "rawMarkdown": "[quote=JeffH;134218]\r\n\r\nThanks, this worked well.\r\n\r\nI had to use a slightly different syntax with MINGW in WIndows\r\n\r\n    sed -i.sav -e 's/T-//g' train_categorical.csv \r\n    sed -i.sav -e 's/T-//g' test_categorical.csv \r\n\r\nNotice, no space after the -i. I thought it wasn't doing anything until I realized it overwrote the original CSV.\r\n\r\n[/quote]\r\n\r\nI'm using sed (GNU sed) 4.2.2 on Linux and had to use this method.",
      "votes": null
    },
    {
      "id": "138094",
      "postDate": "10/07/2016 06:02:22",
      "content": "<p>Hmm... ok, so it wasn't what I wanted. </p>\n\n<p>I decided to keep negative numbers and replace NaNs with 0s. Now I can load categorical data with read_csv as dtype=int64.</p>",
      "rawMarkdown": "Hmm... ok, so it wasn't what I wanted. \r\n\r\nI decided to keep negative numbers and replace NaNs with 0s. Now I can load categorical data with read_csv as dtype=int64.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 133575,
      "author_name": "raphaelrcampos",
      "author_url": "",
      "post_date": "09/02/2016 19:42:47",
      "content": "<p>If you want a non-negative representation, to be more specific, you also have to remove the '-' character that appears in some features. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133692,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "09/03/2016 11:13:46",
      "content": "<p>Is it useful to get rid of negativity?</p>\n\n<p>Note that the following will not remove the negativity, it will remove all content ;)</p>\n\n<pre><code>sed -i .sav 's/T-//g' train_categorical.csv \nsed -i .sav 's/T-//g' test_categorical.csv \n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133810,
      "author_name": "raphaelrcampos",
      "author_url": "",
      "post_date": "09/03/2016 22:02:41",
      "content": "<p>I think it might be useful to save some memory. I have to check, but I am quite sure that there are some huge numbers that may not fit in an integer for example.</p>\n\n<p>By default, python numpy/scipy.sparse uses float64 (double) which consumes 8 bytes, on the other hand you can save half of memory using float32 (float). I am doing this in order to save memory for all datasets.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133883,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "09/04/2016 09:08:21",
      "content": "<p>Right,, but...</p>\n\n<p>If you do the above followed by pandas.factorize, then actual numbers do not matter and you can use <code>int8</code> to encode categorical features if you wish to.  NaNs are encoded with -1 in this case.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133915,
      "author_name": "raphaelrcampos",
      "author_url": "",
      "post_date": "09/04/2016 13:29:08",
      "content": "<p>I see, it makes sense. I am not too familiar with pandas :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133959,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "09/04/2016 15:30:05",
      "content": "<p>No pb, I am learning about pandas too...  It is so rich and evolving so fast...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134218,
      "author_name": "jeffhebert",
      "author_url": "",
      "post_date": "09/05/2016 21:11:48",
      "content": "<p>Thanks, this worked well.</p>\n\n<p>I had to use a slightly different syntax with MINGW in WIndows</p>\n\n<pre><code>sed -i.sav -e 's/T-//g' train_categorical.csv \nsed -i.sav -e 's/T-//g' test_categorical.csv \n</code></pre>\n\n<p>Notice, no space after the -i. I thought it wasn't doing anything until I realized it overwrote the original CSV.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134358,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "09/06/2016 09:50:17",
      "content": "<p>Thanks Jeff, most cases are covered now.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134431,
      "author_name": "zohairzohair4",
      "author_url": "",
      "post_date": "09/06/2016 15:37:09",
      "content": "<p>[quote=CPMP;133959]</p>\n\n<p>No pb, I am learning about pandas too...  It is so rich and evolving so fast...</p>\n\n<p>[/quote]</p>\n\n<p>How can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134432,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "09/06/2016 15:38:30",
      "content": "<p>I don't get your question.  Here we simply get rid of the 'T' character so that the categorical files only contain numerical values.  </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134436,
      "author_name": "fxarte",
      "author_url": "",
      "post_date": "09/06/2016 16:23:06",
      "content": "<p>[quote=ZohairZahid;134431]</p>\n\n<p>How can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other</p>\n\n<p>[/quote]</p>\n\n<p>If I understand your question correctly, in my experience I generally create a method that loads and &quot;pre-process&quot; the datasets, which I call during training or testing(predicting).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134484,
      "author_name": "thawatt",
      "author_url": "",
      "post_date": "09/06/2016 20:49:49",
      "content": "<p>[quote=ZohairZahid;134431]</p>\n\n<p>[quote=CPMP;133959]</p>\n\n<p>No pb, I am learning about pandas too...  It is so rich and evolving so fast...</p>\n\n<p>[/quote]</p>\n\n<p>How can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other</p>\n\n<p>[/quote]</p>\n\n<p>I think Zohair means how do we handle the following...</p>\n\n<p>Train.columnx has the following values in it {'a','b','c'}.\nTest.columnx has the values {'a','c','d'}</p>\n\n<p>When you convert the values to numerical values in each file by itself train converts 'a' to 1, 'b' to 2,'c' to 3.  While test converts 'a' to 1, 'c' to 2, and 'd' to 3.  When you build a model on training data a model looks at c=3, but when predicting on the test set d=3.</p>\n\n<p>Usually you solve this by appending the train and test set together before label encoding so that a=1, b=2, c=3 and d=4 in both sets of data.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134515,
      "author_name": "fxarte",
      "author_url": "",
      "post_date": "09/07/2016 01:52:38",
      "content": "<p>[quote=Michael Maguire;134484]</p>\n\n<p>[quote=ZohairZahid;134431]</p>\n\n<p>[quote=CPMP;133959]</p>\n\n<p>No pb, I am learning about pandas too...  It is so rich and evolving so fast...</p>\n\n<p>[/quote]</p>\n\n<p>How can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other</p>\n\n<p>[/quote]</p>\n\n<p>I think Zohair means how do we handle the following...</p>\n\n<p>Train.columnx has the following values in it {'a','b','c'}.\nTest.columnx has the values {'a','c','d'}</p>\n\n<p>When you convert the values to numerical values in each file by itself train converts 'a' to 1, 'b' to 2,'c' to 3.  While test converts 'a' to 1, 'c' to 2, and 'd' to 3.  When you build a model on training data a model looks at c=3, but when predicting on the test set d=3.</p>\n\n<p>Usually you solve this by appending the train and test set together before label encoding so that a=1, b=2, c=3 and d=4 in both sets of data.</p>\n\n<p>[/quote]</p>\n\n<p>Thanks Michael Maguire, agree</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134548,
      "author_name": "zohairzohair4",
      "author_url": "",
      "post_date": "09/07/2016 07:19:48",
      "content": "<p>[quote=fxarte;134515]</p>\n\n<p>[quote=Michael Maguire;134484]</p>\n\n<p>[quote=ZohairZahid;134431]</p>\n\n<p>[quote=CPMP;133959]</p>\n\n<p>No pb, I am learning about pandas too...  It is so rich and evolving so fast...</p>\n\n<p>[/quote]</p>\n\n<p>How can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other</p>\n\n<p>[/quote]</p>\n\n<p>I think Zohair means how do we handle the following...</p>\n\n<p>Train.columnx has the following values in it {'a','b','c'}.\nTest.columnx has the values {'a','c','d'}</p>\n\n<p>When you convert the values to numerical values in each file by itself train converts 'a' to 1, 'b' to 2,'c' to 3.  While test converts 'a' to 1, 'c' to 2, and 'd' to 3.  When you build a model on training data a model looks at c=3, but when predicting on the test set d=3.</p>\n\n<p>Usually you solve this by appending the train and test set together before label encoding so that a=1, b=2, c=3 and d=4 in both sets of data.</p>\n\n<p>[/quote]</p>\n\n<p>Thanks Michael Maguire, agree</p>\n\n<p>[/quote]</p>\n\n<p>Yes exactly Michael. Thanks</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134561,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "09/07/2016 08:36:54",
      "content": "<p>Michael, thanks.  As said elsewhere, if you follow this with one hot encoding then you just sdrop the values appearing in test that do not appear in train.</p>\n\n<p>I this example Train.columnx has the following values in it {'a','b','c'}. Test.columnx has the values {'a','c','d'}</p>\n\n<p>You replace Train.columnx and Test.columnx with 3 binary features, one for each of {'a','b','c'}.  rows in test that have the value 'd' for columnx will have a value 0 for all three features.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134605,
      "author_name": "thawatt",
      "author_url": "",
      "post_date": "09/07/2016 13:53:32",
      "content": "<p>[quote=CPMP;134561]</p>\n\n<p>Michael, thanks.  As said elsewhere, if you follow this with one hot encoding then you just sdrop the values appearing in test that do not appear in train.</p>\n\n<p>I this example Train.columnx has the following values in it {'a','b','c'}. Test.columnx has the values {'a','c','d'}</p>\n\n<p>You replace Train.columnx and Test.columnx with 3 binary features, one for each of {'a','b','c'}.  rows in test that have the value 'd' for columnx will have a value 0 for all three features.</p>\n\n<p>[/quote]\nSorry, the pandas factorize function was mentioned above and that label encodes the values.  One hot encoding would solve the problem.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134733,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "09/08/2016 07:16:33",
      "content": "<p>Michael,</p>\n\n<p>I think we are in agreement.  If you use pandas factorize before doing ohe then you definitely need to concatenate train and test before calling factorize.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134739,
      "author_name": "zohairzohair4",
      "author_url": "",
      "post_date": "09/08/2016 08:06:42",
      "content": "<p>Hi CMPM, \nI tried to concatenate using pandas but apparently it seems that the data is too big for it . I get the following error.</p>\n\n<pre><code>File &quot;C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\tools\\merge.py&quot;, line 835, in concat\n    return op.get_result()\n  File &quot;C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\tools\\merge.py&quot;, line 1025, in get_result\n    concat_axis=self.axis, copy=self.copy)\n  File &quot;C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py&quot;, line 4474, in concatenate_block_managers\n    for placement, join_units in concat_plan]\n  File &quot;C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py&quot;, line 4567, in concatenate_join_units\n    empty_dtype, upcasted_na = get_empty_dtype_and_na(join_units)\n  File &quot;C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py&quot;, line 4528, in get_empty_dtype_and_na\n    if unit.is_null:\n  File &quot;pandas\\src\\properties.pyx&quot;, line 34, in pandas.lib.cache_readonly.__get__ (pandas\\lib.c:44357)\n  File &quot;C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py&quot;, line 4808, in is_null\n    for i in range(0, total_len, chunk_len):\nOverflowError: Python int too large to convert to C long\n</code></pre>\n\n<p>[quote=CPMP;134733]</p>\n\n<p>Michael,</p>\n\n<p>I think we are in agreement.  If you use pandas factorize before doing ohe then you definitely need to concatenate train and test before calling factorize.</p>\n\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134741,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "09/08/2016 08:14:45",
      "content": "<p>Zohair,</p>\n\n<p>I could load all data in memory using pandas sparse data frames.  Call to_sparse() from your data frames to get sparse versions of them, then you may be able to concatenate them.</p>\n\n<p>For this you need to use the latest pandas version (18.1 if I am not mistaken).  I see you use anaconda.  just type the following in an anaconda prompt on Windows, or in a shell window on linux or macosx:</p>\n\n<pre><code>conda update pandas\n</code></pre>\n\n<p>then restart your notebooks.</p>\n\n<p>Main issue with these sparse data frames is that code is young and has some limit.   I could not save them for instance.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 134803,
      "author_name": "zohairzohair4",
      "author_url": "",
      "post_date": "09/08/2016 14:51:24",
      "content": "<p>CMPM,\nI was able to concatenate but the thing is I think pandas factorize does not take sparse matrix as input so it needs to be converted to dense matrix again. Now that I am factorizing it, it's running for last 2 hours, no idea what state it is in. Any hints or any idea how much time it takes ?</p>\n\n<p>[quote=CPMP;134741]</p>\n\n<p>Zohair,</p>\n\n<p>I could load all data in memory using pandas sparse data frames.  Call to_sparse() from your data frames to get sparse versions of them, then you may be able to concatenate them.</p>\n\n<p>For this you need to use the latest pandas version (18.1 if I am not mistaken).  I see you use anaconda.  just type the following in an anaconda prompt on Windows, or in a shell window on linux or macosx:</p>\n\n<pre><code>conda update pandas\n</code></pre>\n\n<p>then restart your notebooks.</p>\n\n<p>Main issue with these sparse data frames is that code is young and has some limit.   I could not save them for instance.</p>\n\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 138071,
      "author_name": "rshpeley",
      "author_url": "",
      "post_date": "10/07/2016 00:19:28",
      "content": "<p>[quote=JeffH;134218]</p>\n\n<p>Thanks, this worked well.</p>\n\n<p>I had to use a slightly different syntax with MINGW in WIndows</p>\n\n<pre><code>sed -i.sav -e 's/T-//g' train_categorical.csv \nsed -i.sav -e 's/T-//g' test_categorical.csv \n</code></pre>\n\n<p>Notice, no space after the -i. I thought it wasn't doing anything until I realized it overwrote the original CSV.</p>\n\n<p>[/quote]</p>\n\n<p>I'm using sed (GNU sed) 4.2.2 on Linux and had to use this method.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 138094,
      "author_name": "rshpeley",
      "author_url": "",
      "post_date": "10/07/2016 06:02:22",
      "content": "<p>Hmm... ok, so it wasn't what I wanted. </p>\n\n<p>I decided to keep negative numbers and replace NaNs with 0s. Now I can load categorical data with read_csv as dtype=int64.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "133475": "The train_categorical file contains strings and NaNs.  This is not very friendly to pandas.  Here is a simple preprocessing command you can run in Linux or Mac OSX (and maybe cygwin on Windows) to turn all strings into numerical values:\r\n\r\n    sed -i .sav 's/T//g' train_categorical.csv \r\n    sed -i .sav 's/T//g' test_categorical.csv \r\n\r\nIt saves the original files with a sav extension, and produces new files where all data is numeric.  NaNs aren't modified.\r\n\r\nThis saves lots of memory compared to reading the original file, especially if you specify a dtype when reading.",
    "133575": "If you want a non-negative representation, to be more specific, you also have to remove the '-' character that appears in some features.",
    "133692": "Is it useful to get rid of negativity?\r\n\r\nNote that the following will not remove the negativity, it will remove all content ;)\r\n\r\n    sed -i .sav 's/T-//g' train_categorical.csv \r\n    sed -i .sav 's/T-//g' test_categorical.csv",
    "133810": "I think it might be useful to save some memory. I have to check, but I am quite sure that there are some huge numbers that may not fit in an integer for example.\r\n\r\nBy default, python numpy/scipy.sparse uses float64 (double) which consumes 8 bytes, on the other hand you can save half of memory using float32 (float). I am doing this in order to save memory for all datasets.",
    "133883": "Right,, but...\r\n\r\nIf you do the above followed by pandas.factorize, then actual numbers do not matter and you can use `int8` to encode categorical features if you wish to.  NaNs are encoded with -1 in this case.",
    "133915": "I see, it makes sense. I am not too familiar with pandas :)",
    "133959": "No pb, I am learning about pandas too...  It is so rich and evolving so fast...",
    "134218": "Thanks, this worked well.\r\n\r\nI had to use a slightly different syntax with MINGW in WIndows\r\n\r\n    sed -i.sav -e 's/T-//g' train_categorical.csv \r\n    sed -i.sav -e 's/T-//g' test_categorical.csv \r\n\r\nNotice, no space after the -i. I thought it wasn't doing anything until I realized it overwrote the original CSV.",
    "134358": "Thanks Jeff, most cases are covered now.",
    "134431": "[quote=CPMP;133959]\r\n\r\nNo pb, I am learning about pandas too...  It is so rich and evolving so fast...\r\n\r\n[/quote]\r\n\r\nHow can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other",
    "134432": "I don't get your question.  Here we simply get rid of the 'T' character so that the categorical files only contain numerical values.",
    "134436": "[quote=ZohairZahid;134431]\r\n\r\nHow can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other\r\n\r\n[/quote]\r\n\r\nIf I understand your question correctly, in my experience I generally create a method that loads and \"pre-process\" the datasets, which I call during training or testing(predicting).",
    "134484": "[quote=ZohairZahid;134431]\r\n\r\n[quote=CPMP;133959]\r\n\r\nNo pb, I am learning about pandas too...  It is so rich and evolving so fast...\r\n\r\n[/quote]\r\n\r\nHow can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other\r\n\r\n[/quote]\r\n\r\nI think Zohair means how do we handle the following...\r\n\r\nTrain.columnx has the following values in it {'a','b','c'}.\r\nTest.columnx has the values {'a','c','d'}\r\n\r\nWhen you convert the values to numerical values in each file by itself train converts 'a' to 1, 'b' to 2,'c' to 3.  While test converts 'a' to 1, 'c' to 2, and 'd' to 3.  When you build a model on training data a model looks at c=3, but when predicting on the test set d=3.\r\n\r\nUsually you solve this by appending the train and test set together before label encoding so that a=1, b=2, c=3 and d=4 in both sets of data.",
    "134515": "[quote=Michael Maguire;134484]\r\n\r\n[quote=ZohairZahid;134431]\r\n\r\n[quote=CPMP;133959]\r\n\r\nNo pb, I am learning about pandas too...  It is so rich and evolving so fast...\r\n\r\n[/quote]\r\n\r\nHow can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other\r\n\r\n[/quote]\r\n\r\nI think Zohair means how do we handle the following...\r\n\r\nTrain.columnx has the following values in it {'a','b','c'}.\r\nTest.columnx has the values {'a','c','d'}\r\n\r\nWhen you convert the values to numerical values in each file by itself train converts 'a' to 1, 'b' to 2,'c' to 3.  While test converts 'a' to 1, 'c' to 2, and 'd' to 3.  When you build a model on training data a model looks at c=3, but when predicting on the test set d=3.\r\n\r\nUsually you solve this by appending the train and test set together before label encoding so that a=1, b=2, c=3 and d=4 in both sets of data.\r\n\r\n[/quote]\r\n\r\nThanks Michael Maguire, agree",
    "134548": "[quote=fxarte;134515]\r\n\r\n[quote=Michael Maguire;134484]\r\n\r\n[quote=ZohairZahid;134431]\r\n\r\n[quote=CPMP;133959]\r\n\r\nNo pb, I am learning about pandas too...  It is so rich and evolving so fast...\r\n\r\n[/quote]\r\n\r\nHow can we make sure that the conversion from categorical to numerical is done the same way in both Train and Test data, so that the values correspond with each other\r\n\r\n[/quote]\r\n\r\nI think Zohair means how do we handle the following...\r\n\r\nTrain.columnx has the following values in it {'a','b','c'}.\r\nTest.columnx has the values {'a','c','d'}\r\n\r\nWhen you convert the values to numerical values in each file by itself train converts 'a' to 1, 'b' to 2,'c' to 3.  While test converts 'a' to 1, 'c' to 2, and 'd' to 3.  When you build a model on training data a model looks at c=3, but when predicting on the test set d=3.\r\n\r\nUsually you solve this by appending the train and test set together before label encoding so that a=1, b=2, c=3 and d=4 in both sets of data.\r\n\r\n[/quote]\r\n\r\nThanks Michael Maguire, agree\r\n\r\n[/quote]\r\n\r\nYes exactly Michael. Thanks",
    "134561": "Michael, thanks.  As said elsewhere, if you follow this with one hot encoding then you just sdrop the values appearing in test that do not appear in train.\r\n\r\nI this example Train.columnx has the following values in it {'a','b','c'}. Test.columnx has the values {'a','c','d'}\r\n\r\nYou replace Train.columnx and Test.columnx with 3 binary features, one for each of {'a','b','c'}.  rows in test that have the value 'd' for columnx will have a value 0 for all three features.",
    "134605": "[quote=CPMP;134561]\r\n\r\nMichael, thanks.  As said elsewhere, if you follow this with one hot encoding then you just sdrop the values appearing in test that do not appear in train.\r\n\r\nI this example Train.columnx has the following values in it {'a','b','c'}. Test.columnx has the values {'a','c','d'}\r\n\r\nYou replace Train.columnx and Test.columnx with 3 binary features, one for each of {'a','b','c'}.  rows in test that have the value 'd' for columnx will have a value 0 for all three features.\r\n\r\n[/quote]\r\nSorry, the pandas factorize function was mentioned above and that label encodes the values.  One hot encoding would solve the problem.",
    "134733": "Michael,\r\n\r\nI think we are in agreement.  If you use pandas factorize before doing ohe then you definitely need to concatenate train and test before calling factorize.",
    "134739": "Hi CMPM, \r\nI tried to concatenate using pandas but apparently it seems that the data is too big for it . I get the following error.\r\n\r\n    File \"C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\tools\\merge.py\", line 835, in concat\r\n        return op.get_result()\r\n      File \"C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\tools\\merge.py\", line 1025, in get_result\r\n        concat_axis=self.axis, copy=self.copy)\r\n      File \"C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py\", line 4474, in concatenate_block_managers\r\n        for placement, join_units in concat_plan]\r\n      File \"C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py\", line 4567, in concatenate_join_units\r\n        empty_dtype, upcasted_na = get_empty_dtype_and_na(join_units)\r\n      File \"C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py\", line 4528, in get_empty_dtype_and_na\r\n        if unit.is_null:\r\n      File \"pandas\\src\\properties.pyx\", line 34, in pandas.lib.cache_readonly.__get__ (pandas\\lib.c:44357)\r\n      File \"C:\\Users\\Public\\Documents\\anaconda2\\lib\\site-packages\\pandas\\core\\internals.py\", line 4808, in is_null\r\n        for i in range(0, total_len, chunk_len):\r\n    OverflowError: Python int too large to convert to C long\r\n\r\n \r\n\r\n[quote=CPMP;134733]\r\n\r\nMichael,\r\n\r\nI think we are in agreement.  If you use pandas factorize before doing ohe then you definitely need to concatenate train and test before calling factorize.\r\n\r\n[/quote]",
    "134741": "Zohair,\r\n\r\nI could load all data in memory using pandas sparse data frames.  Call to_sparse() from your data frames to get sparse versions of them, then you may be able to concatenate them.\r\n\r\nFor this you need to use the latest pandas version (18.1 if I am not mistaken).  I see you use anaconda.  just type the following in an anaconda prompt on Windows, or in a shell window on linux or macosx:\r\n\r\n    conda update pandas\r\n\r\nthen restart your notebooks.\r\n\r\nMain issue with these sparse data frames is that code is young and has some limit.   I could not save them for instance.",
    "134803": "CMPM,\r\nI was able to concatenate but the thing is I think pandas factorize does not take sparse matrix as input so it needs to be converted to dense matrix again. Now that I am factorizing it, it's running for last 2 hours, no idea what state it is in. Any hints or any idea how much time it takes ?\r\n\r\n[quote=CPMP;134741]\r\n\r\nZohair,\r\n\r\nI could load all data in memory using pandas sparse data frames.  Call to_sparse() from your data frames to get sparse versions of them, then you may be able to concatenate them.\r\n\r\nFor this you need to use the latest pandas version (18.1 if I am not mistaken).  I see you use anaconda.  just type the following in an anaconda prompt on Windows, or in a shell window on linux or macosx:\r\n\r\n    conda update pandas\r\n\r\nthen restart your notebooks.\r\n\r\nMain issue with these sparse data frames is that code is young and has some limit.   I could not save them for instance.\r\n\r\n[/quote]",
    "138071": "[quote=JeffH;134218]\r\n\r\nThanks, this worked well.\r\n\r\nI had to use a slightly different syntax with MINGW in WIndows\r\n\r\n    sed -i.sav -e 's/T-//g' train_categorical.csv \r\n    sed -i.sav -e 's/T-//g' test_categorical.csv \r\n\r\nNotice, no space after the -i. I thought it wasn't doing anything until I realized it overwrote the original CSV.\r\n\r\n[/quote]\r\n\r\nI'm using sed (GNU sed) 4.2.2 on Linux and had to use this method.",
    "138094": "Hmm... ok, so it wasn't what I wanted. \r\n\r\nI decided to keep negative numbers and replace NaNs with 0s. Now I can load categorical data with read_csv as dtype=int64."
  },
  "source": "meta"
}