{
  "id": 116608,
  "title": "Maybe a stupid question about sparse matrix...",
  "url": "/competitions/ashrae-energy-prediction/discussion/116608",
  "author_name": "",
  "post_date": "2019-11-10T09:20:27.045575700Z",
  "votes": 2,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I want to try a linear model. When I try to process the data with OneHotEncoding for some categorical features, I found it invalid to concat a sparse matrix with a dense one.</p>\n\n<p><code>OHE = OneHotEncoding()</code>\n<code>ohe_feature = OHE.fit_transform(df[categorical_feature])</code>\n<code>df = pd.concat([df, ohe_feature])</code>\n<code>df.drop(categorical_feature, axis=1, inplace=1)</code></p>\n\n<p>It raised an error</p>\n\n<p><code>TypeError: cannot concatenate object of type '&lt;class 'scipy.sparse.csr.csr_matrix'&gt;'; only Series and DataFrame objs are valid</code></p>\n\n<p>Ummm…so how can I concatenate a sparse matrix with a dense one? Perhaps transform my numerical matrix into a sparse one?</p>\n\n<p>FEEL SORRY to use so less a powerful linear model but always LightGBM😩</p>",
  "messages": [
    {
      "id": "669637",
      "postDate": "11/10/2019 09:20:27",
      "content": "<p>I want to try a linear model. When I try to process the data with OneHotEncoding for some categorical features, I found it invalid to concat a sparse matrix with a dense one.</p>\n\n<p><code>OHE = OneHotEncoding()</code>\n<code>ohe_feature = OHE.fit_transform(df[categorical_feature])</code>\n<code>df = pd.concat([df, ohe_feature])</code>\n<code>df.drop(categorical_feature, axis=1, inplace=1)</code></p>\n\n<p>It raised an error</p>\n\n<p><code>TypeError: cannot concatenate object of type '&lt;class 'scipy.sparse.csr.csr_matrix'&gt;'; only Series and DataFrame objs are valid</code></p>\n\n<p>Ummm…so how can I concatenate a sparse matrix with a dense one? Perhaps transform my numerical matrix into a sparse one?</p>\n\n<p>FEEL SORRY to use so less a powerful linear model but always LightGBM😩</p>",
      "rawMarkdown": "I want to try a linear model. When I try to process the data with OneHotEncoding for some categorical features, I found it invalid to concat a sparse matrix with a dense one.\n\n`OHE = OneHotEncoding()`\n`ohe_feature = OHE.fit_transform(df[categorical_feature])`\n`df = pd.concat([df, ohe_feature])`\n`df.drop(categorical_feature, axis=1, inplace=1)`\n\nIt raised an error\n\n`TypeError: cannot concatenate object of type '",
      "votes": null
    },
    {
      "id": "669651",
      "postDate": "11/10/2019 09:43:05",
      "content": "<p>Oh..why can't I see all I published...<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1713173%2Fe82a112078a0a6d7a41abd1038566588%2F_20191110174117.png?generation=1573378967752755&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "Oh..why can't I see all I published...![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1713173%2Fe82a112078a0a6d7a41abd1038566588%2F_20191110174117.png?generation=1573378967752755&amp;alt=media)",
      "votes": null
    },
    {
      "id": "669662",
      "postDate": "11/10/2019 09:58:32",
      "content": "<p>Try casting the <code>csr_matrix</code> to <code>DataFrame</code>?</p>",
      "rawMarkdown": "Try casting the `csr_matrix` to `DataFrame`?",
      "votes": null
    },
    {
      "id": "670077",
      "postDate": "11/11/2019 01:03:45",
      "content": "<p>The building_id feature has so many kinds...it may result in a MemoryError...</p>",
      "rawMarkdown": "The building_id feature has so many kinds...it may result in a MemoryError...",
      "votes": null
    },
    {
      "id": "670194",
      "postDate": "11/11/2019 06:48:26",
      "content": "<p>You will be able to do it if you assign <code>np.uint8</code> for every column dtype</p>",
      "rawMarkdown": "You will be able to do it if you assign `np.uint8` for every column dtype",
      "votes": null
    },
    {
      "id": "670265",
      "postDate": "11/11/2019 08:37:48",
      "content": "<p>Thank you! I will try. And I come up with a new idea to do sampling then convert to a DataFrame to avoid MemoryError🙂I will try to implement later. </p>",
      "rawMarkdown": "Thank you! I will try. And I come up with a new idea to do sampling then convert to a DataFrame to avoid MemoryError🙂I will try to implement later.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 669651,
      "author_name": "uestcblackstar",
      "author_url": "",
      "post_date": "11/10/2019 09:43:05",
      "content": "<p>Oh..why can't I see all I published...<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1713173%2Fe82a112078a0a6d7a41abd1038566588%2F_20191110174117.png?generation=1573378967752755&amp;alt=media\" alt=\"\"></p>",
      "votes": null,
      "replies": [
        {
          "id": 669662,
          "author_name": "gunesevitan",
          "author_url": "",
          "post_date": "11/10/2019 09:58:32",
          "content": "<p>Try casting the <code>csr_matrix</code> to <code>DataFrame</code>?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 670077,
          "author_name": "uestcblackstar",
          "author_url": "",
          "post_date": "11/11/2019 01:03:45",
          "content": "<p>The building_id feature has so many kinds...it may result in a MemoryError...</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 670194,
          "author_name": "gunesevitan",
          "author_url": "",
          "post_date": "11/11/2019 06:48:26",
          "content": "<p>You will be able to do it if you assign <code>np.uint8</code> for every column dtype</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 670265,
          "author_name": "uestcblackstar",
          "author_url": "",
          "post_date": "11/11/2019 08:37:48",
          "content": "<p>Thank you! I will try. And I come up with a new idea to do sampling then convert to a DataFrame to avoid MemoryError🙂I will try to implement later. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "669637": "I want to try a linear model. When I try to process the data with OneHotEncoding for some categorical features, I found it invalid to concat a sparse matrix with a dense one.\n\n`OHE = OneHotEncoding()`\n`ohe_feature = OHE.fit_transform(df[categorical_feature])`\n`df = pd.concat([df, ohe_feature])`\n`df.drop(categorical_feature, axis=1, inplace=1)`\n\nIt raised an error\n\n`TypeError: cannot concatenate object of type '",
    "669651": "Oh..why can't I see all I published...![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1713173%2Fe82a112078a0a6d7a41abd1038566588%2F_20191110174117.png?generation=1573378967752755&amp;alt=media)",
    "669662": "Try casting the `csr_matrix` to `DataFrame`?",
    "670077": "The building_id feature has so many kinds...it may result in a MemoryError...",
    "670194": "You will be able to do it if you assign `np.uint8` for every column dtype",
    "670265": "Thank you! I will try. And I come up with a new idea to do sampling then convert to a DataFrame to avoid MemoryError🙂I will try to implement later."
  },
  "source": "meta"
}