I'm new to sparse data formats and I'm a little confused on the actual implementation with regard to zeros. From what I'm reading on github and in the scipy documentation some formats support the retention of "explicit zeros" (I assuming this phrase refers to a zero data value vs. a NaN).
\n\nI'm using scipy's csr format, when I convert to a sparse matrix using that format it seems to retain all of the NaNs as data values. That is, when I convert the first 100 columns of the numeric data (train+test) I get 232,245,416 values (which seems to defeat the purpose of the sparse format), but if I fill all the NaN with 0 then convert, the sparse format retains 61,528,828 values. However, I know that there are 66,032,910 non-NaN values in the data (~4.5M are 0).
\n\nHow do I, or what format should I use, to retain the actual zero data-values when converting to a sparse format without making everything NaN=0 first?
\n\nThanks for any help!!
", "messages": [ { "id": "134977", "postDate": "09/09/2016 17:29:48", "content": "I'm new to sparse data formats and I'm a little confused on the actual implementation with regard to zeros. From what I'm reading on github and in the scipy documentation some formats support the retention of "explicit zeros" (I assuming this phrase refers to a zero data value vs. a NaN).
\n\nI'm using scipy's csr format, when I convert to a sparse matrix using that format it seems to retain all of the NaNs as data values. That is, when I convert the first 100 columns of the numeric data (train+test) I get 232,245,416 values (which seems to defeat the purpose of the sparse format), but if I fill all the NaN with 0 then convert, the sparse format retains 61,528,828 values. However, I know that there are 66,032,910 non-NaN values in the data (~4.5M are 0).
\n\nHow do I, or what format should I use, to retain the actual zero data-values when converting to a sparse format without making everything NaN=0 first?
\n\nThanks for any help!!
", "rawMarkdown": "I'm new to sparse data formats and I'm a little confused on the actual implementation with regard to zeros. From what I'm reading on github and in the scipy documentation some formats support the retention of \"explicit zeros\" (I assuming this phrase refers to a zero data value vs. a NaN). \r\n\r\nI'm using scipy's csr format, when I convert to a sparse matrix using that format it seems to retain all of the NaNs as data values. That is, when I convert the first 100 columns of the numeric data (train+test) I get 232,245,416 values (which seems to defeat the purpose of the sparse format), but if I fill all the NaN with 0 then convert, the sparse format retains 61,528,828 values. However, I know that there are 66,032,910 non-NaN values in the data (~4.5M are 0).\r\n\r\nHow do I, or what format should I use, to retain the actual zero data-values when converting to a sparse format without making everything NaN=0 first?\r\n\r\nThanks for any help!!", "votes": null }, { "id": "134981", "postDate": "09/09/2016 19:44:47", "content": "First of all, sparse matrices typically do not retain zeroes in memory (this is how it is "sparse"). Hence, if there is nothing stored in memory, it is assumed "zero".
\n\n\n\n\nwhat format should I use
\n
In Python, you have different types of matrices, the major three ones for general numeric data:
\n\nAll these four types have advantages and disadvantages. These following scenarios should help you understand what to use when:
\n\nIn short, anything "compressed" should not be used when you want to modify them (although you can still modify them). If you modify them, the whole sparse structure requires to be re-computed from scratch (which may or may not be what you want).
\n\n\n\n\nHow do I retain the actual zero data-values when converting to a sparse format without making everything NaN=0 first?
\n
I'll give you a quick example (I wrote what it should print in Python, not sure but it should be it - have no access to my laptop which crashed at home). This example will NOT work with non-compressed formats. You will need to explicit the zeroes in csc_matrix / csr_matrix before if you intend to use lil_matrix or another matrix.
\n\nEDIT: I noticed I could run an iPython notebook on Kaggle. https://www.kaggle.com/laurae2/bosch-production-line-performance/sparse-matrix-with-explicit-zeroes/
\n\nCorrect me if there is a faster method to impute explicit zeroes in a sparse matrix in Python without using a csc/csr format.
\n\nSimple example (the fake iPython console might have grammatical errors):
\n\n# Example: create a sparse matrix, check its (dense) content, modify it by adding explicit zeroes, check again its (dense) content\n\n# Lets load what we need for the example\nimport numpy as np\nfrom scipy.sparse import lil_matrix\nfrom scipy.sparse import csr_matrix\n\n# Lets create a sparse modifiable matrix (3 rows, 2 columns)\ncsr = csr_matrix(np.array([[0, 0],\n [2, 1],\n [np.nan, np.nan]]))\nOut[1]: csr\n<3x2 sparse matrix of type '<class 'numpy.float64'>'\nwith 4 stored elements in Compressed Sparse Row format>\n\n# Lets check the dense matrix\nIn [2]: csr .todense()\nOut[2]:\nmatrix([[0, 0],\n [2, 1],\n [nan, nan]])\n\n# Can we put explicit zeroes in the first row? Yes we can.\nIn[3]: csr[0, :] = [0, 0]\nblabla about some warning message about how expensive this operation is\n\n# What is inside the sparse matrix now? 6 elements instead of 4!\nIn[4]: csr\nOut[4]: \n<3x2 sparse matrix of type '<class 'numpy.float64'>'\nwith 6 stored elements in Compressed Sparse Row format>\n\n# Do we have what we had at the beginning? Yes!\nIn [5]: csr.todense()\nOut[5]:\nmatrix([[0, 0],\n [2, 1],\n [nan, nan]])\n\n# Indirect conversion to lil_matrix (if you want to make many modifications) is working as intended\nIn [6]: csr.tolil()\nOut [6]:\n<3x2 sparse matrix of type '<class 'numpy.float64'>'\n with 6 stored elements in Linked List format>\n\n# Lets try directly with a lil_matrix!\nIn [7]: lil = lil_matrix(np.array([[0, 0],\n [2, 1],\n [np.nan, np.nan]]))\n\n# Setting up explicit zeroes...\nIn [8]: lil[0, :] = [0, 0]\n\n# NOPE! Specifying explicit zeroes in lil_matrix will NOT work (along with any non-compressed formats)\nIn [9]: lil\nOut[9]:\n<3x2 sparse matrix of type '<class 'numpy.float64'>'\n with 4 stored elements in Linked List format>\n\n# Maybe because the row is full of zeroes...?\nIn [10]: lil[0, :] = [0, 1]\n\n# NOPE! Only 1 element was added, the zero was discarded\nIn [11]: lil\nOut[11]:\n<3x2 sparse matrix of type '<class 'numpy.float64'>'\n with 5 stored elements in Linked List format>\n\n# Only the zero is not explicit!\nIn [12]: lil.tocsr()\nOut[12]:\n<3x2 sparse matrix of type '<class 'numpy.float64'>'\n with 5 stored elements in Compressed Sparse Row format>\n\n# Lets check directly if it is true with the number of non zero entries function: yep, the XYZ in "with [XYZ] stored elements" is the .nnz value\nIn [13]: lil.tocsr().nnz\nOut[13]: 5\nTherefore, how do you make explicit zeroes in a sparse matrix?:
\n\nI would be happy to hear another method which does not use csc/csr matrices but can have explicit zeroes. They are so slow when modifying stuff inside.
", "rawMarkdown": "First of all, sparse matrices typically do not retain zeroes in memory (this is how it is \"sparse\"). Hence, if there is nothing stored in memory, it is assumed \"zero\".\r\n\r\n> what format should I use\r\n\r\nIn Python, you have different types of matrices, the major three ones for general numeric data:\r\n\r\n* scipy.sparse.csc_matrix: compressed sparse column (column-based)\r\n* scipy.sparse.csr_matrix: compressed sparse row (row-based)\r\n* scipy.sparse.lil_matrix: list of lists (row-based)\r\n* (The fourth one often used is scipy.sparse.dok_matrix: dictionary of keys\r\n\r\nAll these four types have advantages and disadvantages. These following scenarios should help you understand what to use when:\r\n\r\n* Do you want to change often the internal structure of your sparse matrix? => lil_matrix or dok_matrix\r\n* Do you want to do fast computation? => csc_matrix (on columns) / csr_matrix (on rows)\r\n\r\nIn short, anything \"compressed\" should not be used when you want to modify them (although you can still modify them). If you modify them, the whole sparse structure requires to be re-computed from scratch (which may or may not be what you want).\r\n\r\n> How do I retain the actual zero data-values when converting to a sparse format without making everything NaN=0 first?\r\n\r\nI'll give you a quick example (I wrote what it should print in Python, not sure but it should be it - have no access to my laptop which crashed at home). This example will NOT work with non-compressed formats. You will need to explicit the zeroes in csc_matrix / csr_matrix before if you intend to use lil_matrix or another matrix.\r\n\r\nEDIT: I noticed I could run an iPython notebook on Kaggle. https://www.kaggle.com/laurae2/bosch-production-line-performance/sparse-matrix-with-explicit-zeroes/\r\n\r\nCorrect me if there is a faster method to impute explicit zeroes in a sparse matrix in Python without using a csc/csr format.\r\n\r\n\r\nSimple example (the fake iPython console might have grammatical errors):\r\n\r\n # Example: create a sparse matrix, check its (dense) content, modify it by adding explicit zeroes, check again its (dense) content\r\n \r\n # Lets load what we need for the example\r\n import numpy as np\r\n from scipy.sparse import lil_matrix\r\n from scipy.sparse import csr_matrix\r\n \r\n # Lets create a sparse modifiable matrix (3 rows, 2 columns)\r\n csr = csr_matrix(np.array([[0, 0],\r\n [2, 1],\r\n [np.nan, np.nan]]))\r\n Out[1]: csr\r\n <3x2 sparse matrix of type 'Thanks Laurae (really! You've been answering a few of my questions lately. I appreciate it!)
\n\nI was afraid that was the answer. I wonder what this will do in various models (XGBoost, NNs, etc.) where every NaN is essentially set equal to 0 when sparse data is used. Does it have an impact? And how big of an impact?
", "rawMarkdown": "Thanks Laurae (really! You've been answering a few of my questions lately. I appreciate it!)\r\n\r\nI was afraid that was the answer. I wonder what this will do in various models (XGBoost, NNs, etc.) where every NaN is essentially set equal to 0 when sparse data is used. Does it have an impact? And how big of an impact?", "votes": null }, { "id": "135070", "postDate": "09/10/2016 18:37:58", "content": "[quote=Michael Maguire;134994]
\n\nI was afraid that was the answer.
\n\n[/quote]
\n\nThere may be other (easier) solutions I am not aware of.
\n\n[quote=Michael Maguire;134994]
\n\nI wonder what this will do in various models (XGBoost, NNs, etc.) where every NaN is essentially set equal to 0 when sparse data is used. Does it have an impact? And how big of an impact?
\n\n[/quote]
\n\nDepends on the implementation of the algorithm and how they are supposed to handle sparse inputs. In the case of xgboost, you can define explicitly what is missing and what is not missing. Thus, it gets easy for xgboost (xgboost always tries to split missing values in the two directions of a node, anyway).
\n\nAs for NNs etc, you have to check the implementation if it mentions anything about how missing sparse values are handled. If there is no documentation about that, you have to delve in the whole code source to find what happen to sparse values. Hence, the impact can be from: nothing to super large.
\n\nFor most dimensionality reduction techniques (like truncated SVDs), sparse values are supposed to not be used to account for the loss objective (unless the algorithm is "badly" implemented). The impact can be severe if sparse missing inputs are considered zeroes when they should not.
", "rawMarkdown": "[quote=Michael Maguire;134994]\r\n\r\nI was afraid that was the answer.\r\n\r\n[/quote]\r\n\r\nThere may be other (easier) solutions I am not aware of.\r\n\r\n[quote=Michael Maguire;134994]\r\n\r\nI wonder what this will do in various models (XGBoost, NNs, etc.) where every NaN is essentially set equal to 0 when sparse data is used. Does it have an impact? And how big of an impact?\r\n\r\n[/quote]\r\n\r\nDepends on the implementation of the algorithm and how they are supposed to handle sparse inputs. In the case of xgboost, you can define explicitly what is missing and what is not missing. Thus, it gets easy for xgboost (xgboost always tries to split missing values in the two directions of a node, anyway).\r\n\r\nAs for NNs etc, you have to check the implementation if it mentions anything about how missing sparse values are handled. If there is no documentation about that, you have to delve in the whole code source to find what happen to sparse values. Hence, the impact can be from: nothing to super large.\r\n\r\nFor most dimensionality reduction techniques (like truncated SVDs), sparse values are supposed to not be used to account for the loss objective (unless the algorithm is \"badly\" implemented). The impact can be severe if sparse missing inputs are considered zeroes when they should not.", "votes": null } ], "comments": [ { "id": 134981, "author_name": "laurae2", "author_url": "", "post_date": "09/09/2016 19:44:47", "content": "First of all, sparse matrices typically do not retain zeroes in memory (this is how it is "sparse"). Hence, if there is nothing stored in memory, it is assumed "zero".
\n\n\n\n\nwhat format should I use
\n
In Python, you have different types of matrices, the major three ones for general numeric data:
\n\nAll these four types have advantages and disadvantages. These following scenarios should help you understand what to use when:
\n\nIn short, anything "compressed" should not be used when you want to modify them (although you can still modify them). If you modify them, the whole sparse structure requires to be re-computed from scratch (which may or may not be what you want).
\n\n\n\n\nHow do I retain the actual zero data-values when converting to a sparse format without making everything NaN=0 first?
\n
I'll give you a quick example (I wrote what it should print in Python, not sure but it should be it - have no access to my laptop which crashed at home). This example will NOT work with non-compressed formats. You will need to explicit the zeroes in csc_matrix / csr_matrix before if you intend to use lil_matrix or another matrix.
\n\nEDIT: I noticed I could run an iPython notebook on Kaggle. https://www.kaggle.com/laurae2/bosch-production-line-performance/sparse-matrix-with-explicit-zeroes/
\n\nCorrect me if there is a faster method to impute explicit zeroes in a sparse matrix in Python without using a csc/csr format.
\n\nSimple example (the fake iPython console might have grammatical errors):
\n\n# Example: create a sparse matrix, check its (dense) content, modify it by adding explicit zeroes, check again its (dense) content\n\n# Lets load what we need for the example\nimport numpy as np\nfrom scipy.sparse import lil_matrix\nfrom scipy.sparse import csr_matrix\n\n# Lets create a sparse modifiable matrix (3 rows, 2 columns)\ncsr = csr_matrix(np.array([[0, 0],\n [2, 1],\n [np.nan, np.nan]]))\nOut[1]: csr\n<3x2 sparse matrix of type '<class 'numpy.float64'>'\nwith 4 stored elements in Compressed Sparse Row format>\n\n# Lets check the dense matrix\nIn [2]: csr .todense()\nOut[2]:\nmatrix([[0, 0],\n [2, 1],\n [nan, nan]])\n\n# Can we put explicit zeroes in the first row? Yes we can.\nIn[3]: csr[0, :] = [0, 0]\nblabla about some warning message about how expensive this operation is\n\n# What is inside the sparse matrix now? 6 elements instead of 4!\nIn[4]: csr\nOut[4]: \n<3x2 sparse matrix of type '<class 'numpy.float64'>'\nwith 6 stored elements in Compressed Sparse Row format>\n\n# Do we have what we had at the beginning? Yes!\nIn [5]: csr.todense()\nOut[5]:\nmatrix([[0, 0],\n [2, 1],\n [nan, nan]])\n\n# Indirect conversion to lil_matrix (if you want to make many modifications) is working as intended\nIn [6]: csr.tolil()\nOut [6]:\n<3x2 sparse matrix of type '<class 'numpy.float64'>'\n with 6 stored elements in Linked List format>\n\n# Lets try directly with a lil_matrix!\nIn [7]: lil = lil_matrix(np.array([[0, 0],\n [2, 1],\n [np.nan, np.nan]]))\n\n# Setting up explicit zeroes...\nIn [8]: lil[0, :] = [0, 0]\n\n# NOPE! Specifying explicit zeroes in lil_matrix will NOT work (along with any non-compressed formats)\nIn [9]: lil\nOut[9]:\n<3x2 sparse matrix of type '<class 'numpy.float64'>'\n with 4 stored elements in Linked List format>\n\n# Maybe because the row is full of zeroes...?\nIn [10]: lil[0, :] = [0, 1]\n\n# NOPE! Only 1 element was added, the zero was discarded\nIn [11]: lil\nOut[11]:\n<3x2 sparse matrix of type '<class 'numpy.float64'>'\n with 5 stored elements in Linked List format>\n\n# Only the zero is not explicit!\nIn [12]: lil.tocsr()\nOut[12]:\n<3x2 sparse matrix of type '<class 'numpy.float64'>'\n with 5 stored elements in Compressed Sparse Row format>\n\n# Lets check directly if it is true with the number of non zero entries function: yep, the XYZ in "with [XYZ] stored elements" is the .nnz value\nIn [13]: lil.tocsr().nnz\nOut[13]: 5\nTherefore, how do you make explicit zeroes in a sparse matrix?:
\n\nI would be happy to hear another method which does not use csc/csr matrices but can have explicit zeroes. They are so slow when modifying stuff inside.
", "votes": null, "replies": [] }, { "id": 134994, "author_name": "thawatt", "author_url": "", "post_date": "09/09/2016 21:51:11", "content": "Thanks Laurae (really! You've been answering a few of my questions lately. I appreciate it!)
\n\nI was afraid that was the answer. I wonder what this will do in various models (XGBoost, NNs, etc.) where every NaN is essentially set equal to 0 when sparse data is used. Does it have an impact? And how big of an impact?
", "votes": null, "replies": [] }, { "id": 135070, "author_name": "laurae2", "author_url": "", "post_date": "09/10/2016 18:37:58", "content": "[quote=Michael Maguire;134994]
\n\nI was afraid that was the answer.
\n\n[/quote]
\n\nThere may be other (easier) solutions I am not aware of.
\n\n[quote=Michael Maguire;134994]
\n\nI wonder what this will do in various models (XGBoost, NNs, etc.) where every NaN is essentially set equal to 0 when sparse data is used. Does it have an impact? And how big of an impact?
\n\n[/quote]
\n\nDepends on the implementation of the algorithm and how they are supposed to handle sparse inputs. In the case of xgboost, you can define explicitly what is missing and what is not missing. Thus, it gets easy for xgboost (xgboost always tries to split missing values in the two directions of a node, anyway).
\n\nAs for NNs etc, you have to check the implementation if it mentions anything about how missing sparse values are handled. If there is no documentation about that, you have to delve in the whole code source to find what happen to sparse values. Hence, the impact can be from: nothing to super large.
\n\nFor most dimensionality reduction techniques (like truncated SVDs), sparse values are supposed to not be used to account for the loss objective (unless the algorithm is "badly" implemented). The impact can be severe if sparse missing inputs are considered zeroes when they should not.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "134977": "I'm new to sparse data formats and I'm a little confused on the actual implementation with regard to zeros. From what I'm reading on github and in the scipy documentation some formats support the retention of \"explicit zeros\" (I assuming this phrase refers to a zero data value vs. a NaN). \r\n\r\nI'm using scipy's csr format, when I convert to a sparse matrix using that format it seems to retain all of the NaNs as data values. That is, when I convert the first 100 columns of the numeric data (train+test) I get 232,245,416 values (which seems to defeat the purpose of the sparse format), but if I fill all the NaN with 0 then convert, the sparse format retains 61,528,828 values. However, I know that there are 66,032,910 non-NaN values in the data (~4.5M are 0).\r\n\r\nHow do I, or what format should I use, to retain the actual zero data-values when converting to a sparse format without making everything NaN=0 first?\r\n\r\nThanks for any help!!", "134981": "First of all, sparse matrices typically do not retain zeroes in memory (this is how it is \"sparse\"). Hence, if there is nothing stored in memory, it is assumed \"zero\".\r\n\r\n> what format should I use\r\n\r\nIn Python, you have different types of matrices, the major three ones for general numeric data:\r\n\r\n* scipy.sparse.csc_matrix: compressed sparse column (column-based)\r\n* scipy.sparse.csr_matrix: compressed sparse row (row-based)\r\n* scipy.sparse.lil_matrix: list of lists (row-based)\r\n* (The fourth one often used is scipy.sparse.dok_matrix: dictionary of keys\r\n\r\nAll these four types have advantages and disadvantages. These following scenarios should help you understand what to use when:\r\n\r\n* Do you want to change often the internal structure of your sparse matrix? => lil_matrix or dok_matrix\r\n* Do you want to do fast computation? => csc_matrix (on columns) / csr_matrix (on rows)\r\n\r\nIn short, anything \"compressed\" should not be used when you want to modify them (although you can still modify them). If you modify them, the whole sparse structure requires to be re-computed from scratch (which may or may not be what you want).\r\n\r\n> How do I retain the actual zero data-values when converting to a sparse format without making everything NaN=0 first?\r\n\r\nI'll give you a quick example (I wrote what it should print in Python, not sure but it should be it - have no access to my laptop which crashed at home). This example will NOT work with non-compressed formats. You will need to explicit the zeroes in csc_matrix / csr_matrix before if you intend to use lil_matrix or another matrix.\r\n\r\nEDIT: I noticed I could run an iPython notebook on Kaggle. https://www.kaggle.com/laurae2/bosch-production-line-performance/sparse-matrix-with-explicit-zeroes/\r\n\r\nCorrect me if there is a faster method to impute explicit zeroes in a sparse matrix in Python without using a csc/csr format.\r\n\r\n\r\nSimple example (the fake iPython console might have grammatical errors):\r\n\r\n # Example: create a sparse matrix, check its (dense) content, modify it by adding explicit zeroes, check again its (dense) content\r\n \r\n # Lets load what we need for the example\r\n import numpy as np\r\n from scipy.sparse import lil_matrix\r\n from scipy.sparse import csr_matrix\r\n \r\n # Lets create a sparse modifiable matrix (3 rows, 2 columns)\r\n csr = csr_matrix(np.array([[0, 0],\r\n [2, 1],\r\n [np.nan, np.nan]]))\r\n Out[1]: csr\r\n <3x2 sparse matrix of type '