will there be no kernel for this competition?
", "messages": [ { "id": 131583, "postDate": "2016-08-18T11:02:30.630Z", "content": "will there be no kernel for this competition?
", "rawMarkdown": "will there be no kernel for this competition?", "votes": 5 }, { "id": 134853, "postDate": "2016-09-08T20:58:42.693Z", "content": "Hi all,
\n\nWe've received many requests for Kernels and have decided we're going to give it a shot on this competition. Be forewarned that it might be a little frustrating to work on data of this size with Kernels' limitations, particularly if our (shared) workers start to hit a lot out-of-memory errors. Nonetheless, we're excited by your enthusiasm and curious to see what's possible to achieve wearing 8GB digital handcuffs. Have fun and don't try loading everything at once!
", "rawMarkdown": "Hi all,\r\n\r\nWe've received many requests for Kernels and have decided we're going to give it a shot on this competition. Be forewarned that it might be a little frustrating to work on data of this size with Kernels' limitations, particularly if our (shared) workers start to hit a lot out-of-memory errors. Nonetheless, we're excited by your enthusiasm and curious to see what's possible to achieve wearing 8GB digital handcuffs. Have fun and don't try loading everything at once!", "votes": 6 }, { "id": 132007, "postDate": "2016-08-21T14:36:45.203Z", "content": "[quote=Prerit Anwekar;132006]
\n\n@jeffd23 How are we gonna do dimensionality reduction on a sparse matrix? Isn't it futile, we will lose information..?
\n\n[/quote]
\n\nUse a package/library that can perform a truncated SVD on a sparse matrix. You almost always lose information when using dimensionality reduction techniques.
\n\nOn a sparse matrix, the loss on sparse values is equal to 0 because they do not need to be interpreted unlike dense matrices where a 0 is 0. It takes only about 100 seconds to perform a truncated SVD with 50 columns on the numerical train dataset with 1e-10 loss, under 8GB RAM usage (for both the train set and SVD computation in memory).
", "rawMarkdown": "[quote=Prerit Anwekar;132006]\r\n\r\n@jeffd23 How are we gonna do dimensionality reduction on a sparse matrix? Isn't it futile, we will lose information..?\r\n\r\n[/quote]\r\n\r\nUse a package/library that can perform a truncated SVD on a sparse matrix. You *almost always* lose information when using dimensionality reduction techniques.\r\n\r\nOn a sparse matrix, the loss on sparse values is equal to 0 because they do not need to be interpreted unlike dense matrices where a 0 is 0. It takes only about 100 seconds to perform a truncated SVD with 50 columns on the numerical train dataset with 1e-10 loss, under 8GB RAM usage (for both the train set and SVD computation in memory).", "votes": 3 }, { "id": 131792, "postDate": "2016-08-19T17:31:26.973Z", "content": "This is what happens when I just load the train numeric dataset.
\n\n> gc()\n used (Mb) gc trigger (Mb) max used (Mb)\nNcells 614931 32.9 1168576 62.5 940480 50.3\nVcells 1058373 8.1 2390182 18.3 2226183 17.0\n> train_numeric <- fread("train_numeric.csv", header = TRUE, sep = ",", stringsAsFactors = FALSE, colClasses = rep("numeric", 970))\nRead 1183747 rows and 970 (of 970) columns from 1.993 GB file in 00:02:29\n> gc()\n used (Mb) gc trigger (Mb) max used (Mb)\nNcells 617120 33.0 1168576 62.5 947860 50.7\nVcells 1149297482 8768.5 1568457548 11966.4 1149391062 8769.2\nOver 8GB, is clearly too large (limit is at something like 8.5 billion bytes at Kaggle for max usage).
\n\nFor a 2GB CSV, it is clearly small, but sparsity has its role. I wonder if Kaggle would still open the usage of scripts, so we can work on the data in a script. Users would need a main tutorial script where it shows how to subsample / select columns to load though (so the first thing that does not happen is "memory limit" error).
\n\nIn case someone needs it: those who cannot load data into R to create a sparse matrix (or cannot convert to sparse directly) due to memory usage, use the following:
\n\ntrain_numeric <- fread(input = "train_numeric.csv",\n select = 2:51\n header = TRUE,\n sep = ",",\n stringsAsFactors = FALSE,\n colClasses = rep("numeric", 970),\n data.table = FALSE)\nTransform it into a sparse matrix, keep loading more and more data (adjust columns selected until you get to 970 - id was intentionally removed here as it is 1; - the 970th is the label "Outcome") while column binding all you load into the sparse matrix. Then, you can save it locally (RDS) for fast load. It fits for a 4GB laptop. Train categorical is about 1GB when sparse.
", "rawMarkdown": "This is what happens when I just load the train numeric dataset.\r\n\r\n > gc()\r\n used (Mb) gc trigger (Mb) max used (Mb)\r\n Ncells 614931 32.9 1168576 62.5 940480 50.3\r\n Vcells 1058373 8.1 2390182 18.3 2226183 17.0\r\n > train_numeric <- fread(\"train_numeric.csv\", header = TRUE, sep = \",\", stringsAsFactors = FALSE, colClasses = rep(\"numeric\", 970))\r\n Read 1183747 rows and 970 (of 970) columns from 1.993 GB file in 00:02:29\r\n > gc()\r\n used (Mb) gc trigger (Mb) max used (Mb)\r\n Ncells 617120 33.0 1168576 62.5 947860 50.7\r\n Vcells 1149297482 8768.5 1568457548 11966.4 1149391062 8769.2\r\n\r\nOver 8GB, is clearly too large (limit is at something like 8.5 billion bytes at Kaggle for max usage).\r\n\r\nFor a 2GB CSV, it is clearly small, but sparsity has its role. I wonder if Kaggle would still open the usage of scripts, so we can work on the data in a script. *Users would need a main tutorial script where it shows how to subsample / select columns to load though (so the first thing that does not happen is \"memory limit\" error).*\r\n\r\nIn case someone needs it: those who cannot load data into R to create a sparse matrix (or cannot convert to sparse directly) due to memory usage, use the following:\r\n\r\n train_numeric <- fread(input = \"train_numeric.csv\",\r\n select = 2:51\r\n header = TRUE,\r\n sep = \",\",\r\n stringsAsFactors = FALSE,\r\n colClasses = rep(\"numeric\", 970),\r\n data.table = FALSE)\r\n\r\nTransform it into a sparse matrix, keep loading more and more data (adjust columns selected until you get to 970 - id was intentionally removed here as it is 1; - the 970th is the label \"Outcome\") while column binding all you load into the sparse matrix. Then, you can save it locally (RDS) for fast load. It fits for a 4GB laptop. Train categorical is about 1GB when sparse.", "votes": 1 }, { "id": 131767, "postDate": "2016-08-19T13:52:15.847Z", "content": "maybe it is because the dataset is too big.
", "rawMarkdown": "maybe it is because the dataset is too big.", "votes": 1 }, { "id": 131948, "postDate": "2016-08-20T21:15:40.927Z", "content": "[quote=jeffd23;131815]
\n\nLoading the data easily maxed out my 32GB of RAM. \n[/quote]
\n\nFYI . . . also maxed out my 64GB of RAM (just blindly reading the data). Of course, there are tricks, as have been mentioned in the forum.
", "rawMarkdown": "[quote=jeffd23;131815]\r\n\r\nLoading the data easily maxed out my 32GB of RAM. \r\n[/quote]\r\n\r\nFYI . . . also maxed out my 64GB of RAM (just blindly reading the data). Of course, there are tricks, as have been mentioned in the forum.", "votes": 2 }, { "id": 131815, "postDate": "2016-08-19T21:29:35.377Z", "content": "Loading the data easily maxed out my 32GB of RAM. I believe Kaggle's kernel docker image has a much lower memory limit. For python users, loading the data in chunks with Pandas, then experimenting with dimensionality reduction is one option...
\n\nChoosing 50 random columns from the numeric set earned a leaderboard score of ~0.14.
\n\ndef read_in_chunks(file_path):\n chunksize = 25000\n chunks = []\n for chunk in pd.read_csv(file_path, chunksize=chunksize):\n # do some dimensionality reduction...\n chunks.append(chunk)\n reduced_data = pd.concat(chunks, axis=0)\n return reduced_data\n@jeffd23: To the best of my knowledge, TruncatedSVD works fine on sparse matrices...
", "rawMarkdown": "@jeffd23: To the best of my knowledge, TruncatedSVD works fine on sparse matrices..." }, { "id": 132011, "postDate": "2016-08-21T17:15:58.997Z", "content": "\n\n\n@jeffd23 How are we gonna do dimensionality reduction on a sparse\n matrix? Isn't it futile, we will lose information..?
\n
That is definitely true for PCA , which requires dense inputs.
\n\nThere are dimensionality reduction techniques for sparse matrices, although I do not have any experience with them, such as SparsePCA in Sckit-learn. Also, you can try to manually isolate the best features - not optimal, but it gets the data down to a manageable size for some initial exporation.
", "rawMarkdown": "> @jeffd23 How are we gonna do dimensionality reduction on a sparse\r\n> matrix? Isn't it futile, we will lose information..?\r\n\r\nThat is definitely true for PCA , which requires dense inputs. \r\n\r\nThere are dimensionality reduction techniques for sparse matrices, although I do not have any experience with them, such as SparsePCA in Sckit-learn. Also, you can try to manually isolate the best features - not optimal, but it gets the data down to a manageable size for some initial exporation. \r\n" }, { "id": 132006, "postDate": "2016-08-21T14:29:54.827Z", "content": "@jeffd23 How are we gonna do dimensionality reduction on a sparse matrix? Isn't it futile, we will lose information..?
", "rawMarkdown": "@jeffd23 How are we gonna do dimensionality reduction on a sparse matrix? Isn't it futile, we will lose information..?" }, { "id": 131785, "postDate": "2016-08-19T16:44:32.010Z", "content": "[quote=junzi;131767]
\n\nmaybe it is because the dataset is too big.
\n\n[/quote]
\n\nThat's what I thought too. But the "Painter by Numbers" competition has a train dataset that is almost 2.5 times bigger, and yet they allow kernels there. It especially doesn't make sense since some of the simplest models can easily fit in Kaggle allocated memory.
", "rawMarkdown": "[quote=junzi;131767]\r\n\r\nmaybe it is because the dataset is too big.\r\n\r\n[/quote]\r\n\r\n\r\nThat's what I thought too. But the \"Painter by Numbers\" competition has a train dataset that is almost 2.5 times bigger, and yet they allow kernels there. It especially doesn't make sense since some of the simplest models can easily fit in Kaggle allocated memory. " }, { "id": 133529, "postDate": "2016-09-02T18:27:28.897Z", "rawMarkdown": "", "isDeleted": true }, { "id": 133527, "postDate": "2016-09-02T18:26:26.403Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 134853, "author_name": "Will Cukierski", "author_url": "", "post_date": "2016-09-08T20:58:42.693000", "content": "Hi all,
\n\nWe've received many requests for Kernels and have decided we're going to give it a shot on this competition. Be forewarned that it might be a little frustrating to work on data of this size with Kernels' limitations, particularly if our (shared) workers start to hit a lot out-of-memory errors. Nonetheless, we're excited by your enthusiasm and curious to see what's possible to achieve wearing 8GB digital handcuffs. Have fun and don't try loading everything at once!
", "votes": 6, "replies": [] }, { "id": 132007, "author_name": "Laurae", "author_url": "", "post_date": "2016-08-21T14:36:45.203000", "content": "[quote=Prerit Anwekar;132006]
\n\n@jeffd23 How are we gonna do dimensionality reduction on a sparse matrix? Isn't it futile, we will lose information..?
\n\n[/quote]
\n\nUse a package/library that can perform a truncated SVD on a sparse matrix. You almost always lose information when using dimensionality reduction techniques.
\n\nOn a sparse matrix, the loss on sparse values is equal to 0 because they do not need to be interpreted unlike dense matrices where a 0 is 0. It takes only about 100 seconds to perform a truncated SVD with 50 columns on the numerical train dataset with 1e-10 loss, under 8GB RAM usage (for both the train set and SVD computation in memory).
", "votes": 3, "replies": [] }, { "id": 131792, "author_name": "Laurae", "author_url": "", "post_date": "2016-08-19T17:31:26.973000", "content": "This is what happens when I just load the train numeric dataset.
\n\n> gc()\n used (Mb) gc trigger (Mb) max used (Mb)\nNcells 614931 32.9 1168576 62.5 940480 50.3\nVcells 1058373 8.1 2390182 18.3 2226183 17.0\n> train_numeric <- fread("train_numeric.csv", header = TRUE, sep = ",", stringsAsFactors = FALSE, colClasses = rep("numeric", 970))\nRead 1183747 rows and 970 (of 970) columns from 1.993 GB file in 00:02:29\n> gc()\n used (Mb) gc trigger (Mb) max used (Mb)\nNcells 617120 33.0 1168576 62.5 947860 50.7\nVcells 1149297482 8768.5 1568457548 11966.4 1149391062 8769.2\nOver 8GB, is clearly too large (limit is at something like 8.5 billion bytes at Kaggle for max usage).
\n\nFor a 2GB CSV, it is clearly small, but sparsity has its role. I wonder if Kaggle would still open the usage of scripts, so we can work on the data in a script. Users would need a main tutorial script where it shows how to subsample / select columns to load though (so the first thing that does not happen is "memory limit" error).
\n\nIn case someone needs it: those who cannot load data into R to create a sparse matrix (or cannot convert to sparse directly) due to memory usage, use the following:
\n\ntrain_numeric <- fread(input = "train_numeric.csv",\n select = 2:51\n header = TRUE,\n sep = ",",\n stringsAsFactors = FALSE,\n colClasses = rep("numeric", 970),\n data.table = FALSE)\nTransform it into a sparse matrix, keep loading more and more data (adjust columns selected until you get to 970 - id was intentionally removed here as it is 1; - the 970th is the label "Outcome") while column binding all you load into the sparse matrix. Then, you can save it locally (RDS) for fast load. It fits for a 4GB laptop. Train categorical is about 1GB when sparse.
", "votes": 1, "replies": [] }, { "id": 131767, "author_name": "junzi", "author_url": "", "post_date": "2016-08-19T13:52:15.847000", "content": "maybe it is because the dataset is too big.
", "votes": 1, "replies": [] }, { "id": 131948, "author_name": "inversion", "author_url": "", "post_date": "2016-08-20T21:15:40.927000", "content": "[quote=jeffd23;131815]
\n\nLoading the data easily maxed out my 32GB of RAM. \n[/quote]
\n\nFYI . . . also maxed out my 64GB of RAM (just blindly reading the data). Of course, there are tricks, as have been mentioned in the forum.
", "votes": 2, "replies": [] }, { "id": 131815, "author_name": "Jeff Delaney", "author_url": "", "post_date": "2016-08-19T21:29:35.377000", "content": "Loading the data easily maxed out my 32GB of RAM. I believe Kaggle's kernel docker image has a much lower memory limit. For python users, loading the data in chunks with Pandas, then experimenting with dimensionality reduction is one option...
\n\nChoosing 50 random columns from the numeric set earned a leaderboard score of ~0.14.
\n\ndef read_in_chunks(file_path):\n chunksize = 25000\n chunks = []\n for chunk in pd.read_csv(file_path, chunksize=chunksize):\n # do some dimensionality reduction...\n chunks.append(chunk)\n reduced_data = pd.concat(chunks, axis=0)\n return reduced_data\n@jeffd23: To the best of my knowledge, TruncatedSVD works fine on sparse matrices...
", "votes": 0, "replies": [] }, { "id": 132011, "author_name": "Jeff Delaney", "author_url": "", "post_date": "2016-08-21T17:15:58.997000", "content": "\n\n\n@jeffd23 How are we gonna do dimensionality reduction on a sparse\n matrix? Isn't it futile, we will lose information..?
\n
That is definitely true for PCA , which requires dense inputs.
\n\nThere are dimensionality reduction techniques for sparse matrices, although I do not have any experience with them, such as SparsePCA in Sckit-learn. Also, you can try to manually isolate the best features - not optimal, but it gets the data down to a manageable size for some initial exporation.
", "votes": 0, "replies": [] }, { "id": 132006, "author_name": "Prerit Anwekar", "author_url": "", "post_date": "2016-08-21T14:29:54.827000", "content": "@jeffd23 How are we gonna do dimensionality reduction on a sparse matrix? Isn't it futile, we will lose information..?
", "votes": 0, "replies": [] }, { "id": 131785, "author_name": "Bojan Tunguz", "author_url": "", "post_date": "2016-08-19T16:44:32.010000", "content": "[quote=junzi;131767]
\n\nmaybe it is because the dataset is too big.
\n\n[/quote]
\n\nThat's what I thought too. But the "Painter by Numbers" competition has a train dataset that is almost 2.5 times bigger, and yet they allow kernels there. It especially doesn't make sense since some of the simplest models can easily fit in Kaggle allocated memory.
", "votes": 0, "replies": [] }, { "id": 133529, "author_name": "", "author_url": "", "post_date": "2016-09-02T18:27:28.897000", "content": "", "votes": 0, "replies": [] }, { "id": 133527, "author_name": "", "author_url": "", "post_date": "2016-09-02T18:26:26.403000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "131583": "will there be no kernel for this competition?", "134853": "Hi all,\r\n\r\nWe've received many requests for Kernels and have decided we're going to give it a shot on this competition. Be forewarned that it might be a little frustrating to work on data of this size with Kernels' limitations, particularly if our (shared) workers start to hit a lot out-of-memory errors. Nonetheless, we're excited by your enthusiasm and curious to see what's possible to achieve wearing 8GB digital handcuffs. Have fun and don't try loading everything at once!", "132007": "[quote=Prerit Anwekar;132006]\r\n\r\n@jeffd23 How are we gonna do dimensionality reduction on a sparse matrix? Isn't it futile, we will lose information..?\r\n\r\n[/quote]\r\n\r\nUse a package/library that can perform a truncated SVD on a sparse matrix. You *almost always* lose information when using dimensionality reduction techniques.\r\n\r\nOn a sparse matrix, the loss on sparse values is equal to 0 because they do not need to be interpreted unlike dense matrices where a 0 is 0. It takes only about 100 seconds to perform a truncated SVD with 50 columns on the numerical train dataset with 1e-10 loss, under 8GB RAM usage (for both the train set and SVD computation in memory).", "131792": "This is what happens when I just load the train numeric dataset.\r\n\r\n > gc()\r\n used (Mb) gc trigger (Mb) max used (Mb)\r\n Ncells 614931 32.9 1168576 62.5 940480 50.3\r\n Vcells 1058373 8.1 2390182 18.3 2226183 17.0\r\n > train_numeric <- fread(\"train_numeric.csv\", header = TRUE, sep = \",\", stringsAsFactors = FALSE, colClasses = rep(\"numeric\", 970))\r\n Read 1183747 rows and 970 (of 970) columns from 1.993 GB file in 00:02:29\r\n > gc()\r\n used (Mb) gc trigger (Mb) max used (Mb)\r\n Ncells 617120 33.0 1168576 62.5 947860 50.7\r\n Vcells 1149297482 8768.5 1568457548 11966.4 1149391062 8769.2\r\n\r\nOver 8GB, is clearly too large (limit is at something like 8.5 billion bytes at Kaggle for max usage).\r\n\r\nFor a 2GB CSV, it is clearly small, but sparsity has its role. I wonder if Kaggle would still open the usage of scripts, so we can work on the data in a script. *Users would need a main tutorial script where it shows how to subsample / select columns to load though (so the first thing that does not happen is \"memory limit\" error).*\r\n\r\nIn case someone needs it: those who cannot load data into R to create a sparse matrix (or cannot convert to sparse directly) due to memory usage, use the following:\r\n\r\n train_numeric <- fread(input = \"train_numeric.csv\",\r\n select = 2:51\r\n header = TRUE,\r\n sep = \",\",\r\n stringsAsFactors = FALSE,\r\n colClasses = rep(\"numeric\", 970),\r\n data.table = FALSE)\r\n\r\nTransform it into a sparse matrix, keep loading more and more data (adjust columns selected until you get to 970 - id was intentionally removed here as it is 1; - the 970th is the label \"Outcome\") while column binding all you load into the sparse matrix. Then, you can save it locally (RDS) for fast load. It fits for a 4GB laptop. Train categorical is about 1GB when sparse.", "131767": "maybe it is because the dataset is too big.", "131948": "[quote=jeffd23;131815]\r\n\r\nLoading the data easily maxed out my 32GB of RAM. \r\n[/quote]\r\n\r\nFYI . . . also maxed out my 64GB of RAM (just blindly reading the data). Of course, there are tricks, as have been mentioned in the forum.", "131815": "Loading the data easily maxed out my 32GB of RAM. I believe Kaggle's kernel docker image has a much lower memory limit. For python users, loading the data in chunks with Pandas, then experimenting with dimensionality reduction is one option...\r\n\r\nChoosing 50 random columns from the numeric set earned a leaderboard score of ~0.14.\r\n\r\n def read_in_chunks(file_path):\r\n chunksize = 25000\r\n chunks = []\r\n for chunk in pd.read_csv(file_path, chunksize=chunksize):\r\n # do some dimensionality reduction...\r\n chunks.append(chunk)\r\n reduced_data = pd.concat(chunks, axis=0)\r\n return reduced_data\r\n \r\n", "132019": "@jeffd23: To the best of my knowledge, TruncatedSVD works fine on sparse matrices...", "132011": "> @jeffd23 How are we gonna do dimensionality reduction on a sparse\r\n> matrix? Isn't it futile, we will lose information..?\r\n\r\nThat is definitely true for PCA , which requires dense inputs. \r\n\r\nThere are dimensionality reduction techniques for sparse matrices, although I do not have any experience with them, such as SparsePCA in Sckit-learn. Also, you can try to manually isolate the best features - not optimal, but it gets the data down to a manageable size for some initial exporation. \r\n", "132006": "@jeffd23 How are we gonna do dimensionality reduction on a sparse matrix? Isn't it futile, we will lose information..?", "131785": "[quote=junzi;131767]\r\n\r\nmaybe it is because the dataset is too big.\r\n\r\n[/quote]\r\n\r\n\r\nThat's what I thought too. But the \"Painter by Numbers\" competition has a train dataset that is almost 2.5 times bigger, and yet they allow kernels there. It especially doesn't make sense since some of the simplest models can easily fit in Kaggle allocated memory. ", "133529": "", "133527": "" } }