Given that we are so close to the ending point, like to learn from all the experts here. Given the high dimensionality of the data assay inputs, what do you use for data dimension reduction ? Anyone use wavelets, pca, svd, isomap, tsne, sommap ? Love\r\n to hear your experiences. Personally, I use only near zero variance reduction. Time is really too short especially these have 15 sets of data...
\r\nThanks.
\r\nDr Patrick
\r\n", "messages": [ { "id": 15731, "postDate": "2012-10-16T18:22:03.367Z", "content": "
nearZeroVar is a function in caret package in R which can be use to find those columns from the data for which variance is near to zero(or zero).
\r\nSo we can reduce the dimension of the data by removing those columns for which varaince is zero, becoz zero variance columns have unique values. So those column doesn't impact the output at all.
", "votes": 3 }, { "id": 174884, "postDate": "2017-04-13T04:26:56.347Z", "content": "Correction: Centering by center of mass has been performed on all data, so Centering is certainly not needed. Here are a few pre-proccessing methods used for this problem in the literature:\n1. De-skewing\n2. 16 x 16 pixel sampling\n3. Noise removal\n4. Blurring\n5. Shiftable edges
", "rawMarkdown": "Correction: Centering by center of mass has been performed on all data, so Centering is certainly not needed. Here are a few pre-proccessing methods used for this problem [in the literature][1]:\n1. De-skewing\n2. 16 x 16 pixel sampling\n3. Noise removal\n4. Blurring\n5. Shiftable edges\n\n\n [1]: http://yann.lecun.com/exdb/mnist/index.html", "votes": -1, "replies": [ { "id": 351183, "postDate": "2018-07-01T14:19:06.723Z", "content": "Wrong competition dude.
", "rawMarkdown": "Wrong competition dude. ", "votes": 3 } ] }, { "id": 15743, "postDate": "2012-10-16T20:14:08.087Z", "content": "My approach was simple. First I used a threshold to filter the columns by. For example, when the frequency of the second most frequent value in the column was below 100, I would remove that column. This is very similar to what nearZeroVar does, which I also\r\n tried later. The resulting "good" column lists from nearZeroVar and thresholding were largely overlapping.
\r\nIt would be nice to hear from someone who successfully used clever feature engineering though.
[quote=Shea Parkes;15899]
\r\nWe definitely loaded the data promptly into sparse matrices ("Matrix" package from R). The standard R tree ensembles (gbm & randomForest) don't accept sparse matrices alas, so you still hit against either dimensionality reduction or RAM limits. I've tried\r\n to skim the python sci-learn docs, but couldn't get an adequately detailed level to understand if their tree algorithms accept sparse data. For dimensionality reduction, we just did a standard old SVD. We did a lot of work trying to adapt to the difference\r\n between testing and training data; I'm pretty sure this was done too simplistically and killed our chances. Just means I've got more reading to do on the subject.
\r\n[/quote]
\r\n\r\nScikit learn tree models don't take sparse matrices as input. That is a planned improvement but not in the current version or probably the next.
" }, { "id": 15948, "postDate": "2012-10-19T12:18:56.053Z", "content": "[quote=Shea Parkes;15899]
\r\nWe definitely loaded the data promptly into sparse matrices ("Matrix" package from R). The standard R tree ensembles (gbm & randomForest) don't accept sparse matrices alas, so you still hit against either dimensionality reduction or RAM limits. I've tried\r\n to skim the python sci-learn docs, but couldn't get an adequately detailed level to understand if their tree algorithms accept sparse data. For dimensionality reduction, we just did a standard old SVD. We did a lot of work trying to adapt to the difference\r\n between testing and training data; I'm pretty sure this was done too simplistically and killed our chances. Just means I've got more reading to do on the subject.
\r\n[/quote]
\r\n\r\n
I drowned in the same boat. Finally my best score was from an unweighted model. I spent almost all the time with different weighting models but unsuccessful.
" }, { "id": 15899, "postDate": "2012-10-18T17:09:21.410Z", "content": "We definitely loaded the data promptly into sparse matrices ("Matrix" package from R). The standard R tree ensembles (gbm & randomForest) don't accept sparse matrices alas, so you still hit against either dimensionality reduction or RAM limits. I've tried\r\n to skim the python sci-learn docs, but couldn't get an adequately detailed level to understand if their tree algorithms accept sparse data. For dimensionality reduction, we just did a standard old SVD. We did a lot of work trying to adapt to the difference\r\n between testing and training data; I'm pretty sure this was done too simplistically and killed our chances. Just means I've got more reading to do on the subject.
" }, { "id": 15893, "postDate": "2012-10-18T15:43:27.600Z", "content": "I tried a home cooked algorithm to remove features that had no effect on the activity.
\r\nFor sets that had a "baseline", any row that was close to the baseline and had features above a threshold value was used to marks those features as ineffective.
Also, parsing the source data and saving as sparse matrices saved a lot of RAM and load time.
" }, { "id": 15787, "postDate": "2012-10-17T05:46:18.287Z", "content": "I didn't know about Caret or zero variance, so ended up cooking up my own scheme which I think accomplishes roughly the same thing.
\r\nFor each column count number of non-zero elements for train (trainNz) and test (testNz).
\r\nFor each column compute product trainNz * testNz
\r\nSort columns in increasing size of trainNz*testNz
\r\nStart removing the columns and keep going until you've removed 1.5% of the total trainNz * testNz sum
This would remove roughly 80-90% of columns from each data set.
\r\nI also did a row strip as I noticed that some of the rows had way different number of non-zero entries. Here I would strip any row in the train set which had less than the minimum number of non-zero entries in the testSet and above the maximum number of\r\n non-zero entries in the testSet. This didn't take out that many rows (roughly 5-30 per subset), but I think it did make an improvement. Though I never really cracked cross validation so can't say by how much.
" }, { "id": 15775, "postDate": "2012-10-17T04:02:55.133Z", "content": "Yes, removing zero variance variables was the only pre-processing that I did for this competition
\r\nSVD was of limited help only
@Sashi
\r\n\r\n
I am not using SVD in my final models but I tried using SVD, You are saying abslutely correct but SVD or Near Zero Variance didn't help me so much but I am pretty much happy about my score as I am just a starter. This competition taught me what we should\r\n do at first place, If I would have done all the things which I tried in last 4-5 days I would have have got my current score in the starting and would have tried to improve from their. Anyhow It was a great learning.
\r\n\r\n
Thanks
\r\n\r\n
" }, { "id": 15745, "postDate": "2012-10-16T21:16:07.693Z", "content": "
[quote=dmitrim;15743]
\r\nMy approach was simple. First I used a threshold to filter the columns by. For example, when the frequency of the second most frequent value in the column was below 100, I would remove that column. This is very similar to what nearZeroVar does, which I also\r\n tried later. The resulting "good" column lists from nearZeroVar and thresholding were largely overlapping.
\r\nIt would be nice to hear from someone who successfully used clever feature engineering though.
[/quote]
\r\nInteresting. I always learning something from these forums.
" }, { "id": 15741, "postDate": "2012-10-16T19:27:55.097Z", "content": "I've tried using no preprocessing and with SVD, but (on a tiny laptop with 4 gigs of ram and a lowly dual core processor) both took at least a week to compute.
" }, { "id": 15740, "postDate": "2012-10-16T19:27:52.710Z", "content": "[quote=Anand Thakur;15731]
\r\nnearZeroVar is a function in caret package in R which can be use to find those columns from the data for which variance is near to zero(or zero).
\r\nSo we can reduce the dimension of the data by removing those columns for which varaince is zero, becoz zero variance columns have unique values. So those column doesn't impact the output at all.
\r\n[/quote]
\r\nHi,
\r\nIf you are using svd & if a column has near zero variance then doesn't that variable automatically get a nearzero weighting in the new dimensions extracted? I used to think this way and left the near zero variance variables in. Since filtering out the near\r\n zero varianace variables prior to svd seemed a little redundant.
" }, { "id": 15716, "postDate": "2012-10-16T17:07:51.963Z", "content": "[quote=Shinmagi;15710]
\r\nI am completely new to data science. I am not using any data processing because I don't know how. It seems to work fine enough for me to be in top 10 on the public score board.
\r\nWhat is the "near zero variance" technique mentioned above? Are we talking about centering and dividing the feature data for equalizing importance across features?
\r\n[/quote]
\r\nThanks for this interesting Challenge and forum discussions. @Shinmagi impressive results, congratulation. You must be using supercomputer\r\n or something. I couldn't get basic benchmark to run on my 32-bit Windows PC. So I ended up picking columns that have the best correlation with Activity. It does better than Merck Internal Benchmark, but not good enough to be in top 80.
\r\n
\r\nI do want to know what is the "near zero variance" technique mentioned above too. Thanks.
I am completely new to data science. I am not using any data processing because I don't know how. It seems to work fine enough for me to be in top 10 on the public score board.\r\n
\r\nWhat is the "near zero variance" technique mentioned above? Are we talking about centering and dividing the feature data for equalizing importance across features?
" }, { "id": 15696, "postDate": "2012-10-16T06:50:19.860Z", "content": "Hi
\r\n\r\n
I tried using SVD and near zero variance both. If you use near zero variance correctly and make prediction using it, you can easily achieve 0.45.
\r\nSVD is not doing so well but it is helping to beat benchmark easily with any simple model like randomForest with 20 trees.
\r\nThere is one draw back of using near zero variance, some time one of the molecule is causing some shift even it's variance is near to zero(note it's not exactly zero, it's near to zero). I think it's better just to delete unique value columns from the data\r\n and then try SVD or other things. I think achieving 0.45 is easy only.
\r\n\r\n
Regards
\r\nThakur
" }, { "id": 15690, "postDate": "2012-10-16T04:29:36.517Z", "content": "Given that we are so close to the ending point, like to learn from all the experts here. Given the high dimensionality of the data assay inputs, what do you use for data dimension reduction ? Anyone use wavelets, pca, svd, isomap, tsne, sommap ? Love\r\n to hear your experiences. Personally, I use only near zero variance reduction. Time is really too short especially these have 15 sets of data...
\r\nThanks.
\r\nDr Patrick
\r\n" }, { "id": 15754, "postDate": "2012-10-16T23:39:47.710Z", "rawMarkdown": "", "votes": 6, "isDeleted": true } ], "comments": [ { "id": 15731, "author_name": "DataGeek", "author_url": "", "post_date": "2012-10-16T18:22:03.367000", "content": "
nearZeroVar is a function in caret package in R which can be use to find those columns from the data for which variance is near to zero(or zero).
\r\nSo we can reduce the dimension of the data by removing those columns for which varaince is zero, becoz zero variance columns have unique values. So those column doesn't impact the output at all.
", "votes": 3, "replies": [] }, { "id": 174884, "author_name": "J.Jones", "author_url": "", "post_date": "2017-04-13T04:26:56.347000", "content": "Correction: Centering by center of mass has been performed on all data, so Centering is certainly not needed. Here are a few pre-proccessing methods used for this problem in the literature:\n1. De-skewing\n2. 16 x 16 pixel sampling\n3. Noise removal\n4. Blurring\n5. Shiftable edges
", "votes": -1, "replies": [ { "id": 351183, "author_name": "ChadRBernier", "author_url": "", "post_date": "2018-07-01T14:19:06.723000", "content": "Wrong competition dude.
", "votes": 3, "replies": [] } ] }, { "id": 15743, "author_name": "dmitrim", "author_url": "", "post_date": "2012-10-16T20:14:08.087000", "content": "My approach was simple. First I used a threshold to filter the columns by. For example, when the frequency of the second most frequent value in the column was below 100, I would remove that column. This is very similar to what nearZeroVar does, which I also\r\n tried later. The resulting "good" column lists from nearZeroVar and thresholding were largely overlapping.
\r\nIt would be nice to hear from someone who successfully used clever feature engineering though.
[quote=Shea Parkes;15899]
\r\nWe definitely loaded the data promptly into sparse matrices ("Matrix" package from R). The standard R tree ensembles (gbm & randomForest) don't accept sparse matrices alas, so you still hit against either dimensionality reduction or RAM limits. I've tried\r\n to skim the python sci-learn docs, but couldn't get an adequately detailed level to understand if their tree algorithms accept sparse data. For dimensionality reduction, we just did a standard old SVD. We did a lot of work trying to adapt to the difference\r\n between testing and training data; I'm pretty sure this was done too simplistically and killed our chances. Just means I've got more reading to do on the subject.
\r\n[/quote]
\r\n\r\nScikit learn tree models don't take sparse matrices as input. That is a planned improvement but not in the current version or probably the next.
", "votes": 0, "replies": [] }, { "id": 15948, "author_name": "José A. Guerrero", "author_url": "", "post_date": "2012-10-19T12:18:56.053000", "content": "[quote=Shea Parkes;15899]
\r\nWe definitely loaded the data promptly into sparse matrices ("Matrix" package from R). The standard R tree ensembles (gbm & randomForest) don't accept sparse matrices alas, so you still hit against either dimensionality reduction or RAM limits. I've tried\r\n to skim the python sci-learn docs, but couldn't get an adequately detailed level to understand if their tree algorithms accept sparse data. For dimensionality reduction, we just did a standard old SVD. We did a lot of work trying to adapt to the difference\r\n between testing and training data; I'm pretty sure this was done too simplistically and killed our chances. Just means I've got more reading to do on the subject.
\r\n[/quote]
\r\n\r\n
I drowned in the same boat. Finally my best score was from an unweighted model. I spent almost all the time with different weighting models but unsuccessful.
", "votes": 0, "replies": [] }, { "id": 15899, "author_name": "Shea Parkes", "author_url": "", "post_date": "2012-10-18T17:09:21.410000", "content": "We definitely loaded the data promptly into sparse matrices ("Matrix" package from R). The standard R tree ensembles (gbm & randomForest) don't accept sparse matrices alas, so you still hit against either dimensionality reduction or RAM limits. I've tried\r\n to skim the python sci-learn docs, but couldn't get an adequately detailed level to understand if their tree algorithms accept sparse data. For dimensionality reduction, we just did a standard old SVD. We did a lot of work trying to adapt to the difference\r\n between testing and training data; I'm pretty sure this was done too simplistically and killed our chances. Just means I've got more reading to do on the subject.
", "votes": 0, "replies": [] }, { "id": 15893, "author_name": "Jonathan Anderson", "author_url": "", "post_date": "2012-10-18T15:43:27.600000", "content": "I tried a home cooked algorithm to remove features that had no effect on the activity.
\r\nFor sets that had a "baseline", any row that was close to the baseline and had features above a threshold value was used to marks those features as ineffective.
Also, parsing the source data and saving as sparse matrices saved a lot of RAM and load time.
", "votes": 0, "replies": [] }, { "id": 15787, "author_name": "s445203", "author_url": "", "post_date": "2012-10-17T05:46:18.287000", "content": "I didn't know about Caret or zero variance, so ended up cooking up my own scheme which I think accomplishes roughly the same thing.
\r\nFor each column count number of non-zero elements for train (trainNz) and test (testNz).
\r\nFor each column compute product trainNz * testNz
\r\nSort columns in increasing size of trainNz*testNz
\r\nStart removing the columns and keep going until you've removed 1.5% of the total trainNz * testNz sum
This would remove roughly 80-90% of columns from each data set.
\r\nI also did a row strip as I noticed that some of the rows had way different number of non-zero entries. Here I would strip any row in the train set which had less than the minimum number of non-zero entries in the testSet and above the maximum number of\r\n non-zero entries in the testSet. This didn't take out that many rows (roughly 5-30 per subset), but I think it did make an improvement. Though I never really cracked cross validation so can't say by how much.
", "votes": 0, "replies": [] }, { "id": 15775, "author_name": "Dr Kiran R", "author_url": "", "post_date": "2012-10-17T04:02:55.133000", "content": "Yes, removing zero variance variables was the only pre-processing that I did for this competition
\r\nSVD was of limited help only
@Sashi
\r\n\r\n
I am not using SVD in my final models but I tried using SVD, You are saying abslutely correct but SVD or Near Zero Variance didn't help me so much but I am pretty much happy about my score as I am just a starter. This competition taught me what we should\r\n do at first place, If I would have done all the things which I tried in last 4-5 days I would have have got my current score in the starting and would have tried to improve from their. Anyhow It was a great learning.
\r\n\r\n
Thanks
\r\n\r\n
", "votes": 0, "replies": [] }, { "id": 15745, "author_name": "Sashikanth Dareddy", "author_url": "", "post_date": "2012-10-16T21:16:07.693000", "content": "
[quote=dmitrim;15743]
\r\nMy approach was simple. First I used a threshold to filter the columns by. For example, when the frequency of the second most frequent value in the column was below 100, I would remove that column. This is very similar to what nearZeroVar does, which I also\r\n tried later. The resulting "good" column lists from nearZeroVar and thresholding were largely overlapping.
\r\nIt would be nice to hear from someone who successfully used clever feature engineering though.
[/quote]
\r\nInteresting. I always learning something from these forums.
", "votes": 0, "replies": [] }, { "id": 15741, "author_name": "miguel", "author_url": "", "post_date": "2012-10-16T19:27:55.097000", "content": "I've tried using no preprocessing and with SVD, but (on a tiny laptop with 4 gigs of ram and a lowly dual core processor) both took at least a week to compute.
", "votes": 0, "replies": [] }, { "id": 15740, "author_name": "Sashikanth Dareddy", "author_url": "", "post_date": "2012-10-16T19:27:52.710000", "content": "[quote=Anand Thakur;15731]
\r\nnearZeroVar is a function in caret package in R which can be use to find those columns from the data for which variance is near to zero(or zero).
\r\nSo we can reduce the dimension of the data by removing those columns for which varaince is zero, becoz zero variance columns have unique values. So those column doesn't impact the output at all.
\r\n[/quote]
\r\nHi,
\r\nIf you are using svd & if a column has near zero variance then doesn't that variable automatically get a nearzero weighting in the new dimensions extracted? I used to think this way and left the near zero variance variables in. Since filtering out the near\r\n zero varianace variables prior to svd seemed a little redundant.
", "votes": 0, "replies": [] }, { "id": 15716, "author_name": "Sarkis", "author_url": "", "post_date": "2012-10-16T17:07:51.963000", "content": "[quote=Shinmagi;15710]
\r\nI am completely new to data science. I am not using any data processing because I don't know how. It seems to work fine enough for me to be in top 10 on the public score board.
\r\nWhat is the "near zero variance" technique mentioned above? Are we talking about centering and dividing the feature data for equalizing importance across features?
\r\n[/quote]
\r\nThanks for this interesting Challenge and forum discussions. @Shinmagi impressive results, congratulation. You must be using supercomputer\r\n or something. I couldn't get basic benchmark to run on my 32-bit Windows PC. So I ended up picking columns that have the best correlation with Activity. It does better than Merck Internal Benchmark, but not good enough to be in top 80.
\r\n
\r\nI do want to know what is the "near zero variance" technique mentioned above too. Thanks.
I am completely new to data science. I am not using any data processing because I don't know how. It seems to work fine enough for me to be in top 10 on the public score board.\r\n
\r\nWhat is the "near zero variance" technique mentioned above? Are we talking about centering and dividing the feature data for equalizing importance across features?
", "votes": 0, "replies": [] }, { "id": 15696, "author_name": "DataGeek", "author_url": "", "post_date": "2012-10-16T06:50:19.860000", "content": "Hi
\r\n\r\n
I tried using SVD and near zero variance both. If you use near zero variance correctly and make prediction using it, you can easily achieve 0.45.
\r\nSVD is not doing so well but it is helping to beat benchmark easily with any simple model like randomForest with 20 trees.
\r\nThere is one draw back of using near zero variance, some time one of the molecule is causing some shift even it's variance is near to zero(note it's not exactly zero, it's near to zero). I think it's better just to delete unique value columns from the data\r\n and then try SVD or other things. I think achieving 0.45 is easy only.
\r\n\r\n
Regards
\r\nThakur
", "votes": 0, "replies": [] }, { "id": 15754, "author_name": "", "author_url": "", "post_date": "2012-10-16T23:39:47.710000", "content": "", "votes": 6, "replies": [] } ], "raw_markdown_by_id": { "15731": "", "174884": "Correction: Centering by center of mass has been performed on all data, so Centering is certainly not needed. Here are a few pre-proccessing methods used for this problem [in the literature][1]:\n1. De-skewing\n2. 16 x 16 pixel sampling\n3. Noise removal\n4. Blurring\n5. Shiftable edges\n\n\n [1]: http://yann.lecun.com/exdb/mnist/index.html", "15743": "", "26554": "", "15948": "", "15899": "", "15893": "", "15787": "", "15775": "", "15760": "", "15745": "", "15741": "", "15740": "", "15716": "", "15710": "", "15696": "", "15690": "", "15754": "" } }