It's close enough to the deadline, and I'm curious. I believe there's no harm in sharing your setup this close to the end. If you don't agree, I respect you, and hey, you can post your rig tomorrow, if you like.
\nI'm curious because there's been a few posts on the forums about utilizing cloud services like AWS. I'm also curious because the top slot on the leaderboard hasn't changed for almost a month, no one's even gotten close. And the top contestant apparently has 32 cpus and 196 gb ram.
\nI'm running on a laptop that's a few years old, 4 gb ram, dual core. I have patience, so my main bottleneck was ram. I have hundreds of memory-dump files, I think I've reduced the life of my harddisk by half in this competition.
\nWhat about you? What hardware did you use?
\n", "messages": [ { "id": "36382", "postDate": "12/19/2013 22:31:35", "content": "
It's close enough to the deadline, and I'm curious. I believe there's no harm in sharing your setup this close to the end. If you don't agree, I respect you, and hey, you can post your rig tomorrow, if you like.
\nI'm curious because there's been a few posts on the forums about utilizing cloud services like AWS. I'm also curious because the top slot on the leaderboard hasn't changed for almost a month, no one's even gotten close. And the top contestant apparently has 32 cpus and 196 gb ram.
\nI'm running on a laptop that's a few years old, 4 gb ram, dual core. I have patience, so my main bottleneck was ram. I have hundreds of memory-dump files, I think I've reduced the life of my harddisk by half in this competition.
\nWhat about you? What hardware did you use?
\n", "rawMarkdown": "", "votes": null }, { "id": "36383", "postDate": "12/19/2013 22:54:04", "content": "
I'm using 8 core laptop with 8 GB of RAM. Like you have a lot of dumps on the disk. My setup essentially is a bunch of Java programs where each reads some file from disk and outputs another file.
\nI was able to achieve 0.78 score with total running time ~2 hours. Then I started to make models more and more complex to improve score and my last model is still running (~30 hours since I started it).
\n\n
\n
", "rawMarkdown": "", "votes": null }, { "id": "36384", "postDate": "12/19/2013 23:00:41", "content": "
I have an i7 with 16gb (generic pc stuff) running linux and python. But I tend not to take advantage of the multi-cores unless the library has built in support for that
\nBelieve it or not, my main issue with this competition was also RAM. This is my first NLP type problem and most of the libraries I used to vectorize the text data resulted in blowing out my ram. I finally got that step accomplished today! (Meanwhile, I have been submitting non-machine learning solutions). I hope to squeeze in a more sophisticated solution in the final hours !
", "rawMarkdown": "", "votes": null }, { "id": "36385", "postDate": "12/19/2013 23:35:12", "content": "Lots of file dumps for me too, but RAM and time are not an issue. My best solution including all the processing runs in less than 30 minutes on a 2 year old low-end laptop (Core i3, 6 GB RAM), using maximum 1.5 GB RAM. I also tried another model for which I needed around and 10 hours of processing, but without better results (yet).
", "rawMarkdown": "", "votes": null }, { "id": "36387", "postDate": "12/20/2013 01:28:42", "content": "My computer is 4core cpus with 6g ram. The ram problem really cause a lot of dumps. So finally, I ran it with pyLucene, and runniing for neally 5 days. I don't think my final result will be good.
", "rawMarkdown": "", "votes": null }, { "id": "36389", "postDate": "12/20/2013 02:10:23", "content": "Quick lunch post.
\nHome desktop. i5 (quad core). 16gb RAM.
\nI'm curious to know (after the competition ends of course) the nature of other people's models/methods, and whether they're essentially mine, qualitatively different, or theoretically different but in effect the same. I'm not going to be doing any more work before the end of the deadline, so my score is a basic benchmark for a decent implementation of mine.
\nI'm genuinely curious because for me its generating the evidence for each tag from the train set that takes the longest amount of time and has to be run in a loop/cull/output to disk process. And its storing total-evidence that easily blows out the ram. Once that's done, and the input data is formatted properly, the actual classification of the records is pretty quick on mine :|
\nI thought at the end of doing something like a fuzzy-join/similarity metric type approach, but its too close to my regular work, so is boring :P But if other people are doing something like that, or running big optimizations/looking for tighter and tighter fits, it might explain the run time differentials. Either way I'm genuinely curious and wanting to learn from other's approaches...
", "rawMarkdown": "", "votes": null }, { "id": "36393", "postDate": "12/20/2013 06:33:13", "content": "Hi Barisumog. Basically I am in the same situation as yours. I am working on a core i3 (dual processor) with 8gb of RAM and I did create some temporary files on hard disk just to speed up the training and prediction by freezing certain passages. My solution, using temporary files, runs in about 6 hours, from scratch it takes about 12 hours.
", "rawMarkdown": "", "votes": null }, { "id": "36395", "postDate": "12/20/2013 08:07:57", "content": "4 cores. 4gb of RAM. SSD hard drive. I used the pandas package in Python to read the csv in chunks.
", "rawMarkdown": "", "votes": null }, { "id": "36398", "postDate": "12/20/2013 08:36:07", "content": "i5 dual core, using only a single core. Main bottleneck is indeed my 8gb RAM.
\nMy solution runs in 1.5 hours and it's from scratch (so it'll be nice to present); no intermediate files.
\nIf I would write things to file, I think I can do all the predictions in roughly 15 minutes. Also, I believe I got a score of roughly 0.76 from scratch in only 1/2 hour.
\nI guess my model is OK in the "low resources vs good result" department.
\nKind of a spoiler/trick, but I basically used 1.5 million posts to allow for new features, then all the other 4.5 million posts are not allowed to add features (as it would blow up the RAM), but they are still very useful to update the estimates. I calculated that with 16gb RAM everything would fit and it would be a great improvement.
\nNow let's hope this last statement won't bite me in the butt when I go for top 10% :-)
\n", "rawMarkdown": "", "votes": null }, { "id": "36399", "postDate": "12/20/2013 10:13:55", "content": "
I've been using an i7 laptop with 32GB RAM. Using a bunch of Python scripts. I preprocessed the data files to speed up training and testing.
\nTraining takes about 4 hours, all in-memory using ~25GB RAM. Some pruning of less important parts of the model was required during training to prevent disk thrashing. Prediction takes about 4 hours too.
", "rawMarkdown": "", "votes": null }, { "id": "36400", "postDate": "12/20/2013 12:56:46", "content": "For the most part I used my home-computer: i7 quad-core with 16GB of RAM and SSD for my first few submissions.
\nTowards the end I tried to brute-force my way up the leaderboard with a 20 core machine with 64 GBs of RAM rented for a night (cost me about 15$, rented from digitalocean.com). I took the first 100'000 samples and trained ridge regression in a OVR for the top 13'000 tags. This takes about 8 hours to run. Unfortunately it improved the score only marginally.
", "rawMarkdown": "", "votes": null }, { "id": "36407", "postDate": "12/20/2013 15:50:13", "content": "I used my i5 dual core laptop with 8gb of ram. got a score around 0.775 with a model which takes around 4 hours for feature selection , 3.5 hours for training and 4 hours for prediction. My last model which scored 0.78385 took 10-12 hours for feature selection, 8 hours for training and 2 hours for prediction. RAM usage was never more than 3gb.
", "rawMarkdown": "", "votes": null }, { "id": "36422", "postDate": "12/20/2013 19:03:45", "content": "I'm using a 2.7GHz Intel Core i7 with 16GB of RAM.
\nI parse the input file with Python for 2 hours. I then feed the result into a C++ program which processes the input again, for about an hour. Final Training/Prediction phase takes about an hour too.
", "rawMarkdown": "", "votes": null }, { "id": "36429", "postDate": "12/20/2013 22:07:30", "content": "I tried with a small Hadoop cluster of 3 nodes (each node having 2 core and 16gb RAM ), but it couldn't score much.
\nFinally, I shifted to my 4 years old laptop having 2 cores , 4GB RAM. My process takes roughly 2 hours including training and prediction.
", "rawMarkdown": "", "votes": null }, { "id": "36433", "postDate": "12/21/2013 00:01:28", "content": "My approach was to use multiple algorithms to divide and conquer. I used a simple logistic regression to model the first 200 tags. I wanted to limit to tags with >5k observations, but ended up finding out that there wasn't much gain after 200. For very rare tags I used a simple "predict tag if tag is in title" approach. For tags falling between top 200 and very rare, I used an associan rules model, limiting the rules to those with p>0.25 and support > 50.
Code wise these models were very simple, the hardest part was to combine them in a way that made sense.
As for the size of the data, with limited time at my disposal, I made the call to get the body trimmed of anything between 'code' and 'a' html tags. With those excluded, the files were much smaller, which allowed me to train the top 200 all in memory. For association rules, I ended up creating a few files, and was keeping in memory only the final dictionary.
I've got to say that this has been by far the competition I learned the most from, and regret having put time in it only in the last two weeks.
I was running a 4 core desktop with 24GB RAM. All of my code was in Python, and I found that RAM was definitely the main bottleneck. I spent a lot of time in the early days trying to get Scikit-learn and LDA model working (using Gensim) but didn't get anywhere, mainly because of memory issues.
\n\n
Then I shifted to using simple word-tag association models, and kept optimising it. In the end, I reached 0.781 on the test set, and 0.535 on my validation set (without using duplicates). Working with small subset of training data while developing the model, and then running it on more later. But at the most, I could training on 4.5M documents, before running into memory issues. Though I am sure, my code can be optimised a lot...
\nI was a bit disappointed towards the end, as I rented an EC2 instance to be able to train on the full dataset (of 6M documents), but the resulting model performed badly due to overfitting. So the money was wasted... :(
\n\n
Did other people have the same experience? I am quite interested in knowing what approaches everyone took, and what worked and what didn't...
", "rawMarkdown": "", "votes": null }, { "id": "36440", "postDate": "12/21/2013 01:21:35", "content": "16 core EC2 machine with 60 GB of ram. I figured the problem was hard enough without constraining myself on hardware. Unfortunately I made some strategic errors early on and so couldn't get close to the highest scores. All 35,000 models are regular expressions with some associated conditional probabilities. Had I taken the time to tokenize properly and try some different types of models I might have done better.
\nTo minimize AWS charges I did the exploratory work on my i5 laptop, but EC2 was essential for fitting and scoring the full model.
", "rawMarkdown": "", "votes": null }, { "id": "36442", "postDate": "12/21/2013 02:43:40", "content": "i also use word-tag association algorithm such as mutual infomation. Here i just use title , not using body. I get 0.4 on my cv. Your result is overperformence more than me . Can you share your method that how you optimism ?
\n\n
Here i want to say: when i use the body, there is so much noise that the score is lower much than the score just using title.
\nand LDA , I try , bad result.
\n\n
Finally, I am curious that when in the train , we get word-tag association, and in the predition func, what is yours?
", "rawMarkdown": "", "votes": null }, { "id": "36444", "postDate": "12/21/2013 02:54:23", "content": "I used an dell xps8300(i7,8gb RAM). The biggest constraint for me is the time. I only got about a month to work on the problem. Had to give up a more comprehensive approach(one model for each tag) and settled with a leaner model with only 7 features due to time constraint. I used python and it took about 6 hours to do feature extraction. Training was about 2 hours. Prediction took about 20 minutes.
", "rawMarkdown": "", "votes": null }, { "id": "36446", "postDate": "12/21/2013 03:24:15", "content": "you take tag prediction as a classfication problem? how do you do it? one tag is should be predict or not is the target?
", "rawMarkdown": "", "votes": null }, { "id": "36447", "postDate": "12/21/2013 04:05:31", "content": "7 features! What were they?
", "rawMarkdown": "", "votes": null }, { "id": "36455", "postDate": "12/21/2013 07:26:29", "content": "I worked on an 8-core mac-mini with 16GB of memory and used Python. As everyone has noted, memory was the main limitation. My solution to this constraint was to process only a sample of the training set and even then I needed to break the sample into parts that were processed separately, storing the results of each part to disk. The total training time (including data munging) my best submission was about 6 hours, and the prediction required an additional 3 hours.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 36383, "author_name": "edunov", "author_url": "", "post_date": "12/19/2013 22:54:04", "content": "I'm using 8 core laptop with 8 GB of RAM. Like you have a lot of dumps on the disk. My setup essentially is a bunch of Java programs where each reads some file from disk and outputs another file.
\nI was able to achieve 0.78 score with total running time ~2 hours. Then I started to make models more and more complex to improve score and my last model is still running (~30 hours since I started it).
\n\n
\n
", "votes": null, "replies": [] }, { "id": 36384, "author_name": "geringer", "author_url": "", "post_date": "12/19/2013 23:00:41", "content": "
I have an i7 with 16gb (generic pc stuff) running linux and python. But I tend not to take advantage of the multi-cores unless the library has built in support for that
\nBelieve it or not, my main issue with this competition was also RAM. This is my first NLP type problem and most of the libraries I used to vectorize the text data resulted in blowing out my ram. I finally got that step accomplished today! (Meanwhile, I have been submitting non-machine learning solutions). I hope to squeeze in a more sophisticated solution in the final hours !
", "votes": null, "replies": [] }, { "id": 36385, "author_name": "andreibozantan", "author_url": "", "post_date": "12/19/2013 23:35:12", "content": "Lots of file dumps for me too, but RAM and time are not an issue. My best solution including all the processing runs in less than 30 minutes on a 2 year old low-end laptop (Core i3, 6 GB RAM), using maximum 1.5 GB RAM. I also tried another model for which I needed around and 10 hours of processing, but without better results (yet).
", "votes": null, "replies": [] }, { "id": 36387, "author_name": "lavizhao", "author_url": "", "post_date": "12/20/2013 01:28:42", "content": "My computer is 4core cpus with 6g ram. The ram problem really cause a lot of dumps. So finally, I ran it with pyLucene, and runniing for neally 5 days. I don't think my final result will be good.
", "votes": null, "replies": [] }, { "id": 36389, "author_name": "damienmelksham", "author_url": "", "post_date": "12/20/2013 02:10:23", "content": "Quick lunch post.
\nHome desktop. i5 (quad core). 16gb RAM.
\nI'm curious to know (after the competition ends of course) the nature of other people's models/methods, and whether they're essentially mine, qualitatively different, or theoretically different but in effect the same. I'm not going to be doing any more work before the end of the deadline, so my score is a basic benchmark for a decent implementation of mine.
\nI'm genuinely curious because for me its generating the evidence for each tag from the train set that takes the longest amount of time and has to be run in a loop/cull/output to disk process. And its storing total-evidence that easily blows out the ram. Once that's done, and the input data is formatted properly, the actual classification of the records is pretty quick on mine :|
\nI thought at the end of doing something like a fuzzy-join/similarity metric type approach, but its too close to my regular work, so is boring :P But if other people are doing something like that, or running big optimizations/looking for tighter and tighter fits, it might explain the run time differentials. Either way I'm genuinely curious and wanting to learn from other's approaches...
", "votes": null, "replies": [] }, { "id": 36393, "author_name": "lucamassaron", "author_url": "", "post_date": "12/20/2013 06:33:13", "content": "Hi Barisumog. Basically I am in the same situation as yours. I am working on a core i3 (dual processor) with 8gb of RAM and I did create some temporary files on hard disk just to speed up the training and prediction by freezing certain passages. My solution, using temporary files, runs in about 6 hours, from scratch it takes about 12 hours.
", "votes": null, "replies": [] }, { "id": 36395, "author_name": "ahallan", "author_url": "", "post_date": "12/20/2013 08:07:57", "content": "4 cores. 4gb of RAM. SSD hard drive. I used the pandas package in Python to read the csv in chunks.
", "votes": null, "replies": [] }, { "id": 36398, "author_name": "pvkooten", "author_url": "", "post_date": "12/20/2013 08:36:07", "content": "i5 dual core, using only a single core. Main bottleneck is indeed my 8gb RAM.
\nMy solution runs in 1.5 hours and it's from scratch (so it'll be nice to present); no intermediate files.
\nIf I would write things to file, I think I can do all the predictions in roughly 15 minutes. Also, I believe I got a score of roughly 0.76 from scratch in only 1/2 hour.
\nI guess my model is OK in the "low resources vs good result" department.
\nKind of a spoiler/trick, but I basically used 1.5 million posts to allow for new features, then all the other 4.5 million posts are not allowed to add features (as it would blow up the RAM), but they are still very useful to update the estimates. I calculated that with 16gb RAM everything would fit and it would be a great improvement.
\nNow let's hope this last statement won't bite me in the butt when I go for top 10% :-)
\n", "votes": null, "replies": [] }, { "id": 36399, "author_name": "philipwahrlich", "author_url": "", "post_date": "12/20/2013 10:13:55", "content": "
I've been using an i7 laptop with 32GB RAM. Using a bunch of Python scripts. I preprocessed the data files to speed up training and testing.
\nTraining takes about 4 hours, all in-memory using ~25GB RAM. Some pruning of less important parts of the model was required during training to prevent disk thrashing. Prediction takes about 4 hours too.
", "votes": null, "replies": [] }, { "id": 36400, "author_name": "mattvonrohr", "author_url": "", "post_date": "12/20/2013 12:56:46", "content": "For the most part I used my home-computer: i7 quad-core with 16GB of RAM and SSD for my first few submissions.
\nTowards the end I tried to brute-force my way up the leaderboard with a 20 core machine with 64 GBs of RAM rented for a night (cost me about 15$, rented from digitalocean.com). I took the first 100'000 samples and trained ridge regression in a OVR for the top 13'000 tags. This takes about 8 hours to run. Unfortunately it improved the score only marginally.
", "votes": null, "replies": [] }, { "id": 36407, "author_name": "shendu", "author_url": "", "post_date": "12/20/2013 15:50:13", "content": "I used my i5 dual core laptop with 8gb of ram. got a score around 0.775 with a model which takes around 4 hours for feature selection , 3.5 hours for training and 4 hours for prediction. My last model which scored 0.78385 took 10-12 hours for feature selection, 8 hours for training and 2 hours for prediction. RAM usage was never more than 3gb.
", "votes": null, "replies": [] }, { "id": 36422, "author_name": "pminkov", "author_url": "", "post_date": "12/20/2013 19:03:45", "content": "I'm using a 2.7GHz Intel Core i7 with 16GB of RAM.
\nI parse the input file with Python for 2 hours. I then feed the result into a C++ program which processes the input again, for about an hour. Final Training/Prediction phase takes about an hour too.
", "votes": null, "replies": [] }, { "id": 36429, "author_name": "chitrasen", "author_url": "", "post_date": "12/20/2013 22:07:30", "content": "I tried with a small Hadoop cluster of 3 nodes (each node having 2 core and 16gb RAM ), but it couldn't score much.
\nFinally, I shifted to my 4 years old laptop having 2 cores , 4GB RAM. My process takes roughly 2 hours including training and prediction.
", "votes": null, "replies": [] }, { "id": 36433, "author_name": "adjgiulio", "author_url": "", "post_date": "12/21/2013 00:01:28", "content": "My approach was to use multiple algorithms to divide and conquer. I used a simple logistic regression to model the first 200 tags. I wanted to limit to tags with >5k observations, but ended up finding out that there wasn't much gain after 200. For very rare tags I used a simple "predict tag if tag is in title" approach. For tags falling between top 200 and very rare, I used an associan rules model, limiting the rules to those with p>0.25 and support > 50.
Code wise these models were very simple, the hardest part was to combine them in a way that made sense.
As for the size of the data, with limited time at my disposal, I made the call to get the body trimmed of anything between 'code' and 'a' html tags. With those excluded, the files were much smaller, which allowed me to train the top 200 all in memory. For association rules, I ended up creating a few files, and was keeping in memory only the final dictionary.
I've got to say that this has been by far the competition I learned the most from, and regret having put time in it only in the last two weeks.
I was running a 4 core desktop with 24GB RAM. All of my code was in Python, and I found that RAM was definitely the main bottleneck. I spent a lot of time in the early days trying to get Scikit-learn and LDA model working (using Gensim) but didn't get anywhere, mainly because of memory issues.
\n\n
Then I shifted to using simple word-tag association models, and kept optimising it. In the end, I reached 0.781 on the test set, and 0.535 on my validation set (without using duplicates). Working with small subset of training data while developing the model, and then running it on more later. But at the most, I could training on 4.5M documents, before running into memory issues. Though I am sure, my code can be optimised a lot...
\nI was a bit disappointed towards the end, as I rented an EC2 instance to be able to train on the full dataset (of 6M documents), but the resulting model performed badly due to overfitting. So the money was wasted... :(
\n\n
Did other people have the same experience? I am quite interested in knowing what approaches everyone took, and what worked and what didn't...
", "votes": null, "replies": [] }, { "id": 36440, "author_name": "jfkingiii", "author_url": "", "post_date": "12/21/2013 01:21:35", "content": "16 core EC2 machine with 60 GB of ram. I figured the problem was hard enough without constraining myself on hardware. Unfortunately I made some strategic errors early on and so couldn't get close to the highest scores. All 35,000 models are regular expressions with some associated conditional probabilities. Had I taken the time to tokenize properly and try some different types of models I might have done better.
\nTo minimize AWS charges I did the exploratory work on my i5 laptop, but EC2 was essential for fitting and scoring the full model.
", "votes": null, "replies": [] }, { "id": 36442, "author_name": "herofan", "author_url": "", "post_date": "12/21/2013 02:43:40", "content": "i also use word-tag association algorithm such as mutual infomation. Here i just use title , not using body. I get 0.4 on my cv. Your result is overperformence more than me . Can you share your method that how you optimism ?
\n\n
Here i want to say: when i use the body, there is so much noise that the score is lower much than the score just using title.
\nand LDA , I try , bad result.
\n\n
Finally, I am curious that when in the train , we get word-tag association, and in the predition func, what is yours?
", "votes": null, "replies": [] }, { "id": 36444, "author_name": "gohigh", "author_url": "", "post_date": "12/21/2013 02:54:23", "content": "I used an dell xps8300(i7,8gb RAM). The biggest constraint for me is the time. I only got about a month to work on the problem. Had to give up a more comprehensive approach(one model for each tag) and settled with a leaner model with only 7 features due to time constraint. I used python and it took about 6 hours to do feature extraction. Training was about 2 hours. Prediction took about 20 minutes.
", "votes": null, "replies": [] }, { "id": 36446, "author_name": "herofan", "author_url": "", "post_date": "12/21/2013 03:24:15", "content": "you take tag prediction as a classfication problem? how do you do it? one tag is should be predict or not is the target?
", "votes": null, "replies": [] }, { "id": 36447, "author_name": "jfkingiii", "author_url": "", "post_date": "12/21/2013 04:05:31", "content": "7 features! What were they?
", "votes": null, "replies": [] }, { "id": 36455, "author_name": "psilogram", "author_url": "", "post_date": "12/21/2013 07:26:29", "content": "I worked on an 8-core mac-mini with 16GB of memory and used Python. As everyone has noted, memory was the main limitation. My solution to this constraint was to process only a sample of the training set and even then I needed to break the sample into parts that were processed separately, storing the results of each part to disk. The total training time (including data munging) my best submission was about 6 hours, and the prediction required an additional 3 hours.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "36382": "", "36383": "", "36384": "", "36385": "", "36387": "", "36389": "", "36393": "", "36395": "", "36398": "", "36399": "", "36400": "", "36407": "", "36422": "", "36429": "", "36433": "", "36436": "", "36440": "", "36442": "", "36444": "", "36446": "", "36447": "", "36455": "" }, "source": "meta" }