I'm a bit concerned that the data set is too large to deal with without using distributed computing methods. So obviously the best solution would be to use hadoop+AWS. My problem with this is that this is not free (if you do not use the free tier).
\nObviously you could do it without distributed computing (eg. sampling), but it seems that distributed computing would offer clear advantages.
\nSo it seems that those competitors who choose to spend money have an advantage over those that do not. Is this a problem?
", "messages": [ { "id": 30592, "postDate": "2013-09-09T19:48:01.057Z", "content": "In Facebook Recruiting 1 I was running everything on a crappy dual-core laptop. Others were spending moderate amounts of money spinning up clusters. If you can do as well on the former as the latter, then that says something.
\nYou also seem to be making the assumption that first place gets an interview and second place doesn't. That's a bad assumption. In FR1 I was never in first place. After I dropped out of the comp I dropped from second to something like 24th. But here I am at Facebook.
", "votes": 4 }, { "id": 30607, "postDate": "2013-09-09T21:16:19.827Z", "content": "As noted in other discussions, the objective of this comp can be multidimensional, and not everyone is interested in the same dimensions. But for those whose ultimate goal is to get a job at Facebook, then simply getting the best score is not necessarily going to end in happiness. For example, if you happened to have access to code that did all the heavy lifting for you, and access to unlimited resources, but had absolutely no idea what the code was actually doing, then you're probably not going to get very far in the interview process.
\nSo, yes, recruiting comps are fundamentally different from other comps, IMHO.
\nObviously, the closer you can get to the top of the pack, the better the chances are that you'll roll through to the next steps. And making use of any talents and (allowed) resources you have is fair game. Being able to make things efficient is definitely a valuable ability.
\nJust try to get the best score you can with the resources available to you!
\nYou can read my only post during FR1 at http://www.kaggle.com/c/FacebookRecruiting/forums/t/2082/0-711-is-the-new-0?page=3 . I dropped out and didn't even know that the comp had finished. But yet I'm here. Counterintuitive, huh? :)
", "votes": 1 }, { "id": 30598, "postDate": "2013-09-09T20:16:32.337Z", "content": "True, more details had to be submitted upfront in FR1. But if you perform well, then the discussion begins.
\nObviously you couldn't try using a '286 and come 367th in the comp and use this strategy; you still have to be competitive. But efficiency is valuable around here. Some of my photos code is executed many billions of times a day. Every millisecond I shaved off it counts.
", "votes": 1 }, { "id": 30591, "postDate": "2013-09-09T19:47:33.457Z", "content": "I'm currently using a 3 year old dual core laptop with 8 GB of RAM for what it's worth. I'm running into constraints at this point though.
", "votes": 1 }, { "id": 30543, "postDate": "2013-09-09T17:07:15.933Z", "content": "I'm a bit concerned that the data set is too large to deal with without using distributed computing methods. So obviously the best solution would be to use hadoop+AWS. My problem with this is that this is not free (if you do not use the free tier).
\nObviously you could do it without distributed computing (eg. sampling), but it seems that distributed computing would offer clear advantages.
\nSo it seems that those competitors who choose to spend money have an advantage over those that do not. Is this a problem?
", "votes": 1 }, { "id": 35387, "postDate": "2013-11-27T22:18:14.987Z", "content": "I also have a 8GB MacBook Pro and was able to train the full training set in a few minutes with an f1 >0.7 - though my training algorithm is written in C so it's bit more efficient compared to other "out of the box" solutions.
" }, { "id": 33447, "postDate": "2013-11-08T22:32:25.153Z", "content": "I haven't submitted an entry yet, but I've coded up just everything except the actual training/prediction bit. I'm working on an i5, with 16gb RAM. I've run out of memory a few times during testing, but so far this was due to my own idiocy...
\nI admit I actually thought I'd have a crack at this one because the size/complexity would exclude a large number of people using "off the shelf"/"out of the box" solutions in the likes of R, and thus thin the crowd/randomness at the top of the board a bit...
\nMy hard drives are nothing special, but low power/slow. I'm not looking at doing distributed anything. While someone up there said that distributed computing offers clear advantages, let me also say that it also offers clear disadvantages: not all algorithms, or even the best solutions, are easily distributable. It also adds a layer of complexity and computational overhead. Plus professionally, I've also noticed a tendency for people to use distributed computing to speed up a slow program 8-10 times, whereas a fast program on a single machine would result in a 10-100 speed-up, is still distributable, but is never considered because "I can just distribute everything!".
\nI've gotten 0.26'ish while testing the F1 score code that grades my predictions. It lines up well with the metrics in the leader board. Doing that barely used memory, required no real training or model, and is FAST. I estimate the time to run on the full test case to be about 1-2 minutes, single core. I'll be very disappointed if i don't supply a better entry by the end of the competition.
\nAs others have hinted, there's more than one way to skin a cat. If you use a program/technique that never requires the full data set to be held in memory (line-by-line, or page-by-page like SAS), then your memory requirements for handling the data become very small, allowing you to set aside more for any model.
\nMy current set-up has a "full read through the training data overhead" of just over 2 minutes, and uses practically little RAM. That's how long it takes to read each record in and out of memory sequentially with no additional work on each observation. I coded up an ability to do random access, but so far I haven't used it. Loading it in, I'm still sitting at just over 1gb memory usage while not doing anything else. I haven't over-optimized or anything at this stage.
\nI did some quick estimations on how long it will take for the technique I'd like to try to build to run on the test set, and came back with a few hours, which seems hopeful to me. Whether it works, or translates into reality, well...that's undetermined at this stage...
\nHopefully that gives people a bit of information.
" }, { "id": 33424, "postDate": "2013-11-08T13:56:48.573Z", "content": "I think a good approach is:
\n" }, { "id": 31105, "postDate": "2013-09-18T17:36:55.957Z", "content": "
To be honest, when you run things line by line, it doesn't require RAM that much. While running predictions, I see my memory usage go up by 1-2% (of 8GB RAM), so that certainly is not the issue, at least it isn't here. My prediction with score 0.26 took 38 minutes on a single core, while it took around 18 minutes for dual core.
\nI second that the information on what people used would be really interesting!
\nMy goal is now to work on getting a better model :-), so I do assume all these numbers will change a lot, soon.
\n" }, { "id": 31100, "postDate": "2013-09-18T17:14:23.617Z", "content": "
I don't know exactly what they mean by "a best submission", but the score of my best submission, 0.46, seems better than the current average score. And as I said before, the program runs in less than one hour (on a mediocre laptop) using less than 1GB of RAM, and this also includes the training part. So maybe they were talking about the whole thing.
\nI'm really curious about the performance of the other submissions. I think that it will be nice to add the possibility to submit some extra information regarding the memory usage, running time and the machine(s) used. And of course, this will be displayed together with a clear message that this information can't be verified to be accurate.
" }, { "id": 31094, "postDate": "2013-09-18T16:06:56.653Z", "content": "I'm pretty sure they are only talking about the prediction part. Training can be done offline so it shouldn't really matter how long it takes (within reason)
" }, { "id": 31091, "postDate": "2013-09-18T15:42:20.527Z", "content": "Solving a large problem with limited resources would be interesting to Facebook. Try to make a best submission that runs with <1GB RAM in under an hour!
\n\n
Does this mean the whole dictionary building and everything? Or is that with all the work done, just making the predictions? I am not sure what people are capable of these days....
" }, { "id": 30805, "postDate": "2013-09-12T20:29:58.650Z", "content": "Agreed. But time is important :) So is RAM
" }, { "id": 30803, "postDate": "2013-09-12T20:18:51.190Z", "content": "If you have a distributed computing algorithm I think that you can also run it on a single machine. Of course it will take much longer, but you can leave your computer to run for as long as required. Maybe AWS will offer a clear advantage for the algorithms which require a huge amount of RAM. Am I wrong?
" }, { "id": 30793, "postDate": "2013-09-12T17:14:23.627Z", "content": "Just to clarify. I did not mean that it is not possible to get a decent score using one machine. I was just saying that distributed computing offers clear advantages (thus increasing your likelihood of getting a better score).
" }, { "id": 30792, "postDate": "2013-09-12T16:50:53.490Z", "content": "My submission with a score of 0.46 is obtained using a single threaded C# program, that uses less than 1Gb of RAM and runs in less than one hour, on a laptop with a Core i3 CPU. And I didn't invest in optimizing it for speed, since this is not important in the competition. Also, speed is not important when running it on my computer. For testing the algorithm offline I can compute an approximate F1 score (+/- 0.04) in less than one minute (using sampling).
" }, { "id": 30771, "postDate": "2013-09-12T04:00:07.380Z", "content": "My 2 submissions were made on a 2GB Ram Linux VM running on only 4 cores in a 4 years old PC. Anything is possible lol.
" }, { "id": 30619, "postDate": "2013-09-09T22:33:25.687Z", "content": "@Brad S.: That's quite impressive! =) William might have mentioned your next challenge: <1GB RAM in under an hour! (I will try to do it if I have an acceptable rank)
\n@John: Your story really inspired me, thanks for sharing!
" }, { "id": 30611, "postDate": "2013-09-09T22:00:20.217Z", "content": "My first solution was done on my multi-core desktop machine. As a challenge to myself, I've since refactored it into a core single-threaded algorithm with a multi-core wrapper. I also made a separate Hadoop wrapper as a nice excuse to learn Apache Whirr, but it wasn't necessary.
" }, { "id": 30606, "postDate": "2013-09-09T21:16:12.297Z", "content": "So my understanding is correct then? This is a multi-objective problem where both score and quality of method have to be maximized (even though they can sometimes be conflicting)
" }, { "id": 30603, "postDate": "2013-09-09T21:03:42.913Z", "content": "The objective is to get the best score, but Facebook is very generous with their examinations of the leaderboard. Consider your accuracy as the thing that gets a foot in the door and the rest (efficiency, novelty, etc.) as the thing that gets you the job.
" }, { "id": 30601, "postDate": "2013-09-09T20:55:14.787Z", "content": "I guess I am a bit confused regarding the objective here. It seems like there might be a bit of a paradox . The point of the competition is to get the best score. Using EC2 clusters would definitely be an advantage (I don't think this is a controversial point). And the point of getting the best score is to get an interview. However, it seems that you are implying that the efficiency of your method and not the score itself is most important in the interview process. So it seems like the objective is not to get the best score at all costs, but to maximize your score while keeping your method as efficient as possible. An efficient solution and one that achieves the highest possible score are not necessarily the same. So it seems like you are saying that getting the best score is not necessarily the only objective (it usually is in these competitions).
" }, { "id": 30595, "postDate": "2013-09-09T19:59:56.987Z", "content": "But you only submit your results. They cannot know the resources you are using.
" }, { "id": 30589, "postDate": "2013-09-09T19:39:52.630Z", "content": "Solving a large problem with limited resources would be interesting to Facebook. Try to make a best submission that runs with <1GB RAM in under an hour!
" }, { "id": 30586, "postDate": "2013-09-09T19:33:18.300Z", "content": "True. But your resources become less limited if spend money on EC2 clusters. It is possible that this could be the deciding factor between first and second place. My point is that it is not exactly an even playing field between those that choose to spend money and those that don't. However, I guess you could say that about any competition, but it just seems like this competition is intrinsically linked to distributed computing (i.e. $).
" }, { "id": 30545, "postDate": "2013-09-09T17:20:48.963Z", "content": "which part?
" }, { "id": 30544, "postDate": "2013-09-09T17:14:36.160Z", "content": "EDIT: My comment was not constructive. Also, I just read this,
\n[quote=AlexM;30543]
\nObviously you could do it without distributed computing (eg. sampling), but it seems that distributed computing would offer clear advantages.
\n[/quote]
\nPartly true. It might be slightly advantageous in terms of time. But then again the competition runs for about 3 months, with a limit of two submissions per day. I think that would be a limiting factor even if you had access to EC2 clusters. As said earlier in the other thread, it's about how you can make best use of limited resources.
\nOn another note, AWS gives away free credit if you are a student. You might want to look into that.
" }, { "id": 35385, "postDate": "2013-11-27T21:54:40.837Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 30592, "author_name": "John Costella", "author_url": "", "post_date": "2013-09-09T19:48:01.057000", "content": "In Facebook Recruiting 1 I was running everything on a crappy dual-core laptop. Others were spending moderate amounts of money spinning up clusters. If you can do as well on the former as the latter, then that says something.
\nYou also seem to be making the assumption that first place gets an interview and second place doesn't. That's a bad assumption. In FR1 I was never in first place. After I dropped out of the comp I dropped from second to something like 24th. But here I am at Facebook.
", "votes": 4, "replies": [] }, { "id": 30607, "author_name": "John Costella", "author_url": "", "post_date": "2013-09-09T21:16:19.827000", "content": "As noted in other discussions, the objective of this comp can be multidimensional, and not everyone is interested in the same dimensions. But for those whose ultimate goal is to get a job at Facebook, then simply getting the best score is not necessarily going to end in happiness. For example, if you happened to have access to code that did all the heavy lifting for you, and access to unlimited resources, but had absolutely no idea what the code was actually doing, then you're probably not going to get very far in the interview process.
\nSo, yes, recruiting comps are fundamentally different from other comps, IMHO.
\nObviously, the closer you can get to the top of the pack, the better the chances are that you'll roll through to the next steps. And making use of any talents and (allowed) resources you have is fair game. Being able to make things efficient is definitely a valuable ability.
\nJust try to get the best score you can with the resources available to you!
\nYou can read my only post during FR1 at http://www.kaggle.com/c/FacebookRecruiting/forums/t/2082/0-711-is-the-new-0?page=3 . I dropped out and didn't even know that the comp had finished. But yet I'm here. Counterintuitive, huh? :)
", "votes": 1, "replies": [] }, { "id": 30598, "author_name": "John Costella", "author_url": "", "post_date": "2013-09-09T20:16:32.337000", "content": "True, more details had to be submitted upfront in FR1. But if you perform well, then the discussion begins.
\nObviously you couldn't try using a '286 and come 367th in the comp and use this strategy; you still have to be competitive. But efficiency is valuable around here. Some of my photos code is executed many billions of times a day. Every millisecond I shaved off it counts.
", "votes": 1, "replies": [] }, { "id": 30591, "author_name": "Alec Radford", "author_url": "", "post_date": "2013-09-09T19:47:33.457000", "content": "I'm currently using a 3 year old dual core laptop with 8 GB of RAM for what it's worth. I'm running into constraints at this point though.
", "votes": 1, "replies": [] }, { "id": 35387, "author_name": "Omar O", "author_url": "", "post_date": "2013-11-27T22:18:14.987000", "content": "I also have a 8GB MacBook Pro and was able to train the full training set in a few minutes with an f1 >0.7 - though my training algorithm is written in C so it's bit more efficient compared to other "out of the box" solutions.
", "votes": 0, "replies": [] }, { "id": 33447, "author_name": "Damien Melksham", "author_url": "", "post_date": "2013-11-08T22:32:25.153000", "content": "I haven't submitted an entry yet, but I've coded up just everything except the actual training/prediction bit. I'm working on an i5, with 16gb RAM. I've run out of memory a few times during testing, but so far this was due to my own idiocy...
\nI admit I actually thought I'd have a crack at this one because the size/complexity would exclude a large number of people using "off the shelf"/"out of the box" solutions in the likes of R, and thus thin the crowd/randomness at the top of the board a bit...
\nMy hard drives are nothing special, but low power/slow. I'm not looking at doing distributed anything. While someone up there said that distributed computing offers clear advantages, let me also say that it also offers clear disadvantages: not all algorithms, or even the best solutions, are easily distributable. It also adds a layer of complexity and computational overhead. Plus professionally, I've also noticed a tendency for people to use distributed computing to speed up a slow program 8-10 times, whereas a fast program on a single machine would result in a 10-100 speed-up, is still distributable, but is never considered because "I can just distribute everything!".
\nI've gotten 0.26'ish while testing the F1 score code that grades my predictions. It lines up well with the metrics in the leader board. Doing that barely used memory, required no real training or model, and is FAST. I estimate the time to run on the full test case to be about 1-2 minutes, single core. I'll be very disappointed if i don't supply a better entry by the end of the competition.
\nAs others have hinted, there's more than one way to skin a cat. If you use a program/technique that never requires the full data set to be held in memory (line-by-line, or page-by-page like SAS), then your memory requirements for handling the data become very small, allowing you to set aside more for any model.
\nMy current set-up has a "full read through the training data overhead" of just over 2 minutes, and uses practically little RAM. That's how long it takes to read each record in and out of memory sequentially with no additional work on each observation. I coded up an ability to do random access, but so far I haven't used it. Loading it in, I'm still sitting at just over 1gb memory usage while not doing anything else. I haven't over-optimized or anything at this stage.
\nI did some quick estimations on how long it will take for the technique I'd like to try to build to run on the test set, and came back with a few hours, which seems hopeful to me. Whether it works, or translates into reality, well...that's undetermined at this stage...
\nHopefully that gives people a bit of information.
", "votes": 0, "replies": [] }, { "id": 33424, "author_name": "Alessandro Sena", "author_url": "", "post_date": "2013-11-08T13:56:48.573000", "content": "I think a good approach is:
\n", "votes": 0, "replies": [] }, { "id": 31105, "author_name": "Pascal", "author_url": "", "post_date": "2013-09-18T17:36:55.957000", "content": "
To be honest, when you run things line by line, it doesn't require RAM that much. While running predictions, I see my memory usage go up by 1-2% (of 8GB RAM), so that certainly is not the issue, at least it isn't here. My prediction with score 0.26 took 38 minutes on a single core, while it took around 18 minutes for dual core.
\nI second that the information on what people used would be really interesting!
\nMy goal is now to work on getting a better model :-), so I do assume all these numbers will change a lot, soon.
\n", "votes": 0, "replies": [] }, { "id": 31100, "author_name": "Andrei Bozantan", "author_url": "", "post_date": "2013-09-18T17:14:23.617000", "content": "
I don't know exactly what they mean by "a best submission", but the score of my best submission, 0.46, seems better than the current average score. And as I said before, the program runs in less than one hour (on a mediocre laptop) using less than 1GB of RAM, and this also includes the training part. So maybe they were talking about the whole thing.
\nI'm really curious about the performance of the other submissions. I think that it will be nice to add the possibility to submit some extra information regarding the memory usage, running time and the machine(s) used. And of course, this will be displayed together with a clear message that this information can't be verified to be accurate.
", "votes": 0, "replies": [] }, { "id": 31094, "author_name": "AlexM", "author_url": "", "post_date": "2013-09-18T16:06:56.653000", "content": "I'm pretty sure they are only talking about the prediction part. Training can be done offline so it shouldn't really matter how long it takes (within reason)
", "votes": 0, "replies": [] }, { "id": 31091, "author_name": "Pascal", "author_url": "", "post_date": "2013-09-18T15:42:20.527000", "content": "Solving a large problem with limited resources would be interesting to Facebook. Try to make a best submission that runs with <1GB RAM in under an hour!
\n\n
Does this mean the whole dictionary building and everything? Or is that with all the work done, just making the predictions? I am not sure what people are capable of these days....
", "votes": 0, "replies": [] }, { "id": 30805, "author_name": "AlexM", "author_url": "", "post_date": "2013-09-12T20:29:58.650000", "content": "Agreed. But time is important :) So is RAM
", "votes": 0, "replies": [] }, { "id": 30803, "author_name": "Andrei Bozantan", "author_url": "", "post_date": "2013-09-12T20:18:51.190000", "content": "If you have a distributed computing algorithm I think that you can also run it on a single machine. Of course it will take much longer, but you can leave your computer to run for as long as required. Maybe AWS will offer a clear advantage for the algorithms which require a huge amount of RAM. Am I wrong?
", "votes": 0, "replies": [] }, { "id": 30793, "author_name": "AlexM", "author_url": "", "post_date": "2013-09-12T17:14:23.627000", "content": "Just to clarify. I did not mean that it is not possible to get a decent score using one machine. I was just saying that distributed computing offers clear advantages (thus increasing your likelihood of getting a better score).
", "votes": 0, "replies": [] }, { "id": 30792, "author_name": "Andrei Bozantan", "author_url": "", "post_date": "2013-09-12T16:50:53.490000", "content": "My submission with a score of 0.46 is obtained using a single threaded C# program, that uses less than 1Gb of RAM and runs in less than one hour, on a laptop with a Core i3 CPU. And I didn't invest in optimizing it for speed, since this is not important in the competition. Also, speed is not important when running it on my computer. For testing the algorithm offline I can compute an approximate F1 score (+/- 0.04) in less than one minute (using sampling).
", "votes": 0, "replies": [] }, { "id": 30771, "author_name": "Gabriel F.", "author_url": "", "post_date": "2013-09-12T04:00:07.380000", "content": "My 2 submissions were made on a 2GB Ram Linux VM running on only 4 cores in a 4 years old PC. Anything is possible lol.
", "votes": 0, "replies": [] }, { "id": 30619, "author_name": "", "author_url": "", "post_date": "2013-09-09T22:33:25.687000", "content": "@Brad S.: That's quite impressive! =) William might have mentioned your next challenge: <1GB RAM in under an hour! (I will try to do it if I have an acceptable rank)
\n@John: Your story really inspired me, thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 30611, "author_name": "Brad S.", "author_url": "", "post_date": "2013-09-09T22:00:20.217000", "content": "My first solution was done on my multi-core desktop machine. As a challenge to myself, I've since refactored it into a core single-threaded algorithm with a multi-core wrapper. I also made a separate Hadoop wrapper as a nice excuse to learn Apache Whirr, but it wasn't necessary.
", "votes": 0, "replies": [] }, { "id": 30606, "author_name": "AlexM", "author_url": "", "post_date": "2013-09-09T21:16:12.297000", "content": "So my understanding is correct then? This is a multi-objective problem where both score and quality of method have to be maximized (even though they can sometimes be conflicting)
", "votes": 0, "replies": [] }, { "id": 30603, "author_name": "Will Cukierski", "author_url": "", "post_date": "2013-09-09T21:03:42.913000", "content": "The objective is to get the best score, but Facebook is very generous with their examinations of the leaderboard. Consider your accuracy as the thing that gets a foot in the door and the rest (efficiency, novelty, etc.) as the thing that gets you the job.
", "votes": 0, "replies": [] }, { "id": 30601, "author_name": "AlexM", "author_url": "", "post_date": "2013-09-09T20:55:14.787000", "content": "I guess I am a bit confused regarding the objective here. It seems like there might be a bit of a paradox . The point of the competition is to get the best score. Using EC2 clusters would definitely be an advantage (I don't think this is a controversial point). And the point of getting the best score is to get an interview. However, it seems that you are implying that the efficiency of your method and not the score itself is most important in the interview process. So it seems like the objective is not to get the best score at all costs, but to maximize your score while keeping your method as efficient as possible. An efficient solution and one that achieves the highest possible score are not necessarily the same. So it seems like you are saying that getting the best score is not necessarily the only objective (it usually is in these competitions).
", "votes": 0, "replies": [] }, { "id": 30595, "author_name": "AlexM", "author_url": "", "post_date": "2013-09-09T19:59:56.987000", "content": "But you only submit your results. They cannot know the resources you are using.
", "votes": 0, "replies": [] }, { "id": 30589, "author_name": "Will Cukierski", "author_url": "", "post_date": "2013-09-09T19:39:52.630000", "content": "Solving a large problem with limited resources would be interesting to Facebook. Try to make a best submission that runs with <1GB RAM in under an hour!
", "votes": 0, "replies": [] }, { "id": 30586, "author_name": "AlexM", "author_url": "", "post_date": "2013-09-09T19:33:18.300000", "content": "True. But your resources become less limited if spend money on EC2 clusters. It is possible that this could be the deciding factor between first and second place. My point is that it is not exactly an even playing field between those that choose to spend money and those that don't. However, I guess you could say that about any competition, but it just seems like this competition is intrinsically linked to distributed computing (i.e. $).
", "votes": 0, "replies": [] }, { "id": 30545, "author_name": "AlexM", "author_url": "", "post_date": "2013-09-09T17:20:48.963000", "content": "which part?
", "votes": 0, "replies": [] }, { "id": 30544, "author_name": "Rohit Sivaprasad", "author_url": "", "post_date": "2013-09-09T17:14:36.160000", "content": "EDIT: My comment was not constructive. Also, I just read this,
\n[quote=AlexM;30543]
\nObviously you could do it without distributed computing (eg. sampling), but it seems that distributed computing would offer clear advantages.
\n[/quote]
\nPartly true. It might be slightly advantageous in terms of time. But then again the competition runs for about 3 months, with a limit of two submissions per day. I think that would be a limiting factor even if you had access to EC2 clusters. As said earlier in the other thread, it's about how you can make best use of limited resources.
\nOn another note, AWS gives away free credit if you are a student. You might want to look into that.
", "votes": 0, "replies": [] }, { "id": 35385, "author_name": "", "author_url": "", "post_date": "2013-11-27T21:54:40.837000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "30592": "", "30607": "", "30598": "", "30591": "", "30543": "", "35387": "", "33447": "", "33424": "", "31105": "", "31100": "", "31094": "", "31091": "", "30805": "", "30803": "", "30793": "", "30792": "", "30771": "", "30619": "", "30611": "", "30606": "", "30603": "", "30601": "", "30595": "", "30589": "", "30586": "", "30545": "", "30544": "", "35385": "" } }