{
  "id": 10987,
  "title": "Dealing with data size",
  "url": "/competitions/avazu-ctr-prediction/discussion/10987",
  "author_name": "",
  "post_date": "2014-11-20T09:35:51.803Z",
  "votes": 3,
  "comment_count": 35,
  "views": 16954,
  "content": "<p>Hey guys</p>\n<p>I'm new to dealing with data of this size. When I try to run my code in R it ends and informs me there is insufficient RAM.</p>\n<p>How do you guys deal with this, besides the obvious of adding more RAM? Is there any technique that could get around this?</p>\n<p>I've tried breaking the data set into smaller sets and fitting models on each set, but R seems to store the data frames in its memory which takes me back to step 1.</p>\n<p>Thanks in advance.</p>",
  "messages": [
    {
      "id": "58504",
      "postDate": "11/20/2014 09:35:51",
      "content": "<p>Hey guys</p>\n<p>I'm new to dealing with data of this size. When I try to run my code in R it ends and informs me there is insufficient RAM.</p>\n<p>How do you guys deal with this, besides the obvious of adding more RAM? Is there any technique that could get around this?</p>\n<p>I've tried breaking the data set into smaller sets and fitting models on each set, but R seems to store the data frames in its memory which takes me back to step 1.</p>\n<p>Thanks in advance.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58525",
      "postDate": "11/20/2014 14:44:28",
      "content": "<p>@MLS7:</p>\n<ol>\n<li><span style=\"line-height: 1.4\">if you're trying to use read.table or somesuch directly,&nbsp;make sure to set&nbsp;stringsAsFactors = FALSE - this should save some memory already</span></li>\n<li><span style=\"line-height: 1.4\">i would recommend the ff package, which allows you to use data.frame functionality while storing stuff on disk</span></li>\n</ol>\n<p>hope this helps,</p>\n<p>K</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58540",
      "postDate": "11/20/2014 16:36:30",
      "content": "<p>Does anyone encode full features successfully? How much ram do you use?</p>\n<p>I can't encode even one column.. with 32 GB memory.. so sad for me : (</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58549",
      "postDate": "11/20/2014 17:00:26",
      "content": "<p>With such RAM you should do everything with this dataset even reading it all in memory</p>\n<p>[quote=rcarson;58540]</p>\n<p>Does anyone encodes full features successfully? How much ram do you use?</p>\n<p>I can't encode even one column.. with 32 GB memory.. so sad for me : (</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58555",
      "postDate": "11/20/2014 17:57:26",
      "content": "<p>@rcarson: could you elaborate a bit? i am a bit fuzzy on what you mean by &quot;encoding full features&quot; - &nbsp;what language are you using / what method?</p>\n\n<p>thanks,</p>\n<p>K</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58557",
      "postDate": "11/20/2014 18:05:32",
      "content": "<p>Hi, I'm using sklearn.feature_extraction.DictVectorizer to do one-hot encoding for categorical features. By full features, I mean every unique value of the feature counts and is encoded, no matter how rare it occurs.</p>\n<p>The code is like:</p>\n<p>from sklearn.feature_extraction import DictVectorizer</p>\n<p>vec= DictVectorizer()</p>\n<p>categorical_feature=[]</p>\n<p>for every feature:</p>\n<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if the feature is a categorical feature:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; train[feature] = map(str, train[feature])<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; test[feature] = map(str, test[feature])<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; categorical_feature.append(feature)</p>\n<p>X_sparse = vec.fit_transform(train[categorical_feature].T.to_dict().values())<br>X_test_sparse = vec.transform(test[categorical_feature].T.to_dict().values())</p>\n<p>where train and test are pandas dataframe</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58562",
      "postDate": "11/20/2014 19:21:47",
      "content": "<p>I don't have access to your amount of ram, so i had to no way to try this myself =&gt; i might be speculating here :-) but fwiw:</p>\n<p>- I seem to recall a script written by Pawel (http://www.kaggle.com/users/26782/pawe) sometime ago that converts all the hashes to integers =&gt; that saves some memory already. unfortunately, i can't find it at the moment - peruse the forum yourself, i think it had a name like 1_convert_raw_features.py or sth similar</p>\n<p>- have you tried using sparse matrices instead of pandas as input? I do believe they are acceptable as input&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58566",
      "postDate": "11/20/2014 19:44:46",
      "content": "<p>Thank you. I am about to write that script anyway. Integer should help.</p>\n<p>I'm actually encoding pandas data frames into sparse matrix.</p>\n<p>[quote=Konrad Banachewicz;58562]</p>\n<p>- have you tried using sparse matrices instead of pandas as input? I do believe they are acceptable as input&nbsp;</p>\n<p>[/quote]</p>\n<p>So how to get a sparse matrix input without encoding? :P</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58569",
      "postDate": "11/20/2014 20:24:49",
      "content": "<p>There are some tools here that might help. For example, in the pandas_extensions there is an algorithm that downsized each column to the smallest dtype necessary.</p>\n<p>https://github.com/gatapia/py_ml_utils</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58575",
      "postDate": "11/20/2014 22:01:07",
      "content": "<p>@Konrad: Here it is :). It reduces the train + test size from 7 GB to&nbsp;3.8 GB. It works on chunks so you don't have to actually load the whole dataset into memory. I used this to load the data to a database.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58576",
      "postDate": "11/20/2014 22:04:33",
      "content": "<p>@Pawel: thanks :-)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58578",
      "postDate": "11/20/2014 22:20:45",
      "content": "<p>[quote=rcarson;58566]</p>\n<p>Thank you. I am about to write that script anyway. Integer should help.</p>\n<p>I'm actually encoding pandas data frames into sparse matrix.</p>\n<p>[quote=Konrad Banachewicz;58562]</p>\n<p>- have you tried using sparse matrices instead of pandas as input? I do believe they are acceptable as input&nbsp;</p>\n<p>[/quote]</p>\n<p>So how to get a sparse matrix input without encoding? :P</p>\n<p>[/quote]</p>\n\n<p>I use Perl to make my matrices - I haven't for this competition yet but that was what I used for Aquired Value which was 22GB file.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58579",
      "postDate": "11/20/2014 23:12:05",
      "content": "<p>Why don't you use a java code to extract data from a csv file. Its running good on my PC.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59480",
      "postDate": "12/03/2014 16:58:42",
      "content": "<p>Hi ,</p>\n<p>How &nbsp; are you &nbsp;exploring &nbsp;this &nbsp; &nbsp; data? Are you using hadoop &nbsp; or &nbsp;some &nbsp;gpu &nbsp; computing?I &nbsp;used open.ff &nbsp;from &nbsp; R &nbsp; &nbsp;it &nbsp; read me &nbsp;the &nbsp;train set in &nbsp;a few minutes &nbsp;but &nbsp;when I am doing &nbsp; simple exploration of the &nbsp;data &nbsp;the &nbsp;computer &nbsp;is &nbsp;blocking.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59482",
      "postDate": "12/03/2014 17:29:48",
      "content": "<p>The dimensionality can be reduced by a lot, by removing one or two columns (check unique values per column). But this loses valuable information.</p>\n<p>Another Kaggle-specific hack&nbsp;to reduce dimensionality is to only keep the categorical features that appear in both the train set and the test set. Most models won't be able to learn from never-encountered-before features in the test set. And if a certain device_id appears in train set, but not in the test set, it only takes up space for a model to learn about it. Do note that for real projects it would be a huge shame to throw away predictability&nbsp;like this. You have to resort to this, when you are on a laptop, you want to avoid this, when you have a server (then look at semi-supervised learning).</p>\n<p>If you use Pawel's script to convert to integers, and do a check for features that appear in both train and test, you do not need a lot of memory to just encode them by integer, skipping the hashing trick. Again, in a real online learning setting such a trick would be not-done.</p>\n<p>[quote=Daia Alexandru;59480]</p>\n<p>How &nbsp; are you &nbsp;exploring &nbsp;this &nbsp;data?</p>\n<p>[/quote]</p>\n<p>One column or chunk at a time. Mostly in <a href=\"http://pandas.pydata.org/pandas-docs/stable/io.html\">Pandas</a>.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59590",
      "postDate": "12/05/2014 02:23:43",
      "content": "<p>Sort of new to data analytics so I might have missed it if you guys already answered this above, but I am trying to import and then wok with the dataset in R. &nbsp;I used 7-fle zip manager to extract the .gz file to a .csv file but can't import it into R or even open it in notepad or excel. &nbsp;I have 6 GB of RAM. &nbsp;How should I go about working with the data? &nbsp;Anyone have sample code for importing only part of a .csv? &nbsp;I also heard that using a paging file might be able to increase working RAM?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59610",
      "postDate": "12/05/2014 08:16:04",
      "content": "<p>@swimmer006: 6GB of RAM is too little in my opinion to deal with data that big. At least if you think about playing around with the dataset.The only option for you is online learning - one observation at a time or a chunk of data (see Triskelion post).&nbsp;Another option is to use a database and analyze it at a database level. I often load the data to the database and use some R package to load queries as dataframes. My workflow is something like this</p>\n<p>df = Q(&quot;SELECT hour, AVG(click) as avg_click FROM data WHERE day = 25 GROUP BY 1&quot;)</p>\n<p>I have a function Q (as for query) which connects to a database and loads the results of the query in a dataframe. Then you can easily plot the results.</p>\n<p>PS. Don't rely on paging - it takes forever to finish some operations.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59612",
      "postDate": "12/05/2014 08:35:27",
      "content": "<p>[quote=swimmer006;59590]</p>\n<p>Sort of new to data analytics so I might have missed it if you guys already answered this above, but I am trying to import and then wok with the dataset in R. &nbsp;I used 7-fle zip manager to extract the .gz file to a .csv file but can't import it into R or even open it in notepad or excel. &nbsp;I have 6 GB of RAM. &nbsp;How should I go about working with the data? &nbsp;Anyone have sample code for importing only part of a .csv? &nbsp;I also heard that using a paging file might be able to increase working RAM?</p>\n<p>[/quote]</p>\n<p>Use :</p>\n<p>library(&quot;ff&quot;)</p>\n<p>x&lt;- read.csv.ffdf(file=&quot;file.csv&quot;, header=F, VERBOSE=TRUE, first.rows=10000, next.rows=50000, colClasses=NA)</p>\n<p>It &nbsp;worked &nbsp;for me(4GB RAM) &nbsp;but &nbsp;when I wan't to manipulate or exlpore this &nbsp;data my Rstduio &nbsp;blocks &nbsp; OS.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59787",
      "postDate": "12/08/2014 14:09:32",
      "content": "<p>I have a very inexpensive way to spin a <a href=\"http://spark.apache.org/\" target=\"_blank\">spark</a> cluster on AWS. It allows me to run a 100 iterations of logistic regression in less than 10 min with all training data. I do this through an ipython notebook connected to the cluster that lets me run&nbsp;<a href=\"https://spark.apache.org/docs/latest/mllib-guide.html\" target=\"_blank\">MLlibs</a> algorithms and do data exploration with sql. I'll be happy to share with the anyone interested in giving it a try. Contact me through kaggle (click on my picture name below to get to my kaggle profile, then clicking on the contact tab should allow you to send me a message) and I will get back to you with instructions.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59788",
      "postDate": "12/08/2014 14:17:03",
      "content": "<p>Paulo, I very interested to try spark, MLlibs in python. How to see this example? I can't contact with you, because</p>\n<p>&quot;You need to obtain more points in competitions before you'll be able to contact other Kaggle users.&quot;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59866",
      "postDate": "12/09/2014 05:25:57",
      "content": "<p>[quote=Paulo;59787]</p>\n<p>I have a very inexpensive way to spin a <a href=\"http://spark.apache.org/\" target=\"_blank\">spark</a> cluster on AWS. It allows me to run a 100 iterations of logistic regression in less than 10 min with all training data. I do this through an ipython notebook connected to the cluster that lets me run&nbsp;<a href=\"https://spark.apache.org/docs/latest/mllib-guide.html\" target=\"_blank\">MLlibs</a> algorithms and do data exploration with sql. I'll be happy to share with the anyone interested in giving it a try. Contact me through kaggle (click on my picture name below to get to my kaggle profile, then clicking on the contact tab should allow you to send me a message) and I will get back to you with instructions.</p>\n<p>[/quote]</p>\n\n<p>Paulo, I very interested to try spark, MLlibs in python. How to see this example? I can't contact with you, because</p>\n<p>&quot;You need to obtain more points in competitions before you'll be able to contact other Kaggle users.&quot;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59881",
      "postDate": "12/09/2014 12:16:00",
      "content": "<p>why not sampling the data?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59882",
      "postDate": "12/09/2014 12:20:40",
      "content": "<p>Well,&nbsp;when&nbsp;sampling you can miss some data behavior.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59883",
      "postDate": "12/09/2014 12:50:59",
      "content": "<p>[quote=dirceusemighini;59882]</p>\n<p>Well,&nbsp;when&nbsp;sampling you can miss some data behavior.</p>\n<p>[/quote]</p>\n\n<p>so if i have 20M, 100M and 500M records, i should use them all for modeling... doesn't make seance to me.</p>\n<p>and with some suggestions here about keeping only values that are part of the &quot;train&quot; and &quot;test&quot;...</p>\n<p>which i find a&nbsp;bit strange advice.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59884",
      "postDate": "12/09/2014 13:05:17",
      "content": "<p>[quote=Tomer kalimi;59883]</p>\n<p>[quote=dirceusemighini;59882]</p>\n<p>Well, when sampling you can miss some data behavior.</p>\n<p>[/quote]</p>\n<p>so if i have 20M, 100M and 500M records, i should use them all for modeling... doesn't make seance to me.</p>\n<p>[/quote]</p>\n<p><code>&quot;Can miss&quot; != &quot;must use all data&quot;.</code></p>\n<p>Use as much as you feels comfort.</p>\n<p>Sometimes it's very hard to deal with all data. But general idea is <em>&quot;more is better&quot;</em>.</p>\n\n<p>[quote=Tomer kalimi;59883]</p>\n<p>and with some suggestions here about keeping only values that are part of the &quot;train&quot; and &quot;test&quot;...</p>\n<p>which i find a bit strange advice.</p>\n<p>[/quote]</p>\n<p>It's simple. In test there are a lot of SiteId's (for example) you've never seen in train. And vice versa - train contains some feature values that are not appear in test.</p>\n<p>So, for <em>this pair of train and test</em>&nbsp;such feature values are just useless.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59888",
      "postDate": "12/09/2014 13:42:26",
      "content": "<p>I'd be interested in seeing something like a plot of LB score versus amount of data used after the competition for the final solutions. I don't have a good intuition for e.g. how much extra benefit you get using 40 million training rows instead of 4 million training rows for various algorithms.</p>\n<p>I was still getting some improvement going from e.g. 800k rows to 4 million rows when playing around with libFM. On the other hand, for SVMs I simply can't run more than 15k rows at a time because of how the algorithm scales - and my results with SVMs aren't particularly good, possibly as a consequence of that. If I could actually push all 40 million rows through the SVMs, I wonder how good they'd be?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59903",
      "postDate": "12/09/2014 20:19:53",
      "content": "<p>[quote=Abhishek Dokania;59866]</p>\n<p>[quote=Paulo;59787]</p>\n<p>I have a very inexpensive way to spin a <a href=\"http://spark.apache.org/\" target=\"_blank\">spark</a> cluster on AWS. It allows me to run a 100 iterations of logistic regression in less than 10 min with all training data. I do this through an ipython notebook connected to the cluster that lets me run&nbsp;<a href=\"https://spark.apache.org/docs/latest/mllib-guide.html\" target=\"_blank\">MLlibs</a> algorithms and do data exploration with sql. I'll be happy to share with the anyone interested in giving it a try. Contact me through kaggle (click on my picture name below to get to my kaggle profile, then clicking on the contact tab should allow you to send me a message) and I will get back to you with instructions.</p>\n<p>[/quote]</p>\n<p>Paulo, I very interested to try spark, MLlibs in python. How to see this example? I can't contact with you, because</p>\n<p>&quot;You need to obtain more points in competitions before you'll be able to contact other Kaggle users.&quot;</p>\n<p>[/quote]</p>\n\n<p>Hi Paulo:</p>\n<p>Kaggle doesn't allow me to email you either. &nbsp;Can you contact me and allow me to look at your code? &nbsp;That would be much appreciated.</p>\n<p>Best regards,</p>\n<p>Jennifer</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59904",
      "postDate": "12/09/2014 21:01:23",
      "content": "<p>[quote=Nicholas Guttenberg;59888]</p>\n<p>I'd be interested in seeing something like a plot of LB score versus amount of data used after the competition for the final solutions. I don't have a good intuition for e.g. how much extra benefit you get using 40 million training rows instead of 4 million training rows for various algorithms.</p>\n<p>[/quote]</p>\n<p>That would be interesting to see. &nbsp;In this <a href=\"http://quinonero.net/Publications/predicting-clicks-facebook.pdf\">Ads CTR prediction paper He et al.</a>&nbsp;plotted the training accuracy v.s. fraction of training data subsampled (Figure 10). &nbsp;It seems that using 10% data the performance reduction&nbsp;was only 1%.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59906",
      "postDate": "12/09/2014 21:02:34",
      "content": "<p>[quote=JD Davis;59903]</p>\n<p>[quote=Abhishek Dokania;59866]</p>\n<p>[quote=Paulo;59787]</p>\n<p>I have a very inexpensive way to spin a <a href=\"http://spark.apache.org/\" target=\"_blank\">spark</a> cluster on AWS. It allows me to run a 100 iterations of logistic regression in less than 10 min with all training data. I do this through an ipython notebook connected to the cluster that lets me run&nbsp;<a href=\"https://spark.apache.org/docs/latest/mllib-guide.html\" target=\"_blank\">MLlibs</a> algorithms and do data exploration with sql. I'll be happy to share with the anyone interested in giving it a try. Contact me through kaggle (click on my picture name below to get to my kaggle profile, then clicking on the contact tab should allow you to send me a message) and I will get back to you with instructions.</p>\n<p>[/quote]</p>\n<p>Paulo, I very interested to try spark, MLlibs in python. How to see this example? I can't contact with you, because</p>\n<p>&quot;You need to obtain more points in competitions before you'll be able to contact other Kaggle users.&quot;</p>\n<p>[/quote]</p>\n<p>Hi Paulo:</p>\n<p>Kaggle doesn't allow me to email you either. &nbsp;Can you contact me and allow me to look at your code? &nbsp;That would be much appreciated.</p>\n<p>Best regards,</p>\n<p>Jennifer</p>\n<p>[/quote]</p>\n\n<p>I'd also be interested in learning how to do this - more specifically how you got sql to work. &nbsp;Is this the sparksql that comes pre-packaged?</p>\n\n<p>@Jennifer - I found some resources to get started with connecting pyspark with AWS:</p>\n<p>http://blog.cloudera.com/blog/2014/08/how-to-use-ipython-notebook-with-apache-spark/</p>\n<p>http://nbviewer.ipython.org/gist/JoshRosen/6856670</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "61557",
      "postDate": "12/19/2014 23:22:15",
      "content": "<p>@Pawel: which db do you use and what criteria did you use to select it?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "61560",
      "postDate": "12/19/2014 23:55:09",
      "content": "<p>[quote=zephyro;61557]</p>\n<p>@Pawel: which db do you use and what criteria did you use to select it?</p>\n<p>[/quote]</p>\n<p>I use PostgreSQL. I used MySQL in the past for similar tasks but I find PostgreSQL much more developer friendly. But this is just a matter of taste. I guess any reasonable database engine could be ok.</p>\n<p>R has really excellent support for querying the databases. Personally I always define a function Q which&nbsp;I use like this:</p>\n<p>df = Q(&quot;SELECT * FROM table&quot;)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "61561",
      "postDate": "12/20/2014 00:14:45",
      "content": "<p>@Pawel: Thanks, I'm running PostgreSQL too. I prefer to use Python rather than R though as I find it better for large datasets. For the hell of it I was thinking of giving PL/Python a try, have you (or anyone else) used it?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "61615",
      "postDate": "12/21/2014 12:14:57",
      "content": "<p>can we use sas?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "61637",
      "postDate": "12/21/2014 21:31:45",
      "content": "<p>Hey... thanks a bunch... saved me time!</p>\n<p>Is not the last line in the script supposed to be:</p>\n<p>_ = write_translated( <span style=\"text-decoration: line-through\">TRAIN</span> TEST_DATA,MERGED_DATA,ids,mode=&quot;a&quot;,start_id = max_id)</p>\n<p>Thanks!</p>\n<p>[quote=Pawe&#322;;58575]</p>\n<p>@Konrad: Here it is :). It reduces the train + test size from 7 GB to&nbsp;3.8 GB. It works on chunks so you don't have to actually load the whole dataset into memory. I used this to load the data to a database.</p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "61638",
      "postDate": "12/21/2014 21:37:07",
      "content": "<p>Yes it should. Sorry for that</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "66544",
      "postDate": "03/16/2015 22:12:14",
      "content": "<p>A few of you showed interested in running spark/ipython on aws so I did a <a href=\"http://www.bigdatamodeling.org/2015/03/using-sparknotebook-for-kaggle-data.html\">little write up </a></p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 58525,
      "author_name": "konradb",
      "author_url": "",
      "post_date": "11/20/2014 14:44:28",
      "content": "<p>@MLS7:</p>\n<ol>\n<li><span style=\"line-height: 1.4\">if you're trying to use read.table or somesuch directly,&nbsp;make sure to set&nbsp;stringsAsFactors = FALSE - this should save some memory already</span></li>\n<li><span style=\"line-height: 1.4\">i would recommend the ff package, which allows you to use data.frame functionality while storing stuff on disk</span></li>\n</ol>\n<p>hope this helps,</p>\n<p>K</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58540,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "11/20/2014 16:36:30",
      "content": "<p>Does anyone encode full features successfully? How much ram do you use?</p>\n<p>I can't encode even one column.. with 32 GB memory.. so sad for me : (</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58549,
      "author_name": "herimanitra",
      "author_url": "",
      "post_date": "11/20/2014 17:00:26",
      "content": "<p>With such RAM you should do everything with this dataset even reading it all in memory</p>\n<p>[quote=rcarson;58540]</p>\n<p>Does anyone encodes full features successfully? How much ram do you use?</p>\n<p>I can't encode even one column.. with 32 GB memory.. so sad for me : (</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58555,
      "author_name": "konradb",
      "author_url": "",
      "post_date": "11/20/2014 17:57:26",
      "content": "<p>@rcarson: could you elaborate a bit? i am a bit fuzzy on what you mean by &quot;encoding full features&quot; - &nbsp;what language are you using / what method?</p>\n\n<p>thanks,</p>\n<p>K</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58557,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "11/20/2014 18:05:32",
      "content": "<p>Hi, I'm using sklearn.feature_extraction.DictVectorizer to do one-hot encoding for categorical features. By full features, I mean every unique value of the feature counts and is encoded, no matter how rare it occurs.</p>\n<p>The code is like:</p>\n<p>from sklearn.feature_extraction import DictVectorizer</p>\n<p>vec= DictVectorizer()</p>\n<p>categorical_feature=[]</p>\n<p>for every feature:</p>\n<p>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if the feature is a categorical feature:<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; train[feature] = map(str, train[feature])<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; test[feature] = map(str, test[feature])<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; categorical_feature.append(feature)</p>\n<p>X_sparse = vec.fit_transform(train[categorical_feature].T.to_dict().values())<br>X_test_sparse = vec.transform(test[categorical_feature].T.to_dict().values())</p>\n<p>where train and test are pandas dataframe</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58562,
      "author_name": "konradb",
      "author_url": "",
      "post_date": "11/20/2014 19:21:47",
      "content": "<p>I don't have access to your amount of ram, so i had to no way to try this myself =&gt; i might be speculating here :-) but fwiw:</p>\n<p>- I seem to recall a script written by Pawel (http://www.kaggle.com/users/26782/pawe) sometime ago that converts all the hashes to integers =&gt; that saves some memory already. unfortunately, i can't find it at the moment - peruse the forum yourself, i think it had a name like 1_convert_raw_features.py or sth similar</p>\n<p>- have you tried using sparse matrices instead of pandas as input? I do believe they are acceptable as input&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58566,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "11/20/2014 19:44:46",
      "content": "<p>Thank you. I am about to write that script anyway. Integer should help.</p>\n<p>I'm actually encoding pandas data frames into sparse matrix.</p>\n<p>[quote=Konrad Banachewicz;58562]</p>\n<p>- have you tried using sparse matrices instead of pandas as input? I do believe they are acceptable as input&nbsp;</p>\n<p>[/quote]</p>\n<p>So how to get a sparse matrix input without encoding? :P</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58569,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "11/20/2014 20:24:49",
      "content": "<p>There are some tools here that might help. For example, in the pandas_extensions there is an algorithm that downsized each column to the smallest dtype necessary.</p>\n<p>https://github.com/gatapia/py_ml_utils</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58575,
      "author_name": "paweljankiewicz",
      "author_url": "",
      "post_date": "11/20/2014 22:01:07",
      "content": "<p>@Konrad: Here it is :). It reduces the train + test size from 7 GB to&nbsp;3.8 GB. It works on chunks so you don't have to actually load the whole dataset into memory. I used this to load the data to a database.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58576,
      "author_name": "konradb",
      "author_url": "",
      "post_date": "11/20/2014 22:04:33",
      "content": "<p>@Pawel: thanks :-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58578,
      "author_name": "domcastro",
      "author_url": "",
      "post_date": "11/20/2014 22:20:45",
      "content": "<p>[quote=rcarson;58566]</p>\n<p>Thank you. I am about to write that script anyway. Integer should help.</p>\n<p>I'm actually encoding pandas data frames into sparse matrix.</p>\n<p>[quote=Konrad Banachewicz;58562]</p>\n<p>- have you tried using sparse matrices instead of pandas as input? I do believe they are acceptable as input&nbsp;</p>\n<p>[/quote]</p>\n<p>So how to get a sparse matrix input without encoding? :P</p>\n<p>[/quote]</p>\n\n<p>I use Perl to make my matrices - I haven't for this competition yet but that was what I used for Aquired Value which was 22GB file.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58579,
      "author_name": "saurabhsaxena1",
      "author_url": "",
      "post_date": "11/20/2014 23:12:05",
      "content": "<p>Why don't you use a java code to extract data from a csv file. Its running good on my PC.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59480,
      "author_name": "",
      "author_url": "",
      "post_date": "12/03/2014 16:58:42",
      "content": "<p>Hi ,</p>\n<p>How &nbsp; are you &nbsp;exploring &nbsp;this &nbsp; &nbsp; data? Are you using hadoop &nbsp; or &nbsp;some &nbsp;gpu &nbsp; computing?I &nbsp;used open.ff &nbsp;from &nbsp; R &nbsp; &nbsp;it &nbsp; read me &nbsp;the &nbsp;train set in &nbsp;a few minutes &nbsp;but &nbsp;when I am doing &nbsp; simple exploration of the &nbsp;data &nbsp;the &nbsp;computer &nbsp;is &nbsp;blocking.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59482,
      "author_name": "triskelion",
      "author_url": "",
      "post_date": "12/03/2014 17:29:48",
      "content": "<p>The dimensionality can be reduced by a lot, by removing one or two columns (check unique values per column). But this loses valuable information.</p>\n<p>Another Kaggle-specific hack&nbsp;to reduce dimensionality is to only keep the categorical features that appear in both the train set and the test set. Most models won't be able to learn from never-encountered-before features in the test set. And if a certain device_id appears in train set, but not in the test set, it only takes up space for a model to learn about it. Do note that for real projects it would be a huge shame to throw away predictability&nbsp;like this. You have to resort to this, when you are on a laptop, you want to avoid this, when you have a server (then look at semi-supervised learning).</p>\n<p>If you use Pawel's script to convert to integers, and do a check for features that appear in both train and test, you do not need a lot of memory to just encode them by integer, skipping the hashing trick. Again, in a real online learning setting such a trick would be not-done.</p>\n<p>[quote=Daia Alexandru;59480]</p>\n<p>How &nbsp; are you &nbsp;exploring &nbsp;this &nbsp;data?</p>\n<p>[/quote]</p>\n<p>One column or chunk at a time. Mostly in <a href=\"http://pandas.pydata.org/pandas-docs/stable/io.html\">Pandas</a>.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59590,
      "author_name": "benmiller",
      "author_url": "",
      "post_date": "12/05/2014 02:23:43",
      "content": "<p>Sort of new to data analytics so I might have missed it if you guys already answered this above, but I am trying to import and then wok with the dataset in R. &nbsp;I used 7-fle zip manager to extract the .gz file to a .csv file but can't import it into R or even open it in notepad or excel. &nbsp;I have 6 GB of RAM. &nbsp;How should I go about working with the data? &nbsp;Anyone have sample code for importing only part of a .csv? &nbsp;I also heard that using a paging file might be able to increase working RAM?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59610,
      "author_name": "paweljankiewicz",
      "author_url": "",
      "post_date": "12/05/2014 08:16:04",
      "content": "<p>@swimmer006: 6GB of RAM is too little in my opinion to deal with data that big. At least if you think about playing around with the dataset.The only option for you is online learning - one observation at a time or a chunk of data (see Triskelion post).&nbsp;Another option is to use a database and analyze it at a database level. I often load the data to the database and use some R package to load queries as dataframes. My workflow is something like this</p>\n<p>df = Q(&quot;SELECT hour, AVG(click) as avg_click FROM data WHERE day = 25 GROUP BY 1&quot;)</p>\n<p>I have a function Q (as for query) which connects to a database and loads the results of the query in a dataframe. Then you can easily plot the results.</p>\n<p>PS. Don't rely on paging - it takes forever to finish some operations.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59612,
      "author_name": "",
      "author_url": "",
      "post_date": "12/05/2014 08:35:27",
      "content": "<p>[quote=swimmer006;59590]</p>\n<p>Sort of new to data analytics so I might have missed it if you guys already answered this above, but I am trying to import and then wok with the dataset in R. &nbsp;I used 7-fle zip manager to extract the .gz file to a .csv file but can't import it into R or even open it in notepad or excel. &nbsp;I have 6 GB of RAM. &nbsp;How should I go about working with the data? &nbsp;Anyone have sample code for importing only part of a .csv? &nbsp;I also heard that using a paging file might be able to increase working RAM?</p>\n<p>[/quote]</p>\n<p>Use :</p>\n<p>library(&quot;ff&quot;)</p>\n<p>x&lt;- read.csv.ffdf(file=&quot;file.csv&quot;, header=F, VERBOSE=TRUE, first.rows=10000, next.rows=50000, colClasses=NA)</p>\n<p>It &nbsp;worked &nbsp;for me(4GB RAM) &nbsp;but &nbsp;when I wan't to manipulate or exlpore this &nbsp;data my Rstduio &nbsp;blocks &nbsp; OS.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59787,
      "author_name": "paulomagalhaes",
      "author_url": "",
      "post_date": "12/08/2014 14:09:32",
      "content": "<p>I have a very inexpensive way to spin a <a href=\"http://spark.apache.org/\" target=\"_blank\">spark</a> cluster on AWS. It allows me to run a 100 iterations of logistic regression in less than 10 min with all training data. I do this through an ipython notebook connected to the cluster that lets me run&nbsp;<a href=\"https://spark.apache.org/docs/latest/mllib-guide.html\" target=\"_blank\">MLlibs</a> algorithms and do data exploration with sql. I'll be happy to share with the anyone interested in giving it a try. Contact me through kaggle (click on my picture name below to get to my kaggle profile, then clicking on the contact tab should allow you to send me a message) and I will get back to you with instructions.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59788,
      "author_name": "shalnovv",
      "author_url": "",
      "post_date": "12/08/2014 14:17:03",
      "content": "<p>Paulo, I very interested to try spark, MLlibs in python. How to see this example? I can't contact with you, because</p>\n<p>&quot;You need to obtain more points in competitions before you'll be able to contact other Kaggle users.&quot;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59866,
      "author_name": "abhi19",
      "author_url": "",
      "post_date": "12/09/2014 05:25:57",
      "content": "<p>[quote=Paulo;59787]</p>\n<p>I have a very inexpensive way to spin a <a href=\"http://spark.apache.org/\" target=\"_blank\">spark</a> cluster on AWS. It allows me to run a 100 iterations of logistic regression in less than 10 min with all training data. I do this through an ipython notebook connected to the cluster that lets me run&nbsp;<a href=\"https://spark.apache.org/docs/latest/mllib-guide.html\" target=\"_blank\">MLlibs</a> algorithms and do data exploration with sql. I'll be happy to share with the anyone interested in giving it a try. Contact me through kaggle (click on my picture name below to get to my kaggle profile, then clicking on the contact tab should allow you to send me a message) and I will get back to you with instructions.</p>\n<p>[/quote]</p>\n\n<p>Paulo, I very interested to try spark, MLlibs in python. How to see this example? I can't contact with you, because</p>\n<p>&quot;You need to obtain more points in competitions before you'll be able to contact other Kaggle users.&quot;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59881,
      "author_name": "tomerkalimi",
      "author_url": "",
      "post_date": "12/09/2014 12:16:00",
      "content": "<p>why not sampling the data?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59882,
      "author_name": "dirceusemighini",
      "author_url": "",
      "post_date": "12/09/2014 12:20:40",
      "content": "<p>Well,&nbsp;when&nbsp;sampling you can miss some data behavior.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59883,
      "author_name": "tomerkalimi",
      "author_url": "",
      "post_date": "12/09/2014 12:50:59",
      "content": "<p>[quote=dirceusemighini;59882]</p>\n<p>Well,&nbsp;when&nbsp;sampling you can miss some data behavior.</p>\n<p>[/quote]</p>\n\n<p>so if i have 20M, 100M and 500M records, i should use them all for modeling... doesn't make seance to me.</p>\n<p>and with some suggestions here about keeping only values that are part of the &quot;train&quot; and &quot;test&quot;...</p>\n<p>which i find a&nbsp;bit strange advice.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59884,
      "author_name": "mikhailtrofimov",
      "author_url": "",
      "post_date": "12/09/2014 13:05:17",
      "content": "<p>[quote=Tomer kalimi;59883]</p>\n<p>[quote=dirceusemighini;59882]</p>\n<p>Well, when sampling you can miss some data behavior.</p>\n<p>[/quote]</p>\n<p>so if i have 20M, 100M and 500M records, i should use them all for modeling... doesn't make seance to me.</p>\n<p>[/quote]</p>\n<p><code>&quot;Can miss&quot; != &quot;must use all data&quot;.</code></p>\n<p>Use as much as you feels comfort.</p>\n<p>Sometimes it's very hard to deal with all data. But general idea is <em>&quot;more is better&quot;</em>.</p>\n\n<p>[quote=Tomer kalimi;59883]</p>\n<p>and with some suggestions here about keeping only values that are part of the &quot;train&quot; and &quot;test&quot;...</p>\n<p>which i find a bit strange advice.</p>\n<p>[/quote]</p>\n<p>It's simple. In test there are a lot of SiteId's (for example) you've never seen in train. And vice versa - train contains some feature values that are not appear in test.</p>\n<p>So, for <em>this pair of train and test</em>&nbsp;such feature values are just useless.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59888,
      "author_name": "ngutten",
      "author_url": "",
      "post_date": "12/09/2014 13:42:26",
      "content": "<p>I'd be interested in seeing something like a plot of LB score versus amount of data used after the competition for the final solutions. I don't have a good intuition for e.g. how much extra benefit you get using 40 million training rows instead of 4 million training rows for various algorithms.</p>\n<p>I was still getting some improvement going from e.g. 800k rows to 4 million rows when playing around with libFM. On the other hand, for SVMs I simply can't run more than 15k rows at a time because of how the algorithm scales - and my results with SVMs aren't particularly good, possibly as a consequence of that. If I could actually push all 40 million rows through the SVMs, I wonder how good they'd be?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59903,
      "author_name": "jddavis100",
      "author_url": "",
      "post_date": "12/09/2014 20:19:53",
      "content": "<p>[quote=Abhishek Dokania;59866]</p>\n<p>[quote=Paulo;59787]</p>\n<p>I have a very inexpensive way to spin a <a href=\"http://spark.apache.org/\" target=\"_blank\">spark</a> cluster on AWS. It allows me to run a 100 iterations of logistic regression in less than 10 min with all training data. I do this through an ipython notebook connected to the cluster that lets me run&nbsp;<a href=\"https://spark.apache.org/docs/latest/mllib-guide.html\" target=\"_blank\">MLlibs</a> algorithms and do data exploration with sql. I'll be happy to share with the anyone interested in giving it a try. Contact me through kaggle (click on my picture name below to get to my kaggle profile, then clicking on the contact tab should allow you to send me a message) and I will get back to you with instructions.</p>\n<p>[/quote]</p>\n<p>Paulo, I very interested to try spark, MLlibs in python. How to see this example? I can't contact with you, because</p>\n<p>&quot;You need to obtain more points in competitions before you'll be able to contact other Kaggle users.&quot;</p>\n<p>[/quote]</p>\n\n<p>Hi Paulo:</p>\n<p>Kaggle doesn't allow me to email you either. &nbsp;Can you contact me and allow me to look at your code? &nbsp;That would be much appreciated.</p>\n<p>Best regards,</p>\n<p>Jennifer</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59904,
      "author_name": "derekzh",
      "author_url": "",
      "post_date": "12/09/2014 21:01:23",
      "content": "<p>[quote=Nicholas Guttenberg;59888]</p>\n<p>I'd be interested in seeing something like a plot of LB score versus amount of data used after the competition for the final solutions. I don't have a good intuition for e.g. how much extra benefit you get using 40 million training rows instead of 4 million training rows for various algorithms.</p>\n<p>[/quote]</p>\n<p>That would be interesting to see. &nbsp;In this <a href=\"http://quinonero.net/Publications/predicting-clicks-facebook.pdf\">Ads CTR prediction paper He et al.</a>&nbsp;plotted the training accuracy v.s. fraction of training data subsampled (Figure 10). &nbsp;It seems that using 10% data the performance reduction&nbsp;was only 1%.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59906,
      "author_name": "sneakyfox",
      "author_url": "",
      "post_date": "12/09/2014 21:02:34",
      "content": "<p>[quote=JD Davis;59903]</p>\n<p>[quote=Abhishek Dokania;59866]</p>\n<p>[quote=Paulo;59787]</p>\n<p>I have a very inexpensive way to spin a <a href=\"http://spark.apache.org/\" target=\"_blank\">spark</a> cluster on AWS. It allows me to run a 100 iterations of logistic regression in less than 10 min with all training data. I do this through an ipython notebook connected to the cluster that lets me run&nbsp;<a href=\"https://spark.apache.org/docs/latest/mllib-guide.html\" target=\"_blank\">MLlibs</a> algorithms and do data exploration with sql. I'll be happy to share with the anyone interested in giving it a try. Contact me through kaggle (click on my picture name below to get to my kaggle profile, then clicking on the contact tab should allow you to send me a message) and I will get back to you with instructions.</p>\n<p>[/quote]</p>\n<p>Paulo, I very interested to try spark, MLlibs in python. How to see this example? I can't contact with you, because</p>\n<p>&quot;You need to obtain more points in competitions before you'll be able to contact other Kaggle users.&quot;</p>\n<p>[/quote]</p>\n<p>Hi Paulo:</p>\n<p>Kaggle doesn't allow me to email you either. &nbsp;Can you contact me and allow me to look at your code? &nbsp;That would be much appreciated.</p>\n<p>Best regards,</p>\n<p>Jennifer</p>\n<p>[/quote]</p>\n\n<p>I'd also be interested in learning how to do this - more specifically how you got sql to work. &nbsp;Is this the sparksql that comes pre-packaged?</p>\n\n<p>@Jennifer - I found some resources to get started with connecting pyspark with AWS:</p>\n<p>http://blog.cloudera.com/blog/2014/08/how-to-use-ipython-notebook-with-apache-spark/</p>\n<p>http://nbviewer.ipython.org/gist/JoshRosen/6856670</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 61557,
      "author_name": "zephyro",
      "author_url": "",
      "post_date": "12/19/2014 23:22:15",
      "content": "<p>@Pawel: which db do you use and what criteria did you use to select it?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 61560,
      "author_name": "paweljankiewicz",
      "author_url": "",
      "post_date": "12/19/2014 23:55:09",
      "content": "<p>[quote=zephyro;61557]</p>\n<p>@Pawel: which db do you use and what criteria did you use to select it?</p>\n<p>[/quote]</p>\n<p>I use PostgreSQL. I used MySQL in the past for similar tasks but I find PostgreSQL much more developer friendly. But this is just a matter of taste. I guess any reasonable database engine could be ok.</p>\n<p>R has really excellent support for querying the databases. Personally I always define a function Q which&nbsp;I use like this:</p>\n<p>df = Q(&quot;SELECT * FROM table&quot;)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 61561,
      "author_name": "zephyro",
      "author_url": "",
      "post_date": "12/20/2014 00:14:45",
      "content": "<p>@Pawel: Thanks, I'm running PostgreSQL too. I prefer to use Python rather than R though as I find it better for large datasets. For the hell of it I was thinking of giving PL/Python a try, have you (or anyone else) used it?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 61615,
      "author_name": "mearul",
      "author_url": "",
      "post_date": "12/21/2014 12:14:57",
      "content": "<p>can we use sas?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 61637,
      "author_name": "ipower2",
      "author_url": "",
      "post_date": "12/21/2014 21:31:45",
      "content": "<p>Hey... thanks a bunch... saved me time!</p>\n<p>Is not the last line in the script supposed to be:</p>\n<p>_ = write_translated( <span style=\"text-decoration: line-through\">TRAIN</span> TEST_DATA,MERGED_DATA,ids,mode=&quot;a&quot;,start_id = max_id)</p>\n<p>Thanks!</p>\n<p>[quote=Pawe&#322;;58575]</p>\n<p>@Konrad: Here it is :). It reduces the train + test size from 7 GB to&nbsp;3.8 GB. It works on chunks so you don't have to actually load the whole dataset into memory. I used this to load the data to a database.</p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 61638,
      "author_name": "paweljankiewicz",
      "author_url": "",
      "post_date": "12/21/2014 21:37:07",
      "content": "<p>Yes it should. Sorry for that</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 66544,
      "author_name": "paulomagalhaes",
      "author_url": "",
      "post_date": "03/16/2015 22:12:14",
      "content": "<p>A few of you showed interested in running spark/ipython on aws so I did a <a href=\"http://www.bigdatamodeling.org/2015/03/using-sparknotebook-for-kaggle-data.html\">little write up </a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "58504": "",
    "58525": "",
    "58540": "",
    "58549": "",
    "58555": "",
    "58557": "",
    "58562": "",
    "58566": "",
    "58569": "",
    "58575": "",
    "58576": "",
    "58578": "",
    "58579": "",
    "59480": "",
    "59482": "",
    "59590": "",
    "59610": "",
    "59612": "",
    "59787": "",
    "59788": "",
    "59866": "",
    "59881": "",
    "59882": "",
    "59883": "",
    "59884": "",
    "59888": "",
    "59903": "",
    "59904": "",
    "59906": "",
    "61557": "",
    "61560": "",
    "61561": "",
    "61615": "",
    "61637": "",
    "61638": "",
    "66544": ""
  },
  "source": "meta"
}