{
  "id": 9566,
  "title": "Finally i got a hadoop machine :) :) ",
  "url": "/competitions/acquire-valued-shoppers-challenge/discussion/9566",
  "author_name": "",
  "post_date": "2014-06-26T03:01:24.460Z",
  "votes": 2,
  "comment_count": 9,
  "views": 2974,
  "content": "<p>Total number of rows in transactions dataset is 349655789, so i was thinking how to pre-process and extract features.</p>\n\n<p>Answer is hadoop and Hive Partitioning and bucketing..</p>\n\n<p>Soon i hope to be in top 10.</p>",
  "messages": [
    {
      "id": "49598",
      "postDate": "06/26/2014 03:01:24",
      "content": "<p>Total number of rows in transactions dataset is 349655789, so i was thinking how to pre-process and extract features.</p>\n\n<p>Answer is hadoop and Hive Partitioning and bucketing..</p>\n\n<p>Soon i hope to be in top 10.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49616",
      "postDate": "06/26/2014 11:27:41",
      "content": "<p>Sounds great! Good luck! Please update this thread. I'm really curious about how hadoop works here!&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49617",
      "postDate": "06/26/2014 11:37:23",
      "content": "<p>[quote=saikumar allaka;49598]</p>\n<p>Total number of rows in transactions dataset is 349655789, so i was thinking how to pre-process and extract features.</p>\n<p>Answer is hadoop and Hive Partitioning and bucketing..</p>\n<p>Soon i hope to be in top 10.</p>\n<p>[/quote]</p>\n\n<p>takes me 20-30 mins to extract ~300 features from the&nbsp;349655789 rows transaction data with less than 2 gb ram use.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49623",
      "postDate": "06/26/2014 13:29:39",
      "content": "<p>@Abhishek : Awesome&nbsp;:) :).. But i would like to implement it on Hadoop.. it should be a practice exercise atleast!!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49634",
      "postDate": "06/26/2014 16:02:42",
      "content": "<p>It sounds very interesting! If it works well, please share some amazing ideas!</p>\n<p>Thanks a lot in advance!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "50062",
      "postDate": "07/05/2014 13:14:01",
      "content": "<p>[quote=Abhishek;49617]</p>\n<p>takes me 20-30 mins to extract ~300 features from the&nbsp;349655789 rows transaction data with less than 2 gb ram use.&nbsp;</p>\n<p>[/quote]</p>\n\n<p>Abhishek, how many cores are in use ? Do you split the file for multi-processing ?</p>\n<p>Only streaming the 22gb file takes me more than 1 hour.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "50066",
      "postDate": "07/05/2014 16:06:24",
      "content": "<p>hint: SSD</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "50077",
      "postDate": "07/06/2014 14:54:47",
      "content": "<p>Plz share the result, thanks in advance</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "50078",
      "postDate": "07/06/2014 14:57:09",
      "content": "<p>[quote=Chitrasen;50062]</p>\n<p>[quote=Abhishek;49617]</p>\n<p>takes me 20-30 mins to extract ~300 features from the&nbsp;349655789 rows transaction data with less than 2 gb ram use.&nbsp;</p>\n<p>[/quote]</p>\n<p>Abhishek, how many cores are in use ? Do you split the file for multi-processing ?</p>\n<p>Only streaming the 22gb file takes me more than 1 hour.</p>\n<p>[/quote]</p>\n\n<p>without splitting and no multiprocessing. Yes, i have an SSD and that makes all the difference ;)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "50101",
      "postDate": "07/07/2014 15:32:23",
      "content": "<p>Abhishek: Just out of curiosity, do you use any framework&nbsp;to process the data or it is just plain python scripts?</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 49616,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "06/26/2014 11:27:41",
      "content": "<p>Sounds great! Good luck! Please update this thread. I'm really curious about how hadoop works here!&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49617,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "06/26/2014 11:37:23",
      "content": "<p>[quote=saikumar allaka;49598]</p>\n<p>Total number of rows in transactions dataset is 349655789, so i was thinking how to pre-process and extract features.</p>\n<p>Answer is hadoop and Hive Partitioning and bucketing..</p>\n<p>Soon i hope to be in top 10.</p>\n<p>[/quote]</p>\n\n<p>takes me 20-30 mins to extract ~300 features from the&nbsp;349655789 rows transaction data with less than 2 gb ram use.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49623,
      "author_name": "saikumarallaka",
      "author_url": "",
      "post_date": "06/26/2014 13:29:39",
      "content": "<p>@Abhishek : Awesome&nbsp;:) :).. But i would like to implement it on Hadoop.. it should be a practice exercise atleast!!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49634,
      "author_name": "shushanc",
      "author_url": "",
      "post_date": "06/26/2014 16:02:42",
      "content": "<p>It sounds very interesting! If it works well, please share some amazing ideas!</p>\n<p>Thanks a lot in advance!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 50062,
      "author_name": "chitrasen",
      "author_url": "",
      "post_date": "07/05/2014 13:14:01",
      "content": "<p>[quote=Abhishek;49617]</p>\n<p>takes me 20-30 mins to extract ~300 features from the&nbsp;349655789 rows transaction data with less than 2 gb ram use.&nbsp;</p>\n<p>[/quote]</p>\n\n<p>Abhishek, how many cores are in use ? Do you split the file for multi-processing ?</p>\n<p>Only streaming the 22gb file takes me more than 1 hour.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 50066,
      "author_name": "spimenov",
      "author_url": "",
      "post_date": "07/05/2014 16:06:24",
      "content": "<p>hint: SSD</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 50077,
      "author_name": "madrilen",
      "author_url": "",
      "post_date": "07/06/2014 14:54:47",
      "content": "<p>Plz share the result, thanks in advance</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 50078,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "07/06/2014 14:57:09",
      "content": "<p>[quote=Chitrasen;50062]</p>\n<p>[quote=Abhishek;49617]</p>\n<p>takes me 20-30 mins to extract ~300 features from the&nbsp;349655789 rows transaction data with less than 2 gb ram use.&nbsp;</p>\n<p>[/quote]</p>\n<p>Abhishek, how many cores are in use ? Do you split the file for multi-processing ?</p>\n<p>Only streaming the 22gb file takes me more than 1 hour.</p>\n<p>[/quote]</p>\n\n<p>without splitting and no multiprocessing. Yes, i have an SSD and that makes all the difference ;)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 50101,
      "author_name": "grigorydymov",
      "author_url": "",
      "post_date": "07/07/2014 15:32:23",
      "content": "<p>Abhishek: Just out of curiosity, do you use any framework&nbsp;to process the data or it is just plain python scripts?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "49598": "",
    "49616": "",
    "49617": "",
    "49623": "",
    "49634": "",
    "50062": "",
    "50066": "",
    "50077": "",
    "50078": "",
    "50101": ""
  },
  "source": "meta"
}