{
  "id": 7689,
  "title": "Which Data Mining Tool for handling rows selectively!",
  "url": "/competitions/acquire-valued-shoppers-challenge/discussion/7689",
  "author_name": "",
  "post_date": "2014-04-15T07:52:55.383Z",
  "votes": null,
  "comment_count": 12,
  "views": 4909,
  "content": "<p>Dear Mates!</p>\n<p>Can I Selectively reduce the Rows using R! Do i need any big data tools like Hadoop for this Challenge! Or R/Rattle Miner is enough! please clarify</p>",
  "messages": [
    {
      "id": "41917",
      "postDate": "04/15/2014 07:52:55",
      "content": "<p>Dear Mates!</p>\n<p>Can I Selectively reduce the Rows using R! Do i need any big data tools like Hadoop for this Challenge! Or R/Rattle Miner is enough! please clarify</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "41925",
      "postDate": "04/15/2014 12:36:49",
      "content": "<p>Hey,</p>\n<p>i think you can use whatever tool you want, ie. i saw people using awk for some sub selection of the data. Reducing rows can be a way, it depends on you're model and you're modeling hypotheses.</p>\n<p>If you're model is extreme time expensive you can think about using hadoop and / or others, but i think with a normal desktop workstation it is possible to calculate you're results.</p>\n\n<p>Hopefully i could answer you're question :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "41941",
      "postDate": "04/15/2014 16:51:44",
      "content": "<p>You could use R/python/any other scripting language to&nbsp;open the file, read it in one line at a time, and then write the lines you want to keep out to another file.</p>\n<p>22GB is really too small for hadoop... if you could fit the file in memory on a medium-sized machine, a cluster will just slow you down.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42275",
      "postDate": "04/18/2014 01:55:33",
      "content": "<p>I haven't gotten very far in this, but it seems like there is little reason to hold all of &quot;transactions&quot; in memory at one time, and that's the big data set. More likely, you're just going to look up transactions involving a specific customer at the time your considering their reaction to a specific offer. &nbsp;My plan was to dump the whole transaction history to MongoDB and just query it as needed, or sample from it to generate any overall statistics.</p>\n<p>Anybody disagree?</p>\n<p>Then again, my computer is getting a little long in the tooth. &nbsp;If it chokes on this challenge, that might be just the excuse I need...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42418",
      "postDate": "04/19/2014 12:35:19",
      "content": "<p>[quote=BuddhaSixFour;42275]</p>\n<p>I haven't gotten very far in this, but it seems like there is little reason to hold all of &quot;transactions&quot; in memory at one time, and that's the big data set. More likely, you're just going to look up transactions involving a specific customer at the time your considering their reaction to a specific offer. &nbsp;My plan was to dump the whole transaction history to MongoDB and just query it as needed, or sample from it to generate any overall statistics.</p>\n<p>Anybody disagree?</p>\n<p>Then again, my computer is getting a little long in the tooth. &nbsp;If it chokes on this challenge, that might be just the excuse I need...</p>\n<p>[/quote]</p>\n<p>Can your share how you dump the whole transaction data into mongoDB. I have just started learning mongoDB ..it will be helpful if you share materials.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42466",
      "postDate": "04/20/2014 04:34:07",
      "content": "<p>If all you want to do is copy the CSV file straight into mongoDB, then&nbsp;<a href=\"http://docs.mongodb.org/manual/reference/program/mongoimport/\">mongoimport</a>&nbsp;will do the trick.</p>\n<p>But, this is such a big file, I did something a little different. &nbsp;The attached .py file uses&nbsp;<a href=\"http://api.mongodb.org/python/2.7rc0/\">pymongo</a>. &nbsp;It loads the &quot;offers&quot;, &quot;trainHistory&quot;, and &quot;testHistory&quot; as is with only some basic datatype conversions.&nbsp;</p>\n<p>It does more with &quot;transactions&quot;. It splits the entries into three categories: 1) entries where 'category' is also contained in 'offers'. This mirrors&nbsp;BreakfastPirate's <a href=\"http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/7666/getting-started-data-reduction\">grep approach</a>. &nbsp;2) If an entry doesn't match that criteria, 10% of it is retained in a &quot;sampled&quot; collection, and the rest is put in 3) a &quot;remainder&quot; collection. &nbsp;Note that the &quot;sample&quot; portion is by ID and not by line. &nbsp;It keeps 10% of IDs in the sample, so you will have all entries for those particular IDs.</p>\n<p>The idea there is to keep a piece of the remaining data available to play around with without having to store the whole thing right now. &nbsp;If you look at the settings, you can set how much gets sampled and which of the three transaction collections get saved. &nbsp;Right now, it defaults to 10% sampled, and the remainder is discarded.</p>\n<p>If you want to play around with it, you can set settings['load_maximum'] to a value other than -1, say 1,000,000. &nbsp;That will prevent it from trying to load everything. &nbsp;Set it to -1 to have it do everything again.</p>\n<p>Its not the quickest code in the world, but it works.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42523",
      "postDate": "04/20/2014 16:11:03",
      "content": "<p>Just curious, how much time would it take for MongoDB to return equivalent of following SQL:</p>\n<p>SELECT&nbsp;company, sum(purchaseamount) FROM transactions GROUP BY company&nbsp;</p>\n<p>over all 350M transactions?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42528",
      "postDate": "04/20/2014 17:28:16",
      "content": "<p>Has anyone tried loading the data into MySQL, SQLlite, or PostgreSQL? &nbsp;What kind of performance do you see for aggregations?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42540",
      "postDate": "04/20/2014 19:41:12",
      "content": "<p>[quote=grizzli3k;42523]</p>\n<p>Just curious, how much time would it take for MongoDB to return equivalent of following SQL:</p>\n<p>SELECT&nbsp;company, sum(purchaseamount) FROM transactions GROUP BY company&nbsp;</p>\n<p>over all 350M transactions?</p>\n<p>[/quote]</p>\n<p>On the 32.1M&nbsp;I kept as a random sample, the answer is&nbsp;171416ms on a 4-year old MacBook Pro after building an index on &quot;company&quot;. &nbsp;I'm curious to hear what people are getting on MySQL. &nbsp;The one advantage to MySQL in this case is that it doesn't have to write the schema to your hard-drive 350M times... certainly more efficient on storage.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42567",
      "postDate": "04/21/2014 02:59:22",
      "content": "<p>You can true MySQL. It's really fast by creating index.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42640",
      "postDate": "04/21/2014 19:53:56",
      "content": "<p>Thanks, BuddhaSixFour</p>\n<p>just tried this&nbsp;query over 350M in Oracle 11g running in a virtual box on&nbsp;my &nbsp;3 year old Lenovo laptop.</p>\n<p>2 mins 14 secs</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49388",
      "postDate": "06/22/2014 22:11:45",
      "content": "<p>i converted transactions.csv to a numpy table and np.save'd that to a local ssd.</p>\n<p>reading it from the&nbsp;ssd&nbsp;with np.load takes 36 seconds. &nbsp;(15 gb npy file).</p>\n\n<p>now that i can query the data quickly i can see that you can probably represent all the transaction data with&nbsp;int8/int16 which would help alot...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49686",
      "postDate": "06/27/2014 08:12:51",
      "content": "<p>I've uploaded using qlikview. &nbsp;Handles it really well.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 41925,
      "author_name": "robstr",
      "author_url": "",
      "post_date": "04/15/2014 12:36:49",
      "content": "<p>Hey,</p>\n<p>i think you can use whatever tool you want, ie. i saw people using awk for some sub selection of the data. Reducing rows can be a way, it depends on you're model and you're modeling hypotheses.</p>\n<p>If you're model is extreme time expensive you can think about using hadoop and / or others, but i think with a normal desktop workstation it is possible to calculate you're results.</p>\n\n<p>Hopefully i could answer you're question :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 41941,
      "author_name": "zachmayer",
      "author_url": "",
      "post_date": "04/15/2014 16:51:44",
      "content": "<p>You could use R/python/any other scripting language to&nbsp;open the file, read it in one line at a time, and then write the lines you want to keep out to another file.</p>\n<p>22GB is really too small for hadoop... if you could fit the file in memory on a medium-sized machine, a cluster will just slow you down.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42275,
      "author_name": "buddhasixfour",
      "author_url": "",
      "post_date": "04/18/2014 01:55:33",
      "content": "<p>I haven't gotten very far in this, but it seems like there is little reason to hold all of &quot;transactions&quot; in memory at one time, and that's the big data set. More likely, you're just going to look up transactions involving a specific customer at the time your considering their reaction to a specific offer. &nbsp;My plan was to dump the whole transaction history to MongoDB and just query it as needed, or sample from it to generate any overall statistics.</p>\n<p>Anybody disagree?</p>\n<p>Then again, my computer is getting a little long in the tooth. &nbsp;If it chokes on this challenge, that might be just the excuse I need...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42418,
      "author_name": "parthibangowthaman",
      "author_url": "",
      "post_date": "04/19/2014 12:35:19",
      "content": "<p>[quote=BuddhaSixFour;42275]</p>\n<p>I haven't gotten very far in this, but it seems like there is little reason to hold all of &quot;transactions&quot; in memory at one time, and that's the big data set. More likely, you're just going to look up transactions involving a specific customer at the time your considering their reaction to a specific offer. &nbsp;My plan was to dump the whole transaction history to MongoDB and just query it as needed, or sample from it to generate any overall statistics.</p>\n<p>Anybody disagree?</p>\n<p>Then again, my computer is getting a little long in the tooth. &nbsp;If it chokes on this challenge, that might be just the excuse I need...</p>\n<p>[/quote]</p>\n<p>Can your share how you dump the whole transaction data into mongoDB. I have just started learning mongoDB ..it will be helpful if you share materials.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42466,
      "author_name": "buddhasixfour",
      "author_url": "",
      "post_date": "04/20/2014 04:34:07",
      "content": "<p>If all you want to do is copy the CSV file straight into mongoDB, then&nbsp;<a href=\"http://docs.mongodb.org/manual/reference/program/mongoimport/\">mongoimport</a>&nbsp;will do the trick.</p>\n<p>But, this is such a big file, I did something a little different. &nbsp;The attached .py file uses&nbsp;<a href=\"http://api.mongodb.org/python/2.7rc0/\">pymongo</a>. &nbsp;It loads the &quot;offers&quot;, &quot;trainHistory&quot;, and &quot;testHistory&quot; as is with only some basic datatype conversions.&nbsp;</p>\n<p>It does more with &quot;transactions&quot;. It splits the entries into three categories: 1) entries where 'category' is also contained in 'offers'. This mirrors&nbsp;BreakfastPirate's <a href=\"http://www.kaggle.com/c/acquire-valued-shoppers-challenge/forums/t/7666/getting-started-data-reduction\">grep approach</a>. &nbsp;2) If an entry doesn't match that criteria, 10% of it is retained in a &quot;sampled&quot; collection, and the rest is put in 3) a &quot;remainder&quot; collection. &nbsp;Note that the &quot;sample&quot; portion is by ID and not by line. &nbsp;It keeps 10% of IDs in the sample, so you will have all entries for those particular IDs.</p>\n<p>The idea there is to keep a piece of the remaining data available to play around with without having to store the whole thing right now. &nbsp;If you look at the settings, you can set how much gets sampled and which of the three transaction collections get saved. &nbsp;Right now, it defaults to 10% sampled, and the remainder is discarded.</p>\n<p>If you want to play around with it, you can set settings['load_maximum'] to a value other than -1, say 1,000,000. &nbsp;That will prevent it from trying to load everything. &nbsp;Set it to -1 to have it do everything again.</p>\n<p>Its not the quickest code in the world, but it works.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42523,
      "author_name": "grizzli3k",
      "author_url": "",
      "post_date": "04/20/2014 16:11:03",
      "content": "<p>Just curious, how much time would it take for MongoDB to return equivalent of following SQL:</p>\n<p>SELECT&nbsp;company, sum(purchaseamount) FROM transactions GROUP BY company&nbsp;</p>\n<p>over all 350M transactions?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42528,
      "author_name": "zachmayer",
      "author_url": "",
      "post_date": "04/20/2014 17:28:16",
      "content": "<p>Has anyone tried loading the data into MySQL, SQLlite, or PostgreSQL? &nbsp;What kind of performance do you see for aggregations?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42540,
      "author_name": "buddhasixfour",
      "author_url": "",
      "post_date": "04/20/2014 19:41:12",
      "content": "<p>[quote=grizzli3k;42523]</p>\n<p>Just curious, how much time would it take for MongoDB to return equivalent of following SQL:</p>\n<p>SELECT&nbsp;company, sum(purchaseamount) FROM transactions GROUP BY company&nbsp;</p>\n<p>over all 350M transactions?</p>\n<p>[/quote]</p>\n<p>On the 32.1M&nbsp;I kept as a random sample, the answer is&nbsp;171416ms on a 4-year old MacBook Pro after building an index on &quot;company&quot;. &nbsp;I'm curious to hear what people are getting on MySQL. &nbsp;The one advantage to MySQL in this case is that it doesn't have to write the schema to your hard-drive 350M times... certainly more efficient on storage.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42567,
      "author_name": "lavizhao",
      "author_url": "",
      "post_date": "04/21/2014 02:59:22",
      "content": "<p>You can true MySQL. It's really fast by creating index.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42640,
      "author_name": "grizzli3k",
      "author_url": "",
      "post_date": "04/21/2014 19:53:56",
      "content": "<p>Thanks, BuddhaSixFour</p>\n<p>just tried this&nbsp;query over 350M in Oracle 11g running in a virtual box on&nbsp;my &nbsp;3 year old Lenovo laptop.</p>\n<p>2 mins 14 secs</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49388,
      "author_name": "richm63001",
      "author_url": "",
      "post_date": "06/22/2014 22:11:45",
      "content": "<p>i converted transactions.csv to a numpy table and np.save'd that to a local ssd.</p>\n<p>reading it from the&nbsp;ssd&nbsp;with np.load takes 36 seconds. &nbsp;(15 gb npy file).</p>\n\n<p>now that i can query the data quickly i can see that you can probably represent all the transaction data with&nbsp;int8/int16 which would help alot...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49686,
      "author_name": "dapahughes",
      "author_url": "",
      "post_date": "06/27/2014 08:12:51",
      "content": "<p>I've uploaded using qlikview. &nbsp;Handles it really well.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "41917": "",
    "41925": "",
    "41941": "",
    "42275": "",
    "42418": "",
    "42466": "",
    "42523": "",
    "42528": "",
    "42540": "",
    "42567": "",
    "42640": "",
    "49388": "",
    "49686": ""
  },
  "source": "meta"
}