{
  "id": 1180,
  "title": "Breaking up of Training Data",
  "url": "/competitions/AlgorithmicTradingChallenge/discussion/1180",
  "author_name": "",
  "post_date": "2011-12-19T15:05:26.783Z",
  "votes": null,
  "comment_count": 7,
  "views": 5969,
  "content": "<p>Would it be possible for admin (or some one else ) to break up the training data randomly into smaller sets , say 10000 rows each. My outdated PC has given up trying to process the complete file. I have been able to break the data serially using a software\r\n but that doesnt solve my problem.</p>",
  "messages": [
    {
      "id": "7335",
      "postDate": "12/19/2011 15:05:26",
      "content": "<p>Would it be possible for admin (or some one else ) to break up the training data randomly into smaller sets , say 10000 rows each. My outdated PC has given up trying to process the complete file. I have been able to break the data serially using a software\r\n but that doesnt solve my problem.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "7349",
      "postDate": "12/20/2011 01:29:12",
      "content": "<p>I have been using matlab and excel(and just started R) which are unable to process. &nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "7351",
      "postDate": "12/20/2011 01:56:13",
      "content": "<p>How much memory do you have? How much can you add?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "7354",
      "postDate": "12/20/2011 02:27:50",
      "content": "<p>I am comfortable with around 20000 rows max.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "7372",
      "postDate": "12/20/2011 14:46:58",
      "content": "<p>[quote=amit rajora;7354]</p>\r\n<p>I am comfortable with around 20000 rows max.</p>\r\n<p>[/quote]</p>\r\n<p>This is not exactly what I asked. &nbsp;Anyway, if you are using Matlab, then read data file in chunks you are comfortable with and save them as .mat files. It will make you life much easier.</p>\r\n<p>Do not expect anybody to slice data for you. If you want to compete you need at least to know how to manipulate data and how to deal with big data sets (and this set is not that big).</p>\r\n<p>I apologize for being so direct, but if you want to compete in car race you need to know how to open the car door and start the engine.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "7378",
      "postDate": "12/20/2011 15:36:22",
      "content": "<p>Problem solved !</p>\r\n<p>Thanks to :-</p>\r\n<p>Admin- for the pointer</p>\r\n<p>Sergey-for the encouragement</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "7629",
      "postDate": "01/03/2012 02:08:11",
      "content": "<p>Quick. At this late stage I think I am having trouble unpacking &quot;training.zip&quot;.&nbsp; Can anyone tell me how many rows there are?&nbsp; I get variable result around 544,111 to 544,351 !!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "7630",
      "postDate": "01/03/2012 02:42:32",
      "content": "<p>Not counting the header row, there should be 754,018.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 7349,
      "author_name": "bergeisubka",
      "author_url": "",
      "post_date": "12/20/2011 01:29:12",
      "content": "<p>I have been using matlab and excel(and just started R) which are unable to process. &nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 7351,
      "author_name": "ccccat",
      "author_url": "",
      "post_date": "12/20/2011 01:56:13",
      "content": "<p>How much memory do you have? How much can you add?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 7354,
      "author_name": "bergeisubka",
      "author_url": "",
      "post_date": "12/20/2011 02:27:50",
      "content": "<p>I am comfortable with around 20000 rows max.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 7372,
      "author_name": "ccccat",
      "author_url": "",
      "post_date": "12/20/2011 14:46:58",
      "content": "<p>[quote=amit rajora;7354]</p>\r\n<p>I am comfortable with around 20000 rows max.</p>\r\n<p>[/quote]</p>\r\n<p>This is not exactly what I asked. &nbsp;Anyway, if you are using Matlab, then read data file in chunks you are comfortable with and save them as .mat files. It will make you life much easier.</p>\r\n<p>Do not expect anybody to slice data for you. If you want to compete you need at least to know how to manipulate data and how to deal with big data sets (and this set is not that big).</p>\r\n<p>I apologize for being so direct, but if you want to compete in car race you need to know how to open the car door and start the engine.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 7378,
      "author_name": "bergeisubka",
      "author_url": "",
      "post_date": "12/20/2011 15:36:22",
      "content": "<p>Problem solved !</p>\r\n<p>Thanks to :-</p>\r\n<p>Admin- for the pointer</p>\r\n<p>Sergey-for the encouragement</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 7629,
      "author_name": "jc3626439",
      "author_url": "",
      "post_date": "01/03/2012 02:08:11",
      "content": "<p>Quick. At this late stage I think I am having trouble unpacking &quot;training.zip&quot;.&nbsp; Can anyone tell me how many rows there are?&nbsp; I get variable result around 544,111 to 544,351 !!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 7630,
      "author_name": "bcragin",
      "author_url": "",
      "post_date": "01/03/2012 02:42:32",
      "content": "<p>Not counting the header row, there should be 754,018.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "7335": "",
    "7349": "",
    "7351": "",
    "7354": "",
    "7372": "",
    "7378": "",
    "7629": "",
    "7630": ""
  },
  "source": "meta"
}