{
  "id": 11102,
  "title": "Running through SAS software?",
  "url": "/competitions/avazu-ctr-prediction/discussion/11102",
  "author_name": "",
  "post_date": "2014-12-01T07:06:26.383Z",
  "votes": null,
  "comment_count": 14,
  "views": 3919,
  "content": "<p>Hi,</p>\n<p>Can we run this analysis through SAS software?</p>",
  "messages": [
    {
      "id": "59311",
      "postDate": "12/01/2014 07:06:26",
      "content": "<p>Hi,</p>\n<p>Can we run this analysis through SAS software?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59313",
      "postDate": "12/01/2014 08:19:05",
      "content": "<p>SAS is forbidden</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59317",
      "postDate": "12/01/2014 09:08:39",
      "content": "<p>Thanks Steve.But &quot;R&quot; is taking much time to import&nbsp;the data into it.:(</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59329",
      "postDate": "12/01/2014 13:51:28",
      "content": "<p>You should checkout tinrgtu's benchmark code:&nbsp;http://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10927/beat-the-benchmark-with-less-than-1mb-of-memory</p>\n<ul>\n<li>it's well written and uses only default Python modules, so very easy to use</li>\n<li>it's an online algorithm and doesn't waste too much memory so you don't need a lot of RAM</li>\n<li>it's based on state of the art research by Google so you can get very good results (top 10 LB)</li>\n</ul>\n\n<p>If you use this script with PyPy, you should be able to train over the whole dataset&nbsp;fairly quickly.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59333",
      "postDate": "12/01/2014 15:17:07",
      "content": "<p>Try fread from data.table</p>\n<p>http://cran.r-project.org/web/packages/data.table/index.html</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59340",
      "postDate": "12/01/2014 16:53:22",
      "content": "<p>Is it ok to use Vowpal Wabbit?&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59342",
      "postDate": "12/01/2014 16:55:06",
      "content": "<p>[quote=Sandeep Kumar Kotha;59311]</p>\n<p>Hi,</p>\n<p>Can we run this analysis through SAS software?</p>\n<p>[/quote]</p>\n<p>I can be wrong, but I think you should only be concerned if you think you'll do good enough to be eligible for prizes. Otherwise, you could use whatever software you want. Plus, I don't see a reason why you shouldn't use SAS, if you're more familiar with it, to quickly test hypothesis. And if you really end up in a top spot, you can always re-write the SAS code in Python or R.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59350",
      "postDate": "12/01/2014 18:36:16",
      "content": "<p>Just to read data, and get a quick summary, try data.table package. It will load the entire data into memory in a couple of mins.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59356",
      "postDate": "12/01/2014 20:48:32",
      "content": "<p>[quote=Pavitrakumar;59340]</p>\n<p>Is it ok to use Vowpal Wabbit?&nbsp;</p>\n<p>[/quote]</p>\n\n<p>Yes, based on every other competition I've ever been in.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59373",
      "postDate": "12/02/2014 04:33:13",
      "content": "<p>HI All,&nbsp;</p>\n<p>Thanks for the reply.I used the command <strong>fread</strong> to import the dataset but it&nbsp;took one hour and still 0.8% of whole has been imported &nbsp;so far.</p>\n<p>This is the code</p>\n<p>s1=fread(&quot;train.csv&quot;,colclasses='character',header=T)</p>\n<p>Please let me know am I missing something here?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59379",
      "postDate": "12/02/2014 07:58:04",
      "content": "<p>[quote=Sandeep Kumar Kotha;59373]</p>\n<p>HI All,&nbsp;</p>\n<p>Thanks for the reply.I used the command <strong>fread</strong> to import the dataset but it&nbsp;took one hour and still 0.8% of whole has been imported &nbsp;so far.</p>\n<p>This is the code</p>\n<p>s1=fread(&quot;train.csv&quot;,colclasses='character',header=T)</p>\n<p>Please let me know am I missing something here?</p>\n<p>[/quote]</p>\n<p>With colclasses='character' took 18 mins for both files. Much more than without this parameter.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59385",
      "postDate": "12/02/2014 09:51:43",
      "content": "<p>[quote=Jos&#233; A. Guerrero;59379]</p>\n<p>With colclasses='character' took 18 mins for both files. Much more than without this parameter.&nbsp;</p>\n<p>[/quote]</p>\n<p>train &lt;- fread(&quot;./train.csv&quot;,stringsAsFactors=F) &nbsp; &nbsp; runs in about 4 mins on my Windows machine.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60320",
      "postDate": "12/16/2014 04:33:23",
      "content": "<p>[quote=Steve Wang;59313]</p>\n<p>SAS is forbidden</p>\n<p>[/quote]</p>\n<p>Seriously?!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60325",
      "postDate": "12/16/2014 06:50:27",
      "content": "<p>This question is literally just a question about facts. Nothing more or less.</p>\n<p>Has there even been a Kaggle winning solution (top 3) that has used SAS in the history of Kaggle?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60364",
      "postDate": "12/16/2014 16:03:54",
      "content": "<p>[quote=Mike Kim;60325]</p>\n<p>This question is literally just a question about facts. Nothing more or less.</p>\n<p>Has there even been a Kaggle winning solution (top 3) that has used SAS in the history of Kaggle?</p>\n<p>[/quote]</p>\n<p>I don't recall any since I started a year ago, but I clearly recall a winner solution post (that I cannot find now) where &nbsp;a SAS model was part of the winning ensemble. But a simple google search brings up things like this:</p>\n<p>http://blogs.sas.com/content/sascom/2012/03/05/kaggles-real-predictive-analytics-winners/</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 59313,
      "author_name": "stevewang0",
      "author_url": "",
      "post_date": "12/01/2014 08:19:05",
      "content": "<p>SAS is forbidden</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59317,
      "author_name": "sandeepkumar",
      "author_url": "",
      "post_date": "12/01/2014 09:08:39",
      "content": "<p>Thanks Steve.But &quot;R&quot; is taking much time to import&nbsp;the data into it.:(</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59329,
      "author_name": "alexistaugeron",
      "author_url": "",
      "post_date": "12/01/2014 13:51:28",
      "content": "<p>You should checkout tinrgtu's benchmark code:&nbsp;http://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10927/beat-the-benchmark-with-less-than-1mb-of-memory</p>\n<ul>\n<li>it's well written and uses only default Python modules, so very easy to use</li>\n<li>it's an online algorithm and doesn't waste too much memory so you don't need a lot of RAM</li>\n<li>it's based on state of the art research by Google so you can get very good results (top 10 LB)</li>\n</ul>\n\n<p>If you use this script with PyPy, you should be able to train over the whole dataset&nbsp;fairly quickly.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59333,
      "author_name": "adamszalucha",
      "author_url": "",
      "post_date": "12/01/2014 15:17:07",
      "content": "<p>Try fread from data.table</p>\n<p>http://cran.r-project.org/web/packages/data.table/index.html</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59340,
      "author_name": "pavitrakumar78",
      "author_url": "",
      "post_date": "12/01/2014 16:53:22",
      "content": "<p>Is it ok to use Vowpal Wabbit?&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59342,
      "author_name": "adjgiulio",
      "author_url": "",
      "post_date": "12/01/2014 16:55:06",
      "content": "<p>[quote=Sandeep Kumar Kotha;59311]</p>\n<p>Hi,</p>\n<p>Can we run this analysis through SAS software?</p>\n<p>[/quote]</p>\n<p>I can be wrong, but I think you should only be concerned if you think you'll do good enough to be eligible for prizes. Otherwise, you could use whatever software you want. Plus, I don't see a reason why you shouldn't use SAS, if you're more familiar with it, to quickly test hypothesis. And if you really end up in a top spot, you can always re-write the SAS code in Python or R.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59350,
      "author_name": "xiaozhouwang",
      "author_url": "",
      "post_date": "12/01/2014 18:36:16",
      "content": "<p>Just to read data, and get a quick summary, try data.table package. It will load the entire data into memory in a couple of mins.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59356,
      "author_name": "jfkingiii",
      "author_url": "",
      "post_date": "12/01/2014 20:48:32",
      "content": "<p>[quote=Pavitrakumar;59340]</p>\n<p>Is it ok to use Vowpal Wabbit?&nbsp;</p>\n<p>[/quote]</p>\n\n<p>Yes, based on every other competition I've ever been in.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59373,
      "author_name": "sandeepkumar",
      "author_url": "",
      "post_date": "12/02/2014 04:33:13",
      "content": "<p>HI All,&nbsp;</p>\n<p>Thanks for the reply.I used the command <strong>fread</strong> to import the dataset but it&nbsp;took one hour and still 0.8% of whole has been imported &nbsp;so far.</p>\n<p>This is the code</p>\n<p>s1=fread(&quot;train.csv&quot;,colclasses='character',header=T)</p>\n<p>Please let me know am I missing something here?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59379,
      "author_name": "blindape",
      "author_url": "",
      "post_date": "12/02/2014 07:58:04",
      "content": "<p>[quote=Sandeep Kumar Kotha;59373]</p>\n<p>HI All,&nbsp;</p>\n<p>Thanks for the reply.I used the command <strong>fread</strong> to import the dataset but it&nbsp;took one hour and still 0.8% of whole has been imported &nbsp;so far.</p>\n<p>This is the code</p>\n<p>s1=fread(&quot;train.csv&quot;,colclasses='character',header=T)</p>\n<p>Please let me know am I missing something here?</p>\n<p>[/quote]</p>\n<p>With colclasses='character' took 18 mins for both files. Much more than without this parameter.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59385,
      "author_name": "rohanrao",
      "author_url": "",
      "post_date": "12/02/2014 09:51:43",
      "content": "<p>[quote=Jos&#233; A. Guerrero;59379]</p>\n<p>With colclasses='character' took 18 mins for both files. Much more than without this parameter.&nbsp;</p>\n<p>[/quote]</p>\n<p>train &lt;- fread(&quot;./train.csv&quot;,stringsAsFactors=F) &nbsp; &nbsp; runs in about 4 mins on my Windows machine.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60320,
      "author_name": "louify",
      "author_url": "",
      "post_date": "12/16/2014 04:33:23",
      "content": "<p>[quote=Steve Wang;59313]</p>\n<p>SAS is forbidden</p>\n<p>[/quote]</p>\n<p>Seriously?!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60325,
      "author_name": "mikeskim",
      "author_url": "",
      "post_date": "12/16/2014 06:50:27",
      "content": "<p>This question is literally just a question about facts. Nothing more or less.</p>\n<p>Has there even been a Kaggle winning solution (top 3) that has used SAS in the history of Kaggle?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60364,
      "author_name": "adjgiulio",
      "author_url": "",
      "post_date": "12/16/2014 16:03:54",
      "content": "<p>[quote=Mike Kim;60325]</p>\n<p>This question is literally just a question about facts. Nothing more or less.</p>\n<p>Has there even been a Kaggle winning solution (top 3) that has used SAS in the history of Kaggle?</p>\n<p>[/quote]</p>\n<p>I don't recall any since I started a year ago, but I clearly recall a winner solution post (that I cannot find now) where &nbsp;a SAS model was part of the winning ensemble. But a simple google search brings up things like this:</p>\n<p>http://blogs.sas.com/content/sascom/2012/03/05/kaggles-real-predictive-analytics-winners/</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "59311": "",
    "59313": "",
    "59317": "",
    "59329": "",
    "59333": "",
    "59340": "",
    "59342": "",
    "59350": "",
    "59356": "",
    "59373": "",
    "59379": "",
    "59385": "",
    "60320": "",
    "60325": "",
    "60364": ""
  },
  "source": "meta"
}