{ "id": 11136, "title": "Ideas on using Python or R on Mac OS X?", "url": "/competitions/avazu-ctr-prediction/discussion/11136", "author_name": "", "post_date": "2014-12-04T21:50:13.403Z", "votes": null, "comment_count": 14, "views": 3121, "content": "

Hey all!

\n

I'm working on this competition for a school project, essentially trying to get the best score I can for my group (doing this for the grade, not the money).

\n

I was wondering if anyone using a Mac was having any luck getting Python (Pypy or 2.7.8) or perhaps R Studio to generate successful submissions (not necessarily the best, but scoring well hopefully)

\n

I'm fairly new to both R and Python, and was wondering if anyone had any ideas that might help us get the ball rolling as we feel a bit stalled at the moment. Any input would be greatly appreciated! 

", "messages": [ { "id": "59576", "postDate": "12/04/2014 21:50:13", "content": "

Hey all!

\n

I'm working on this competition for a school project, essentially trying to get the best score I can for my group (doing this for the grade, not the money).

\n

I was wondering if anyone using a Mac was having any luck getting Python (Pypy or 2.7.8) or perhaps R Studio to generate successful submissions (not necessarily the best, but scoring well hopefully)

\n

I'm fairly new to both R and Python, and was wondering if anyone had any ideas that might help us get the ball rolling as we feel a bit stalled at the moment. Any input would be greatly appreciated! 

", "rawMarkdown": "", "votes": null }, { "id": "59578", "postDate": "12/04/2014 22:08:33", "content": "

Yep, I use a Mac.  Python and RStudio both work just fine.  What kinds of problems are you having?  What have you tried?  If you haven't tried tinrtgu's Python code, I'd take a look at it.  It should work pretty much out of the box, without any additional libraries, etc.  I (and likely most of the other folks here) would be happy to help get you on track... just let us know where you're stuck.

", "rawMarkdown": "", "votes": null }, { "id": "59579", "postDate": "12/04/2014 22:10:45", "content": "

I have a Mac. I have pypy and Anaconda python 2.7. I also use R.

\n

Start by running the Beat Benchmark with less than 1mg .... code

\n

If I remember rightly, I just downloaded pypy and just call it from where I downloaded it to

", "rawMarkdown": "", "votes": null }, { "id": "59581", "postDate": "12/04/2014 22:44:08", "content": "

It's fine to using Python or R. I think either one is easy to get started since they are scripting language. I also recommend starting playing with tinrtgu's benchmark python scripts with pypy.

", "rawMarkdown": "", "votes": null }, { "id": "59584", "postDate": "12/04/2014 23:44:07", "content": "

R support for very large datasets and sparse matrices is not as good as Pythons, and as an R user I spend a lot of time working with smaller samples that can fit into memory (I do not use the streaming DB approach in R). Also I often need to split responses into multiple files and then merge them for submission. I have my own software libraries for this, as I have an IT background.

\n

This is a lot of work that could be avoided using Python, but I like to do it this way because it's better practice for the real world, and because I am slightly masochistic ;-)

\n

So, given that both R and Python are new to you, I would go with Python for this challenge. It's easier to learn (R's syntax is powerful, but takes longer to get your head around, especially if you are already a programmer) and the sparse matrix support should greatly help.

", "rawMarkdown": "", "votes": null }, { "id": "59607", "postDate": "12/05/2014 07:56:09", "content": "

I was using the code posted by tinrtgu, but I wasn't able to generate any results, though after experimenting with pasting all the code in the perminal window for pypy, I was able to get the ID and click columns to show up in my submission document. Did you guys alter tinrtgu' score at all to account for the different locations that your any have had the test and train datasets saved, or did you just leave the code as was? I moved everything to the desktop, including the folder for pypy, the datasets, and the scripts, but when entering 'pypy fast_solution.py', I always encounter an error referring to 'stdin' on Line 1. Any ideas on what I could try? Perhaps the locations of my files may be causing some issues?

\n

I've also tried copying all the text of the script into the terminal window, which gave me just the column titles, but returned errors with indentation, and I've also tried substituting the script name with the file path in quotes 'like this', but still have had no luck thus far. Any ideas on what might be flawed in my approach?

\n

Again, thank you guys so much!

", "rawMarkdown": "", "votes": null }, { "id": "59608", "postDate": "12/05/2014 08:03:57", "content": "

One of my questions was meant to be "did you alter tinrtgu's code" but corrected to "tinrtgu' score", sorry for the typo!

", "rawMarkdown": "", "votes": null }, { "id": "59614", "postDate": "12/05/2014 09:43:22", "content": "

If you want to get pypy (or other soft like vw) on mac without any troubles, use homebrew.

\n

http://brew.sh/

", "rawMarkdown": "", "votes": null }, { "id": "59615", "postDate": "12/05/2014 10:09:24", "content": "

The original code will give scores for sure. I guess you did not specify train and test file locations correctly. You have to change the path accordingly.

", "rawMarkdown": "", "votes": null }, { "id": "59617", "postDate": "12/05/2014 11:04:02", "content": "

[quote=simeng;59615]

\n

The original code will give scores for sure. I guess you did not specify train and test file locations correctly. You have to change the path accordingly.

\n

[/quote]

\n\n

Agreed

", "rawMarkdown": "", "votes": null }, { "id": "59621", "postDate": "12/05/2014 14:43:14", "content": "

I did try to edit the code (using Xcode which it defaulted to) to specify the locations of my files at the section at the top where it asked for file paths. Would I need to specify anything in pypy as opposed to the script code?

", "rawMarkdown": "", "votes": null }, { "id": "59622", "postDate": "12/05/2014 15:10:32", "content": "

are you getting any errors?

", "rawMarkdown": "", "votes": null }, { "id": "59623", "postDate": "12/05/2014 15:54:25", "content": "

I work on a mac using mostly python and R. If you've got pypy and the Anaconda distro installed, you should only need to modify a few lines in tingrtu's script for it to run: the paths to train, test, and submission. You also need to make the date correction caught in the forum.

\n

I'd suggest not using Xcode. Work at the terminal instead (in emacs, say) for editing the files. You have a .bash_profile file (google 'modify bash profile mac') that you should modify so that you can call pypy from whatever location your files are at. (Mac's don't have this file by default, but Anaconda creates one when you install it). Then you can put tinrtgu's script, test, train all in the same location and run with very little change to the original script. My lines 32 - 34 are

\n

train = 'train'

\n

test = 'test'

\n

submission = 'fast_submission.csv' 

\n

Good luck.  

", "rawMarkdown": "", "votes": null }, { "id": "59643", "postDate": "12/05/2014 20:57:27", "content": "

As this is a learning endeavour, one thing that I would suggest you to do is just take a sample of training data and try to use scikit learn on it. 

\n

Some have tried that approach and have got LB scores of .39 just using a sample of 2 million

", "rawMarkdown": "", "votes": null }, { "id": "59692", "postDate": "12/06/2014 20:51:25", "content": "

Thanks so much for your suggestions, guys! I ended up using Spyder, and specified the file locations after changing the directory to where my files were located. I was able to get a submission in and our scores look amazing!

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 59578, "author_name": "philculliton", "author_url": "", "post_date": "12/04/2014 22:08:33", "content": "

Yep, I use a Mac.  Python and RStudio both work just fine.  What kinds of problems are you having?  What have you tried?  If you haven't tried tinrtgu's Python code, I'd take a look at it.  It should work pretty much out of the box, without any additional libraries, etc.  I (and likely most of the other folks here) would be happy to help get you on track... just let us know where you're stuck.

", "votes": null, "replies": [] }, { "id": 59579, "author_name": "domcastro", "author_url": "", "post_date": "12/04/2014 22:10:45", "content": "

I have a Mac. I have pypy and Anaconda python 2.7. I also use R.

\n

Start by running the Beat Benchmark with less than 1mg .... code

\n

If I remember rightly, I just downloaded pypy and just call it from where I downloaded it to

", "votes": null, "replies": [] }, { "id": 59581, "author_name": "simengy82", "author_url": "", "post_date": "12/04/2014 22:44:08", "content": "

It's fine to using Python or R. I think either one is easy to get started since they are scripting language. I also recommend starting playing with tinrtgu's benchmark python scripts with pypy.

", "votes": null, "replies": [] }, { "id": 59584, "author_name": "heisenbug", "author_url": "", "post_date": "12/04/2014 23:44:07", "content": "

R support for very large datasets and sparse matrices is not as good as Pythons, and as an R user I spend a lot of time working with smaller samples that can fit into memory (I do not use the streaming DB approach in R). Also I often need to split responses into multiple files and then merge them for submission. I have my own software libraries for this, as I have an IT background.

\n

This is a lot of work that could be avoided using Python, but I like to do it this way because it's better practice for the real world, and because I am slightly masochistic ;-)

\n

So, given that both R and Python are new to you, I would go with Python for this challenge. It's easier to learn (R's syntax is powerful, but takes longer to get your head around, especially if you are already a programmer) and the sparse matrix support should greatly help.

", "votes": null, "replies": [] }, { "id": 59607, "author_name": "manas6", "author_url": "", "post_date": "12/05/2014 07:56:09", "content": "

I was using the code posted by tinrtgu, but I wasn't able to generate any results, though after experimenting with pasting all the code in the perminal window for pypy, I was able to get the ID and click columns to show up in my submission document. Did you guys alter tinrtgu' score at all to account for the different locations that your any have had the test and train datasets saved, or did you just leave the code as was? I moved everything to the desktop, including the folder for pypy, the datasets, and the scripts, but when entering 'pypy fast_solution.py', I always encounter an error referring to 'stdin' on Line 1. Any ideas on what I could try? Perhaps the locations of my files may be causing some issues?

\n

I've also tried copying all the text of the script into the terminal window, which gave me just the column titles, but returned errors with indentation, and I've also tried substituting the script name with the file path in quotes 'like this', but still have had no luck thus far. Any ideas on what might be flawed in my approach?

\n

Again, thank you guys so much!

", "votes": null, "replies": [] }, { "id": 59608, "author_name": "manas6", "author_url": "", "post_date": "12/05/2014 08:03:57", "content": "

One of my questions was meant to be "did you alter tinrtgu's code" but corrected to "tinrtgu' score", sorry for the typo!

", "votes": null, "replies": [] }, { "id": 59614, "author_name": "adamszalucha", "author_url": "", "post_date": "12/05/2014 09:43:22", "content": "

If you want to get pypy (or other soft like vw) on mac without any troubles, use homebrew.

\n

http://brew.sh/

", "votes": null, "replies": [] }, { "id": 59615, "author_name": "simengy82", "author_url": "", "post_date": "12/05/2014 10:09:24", "content": "

The original code will give scores for sure. I guess you did not specify train and test file locations correctly. You have to change the path accordingly.

", "votes": null, "replies": [] }, { "id": 59617, "author_name": "domcastro", "author_url": "", "post_date": "12/05/2014 11:04:02", "content": "

[quote=simeng;59615]

\n

The original code will give scores for sure. I guess you did not specify train and test file locations correctly. You have to change the path accordingly.

\n

[/quote]

\n\n

Agreed

", "votes": null, "replies": [] }, { "id": 59621, "author_name": "manas6", "author_url": "", "post_date": "12/05/2014 14:43:14", "content": "

I did try to edit the code (using Xcode which it defaulted to) to specify the locations of my files at the section at the top where it asked for file paths. Would I need to specify anything in pypy as opposed to the script code?

", "votes": null, "replies": [] }, { "id": 59622, "author_name": "domcastro", "author_url": "", "post_date": "12/05/2014 15:10:32", "content": "

are you getting any errors?

", "votes": null, "replies": [] }, { "id": 59623, "author_name": "jwjohnson314", "author_url": "", "post_date": "12/05/2014 15:54:25", "content": "

I work on a mac using mostly python and R. If you've got pypy and the Anaconda distro installed, you should only need to modify a few lines in tingrtu's script for it to run: the paths to train, test, and submission. You also need to make the date correction caught in the forum.

\n

I'd suggest not using Xcode. Work at the terminal instead (in emacs, say) for editing the files. You have a .bash_profile file (google 'modify bash profile mac') that you should modify so that you can call pypy from whatever location your files are at. (Mac's don't have this file by default, but Anaconda creates one when you install it). Then you can put tinrtgu's script, test, train all in the same location and run with very little change to the original script. My lines 32 - 34 are

\n

train = 'train'

\n

test = 'test'

\n

submission = 'fast_submission.csv' 

\n

Good luck.  

", "votes": null, "replies": [] }, { "id": 59643, "author_name": "mlwhiz", "author_url": "", "post_date": "12/05/2014 20:57:27", "content": "

As this is a learning endeavour, one thing that I would suggest you to do is just take a sample of training data and try to use scikit learn on it. 

\n

Some have tried that approach and have got LB scores of .39 just using a sample of 2 million

", "votes": null, "replies": [] }, { "id": 59692, "author_name": "manas6", "author_url": "", "post_date": "12/06/2014 20:51:25", "content": "

Thanks so much for your suggestions, guys! I ended up using Spyder, and specified the file locations after changing the directory to where my files were located. I was able to get a submission in and our scores look amazing!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "59576": "", "59578": "", "59579": "", "59581": "", "59584": "", "59607": "", "59608": "", "59614": "", "59615": "", "59617": "", "59621": "", "59622": "", "59623": "", "59643": "", "59692": "" }, "source": "meta" }