{
  "id": 23719,
  "title": "Figuring out which AWS instance will work the best",
  "url": "/competitions/bosch-production-line-performance/discussion/23719",
  "author_name": "",
  "post_date": "2016-09-17T04:52:51.143Z",
  "votes": null,
  "comment_count": 4,
  "views": 811,
  "content": "<p>Hi guys,</p>\n\n<p>I launched a rstudio server ami with m4.xlarge instance. I loaded the data, but it was taking ages to read the data files.</p>\n\n<p>Any pointers on which will be the best instance for this competition?</p>\n\n<p>In addition, can somebody explain how to use python using the same AMI. Will I have to connect to the Ubuntu terminal through SSH to use python? Or can I directly open a iPython notebook in my browser just like I did with rstudio.</p>\n\n<p>Thanks and regards,\nChaitanya.</p>",
  "messages": [
    {
      "id": "135804",
      "postDate": "09/17/2016 04:52:51",
      "content": "<p>Hi guys,</p>\n\n<p>I launched a rstudio server ami with m4.xlarge instance. I loaded the data, but it was taking ages to read the data files.</p>\n\n<p>Any pointers on which will be the best instance for this competition?</p>\n\n<p>In addition, can somebody explain how to use python using the same AMI. Will I have to connect to the Ubuntu terminal through SSH to use python? Or can I directly open a iPython notebook in my browser just like I did with rstudio.</p>\n\n<p>Thanks and regards,\nChaitanya.</p>",
      "rawMarkdown": "Hi guys,\r\n\r\nI launched a rstudio server ami with m4.xlarge instance. I loaded the data, but it was taking ages to read the data files.\r\n\r\nAny pointers on which will be the best instance for this competition?\r\n\r\nIn addition, can somebody explain how to use python using the same AMI. Will I have to connect to the Ubuntu terminal through SSH to use python? Or can I directly open a iPython notebook in my browser just like I did with rstudio.\r\n\r\nThanks and regards,\r\nChaitanya.",
      "votes": null
    },
    {
      "id": "135836",
      "postDate": "09/17/2016 13:50:47",
      "content": "<p>Chaitanya - I usually use an r3 instance for bigger data sets in R because they come with more RAM than the equivalent m4. r3.xlarge should be enough for most things. I ran into some limits with this data on that type of instance, but it's because I wasn't managing my memory at all. With some of the techniques elsewhere in the forum, you can get by with far less.</p>\n\n<p>You can open a Jupyter/iPython notebook directly on your browser if it's set up right. The <a href=\"http://www.louisaslett.com/RStudio_AMI/\">Louis Aslett AMIs</a> have notebooks and Python built in now and you can run both at once. Follow the directions there. I had performance issues with using both though, and will go back to using my own image for notebooks.</p>\n\n<p>Good luck!</p>",
      "rawMarkdown": "Chaitanya - I usually use an r3 instance for bigger data sets in R because they come with more RAM than the equivalent m4. r3.xlarge should be enough for most things. I ran into some limits with this data on that type of instance, but it's because I wasn't managing my memory at all. With some of the techniques elsewhere in the forum, you can get by with far less.\r\n\r\nYou can open a Jupyter/iPython notebook directly on your browser if it's set up right. The [Louis Aslett AMIs](http://www.louisaslett.com/RStudio_AMI/) have notebooks and Python built in now and you can run both at once. Follow the directions there. I had performance issues with using both though, and will go back to using my own image for notebooks.\r\n\r\nGood luck!",
      "votes": null
    },
    {
      "id": "135849",
      "postDate": "09/17/2016 16:38:56",
      "content": "<p>Thanks a lot John.</p>\n\n<p>This really helps.</p>\n\n<p>Cheers!</p>",
      "rawMarkdown": "Thanks a lot John.\r\n\r\nThis really helps.\r\n\r\nCheers!",
      "votes": null
    },
    {
      "id": "136755",
      "postDate": "09/24/2016 20:03:55",
      "content": "<p>@ChaitanyaGokhale - which part was more helpful, using an r3.xlarge instance or using different techniques to get by with less memory?</p>",
      "rawMarkdown": "ChaitanyaGokhale - which part was more helpful, using an r3.xlarge instance or using different techniques to get by with less memory?",
      "votes": null
    },
    {
      "id": "136758",
      "postDate": "09/24/2016 21:05:05",
      "content": "<p>I work in the cloud frequently for my day job and I've found Google Cloud Engine more user friendly for doing Kaggle stuff.  </p>\n\n<p>I wrote a blog post about setting up <a href=\"https://jeffdelaney.me/blog/running-jupyter-notebook-google-cloud-platform/\">GCE for iPython/Jupyter Notebooks</a>, which includes instructions for connecting via HTTP and saving the disk image to easily reproduce your environment. It's easy to build your own custom image, rather than use an AMI, if you're particular about your environment config. All this can be done in AWS as well. </p>\n\n<p>If you go this route, you'll probably want to use the v16 High Memory instance for this competition. </p>",
      "rawMarkdown": "I work in the cloud frequently for my day job and I've found Google Cloud Engine more user friendly for doing Kaggle stuff.  \r\n\r\nI wrote a blog post about setting up [GCE for iPython/Jupyter Notebooks][1], which includes instructions for connecting via HTTP and saving the disk image to easily reproduce your environment. It's easy to build your own custom image, rather than use an AMI, if you're particular about your environment config. All this can be done in AWS as well. \r\n\r\nIf you go this route, you'll probably want to use the v16 High Memory instance for this competition. \r\n\r\n\r\n  [1]: https://jeffdelaney.me/blog/running-jupyter-notebook-google-cloud-platform/",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 135836,
      "author_name": "jpmiller",
      "author_url": "",
      "post_date": "09/17/2016 13:50:47",
      "content": "<p>Chaitanya - I usually use an r3 instance for bigger data sets in R because they come with more RAM than the equivalent m4. r3.xlarge should be enough for most things. I ran into some limits with this data on that type of instance, but it's because I wasn't managing my memory at all. With some of the techniques elsewhere in the forum, you can get by with far less.</p>\n\n<p>You can open a Jupyter/iPython notebook directly on your browser if it's set up right. The <a href=\"http://www.louisaslett.com/RStudio_AMI/\">Louis Aslett AMIs</a> have notebooks and Python built in now and you can run both at once. Follow the directions there. I had performance issues with using both though, and will go back to using my own image for notebooks.</p>\n\n<p>Good luck!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 135849,
      "author_name": "statchaitya",
      "author_url": "",
      "post_date": "09/17/2016 16:38:56",
      "content": "<p>Thanks a lot John.</p>\n\n<p>This really helps.</p>\n\n<p>Cheers!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 136755,
      "author_name": "",
      "author_url": "",
      "post_date": "09/24/2016 20:03:55",
      "content": "<p>@ChaitanyaGokhale - which part was more helpful, using an r3.xlarge instance or using different techniques to get by with less memory?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 136758,
      "author_name": "jeffd23",
      "author_url": "",
      "post_date": "09/24/2016 21:05:05",
      "content": "<p>I work in the cloud frequently for my day job and I've found Google Cloud Engine more user friendly for doing Kaggle stuff.  </p>\n\n<p>I wrote a blog post about setting up <a href=\"https://jeffdelaney.me/blog/running-jupyter-notebook-google-cloud-platform/\">GCE for iPython/Jupyter Notebooks</a>, which includes instructions for connecting via HTTP and saving the disk image to easily reproduce your environment. It's easy to build your own custom image, rather than use an AMI, if you're particular about your environment config. All this can be done in AWS as well. </p>\n\n<p>If you go this route, you'll probably want to use the v16 High Memory instance for this competition. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "135804": "Hi guys,\r\n\r\nI launched a rstudio server ami with m4.xlarge instance. I loaded the data, but it was taking ages to read the data files.\r\n\r\nAny pointers on which will be the best instance for this competition?\r\n\r\nIn addition, can somebody explain how to use python using the same AMI. Will I have to connect to the Ubuntu terminal through SSH to use python? Or can I directly open a iPython notebook in my browser just like I did with rstudio.\r\n\r\nThanks and regards,\r\nChaitanya.",
    "135836": "Chaitanya - I usually use an r3 instance for bigger data sets in R because they come with more RAM than the equivalent m4. r3.xlarge should be enough for most things. I ran into some limits with this data on that type of instance, but it's because I wasn't managing my memory at all. With some of the techniques elsewhere in the forum, you can get by with far less.\r\n\r\nYou can open a Jupyter/iPython notebook directly on your browser if it's set up right. The [Louis Aslett AMIs](http://www.louisaslett.com/RStudio_AMI/) have notebooks and Python built in now and you can run both at once. Follow the directions there. I had performance issues with using both though, and will go back to using my own image for notebooks.\r\n\r\nGood luck!",
    "135849": "Thanks a lot John.\r\n\r\nThis really helps.\r\n\r\nCheers!",
    "136755": "ChaitanyaGokhale - which part was more helpful, using an r3.xlarge instance or using different techniques to get by with less memory?",
    "136758": "I work in the cloud frequently for my day job and I've found Google Cloud Engine more user friendly for doing Kaggle stuff.  \r\n\r\nI wrote a blog post about setting up [GCE for iPython/Jupyter Notebooks][1], which includes instructions for connecting via HTTP and saving the disk image to easily reproduce your environment. It's easy to build your own custom image, rather than use an AMI, if you're particular about your environment config. All this can be done in AWS as well. \r\n\r\nIf you go this route, you'll probably want to use the v16 High Memory instance for this competition. \r\n\r\n\r\n  [1]: https://jeffdelaney.me/blog/running-jupyter-notebook-google-cloud-platform/"
  },
  "source": "meta"
}