{
  "id": 71499,
  "title": "Suggestions for handling the big data problem",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/71499",
  "author_name": "",
  "post_date": "2018-11-14T07:18:25.520089300Z",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi chaps, for those fellow kagglers working with this big data problem on their tiny puny PCs or laptops, here are my humble suggestions from a fellow kaggler with a budget of $0:</p>\n\n<p>In Jupyter notebook use the following code:</p>\n\n<p>*import pandas as pd\ndf = pd.read_csv('./downloads/gstore/train_v2.csv', dtype={'fullVisitorId': object}, nrows=1000)\ndf.count()*</p>\n\n<p>This loads the 1st 1000 rows for you to do your initial data exploration.\nOnce you get a feel of what you want to do, you then have to read your data in batches:</p>\n\n<p>*df = pd.read_csv('./downloads/gstore/train_v2.csv', dtype={'fullVisitorId': object}, nrows=100001, skiprows=range(1, 100000))*</p>\n\n<p>The code above reads 100,001 rows, but skips row 1-100,000. So you essentially read row 0 (for the column headers, rows 100,001 to 200,000 in the original file.</p>",
  "messages": [
    {
      "id": "420829",
      "postDate": "11/14/2018 07:18:25",
      "content": "<p>Hi chaps, for those fellow kagglers working with this big data problem on their tiny puny PCs or laptops, here are my humble suggestions from a fellow kaggler with a budget of $0:</p>\n\n<p>In Jupyter notebook use the following code:</p>\n\n<p>*import pandas as pd\ndf = pd.read_csv('./downloads/gstore/train_v2.csv', dtype={'fullVisitorId': object}, nrows=1000)\ndf.count()*</p>\n\n<p>This loads the 1st 1000 rows for you to do your initial data exploration.\nOnce you get a feel of what you want to do, you then have to read your data in batches:</p>\n\n<p>*df = pd.read_csv('./downloads/gstore/train_v2.csv', dtype={'fullVisitorId': object}, nrows=100001, skiprows=range(1, 100000))*</p>\n\n<p>The code above reads 100,001 rows, but skips row 1-100,000. So you essentially read row 0 (for the column headers, rows 100,001 to 200,000 in the original file.</p>",
      "rawMarkdown": "Hi chaps, for those fellow kagglers working with this big data problem on their tiny puny PCs or laptops, here are my humble suggestions from a fellow kaggler with a budget of $0:\n\nIn Jupyter notebook use the following code:\n\n*import pandas as pd\ndf = pd.read_csv('./downloads/gstore/train_v2.csv', dtype={'fullVisitorId': object}, nrows=1000)\ndf.count()*\n\nThis loads the 1st 1000 rows for you to do your initial data exploration.\nOnce you get a feel of what you want to do, you then have to read your data in batches:\n\n*df = pd.read_csv('./downloads/gstore/train_v2.csv', dtype={'fullVisitorId': object}, nrows=100001, skiprows=range(1, 100000))*\n\nThe code above reads 100,001 rows, but skips row 1-100,000. So you essentially read row 0 (for the column headers, rows 100,001 to 200,000 in the original file.",
      "votes": null
    },
    {
      "id": "421009",
      "postDate": "11/14/2018 13:15:16",
      "content": "<p>I use this same tip, thanks teesoong for sharing.</p>",
      "rawMarkdown": "I use this same tip, thanks teesoong for sharing.",
      "votes": null
    },
    {
      "id": "426557",
      "postDate": "11/23/2018 12:48:36",
      "content": "<p>See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\n<a href=\"https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd\">https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd</a></p>",
      "rawMarkdown": "See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\nhttps://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 421009,
      "author_name": "pierreco",
      "author_url": "",
      "post_date": "11/14/2018 13:15:16",
      "content": "<p>I use this same tip, thanks teesoong for sharing.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 426557,
      "author_name": "mlisovyi",
      "author_url": "",
      "post_date": "11/23/2018 12:48:36",
      "content": "<p>See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\n<a href=\"https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd\">https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "420829": "Hi chaps, for those fellow kagglers working with this big data problem on their tiny puny PCs or laptops, here are my humble suggestions from a fellow kaggler with a budget of $0:\n\nIn Jupyter notebook use the following code:\n\n*import pandas as pd\ndf = pd.read_csv('./downloads/gstore/train_v2.csv', dtype={'fullVisitorId': object}, nrows=1000)\ndf.count()*\n\nThis loads the 1st 1000 rows for you to do your initial data exploration.\nOnce you get a feel of what you want to do, you then have to read your data in batches:\n\n*df = pd.read_csv('./downloads/gstore/train_v2.csv', dtype={'fullVisitorId': object}, nrows=100001, skiprows=range(1, 100000))*\n\nThe code above reads 100,001 rows, but skips row 1-100,000. So you essentially read row 0 (for the column headers, rows 100,001 to 200,000 in the original file.",
    "421009": "I use this same tip, thanks teesoong for sharing.",
    "426557": "See this kernel on how to do it using dask, which allows to process data in chunks without reading the whole dataframe in memory:\nhttps://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd"
  },
  "source": "meta"
}