{
  "id": 21836,
  "title": "PANDAS seems to be too slow for this project",
  "url": "/competitions/grupo-bimbo-inventory-demand/discussion/21836",
  "author_name": "",
  "post_date": "2016-06-22T07:31:10.967Z",
  "votes": 2,
  "comment_count": 2,
  "views": 916,
  "content": "<p>I am using my own laptop (MAC Pro) with Python PANDAS to do this project, I am trying to calculate each entry to the submission file(around 7million rows), I roughly estimated the time consumption to do the all the calculation ~ on the order of 2 weeks! </p>\n\n<p>What I essentially do is to read in information from test.csv(CustomerID, ProductID, etc.) and compute an estimate using some model according to the relevant rows in the training set dataframe .</p>\n\n<p>It would be really helpful to know how people generally deal with this kind of situation, is there any easy way to enhance the performance of PANDAS, or should I just use another tool.</p>\n\n<p>Thank you</p>",
  "messages": [
    {
      "id": "124773",
      "postDate": "06/22/2016 07:31:10",
      "content": "<p>I am using my own laptop (MAC Pro) with Python PANDAS to do this project, I am trying to calculate each entry to the submission file(around 7million rows), I roughly estimated the time consumption to do the all the calculation ~ on the order of 2 weeks! </p>\n\n<p>What I essentially do is to read in information from test.csv(CustomerID, ProductID, etc.) and compute an estimate using some model according to the relevant rows in the training set dataframe .</p>\n\n<p>It would be really helpful to know how people generally deal with this kind of situation, is there any easy way to enhance the performance of PANDAS, or should I just use another tool.</p>\n\n<p>Thank you</p>",
      "rawMarkdown": "I am using my own laptop (MAC Pro) with Python PANDAS to do this project, I am trying to calculate each entry to the submission file(around 7million rows), I roughly estimated the time consumption to do the all the calculation ~ on the order of 2 weeks! \r\n\r\nWhat I essentially do is to read in information from test.csv(CustomerID, ProductID, etc.) and compute an estimate using some model according to the relevant rows in the training set dataframe .\r\n\r\nIt would be really helpful to know how people generally deal with this kind of situation, is there any easy way to enhance the performance of PANDAS, or should I just use another tool.\r\n\r\nThank you",
      "votes": null
    },
    {
      "id": "124783",
      "postDate": "06/22/2016 10:05:08",
      "content": "<p>Are you looping over Pandas dataframes using indexes and ix, iloc or loc?</p>\n\n<p>If so then you should switch to the underlying Numpy arrays for each series of the dataframe.  You'll get at least a 10x speedup.</p>\n\n<p>This is explained in pandas documentation: <a href=\"http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html\">http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html</a></p>\n\n<p>Here are also some blog posts about how to make Python code run fast:</p>\n\n<p><a href=\"https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Meets_Julia_Micro_Performance?lang=en\">https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Meets_Julia_Micro_Performance?lang=en</a></p>\n\n<p><a href=\"https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Is_Not_C_Take_Two?lang=en\">https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Is_Not_C_Take_Two?lang=en</a></p>",
      "rawMarkdown": "Are you looping over Pandas dataframes using indexes and ix, iloc or loc?\r\n\r\nIf so then you should switch to the underlying Numpy arrays for each series of the dataframe.  You'll get at least a 10x speedup.\r\n\r\nThis is explained in pandas documentation: http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html\r\n\r\nHere are also some blog posts about how to make Python code run fast:\r\n\r\nhttps://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Meets_Julia_Micro_Performance?lang=en\r\n\r\nhttps://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Is_Not_C_Take_Two?lang=en",
      "votes": null
    },
    {
      "id": "124845",
      "postDate": "06/22/2016 18:36:03",
      "content": "<p>[quote=CPMP;124783]</p>\n\n<p>Are you looping over Pandas dataframes using indexes and ix, iloc or loc?</p>\n\n<p>If so then you should switch to the underlying Numpy arrays for each series of the dataframe.  You'll get at least a 10x speedup.</p>\n\n<p>This is explained in pandas documentation: <a href=\"http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html\">http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html</a></p>\n\n<p>Here are also some blog posts about how to make Python code run fast:</p>\n\n<p><a href=\"https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Meets_Julia_Micro_Performance?lang=en\">https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Meets_Julia_Micro_Performance?lang=en</a></p>\n\n<p><a href=\"https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Is_Not_C_Take_Two?lang=en\">https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Is_Not_C_Take_Two?lang=en</a></p>\n\n<p>[/quote]</p>\n\n<p>Thank you for sharing your wisdom&#65292;best of luck with this project!</p>",
      "rawMarkdown": "[quote=CPMP;124783]\r\n\r\nAre you looping over Pandas dataframes using indexes and ix, iloc or loc?\r\n\r\nIf so then you should switch to the underlying Numpy arrays for each series of the dataframe.  You'll get at least a 10x speedup.\r\n\r\nThis is explained in pandas documentation: http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html\r\n\r\nHere are also some blog posts about how to make Python code run fast:\r\n\r\nhttps://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Meets_Julia_Micro_Performance?lang=en\r\n\r\nhttps://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Is_Not_C_Take_Two?lang=en\r\n\r\n\r\n\r\n[/quote]\r\n\r\nThank you for sharing your wisdom，best of luck with this project!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 124783,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "06/22/2016 10:05:08",
      "content": "<p>Are you looping over Pandas dataframes using indexes and ix, iloc or loc?</p>\n\n<p>If so then you should switch to the underlying Numpy arrays for each series of the dataframe.  You'll get at least a 10x speedup.</p>\n\n<p>This is explained in pandas documentation: <a href=\"http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html\">http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html</a></p>\n\n<p>Here are also some blog posts about how to make Python code run fast:</p>\n\n<p><a href=\"https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Meets_Julia_Micro_Performance?lang=en\">https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Meets_Julia_Micro_Performance?lang=en</a></p>\n\n<p><a href=\"https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Is_Not_C_Take_Two?lang=en\">https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Is_Not_C_Take_Two?lang=en</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 124845,
      "author_name": "wwang2",
      "author_url": "",
      "post_date": "06/22/2016 18:36:03",
      "content": "<p>[quote=CPMP;124783]</p>\n\n<p>Are you looping over Pandas dataframes using indexes and ix, iloc or loc?</p>\n\n<p>If so then you should switch to the underlying Numpy arrays for each series of the dataframe.  You'll get at least a 10x speedup.</p>\n\n<p>This is explained in pandas documentation: <a href=\"http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html\">http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html</a></p>\n\n<p>Here are also some blog posts about how to make Python code run fast:</p>\n\n<p><a href=\"https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Meets_Julia_Micro_Performance?lang=en\">https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Meets_Julia_Micro_Performance?lang=en</a></p>\n\n<p><a href=\"https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Is_Not_C_Take_Two?lang=en\">https://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Is_Not_C_Take_Two?lang=en</a></p>\n\n<p>[/quote]</p>\n\n<p>Thank you for sharing your wisdom&#65292;best of luck with this project!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "124773": "I am using my own laptop (MAC Pro) with Python PANDAS to do this project, I am trying to calculate each entry to the submission file(around 7million rows), I roughly estimated the time consumption to do the all the calculation ~ on the order of 2 weeks! \r\n\r\nWhat I essentially do is to read in information from test.csv(CustomerID, ProductID, etc.) and compute an estimate using some model according to the relevant rows in the training set dataframe .\r\n\r\nIt would be really helpful to know how people generally deal with this kind of situation, is there any easy way to enhance the performance of PANDAS, or should I just use another tool.\r\n\r\nThank you",
    "124783": "Are you looping over Pandas dataframes using indexes and ix, iloc or loc?\r\n\r\nIf so then you should switch to the underlying Numpy arrays for each series of the dataframe.  You'll get at least a 10x speedup.\r\n\r\nThis is explained in pandas documentation: http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html\r\n\r\nHere are also some blog posts about how to make Python code run fast:\r\n\r\nhttps://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Meets_Julia_Micro_Performance?lang=en\r\n\r\nhttps://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Is_Not_C_Take_Two?lang=en",
    "124845": "[quote=CPMP;124783]\r\n\r\nAre you looping over Pandas dataframes using indexes and ix, iloc or loc?\r\n\r\nIf so then you should switch to the underlying Numpy arrays for each series of the dataframe.  You'll get at least a 10x speedup.\r\n\r\nThis is explained in pandas documentation: http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html\r\n\r\nHere are also some blog posts about how to make Python code run fast:\r\n\r\nhttps://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Meets_Julia_Micro_Performance?lang=en\r\n\r\nhttps://www.ibm.com/developerworks/community/blogs/jfp/entry/Python_Is_Not_C_Take_Two?lang=en\r\n\r\n\r\n\r\n[/quote]\r\n\r\nThank you for sharing your wisdom，best of luck with this project!"
  },
  "source": "meta"
}