{
  "id": 22093,
  "title": "How to improve pandas performance with this data ",
  "url": "/competitions/grupo-bimbo-inventory-demand/discussion/22093",
  "author_name": "",
  "post_date": "2016-07-07T17:13:05.190Z",
  "votes": null,
  "comment_count": 4,
  "views": 1202,
  "content": "<p>Hi! </p>\n\n<p>How are you guys doing? </p>\n\n<p>I was thinking of doing a Moving Avg of the demand to plot the behavior of products and clients over time. </p>\n\n<p>My main problem is that using pandas is taking to long for all the train data (I little over an hour iterating over smaller chunks)</p>\n\n<p>This is a sample of the code I'm using:</p>\n\n<pre><code> df = load_bimbo_data()\n#Using all the data \n\n%time df['Last'] = df.groupby(['Cliente_ID','Producto_ID'])['Demanda_uni_equil'].shift(1)\n%time df['MovingAvg'] = df.groupby(['Cliente_ID','Producto_ID'])['Last'].apply(pd.rolling_mean,3)\n\n\n# by chuncks \n\nproducts_chunk = df['Cliente_ID'].unique()\nproducts_chunk = np.array_split(products_chunk,20) # Avoid Ram issues by taking chunks\n\nfor chunk in products_chunk:\n    start = time.time()        \n\n    df.loc[df.Cliente_ID.isin(chunk)]['Last']  = df[df.Cliente_ID.isin(chunk)].groupby(['Cliente_ID','Producto_ID'])['Demanda_uni_equil'].shift(1)\n    %time df.loc[df.Cliente_ID.isin(chunk)]['MovingAvg'] = df[df.Cliente_ID.isin(chunk)].groupby(['Cliente_ID','Producto_ID'])['Last'].apply(pd.rolling_mean,3)\n\n\n    print(&quot;total time taken this loop: &quot;, time.time() - start)\n</code></pre>\n\n<p>Improvement Ideas:</p>\n\n<ul>\n<li>Moving to Spark Clusters using databricks</li>\n<li>Try a parallel approach (<a href=\"http://stackoverflow.com/questions/26187759/parallelize-apply-after-pandas-groupby\">http://stackoverflow.com/questions/26187759/parallelize-apply-after-pandas-groupby</a>) </li>\n<li>Try Blaze lib? or Dask? </li>\n</ul>\n\n<p>Any thoughts? </p>",
  "messages": [
    {
      "id": "126322",
      "postDate": "07/07/2016 17:13:05",
      "content": "<p>Hi! </p>\n\n<p>How are you guys doing? </p>\n\n<p>I was thinking of doing a Moving Avg of the demand to plot the behavior of products and clients over time. </p>\n\n<p>My main problem is that using pandas is taking to long for all the train data (I little over an hour iterating over smaller chunks)</p>\n\n<p>This is a sample of the code I'm using:</p>\n\n<pre><code> df = load_bimbo_data()\n#Using all the data \n\n%time df['Last'] = df.groupby(['Cliente_ID','Producto_ID'])['Demanda_uni_equil'].shift(1)\n%time df['MovingAvg'] = df.groupby(['Cliente_ID','Producto_ID'])['Last'].apply(pd.rolling_mean,3)\n\n\n# by chuncks \n\nproducts_chunk = df['Cliente_ID'].unique()\nproducts_chunk = np.array_split(products_chunk,20) # Avoid Ram issues by taking chunks\n\nfor chunk in products_chunk:\n    start = time.time()        \n\n    df.loc[df.Cliente_ID.isin(chunk)]['Last']  = df[df.Cliente_ID.isin(chunk)].groupby(['Cliente_ID','Producto_ID'])['Demanda_uni_equil'].shift(1)\n    %time df.loc[df.Cliente_ID.isin(chunk)]['MovingAvg'] = df[df.Cliente_ID.isin(chunk)].groupby(['Cliente_ID','Producto_ID'])['Last'].apply(pd.rolling_mean,3)\n\n\n    print(&quot;total time taken this loop: &quot;, time.time() - start)\n</code></pre>\n\n<p>Improvement Ideas:</p>\n\n<ul>\n<li>Moving to Spark Clusters using databricks</li>\n<li>Try a parallel approach (<a href=\"http://stackoverflow.com/questions/26187759/parallelize-apply-after-pandas-groupby\">http://stackoverflow.com/questions/26187759/parallelize-apply-after-pandas-groupby</a>) </li>\n<li>Try Blaze lib? or Dask? </li>\n</ul>\n\n<p>Any thoughts? </p>",
      "rawMarkdown": "Hi! \r\n\r\nHow are you guys doing? \r\n\r\nI was thinking of doing a Moving Avg of the demand to plot the behavior of products and clients over time. \r\n\r\nMy main problem is that using pandas is taking to long for all the train data (I little over an hour iterating over smaller chunks)\r\n\r\nThis is a sample of the code I'm using:\r\n\r\n   \r\n\r\n     df = load_bimbo_data()\r\n    #Using all the data \r\n        \r\n    %time df['Last'] = df.groupby(['Cliente_ID','Producto_ID'])['Demanda_uni_equil'].shift(1)\r\n    %time df['MovingAvg'] = df.groupby(['Cliente_ID','Producto_ID'])['Last'].apply(pd.rolling_mean,3)\r\n    \r\n        \r\n    # by chuncks \r\n        \r\n    products_chunk = df['Cliente_ID'].unique()\r\n    products_chunk = np.array_split(products_chunk,20) # Avoid Ram issues by taking chunks\r\n        \r\n    for chunk in products_chunk:\r\n        start = time.time()        \r\n            \r\n        df.loc[df.Cliente_ID.isin(chunk)]['Last']  = df[df.Cliente_ID.isin(chunk)].groupby(['Cliente_ID','Producto_ID'])['Demanda_uni_equil'].shift(1)\r\n        %time df.loc[df.Cliente_ID.isin(chunk)]['MovingAvg'] = df[df.Cliente_ID.isin(chunk)].groupby(['Cliente_ID','Producto_ID'])['Last'].apply(pd.rolling_mean,3)\r\n    \r\n                    \r\n        print(\"total time taken this loop: \", time.time() - start)\r\n\r\nImprovement Ideas:\r\n\r\n* Moving to Spark Clusters using databricks\r\n* Try a parallel approach (http://stackoverflow.com/questions/26187759/parallelize-apply-after-pandas-groupby) \r\n* Try Blaze lib? or Dask? \r\n\r\nAny thoughts?",
      "votes": null
    },
    {
      "id": "126368",
      "postDate": "07/07/2016 23:05:24",
      "content": "<p>You could try reading only the client id, product id, and demand into memory using <code>usecols</code> and <code>dtype</code>. That might allow you to process everything without splitting the data into chunks.</p>\n\n<p>As for your improvement ideas, I've only ever used dask, and it's an extremely limited library. A lot of it is unfinished.</p>\n\n<p>My last thought is to iterate through the train/test files with the csv module, calculating the moving average and writing the results to new train/test files. This would take a while to process, but once it is finished you would never need to rerun the calculations.</p>",
      "rawMarkdown": "You could try reading only the client id, product id, and demand into memory using `usecols` and `dtype`. That might allow you to process everything without splitting the data into chunks.\r\n\r\nAs for your improvement ideas, I've only ever used dask, and it's an extremely limited library. A lot of it is unfinished.\r\n\r\nMy last thought is to iterate through the train/test files with the csv module, calculating the moving average and writing the results to new train/test files. This would take a while to process, but once it is finished you would never need to rerun the calculations.",
      "votes": null
    },
    {
      "id": "126414",
      "postDate": "07/08/2016 09:00:14",
      "content": "<p>.loc is extremely slow.  Move to the underlying Numpy arrays for each series of data frames, use cython or numba, see: <a href=\"http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html\">http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html</a></p>",
      "rawMarkdown": ".loc is extremely slow.  Move to the underlying Numpy arrays for each series of data frames, use cython or numba, see: http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html",
      "votes": null
    },
    {
      "id": "126476",
      "postDate": "07/08/2016 18:20:41",
      "content": "<p>@Kevin Kirchhoff yes! I forgot to include that I only use client_id, product_id and demand. </p>\n\n<p>I will definitely run this only once and save as a .csv file. But it is a fun problem and I wanted to try and optimize. </p>\n\n<p>@CPMP  I thought of using cython but I  wanted to explore other available options, I will probably give it a go when I have some spare time </p>",
      "rawMarkdown": "Kevin Kirchhoff yes! I forgot to include that I only use client_id, product_id and demand. \r\n\r\nI will definitely run this only once and save as a .csv file. But it is a fun problem and I wanted to try and optimize. \r\n\r\n@CPMP  I thought of using cython but I  wanted to explore other available options, I will probably give it a go when I have some spare time",
      "votes": null
    },
    {
      "id": "126484",
      "postDate": "07/08/2016 20:02:00",
      "content": "<p>Switching to Numy will buy you significant speedup, even without cython or numba.</p>",
      "rawMarkdown": "Switching to Numy will buy you significant speedup, even without cython or numba.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 126368,
      "author_name": "kevinkirchhoff",
      "author_url": "",
      "post_date": "07/07/2016 23:05:24",
      "content": "<p>You could try reading only the client id, product id, and demand into memory using <code>usecols</code> and <code>dtype</code>. That might allow you to process everything without splitting the data into chunks.</p>\n\n<p>As for your improvement ideas, I've only ever used dask, and it's an extremely limited library. A lot of it is unfinished.</p>\n\n<p>My last thought is to iterate through the train/test files with the csv module, calculating the moving average and writing the results to new train/test files. This would take a while to process, but once it is finished you would never need to rerun the calculations.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126414,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "07/08/2016 09:00:14",
      "content": "<p>.loc is extremely slow.  Move to the underlying Numpy arrays for each series of data frames, use cython or numba, see: <a href=\"http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html\">http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126476,
      "author_name": "jherna17",
      "author_url": "",
      "post_date": "07/08/2016 18:20:41",
      "content": "<p>@Kevin Kirchhoff yes! I forgot to include that I only use client_id, product_id and demand. </p>\n\n<p>I will definitely run this only once and save as a .csv file. But it is a fun problem and I wanted to try and optimize. </p>\n\n<p>@CPMP  I thought of using cython but I  wanted to explore other available options, I will probably give it a go when I have some spare time </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 126484,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "07/08/2016 20:02:00",
      "content": "<p>Switching to Numy will buy you significant speedup, even without cython or numba.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "126322": "Hi! \r\n\r\nHow are you guys doing? \r\n\r\nI was thinking of doing a Moving Avg of the demand to plot the behavior of products and clients over time. \r\n\r\nMy main problem is that using pandas is taking to long for all the train data (I little over an hour iterating over smaller chunks)\r\n\r\nThis is a sample of the code I'm using:\r\n\r\n   \r\n\r\n     df = load_bimbo_data()\r\n    #Using all the data \r\n        \r\n    %time df['Last'] = df.groupby(['Cliente_ID','Producto_ID'])['Demanda_uni_equil'].shift(1)\r\n    %time df['MovingAvg'] = df.groupby(['Cliente_ID','Producto_ID'])['Last'].apply(pd.rolling_mean,3)\r\n    \r\n        \r\n    # by chuncks \r\n        \r\n    products_chunk = df['Cliente_ID'].unique()\r\n    products_chunk = np.array_split(products_chunk,20) # Avoid Ram issues by taking chunks\r\n        \r\n    for chunk in products_chunk:\r\n        start = time.time()        \r\n            \r\n        df.loc[df.Cliente_ID.isin(chunk)]['Last']  = df[df.Cliente_ID.isin(chunk)].groupby(['Cliente_ID','Producto_ID'])['Demanda_uni_equil'].shift(1)\r\n        %time df.loc[df.Cliente_ID.isin(chunk)]['MovingAvg'] = df[df.Cliente_ID.isin(chunk)].groupby(['Cliente_ID','Producto_ID'])['Last'].apply(pd.rolling_mean,3)\r\n    \r\n                    \r\n        print(\"total time taken this loop: \", time.time() - start)\r\n\r\nImprovement Ideas:\r\n\r\n* Moving to Spark Clusters using databricks\r\n* Try a parallel approach (http://stackoverflow.com/questions/26187759/parallelize-apply-after-pandas-groupby) \r\n* Try Blaze lib? or Dask? \r\n\r\nAny thoughts?",
    "126368": "You could try reading only the client id, product id, and demand into memory using `usecols` and `dtype`. That might allow you to process everything without splitting the data into chunks.\r\n\r\nAs for your improvement ideas, I've only ever used dask, and it's an extremely limited library. A lot of it is unfinished.\r\n\r\nMy last thought is to iterate through the train/test files with the csv module, calculating the moving average and writing the results to new train/test files. This would take a while to process, but once it is finished you would never need to rerun the calculations.",
    "126414": ".loc is extremely slow.  Move to the underlying Numpy arrays for each series of data frames, use cython or numba, see: http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html",
    "126476": "Kevin Kirchhoff yes! I forgot to include that I only use client_id, product_id and demand. \r\n\r\nI will definitely run this only once and save as a .csv file. But it is a fun problem and I wanted to try and optimize. \r\n\r\n@CPMP  I thought of using cython but I  wanted to explore other available options, I will probably give it a go when I have some spare time",
    "126484": "Switching to Numy will buy you significant speedup, even without cython or numba."
  },
  "source": "meta"
}