{
  "id": 76359,
  "title": "is there a way to compute groupby-agg in faster version with numpy?",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/76359",
  "author_name": "",
  "post_date": "2019-01-01T19:37:35.687860200Z",
  "votes": null,
  "comment_count": 4,
  "views": 0,
  "content": "<p>My computer is a little bit slow on doing groupby-agg and i want to make it faster. Is there any way to use numpy array to make it much more faster? I think it will be usefull!</p>",
  "messages": [
    {
      "id": "448660",
      "postDate": "01/01/2019 19:37:35",
      "content": "<p>My computer is a little bit slow on doing groupby-agg and i want to make it faster. Is there any way to use numpy array to make it much more faster? I think it will be usefull!</p>",
      "rawMarkdown": "My computer is a little bit slow on doing groupby-agg and i want to make it faster. Is there any way to use numpy array to make it much more faster? I think it will be usefull!",
      "votes": null
    },
    {
      "id": "448665",
      "postDate": "01/01/2019 19:47:56",
      "content": "<p><a href=\"https://colab.research.google.com/\">https://colab.research.google.com/</a></p>",
      "rawMarkdown": "https://colab.research.google.com/",
      "votes": null
    },
    {
      "id": "448681",
      "postDate": "01/01/2019 20:42:16",
      "content": "<p>Hi! Ka.Kr, \nAre you using the data frame memory optimization function (reduce_mem_usage), that is quite popular around this competition before running the aggregation, another option could be to use Dask and distribute the aggregation by feature.\nwith a quick google search came across this:\n<a href=\"https://jakevdp.github.io/PythonDataScienceHandbook/02.04-computation-on-arrays-aggregates.html\">https://jakevdp.github.io/PythonDataScienceHandbook/02.04-computation-on-arrays-aggregates.html</a></p>",
      "rawMarkdown": "Hi! Ka.Kr, \nAre you using the data frame memory optimization function (reduce_mem_usage), that is quite popular around this competition before running the aggregation, another option could be to use Dask and distribute the aggregation by feature.\nwith a quick google search came across this:\nhttps://jakevdp.github.io/PythonDataScienceHandbook/02.04-computation-on-arrays-aggregates.html",
      "votes": null
    },
    {
      "id": "448694",
      "postDate": "01/01/2019 21:32:30",
      "content": "<p>An approach is to chop the large dataframe into pieces along the columns wanted to perform groupby, then apply multiprocessing to execute groupby-agg. Or you might want to do it by dask but its built-in agg functions are limited compared to pandas. </p>",
      "rawMarkdown": "An approach is to chop the large dataframe into pieces along the columns wanted to perform groupby, then apply multiprocessing to execute groupby-agg. Or you might want to do it by dask but its built-in agg functions are limited compared to pandas.",
      "votes": null
    },
    {
      "id": "448799",
      "postDate": "01/02/2019 05:25:06",
      "content": "<p>I find it much faster to run agg for one column at a time and concatenate the results, rather than run one giant groupby-agg.</p>",
      "rawMarkdown": "I find it much faster to run agg for one column at a time and concatenate the results, rather than run one giant groupby-agg.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 448665,
      "author_name": "alialhumaid",
      "author_url": "",
      "post_date": "01/01/2019 19:47:56",
      "content": "<p><a href=\"https://colab.research.google.com/\">https://colab.research.google.com/</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 448681,
      "author_name": "cv13j0",
      "author_url": "",
      "post_date": "01/01/2019 20:42:16",
      "content": "<p>Hi! Ka.Kr, \nAre you using the data frame memory optimization function (reduce_mem_usage), that is quite popular around this competition before running the aggregation, another option could be to use Dask and distribute the aggregation by feature.\nwith a quick google search came across this:\n<a href=\"https://jakevdp.github.io/PythonDataScienceHandbook/02.04-computation-on-arrays-aggregates.html\">https://jakevdp.github.io/PythonDataScienceHandbook/02.04-computation-on-arrays-aggregates.html</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 448694,
      "author_name": "cttsai",
      "author_url": "",
      "post_date": "01/01/2019 21:32:30",
      "content": "<p>An approach is to chop the large dataframe into pieces along the columns wanted to perform groupby, then apply multiprocessing to execute groupby-agg. Or you might want to do it by dask but its built-in agg functions are limited compared to pandas. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 448799,
      "author_name": "peterhurford",
      "author_url": "",
      "post_date": "01/02/2019 05:25:06",
      "content": "<p>I find it much faster to run agg for one column at a time and concatenate the results, rather than run one giant groupby-agg.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "448660": "My computer is a little bit slow on doing groupby-agg and i want to make it faster. Is there any way to use numpy array to make it much more faster? I think it will be usefull!",
    "448665": "https://colab.research.google.com/",
    "448681": "Hi! Ka.Kr, \nAre you using the data frame memory optimization function (reduce_mem_usage), that is quite popular around this competition before running the aggregation, another option could be to use Dask and distribute the aggregation by feature.\nwith a quick google search came across this:\nhttps://jakevdp.github.io/PythonDataScienceHandbook/02.04-computation-on-arrays-aggregates.html",
    "448694": "An approach is to chop the large dataframe into pieces along the columns wanted to perform groupby, then apply multiprocessing to execute groupby-agg. Or you might want to do it by dask but its built-in agg functions are limited compared to pandas.",
    "448799": "I find it much faster to run agg for one column at a time and concatenate the results, rather than run one giant groupby-agg."
  },
  "source": "meta"
}