Hey,
\n\nI publish a bit lately my solution.\nhttps://github.com/jacquespeeters/instacart-market-basket-analysis
\n\nMost of my code is done with Pandas not in pure python (i'm an R user, and therefore i use dplyr/data.table a lot) I'm still learning Python, therefore some code might not be good.
\n\nMy strongest feature is a binary encoding of the purchase pattern of products by users. Binary encoding is a nice way to encode a series of 1/0, isn't it?
\n\n\norder_prior[\"UP_order_strike\"] = 1 / 2 ** (order_prior[\"order_number_reverse\"])\nusers_products = order_prior. \\\n groupby([\"user_id\", \"product_id\"]). \\\n agg({..., \\\n 'UP_order_strike': {\"UP_order_strike\": \"sum\"}})\n
I created a model specifically to predict None.
\n\nThe main idea is to re-use feature engineering already done for the main model (like the user profile) and to create new features based on the prediction of the main model.
\n\nHere is the piece of code highlighting the new features created in order to predict None.
\n\n\ndf_pred = df_pred.groupby([\"order_id\", \"user_id\"]). \\\n agg({'pred_minus': {'pred_none_prod': \"prod\"}, \\\n 'pred': {'pred_basket_sum': \"sum\", 'pred_basket_std':'std'}})\n
Obviously it is done in a cross-validation way in order to avoid any leakage.
\n\nIt gave me a nice overall boost of +0.01
\n\nI should write something about it, because you don't want to // every group because it creates a lot of data chunk. One would prefer to create fewer chunk and apply groupby on these chunk (i use a rule of thumb of 3 chunk per core).\nBecause transferring data chunk to cores cost computing power, therefore you only want to parallelize the minimal amount of chunk. But you also need more chunks than cores, because some cores might be faster than others, therefore faster cores must be able to process multiple chunks and slower cores fewer.
\n\nI'm happy to won my first silver medal but i'm a not bitter given the fact i was better placed before Faron and Sh1ng script. Anyway i'll do better next time :)
", "comments": [ { "id": 416750, "author_name": "LongYin/杰少", "author_url": "", "post_date": "2018-11-07T08:13:30.743000", "content": "Thanks for sharing, from your git, I see the following running commands,\n1. python3 create_orders.py\n2. python3 word2vector.py\n3. python3 create_old_orders.py\n4. python3 main.py\nBut I did not see the file create_orders.py, am I wrong?
", "votes": 0, "replies": [] }, { "id": 214773, "author_name": "yue yu", "author_url": "", "post_date": "2017-08-18T08:20:26.397000", "content": "Thanks for sharing.Helps me a lot
", "votes": 0, "replies": [] } ], "comment_count": 2, "messages": [ { "id": 214613, "postDate": "2017-08-17T15:53:57.033Z", "content": "Hey,
\n\nI publish a bit lately my solution.\nhttps://github.com/jacquespeeters/instacart-market-basket-analysis
\n\nMost of my code is done with Pandas not in pure python (i'm an R user, and therefore i use dplyr/data.table a lot) I'm still learning Python, therefore some code might not be good.
\n\nMy strongest feature is a binary encoding of the purchase pattern of products by users. Binary encoding is a nice way to encode a series of 1/0, isn't it?
\n\n\norder_prior[\"UP_order_strike\"] = 1 / 2 ** (order_prior[\"order_number_reverse\"])\nusers_products = order_prior. \\\n groupby([\"user_id\", \"product_id\"]). \\\n agg({..., \\\n 'UP_order_strike': {\"UP_order_strike\": \"sum\"}})\n
I created a model specifically to predict None.
\n\nThe main idea is to re-use feature engineering already done for the main model (like the user profile) and to create new features based on the prediction of the main model.
\n\nHere is the piece of code highlighting the new features created in order to predict None.
\n\n\ndf_pred = df_pred.groupby([\"order_id\", \"user_id\"]). \\\n agg({'pred_minus': {'pred_none_prod': \"prod\"}, \\\n 'pred': {'pred_basket_sum': \"sum\", 'pred_basket_std':'std'}})\n
Obviously it is done in a cross-validation way in order to avoid any leakage.
\n\nIt gave me a nice overall boost of +0.01
\n\nI should write something about it, because you don't want to // every group because it creates a lot of data chunk. One would prefer to create fewer chunk and apply groupby on these chunk (i use a rule of thumb of 3 chunk per core).\nBecause transferring data chunk to cores cost computing power, therefore you only want to parallelize the minimal amount of chunk. But you also need more chunks than cores, because some cores might be faster than others, therefore faster cores must be able to process multiple chunks and slower cores fewer.
\n\nI'm happy to won my first silver medal but i'm a not bitter given the fact i was better placed before Faron and Sh1ng script. Anyway i'll do better next time :)
", "rawMarkdown": "Hey,\n\nI publish a bit lately my solution.\nhttps://github.com/jacquespeeters/instacart-market-basket-analysis\n\nMost of my code is done with Pandas not in pure python (i'm an R user, and therefore i use dplyr/data.table a lot) I'm still learning Python, therefore some code might not be good. \n\n### Strongest feature\nMy strongest feature is a binary encoding of the purchase pattern of products by users. Binary encoding is a nice way to encode a series of 1/0, isn't it?\n\n```\norder_prior[\"UP_order_strike\"] = 1 / 2 ** (order_prior[\"order_number_reverse\"])\nusers_products = order_prior. \\\n groupby([\"user_id\", \"product_id\"]). \\\n agg({..., \\\n 'UP_order_strike': {\"UP_order_strike\": \"sum\"}})\n```\n\n### None handling\nI created a model specifically to predict None.\n\nThe main idea is to re-use feature engineering already done for the main model (like the user profile) and to create new features based on the prediction of the main model. \n\nHere is the piece of code highlighting the new features created in order to predict None.\n\n\n\n```\ndf_pred = df_pred.groupby([\"order_id\", \"user_id\"]). \\\n agg({'pred_minus': {'pred_none_prod': \"prod\"}, \\\n 'pred': {'pred_basket_sum': \"sum\", 'pred_basket_std':'std'}})\n```\n\nObviously it is done in a cross-validation way in order to avoid any leakage.\n\nIt gave me a nice overall boost of +0.01 \n\n### Parallel groupby in Pandas\nI should write something about it, because you don't want to // every group because it creates a lot of data chunk. One would prefer to create fewer chunk and apply groupby on these chunk (i use a rule of thumb of 3 chunk per core).\nBecause transferring data chunk to cores cost computing power, therefore you only want to parallelize the minimal amount of chunk. But you also need more chunks than cores, because some cores might be faster than others, therefore faster cores must be able to process multiple chunks and slower cores fewer.\n\n## Conclusion\nI'm happy to won my first silver medal but i'm a not bitter given the fact i was better placed before Faron and Sh1ng script. Anyway i'll do better next time :)\n", "votes": 16 }, { "id": 416750, "postDate": "2018-11-07T08:13:30.743Z", "content": "Thanks for sharing, from your git, I see the following running commands,\n1. python3 create_orders.py\n2. python3 word2vector.py\n3. python3 create_old_orders.py\n4. python3 main.py\nBut I did not see the file create_orders.py, am I wrong?
", "rawMarkdown": "Thanks for sharing, from your git, I see the following running commands,\n1. python3 create_orders.py\n2. python3 word2vector.py\n3. python3 create_old_orders.py\n4. python3 main.py\nBut I did not see the file **create_orders.py**, am I wrong?" }, { "id": 214773, "postDate": "2017-08-18T08:20:26.397Z", "content": "Thanks for sharing.Helps me a lot
", "rawMarkdown": "Thanks for sharing.Helps me a lot\n" } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T02:01:57Z", "competition_id": 6644, "competition_slug": "basket-analysis", "team_id": 718087, "solution_url": "/competitions/basket-analysis/writeups/jacques-peeters-54th-solution-with-a-nice-user-pro" }