Hello everyone:
\n\nI'm a bit confused with how people are using GroupKFold in their kernels. Hopefully I'm not having a brain fart but it seems to me that we are not using it the way it was intended.
\n\nThis is the code being used in a dozen or so kernels:
\n\ndef get_folds(df=None, n_splits=5):\n\"\"\"Returns dataframe indices corresponding to Visitors Group KFold\"\"\"\n# Get sorted unique visitors\nunique_vis = np.array(sorted(df['fullVisitorId'].unique()))\n\n# Get folds\nfolds = GroupKFold(n_splits=n_splits)\nfold_ids = []\nids = np.arange(df.shape[0])\nfor trn_vis, val_vis in folds.split(X=unique_vis, y=unique_vis, groups=unique_vis):\n fold_ids.append(\n [\n ids[df['fullVisitorId'].isin(unique_vis[trn_vis])],\n ids[df['fullVisitorId'].isin(unique_vis[val_vis])]\n ]\n )\n\nreturn fold_ids\nI compared that to the doc here: \nhttp://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html
\n\nThis is what I understand the code above does:
\n\ncreate an array of unique visitor IDs
use GroupKFold to split that array into folds.
map those visitorId folds back to the original dataFrame
It appears to me that we are using GroupKFold passing data where all the groups have a unique item (since we pass groups = unique_vis, and unique_vis contains all unique visitorIDs)
\n\nTherefore, I'm guessing that our code would get the same result using vanilla KFold.
\n\nMy understanding of GroupKFold is that it would normally be used in the following manner:
\n\ngroup_kfold.split(train[cols], train['target'], train['fullVisitorId'])
\n\nI wonder if by using GroupKFold the way we are, we are missing out on some features such as balancing the group sizes between folds.
\n\nOr maybe we don't want to use GroupKFold at all; and instead of splitting visitorIDs using vanilla KFold, we could use StratifiedKFold (based on whether a visitorID generates revenue or not)
\n\nAppreciating any thoughts!
", "messages": [ { "id": "399736", "postDate": "10/06/2018 15:56:35", "content": "Hello everyone:
\n\nI'm a bit confused with how people are using GroupKFold in their kernels. Hopefully I'm not having a brain fart but it seems to me that we are not using it the way it was intended.
\n\nThis is the code being used in a dozen or so kernels:
\n\ndef get_folds(df=None, n_splits=5):\n\"\"\"Returns dataframe indices corresponding to Visitors Group KFold\"\"\"\n# Get sorted unique visitors\nunique_vis = np.array(sorted(df['fullVisitorId'].unique()))\n\n# Get folds\nfolds = GroupKFold(n_splits=n_splits)\nfold_ids = []\nids = np.arange(df.shape[0])\nfor trn_vis, val_vis in folds.split(X=unique_vis, y=unique_vis, groups=unique_vis):\n fold_ids.append(\n [\n ids[df['fullVisitorId'].isin(unique_vis[trn_vis])],\n ids[df['fullVisitorId'].isin(unique_vis[val_vis])]\n ]\n )\n\nreturn fold_ids\nI compared that to the doc here: \nhttp://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html
\n\nThis is what I understand the code above does:
\n\ncreate an array of unique visitor IDs
use GroupKFold to split that array into folds.
map those visitorId folds back to the original dataFrame
It appears to me that we are using GroupKFold passing data where all the groups have a unique item (since we pass groups = unique_vis, and unique_vis contains all unique visitorIDs)
\n\nTherefore, I'm guessing that our code would get the same result using vanilla KFold.
\n\nMy understanding of GroupKFold is that it would normally be used in the following manner:
\n\ngroup_kfold.split(train[cols], train['target'], train['fullVisitorId'])
\n\nI wonder if by using GroupKFold the way we are, we are missing out on some features such as balancing the group sizes between folds.
\n\nOr maybe we don't want to use GroupKFold at all; and instead of splitting visitorIDs using vanilla KFold, we could use StratifiedKFold (based on whether a visitorID generates revenue or not)
\n\nAppreciating any thoughts!
", "rawMarkdown": "Hello everyone:\n\nI'm a bit confused with how people are using GroupKFold in their kernels. Hopefully I'm not having a brain fart but it seems to me that we are not using it the way it was intended.\n\nThis is the code being used in a dozen or so kernels:\n\n\n def get_folds(df=None, n_splits=5):\n \"\"\"Returns dataframe indices corresponding to Visitors Group KFold\"\"\"\n # Get sorted unique visitors\n unique_vis = np.array(sorted(df['fullVisitorId'].unique()))\n\n # Get folds\n folds = GroupKFold(n_splits=n_splits)\n fold_ids = []\n ids = np.arange(df.shape[0])\n for trn_vis, val_vis in folds.split(X=unique_vis, y=unique_vis, groups=unique_vis):\n fold_ids.append(\n [\n ids[df['fullVisitorId'].isin(unique_vis[trn_vis])],\n ids[df['fullVisitorId'].isin(unique_vis[val_vis])]\n ]\n )\n\n return fold_ids\n\n\nI compared that to the doc here: \nhttp://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html\n\nThis is what I understand the code above does:\n\n - create an array of unique visitor IDs\n\n - use GroupKFold to split that array into folds. \n\n - map those visitorId folds back to the original dataFrame\n\nIt appears to me that we are using GroupKFold passing data where all the groups have a unique item (since we pass groups = unique_vis, and unique_vis contains all unique visitorIDs)\n\nTherefore, I'm guessing that our code would get the same result using vanilla KFold. \n\nMy understanding of GroupKFold is that it would normally be used in the following manner:\n\ngroup_kfold.split(train[cols], train['target'], train['fullVisitorId'])\n\n\nI wonder if by using GroupKFold the way we are, we are missing out on some features such as balancing the group sizes between folds.\n\nOr maybe we don't want to use GroupKFold at all; and instead of splitting visitorIDs using vanilla KFold, we could use StratifiedKFold (based on whether a visitorID generates revenue or not)\n\nAppreciating any thoughts!", "votes": null }, { "id": "399819", "postDate": "10/06/2018 21:15:21", "content": "This is all my fault :(
\n\nI think you're absolutely right. I could have used simple KFold in this method.
\n\nI originally used GroupKFold the way you describe but I changed my thoughts to make sure I could use it at session and then at visitor level to avoid leakage...
", "rawMarkdown": "This is all my fault :( \n\nI think you're absolutely right. I could have used simple KFold in this method.\n\nI originally used GroupKFold the way you describe but I changed my thoughts to make sure I could use it at session and then at visitor level to avoid leakage...", "votes": null }, { "id": "399842", "postDate": "10/06/2018 22:36:37", "content": "Hi @olivier,
\n\nI am a newbie so bear with me if I'm asking some dumb question...\nSo the original version (the code shown) did not split the dataset based on VISITORID.\nWhat we should do is using the format GroupKFold(x, y, label) to do GroupKFold?
Or we just use KFold? (But I kind of confuse how KFold can split data based on ID.....)\nThank you guys in advance...Both OP and Olivier....
", "rawMarkdown": "Hi @olivier, \n\nI am a newbie so bear with me if I'm asking some dumb question...\nSo the original version (the code shown) did not split the dataset based on `VISITORID`.\nWhat we should do is using the format GroupKFold(x, y, label) to do GroupKFold?\n\nOr we just use KFold? (But I kind of confuse how KFold can split data based on ID.....)\nThank you guys in advance...Both OP and Olivier....", "votes": null }, { "id": "399856", "postDate": "10/07/2018 00:07:28", "content": "@olivier your code oddities are my learning opportunities! Thanks for sharing your kernels.
\n\n@Justin \nEdit to add: The code above does split the dataset based on visitorID, but it is not balancing the folds the way GroupKFold is supposed to do.
\n\nThe method quoted in the OP first splits an array of unique visitorID's. Since those visitorIDs are unique (in that array), we are dealing with groups of 1. Therefore KFold and GroupKFold are equivalent.
\n\nI personally will use GroupKFold(x, y, label) since it is simpler than the above method and balances the folds better in terms of group sizes.
\n\nAnother option could be to improve on the quoted method and use Stratified K Fold at the visitor ID level.
", "rawMarkdown": "olivier your code oddities are my learning opportunities! Thanks for sharing your kernels.\n\n@Justin \nEdit to add: The code above does split the dataset based on visitorID, but it is not balancing the folds the way GroupKFold is supposed to do.\n\nThe method quoted in the OP first splits an array of unique visitorID's. Since those visitorIDs are unique (in that array), we are dealing with groups of 1. Therefore KFold and GroupKFold are equivalent.\n\nI personally will use GroupKFold(x, y, label) since it is simpler than the above method and balances the folds better in terms of group sizes.\n\nAnother option could be to improve on the quoted method and use Stratified K Fold at the visitor ID level.", "votes": null }, { "id": "399857", "postDate": "10/07/2018 00:12:07", "content": "Thank you for explanation! I understand now~
", "rawMarkdown": "Thank you for explanation! I understand now~", "votes": null }, { "id": "400224", "postDate": "10/07/2018 23:39:53", "content": "Hi @olivier,
\n\nI was also confused about the GroupKFold. But then I thought the reason for using GroupKFold was to avoid leakage; e.g., using an user's future session revenue to predict the previous session revenue would be a leakage. Did I get it right? Thanks.
\n\nL
", "rawMarkdown": "Hi @olivier,\n\nI was also confused about the GroupKFold. But then I thought the reason for using GroupKFold was to avoid leakage; e.g., using an user's future session revenue to predict the previous session revenue would be a leakage. Did I get it right? Thanks.\n\nL", "votes": null }, { "id": "404414", "postDate": "10/15/2018 17:00:55", "content": "@vett93 can you elaborate on \" But then I thought the reason for using GroupKFold was to avoid leakage; e.g., using an user's future session revenue to predict the previous session revenue would be a leakage. \"
\n\nSorry for the dumb question
", "rawMarkdown": "vett93 can you elaborate on \" But then I thought the reason for using GroupKFold was to avoid leakage; e.g., using an user's future session revenue to predict the previous session revenue would be a leakage. \" \n\nSorry for the dumb question", "votes": null }, { "id": "404418", "postDate": "10/15/2018 17:05:40", "content": "Hi Lone Coder, as S D said the way the code works you could use just KFold. It's only the fact I do this on all distinct visortor id that IMHO avoids leakage.
", "rawMarkdown": "Hi Lone Coder, as S D said the way the code works you could use just KFold. It's only the fact I do this on all distinct visortor id that IMHO avoids leakage.", "votes": null }, { "id": "404419", "postDate": "10/15/2018 17:09:02", "content": "Learner2345, to be totally specific what I mean is avoiding leakage between stacking level (or between trained model) not leakage from using future sessions.
", "rawMarkdown": "Learner2345, to be totally specific what I mean is avoiding leakage between stacking level (or between trained model) not leakage from using future sessions.", "votes": null } ], "comments": [ { "id": 399819, "author_name": "ogrellier", "author_url": "", "post_date": "10/06/2018 21:15:21", "content": "This is all my fault :(
\n\nI think you're absolutely right. I could have used simple KFold in this method.
\n\nI originally used GroupKFold the way you describe but I changed my thoughts to make sure I could use it at session and then at visitor level to avoid leakage...
", "votes": null, "replies": [ { "id": 399842, "author_name": "reputation18", "author_url": "", "post_date": "10/06/2018 22:36:37", "content": "Hi @olivier,
\n\nI am a newbie so bear with me if I'm asking some dumb question...\nSo the original version (the code shown) did not split the dataset based on VISITORID.\nWhat we should do is using the format GroupKFold(x, y, label) to do GroupKFold?
Or we just use KFold? (But I kind of confuse how KFold can split data based on ID.....)\nThank you guys in advance...Both OP and Olivier....
", "votes": null, "replies": [] }, { "id": 399856, "author_name": "sdoria", "author_url": "", "post_date": "10/07/2018 00:07:28", "content": "@olivier your code oddities are my learning opportunities! Thanks for sharing your kernels.
\n\n@Justin \nEdit to add: The code above does split the dataset based on visitorID, but it is not balancing the folds the way GroupKFold is supposed to do.
\n\nThe method quoted in the OP first splits an array of unique visitorID's. Since those visitorIDs are unique (in that array), we are dealing with groups of 1. Therefore KFold and GroupKFold are equivalent.
\n\nI personally will use GroupKFold(x, y, label) since it is simpler than the above method and balances the folds better in terms of group sizes.
\n\nAnother option could be to improve on the quoted method and use Stratified K Fold at the visitor ID level.
", "votes": null, "replies": [] }, { "id": 399857, "author_name": "reputation18", "author_url": "", "post_date": "10/07/2018 00:12:07", "content": "Thank you for explanation! I understand now~
", "votes": null, "replies": [] }, { "id": 400224, "author_name": "vett93", "author_url": "", "post_date": "10/07/2018 23:39:53", "content": "Hi @olivier,
\n\nI was also confused about the GroupKFold. But then I thought the reason for using GroupKFold was to avoid leakage; e.g., using an user's future session revenue to predict the previous session revenue would be a leakage. Did I get it right? Thanks.
\n\nL
", "votes": null, "replies": [] }, { "id": 404418, "author_name": "ogrellier", "author_url": "", "post_date": "10/15/2018 17:05:40", "content": "Hi Lone Coder, as S D said the way the code works you could use just KFold. It's only the fact I do this on all distinct visortor id that IMHO avoids leakage.
", "votes": null, "replies": [] } ] }, { "id": 404414, "author_name": "", "author_url": "", "post_date": "10/15/2018 17:00:55", "content": "@vett93 can you elaborate on \" But then I thought the reason for using GroupKFold was to avoid leakage; e.g., using an user's future session revenue to predict the previous session revenue would be a leakage. \"
\n\nSorry for the dumb question
", "votes": null, "replies": [ { "id": 404419, "author_name": "ogrellier", "author_url": "", "post_date": "10/15/2018 17:09:02", "content": "Learner2345, to be totally specific what I mean is avoiding leakage between stacking level (or between trained model) not leakage from using future sessions.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "399736": "Hello everyone:\n\nI'm a bit confused with how people are using GroupKFold in their kernels. Hopefully I'm not having a brain fart but it seems to me that we are not using it the way it was intended.\n\nThis is the code being used in a dozen or so kernels:\n\n\n def get_folds(df=None, n_splits=5):\n \"\"\"Returns dataframe indices corresponding to Visitors Group KFold\"\"\"\n # Get sorted unique visitors\n unique_vis = np.array(sorted(df['fullVisitorId'].unique()))\n\n # Get folds\n folds = GroupKFold(n_splits=n_splits)\n fold_ids = []\n ids = np.arange(df.shape[0])\n for trn_vis, val_vis in folds.split(X=unique_vis, y=unique_vis, groups=unique_vis):\n fold_ids.append(\n [\n ids[df['fullVisitorId'].isin(unique_vis[trn_vis])],\n ids[df['fullVisitorId'].isin(unique_vis[val_vis])]\n ]\n )\n\n return fold_ids\n\n\nI compared that to the doc here: \nhttp://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GroupKFold.html\n\nThis is what I understand the code above does:\n\n - create an array of unique visitor IDs\n\n - use GroupKFold to split that array into folds. \n\n - map those visitorId folds back to the original dataFrame\n\nIt appears to me that we are using GroupKFold passing data where all the groups have a unique item (since we pass groups = unique_vis, and unique_vis contains all unique visitorIDs)\n\nTherefore, I'm guessing that our code would get the same result using vanilla KFold. \n\nMy understanding of GroupKFold is that it would normally be used in the following manner:\n\ngroup_kfold.split(train[cols], train['target'], train['fullVisitorId'])\n\n\nI wonder if by using GroupKFold the way we are, we are missing out on some features such as balancing the group sizes between folds.\n\nOr maybe we don't want to use GroupKFold at all; and instead of splitting visitorIDs using vanilla KFold, we could use StratifiedKFold (based on whether a visitorID generates revenue or not)\n\nAppreciating any thoughts!", "399819": "This is all my fault :( \n\nI think you're absolutely right. I could have used simple KFold in this method.\n\nI originally used GroupKFold the way you describe but I changed my thoughts to make sure I could use it at session and then at visitor level to avoid leakage...", "399842": "Hi @olivier, \n\nI am a newbie so bear with me if I'm asking some dumb question...\nSo the original version (the code shown) did not split the dataset based on `VISITORID`.\nWhat we should do is using the format GroupKFold(x, y, label) to do GroupKFold?\n\nOr we just use KFold? (But I kind of confuse how KFold can split data based on ID.....)\nThank you guys in advance...Both OP and Olivier....", "399856": "olivier your code oddities are my learning opportunities! Thanks for sharing your kernels.\n\n@Justin \nEdit to add: The code above does split the dataset based on visitorID, but it is not balancing the folds the way GroupKFold is supposed to do.\n\nThe method quoted in the OP first splits an array of unique visitorID's. Since those visitorIDs are unique (in that array), we are dealing with groups of 1. Therefore KFold and GroupKFold are equivalent.\n\nI personally will use GroupKFold(x, y, label) since it is simpler than the above method and balances the folds better in terms of group sizes.\n\nAnother option could be to improve on the quoted method and use Stratified K Fold at the visitor ID level.", "399857": "Thank you for explanation! I understand now~", "400224": "Hi @olivier,\n\nI was also confused about the GroupKFold. But then I thought the reason for using GroupKFold was to avoid leakage; e.g., using an user's future session revenue to predict the previous session revenue would be a leakage. Did I get it right? Thanks.\n\nL", "404414": "vett93 can you elaborate on \" But then I thought the reason for using GroupKFold was to avoid leakage; e.g., using an user's future session revenue to predict the previous session revenue would be a leakage. \" \n\nSorry for the dumb question", "404418": "Hi Lone Coder, as S D said the way the code works you could use just KFold. It's only the fact I do this on all distinct visortor id that IMHO avoids leakage.", "404419": "Learner2345, to be totally specific what I mean is avoiding leakage between stacking level (or between trained model) not leakage from using future sessions." }, "source": "meta" }