While working on my EDA, I found out that 898 sessionId's in the train set are duplicate. I think the sessionId should be the unique identifier, as it is a combination of fullVisitorId and visitStartTime. The only explanation that I can think of is that those users have started a new session within a second. Has anyone looked into this yet (should we remove duplicates?)?
", "messages": [ { "id": "389158", "postDate": "09/18/2018 09:18:09", "content": "While working on my EDA, I found out that 898 sessionId's in the train set are duplicate. I think the sessionId should be the unique identifier, as it is a combination of fullVisitorId and visitStartTime. The only explanation that I can think of is that those users have started a new session within a second. Has anyone looked into this yet (should we remove duplicates?)?
", "rawMarkdown": "While working on my EDA, I found out that 898 sessionId's in the train set are duplicate. I think the sessionId should be the unique identifier, as it is a combination of fullVisitorId and visitStartTime. The only explanation that I can think of is that those users have started a new session within a second. Has anyone looked into this yet (should we remove duplicates?)?", "votes": null }, { "id": "389239", "postDate": "09/18/2018 12:06:40", "content": "I also found the same thing, but I don't think they created a new session within an instant. We can see this from the visitStartTime(they seem to different for the rows with same sessionId).\nLet me know your thoughts.
", "rawMarkdown": "I also found the same thing, but I don't think they created a new session within an instant. We can see this from the visitStartTime(they seem to different for the rows with same sessionId).\nLet me know your thoughts.", "votes": null }, { "id": "389249", "postDate": "09/18/2018 12:19:42", "content": "visitStartTime really is the same as visitId (but I converted visitStartTime into POSIXct, as I think that time between sessions could be a good feature). Have not checked yet if those obs with identical sessionId's are exactly the same with regards to all features.....
", "rawMarkdown": "visitStartTime really is the same as visitId (but I converted visitStartTime into POSIXct, as I think that time between sessions could be a good feature). Have not checked yet if those obs with identical sessionId's are exactly the same with regards to all features.....", "votes": null }, { "id": "389362", "postDate": "09/18/2018 15:37:35", "content": "I think these duplicates are one session that spans midnight in the respective timezone. I remember having read somewhere that such sessions are split into two, but I can't find the reference at the moment.
\n\nEdit: I found that reference
", "rawMarkdown": "I think these duplicates are one session that spans midnight in the respective timezone. I remember having read somewhere that such sessions are split into two, but I can't find the reference at the moment.\n\nEdit: I found that [reference][1]\n\n\n [1]: https://www.kaggle.com/c/google-analytics-customer-revenue-prediction/discussion/65945#388721", "votes": null }, { "id": "389376", "postDate": "09/18/2018 16:01:11", "content": "Nice find! Please keep us posted (at the moment I'm too lazy to figure out ;-))!
", "rawMarkdown": "Nice find! Please keep us posted (at the moment I'm too lazy to figure out ;-))!", "votes": null }, { "id": "389603", "postDate": "09/19/2018 01:18:34", "content": "We have sessionId = fullVisitorId + '_' + visitId. I think some visitId are broken, and it causes the duplication. I wrote a kernel and found that 895 of 898 duplication are caused by setting the visitStartTime of the previous visit of the user. Could you let me know your thoughts?
That sounds very feasible! Thanks, I will definitely look at your kernel!
", "rawMarkdown": "That sounds very feasible! Thanks, I will definitely look at your kernel!", "votes": null } ], "comments": [ { "id": 389239, "author_name": "sahil711", "author_url": "", "post_date": "09/18/2018 12:06:40", "content": "I also found the same thing, but I don't think they created a new session within an instant. We can see this from the visitStartTime(they seem to different for the rows with same sessionId).\nLet me know your thoughts.
", "votes": null, "replies": [ { "id": 389249, "author_name": "erikbruin", "author_url": "", "post_date": "09/18/2018 12:19:42", "content": "visitStartTime really is the same as visitId (but I converted visitStartTime into POSIXct, as I think that time between sessions could be a good feature). Have not checked yet if those obs with identical sessionId's are exactly the same with regards to all features.....
", "votes": null, "replies": [] } ] }, { "id": 389362, "author_name": "tobycheese", "author_url": "", "post_date": "09/18/2018 15:37:35", "content": "I think these duplicates are one session that spans midnight in the respective timezone. I remember having read somewhere that such sessions are split into two, but I can't find the reference at the moment.
\n\nEdit: I found that reference
", "votes": null, "replies": [ { "id": 389376, "author_name": "erikbruin", "author_url": "", "post_date": "09/18/2018 16:01:11", "content": "Nice find! Please keep us posted (at the moment I'm too lazy to figure out ;-))!
", "votes": null, "replies": [] } ] }, { "id": 389603, "author_name": "zaburo", "author_url": "", "post_date": "09/19/2018 01:18:34", "content": "We have sessionId = fullVisitorId + '_' + visitId. I think some visitId are broken, and it causes the duplication. I wrote a kernel and found that 895 of 898 duplication are caused by setting the visitStartTime of the previous visit of the user. Could you let me know your thoughts?
That sounds very feasible! Thanks, I will definitely look at your kernel!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "389158": "While working on my EDA, I found out that 898 sessionId's in the train set are duplicate. I think the sessionId should be the unique identifier, as it is a combination of fullVisitorId and visitStartTime. The only explanation that I can think of is that those users have started a new session within a second. Has anyone looked into this yet (should we remove duplicates?)?", "389239": "I also found the same thing, but I don't think they created a new session within an instant. We can see this from the visitStartTime(they seem to different for the rows with same sessionId).\nLet me know your thoughts.", "389249": "visitStartTime really is the same as visitId (but I converted visitStartTime into POSIXct, as I think that time between sessions could be a good feature). Have not checked yet if those obs with identical sessionId's are exactly the same with regards to all features.....", "389362": "I think these duplicates are one session that spans midnight in the respective timezone. I remember having read somewhere that such sessions are split into two, but I can't find the reference at the moment.\n\nEdit: I found that [reference][1]\n\n\n [1]: https://www.kaggle.com/c/google-analytics-customer-revenue-prediction/discussion/65945#388721", "389376": "Nice find! Please keep us posted (at the moment I'm too lazy to figure out ;-))!", "389603": "We have `sessionId` = `fullVisitorId` + '_' + `visitId`. I think some `visitId` are broken, and it causes the duplication. I wrote [a kernel](https://www.kaggle.com/zaburo/visitid-seems-to-be-broken) and found that 895 of 898 duplication are caused by setting the `visitStartTime` of the previous visit of the user. Could you let me know your thoughts?", "389737": "That sounds very feasible! Thanks, I will definitely look at your kernel!" }, "source": "meta" }