![\"enter](\"https://i.imgur.com/uJaqJl2.png\")
Some of you may have noticed that for some records the date extracted from visitStartTime do not match the field date. I will shed some light on this issue without being technical.
As per BigQuery Export schema, visitStartTime is 'The timestamp (expressed as POSIX time)' and as such it is expressed in UTC.
On the other hand date (or ga:date to be more specific) comes from Google Analytics and it is expressed on the timezone that the GA administrator has configured in the specific view's settings.
\nIn layman terms, it is a property of the website, not the client/web browser and it should be consistent in all rows if they came from the same view irrespectively of the timezone of the visitor.
That being said, and knowing that the data come from Google Merchandise Store which resides in California (which is UTC -7), every session started between 00:00 and 07:00 it is accounted in the previous day \"date wise\" and to the next day \"visitStartTime wise\".
\n\nI hope the above is clear, otherwise please let me know.
\n\nCheers
", "messages": [ { "id": "393675", "postDate": "09/25/2018 18:35:58", "content": "Some of you may have noticed that for some records the date extracted from visitStartTime do not match the field date. I will shed some light on this issue without being technical.
As per BigQuery Export schema, visitStartTime is 'The timestamp (expressed as POSIX time)' and as such it is expressed in UTC.
On the other hand date (or ga:date to be more specific) comes from Google Analytics and it is expressed on the timezone that the GA administrator has configured in the specific view's settings.
\nIn layman terms, it is a property of the website, not the client/web browser and it should be consistent in all rows if they came from the same view irrespectively of the timezone of the visitor.
That being said, and knowing that the data come from Google Merchandise Store which resides in California (which is UTC -7), every session started between 00:00 and 07:00 it is accounted in the previous day \"date wise\" and to the next day \"visitStartTime wise\".
\n\nI hope the above is clear, otherwise please let me know.
\n\nCheers
", "rawMarkdown": "Some of you may have noticed that for some records the date extracted from `visitStartTime` do not match the field `date`. I will shed some light on this issue without being technical.\n\nAs per [BigQuery Export schema](https://support.google.com/analytics/answer/3437719?hl=en), `visitStartTime` is 'The timestamp (expressed as POSIX time)' and as such it is **expressed in UTC**. \n\nOn the other hand `date` (or [ga:date](https://developers.google.com/analytics/devguides/reporting/core/dimsmets#view=detail&group=time&jump=ga_date) to be more specific) comes from Google Analytics and it is expressed on the timezone **that the GA administrator has configured in the specific [view's](https://support.google.com/analytics/answer/2649553?hl=en) settings**. \nIn layman terms, **it is a property of the website, not the client/web browser** and it should be consistent in all rows if they came from the same view irrespectively of the timezone of the visitor.\n\nThat being said, and knowing that the data come from Google Merchandise Store which resides in California (which is UTC -7), **every session started between 00:00 and 07:00 it is accounted in the previous day \"date wise\" and to the next day \"visitStartTime wise\"**.\n\nI hope the above is clear, otherwise please let me know.\n\nCheers", "votes": null }, { "id": "393681", "postDate": "09/25/2018 18:44:12", "content": "Good to know! Thank you!
", "rawMarkdown": "Good to know! Thank you!", "votes": null }, { "id": "393687", "postDate": "09/25/2018 18:58:58", "content": "Nice catch! Thanks for the tip!
", "rawMarkdown": "Nice catch! Thanks for the tip!", "votes": null }, { "id": "393693", "postDate": "09/25/2018 19:05:32", "content": "Thanks Yannis for the insight. Could you please elaborate how to explain the shift in the minimum of visitStartTime between countries in different timezones? For example, see the plots attached in https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65691#387658
I've seen a similar thing. If you look at duplicate sessionID's, they stem from that difference in timezone. As you said, those timezones are UTC and GA settings (i.e. California). There will be two rows for one sessionID when the visit starts right before midnight and ends after midnight (I forget in which of the timezones)
\n\nI have to add one thing to your post. Sometimes the difference is 8 hours, not 7 hours. Probably a daily savings issue.
", "rawMarkdown": "I've seen a similar thing. If you look at duplicate sessionID's, they stem from that difference in timezone. As you said, those timezones are UTC and GA settings (i.e. California). There will be two rows for one sessionID when the visit starts right before midnight and ends after midnight (I forget in which of the timezones)\n\nI have to add one thing to your post. Sometimes the difference is 8 hours, not 7 hours. Probably a daily savings issue.", "votes": null }, { "id": "393797", "postDate": "09/25/2018 22:38:18", "content": "Very interesting. Have you done any exploration with respect to daylight savings time and how that might affect the UTC to Pacific Standard/Daylight Time conversion?
", "rawMarkdown": "Very interesting. Have you done any exploration with respect to daylight savings time and how that might affect the UTC to Pacific Standard/Daylight Time conversion?", "votes": null }, { "id": "393809", "postDate": "09/25/2018 23:01:19", "content": "I haven't dug into it deeply, but the few cases I inspected indicated that DST does affect the conversion.
", "rawMarkdown": "I haven't dug into it deeply, but the few cases I inspected indicated that DST does affect the conversion.", "votes": null }, { "id": "394069", "postDate": "09/26/2018 10:18:57", "content": "If my code is right the max difference is 6:43:47.\n
Could you point me to a row where this difference is greater?
", "rawMarkdown": "If my code is right the max difference is 6:43:47.\n![enter image description here][1]\n\nCould you point me to a row where this difference is greater?\n\n [1]: https://i.imgur.com/uJaqJl2.png", "votes": null }, { "id": "394082", "postDate": "09/26/2018 10:40:53", "content": "It looks reasonable to me.
\nKeep in mind that the time in both plots is in UTC. Thus, it makes perfect sense that the lowest traffic between two time zones with X hours difference is also shifted by X hours.
As an example, if at 5:00 AM (local time) we have the lowest traffic:
\n\nNo, I haven't but I don't think it will make any difference.
\n\nIn general, I don't know if the date field has any importance. I was thinking about using the date from visitStartTime as a more consistent variable.
A good idea would be to use visitStartTime and geo-information (where is present) to create an extra feature representing the localTime of the visit. My intuition says that this may be important for the visitors' behavior (maybe they buy more during the morning or night etc.)
Oh, I just noticed that with the code above I just got a sample.
\n\nYes, 8 hours is possible based on if a daylight saving time is in effect or not.
\n\nhttps://www.timeanddate.com/time/map/
", "rawMarkdown": "Oh, I just noticed that with the code above I just got a sample.\n\nYes, 8 hours is possible based on if a daylight saving time is in effect or not.\n\nhttps://www.timeanddate.com/time/map/", "votes": null }, { "id": "394121", "postDate": "09/26/2018 11:36:08", "content": "So if visitStartTime is in UTC, it may indeed be prudent to account for daylight savings when creating a localTime feature?
Sure, because UTC is not affected by daylight changes in contrast with the local times.
\nThis should not be a problem though if you use something similar to pandas.Series.tz_localize to convert them.
Good point. Not the first time I have foolishly considered handling dates myself... They're always deceptively complicated and there are always well-vetted packages out there for handling them
", "rawMarkdown": "Good point. Not the first time I have foolishly considered handling dates myself... They're always deceptively complicated and there are always well-vetted packages out there for handling them", "votes": null }, { "id": "394158", "postDate": "09/26/2018 12:40:10", "content": "Thanks for sharing, just curious how you find out this? Is it stated somewhere? Can you share the source with us?
", "rawMarkdown": "Thanks for sharing, just curious how you find out this? Is it stated somewhere? Can you share the source with us?", "votes": null }, { "id": "394171", "postDate": "09/26/2018 13:02:00", "content": "I picked a couple example from ones that have duplicate session IDs (which occurs when visits span around midnight).
\nThere are two rows for each of those, so make sure you look at both.\nsessionID='0098453420096961116_1498458618' => 20170625/20170626 => 2017-06-26 07:00:16
'738204402836923511_1487404770' => 20170217/ 20170218 => 2017-02-18 08:00:10
\n\nNote how the first one is in the summer, while the second one is in the winter.
", "rawMarkdown": "I picked a couple example from ones that have duplicate session IDs (which occurs when visits span around midnight). \nThere are two rows for each of those, so make sure you look at both.\nsessionID='0098453420096961116_1498458618' => 20170625/20170626 => 2017-06-26 07:00:16\n\n'738204402836923511_1487404770' => 20170217/ 20170218 => 2017-02-18 08:00:10\n\nNote how the first one is in the summer, while the second one is in the winter.", "votes": null }, { "id": "394178", "postDate": "09/26/2018 13:12:31", "content": "Yannis, I believe your intuition will be proved to be correct. You don't even need to convert time to do so. E.g. look at hourly California data
", "rawMarkdown": "Yannis, I believe your intuition will be proved to be correct. You don't even need to convert time to do so. E.g. look at hourly California data", "votes": null }, { "id": "394197", "postDate": "09/26/2018 13:35:08", "content": "I had been working for some time as an analyst with Google Analytics data so when I noticed this discrepancy I figured out what was the issue.
\nThen I saw that this troubles other members also and I shared it.
I believe that visitStartTime is in UTC is stated in the projects' overview and also in BigQuery Export schema (check the link in my post).
\nThe fact that the date is in the time zone that the administrator had set, was by experience but you can find it also in GA documentation
Yes, you are right. It is obvious. I was confused.
", "rawMarkdown": "Yes, you are right. It is obvious. I was confused.", "votes": null }, { "id": "395093", "postDate": "09/28/2018 00:37:23", "content": "A few kernels use this visitStartTime as a numerical variables and it get high importance. Since train and test data are not on the same period, it shouldn't matter right ?
\n\nI tried removing it but lower both the CV and the LB score. I also tried replacing this variable by a float representing the moment during the day, but same thing the score was not as good... Any thought on this ?
", "rawMarkdown": "A few kernels use this visitStartTime as a numerical variables and it get high importance. Since train and test data are not on the same period, it shouldn't matter right ? \n\nI tried removing it but lower both the CV and the LB score. I also tried replacing this variable by a float representing the moment during the day, but same thing the score was not as good... Any thought on this ?", "votes": null }, { "id": "395105", "postDate": "09/28/2018 01:26:30", "content": "some users appear in both train and test
", "rawMarkdown": "some users appear in both train and test", "votes": null }, { "id": "395300", "postDate": "09/28/2018 09:12:13", "content": "@areveillon It may matter because Unix time continuous variable increasing with the time. So, if the revenue also grows with the time, there might be a correlation, but you get this information from the date also.
I believe the power of visitStartTime comes when you combine it with the timezone to calculate the local time of the user.
Let consider situation when some tree model is used and there is a split at the end of train data like before/after 16 Jun 2017. This rule will be used for all test, so using visitStartTime makes sense.\nAs for why there is a difference between visitStartTime and its float counter part, they should behave the same way, I may assume there is an error when converting to float. \nBtw, why to convert visitStartTime to float?
\n\nFeatures like month, day of month, day of year, etc are more appropriate as they have good overlapping between train and test.
", "rawMarkdown": "Let consider situation when some tree model is used and there is a split at the end of train data like before/after 16 Jun 2017. This rule will be used for all test, so using visitStartTime makes sense.\nAs for why there is a difference between visitStartTime and its float counter part, they should behave the same way, I may assume there is an error when converting to float. \nBtw, why to convert visitStartTime to float?\n\nFeatures like month, day of month, day of year, etc are more appropriate as they have good overlapping between train and test.", "votes": null }, { "id": "395334", "postDate": "09/28/2018 11:01:52", "content": "If I get it right from
\n\n\n\n\nalso tried replacing this variable by a float representing the moment during the day
\n
what Antoine does is a nice trick cause he gets a \"time of the day\" feature.
\nBut Antoine, keep in mind that this will be the time of the purchase in another timezone (UTC) if you don't adjust it to the local time zone.
I think a tree based model can handle easily this situation building its branch separation conditions with both UTC time and one location feature, if it considers that it optimizes the model, isn't it?
", "rawMarkdown": "I think a tree based model can handle easily this situation building its branch separation conditions with both UTC time and one location feature, if it considers that it optimizes the model, isn't it?", "votes": null }, { "id": "395342", "postDate": "09/28/2018 11:31:25", "content": "I assume in theory you are right but the inference (from the model) of the local time from UTC and location features is a small regression problem by itself. Probably you may lose some accuracy \"inferring\" it instead of \"calculating\" it (with an external datasource, API, etc.).
\n\nAm I thinking it right?
", "rawMarkdown": "I assume in theory you are right but the inference (from the model) of the local time from UTC and location features is a small regression problem by itself. Probably you may lose some accuracy \"inferring\" it instead of \"calculating\" it (with an external datasource, API, etc.). \n\nAm I thinking it right?", "votes": null }, { "id": "395351", "postDate": "09/28/2018 12:14:41", "content": "The main problem with accuracy here is that 56% of records do not have city/metro/region, so we have to work with country level which sometimes means 6-8 hours spread.
", "rawMarkdown": "The main problem with accuracy here is that 56% of records do not have city/metro/region, so we have to work with country level which sometimes means 6-8 hours spread.", "votes": null }, { "id": "395365", "postDate": "09/28/2018 12:48:02", "content": "I agree with you @emorej, the model should theoretically be able to do it, but you don't really know what split lightgbm will do... Especially if you have hundred of features. It would be better to have a clear feature with local time, but it might not be very precise with all missing data.
\n\nSo I tried to remove the visitStartTime, and add day / month / year + a float representing the moment of the day. Since there is no overlapping between Train and Test I would expect the visitStartTime to overfit the training set. Maybe there is a side effect as you were saying @alexfir but I'm not aware of such situation.
\n\nAnyway by adding those variable I don't get the same CV LB that I get with visitStartTime ? @alexfir is your model based on visitStartTime ? or were you able to completely remove it ?
\n\ntrain min 1470034812\ntrain max 1501657193
\n\ntest min 1501657203\ntest max 1525157818
", "rawMarkdown": "I agree with you @emorej, the model should theoretically be able to do it, but you don't really know what split lightgbm will do... Especially if you have hundred of features. It would be better to have a clear feature with local time, but it might not be very precise with all missing data.\n\nSo I tried to remove the visitStartTime, and add day / month / year + a float representing the moment of the day. Since there is no overlapping between Train and Test I would expect the visitStartTime to overfit the training set. Maybe there is a side effect as you were saying @alexfir but I'm not aware of such situation.\n\nAnyway by adding those variable I don't get the same CV LB that I get with visitStartTime ? @alexfir is your model based on visitStartTime ? or were you able to completely remove it ?\n\ntrain min 1470034812\ntrain max 1501657193\n\ntest min 1501657203\ntest max 1525157818", "votes": null }, { "id": "395391", "postDate": "09/28/2018 13:43:09", "content": "@areveillon, currently I am using plain visitStartTime as a feature, but I did not select features on this stage. \"visitStartTime\" is on 13rd place by importance, while plain \"date\" is on 7th while representing the same data.
\n\nYou probably may get more from \"visitStartTime\" if converting to some kind of relative feature rather than absolute one. For example, how long time ago the user visited GStore for the first time. In this case model can treat users that visited for the first time 20 days ago in the same way in both train and test.
", "rawMarkdown": "areveillon, currently I am using plain visitStartTime as a feature, but I did not select features on this stage. \"visitStartTime\" is on 13rd place by importance, while plain \"date\" is on 7th while representing the same data. \n\nYou probably may get more from \"visitStartTime\" if converting to some kind of relative feature rather than absolute one. For example, how long time ago the user visited GStore for the first time. In this case model can treat users that visited for the first time 20 days ago in the same way in both train and test.", "votes": null }, { "id": "395426", "postDate": "09/28/2018 14:55:21", "content": "I agree with you, relative feature for time should be more helpful... I did include some features like this, and they were not as good as the visitStartTime or date...
\n\nI was using time split CV btw, and even locally the validation set is better when I include those time feature... I was just curious to understand why since the interval don't overlap.
", "rawMarkdown": "I agree with you, relative feature for time should be more helpful... I did include some features like this, and they were not as good as the visitStartTime or date...\n\nI was using time split CV btw, and even locally the validation set is better when I include those time feature... I was just curious to understand why since the interval don't overlap.", "votes": null }, { "id": "395728", "postDate": "09/29/2018 08:21:11", "content": "@emorej The trick is that such optimisation would require to look at 2 variables simultaneously. But GBMs at each split considers only one variable.
", "rawMarkdown": "emorej The trick is that such optimisation would require to look at 2 variables simultaneously. But GBMs at each split considers only one variable.", "votes": null }, { "id": "395855", "postDate": "09/29/2018 13:41:29", "content": "@Misha yes you're right so even if this optimization could be found in depth tree, it's sensibly easier to calc directly a local time feature I admit it :)
", "rawMarkdown": "Misha yes you're right so even if this optimization could be found in depth tree, it's sensibly easier to calc directly a local time feature I admit it :)", "votes": null }, { "id": "397205", "postDate": "10/02/2018 05:04:44", "content": "I have a question about converting visitStartTime to local time.\nHow can we find the timezone of the users?
I have read this topic, which points out the discrepancy between geoNetwork_country and geoNetwork_city (for example, when geoNetwork_city is Toronto, some of geoNetwork_country are not Canada).
So, I wonder how we find the right place of the users.
", "rawMarkdown": "I have a question about converting ```visitStartTime``` to local time.\nHow can we find the timezone of the users?\n\nI have read [this topic](https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66627), which points out the discrepancy between ```geoNetwork_country``` and ```geoNetwork_city``` (for example, when ```geoNetwork_city``` is Toronto, some of ```geoNetwork_country``` are not Canada).\n\nSo, I wonder how we find the right place of the users.", "votes": null }, { "id": "413871", "postDate": "11/01/2018 17:17:26", "content": "Having said that, would you recommend using only one of those features? Since they are just a matter of +7 difference? And if so, we can drop date and be sure we also don't have the timezone effect. So keeping VisitStartTime keeps everything standardised and in one timezone (UTC - 7)?
", "rawMarkdown": "Having said that, would you recommend using only one of those features? Since they are just a matter of +7 difference? And if so, we can drop date and be sure we also don't have the timezone effect. So keeping VisitStartTime keeps everything standardised and in one timezone (UTC - 7)?", "votes": null }, { "id": "414328", "postDate": "11/02/2018 14:59:14", "content": "But we dont have a timezone field. Should we get it from the name of the country maybe
", "rawMarkdown": "But we dont have a timezone field. Should we get it from the name of the country maybe", "votes": null } ], "comments": [ { "id": 393681, "author_name": "leonidg", "author_url": "", "post_date": "09/25/2018 18:44:12", "content": "Good to know! Thank you!
", "votes": null, "replies": [] }, { "id": 393687, "author_name": "divee9", "author_url": "", "post_date": "09/25/2018 18:58:58", "content": "Nice catch! Thanks for the tip!
", "votes": null, "replies": [] }, { "id": 393693, "author_name": "mlisovyi", "author_url": "", "post_date": "09/25/2018 19:05:32", "content": "Thanks Yannis for the insight. Could you please elaborate how to explain the shift in the minimum of visitStartTime between countries in different timezones? For example, see the plots attached in https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65691#387658
It looks reasonable to me.
\nKeep in mind that the time in both plots is in UTC. Thus, it makes perfect sense that the lowest traffic between two time zones with X hours difference is also shifted by X hours.
As an example, if at 5:00 AM (local time) we have the lowest traffic:
\n\nYes, you are right. It is obvious. I was confused.
", "votes": null, "replies": [] } ] }, { "id": 393701, "author_name": "sdoria", "author_url": "", "post_date": "09/25/2018 19:14:20", "content": "I've seen a similar thing. If you look at duplicate sessionID's, they stem from that difference in timezone. As you said, those timezones are UTC and GA settings (i.e. California). There will be two rows for one sessionID when the visit starts right before midnight and ends after midnight (I forget in which of the timezones)
\n\nI have to add one thing to your post. Sometimes the difference is 8 hours, not 7 hours. Probably a daily savings issue.
", "votes": null, "replies": [ { "id": 394069, "author_name": "yannisp", "author_url": "", "post_date": "09/26/2018 10:18:57", "content": "If my code is right the max difference is 6:43:47.\n
Could you point me to a row where this difference is greater?
", "votes": null, "replies": [] }, { "id": 394097, "author_name": "yannisp", "author_url": "", "post_date": "09/26/2018 11:02:04", "content": "Oh, I just noticed that with the code above I just got a sample.
\n\nYes, 8 hours is possible based on if a daylight saving time is in effect or not.
\n\nhttps://www.timeanddate.com/time/map/
", "votes": null, "replies": [] }, { "id": 394171, "author_name": "sdoria", "author_url": "", "post_date": "09/26/2018 13:02:00", "content": "I picked a couple example from ones that have duplicate session IDs (which occurs when visits span around midnight).
\nThere are two rows for each of those, so make sure you look at both.\nsessionID='0098453420096961116_1498458618' => 20170625/20170626 => 2017-06-26 07:00:16
'738204402836923511_1487404770' => 20170217/ 20170218 => 2017-02-18 08:00:10
\n\nNote how the first one is in the summer, while the second one is in the winter.
", "votes": null, "replies": [] } ] }, { "id": 393797, "author_name": "nnnnick", "author_url": "", "post_date": "09/25/2018 22:38:18", "content": "Very interesting. Have you done any exploration with respect to daylight savings time and how that might affect the UTC to Pacific Standard/Daylight Time conversion?
", "votes": null, "replies": [ { "id": 393809, "author_name": "sdoria", "author_url": "", "post_date": "09/25/2018 23:01:19", "content": "I haven't dug into it deeply, but the few cases I inspected indicated that DST does affect the conversion.
", "votes": null, "replies": [] }, { "id": 394094, "author_name": "yannisp", "author_url": "", "post_date": "09/26/2018 10:55:24", "content": "No, I haven't but I don't think it will make any difference.
\n\nIn general, I don't know if the date field has any importance. I was thinking about using the date from visitStartTime as a more consistent variable.
A good idea would be to use visitStartTime and geo-information (where is present) to create an extra feature representing the localTime of the visit. My intuition says that this may be important for the visitors' behavior (maybe they buy more during the morning or night etc.)
So if visitStartTime is in UTC, it may indeed be prudent to account for daylight savings when creating a localTime feature?
Sure, because UTC is not affected by daylight changes in contrast with the local times.
\nThis should not be a problem though if you use something similar to pandas.Series.tz_localize to convert them.
Good point. Not the first time I have foolishly considered handling dates myself... They're always deceptively complicated and there are always well-vetted packages out there for handling them
", "votes": null, "replies": [] }, { "id": 394178, "author_name": "sdoria", "author_url": "", "post_date": "09/26/2018 13:12:31", "content": "Yannis, I believe your intuition will be proved to be correct. You don't even need to convert time to do so. E.g. look at hourly California data
", "votes": null, "replies": [] } ] }, { "id": 394158, "author_name": "joe1995", "author_url": "", "post_date": "09/26/2018 12:40:10", "content": "Thanks for sharing, just curious how you find out this? Is it stated somewhere? Can you share the source with us?
", "votes": null, "replies": [ { "id": 394197, "author_name": "yannisp", "author_url": "", "post_date": "09/26/2018 13:35:08", "content": "I had been working for some time as an analyst with Google Analytics data so when I noticed this discrepancy I figured out what was the issue.
\nThen I saw that this troubles other members also and I shared it.
I believe that visitStartTime is in UTC is stated in the projects' overview and also in BigQuery Export schema (check the link in my post).
\nThe fact that the date is in the time zone that the administrator had set, was by experience but you can find it also in GA documentation
A few kernels use this visitStartTime as a numerical variables and it get high importance. Since train and test data are not on the same period, it shouldn't matter right ?
\n\nI tried removing it but lower both the CV and the LB score. I also tried replacing this variable by a float representing the moment during the day, but same thing the score was not as good... Any thought on this ?
", "votes": null, "replies": [ { "id": 395105, "author_name": "echowuhao", "author_url": "", "post_date": "09/28/2018 01:26:30", "content": "some users appear in both train and test
", "votes": null, "replies": [] }, { "id": 395300, "author_name": "yannisp", "author_url": "", "post_date": "09/28/2018 09:12:13", "content": "@areveillon It may matter because Unix time continuous variable increasing with the time. So, if the revenue also grows with the time, there might be a correlation, but you get this information from the date also.
I believe the power of visitStartTime comes when you combine it with the timezone to calculate the local time of the user.
Let consider situation when some tree model is used and there is a split at the end of train data like before/after 16 Jun 2017. This rule will be used for all test, so using visitStartTime makes sense.\nAs for why there is a difference between visitStartTime and its float counter part, they should behave the same way, I may assume there is an error when converting to float. \nBtw, why to convert visitStartTime to float?
\n\nFeatures like month, day of month, day of year, etc are more appropriate as they have good overlapping between train and test.
", "votes": null, "replies": [] }, { "id": 395334, "author_name": "yannisp", "author_url": "", "post_date": "09/28/2018 11:01:52", "content": "If I get it right from
\n\n\n\n\nalso tried replacing this variable by a float representing the moment during the day
\n
what Antoine does is a nice trick cause he gets a \"time of the day\" feature.
\nBut Antoine, keep in mind that this will be the time of the purchase in another timezone (UTC) if you don't adjust it to the local time zone.
I think a tree based model can handle easily this situation building its branch separation conditions with both UTC time and one location feature, if it considers that it optimizes the model, isn't it?
", "votes": null, "replies": [] }, { "id": 395342, "author_name": "yannisp", "author_url": "", "post_date": "09/28/2018 11:31:25", "content": "I assume in theory you are right but the inference (from the model) of the local time from UTC and location features is a small regression problem by itself. Probably you may lose some accuracy \"inferring\" it instead of \"calculating\" it (with an external datasource, API, etc.).
\n\nAm I thinking it right?
", "votes": null, "replies": [] }, { "id": 395351, "author_name": "alexfir", "author_url": "", "post_date": "09/28/2018 12:14:41", "content": "The main problem with accuracy here is that 56% of records do not have city/metro/region, so we have to work with country level which sometimes means 6-8 hours spread.
", "votes": null, "replies": [] }, { "id": 395365, "author_name": "areveillon", "author_url": "", "post_date": "09/28/2018 12:48:02", "content": "I agree with you @emorej, the model should theoretically be able to do it, but you don't really know what split lightgbm will do... Especially if you have hundred of features. It would be better to have a clear feature with local time, but it might not be very precise with all missing data.
\n\nSo I tried to remove the visitStartTime, and add day / month / year + a float representing the moment of the day. Since there is no overlapping between Train and Test I would expect the visitStartTime to overfit the training set. Maybe there is a side effect as you were saying @alexfir but I'm not aware of such situation.
\n\nAnyway by adding those variable I don't get the same CV LB that I get with visitStartTime ? @alexfir is your model based on visitStartTime ? or were you able to completely remove it ?
\n\ntrain min 1470034812\ntrain max 1501657193
\n\ntest min 1501657203\ntest max 1525157818
", "votes": null, "replies": [] }, { "id": 395391, "author_name": "alexfir", "author_url": "", "post_date": "09/28/2018 13:43:09", "content": "@areveillon, currently I am using plain visitStartTime as a feature, but I did not select features on this stage. \"visitStartTime\" is on 13rd place by importance, while plain \"date\" is on 7th while representing the same data.
\n\nYou probably may get more from \"visitStartTime\" if converting to some kind of relative feature rather than absolute one. For example, how long time ago the user visited GStore for the first time. In this case model can treat users that visited for the first time 20 days ago in the same way in both train and test.
", "votes": null, "replies": [] }, { "id": 395426, "author_name": "areveillon", "author_url": "", "post_date": "09/28/2018 14:55:21", "content": "I agree with you, relative feature for time should be more helpful... I did include some features like this, and they were not as good as the visitStartTime or date...
\n\nI was using time split CV btw, and even locally the validation set is better when I include those time feature... I was just curious to understand why since the interval don't overlap.
", "votes": null, "replies": [] }, { "id": 395728, "author_name": "mlisovyi", "author_url": "", "post_date": "09/29/2018 08:21:11", "content": "@emorej The trick is that such optimisation would require to look at 2 variables simultaneously. But GBMs at each split considers only one variable.
", "votes": null, "replies": [] }, { "id": 395855, "author_name": "emorej", "author_url": "", "post_date": "09/29/2018 13:41:29", "content": "@Misha yes you're right so even if this optimization could be found in depth tree, it's sensibly easier to calc directly a local time feature I admit it :)
", "votes": null, "replies": [] }, { "id": 414328, "author_name": "ramonafli", "author_url": "", "post_date": "11/02/2018 14:59:14", "content": "But we dont have a timezone field. Should we get it from the name of the country maybe
", "votes": null, "replies": [] } ] }, { "id": 397205, "author_name": "takaishikawa", "author_url": "", "post_date": "10/02/2018 05:04:44", "content": "I have a question about converting visitStartTime to local time.\nHow can we find the timezone of the users?
I have read this topic, which points out the discrepancy between geoNetwork_country and geoNetwork_city (for example, when geoNetwork_city is Toronto, some of geoNetwork_country are not Canada).
So, I wonder how we find the right place of the users.
", "votes": null, "replies": [] }, { "id": 413871, "author_name": "ramonafli", "author_url": "", "post_date": "11/01/2018 17:17:26", "content": "Having said that, would you recommend using only one of those features? Since they are just a matter of +7 difference? And if so, we can drop date and be sure we also don't have the timezone effect. So keeping VisitStartTime keeps everything standardised and in one timezone (UTC - 7)?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "393675": "Some of you may have noticed that for some records the date extracted from `visitStartTime` do not match the field `date`. I will shed some light on this issue without being technical.\n\nAs per [BigQuery Export schema](https://support.google.com/analytics/answer/3437719?hl=en), `visitStartTime` is 'The timestamp (expressed as POSIX time)' and as such it is **expressed in UTC**. \n\nOn the other hand `date` (or [ga:date](https://developers.google.com/analytics/devguides/reporting/core/dimsmets#view=detail&group=time&jump=ga_date) to be more specific) comes from Google Analytics and it is expressed on the timezone **that the GA administrator has configured in the specific [view's](https://support.google.com/analytics/answer/2649553?hl=en) settings**. \nIn layman terms, **it is a property of the website, not the client/web browser** and it should be consistent in all rows if they came from the same view irrespectively of the timezone of the visitor.\n\nThat being said, and knowing that the data come from Google Merchandise Store which resides in California (which is UTC -7), **every session started between 00:00 and 07:00 it is accounted in the previous day \"date wise\" and to the next day \"visitStartTime wise\"**.\n\nI hope the above is clear, otherwise please let me know.\n\nCheers", "393681": "Good to know! Thank you!", "393687": "Nice catch! Thanks for the tip!", "393693": "Thanks Yannis for the insight. Could you please elaborate how to explain the shift in the minimum of `visitStartTime` between countries in different timezones? For example, see the plots attached in https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/65691#387658", "393701": "I've seen a similar thing. If you look at duplicate sessionID's, they stem from that difference in timezone. As you said, those timezones are UTC and GA settings (i.e. California). There will be two rows for one sessionID when the visit starts right before midnight and ends after midnight (I forget in which of the timezones)\n\nI have to add one thing to your post. Sometimes the difference is 8 hours, not 7 hours. Probably a daily savings issue.", "393797": "Very interesting. Have you done any exploration with respect to daylight savings time and how that might affect the UTC to Pacific Standard/Daylight Time conversion?", "393809": "I haven't dug into it deeply, but the few cases I inspected indicated that DST does affect the conversion.", "394069": "If my code is right the max difference is 6:43:47.\n![enter image description here][1]\n\nCould you point me to a row where this difference is greater?\n\n [1]: https://i.imgur.com/uJaqJl2.png", "394082": "It looks reasonable to me. \nKeep in mind that the time in both plots is in UTC. Thus, it makes perfect sense that the lowest traffic between two time zones with X hours difference is also shifted by X hours. \n\nAs an example, if at 5:00 AM (local time) we have the lowest traffic: \n\n - 5:00 (AM) in the UK is 5:00 UTC\n - 5:00 (AM) in the USA is 9:00 - 13:00 UTC (depending on the specific time zone)", "394094": "No, I haven't but I don't think it will make any difference.\n\nIn general, I don't know if the `date` field has any importance. I was thinking about using the date from `visitStartTime` as a more consistent variable.\n\nA good idea would be to use `visitStartTime` and geo-information (where is present) to create an extra feature representing the `localTime` of the visit. My intuition says that this may be important for the visitors' behavior (maybe they buy more during the morning or night etc.)", "394097": "Oh, I just noticed that with the code above I just got a sample.\n\nYes, 8 hours is possible based on if a daylight saving time is in effect or not.\n\nhttps://www.timeanddate.com/time/map/", "394121": "So if `visitStartTime` is in UTC, it may indeed be prudent to account for daylight savings when creating a `localTime` feature?", "394126": "Sure, because UTC is not affected by daylight changes in contrast with the local times. \nThis should not be a problem though if you use something similar to [pandas.Series.tz_localize](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.tz_localize.html) to convert them.", "394133": "Good point. Not the first time I have foolishly considered handling dates myself... They're always deceptively complicated and there are always well-vetted packages out there for handling them", "394158": "Thanks for sharing, just curious how you find out this? Is it stated somewhere? Can you share the source with us?", "394171": "I picked a couple example from ones that have duplicate session IDs (which occurs when visits span around midnight). \nThere are two rows for each of those, so make sure you look at both.\nsessionID='0098453420096961116_1498458618' => 20170625/20170626 => 2017-06-26 07:00:16\n\n'738204402836923511_1487404770' => 20170217/ 20170218 => 2017-02-18 08:00:10\n\nNote how the first one is in the summer, while the second one is in the winter.", "394178": "Yannis, I believe your intuition will be proved to be correct. You don't even need to convert time to do so. E.g. look at hourly California data", "394197": "I had been working for some time as an analyst with Google Analytics data so when I noticed this discrepancy I figured out what was the issue. \nThen I saw that this troubles other members also and I shared it.\n\nI believe that `visitStartTime` is in UTC is stated in the projects' overview and also in BigQuery Export schema (check the link in my post). \nThe fact that the `date` is in the time zone that the administrator had set, was by experience but you can find it also in [GA documentation](https://support.google.com/analytics/answer/1010249?hl=en)", "394383": "Yes, you are right. It is obvious. I was confused.", "395093": "A few kernels use this visitStartTime as a numerical variables and it get high importance. Since train and test data are not on the same period, it shouldn't matter right ? \n\nI tried removing it but lower both the CV and the LB score. I also tried replacing this variable by a float representing the moment during the day, but same thing the score was not as good... Any thought on this ?", "395105": "some users appear in both train and test", "395300": "areveillon It may matter because Unix time continuous variable increasing with the time. So, if the revenue also grows with the time, there might be a correlation, but you get this information from the `date` also.\n\nI believe the power of `visitStartTime` comes when you combine it with the timezone to calculate the local time of the user.", "395323": "Let consider situation when some tree model is used and there is a split at the end of train data like before/after 16 Jun 2017. This rule will be used for all test, so using visitStartTime makes sense.\nAs for why there is a difference between visitStartTime and its float counter part, they should behave the same way, I may assume there is an error when converting to float. \nBtw, why to convert visitStartTime to float?\n\nFeatures like month, day of month, day of year, etc are more appropriate as they have good overlapping between train and test.", "395334": "If I get it right from \n\n> also tried replacing this variable by a float **representing the moment during the day**\n\nwhat Antoine does is a nice trick cause he gets a \"time of the day\" feature. \nBut Antoine, keep in mind that this will be the time of the purchase in another timezone (UTC) if you don't adjust it to the local time zone.", "395340": "I think a tree based model can handle easily this situation building its branch separation conditions with both UTC time and one location feature, if it considers that it optimizes the model, isn't it?", "395342": "I assume in theory you are right but the inference (from the model) of the local time from UTC and location features is a small regression problem by itself. Probably you may lose some accuracy \"inferring\" it instead of \"calculating\" it (with an external datasource, API, etc.). \n\nAm I thinking it right?", "395351": "The main problem with accuracy here is that 56% of records do not have city/metro/region, so we have to work with country level which sometimes means 6-8 hours spread.", "395365": "I agree with you @emorej, the model should theoretically be able to do it, but you don't really know what split lightgbm will do... Especially if you have hundred of features. It would be better to have a clear feature with local time, but it might not be very precise with all missing data.\n\nSo I tried to remove the visitStartTime, and add day / month / year + a float representing the moment of the day. Since there is no overlapping between Train and Test I would expect the visitStartTime to overfit the training set. Maybe there is a side effect as you were saying @alexfir but I'm not aware of such situation.\n\nAnyway by adding those variable I don't get the same CV LB that I get with visitStartTime ? @alexfir is your model based on visitStartTime ? or were you able to completely remove it ?\n\ntrain min 1470034812\ntrain max 1501657193\n\ntest min 1501657203\ntest max 1525157818", "395391": "areveillon, currently I am using plain visitStartTime as a feature, but I did not select features on this stage. \"visitStartTime\" is on 13rd place by importance, while plain \"date\" is on 7th while representing the same data. \n\nYou probably may get more from \"visitStartTime\" if converting to some kind of relative feature rather than absolute one. For example, how long time ago the user visited GStore for the first time. In this case model can treat users that visited for the first time 20 days ago in the same way in both train and test.", "395426": "I agree with you, relative feature for time should be more helpful... I did include some features like this, and they were not as good as the visitStartTime or date...\n\nI was using time split CV btw, and even locally the validation set is better when I include those time feature... I was just curious to understand why since the interval don't overlap.", "395728": "emorej The trick is that such optimisation would require to look at 2 variables simultaneously. But GBMs at each split considers only one variable.", "395855": "Misha yes you're right so even if this optimization could be found in depth tree, it's sensibly easier to calc directly a local time feature I admit it :)", "397205": "I have a question about converting ```visitStartTime``` to local time.\nHow can we find the timezone of the users?\n\nI have read [this topic](https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/66627), which points out the discrepancy between ```geoNetwork_country``` and ```geoNetwork_city``` (for example, when ```geoNetwork_city``` is Toronto, some of ```geoNetwork_country``` are not Canada).\n\nSo, I wonder how we find the right place of the users.", "413871": "Having said that, would you recommend using only one of those features? Since they are just a matter of +7 difference? And if so, we can drop date and be sure we also don't have the timezone effect. So keeping VisitStartTime keeps everything standardised and in one timezone (UTC - 7)?", "414328": "But we dont have a timezone field. Should we get it from the name of the country maybe" }, "source": "meta" }