The data is now ready to be downloaded.
\n\nWe have released a new dataset for Google Analytics Customer Revenue Prediction! This is intended to aid competitors in predicting future data (December 1st 2018 - January 31st 2019). We will be predicting the log of the sum of user revenue for this future time period for all users represented in the posted test set: test_v2.csv, which includes all transactions from May 1st 2018 to October 15th 2018.
We have included columns of the dataset that were previously excluded, in order to match the full set of data that’s available from the demo account. The train set and test set both contain all relevant data for all transactions in the data's time period, including the totals.transactionRevenue. Note that this means the public leaderboard is calculated on all users in test_v2.csv for their transactions between May 1st 2018 to October 15th 2018. Your true performance and final standing will be based on the private future-looking dataset, which will also be calculated for all users in test_v2.csv, but for the time period December 1st 2018 to January 31st 2019. As such, we recommend that you use caution when interpreting the results of the public leaderboard.
We are aware that we are missing the most recent data, from the weeks between October 16th 2018 and November 30th 2018. Unfortunately the process of preparing this data with many different groups involved is extremely heavy; there was no way to deliver this newer data while also providing enough time for competitors to work with it. I apologize in advance for this lack of recency, but you are still permitted to use it as external data if you extract it from the demo account.
\n\nPlease feel free to ask questions here.
\n\nGood luck!
\n\nNote: Edited to clarify that ALL users in test_v2.csv should have predictions generated for them, and final scores will be based on all predictions.
", "messages": [ { "id": 417914, "postDate": "2018-11-09T01:12:49.357Z", "content": "The data is now ready to be downloaded.
\n\nWe have released a new dataset for Google Analytics Customer Revenue Prediction! This is intended to aid competitors in predicting future data (December 1st 2018 - January 31st 2019). We will be predicting the log of the sum of user revenue for this future time period for all users represented in the posted test set: test_v2.csv, which includes all transactions from May 1st 2018 to October 15th 2018.
We have included columns of the dataset that were previously excluded, in order to match the full set of data that’s available from the demo account. The train set and test set both contain all relevant data for all transactions in the data's time period, including the totals.transactionRevenue. Note that this means the public leaderboard is calculated on all users in test_v2.csv for their transactions between May 1st 2018 to October 15th 2018. Your true performance and final standing will be based on the private future-looking dataset, which will also be calculated for all users in test_v2.csv, but for the time period December 1st 2018 to January 31st 2019. As such, we recommend that you use caution when interpreting the results of the public leaderboard.
We are aware that we are missing the most recent data, from the weeks between October 16th 2018 and November 30th 2018. Unfortunately the process of preparing this data with many different groups involved is extremely heavy; there was no way to deliver this newer data while also providing enough time for competitors to work with it. I apologize in advance for this lack of recency, but you are still permitted to use it as external data if you extract it from the demo account.
\n\nPlease feel free to ask questions here.
\n\nGood luck!
\n\nNote: Edited to clarify that ALL users in test_v2.csv should have predictions generated for them, and final scores will be based on all predictions.
", "rawMarkdown": "**The data is now ready to be downloaded.**\n\nWe have released a new dataset for Google Analytics Customer Revenue Prediction! This is intended to aid competitors in predicting future data (December 1st 2018 - January 31st 2019). We will be predicting the log of the sum of user revenue for this future time period for *all users represented in the posted test set: `test_v2.csv`, which includes all transactions from May 1st 2018 to October 15th 2018*. \n\nWe have included columns of the dataset that were previously excluded, in order to match the full set of data that’s available from the demo account. The train set and test set both contain ***all*** relevant data for all transactions in the data's time period, including the `totals.transactionRevenue`. Note that this means the public leaderboard is calculated on all users in `test_v2.csv` for their transactions between May 1st 2018 to October 15th 2018. Your true performance and final standing will be based on the private future-looking dataset, which will also be calculated for all users in `test_v2.csv`, but for the time period December 1st 2018 to January 31st 2019. As such, we recommend that you use caution when interpreting the results of the public leaderboard.\n\nWe are aware that we are missing the most recent data, from the weeks between October 16th 2018 and November 30th 2018. Unfortunately the process of preparing this data with many different groups involved is extremely heavy; there was no way to deliver this newer data while also providing enough time for competitors to work with it. I apologize in advance for this lack of recency, but you are still permitted to use it as external data if you extract it from the demo account.\n\nPlease feel free to ask questions here.\n\nGood luck!\n\nNote: Edited to clarify that ALL users in test_v2.csv should have predictions generated for them, and final scores will be based on all predictions.", "votes": 25 }, { "id": 418394, "postDate": "2018-11-09T20:49:17.570Z", "content": "Can you clear the existing submissions? \nSome people might think about \"quitting\" the competition and might want their past submissions which are now more or less useless (in regard to the private leaderboard) not to be counted.
", "rawMarkdown": "Can you clear the existing submissions? \nSome people might think about \"quitting\" the competition and might want their past submissions which are now more or less useless (in regard to the private leaderboard) not to be counted. ", "votes": 11, "replies": [ { "id": 418619, "postDate": "2018-11-10T09:18:25.323Z", "content": "I agree for leaderboard reset. Dear @philculliton, will you consider to reset the previous leaderboard (or submissions) again? As it's no longer relevant.
", "rawMarkdown": "I agree for leaderboard reset. Dear @philculliton, will you consider to reset the previous leaderboard (or submissions) again? As it's no longer relevant.", "votes": 2 }, { "id": 419150, "postDate": "2018-11-11T11:54:05.973Z", "rawMarkdown": "", "isDeleted": true }, { "id": 419998, "postDate": "2018-11-12T22:51:56.897Z", "content": "Thanks for the questions, and for bringing this to our attention - yep, I'll be resetting the previous leaderboard.
", "rawMarkdown": "Thanks for the questions, and for bringing this to our attention - yep, I'll be resetting the previous leaderboard.", "votes": 2 }, { "id": 420215, "postDate": "2018-11-13T09:47:16.187Z", "content": "Cool, thanks!
", "rawMarkdown": "Cool, thanks!" }, { "id": 420743, "postDate": "2018-11-14T04:07:38.247Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" } ] }, { "id": 418261, "postDate": "2018-11-09T15:01:11.013Z", "content": "Hi @philculliton, can you please clear one thing for us? What if we will submit every user as if he will return? Will you calculate score only for valid users or there will be some kind of penalty for incorrect predictions?\nThanks in advance
", "rawMarkdown": "Hi @philculliton, can you please clear one thing for us? What if we will submit every user as if he will return? Will you calculate score only for valid users or there will be some kind of penalty for incorrect predictions?\nThanks in advance", "votes": 9 }, { "id": 419152, "postDate": "2018-11-11T11:55:46.050Z", "content": "So the rules enterely change it doesn't look like a big change but it does affect all of our models and methodology, first off all we have to make predictions based on information that doesn't even exist, if you are using features with information of the previous weeks or time series then you need to make predictions for the hole period that is missing ( all the time from October to the end of November). We don't have the time or the resources to do this, also we don't have information from the prevoious year so whatever the predictions are they'll be really random so it doesn't really matter how \"good\" our models are if we don't have the information or even a board for guidance. I just think that all the efford and time that we invested in this challenge was for nothing.
", "rawMarkdown": "So the rules enterely change it doesn't look like a big change but it does affect all of our models and methodology, first off all we have to make predictions based on information that doesn't even exist, if you are using features with information of the previous weeks or time series then you need to make predictions for the hole period that is missing ( all the time from October to the end of November). We don't have the time or the resources to do this, also we don't have information from the prevoious year so whatever the predictions are they'll be really random so it doesn't really matter how \"good\" our models are if we don't have the information or even a board for guidance. I just think that all the efford and time that we invested in this challenge was for nothing.", "votes": 7, "replies": [ { "id": 419697, "postDate": "2018-11-12T12:17:02.427Z", "content": "I think kaggler should consider this.@philculliton
", "rawMarkdown": "I think kaggler should consider this.@philculliton", "votes": 1 }, { "id": 420227, "postDate": "2018-11-13T10:06:09.360Z", "content": "We are waiting an answer!
", "rawMarkdown": "We are waiting an answer!", "votes": 1 } ] }, { "id": 418114, "postDate": "2018-11-09T09:54:44.190Z", "content": "Data got too. large. Trying to read csv, but notebook restarts after some minutes without yielding any output.
", "rawMarkdown": "**Data got too. large.** Trying to read csv, but notebook **restarts** after some minutes without yielding any output.\n", "votes": 7, "replies": [ { "id": 418115, "postDate": "2018-11-09T10:04:27.260Z", "content": "True
", "rawMarkdown": "True", "votes": 2 }, { "id": 418481, "postDate": "2018-11-10T00:53:02.777Z", "content": "i was seeing the same, please let me know if you get that, thanks
", "rawMarkdown": "i was seeing the same, please let me know if you get that, thanks" }, { "id": 418499, "postDate": "2018-11-10T01:42:30.193Z", "content": "My local machine's memory is smaller than Kaggle, so i just read a chunk every time.
", "rawMarkdown": "My local machine's memory is smaller than Kaggle, so i just read a chunk every time." } ] }, { "id": 425819, "postDate": "2018-11-22T06:59:48.210Z", "content": "I get it now, this column looks like how you would hard-code a list in Python. It seems like the way to input that column is to make Python execute it as a code.
\n\nIt would've been helpful to explain how to parse that column, especially since it looks like a badly prepared JSON format data, and not everyone is familiar enough with Python to recognize it as a Python code. This also seems like a sloppy and super dangerous way to encode the data, so maybe the reluctance to document it is not that surprising.
\n\nI'm starting to think that this competition should've been euthanized instead of being halfheartedly dragged along to the finish line.
", "rawMarkdown": "I get it now, this column looks like how you would hard-code a list in Python. It seems like the way to input that column is to make Python execute it as a code.\n\nIt would've been helpful to explain how to parse that column, especially since it looks like a badly prepared JSON format data, and not everyone is familiar enough with Python to recognize it as a Python code. This also seems like a sloppy and super dangerous way to encode the data, so maybe the reluctance to document it is not that surprising.\n\nI'm starting to think that this competition should've been euthanized instead of being halfheartedly dragged along to the finish line.", "votes": 6, "replies": [ { "id": 425852, "postDate": "2018-11-22T07:51:24.540Z", "content": "Indeed, this competition should have been ended when the first leaked submission appeared on the LB. I still find it so funny and annoying that in the first form of the competition, we literally had the test target publicly on the internet. Isn't that so irresponsible? For the new form of the competition, I see that there is still many problems and people are so confused with the new data. From the very beginning, this competition is just a mess.
", "rawMarkdown": "Indeed, this competition should have been ended when the first leaked submission appeared on the LB. I still find it so funny and annoying that in the first form of the competition, we literally had the test target publicly on the internet. Isn't that so irresponsible? For the new form of the competition, I see that there is still many problems and people are so confused with the new data. From the very beginning, this competition is just a mess.", "votes": 6 }, { "id": 426676, "postDate": "2018-11-23T16:22:52.247Z", "content": "Hey Dmitriy -
\n\nThat's on me and I apologize for the state of that column. The format that the hits column is in, is how it is stored in - and extracts from - BigQuery.
In the first go-round we were doing our best to match the BigQuery formatting because we wanted code to be easily portable between working with data from BQ and from our CSV files, so my processing code deliberately didn't try to reformat columns. I followed that process again this time because I was hoping to still allow BQ code to work. So - that was my call and I don't think it worked out particularly well, but that is the reason it is that way.
", "rawMarkdown": "Hey Dmitriy -\n\nThat's on me and I apologize for the state of that column. The format that the `hits` column is in, is how it is stored in - and extracts from - BigQuery.\n\nIn the first go-round we were doing our best to match the BigQuery formatting because we wanted code to be easily portable between working with data from BQ and from our CSV files, so my processing code deliberately didn't try to reformat columns. I followed that process again this time because I was hoping to still allow BQ code to work. So - that was my call and I don't think it worked out particularly well, but that is the reason it is that way.", "votes": 2 } ] }, { "id": 418393, "postDate": "2018-11-09T20:42:04.950Z", "content": "Maybe it would be better to upload .zip archives instead of plain .csv files?
\n\nThat would reduce file sizes a lot, making them much easier/faster to download.
", "rawMarkdown": "Maybe it would be better to upload .zip archives instead of plain .csv files? \n\nThat would reduce file sizes a lot, making them much easier/faster to download.", "votes": 5, "replies": [ { "id": 418590, "postDate": "2018-11-10T08:02:35.453Z", "content": "Use \"Download All\" on data page, it allows downloading all files as single zip.
", "rawMarkdown": "Use \"Download All\" on data page, it allows downloading all files as single zip.", "votes": 2 } ] }, { "id": 420332, "postDate": "2018-11-13T13:12:18.020Z", "content": "who knows why we have two fields : \"totals.transactionRevenue\" and \"totals.totalTransactionRevenue\" ?\nand which of them must be TARGET ?
", "rawMarkdown": "who knows why we have two fields : \"totals.transactionRevenue\" and \"totals.totalTransactionRevenue\" ?\nand which of them must be TARGET ?", "votes": 6, "replies": [ { "id": 420741, "postDate": "2018-11-14T04:04:41.673Z", "content": "have the same question..
", "rawMarkdown": "have the same question.." } ] }, { "id": 418149, "postDate": "2018-11-09T11:22:27.430Z", "content": "It sounds little bit strange for me that Google already has a data of future transactions to test models performance...
", "rawMarkdown": "It sounds little bit strange for me that Google already has a data of future transactions to test models performance... ", "votes": 6 }, { "id": 420473, "postDate": "2018-11-13T17:24:25.377Z", "content": "Hi all - I wanted to clarify two things. I apologize for the confusion, and thanks to everyone who has asked questions and provided feedback. We are doing our level best to create something interesting, and your responses have been incredibly helpful. We really appreciate your patience.
\n\nFirst, I want to confirm that this is a future prediction problem, and you will need to make predictions on unseen data (December 2018-January 2019) for all users that visited the store between May 1st 2018 and October 15th 2018. These are represented by the set of fullVisitorIds in sample_submission_v2.csv.
Second, I want to confirm that predictions for all users from that time period will influence the score calculation. The objective of \"figure out how much a small number of users might spend, ignoring all others\" is extremely narrow and risks being broken in a number of ways. \"Figure out which users will spend again, and how much\" is much more robust, and much more likely to produce interesting / useful models and results.
\n\nThanks again for your feedback!
", "rawMarkdown": "Hi all - I wanted to clarify two things. I apologize for the confusion, and thanks to everyone who has asked questions and provided feedback. We are doing our level best to create something interesting, and your responses have been incredibly helpful. We really appreciate your patience.\n\nFirst, I want to confirm that this is a future prediction problem, and you will need to make predictions on unseen data (December 2018-January 2019) for all users that visited the store between May 1st 2018 and October 15th 2018. These are represented by the set of `fullVisitorId`s in `sample_submission_v2.csv`.\n\nSecond, I want to confirm that predictions for *all* users from that time period will influence the score calculation. The objective of \"figure out how much a small number of users might spend, ignoring all others\" is extremely narrow and risks being broken in a number of ways. \"Figure out which users will spend again, and how much\" is much more robust, and much more likely to produce interesting / useful models and results.\n\nThanks again for your feedback!", "votes": 4, "replies": [ { "id": 420479, "postDate": "2018-11-13T17:38:27.323Z", "content": "Thanks for the answer. My main question is: on what data will we make the final predictions? Will this competition turn into competition with 2 stages? Or should we treat this as a time-series problem with predictions for future periods?
", "rawMarkdown": "Thanks for the answer. My main question is: on what data will we make the final predictions? Will this competition turn into competition with 2 stages? Or should we treat this as a time-series problem with predictions for future periods?", "votes": 1 }, { "id": 420529, "postDate": "2018-11-13T19:22:19.270Z", "content": "Great question.
\n\nYou're predicting based on the data you already have - treating it as a time-series problem with predictions for future periods.
\n\nThe population of fullVisitorIds that you are predicting for are only those who showed up in the May 1st 2018 to October 15th 2018 data, matching the fullVisitorIds from sample_submission_v2.csv. If you are predicting that a user in that dataset does not return during the December 2018-January 2019 timeframe, then for that user you would predict 0. You are not being asked to predict the target for any users outside of that timeframe.
So, Can you make a explanation about the \"one line perfect score\", which makes the LB boring, many people just submit that result.\nThank you!
", "rawMarkdown": "So, Can you make a explanation about the \"one line perfect score\", which makes the LB boring, many people just submit that result.\nThank you!", "votes": 1 }, { "id": 420871, "postDate": "2018-11-14T09:07:54.177Z", "content": "Hi, Does the train set have sufficient repeat purchasers to be able to create a validation dataset ? In the earlier train set there were very few repeat purchasers in train, most users have only one purchase.
", "rawMarkdown": "Hi, Does the train set have sufficient repeat purchasers to be able to create a validation dataset ? In the earlier train set there were very few repeat purchasers in train, most users have only one purchase." }, { "id": 420964, "postDate": "2018-11-14T12:03:43.027Z", "content": "@GreatDataScientist,
\n\nAs I wrote in this post here:\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71427
\n\nValidation and also the predictive power, seem to be very weak!\nI hope i'm not doing anything wrong there, but i triple checked my scenario and it seems quite bad.
", "rawMarkdown": "@GreatDataScientist,\n\nAs I wrote in this post here:\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71427\n\nValidation and also the predictive power, seem to be very weak!\nI hope i'm not doing anything wrong there, but i triple checked my scenario and it seems quite bad.\n" }, { "id": 421035, "postDate": "2018-11-14T13:57:36.883Z", "content": "thanks for letting us know.
", "rawMarkdown": "thanks for letting us know." }, { "id": 421890, "postDate": "2018-11-15T14:39:42.697Z", "content": "The LB had already become fake when the test data leaked. The current LB has no value. As always you should rely on your validation to choose your best model.
", "rawMarkdown": "The LB had already become fake when the test data leaked. The current LB has no value. As always you should rely on your validation to choose your best model.", "votes": 2 } ] }, { "id": 420375, "postDate": "2018-11-13T14:11:56.573Z", "content": "Dear @philculliton\nI was working on this challenge in order to write a nice code and showcase it for my job hunt which turned out to be not the best choice. I would like to start with another competition and would rather not continue with the new datasets here. I just would like to know if it is possible that I can have access to the premier datasets (train.csv and test.csv) in order to be able to share my work. Because now I cannot load them anymore. Could you help me with that?
\n\nMany thanks
", "rawMarkdown": "Dear @philculliton\nI was working on this challenge in order to write a nice code and showcase it for my job hunt which turned out to be not the best choice. I would like to start with another competition and would rather not continue with the new datasets here. I just would like to know if it is possible that I can have access to the premier datasets (train.csv and test.csv) in order to be able to share my work. Because now I cannot load them anymore. Could you help me with that?\n\nMany thanks", "votes": 3, "replies": [ { "id": 421512, "postDate": "2018-11-15T04:53:48.807Z", "content": "Seconded. Could we have access to the old datasets? I too was using this for my job search so I'd love to be able to use the code I have to produce predictions.
", "rawMarkdown": "Seconded. Could we have access to the old datasets? I too was using this for my job search so I'd love to be able to use the code I have to produce predictions. ", "votes": 2 } ] }, { "id": 418868, "postDate": "2018-11-10T19:01:36.200Z", "content": "The leaderboard is really frustrating as one cannot see how well his solution did compared to other competitors until the very end.
", "rawMarkdown": "The leaderboard is really frustrating as one cannot see how well his solution did compared to other competitors until the very end.", "votes": 3, "replies": [ { "id": 419099, "postDate": "2018-11-11T09:12:34.950Z", "content": "I think the Lb with score = 0 should be clear,it's looks so boring and meaningless.
", "rawMarkdown": "I think the Lb with score = 0 should be clear,it's looks so boring and meaningless.", "votes": 1 } ] }, { "id": 418177, "postDate": "2018-11-09T12:28:57.623Z", "content": "Hi all, \nSo what I understanding about this competition is , public score has nothing to do with final standing right? What we can trust is our cross-validating score right?
", "rawMarkdown": "Hi all, \nSo what I understanding about this competition is , public score has nothing to do with final standing right? What we can trust is our cross-validating score right?", "votes": 3, "replies": [ { "id": 418281, "postDate": "2018-11-09T15:34:53.490Z", "content": "I believe that in every competition we have to rely on our CV score, but even more on this one as we can't test our CV on the LB.. Makes it even more exciting !
", "rawMarkdown": "I believe that in every competition we have to rely on our CV score, but even more on this one as we can't test our CV on the LB.. Makes it even more exciting !", "votes": 2 }, { "id": 418791, "postDate": "2018-11-10T16:02:22.147Z", "content": "Hi, u'r right. My second question is quite stupid haha. May I know do u use any external data that's helpful
", "rawMarkdown": "Hi, u'r right. My second question is quite stupid haha. May I know do u use any external data that's helpful" } ] }, { "id": 420939, "postDate": "2018-11-14T11:10:34.677Z", "content": "Actually reset the previous leaderboard is unfair. Especially submissions team was changed rapidly from more than 3000 to 300. because it will influence the final ranking. Even if the question changed, it's not their fault.
", "rawMarkdown": "Actually reset the previous leaderboard is unfair. Especially submissions team was changed rapidly from more than 3000 to 300. because it will influence the final ranking. Even if the question changed, it's not their fault.", "votes": 4, "replies": [ { "id": 420954, "postDate": "2018-11-14T11:50:23.917Z", "content": "Yes, many work is just in vain. This competition seems dead, only few day to go, but there are little meaningful submission.
", "rawMarkdown": "Yes, many work is just in vain. This competition seems dead, only few day to go, but there are little meaningful submission.", "votes": 2 }, { "id": 421025, "postDate": "2018-11-14T13:47:26.083Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 419349, "postDate": "2018-11-11T19:12:09.833Z", "content": "Dear @philculliton,
\n\nCould you help us to confirm one thing about the evaluation? Is it true that: \n- The predictions for customers who do not present between December 1st 2018 to January 31st 2019 will not influence the score calculation
\n\nAs this will decide how should we implment our own evaluation method.
\n\nThanks
", "rawMarkdown": "Dear @philculliton,\n\nCould you help us to confirm one thing about the evaluation? Is it true that: \n- The predictions for customers who do not present between December 1st 2018 to January 31st 2019 will not influence the score calculation\n\nAs this will decide how should we implment our own evaluation method.\n\nThanks\n", "votes": 4, "replies": [ { "id": 419905, "postDate": "2018-11-12T18:42:39.810Z", "content": "That seems to contradict Julia's post from a month ago. It's unclear which interpretation is the correct one.
", "rawMarkdown": "That seems to contradict Julia's post from a month ago. It's unclear which interpretation is the correct one." }, { "id": 420466, "postDate": "2018-11-13T17:03:31.630Z", "content": "Hi - sorry for the confusion, and thanks for the question. The predictions for customers who do not present between December 1st 2018 and January 31st 2019 will influence the score calculation.
", "rawMarkdown": "Hi - sorry for the confusion, and thanks for the question. The predictions for customers who do *not* present between December 1st 2018 and January 31st 2019 *will* influence the score calculation.", "votes": 4 }, { "id": 422184, "postDate": "2018-11-15T22:04:05.187Z", "content": "Thanks for your confirmation, it's helpful.
", "rawMarkdown": "Thanks for your confirmation, it's helpful. " }, { "id": 422735, "postDate": "2018-11-16T17:42:05.200Z", "content": "@philculliton That contradicts your statement:
\n\n\n\n\nYour true performance and final standing will be based on the private\n future-looking dataset, and only calculated for those users in\n test_v2.csv who returned during the future timeframe of December 1st\n 2018 to January 31st 2019.
\n
If we predict visitor A will spend $200, but visitor A doesn't visit between Dec 1 - Jan 31, will this affect our score? I'm confused because from my understanding, you've said two opposite things.
", "rawMarkdown": "@philculliton That contradicts your statement:\n\n> Your true performance and final standing will be based on the private\n> future-looking dataset, and only calculated for those users in\n> test_v2.csv who returned during the future timeframe of December 1st\n> 2018 to January 31st 2019.\n\nIf we predict visitor A will spend $200, but visitor A doesn't visit between Dec 1 - Jan 31, *will this affect our score?* I'm confused because from my understanding, you've said two opposite things." }, { "id": 422812, "postDate": "2018-11-16T20:04:34.623Z", "content": "Hey CoreyLevinson -
\n\nThanks! Good question, sorry for the confusion. I've updated my original post to reflect the clarification I made here - namely, that visitor A's spending will effect your score, even if they don't visit between Dec 1 - Jan 31.
", "rawMarkdown": "Hey CoreyLevinson -\n\nThanks! Good question, sorry for the confusion. I've updated my original post to reflect the clarification I made here - namely, that visitor A's spending *will* effect your score, even if they don't visit between Dec 1 - Jan 31." } ] }, { "id": 418478, "postDate": "2018-11-10T00:50:01.883Z", "content": "please help me as i am not able to load the new data. it keep on saying The kernel appears to have died. It will restart automatically.\nwhat is the problem \nplease help \nthanks in advance
", "rawMarkdown": "please help me as i am not able to load the new data. it keep on saying The kernel appears to have died. It will restart automatically.\nwhat is the problem \nplease help \nthanks in advance", "votes": 4, "replies": [ { "id": 418520, "postDate": "2018-11-10T02:47:37.200Z", "content": "Going through the same issue. Let me know please, if you find a work around for this. Thanks.
", "rawMarkdown": "Going through the same issue. Let me know please, if you find a work around for this. Thanks." }, { "id": 418524, "postDate": "2018-11-10T03:04:02.893Z", "content": "Read the huge data with a chunksize maybe work, i'm work on this.
", "rawMarkdown": "Read the huge data with a chunksize maybe work, i'm work on this." }, { "id": 418539, "postDate": "2018-11-10T04:24:01.043Z", "content": "You can use this to process, the result memory is still huge. https://www.kaggle.com/codlife/pre-processing-for-huge-train-data-with-chunksize.
", "rawMarkdown": "You can use this to process, the result memory is still huge. https://www.kaggle.com/codlife/pre-processing-for-huge-train-data-with-chunksize. " }, { "id": 419887, "postDate": "2018-11-12T17:46:28.973Z", "content": "I could not find anything page 404 is displayed\nthanks for the help
", "rawMarkdown": "I could not find anything page 404 is displayed\nthanks for the help" }, { "id": 420034, "postDate": "2018-11-13T00:40:56.757Z", "content": "Oh no， you should remove the period in the sentence last.
", "rawMarkdown": "Oh no， you should remove the period in the sentence last." }, { "id": 420428, "postDate": "2018-11-13T16:02:28.753Z", "content": "ooohh my bad got it thankyou
", "rawMarkdown": "ooohh my bad got it thankyou" } ] }, { "id": 418119, "postDate": "2018-11-09T10:19:24.937Z", "content": "How exactly are we going to predict data up to January 31 2019 if we have test data only up to October 15 2018? Or does his turn into simple time-series problem?
", "rawMarkdown": "How exactly are we going to predict data up to January 31 2019 if we have test data only up to October 15 2018? Or does his turn into simple time-series problem?", "votes": 3 }, { "id": 429418, "postDate": "2018-11-28T20:28:24.273Z", "content": "No answer for two days so I'll repost\nHi Phil - the prediction of totals.transactionRevenue is for the December 1st 2018 to January 31st 2019 and not for a May 1st 2018 to January 31st 2019.
", "rawMarkdown": "No answer for two days so I'll repost\nHi Phil - the prediction of totals.transactionRevenue is for the December 1st 2018 to January 31st 2019 and not for a May 1st 2018 to January 31st 2019.", "votes": 1, "replies": [ { "id": 430077, "postDate": "2018-11-29T19:39:01.407Z", "content": "I believe, this has been answered below, for example here: \"The Private LB is being calculated on the future-looking timeframe of 12/1/18 to 1/31/19 - for those same set of users. \"
", "rawMarkdown": "I believe, this has been answered below, for example [here](https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/70982#422799): _\"The Private LB is being calculated on the future-looking timeframe of 12/1/18 to 1/31/19 - for those same set of users. \"_", "votes": 1 } ] }, { "id": 425653, "postDate": "2018-11-21T22:56:10.580Z", "content": "Has anyone managed to parse the \"hits\" column? I don't have much experience with JSON, but I think it's not put together correctly as a JSON column.
", "rawMarkdown": "Has anyone managed to parse the \"hits\" column? I don't have much experience with JSON, but I think it's not put together correctly as a JSON column.", "votes": 1, "replies": [ { "id": 425772, "postDate": "2018-11-22T04:55:55.107Z", "content": "I think it's not json ( I may be wrong). I just used ast.literal_eval to parse hits.
", "rawMarkdown": "I think it's not json ( I may be wrong). I just used ast.literal_eval to parse hits.", "votes": 1 }, { "id": 425773, "postDate": "2018-11-22T04:58:22.673Z", "content": "Is this a python thing?
", "rawMarkdown": "Is this a python thing?" }, { "id": 425872, "postDate": "2018-11-22T08:25:40.427Z", "content": "yeah I use python
", "rawMarkdown": "yeah I use python " }, { "id": 425889, "postDate": "2018-11-22T08:54:52.690Z", "content": "Hello Dmitriy,
\n\nThis kernel can help you to flatten the hits column : https://www.kaggle.com/burritodan/gstore-pythonic-flatten-only-hits-column
", "rawMarkdown": "Hello Dmitriy,\n\nThis kernel can help you to flatten the hits column : https://www.kaggle.com/burritodan/gstore-pythonic-flatten-only-hits-column", "votes": 1 }, { "id": 426832, "postDate": "2018-11-24T00:44:29.717Z", "content": "My sense is that the hits column is multi-layer encoded (i.e. JSON embedded in JSON): hits-Product, for example. So its extracted form is not 1:1 with the test/train datasets.
\n\nMy process decodes the hits string once, and then calls json_normalize() on each record for each hits sub-column that is embedded in JSON and appends them to a separate data set, where I calc various aggregates by visit before joining back to source data.
", "rawMarkdown": "My sense is that the hits column is multi-layer encoded (i.e. JSON embedded in JSON): hits-Product, for example. So its extracted form is not 1:1 with the test/train datasets. \n\nMy process decodes the hits string once, and then calls json_normalize() on each record for each hits sub-column that is embedded in JSON and appends them to a separate data set, where I calc various aggregates by visit before joining back to source data." } ] }, { "id": 418527, "postDate": "2018-11-10T03:24:43.320Z", "content": "Since the competition is already relaunched, will the deadline change? Do we still need to submit our result before 30th, November? That's too rush!
", "rawMarkdown": "Since the competition is already relaunched, will the deadline change? Do we still need to submit our result before 30th, November? That's too rush!", "votes": 1, "replies": [ { "id": 419159, "postDate": "2018-11-11T12:06:28.590Z", "content": "Since the evaluation is based on transactions from Dec 1, there is no way to postpone submissions after Nov. 30...
", "rawMarkdown": "Since the evaluation is based on transactions from Dec 1, there is no way to postpone submissions after Nov. 30..." } ] }, { "id": 418288, "postDate": "2018-11-09T15:43:29.733Z", "content": "Small issue: sessionId seems not to be present in the updated data, but the Data tab still mentions it.
Hi , I also cannot find it.
", "rawMarkdown": "Hi , I also cannot find it." }, { "id": 419996, "postDate": "2018-11-12T22:50:35.410Z", "content": "Hi Misha - thanks for pointing this out! That's correct - sessionId is not included in the current dataset. fullVisitorId is the critical / unique ID in the dataset. sessionId was an ID created for the original dataset when the problem formulation hadn't been set yet, and has been dropped. I'll remove it from the Data tab.
\"[{'index': '4', 'value': 'EMEA'}]\" is not valid JSON because of using single quotes. \n\"There are multiple columns which contain JSON blobs of varying depth\" is not correct. \nYou can find such values in \"customDimensions\" and \"hits\" column.
", "rawMarkdown": "\"[{'index': '4', 'value': 'EMEA'}]\" is not valid JSON because of using single quotes. \n\"There are multiple columns which contain JSON blobs of varying depth\" is not correct. \nYou can find such values in \"customDimensions\" and \"hits\" column.", "votes": 1, "replies": [ { "id": 418269, "postDate": "2018-11-09T15:14:44.903Z", "content": "I used df = df.replace(to_replace=r'\\'', value='\\\"', regex=True) to make it \"JSON-like\"
", "rawMarkdown": "I used df = df.replace(to_replace=r'\\'', value='\\\"', regex=True) to make it \"JSON-like\"", "votes": 3 }, { "id": 418273, "postDate": "2018-11-09T15:21:16.960Z", "content": "Thanks! \nHow do you deal with values like this \"\"YouTube Men's Short Sleeve Hero Tee Black\"\" ?
", "rawMarkdown": "Thanks! \nHow do you deal with values like this \"\"YouTube Men's Short Sleeve Hero Tee Black\"\" ?" }, { "id": 418275, "postDate": "2018-11-09T15:23:20.367Z", "content": "Haven't seen this value yet (only working on nrows=10000 for now and trying to reduce the size of the dataframe..) in which column is it ?
", "rawMarkdown": "Haven't seen this value yet (only working on nrows=10000 for now and trying to reduce the size of the dataframe..) in which column is it ? " }, { "id": 418277, "postDate": "2018-11-09T15:27:20.637Z", "content": "This is 4th record in train_v2, column \"hits\"
\n\nI found solution. Did not check for all data yet:
\n\nimport ast\ns = \"{'username':'dfdsfdsf'}\"\nast.literal_eval(s)\nNop, this does not work.
", "rawMarkdown": "This is 4th record in train_v2, column \"hits\"\n\nI found [solution][1]. Did not check for all data yet:\n\n import ast\n s = \"{'username':'dfdsfdsf'}\"\n ast.literal_eval(s)\n\nNop, this does not work.\n\n [1]: https://stackoverflow.com/questions/4162642/single-vs-double-quotes-in-json", "votes": 1 }, { "id": 418280, "postDate": "2018-11-09T15:33:33.580Z", "content": "I didn't load my csv with this column.. i'm wondering if the \"hit\" column will be in the future-looking dataset, what do you think ?
", "rawMarkdown": "I didn't load my csv with this column.. i'm wondering if the \"hit\" column will be in the future-looking dataset, what do you think ?" }, { "id": 418282, "postDate": "2018-11-09T15:35:54.853Z", "content": "There will not be any other dataset, only train_v2 and test_v2. We should predict revenue for visitors from test_v2 using only this data.
", "rawMarkdown": "There will not be any other dataset, only train_v2 and test_v2. We should predict revenue for visitors from test_v2 using only this data." }, { "id": 418285, "postDate": "2018-11-09T15:38:26.923Z", "content": "When I was talking about the \"future-looking dataset\" I meant the dataset used to score our submissions in the final stand :)
", "rawMarkdown": "When I was talking about the \"future-looking dataset\" I meant the dataset used to score our submissions in the final stand :) " }, { "id": 418292, "postDate": "2018-11-09T15:50:43.777Z", "rawMarkdown": "", "isDeleted": true }, { "id": 418296, "postDate": "2018-11-09T15:54:23.893Z", "content": "Then I still do not understand what you meant :) We will do prediction for future. \nDataset to score is just visitorId and revenue for time range December 1st 2018 - January 31st 2019.
", "rawMarkdown": "Then I still do not understand what you meant :) We will do prediction for future. \nDataset to score is just visitorId and revenue for time range December 1st 2018 - January 31st 2019." }, { "id": 418299, "postDate": "2018-11-09T16:00:05.623Z", "content": "You are right, my bad, I'm confused with all those changes in the competition :) \nSo i'll put hits in my dataset again and see how to handle the \"\"YouTube Men's Short Sleeve Hero Tee Black\"\"
", "rawMarkdown": "You are right, my bad, I'm confused with all those changes in the competition :) \nSo i'll put hits in my dataset again and see how to handle the \"\"YouTube Men's Short Sleeve Hero Tee Black\"\" " }, { "id": 418306, "postDate": "2018-11-09T16:13:42.020Z", "content": "I believe the main problem of 'hits' column is not invalid JSON, but the fact that the columns contains information about all hits within this session. \nFor example, first record from train contains single hit, so 1-1 relation between session ans hits.\nSecond has two hits, so the session has multiple hit objects, which is hard to convert to flat table.
\n\nSolution is to have separate table for hits, but we should invent a key since session_id is not unique (or maybe this is ok)
", "rawMarkdown": "I believe the main problem of 'hits' column is not invalid JSON, but the fact that the columns contains information about all hits within this session. \nFor example, first record from train contains single hit, so 1-1 relation between session ans hits.\nSecond has two hits, so the session has multiple hit objects, which is hard to convert to flat table.\n\nSolution is to have separate table for hits, but we should invent a key since session_id is not unique (or maybe this is ok)" }, { "id": 418315, "postDate": "2018-11-09T16:29:47.520Z", "content": "Reposting deleted....\nCheck cell 8 in this public kernel https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue ... It shows some columns having constant values. Maybe excluding them might save some memory. Also check https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71048#418308 . According to vincent, dropping 'hits' would bring train_v2.csv to 800 MB.
", "rawMarkdown": "Reposting deleted....\nCheck cell 8 in this public kernel https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue ... It shows some columns having constant values. Maybe excluding them might save some memory. Also check https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71048#418308 . According to vincent, dropping 'hits' would bring train_v2.csv to 800 MB." }, { "id": 418320, "postDate": "2018-11-09T16:36:54.740Z", "content": "Thanks! But I do not have problems with memory so far :)\nFirst kernel you mentioned is based on old data, be careful.
", "rawMarkdown": "Thanks! But I do not have problems with memory so far :)\nFirst kernel you mentioned is based on old data, be careful." }, { "id": 418324, "postDate": "2018-11-09T16:42:06.447Z", "content": "Yes. But assuming it would have not changed, it would be a good option to try skip them. Also, are you running your notebook on local machine or on kaggle??? Because I am not even able to read train_v2.csv ...\nAlso, as you don't have memory problem, can you please tell if those columns have constant value in new dataset or not???
", "rawMarkdown": "Yes. But assuming it would have not changed, it would be a good option to try skip them. Also, are you running your notebook on local machine or on kaggle??? Because I am not even able to read train_v2.csv ...\nAlso, as you don't have memory problem, can you please tell if those columns have constant value in new dataset or not???" }, { "id": 418329, "postDate": "2018-11-09T16:55:26.310Z", "rawMarkdown": "", "isDeleted": true }, { "id": 418370, "postDate": "2018-11-09T18:40:39.387Z", "content": "Pratik, I do not have memory issues because I still have no idea how to read 'hits' and 'customDimensions' columns. It is not a problem to read all data excluding these columns.\nYou can safely drop the following fields because they are constant in new data sets:
\n\n'socialEngagementType', 'device.browserSize', 'device.browserVersion', 'device.flashVersion', 'device.language', 'device.mobileDeviceBranding', 'device.mobileDeviceInfo', 'device.mobileDeviceMarketingName', 'device.mobileDeviceModel', 'device.mobileInputSelector', 'device.operatingSystemVersion', 'device.screenColors', 'device.screenResolution', 'geoNetwork.cityId', 'geoNetwork.latitude', 'geoNetwork.longitude', 'geoNetwork.networkLocation', 'totals.visits', 'trafficSource.adwordsClickInfo.criteriaParameters'
\n\nThe problem is that many of these fields are part of json blobs, therefore you need read json fields completely, convert to flat table and only on this stage remove constant columns. in other words, knowing that some json fields are not needed cannot be used to minimize memory usage while reading.
", "rawMarkdown": "Pratik, I do not have memory issues because I still have no idea how to read 'hits' and 'customDimensions' columns. It is not a problem to read all data excluding these columns.\nYou can safely drop the following fields because they are constant in new data sets:\n\n'socialEngagementType', 'device.browserSize', 'device.browserVersion', 'device.flashVersion', 'device.language', 'device.mobileDeviceBranding', 'device.mobileDeviceInfo', 'device.mobileDeviceMarketingName', 'device.mobileDeviceModel', 'device.mobileInputSelector', 'device.operatingSystemVersion', 'device.screenColors', 'device.screenResolution', 'geoNetwork.cityId', 'geoNetwork.latitude', 'geoNetwork.longitude', 'geoNetwork.networkLocation', 'totals.visits', 'trafficSource.adwordsClickInfo.criteriaParameters'\n\nThe problem is that many of these fields are part of json blobs, therefore you need read json fields completely, convert to flat table and only on this stage remove constant columns. in other words, knowing that some json fields are not needed cannot be used to minimize memory usage while reading.", "votes": 2 }, { "id": 418498, "postDate": "2018-11-10T01:41:32.350Z", "content": "Yeah right.
", "rawMarkdown": "Yeah right." } ] }, { "id": 430729, "postDate": "2018-11-30T22:29:17.737Z", "content": "hi it says 3 months to go was the deadline changed
", "rawMarkdown": "hi it says 3 months to go was the deadline changed\n", "votes": 2, "replies": [ { "id": 431112, "postDate": "2018-12-01T16:18:47.217Z", "content": "That's because the results would be published on 15.2.19 on data until 31.1.19. Submission is of course closed, as we are in the test period.
", "rawMarkdown": "That's because the results would be published on 15.2.19 on data until 31.1.19. Submission is of course closed, as we are in the test period.", "votes": 1 } ] }, { "id": 422567, "postDate": "2018-11-16T12:19:45.313Z", "content": "@philculliton please can you confirm how we submit to the private and public leaderboards. Usually the two leaderboards are disjoint subsets of the test set. In this case, the predictions for the public leaderboard are a different timeframe from the predictions for the private leaderboard.
\n\nIf I make a prediction using the future prediction (Dec-Jan) it will score horribly on the public leaderboard, which is a past prediction for the (May-15/Oct) timeframe. But I guess that is what you want? It's a bit counterintuitive!
", "rawMarkdown": "@philculliton please can you confirm how we submit to the private and public leaderboards. Usually the two leaderboards are disjoint subsets of the test set. In this case, the predictions for the public leaderboard are a different timeframe from the predictions for the private leaderboard. \n\nIf I make a prediction using the future prediction (Dec-Jan) it will score horribly on the public leaderboard, which is a past prediction for the (May-15/Oct) timeframe. But I guess that is what you want? It's a bit counterintuitive!", "votes": 2, "replies": [ { "id": 422636, "postDate": "2018-11-16T14:41:39.770Z", "content": "That's it. You should rely only on your validation when it comes to real data forecasts.
", "rawMarkdown": "That's it. You should rely only on your validation when it comes to real data forecasts.", "votes": 3 }, { "id": 422799, "postDate": "2018-11-16T19:44:01.237Z", "content": "In this competition, the sample_submission_v2 file on the data page is composed of all fullVisitorIds for the 5/1/18 to 10/15/18 time period. The public leaderboard submission for this file is based on a separate timeframe than the private leaderboard.
As specified by Phil, above, and on the data page, the Public LB is being calculated for those visitors during the same timeframe of 5/1/18 to 10/15/18. This is all publicly-available for the target prediction, but intended to provide some means of signal for those who wish to use it in that way. The Private LB is being calculated on the future-looking timeframe of 12/1/18 to 1/31/19 - for those same set of users.
\n\nTherefore, your submission that is intended for the public LB timeframe will be different from the private LB timeframe, which will be rescored/recalculated on the future timeframe.
\n\nKnowing this, I expect that competitors will need to make explicit final submission selections for those submissions which represent what they expect those future-looking predictions to be. These final submission selections for which this competition permits 2, per the rules, can be made under \"My Submissions\" by the final submission deadline.
", "rawMarkdown": "In this competition, the `sample_submission_v2` file on the data page is composed of all `fullVisitorId`s for the 5/1/18 to 10/15/18 time period. The public leaderboard submission for this file is based on a separate timeframe than the private leaderboard. \n\nAs specified by Phil, above, and on the data page, the Public LB is being calculated for those visitors during the same timeframe of 5/1/18 to 10/15/18. This is all publicly-available for the target prediction, but intended to provide some means of signal for those who wish to use it in that way. The Private LB is being calculated on the future-looking timeframe of 12/1/18 to 1/31/19 - for those *same* set of users. \n\nTherefore, your submission that is intended for the public LB timeframe **will be different** from the private LB timeframe, which will be rescored/recalculated on the future timeframe.\n\nKnowing this, I expect that competitors will need to make explicit final submission selections for those submissions which represent what they expect those future-looking predictions to be. These final submission selections for which this competition permits 2, per the rules, can be made under \"My Submissions\" by the final submission deadline.", "votes": 2 }, { "id": 422806, "postDate": "2018-11-16T19:50:29.303Z", "content": "Can you please confirm about the target. Which is the correct one totals.transactionRevenue(this is given in data page) or totals.TotalTransactionRevenue ?
", "rawMarkdown": "Can you please confirm about the target. Which is the correct one totals.transactionRevenue(this is given in data page) or totals.TotalTransactionRevenue ?\n", "votes": 3 }, { "id": 422808, "postDate": "2018-11-16T19:52:12.143Z", "content": "Hi Suchith - the correct target is totals.transactionRevenue.
Phil, can you please respond to my post here:\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71779
\n\nI believe I have evidence against totals.transactionRevenue (and in favour of totals.totalTransactionRevenue) to be a meaningful metric to analyse.\nThank you!
", "rawMarkdown": "Phil, can you please respond to my post here:\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71779\n\nI believe I have evidence against totals.transactionRevenue (and in favour of totals.totalTransactionRevenue) to be a meaningful metric to analyse.\nThank you!", "votes": 1 }, { "id": 424073, "postDate": "2018-11-19T14:13:50.440Z", "content": "@philculliton, I would still like to hear from you regarding the meaningfulness of totals.transactionRevenue. In the post above I provided the link to the post where I laid out my arguments. And judging by the upvotes, they are not complete nonsense. \nThank you a lot in advance!
", "rawMarkdown": "@philculliton, I would still like to hear from you regarding the meaningfulness of totals.transactionRevenue. In the post above I provided the link to the post where I laid out my arguments. And judging by the upvotes, they are not complete nonsense. \nThank you a lot in advance!", "votes": 1 }, { "id": 424183, "postDate": "2018-11-19T18:00:25.817Z", "content": "Hey @antoniorossi - thanks for the heads up! I'm looking into it now.
", "rawMarkdown": "Hey @antoniorossi - thanks for the heads up! I'm looking into it now." } ] }, { "id": 418098, "postDate": "2018-11-09T09:31:22.460Z", "content": "\"Note that this means the public leaderboard is calculated on all users in test_v2.csv for their transactions between May 1st 2018 to October 15th 2018. Your true performance and final standing will be based on the private future-looking dataset\" AND \n\"The train set and test set both contain all relevant data for all transactions in the data's time period, including the totals.transactionRevenue\"
\n\nDoes it mean that someone can just copy the transaction revenue from the test set, make the sum of logs, post it and get the perfect model?
", "rawMarkdown": "\"Note that this means the public leaderboard is calculated on all users in test_v2.csv for their transactions between May 1st 2018 to October 15th 2018. Your true performance and final standing will be based on the private future-looking dataset\" AND \n\"The train set and test set both contain all relevant data for all transactions in the data's time period, including the totals.transactionRevenue\"\n\nDoes it mean that someone can just copy the transaction revenue from the test set, make the sum of logs, post it and get the perfect model? ", "votes": 2, "replies": [ { "id": 418130, "postDate": "2018-11-09T10:50:03.250Z", "content": "yep, but i suspect no one will be impressed!
", "rawMarkdown": "yep, but i suspect no one will be impressed!", "votes": 3 }, { "id": 423304, "postDate": "2018-11-17T22:03:10.483Z", "content": "I did it just as test but i didn't get the perfect score.
", "rawMarkdown": "I did it just as test but i didn't get the perfect score." }, { "id": 423418, "postDate": "2018-11-18T07:52:51.303Z", "content": "\n\n\nDoes it mean that someone can just copy the transaction revenue from the test set, make the sum of logs, post it and get the perfect model?
\n
It is log of sum...
", "rawMarkdown": ">Does it mean that someone can just copy the transaction revenue from the test set, make the **sum of logs**, post it and get the perfect model?\n\nIt is log of sum..." } ] }, { "id": 430902, "postDate": "2018-12-01T07:05:28.327Z", "content": "Hello Phil, when will the private leaderboard get reflected with the final score for this competition, since I see it says 2 months go.
", "rawMarkdown": "Hello Phil, when will the private leaderboard get reflected with the final score for this competition, since I see it says 2 months go.", "replies": [ { "id": 431109, "postDate": "2018-12-01T16:17:08.730Z", "content": "I'm the timline section it says 15.2.19
", "rawMarkdown": "I'm the timline section it says 15.2.19" } ] }, { "id": 419048, "postDate": "2018-11-11T06:21:33.243Z", "content": "Why Error while uploading
", "rawMarkdown": "Why Error while uploading", "votes": -2 }, { "id": 419014, "postDate": "2018-11-11T04:51:13.633Z", "content": "what happened？ I can't submit！
", "rawMarkdown": "what happened？ I can't submit！", "votes": -2 }, { "id": 429195, "postDate": "2018-11-28T13:55:31.723Z", "content": "Hi.\nThis is my first Kaggle project. I want to know my score. But it wasn't until today did I found out that the rules were completely changed. I just finished predicting the original dataset but all my efforts samed to be in vain. I could not submit my answer because the rows didn't match (sure, due to different data sets.) .\nWhat shell I do? How do I get a score? Or can I compare my prediction to the \"leaked\" test data myself? Where can I find it?
", "rawMarkdown": "Hi.\nThis is my first Kaggle project. I want to know my score. But it wasn't until today did I found out that the rules were completely changed. I just finished predicting the original dataset but all my efforts samed to be in vain. I could not submit my answer because the rows didn't match (sure, due to different data sets.) .\nWhat shell I do? How do I get a score? Or can I compare my prediction to the \"leaked\" test data myself? Where can I find it?", "replies": [ { "id": 430205, "postDate": "2018-11-30T02:38:36.630Z", "content": "the column totals. totals.transactionRevenue is the target
", "rawMarkdown": "the column totals. totals.transactionRevenue is the target" } ] }, { "id": 428272, "postDate": "2018-11-27T02:14:56.177Z", "content": "I see the totalTransactionRevenue in feild totals in file *test_v2.csv*. Does file *test_v2.csv* have something wrong. Does anyone can explain for me?
", "rawMarkdown": "I see the *totalTransactionRevenue* in feild *totals* in file *test_v2.csv*. Does file *test_v2.csv* have something wrong. Does anyone can explain for me?", "replies": [ { "id": 430442, "postDate": "2018-11-30T11:44:46.070Z", "content": "As explained in the topic, the testv2 file is actually another train file. The real test will be the transactionRevenue during 1/12/18 - 31/1/19 of the ID appearing in test2 (including those who won't even enter the google store in that period), and that's what you are to predict.
", "rawMarkdown": "As explained in the topic, the testv2 file is actually another train file. The real test will be the transactionRevenue during 1/12/18 - 31/1/19 of the ID appearing in test2 (including those who won't even enter the google store in that period), and that's what you are to predict." } ] }, { "id": 428045, "postDate": "2018-11-26T16:54:12.487Z", "content": "Hi Phil - the prediction of totals.transactionRevenue is for the December 1st 2018 to January 31st 2019 and not for a May 1st 2018 to January 31st 2019.
", "rawMarkdown": "Hi Phil - the prediction of totals.transactionRevenue is for the December 1st 2018 to January 31st 2019 and not for a May 1st 2018 to January 31st 2019." }, { "id": 427083, "postDate": "2018-11-24T14:17:27.483Z", "content": "@phil Are we predicting the log of revenue or Log1p of revenue? The target is Log1p, but the competition rules imply log for the prediction. It could make a big differences if you think the probability (ie expected spend) is close to zero.
", "rawMarkdown": "@phil Are we predicting the log of revenue or Log1p of revenue? The target is Log1p, but the competition rules imply log for the prediction. It could make a big differences if you think the probability (ie expected spend) is close to zero.", "replies": [ { "id": 427133, "postDate": "2018-11-24T17:35:31.763Z", "content": "log1p I think (it's mentioned in the data page)
", "rawMarkdown": "log1p I think (it's mentioned in the data page)", "votes": 1 } ] }, { "id": 423949, "postDate": "2018-11-19T10:10:27.887Z", "content": "From October 15th onwards: Traffic on the site has been stable, but purchase activity has dropped drastically, according to the GA demo account reports. It actually looks like there is a different reality now when it comes to users being able to complete transactions.
", "rawMarkdown": "From October 15th onwards: Traffic on the site has been stable, but purchase activity has dropped drastically, according to the GA demo account reports. It actually looks like there is a different reality now when it comes to users being able to complete transactions. " }, { "id": 422648, "postDate": "2018-11-16T15:10:47.227Z", "content": "@philculliton\nThis question may be already asked somewhere, but I want to make things clear about the duration of the data.
\nBefore the submission deadline, November 30th 2018, we can get the train and test data like this:
\n- Train data(train_v2): Aug 1st 2016 to Apr 30th 2018
\n- Test data(test_v2): May 1st 2018 to October 15th 2018
\nThen, after the submission deadline, we cannot re-run our scripts.
And, here is my question: after Feb 1st 2019, will the datasets change like this (and scripts re-run)?:
\n- Train data: Aug 1st 2016 to October 15th 2018
\n- Test data: December 1st 2018 to January 31st 2019
I have not ever join the 2 stage competition, so it confuses me a lot.
\nThanks if you help me.
Hi Tak! There will be no additional datasets in this competition. You're being asked to make predictions about the future behavior of all fullVisitorIds in test_v2.csv without ever seeing that data.
Hi Tak, your understanding is mostly correct. Your task is to take the visitors from 5/1/18 to 10/15/18, represented in full by sample_submission_v2.csv on the Data page, and predict the target variable for those visitors during the future-looking timeframe of 12/1/18 to 1/31/19. Review the Evaluation page for details on the target variable and how accuracy of your predictions is evaluated.
To do this, you may use any of the existing training data provided, and any external data, if you wish. The data we've been able to provide is only for the target variable from 12/1/16 up to 10/15/18, and that is available on the data page. Some may opt to pull the data after 10/15/18 and use it as external data.
\n\nFollowing 1/31/19, we will be pulling the data for those visitors during that private test period (12/1/18 to 1/31/19) and refreshing the private leaderboard with the actual scores for the submissions made.
\n\nReview my response below for more details on how the public & private LB submissions differ.
", "rawMarkdown": "Hi Tak, your understanding is mostly correct. Your task is to take the visitors from 5/1/18 to 10/15/18, represented in full by `sample_submission_v2.csv` on the [Data page](https://www.kaggle.com/c/ga-customer-revenue-prediction/data), and predict the target variable for those visitors during the future-looking timeframe of 12/1/18 to 1/31/19. Review the [Evaluation page](https://www.kaggle.com/c/ga-customer-revenue-prediction#evaluation) for details on the target variable and how accuracy of your predictions is evaluated. \n\nTo do this, you may use any of the existing training data provided, and any external data, if you wish. The data we've been able to provide is only for the target variable from 12/1/16 up to 10/15/18, and that is available on the data page. Some may opt to pull the data after 10/15/18 and use it as external data.\n\nFollowing 1/31/19, we will be pulling the data for those visitors during that private test period (12/1/18 to 1/31/19) and refreshing the private leaderboard with the actual scores for the submissions made.\n\nReview my [response below](https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/70982#422799) for more details on how the public & private LB submissions differ.", "votes": 1 }, { "id": 422882, "postDate": "2018-11-16T23:56:43.220Z", "content": "Thanks for your quick resoponses, Phil and Julia! I completely understand.
\n\nWe predict the revenue of the fullVisitorIds in test_v2.csv. The score is calculated in Public LB in regard to the timeframe of 5/1/18 to 10/15/18, and will be calculated in Private LB in regard to the timeframe of 12/1/18 to 1/31/19.
Hello,\nbefore your change, I was very well ranked and now I do not want to participate in the contests anymore.\nI do not find it professional to change the rules and leave us little time.\nHow can I unsubscribe?
", "rawMarkdown": "\nHello,\nbefore your change, I was very well ranked and now I do not want to participate in the contests anymore.\nI do not find it professional to change the rules and leave us little time.\nHow can I unsubscribe?" }, { "id": 421834, "postDate": "2018-11-15T13:28:49.483Z", "content": "I find all of my submissions are marked with error! Can anyone explain to me why does this happen !
", "rawMarkdown": "I find all of my submissions are marked with error! Can anyone explain to me why does this happen !", "replies": [ { "id": 422046, "postDate": "2018-11-15T18:07:12.320Z", "content": "maybe try to load the fullvistorID as character?
", "rawMarkdown": "maybe try to load the fullvistorID as character?", "votes": 1 } ] }, { "id": 421735, "postDate": "2018-11-15T10:49:22.230Z", "content": "@philculliton will the Private LB live (updated daily or weekly) ? Or are we going to wait until February to see how well our predictions did?
", "rawMarkdown": "@philculliton will the Private LB live (updated daily or weekly) ? Or are we going to wait until February to see how well our predictions did?", "replies": [ { "id": 422800, "postDate": "2018-11-16T19:46:15.087Z", "content": "The Private LB will be updated all at once following the 1/31/2019 evaluation timeframe. There will be no intermittent update. Note there will be a time gap until results are released, while the data is being pulled and rescored.
", "rawMarkdown": "The Private LB will be updated all at once following the 1/31/2019 evaluation timeframe. There will be no intermittent update. Note there will be a time gap until results are released, while the data is being pulled and rescored.", "votes": 1 } ] }, { "id": 421600, "postDate": "2018-11-15T07:08:42.230Z", "content": "The fields \"totals_totalTransactionRevenue\" contains information about target………..\nWhat am I predicting? \"totals_totalTransactionRevenue\" or \"totals_transactionRevenue\"?\nis \"totals_transactionRevenue\" I think
", "rawMarkdown": "The fields \"totals_totalTransactionRevenue\" contains information about target………..\nWhat am I predicting? \"totals_totalTransactionRevenue\" or \"totals_transactionRevenue\"?\nis \"totals_transactionRevenue\" I think", "replies": [ { "id": 421636, "postDate": "2018-11-15T08:13:49.670Z", "content": "You're going to predict \"totalstotalTransactionRevenue\" but it seems like it's impossible to submit now
", "rawMarkdown": "You're going to predict \"totalstotalTransactionRevenue\" but it seems like it's impossible to submit now " }, { "id": 421884, "postDate": "2018-11-15T14:34:10.060Z", "content": "The goal now is to provide totals.transactionRevenue for the future period, from December 2018 to January 2019. The LB are being calculated based on a past period.
", "rawMarkdown": "The goal now is to provide **totals.transactionRevenue** for the future period, from December 2018 to January 2019. The LB are being calculated based on a past period.", "votes": 2 } ] }, { "id": 420166, "postDate": "2018-11-13T07:16:46.947Z", "content": "If a visitor returns a few times during the prediction period, are we to predict the sum of his transactions, the mean, the median or just for the first visit?
", "rawMarkdown": "If a visitor returns a few times during the prediction period, are we to predict the sum of his transactions, the mean, the median or just for the first visit?", "replies": [ { "id": 420234, "postDate": "2018-11-13T10:21:04.737Z", "content": "sum
", "rawMarkdown": "sum", "votes": 1 }, { "id": 420294, "postDate": "2018-11-13T12:29:07.560Z", "rawMarkdown": "", "isDeleted": true }, { "id": 420308, "postDate": "2018-11-13T12:46:44.660Z", "content": "df.groupby('fullVisitorId').sum().reset_index() ... or submitting directly gives duplicate entries error... no guessing.... also find natural log of final result...
", "rawMarkdown": "df.groupby('fullVisitorId').sum().reset_index() ... or submitting directly gives duplicate entries error... no guessing.... also find natural log of final result...", "votes": 1 } ] }, { "id": 420092, "postDate": "2018-11-13T03:33:14.163Z", "content": "I can't find transationRevenue in column totals of train dataset
", "rawMarkdown": "I can't find transationRevenue in column totals of train dataset" }, { "id": 419339, "postDate": "2018-11-11T18:39:31.230Z", "content": "So, will be there the test_v3.csv file with future timeframe of December 1st 2018 to January 31st 2019? Because otherwise I do not see any point in creating ML model with current features. I could just use random number generator.
", "rawMarkdown": "So, will be there the test_v3.csv file with future timeframe of December 1st 2018 to January 31st 2019? Because otherwise I do not see any point in creating ML model with current features. I could just use random number generator." }, { "id": 419179, "postDate": "2018-11-11T12:43:49.477Z", "content": "I might be missing a HUGE thing here, but for what I understand, in the most fundamental level, we are to predict targets without related features. Though there is some thinking to do here, it doesn't look like a DS / ML competition to me.
", "rawMarkdown": "I might be missing a HUGE thing here, but for what I understand, in the most fundamental level, we are to predict targets without related features. Though there is some thinking to do here, it doesn't look like a DS / ML competition to me.", "replies": [ { "id": 419271, "postDate": "2018-11-11T16:03:47.310Z", "content": "I think it is still a ML competition if you manage to implement the sorcerer's AI algorithm
", "rawMarkdown": "I think it is still a ML competition if you manage to implement the sorcerer's AI algorithm" }, { "id": 419350, "postDate": "2018-11-11T19:13:51.867Z", "content": "Can I know what do you mean by \"sorcerer's AI algorithm\"? Failed to google it,
", "rawMarkdown": "Can I know what do you mean by \"sorcerer's AI algorithm\"? Failed to google it," } ] }, { "id": 418616, "postDate": "2018-11-10T09:17:20.077Z", "content": "Hi Kaggle and all,\nCan I confirm if there is no sessionId in train_v2 and test_v2 ??
", "rawMarkdown": "Hi Kaggle and all,\nCan I confirm if there is no sessionId in train_v2 and test_v2 ?? ", "replies": [ { "id": 420000, "postDate": "2018-11-12T22:53:01.713Z", "content": "Correct - sessionId has been dropped. I've removed it from the Data page.
We're working on an issue with uploading the data - it should be ready tomorrow!
", "rawMarkdown": "We're working on an issue with uploading the data - it should be ready tomorrow!", "replies": [ { "id": 418109, "postDate": "2018-11-09T09:42:28.673Z", "content": "Now the files are there but they cant be downloaded.
", "rawMarkdown": "Now the files are there but they cant be downloaded." }, { "id": 418145, "postDate": "2018-11-09T11:15:09.527Z", "content": "Hi @Phil
\n\ntrain_v2 is 25.4GB.\ntest_v2 is 7.6GB.
\n\nIs this normal?
\n\nIt is bigger than I assumed.
", "rawMarkdown": "Hi @Phil\n\ntrain_v2 is 25.4GB.\ntest_v2 is 7.6GB.\n\nIs this normal?\n\nIt is bigger than I assumed.", "votes": 2 }, { "id": 418460, "postDate": "2018-11-09T23:34:59.157Z", "content": "It’s the inclusion of the hits data. Each record is an entire multilevel dataframe of stuff.
", "rawMarkdown": "It’s the inclusion of the hits data. Each record is an entire multilevel dataframe of stuff.", "votes": 1 } ] }, { "id": 417968, "postDate": "2018-11-09T04:10:34.007Z", "content": "Hello Phil\nData page says that you need to download test_v2 and train_v2. But I can't see any files named train v2 and test v2. \nOr we have to download train.csv and test.csv files ?
", "rawMarkdown": "Hello Phil\nData page says that you need to download test_v2 and train_v2. But I can't see any files named train v2 and test v2. \nOr we have to download train.csv and test.csv files ?", "replies": [ { "id": 418204, "postDate": "2018-11-09T13:27:48.197Z", "content": "It has been uploaded :)
\n\nBut trainv2 & testv2 are huge ! A bit weird...
", "rawMarkdown": "It has been uploaded :) \n\nBut trainv2 & testv2 are huge ! A bit weird...", "votes": 1 } ] }, { "id": 430736, "postDate": "2018-11-30T22:37:27.673Z", "rawMarkdown": "", "isDeleted": true }, { "id": 426838, "postDate": "2018-11-24T01:02:12.860Z", "rawMarkdown": "", "isDeleted": true }, { "id": 418782, "postDate": "2018-11-10T15:46:48.083Z", "rawMarkdown": "", "isDeleted": true }, { "id": 418389, "postDate": "2018-11-09T20:29:04.077Z", "rawMarkdown": "", "votes": 5, "isDeleted": true, "replies": [ { "id": 419997, "postDate": "2018-11-12T22:51:22.210Z", "content": "Hi Anton - good question! We'll be invalidating all old submissions.
", "rawMarkdown": "Hi Anton - good question! We'll be invalidating all old submissions.", "votes": 1 } ] }, { "id": 418189, "postDate": "2018-11-09T12:56:28.157Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ], "comments": [ { "id": 418394, "author_name": "randommodnar", "author_url": "", "post_date": "2018-11-09T20:49:17.570000", "content": "Can you clear the existing submissions? \nSome people might think about \"quitting\" the competition and might want their past submissions which are now more or less useless (in regard to the private leaderboard) not to be counted.
", "votes": 11, "replies": [ { "id": 418619, "author_name": "SDA", "author_url": "", "post_date": "2018-11-10T09:18:25.323000", "content": "I agree for leaderboard reset. Dear @philculliton, will you consider to reset the previous leaderboard (or submissions) again? As it's no longer relevant.
", "votes": 2, "replies": [] }, { "id": 419150, "author_name": "", "author_url": "", "post_date": "2018-11-11T11:54:05.973000", "content": "", "votes": 0, "replies": [] }, { "id": 419998, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-11-12T22:51:56.897000", "content": "Thanks for the questions, and for bringing this to our attention - yep, I'll be resetting the previous leaderboard.
", "votes": 2, "replies": [] }, { "id": 420215, "author_name": "SDA", "author_url": "", "post_date": "2018-11-13T09:47:16.187000", "content": "Cool, thanks!
", "votes": 0, "replies": [] }, { "id": 420743, "author_name": "randommodnar", "author_url": "", "post_date": "2018-11-14T04:07:38.247000", "content": "Thank you!
", "votes": 0, "replies": [] } ] }, { "id": 418261, "author_name": "WispZero", "author_url": "", "post_date": "2018-11-09T15:01:11.013000", "content": "Hi @philculliton, can you please clear one thing for us? What if we will submit every user as if he will return? Will you calculate score only for valid users or there will be some kind of penalty for incorrect predictions?\nThanks in advance
", "votes": 9, "replies": [] }, { "id": 419152, "author_name": "AlejandroCoronado", "author_url": "", "post_date": "2018-11-11T11:55:46.050000", "content": "So the rules enterely change it doesn't look like a big change but it does affect all of our models and methodology, first off all we have to make predictions based on information that doesn't even exist, if you are using features with information of the previous weeks or time series then you need to make predictions for the hole period that is missing ( all the time from October to the end of November). We don't have the time or the resources to do this, also we don't have information from the prevoious year so whatever the predictions are they'll be really random so it doesn't really matter how \"good\" our models are if we don't have the information or even a board for guidance. I just think that all the efford and time that we invested in this challenge was for nothing.
", "votes": 7, "replies": [ { "id": 419697, "author_name": "Jianfei Wang", "author_url": "", "post_date": "2018-11-12T12:17:02.427000", "content": "I think kaggler should consider this.@philculliton
", "votes": 1, "replies": [] }, { "id": 420227, "author_name": "data", "author_url": "", "post_date": "2018-11-13T10:06:09.360000", "content": "We are waiting an answer!
", "votes": 1, "replies": [] } ] }, { "id": 418114, "author_name": "Pratik Mangesh Sonar", "author_url": "", "post_date": "2018-11-09T09:54:44.190000", "content": "Data got too. large. Trying to read csv, but notebook restarts after some minutes without yielding any output.
", "votes": 7, "replies": [ { "id": 418115, "author_name": "Ramona Fli", "author_url": "", "post_date": "2018-11-09T10:04:27.260000", "content": "True
", "votes": 2, "replies": [] }, { "id": 418481, "author_name": "Kalyan", "author_url": "", "post_date": "2018-11-10T00:53:02.777000", "content": "i was seeing the same, please let me know if you get that, thanks
", "votes": 0, "replies": [] }, { "id": 418499, "author_name": "Jianfei Wang", "author_url": "", "post_date": "2018-11-10T01:42:30.193000", "content": "My local machine's memory is smaller than Kaggle, so i just read a chunk every time.
", "votes": 0, "replies": [] } ] }, { "id": 425819, "author_name": "Dmitriy Guller", "author_url": "", "post_date": "2018-11-22T06:59:48.210000", "content": "I get it now, this column looks like how you would hard-code a list in Python. It seems like the way to input that column is to make Python execute it as a code.
\n\nIt would've been helpful to explain how to parse that column, especially since it looks like a badly prepared JSON format data, and not everyone is familiar enough with Python to recognize it as a Python code. This also seems like a sloppy and super dangerous way to encode the data, so maybe the reluctance to document it is not that surprising.
\n\nI'm starting to think that this competition should've been euthanized instead of being halfheartedly dragged along to the finish line.
", "votes": 6, "replies": [ { "id": 425852, "author_name": "Fatih Öztürk", "author_url": "", "post_date": "2018-11-22T07:51:24.540000", "content": "Indeed, this competition should have been ended when the first leaked submission appeared on the LB. I still find it so funny and annoying that in the first form of the competition, we literally had the test target publicly on the internet. Isn't that so irresponsible? For the new form of the competition, I see that there is still many problems and people are so confused with the new data. From the very beginning, this competition is just a mess.
", "votes": 6, "replies": [] }, { "id": 426676, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-11-23T16:22:52.247000", "content": "Hey Dmitriy -
\n\nThat's on me and I apologize for the state of that column. The format that the hits column is in, is how it is stored in - and extracts from - BigQuery.
In the first go-round we were doing our best to match the BigQuery formatting because we wanted code to be easily portable between working with data from BQ and from our CSV files, so my processing code deliberately didn't try to reformat columns. I followed that process again this time because I was hoping to still allow BQ code to work. So - that was my call and I don't think it worked out particularly well, but that is the reason it is that way.
", "votes": 2, "replies": [] } ] }, { "id": 418393, "author_name": "alijs", "author_url": "", "post_date": "2018-11-09T20:42:04.950000", "content": "Maybe it would be better to upload .zip archives instead of plain .csv files?
\n\nThat would reduce file sizes a lot, making them much easier/faster to download.
", "votes": 5, "replies": [ { "id": 418590, "author_name": "Alexander Firsov", "author_url": "", "post_date": "2018-11-10T08:02:35.453000", "content": "Use \"Download All\" on data page, it allows downloading all files as single zip.
", "votes": 2, "replies": [] } ] }, { "id": 420332, "author_name": "Valeriy Garipov", "author_url": "", "post_date": "2018-11-13T13:12:18.020000", "content": "who knows why we have two fields : \"totals.transactionRevenue\" and \"totals.totalTransactionRevenue\" ?\nand which of them must be TARGET ?
", "votes": 6, "replies": [ { "id": 420741, "author_name": "Zhiwei Zhang", "author_url": "", "post_date": "2018-11-14T04:04:41.673000", "content": "have the same question..
", "votes": 0, "replies": [] } ] }, { "id": 418149, "author_name": "Max Lukyanenko", "author_url": "", "post_date": "2018-11-09T11:22:27.430000", "content": "It sounds little bit strange for me that Google already has a data of future transactions to test models performance...
", "votes": 6, "replies": [] }, { "id": 420473, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-11-13T17:24:25.377000", "content": "Hi all - I wanted to clarify two things. I apologize for the confusion, and thanks to everyone who has asked questions and provided feedback. We are doing our level best to create something interesting, and your responses have been incredibly helpful. We really appreciate your patience.
\n\nFirst, I want to confirm that this is a future prediction problem, and you will need to make predictions on unseen data (December 2018-January 2019) for all users that visited the store between May 1st 2018 and October 15th 2018. These are represented by the set of fullVisitorIds in sample_submission_v2.csv.
Second, I want to confirm that predictions for all users from that time period will influence the score calculation. The objective of \"figure out how much a small number of users might spend, ignoring all others\" is extremely narrow and risks being broken in a number of ways. \"Figure out which users will spend again, and how much\" is much more robust, and much more likely to produce interesting / useful models and results.
\n\nThanks again for your feedback!
", "votes": 4, "replies": [ { "id": 420479, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2018-11-13T17:38:27.323000", "content": "Thanks for the answer. My main question is: on what data will we make the final predictions? Will this competition turn into competition with 2 stages? Or should we treat this as a time-series problem with predictions for future periods?
", "votes": 1, "replies": [] }, { "id": 420529, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-11-13T19:22:19.270000", "content": "Great question.
\n\nYou're predicting based on the data you already have - treating it as a time-series problem with predictions for future periods.
\n\nThe population of fullVisitorIds that you are predicting for are only those who showed up in the May 1st 2018 to October 15th 2018 data, matching the fullVisitorIds from sample_submission_v2.csv. If you are predicting that a user in that dataset does not return during the December 2018-January 2019 timeframe, then for that user you would predict 0. You are not being asked to predict the target for any users outside of that timeframe.
So, Can you make a explanation about the \"one line perfect score\", which makes the LB boring, many people just submit that result.\nThank you!
", "votes": 1, "replies": [] }, { "id": 420871, "author_name": "Chinta", "author_url": "", "post_date": "2018-11-14T09:07:54.177000", "content": "Hi, Does the train set have sufficient repeat purchasers to be able to create a validation dataset ? In the earlier train set there were very few repeat purchasers in train, most users have only one purchase.
", "votes": 0, "replies": [] }, { "id": 420964, "author_name": "AmirH", "author_url": "", "post_date": "2018-11-14T12:03:43.027000", "content": "@GreatDataScientist,
\n\nAs I wrote in this post here:\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71427
\n\nValidation and also the predictive power, seem to be very weak!\nI hope i'm not doing anything wrong there, but i triple checked my scenario and it seems quite bad.
", "votes": 0, "replies": [] }, { "id": 421035, "author_name": "Chinta", "author_url": "", "post_date": "2018-11-14T13:57:36.883000", "content": "thanks for letting us know.
", "votes": 0, "replies": [] }, { "id": 421890, "author_name": "Paulo Pinto", "author_url": "", "post_date": "2018-11-15T14:39:42.697000", "content": "The LB had already become fake when the test data leaked. The current LB has no value. As always you should rely on your validation to choose your best model.
", "votes": 2, "replies": [] } ] }, { "id": 420375, "author_name": "Ramona Fli", "author_url": "", "post_date": "2018-11-13T14:11:56.573000", "content": "Dear @philculliton\nI was working on this challenge in order to write a nice code and showcase it for my job hunt which turned out to be not the best choice. I would like to start with another competition and would rather not continue with the new datasets here. I just would like to know if it is possible that I can have access to the premier datasets (train.csv and test.csv) in order to be able to share my work. Because now I cannot load them anymore. Could you help me with that?
\n\nMany thanks
", "votes": 3, "replies": [ { "id": 421512, "author_name": "iGotsCats", "author_url": "", "post_date": "2018-11-15T04:53:48.807000", "content": "Seconded. Could we have access to the old datasets? I too was using this for my job search so I'd love to be able to use the code I have to produce predictions.
", "votes": 2, "replies": [] } ] }, { "id": 418868, "author_name": "restricteur", "author_url": "", "post_date": "2018-11-10T19:01:36.200000", "content": "The leaderboard is really frustrating as one cannot see how well his solution did compared to other competitors until the very end.
", "votes": 3, "replies": [ { "id": 419099, "author_name": "Jianfei Wang", "author_url": "", "post_date": "2018-11-11T09:12:34.950000", "content": "I think the Lb with score = 0 should be clear,it's looks so boring and meaningless.
", "votes": 1, "replies": [] } ] }, { "id": 418177, "author_name": "Yunrui-li", "author_url": "", "post_date": "2018-11-09T12:28:57.623000", "content": "Hi all, \nSo what I understanding about this competition is , public score has nothing to do with final standing right? What we can trust is our cross-validating score right?
", "votes": 3, "replies": [ { "id": 418281, "author_name": "Y Assous", "author_url": "", "post_date": "2018-11-09T15:34:53.490000", "content": "I believe that in every competition we have to rely on our CV score, but even more on this one as we can't test our CV on the LB.. Makes it even more exciting !
", "votes": 2, "replies": [] }, { "id": 418791, "author_name": "Yunrui-li", "author_url": "", "post_date": "2018-11-10T16:02:22.147000", "content": "Hi, u'r right. My second question is quite stupid haha. May I know do u use any external data that's helpful
", "votes": 0, "replies": [] } ] }, { "id": 420939, "author_name": "Ping Ye", "author_url": "", "post_date": "2018-11-14T11:10:34.677000", "content": "Actually reset the previous leaderboard is unfair. Especially submissions team was changed rapidly from more than 3000 to 300. because it will influence the final ranking. Even if the question changed, it's not their fault.
", "votes": 4, "replies": [ { "id": 420954, "author_name": "Jianfei Wang", "author_url": "", "post_date": "2018-11-14T11:50:23.917000", "content": "Yes, many work is just in vain. This competition seems dead, only few day to go, but there are little meaningful submission.
", "votes": 2, "replies": [] }, { "id": 421025, "author_name": "", "author_url": "", "post_date": "2018-11-14T13:47:26.083000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 419349, "author_name": "Fangzheng Hu", "author_url": "", "post_date": "2018-11-11T19:12:09.833000", "content": "Dear @philculliton,
\n\nCould you help us to confirm one thing about the evaluation? Is it true that: \n- The predictions for customers who do not present between December 1st 2018 to January 31st 2019 will not influence the score calculation
\n\nAs this will decide how should we implment our own evaluation method.
\n\nThanks
", "votes": 4, "replies": [ { "id": 419905, "author_name": "Allen", "author_url": "", "post_date": "2018-11-12T18:42:39.810000", "content": "That seems to contradict Julia's post from a month ago. It's unclear which interpretation is the correct one.
", "votes": 0, "replies": [] }, { "id": 420466, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-11-13T17:03:31.630000", "content": "Hi - sorry for the confusion, and thanks for the question. The predictions for customers who do not present between December 1st 2018 and January 31st 2019 will influence the score calculation.
", "votes": 4, "replies": [] }, { "id": 422184, "author_name": "Fangzheng Hu", "author_url": "", "post_date": "2018-11-15T22:04:05.187000", "content": "Thanks for your confirmation, it's helpful.
", "votes": 0, "replies": [] }, { "id": 422735, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-11-16T17:42:05.200000", "content": "@philculliton That contradicts your statement:
\n\n\n\n\nYour true performance and final standing will be based on the private\n future-looking dataset, and only calculated for those users in\n test_v2.csv who returned during the future timeframe of December 1st\n 2018 to January 31st 2019.
\n
If we predict visitor A will spend $200, but visitor A doesn't visit between Dec 1 - Jan 31, will this affect our score? I'm confused because from my understanding, you've said two opposite things.
", "votes": 0, "replies": [] }, { "id": 422812, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-11-16T20:04:34.623000", "content": "Hey CoreyLevinson -
\n\nThanks! Good question, sorry for the confusion. I've updated my original post to reflect the clarification I made here - namely, that visitor A's spending will effect your score, even if they don't visit between Dec 1 - Jan 31.
", "votes": 0, "replies": [] } ] }, { "id": 418478, "author_name": "Kalyan", "author_url": "", "post_date": "2018-11-10T00:50:01.883000", "content": "please help me as i am not able to load the new data. it keep on saying The kernel appears to have died. It will restart automatically.\nwhat is the problem \nplease help \nthanks in advance
", "votes": 4, "replies": [ { "id": 418520, "author_name": "Aafrin Dabhoiwala", "author_url": "", "post_date": "2018-11-10T02:47:37.200000", "content": "Going through the same issue. Let me know please, if you find a work around for this. Thanks.
", "votes": 0, "replies": [] }, { "id": 418524, "author_name": "Jianfei Wang", "author_url": "", "post_date": "2018-11-10T03:04:02.893000", "content": "Read the huge data with a chunksize maybe work, i'm work on this.
", "votes": 0, "replies": [] }, { "id": 418539, "author_name": "Jianfei Wang", "author_url": "", "post_date": "2018-11-10T04:24:01.043000", "content": "You can use this to process, the result memory is still huge. https://www.kaggle.com/codlife/pre-processing-for-huge-train-data-with-chunksize.
", "votes": 0, "replies": [] }, { "id": 419887, "author_name": "Kalyan", "author_url": "", "post_date": "2018-11-12T17:46:28.973000", "content": "I could not find anything page 404 is displayed\nthanks for the help
", "votes": 0, "replies": [] }, { "id": 420034, "author_name": "Jianfei Wang", "author_url": "", "post_date": "2018-11-13T00:40:56.757000", "content": "Oh no， you should remove the period in the sentence last.
", "votes": 0, "replies": [] }, { "id": 420428, "author_name": "Kalyan", "author_url": "", "post_date": "2018-11-13T16:02:28.753000", "content": "ooohh my bad got it thankyou
", "votes": 0, "replies": [] } ] }, { "id": 418119, "author_name": "Andrey Lukyanenko", "author_url": "", "post_date": "2018-11-09T10:19:24.937000", "content": "How exactly are we going to predict data up to January 31 2019 if we have test data only up to October 15 2018? Or does his turn into simple time-series problem?
", "votes": 3, "replies": [] }, { "id": 429418, "author_name": "Scirpus", "author_url": "", "post_date": "2018-11-28T20:28:24.273000", "content": "No answer for two days so I'll repost\nHi Phil - the prediction of totals.transactionRevenue is for the December 1st 2018 to January 31st 2019 and not for a May 1st 2018 to January 31st 2019.
", "votes": 1, "replies": [ { "id": 430077, "author_name": "Misha Lisovyi", "author_url": "", "post_date": "2018-11-29T19:39:01.407000", "content": "I believe, this has been answered below, for example here: \"The Private LB is being calculated on the future-looking timeframe of 12/1/18 to 1/31/19 - for those same set of users. \"
", "votes": 1, "replies": [] } ] }, { "id": 425653, "author_name": "Dmitriy Guller", "author_url": "", "post_date": "2018-11-21T22:56:10.580000", "content": "Has anyone managed to parse the \"hits\" column? I don't have much experience with JSON, but I think it's not put together correctly as a JSON column.
", "votes": 1, "replies": [ { "id": 425772, "author_name": "redshellspy", "author_url": "", "post_date": "2018-11-22T04:55:55.107000", "content": "I think it's not json ( I may be wrong). I just used ast.literal_eval to parse hits.
", "votes": 1, "replies": [] }, { "id": 425773, "author_name": "Dmitriy Guller", "author_url": "", "post_date": "2018-11-22T04:58:22.673000", "content": "Is this a python thing?
", "votes": 0, "replies": [] }, { "id": 425872, "author_name": "redshellspy", "author_url": "", "post_date": "2018-11-22T08:25:40.427000", "content": "yeah I use python
", "votes": 0, "replies": [] }, { "id": 425889, "author_name": "Y Assous", "author_url": "", "post_date": "2018-11-22T08:54:52.690000", "content": "Hello Dmitriy,
\n\nThis kernel can help you to flatten the hits column : https://www.kaggle.com/burritodan/gstore-pythonic-flatten-only-hits-column
", "votes": 1, "replies": [] }, { "id": 426832, "author_name": "crownburntdown", "author_url": "", "post_date": "2018-11-24T00:44:29.717000", "content": "My sense is that the hits column is multi-layer encoded (i.e. JSON embedded in JSON): hits-Product, for example. So its extracted form is not 1:1 with the test/train datasets.
\n\nMy process decodes the hits string once, and then calls json_normalize() on each record for each hits sub-column that is embedded in JSON and appends them to a separate data set, where I calc various aggregates by visit before joining back to source data.
", "votes": 0, "replies": [] } ] }, { "id": 418527, "author_name": "Ethan Huang", "author_url": "", "post_date": "2018-11-10T03:24:43.320000", "content": "Since the competition is already relaunched, will the deadline change? Do we still need to submit our result before 30th, November? That's too rush!
", "votes": 1, "replies": [ { "id": 419159, "author_name": "Arye Yerushalmi", "author_url": "", "post_date": "2018-11-11T12:06:28.590000", "content": "Since the evaluation is based on transactions from Dec 1, there is no way to postpone submissions after Nov. 30...
", "votes": 0, "replies": [] } ] }, { "id": 418288, "author_name": "Misha Lisovyi", "author_url": "", "post_date": "2018-11-09T15:43:29.733000", "content": "Small issue: sessionId seems not to be present in the updated data, but the Data tab still mentions it.
Hi , I also cannot find it.
", "votes": 0, "replies": [] }, { "id": 419996, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-11-12T22:50:35.410000", "content": "Hi Misha - thanks for pointing this out! That's correct - sessionId is not included in the current dataset. fullVisitorId is the critical / unique ID in the dataset. sessionId was an ID created for the original dataset when the problem formulation hadn't been set yet, and has been dropped. I'll remove it from the Data tab.
\"[{'index': '4', 'value': 'EMEA'}]\" is not valid JSON because of using single quotes. \n\"There are multiple columns which contain JSON blobs of varying depth\" is not correct. \nYou can find such values in \"customDimensions\" and \"hits\" column.
", "votes": 1, "replies": [ { "id": 418269, "author_name": "Y Assous", "author_url": "", "post_date": "2018-11-09T15:14:44.903000", "content": "I used df = df.replace(to_replace=r'\\'', value='\\\"', regex=True) to make it \"JSON-like\"
", "votes": 3, "replies": [] }, { "id": 418273, "author_name": "Alexander Firsov", "author_url": "", "post_date": "2018-11-09T15:21:16.960000", "content": "Thanks! \nHow do you deal with values like this \"\"YouTube Men's Short Sleeve Hero Tee Black\"\" ?
", "votes": 0, "replies": [] }, { "id": 418275, "author_name": "Y Assous", "author_url": "", "post_date": "2018-11-09T15:23:20.367000", "content": "Haven't seen this value yet (only working on nrows=10000 for now and trying to reduce the size of the dataframe..) in which column is it ?
", "votes": 0, "replies": [] }, { "id": 418277, "author_name": "Alexander Firsov", "author_url": "", "post_date": "2018-11-09T15:27:20.637000", "content": "This is 4th record in train_v2, column \"hits\"
\n\nI found solution. Did not check for all data yet:
\n\nimport ast\ns = \"{'username':'dfdsfdsf'}\"\nast.literal_eval(s)\nNop, this does not work.
", "votes": 1, "replies": [] }, { "id": 418280, "author_name": "Y Assous", "author_url": "", "post_date": "2018-11-09T15:33:33.580000", "content": "I didn't load my csv with this column.. i'm wondering if the \"hit\" column will be in the future-looking dataset, what do you think ?
", "votes": 0, "replies": [] }, { "id": 418282, "author_name": "Alexander Firsov", "author_url": "", "post_date": "2018-11-09T15:35:54.853000", "content": "There will not be any other dataset, only train_v2 and test_v2. We should predict revenue for visitors from test_v2 using only this data.
", "votes": 0, "replies": [] }, { "id": 418285, "author_name": "Y Assous", "author_url": "", "post_date": "2018-11-09T15:38:26.923000", "content": "When I was talking about the \"future-looking dataset\" I meant the dataset used to score our submissions in the final stand :)
", "votes": 0, "replies": [] }, { "id": 418292, "author_name": "", "author_url": "", "post_date": "2018-11-09T15:50:43.777000", "content": "", "votes": 0, "replies": [] }, { "id": 418296, "author_name": "Alexander Firsov", "author_url": "", "post_date": "2018-11-09T15:54:23.893000", "content": "Then I still do not understand what you meant :) We will do prediction for future. \nDataset to score is just visitorId and revenue for time range December 1st 2018 - January 31st 2019.
", "votes": 0, "replies": [] }, { "id": 418299, "author_name": "Y Assous", "author_url": "", "post_date": "2018-11-09T16:00:05.623000", "content": "You are right, my bad, I'm confused with all those changes in the competition :) \nSo i'll put hits in my dataset again and see how to handle the \"\"YouTube Men's Short Sleeve Hero Tee Black\"\"
", "votes": 0, "replies": [] }, { "id": 418306, "author_name": "Alexander Firsov", "author_url": "", "post_date": "2018-11-09T16:13:42.020000", "content": "I believe the main problem of 'hits' column is not invalid JSON, but the fact that the columns contains information about all hits within this session. \nFor example, first record from train contains single hit, so 1-1 relation between session ans hits.\nSecond has two hits, so the session has multiple hit objects, which is hard to convert to flat table.
\n\nSolution is to have separate table for hits, but we should invent a key since session_id is not unique (or maybe this is ok)
", "votes": 0, "replies": [] }, { "id": 418315, "author_name": "Pratik Mangesh Sonar", "author_url": "", "post_date": "2018-11-09T16:29:47.520000", "content": "Reposting deleted....\nCheck cell 8 in this public kernel https://www.kaggle.com/sudalairajkumar/simple-exploration-baseline-ga-customer-revenue ... It shows some columns having constant values. Maybe excluding them might save some memory. Also check https://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71048#418308 . According to vincent, dropping 'hits' would bring train_v2.csv to 800 MB.
", "votes": 0, "replies": [] }, { "id": 418320, "author_name": "Alexander Firsov", "author_url": "", "post_date": "2018-11-09T16:36:54.740000", "content": "Thanks! But I do not have problems with memory so far :)\nFirst kernel you mentioned is based on old data, be careful.
", "votes": 0, "replies": [] }, { "id": 418324, "author_name": "Pratik Mangesh Sonar", "author_url": "", "post_date": "2018-11-09T16:42:06.447000", "content": "Yes. But assuming it would have not changed, it would be a good option to try skip them. Also, are you running your notebook on local machine or on kaggle??? Because I am not even able to read train_v2.csv ...\nAlso, as you don't have memory problem, can you please tell if those columns have constant value in new dataset or not???
", "votes": 0, "replies": [] }, { "id": 418329, "author_name": "", "author_url": "", "post_date": "2018-11-09T16:55:26.310000", "content": "", "votes": 0, "replies": [] }, { "id": 418370, "author_name": "Alexander Firsov", "author_url": "", "post_date": "2018-11-09T18:40:39.387000", "content": "Pratik, I do not have memory issues because I still have no idea how to read 'hits' and 'customDimensions' columns. It is not a problem to read all data excluding these columns.\nYou can safely drop the following fields because they are constant in new data sets:
\n\n'socialEngagementType', 'device.browserSize', 'device.browserVersion', 'device.flashVersion', 'device.language', 'device.mobileDeviceBranding', 'device.mobileDeviceInfo', 'device.mobileDeviceMarketingName', 'device.mobileDeviceModel', 'device.mobileInputSelector', 'device.operatingSystemVersion', 'device.screenColors', 'device.screenResolution', 'geoNetwork.cityId', 'geoNetwork.latitude', 'geoNetwork.longitude', 'geoNetwork.networkLocation', 'totals.visits', 'trafficSource.adwordsClickInfo.criteriaParameters'
\n\nThe problem is that many of these fields are part of json blobs, therefore you need read json fields completely, convert to flat table and only on this stage remove constant columns. in other words, knowing that some json fields are not needed cannot be used to minimize memory usage while reading.
", "votes": 2, "replies": [] }, { "id": 418498, "author_name": "Pratik Mangesh Sonar", "author_url": "", "post_date": "2018-11-10T01:41:32.350000", "content": "Yeah right.
", "votes": 0, "replies": [] } ] }, { "id": 430729, "author_name": "Prasen Jit Singh", "author_url": "", "post_date": "2018-11-30T22:29:17.737000", "content": "hi it says 3 months to go was the deadline changed
", "votes": 2, "replies": [ { "id": 431112, "author_name": "Arye Yerushalmi", "author_url": "", "post_date": "2018-12-01T16:18:47.217000", "content": "That's because the results would be published on 15.2.19 on data until 31.1.19. Submission is of course closed, as we are in the test period.
", "votes": 1, "replies": [] } ] }, { "id": 422567, "author_name": "Burrito Dan", "author_url": "", "post_date": "2018-11-16T12:19:45.313000", "content": "@philculliton please can you confirm how we submit to the private and public leaderboards. Usually the two leaderboards are disjoint subsets of the test set. In this case, the predictions for the public leaderboard are a different timeframe from the predictions for the private leaderboard.
\n\nIf I make a prediction using the future prediction (Dec-Jan) it will score horribly on the public leaderboard, which is a past prediction for the (May-15/Oct) timeframe. But I guess that is what you want? It's a bit counterintuitive!
", "votes": 2, "replies": [ { "id": 422636, "author_name": "Paulo Pinto", "author_url": "", "post_date": "2018-11-16T14:41:39.770000", "content": "That's it. You should rely only on your validation when it comes to real data forecasts.
", "votes": 3, "replies": [] }, { "id": 422799, "author_name": "Julia Elliott", "author_url": "", "post_date": "2018-11-16T19:44:01.237000", "content": "In this competition, the sample_submission_v2 file on the data page is composed of all fullVisitorIds for the 5/1/18 to 10/15/18 time period. The public leaderboard submission for this file is based on a separate timeframe than the private leaderboard.
As specified by Phil, above, and on the data page, the Public LB is being calculated for those visitors during the same timeframe of 5/1/18 to 10/15/18. This is all publicly-available for the target prediction, but intended to provide some means of signal for those who wish to use it in that way. The Private LB is being calculated on the future-looking timeframe of 12/1/18 to 1/31/19 - for those same set of users.
\n\nTherefore, your submission that is intended for the public LB timeframe will be different from the private LB timeframe, which will be rescored/recalculated on the future timeframe.
\n\nKnowing this, I expect that competitors will need to make explicit final submission selections for those submissions which represent what they expect those future-looking predictions to be. These final submission selections for which this competition permits 2, per the rules, can be made under \"My Submissions\" by the final submission deadline.
", "votes": 2, "replies": [] }, { "id": 422806, "author_name": "redshellspy", "author_url": "", "post_date": "2018-11-16T19:50:29.303000", "content": "Can you please confirm about the target. Which is the correct one totals.transactionRevenue(this is given in data page) or totals.TotalTransactionRevenue ?
", "votes": 3, "replies": [] }, { "id": 422808, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-11-16T19:52:12.143000", "content": "Hi Suchith - the correct target is totals.transactionRevenue.
Phil, can you please respond to my post here:\nhttps://www.kaggle.com/c/ga-customer-revenue-prediction/discussion/71779
\n\nI believe I have evidence against totals.transactionRevenue (and in favour of totals.totalTransactionRevenue) to be a meaningful metric to analyse.\nThank you!
", "votes": 1, "replies": [] }, { "id": 424073, "author_name": "Mikhail Zolotko", "author_url": "", "post_date": "2018-11-19T14:13:50.440000", "content": "@philculliton, I would still like to hear from you regarding the meaningfulness of totals.transactionRevenue. In the post above I provided the link to the post where I laid out my arguments. And judging by the upvotes, they are not complete nonsense. \nThank you a lot in advance!
", "votes": 1, "replies": [] }, { "id": 424183, "author_name": "Phil Culliton", "author_url": "", "post_date": "2018-11-19T18:00:25.817000", "content": "Hey @antoniorossi - thanks for the heads up! I'm looking into it now.
", "votes": 0, "replies": [] } ] }, { "id": 418098, "author_name": "Ramona Fli", "author_url": "", "post_date": "2018-11-09T09:31:22.460000", "content": "\"Note that this means the public leaderboard is calculated on all users in test_v2.csv for their transactions between May 1st 2018 to October 15th 2018. Your true performance and final standing will be based on the private future-looking dataset\" AND \n\"The train set and test set both contain all relevant data for all transactions in the data's time period, including the totals.transactionRevenue\"
\n\nDoes it mean that someone can just copy the transaction revenue from the test set, make the sum of logs, post it and get the perfect model?
", "votes": 2, "replies": [ { "id": 418130, "author_name": "Burrito Dan", "author_url": "", "post_date": "2018-11-09T10:50:03.250000", "content": "yep, but i suspect no one will be impressed!
", "votes": 3, "replies": [] }, { "id": 423304, "author_name": "merazga7", "author_url": "", "post_date": "2018-11-17T22:03:10.483000", "content": "I did it just as test but i didn't get the perfect score.
", "votes": 0, "replies": [] }, { "id": 423418, "author_name": "Pratik Mangesh Sonar", "author_url": "", "post_date": "2018-11-18T07:52:51.303000", "content": "\n\n\nDoes it mean that someone can just copy the transaction revenue from the test set, make the sum of logs, post it and get the perfect model?
\n
It is log of sum...
", "votes": 0, "replies": [] } ] }, { "id": 430902, "author_name": "Arunkumar Venkataramanan", "author_url": "", "post_date": "2018-12-01T07:05:28.327000", "content": "Hello Phil, when will the private leaderboard get reflected with the final score for this competition, since I see it says 2 months go.
", "votes": 0, "replies": [ { "id": 431109, "author_name": "Arye Yerushalmi", "author_url": "", "post_date": "2018-12-01T16:17:08.730000", "content": "I'm the timline section it says 15.2.19
", "votes": 0, "replies": [] } ] }, { "id": 419048, "author_name": "JunLuDing", "author_url": "", "post_date": "2018-11-11T06:21:33.243000", "content": "Why Error while uploading
", "votes": -2, "replies": [] }, { "id": 419014, "author_name": "JunLuDing", "author_url": "", "post_date": "2018-11-11T04:51:13.633000", "content": "what happened？ I can't submit！
", "votes": -2, "replies": [] }, { "id": 429195, "author_name": "Xiaotong Lu", "author_url": "", "post_date": "2018-11-28T13:55:31.723000", "content": "Hi.\nThis is my first Kaggle project. I want to know my score. But it wasn't until today did I found out that the rules were completely changed. I just finished predicting the original dataset but all my efforts samed to be in vain. I could not submit my answer because the rows didn't match (sure, due to different data sets.) .\nWhat shell I do? How do I get a score? Or can I compare my prediction to the \"leaked\" test data myself? Where can I find it?
", "votes": 0, "replies": [ { "id": 430205, "author_name": "yangDDD", "author_url": "", "post_date": "2018-11-30T02:38:36.630000", "content": "the column totals. totals.transactionRevenue is the target
", "votes": 0, "replies": [] } ] }, { "id": 428272, "author_name": "ducvu", "author_url": "", "post_date": "2018-11-27T02:14:56.177000", "content": "I see the totalTransactionRevenue in feild totals in file *test_v2.csv*. Does file *test_v2.csv* have something wrong. Does anyone can explain for me?
", "votes": 0, "replies": [ { "id": 430442, "author_name": "Arye Yerushalmi", "author_url": "", "post_date": "2018-11-30T11:44:46.070000", "content": "As explained in the topic, the testv2 file is actually another train file. The real test will be the transactionRevenue during 1/12/18 - 31/1/19 of the ID appearing in test2 (including those who won't even enter the google store in that period), and that's what you are to predict.
", "votes": 0, "replies": [] } ] }, { "id": 428045, "author_name": "", "author_url": "", "post_date": "2018-11-26T16:54:12.487000", "content": "", "votes": 0, "replies": [] }, { "id": 427083, "author_name": "", "author_url": "", "post_date": "2018-11-24T14:17:27.483000", "content": "", "votes": 0, "replies": [ { "id": 427133, "author_name": "", "author_url": "", "post_date": "2018-11-24T17:35:31.763000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 423949, "author_name": "", "author_url": "", "post_date": "2018-11-19T10:10:27.887000", "content": "", "votes": 0, "replies": [] }, { "id": 422648, "author_name": "", "author_url": "", "post_date": "2018-11-16T15:10:47.227000", "content": "", "votes": 0, "replies": [ { "id": 422807, "author_name": "", "author_url": "", "post_date": "2018-11-16T19:51:36.050000", "content": "", "votes": 1, "replies": [] }, { "id": 422809, "author_name": "", "author_url": "", "post_date": "2018-11-16T19:58:18.490000", "content": "", "votes": 1, "replies": [] }, { "id": 422882, "author_name": "", "author_url": "", "post_date": "2018-11-16T23:56:43.220000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 422140, "author_name": "", "author_url": "", "post_date": "2018-11-15T20:42:39.153000", "content": "", "votes": 0, "replies": [] }, { "id": 421834, "author_name": "", "author_url": "", "post_date": "2018-11-15T13:28:49.483000", "content": "", "votes": 0, "replies": [ { "id": 422046, "author_name": "", "author_url": "", "post_date": "2018-11-15T18:07:12.320000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 421735, "author_name": "", "author_url": "", "post_date": "2018-11-15T10:49:22.230000", "content": "", "votes": 0, "replies": [ { "id": 422800, "author_name": "", "author_url": "", "post_date": "2018-11-16T19:46:15.087000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 421600, "author_name": "", "author_url": "", "post_date": "2018-11-15T07:08:42.230000", "content": "", "votes": 0, "replies": [ { "id": 421636, "author_name": "", "author_url": "", "post_date": "2018-11-15T08:13:49.670000", "content": "", "votes": 0, "replies": [] }, { "id": 421884, "author_name": "", "author_url": "", "post_date": "2018-11-15T14:34:10.060000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 420166, "author_name": "", "author_url": "", "post_date": "2018-11-13T07:16:46.947000", "content": "", "votes": 0, "replies": [ { "id": 420234, "author_name": "", "author_url": "", "post_date": "2018-11-13T10:21:04.737000", "content": "", "votes": 1, "replies": [] }, { "id": 420294, "author_name": "", "author_url": "", "post_date": "2018-11-13T12:29:07.560000", "content": "", "votes": 0, "replies": [] }, { "id": 420308, "author_name": "", "author_url": "", "post_date": "2018-11-13T12:46:44.660000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 420092, "author_name": "", "author_url": "", "post_date": "2018-11-13T03:33:14.163000", "content": "", "votes": 0, "replies": [] }, { "id": 419339, "author_name": "", "author_url": "", "post_date": "2018-11-11T18:39:31.230000", "content": "", "votes": 0, "replies": [] }, { "id": 419179, "author_name": "", "author_url": "", "post_date": "2018-11-11T12:43:49.477000", "content": "", "votes": 0, "replies": [ { "id": 419271, "author_name": "", "author_url": "", "post_date": "2018-11-11T16:03:47.310000", "content": "", "votes": 0, "replies": [] }, { "id": 419350, "author_name": "", "author_url": "", "post_date": "2018-11-11T19:13:51.867000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 418616, "author_name": "", "author_url": "", "post_date": "2018-11-10T09:17:20.077000", "content": "", "votes": 0, "replies": [ { "id": 420000, "author_name": "", "author_url": "", "post_date": "2018-11-12T22:53:01.713000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 418030, "author_name": "", "author_url": "", "post_date": "2018-11-09T06:22:43.920000", "content": "", "votes": 0, "replies": [ { "id": 418109, "author_name": "", "author_url": "", "post_date": "2018-11-09T09:42:28.673000", "content": "", "votes": 0, "replies": [] }, { "id": 418145, "author_name": "", "author_url": "", "post_date": "2018-11-09T11:15:09.527000", "content": "", "votes": 2, "replies": [] }, { "id": 418460, "author_name": "", "author_url": "", "post_date": "2018-11-09T23:34:59.157000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 417968, "author_name": "", "author_url": "", "post_date": "2018-11-09T04:10:34.007000", "content": "", "votes": 0, "replies": [ { "id": 418204, "author_name": "", "author_url": "", "post_date": "2018-11-09T13:27:48.197000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 430736, "author_name": "", "author_url": "", "post_date": "2018-11-30T22:37:27.673000", "content": "", "votes": 0, "replies": [] }, { "id": 426838, "author_name": "", "author_url": "", "post_date": "2018-11-24T01:02:12.860000", "content": "", "votes": 0, "replies": [] }, { "id": 418782, "author_name": "", "author_url": "", "post_date": "2018-11-10T15:46:48.083000", "content": "", "votes": 0, "replies": [] }, { "id": 418389, "author_name": "", "author_url": "", "post_date": "2018-11-09T20:29:04.077000", "content": "", "votes": 5, "replies": [ { "id": 419997, "author_name": "", "author_url": "", "post_date": "2018-11-12T22:51:22.210000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 418189, "author_name": "", "author_url": "", "post_date": "2018-11-09T12:56:28.157000", "content": "", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "417914": "**The data is now ready to be downloaded.**\n\nWe have released a new dataset for Google Analytics Customer Revenue Prediction! This is intended to aid competitors in predicting future data (December 1st 2018 - January 31st 2019). We will be predicting the log of the sum of user revenue for this future time period for *all users represented in the posted test set: `test_v2.csv`, which includes all transactions from May 1st 2018 to October 15th 2018*. \n\nWe have included columns of the dataset that were previously excluded, in order to match the full set of data that’s available from the demo account. The train set and test set both contain ***all*** relevant data for all transactions in the data's time period, including the `totals.transactionRevenue`. Note that this means the public leaderboard is calculated on all users in `test_v2.csv` for their transactions between May 1st 2018 to October 15th 2018. Your true performance and final standing will be based on the private future-looking dataset, which will also be calculated for all users in `test_v2.csv`, but for the time period December 1st 2018 to January 31st 2019. As such, we recommend that you use caution when interpreting the results of the public leaderboard.\n\nWe are aware that we are missing the most recent data, from the weeks between October 16th 2018 and November 30th 2018. Unfortunately the process of preparing this data with many different groups involved is extremely heavy; there was no way to deliver this newer data while also providing enough time for competitors to work with it. I apologize in advance for this lack of recency, but you are still permitted to use it as external data if you extract it from the demo account.\n\nPlease feel free to ask questions here.\n\nGood luck!\n\nNote: Edited to clarify that ALL users in test_v2.csv should have predictions generated for them, and final scores will be based on all predictions.", "418394": "Can you clear the existing submissions? \nSome people might think about \"quitting\" the competition and might want their past submissions which are now more or less useless (in regard to the private leaderboard) not to be counted. ", "418261": "Hi @philculliton, can you please clear one thing for us? What if we will submit every user as if he will return? Will you calculate score only for valid users or there will be some kind of penalty for incorrect predictions?\nThanks in advance", "419152": "So the rules enterely change it doesn't look like a big change but it does affect all of our models and methodology, first off all we have to make predictions based on information that doesn't even exist, if you are using features with information of the previous weeks or time series then you need to make predictions for the hole period that is missing ( all the time from October to the end of November). We don't have the time or the resources to do this, also we don't have information from the prevoious year so whatever the predictions are they'll be really random so it doesn't really matter how \"good\" our models are if we don't have the information or even a board for guidance. I just think that all the efford and time that we invested in this challenge was for nothing.", "418114": "**Data got too. large.** Trying to read csv, but notebook **restarts** after some minutes without yielding any output.\n", "425819": "I get it now, this column looks like how you would hard-code a list in Python. It seems like the way to input that column is to make Python execute it as a code.\n\nIt would've been helpful to explain how to parse that column, especially since it looks like a badly prepared JSON format data, and not everyone is familiar enough with Python to recognize it as a Python code. This also seems like a sloppy and super dangerous way to encode the data, so maybe the reluctance to document it is not that surprising.\n\nI'm starting to think that this competition should've been euthanized instead of being halfheartedly dragged along to the finish line.", "418393": "Maybe it would be better to upload .zip archives instead of plain .csv files? \n\nThat would reduce file sizes a lot, making them much easier/faster to download.", "420332": "who knows why we have two fields : \"totals.transactionRevenue\" and \"totals.totalTransactionRevenue\" ?\nand which of them must be TARGET ?", "418149": "It sounds little bit strange for me that Google already has a data of future transactions to test models performance... ", "420473": "Hi all - I wanted to clarify two things. I apologize for the confusion, and thanks to everyone who has asked questions and provided feedback. We are doing our level best to create something interesting, and your responses have been incredibly helpful. We really appreciate your patience.\n\nFirst, I want to confirm that this is a future prediction problem, and you will need to make predictions on unseen data (December 2018-January 2019) for all users that visited the store between May 1st 2018 and October 15th 2018. These are represented by the set of `fullVisitorId`s in `sample_submission_v2.csv`.\n\nSecond, I want to confirm that predictions for *all* users from that time period will influence the score calculation. The objective of \"figure out how much a small number of users might spend, ignoring all others\" is extremely narrow and risks being broken in a number of ways. \"Figure out which users will spend again, and how much\" is much more robust, and much more likely to produce interesting / useful models and results.\n\nThanks again for your feedback!", "420375": "Dear @philculliton\nI was working on this challenge in order to write a nice code and showcase it for my job hunt which turned out to be not the best choice. I would like to start with another competition and would rather not continue with the new datasets here. I just would like to know if it is possible that I can have access to the premier datasets (train.csv and test.csv) in order to be able to share my work. Because now I cannot load them anymore. Could you help me with that?\n\nMany thanks", "418868": "The leaderboard is really frustrating as one cannot see how well his solution did compared to other competitors until the very end.", "418177": "Hi all, \nSo what I understanding about this competition is , public score has nothing to do with final standing right? What we can trust is our cross-validating score right?", "420939": "Actually reset the previous leaderboard is unfair. Especially submissions team was changed rapidly from more than 3000 to 300. because it will influence the final ranking. Even if the question changed, it's not their fault.", "419349": "Dear @philculliton,\n\nCould you help us to confirm one thing about the evaluation? Is it true that: \n- The predictions for customers who do not present between December 1st 2018 to January 31st 2019 will not influence the score calculation\n\nAs this will decide how should we implment our own evaluation method.\n\nThanks\n", "418478": "please help me as i am not able to load the new data. it keep on saying The kernel appears to have died. It will restart automatically.\nwhat is the problem \nplease help \nthanks in advance", "418119": "How exactly are we going to predict data up to January 31 2019 if we have test data only up to October 15 2018? Or does his turn into simple time-series problem?", "429418": "No answer for two days so I'll repost\nHi Phil - the prediction of totals.transactionRevenue is for the December 1st 2018 to January 31st 2019 and not for a May 1st 2018 to January 31st 2019.", "425653": "Has anyone managed to parse the \"hits\" column? I don't have much experience with JSON, but I think it's not put together correctly as a JSON column.", "418527": "Since the competition is already relaunched, will the deadline change? Do we still need to submit our result before 30th, November? That's too rush!", "418288": "Small issue: `sessionId` seems not to be present in the updated data, but the Data tab still mentions it.", "418267": "\"[{'index': '4', 'value': 'EMEA'}]\" is not valid JSON because of using single quotes. \n\"There are multiple columns which contain JSON blobs of varying depth\" is not correct. \nYou can find such values in \"customDimensions\" and \"hits\" column.", "430729": "hi it says 3 months to go was the deadline changed\n", "422567": "@philculliton please can you confirm how we submit to the private and public leaderboards. Usually the two leaderboards are disjoint subsets of the test set. In this case, the predictions for the public leaderboard are a different timeframe from the predictions for the private leaderboard. \n\nIf I make a prediction using the future prediction (Dec-Jan) it will score horribly on the public leaderboard, which is a past prediction for the (May-15/Oct) timeframe. But I guess that is what you want? It's a bit counterintuitive!", "418098": "\"Note that this means the public leaderboard is calculated on all users in test_v2.csv for their transactions between May 1st 2018 to October 15th 2018. Your true performance and final standing will be based on the private future-looking dataset\" AND \n\"The train set and test set both contain all relevant data for all transactions in the data's time period, including the totals.transactionRevenue\"\n\nDoes it mean that someone can just copy the transaction revenue from the test set, make the sum of logs, post it and get the perfect model? ", "430902": "Hello Phil, when will the private leaderboard get reflected with the final score for this competition, since I see it says 2 months go.", "419048": "Why Error while uploading", "419014": "what happened？ I can't submit！", "429195": "Hi.\nThis is my first Kaggle project. I want to know my score. But it wasn't until today did I found out that the rules were completely changed. I just finished predicting the original dataset but all my efforts samed to be in vain. I could not submit my answer because the rows didn't match (sure, due to different data sets.) .\nWhat shell I do? How do I get a score? Or can I compare my prediction to the \"leaked\" test data myself? Where can I find it?", "428272": "I see the *totalTransactionRevenue* in feild *totals* in file *test_v2.csv*. Does file *test_v2.csv* have something wrong. Does anyone can explain for me?", "428045": "Hi Phil - the prediction of totals.transactionRevenue is for the December 1st 2018 to January 31st 2019 and not for a May 1st 2018 to January 31st 2019.", "427083": "@phil Are we predicting the log of revenue or Log1p of revenue? The target is Log1p, but the competition rules imply log for the prediction. It could make a big differences if you think the probability (ie expected spend) is close to zero.", "423949": "From October 15th onwards: Traffic on the site has been stable, but purchase activity has dropped drastically, according to the GA demo account reports. It actually looks like there is a different reality now when it comes to users being able to complete transactions. ", "422648": "@philculliton\nThis question may be already asked somewhere, but I want to make things clear about **the duration of the data**. \nBefore the submission deadline, November 30th 2018, we can get the train and test data like this: \n- Train data(train_v2): *Aug 1st 2016 to Apr 30th 2018* \n- Test data(test_v2): *May 1st 2018 to October 15th 2018* \nThen, after the submission deadline, we cannot re-run our scripts. \n\nAnd, **here is my question**: after Feb 1st 2019, will the datasets change like this (and scripts re-run)?: \n- Train data: *Aug 1st 2016 to October 15th 2018* \n- Test data: *December 1st 2018 to January 31st 2019* \n\nI have not ever join the 2 stage competition, so it confuses me a lot. \nThanks if you help me. ", "422140": "\nHello,\nbefore your change, I was very well ranked and now I do not want to participate in the contests anymore.\nI do not find it professional to change the rules and leave us little time.\nHow can I unsubscribe?", "421834": "I find all of my submissions are marked with error! Can anyone explain to me why does this happen !", "421735": "@philculliton will the Private LB live (updated daily or weekly) ? Or are we going to wait until February to see how well our predictions did?", "421600": "The fields \"totals_totalTransactionRevenue\" contains information about target………..\nWhat am I predicting? \"totals_totalTransactionRevenue\" or \"totals_transactionRevenue\"?\nis \"totals_transactionRevenue\" I think", "420166": "If a visitor returns a few times during the prediction period, are we to predict the sum of his transactions, the mean, the median or just for the first visit?", "420092": "I can't find transationRevenue in column totals of train dataset", "419339": "So, will be there the test_v3.csv file with future timeframe of December 1st 2018 to January 31st 2019? Because otherwise I do not see any point in creating ML model with current features. I could just use random number generator.", "419179": "I might be missing a HUGE thing here, but for what I understand, in the most fundamental level, we are to predict targets without related features. Though there is some thinking to do here, it doesn't look like a DS / ML competition to me.", "418616": "Hi Kaggle and all,\nCan I confirm if there is no sessionId in train_v2 and test_v2 ?? ", "418030": "We're working on an issue with uploading the data - it should be ready tomorrow!", "417968": "Hello Phil\nData page says that you need to download test_v2 and train_v2. But I can't see any files named train v2 and test v2. \nOr we have to download train.csv and test.csv files ?", "430736": "", "426838": "", "418782": "", "418389": "", "418189": "" } }