For those who don't want to read it all through: I am claiming the following:
\n\ntransactionRevenue is a wrong thing to look at\nit's totalTransactionRevenue that makes sense to analyse and to predict.
\n\nHowever Kaggle wants us to analyse the former.
\n\nSince it is now possible to submit the known values of the predicted variable for the test period and so get a perfect score, it is obvious that what Kaggle wants us to predict is really transactionRevenue and not totalTransactionRevenue. If you submit transactionRevenue, you will score 0, if you submit totalTransactionRevenue, only 0.03.\n This is actually what it quite clearly says in the Data section, paragraph \"What should I expect the data format to be?\".
\n\nThe thing is that we also have some revenue data buried deeply inside the infamous Hits field. I was doing some consistency checks between these different views on the revenue and it turned out that (as you probably know) in the Hits field we have json datasets for each hit, which includes some revenue figures for a transaction if such a hit resulted in a transaction.
\n\nNow, based on at least three examples I can conclude that the field transactionRevenue corresponds only to the revenue resulting from one certain transaction during a visit, whereas totalTransactionRevenue represents the whole revenue resulting from the whole visit. That is why any aggregations of transactionRevenue on the user level represent some (random) fractions of the aggregate user revenue over a certain period of time, and if this is really the case, this number is unfortunately fairly difficult to model and senseless to predict.
\n\nHere are the examples. All come from the train_v2.csv, the index is the row range index (zero-based, python style!)
\n\n1) row 4536: \ntotals: 'totalTransactionRevenue': '2103690000', 'transactionRevenue': '546320000'\nhits: two available values for transactionRevenue are 1557370000 and 548320000 \nWe see that the second value from hits corresponds to the transactionRevenue and their sum (2105690000) corresponds to the totalTransactionRevenue. Why they are not precisely equal is another question, but this is apparently not because of 'transactionShipping' or 'transactionTax', two other fields also available in the hits jsons. I've tried adding and subtracting them, to no avail. Expressed in dollars this deviation is in fact just $2.
\n\n2) row 4639: \ntotals: 'totalTransactionRevenue': '672960000', 'transactionRevenue': '338970000',\nhits: two available values for transactionRevenue are 333990000 and 341970000\nAgain, the second hits transactionRevenue corresponds to transactionRevenue from totals, the sum (675960000) corresponds to totals totalTransactionRevenue. Again, they do not match exactly, the deviation this time is $3.
\n\n3) row 9761: \ntotals: 'totalTransactionRevenue': '1750210000', 'transactionRevenue': '1077000000'\nhits: two available values for transactionRevenue are 673210000 and 1079000000. \nAgain, the second hits value corresponds to transactionRevenue and their sum (1752210000) corresponds to totalTransactionRevenue. Deviation = $2.
\n\nIs there any reason why only those certain transactions were included in transactionRevenue and certain other transactions were disregarded? It doesn't seem like this. For many other metrics we have numbers for the entire session (subfields of total: timeOnSite, page views, hits, even the number of transactions in the transactions field.)
\n\nOf course, most users conduct just one transaction per visit (if any at all), in this case there is no difference between transactionRevenue and totalTransactionRevenue. But I assumed it is precisely the users who spend a little more is whose behaviour we are really interested in.
\n\nAnother thing: of course the difference between transactionRevenue and totalTransactionRevenue is further reduced by taking logarithms. This also leads to only a minor difference in the scores (0 vs. 0.03). But is it actually a log of revenue that matters in reality? I don’t remember any log revenues or profits in companies’ financial statements. So I would prefer if the RMSE were calculated based on raw (i.e. not logarithmised) revenue numbers, and of course total and not partial ones.
\n\nAs a conclusion, probably my strategy won’t change much, but if the score were calculated the way I suggest, it could have significant implications for the final standings.
", "messages": [ { "id": "422579", "postDate": "11/16/2018 12:47:05", "content": "For those who don't want to read it all through: I am claiming the following:
\n\ntransactionRevenue is a wrong thing to look at\nit's totalTransactionRevenue that makes sense to analyse and to predict.
\n\nHowever Kaggle wants us to analyse the former.
\n\nSince it is now possible to submit the known values of the predicted variable for the test period and so get a perfect score, it is obvious that what Kaggle wants us to predict is really transactionRevenue and not totalTransactionRevenue. If you submit transactionRevenue, you will score 0, if you submit totalTransactionRevenue, only 0.03.\n This is actually what it quite clearly says in the Data section, paragraph \"What should I expect the data format to be?\".
\n\nThe thing is that we also have some revenue data buried deeply inside the infamous Hits field. I was doing some consistency checks between these different views on the revenue and it turned out that (as you probably know) in the Hits field we have json datasets for each hit, which includes some revenue figures for a transaction if such a hit resulted in a transaction.
\n\nNow, based on at least three examples I can conclude that the field transactionRevenue corresponds only to the revenue resulting from one certain transaction during a visit, whereas totalTransactionRevenue represents the whole revenue resulting from the whole visit. That is why any aggregations of transactionRevenue on the user level represent some (random) fractions of the aggregate user revenue over a certain period of time, and if this is really the case, this number is unfortunately fairly difficult to model and senseless to predict.
\n\nHere are the examples. All come from the train_v2.csv, the index is the row range index (zero-based, python style!)
\n\n1) row 4536: \ntotals: 'totalTransactionRevenue': '2103690000', 'transactionRevenue': '546320000'\nhits: two available values for transactionRevenue are 1557370000 and 548320000 \nWe see that the second value from hits corresponds to the transactionRevenue and their sum (2105690000) corresponds to the totalTransactionRevenue. Why they are not precisely equal is another question, but this is apparently not because of 'transactionShipping' or 'transactionTax', two other fields also available in the hits jsons. I've tried adding and subtracting them, to no avail. Expressed in dollars this deviation is in fact just $2.
\n\n2) row 4639: \ntotals: 'totalTransactionRevenue': '672960000', 'transactionRevenue': '338970000',\nhits: two available values for transactionRevenue are 333990000 and 341970000\nAgain, the second hits transactionRevenue corresponds to transactionRevenue from totals, the sum (675960000) corresponds to totals totalTransactionRevenue. Again, they do not match exactly, the deviation this time is $3.
\n\n3) row 9761: \ntotals: 'totalTransactionRevenue': '1750210000', 'transactionRevenue': '1077000000'\nhits: two available values for transactionRevenue are 673210000 and 1079000000. \nAgain, the second hits value corresponds to transactionRevenue and their sum (1752210000) corresponds to totalTransactionRevenue. Deviation = $2.
\n\nIs there any reason why only those certain transactions were included in transactionRevenue and certain other transactions were disregarded? It doesn't seem like this. For many other metrics we have numbers for the entire session (subfields of total: timeOnSite, page views, hits, even the number of transactions in the transactions field.)
\n\nOf course, most users conduct just one transaction per visit (if any at all), in this case there is no difference between transactionRevenue and totalTransactionRevenue. But I assumed it is precisely the users who spend a little more is whose behaviour we are really interested in.
\n\nAnother thing: of course the difference between transactionRevenue and totalTransactionRevenue is further reduced by taking logarithms. This also leads to only a minor difference in the scores (0 vs. 0.03). But is it actually a log of revenue that matters in reality? I don’t remember any log revenues or profits in companies’ financial statements. So I would prefer if the RMSE were calculated based on raw (i.e. not logarithmised) revenue numbers, and of course total and not partial ones.
\n\nAs a conclusion, probably my strategy won’t change much, but if the score were calculated the way I suggest, it could have significant implications for the final standings.
", "rawMarkdown": "For those who don't want to read it all through: I am claiming the following:\n\ntransactionRevenue is a wrong thing to look at\nit's **totalTransactionRevenue** that makes sense to analyse and to predict.\n\nHowever Kaggle wants us to analyse the former.\n\n****************************************************************\n\n\n\nSince it is now possible to submit the known values of the predicted variable for the test period and so get a perfect score, it is obvious that what Kaggle wants us to predict is really transactionRevenue and not totalTransactionRevenue. If you submit transactionRevenue, you will score 0, if you submit totalTransactionRevenue, only 0.03.\n This is actually what it quite clearly says in the Data section, paragraph \"What should I expect the data format to be?\".\n\nThe thing is that we also have some revenue data buried deeply inside the infamous Hits field. I was doing some consistency checks between these different views on the revenue and it turned out that (as you probably know) in the Hits field we have json datasets for each hit, which includes some revenue figures for a transaction if such a hit resulted in a transaction.\n\nNow, based on at least three examples I can conclude that the field transactionRevenue corresponds only to the revenue resulting from one certain transaction during a visit, whereas **totalTransactionRevenue** represents the whole revenue resulting from the whole visit. That is why any aggregations of transactionRevenue on the user level represent some (random) fractions of the aggregate user revenue over a certain period of time, and if this is really the case, this number is unfortunately fairly difficult to model and senseless to predict.\n\nHere are the examples. All come from the train_v2.csv, the index is the row range index (zero-based, python style!)\n\n1) row 4536: \ntotals: 'totalTransactionRevenue': '2103690000', 'transactionRevenue': '546320000'\nhits: two available values for transactionRevenue are 1557370000 and 548320000 \nWe see that the second value from hits corresponds to the transactionRevenue and their sum (2105690000) corresponds to the totalTransactionRevenue. Why they are not precisely equal is another question, but this is apparently not because of 'transactionShipping' or 'transactionTax', two other fields also available in the hits jsons. I've tried adding and subtracting them, to no avail. Expressed in dollars this deviation is in fact just $2.\n\n2) row 4639: \ntotals: 'totalTransactionRevenue': '672960000', 'transactionRevenue': '338970000',\nhits: two available values for transactionRevenue are 333990000 and 341970000\nAgain, the second hits transactionRevenue corresponds to transactionRevenue from totals, the sum (675960000) corresponds to totals totalTransactionRevenue. Again, they do not match exactly, the deviation this time is $3.\n\n3) row 9761: \ntotals: 'totalTransactionRevenue': '1750210000', 'transactionRevenue': '1077000000'\nhits: two available values for transactionRevenue are 673210000 and 1079000000. \nAgain, the second hits value corresponds to transactionRevenue and their sum (1752210000) corresponds to totalTransactionRevenue. Deviation = $2.\n\nIs there any reason why only those certain transactions were included in transactionRevenue and certain other transactions were disregarded? It doesn't seem like this. For many other metrics we have numbers for the entire session (subfields of total: timeOnSite, page views, hits, even the number of transactions in the transactions field.)\n\nOf course, most users conduct just one transaction per visit (if any at all), in this case there is no difference between transactionRevenue and totalTransactionRevenue. But I assumed it is precisely the users who spend a little more is whose behaviour we are really interested in. \n\nAnother thing: of course the difference between transactionRevenue and totalTransactionRevenue is further reduced by taking logarithms. This also leads to only a minor difference in the scores (0 vs. 0.03). But is it actually a log of revenue that matters in reality? I don’t remember any log revenues or profits in companies’ financial statements. So I would prefer if the RMSE were calculated based on raw (i.e. not logarithmised) revenue numbers, and of course total and not partial ones.\n\nAs a conclusion, probably my strategy won’t change much, but if the score were calculated the way I suggest, it could have significant implications for the final standings.", "votes": null }, { "id": "422585", "postDate": "11/16/2018 12:53:06", "content": "For those who don't want to read it all through: I am claiming the following:
\n\ntransactionRevenue is wrong\ntotalTransactionRevenue is correct
", "rawMarkdown": "For those who don't want to read it all through: I am claiming the following:\n\ntransactionRevenue is wrong\ntotalTransactionRevenue is correct", "votes": null }, { "id": "422718", "postDate": "11/16/2018 17:01:33", "content": "Thanks for your post. I think you are right but we need competition manager to confirm that.
", "rawMarkdown": "Thanks for your post. I think you are right but we need competition manager to confirm that.", "votes": null }, { "id": "424170", "postDate": "11/19/2018 17:38:53", "content": "Forgive me for asking this silly question, but if we are only suppose to submit results and NOT a model.\nWhat is the point of making the entire set a \"train set\"?
", "rawMarkdown": "Forgive me for asking this silly question, but if we are only suppose to submit results and NOT a model.\nWhat is the point of making the entire set a \"train set\"?", "votes": null }, { "id": "424296", "postDate": "11/19/2018 21:24:30", "content": "Not sure what you mean. Are you referring to my initial post?
", "rawMarkdown": "Not sure what you mean. Are you referring to my initial post?", "votes": null }, { "id": "424645", "postDate": "11/20/2018 13:29:59", "content": "Antonio, you are right transactionRevenue is last purchase within a session, totalTransactionRevenue is sum of all purchases within a session. I know this from comparing demo account data and train/test data.
", "rawMarkdown": "Antonio, you are right transactionRevenue is last purchase within a session, totalTransactionRevenue is sum of all purchases within a session. I know this from comparing demo account data and train/test data.", "votes": null }, { "id": "424769", "postDate": "11/20/2018 16:21:51", "content": "Hey Antonio - thanks for checking this out and letting us know! I appreciate it. I've looked into it and you're right that totals.totalTransactionRevenue is the new, corrected version of the target variable. However, we won't be changing the target at this point for two reasons:
1) Time is too short. There are no guarantees that everyone would get the message to switch, as there's no good way to invalidate submissions made with one or the other. Keeping the target as-is means that at least everyone is working on the same objective for this final week or so.
\n\n2) totals.transactionRevenue and totals.totalTransactionRevenue are always zero - or non-zero - together. While totals.transactionRevenue is definitely flawed, it doesn't appear to be invalid as a measure of whether or not someone spent money at the store.
I do apologize - I think this is the most viable way to go at this point. Thanks again. This was definitely helpful in checking out the problem.
\n\nRe: the log of revenue, that's a fair point! I chose logs because we couldn't change the values and still line up with the public BigQuery set, but I didn't want to penalize based on the massive multiplier used for revenue. We were somewhat locked in by having multiple data sources, including one that we (essentially) could not change.
", "rawMarkdown": "Hey Antonio - thanks for checking this out and letting us know! I appreciate it. I've looked into it and you're right that `totals.totalTransactionRevenue` is the new, corrected version of the target variable. However, we won't be changing the target at this point for two reasons:\n\n1) Time is too short. There are no guarantees that everyone would get the message to switch, as there's no good way to invalidate submissions made with one or the other. Keeping the target as-is means that at least everyone is working on the same objective for this final week or so.\n\n2) `totals.transactionRevenue` and `totals.totalTransactionRevenue` are always zero - or non-zero - together. While `totals.transactionRevenue` is definitely flawed, it doesn't appear to be invalid as a measure of whether or not someone spent money at the store.\n\nI do apologize - I think this is the most viable way to go at this point. Thanks again. This was definitely helpful in checking out the problem.\n\nRe: the log of revenue, that's a fair point! I chose logs because we couldn't change the values and still line up with the public BigQuery set, but I didn't want to penalize based on the massive multiplier used for revenue. We were somewhat locked in by having multiple data sources, including one that we (essentially) could not change.", "votes": null }, { "id": "425665", "postDate": "11/21/2018 23:40:16", "content": "Thanks for you reply Phil! Yes, I understand your point and agree that there is not much time left. However, concerning your last point on logarithms, what I still don't understand (although I assume it is too late for any changes on that as well) is if it were possible to define the metric as an RMSE between actual and predicted figures that are first divided by 1e6, i.e. to effectively compare revenues expressed in dollars. Taking logs is a transformation in a certain sense not fundamentally different from division by a million (i.e. also a monotonic transformation), so if you are allowed to do one, why are you not allowed to do the other?
", "rawMarkdown": "Thanks for you reply Phil! Yes, I understand your point and agree that there is not much time left. However, concerning your last point on logarithms, what I still don't understand (although I assume it is too late for any changes on that as well) is if it were possible to define the metric as an RMSE between actual and predicted figures that are first divided by 1e6, i.e. to effectively compare revenues expressed in dollars. Taking logs is a transformation in a certain sense not fundamentally different from division by a million (i.e. also a monotonic transformation), so if you are allowed to do one, why are you not allowed to do the other?", "votes": null }, { "id": "426541", "postDate": "11/23/2018 12:16:09", "content": "yes.
", "rawMarkdown": "yes.", "votes": null }, { "id": "426654", "postDate": "11/23/2018 15:45:53", "content": "Sorry, I don't understand your question, especially in the context of my post.
", "rawMarkdown": "Sorry, I don't understand your question, especially in the context of my post.", "votes": null }, { "id": "426669", "postDate": "11/23/2018 16:12:41", "content": "Hey Antonio - that's a great question and a good point. I could have simply asked people to divide by the multiplier. I had a couple of possible paths and chose what I felt was the more common (and I felt, less magical) log transformation. Because I couldn't transform the data prior to people getting it, I wanted to use something that didn't involve some special knowledge (that there was a x1000000 multiplier).
", "rawMarkdown": "Hey Antonio - that's a great question and a good point. I could have simply asked people to divide by the multiplier. I had a couple of possible paths and chose what I felt was the more common (and I felt, less magical) log transformation. Because I couldn't transform the data prior to people getting it, I wanted to use something that didn't involve some special knowledge (that there was a x1000000 multiplier).", "votes": null }, { "id": "430320", "postDate": "11/30/2018 07:44:32", "content": "Thanks Antonio. It seems your finding reveals that we are in a lottery. Not as an analogy but literally.
", "rawMarkdown": "Thanks Antonio. It seems your finding reveals that we are in a lottery. Not as an analogy but literally.", "votes": null }, { "id": "442057", "postDate": "12/19/2018 12:24:39", "content": "I was asking what is the point of prediction if the entire set is known
", "rawMarkdown": "I was asking what is the point of prediction if the entire set is known", "votes": null }, { "id": "443371", "postDate": "12/21/2018 14:10:45", "content": "The figures we were supposed to submit were not known because they related to the future at the moment of submission. The target values were indeed known for the \"test\" part of the historical dataset, which in effect became part of the training set. The point of submission of these known values was only to make sure that you calculate the target the way Kaggle expects us to do it (i.e. in terms of summation, taking logs and of course choosing between transactionRevenue and totalTransactionRevenue). I believe this was extensively discussed in numerous threads and I frankly still don't understand how this is relevant in this thread. I was making a point about the difference between transactionRevenue and totalTransactionRevenue.
", "rawMarkdown": "The figures we were supposed to submit were not known because they related to the future at the moment of submission. The target values were indeed known for the \"test\" part of the historical dataset, which in effect became part of the training set. The point of submission of these known values was only to make sure that you calculate the target the way Kaggle expects us to do it (i.e. in terms of summation, taking logs and of course choosing between transactionRevenue and totalTransactionRevenue). I believe this was extensively discussed in numerous threads and I frankly still don't understand how this is relevant in this thread. I was making a point about the difference between transactionRevenue and totalTransactionRevenue.", "votes": null } ], "comments": [ { "id": 422585, "author_name": "antoniorossi", "author_url": "", "post_date": "11/16/2018 12:53:06", "content": "For those who don't want to read it all through: I am claiming the following:
\n\ntransactionRevenue is wrong\ntotalTransactionRevenue is correct
", "votes": null, "replies": [] }, { "id": 422718, "author_name": "zhangchi9", "author_url": "", "post_date": "11/16/2018 17:01:33", "content": "Thanks for your post. I think you are right but we need competition manager to confirm that.
", "votes": null, "replies": [] }, { "id": 424170, "author_name": "ybond92", "author_url": "", "post_date": "11/19/2018 17:38:53", "content": "Forgive me for asking this silly question, but if we are only suppose to submit results and NOT a model.\nWhat is the point of making the entire set a \"train set\"?
", "votes": null, "replies": [ { "id": 424296, "author_name": "antoniorossi", "author_url": "", "post_date": "11/19/2018 21:24:30", "content": "Not sure what you mean. Are you referring to my initial post?
", "votes": null, "replies": [] }, { "id": 426541, "author_name": "ybond92", "author_url": "", "post_date": "11/23/2018 12:16:09", "content": "yes.
", "votes": null, "replies": [] }, { "id": 426654, "author_name": "antoniorossi", "author_url": "", "post_date": "11/23/2018 15:45:53", "content": "Sorry, I don't understand your question, especially in the context of my post.
", "votes": null, "replies": [] }, { "id": 442057, "author_name": "ybond92", "author_url": "", "post_date": "12/19/2018 12:24:39", "content": "I was asking what is the point of prediction if the entire set is known
", "votes": null, "replies": [] }, { "id": 443371, "author_name": "antoniorossi", "author_url": "", "post_date": "12/21/2018 14:10:45", "content": "The figures we were supposed to submit were not known because they related to the future at the moment of submission. The target values were indeed known for the \"test\" part of the historical dataset, which in effect became part of the training set. The point of submission of these known values was only to make sure that you calculate the target the way Kaggle expects us to do it (i.e. in terms of summation, taking logs and of course choosing between transactionRevenue and totalTransactionRevenue). I believe this was extensively discussed in numerous threads and I frankly still don't understand how this is relevant in this thread. I was making a point about the difference between transactionRevenue and totalTransactionRevenue.
", "votes": null, "replies": [] } ] }, { "id": 424645, "author_name": "alexfir", "author_url": "", "post_date": "11/20/2018 13:29:59", "content": "Antonio, you are right transactionRevenue is last purchase within a session, totalTransactionRevenue is sum of all purchases within a session. I know this from comparing demo account data and train/test data.
", "votes": null, "replies": [] }, { "id": 424769, "author_name": "philculliton", "author_url": "", "post_date": "11/20/2018 16:21:51", "content": "Hey Antonio - thanks for checking this out and letting us know! I appreciate it. I've looked into it and you're right that totals.totalTransactionRevenue is the new, corrected version of the target variable. However, we won't be changing the target at this point for two reasons:
1) Time is too short. There are no guarantees that everyone would get the message to switch, as there's no good way to invalidate submissions made with one or the other. Keeping the target as-is means that at least everyone is working on the same objective for this final week or so.
\n\n2) totals.transactionRevenue and totals.totalTransactionRevenue are always zero - or non-zero - together. While totals.transactionRevenue is definitely flawed, it doesn't appear to be invalid as a measure of whether or not someone spent money at the store.
I do apologize - I think this is the most viable way to go at this point. Thanks again. This was definitely helpful in checking out the problem.
\n\nRe: the log of revenue, that's a fair point! I chose logs because we couldn't change the values and still line up with the public BigQuery set, but I didn't want to penalize based on the massive multiplier used for revenue. We were somewhat locked in by having multiple data sources, including one that we (essentially) could not change.
", "votes": null, "replies": [ { "id": 425665, "author_name": "antoniorossi", "author_url": "", "post_date": "11/21/2018 23:40:16", "content": "Thanks for you reply Phil! Yes, I understand your point and agree that there is not much time left. However, concerning your last point on logarithms, what I still don't understand (although I assume it is too late for any changes on that as well) is if it were possible to define the metric as an RMSE between actual and predicted figures that are first divided by 1e6, i.e. to effectively compare revenues expressed in dollars. Taking logs is a transformation in a certain sense not fundamentally different from division by a million (i.e. also a monotonic transformation), so if you are allowed to do one, why are you not allowed to do the other?
", "votes": null, "replies": [] }, { "id": 426669, "author_name": "philculliton", "author_url": "", "post_date": "11/23/2018 16:12:41", "content": "Hey Antonio - that's a great question and a good point. I could have simply asked people to divide by the multiplier. I had a couple of possible paths and chose what I felt was the more common (and I felt, less magical) log transformation. Because I couldn't transform the data prior to people getting it, I wanted to use something that didn't involve some special knowledge (that there was a x1000000 multiplier).
", "votes": null, "replies": [] } ] }, { "id": 430320, "author_name": "fatihozturk", "author_url": "", "post_date": "11/30/2018 07:44:32", "content": "Thanks Antonio. It seems your finding reveals that we are in a lottery. Not as an analogy but literally.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "422579": "For those who don't want to read it all through: I am claiming the following:\n\ntransactionRevenue is a wrong thing to look at\nit's **totalTransactionRevenue** that makes sense to analyse and to predict.\n\nHowever Kaggle wants us to analyse the former.\n\n****************************************************************\n\n\n\nSince it is now possible to submit the known values of the predicted variable for the test period and so get a perfect score, it is obvious that what Kaggle wants us to predict is really transactionRevenue and not totalTransactionRevenue. If you submit transactionRevenue, you will score 0, if you submit totalTransactionRevenue, only 0.03.\n This is actually what it quite clearly says in the Data section, paragraph \"What should I expect the data format to be?\".\n\nThe thing is that we also have some revenue data buried deeply inside the infamous Hits field. I was doing some consistency checks between these different views on the revenue and it turned out that (as you probably know) in the Hits field we have json datasets for each hit, which includes some revenue figures for a transaction if such a hit resulted in a transaction.\n\nNow, based on at least three examples I can conclude that the field transactionRevenue corresponds only to the revenue resulting from one certain transaction during a visit, whereas **totalTransactionRevenue** represents the whole revenue resulting from the whole visit. That is why any aggregations of transactionRevenue on the user level represent some (random) fractions of the aggregate user revenue over a certain period of time, and if this is really the case, this number is unfortunately fairly difficult to model and senseless to predict.\n\nHere are the examples. All come from the train_v2.csv, the index is the row range index (zero-based, python style!)\n\n1) row 4536: \ntotals: 'totalTransactionRevenue': '2103690000', 'transactionRevenue': '546320000'\nhits: two available values for transactionRevenue are 1557370000 and 548320000 \nWe see that the second value from hits corresponds to the transactionRevenue and their sum (2105690000) corresponds to the totalTransactionRevenue. Why they are not precisely equal is another question, but this is apparently not because of 'transactionShipping' or 'transactionTax', two other fields also available in the hits jsons. I've tried adding and subtracting them, to no avail. Expressed in dollars this deviation is in fact just $2.\n\n2) row 4639: \ntotals: 'totalTransactionRevenue': '672960000', 'transactionRevenue': '338970000',\nhits: two available values for transactionRevenue are 333990000 and 341970000\nAgain, the second hits transactionRevenue corresponds to transactionRevenue from totals, the sum (675960000) corresponds to totals totalTransactionRevenue. Again, they do not match exactly, the deviation this time is $3.\n\n3) row 9761: \ntotals: 'totalTransactionRevenue': '1750210000', 'transactionRevenue': '1077000000'\nhits: two available values for transactionRevenue are 673210000 and 1079000000. \nAgain, the second hits value corresponds to transactionRevenue and their sum (1752210000) corresponds to totalTransactionRevenue. Deviation = $2.\n\nIs there any reason why only those certain transactions were included in transactionRevenue and certain other transactions were disregarded? It doesn't seem like this. For many other metrics we have numbers for the entire session (subfields of total: timeOnSite, page views, hits, even the number of transactions in the transactions field.)\n\nOf course, most users conduct just one transaction per visit (if any at all), in this case there is no difference between transactionRevenue and totalTransactionRevenue. But I assumed it is precisely the users who spend a little more is whose behaviour we are really interested in. \n\nAnother thing: of course the difference between transactionRevenue and totalTransactionRevenue is further reduced by taking logarithms. This also leads to only a minor difference in the scores (0 vs. 0.03). But is it actually a log of revenue that matters in reality? I don’t remember any log revenues or profits in companies’ financial statements. So I would prefer if the RMSE were calculated based on raw (i.e. not logarithmised) revenue numbers, and of course total and not partial ones.\n\nAs a conclusion, probably my strategy won’t change much, but if the score were calculated the way I suggest, it could have significant implications for the final standings.", "422585": "For those who don't want to read it all through: I am claiming the following:\n\ntransactionRevenue is wrong\ntotalTransactionRevenue is correct", "422718": "Thanks for your post. I think you are right but we need competition manager to confirm that.", "424170": "Forgive me for asking this silly question, but if we are only suppose to submit results and NOT a model.\nWhat is the point of making the entire set a \"train set\"?", "424296": "Not sure what you mean. Are you referring to my initial post?", "424645": "Antonio, you are right transactionRevenue is last purchase within a session, totalTransactionRevenue is sum of all purchases within a session. I know this from comparing demo account data and train/test data.", "424769": "Hey Antonio - thanks for checking this out and letting us know! I appreciate it. I've looked into it and you're right that `totals.totalTransactionRevenue` is the new, corrected version of the target variable. However, we won't be changing the target at this point for two reasons:\n\n1) Time is too short. There are no guarantees that everyone would get the message to switch, as there's no good way to invalidate submissions made with one or the other. Keeping the target as-is means that at least everyone is working on the same objective for this final week or so.\n\n2) `totals.transactionRevenue` and `totals.totalTransactionRevenue` are always zero - or non-zero - together. While `totals.transactionRevenue` is definitely flawed, it doesn't appear to be invalid as a measure of whether or not someone spent money at the store.\n\nI do apologize - I think this is the most viable way to go at this point. Thanks again. This was definitely helpful in checking out the problem.\n\nRe: the log of revenue, that's a fair point! I chose logs because we couldn't change the values and still line up with the public BigQuery set, but I didn't want to penalize based on the massive multiplier used for revenue. We were somewhat locked in by having multiple data sources, including one that we (essentially) could not change.", "425665": "Thanks for you reply Phil! Yes, I understand your point and agree that there is not much time left. However, concerning your last point on logarithms, what I still don't understand (although I assume it is too late for any changes on that as well) is if it were possible to define the metric as an RMSE between actual and predicted figures that are first divided by 1e6, i.e. to effectively compare revenues expressed in dollars. Taking logs is a transformation in a certain sense not fundamentally different from division by a million (i.e. also a monotonic transformation), so if you are allowed to do one, why are you not allowed to do the other?", "426541": "yes.", "426654": "Sorry, I don't understand your question, especially in the context of my post.", "426669": "Hey Antonio - that's a great question and a good point. I could have simply asked people to divide by the multiplier. I had a couple of possible paths and chose what I felt was the more common (and I felt, less magical) log transformation. Because I couldn't transform the data prior to people getting it, I wanted to use something that didn't involve some special knowledge (that there was a x1000000 multiplier).", "430320": "Thanks Antonio. It seems your finding reveals that we are in a lottery. Not as an analogy but literally.", "442057": "I was asking what is the point of prediction if the entire set is known", "443371": "The figures we were supposed to submit were not known because they related to the future at the moment of submission. The target values were indeed known for the \"test\" part of the historical dataset, which in effect became part of the training set. The point of submission of these known values was only to make sure that you calculate the target the way Kaggle expects us to do it (i.e. in terms of summation, taking logs and of course choosing between transactionRevenue and totalTransactionRevenue). I believe this was extensively discussed in numerous threads and I frankly still don't understand how this is relevant in this thread. I was making a point about the difference between transactionRevenue and totalTransactionRevenue." }, "source": "meta" }