{ "id": 69592, "title": "2 issues of the new competition settings", "url": "/competitions/ga-customer-revenue-prediction/discussion/69592", "author_name": "comsaint", "post_date": "2018-10-25T06:59:28.623000", "votes": 17, "comment_count": 17, "views": 0, "content": "

Greetings.

\n\n

After previous data leak, the organizer updated the rules of the competition, which are greeted warmly by the community in general. However, I see two potential problems with the new setting:

\n\n
    \n
  1. As the ID of the demo GA profile is leaked, one can send fake PV/ transactions to it anytime, polluting it to s/he favor.
    2. \n
  2. The statement by organizer,
    3. \n
\n\n
\n

The Google Merchandise Store Demo Account effectively becomes freely available to use public data, but will no longer provide any additional benefit to final standings.

\n
\n\n

is plainly false.

\n\n

Let's start with #1.

\n\n

Problem 1: Inject fake booking (with fids already exist in before Oct-15)

\n\n

A bit of background knowledge - the provided data we have come from Google Analytics, which collects data on the website and exports to Google BigQuery. Google Analytics collects data via a piece of JS ('tag') implemented on website, which passes information via standard HTTP POST calls.

\n\n

Our data set contains a field fullVisitorId (available in BigQuery's schema), or fid, which is a hashed version of the client ID (cid) used by GA's cookie. In this kernel, I have shown a way to match an fid with a cid, thus will not elaborate here.

\n\n

Now the crucial part: given a GA profile ID (UA-54516992-1 in our case) and a cid, one can generate any fake user behaviors and purchases.

\n\n

Firstly, some references from Google:

\n\n
    \n

  1. Document on the protocol used by GA to pass information from website to server

    2. \n

  2. A handy tool to generate valid calls to GA

    3. \n
\n\n

I am going to skip the details on the parameters used in my calls, but simply attaching the screenshots of my setting and result. Please refer to the attachments, where I generate a booking which can be observed in the demo account. You can get creative and forge any custom call with fake geo-location/channel/page path/purchase etc. by referencing the the protocol above.

\n\n

The implication is that everyone can fake purchases by past visitors during evaluation period, as far as they have the correct GA profile ID and cid.

\n\n

Notes:

\n\n

Let's revisit the assumptions carefully:

\n\n

1) Is the profile ID UA-54516992-1 really the ID used in this competition?

\n\n

A: I do not know, but guess not. Checking the site, I can see a number of profile IDs on it, and the organizer can be using any one of them. However, likely the 'true' profile is among them (well, data have to come from somewhere), so we can simply send the same web calls to all profiles.

\n\n

2) If the profile ID is different, is cid consistent across profiles?

\n\n

A: I think so. In my work we manage lots of websites, and the cid is consistent across different profiles (even across GA accounts). Likely GA is using same cookie for different sites, which is quite logical.

\n\n

Possible solutions

\n\n

Finally, solution I can think of (any ideas?):

\n\n

Solution 1 - Validate all bookings during test period. This is the most obvious solution, checking with real-world to see if a purchase really happens.

\n\n

However, there is still a problem: given purchases are rare and the limited predictability of current setting (predicting revenue of far future), one can edit their own cookie to pretend as any cid, make a purchase as normal customer, and s/he will immediately gain a good advantage over all competitors (someone volunteers to do the math for the bias of a $20 purchase?). This is further encouraged by the monetary prize of competition. (And we have not touched on refund policy yet...)

\n\n

Let's pretend as an existing cid and do something.

\n\n
    \n

  1. Firstly, we need a cid to begin with. Let's pick 952749702.1498272983 as our cid, which is also known to be fid 2658678838785413275 (refer to my notebook), as seen in the training set.

    2. \n

  2. Enters the shop at https://shop.googlemerchandisestore.com/. If you are interested, you can explore the cookies in browser.

    3. \n

  3. Delete all existing cookies on the page (with a cookie editor for example).

    4. \n

  4. Enters developer view (in Chrome, press F12. Check the setting of your favorite browser), and go to \"console\" tab (the one you can type Javascript code).

    5. \n

  5. Create a new GA tracker by entering the following command:

    \n\n

    ga.create('UA-545169921',{'clientId':'952749702.1498272983'});

    \n\n

    Note that clientId cannot be altered after GA tracker is created, so it needs to be set now. You can pass additional information e.g. source/medium, following the protocol.

    6. \n

  6. Refresh the page and GA will recognize you as cid 952749702.1498272983. You can verify by entering the following command in console:

    \n\n

    for (i=0; i<ga.getAll().length; i++){console.log(ga.getAll()[i].get('clientId'));}

    \n\n

    In my case there are 9 trackers on the page, all recognize me as 952749702.1498272983.

    7. \n

  7. Finally, check the User Explorer view in demo account to confirm that everything goes through. You browsing history should show within a few minutes. (see my screenshot pretend as 2658678838785413275). You can of course make a real purchase now.

    8. \n
\n\n

This implies everyone can pretend as any cid and create new purchase history any time, so verifying with real purchases does not help.

\n\n

Solution 2 - Just use a new data set. This is the best alternative I can think of. Be sure the profile ID and data do not leak!

\n\n

Problem 2: Extra data from Demo account give great advantage

\n\n

Let's revisit the new rules. We are provided some data from a few years back up to 2018 Oct 15th, followed by a gap period from Oct 16th to Nov 30th, and we are asked to predict the revenue for the months of 2018 Dec and 2019 Jan, i.e. there is a gap of 1.5 months between training set and evaluation period.

\n\n

The source of data leak, Google demo account UA-54516992-1, provides data close to real-time, so a user can use it as data source of the gap period. As we have seen earlier how to match an fid and cid, we can leverage the demo account to check if an fid has any visit/purchase behavior during gap period, giving much advantage.

\n\n

Is this an issue? Not by rule, as external data source is permitted at the moment. However, what troubles me is this external data may be even more valuable than the provided training data due to its recency.

\n\n

A bit of shameless self-advertising: I have previously completed a similar project together with Google on our e-commerce site (data volume a couple orders of magnitude bigger BTW), so I suppose I am qualified to make this statement. In our project, we use the same data source (the full BigQuery data schema, plus DoubleClick data too) to predict whether a customer will return and purchase in following week. One thing we learn is that data recency is (again) key to good model, as in most prediction tasks. Features reflecting events a few weeks past become useless fast. Another factor that contributes to this phenomenon is due to the nature of identity tracking via cookies, which gets into trouble with cross-device/browser purchases, making their prediction power short-lived. Not to mention we are to predict not the happening of purchase event, but the actual revenue.

\n\n

And do not forget that overall sales trend can assist in fine-tuning individual revenue too.

\n\n

The takeaway is if you want to win this competition, make sure you are prepared to scrape as much data as from the demo account as possible, which may hold much predictive power than all provided data put together (needless to say, combining them gives best result).

\n\n

Anyway, the statement by organizer is, at least, inaccurate.

\n\n

I hope the organizer have taken precautions of these issues, but raise them here just in case.

", "messages": [ { "id": 409989, "postDate": "2018-10-25T06:59:28.623Z", "content": "

Greetings.

\n\n

After previous data leak, the organizer updated the rules of the competition, which are greeted warmly by the community in general. However, I see two potential problems with the new setting:

\n\n
    \n
  1. As the ID of the demo GA profile is leaked, one can send fake PV/ transactions to it anytime, polluting it to s/he favor.
    2. \n
  2. The statement by organizer,
    3. \n
\n\n
\n

The Google Merchandise Store Demo Account effectively becomes freely available to use public data, but will no longer provide any additional benefit to final standings.

\n
\n\n

is plainly false.

\n\n

Let's start with #1.

\n\n

Problem 1: Inject fake booking (with fids already exist in before Oct-15)

\n\n

A bit of background knowledge - the provided data we have come from Google Analytics, which collects data on the website and exports to Google BigQuery. Google Analytics collects data via a piece of JS ('tag') implemented on website, which passes information via standard HTTP POST calls.

\n\n

Our data set contains a field fullVisitorId (available in BigQuery's schema), or fid, which is a hashed version of the client ID (cid) used by GA's cookie. In this kernel, I have shown a way to match an fid with a cid, thus will not elaborate here.

\n\n

Now the crucial part: given a GA profile ID (UA-54516992-1 in our case) and a cid, one can generate any fake user behaviors and purchases.

\n\n

Firstly, some references from Google:

\n\n
    \n

  1. Document on the protocol used by GA to pass information from website to server

    2. \n

  2. A handy tool to generate valid calls to GA

    3. \n
\n\n

I am going to skip the details on the parameters used in my calls, but simply attaching the screenshots of my setting and result. Please refer to the attachments, where I generate a booking which can be observed in the demo account. You can get creative and forge any custom call with fake geo-location/channel/page path/purchase etc. by referencing the the protocol above.

\n\n

The implication is that everyone can fake purchases by past visitors during evaluation period, as far as they have the correct GA profile ID and cid.

\n\n

Notes:

\n\n

Let's revisit the assumptions carefully:

\n\n

1) Is the profile ID UA-54516992-1 really the ID used in this competition?

\n\n

A: I do not know, but guess not. Checking the site, I can see a number of profile IDs on it, and the organizer can be using any one of them. However, likely the 'true' profile is among them (well, data have to come from somewhere), so we can simply send the same web calls to all profiles.

\n\n

2) If the profile ID is different, is cid consistent across profiles?

\n\n

A: I think so. In my work we manage lots of websites, and the cid is consistent across different profiles (even across GA accounts). Likely GA is using same cookie for different sites, which is quite logical.

\n\n

Possible solutions

\n\n

Finally, solution I can think of (any ideas?):

\n\n

Solution 1 - Validate all bookings during test period. This is the most obvious solution, checking with real-world to see if a purchase really happens.

\n\n

However, there is still a problem: given purchases are rare and the limited predictability of current setting (predicting revenue of far future), one can edit their own cookie to pretend as any cid, make a purchase as normal customer, and s/he will immediately gain a good advantage over all competitors (someone volunteers to do the math for the bias of a $20 purchase?). This is further encouraged by the monetary prize of competition. (And we have not touched on refund policy yet...)

\n\n

Let's pretend as an existing cid and do something.

\n\n
    \n

  1. Firstly, we need a cid to begin with. Let's pick 952749702.1498272983 as our cid, which is also known to be fid 2658678838785413275 (refer to my notebook), as seen in the training set.

    2. \n

  2. Enters the shop at https://shop.googlemerchandisestore.com/. If you are interested, you can explore the cookies in browser.

    3. \n

  3. Delete all existing cookies on the page (with a cookie editor for example).

    4. \n

  4. Enters developer view (in Chrome, press F12. Check the setting of your favorite browser), and go to \"console\" tab (the one you can type Javascript code).

    5. \n

  5. Create a new GA tracker by entering the following command:

    \n\n

    ga.create('UA-545169921',{'clientId':'952749702.1498272983'});

    \n\n

    Note that clientId cannot be altered after GA tracker is created, so it needs to be set now. You can pass additional information e.g. source/medium, following the protocol.

    6. \n

  6. Refresh the page and GA will recognize you as cid 952749702.1498272983. You can verify by entering the following command in console:

    \n\n

    for (i=0; i<ga.getAll().length; i++){console.log(ga.getAll()[i].get('clientId'));}

    \n\n

    In my case there are 9 trackers on the page, all recognize me as 952749702.1498272983.

    7. \n

  7. Finally, check the User Explorer view in demo account to confirm that everything goes through. You browsing history should show within a few minutes. (see my screenshot pretend as 2658678838785413275). You can of course make a real purchase now.

    8. \n
\n\n

This implies everyone can pretend as any cid and create new purchase history any time, so verifying with real purchases does not help.

\n\n

Solution 2 - Just use a new data set. This is the best alternative I can think of. Be sure the profile ID and data do not leak!

\n\n

Problem 2: Extra data from Demo account give great advantage

\n\n

Let's revisit the new rules. We are provided some data from a few years back up to 2018 Oct 15th, followed by a gap period from Oct 16th to Nov 30th, and we are asked to predict the revenue for the months of 2018 Dec and 2019 Jan, i.e. there is a gap of 1.5 months between training set and evaluation period.

\n\n

The source of data leak, Google demo account UA-54516992-1, provides data close to real-time, so a user can use it as data source of the gap period. As we have seen earlier how to match an fid and cid, we can leverage the demo account to check if an fid has any visit/purchase behavior during gap period, giving much advantage.

\n\n

Is this an issue? Not by rule, as external data source is permitted at the moment. However, what troubles me is this external data may be even more valuable than the provided training data due to its recency.

\n\n

A bit of shameless self-advertising: I have previously completed a similar project together with Google on our e-commerce site (data volume a couple orders of magnitude bigger BTW), so I suppose I am qualified to make this statement. In our project, we use the same data source (the full BigQuery data schema, plus DoubleClick data too) to predict whether a customer will return and purchase in following week. One thing we learn is that data recency is (again) key to good model, as in most prediction tasks. Features reflecting events a few weeks past become useless fast. Another factor that contributes to this phenomenon is due to the nature of identity tracking via cookies, which gets into trouble with cross-device/browser purchases, making their prediction power short-lived. Not to mention we are to predict not the happening of purchase event, but the actual revenue.

\n\n

And do not forget that overall sales trend can assist in fine-tuning individual revenue too.

\n\n

The takeaway is if you want to win this competition, make sure you are prepared to scrape as much data as from the demo account as possible, which may hold much predictive power than all provided data put together (needless to say, combining them gives best result).

\n\n

Anyway, the statement by organizer is, at least, inaccurate.

\n\n

I hope the organizer have taken precautions of these issues, but raise them here just in case.

", "rawMarkdown": "Greetings. \n\nAfter previous data leak, the organizer updated the rules of the competition, which are greeted warmly by the community in general. However, I see two potential problems with the new setting:\n\n 1. As the ID of the demo GA profile is leaked, one can send fake PV/ transactions to it anytime, polluting it to s/he favor.\n 2. The statement by organizer,\n\n> The Google Merchandise Store Demo Account effectively becomes freely available to use public data, but will no longer provide any additional benefit to final standings.\n\nis plainly false.\n\nLet's start with #1.\n\n## Problem 1: Inject fake booking (with `fid`s already exist in before Oct-15)\n\nA bit of background knowledge - the provided data we have come from Google Analytics, which collects data on the website and exports to Google BigQuery. Google Analytics collects data via a piece of JS ('tag') implemented on website, which passes information via standard HTTP POST calls.\n\nOur data set contains a field `fullVisitorId` (available in BigQuery's schema), or `fid`, which is a hashed version of the client ID (`cid`) used by GA's cookie. In [this][1] kernel, I have shown a way to match an `fid` with a `cid`, thus will not elaborate here.\n\nNow the crucial part: given a GA profile ID (`UA-54516992-1` in our case) and a `cid`, one can generate any fake user behaviors and purchases. \n\nFirstly, some references from Google:\n\n 1. [Document][2] on the protocol used by GA to pass information from website to server\n\n 2. [A handy tool][3] to generate valid calls to GA\n\nI am going to skip the details on the parameters used in my calls, but simply attaching the screenshots of my setting and result. Please refer to the attachments, where I generate a booking which can be observed in the demo account. You can get creative and forge any custom call with fake geo-location/channel/page path/purchase etc. by referencing the the protocol above.\n\nThe implication is that everyone can **fake purchases by past visitors** during evaluation period, as far as they have the correct GA profile ID and `cid`. \n\n### Notes:\n\nLet's revisit the assumptions carefully:\n\n1) Is the profile ID `UA-54516992-1` really the ID used in this competition?\n\nA: I do not know, but guess not. Checking the [site][4], I can see a number of profile IDs on it, and the organizer can be using any one of them. However, likely the 'true' profile is among them (well, data have to come from somewhere), so we can simply send the same web calls to all profiles.\n\n2) If the profile ID is different, is `cid` consistent across profiles?\n\nA: I think so. In my work we manage lots of websites, and the `cid` is consistent across different profiles (even across GA accounts). Likely GA is using same cookie for different sites, which is quite logical.\n\n### Possible solutions\n\nFinally, solution I can think of (any ideas?):\n\n**Solution 1** - Validate all bookings during test period. This is the most obvious solution, checking with real-world to see if a purchase really happens. \n\nHowever, there is still a problem: given purchases are rare and the limited predictability of current setting (predicting revenue of far future), one can edit their own cookie to pretend as any `cid`, make a purchase as normal customer, and s/he will immediately gain a good advantage over all competitors (someone volunteers to do the math for the bias of a $20 purchase?). This is further encouraged by the monetary prize of competition. (And we have not touched on refund policy yet...)\n\nLet's pretend as an existing `cid` and do something. \n\n 1. Firstly, we need a `cid` to begin with. Let's pick **952749702.1498272983** as our `cid`, which is also known to be `fid` **2658678838785413275** (refer to [my notebook][1]), as seen in the training set.\n\n 2. Enters the shop at https://shop.googlemerchandisestore.com/. If you are interested, you can explore the cookies in browser.\n\n 3. Delete all existing cookies on the page (with a cookie editor for example).\n\n 4. Enters developer view (in Chrome, press F12. Check the setting of your favorite browser), and go to \"console\" tab (the one you can type Javascript code).\n\n 5. Create a new GA tracker by entering the following command:\n\n `ga.create('UA-545169921',{'clientId':'952749702.1498272983'});`\n\n Note that `clientId` cannot be altered after GA tracker is created, so it needs to be set now. You can pass additional information e.g. source/medium, following the [protocol][2].\n\n 6. Refresh the page and GA will recognize you as `cid` **952749702.1498272983**. You can verify by entering the following command in console:\n\n `for (i=0; i", "votes": 16 }, { "id": 410000, "postDate": "2018-10-25T07:21:22.923Z", "content": "

Hi @Comasaint,\nI wrote a comment earlier before reading the entire post.\nI deleted it as it was irrelevant to the main point you were trying to convey.

\n\n

Indeed if someone can, during the future test period, insert purchases as other, existing visitorIDs - that is very bad.

", "rawMarkdown": "Hi @Comasaint,\nI wrote a comment earlier before reading the entire post.\nI deleted it as it was irrelevant to the main point you were trying to convey.\n\nIndeed if someone can, during the future test period, insert purchases as other, existing visitorIDs - that is very bad.", "votes": 1, "replies": [ { "id": 410003, "postDate": "2018-10-25T07:25:25.307Z", "content": "

@AmirH - No problem, I also delete my reply. Thanks!

", "rawMarkdown": "@AmirH - No problem, I also delete my reply. Thanks!", "votes": 1 } ] }, { "id": 417312, "postDate": "2018-11-08T04:34:31.867Z", "content": "

Thx @comsaint! Learned a lot about GA from your post.

", "rawMarkdown": "Thx @comsaint! Learned a lot about GA from your post." }, { "id": 413294, "postDate": "2018-10-31T16:34:51.670Z", "content": "

Thank you for detailing your concerns. We appreciate the transparency!

\n\n

First, you should expect that we have measures in place to identify fake injected data during the test set period and omit from evaluation.

\n\n

Second, any team identified as having committed these acts to intentionally insert invalid purchases in the manner you describe would be considered in violation of the rules which prohibit attempts \"...to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses...\" Such teams will be disqualified and removed from the competition.

", "rawMarkdown": "Thank you for detailing your concerns. We appreciate the transparency!\n\nFirst, you should expect that we have measures in place to identify fake injected data during the test set period and omit from evaluation. \n\nSecond, any team identified as having committed these acts to intentionally insert invalid purchases in the manner you describe would be considered in violation of the rules which prohibit attempts *\"...to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses...\"* Such teams will be disqualified and removed from the competition.", "replies": [ { "id": 413306, "postDate": "2018-10-31T16:53:27.920Z", "content": "

Glad to hear this!\nJust to remind: make sure your measure also covers true booking where competitors pretend as a previous visitor. To me it seems hard, so please share with me after the competition! I am a field-user of GA, and love to learn such a method too.

", "rawMarkdown": "Glad to hear this!\nJust to remind: make sure your measure also covers **true** booking where competitors pretend as a previous visitor. To me it seems hard, so please share with me after the competition! I am a field-user of GA, and love to learn such a method too.", "votes": 2 } ] }, { "id": 413140, "postDate": "2018-10-31T11:01:37.207Z", "content": "

@philculliton,\nAny comments about this?\nIf this is true, this can wreck the new rules =/

\n\n

@Comsaint says he found a way to insert transactions to the data posing as a different CustomerID.\nWhich allows people to create data to the final future unseen test set.

", "rawMarkdown": "@philculliton,\nAny comments about this?\nIf this is true, this can wreck the new rules =/\n\n@Comsaint says he found a way to insert transactions to the data posing as a different CustomerID.\nWhich allows people to create data to the final future unseen test set.\n", "replies": [ { "id": 413203, "postDate": "2018-10-31T13:39:32.960Z", "content": "

I updated my post to add a way to pretend and make real booking, which I believe is even harder to detect.

", "rawMarkdown": "I updated my post to add a way to pretend and make real booking, which I believe is even harder to detect." } ] }, { "id": 410768, "postDate": "2018-10-26T16:18:52.847Z", "content": "

But to win the money, you must share your code. It will be apparent you’ve cheated by hard coding predictions. But maybe less obvious if you overfit in the “right” way.

\n\n

Easy solution for organisers. Hold-out data, or an unnannounced date range to separate real winners from the overfit solutions.

\n\n

For example, if they suspect false purchases made by users in October, once they announced the rule change, just exclude the users from the submission list. Or subset the users to the day before they announced the rule change. To be clear, we are asked to predict revenue from repeat users, and all new users (after 15/Oct) are excluded.

\n\n

PS I really liked your analysis. Loads of insights into how the comp and GA works!

", "rawMarkdown": "But to win the money, you must share your code. It will be apparent you’ve cheated by hard coding predictions. But maybe less obvious if you overfit in the “right” way.\n\nEasy solution for organisers. Hold-out data, or an unnannounced date range to separate real winners from the overfit solutions. \n\nFor example, if they suspect false purchases made by users in October, once they announced the rule change, just exclude the users from the submission list. Or subset the users to the day before they announced the rule change. To be clear, we are asked to predict revenue from repeat users, and all new users (after 15/Oct) are excluded.\n\nPS I really liked your analysis. Loads of insights into how the comp and GA works!", "replies": [ { "id": 410853, "postDate": "2018-10-26T18:54:59.083Z", "content": "

Hard-coding is a stupid way to do this, for sure (much like shouting \"please catch me\"). To be honest, I do not know how to overfit a model with extra data without raising suspicions (the education system did not teach that, shame). Anyone has an idea?

\n\n
\n

if they suspect false purchases made by users in October

\n
\n\n

This is the root of problem - how do you tell if a purchase is false or not? I can build a model, take one cid that has a 30% chance to convert, and make a real purchase in the store with it. How do you tell if it was me who did it, or the real person (remember, I can pretend to have same geo-location/IP/channel etc.)?

\n\n
\n

Hold-out data

\n
\n\n

The whole test set is essentially hold-out now (as it happens in future). The problem is I can choose to insert lots of fake purchases in the test set (which is easy to fix, see my solution in post), or stay low and make a few true bookings with a couple IDs that only I know (or both!).

\n\n
\n

an unannounced date range to separate real winners from the overfit solutions

\n
\n\n

Good reminder. I shall remember to inject some purchases before and after test period, and make sure my model knows them in advance. Let's see how long before the organizer tired out... But yes, you caught me - I did not see this option.

\n\n

Just kidding. It is not possible to have another date range other than 2018 Dec and 2019 Jan, simply because everyone's models are predicting for this period. 2019 Feb does not have a Christmas, and 2019 Mar will not have a New Year's Day.

\n\n
\n

To be clear, we are asked to predict revenue from repeat users, and all new users (after 15/Oct) are excluded.

\n
\n\n

And what I have shown is, anyone can pretend as a user who existed before 15/Oct, and make (fake or real) purchases in future. Sorry, but please read my post again. I will make a few edit to my post to emphasis that.

", "rawMarkdown": "Hard-coding is a stupid way to do this, for sure (much like shouting \"please catch me\"). To be honest, I do not know how to overfit a model with extra data without raising suspicions (the education system did not teach that, shame). Anyone has an idea?\n\n> if they suspect false purchases made by users in October\n\nThis is the root of problem - how do you tell if a purchase is false or not? I can build a model, take one `cid` that has a 30% chance to convert, and make a real purchase in the store with it. How do you tell if it was me who did it, or the real person (remember, I can pretend to have same geo-location/IP/channel etc.)?\n\n> Hold-out data\n\nThe whole test set is essentially hold-out now (as it happens in future). The problem is I can choose to insert lots of fake purchases in the test set (which is easy to fix, see my solution in post), *or stay low and make a few true bookings with a couple IDs that only I know* (or both!). \n\n> an unannounced date range to separate real winners from the overfit solutions\n\nGood reminder. I shall remember to inject some purchases before and after test period, and make sure my model knows them in advance. Let's see how long before the organizer tired out... But yes, you caught me - I did not see this option.\n\nJust kidding. It is not possible to have another date range other than 2018 Dec and 2019 Jan, simply because everyone's models are predicting for this period. 2019 Feb does not have a Christmas, and 2019 Mar will not have a New Year's Day.\n\n> To be clear, we are asked to predict revenue from repeat users, and all new users (after 15/Oct) are excluded.\n\nAnd what I have shown is, anyone can pretend as a user who existed **before** 15/Oct, and make (fake or real) purchases in future. Sorry, but please read my post again. I will make a few edit to my post to emphasis that.\n", "votes": 1 }, { "id": 410928, "postDate": "2018-10-26T23:02:39.213Z", "content": "

@comsaint These flaws in the competition move me to believe that the competition should be closed and a different dataset should be chosen. Recall the goal of the competition is to showcase data analysis skills using R (to promote RStudio). They should quit the Google Merchandise data and take a different dataset.

\n\n

Sorry, everyone's time will be lost building models here, but it's the right thing to do because the test dataset is subject to attack using the methods you've detailed.

", "rawMarkdown": "@comsaint These flaws in the competition move me to believe that the competition should be closed and a different dataset should be chosen. Recall the goal of the competition is to showcase data analysis skills using R (to promote RStudio). They should quit the Google Merchandise data and take a different dataset.\n\nSorry, everyone's time will be lost building models here, but it's the right thing to do because the test dataset is subject to attack using the methods you've detailed.", "votes": 1 }, { "id": 410948, "postDate": "2018-10-27T01:47:24.897Z", "content": "

@CoreyLevinson I have put myself in the shoes of organizer but cannot find a way to rectify, except using a new data set.

\n\n

On the other hand, things are not that bad - as far as they can find another GA profile, the schema would be identical. So we can reuse part of the work & knowledge at least.

", "rawMarkdown": "@CoreyLevinson I have put myself in the shoes of organizer but cannot find a way to rectify, except using a new data set.\n\nOn the other hand, things are not that bad - as far as they can find another GA profile, the schema would be identical. So we can reuse part of the work & knowledge at least.", "votes": 1 }, { "id": 411064, "postDate": "2018-10-27T10:30:38.967Z", "content": "

You are cunning!

\n\n

So there’s no way to detect a spoofed clientId? That’s whack

\n\n

To overfit, just do a tonne of feature engineering, and assign a higher weight to the spoofed sessions. This will help you isolate the features that separate the higher weights. You can then use these features in a production model. Probably very hard in practice, and a totally insane thing to do with your time, even with the prize money!

\n\n

Another solution for organisers: a beauty pageant for solutions, with the score as a qualifier.

", "rawMarkdown": "You are cunning! \n\nSo there’s no way to detect a spoofed clientId? That’s whack\n\nTo overfit, just do a tonne of feature engineering, and assign a higher weight to the spoofed sessions. This will help you isolate the features that separate the higher weights. You can then use these features in a production model. Probably very hard in practice, and a totally insane thing to do with your time, even with the prize money!\n\nAnother solution for organisers: a beauty pageant for solutions, with the score as a qualifier.\n", "votes": -2 }, { "id": 411106, "postDate": "2018-10-27T12:34:36.270Z", "content": "

Money can make people do things.

\n\n
\n

Another solution for organisers: a beauty pageant for solutions, with the score as a qualifier.

\n
\n\n

Sorry I do not get what you mean by this. Mind to elaborate?

", "rawMarkdown": "Money can make people do things.\n\n> Another solution for organisers: a beauty pageant for solutions, with the score as a qualifier.\n\nSorry I do not get what you mean by this. Mind to elaborate?" }, { "id": 411167, "postDate": "2018-10-27T14:34:58.033Z", "content": "

Take the top 20 high scoring entries, then look at all their code. Which is the most elegant solution?

", "rawMarkdown": "Take the top 20 high scoring entries, then look at all their code. Which is the most elegant solution?", "votes": 1 }, { "id": 411183, "postDate": "2018-10-27T15:07:38.573Z", "content": "

No more words with you, Burrito Dan. Sorry.

", "rawMarkdown": "No more words with you, Burrito Dan. Sorry.", "votes": 1 } ] }, { "id": 409997, "postDate": "2018-10-25T07:15:02.323Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 410001, "postDate": "2018-10-25T07:24:03.773Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 410000, "author_name": "AmirH", "author_url": "", "post_date": "2018-10-25T07:21:22.923000", "content": "

Hi @Comasaint,\nI wrote a comment earlier before reading the entire post.\nI deleted it as it was irrelevant to the main point you were trying to convey.

\n\n

Indeed if someone can, during the future test period, insert purchases as other, existing visitorIDs - that is very bad.

", "votes": 1, "replies": [ { "id": 410003, "author_name": "comsaint", "author_url": "", "post_date": "2018-10-25T07:25:25.307000", "content": "

@AmirH - No problem, I also delete my reply. Thanks!

", "votes": 1, "replies": [] } ] }, { "id": 417312, "author_name": "Shelia Xin", "author_url": "", "post_date": "2018-11-08T04:34:31.867000", "content": "

Thx @comsaint! Learned a lot about GA from your post.

", "votes": 0, "replies": [] }, { "id": 413294, "author_name": "Julia Elliott", "author_url": "", "post_date": "2018-10-31T16:34:51.670000", "content": "

Thank you for detailing your concerns. We appreciate the transparency!

\n\n

First, you should expect that we have measures in place to identify fake injected data during the test set period and omit from evaluation.

\n\n

Second, any team identified as having committed these acts to intentionally insert invalid purchases in the manner you describe would be considered in violation of the rules which prohibit attempts \"...to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses...\" Such teams will be disqualified and removed from the competition.

", "votes": 0, "replies": [ { "id": 413306, "author_name": "comsaint", "author_url": "", "post_date": "2018-10-31T16:53:27.920000", "content": "

Glad to hear this!\nJust to remind: make sure your measure also covers true booking where competitors pretend as a previous visitor. To me it seems hard, so please share with me after the competition! I am a field-user of GA, and love to learn such a method too.

", "votes": 2, "replies": [] } ] }, { "id": 413140, "author_name": "AmirH", "author_url": "", "post_date": "2018-10-31T11:01:37.207000", "content": "

@philculliton,\nAny comments about this?\nIf this is true, this can wreck the new rules =/

\n\n

@Comsaint says he found a way to insert transactions to the data posing as a different CustomerID.\nWhich allows people to create data to the final future unseen test set.

", "votes": 0, "replies": [ { "id": 413203, "author_name": "comsaint", "author_url": "", "post_date": "2018-10-31T13:39:32.960000", "content": "

I updated my post to add a way to pretend and make real booking, which I believe is even harder to detect.

", "votes": 0, "replies": [] } ] }, { "id": 410768, "author_name": "Burrito Dan", "author_url": "", "post_date": "2018-10-26T16:18:52.847000", "content": "

But to win the money, you must share your code. It will be apparent you’ve cheated by hard coding predictions. But maybe less obvious if you overfit in the “right” way.

\n\n

Easy solution for organisers. Hold-out data, or an unnannounced date range to separate real winners from the overfit solutions.

\n\n

For example, if they suspect false purchases made by users in October, once they announced the rule change, just exclude the users from the submission list. Or subset the users to the day before they announced the rule change. To be clear, we are asked to predict revenue from repeat users, and all new users (after 15/Oct) are excluded.

\n\n

PS I really liked your analysis. Loads of insights into how the comp and GA works!

", "votes": 0, "replies": [ { "id": 410853, "author_name": "comsaint", "author_url": "", "post_date": "2018-10-26T18:54:59.083000", "content": "

Hard-coding is a stupid way to do this, for sure (much like shouting \"please catch me\"). To be honest, I do not know how to overfit a model with extra data without raising suspicions (the education system did not teach that, shame). Anyone has an idea?

\n\n
\n

if they suspect false purchases made by users in October

\n
\n\n

This is the root of problem - how do you tell if a purchase is false or not? I can build a model, take one cid that has a 30% chance to convert, and make a real purchase in the store with it. How do you tell if it was me who did it, or the real person (remember, I can pretend to have same geo-location/IP/channel etc.)?

\n\n
\n

Hold-out data

\n
\n\n

The whole test set is essentially hold-out now (as it happens in future). The problem is I can choose to insert lots of fake purchases in the test set (which is easy to fix, see my solution in post), or stay low and make a few true bookings with a couple IDs that only I know (or both!).

\n\n
\n

an unannounced date range to separate real winners from the overfit solutions

\n
\n\n

Good reminder. I shall remember to inject some purchases before and after test period, and make sure my model knows them in advance. Let's see how long before the organizer tired out... But yes, you caught me - I did not see this option.

\n\n

Just kidding. It is not possible to have another date range other than 2018 Dec and 2019 Jan, simply because everyone's models are predicting for this period. 2019 Feb does not have a Christmas, and 2019 Mar will not have a New Year's Day.

\n\n
\n

To be clear, we are asked to predict revenue from repeat users, and all new users (after 15/Oct) are excluded.

\n
\n\n

And what I have shown is, anyone can pretend as a user who existed before 15/Oct, and make (fake or real) purchases in future. Sorry, but please read my post again. I will make a few edit to my post to emphasis that.

", "votes": 1, "replies": [] }, { "id": 410928, "author_name": "CoreyJamesLevinson", "author_url": "", "post_date": "2018-10-26T23:02:39.213000", "content": "

@comsaint These flaws in the competition move me to believe that the competition should be closed and a different dataset should be chosen. Recall the goal of the competition is to showcase data analysis skills using R (to promote RStudio). They should quit the Google Merchandise data and take a different dataset.

\n\n

Sorry, everyone's time will be lost building models here, but it's the right thing to do because the test dataset is subject to attack using the methods you've detailed.

", "votes": 1, "replies": [] }, { "id": 410948, "author_name": "comsaint", "author_url": "", "post_date": "2018-10-27T01:47:24.897000", "content": "

@CoreyLevinson I have put myself in the shoes of organizer but cannot find a way to rectify, except using a new data set.

\n\n

On the other hand, things are not that bad - as far as they can find another GA profile, the schema would be identical. So we can reuse part of the work & knowledge at least.

", "votes": 1, "replies": [] }, { "id": 411064, "author_name": "Burrito Dan", "author_url": "", "post_date": "2018-10-27T10:30:38.967000", "content": "

You are cunning!

\n\n

So there’s no way to detect a spoofed clientId? That’s whack

\n\n

To overfit, just do a tonne of feature engineering, and assign a higher weight to the spoofed sessions. This will help you isolate the features that separate the higher weights. You can then use these features in a production model. Probably very hard in practice, and a totally insane thing to do with your time, even with the prize money!

\n\n

Another solution for organisers: a beauty pageant for solutions, with the score as a qualifier.

", "votes": -2, "replies": [] }, { "id": 411106, "author_name": "comsaint", "author_url": "", "post_date": "2018-10-27T12:34:36.270000", "content": "

Money can make people do things.

\n\n
\n

Another solution for organisers: a beauty pageant for solutions, with the score as a qualifier.

\n
\n\n

Sorry I do not get what you mean by this. Mind to elaborate?

", "votes": 0, "replies": [] }, { "id": 411167, "author_name": "Burrito Dan", "author_url": "", "post_date": "2018-10-27T14:34:58.033000", "content": "

Take the top 20 high scoring entries, then look at all their code. Which is the most elegant solution?

", "votes": 1, "replies": [] }, { "id": 411183, "author_name": "comsaint", "author_url": "", "post_date": "2018-10-27T15:07:38.573000", "content": "

No more words with you, Burrito Dan. Sorry.

", "votes": 1, "replies": [] } ] }, { "id": 409997, "author_name": "", "author_url": "", "post_date": "2018-10-25T07:15:02.323000", "content": "", "votes": 0, "replies": [ { "id": 410001, "author_name": "", "author_url": "", "post_date": "2018-10-25T07:24:03.773000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "409989": "Greetings. \n\nAfter previous data leak, the organizer updated the rules of the competition, which are greeted warmly by the community in general. However, I see two potential problems with the new setting:\n\n 1. As the ID of the demo GA profile is leaked, one can send fake PV/ transactions to it anytime, polluting it to s/he favor.\n 2. The statement by organizer,\n\n> The Google Merchandise Store Demo Account effectively becomes freely available to use public data, but will no longer provide any additional benefit to final standings.\n\nis plainly false.\n\nLet's start with #1.\n\n## Problem 1: Inject fake booking (with `fid`s already exist in before Oct-15)\n\nA bit of background knowledge - the provided data we have come from Google Analytics, which collects data on the website and exports to Google BigQuery. Google Analytics collects data via a piece of JS ('tag') implemented on website, which passes information via standard HTTP POST calls.\n\nOur data set contains a field `fullVisitorId` (available in BigQuery's schema), or `fid`, which is a hashed version of the client ID (`cid`) used by GA's cookie. In [this][1] kernel, I have shown a way to match an `fid` with a `cid`, thus will not elaborate here.\n\nNow the crucial part: given a GA profile ID (`UA-54516992-1` in our case) and a `cid`, one can generate any fake user behaviors and purchases. \n\nFirstly, some references from Google:\n\n 1. [Document][2] on the protocol used by GA to pass information from website to server\n\n 2. [A handy tool][3] to generate valid calls to GA\n\nI am going to skip the details on the parameters used in my calls, but simply attaching the screenshots of my setting and result. Please refer to the attachments, where I generate a booking which can be observed in the demo account. You can get creative and forge any custom call with fake geo-location/channel/page path/purchase etc. by referencing the the protocol above.\n\nThe implication is that everyone can **fake purchases by past visitors** during evaluation period, as far as they have the correct GA profile ID and `cid`. \n\n### Notes:\n\nLet's revisit the assumptions carefully:\n\n1) Is the profile ID `UA-54516992-1` really the ID used in this competition?\n\nA: I do not know, but guess not. Checking the [site][4], I can see a number of profile IDs on it, and the organizer can be using any one of them. However, likely the 'true' profile is among them (well, data have to come from somewhere), so we can simply send the same web calls to all profiles.\n\n2) If the profile ID is different, is `cid` consistent across profiles?\n\nA: I think so. In my work we manage lots of websites, and the `cid` is consistent across different profiles (even across GA accounts). Likely GA is using same cookie for different sites, which is quite logical.\n\n### Possible solutions\n\nFinally, solution I can think of (any ideas?):\n\n**Solution 1** - Validate all bookings during test period. This is the most obvious solution, checking with real-world to see if a purchase really happens. \n\nHowever, there is still a problem: given purchases are rare and the limited predictability of current setting (predicting revenue of far future), one can edit their own cookie to pretend as any `cid`, make a purchase as normal customer, and s/he will immediately gain a good advantage over all competitors (someone volunteers to do the math for the bias of a $20 purchase?). This is further encouraged by the monetary prize of competition. (And we have not touched on refund policy yet...)\n\nLet's pretend as an existing `cid` and do something. \n\n 1. Firstly, we need a `cid` to begin with. Let's pick **952749702.1498272983** as our `cid`, which is also known to be `fid` **2658678838785413275** (refer to [my notebook][1]), as seen in the training set.\n\n 2. Enters the shop at https://shop.googlemerchandisestore.com/. If you are interested, you can explore the cookies in browser.\n\n 3. Delete all existing cookies on the page (with a cookie editor for example).\n\n 4. Enters developer view (in Chrome, press F12. Check the setting of your favorite browser), and go to \"console\" tab (the one you can type Javascript code).\n\n 5. Create a new GA tracker by entering the following command:\n\n `ga.create('UA-545169921',{'clientId':'952749702.1498272983'});`\n\n Note that `clientId` cannot be altered after GA tracker is created, so it needs to be set now. You can pass additional information e.g. source/medium, following the [protocol][2].\n\n 6. Refresh the page and GA will recognize you as `cid` **952749702.1498272983**. You can verify by entering the following command in console:\n\n `for (i=0; i", "410000": "Hi @Comasaint,\nI wrote a comment earlier before reading the entire post.\nI deleted it as it was irrelevant to the main point you were trying to convey.\n\nIndeed if someone can, during the future test period, insert purchases as other, existing visitorIDs - that is very bad.", "417312": "Thx @comsaint! Learned a lot about GA from your post.", "413294": "Thank you for detailing your concerns. We appreciate the transparency!\n\nFirst, you should expect that we have measures in place to identify fake injected data during the test set period and omit from evaluation. \n\nSecond, any team identified as having committed these acts to intentionally insert invalid purchases in the manner you describe would be considered in violation of the rules which prohibit attempts *\"...to undermine the legitimate operation of the Competition by cheating, deception, or other unfair playing practices or abuses...\"* Such teams will be disqualified and removed from the competition.", "413140": "@philculliton,\nAny comments about this?\nIf this is true, this can wreck the new rules =/\n\n@Comsaint says he found a way to insert transactions to the data posing as a different CustomerID.\nWhich allows people to create data to the final future unseen test set.\n", "410768": "But to win the money, you must share your code. It will be apparent you’ve cheated by hard coding predictions. But maybe less obvious if you overfit in the “right” way.\n\nEasy solution for organisers. Hold-out data, or an unnannounced date range to separate real winners from the overfit solutions. \n\nFor example, if they suspect false purchases made by users in October, once they announced the rule change, just exclude the users from the submission list. Or subset the users to the day before they announced the rule change. To be clear, we are asked to predict revenue from repeat users, and all new users (after 15/Oct) are excluded.\n\nPS I really liked your analysis. Loads of insights into how the comp and GA works!", "409997": "" } }