{
  "id": 8328,
  "title": "Not able to regenerate benchmark model",
  "url": "/competitions/acquire-valued-shoppers-challenge/discussion/8328",
  "author_name": "",
  "post_date": "2014-05-30T10:26:13.470Z",
  "votes": 1,
  "comment_count": 15,
  "views": 3812,
  "content": "<p>Hi I am trying to regenerate Prior (Brand &amp; Company &amp; Category) Benchmark</p>\n\n<p>I marked cases having atleast one transaction involving same brand company, category in offer as 1.</p>\n<p>I am still not able to match benchmark score</p>\n\n<p>Is there anything I am missing?</p>",
  "messages": [
    {
      "id": "47324",
      "postDate": "05/30/2014 10:26:13",
      "content": "<p>Hi I am trying to regenerate Prior (Brand &amp; Company &amp; Category) Benchmark</p>\n\n<p>I marked cases having atleast one transaction involving same brand company, category in offer as 1.</p>\n<p>I am still not able to match benchmark score</p>\n\n<p>Is there anything I am missing?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "47326",
      "postDate": "05/30/2014 11:09:31",
      "content": "<p>May be you can try to check yourself by count of nonzeros.</p>\n<p>How many nonzero values&nbsp;do you have for test set?</p>\n<p>When I calculated this I get&nbsp;25431 &quot;1&quot; records.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "47334",
      "postDate": "05/30/2014 12:39:18",
      "content": "<p>Thanks Monkey, i am getting only 21206.</p>\n<p>I have put transactions.csv in database with index on id field</p>\n<p>All i am doing is for every id in test file, i am retrieving all cases from transactions with that id.</p>\n<p>If any transaction has same brand,category and company, i am marking it 1</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "47336",
      "postDate": "05/30/2014 13:06:47",
      "content": "<p>Looks perfectly correct, try also count how many different keys (users) you have (should be about 310000 train + test), how many records you have overall (should be about 350 * 10^6)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "48467",
      "postDate": "06/02/2014 08:30:45",
      "content": "<p>Hi Monkey data in database seems alright. I have 349600001 entries in transactions and 311265 unique Ids in transactions file.</p>\n<p>I am debugging matching logic now</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "48468",
      "postDate": "06/02/2014 08:43:09",
      "content": "<p>You can also post your code here if you want. If I will understand what may cause match problem, I will point it.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "48760",
      "postDate": "06/06/2014 07:23:51",
      "content": "<p>Hi Monkey</p>\n\n<p>I reached benchmark finally!!</p>\n\n<p>There were two issues, some offers were not loading properly in my code and some data wasnt indexed in database.</p>\n\n<p>NUmbers you gave were extremely important for me to debug this issue.</p>\n\n<p>Thanks for helping out</p>\n\n\n\n<p>Premraj</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "48761",
      "postDate": "06/06/2014 08:41:50",
      "content": "<p>I'm glad for your advance)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49160",
      "postDate": "06/15/2014 18:07:27",
      "content": "<p>I am also trying to replicate the prior category/company/brand benchmark, yet my score is higher than the benchmark score, which makes me concerned that there might be errors in my data aggregation. It's also possible that I misunderstood what the benchmark means.</p>\n\n<p>Basically for each shopper in the test data I look to see whether there is at least one transaction whose category, company, and brand all match the category, company, and brand of the offer made to that shopper. I found 45265 such shoppers in the test data, a lot higher than 25431 in MonkeyPatching's post.</p>\n\n<p>I was able to replicate the prior brand benchmark though. My count of shoppers in the test data with at least one transaction of the same brand is 65050.</p>\n\n<p>Am I missing something?&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49184",
      "postDate": "06/16/2014 07:35:44",
      "content": "<p>I think you can double check category and company columns handling because I also get 65050 rows when counted prior brand baseline</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49196",
      "postDate": "06/16/2014 12:27:54",
      "content": "<p>For category I get 123563 nonzero rows, for company I get 121843 nonzero rows. The scores for both match the corresponding benchmark scores.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49202",
      "postDate": "06/16/2014 15:05:25",
      "content": "<p>nw3097, it could be some problem either in your if-condition or exiting loop when counting, because the 45265 shoppers you find are quite close to the total count of customer transactions satisfying the 3 conditions at once:&nbsp; 46055.</p>\n\n<p>What I mean with &quot;total count of customer transactions&quot; is just that if customer A has bought once an item from the given company brand and category and customer B has bought twice an item from the given company brand and category then &quot;total count&quot; would be 3, but to get to the 25431 you should set the count of these&nbsp; 2 customers to 2 (i.e a boolean: bought once or more--&gt; count=1 else count=0)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49206",
      "postDate": "06/16/2014 15:38:41",
      "content": "<p>I think I know where I went wrong. 45265 is the number of shoppers who have made positive purchases in the same category, company, and brand overall, but it doesn't follow that they all have at least one individual transaction that satisfies all three conditions. For example, if someone has three transactions in total, one with the right category, one with the right company, and one with the right brand, I would have included him, which I shouldn't for the benchmark.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49918",
      "postDate": "07/01/2014 17:47:55",
      "content": "<p>I'm also having trouble replicating the prior brand+category+company benchmark.</p>\n\n<p>This thread was helpful in diagnosing one issue (I initially made the same error&nbsp;as nw3097), but now I believe I am correctly calculating whether each customer has made a prior transaction for the same category+company+brand (in a single transaction) as the&nbsp;offer. My submission vector has 25431 &quot;1&quot; records. But when I submitted it, I got a score of 0.49815 instead of the expected 0.56619.</p>\n\n<p>I verified that the id values are the same as in the testHistory file and in the same order. My submission file starts out</p>\n\n<p><code>id,repeatProbability<br>12262064,0<br>12277270,0<br>12332190,1<br>12524696,0<br>13074629,0<br>13387341,0<br>13501141,0<br>13558712,0<br>13563017,0<br></code></p>\n\n<p>What am I misunderstanding about this competition's scoring?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49937",
      "postDate": "07/02/2014 06:18:02",
      "content": "<p>Lokks like you have problem with count from zero and from1 or something like that. I say this because my file for the benchmark looks like:</p>\n<p>id,repeatProbability<br> 12262064, 0.00<br> 12277270, 1.00<br> 12332190, 0.00<br> 12524696, 0.00<br> 13074629, 0.00<br> 13387341, 0.00<br> 13501141, 0.00<br> 13558712, 0.00<br> 13563017, 0.00<br> 13584134, 0.00<br> 13744500, 1.00</p>\n\n<p>i.e. the customer before yours is a repeater...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "49940",
      "postDate": "07/02/2014 06:52:07",
      "content": "<p>Awesome, thanks! Indeed, it appears to be an off-by-one error in an index.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 47326,
      "author_name": "dremovd",
      "author_url": "",
      "post_date": "05/30/2014 11:09:31",
      "content": "<p>May be you can try to check yourself by count of nonzeros.</p>\n<p>How many nonzero values&nbsp;do you have for test set?</p>\n<p>When I calculated this I get&nbsp;25431 &quot;1&quot; records.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 47334,
      "author_name": "premrajnarkhede",
      "author_url": "",
      "post_date": "05/30/2014 12:39:18",
      "content": "<p>Thanks Monkey, i am getting only 21206.</p>\n<p>I have put transactions.csv in database with index on id field</p>\n<p>All i am doing is for every id in test file, i am retrieving all cases from transactions with that id.</p>\n<p>If any transaction has same brand,category and company, i am marking it 1</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 47336,
      "author_name": "dremovd",
      "author_url": "",
      "post_date": "05/30/2014 13:06:47",
      "content": "<p>Looks perfectly correct, try also count how many different keys (users) you have (should be about 310000 train + test), how many records you have overall (should be about 350 * 10^6)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 48467,
      "author_name": "premrajnarkhede",
      "author_url": "",
      "post_date": "06/02/2014 08:30:45",
      "content": "<p>Hi Monkey data in database seems alright. I have 349600001 entries in transactions and 311265 unique Ids in transactions file.</p>\n<p>I am debugging matching logic now</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 48468,
      "author_name": "dremovd",
      "author_url": "",
      "post_date": "06/02/2014 08:43:09",
      "content": "<p>You can also post your code here if you want. If I will understand what may cause match problem, I will point it.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 48760,
      "author_name": "premrajnarkhede",
      "author_url": "",
      "post_date": "06/06/2014 07:23:51",
      "content": "<p>Hi Monkey</p>\n\n<p>I reached benchmark finally!!</p>\n\n<p>There were two issues, some offers were not loading properly in my code and some data wasnt indexed in database.</p>\n\n<p>NUmbers you gave were extremely important for me to debug this issue.</p>\n\n<p>Thanks for helping out</p>\n\n\n\n<p>Premraj</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 48761,
      "author_name": "dremovd",
      "author_url": "",
      "post_date": "06/06/2014 08:41:50",
      "content": "<p>I'm glad for your advance)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49160,
      "author_name": "nw3097",
      "author_url": "",
      "post_date": "06/15/2014 18:07:27",
      "content": "<p>I am also trying to replicate the prior category/company/brand benchmark, yet my score is higher than the benchmark score, which makes me concerned that there might be errors in my data aggregation. It's also possible that I misunderstood what the benchmark means.</p>\n\n<p>Basically for each shopper in the test data I look to see whether there is at least one transaction whose category, company, and brand all match the category, company, and brand of the offer made to that shopper. I found 45265 such shoppers in the test data, a lot higher than 25431 in MonkeyPatching's post.</p>\n\n<p>I was able to replicate the prior brand benchmark though. My count of shoppers in the test data with at least one transaction of the same brand is 65050.</p>\n\n<p>Am I missing something?&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49184,
      "author_name": "dremovd",
      "author_url": "",
      "post_date": "06/16/2014 07:35:44",
      "content": "<p>I think you can double check category and company columns handling because I also get 65050 rows when counted prior brand baseline</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49196,
      "author_name": "nw3097",
      "author_url": "",
      "post_date": "06/16/2014 12:27:54",
      "content": "<p>For category I get 123563 nonzero rows, for company I get 121843 nonzero rows. The scores for both match the corresponding benchmark scores.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49202,
      "author_name": "sandros",
      "author_url": "",
      "post_date": "06/16/2014 15:05:25",
      "content": "<p>nw3097, it could be some problem either in your if-condition or exiting loop when counting, because the 45265 shoppers you find are quite close to the total count of customer transactions satisfying the 3 conditions at once:&nbsp; 46055.</p>\n\n<p>What I mean with &quot;total count of customer transactions&quot; is just that if customer A has bought once an item from the given company brand and category and customer B has bought twice an item from the given company brand and category then &quot;total count&quot; would be 3, but to get to the 25431 you should set the count of these&nbsp; 2 customers to 2 (i.e a boolean: bought once or more--&gt; count=1 else count=0)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49206,
      "author_name": "nw3097",
      "author_url": "",
      "post_date": "06/16/2014 15:38:41",
      "content": "<p>I think I know where I went wrong. 45265 is the number of shoppers who have made positive purchases in the same category, company, and brand overall, but it doesn't follow that they all have at least one individual transaction that satisfies all three conditions. For example, if someone has three transactions in total, one with the right category, one with the right company, and one with the right brand, I would have included him, which I shouldn't for the benchmark.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49918,
      "author_name": "brundage",
      "author_url": "",
      "post_date": "07/01/2014 17:47:55",
      "content": "<p>I'm also having trouble replicating the prior brand+category+company benchmark.</p>\n\n<p>This thread was helpful in diagnosing one issue (I initially made the same error&nbsp;as nw3097), but now I believe I am correctly calculating whether each customer has made a prior transaction for the same category+company+brand (in a single transaction) as the&nbsp;offer. My submission vector has 25431 &quot;1&quot; records. But when I submitted it, I got a score of 0.49815 instead of the expected 0.56619.</p>\n\n<p>I verified that the id values are the same as in the testHistory file and in the same order. My submission file starts out</p>\n\n<p><code>id,repeatProbability<br>12262064,0<br>12277270,0<br>12332190,1<br>12524696,0<br>13074629,0<br>13387341,0<br>13501141,0<br>13558712,0<br>13563017,0<br></code></p>\n\n<p>What am I misunderstanding about this competition's scoring?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49937,
      "author_name": "sandros",
      "author_url": "",
      "post_date": "07/02/2014 06:18:02",
      "content": "<p>Lokks like you have problem with count from zero and from1 or something like that. I say this because my file for the benchmark looks like:</p>\n<p>id,repeatProbability<br> 12262064, 0.00<br> 12277270, 1.00<br> 12332190, 0.00<br> 12524696, 0.00<br> 13074629, 0.00<br> 13387341, 0.00<br> 13501141, 0.00<br> 13558712, 0.00<br> 13563017, 0.00<br> 13584134, 0.00<br> 13744500, 1.00</p>\n\n<p>i.e. the customer before yours is a repeater...</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 49940,
      "author_name": "brundage",
      "author_url": "",
      "post_date": "07/02/2014 06:52:07",
      "content": "<p>Awesome, thanks! Indeed, it appears to be an off-by-one error in an index.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "47324": "",
    "47326": "",
    "47334": "",
    "47336": "",
    "48467": "",
    "48468": "",
    "48760": "",
    "48761": "",
    "49160": "",
    "49184": "",
    "49196": "",
    "49202": "",
    "49206": "",
    "49918": "",
    "49937": "",
    "49940": ""
  },
  "source": "meta"
}