{
  "id": 7665,
  "title": "Data in transactions.csv",
  "url": "/competitions/acquire-valued-shoppers-challenge/discussion/7665",
  "author_name": "",
  "post_date": "2014-04-11T14:08:33.653Z",
  "votes": null,
  "comment_count": 13,
  "views": 5479,
  "content": "<p>In file &quot;transactions.csv&quot; in&nbsp;column &quot;purchaseamount&quot; there are several negative values. What do they mean?</p>",
  "messages": [
    {
      "id": "41718",
      "postDate": "04/11/2014 14:08:33",
      "content": "<p>In file &quot;transactions.csv&quot; in&nbsp;column &quot;purchaseamount&quot; there are several negative values. What do they mean?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "41719",
      "postDate": "04/11/2014 14:09:47",
      "content": "<p>A negative value in productquantity and purchaseamount indicates a return</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "41720",
      "postDate": "04/11/2014 14:10:07",
      "content": "<p>Are all test users in testhistory present in transactions.csv?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "41721",
      "postDate": "04/11/2014 14:36:55",
      "content": "<p>Did I get right that &quot;purchaseamount&quot; == 0 means that product is a present for buying another one?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "41722",
      "postDate": "04/11/2014 14:57:46",
      "content": "<p>[quote=Abhishek;41720]</p>\n<p>Are all test users in testhistory present in transactions.csv?</p>\n<p>[/quote]</p>\n<p>Questions like these lead to&nbsp;heart palpitations for us poor admins, causing us to switch from what we're doing to double check that the data isn't broken. Did you make an honest attempt to answer this yourself before asking? I checked using this method:</p>\n<p>gzcat transactions.csv.gz | cut -d, -f1 | uniq | sort -g &gt; somefile<br>gzcat testHistory.csv.gz | cut -d, -f1 | uniq | sort -g &gt; someotherfile</p>\n<p>(the rest is left as an exercise for the reader). There's probably a python one-liner to do it as well. My point is, try first, then ask. If you find a problem, post it to the forums and be specific about why you think it's a problem. If you cry wolf, we eventually have to ignore forum questions because of the context switching costs.</p>\n<p>This is not to say we don't appreciate when you guys and gals catch our errors. We Just ask that you make an honest effort to answer questions yourself first, and share&nbsp;evidence when you do find something wrong.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "41723",
      "postDate": "04/11/2014 15:00:54",
      "content": "<p>Yeah. got that. There was something wrong with what I was doing and was getting less number ;) . thanks anyways</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "41789",
      "postDate": "04/12/2014 16:05:14",
      "content": "<p>After looking at the min and max values of the variables, except the date, I have several questions.</p>\n<p>1. the min for dept, category, brand, and purchase size is 0. Does 0 in this case mean that the real value is unknown? Or it mean that there actually is a brand 0, dept 0, category 0, and productsize 0? 0 productsize for example could represent a service offered.</p>\n<p>2. are the extreme values of purchasequantity and purchaseamount valid?&nbsp;</p>\n<p>3. would it be possible for the competition admins to provide the min max values of the variables?</p>\n<p>4. did the competition admins add noise to anonymize the data?</p>\n<p>VAR &nbsp; MIN &nbsp; &nbsp;MAX</p>\n<p>id &nbsp;86246 4853598737</p>\n<p>chain 2 526</p>\n<p>dept 0 99&nbsp;</p>\n<p>category 0 9999&nbsp;</p>\n<p>company 10000 10999999999&nbsp;</p>\n<p>brand 0 108689&nbsp;</p>\n<p>productsize 0 6000&nbsp;</p>\n<p>purchasequantity -32255.00 54800.00&nbsp;</p>\n<p>purchaseamount -8593791.00 58658.76&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "41965",
      "postDate": "04/15/2014 20:14:58",
      "content": "<p>Q1:&nbsp; 0 = missing value</p>\n\n<p>Q2:&nbsp; These values have been observed in the data</p>\n\n<p>Q3:&nbsp; No</p>\n\n<p>Q4:&nbsp; No</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42601",
      "postDate": "04/21/2014 14:19:44",
      "content": "<p>So brand 0 means brand is unknown or missing? &nbsp;Similar for category and department?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42609",
      "postDate": "04/21/2014 15:40:08",
      "content": "<p>Is &quot;10000&quot; the code for an unknown or missing company?</p>\n<p>(I only started analyzing the data. But for brand 0 and company&nbsp;10000 I get very strange data.)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "42658",
      "postDate": "04/22/2014 00:49:01",
      "content": "<p>[quote=Leo Buettiker;42609]</p>\n<p>Is &quot;10000&quot; the code for an unknown or missing company?</p>\n<p>(I only started analyzing the data. But for brand 0 and company&nbsp;10000 I get very strange data.)</p>\n<p>[/quote]</p>\n\n<p>For brand and category, 0 seems to mean missing.</p>\n\n<p>For company, I'm not sure, but &quot;10000&quot; does seem odd</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "43106",
      "postDate": "04/26/2014 22:05:44",
      "content": "<p>Odd in what way? In terms of occurrence frequency it's 2nd from top, but there are many other high frequency IDs in the same ballpark.</p>\n<p>What's more odd is the number of company IDs with very low frequencies, presumably these are products local to a specific store or town and not widely available?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "43108",
      "postDate": "04/26/2014 23:19:28",
      "content": "<p>Are you sure 0 = missing for dept? The others have IDs much higher than zero, and also zero, whereas dept has a continuous span of IDs starting from zero. Also, looking at the frequency of transactions for each dept ID, zero isn't an outlier at all.</p>\n<p>Think I'll assume for now that dept 0 is a real 'dept' (also, strange choice of field names in this data huh).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "44385",
      "postDate": "05/11/2014 23:28:47",
      "content": "<p>There seem to be transactions for which :</p>\n<p>a) purchasequantity is positive, but purchaseamount is zero.</p>\n<p>1000714152,46,35,3509,103320030,875,2013-01-25,60.5,OZ,1,0</p>\n<p>b) purchasequantity negative, but purchaseamount is zero.</p>\n<p>122307580,4,22,2211,103700030,2246,2013-01-08,120,CT,-1,0</p>\n<p>c) purchasequantity is zero, but&nbsp;purchaseamount is positive.</p>\n<p>1012000746,214,3,305,103320030,875,2012-10-13,8,OZ,0,0.96</p>\n<p>d) purchasequantity is zero, but purchaseamount is negative.</p>\n<p>100017875,3,26,2614,103700030,514,2012-08-15,60,CT,0,-0.7</p>\n<p>e) Both zero</p>\n<p>100084808,20,69,6901,103700030,16139,2012-06-26,0.5,OZ,0,0</p>\n<p>f) purchasequantity is negative, but purchaseamount is positive.</p>\n<p>1050021843,214,41,4109,105100050,2820,2012-08-06,11,OZ,-4,1.6</p>\n<p>g) purchasequantity is positive, but purchaseamount is negative.</p>\n<p>100022923,95,26,2628,103700030,2248,2012-05-29,2,RL,1,-14.82</p>\n\n<p>Is there any explanation for these? (for e.g. missing values indicated by zero)</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 41719,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "04/11/2014 14:09:47",
      "content": "<p>A negative value in productquantity and purchaseamount indicates a return</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 41720,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "04/11/2014 14:10:07",
      "content": "<p>Are all test users in testhistory present in transactions.csv?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 41721,
      "author_name": "mishutka",
      "author_url": "",
      "post_date": "04/11/2014 14:36:55",
      "content": "<p>Did I get right that &quot;purchaseamount&quot; == 0 means that product is a present for buying another one?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 41722,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "04/11/2014 14:57:46",
      "content": "<p>[quote=Abhishek;41720]</p>\n<p>Are all test users in testhistory present in transactions.csv?</p>\n<p>[/quote]</p>\n<p>Questions like these lead to&nbsp;heart palpitations for us poor admins, causing us to switch from what we're doing to double check that the data isn't broken. Did you make an honest attempt to answer this yourself before asking? I checked using this method:</p>\n<p>gzcat transactions.csv.gz | cut -d, -f1 | uniq | sort -g &gt; somefile<br>gzcat testHistory.csv.gz | cut -d, -f1 | uniq | sort -g &gt; someotherfile</p>\n<p>(the rest is left as an exercise for the reader). There's probably a python one-liner to do it as well. My point is, try first, then ask. If you find a problem, post it to the forums and be specific about why you think it's a problem. If you cry wolf, we eventually have to ignore forum questions because of the context switching costs.</p>\n<p>This is not to say we don't appreciate when you guys and gals catch our errors. We Just ask that you make an honest effort to answer questions yourself first, and share&nbsp;evidence when you do find something wrong.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 41723,
      "author_name": "abhishek",
      "author_url": "",
      "post_date": "04/11/2014 15:00:54",
      "content": "<p>Yeah. got that. There was something wrong with what I was doing and was getting less number ;) . thanks anyways</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 41789,
      "author_name": "artghazaryan",
      "author_url": "",
      "post_date": "04/12/2014 16:05:14",
      "content": "<p>After looking at the min and max values of the variables, except the date, I have several questions.</p>\n<p>1. the min for dept, category, brand, and purchase size is 0. Does 0 in this case mean that the real value is unknown? Or it mean that there actually is a brand 0, dept 0, category 0, and productsize 0? 0 productsize for example could represent a service offered.</p>\n<p>2. are the extreme values of purchasequantity and purchaseamount valid?&nbsp;</p>\n<p>3. would it be possible for the competition admins to provide the min max values of the variables?</p>\n<p>4. did the competition admins add noise to anonymize the data?</p>\n<p>VAR &nbsp; MIN &nbsp; &nbsp;MAX</p>\n<p>id &nbsp;86246 4853598737</p>\n<p>chain 2 526</p>\n<p>dept 0 99&nbsp;</p>\n<p>category 0 9999&nbsp;</p>\n<p>company 10000 10999999999&nbsp;</p>\n<p>brand 0 108689&nbsp;</p>\n<p>productsize 0 6000&nbsp;</p>\n<p>purchasequantity -32255.00 54800.00&nbsp;</p>\n<p>purchaseamount -8593791.00 58658.76&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 41965,
      "author_name": "dmdave",
      "author_url": "",
      "post_date": "04/15/2014 20:14:58",
      "content": "<p>Q1:&nbsp; 0 = missing value</p>\n\n<p>Q2:&nbsp; These values have been observed in the data</p>\n\n<p>Q3:&nbsp; No</p>\n\n<p>Q4:&nbsp; No</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42601,
      "author_name": "zachmayer",
      "author_url": "",
      "post_date": "04/21/2014 14:19:44",
      "content": "<p>So brand 0 means brand is unknown or missing? &nbsp;Similar for category and department?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42609,
      "author_name": "leobuettiker",
      "author_url": "",
      "post_date": "04/21/2014 15:40:08",
      "content": "<p>Is &quot;10000&quot; the code for an unknown or missing company?</p>\n<p>(I only started analyzing the data. But for brand 0 and company&nbsp;10000 I get very strange data.)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 42658,
      "author_name": "zachmayer",
      "author_url": "",
      "post_date": "04/22/2014 00:49:01",
      "content": "<p>[quote=Leo Buettiker;42609]</p>\n<p>Is &quot;10000&quot; the code for an unknown or missing company?</p>\n<p>(I only started analyzing the data. But for brand 0 and company&nbsp;10000 I get very strange data.)</p>\n<p>[/quote]</p>\n\n<p>For brand and category, 0 seems to mean missing.</p>\n\n<p>For company, I'm not sure, but &quot;10000&quot; does seem odd</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 43106,
      "author_name": "colingreen",
      "author_url": "",
      "post_date": "04/26/2014 22:05:44",
      "content": "<p>Odd in what way? In terms of occurrence frequency it's 2nd from top, but there are many other high frequency IDs in the same ballpark.</p>\n<p>What's more odd is the number of company IDs with very low frequencies, presumably these are products local to a specific store or town and not widely available?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 43108,
      "author_name": "colingreen",
      "author_url": "",
      "post_date": "04/26/2014 23:19:28",
      "content": "<p>Are you sure 0 = missing for dept? The others have IDs much higher than zero, and also zero, whereas dept has a continuous span of IDs starting from zero. Also, looking at the frequency of transactions for each dept ID, zero isn't an outlier at all.</p>\n<p>Think I'll assume for now that dept 0 is a real 'dept' (also, strange choice of field names in this data huh).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 44385,
      "author_name": "u90337",
      "author_url": "",
      "post_date": "05/11/2014 23:28:47",
      "content": "<p>There seem to be transactions for which :</p>\n<p>a) purchasequantity is positive, but purchaseamount is zero.</p>\n<p>1000714152,46,35,3509,103320030,875,2013-01-25,60.5,OZ,1,0</p>\n<p>b) purchasequantity negative, but purchaseamount is zero.</p>\n<p>122307580,4,22,2211,103700030,2246,2013-01-08,120,CT,-1,0</p>\n<p>c) purchasequantity is zero, but&nbsp;purchaseamount is positive.</p>\n<p>1012000746,214,3,305,103320030,875,2012-10-13,8,OZ,0,0.96</p>\n<p>d) purchasequantity is zero, but purchaseamount is negative.</p>\n<p>100017875,3,26,2614,103700030,514,2012-08-15,60,CT,0,-0.7</p>\n<p>e) Both zero</p>\n<p>100084808,20,69,6901,103700030,16139,2012-06-26,0.5,OZ,0,0</p>\n<p>f) purchasequantity is negative, but purchaseamount is positive.</p>\n<p>1050021843,214,41,4109,105100050,2820,2012-08-06,11,OZ,-4,1.6</p>\n<p>g) purchasequantity is positive, but purchaseamount is negative.</p>\n<p>100022923,95,26,2628,103700030,2248,2012-05-29,2,RL,1,-14.82</p>\n\n<p>Is there any explanation for these? (for e.g. missing values indicated by zero)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "41718": "",
    "41719": "",
    "41720": "",
    "41721": "",
    "41722": "",
    "41723": "",
    "41789": "",
    "41965": "",
    "42601": "",
    "42609": "",
    "42658": "",
    "43106": "",
    "43108": "",
    "44385": ""
  },
  "source": "meta"
}