{
  "id": 77666,
  "title": "how to deal with purchase_amount",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/77666",
  "author_name": "dangdang",
  "post_date": "2019-01-15T11:25:48.588000",
  "votes": 10,
  "comment_count": 19,
  "views": 0,
  "content": "<p>The feature named purchase_amount should be normalized according to the instructions.However,there are many values which is bigger than 1,(the biggest one is bigger than 6000000).So,I wonder how you deal with it during preprocessing period?Thanks a lot!</p>",
  "messages": [
    {
      "id": 456221,
      "postDate": "2019-01-15T11:25:48.590Z",
      "content": "<p>The feature named purchase_amount should be normalized according to the instructions.However,there are many values which is bigger than 1,(the biggest one is bigger than 6000000).So,I wonder how you deal with it during preprocessing period?Thanks a lot!</p>",
      "rawMarkdown": "The feature named purchase_amount should be normalized according to the instructions.However,there are many values which is bigger than 1,(the biggest one is bigger than 6000000).So,I wonder how you deal with it during preprocessing period?Thanks a lot!",
      "votes": 10
    },
    {
      "id": 456758,
      "postDate": "2019-01-16T13:30:49.253Z",
      "content": "<p><code>df['purchase_amount'].hist()</code> looks like rubbish due to the outliers</p>\n\n<p>try</p>\n\n<p>```\nnp.percentile(df['purchase_amount'], [1, 5, 50, 95, 99])</p>\n\n<blockquote>\n  <blockquote>\n    <blockquote>\n      <p>array([-0.74273043, -0.73789189, -0.69446527, -0.29611244,  0.80202504])\n      ```</p>\n    </blockquote>\n  </blockquote>\n</blockquote>\n\n<p>then chop the outliers </p>\n\n<p><code>\ndf['purchase_amount'].loc[np.logical_and(\n    df['purchase_amount'] &gt; -0.74273043,\n    df['purchase_amount'] &lt; 0.80202504\n)].hist()\n</code></p>\n\n<p>you only need to chop the positive end, e.g. <code>df['purchase_amount'].loc[df['purchase_amount'] &gt; -0.74273043].hist()</code> looks the same.</p>\n\n<p>This what I am going to do</p>\n\n<ul>\n<li>Trim all values greater than <code>0.8</code> to <code>0.8</code>, e.g. <code>X_new = min(X_old, 0.8)</code> (or whatever value you like)</li>\n<li>Add a boolean indicator variable <code>IsaOutlier</code> to your dataset</li>\n</ul>",
      "rawMarkdown": "`df['purchase_amount'].hist()` looks like rubbish due to the outliers\n\ntry\n\n```\nnp.percentile(df['purchase_amount'], [1, 5, 50, 95, 99])\n&gt;&gt;&gt; array([-0.74273043, -0.73789189, -0.69446527, -0.29611244,  0.80202504])\n```\n\nthen chop the outliers \n\n```\ndf['purchase_amount'].loc[np.logical_and(\n    df['purchase_amount'] &gt; -0.74273043,\n    df['purchase_amount'] &lt; 0.80202504\n)].hist()\n```\n\nyou only need to chop the positive end, e.g. `df['purchase_amount'].loc[df['purchase_amount'] &gt; -0.74273043].hist()` looks the same.\n\nThis what I am going to do\n\n- Trim all values greater than `0.8` to `0.8`, e.g. `X_new = min(X_old, 0.8)` (or whatever value you like)\n- Add a boolean indicator variable `IsaOutlier` to your dataset\n",
      "votes": 3,
      "replies": [
        {
          "id": 457749,
          "postDate": "2019-01-18T03:03:53.690Z",
          "content": "<p>good way to process this outlier data, but seem no help in LB.</p>",
          "rawMarkdown": "good way to process this outlier data, but seem no help in LB.",
          "votes": 1
        },
        {
          "id": 457759,
          "postDate": "2019-01-18T03:47:44.283Z",
          "content": "<p>thank u very much!Very useful advice and  I will try it out later! </p>",
          "rawMarkdown": "thank u very much!Very useful advice and  I will try it out later! "
        },
        {
          "id": 459314,
          "postDate": "2019-01-21T15:16:35.643Z",
          "content": "<p>with seaborn, I coded following, and it also worked. <br>\nyou can obtain a easy-to-see distribution chart of purchase amount.  </p>\n\n<blockquote>\n  <p>plt.figure(figsize=(10, 6)) <br>\n  plt.title('Purchase Amounts Distribution') <br>\n  sns.despine() <br>\n  sns.set_context(\"notebook\", font_scale=1.5, rc={\"lines.linewidth\": 2.5}) <br>\n  sns.distplot(np.clip(df['purchase_amount'], -0.8, +0.8), hist=True, rug=False, norm_hist=True)  </p>\n</blockquote>",
          "rawMarkdown": "with seaborn, I coded following, and it also worked.  \nyou can obtain a easy-to-see distribution chart of purchase amount.  \n&gt; plt.figure(figsize=(10, 6))  \nplt.title('Purchase Amounts Distribution')  \nsns.despine()  \nsns.set_context(\"notebook\", font\\_scale=1.5, rc={\"lines.linewidth\": 2.5})  \nsns.distplot(np.clip(df['purchase\\_amount'], -0.8, +0.8), hist=True, rug=False, norm_hist=True)  \n\n",
          "votes": 2
        },
        {
          "id": 467538,
          "postDate": "2019-02-07T09:41:36.563Z",
          "content": "<p>Clipping purchase amount might help. By clipping i mean, (R code below)</p>\n\n<p>q_25 = quantile(n_merc$target,na.rm = TRUE)[2] --25th percentile\nq_75 = quantile(n_merc$target,na.rm = TRUE)[4] --75th percentile\niqr =  q_75 - q_25 --Inter Quartile Range\nabove = (iqr * 3) + q_75\nbelow = q_25 - (iqr * 3)\nq_95 = quantile(n_merc$target,.95,na.rm = TRUE)\nq_5  = quantile(n_merc$target,.5,na.rm = TRUE)\nn_merc$target = ifelse((n_merc$target&gt;above),q_95,ifelse((n_merc$target</p>",
          "rawMarkdown": "Clipping purchase amount might help. By clipping i mean, (R code below)\n\nq_25 = quantile(n_merc$target,na.rm = TRUE)[2] --25th percentile\nq_75 = quantile(n_merc$target,na.rm = TRUE)[4] --75th percentile\niqr =  q_75 - q_25 --Inter Quartile Range\nabove = (iqr * 3) + q_75\nbelow = q_25 - (iqr * 3)\nq_95 = quantile(n_merc$target,.95,na.rm = TRUE)\nq_5  = quantile(n_merc$target,.5,na.rm = TRUE)\nn_merc$target = ifelse((n_merc$target&gt;above),q_95,ifelse((n_merc$target"
        }
      ]
    },
    {
      "id": 457741,
      "postDate": "2019-01-18T02:22:41.023Z",
      "content": "<p>For purchase_amount, I have another question, why is there a value less than 0? What business meaning is less than 0? Or is the problem of creating fake data.</p>",
      "rawMarkdown": "For purchase_amount, I have another question, why is there a value less than 0? What business meaning is less than 0? Or is the problem of creating fake data.",
      "votes": 1,
      "replies": [
        {
          "id": 457748,
          "postDate": "2019-01-18T03:03:02.013Z",
          "content": "<p>Yes, for true value of purchase_amount, negative value make no sense, but as the definition say it's normalized. </p>",
          "rawMarkdown": "Yes, for true value of purchase_amount, negative value make no sense, but as the definition say it's normalized. "
        },
        {
          "id": 457764,
          "postDate": "2019-01-18T03:53:50.650Z",
          "content": "<p>For the feature purchase_amount feature, how do you use it? If this value is standardized, I think he has lost his meaning. What do you think?</p>",
          "rawMarkdown": "For the feature purchase_amount feature, how do you use it? If this value is standardized, I think he has lost his meaning. What do you think?"
        },
        {
          "id": 457771,
          "postDate": "2019-01-18T04:26:38.087Z",
          "content": "<p>why lost his meaning, you can just think it as a scale, make no big difference, right? see the public kernels, this is a useful feature。</p>",
          "rawMarkdown": "why lost his meaning, you can just think it as a scale, make no big difference, right? see the public kernels, this is a useful feature。",
          "votes": 2
        }
      ]
    },
    {
      "id": 456953,
      "postDate": "2019-01-16T19:31:10.507Z",
      "content": "<p>I think that removing outliers is important for this feature! In particular, the value of 600,000 seems way too big. Using a boxplot allows to easily spot outliers, and from there you can decide whether you would like to keep them or not. My intuition is that some purchase amounts were not normalized, and we could make the transformation ourselves if we had the original mean and standard deviation, which I do not believe we do...</p>",
      "rawMarkdown": "I think that removing outliers is important for this feature! In particular, the value of 600,000 seems way too big. Using a boxplot allows to easily spot outliers, and from there you can decide whether you would like to keep them or not. My intuition is that some purchase amounts were not normalized, and we could make the transformation ourselves if we had the original mean and standard deviation, which I do not believe we do...",
      "votes": 2,
      "replies": [
        {
          "id": 456977,
          "postDate": "2019-01-16T20:26:13.853Z",
          "content": "<p>Note that all significant outliers are from unauthorized transactions, i.e. for which authorized_flag == 0. </p>\n\n<p>Further note that there are no unauthorized transactions in new merchants.</p>",
          "rawMarkdown": "Note that all significant outliers are from unauthorized transactions, i.e. for which authorized_flag == 0. \n\nFurther note that there are no unauthorized transactions in new merchants.",
          "votes": 10
        },
        {
          "id": 457733,
          "postDate": "2019-01-18T02:08:03.810Z",
          "content": "<p>nice finding, but what can we do with this finding? I am a bit confused.</p>",
          "rawMarkdown": "nice finding, but what can we do with this finding? I am a bit confused."
        },
        {
          "id": 457761,
          "postDate": "2019-01-18T03:52:09.323Z",
          "content": "<p>Do you mean that we should only use the authorized data when calculate purchase amount,  and use the unauthorized data to calculate other features except purchase amount? waiting for your advice</p>",
          "rawMarkdown": "Do you mean that we should only use the authorized data when calculate purchase amount,  and use the unauthorized data to calculate other features except purchase amount? waiting for your advice"
        },
        {
          "id": 458092,
          "postDate": "2019-01-18T18:56:35.460Z",
          "content": "<p>That's what I do. Whether correct or not - I cannot tell :-(</p>",
          "rawMarkdown": "That's what I do. Whether correct or not - I cannot tell :-(",
          "votes": 1
        },
        {
          "id": 458612,
          "postDate": "2019-01-20T03:46:42.900Z",
          "content": "<p>thx for sharing your finding. </p>",
          "rawMarkdown": "thx for sharing your finding. "
        }
      ]
    },
    {
      "id": 458912,
      "postDate": "2019-01-20T19:45:41.497Z",
      "content": "<p>From an accounting perspective, a negative purchase_amount could represent a return which would nullify the initial purchase and correctly state cash and stock positions. My 2 cents</p>",
      "rawMarkdown": "From an accounting perspective, a negative purchase_amount could represent a return which would nullify the initial purchase and correctly state cash and stock positions. My 2 cents"
    },
    {
      "id": 458776,
      "postDate": "2019-01-20T13:28:28.693Z",
      "content": "<p>In my models features with purchase amount (containing outliers) are important.</p>",
      "rawMarkdown": "In my models features with purchase amount (containing outliers) are important."
    },
    {
      "id": 456762,
      "postDate": "2019-01-16T13:37:44.787Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 456761,
      "postDate": "2019-01-16T13:34:26.113Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 456758,
      "author_name": "bubblride",
      "author_url": "",
      "post_date": "2019-01-16T13:30:49.253000",
      "content": "<p><code>df['purchase_amount'].hist()</code> looks like rubbish due to the outliers</p>\n\n<p>try</p>\n\n<p>```\nnp.percentile(df['purchase_amount'], [1, 5, 50, 95, 99])</p>\n\n<blockquote>\n  <blockquote>\n    <blockquote>\n      <p>array([-0.74273043, -0.73789189, -0.69446527, -0.29611244,  0.80202504])\n      ```</p>\n    </blockquote>\n  </blockquote>\n</blockquote>\n\n<p>then chop the outliers </p>\n\n<p><code>\ndf['purchase_amount'].loc[np.logical_and(\n    df['purchase_amount'] &gt; -0.74273043,\n    df['purchase_amount'] &lt; 0.80202504\n)].hist()\n</code></p>\n\n<p>you only need to chop the positive end, e.g. <code>df['purchase_amount'].loc[df['purchase_amount'] &gt; -0.74273043].hist()</code> looks the same.</p>\n\n<p>This what I am going to do</p>\n\n<ul>\n<li>Trim all values greater than <code>0.8</code> to <code>0.8</code>, e.g. <code>X_new = min(X_old, 0.8)</code> (or whatever value you like)</li>\n<li>Add a boolean indicator variable <code>IsaOutlier</code> to your dataset</li>\n</ul>",
      "votes": 3,
      "replies": [
        {
          "id": 457749,
          "author_name": "Jianfei Wang",
          "author_url": "",
          "post_date": "2019-01-18T03:03:53.690000",
          "content": "<p>good way to process this outlier data, but seem no help in LB.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 457759,
          "author_name": "dangdang",
          "author_url": "",
          "post_date": "2019-01-18T03:47:44.283000",
          "content": "<p>thank u very much!Very useful advice and  I will try it out later! </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 459314,
          "author_name": "Yukou Takahashi",
          "author_url": "",
          "post_date": "2019-01-21T15:16:35.643000",
          "content": "<p>with seaborn, I coded following, and it also worked. <br>\nyou can obtain a easy-to-see distribution chart of purchase amount.  </p>\n\n<blockquote>\n  <p>plt.figure(figsize=(10, 6)) <br>\n  plt.title('Purchase Amounts Distribution') <br>\n  sns.despine() <br>\n  sns.set_context(\"notebook\", font_scale=1.5, rc={\"lines.linewidth\": 2.5}) <br>\n  sns.distplot(np.clip(df['purchase_amount'], -0.8, +0.8), hist=True, rug=False, norm_hist=True)  </p>\n</blockquote>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 467538,
          "author_name": "Prashanth Thangavel",
          "author_url": "",
          "post_date": "2019-02-07T09:41:36.563000",
          "content": "<p>Clipping purchase amount might help. By clipping i mean, (R code below)</p>\n\n<p>q_25 = quantile(n_merc$target,na.rm = TRUE)[2] --25th percentile\nq_75 = quantile(n_merc$target,na.rm = TRUE)[4] --75th percentile\niqr =  q_75 - q_25 --Inter Quartile Range\nabove = (iqr * 3) + q_75\nbelow = q_25 - (iqr * 3)\nq_95 = quantile(n_merc$target,.95,na.rm = TRUE)\nq_5  = quantile(n_merc$target,.5,na.rm = TRUE)\nn_merc$target = ifelse((n_merc$target&gt;above),q_95,ifelse((n_merc$target</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 457741,
      "author_name": "banfenqian",
      "author_url": "",
      "post_date": "2019-01-18T02:22:41.023000",
      "content": "<p>For purchase_amount, I have another question, why is there a value less than 0? What business meaning is less than 0? Or is the problem of creating fake data.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 457748,
          "author_name": "Jianfei Wang",
          "author_url": "",
          "post_date": "2019-01-18T03:03:02.013000",
          "content": "<p>Yes, for true value of purchase_amount, negative value make no sense, but as the definition say it's normalized. </p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 457764,
          "author_name": "banfenqian",
          "author_url": "",
          "post_date": "2019-01-18T03:53:50.650000",
          "content": "<p>For the feature purchase_amount feature, how do you use it? If this value is standardized, I think he has lost his meaning. What do you think?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 457771,
          "author_name": "Jianfei Wang",
          "author_url": "",
          "post_date": "2019-01-18T04:26:38.087000",
          "content": "<p>why lost his meaning, you can just think it as a scale, make no big difference, right? see the public kernels, this is a useful feature。</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 456953,
      "author_name": "Celestin Hermez",
      "author_url": "",
      "post_date": "2019-01-16T19:31:10.507000",
      "content": "<p>I think that removing outliers is important for this feature! In particular, the value of 600,000 seems way too big. Using a boxplot allows to easily spot outliers, and from there you can decide whether you would like to keep them or not. My intuition is that some purchase amounts were not normalized, and we could make the transformation ourselves if we had the original mean and standard deviation, which I do not believe we do...</p>",
      "votes": 2,
      "replies": [
        {
          "id": 456977,
          "author_name": "lennart",
          "author_url": "",
          "post_date": "2019-01-16T20:26:13.853000",
          "content": "<p>Note that all significant outliers are from unauthorized transactions, i.e. for which authorized_flag == 0. </p>\n\n<p>Further note that there are no unauthorized transactions in new merchants.</p>",
          "votes": 10,
          "replies": []
        },
        {
          "id": 457733,
          "author_name": "good good study",
          "author_url": "",
          "post_date": "2019-01-18T02:08:03.810000",
          "content": "<p>nice finding, but what can we do with this finding? I am a bit confused.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 457761,
          "author_name": "dangdang",
          "author_url": "",
          "post_date": "2019-01-18T03:52:09.323000",
          "content": "<p>Do you mean that we should only use the authorized data when calculate purchase amount,  and use the unauthorized data to calculate other features except purchase amount? waiting for your advice</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 458092,
          "author_name": "lennart",
          "author_url": "",
          "post_date": "2019-01-18T18:56:35.460000",
          "content": "<p>That's what I do. Whether correct or not - I cannot tell :-(</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 458612,
          "author_name": "bubblride",
          "author_url": "",
          "post_date": "2019-01-20T03:46:42.900000",
          "content": "<p>thx for sharing your finding. </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 458912,
      "author_name": "TChidi",
      "author_url": "",
      "post_date": "2019-01-20T19:45:41.497000",
      "content": "<p>From an accounting perspective, a negative purchase_amount could represent a return which would nullify the initial purchase and correctly state cash and stock positions. My 2 cents</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 458776,
      "author_name": "bezul",
      "author_url": "",
      "post_date": "2019-01-20T13:28:28.693000",
      "content": "<p>In my models features with purchase amount (containing outliers) are important.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 456762,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-01-16T13:37:44.787000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 456761,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-01-16T13:34:26.113000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "456221": "The feature named purchase_amount should be normalized according to the instructions.However,there are many values which is bigger than 1,(the biggest one is bigger than 6000000).So,I wonder how you deal with it during preprocessing period?Thanks a lot!",
    "456758": "`df['purchase_amount'].hist()` looks like rubbish due to the outliers\n\ntry\n\n```\nnp.percentile(df['purchase_amount'], [1, 5, 50, 95, 99])\n&gt;&gt;&gt; array([-0.74273043, -0.73789189, -0.69446527, -0.29611244,  0.80202504])\n```\n\nthen chop the outliers \n\n```\ndf['purchase_amount'].loc[np.logical_and(\n    df['purchase_amount'] &gt; -0.74273043,\n    df['purchase_amount'] &lt; 0.80202504\n)].hist()\n```\n\nyou only need to chop the positive end, e.g. `df['purchase_amount'].loc[df['purchase_amount'] &gt; -0.74273043].hist()` looks the same.\n\nThis what I am going to do\n\n- Trim all values greater than `0.8` to `0.8`, e.g. `X_new = min(X_old, 0.8)` (or whatever value you like)\n- Add a boolean indicator variable `IsaOutlier` to your dataset\n",
    "457741": "For purchase_amount, I have another question, why is there a value less than 0? What business meaning is less than 0? Or is the problem of creating fake data.",
    "456953": "I think that removing outliers is important for this feature! In particular, the value of 600,000 seems way too big. Using a boxplot allows to easily spot outliers, and from there you can decide whether you would like to keep them or not. My intuition is that some purchase amounts were not normalized, and we could make the transformation ourselves if we had the original mean and standard deviation, which I do not believe we do...",
    "458912": "From an accounting perspective, a negative purchase_amount could represent a return which would nullify the initial purchase and correctly state cash and stock positions. My 2 cents",
    "458776": "In my models features with purchase amount (containing outliers) are important.",
    "456762": "",
    "456761": ""
  }
}