{
  "id": 74120,
  "title": "Difference between historical_transactions and new_merchant_transactions",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/74120",
  "author_name": "",
  "post_date": "2018-12-08T20:05:03.882133300Z",
  "votes": 3,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hi, </p>\n\n<p>We are struggling to understand what are the differences between the historical_transactions and new_merchant_transactions. Can anyone give us any clue?</p>\n\n<p>According to the data description tab, new_merchant_transactions contains two months' worth of data for each card_id containing ALL purchases that card_id made at merchant_ids that were not visited in the historical data. However, what is the underlying cause for a purchase to be or not to be included in the historical data?</p>\n\n<p>Thanks in advance!</p>",
  "messages": [
    {
      "id": "435806",
      "postDate": "12/08/2018 20:05:03",
      "content": "<p>Hi, </p>\n\n<p>We are struggling to understand what are the differences between the historical_transactions and new_merchant_transactions. Can anyone give us any clue?</p>\n\n<p>According to the data description tab, new_merchant_transactions contains two months' worth of data for each card_id containing ALL purchases that card_id made at merchant_ids that were not visited in the historical data. However, what is the underlying cause for a purchase to be or not to be included in the historical data?</p>\n\n<p>Thanks in advance!</p>",
      "rawMarkdown": "Hi, \n\nWe are struggling to understand what are the differences between the historical_transactions and new_merchant_transactions. Can anyone give us any clue?\n\nAccording to the data description tab, new_merchant_transactions contains two months' worth of data for each card_id containing ALL purchases that card_id made at merchant_ids that were not visited in the historical data. However, what is the underlying cause for a purchase to be or not to be included in the historical data?\n\nThanks in advance!",
      "votes": null
    },
    {
      "id": "435855",
      "postDate": "12/08/2018 22:59:28",
      "content": "<p>That confused me also.  I've handled it by ignoring that paragraph, since its hard to tell what it means.  But the card_id's are disjoint sets between train and test so I just combined the two transaction files, then split them again based on the card_ids into train_transactions and test transactions.  Some of the answers provided by the admins imply we can do whatever we want to predict the target score so ...</p>",
      "rawMarkdown": "That confused me also.  I've handled it by ignoring that paragraph, since its hard to tell what it means.  But the card_id's are disjoint sets between train and test so I just combined the two transaction files, then split them again based on the card_ids into train_transactions and test transactions.  Some of the answers provided by the admins imply we can do whatever we want to predict the target score so ...",
      "votes": null
    },
    {
      "id": "436037",
      "postDate": "12/09/2018 11:57:06",
      "content": "<p>Ok, thanks for the answer. </p>",
      "rawMarkdown": "Ok, thanks for the answer.",
      "votes": null
    },
    {
      "id": "436253",
      "postDate": "12/10/2018 01:51:33",
      "content": "<p>I'm also study about <code>historical_transactions</code> and <code>new_merchant_transactions</code>. In <strong>Data Description</strong>:\n- historical_transactions.csv contains up to 3 months' worth of transactions for every card at any of the provided merchant_ids.\n- new_merchant_transactions.csv contains the transactions at new merchants (merchant_ids that this particular card_id has not yet visited) over a period of two months.</p>\n\n<p>I think there are at least 2 differences:\n1. <code>new_merchant_transaction</code> is latter than <code>historical_transactions</code>. Actually, <code>historical_transactions</code> : 2017/01/01 ~ 2018/02/28; <code>new_merchant_transaction</code>: 2018/03/01 ~ 2018/04/30.\n```python\nhist_trans = pd.read_csv('input/historical_transactions.csv')\nnew_trans = pd.read_csv('input/new_merchant_transactions.csv')</p>\n\n<p>hist_time_range = hist_trans.groupby('card_id')['purchase_date'].agg([np.min, np.max])\nnew_time_range = new_trans.groupby('card_id')['purchase_date'].agg([np.min, np.max])</p>\n\n<p>time_range = hist_time_range .join(new_time_range , how='outer', lsuffix='_hist', rsuffix='_new')</p>\n\n<p>for col in time_range:\n    time_range[col] = pd.to_datetime(time_range[col])</p>\n\n<p>time_range = time_range.dropna()</p>\n\n<p>(time_range.amax_hist &gt; time_range.amin_new).sum()    # returns 0\n```\n<img src=\"http://youdao.pythonplus.cn/time_range_head.PNG\" alt=\"time_range.head()\"></p>\n\n<ol>\n<li>for every <code>card_id</code>, <code>hist_trans.merchant_id</code> will not appear in <code>new_trans.merchant_id</code>.</li>\n</ol>\n\n<p>In summary, for a particular <code>card_id</code>, we record transactions a period of time into <code>historical_transactions</code>, and latter, we record transactions(only for the merchant he first visit) for another period of time into 'new_merchant_transactions`.</p>\n\n<p>( Forgive for my poor English.)</p>",
      "rawMarkdown": "I'm also study about `historical_transactions` and `new_merchant_transactions`. In **Data Description**:\n- historical_transactions.csv contains up to 3 months' worth of transactions for every card at any of the provided merchant_ids.\n- new_merchant_transactions.csv contains the transactions at new merchants (merchant_ids that this particular card_id has not yet visited) over a period of two months.\n\nI think there are at least 2 differences:\n1. `new_merchant_transaction` is latter than `historical_transactions`. Actually, `historical_transactions` : 2017/01/01 ~ 2018/02/28; `new_merchant_transaction`: 2018/03/01 ~ 2018/04/30.\n```python\nhist_trans = pd.read_csv('input/historical_transactions.csv')\nnew_trans = pd.read_csv('input/new_merchant_transactions.csv')\n\nhist_time_range = hist_trans.groupby('card_id')['purchase_date'].agg([np.min, np.max])\nnew_time_range = new_trans.groupby('card_id')['purchase_date'].agg([np.min, np.max])\n\ntime_range = hist_time_range .join(new_time_range , how='outer', lsuffix='_hist', rsuffix='_new')\n\nfor col in time_range:\n    time_range[col] = pd.to_datetime(time_range[col])\n\ntime_range = time_range.dropna()\n\n(time_range.amax_hist &gt; time_range.amin_new).sum()    # returns 0\n```\n![time_range.head()](http://youdao.pythonplus.cn/time_range_head.PNG)\n\n2.  for every `card_id`, `hist_trans.merchant_id` will not appear in `new_trans.merchant_id`.\n\nIn summary, for a particular `card_id`, we record transactions a period of time into `historical_transactions`, and latter, we record transactions(only for the merchant he first visit) for another period of time into 'new_merchant_transactions`.\n\n( Forgive for my poor English.)",
      "votes": null
    },
    {
      "id": "436839",
      "postDate": "12/11/2018 02:34:24",
      "content": "<p>Thank you for your answer, it helps a lot.</p>",
      "rawMarkdown": "Thank you for your answer, it helps a lot.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 435855,
      "author_name": "thawatt",
      "author_url": "",
      "post_date": "12/08/2018 22:59:28",
      "content": "<p>That confused me also.  I've handled it by ignoring that paragraph, since its hard to tell what it means.  But the card_id's are disjoint sets between train and test so I just combined the two transaction files, then split them again based on the card_ids into train_transactions and test transactions.  Some of the answers provided by the admins imply we can do whatever we want to predict the target score so ...</p>",
      "votes": null,
      "replies": [
        {
          "id": 436037,
          "author_name": "andreip",
          "author_url": "",
          "post_date": "12/09/2018 11:57:06",
          "content": "<p>Ok, thanks for the answer. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 436253,
      "author_name": "jiazhuang",
      "author_url": "",
      "post_date": "12/10/2018 01:51:33",
      "content": "<p>I'm also study about <code>historical_transactions</code> and <code>new_merchant_transactions</code>. In <strong>Data Description</strong>:\n- historical_transactions.csv contains up to 3 months' worth of transactions for every card at any of the provided merchant_ids.\n- new_merchant_transactions.csv contains the transactions at new merchants (merchant_ids that this particular card_id has not yet visited) over a period of two months.</p>\n\n<p>I think there are at least 2 differences:\n1. <code>new_merchant_transaction</code> is latter than <code>historical_transactions</code>. Actually, <code>historical_transactions</code> : 2017/01/01 ~ 2018/02/28; <code>new_merchant_transaction</code>: 2018/03/01 ~ 2018/04/30.\n```python\nhist_trans = pd.read_csv('input/historical_transactions.csv')\nnew_trans = pd.read_csv('input/new_merchant_transactions.csv')</p>\n\n<p>hist_time_range = hist_trans.groupby('card_id')['purchase_date'].agg([np.min, np.max])\nnew_time_range = new_trans.groupby('card_id')['purchase_date'].agg([np.min, np.max])</p>\n\n<p>time_range = hist_time_range .join(new_time_range , how='outer', lsuffix='_hist', rsuffix='_new')</p>\n\n<p>for col in time_range:\n    time_range[col] = pd.to_datetime(time_range[col])</p>\n\n<p>time_range = time_range.dropna()</p>\n\n<p>(time_range.amax_hist &gt; time_range.amin_new).sum()    # returns 0\n```\n<img src=\"http://youdao.pythonplus.cn/time_range_head.PNG\" alt=\"time_range.head()\"></p>\n\n<ol>\n<li>for every <code>card_id</code>, <code>hist_trans.merchant_id</code> will not appear in <code>new_trans.merchant_id</code>.</li>\n</ol>\n\n<p>In summary, for a particular <code>card_id</code>, we record transactions a period of time into <code>historical_transactions</code>, and latter, we record transactions(only for the merchant he first visit) for another period of time into 'new_merchant_transactions`.</p>\n\n<p>( Forgive for my poor English.)</p>",
      "votes": null,
      "replies": [
        {
          "id": 436839,
          "author_name": "dandaner",
          "author_url": "",
          "post_date": "12/11/2018 02:34:24",
          "content": "<p>Thank you for your answer, it helps a lot.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "435806": "Hi, \n\nWe are struggling to understand what are the differences between the historical_transactions and new_merchant_transactions. Can anyone give us any clue?\n\nAccording to the data description tab, new_merchant_transactions contains two months' worth of data for each card_id containing ALL purchases that card_id made at merchant_ids that were not visited in the historical data. However, what is the underlying cause for a purchase to be or not to be included in the historical data?\n\nThanks in advance!",
    "435855": "That confused me also.  I've handled it by ignoring that paragraph, since its hard to tell what it means.  But the card_id's are disjoint sets between train and test so I just combined the two transaction files, then split them again based on the card_ids into train_transactions and test transactions.  Some of the answers provided by the admins imply we can do whatever we want to predict the target score so ...",
    "436037": "Ok, thanks for the answer.",
    "436253": "I'm also study about `historical_transactions` and `new_merchant_transactions`. In **Data Description**:\n- historical_transactions.csv contains up to 3 months' worth of transactions for every card at any of the provided merchant_ids.\n- new_merchant_transactions.csv contains the transactions at new merchants (merchant_ids that this particular card_id has not yet visited) over a period of two months.\n\nI think there are at least 2 differences:\n1. `new_merchant_transaction` is latter than `historical_transactions`. Actually, `historical_transactions` : 2017/01/01 ~ 2018/02/28; `new_merchant_transaction`: 2018/03/01 ~ 2018/04/30.\n```python\nhist_trans = pd.read_csv('input/historical_transactions.csv')\nnew_trans = pd.read_csv('input/new_merchant_transactions.csv')\n\nhist_time_range = hist_trans.groupby('card_id')['purchase_date'].agg([np.min, np.max])\nnew_time_range = new_trans.groupby('card_id')['purchase_date'].agg([np.min, np.max])\n\ntime_range = hist_time_range .join(new_time_range , how='outer', lsuffix='_hist', rsuffix='_new')\n\nfor col in time_range:\n    time_range[col] = pd.to_datetime(time_range[col])\n\ntime_range = time_range.dropna()\n\n(time_range.amax_hist &gt; time_range.amin_new).sum()    # returns 0\n```\n![time_range.head()](http://youdao.pythonplus.cn/time_range_head.PNG)\n\n2.  for every `card_id`, `hist_trans.merchant_id` will not appear in `new_trans.merchant_id`.\n\nIn summary, for a particular `card_id`, we record transactions a period of time into `historical_transactions`, and latter, we record transactions(only for the merchant he first visit) for another period of time into 'new_merchant_transactions`.\n\n( Forgive for my poor English.)",
    "436839": "Thank you for your answer, it helps a lot."
  },
  "source": "meta"
}