{
  "id": 80235,
  "title": "Temporal aspect of  reference date",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/80235",
  "author_name": "sergeman",
  "post_date": "2019-02-12T02:12:21.190000",
  "votes": 4,
  "comment_count": 0,
  "views": 0,
  "content": "<p><em>For more details about the data analysis that led me to these conclusions can be fond in a public kernel <a href=\"https://www.kaggle.com/calabist/resolving-mystery-of-reference-date\">here</a></em></p>\n\n<p>The transaction data released by ELO exhibits a strong temporal component revealed in the month lag and purchase date columns of the transaction data.</p>\n\n<p>Contrary to popular belief, the  reference date is not a fixed date for entire dataset. Reference date is a date relative to a group of card ids selected for promotion. Historic transactions for selected card ids  are collected up to the reference date. New transactions contain two month worth of data for the cards that are on promotion. After end of two month period no data is collected for selected cards in both transactions files.</p>\n\n<p>A new batch of promotions is released every  month, hence the reference date slides along the dataset with a monthly cadence. </p>\n\n<p>Duration of data collection prior to the reference date varies from card to card. It might be because a card with a shorter pre reference date history could have been opened after the data collection in the historic data set started. </p>\n\n<p>All these details about reference date raise a couple of questions\n- How should reference date logic inform creation of the training set features?\n- How a proper validation set should be constructed?</p>\n\n<p><strong>Reference date  schedule</strong></p>\n\n<p>New merchant_transactions file contains transactions for cards exposed to suggestions of new merchants to existing customers. ELO ran  promotions for new merchants at slow pace since the beginning of record in March 1st of 2017 gradually increasing exposure. In March of 2017 number of cards with month_lag == 1 was 213. By January 1st of 2018 there were 18696 cards exposed and by February of 2018  the promotion went out to 173574 card_ids</p>\n\n<p>On March 1st 2018 the promotion was released to a broader population . Sales to new merchants spiked and historic merchants were on declining trend, but we do not have data in historic transactions to substantiate that . I guess that this spike led  ELO to a decision that the loyalty score that they have created does not explain true customer loyalty behavior. There was a need for new way of computing loyalty score differently. ELO decided to do this competition. </p>\n\n<p><strong>Two month difference between the historic and new transactions datasets</strong></p>\n\n<p>The historic transactions file contain sales leading up to the month when promotion started, but omits data about historic transactions during the time period covered in new_transactions.  This explains the two months lag between historic transactions and new merchant transactions.</p>\n\n<p>Once promotion was launched to a selected group of customers, the new_transactions dataset was started. The new_transactions dataset contains  data about all card_ids that were exposed to promotion and transactions outside of the core historic group that were a result of promotion were recorded. Hence the new_transaction dataset  contains record of all card_ids that completed transactions as a result of promotion. </p>\n\n<p>This data collection procedure explains why we have the two month lag between the historic transactions and new merchant transactions time frames. </p>",
  "messages": [
    {
      "id": 469890,
      "postDate": "2019-02-12T02:12:21.190Z",
      "content": "<p><em>For more details about the data analysis that led me to these conclusions can be fond in a public kernel <a href=\"https://www.kaggle.com/calabist/resolving-mystery-of-reference-date\">here</a></em></p>\n\n<p>The transaction data released by ELO exhibits a strong temporal component revealed in the month lag and purchase date columns of the transaction data.</p>\n\n<p>Contrary to popular belief, the  reference date is not a fixed date for entire dataset. Reference date is a date relative to a group of card ids selected for promotion. Historic transactions for selected card ids  are collected up to the reference date. New transactions contain two month worth of data for the cards that are on promotion. After end of two month period no data is collected for selected cards in both transactions files.</p>\n\n<p>A new batch of promotions is released every  month, hence the reference date slides along the dataset with a monthly cadence. </p>\n\n<p>Duration of data collection prior to the reference date varies from card to card. It might be because a card with a shorter pre reference date history could have been opened after the data collection in the historic data set started. </p>\n\n<p>All these details about reference date raise a couple of questions\n- How should reference date logic inform creation of the training set features?\n- How a proper validation set should be constructed?</p>\n\n<p><strong>Reference date  schedule</strong></p>\n\n<p>New merchant_transactions file contains transactions for cards exposed to suggestions of new merchants to existing customers. ELO ran  promotions for new merchants at slow pace since the beginning of record in March 1st of 2017 gradually increasing exposure. In March of 2017 number of cards with month_lag == 1 was 213. By January 1st of 2018 there were 18696 cards exposed and by February of 2018  the promotion went out to 173574 card_ids</p>\n\n<p>On March 1st 2018 the promotion was released to a broader population . Sales to new merchants spiked and historic merchants were on declining trend, but we do not have data in historic transactions to substantiate that . I guess that this spike led  ELO to a decision that the loyalty score that they have created does not explain true customer loyalty behavior. There was a need for new way of computing loyalty score differently. ELO decided to do this competition. </p>\n\n<p><strong>Two month difference between the historic and new transactions datasets</strong></p>\n\n<p>The historic transactions file contain sales leading up to the month when promotion started, but omits data about historic transactions during the time period covered in new_transactions.  This explains the two months lag between historic transactions and new merchant transactions.</p>\n\n<p>Once promotion was launched to a selected group of customers, the new_transactions dataset was started. The new_transactions dataset contains  data about all card_ids that were exposed to promotion and transactions outside of the core historic group that were a result of promotion were recorded. Hence the new_transaction dataset  contains record of all card_ids that completed transactions as a result of promotion. </p>\n\n<p>This data collection procedure explains why we have the two month lag between the historic transactions and new merchant transactions time frames. </p>",
      "rawMarkdown": "*For more details about the data analysis that led me to these conclusions can be fond in a public kernel [here][1]*\n\nThe transaction data released by ELO exhibits a strong temporal component revealed in the month lag and purchase date columns of the transaction data.\n\nContrary to popular belief, the  reference date is not a fixed date for entire dataset. Reference date is a date relative to a group of card ids selected for promotion. Historic transactions for selected card ids  are collected up to the reference date. New transactions contain two month worth of data for the cards that are on promotion. After end of two month period no data is collected for selected cards in both transactions files.\n \nA new batch of promotions is released every  month, hence the reference date slides along the dataset with a monthly cadence. \n\nDuration of data collection prior to the reference date varies from card to card. It might be because a card with a shorter pre reference date history could have been opened after the data collection in the historic data set started. \n\nAll these details about reference date raise a couple of questions\n- How should reference date logic inform creation of the training set features?\n- How a proper validation set should be constructed?\n\n**Reference date  schedule**\n\nNew merchant_transactions file contains transactions for cards exposed to suggestions of new merchants to existing customers. ELO ran  promotions for new merchants at slow pace since the beginning of record in March 1st of 2017 gradually increasing exposure. In March of 2017 number of cards with month_lag == 1 was 213. By January 1st of 2018 there were 18696 cards exposed and by February of 2018  the promotion went out to 173574 card_ids\n\nOn March 1st 2018 the promotion was released to a broader population . Sales to new merchants spiked and historic merchants were on declining trend, but we do not have data in historic transactions to substantiate that . I guess that this spike led  ELO to a decision that the loyalty score that they have created does not explain true customer loyalty behavior. There was a need for new way of computing loyalty score differently. ELO decided to do this competition. \n\n**Two month difference between the historic and new transactions datasets**\n\nThe historic transactions file contain sales leading up to the month when promotion started, but omits data about historic transactions during the time period covered in new_transactions.  This explains the two months lag between historic transactions and new merchant transactions.\n\nOnce promotion was launched to a selected group of customers, the new_transactions dataset was started. The new_transactions dataset contains  data about all card_ids that were exposed to promotion and transactions outside of the core historic group that were a result of promotion were recorded. Hence the new_transaction dataset  contains record of all card_ids that completed transactions as a result of promotion. \n\nThis data collection procedure explains why we have the two month lag between the historic transactions and new merchant transactions time frames. \n\n  [1]: https://www.kaggle.com/calabist/resolving-mystery-of-reference-date",
      "votes": 4
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "469890": "*For more details about the data analysis that led me to these conclusions can be fond in a public kernel [here][1]*\n\nThe transaction data released by ELO exhibits a strong temporal component revealed in the month lag and purchase date columns of the transaction data.\n\nContrary to popular belief, the  reference date is not a fixed date for entire dataset. Reference date is a date relative to a group of card ids selected for promotion. Historic transactions for selected card ids  are collected up to the reference date. New transactions contain two month worth of data for the cards that are on promotion. After end of two month period no data is collected for selected cards in both transactions files.\n \nA new batch of promotions is released every  month, hence the reference date slides along the dataset with a monthly cadence. \n\nDuration of data collection prior to the reference date varies from card to card. It might be because a card with a shorter pre reference date history could have been opened after the data collection in the historic data set started. \n\nAll these details about reference date raise a couple of questions\n- How should reference date logic inform creation of the training set features?\n- How a proper validation set should be constructed?\n\n**Reference date  schedule**\n\nNew merchant_transactions file contains transactions for cards exposed to suggestions of new merchants to existing customers. ELO ran  promotions for new merchants at slow pace since the beginning of record in March 1st of 2017 gradually increasing exposure. In March of 2017 number of cards with month_lag == 1 was 213. By January 1st of 2018 there were 18696 cards exposed and by February of 2018  the promotion went out to 173574 card_ids\n\nOn March 1st 2018 the promotion was released to a broader population . Sales to new merchants spiked and historic merchants were on declining trend, but we do not have data in historic transactions to substantiate that . I guess that this spike led  ELO to a decision that the loyalty score that they have created does not explain true customer loyalty behavior. There was a need for new way of computing loyalty score differently. ELO decided to do this competition. \n\n**Two month difference between the historic and new transactions datasets**\n\nThe historic transactions file contain sales leading up to the month when promotion started, but omits data about historic transactions during the time period covered in new_transactions.  This explains the two months lag between historic transactions and new merchant transactions.\n\nOnce promotion was launched to a selected group of customers, the new_transactions dataset was started. The new_transactions dataset contains  data about all card_ids that were exposed to promotion and transactions outside of the core historic group that were a result of promotion were recorded. Hence the new_transaction dataset  contains record of all card_ids that completed transactions as a result of promotion. \n\nThis data collection procedure explains why we have the two month lag between the historic transactions and new merchant transactions time frames. \n\n  [1]: https://www.kaggle.com/calabist/resolving-mystery-of-reference-date"
  }
}