Hi,\nThis is my first go at Kaggle so be gentle. :-)\nI'm reading the descriptions and I don't quite get the New Merchants description. It says these are transactions at New Merchants (ones that the card has not yet visited in 2 months).
\nMy question is ... if they've not yet visited these merchants, how can there by transactions for that card id in the file ??
\nSomeone please help me understand how the New Merchants file is working.\nThanks,\nShaun
These two files have data for two different periods and for two different groups of merchants. If you look at the purchase dates in both files, you'll see that historical_transactions.csv shows data for few months in 2017, while new_merchants_transactions shows data for a few months in 2018 for merchants that were not visited in the historical data. This means that even if customers purchased from the same merchant in both periods, these merchants will not be shown in new merchants, only those that are new (not visited in first period) will be shown in second.
", "rawMarkdown": "These two files have data for two different periods and for two different groups of merchants. If you look at the purchase dates in both files, you'll see that historical_transactions.csv shows data for few months in 2017, while new_merchants_transactions shows data for a few months in 2018 for merchants that were not visited in the historical data. This means that even if customers purchased from the same merchant in both periods, these merchants will not be shown in new merchants, only those that are new (not visited in first period) will be shown in second. ", "votes": 16, "replies": [ { "id": 429489, "postDate": "2018-11-29T00:00:45.900Z", "content": "Ha. Ok Diana that sounds like it makes sense. I will have a think about what you wrote and ask more if I'm still puzzled.
", "rawMarkdown": "Ha. Ok Diana that sounds like it makes sense. I will have a think about what you wrote and ask more if I'm still puzzled. " }, { "id": 439352, "postDate": "2018-12-15T08:18:52.970Z", "content": "I think I understand the difference between new_merchant_transactions.csv and merchants.csv. But why is it even split like this? the new_merchant_transactions file is essentially censoring particular merchants ... is it a matter of anonymising the data?
", "rawMarkdown": "I think I understand the difference between new_merchant_transactions.csv and merchants.csv. But why is it even split like this? the new_merchant_transactions file is essentially censoring particular merchants ... is it a matter of anonymising the data?" } ] }, { "id": 429378, "postDate": "2018-11-28T19:12:00.937Z", "content": "Hi,\nThis is my first go at Kaggle so be gentle. :-)\nI'm reading the descriptions and I don't quite get the New Merchants description. It says these are transactions at New Merchants (ones that the card has not yet visited in 2 months).
\nMy question is ... if they've not yet visited these merchants, how can there by transactions for that card id in the file ??
\nSomeone please help me understand how the New Merchants file is working.\nThanks,\nShaun
I've been competing on Kaggle for awhile now and I'm also confused by this.
", "rawMarkdown": "I've been competing on Kaggle for awhile now and I'm also confused by this.", "votes": 3 }, { "id": 430074, "postDate": "2018-11-29T19:34:35.113Z", "content": "Thanks @Diana Kolusheva,
\n@blissweb, I am new too :-) I gave it a shot while exploring the txn data.
Following are my observations
\n\nThere are 217808 common merchant_id and 290001 common card_id between the two txn tables when looked in isolation
\n\nprint(len(set(df_newMerchTxn['merchant_id']).intersection(set(df_histTxn['merchant_id']))))\n print(len(set(df_newMerchTxn['card_id']).intersection(set(df_histTxn['card_id']))))\n\n
however if you create the unique id by combining/concatenating the merchent_id and Card_id in both datasets, there are no common unique ids (except 1 nan due to the missing combination attributed to the data quality)
\n\n`df_newMerchTxn['merchant_card_id']= df_newMerchTxn['merchant_id']+df_newMerchTxn['card_id']\ndf_histTxn['merchant_card_id']= df_histTxn['merchant_id']+df_histTxn['card_id']\nprint((set(df_newMerchTxn['merchant_card_id']).intersection(set(df_histTxn['merchant_card_id']))))`\n\n
This means, the way to read the provided comments about new_merchant_transactions table is \"combination of customers [i] who shopped at merchent [y] in historical_transactions table will not be available in the new_merchant_transactions and vice-versa\"
\n\nI hope this helps, please feel free to comment/ correct me if this is not right.
", "rawMarkdown": "Thanks @[Diana Kolusheva][1], \n@blissweb, I am new too :-) I gave it a shot while exploring the txn data. \n\nFollowing are my observations\nprint(len(set(df_newMerchTxn['merchant_id']).intersection(set(df_histTxn['merchant_id']))))\n print(len(set(df_newMerchTxn['card_id']).intersection(set(df_histTxn['card_id']))))\n\nhowever if you create the unique id by combining/concatenating the merchent_id and Card_id in both datasets, there are no common unique ids (except 1 nan due to the missing combination attributed to the data quality)\n\n
`df_newMerchTxn['merchant_card_id']= df_newMerchTxn['merchant_id']+df_newMerchTxn['card_id']\ndf_histTxn['merchant_card_id']= df_histTxn['merchant_id']+df_histTxn['card_id']\nprint((set(df_newMerchTxn['merchant_card_id']).intersection(set(df_histTxn['merchant_card_id']))))`\n\nThis means, the way to read the provided comments about new_merchant_transactions table is \"combination of customers [i] who shopped at merchent [y] in historical_transactions table will not be available in the new_merchant_transactions and vice-versa\"\n\nI hope this helps, please feel free to comment/ correct me if this is not right.\n\n\n [1]: https://www.kaggle.com/dianakolusheva", "votes": 4 }, { "id": 431751, "postDate": "2018-12-02T21:31:35.880Z", "content": "
Ran some analysis, it seems we can indeed get some insights about loyalty score from the difference between historical and new merchants. Hope this helps
\n\n", "rawMarkdown": "Ran some analysis, it seems we can indeed get some insights about loyalty score from the difference between historical and new merchants. Hope this helps \n\nhttps://www.kaggle.com/firstprayer/understand-loyalty-score-w-r-t-hist-vs-new?scriptVersionId=8008055 ", "votes": 1 }, { "id": 431416, "postDate": "2018-12-02T08:37:20.970Z", "content": "The explanations from @deepali clarified some of things, but I'm still confused by the relationship of the hist/new transactions with train/test set.
\n\nHere's the overlap of card ids I got :
\n\nprint(len(set(hist_trans[\"card_id\"]) & set(train[\"card_id\"])))\nprint(len(set(hist_trans[\"card_id\"]) & set(test[\"card_id\"])))\nprint(len(set(new_trans[\"card_id\"]) & set(train[\"card_id\"])))\nprint(len(set(new_trans[\"card_id\"]) & set(test[\"card_id\"])))
\n\n201917\n123623\n179986\n110015
\n\nBoth train and test set have overlap with old and new data.
\n\nIf the train label (the loyalty score) is computed based on both hist and new data, and so is the predicted score on test set, then why do we even need to distinguish between hist and new transactions?
\n\nSo it seems the logical explanation would be that, the training label are measured based on only historical transactions, whereas when we're making predictions on test set, we need to consider both hist and new transactions?
\n\nAnother possible interpretation would be the loyalty score correlates with the DELTA between the two time periods? For instance, if a card_id appear in hist transaction but no in new transactions, it means the card holder doesn't use ELO payment for new merchants (but may still stick with old merchants), and we expect the loyalty score might be lower in this case?
\n\nCan someone confirm/comment on my conjectures above?
", "rawMarkdown": "The explanations from @deepali clarified some of things, but I'm still confused by the relationship of the hist/new transactions with train/test set.\n\nHere's the overlap of card ids I got :\n\nprint(len(set(hist_trans[\"card_id\"]) & set(train[\"card_id\"])))\nprint(len(set(hist_trans[\"card_id\"]) & set(test[\"card_id\"])))\nprint(len(set(new_trans[\"card_id\"]) & set(train[\"card_id\"])))\nprint(len(set(new_trans[\"card_id\"]) & set(test[\"card_id\"])))\n\n201917\n123623\n179986\n110015\n\nBoth train and test set have overlap with old and new data. \n\nIf the train label (the loyalty score) is computed based on both hist and new data, and so is the predicted score on test set, then why do we even need to distinguish between hist and new transactions?\n\nSo it seems the logical explanation would be that, the training label are measured based on only historical transactions, whereas when we're making predictions on test set, we need to consider both hist and new transactions?\n\nAnother possible interpretation would be the loyalty score correlates with the DELTA between the two time periods? For instance, if a card_id appear in hist transaction but no in new transactions, it means the card holder doesn't use ELO payment for new merchants (but may still stick with old merchants), and we expect the loyalty score might be lower in this case?\n\nCan someone confirm/comment on my conjectures above?", "votes": 2 }, { "id": 439468, "postDate": "2018-12-15T14:37:28.427Z", "content": "hi sir ! am doing my final year project in analysis of tweets for election , i used your dataset but cannot understand some attributes like contribution coordination indices entities etc can you please help out me so i can understand tyhe dataset and can you provide me the dataset in csv format??
", "rawMarkdown": "hi sir ! am doing my final year project in analysis of tweets for election , i used your dataset but cannot understand some attributes like contribution coordination indices entities etc can you please help out me so i can understand tyhe dataset and can you provide me the dataset in csv format??\n", "votes": -1 }, { "id": 429860, "postDate": "2018-11-29T13:24:47.877Z", "content": "I think it is a matter of terminology, as Diana nicely explained. Cards (infact card holders) have visited merchants in 2017 and 2018. Mechants that were visited only in 2017, in 2017 and 2018 are not new. However, merchants that were visited in 2018 but were visited in 2017 are new merchants.
", "rawMarkdown": "I think it is a matter of terminology, as Diana nicely explained. Cards (infact card holders) have visited merchants in 2017 and 2018. Mechants that were visited only in 2017, in 2017 and 2018 are not new. However, merchants that were visited in 2018 but were visited in 2017 are new merchants. ", "replies": [ { "id": 430161, "postDate": "2018-11-29T23:30:24.247Z", "content": "\"merchants that were visited in 2018 but were visited in 2018 are new merchants.\"
\n\nI'm even more confused now.
", "rawMarkdown": "\"merchants that were visited in 2018 but were visited in 2018 are new merchants.\"\n\nI'm even more confused now." }, { "id": 430216, "postDate": "2018-11-30T03:12:12.567Z", "content": "I think they meant to say 'merchants that weren't visited in 2017 but were visited in 2018 are new merchants'
", "rawMarkdown": "I think they meant to say 'merchants that weren't visited in 2017 but were visited in 2018 are new merchants'", "votes": 3 }, { "id": 430235, "postDate": "2018-11-30T03:37:56.443Z", "content": "Yeah I figured. I thought it was just sort of funny that we're guessing about what a line offering clarification means. :)
", "rawMarkdown": "Yeah I figured. I thought it was just sort of funny that we're guessing about what a line offering clarification means. :)" } ] }, { "id": 439350, "postDate": "2018-12-15T08:18:00.123Z", "rawMarkdown": "", "isDeleted": true }, { "id": 431564, "postDate": "2018-12-02T13:55:06.063Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 429445, "author_name": "Diana Kolusheva", "author_url": "", "post_date": "2018-11-28T21:24:42.647000", "content": "These two files have data for two different periods and for two different groups of merchants. If you look at the purchase dates in both files, you'll see that historical_transactions.csv shows data for few months in 2017, while new_merchants_transactions shows data for a few months in 2018 for merchants that were not visited in the historical data. This means that even if customers purchased from the same merchant in both periods, these merchants will not be shown in new merchants, only those that are new (not visited in first period) will be shown in second.
", "votes": 16, "replies": [ { "id": 429489, "author_name": "blissweb", "author_url": "", "post_date": "2018-11-29T00:00:45.900000", "content": "Ha. Ok Diana that sounds like it makes sense. I will have a think about what you wrote and ask more if I'm still puzzled.
", "votes": 0, "replies": [] }, { "id": 439352, "author_name": "_krisha_n_", "author_url": "", "post_date": "2018-12-15T08:18:52.970000", "content": "I think I understand the difference between new_merchant_transactions.csv and merchants.csv. But why is it even split like this? the new_merchant_transactions file is essentially censoring particular merchants ... is it a matter of anonymising the data?
", "votes": 0, "replies": [] } ] }, { "id": 429439, "author_name": "Peter Hurford", "author_url": "", "post_date": "2018-11-28T21:11:42.183000", "content": "I've been competing on Kaggle for awhile now and I'm also confused by this.
", "votes": 3, "replies": [] }, { "id": 430074, "author_name": "deepali agnihotri", "author_url": "", "post_date": "2018-11-29T19:34:35.113000", "content": "Thanks @Diana Kolusheva,
\n@blissweb, I am new too :-) I gave it a shot while exploring the txn data.
Following are my observations
\n\nThere are 217808 common merchant_id and 290001 common card_id between the two txn tables when looked in isolation
\n\nprint(len(set(df_newMerchTxn['merchant_id']).intersection(set(df_histTxn['merchant_id']))))\n print(len(set(df_newMerchTxn['card_id']).intersection(set(df_histTxn['card_id']))))\n\n
however if you create the unique id by combining/concatenating the merchent_id and Card_id in both datasets, there are no common unique ids (except 1 nan due to the missing combination attributed to the data quality)
\n\n`df_newMerchTxn['merchant_card_id']= df_newMerchTxn['merchant_id']+df_newMerchTxn['card_id']\ndf_histTxn['merchant_card_id']= df_histTxn['merchant_id']+df_histTxn['card_id']\nprint((set(df_newMerchTxn['merchant_card_id']).intersection(set(df_histTxn['merchant_card_id']))))`\n\n
This means, the way to read the provided comments about new_merchant_transactions table is \"combination of customers [i] who shopped at merchent [y] in historical_transactions table will not be available in the new_merchant_transactions and vice-versa\"
\n\nI hope this helps, please feel free to comment/ correct me if this is not right.
", "votes": 4, "replies": [] }, { "id": 431751, "author_name": "Taylor Zhang", "author_url": "", "post_date": "2018-12-02T21:31:35.880000", "content": "Ran some analysis, it seems we can indeed get some insights about loyalty score from the difference between historical and new merchants. Hope this helps
\n\n", "votes": 1, "replies": [] }, { "id": 431416, "author_name": "Taylor Zhang", "author_url": "", "post_date": "2018-12-02T08:37:20.970000", "content": "The explanations from @deepali clarified some of things, but I'm still confused by the relationship of the hist/new transactions with train/test set.
\n\nHere's the overlap of card ids I got :
\n\nprint(len(set(hist_trans[\"card_id\"]) & set(train[\"card_id\"])))\nprint(len(set(hist_trans[\"card_id\"]) & set(test[\"card_id\"])))\nprint(len(set(new_trans[\"card_id\"]) & set(train[\"card_id\"])))\nprint(len(set(new_trans[\"card_id\"]) & set(test[\"card_id\"])))
\n\n201917\n123623\n179986\n110015
\n\nBoth train and test set have overlap with old and new data.
\n\nIf the train label (the loyalty score) is computed based on both hist and new data, and so is the predicted score on test set, then why do we even need to distinguish between hist and new transactions?
\n\nSo it seems the logical explanation would be that, the training label are measured based on only historical transactions, whereas when we're making predictions on test set, we need to consider both hist and new transactions?
\n\nAnother possible interpretation would be the loyalty score correlates with the DELTA between the two time periods? For instance, if a card_id appear in hist transaction but no in new transactions, it means the card holder doesn't use ELO payment for new merchants (but may still stick with old merchants), and we expect the loyalty score might be lower in this case?
\n\nCan someone confirm/comment on my conjectures above?
", "votes": 2, "replies": [] }, { "id": 439468, "author_name": "junaid abbas", "author_url": "", "post_date": "2018-12-15T14:37:28.427000", "content": "hi sir ! am doing my final year project in analysis of tweets for election , i used your dataset but cannot understand some attributes like contribution coordination indices entities etc can you please help out me so i can understand tyhe dataset and can you provide me the dataset in csv format??
", "votes": -1, "replies": [] }, { "id": 429860, "author_name": "predictor", "author_url": "", "post_date": "2018-11-29T13:24:47.877000", "content": "I think it is a matter of terminology, as Diana nicely explained. Cards (infact card holders) have visited merchants in 2017 and 2018. Mechants that were visited only in 2017, in 2017 and 2018 are not new. However, merchants that were visited in 2018 but were visited in 2017 are new merchants.
", "votes": 0, "replies": [ { "id": 430161, "author_name": "Jesse Nicholson", "author_url": "", "post_date": "2018-11-29T23:30:24.247000", "content": "\"merchants that were visited in 2018 but were visited in 2018 are new merchants.\"
\n\nI'm even more confused now.
", "votes": 0, "replies": [] }, { "id": 430216, "author_name": "Emma Walker", "author_url": "", "post_date": "2018-11-30T03:12:12.567000", "content": "I think they meant to say 'merchants that weren't visited in 2017 but were visited in 2018 are new merchants'
", "votes": 3, "replies": [] }, { "id": 430235, "author_name": "Jesse Nicholson", "author_url": "", "post_date": "2018-11-30T03:37:56.443000", "content": "Yeah I figured. I thought it was just sort of funny that we're guessing about what a line offering clarification means. :)
", "votes": 0, "replies": [] } ] }, { "id": 439350, "author_name": "", "author_url": "", "post_date": "2018-12-15T08:18:00.123000", "content": "", "votes": 0, "replies": [] }, { "id": 431564, "author_name": "", "author_url": "", "post_date": "2018-12-02T13:55:06.063000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "429445": "These two files have data for two different periods and for two different groups of merchants. If you look at the purchase dates in both files, you'll see that historical_transactions.csv shows data for few months in 2017, while new_merchants_transactions shows data for a few months in 2018 for merchants that were not visited in the historical data. This means that even if customers purchased from the same merchant in both periods, these merchants will not be shown in new merchants, only those that are new (not visited in first period) will be shown in second. ", "429378": "Hi,\nThis is my first go at Kaggle so be gentle. :-)\nI'm reading the descriptions and I don't quite get the New Merchants description. It says these are transactions at New Merchants (ones that the card has not yet visited in 2 months). \nMy question is ... if they've not yet visited these merchants, how can there by transactions for that card id in the file ?? \nSomeone please help me understand how the New Merchants file is working.\nThanks,\nShaun", "429439": "I've been competing on Kaggle for awhile now and I'm also confused by this.", "430074": "Thanks @[Diana Kolusheva][1], \n@blissweb, I am new too :-) I gave it a shot while exploring the txn data. \n\nFollowing are my observations\nprint(len(set(df_newMerchTxn['merchant_id']).intersection(set(df_histTxn['merchant_id']))))\n print(len(set(df_newMerchTxn['card_id']).intersection(set(df_histTxn['card_id']))))\n\nhowever if you create the unique id by combining/concatenating the merchent_id and Card_id in both datasets, there are no common unique ids (except 1 nan due to the missing combination attributed to the data quality)\n\n
`df_newMerchTxn['merchant_card_id']= df_newMerchTxn['merchant_id']+df_newMerchTxn['card_id']\ndf_histTxn['merchant_card_id']= df_histTxn['merchant_id']+df_histTxn['card_id']\nprint((set(df_newMerchTxn['merchant_card_id']).intersection(set(df_histTxn['merchant_card_id']))))`\n\nThis means, the way to read the provided comments about new_merchant_transactions table is \"combination of customers [i] who shopped at merchent [y] in historical_transactions table will not be available in the new_merchant_transactions and vice-versa\"\n\nI hope this helps, please feel free to comment/ correct me if this is not right.\n\n\n [1]: https://www.kaggle.com/dianakolusheva", "431751": "Ran some analysis, it seems we can indeed get some insights about loyalty score from the difference between historical and new merchants. Hope this helps \n\nhttps://www.kaggle.com/firstprayer/understand-loyalty-score-w-r-t-hist-vs-new?scriptVersionId=8008055 ", "431416": "The explanations from @deepali clarified some of things, but I'm still confused by the relationship of the hist/new transactions with train/test set.\n\nHere's the overlap of card ids I got :\n\nprint(len(set(hist_trans[\"card_id\"]) & set(train[\"card_id\"])))\nprint(len(set(hist_trans[\"card_id\"]) & set(test[\"card_id\"])))\nprint(len(set(new_trans[\"card_id\"]) & set(train[\"card_id\"])))\nprint(len(set(new_trans[\"card_id\"]) & set(test[\"card_id\"])))\n\n201917\n123623\n179986\n110015\n\nBoth train and test set have overlap with old and new data. \n\nIf the train label (the loyalty score) is computed based on both hist and new data, and so is the predicted score on test set, then why do we even need to distinguish between hist and new transactions?\n\nSo it seems the logical explanation would be that, the training label are measured based on only historical transactions, whereas when we're making predictions on test set, we need to consider both hist and new transactions?\n\nAnother possible interpretation would be the loyalty score correlates with the DELTA between the two time periods? For instance, if a card_id appear in hist transaction but no in new transactions, it means the card holder doesn't use ELO payment for new merchants (but may still stick with old merchants), and we expect the loyalty score might be lower in this case?\n\nCan someone confirm/comment on my conjectures above?", "439468": "hi sir ! am doing my final year project in analysis of tweets for election , i used your dataset but cannot understand some attributes like contribution coordination indices entities etc can you please help out me so i can understand tyhe dataset and can you provide me the dataset in csv format??\n", "429860": "I think it is a matter of terminology, as Diana nicely explained. Cards (infact card holders) have visited merchants in 2017 and 2018. Mechants that were visited only in 2017, in 2017 and 2018 are not new. However, merchants that were visited in 2018 but were visited in 2017 are new merchants. ", "439350": "", "431564": "" } }