There are 63 merchant_ids duplicated. How can we interpret this?
\n\nOne solution is to just delete the duplicated rows, but the issue is that some other features have different values, so we would have to combine them in some way or choose arbitrarily one or the other.
\n\nFor example, look at id: M_ID_c0b712e11a
", "messages": [ { "id": "429125", "postDate": "11/28/2018 11:40:22", "content": "There are 63 merchant_ids duplicated. How can we interpret this?
\n\nOne solution is to just delete the duplicated rows, but the issue is that some other features have different values, so we would have to combine them in some way or choose arbitrarily one or the other.
\n\nFor example, look at id: M_ID_c0b712e11a
", "rawMarkdown": "There are 63 merchant_ids duplicated. How can we interpret this? \n\nOne solution is to just delete the duplicated rows, but the issue is that some other features have different values, so we would have to combine them in some way or choose arbitrarily one or the other.\n\nFor example, look at id: M_ID_c0b712e11a", "votes": null }, { "id": "429153", "postDate": "11/28/2018 12:30:02", "content": "Good catch, thanks.
", "rawMarkdown": "Good catch, thanks.", "votes": null }, { "id": "429233", "postDate": "11/28/2018 15:03:39", "content": "i try to do merchant.drop_duplicates(subset=['merchant_id', 'subsector_id', 'merchant_category_id'], keep='last') , but it still has 1 merchant_ids duplicated
", "rawMarkdown": "i try to do merchant.drop_duplicates(subset=['merchant_id', 'subsector_id', 'merchant_category_id'], keep='last') , but it still has 1 merchant_ids duplicated", "votes": null }, { "id": "429449", "postDate": "11/28/2018 21:35:52", "content": "Can it be that the merchant belongs to more than one category? I haven't checked it yet, which features have different values?
", "rawMarkdown": "Can it be that the merchant belongs to more than one category? I haven't checked it yet, which features have different values?", "votes": null }, { "id": "431210", "postDate": "12/01/2018 21:03:34", "content": "Actually, there are 41 merchant_ids that have more than one entry in merchants.csv: 30 merchant_ids have 2 entries and 11 have 4 entries.
\n\nWith one exception (M_ID_645a6af169), all the duplicates seem to have the same information for merchant_id, merchant_category_id, subsector_id, city_id, state_id, and the differences are in combinations of other parameters (numerical_1, numerical_2, *avg_sales_lagN, avg_purchases_lagN, etc.). My guess is that it is the same merchant with a different snapshot of information taken at different times.
\n\nThe two rows for merchant_id M_ID_645a6af169 differ in city, state and category information. To me this might mean an error in the identification of the merchant.
\n\nWhat perturbs me the most is the difference in some of these categories between merchants.csv and *historical_transactions.csv*. The columns merchant_id, merchant_category_id, subsector_id, city_id, state_id appear in both files but there seems to be significant differences in city_id and state_id between the two.
\n\nIn merchants.csv, there are 105K merchant_ids (or 31.5%) that have a city_id equal to -1. Of these, only about 10K merchant_ids also have a city_id equal to -1 in *historical_transactions.csv*. Another 97.5k merchant_ids have a city_id other than -1. There is about 2.5K merchant_ids that have city_id equal to -1 and then another value.
\n\nSo trying to merge *historical_transactions.csv* with merchants.csv over the 5 id parameters proves impossible. Roughly 35% of rows in *historical_transactions.csv* have no match in merchants.csv when you try to merge using merchant_id, merchant_category_id, subsector_id, city_id, and state_id.
", "rawMarkdown": "Actually, there are 41 *merchant\\_ids* that have more than one entry in *merchants.csv*: 30 *merchant\\_ids* have 2 entries and 11 have 4 entries.\n\nWith one exception (*M\\_ID\\_645a6af169*), all the duplicates seem to have the same information for *merchant\\_id*, *merchant\\_category\\_id*, *subsector\\_id*, *city\\_id*, *state\\_id*, and the differences are in combinations of other parameters (*numerical\\_1*, *numerical\\_2*, *avg\\_sales\\_lagN, *avg\\_purchases\\_lagN*, etc.). My guess is that it is the same merchant with a different snapshot of information taken at different times.\n\nThe two rows for merchant\\_id *M\\_ID\\_645a6af169* differ in city, state and category information. To me this might mean an error in the identification of the merchant. \n\nWhat perturbs me the most is the difference in some of these categories between *merchants.csv* and *historical_transactions.csv*. The columns *merchant\\_id*, *merchant\\_category\\_id*, *subsector\\_id*, *city\\_id*, *state\\_id* appear in both files but there seems to be significant differences in *city\\_id* and *state\\_id* between the two.\n\nIn *merchants.csv*, there are 105K *merchant\\_ids* (or 31.5%) that have a *city\\_id* equal to -1. Of these, only about 10K *merchant\\_ids* also have a *city\\_id* equal to -1 in *historical_transactions.csv*. Another 97.5k *merchant\\_ids* have a *city\\_id* other than -1. There is about 2.5K *merchant\\_ids* that have *city\\_id* equal to -1 and then another value. \n\nSo trying to merge *historical_transactions.csv* with *merchants.csv* over the 5 id parameters proves impossible. Roughly 35% of rows in *historical_transactions.csv* have no match in *merchants.csv* when you try to merge using *merchant\\_id*, *merchant\\_category\\_id*, *subsector\\_id*, *city\\_id*, and *state\\_id*.", "votes": null }, { "id": "431237", "postDate": "12/01/2018 22:58:13", "content": "Generaly no. We have exceptation - 41 of the merchnat_id are duplicated. They have the same category_1 and category_4 parameters, but for example merchant_group_id can be different (for example M_ID_1802942aaf ). But we have most of this parameters also in transaction table, so we probably (I have not checked yet) distinguish which of the duplicated record is the proper one.
", "rawMarkdown": "Generaly no. We have exceptation - 41 of the merchnat_id are duplicated. They have the same category_1 and category_4 parameters, but for example merchant_group_id can be different (for example M_ID_1802942aaf ). But we have most of this parameters also in transaction table, so we probably (I have not checked yet) distinguish which of the duplicated record is the proper one.", "votes": null }, { "id": "431440", "postDate": "12/02/2018 09:19:41", "content": "Is it possible that the location data(state/city etc) in merchants.csv relates to the registered location of the merchant whereas in the transactions file refers to the location where the transaction took place. E.g. if I order food from Uber Eats in Bangalore, the transactions file will register it as Bangalore, whereas the merchants file will have Uber Eats' registered location.
", "rawMarkdown": "Is it possible that the location data(state/city etc) in merchants.csv relates to the registered location of the merchant whereas in the transactions file refers to the location where the transaction took place. E.g. if I order food from Uber Eats in Bangalore, the transactions file will register it as Bangalore, whereas the merchants file will have Uber Eats' registered location.", "votes": null }, { "id": "432125", "postDate": "12/03/2018 12:40:34", "content": "That is a possibility but I don't think the \"-1\" means a registered location, more an unknown or ambiguous location.
", "rawMarkdown": "That is a possibility but I don't think the \"-1\" means a registered location, more an unknown or ambiguous location.", "votes": null }, { "id": "433554", "postDate": "12/05/2018 07:09:04", "content": "you are probably right there
", "rawMarkdown": "you are probably right there", "votes": null }, { "id": "445032", "postDate": "12/25/2018 12:05:29", "content": "Nice!
", "rawMarkdown": "Nice!", "votes": null }, { "id": "445419", "postDate": "12/26/2018 12:11:03", "content": "Thank you, Rubin. I am confuse about this problem too, and I am puzzled about the facts that the same merchant_id has different merchant_category_id, subsector_id, etc. in historical_transactions.csv and merchants.csv file.
It is better to merge with more information rather than just merchant id, like you can merge with merchant_id,category_1,etc. This can help a little.
", "rawMarkdown": "It is better to merge with more information rather than just merchant id, like you can merge with merchant_id,category_1,etc. This can help a little.", "votes": null }, { "id": "463601", "postDate": "01/30/2019 09:12:44", "content": "Few categorical variables for the duplicate merchants_ids are also different! (If you compare the 1st id itself, it repeats 4 times, with different categorical variable values). Norman's guess could be right that it's the same merchant with a different snapshot of information taken at different times. But how do I group these together? I can group continuous variables by making columns like mean, std dev etc., but how to treat cat vars? \nThanks.
", "rawMarkdown": "Few categorical variables for the duplicate merchants_ids are also different! (If you compare the 1st id itself, it repeats 4 times, with different categorical variable values). Norman's guess could be right that it's the same merchant with a different snapshot of information taken at different times. But how do I group these together? I can group continuous variables by making columns like mean, std dev etc., but how to treat cat vars? \nThanks.", "votes": null } ], "comments": [ { "id": 429153, "author_name": "konradb", "author_url": "", "post_date": "11/28/2018 12:30:02", "content": "Good catch, thanks.
", "votes": null, "replies": [] }, { "id": 429233, "author_name": "chenxin1991", "author_url": "", "post_date": "11/28/2018 15:03:39", "content": "i try to do merchant.drop_duplicates(subset=['merchant_id', 'subsector_id', 'merchant_category_id'], keep='last') , but it still has 1 merchant_ids duplicated
", "votes": null, "replies": [] }, { "id": 429449, "author_name": "dianakolusheva", "author_url": "", "post_date": "11/28/2018 21:35:52", "content": "Can it be that the merchant belongs to more than one category? I haven't checked it yet, which features have different values?
", "votes": null, "replies": [ { "id": 431237, "author_name": "malina015", "author_url": "", "post_date": "12/01/2018 22:58:13", "content": "Generaly no. We have exceptation - 41 of the merchnat_id are duplicated. They have the same category_1 and category_4 parameters, but for example merchant_group_id can be different (for example M_ID_1802942aaf ). But we have most of this parameters also in transaction table, so we probably (I have not checked yet) distinguish which of the duplicated record is the proper one.
", "votes": null, "replies": [] } ] }, { "id": 431210, "author_name": "nsecord", "author_url": "", "post_date": "12/01/2018 21:03:34", "content": "Actually, there are 41 merchant_ids that have more than one entry in merchants.csv: 30 merchant_ids have 2 entries and 11 have 4 entries.
\n\nWith one exception (M_ID_645a6af169), all the duplicates seem to have the same information for merchant_id, merchant_category_id, subsector_id, city_id, state_id, and the differences are in combinations of other parameters (numerical_1, numerical_2, *avg_sales_lagN, avg_purchases_lagN, etc.). My guess is that it is the same merchant with a different snapshot of information taken at different times.
\n\nThe two rows for merchant_id M_ID_645a6af169 differ in city, state and category information. To me this might mean an error in the identification of the merchant.
\n\nWhat perturbs me the most is the difference in some of these categories between merchants.csv and *historical_transactions.csv*. The columns merchant_id, merchant_category_id, subsector_id, city_id, state_id appear in both files but there seems to be significant differences in city_id and state_id between the two.
\n\nIn merchants.csv, there are 105K merchant_ids (or 31.5%) that have a city_id equal to -1. Of these, only about 10K merchant_ids also have a city_id equal to -1 in *historical_transactions.csv*. Another 97.5k merchant_ids have a city_id other than -1. There is about 2.5K merchant_ids that have city_id equal to -1 and then another value.
\n\nSo trying to merge *historical_transactions.csv* with merchants.csv over the 5 id parameters proves impossible. Roughly 35% of rows in *historical_transactions.csv* have no match in merchants.csv when you try to merge using merchant_id, merchant_category_id, subsector_id, city_id, and state_id.
", "votes": null, "replies": [ { "id": 431440, "author_name": "abhimanyud", "author_url": "", "post_date": "12/02/2018 09:19:41", "content": "Is it possible that the location data(state/city etc) in merchants.csv relates to the registered location of the merchant whereas in the transactions file refers to the location where the transaction took place. E.g. if I order food from Uber Eats in Bangalore, the transactions file will register it as Bangalore, whereas the merchants file will have Uber Eats' registered location.
", "votes": null, "replies": [] }, { "id": 432125, "author_name": "nsecord", "author_url": "", "post_date": "12/03/2018 12:40:34", "content": "That is a possibility but I don't think the \"-1\" means a registered location, more an unknown or ambiguous location.
", "votes": null, "replies": [] }, { "id": 433554, "author_name": "abhimanyud", "author_url": "", "post_date": "12/05/2018 07:09:04", "content": "you are probably right there
", "votes": null, "replies": [] } ] }, { "id": 445032, "author_name": "maparla", "author_url": "", "post_date": "12/25/2018 12:05:29", "content": "Nice!
", "votes": null, "replies": [] }, { "id": 445419, "author_name": "liewmanchoi", "author_url": "", "post_date": "12/26/2018 12:11:03", "content": "Thank you, Rubin. I am confuse about this problem too, and I am puzzled about the facts that the same merchant_id has different merchant_category_id, subsector_id, etc. in historical_transactions.csv and merchants.csv file.
It is better to merge with more information rather than just merchant id, like you can merge with merchant_id,category_1,etc. This can help a little.
", "votes": null, "replies": [] }, { "id": 463601, "author_name": "pranavmahajan725", "author_url": "", "post_date": "01/30/2019 09:12:44", "content": "Few categorical variables for the duplicate merchants_ids are also different! (If you compare the 1st id itself, it repeats 4 times, with different categorical variable values). Norman's guess could be right that it's the same merchant with a different snapshot of information taken at different times. But how do I group these together? I can group continuous variables by making columns like mean, std dev etc., but how to treat cat vars? \nThanks.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "429125": "There are 63 merchant_ids duplicated. How can we interpret this? \n\nOne solution is to just delete the duplicated rows, but the issue is that some other features have different values, so we would have to combine them in some way or choose arbitrarily one or the other.\n\nFor example, look at id: M_ID_c0b712e11a", "429153": "Good catch, thanks.", "429233": "i try to do merchant.drop_duplicates(subset=['merchant_id', 'subsector_id', 'merchant_category_id'], keep='last') , but it still has 1 merchant_ids duplicated", "429449": "Can it be that the merchant belongs to more than one category? I haven't checked it yet, which features have different values?", "431210": "Actually, there are 41 *merchant\\_ids* that have more than one entry in *merchants.csv*: 30 *merchant\\_ids* have 2 entries and 11 have 4 entries.\n\nWith one exception (*M\\_ID\\_645a6af169*), all the duplicates seem to have the same information for *merchant\\_id*, *merchant\\_category\\_id*, *subsector\\_id*, *city\\_id*, *state\\_id*, and the differences are in combinations of other parameters (*numerical\\_1*, *numerical\\_2*, *avg\\_sales\\_lagN, *avg\\_purchases\\_lagN*, etc.). My guess is that it is the same merchant with a different snapshot of information taken at different times.\n\nThe two rows for merchant\\_id *M\\_ID\\_645a6af169* differ in city, state and category information. To me this might mean an error in the identification of the merchant. \n\nWhat perturbs me the most is the difference in some of these categories between *merchants.csv* and *historical_transactions.csv*. The columns *merchant\\_id*, *merchant\\_category\\_id*, *subsector\\_id*, *city\\_id*, *state\\_id* appear in both files but there seems to be significant differences in *city\\_id* and *state\\_id* between the two.\n\nIn *merchants.csv*, there are 105K *merchant\\_ids* (or 31.5%) that have a *city\\_id* equal to -1. Of these, only about 10K *merchant\\_ids* also have a *city\\_id* equal to -1 in *historical_transactions.csv*. Another 97.5k *merchant\\_ids* have a *city\\_id* other than -1. There is about 2.5K *merchant\\_ids* that have *city\\_id* equal to -1 and then another value. \n\nSo trying to merge *historical_transactions.csv* with *merchants.csv* over the 5 id parameters proves impossible. Roughly 35% of rows in *historical_transactions.csv* have no match in *merchants.csv* when you try to merge using *merchant\\_id*, *merchant\\_category\\_id*, *subsector\\_id*, *city\\_id*, and *state\\_id*.", "431237": "Generaly no. We have exceptation - 41 of the merchnat_id are duplicated. They have the same category_1 and category_4 parameters, but for example merchant_group_id can be different (for example M_ID_1802942aaf ). But we have most of this parameters also in transaction table, so we probably (I have not checked yet) distinguish which of the duplicated record is the proper one.", "431440": "Is it possible that the location data(state/city etc) in merchants.csv relates to the registered location of the merchant whereas in the transactions file refers to the location where the transaction took place. E.g. if I order food from Uber Eats in Bangalore, the transactions file will register it as Bangalore, whereas the merchants file will have Uber Eats' registered location.", "432125": "That is a possibility but I don't think the \"-1\" means a registered location, more an unknown or ambiguous location.", "433554": "you are probably right there", "445032": "Nice!", "445419": "Thank you, Rubin. I am confuse about this problem too, and I am puzzled about the facts that the same `merchant_id` has different `merchant_category_id`, `subsector_id`, etc. in `historical_transactions.csv` and `merchants.csv` file.", "457871": "It is better to merge with more information rather than just merchant id, like you can merge with merchant_id,category_1,etc. This can help a little.", "463601": "Few categorical variables for the duplicate merchants_ids are also different! (If you compare the 1st id itself, it repeats 4 times, with different categorical variable values). Norman's guess could be right that it's the same merchant with a different snapshot of information taken at different times. But how do I group these together? I can group continuous variables by making columns like mean, std dev etc., but how to treat cat vars? \nThanks." }, "source": "meta" }