I was able to come up with a way to use subsector_id in my model. I found the three most common sectors ids for each card_id and added them as sector_1, sector_2, and sector_3. When I tried adding a sector_4 variable or deleting the sector_3 variable, it decreased the accuracy of my model. Here is the r code that I used to do this.
\n\ntransactionData$subsector_id=as.factor(transactionData$subsector_id)\nhistTransactSummariesSector <- transactionData %>%\n group_by(card_id, subsector_id) %>% \n summarise(n=n()) %>%\n ungroup() %>%\n spread(subsector_id, n, fill=0)
\n\nsectorMatrix=as.matrix(histTransactSummariesSector[,-1])\ntop3 <- t(apply(sectorMatrix, 1, function(a) order(a,decreasing=TRUE)[1:3]))\ntop3[] <- colnames(histTransactSummariesSector[,-1])[top3]
\n\nhistTransactSummaries$sector_1=as.numeric(top3[,1])\nhistTransactSummaries$sector_2=as.numeric(top3[,2])\nhistTransactSummaries$sector_3=as.numeric(top3[,3])
", "messages": [ { "id": "467414", "postDate": "02/07/2019 03:38:01", "content": "I was able to come up with a way to use subsector_id in my model. I found the three most common sectors ids for each card_id and added them as sector_1, sector_2, and sector_3. When I tried adding a sector_4 variable or deleting the sector_3 variable, it decreased the accuracy of my model. Here is the r code that I used to do this.
\n\ntransactionData$subsector_id=as.factor(transactionData$subsector_id)\nhistTransactSummariesSector <- transactionData %>%\n group_by(card_id, subsector_id) %>% \n summarise(n=n()) %>%\n ungroup() %>%\n spread(subsector_id, n, fill=0)
\n\nsectorMatrix=as.matrix(histTransactSummariesSector[,-1])\ntop3 <- t(apply(sectorMatrix, 1, function(a) order(a,decreasing=TRUE)[1:3]))\ntop3[] <- colnames(histTransactSummariesSector[,-1])[top3]
\n\nhistTransactSummaries$sector_1=as.numeric(top3[,1])\nhistTransactSummaries$sector_2=as.numeric(top3[,2])\nhistTransactSummaries$sector_3=as.numeric(top3[,3])
", "rawMarkdown": "I was able to come up with a way to use subsector_id in my model. I found the three most common sectors ids for each card_id and added them as sector_1, sector_2, and sector_3. When I tried adding a sector_4 variable or deleting the sector_3 variable, it decreased the accuracy of my model. Here is the r code that I used to do this.\n\ntransactionData$subsector_id=as.factor(transactionData$subsector_id)\nhistTransactSummariesSector <- transactionData %>%\n group_by(card_id, subsector_id) %>% \n summarise(n=n()) %>%\n ungroup() %>%\n spread(subsector_id, n, fill=0)\n\n\nsectorMatrix=as.matrix(histTransactSummariesSector[,-1])\ntop3 <- t(apply(sectorMatrix, 1, function(a) order(a,decreasing=TRUE)[1:3]))\ntop3[] <- colnames(histTransactSummariesSector[,-1])[top3]\n\n\nhistTransactSummaries$sector_1=as.numeric(top3[,1])\nhistTransactSummaries$sector_2=as.numeric(top3[,2])\nhistTransactSummaries$sector_3=as.numeric(top3[,3])", "votes": null }, { "id": "467450", "postDate": "02/07/2019 06:43:02", "content": "I also attempted to extract the mode value from the transaction and use subsector_id and city_id. As a result, it was confirmed that excessive learning occurred and accuracy got worse. However, by performing one-hot-encording at the stage of transaction data, accuracy will improve somewhat.\nHowever, because the data is very large, it is difficult to create the data on an ordinary PC.
", "rawMarkdown": "I also attempted to extract the mode value from the transaction and use subsector_id and city_id. As a result, it was confirmed that excessive learning occurred and accuracy got worse. However, by performing one-hot-encording at the stage of transaction data, accuracy will improve somewhat.\nHowever, because the data is very large, it is difficult to create the data on an ordinary PC.", "votes": null }, { "id": "469917", "postDate": "02/12/2019 03:33:32", "content": "@ryuichigt, thank you for your insight!
\n\nMay I know more about \"by performing one-hot-encording at the stage of transaction data\"? Do you mean before feature aggregation for hist/new transaction data?
", "rawMarkdown": "ryuichigt, thank you for your insight!\n\nMay I know more about \"by performing one-hot-encording at the stage of transaction data\"? Do you mean before feature aggregation for hist/new transaction data?", "votes": null }, { "id": "469981", "postDate": "02/12/2019 07:01:23", "content": "exactly.\nWe used a dummy for transaction data and calculated the average purchasing district and so on. By using that data, you can lower it to 3.648.
", "rawMarkdown": "exactly.\nWe used a dummy for transaction data and calculated the average purchasing district and so on. By using that data, you can lower it to 3.648.", "votes": null }, { "id": "470084", "postDate": "02/12/2019 10:48:58", "content": "@ryuichigt, thank you again for your hint! We got stuck for a while. May I know whether you have done any parameter tuning after creating such features? We have no luck after creating new features, probably because of parameter tuning. It has been two weeks and it's really frustrating.
", "rawMarkdown": "ryuichigt, thank you again for your hint! We got stuck for a while. May I know whether you have done any parameter tuning after creating such features? We have no luck after creating new features, probably because of parameter tuning. It has been two weeks and it's really frustrating.", "votes": null }, { "id": "470085", "postDate": "02/12/2019 10:49:23", "content": "@ryuichigt, again, can't thank you more for your insight.
", "rawMarkdown": "ryuichigt, again, can't thank you more for your insight.", "votes": null }, { "id": "470155", "postDate": "02/12/2019 13:34:21", "content": "I am glad you said so.\nIn addition, please check out more Brazilian events. I can find features with a correlation coefficient of 0.14 with the target. They will help you.
", "rawMarkdown": "I am glad you said so.\nIn addition, please check out more Brazilian events. I can find features with a correlation coefficient of 0.14 with the target. They will help you.", "votes": null } ], "comments": [ { "id": 467450, "author_name": "ryuichigt", "author_url": "", "post_date": "02/07/2019 06:43:02", "content": "I also attempted to extract the mode value from the transaction and use subsector_id and city_id. As a result, it was confirmed that excessive learning occurred and accuracy got worse. However, by performing one-hot-encording at the stage of transaction data, accuracy will improve somewhat.\nHowever, because the data is very large, it is difficult to create the data on an ordinary PC.
", "votes": null, "replies": [ { "id": 469917, "author_name": "pukkinming", "author_url": "", "post_date": "02/12/2019 03:33:32", "content": "@ryuichigt, thank you for your insight!
\n\nMay I know more about \"by performing one-hot-encording at the stage of transaction data\"? Do you mean before feature aggregation for hist/new transaction data?
", "votes": null, "replies": [] }, { "id": 469981, "author_name": "ryuichigt", "author_url": "", "post_date": "02/12/2019 07:01:23", "content": "exactly.\nWe used a dummy for transaction data and calculated the average purchasing district and so on. By using that data, you can lower it to 3.648.
", "votes": null, "replies": [] }, { "id": 470084, "author_name": "pukkinming", "author_url": "", "post_date": "02/12/2019 10:48:58", "content": "@ryuichigt, thank you again for your hint! We got stuck for a while. May I know whether you have done any parameter tuning after creating such features? We have no luck after creating new features, probably because of parameter tuning. It has been two weeks and it's really frustrating.
", "votes": null, "replies": [] }, { "id": 470085, "author_name": "pukkinming", "author_url": "", "post_date": "02/12/2019 10:49:23", "content": "@ryuichigt, again, can't thank you more for your insight.
", "votes": null, "replies": [] }, { "id": 470155, "author_name": "ryuichigt", "author_url": "", "post_date": "02/12/2019 13:34:21", "content": "I am glad you said so.\nIn addition, please check out more Brazilian events. I can find features with a correlation coefficient of 0.14 with the target. They will help you.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "467414": "I was able to come up with a way to use subsector_id in my model. I found the three most common sectors ids for each card_id and added them as sector_1, sector_2, and sector_3. When I tried adding a sector_4 variable or deleting the sector_3 variable, it decreased the accuracy of my model. Here is the r code that I used to do this.\n\ntransactionData$subsector_id=as.factor(transactionData$subsector_id)\nhistTransactSummariesSector <- transactionData %>%\n group_by(card_id, subsector_id) %>% \n summarise(n=n()) %>%\n ungroup() %>%\n spread(subsector_id, n, fill=0)\n\n\nsectorMatrix=as.matrix(histTransactSummariesSector[,-1])\ntop3 <- t(apply(sectorMatrix, 1, function(a) order(a,decreasing=TRUE)[1:3]))\ntop3[] <- colnames(histTransactSummariesSector[,-1])[top3]\n\n\nhistTransactSummaries$sector_1=as.numeric(top3[,1])\nhistTransactSummaries$sector_2=as.numeric(top3[,2])\nhistTransactSummaries$sector_3=as.numeric(top3[,3])", "467450": "I also attempted to extract the mode value from the transaction and use subsector_id and city_id. As a result, it was confirmed that excessive learning occurred and accuracy got worse. However, by performing one-hot-encording at the stage of transaction data, accuracy will improve somewhat.\nHowever, because the data is very large, it is difficult to create the data on an ordinary PC.", "469917": "ryuichigt, thank you for your insight!\n\nMay I know more about \"by performing one-hot-encording at the stage of transaction data\"? Do you mean before feature aggregation for hist/new transaction data?", "469981": "exactly.\nWe used a dummy for transaction data and calculated the average purchasing district and so on. By using that data, you can lower it to 3.648.", "470084": "ryuichigt, thank you again for your hint! We got stuck for a while. May I know whether you have done any parameter tuning after creating such features? We have no luck after creating new features, probably because of parameter tuning. It has been two weeks and it's really frustrating.", "470085": "ryuichigt, again, can't thank you more for your insight.", "470155": "I am glad you said so.\nIn addition, please check out more Brazilian events. I can find features with a correlation coefficient of 0.14 with the target. They will help you." }, "source": "meta" }