![\"Feature](\"https://inclass.kaggle.com/blobs/download/forum-message-attachment-files/6607/patient8_feature_matrix.png\" "\\"\\"")
The model I used for this competition is XGBoost. Some simple ensembling methods were tried but unsuccessful. So, most of my submissions including the two I finally selected are directly generated from XGBoost.
\n\nThere are 14322 features in total for each instance (patient). After data augment, the training data is an 837552 * 14322 sparse matrix, and the test data a 279152 * 14322 sparse matrix.
\n\nThen I will describe the features and data augment method in details.
\n\nThe first two features are YOB (year of birth) and M/F (gender), and the rest is based on ATC codes.\n716 ATC codes have been considered. Each of them has 20 features. Therefore, 716*20 + 2 = 14322.
\n\nThe figure below is the feature matrix for patient 8 (also attached).\n
The leftmost column is the ATC code column. # is a special symbol can be treated as Level 0 ATC code which contains every drug. Features are calculated for each ATC code from Level 0 to Level 4 that is seen in the test set. In total, there are 716 ATC codes (including #) seen in the test set.
\n\nBeg/End is the first/last transaction date of drugs under each ATC code.\nBeg is in its original format, however, End is represented by the difference between it and 2016-01-01, e.g. -5 means 2015-12-27. The reason will be explained in the Data Augment section.\nDur is the number of days between Beg and End.
\n\nGMed and GMax are the median and maximum gaps between dispense weeks of drugs under each ATC code. For example, a person has purchased C09 drugs for only 4 times, which were in week 2017-04-16, 2017-04-30, 2017-05-07, and 2017-05-28. Then the gaps are 14 days, 7 days, 21 days. The median is 14, the maximum is 21. Duplicates will be removed before calculating GMed and GMax. For example, 2017-04-16, 2017-04-30, 2017-05-07, 2017-05-07, 2017-05-07, and 2017-05-28 will give the same result.
\n\ny-3, y-2, y-1, h1, q3, q4 are histogram features. All of them are counts of transactions. y-3, y-2, y-1 are year 2012, 2013, 2014. h1 is the first half of year 2015. q3 and q4 are the 3rd and 4th quarters of year 2015.
\n\nm06, m12, m24, m36, m48 are cumulative sums of the above histogram features. m06 = q3 + q4. m12 = h1 + m06. m24 = y-1 + m12. m36 = y-2 + m24. m48 = y-3 + m36. They can be considered as numbers of transactions in the last XX months.
\n\nBdif = abs(Beg - #_Beg) + 1, Edif = abs(End - #_End) + 1. For example, #_Beg and #_End for patient 8 are 2011-12-11, and -5 (or 2015-12-27). So A_Edif = abs(A_End - #_End) + 1 = abs(-173-(-5)) + 1 = 169, and A_Bdif = abs(A_Beg - #_Beg) + 1 = 1.
\n\nIntuitively, both features consider the interaction between drugs. Edif is the number of days between the last transaction date of one ATC code drugs and the last transaction date of all drugs. So Edif can help us to identify whether a patient has stopped purchasing some ATC code drugs. Bdif is similar. “+1” is to avoid 0. For patient 8, C07_End == #_End, however, if a patient has never purchased C07 drugs, then his C07_Edif is also 0. “+1” can avoid this ambiguity.
\n\nNPD1 = End + GMed, NPD2 = End + GMax. They are two very simple ways to estimate the next purchase date. When both features of an ATC code are very large negative/positive numbers, that patient is unlikely to purchase that ATC code drugs anymore.
\n\nTo generate the above features, 3 columns, Patient_ID, Drug_ID, and Dispense_Week are first extracted from all the transaction files, then drugs that don’t have ATC codes in the Drug_LookUp.txt are removed from the transactions, and drugs whose ATC codes are not seen in the test set are also removed.
\n\nThe goal of training a model is to learn transferable knowledge. For example, we train a model on one group of patients, then the model should work on another group of patients as well, otherwise it is not a good model.
\n\nSimilarly, the knowledge in a good model should also be transferable across time. We can build a model on transactions between 11-15 to predict 2016. We can also build a model on transactions between 11-14 to predict 2015. And both models should learn similar knowledge. So, it’s reasonable to train one single model which works for both tasks.
\n\nWe originally have
\n\nThen we can create two more training sets:
\n\n(1) + (3) + (4) have 279200 + 279200 + 279152 = 837552 patients in total.
\n\nTo make this possible, we need to design the features so that they are transferable across time. That’s why Beg and End are both dates but have different representation. Beg can be transferable, but End can’t, unless we represent it as the difference between the true End date and a reference date (2015-01-01 or 2016-01-01). This is also true for the histogram features and their cumulative sums. In two cases, the bins used for the histogram are shifted by one year, and the meaning of cumulative sums is also changed accordingly.
\n\nIn this way, we obtain a training set 3 times larger than the original one, which is helpful to prevent overfitting. Besides, when time is not enough for running cross validation, this provides us an alternative way to select model/parameters: use (1) + (4) as the training set and (3) as a development set.
", "comments": [ { "id": 192975, "author_name": "datasings", "author_url": "", "post_date": "2017-06-15T08:49:12.023000", "content": "May I know what's your computer configuration? What kind of tools or packages did you use? and how long it took the computer to train the model? Do you mind to share some of your code ?
", "votes": 0, "replies": [] }, { "id": 188141, "author_name": "JHL", "author_url": "", "post_date": "2017-06-02T04:30:35.520000", "content": "Good stuff :D
", "votes": 0, "replies": [] }, { "id": 188140, "author_name": "Yuan Li", "author_url": "", "post_date": "2017-06-02T04:15:19.660000", "content": "Thanks Jay Han. Yes, that gave rise to a large improvement. Without that, the best two submissions are 0.97032/0.97120 (public/private) and 0.97000/0.97132. With data augment and using the same features, the best two are 0.97116/0.97191 and 0.97101/0.97206.
", "votes": 0, "replies": [] }, { "id": 188124, "author_name": "JHL", "author_url": "", "post_date": "2017-06-02T02:50:43.820000", "content": "This is really good. Thanks for sharing this.
\n\nI have a feeling that possibly the biggest impact is the clever use of test data (training set (4)). Do you know what performance did you get if you only train on (1) and test on (2)? I am curious how much performance gain you get by adding 2 times more data.
", "votes": 0, "replies": [] } ], "comment_count": 4, "messages": [ { "id": 188061, "postDate": "2017-06-01T22:17:26.397Z", "content": "The model I used for this competition is XGBoost. Some simple ensembling methods were tried but unsuccessful. So, most of my submissions including the two I finally selected are directly generated from XGBoost.
\n\nThere are 14322 features in total for each instance (patient). After data augment, the training data is an 837552 * 14322 sparse matrix, and the test data a 279152 * 14322 sparse matrix.
\n\nThen I will describe the features and data augment method in details.
\n\nThe first two features are YOB (year of birth) and M/F (gender), and the rest is based on ATC codes.\n716 ATC codes have been considered. Each of them has 20 features. Therefore, 716*20 + 2 = 14322.
\n\nThe figure below is the feature matrix for patient 8 (also attached).\n
The leftmost column is the ATC code column. # is a special symbol can be treated as Level 0 ATC code which contains every drug. Features are calculated for each ATC code from Level 0 to Level 4 that is seen in the test set. In total, there are 716 ATC codes (including #) seen in the test set.
\n\nBeg/End is the first/last transaction date of drugs under each ATC code.\nBeg is in its original format, however, End is represented by the difference between it and 2016-01-01, e.g. -5 means 2015-12-27. The reason will be explained in the Data Augment section.\nDur is the number of days between Beg and End.
\n\nGMed and GMax are the median and maximum gaps between dispense weeks of drugs under each ATC code. For example, a person has purchased C09 drugs for only 4 times, which were in week 2017-04-16, 2017-04-30, 2017-05-07, and 2017-05-28. Then the gaps are 14 days, 7 days, 21 days. The median is 14, the maximum is 21. Duplicates will be removed before calculating GMed and GMax. For example, 2017-04-16, 2017-04-30, 2017-05-07, 2017-05-07, 2017-05-07, and 2017-05-28 will give the same result.
\n\ny-3, y-2, y-1, h1, q3, q4 are histogram features. All of them are counts of transactions. y-3, y-2, y-1 are year 2012, 2013, 2014. h1 is the first half of year 2015. q3 and q4 are the 3rd and 4th quarters of year 2015.
\n\nm06, m12, m24, m36, m48 are cumulative sums of the above histogram features. m06 = q3 + q4. m12 = h1 + m06. m24 = y-1 + m12. m36 = y-2 + m24. m48 = y-3 + m36. They can be considered as numbers of transactions in the last XX months.
\n\nBdif = abs(Beg - #_Beg) + 1, Edif = abs(End - #_End) + 1. For example, #_Beg and #_End for patient 8 are 2011-12-11, and -5 (or 2015-12-27). So A_Edif = abs(A_End - #_End) + 1 = abs(-173-(-5)) + 1 = 169, and A_Bdif = abs(A_Beg - #_Beg) + 1 = 1.
\n\nIntuitively, both features consider the interaction between drugs. Edif is the number of days between the last transaction date of one ATC code drugs and the last transaction date of all drugs. So Edif can help us to identify whether a patient has stopped purchasing some ATC code drugs. Bdif is similar. “+1” is to avoid 0. For patient 8, C07_End == #_End, however, if a patient has never purchased C07 drugs, then his C07_Edif is also 0. “+1” can avoid this ambiguity.
\n\nNPD1 = End + GMed, NPD2 = End + GMax. They are two very simple ways to estimate the next purchase date. When both features of an ATC code are very large negative/positive numbers, that patient is unlikely to purchase that ATC code drugs anymore.
\n\nTo generate the above features, 3 columns, Patient_ID, Drug_ID, and Dispense_Week are first extracted from all the transaction files, then drugs that don’t have ATC codes in the Drug_LookUp.txt are removed from the transactions, and drugs whose ATC codes are not seen in the test set are also removed.
\n\nThe goal of training a model is to learn transferable knowledge. For example, we train a model on one group of patients, then the model should work on another group of patients as well, otherwise it is not a good model.
\n\nSimilarly, the knowledge in a good model should also be transferable across time. We can build a model on transactions between 11-15 to predict 2016. We can also build a model on transactions between 11-14 to predict 2015. And both models should learn similar knowledge. So, it’s reasonable to train one single model which works for both tasks.
\n\nWe originally have
\n\nThen we can create two more training sets:
\n\n(1) + (3) + (4) have 279200 + 279200 + 279152 = 837552 patients in total.
\n\nTo make this possible, we need to design the features so that they are transferable across time. That’s why Beg and End are both dates but have different representation. Beg can be transferable, but End can’t, unless we represent it as the difference between the true End date and a reference date (2015-01-01 or 2016-01-01). This is also true for the histogram features and their cumulative sums. In two cases, the bins used for the histogram are shifted by one year, and the meaning of cumulative sums is also changed accordingly.
\n\nIn this way, we obtain a training set 3 times larger than the original one, which is helpful to prevent overfitting. Besides, when time is not enough for running cross validation, this provides us an alternative way to select model/parameters: use (1) + (4) as the training set and (3) as a development set.
", "rawMarkdown": "Overview\r\n--------\r\n\r\nThe model I used for this competition is XGBoost. Some simple ensembling methods were tried but unsuccessful. So, most of my submissions including the two I finally selected are directly generated from XGBoost.\r\n\r\nThere are 14322 features in total for each instance (patient). After data augment, the training data is an 837552 * 14322 sparse matrix, and the test data a 279152 * 14322 sparse matrix.\r\n\r\nThen I will describe the features and data augment method in details.\r\n\r\nFeatures\r\n--------\r\n\r\nThe first two features are YOB (year of birth) and M/F (gender), and the rest is based on ATC codes.\r\n716 ATC codes have been considered. Each of them has 20 features. Therefore, 716*20 + 2 = 14322.\r\n\r\nThe figure below is the feature matrix for patient 8 (also attached).\r\n![Feature matrix for patient 8][1]\r\n\r\nThe leftmost column is the ATC code column. # is a special symbol can be treated as Level 0 ATC code which contains every drug. Features are calculated for each ATC code from Level 0 to Level 4 that is seen in the test set. In total, there are 716 ATC codes (including #) seen in the test set.\r\n\r\nBeg/End is the first/last transaction date of drugs under each ATC code.\r\nBeg is in its original format, however, End is represented by the difference between it and 2016-01-01, e.g. -5 means 2015-12-27. The reason will be explained in the Data Augment section.\r\nDur is the number of days between Beg and End.\r\n\r\nGMed and GMax are the median and maximum gaps between dispense weeks of drugs under each ATC code. For example, a person has purchased C09 drugs for only 4 times, which were in week 2017-04-16, 2017-04-30, 2017-05-07, and 2017-05-28. Then the gaps are 14 days, 7 days, 21 days. The median is 14, the maximum is 21. Duplicates will be removed before calculating GMed and GMax. For example, 2017-04-16, 2017-04-30, 2017-05-07, 2017-05-07, 2017-05-07, and 2017-05-28 will give the same result.\r\n\r\ny-3, y-2, y-1, h1, q3, q4 are histogram features. All of them are counts of transactions. y-3, y-2, y-1 are year 2012, 2013, 2014. h1 is the first half of year 2015. q3 and q4 are the 3rd and 4th quarters of year 2015.\r\n\r\nm06, m12, m24, m36, m48 are cumulative sums of the above histogram features. m06 = q3 + q4. m12 = h1 + m06. m24 = y-1 + m12. m36 = y-2 + m24. m48 = y-3 + m36. They can be considered as numbers of transactions in the last XX months.\r\n\r\nBdif = abs(Beg - #_Beg) + 1, Edif = abs(End - #_End) + 1. For example, #_Beg and #_End for patient 8 are 2011-12-11, and -5 (or 2015-12-27). So A_Edif = abs(A_End - #_End) + 1 = abs(-173-(-5)) + 1 = 169, and A_Bdif = abs(A_Beg - #_Beg) + 1 = 1.\r\n\r\nIntuitively, both features consider the interaction between drugs. Edif is the number of days between the last transaction date of one ATC code drugs and the last transaction date of all drugs. So Edif can help us to identify whether a patient has stopped purchasing some ATC code drugs. Bdif is similar. “+1” is to avoid 0. For patient 8, C07_End == #_End, however, if a patient has never purchased C07 drugs, then his C07_Edif is also 0. “+1” can avoid this ambiguity. \r\n\r\nNPD1 = End + GMed, NPD2 = End + GMax. They are two very simple ways to estimate the next purchase date. When both features of an ATC code are very large negative/positive numbers, that patient is unlikely to purchase that ATC code drugs anymore. \r\n\r\nTo generate the above features, 3 columns, Patient_ID, Drug_ID, and Dispense_Week are first extracted from all the transaction files, then drugs that don’t have ATC codes in the Drug_LookUp.txt are removed from the transactions, and drugs whose ATC codes are not seen in the test set are also removed.\r\n\r\nData Augment\r\n------------\r\n\r\nThe goal of training a model is to learn transferable knowledge. For example, we train a model on one group of patients, then the model should work on another group of patients as well, otherwise it is not a good model.\r\n\r\nSimilarly, the knowledge in a good model should also be transferable across time. We can build a model on transactions between 11-15 to predict 2016. We can also build a model on transactions between 11-14 to predict 2015. And both models should learn similar knowledge. So, it’s reasonable to train one single model which works for both tasks.\r\n\r\nWe originally have\r\n\r\n - (1) train set (11-15) with ground truth (2016), and\r\n - (2) test set (11-15) without ground truth\r\n\r\nThen we can create two more training sets: \r\n\r\n - (3) train set (11-14) with ground truth (2015), and\r\n - (4) test set (11-14) with ground truth (2015) ----- this is a training set now.\r\n\r\n(1) + (3) + (4) have 279200 + 279200 + 279152 = 837552 patients in total.\r\n\r\nTo make this possible, we need to design the features so that they are transferable across time. That’s why Beg and End are both dates but have different representation. Beg can be transferable, but End can’t, unless we represent it as the difference between the true End date and a reference date (2015-01-01 or 2016-01-01). This is also true for the histogram features and their cumulative sums. In two cases, the bins used for the histogram are shifted by one year, and the meaning of cumulative sums is also changed accordingly.\r\n\r\nIn this way, we obtain a training set 3 times larger than the original one, which is helpful to prevent overfitting. Besides, when time is not enough for running cross validation, this provides us an alternative way to select model/parameters: use (1) + (4) as the training set and (3) as a development set.\r\n\r\n\r\n [1]: https://inclass.kaggle.com/blobs/download/forum-message-attachment-files/6607/patient8_feature_matrix.png", "votes": 3 }, { "id": 192975, "postDate": "2017-06-15T08:49:12.023Z", "content": "May I know what's your computer configuration? What kind of tools or packages did you use? and how long it took the computer to train the model? Do you mind to share some of your code ?
", "rawMarkdown": "May I know what's your computer configuration? What kind of tools or packages did you use? and how long it took the computer to train the model? Do you mind to share some of your code ?" }, { "id": 188141, "postDate": "2017-06-02T04:30:35.520Z", "content": "Good stuff :D
", "rawMarkdown": "Good stuff :D" }, { "id": 188140, "postDate": "2017-06-02T04:15:19.660Z", "content": "Thanks Jay Han. Yes, that gave rise to a large improvement. Without that, the best two submissions are 0.97032/0.97120 (public/private) and 0.97000/0.97132. With data augment and using the same features, the best two are 0.97116/0.97191 and 0.97101/0.97206.
", "rawMarkdown": "Thanks Jay Han. Yes, that gave rise to a large improvement. Without that, the best two submissions are 0.97032/0.97120 (public/private) and 0.97000/0.97132. With data augment and using the same features, the best two are 0.97116/0.97191 and 0.97101/0.97206." }, { "id": 188124, "postDate": "2017-06-02T02:50:43.820Z", "content": "This is really good. Thanks for sharing this.
\n\nI have a feeling that possibly the biggest impact is the clever use of test data (training set (4)). Do you know what performance did you get if you only train on (1) and test on (2)? I am curious how much performance gain you get by adding 2 times more data.
", "rawMarkdown": "This is really good. Thanks for sharing this.\n\nI have a feeling that possibly the biggest impact is the clever use of test data (training set (4)). Do you know what performance did you get if you only train on (1) and test on (2)? I am curious how much performance gain you get by adding 2 times more data." } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T05:01:02Z", "competition_id": 6467, "competition_slug": "dsm2017", "team_id": 609114, "solution_url": "/competitions/dsm2017/writeups/yuan-li-my-solution-to-datathon-2017" }