![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1605656%2F78817ef98c36f3bcfd21d89b7f70ac86%2FUntitled.jpg?generation=1579795238633405&alt=media\")
First of all, congrats to all winners! Was an interesting competition in the end, with not too much signal in the data and a very weirdly behaving metric. This is not a gold medal solution, maybe a few points are still interesting to some of you.
\n\nI joined the competition only a bit more than two weeks before end without any expectations. The main reason to jump in, was that I had a few ideas I wanted to check out on data like this using neural networks which is also what I started with.
\n\nNeural network
\n\nIn NFL competition I learned that you can use convolution neural networks really well on non-temporal data by just using kernels with size 1. So my idea was to use it as a form of feature engineering / understanding without me spending too much time on the engineering part myself.
\n\nI observed that a few things are important for predicting the success here. The two most important things are using information from previous assessments and previous other activities. So I generated two forms of sequences for each training data: sequence of all previous sessions, and sequence of all previous assessments. Each step in a sequence can then have multiple features, like one-hot-encoded or embedded title. So the assessment sequence could look like: assessment title 1, assessment title 2, etc.
\n\nAfter padding and reshaping, I tried to run LSTM and CNN on top, but quite quickly saw that there is little temporal information, so I just used CNN with kernel size 2 in the end for the session sequence, and kernel size 1 for the assessment sequence. I had as additional input the current assessment title.
\n\nAfter a bit of tuning, this NN scored around 0.520 on public LB without any threshold tuning. I always only did fast submission though and only used the training data externally. And then I probably made a mistake.
\n\nI decided to use my evaluation routine (more on that a bit later) on one of the public kernels to test it out. I also was fixed on the idea to use all the data available from test to train on, so I definitely wanted to train my model in kernel. The public kernel then quickly scored 0.557 with me adding my routine and the extra data. So I thought: wow I have a good setup let me try to improve that. So I decided to not port the NN to the kernel, because my code was very memory heavy and I would have needed to work a few days on adjusting it and also check how it works with training on more data. So from this point on, I do not use this NN anylonger, but I still believe it has potential if properly tuned and adjusted. What I also want to mention is that the public kernels had a lot of bugs, and it took me quite some time to find most of them. I think it is way better to start from scratch the next time. It is the first time I started with some public kernels.
\n\nCV and thresholds
\n\nI believe I came up with a quite nice and robust CV setup including a nice way to optimize thresholds. So what I did was to use stratified group kfold, and final CV is based on the median score of a few thousand truncated samples. I believe a few evaluated their models like that. What I did with thresholds though, was to optimize the thresholds in a way that they optimize the median score of these truncated samples. I then used these thresholds to predict the test set. To improve the threshold optimization I initialized the Nelder Mead algorithm with the histogram of the target.
\n\nFeatures and models
\n\nAs said earlier, I don't think there is a tremendous amount of signal in the data. So I did not spend too much time on FE, even though I believe that some carefully crafted features can help quite a bit. In the end I used event codes, event ids, a few assessment related features and a handful extracted from event data. I tried other things like tfidf on json data etc. without too much success. I train on all samples, meaning also those extracted from test data.
\n\nI focused on LGB and Catboost in the end. With catboost I explicitely utilize the has_time parameter which is perfect for this competition as it encoded categorical variables based on time information it has, so only using samples before that timepoint. I explicitely added also the assessment_id as categorical variable as local tests suggested that it will help me on private LB if I add the previous records to training.
Blending
\n\nI tried to be as robust as possible, so I decided on the following schema. 15 times 10 fold, for each of those 15 bags fit catboost and lgb, blend them with rank average using catboost 25% and lgb 75%. Optimize truncated thresholds as above. Predict test based also on rank avg mean of all 10 folds and then finally do majority voting on all 15 bags. I am quite happy with that because I managed to pick one of my best private LB scores in the end.
\n\nCrazy ideas that did not make it
\n\nI had two \"crazy\" ideas. I think the first one is quite simple and should have worked, but I had a bug on the kernel at the last day and didn't select it even though it scored highest on private LB even with this bug. The idea is quite simple, fit a MinMaxScaler or QuantileTransformer on the test predictions, transform oof and test predictions with it. Then do the threshold optimization on oof, and then apply it to test. This brings them on a similar range and scale and has some benefits over ranking the predictions. In nearly all my local experiments with simulating test data, this improved the QWK on the test data. I think this can bring a few points, but I have to test it again.
\n\nThe second idea involves again CNNs. The idea is to use CNNs to find the optimal thresholds for a sample. My idea was to do repeated subsampling of predictions on oof (can also be truncated) and calculate the optimal thresholds for these subsamples. The predictions of these subsamples are then the training data, and the thresholds the targets to predict. But again, you have no order on the training data but rather a set of predictions. So you can take a 1 kernel CNN and fit it on the set in order to predict the three thresholds optimizing something like MSE. Evaluation is then how close predicted thresholds are to the optimal thresholds. This actually also worked really well locally, but again I did not have time to tune it properly and port it to kernels. But might be worth a shot to test this further.
\n\nI tried really hard to find a good gamble the last few days using local test simulations, but I just could not find any. Ideas like sampling according to the test distribution etc.
\n\nThis competition again is a good example of why you should not chase the public LB. I was also tempted to do it after jumping to position 50 or so, but relying on a robust CV setup simulating how test data looks like is usually the better idea.
\n\nKernel: https://www.kaggle.com/philippsinger/lgb-catboost-17-10fold-15bag-3k-majority-rank-v4?
", "comments": [ { "id": 727321, "author_name": "Kurian Benoy", "author_url": "", "post_date": "2020-01-23T16:20:31.307000", "content": "\n\n\nWhat I also want to mention is that the public kernels had a lot of bugs, and it took me quite some time to find most of them. I think it is way better to start from scratch the next time. It is the first time I started with some public kernels.
\n
I will always keep this in mind whenever I am participating in any competitions next time
", "votes": 3, "replies": [ { "id": 727338, "author_name": "Anthony Chiu", "author_url": "", "post_date": "2020-01-23T16:35:42.110000", "content": "I actually spent 3 days to code the loop from the beginning, 2 weeks ago. The For-loop here is more like a coding test for me haha. The first successful submission (2 weeks ago) actually got the highest private score of 0.552, which means my only useful effort here is writing the feature generation For-loop.😓
", "votes": 1, "replies": [] }, { "id": 727396, "author_name": "Psi", "author_url": "", "post_date": "2020-01-23T17:27:43.597000", "content": "I think you can use the basic functionality framework like the groupby routine, but in best case you then remove every logical part and code it from scratch. That also enables you to think about features and setup better.
", "votes": 0, "replies": [] }, { "id": 727487, "author_name": "Ted K", "author_url": "", "post_date": "2020-01-23T18:42:01.613000", "content": "I think this is a good lesson for me as well. I ripped out code from the public kernel in the last week. Psi, I also want to compliment you on a very clear, well written write up. Thank you for sharing!
", "votes": 2, "replies": [] } ] }, { "id": 726927, "author_name": "CPMP", "author_url": "", "post_date": "2020-01-23T10:17:43.043000", "content": "Thanks for sharing. You definitely are great at dealing with train vs test distributions, here with your first crazy idea. I have to learn from you on that !
\n\nAnd congrats on the result obtained in a short time frame!
", "votes": 2, "replies": [ { "id": 726950, "author_name": "Psi", "author_url": "", "post_date": "2020-01-23T10:31:30.550000", "content": "Thanks! And again congrats on your in-the-money finish in the TF competition.
", "votes": 3, "replies": [] } ] }, { "id": 727295, "author_name": "Massoud Hosseinali", "author_url": "", "post_date": "2020-01-23T16:01:31.323000", "content": "Thanks for sharing and congratulations on your impressive rank in such a short time period. We used the same strategy for CV and optimizing thresholds. The red line in the following graph shows the threshold if train distribution is used for thresholds without optimization. The figure below on the bottom shows the CV we used for a while; we used the median of distribution as the local QWK but it was not correlated with the public LB and at some point, we decided not to continue with it which was a mistake.
\n\n
thanks for sharing, it is amazing you can build such a powerful and stable solution in a short time!
", "votes": 1, "replies": [ { "id": 727183, "author_name": "Psi", "author_url": "", "post_date": "2020-01-23T14:15:41.237000", "content": "Thank you and big congrats to you.
", "votes": 0, "replies": [] } ] }, { "id": 727135, "author_name": "Psi", "author_url": "", "post_date": "2020-01-23T13:39:26.697000", "content": "I opened the kernel of what I selected: https://www.kaggle.com/philippsinger/lgb-catboost-17-10fold-15bag-3k-majority-rank-v4?
", "votes": 1, "replies": [] }, { "id": 726904, "author_name": "davidlee0314", "author_url": "", "post_date": "2020-01-23T10:08:53.863000", "content": "Nice writeup and thanks for sharing! Although I am not familiar with NN, I believe it can have greater prediction power than other models. Probably start to polish my NN skills and back to review your approach haha.
\n\nSpeaking of LB chase, I was also doing the same thing. Worry about score and keep crafting features without validating it well. As you say, a good CV setup matters.
", "votes": 1, "replies": [] }, { "id": 726871, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-23T09:43:48.583000", "content": "Congrats and thanks for sharing indeed interesting ideas
", "votes": 1, "replies": [] }, { "id": 730877, "author_name": "keeeeei79", "author_url": "", "post_date": "2020-01-28T03:39:24.060000", "content": "Thank you for your sharing.\nHow did you experiment to see whether or not a new feature is useful?\nI encountered the problem of overfitting for KFold and LightGBM's seed.\n(One feature improves a loss in one seed. But using another seed, that feature worsens a loss.)\nHow did you deal with it?
", "votes": 0, "replies": [ { "id": 731609, "author_name": "Psi", "author_url": "", "post_date": "2020-01-28T20:10:44.357000", "content": "To be fair, I did not bother too much with feature selection here, rather set feature_fraction to a low number. But usually I try to run at least the whole CV and maybe bag it.
", "votes": 1, "replies": [] }, { "id": 732047, "author_name": "keeeeei79", "author_url": "", "post_date": "2020-01-29T12:20:39.850000", "content": "You mean you will mostly adopt new features even if new features worse loss a little?
", "votes": 0, "replies": [] }, { "id": 732078, "author_name": "Psi", "author_url": "", "post_date": "2020-01-29T13:23:10.717000", "content": "No, I tried to limit features, but I am not doing any sophisticated method for it, just quickly test if it works on full CV or not.
", "votes": 1, "replies": [] }, { "id": 732149, "author_name": "keeeeei79", "author_url": "", "post_date": "2020-01-29T14:34:28.447000", "content": "I got it. Thank you very much !!
", "votes": 0, "replies": [] } ] }, { "id": 728310, "author_name": "Trigram", "author_url": "", "post_date": "2020-01-24T15:38:33.850000", "content": "Congrats on becoming GM @philippsinger !
", "votes": 0, "replies": [] }, { "id": 726983, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-01-23T10:50:36.123000", "content": "Could you explain what is non - temporal data
", "votes": 0, "replies": [ { "id": 727042, "author_name": "Psi", "author_url": "", "post_date": "2020-01-23T12:02:50.340000", "content": "When you don't have any implied order in the data. So basically non-sequential data.
", "votes": 0, "replies": [] } ] }, { "id": 726890, "author_name": "Hanjoon Choe", "author_url": "", "post_date": "2020-01-23T09:53:38.933000", "content": "I was one of those who was curious about your approach. Thank you for sharing ;-) Congratulations!
", "votes": 0, "replies": [] }, { "id": 726849, "author_name": "1ira", "author_url": "", "post_date": "2020-01-23T09:24:17.717000", "content": "Thank you this is great, i find the first chapter on NN fascinating. Could you perhaps suggest any good sources, such as kernels or some out of kaggle matterials to learn more about the approach?
", "votes": 0, "replies": [] }, { "id": 727125, "author_name": "", "author_url": "", "post_date": "2020-01-23T13:27:32.860000", "content": "", "votes": 0, "replies": [ { "id": 727126, "author_name": "Psi", "author_url": "", "post_date": "2020-01-23T13:28:58.497000", "content": "out of fold predictions
", "votes": 0, "replies": [] }, { "id": 727129, "author_name": "", "author_url": "", "post_date": "2020-01-23T13:30:10.473000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 726865, "author_name": "", "author_url": "", "post_date": "2020-01-23T09:36:59.633000", "content": "", "votes": 0, "replies": [] }, { "id": 732570, "author_name": "corochann", "author_url": "", "post_date": "2020-01-30T00:27:28.303000", "content": "Thanks for sharing.
", "votes": 0, "replies": [] } ], "comment_count": 27, "messages": [ { "id": 726813, "postDate": "2020-01-23T08:50:12.123Z", "content": "First of all, congrats to all winners! Was an interesting competition in the end, with not too much signal in the data and a very weirdly behaving metric. This is not a gold medal solution, maybe a few points are still interesting to some of you.
\n\nI joined the competition only a bit more than two weeks before end without any expectations. The main reason to jump in, was that I had a few ideas I wanted to check out on data like this using neural networks which is also what I started with.
\n\nNeural network
\n\nIn NFL competition I learned that you can use convolution neural networks really well on non-temporal data by just using kernels with size 1. So my idea was to use it as a form of feature engineering / understanding without me spending too much time on the engineering part myself.
\n\nI observed that a few things are important for predicting the success here. The two most important things are using information from previous assessments and previous other activities. So I generated two forms of sequences for each training data: sequence of all previous sessions, and sequence of all previous assessments. Each step in a sequence can then have multiple features, like one-hot-encoded or embedded title. So the assessment sequence could look like: assessment title 1, assessment title 2, etc.
\n\nAfter padding and reshaping, I tried to run LSTM and CNN on top, but quite quickly saw that there is little temporal information, so I just used CNN with kernel size 2 in the end for the session sequence, and kernel size 1 for the assessment sequence. I had as additional input the current assessment title.
\n\nAfter a bit of tuning, this NN scored around 0.520 on public LB without any threshold tuning. I always only did fast submission though and only used the training data externally. And then I probably made a mistake.
\n\nI decided to use my evaluation routine (more on that a bit later) on one of the public kernels to test it out. I also was fixed on the idea to use all the data available from test to train on, so I definitely wanted to train my model in kernel. The public kernel then quickly scored 0.557 with me adding my routine and the extra data. So I thought: wow I have a good setup let me try to improve that. So I decided to not port the NN to the kernel, because my code was very memory heavy and I would have needed to work a few days on adjusting it and also check how it works with training on more data. So from this point on, I do not use this NN anylonger, but I still believe it has potential if properly tuned and adjusted. What I also want to mention is that the public kernels had a lot of bugs, and it took me quite some time to find most of them. I think it is way better to start from scratch the next time. It is the first time I started with some public kernels.
\n\nCV and thresholds
\n\nI believe I came up with a quite nice and robust CV setup including a nice way to optimize thresholds. So what I did was to use stratified group kfold, and final CV is based on the median score of a few thousand truncated samples. I believe a few evaluated their models like that. What I did with thresholds though, was to optimize the thresholds in a way that they optimize the median score of these truncated samples. I then used these thresholds to predict the test set. To improve the threshold optimization I initialized the Nelder Mead algorithm with the histogram of the target.
\n\nFeatures and models
\n\nAs said earlier, I don't think there is a tremendous amount of signal in the data. So I did not spend too much time on FE, even though I believe that some carefully crafted features can help quite a bit. In the end I used event codes, event ids, a few assessment related features and a handful extracted from event data. I tried other things like tfidf on json data etc. without too much success. I train on all samples, meaning also those extracted from test data.
\n\nI focused on LGB and Catboost in the end. With catboost I explicitely utilize the has_time parameter which is perfect for this competition as it encoded categorical variables based on time information it has, so only using samples before that timepoint. I explicitely added also the assessment_id as categorical variable as local tests suggested that it will help me on private LB if I add the previous records to training.
Blending
\n\nI tried to be as robust as possible, so I decided on the following schema. 15 times 10 fold, for each of those 15 bags fit catboost and lgb, blend them with rank average using catboost 25% and lgb 75%. Optimize truncated thresholds as above. Predict test based also on rank avg mean of all 10 folds and then finally do majority voting on all 15 bags. I am quite happy with that because I managed to pick one of my best private LB scores in the end.
\n\nCrazy ideas that did not make it
\n\nI had two \"crazy\" ideas. I think the first one is quite simple and should have worked, but I had a bug on the kernel at the last day and didn't select it even though it scored highest on private LB even with this bug. The idea is quite simple, fit a MinMaxScaler or QuantileTransformer on the test predictions, transform oof and test predictions with it. Then do the threshold optimization on oof, and then apply it to test. This brings them on a similar range and scale and has some benefits over ranking the predictions. In nearly all my local experiments with simulating test data, this improved the QWK on the test data. I think this can bring a few points, but I have to test it again.
\n\nThe second idea involves again CNNs. The idea is to use CNNs to find the optimal thresholds for a sample. My idea was to do repeated subsampling of predictions on oof (can also be truncated) and calculate the optimal thresholds for these subsamples. The predictions of these subsamples are then the training data, and the thresholds the targets to predict. But again, you have no order on the training data but rather a set of predictions. So you can take a 1 kernel CNN and fit it on the set in order to predict the three thresholds optimizing something like MSE. Evaluation is then how close predicted thresholds are to the optimal thresholds. This actually also worked really well locally, but again I did not have time to tune it properly and port it to kernels. But might be worth a shot to test this further.
\n\nI tried really hard to find a good gamble the last few days using local test simulations, but I just could not find any. Ideas like sampling according to the test distribution etc.
\n\nThis competition again is a good example of why you should not chase the public LB. I was also tempted to do it after jumping to position 50 or so, but relying on a robust CV setup simulating how test data looks like is usually the better idea.
\n\nKernel: https://www.kaggle.com/philippsinger/lgb-catboost-17-10fold-15bag-3k-majority-rank-v4?
", "rawMarkdown": "First of all, congrats to all winners! Was an interesting competition in the end, with not too much signal in the data and a very weirdly behaving metric. This is not a gold medal solution, maybe a few points are still interesting to some of you.\n\nI joined the competition only a bit more than two weeks before end without any expectations. The main reason to jump in, was that I had a few ideas I wanted to check out on data like this using neural networks which is also what I started with.\n\n**Neural network**\n\nIn NFL competition I learned that you can use convolution neural networks really well on non-temporal data by just using kernels with size 1. So my idea was to use it as a form of feature engineering / understanding without me spending too much time on the engineering part myself. \n\nI observed that a few things are important for predicting the success here. The two most important things are using information from previous assessments and previous other activities. So I generated two forms of sequences for each training data: sequence of all previous sessions, and sequence of all previous assessments. Each step in a sequence can then have multiple features, like one-hot-encoded or embedded title. So the assessment sequence could look like: assessment title 1, assessment title 2, etc.\n\nAfter padding and reshaping, I tried to run LSTM and CNN on top, but quite quickly saw that there is little temporal information, so I just used CNN with kernel size 2 in the end for the session sequence, and kernel size 1 for the assessment sequence. I had as additional input the current assessment title.\n\nAfter a bit of tuning, this NN scored around 0.520 on public LB without any threshold tuning. I always only did fast submission though and only used the training data externally. And then I probably made a mistake. \n\nI decided to use my evaluation routine (more on that a bit later) on one of the public kernels to test it out. I also was fixed on the idea to use all the data available from test to train on, so I definitely wanted to train my model in kernel. The public kernel then quickly scored 0.557 with me adding my routine and the extra data. So I thought: wow I have a good setup let me try to improve that. So I decided to not port the NN to the kernel, because my code was very memory heavy and I would have needed to work a few days on adjusting it and also check how it works with training on more data. So from this point on, I do not use this NN anylonger, but I still believe it has potential if properly tuned and adjusted. What I also want to mention is that the public kernels had a lot of bugs, and it took me quite some time to find most of them. I think it is way better to start from scratch the next time. It is the first time I started with some public kernels.\n\n**CV and thresholds**\n\nI believe I came up with a quite nice and robust CV setup including a nice way to optimize thresholds. So what I did was to use stratified group kfold, and final CV is based on the median score of a few thousand truncated samples. I believe a few evaluated their models like that. What I did with thresholds though, was to **optimize the thresholds in a way that they optimize the median score of these truncated samples**. I then used these thresholds to predict the test set. To improve the threshold optimization I initialized the Nelder Mead algorithm with the histogram of the target.\n\n**Features and models**\n\nAs said earlier, I don't think there is a tremendous amount of signal in the data. So I did not spend too much time on FE, even though I believe that some carefully crafted features can help quite a bit. In the end I used event codes, event ids, a few assessment related features and a handful extracted from event data. I tried other things like tfidf on json data etc. without too much success. I train on **all samples**, meaning also those extracted from test data.\n\nI focused on LGB and Catboost in the end. With catboost I explicitely utilize the `has_time` parameter which is perfect for this competition as it encoded categorical variables based on time information it has, so only using samples before that timepoint. I explicitely added also the `assessment_id` as categorical variable as local tests suggested that it will help me on private LB if I add the previous records to training.\n\n**Blending**\n\nI tried to be as robust as possible, so I decided on the following schema. 15 times 10 fold, for each of those 15 bags fit catboost and lgb, blend them with rank average using catboost 25% and lgb 75%. Optimize truncated thresholds as above. Predict test based also on rank avg mean of all 10 folds and then finally do majority voting on all 15 bags. I am quite happy with that because I managed to pick one of my best private LB scores in the end.\n\n**Crazy ideas that did not make it**\n\nI had two \"crazy\" ideas. I think the first one is quite simple and should have worked, but I had a bug on the kernel at the last day and didn't select it even though it scored highest on private LB even with this bug. The idea is quite simple, fit a MinMaxScaler or QuantileTransformer on the test predictions, transform oof and test predictions with it. Then do the threshold optimization on oof, and then apply it to test. This brings them on a similar range and scale and has some benefits over ranking the predictions. In nearly all my local experiments with simulating test data, this improved the QWK on the test data. I think this can bring a few points, but I have to test it again.\n\nThe second idea involves again CNNs. The idea is to use CNNs to find the optimal thresholds for a sample. My idea was to do repeated subsampling of predictions on oof (can also be truncated) and calculate the optimal thresholds for these subsamples. The predictions of these subsamples are then the training data, and the thresholds the targets to predict. But again, you have no order on the training data but rather a set of predictions. So you can take a 1 kernel CNN and fit it on the set in order to predict the three thresholds optimizing something like MSE. Evaluation is then how close predicted thresholds are to the optimal thresholds. This actually also worked really well locally, but again I did not have time to tune it properly and port it to kernels. But might be worth a shot to test this further.\n\nI tried really hard to find a good gamble the last few days using local test simulations, but I just could not find any. Ideas like sampling according to the test distribution etc. \n\nThis competition again is a good example of why you should not chase the public LB. I was also tempted to do it after jumping to position 50 or so, but relying on a robust CV setup simulating how test data looks like is usually the better idea.\n\nKernel: https://www.kaggle.com/philippsinger/lgb-catboost-17-10fold-15bag-3k-majority-rank-v4?\n", "votes": 53 }, { "id": 727321, "postDate": "2020-01-23T16:20:31.307Z", "content": "\n\n\nWhat I also want to mention is that the public kernels had a lot of bugs, and it took me quite some time to find most of them. I think it is way better to start from scratch the next time. It is the first time I started with some public kernels.
\n
I will always keep this in mind whenever I am participating in any competitions next time
", "rawMarkdown": "> What I also want to mention is that the public kernels had a lot of bugs, and it took me quite some time to find most of them. I think it is way better to start from scratch the next time. It is the first time I started with some public kernels.\n\nI will always keep this in mind whenever I am participating in any competitions next time", "votes": 3, "replies": [ { "id": 727338, "postDate": "2020-01-23T16:35:42.110Z", "content": "I actually spent 3 days to code the loop from the beginning, 2 weeks ago. The For-loop here is more like a coding test for me haha. The first successful submission (2 weeks ago) actually got the highest private score of 0.552, which means my only useful effort here is writing the feature generation For-loop.😓
", "rawMarkdown": "I actually spent 3 days to code the loop from the beginning, 2 weeks ago. The For-loop here is more like a coding test for me haha. The first successful submission (2 weeks ago) actually got the highest private score of 0.552, which means my only useful effort here is writing the feature generation For-loop.😓 ", "votes": 1 }, { "id": 727396, "postDate": "2020-01-23T17:27:43.597Z", "content": "I think you can use the basic functionality framework like the groupby routine, but in best case you then remove every logical part and code it from scratch. That also enables you to think about features and setup better.
", "rawMarkdown": "I think you can use the basic functionality framework like the groupby routine, but in best case you then remove every logical part and code it from scratch. That also enables you to think about features and setup better." }, { "id": 727487, "postDate": "2020-01-23T18:42:01.613Z", "content": "I think this is a good lesson for me as well. I ripped out code from the public kernel in the last week. Psi, I also want to compliment you on a very clear, well written write up. Thank you for sharing!
", "rawMarkdown": "I think this is a good lesson for me as well. I ripped out code from the public kernel in the last week. Psi, I also want to compliment you on a very clear, well written write up. Thank you for sharing!", "votes": 2 } ] }, { "id": 726927, "postDate": "2020-01-23T10:17:43.043Z", "content": "Thanks for sharing. You definitely are great at dealing with train vs test distributions, here with your first crazy idea. I have to learn from you on that !
\n\nAnd congrats on the result obtained in a short time frame!
", "rawMarkdown": "Thanks for sharing. You definitely are great at dealing with train vs test distributions, here with your first crazy idea. I have to learn from you on that !\n\nAnd congrats on the result obtained in a short time frame!", "votes": 2, "replies": [ { "id": 726950, "postDate": "2020-01-23T10:31:30.550Z", "content": "Thanks! And again congrats on your in-the-money finish in the TF competition.
", "rawMarkdown": "Thanks! And again congrats on your in-the-money finish in the TF competition.", "votes": 3 } ] }, { "id": 727295, "postDate": "2020-01-23T16:01:31.323Z", "content": "Thanks for sharing and congratulations on your impressive rank in such a short time period. We used the same strategy for CV and optimizing thresholds. The red line in the following graph shows the threshold if train distribution is used for thresholds without optimization. The figure below on the bottom shows the CV we used for a while; we used the median of distribution as the local QWK but it was not correlated with the public LB and at some point, we decided not to continue with it which was a mistake.
\n\n
thanks for sharing, it is amazing you can build such a powerful and stable solution in a short time!
", "rawMarkdown": "thanks for sharing, it is amazing you can build such a powerful and stable solution in a short time!", "votes": 1, "replies": [ { "id": 727183, "postDate": "2020-01-23T14:15:41.237Z", "content": "Thank you and big congrats to you.
", "rawMarkdown": "Thank you and big congrats to you." } ] }, { "id": 727135, "postDate": "2020-01-23T13:39:26.697Z", "content": "I opened the kernel of what I selected: https://www.kaggle.com/philippsinger/lgb-catboost-17-10fold-15bag-3k-majority-rank-v4?
", "rawMarkdown": "I opened the kernel of what I selected: https://www.kaggle.com/philippsinger/lgb-catboost-17-10fold-15bag-3k-majority-rank-v4?", "votes": 1 }, { "id": 726904, "postDate": "2020-01-23T10:08:53.863Z", "content": "Nice writeup and thanks for sharing! Although I am not familiar with NN, I believe it can have greater prediction power than other models. Probably start to polish my NN skills and back to review your approach haha.
\n\nSpeaking of LB chase, I was also doing the same thing. Worry about score and keep crafting features without validating it well. As you say, a good CV setup matters.
", "rawMarkdown": "Nice writeup and thanks for sharing! Although I am not familiar with NN, I believe it can have greater prediction power than other models. Probably start to polish my NN skills and back to review your approach haha.\n\nSpeaking of LB chase, I was also doing the same thing. Worry about score and keep crafting features without validating it well. As you say, a good CV setup matters.", "votes": 1 }, { "id": 726871, "postDate": "2020-01-23T09:43:48.583Z", "content": "Congrats and thanks for sharing indeed interesting ideas
", "rawMarkdown": "Congrats and thanks for sharing indeed interesting ideas", "votes": 1 }, { "id": 730877, "postDate": "2020-01-28T03:39:24.060Z", "content": "Thank you for your sharing.\nHow did you experiment to see whether or not a new feature is useful?\nI encountered the problem of overfitting for KFold and LightGBM's seed.\n(One feature improves a loss in one seed. But using another seed, that feature worsens a loss.)\nHow did you deal with it?
", "rawMarkdown": "Thank you for your sharing.\nHow did you experiment to see whether or not a new feature is useful?\nI encountered the problem of overfitting for KFold and LightGBM's seed.\n(One feature improves a loss in one seed. But using another seed, that feature worsens a loss.)\nHow did you deal with it?", "replies": [ { "id": 731609, "postDate": "2020-01-28T20:10:44.357Z", "content": "To be fair, I did not bother too much with feature selection here, rather set feature_fraction to a low number. But usually I try to run at least the whole CV and maybe bag it.
", "rawMarkdown": "To be fair, I did not bother too much with feature selection here, rather set feature_fraction to a low number. But usually I try to run at least the whole CV and maybe bag it.", "votes": 1 }, { "id": 732047, "postDate": "2020-01-29T12:20:39.850Z", "content": "You mean you will mostly adopt new features even if new features worse loss a little?
", "rawMarkdown": "You mean you will mostly adopt new features even if new features worse loss a little?" }, { "id": 732078, "postDate": "2020-01-29T13:23:10.717Z", "content": "No, I tried to limit features, but I am not doing any sophisticated method for it, just quickly test if it works on full CV or not.
", "rawMarkdown": "No, I tried to limit features, but I am not doing any sophisticated method for it, just quickly test if it works on full CV or not.", "votes": 1 }, { "id": 732149, "postDate": "2020-01-29T14:34:28.447Z", "content": "I got it. Thank you very much !!
", "rawMarkdown": "I got it. Thank you very much !!" } ] }, { "id": 728310, "postDate": "2020-01-24T15:38:33.850Z", "content": "Congrats on becoming GM @philippsinger !
", "rawMarkdown": "Congrats on becoming GM @philippsinger !" }, { "id": 726983, "postDate": "2020-01-23T10:50:36.123Z", "content": "Could you explain what is non - temporal data
", "rawMarkdown": "Could you explain what is non - temporal data", "replies": [ { "id": 727042, "postDate": "2020-01-23T12:02:50.340Z", "content": "When you don't have any implied order in the data. So basically non-sequential data.
", "rawMarkdown": "When you don't have any implied order in the data. So basically non-sequential data." } ] }, { "id": 726890, "postDate": "2020-01-23T09:53:38.933Z", "content": "I was one of those who was curious about your approach. Thank you for sharing ;-) Congratulations!
", "rawMarkdown": "I was one of those who was curious about your approach. Thank you for sharing ;-) Congratulations! " }, { "id": 726849, "postDate": "2020-01-23T09:24:17.717Z", "content": "Thank you this is great, i find the first chapter on NN fascinating. Could you perhaps suggest any good sources, such as kernels or some out of kaggle matterials to learn more about the approach?
", "rawMarkdown": "Thank you this is great, i find the first chapter on NN fascinating. Could you perhaps suggest any good sources, such as kernels or some out of kaggle matterials to learn more about the approach?" }, { "id": 727125, "postDate": "2020-01-23T13:27:32.860Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 727126, "postDate": "2020-01-23T13:28:58.497Z", "content": "out of fold predictions
", "rawMarkdown": "out of fold predictions" }, { "id": 727129, "postDate": "2020-01-23T13:30:10.473Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 726865, "postDate": "2020-01-23T09:36:59.633Z", "rawMarkdown": "", "isDeleted": true }, { "id": 732570, "postDate": "2020-01-30T00:27:28.303Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing." } ], "complete": true, "source": "live", "fetched_at": "2026-10-22T10:52:33Z", "competition_id": 16531, "competition_slug": "data-science-bowl-2019", "team_id": 3855192, "solution_url": "/competitions/data-science-bowl-2019/writeups/psi-solution-and-some-ideas" }