{ "id": 127213, "title": "7th Place Solution", "content": "

Approach

\n\n\n\n

Results

\n\n\n\n

Final Remarks

\n\n\n\n

Validation setup

\n\n\n\n

Update 1: Features

\n\n

Motivation:\nI was impacted by the ideas presented in this paper by Francois Chollet: On the measure of intelligence. There are tons of interesting and powerful thoughts there. I was mostly stimulated by a discussion on how to measure intelligence: \n- A/ by overall-skill-level \n- B/ by skill-acquisition-tempo

\n\n

In our case, we are measured by A/, which can be broken into two drivers:\n- experience, i.e. how much time/effort the child has spent on various actitivities in the game. This formed my first group of features\n- accuracy - how accurate was this child in her journey. This formed my second group of features.

\n\n

However, skill-acquisition-tempo is a very interesting way to capture how quickly children are learning (features like minutes per level, events per level, etc.). This formed by 3rd group of features

\n\n

I love competitions with manual feature engineering. Combination human+machine wins, which represents my view on how AI will impact the world.

\n\n

Update 2: Feature selection

\n\n", "comments": [ { "id": 726486, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2020-01-23T03:08:44.887000", "content": "

Congrats! Could you shed some light on your feature selection approach? How did you choose those 51?

", "votes": 4, "replies": [ { "id": 726515, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-23T03:29:13.950000", "content": "

Sure - please see updated post above

", "votes": 2, "replies": [] }, { "id": 726517, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2020-01-23T03:31:14.723000", "content": "

Thanks!

", "votes": 1, "replies": [] } ] }, { "id": 726269, "author_name": "Nanashi", "author_url": "", "post_date": "2020-01-23T00:35:03.750000", "content": "

thank you so much @narsil \ncould you share those 51 features or how to produce them?\nhow did you add test assesments as part of the train? (just trying to learn)

", "votes": 4, "replies": [ { "id": 726371, "author_name": "davidlee0314", "author_url": "", "post_date": "2020-01-23T01:31:32.383000", "content": "

I am also curious about the features, since I didn't do good feature selection.

", "votes": 1, "replies": [] }, { "id": 726410, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-23T02:05:27.620000", "content": "

Sure, please see the updated post above. Feature engineering was the most fun part of this competition.

", "votes": 0, "replies": [] }, { "id": 726521, "author_name": "Khurram Siddiqui", "author_url": "", "post_date": "2020-01-23T03:48:11.007000", "content": "

@narsil , how new feature engineering worked with you , after my initial hypothesis the minute i add up new feature e.g. number of distraction before an assessment from event data or number of incorrect assessment for the same assessment world , my public LB got made and decreased from 0.40 to less than 0.30

\n\n

Top 103rd Solution

", "votes": 0, "replies": [] }, { "id": 727647, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-23T23:58:42.457000", "content": "

I always try to find some common sense logic and build a framework to guide me in the feature engineering process. This time I did some brainstormin and used the paper from Francois Chollet. This initial 'framing' stage helps me make sure I don't miss an important feature group.

", "votes": 1, "replies": [] }, { "id": 727899, "author_name": "Khurram Siddiqui", "author_url": "", "post_date": "2020-01-24T07:03:52.720000", "content": "

@narsil can you refer this paper please?

", "votes": 1, "replies": [] }, { "id": 727901, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-24T07:07:25.797000", "content": "

Sure: https://arxiv.org/abs/1911.01547\nIt is also in the write-up, section Update 1: Features

", "votes": 1, "replies": [] }, { "id": 727919, "author_name": "Khurram Siddiqui", "author_url": "", "post_date": "2020-01-24T07:43:53.463000", "content": "

@narsil thanks mate

", "votes": 0, "replies": [] } ] }, { "id": 726935, "author_name": "Paulo Alberto Buoro", "author_url": "", "post_date": "2020-01-23T10:19:28.653000", "content": "

great, congrats!\nWhat exactly 20-fold bagging means?\nIs that you used bootstrap to resample and create 20 bags? Did you use replacement? What was the size of each bag?\nThanks

", "votes": 2, "replies": [ { "id": 727651, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-24T00:01:25.687000", "content": "

I split data into 20 folds. I build model separately for each fold and\n- for cv I take oof predictions for every fold\n- for submission I take the average of predictions across all folds

", "votes": 1, "replies": [] } ] }, { "id": 727956, "author_name": "ShawnSalm", "author_url": "", "post_date": "2020-01-24T08:52:25.343000", "content": "

Congratulations ,and thank you for sharing insight into your process!!! I'm new to ML and this was my first project and Kaggle competition. Still I was a bit surprised and disappointed on how badly I got my butt kicked. I also saw that the best correlation I could find was the assessment that the kids were taking (I guess some assessments are easier than others.) But I was hoping that I could find or combine other features that would also be good (I found a couple that improved a little with by logging then, but that's it). I didn't end up using a NN because I couldn't seem to get it to reach a global minimum. Anyway, any advice, other than what you have already given (which is great) that could help me in the future with Kaggle competitions as well as on ML projects in general? I guess if we had cleaner training data (devices not shared by other people) we might have seen more useful patterns in the kids behavior. I did try to filter out some records that I thought might be the case such as when the first action on the device was an assignment and the person got it correct the first time. I assumed that was a parent or teacher trying it out, but removing them only made my public score worse. As well, my cv score was .58 but my private score ended up being around 4.3. So big difference there. I ended up only using one lgbmclassifier. So I guess one take away is to use bagging with multiple classifiers. Anyway, thanks again. Any more insight you could provide when you have time would be great.

", "votes": 1, "replies": [ { "id": 728638, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-25T02:02:34.300000", "content": "

Hey.

\n\n

Great to be after your 1st competition - you know more how Kaggle works now.\nFor me - it took 2-3 competitions before I realized what is going on. Since 4th or 5th competition I could compete for real.\nIf your score dropped from 0.58 to 4.3, it means there must be a bug in the code. Just by saying 2.0 for every assessment, your score could be around 1.0.

\n\n

My advice: keep going and have fun. Zero expectations on score, but maximize learning.

", "votes": 0, "replies": [] }, { "id": 728783, "author_name": "ShawnSalm", "author_url": "", "post_date": "2020-01-25T08:54:44.687000", "content": "

Good advice! Maximizing learning seems like the best goal (although $100,000 wouldn't have been a bad thing as well :-) I wonder if it was a mistake for me to use the class weights while training? If I didn't use them though, my model would end up predicting only two or three category 2s in the test set. I have much to learn! Thanks again!

", "votes": 1, "replies": [] } ] }, { "id": 727227, "author_name": "samshipengs", "author_url": "", "post_date": "2020-01-23T14:43:34.757000", "content": "

Thank you for the write-up, i have a few basic questions:\n1)

\n\n
\n

20 fold-bagging for all models

\n
\n\n

from what i understand bagging is the sampling with replacement to form a new data set, is the new data set you sampled the same size as the training data (or smaller/bigger)? then what do you use as your validation set? (the ones did not get sampled?) And did you do anything special with sampling e.g. sample same number of data points from each installation_id instead of sampling uniformly from all available training data points (ignoring groups)?

\n\n

2)

\n\n
\n

Truncate - select one assessment randomly for every child to reflect test set structure

\n
\n\n

Is the select one only happens once per installation_id for one validation set, or you repeatedly sample (with replacement) one from each installation_id? otherwise i imagine the validation set would be small and unstable

\n\n

3)

\n\n
\n

Calculated cv score after dropping a feature - did this individually for all ~150 features

\n
\n\n

its nice that it worked out for you, although did you look into feature selections with Shapley value (shap library) i.e. drop features with small or zero shap importance on validation set? One potential draw-back for evaluating features one by one, what if there are feature interactions among them that each features may not matter much but a certain combination would?

\n\n

4) Did you look into adversarial validation? if not, is it because the public test set is small? and what are your thoughts on adjusting the feature values by comparing to test features statistic (as some public kernel did)?

\n\n

5) At last, a very general question, normally how do you validate the features your created? such as making sure there are no obvious bugs or silent bugs? (sometimes I discover by chance that there's silent bug in my code, i didnt notice as they did not throw errors and the value seems like to be in correct range, do you do say unitest on your feature generation code? thats sounds a bit crazy)

\n\n

thanks for reading, and congrats on the winning!

", "votes": 1, "replies": [ { "id": 727657, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-24T00:12:53.663000", "content": "

Hello.

\n\n
    \n
  1. I reply below in the post from @pabuoro
    2. \n
  2. I do it only once and get an indeed small sample, but then I repeat this small sample 100 times to get a stable result
    3. \n
  3. I did a final check on CV score after dropping the features - to see exactly what you are pointing out.
    4. \n
  4. No. I should have, but didn't have time, and it wasn't high on my priorities. Adjusting the feature values - did not put too much thought on this, but at first glance seems very dangerous to me,
    5. \n
  5. I do indeed some tests on my features to prevent bugs - normally I look at their distributions and CV scores after inclusion in the model
    6. \n
", "votes": 1, "replies": [] } ] }, { "id": 726951, "author_name": "Hi Im Ryo", "author_url": "", "post_date": "2020-01-23T10:34:56.730000", "content": "

Congratulations ,and thank you for sharing your solution!\nI have a question about value of feature selection.\nDo you also need feature selection when you don't use NN?\nI've learned you don't have to worry too much about noise when you use only GBDTs.

", "votes": 1, "replies": [ { "id": 727652, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-24T00:02:47.963000", "content": "

My general experience is that the simpler the model, the more generalization power it has

\n\n

Complex models can fit the past data very well but are not so good at predicting the future. Simple models don't fit the past very well, but whatever they fit has very similar predictive power for the future

", "votes": 2, "replies": [] }, { "id": 727833, "author_name": "Hi Im Ryo", "author_url": "", "post_date": "2020-01-24T05:13:14.010000", "content": "

Thanks!

", "votes": 1, "replies": [] } ] }, { "id": 726862, "author_name": "Alvor", "author_url": "", "post_date": "2020-01-23T09:33:51.210000", "content": "

Congrats on your first Solo Gold!

", "votes": 1, "replies": [ { "id": 727650, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-23T23:59:49.067000", "content": "

Thanks Sasza :) Remember out deal? We are going to pimiento when we visit Krakow :)

", "votes": 1, "replies": [] } ] }, { "id": 726665, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-01-23T06:39:36.507000", "content": "

Your solution is concise and elegant. Congrats!

", "votes": 1, "replies": [ { "id": 727645, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-23T23:56:26.600000", "content": "

Thanks! Congrats to your team as well - you managed to stay in the gold zone and survive the shakeup, which is impressive

", "votes": 1, "replies": [] } ] }, { "id": 726616, "author_name": "Kurian Benoy", "author_url": "", "post_date": "2020-01-23T06:03:33.770000", "content": "

Can you share the code you used for the competition?

", "votes": 1, "replies": [ { "id": 727648, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-23T23:59:09.853000", "content": "

I do not plan to release my code this time

", "votes": 1, "replies": [] } ] }, { "id": 726508, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-01-23T03:23:10.210000", "content": "

Congratulations\nThanks for sharing your Approach & Insights!! @narsil

", "votes": 1, "replies": [] }, { "id": 726451, "author_name": "", "author_url": "", "post_date": "2020-01-23T02:34:42.323000", "content": "

Thanks for sharing and Congrats on your gold medal!

\n\n

\nOne “trick” – use assessments from test set “blindly” (as we did not see this data) as samples in training. Data was scarce – so I was looking for every way to increase the number of training samples\n\nBTW, could you say a bit more about using assessment from test set? I tried to mix up train and test set. Public LB decrease by about 0.003.(Not sure about private LB).

", "votes": 1, "replies": [ { "id": 726482, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-23T03:06:17.227000", "content": "

This will require some work from me, but I will try to do this experiment and see results. \nI assumed early in the competition that additional data is good.

\n\n

I know, I know: \"assumed\" - the most dangerous word of all. But we need to have some cornerstones of our strategy, and having more data is a good one almost always. Especially when you realize that you add data exactly for the children that are in the private LB.

", "votes": 2, "replies": [] }, { "id": 726494, "author_name": "", "author_url": "", "post_date": "2020-01-23T03:13:51.097000", "content": "

Thanks. Look forward to your experiment.

", "votes": 1, "replies": [] }, { "id": 726501, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2020-01-23T03:17:44.020000", "content": "

Actually you can probing changing of CV score instead of assuming ;)

", "votes": 1, "replies": [] }, { "id": 726506, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-23T03:21:27.897000", "content": "

Fair comment. \"Assumed\" was a shortcut.

\n\n

In my case, after adding test data:\n- CV increased a little. I expected more, but I was relieved to see it does not destroy the score.\n- Public LB dropped a little. So I \"assumed\" that it would still be good for the private LB.

", "votes": 1, "replies": [] } ] }, { "id": 726424, "author_name": "KhanhVD", "author_url": "", "post_date": "2020-01-23T02:18:57.160000", "content": "

Congrats on your gold medal @narsil !

", "votes": 1, "replies": [ { "id": 726581, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-23T05:20:40.933000", "content": "

Thanks so much!

", "votes": 0, "replies": [] } ] }, { "id": 726415, "author_name": "Stephen Keller", "author_url": "", "post_date": "2020-01-23T02:10:32.047000", "content": "

I want to save this comment. Beautifully written and quickly explains the system you built. A great example for students. Well done and UP voted! Congrats on your gold medal !

", "votes": 1, "replies": [ { "id": 726582, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-23T05:20:45.420000", "content": "

Thanks so much!

", "votes": 0, "replies": [] } ] }, { "id": 733269, "author_name": "corochann", "author_url": "", "post_date": "2020-01-30T22:58:50.533000", "content": "

Congrats and thanks for sharing.

", "votes": 0, "replies": [] } ], "comment_count": 38, "messages": [ { "id": 726254, "postDate": "2020-01-23T00:23:00.793Z", "content": "

Approach

\n\n\n\n

Results

\n\n\n\n

Final Remarks

\n\n\n\n

Validation setup

\n\n\n\n

Update 1: Features

\n\n

Motivation:\nI was impacted by the ideas presented in this paper by Francois Chollet: On the measure of intelligence. There are tons of interesting and powerful thoughts there. I was mostly stimulated by a discussion on how to measure intelligence: \n- A/ by overall-skill-level \n- B/ by skill-acquisition-tempo

\n\n

In our case, we are measured by A/, which can be broken into two drivers:\n- experience, i.e. how much time/effort the child has spent on various actitivities in the game. This formed my first group of features\n- accuracy - how accurate was this child in her journey. This formed my second group of features.

\n\n

However, skill-acquisition-tempo is a very interesting way to capture how quickly children are learning (features like minutes per level, events per level, etc.). This formed by 3rd group of features

\n\n

I love competitions with manual feature engineering. Combination human+machine wins, which represents my view on how AI will impact the world.

\n\n

Update 2: Feature selection

\n\n", "rawMarkdown": "# Approach\n- The best part of the competition for me was feature engineering. In the end I used 51 features, truncated down from 150.\n- By far strongest features were base distributions of each Assessment. However, they were used by everyone. Individual features, given small data, were not so important, but they decided winners I think.\n- Final model was an ensemble: 0.3 LGB, 0.3 CATB, 0.4 NN\n- 20 fold-bagging for all models, for NN additionally averaging across 3 seeds \n- One “trick” – use assessments from test set “blindly” (as we did not see this data) as samples in training. Data was scarce – so I was looking for every way to increase the number of training samples. Especially that you add data exactly for the children that are in the private LB. \n\n# Results\n- Truncated CV: 0.575\n- Private LB: 0.559\n- Public LB: 0.559\n\n# Final Remarks\n- Congrats to winners - looking forward to your solutions\n- Thanks to the organizers for the competition with event data – love those 😊 It unleashes your creativity in feature engineering\n\n# Validation setup\n- Truncate - select one assessment randomly for every child to reflect test set structure\n\n# Update 1: Features\nMotivation:\nI was impacted by the ideas presented in this paper by Francois Chollet: [On the measure of intelligence](https://arxiv.org/abs/1911.01547). There are tons of interesting and powerful thoughts there. I was mostly stimulated by a discussion on how to measure intelligence: \n- A/ by overall-skill-level \n- B/ by skill-acquisition-tempo\n\nIn our case, we are measured by A/, which can be broken into two drivers:\n- experience, i.e. how much time/effort the child has spent on various actitivities in the game. This formed my first group of features\n- accuracy - how accurate was this child in her journey. This formed my second group of features.\n\nHowever, skill-acquisition-tempo is a very interesting way to capture how quickly children are learning (features like minutes per level, events per level, etc.). This formed by 3rd group of features\n\nI love competitions with manual feature engineering. Combination human+machine wins, which represents my view on how AI will impact the world.\n\n#Update 2: Feature selection\n- Calculated cv score after dropping a feature - did this individually for all ~150 features\n- Dropped all features which brought an improvement of less than 0.0001 on QWK score - I treat them as noise. Found ~100 noise features in this way.\n- Recalcuated CV once more to see that overall score improved slightly after removing 100 noise features\n", "votes": 81 }, { "id": 726486, "postDate": "2020-01-23T03:08:44.887Z", "content": "

Congrats! Could you shed some light on your feature selection approach? How did you choose those 51?

", "rawMarkdown": "Congrats! Could you shed some light on your feature selection approach? How did you choose those 51?", "votes": 4, "replies": [ { "id": 726515, "postDate": "2020-01-23T03:29:13.950Z", "content": "

Sure - please see updated post above

", "rawMarkdown": "Sure - please see updated post above", "votes": 2 }, { "id": 726517, "postDate": "2020-01-23T03:31:14.723Z", "content": "

Thanks!

", "rawMarkdown": " Thanks! ", "votes": 1 } ] }, { "id": 726269, "postDate": "2020-01-23T00:35:03.750Z", "content": "

thank you so much @narsil \ncould you share those 51 features or how to produce them?\nhow did you add test assesments as part of the train? (just trying to learn)

", "rawMarkdown": "thank you so much @narsil \ncould you share those 51 features or how to produce them?\nhow did you add test assesments as part of the train? (just trying to learn)", "votes": 4, "replies": [ { "id": 726371, "postDate": "2020-01-23T01:31:32.383Z", "content": "

I am also curious about the features, since I didn't do good feature selection.

", "rawMarkdown": "I am also curious about the features, since I didn't do good feature selection. ", "votes": 1 }, { "id": 726410, "postDate": "2020-01-23T02:05:27.620Z", "content": "

Sure, please see the updated post above. Feature engineering was the most fun part of this competition.

", "rawMarkdown": "Sure, please see the updated post above. Feature engineering was the most fun part of this competition. " }, { "id": 726521, "postDate": "2020-01-23T03:48:11.007Z", "content": "

@narsil , how new feature engineering worked with you , after my initial hypothesis the minute i add up new feature e.g. number of distraction before an assessment from event data or number of incorrect assessment for the same assessment world , my public LB got made and decreased from 0.40 to less than 0.30

\n\n

Top 103rd Solution

", "rawMarkdown": "@narsil , how new feature engineering worked with you , after my initial hypothesis the minute i add up new feature e.g. number of distraction before an assessment from event data or number of incorrect assessment for the same assessment world , my public LB got made and decreased from 0.40 to less than 0.30\n\n[Top 103rd Solution](https://www.kaggle.com/c/data-science-bowl-2019/discussion/127244#726470)" }, { "id": 727647, "postDate": "2020-01-23T23:58:42.457Z", "content": "

I always try to find some common sense logic and build a framework to guide me in the feature engineering process. This time I did some brainstormin and used the paper from Francois Chollet. This initial 'framing' stage helps me make sure I don't miss an important feature group.

", "rawMarkdown": "I always try to find some common sense logic and build a framework to guide me in the feature engineering process. This time I did some brainstormin and used the paper from Francois Chollet. This initial 'framing' stage helps me make sure I don't miss an important feature group.", "votes": 1 }, { "id": 727899, "postDate": "2020-01-24T07:03:52.720Z", "content": "

@narsil can you refer this paper please?

", "rawMarkdown": "@narsil can you refer this paper please?", "votes": 1 }, { "id": 727901, "postDate": "2020-01-24T07:07:25.797Z", "content": "

Sure: https://arxiv.org/abs/1911.01547\nIt is also in the write-up, section Update 1: Features

", "rawMarkdown": "Sure: https://arxiv.org/abs/1911.01547\nIt is also in the write-up, section Update 1: Features", "votes": 1 }, { "id": 727919, "postDate": "2020-01-24T07:43:53.463Z", "content": "

@narsil thanks mate

", "rawMarkdown": "@narsil thanks mate " } ] }, { "id": 726935, "postDate": "2020-01-23T10:19:28.653Z", "content": "

great, congrats!\nWhat exactly 20-fold bagging means?\nIs that you used bootstrap to resample and create 20 bags? Did you use replacement? What was the size of each bag?\nThanks

", "rawMarkdown": "great, congrats!\nWhat exactly 20-fold bagging means?\nIs that you used bootstrap to resample and create 20 bags? Did you use replacement? What was the size of each bag?\nThanks", "votes": 2, "replies": [ { "id": 727651, "postDate": "2020-01-24T00:01:25.687Z", "content": "

I split data into 20 folds. I build model separately for each fold and\n- for cv I take oof predictions for every fold\n- for submission I take the average of predictions across all folds

", "rawMarkdown": "I split data into 20 folds. I build model separately for each fold and\n- for cv I take oof predictions for every fold\n- for submission I take the average of predictions across all folds", "votes": 1 } ] }, { "id": 727956, "postDate": "2020-01-24T08:52:25.343Z", "content": "

Congratulations ,and thank you for sharing insight into your process!!! I'm new to ML and this was my first project and Kaggle competition. Still I was a bit surprised and disappointed on how badly I got my butt kicked. I also saw that the best correlation I could find was the assessment that the kids were taking (I guess some assessments are easier than others.) But I was hoping that I could find or combine other features that would also be good (I found a couple that improved a little with by logging then, but that's it). I didn't end up using a NN because I couldn't seem to get it to reach a global minimum. Anyway, any advice, other than what you have already given (which is great) that could help me in the future with Kaggle competitions as well as on ML projects in general? I guess if we had cleaner training data (devices not shared by other people) we might have seen more useful patterns in the kids behavior. I did try to filter out some records that I thought might be the case such as when the first action on the device was an assignment and the person got it correct the first time. I assumed that was a parent or teacher trying it out, but removing them only made my public score worse. As well, my cv score was .58 but my private score ended up being around 4.3. So big difference there. I ended up only using one lgbmclassifier. So I guess one take away is to use bagging with multiple classifiers. Anyway, thanks again. Any more insight you could provide when you have time would be great.

", "rawMarkdown": "Congratulations ,and thank you for sharing insight into your process!!! I'm new to ML and this was my first project and Kaggle competition. Still I was a bit surprised and disappointed on how badly I got my butt kicked. I also saw that the best correlation I could find was the assessment that the kids were taking (I guess some assessments are easier than others.) But I was hoping that I could find or combine other features that would also be good (I found a couple that improved a little with by logging then, but that's it). I didn't end up using a NN because I couldn't seem to get it to reach a global minimum. Anyway, any advice, other than what you have already given (which is great) that could help me in the future with Kaggle competitions as well as on ML projects in general? I guess if we had cleaner training data (devices not shared by other people) we might have seen more useful patterns in the kids behavior. I did try to filter out some records that I thought might be the case such as when the first action on the device was an assignment and the person got it correct the first time. I assumed that was a parent or teacher trying it out, but removing them only made my public score worse. As well, my cv score was .58 but my private score ended up being around 4.3. So big difference there. I ended up only using one lgbmclassifier. So I guess one take away is to use bagging with multiple classifiers. Anyway, thanks again. Any more insight you could provide when you have time would be great.", "votes": 1, "replies": [ { "id": 728638, "postDate": "2020-01-25T02:02:34.300Z", "content": "

Hey.

\n\n

Great to be after your 1st competition - you know more how Kaggle works now.\nFor me - it took 2-3 competitions before I realized what is going on. Since 4th or 5th competition I could compete for real.\nIf your score dropped from 0.58 to 4.3, it means there must be a bug in the code. Just by saying 2.0 for every assessment, your score could be around 1.0.

\n\n

My advice: keep going and have fun. Zero expectations on score, but maximize learning.

", "rawMarkdown": "Hey. \n\nGreat to be after your 1st competition - you know more how Kaggle works now.\nFor me - it took 2-3 competitions before I realized what is going on. Since 4th or 5th competition I could compete for real.\nIf your score dropped from 0.58 to 4.3, it means there must be a bug in the code. Just by saying 2.0 for every assessment, your score could be around 1.0.\n\nMy advice: keep going and have fun. Zero expectations on score, but maximize learning. \n" }, { "id": 728783, "postDate": "2020-01-25T08:54:44.687Z", "content": "

Good advice! Maximizing learning seems like the best goal (although $100,000 wouldn't have been a bad thing as well :-) I wonder if it was a mistake for me to use the class weights while training? If I didn't use them though, my model would end up predicting only two or three category 2s in the test set. I have much to learn! Thanks again!

", "rawMarkdown": "Good advice! Maximizing learning seems like the best goal (although $100,000 wouldn't have been a bad thing as well :-) I wonder if it was a mistake for me to use the class weights while training? If I didn't use them though, my model would end up predicting only two or three category 2s in the test set. I have much to learn! Thanks again!", "votes": 1 } ] }, { "id": 727227, "postDate": "2020-01-23T14:43:34.757Z", "content": "

Thank you for the write-up, i have a few basic questions:\n1)

\n\n
\n

20 fold-bagging for all models

\n
\n\n

from what i understand bagging is the sampling with replacement to form a new data set, is the new data set you sampled the same size as the training data (or smaller/bigger)? then what do you use as your validation set? (the ones did not get sampled?) And did you do anything special with sampling e.g. sample same number of data points from each installation_id instead of sampling uniformly from all available training data points (ignoring groups)?

\n\n

2)

\n\n
\n

Truncate - select one assessment randomly for every child to reflect test set structure

\n
\n\n

Is the select one only happens once per installation_id for one validation set, or you repeatedly sample (with replacement) one from each installation_id? otherwise i imagine the validation set would be small and unstable

\n\n

3)

\n\n
\n

Calculated cv score after dropping a feature - did this individually for all ~150 features

\n
\n\n

its nice that it worked out for you, although did you look into feature selections with Shapley value (shap library) i.e. drop features with small or zero shap importance on validation set? One potential draw-back for evaluating features one by one, what if there are feature interactions among them that each features may not matter much but a certain combination would?

\n\n

4) Did you look into adversarial validation? if not, is it because the public test set is small? and what are your thoughts on adjusting the feature values by comparing to test features statistic (as some public kernel did)?

\n\n

5) At last, a very general question, normally how do you validate the features your created? such as making sure there are no obvious bugs or silent bugs? (sometimes I discover by chance that there's silent bug in my code, i didnt notice as they did not throw errors and the value seems like to be in correct range, do you do say unitest on your feature generation code? thats sounds a bit crazy)

\n\n

thanks for reading, and congrats on the winning!

", "rawMarkdown": "Thank you for the write-up, i have a few basic questions:\n1)\n> 20 fold-bagging for all models\n\nfrom what i understand `bagging` is the sampling with replacement to form a new data set, is the new data set you sampled the same size as the training data (or smaller/bigger)? then what do you use as your validation set? (the ones did not get sampled?) And did you do anything special with sampling e.g. sample same number of data points from each `installation_id` instead of sampling uniformly from all available training data points (ignoring groups)?\n\n2) \n> Truncate - select one assessment randomly for every child to reflect test set structure\n\nIs the `select one` only happens once per `installation_id` for one validation set, or you repeatedly sample (with replacement) one from each installation_id? otherwise i imagine the validation set would be small and unstable\n\n3) \n>Calculated cv score after dropping a feature - did this individually for all ~150 features\n\nits nice that it worked out for you, although did you look into feature selections with Shapley value (shap library) i.e. drop features with small or zero shap importance on validation set? One potential draw-back for evaluating features one by one, what if there are feature interactions among them that each features may not matter much but a certain combination would?\n\n4) Did you look into adversarial validation? if not, is it because the public test set is small? and what are your thoughts on adjusting the feature values by comparing to test features statistic (as some public kernel did)?\n\n5) At last, a very general question, normally how do you validate the features your created? such as making sure there are no obvious bugs or silent bugs? (sometimes I discover by chance that there's silent bug in my code, i didnt notice as they did not throw errors and the value seems like to be in correct range, do you do say unitest on your feature generation code? thats sounds a bit crazy)\n\nthanks for reading, and congrats on the winning!", "votes": 1, "replies": [ { "id": 727657, "postDate": "2020-01-24T00:12:53.663Z", "content": "

Hello.

\n\n
    \n
  1. I reply below in the post from @pabuoro
    2. \n
  2. I do it only once and get an indeed small sample, but then I repeat this small sample 100 times to get a stable result
    3. \n
  3. I did a final check on CV score after dropping the features - to see exactly what you are pointing out.
    4. \n
  4. No. I should have, but didn't have time, and it wasn't high on my priorities. Adjusting the feature values - did not put too much thought on this, but at first glance seems very dangerous to me,
    5. \n
  5. I do indeed some tests on my features to prevent bugs - normally I look at their distributions and CV scores after inclusion in the model
    6. \n
", "rawMarkdown": "Hello.\n\n1. I reply below in the post from @pabuoro \n2. I do it only once and get an indeed small sample, but then I repeat this small sample 100 times to get a stable result\n3. I did a final check on CV score after dropping the features - to see exactly what you are pointing out.\n4. No. I should have, but didn't have time, and it wasn't high on my priorities. Adjusting the feature values - did not put too much thought on this, but at first glance seems very dangerous to me,\n5. I do indeed some tests on my features to prevent bugs - normally I look at their distributions and CV scores after inclusion in the model", "votes": 1 } ] }, { "id": 726951, "postDate": "2020-01-23T10:34:56.730Z", "content": "

Congratulations ,and thank you for sharing your solution!\nI have a question about value of feature selection.\nDo you also need feature selection when you don't use NN?\nI've learned you don't have to worry too much about noise when you use only GBDTs.

", "rawMarkdown": "Congratulations ,and thank you for sharing your solution!\nI have a question about value of feature selection.\nDo you also need feature selection when you don't use NN?\nI've learned you don't have to worry too much about noise when you use only GBDTs.", "votes": 1, "replies": [ { "id": 727652, "postDate": "2020-01-24T00:02:47.963Z", "content": "

My general experience is that the simpler the model, the more generalization power it has

\n\n

Complex models can fit the past data very well but are not so good at predicting the future. Simple models don't fit the past very well, but whatever they fit has very similar predictive power for the future

", "rawMarkdown": "My general experience is that the simpler the model, the more generalization power it has\n\nComplex models can fit the past data very well but are not so good at predicting the future. Simple models don't fit the past very well, but whatever they fit has very similar predictive power for the future", "votes": 2 }, { "id": 727833, "postDate": "2020-01-24T05:13:14.010Z", "content": "

Thanks!

", "rawMarkdown": "Thanks!", "votes": 1 } ] }, { "id": 726862, "postDate": "2020-01-23T09:33:51.210Z", "content": "

Congrats on your first Solo Gold!

", "rawMarkdown": "Congrats on your first Solo Gold! ", "votes": 1, "replies": [ { "id": 727650, "postDate": "2020-01-23T23:59:49.067Z", "content": "

Thanks Sasza :) Remember out deal? We are going to pimiento when we visit Krakow :)

", "rawMarkdown": "Thanks Sasza :) Remember out deal? We are going to pimiento when we visit Krakow :)", "votes": 1 } ] }, { "id": 726665, "postDate": "2020-01-23T06:39:36.507Z", "content": "

Your solution is concise and elegant. Congrats!

", "rawMarkdown": "Your solution is concise and elegant. Congrats!", "votes": 1, "replies": [ { "id": 727645, "postDate": "2020-01-23T23:56:26.600Z", "content": "

Thanks! Congrats to your team as well - you managed to stay in the gold zone and survive the shakeup, which is impressive

", "rawMarkdown": "Thanks! Congrats to your team as well - you managed to stay in the gold zone and survive the shakeup, which is impressive", "votes": 1 } ] }, { "id": 726616, "postDate": "2020-01-23T06:03:33.770Z", "content": "

Can you share the code you used for the competition?

", "rawMarkdown": "Can you share the code you used for the competition?", "votes": 1, "replies": [ { "id": 727648, "postDate": "2020-01-23T23:59:09.853Z", "content": "

I do not plan to release my code this time

", "rawMarkdown": "I do not plan to release my code this time", "votes": 1 } ] }, { "id": 726508, "postDate": "2020-01-23T03:23:10.210Z", "content": "

Congratulations\nThanks for sharing your Approach & Insights!! @narsil

", "rawMarkdown": "Congratulations\nThanks for sharing your Approach & Insights!! @narsil ", "votes": 1 }, { "id": 726451, "postDate": "2020-01-23T02:34:42.323Z", "content": "

Thanks for sharing and Congrats on your gold medal!

\n\n

\nOne “trick” – use assessments from test set “blindly” (as we did not see this data) as samples in training. Data was scarce – so I was looking for every way to increase the number of training samples\n\nBTW, could you say a bit more about using assessment from test set? I tried to mix up train and test set. Public LB decrease by about 0.003.(Not sure about private LB).

", "rawMarkdown": "Thanks for sharing and Congrats on your gold medal!\n\n```\nOne “trick” – use assessments from test set “blindly” (as we did not see this data) as samples in training. Data was scarce – so I was looking for every way to increase the number of training samples\n```\nBTW, could you say a bit more about using assessment from test set? I tried to mix up train and test set. Public LB decrease by about 0.003.(Not sure about private LB).", "votes": 1, "replies": [ { "id": 726482, "postDate": "2020-01-23T03:06:17.227Z", "content": "

This will require some work from me, but I will try to do this experiment and see results. \nI assumed early in the competition that additional data is good.

\n\n

I know, I know: \"assumed\" - the most dangerous word of all. But we need to have some cornerstones of our strategy, and having more data is a good one almost always. Especially when you realize that you add data exactly for the children that are in the private LB.

", "rawMarkdown": "This will require some work from me, but I will try to do this experiment and see results. \nI assumed early in the competition that additional data is good. \n\nI know, I know: \"assumed\" - the most dangerous word of all. But we need to have some cornerstones of our strategy, and having more data is a good one almost always. Especially when you realize that you add data exactly for the children that are in the private LB.", "votes": 2 }, { "id": 726494, "postDate": "2020-01-23T03:13:51.097Z", "content": "

Thanks. Look forward to your experiment.

", "rawMarkdown": "Thanks. Look forward to your experiment.", "votes": 1 }, { "id": 726501, "postDate": "2020-01-23T03:17:44.020Z", "content": "

Actually you can probing changing of CV score instead of assuming ;)

", "rawMarkdown": "Actually you can probing changing of CV score instead of assuming ;) ", "votes": 1 }, { "id": 726506, "postDate": "2020-01-23T03:21:27.897Z", "content": "

Fair comment. \"Assumed\" was a shortcut.

\n\n

In my case, after adding test data:\n- CV increased a little. I expected more, but I was relieved to see it does not destroy the score.\n- Public LB dropped a little. So I \"assumed\" that it would still be good for the private LB.

", "rawMarkdown": "Fair comment. \"Assumed\" was a shortcut.\n\nIn my case, after adding test data:\n- CV increased a little. I expected more, but I was relieved to see it does not destroy the score.\n- Public LB dropped a little. So I \"assumed\" that it would still be good for the private LB.", "votes": 1 } ] }, { "id": 726424, "postDate": "2020-01-23T02:18:57.160Z", "content": "

Congrats on your gold medal @narsil !

", "rawMarkdown": "Congrats on your gold medal @narsil !", "votes": 1, "replies": [ { "id": 726581, "postDate": "2020-01-23T05:20:40.933Z", "content": "

Thanks so much!

", "rawMarkdown": "Thanks so much!" } ] }, { "id": 726415, "postDate": "2020-01-23T02:10:32.047Z", "content": "

I want to save this comment. Beautifully written and quickly explains the system you built. A great example for students. Well done and UP voted! Congrats on your gold medal !

", "rawMarkdown": "I want to save this comment. Beautifully written and quickly explains the system you built. A great example for students. Well done and UP voted! Congrats on your gold medal ! ", "votes": 1, "replies": [ { "id": 726582, "postDate": "2020-01-23T05:20:45.420Z", "content": "

Thanks so much!

", "rawMarkdown": "Thanks so much!" } ] }, { "id": 733269, "postDate": "2020-01-30T22:58:50.533Z", "content": "

Congrats and thanks for sharing.

", "rawMarkdown": "Congrats and thanks for sharing." } ], "complete": true, "source": "live", "fetched_at": "2026-10-22T10:52:46Z", "competition_id": 16531, "competition_slug": "data-science-bowl-2019", "team_id": 3877268, "solution_url": "/competitions/data-science-bowl-2019/writeups/pawel-z-lublina-7th-place-solution" }