![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1658108%2F6a4d5317015ad5235b526b46d1327fee%2Fdsb20192.png?generation=1582589525333424&alt=media\")
I'm sorry. It was a traditional Korean holidays until today, so I didn't have time to write this. Thank you for your patience.
\n\nFirst of all, I would like to thank Booz Allen Hamilton for hosting this interesting competition. And congratulates to the all participants and especially the winners!
\n\nI like deep neural networks so I prefer to solve all the problems with a deep neural networks. π
\n\nI focus on the structure of the input data rather than understanding the input data. And concentrate on making the model's input by avoiding missing information as much as possible, hoping that the model will do more than I expected. π
\n\nIn other words, I focus less on feature engineering and more on finding a neural net model architecture that fits the data.
The sequence of installation_id is too long to be used as it is. So I aggregated log data (train_df) by game_session. Please see the example below.\n\ndf = train_df\nevent_code = pd.crosstab(df['game_session'], df['event_code'])\nevent_id = pd.crosstab(df['game_session'], df['event_id'])\n...\nagged_df = pd.concat([event_code, event_id, game_accuracy, max_round])\nsession_df = df.drop_duplicates('game_session', keep='last').reset_index(drop=True)\nsession_df = session_df.merge(agged_df, how='left', on='game_session')\n
The LSTM and TRANSFORMER models in NLP receive sequence of words (or sentence) as input. Similarly, I will use the sequence of game_sessions (or installation_id) as input here.
\n\nBest private score: 0.564\nSingle transformer model used.
\n\n
Prediction from game_sessions of an installation_id
\n\nCategorical columns (such as event_code, title, world, etc...) were embedded respectively. Then, the categorical_vector were obtained by concatenating the embeddings. Next the nn.linear layer is applied for the dimension reduction of the categorical vector.\n\nself.categorical_proj = nn.Sequential(\n nn.Linear(cfg.emb_size*num_categorical_columns, cfg.hidden_size//2),\n nn.LayerNorm(cfg.hidden_size//2),\n )
\n
Continuous columns were embedded directly using a linear layer.\n\nself.continuous_emb = nn.Sequential( \n* I used np.log1p for normalization of continuous columns.
\n nn.Linear(num_continuous_columns, cfg.hidden_size//2),\n nn.LayerNorm(cfg.hidden_size//2),\n )\n
https://www.kaggle.com/c/data-science-bowl-2019/discussion/124836\nAs mentioned in this link, the 0 and 3 classes of the accuracy_group may be very close. \nnum_correct can have 0 or 1, if the num_correct has 1 then the accuracy_group increases 3 points.\nOn the other hand, num_incorrect decreases 1 point when num_incorrect has 1 and decreases 2 points when num_incorrect has 2 or more.
\n\nThis could be expressed as \n```
\n\nnew_accuracy_group = 3 * num_correct - num_incorrect\n```
\n\nUsing the above equation, we can calculate the real values of 0 to 3 from num_correct, num_incorrect.\nTherefore, the prediction of the model is set to [num_correct_pred, num_incorrect_pred]
\n\n```\nprediction = model(x) # prediction = [num_correct_pred, num_incorrect_pred]
\n\n\n**Then train the model with the modified_loss below.**\n\nmodified_loss = mse_loss( prediction, target )\n```
After the training is done, we can use the new_accuracy_group calculated from \"num_correct_pred, num_incorrect_pred\".\n\nnum_correct_pred, num_incorrect_pred = prediction\nnew_accuracy_group = 3 * num_correct_pred - num_incorrect_pred\n
We can also use original accuracy_group to slightly improve performance.\n```\nprediction = model(x) # prediction = [accuracy_group_pred, num_correct_pred, num_incorrect_pred]
\n\n```
\n\nThe final_accuracy_group is calculated as below.\n\nnew_accuracy_group = 3 * num_correct_pred - num_incorrect_pred\nfinal_accuracy_group = (accuracy_group_pred + new_accuracy_group) / 2\n
I generated an additional label for game_sessions, where the type is Game. From the \"correct\":true, βcorrectβ:false of event_data, I was able to create num_correct and num_incorrect, and likewise I was able to create an accuracy_group.\nThe number of additional training samples generated is 41,194.
\n\nPre-training and fine-tuning steps\n\nhttps://www.kaggle.com/limerobot/dsb2019-v77-tr-dt-aug0-5-3tta?scriptVersionId=27448615
\n\nIt is a shame to me sharing the uncleane code. But first I decided to share the kernel and make a clean code. Maybe in two weeks? ;)
\n\nIt is a shame to me sharing the uncleane code. \nBut first I decided to share the kernel and make a clean code. Maybe in two weeks?\nhttps://www.kaggle.com/limerobot/dsb2019-v77-tr-dt-aug0-5-3tta?scriptVersionId=27448615
", "votes": 5, "replies": [ { "id": 731937, "author_name": "Y. O.", "author_url": "", "post_date": "2020-01-29T09:17:52.630000", "content": "Thank you so much. Iβm sorry for creating the extra work.
", "votes": 1, "replies": [] }, { "id": 732212, "author_name": "Limerobot", "author_url": "", "post_date": "2020-01-29T15:22:09.887000", "content": "It's ok π
", "votes": 1, "replies": [] } ] }, { "id": 732690, "author_name": "Yan Linn Aung", "author_url": "", "post_date": "2020-01-30T05:39:07.343000", "content": "Great job. Thanks for sharing!
", "votes": 2, "replies": [ { "id": 732692, "author_name": "Agupon", "author_url": "", "post_date": "2020-01-30T05:42:49.400000", "content": "*YEAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!πͺ π π *
", "votes": 0, "replies": [] } ] }, { "id": 731558, "author_name": "kensler", "author_url": "", "post_date": "2020-01-28T19:33:18.833000", "content": "@limerobot, congratulations and thanks for the interesting writeup! This was my first Kaggle competition and I invested a ton of time on it. Very frustrating that we had kernels that would have been solid silver but we ended up just missing Bronze. I had an idea for and LSTM using train summarized by session but we didnt quite get it working. I'd really like to understand all the details of how you manipulated the data and built your model. Might you share the code?
", "votes": 2, "replies": [ { "id": 731727, "author_name": "Limerobot", "author_url": "", "post_date": "2020-01-29T01:04:35.370000", "content": "Please check your e-mail π
", "votes": 2, "replies": [] }, { "id": 731840, "author_name": "kensler", "author_url": "", "post_date": "2020-01-29T06:08:36.617000", "content": "Many thanks! So much to learn!!! π
", "votes": 0, "replies": [] } ] }, { "id": 731500, "author_name": "Abdur Rehman", "author_url": "", "post_date": "2020-01-28T17:56:49.867000", "content": "@limerobot Great Solution and Congrats on winning. Can you share your kernel please? I want to give a try to your approach and expecting a lot to learn from your solution.
", "votes": 2, "replies": [ { "id": 731728, "author_name": "Limerobot", "author_url": "", "post_date": "2020-01-29T01:04:42.010000", "content": "Please check your e-mail π
", "votes": 3, "replies": [] }, { "id": 731961, "author_name": "toshi_k", "author_url": "", "post_date": "2020-01-29T09:53:45.893000", "content": "@limerobot Could you share your kernel with me too ? I want to compare my model with yours.
\n\nEDIT: Sorry, I've missed the link. Thank you for sharing !!
", "votes": 1, "replies": [] }, { "id": 731998, "author_name": "Abdur Rehman", "author_url": "", "post_date": "2020-01-29T11:10:31.620000", "content": "Thanks @limerobot
", "votes": 0, "replies": [] } ] }, { "id": 1058685, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-10-24T05:20:21.173000", "content": "Could you please share your kernel..
", "votes": 1, "replies": [] }, { "id": 754013, "author_name": "Jye", "author_url": "", "post_date": "2020-02-22T23:23:08.630000", "content": "Great solution, and thanks for sharing the code and write-up.
\n\nI'm just curious about one thing. The goal of the competition was to \"Uncover the factors to help measure how young children learn\". With something like a decision tree we can quite easily have an idea about the most important factors, but with a Transformer model (or nn in general) that seems to be a lot harder.
\n\nHow do you think your model could be used to identify the best learning patterns in children? Do you think it's better to have a less accurate but more readable model for the use case of the competition organisers?
\n\nWould be great to hear your thoughts!
", "votes": 1, "replies": [ { "id": 772339, "author_name": "Limerobot", "author_url": "", "post_date": "2020-03-15T11:28:33.580000", "content": "@jyesawtellrickson I'm really sorry for late reply. Please consider using notifications(@) to me. π±
\n\nI think the results of various models will help competition organisers.
\n\nFor example, the fact that performance has decreased when sequential information is used can help clarify the relationship between random path and linear progression.\nBelow is what competition organiser mentioned.
\n\nhttps://www.kaggle.com/c/data-science-bowl-2019/discussion/115034
\n\n\n", "votes": 1, "replies": [] } ] }, { "id": 741660, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2020-02-10T21:34:51.520000", "content": "The app is designed to try to guide the kids through an idealized learning path, which is intended to present players with a pattern of exposure->exploration->practice->demonstration (as in demonstration of knowledge). Each of the worlds in the app may have one or more such sequences of media objects, and sometimes the app does not follow this exact formula.
\n \n\n
However, kids are not required to follow the path that is laid out for them, and whether the suggested linear progression leads to better learning outcomes than a random path is not yet clear.Perhaps this competition will give us some insights into this question as well!
So much great ideas. And I should get better at applying transformers. Thanks for sharing the insights and the kernel!
", "votes": 1, "replies": [] }, { "id": 737144, "author_name": "Yuan", "author_url": "", "post_date": "2020-02-05T00:48:49.720000", "content": "Congratz and thanks for the big sharing! It is super helpful for learning! π π€
", "votes": 1, "replies": [] }, { "id": 733698, "author_name": "Akhil Gupta", "author_url": "", "post_date": "2020-01-31T12:47:46.153000", "content": "Nice work
", "votes": 1, "replies": [] }, { "id": 733639, "author_name": "HELLOWORLD", "author_url": "", "post_date": "2020-01-31T11:30:18.540000", "content": "what is transformer? i dont know that. can you describe some?
", "votes": 1, "replies": [ { "id": 772342, "author_name": "Limerobot", "author_url": "", "post_date": "2020-03-15T11:33:21.797000", "content": "Please refer to the \"Attention Is All You Need\" paper or a well-organized blog on the Internet. π \nI think that's a better way.
", "votes": 0, "replies": [] } ] }, { "id": 733511, "author_name": "Parikshit Agarwal", "author_url": "", "post_date": "2020-01-31T08:35:49.117000", "content": "very impressive!!
", "votes": 1, "replies": [] }, { "id": 733462, "author_name": "Nandesh Prasad", "author_url": "", "post_date": "2020-01-31T06:47:32.763000", "content": "Great Job Buddy
", "votes": 1, "replies": [] }, { "id": 733317, "author_name": "kevin_hu", "author_url": "", "post_date": "2020-01-31T00:28:39.570000", "content": "very interesting kernel.
", "votes": 1, "replies": [] }, { "id": 733138, "author_name": "Patrick Chan", "author_url": "", "post_date": "2020-01-30T18:09:16.540000", "content": "Congrats Limerobot. Very good impressive work with a single model. Appreciate if you can also point us to your training kernel and code. Many thanks.
", "votes": 1, "replies": [] }, { "id": 732670, "author_name": "Kuldeep Arya", "author_url": "", "post_date": "2020-01-30T04:42:15.660000", "content": "it's very interesting and great kernal its very useful for learning purpose. thanks for sharing
", "votes": 1, "replies": [] }, { "id": 732566, "author_name": "corochann", "author_url": "", "post_date": "2020-01-30T00:13:03.790000", "content": "Interesting approach. Congrats!
", "votes": 1, "replies": [] }, { "id": 732295, "author_name": "Miyabon", "author_url": "", "post_date": "2020-01-29T16:39:04.940000", "content": "Congrats & Thanks for sharing!!π π π
", "votes": 1, "replies": [] }, { "id": 732232, "author_name": "S.Sakai_65", "author_url": "", "post_date": "2020-01-29T15:43:13.933000", "content": "Great job. Really helpful. Thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 732222, "author_name": "Wonho Song", "author_url": "", "post_date": "2020-01-29T15:32:23.920000", "content": "It's a really beautiful write-up. Thank you for sharing !
", "votes": 1, "replies": [] }, { "id": 732009, "author_name": "Kevin_Tran", "author_url": "", "post_date": "2020-01-29T11:30:20.637000", "content": "very helpful :)
", "votes": 1, "replies": [] }, { "id": 731989, "author_name": "hendrikjoosten", "author_url": "", "post_date": "2020-01-29T10:43:42.053000", "content": "This is a very good explanation, looking forward to the clean code release!
", "votes": 1, "replies": [] }, { "id": 731979, "author_name": "Dan Ofer", "author_url": "", "post_date": "2020-01-29T10:26:49.573000", "content": "Dang, 3d place with just 1 model and relatively little FE? Neat! \n(It's interesting to see how Attention models worked well for some and not others. How much worse was your model with the PE?).
", "votes": 1, "replies": [ { "id": 732219, "author_name": "Limerobot", "author_url": "", "post_date": "2020-01-29T15:29:18.027000", "content": "I remember with PE decreased the local CV score about 0.02~0.03
", "votes": 0, "replies": [] } ] }, { "id": 731909, "author_name": "Changmin Choi", "author_url": "", "post_date": "2020-01-29T08:14:36.767000", "content": "Congrats on winning the competition and thank you for sharing the approach!! \nCould you share your code please?
", "votes": 1, "replies": [ { "id": 732217, "author_name": "Limerobot", "author_url": "", "post_date": "2020-01-29T15:24:24.443000", "content": "Please see the link above π
", "votes": 0, "replies": [] } ] }, { "id": 731028, "author_name": "toshi_k", "author_url": "", "post_date": "2020-01-28T09:18:34.020000", "content": "Thank you for sharing !\nHow did you aggregate the output of Transformer ? Are the numbers of game_session different for each installation_id ?
Yep, installation_id is a variable-length sequence.\nSo I set the sequence length to 100 and I added PAD tokens to the front in the case of a short sequence. Then like the figure above, I took the last output of the sequence.
", "votes": 1, "replies": [] } ] }, { "id": 730885, "author_name": "Raghawendra Singh", "author_url": "", "post_date": "2020-01-28T04:03:51.453000", "content": "Very impressive!!! Thanks for sharing your solution.
", "votes": 1, "replies": [] }, { "id": 730884, "author_name": "Hanjoon Choe", "author_url": "", "post_date": "2020-01-28T03:59:26.780000", "content": "Thank you for sharing! I should learn this! ;-) Congratulations!
", "votes": 1, "replies": [] }, { "id": 730835, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-01-28T02:01:00.640000", "content": "@limerobot Great solution! May I ask would you share your kernel? I want to try to blend your method with mine and see the effect.
", "votes": 1, "replies": [ { "id": 731143, "author_name": "Limerobot", "author_url": "", "post_date": "2020-01-28T11:55:20.873000", "content": "Please check your e-mail π
", "votes": 1, "replies": [] }, { "id": 731851, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-01-29T06:16:59.333000", "content": "Thanks. I will do that in 2 weeks and let you know the result.
", "votes": 1, "replies": [] } ] }, { "id": 730825, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-28T01:39:23.410000", "content": "Amazing work. Congrats on 3rd place and GM status - you earned it.
", "votes": 1, "replies": [] }, { "id": 730578, "author_name": "Gammer", "author_url": "", "post_date": "2020-01-27T16:44:44.783000", "content": "Great job. I was curious about your solution. Thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 730430, "author_name": "", "author_url": "", "post_date": "2020-01-27T13:30:50.150000", "content": "", "votes": 1, "replies": [ { "id": 730814, "author_name": "", "author_url": "", "post_date": "2020-01-28T00:35:43.240000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 3455513, "author_name": "", "author_url": "", "post_date": "2026-05-09T15:25:48.787000", "content": "", "votes": 0, "replies": [] }, { "id": 731829, "author_name": "", "author_url": "", "post_date": "2020-01-29T05:56:25.900000", "content": "", "votes": 0, "replies": [ { "id": 732215, "author_name": "", "author_url": "", "post_date": "2020-01-29T15:24:05.560000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 731801, "author_name": "", "author_url": "", "post_date": "2020-01-29T04:40:24.670000", "content": "", "votes": 0, "replies": [ { "id": 732214, "author_name": "", "author_url": "", "post_date": "2020-01-29T15:23:56.983000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 751327, "author_name": "", "author_url": "", "post_date": "2020-02-20T05:58:04.840000", "content": "", "votes": 0, "replies": [] }, { "id": 732707, "author_name": "", "author_url": "", "post_date": "2020-01-30T06:11:06.540000", "content": "", "votes": 1, "replies": [] }, { "id": 731868, "author_name": "", "author_url": "", "post_date": "2020-01-29T06:49:21.770000", "content": "", "votes": 0, "replies": [ { "id": 732216, "author_name": "", "author_url": "", "post_date": "2020-01-29T15:24:12.420000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 736519, "author_name": "", "author_url": "", "post_date": "2020-02-04T09:43:54.930000", "content": "", "votes": 1, "replies": [] }, { "id": 733059, "author_name": "", "author_url": "", "post_date": "2020-01-30T15:44:42.570000", "content": "", "votes": 1, "replies": [] }, { "id": 731859, "author_name": "", "author_url": "", "post_date": "2020-01-29T06:27:23.977000", "content": "", "votes": 1, "replies": [] } ], "comment_count": 58, "messages": [ { "id": 730414, "postDate": "2020-01-27T13:19:21.023Z", "content": "I'm sorry. It was a traditional Korean holidays until today, so I didn't have time to write this. Thank you for your patience.
\n\nFirst of all, I would like to thank Booz Allen Hamilton for hosting this interesting competition. And congratulates to the all participants and especially the winners!
\n\nI like deep neural networks so I prefer to solve all the problems with a deep neural networks. π
\n\nI focus on the structure of the input data rather than understanding the input data. And concentrate on making the model's input by avoiding missing information as much as possible, hoping that the model will do more than I expected. π
\n\nIn other words, I focus less on feature engineering and more on finding a neural net model architecture that fits the data.
The sequence of installation_id is too long to be used as it is. So I aggregated log data (train_df) by game_session. Please see the example below.\n\ndf = train_df\nevent_code = pd.crosstab(df['game_session'], df['event_code'])\nevent_id = pd.crosstab(df['game_session'], df['event_id'])\n...\nagged_df = pd.concat([event_code, event_id, game_accuracy, max_round])\nsession_df = df.drop_duplicates('game_session', keep='last').reset_index(drop=True)\nsession_df = session_df.merge(agged_df, how='left', on='game_session')\n
The LSTM and TRANSFORMER models in NLP receive sequence of words (or sentence) as input. Similarly, I will use the sequence of game_sessions (or installation_id) as input here.
\n\nBest private score: 0.564\nSingle transformer model used.
\n\n
Prediction from game_sessions of an installation_id
\n\nCategorical columns (such as event_code, title, world, etc...) were embedded respectively. Then, the categorical_vector were obtained by concatenating the embeddings. Next the nn.linear layer is applied for the dimension reduction of the categorical vector.\n\nself.categorical_proj = nn.Sequential(\n nn.Linear(cfg.emb_size*num_categorical_columns, cfg.hidden_size//2),\n nn.LayerNorm(cfg.hidden_size//2),\n )
\n
Continuous columns were embedded directly using a linear layer.\n\nself.continuous_emb = nn.Sequential( \n* I used np.log1p for normalization of continuous columns.
\n nn.Linear(num_continuous_columns, cfg.hidden_size//2),\n nn.LayerNorm(cfg.hidden_size//2),\n )\n
https://www.kaggle.com/c/data-science-bowl-2019/discussion/124836\nAs mentioned in this link, the 0 and 3 classes of the accuracy_group may be very close. \nnum_correct can have 0 or 1, if the num_correct has 1 then the accuracy_group increases 3 points.\nOn the other hand, num_incorrect decreases 1 point when num_incorrect has 1 and decreases 2 points when num_incorrect has 2 or more.
\n\nThis could be expressed as \n```
\n\nnew_accuracy_group = 3 * num_correct - num_incorrect\n```
\n\nUsing the above equation, we can calculate the real values of 0 to 3 from num_correct, num_incorrect.\nTherefore, the prediction of the model is set to [num_correct_pred, num_incorrect_pred]
\n\n```\nprediction = model(x) # prediction = [num_correct_pred, num_incorrect_pred]
\n\n\n**Then train the model with the modified_loss below.**\n\nmodified_loss = mse_loss( prediction, target )\n```
After the training is done, we can use the new_accuracy_group calculated from \"num_correct_pred, num_incorrect_pred\".\n\nnum_correct_pred, num_incorrect_pred = prediction\nnew_accuracy_group = 3 * num_correct_pred - num_incorrect_pred\n
We can also use original accuracy_group to slightly improve performance.\n```\nprediction = model(x) # prediction = [accuracy_group_pred, num_correct_pred, num_incorrect_pred]
\n\n```
\n\nThe final_accuracy_group is calculated as below.\n\nnew_accuracy_group = 3 * num_correct_pred - num_incorrect_pred\nfinal_accuracy_group = (accuracy_group_pred + new_accuracy_group) / 2\n
I generated an additional label for game_sessions, where the type is Game. From the \"correct\":true, βcorrectβ:false of event_data, I was able to create num_correct and num_incorrect, and likewise I was able to create an accuracy_group.\nThe number of additional training samples generated is 41,194.
\n\nPre-training and fine-tuning steps\n\nhttps://www.kaggle.com/limerobot/dsb2019-v77-tr-dt-aug0-5-3tta?scriptVersionId=27448615
\n\nIt is a shame to me sharing the uncleane code. But first I decided to share the kernel and make a clean code. Maybe in two weeks? ;)
\n\nIt is a shame to me sharing the uncleane code. \nBut first I decided to share the kernel and make a clean code. Maybe in two weeks?\nhttps://www.kaggle.com/limerobot/dsb2019-v77-tr-dt-aug0-5-3tta?scriptVersionId=27448615
", "rawMarkdown": "It is a shame to me sharing the uncleane code. \nBut first I decided to share the kernel and make a clean code. Maybe in two weeks?\nhttps://www.kaggle.com/limerobot/dsb2019-v77-tr-dt-aug0-5-3tta?scriptVersionId=27448615", "votes": 5, "replies": [ { "id": 731937, "postDate": "2020-01-29T09:17:52.630Z", "content": "Thank you so much. Iβm sorry for creating the extra work.
", "rawMarkdown": "Thank you so much. Iβm sorry for creating the extra work.", "votes": 1 }, { "id": 732212, "postDate": "2020-01-29T15:22:09.887Z", "content": "It's ok π
", "rawMarkdown": "It's ok π ", "votes": 1 } ] }, { "id": 732690, "postDate": "2020-01-30T05:39:07.343Z", "content": "Great job. Thanks for sharing!
", "rawMarkdown": "Great job. Thanks for sharing!", "votes": 2, "replies": [ { "id": 732692, "postDate": "2020-01-30T05:42:49.400Z", "content": "*YEAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!πͺ π π *
", "rawMarkdown": "*YEAAAAAAAAAAAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!**πͺ π π ***" } ] }, { "id": 731558, "postDate": "2020-01-28T19:33:18.833Z", "content": "@limerobot, congratulations and thanks for the interesting writeup! This was my first Kaggle competition and I invested a ton of time on it. Very frustrating that we had kernels that would have been solid silver but we ended up just missing Bronze. I had an idea for and LSTM using train summarized by session but we didnt quite get it working. I'd really like to understand all the details of how you manipulated the data and built your model. Might you share the code?
", "rawMarkdown": "@limerobot, congratulations and thanks for the interesting writeup! This was my first Kaggle competition and I invested a ton of time on it. Very frustrating that we had kernels that would have been solid silver but we ended up just missing Bronze. I had an idea for and LSTM using train summarized by session but we didnt quite get it working. I'd really like to understand all the details of how you manipulated the data and built your model. Might you share the code?", "votes": 2, "replies": [ { "id": 731727, "postDate": "2020-01-29T01:04:35.370Z", "content": "Please check your e-mail π
", "rawMarkdown": "Please check your e-mail π", "votes": 2 }, { "id": 731840, "postDate": "2020-01-29T06:08:36.617Z", "content": "Many thanks! So much to learn!!! π
", "rawMarkdown": "Many thanks! So much to learn!!! π " } ] }, { "id": 731500, "postDate": "2020-01-28T17:56:49.867Z", "content": "@limerobot Great Solution and Congrats on winning. Can you share your kernel please? I want to give a try to your approach and expecting a lot to learn from your solution.
", "rawMarkdown": "@limerobot Great Solution and Congrats on winning. Can you share your kernel please? I want to give a try to your approach and expecting a lot to learn from your solution.", "votes": 2, "replies": [ { "id": 731728, "postDate": "2020-01-29T01:04:42.010Z", "content": "Please check your e-mail π
", "rawMarkdown": "Please check your e-mail π", "votes": 3 }, { "id": 731961, "postDate": "2020-01-29T09:53:45.893Z", "content": "@limerobot Could you share your kernel with me too ? I want to compare my model with yours.
\n\nEDIT: Sorry, I've missed the link. Thank you for sharing !!
", "rawMarkdown": "@limerobot Could you share your kernel with me too ? I want to compare my model with yours.\n\nEDIT: Sorry, I've missed the link. Thank you for sharing !!", "votes": 1 }, { "id": 731998, "postDate": "2020-01-29T11:10:31.620Z", "content": "Thanks @limerobot
", "rawMarkdown": "Thanks @limerobot " } ] }, { "id": 1058685, "postDate": "2020-10-24T05:20:21.173Z", "content": "Could you please share your kernel..
", "rawMarkdown": "Could you please share your kernel..", "votes": 1 }, { "id": 754013, "postDate": "2020-02-22T23:23:08.630Z", "content": "Great solution, and thanks for sharing the code and write-up.
\n\nI'm just curious about one thing. The goal of the competition was to \"Uncover the factors to help measure how young children learn\". With something like a decision tree we can quite easily have an idea about the most important factors, but with a Transformer model (or nn in general) that seems to be a lot harder.
\n\nHow do you think your model could be used to identify the best learning patterns in children? Do you think it's better to have a less accurate but more readable model for the use case of the competition organisers?
\n\nWould be great to hear your thoughts!
", "rawMarkdown": "Great solution, and thanks for sharing the code and write-up. \n\nI'm just curious about one thing. The goal of the competition was to \"Uncover the factors to help measure how young children learn\". With something like a decision tree we can quite easily have an idea about the most important factors, but with a Transformer model (or nn in general) that seems to be a lot harder. \n\nHow do you think your model could be used to identify the best learning patterns in children? Do you think it's better to have a less accurate but more readable model for the use case of the competition organisers?\n\nWould be great to hear your thoughts!", "votes": 1, "replies": [ { "id": 772339, "postDate": "2020-03-15T11:28:33.580Z", "content": "@jyesawtellrickson I'm really sorry for late reply. Please consider using notifications(@) to me. π±
\n\nI think the results of various models will help competition organisers.
\n\nFor example, the fact that performance has decreased when sequential information is used can help clarify the relationship between random path and linear progression.\nBelow is what competition organiser mentioned.
\n\nhttps://www.kaggle.com/c/data-science-bowl-2019/discussion/115034
\n\n\n", "rawMarkdown": "@jyesawtellrickson I'm really sorry for late reply. Please consider using notifications(@) to me. π± \n\nI think the results of various models will help competition organisers.\n\nFor example, the fact that performance has decreased when sequential information is used can help clarify the relationship between random path and linear progression.\nBelow is what competition organiser mentioned.\n\nhttps://www.kaggle.com/c/data-science-bowl-2019/discussion/115034\n\n> The app is designed to try to guide the kids through an idealized learning path, which is intended to present players with a pattern of exposure->exploration->practice->demonstration (as in demonstration of knowledge). Each of the worlds in the app may have one or more such sequences of media objects, and sometimes the app does not follow this exact formula. \n\n> `However, kids are not required to follow the path that is laid out for them, and whether the suggested linear progression leads to better learning outcomes than a random path is not yet clear.` Perhaps this competition will give us some insights into this question as well!\n\n", "votes": 1 } ] }, { "id": 741660, "postDate": "2020-02-10T21:34:51.520Z", "content": "The app is designed to try to guide the kids through an idealized learning path, which is intended to present players with a pattern of exposure->exploration->practice->demonstration (as in demonstration of knowledge). Each of the worlds in the app may have one or more such sequences of media objects, and sometimes the app does not follow this exact formula.
\n \n\n
However, kids are not required to follow the path that is laid out for them, and whether the suggested linear progression leads to better learning outcomes than a random path is not yet clear.Perhaps this competition will give us some insights into this question as well!
So much great ideas. And I should get better at applying transformers. Thanks for sharing the insights and the kernel!
", "rawMarkdown": "So much great ideas. And I should get better at applying transformers. Thanks for sharing the insights and the kernel! ", "votes": 1 }, { "id": 737144, "postDate": "2020-02-05T00:48:49.720Z", "content": "Congratz and thanks for the big sharing! It is super helpful for learning! π π€
", "rawMarkdown": "Congratz and thanks for the big sharing! It is super helpful for learning! π π€ ", "votes": 1 }, { "id": 733698, "postDate": "2020-01-31T12:47:46.153Z", "content": "Nice work
", "rawMarkdown": "Nice work", "votes": 1 }, { "id": 733639, "postDate": "2020-01-31T11:30:18.540Z", "content": "what is transformer? i dont know that. can you describe some?
", "rawMarkdown": "what is transformer? i dont know that. can you describe some? ", "votes": 1, "replies": [ { "id": 772342, "postDate": "2020-03-15T11:33:21.797Z", "content": "Please refer to the \"Attention Is All You Need\" paper or a well-organized blog on the Internet. π \nI think that's a better way.
", "rawMarkdown": "Please refer to the \"Attention Is All You Need\" paper or a well-organized blog on the Internet. π \nI think that's a better way." } ] }, { "id": 733511, "postDate": "2020-01-31T08:35:49.117Z", "content": "very impressive!!
", "rawMarkdown": "very impressive!!", "votes": 1 }, { "id": 733462, "postDate": "2020-01-31T06:47:32.763Z", "content": "Great Job Buddy
", "rawMarkdown": "Great Job Buddy ", "votes": 1 }, { "id": 733317, "postDate": "2020-01-31T00:28:39.570Z", "content": "very interesting kernel.
", "rawMarkdown": "very interesting kernel.", "votes": 1 }, { "id": 733138, "postDate": "2020-01-30T18:09:16.540Z", "content": "Congrats Limerobot. Very good impressive work with a single model. Appreciate if you can also point us to your training kernel and code. Many thanks.
", "rawMarkdown": "Congrats Limerobot. Very good impressive work with a single model. Appreciate if you can also point us to your training kernel and code. Many thanks.", "votes": 1 }, { "id": 732670, "postDate": "2020-01-30T04:42:15.660Z", "content": "it's very interesting and great kernal its very useful for learning purpose. thanks for sharing
", "rawMarkdown": "it's very interesting and great kernal its very useful for learning purpose. thanks for sharing ", "votes": 1 }, { "id": 732566, "postDate": "2020-01-30T00:13:03.790Z", "content": "Interesting approach. Congrats!
", "rawMarkdown": "Interesting approach. Congrats!", "votes": 1 }, { "id": 732295, "postDate": "2020-01-29T16:39:04.940Z", "content": "Congrats & Thanks for sharing!!π π π
", "rawMarkdown": "Congrats & Thanks for sharing!!π π π ", "votes": 1 }, { "id": 732232, "postDate": "2020-01-29T15:43:13.933Z", "content": "Great job. Really helpful. Thanks for sharing.
", "rawMarkdown": "Great job. Really helpful. Thanks for sharing.", "votes": 1 }, { "id": 732222, "postDate": "2020-01-29T15:32:23.920Z", "content": "It's a really beautiful write-up. Thank you for sharing !
", "rawMarkdown": "It's a really beautiful write-up. Thank you for sharing !", "votes": 1 }, { "id": 732009, "postDate": "2020-01-29T11:30:20.637Z", "content": "very helpful :)
", "rawMarkdown": "very helpful :)", "votes": 1 }, { "id": 731989, "postDate": "2020-01-29T10:43:42.053Z", "content": "This is a very good explanation, looking forward to the clean code release!
", "rawMarkdown": "This is a very good explanation, looking forward to the clean code release!", "votes": 1 }, { "id": 731979, "postDate": "2020-01-29T10:26:49.573Z", "content": "Dang, 3d place with just 1 model and relatively little FE? Neat! \n(It's interesting to see how Attention models worked well for some and not others. How much worse was your model with the PE?).
", "rawMarkdown": "Dang, 3d place with just 1 model and relatively little FE? Neat! \n(It's interesting to see how Attention models worked well for some and not others. How much worse was your model with the PE?).", "votes": 1, "replies": [ { "id": 732219, "postDate": "2020-01-29T15:29:18.027Z", "content": "I remember with PE decreased the local CV score about 0.02~0.03
", "rawMarkdown": "I remember with PE decreased the local CV score about 0.02~0.03" } ] }, { "id": 731909, "postDate": "2020-01-29T08:14:36.767Z", "content": "Congrats on winning the competition and thank you for sharing the approach!! \nCould you share your code please?
", "rawMarkdown": "Congrats on winning the competition and thank you for sharing the approach!! \nCould you share your code please? ", "votes": 1, "replies": [ { "id": 732217, "postDate": "2020-01-29T15:24:24.443Z", "content": "Please see the link above π
", "rawMarkdown": "Please see the link above π " } ] }, { "id": 731028, "postDate": "2020-01-28T09:18:34.020Z", "content": "Thank you for sharing !\nHow did you aggregate the output of Transformer ? Are the numbers of game_session different for each installation_id ?
Yep, installation_id is a variable-length sequence.\nSo I set the sequence length to 100 and I added PAD tokens to the front in the case of a short sequence. Then like the figure above, I took the last output of the sequence.
", "rawMarkdown": "Yep, installation_id is a variable-length sequence.\nSo I set the sequence length to 100 and I added PAD tokens to the front in the case of a short sequence. Then like the figure above, I took the last output of the sequence.", "votes": 1 } ] }, { "id": 730885, "postDate": "2020-01-28T04:03:51.453Z", "content": "Very impressive!!! Thanks for sharing your solution.
", "rawMarkdown": "Very impressive!!! Thanks for sharing your solution.", "votes": 1 }, { "id": 730884, "postDate": "2020-01-28T03:59:26.780Z", "content": "Thank you for sharing! I should learn this! ;-) Congratulations!
", "rawMarkdown": "Thank you for sharing! I should learn this! ;-) Congratulations!", "votes": 1 }, { "id": 730835, "postDate": "2020-01-28T02:01:00.640Z", "content": "@limerobot Great solution! May I ask would you share your kernel? I want to try to blend your method with mine and see the effect.
", "rawMarkdown": "@limerobot Great solution! May I ask would you share your kernel? I want to try to blend your method with mine and see the effect. ", "votes": 1, "replies": [ { "id": 731143, "postDate": "2020-01-28T11:55:20.873Z", "content": "Please check your e-mail π
", "rawMarkdown": "Please check your e-mail π ", "votes": 1 }, { "id": 731851, "postDate": "2020-01-29T06:16:59.333Z", "content": "Thanks. I will do that in 2 weeks and let you know the result.
", "rawMarkdown": "Thanks. I will do that in 2 weeks and let you know the result. ", "votes": 1 } ] }, { "id": 730825, "postDate": "2020-01-28T01:39:23.410Z", "content": "Amazing work. Congrats on 3rd place and GM status - you earned it.
", "rawMarkdown": "Amazing work. Congrats on 3rd place and GM status - you earned it. ", "votes": 1 }, { "id": 730578, "postDate": "2020-01-27T16:44:44.783Z", "content": "Great job. I was curious about your solution. Thanks for sharing.
", "rawMarkdown": "Great job. I was curious about your solution. Thanks for sharing.", "votes": 1 }, { "id": 730430, "postDate": "2020-01-27T13:30:50.150Z", "content": "Congrats! Very impressive job. Curious how many heads did the Transformer layers have?\nI also found that adding positional encodings made the transformer worse. I did add in game time and seconds elapsed (from first event to last event for installation_id) though so it had some sense of timing. Also curious exactly what features you used? Thanks.
", "rawMarkdown": "Congrats! Very impressive job. Curious how many heads did the Transformer layers have?\nI also found that adding positional encodings made the transformer worse. I did add in game time and seconds elapsed (from first event to last event for installation_id) though so it had some sense of timing. Also curious exactly what features you used? Thanks.", "votes": 1, "replies": [ { "id": 730814, "postDate": "2020-01-28T00:35:43.240Z", "content": "Only one head was used.\n\ndef get_reg():\n return nn.Sequential(\n nn.Linear(cfg.hidden_size, cfg.hidden_size),\n nn.LayerNorm(cfg.hidden_size),\n nn.Dropout(cfg.dropout),\n nn.ReLU(),\n nn.Linear(cfg.hidden_size, cfg.hidden_size),\n nn.LayerNorm(cfg.hidden_size),\n nn.Dropout(cfg.dropout),\n nn.ReLU(),\n nn.Linear(cfg.hidden_size, cfg.target_size), \nI didn't consider
\n )
\nself.reg_layer = get_reg()\nseconds elapsed π ; it looks like a good feature.
The features I used are \nCategorical columns = ['title', 'type', 'world']\nContinuous columns = ['event_count', 'game_time', 'max_game_time']
\n\nThe rest columns you can see the agged_df I mentioned above. π
", "rawMarkdown": "Only one head was used.\n```\ndef get_reg():\n return nn.Sequential(\n nn.Linear(cfg.hidden_size, cfg.hidden_size),\n nn.LayerNorm(cfg.hidden_size),\n nn.Dropout(cfg.dropout),\n nn.ReLU(),\n nn.Linear(cfg.hidden_size, cfg.hidden_size),\n nn.LayerNorm(cfg.hidden_size),\n nn.Dropout(cfg.dropout),\n nn.ReLU(),\n nn.Linear(cfg.hidden_size, cfg.target_size), \n ) \nself.reg_layer = get_reg()\n```\nI didn't consider `seconds elapsed` π ; it looks like a good feature.\n\nThe features I used are \nCategorical columns = ['title', 'type', 'world']\nContinuous columns = ['event\\_count', 'game\\_time', 'max\\_game_time']\n\nThe rest columns you can see the agged\\_df I mentioned above. π ", "votes": 2 } ] }, { "id": 3455513, "postDate": "2026-05-09T15:25:48.787Z", "content": "I learned a lot from here. Thanks!
", "rawMarkdown": " I learned a lot from here. Thanks!" }, { "id": 731829, "postDate": "2020-01-29T05:56:25.900Z", "content": "@limerobot Congratulations for this Post and your achievement. Most of the top result posts mostly discussed about Feature Engineering. But your post explained less about feature engineering and more about Model building(NN). Since I started to NN, I would like to know your way of handling the features and how you built the NN Model with Transformers. Please share the Notebook link for my learning.
", "rawMarkdown": "@limerobot Congratulations for this Post and your achievement. Most of the top result posts mostly discussed about Feature Engineering. But your post explained less about feature engineering and more about Model building(NN). Since I started to NN, I would like to know your way of handling the features and how you built the NN Model with Transformers. Please share the Notebook link for my learning.", "replies": [ { "id": 732215, "postDate": "2020-01-29T15:24:05.560Z", "content": "Please see the link above π
", "rawMarkdown": "Please see the link above π " } ] }, { "id": 731801, "postDate": "2020-01-29T04:40:24.670Z", "content": "@limerobot Congrats for winning the prize and thanks for great summary!!!\nI had been competing in this competition for two months. My solution is based on LGB and XGB. I found it is very important for Boosting models to feed features which describe what game did an user played before the assessment (time-series features.) Therefore, if I had have enough experiences, I would have built attention-base model/LSTM. \nDuring competition, you were stepping up in LB day by day even though you joined this comp late (I think you said so in discussion), and I was really waiting your solution. After reading your summary, I am getting really curious about how you implemented in actual coding. If possible, could I read your code? I am summarizing my solution and share it in near future (under construction).
", "rawMarkdown": "@limerobot Congrats for winning the prize and thanks for great summary!!!\nI had been competing in this competition for two months. My solution is based on LGB and XGB. I found it is very important for Boosting models to feed features which describe what game did an user played before the assessment (time-series features.) Therefore, if I had have enough experiences, I would have built attention-base model/LSTM. \nDuring competition, you were stepping up in LB day by day even though you joined this comp late (I think you said so in discussion), and I was really waiting your solution. After reading your summary, I am getting really curious about how you implemented in actual coding. If possible, could I read your code? I am summarizing my solution and share it in near future ([under construction](https://github.com/yuooka/DSB2019-104th)).", "replies": [ { "id": 732214, "postDate": "2020-01-29T15:23:56.983Z", "content": "Please see the link above π
", "rawMarkdown": "Please see the link above π " } ] }, { "id": 751327, "postDate": "2020-02-20T05:58:04.840Z", "rawMarkdown": "", "isDeleted": true }, { "id": 732707, "postDate": "2020-01-30T06:11:06.540Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 731868, "postDate": "2020-01-29T06:49:21.770Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 732216, "postDate": "2020-01-29T15:24:12.420Z", "content": "Please see the link above π
", "rawMarkdown": "Please see the link above π " } ] }, { "id": 736519, "postDate": "2020-02-04T09:43:54.930Z", "content": "Congrats and thanks for sharing!
", "rawMarkdown": "Congrats and thanks for sharing!", "votes": 1 }, { "id": 733059, "postDate": "2020-01-30T15:44:42.570Z", "content": "Really interesting. Thank you !
", "rawMarkdown": "Really interesting. Thank you !", "votes": 1 }, { "id": 731859, "postDate": "2020-01-29T06:27:23.977Z", "content": "Thanks for this blog, Really helpful π
", "rawMarkdown": "Thanks for this blog, Really helpful π ", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-23T23:50:50Z", "competition_id": 16531, "competition_slug": "data-science-bowl-2019", "team_id": 3957064, "solution_url": "/competitions/data-science-bowl-2019/writeups/limerobot-3rd-solution-single-transformer-model-li" }