This was a very challenging and fun competition. I want to thank the sponsor, Kaggle, and of course all the amazing competitors!
\n\nI also want to give a shout out to the Kaggle/Google engineers. I've been incredibly impressed with the website/kernels. I am currently a full stack engineer at Amazon so I know how hard it is to pull that off, so thanks for making such an amazing product!
\n\nI did some minor edits to event codes. I broke up 4020, 4010, 4025 into whether correct was true or false, so I ended up with 40201, 40200, 41001, 41000, 40251, 40250.
\n\nI transformed the data into \"histories\". Basically a history is all of the data leading up to the target assessment. I then processed these histories into a large numpy array.
\n\nDue to some histories being enormous due to shared devices I decided to take the last X game sessions per target assessment and also the last Y events per game session. This made sense to me as the recent data should be more important and it is an easy way to deal with shared devices.
\n\nI found that x = 80 and y = 100 gave the best results, so I ended up with a sparse np array:\n(# histories, 80, 100, # features)
\n\nI added \"blank\" categories to the title, event_code, and accuracy group embeddings. This informed the model that these did not exist. (History was shorter then 100 events or shorter than 80 game sessions.) I tried masking the input to the transformer but it killed the performance and the score decreased.
\n\nYou can see the features below in the model diagram. The Assess Target Title and Assess Target Time are fed into every event. I did try inputting these once at the end of the model but the performance got slightly worse. I also tried inserting game session features into the game session embedding but none of the features I tried helped.
\n\nNear the end of the competition I played around with adding in the OOF models' predictions and the models' prediction groups. This seemed to help a lot on Local CV but not as much on Test. I think perhaps I was doing something wrong with how I was then creating these values for the Test Assessments.
\n\nMy original idea was to use a double transformer network. One transformer for the events of a game session and then use those outputs to have one transformer take in each game session embedding. This did work but I discovered it was better (and much faster) with the first transformer being a \"Zero Head\", which means I just removed the attention part and left in the shared FC layers:
\n\n\n\n\nevents2 = self.linear2(self.dropout1(F.relu(self.linear1(events))))\n events = events + self.dropout2(events2)\n events = self.norm2(events)
\n
One key idea came from NLP where they tie the word embeddings in the input and in the output to improve generalization. I did this with the accuracy group and saw a nice bump in QWK.
\n\n.png?generation=1579876333416350&alt=media)
This was one of the more interesting ideas that I tried. If you consider the fact that 3-5 year olds are incredibly noisy in general you can view the labels as being fairly noisy. I read a bunch of papers on dealing with noisy labels and they all basically dealt with the model (or a second model) learning the noise. I decided to save the OOF predictions and then blend these with the actual targets when training new models hoping the single model would be able to learn about some of the noise patterns. Turned out this was very difficult to tune correctly as it seem to leak into my 5 fold CV and also made the train loss hard to reason about. Basically it made it very easy to overfit to the train data.
\n\nI'm also not sure if the way I did the OOF predictions was the best. I would save it after every 5 fold run and then would just average the predictions from all of the past. I think now that this may have increased the confirmation bias and I would have better off with just getting some initial predictions from models that were not blended with new targets and sticking with those.
\n\nIn the end it did boost my private test score from .550 to .554. My best blend was 85 epochs starting from all actual targets to 50/50 at epoch 50 and then increasing back in the actual targets. The blend was linear changing by 1% every epoch so old/new: 1/0 -> .5/.5 -> .85/.15
\n\nHere was the main paper that I got this idea from. It's on pseudo labeling but I think it applies just as well to dealing with noisy labels. I had mixup augmentation on my todo list as I think it would have greatly enhanced the Noisy label technique, but I never got to try it.
\n\n\n\n\nPseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning\n Eric Arazo, Diego Ortego, Paul Albert, Noel E. O’Connor & Kevin McGuinness\n https://arxiv.org/pdf/1908.02983.pdf
\n
I'm working on cleaning up the code and hope to make it public soon. Thanks for reading!
", "comments": [ { "id": 728340, "author_name": "Psi", "author_url": "", "post_date": "2020-01-24T16:22:31.383000", "content": "Nice work. Did you by any chance try to take the first k sessions instead of the last k sessions? When I was playing with NNs I interestingly got quite better results by padding in the end instead of beginning.
", "votes": 1, "replies": [ { "id": 728374, "author_name": "Zac Warren", "author_url": "", "post_date": "2020-01-24T16:48:25.823000", "content": "Interesting. I never tried it. I figured it would be a mess for the installation_ids that had 1000s of game sessions. At one point I did try one single transformer on the last 2500 events. It was surprisingly good but not as good so I dropped it.
\n\nAlso congrats again on the NFL solution! Your solution definitely helped me here. I think the \"Zero Head\" transformer is almost identical to a 1x1 convolution.
", "votes": 0, "replies": [] }, { "id": 728414, "author_name": "Psi", "author_url": "", "post_date": "2020-01-24T17:37:09.747000", "content": "Thanks! A competing hypothesis would be that the behavior in the beginning of using the app has more impact on the outcome than the latest behavior. You can try it if you have time, I would be curious.
", "votes": 0, "replies": [] }, { "id": 728640, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2020-01-25T02:08:41.537000", "content": "I also spent some time considering first vs last activities. I came to the following conclusions:\n- Last sessions should be most important since they carry most recent, and presumably most relevant info\n- However, first sessions show how quickly you learn, unbiased by the amount of your experience, so they measure more your general intelligence. Which could also be an important feature
\n\nIn the end, my lstm nn was looking at 50 sessions, so for the majority of sessions I captured both first and last sessions.
", "votes": 1, "replies": [] } ] }, { "id": 728769, "author_name": "LeiZhou", "author_url": "", "post_date": "2020-01-25T08:24:06.283000", "content": "congrats! elegant solution!
", "votes": 0, "replies": [] }, { "id": 728458, "author_name": "Fernando Pereira", "author_url": "", "post_date": "2020-01-24T18:37:42.257000", "content": "Cool write-up. I am looking forward to check your code architecture!
", "votes": 0, "replies": [] }, { "id": 728335, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2020-01-24T16:16:30.490000", "content": "Congratulations :) its awesome to see Transformers in this competition 👍
", "votes": 0, "replies": [] }, { "id": 728354, "author_name": "", "author_url": "", "post_date": "2020-01-24T16:32:05.803000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 8, "messages": [ { "id": 728328, "postDate": "2020-01-24T16:05:38.687Z", "content": "This was a very challenging and fun competition. I want to thank the sponsor, Kaggle, and of course all the amazing competitors!
\n\nI also want to give a shout out to the Kaggle/Google engineers. I've been incredibly impressed with the website/kernels. I am currently a full stack engineer at Amazon so I know how hard it is to pull that off, so thanks for making such an amazing product!
\n\nI did some minor edits to event codes. I broke up 4020, 4010, 4025 into whether correct was true or false, so I ended up with 40201, 40200, 41001, 41000, 40251, 40250.
\n\nI transformed the data into \"histories\". Basically a history is all of the data leading up to the target assessment. I then processed these histories into a large numpy array.
\n\nDue to some histories being enormous due to shared devices I decided to take the last X game sessions per target assessment and also the last Y events per game session. This made sense to me as the recent data should be more important and it is an easy way to deal with shared devices.
\n\nI found that x = 80 and y = 100 gave the best results, so I ended up with a sparse np array:\n(# histories, 80, 100, # features)
\n\nI added \"blank\" categories to the title, event_code, and accuracy group embeddings. This informed the model that these did not exist. (History was shorter then 100 events or shorter than 80 game sessions.) I tried masking the input to the transformer but it killed the performance and the score decreased.
\n\nYou can see the features below in the model diagram. The Assess Target Title and Assess Target Time are fed into every event. I did try inputting these once at the end of the model but the performance got slightly worse. I also tried inserting game session features into the game session embedding but none of the features I tried helped.
\n\nNear the end of the competition I played around with adding in the OOF models' predictions and the models' prediction groups. This seemed to help a lot on Local CV but not as much on Test. I think perhaps I was doing something wrong with how I was then creating these values for the Test Assessments.
\n\nMy original idea was to use a double transformer network. One transformer for the events of a game session and then use those outputs to have one transformer take in each game session embedding. This did work but I discovered it was better (and much faster) with the first transformer being a \"Zero Head\", which means I just removed the attention part and left in the shared FC layers:
\n\n\n\n\nevents2 = self.linear2(self.dropout1(F.relu(self.linear1(events))))\n events = events + self.dropout2(events2)\n events = self.norm2(events)
\n
One key idea came from NLP where they tie the word embeddings in the input and in the output to improve generalization. I did this with the accuracy group and saw a nice bump in QWK.
\n\n.png?generation=1579876333416350&alt=media)
This was one of the more interesting ideas that I tried. If you consider the fact that 3-5 year olds are incredibly noisy in general you can view the labels as being fairly noisy. I read a bunch of papers on dealing with noisy labels and they all basically dealt with the model (or a second model) learning the noise. I decided to save the OOF predictions and then blend these with the actual targets when training new models hoping the single model would be able to learn about some of the noise patterns. Turned out this was very difficult to tune correctly as it seem to leak into my 5 fold CV and also made the train loss hard to reason about. Basically it made it very easy to overfit to the train data.
\n\nI'm also not sure if the way I did the OOF predictions was the best. I would save it after every 5 fold run and then would just average the predictions from all of the past. I think now that this may have increased the confirmation bias and I would have better off with just getting some initial predictions from models that were not blended with new targets and sticking with those.
\n\nIn the end it did boost my private test score from .550 to .554. My best blend was 85 epochs starting from all actual targets to 50/50 at epoch 50 and then increasing back in the actual targets. The blend was linear changing by 1% every epoch so old/new: 1/0 -> .5/.5 -> .85/.15
\n\nHere was the main paper that I got this idea from. It's on pseudo labeling but I think it applies just as well to dealing with noisy labels. I had mixup augmentation on my todo list as I think it would have greatly enhanced the Noisy label technique, but I never got to try it.
\n\n\n\n\nPseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning\n Eric Arazo, Diego Ortego, Paul Albert, Noel E. O’Connor & Kevin McGuinness\n https://arxiv.org/pdf/1908.02983.pdf
\n
I'm working on cleaning up the code and hope to make it public soon. Thanks for reading!
", "rawMarkdown": "This was a very challenging and fun competition. I want to thank the sponsor, Kaggle, and of course all the amazing competitors!\n\nI also want to give a shout out to the Kaggle/Google engineers. I've been incredibly impressed with the website/kernels. I am currently a full stack engineer at Amazon so I know how hard it is to pull that off, so thanks for making such an amazing product!\n\n## Data Preparation\nI did some minor edits to event codes. I broke up 4020, 4010, 4025 into whether correct was true or false, so I ended up with 40201, 40200, 41001, 41000, 40251, 40250.\n\nI transformed the data into \"histories\". Basically a history is all of the data leading up to the target assessment. I then processed these histories into a large numpy array.\n\nDue to some histories being enormous due to shared devices I decided to take the last X game sessions per target assessment and also the last Y events per game session. This made sense to me as the recent data should be more important and it is an easy way to deal with shared devices.\n\nI found that x = 80 and y = 100 gave the best results, so I ended up with a sparse np array:\n(# histories, 80, 100, # features)\n\nI added \"blank\" categories to the title, event_code, and accuracy group embeddings. This informed the model that these did not exist. (History was shorter then 100 events or shorter than 80 game sessions.) I tried masking the input to the transformer but it killed the performance and the score decreased.\n\n## Features\nYou can see the features below in the model diagram. The Assess Target Title and Assess Target Time are fed into every event. I did try inputting these once at the end of the model but the performance got slightly worse. I also tried inserting game session features into the game session embedding but none of the features I tried helped.\n\nNear the end of the competition I played around with adding in the OOF models' predictions and the models' prediction groups. This seemed to help a lot on Local CV but not as much on Test. I think perhaps I was doing something wrong with how I was then creating these values for the Test Assessments.\n\n## Model\nMy original idea was to use a double transformer network. One transformer for the events of a game session and then use those outputs to have one transformer take in each game session embedding. This did work but I discovered it was better (and much faster) with the first transformer being a \"Zero Head\", which means I just removed the attention part and left in the shared FC layers:\n\n >events2 = self.linear2(self.dropout1(F.relu(self.linear1(events))))\n >events = events + self.dropout2(events2)\n >events = self.norm2(events)\n\nOne key idea came from NLP where they tie the word embeddings in the input and in the output to improve generalization. I did this with the accuracy group and saw a nice bump in QWK.\n\n.png?generation=1579876333416350&alt=media)\n\n## Noisy Labels\nThis was one of the more interesting ideas that I tried. If you consider the fact that 3-5 year olds are incredibly noisy in general you can view the labels as being fairly noisy. I read a bunch of papers on dealing with noisy labels and they all basically dealt with the model (or a second model) learning the noise. I decided to save the OOF predictions and then blend these with the actual targets when training new models hoping the single model would be able to learn about some of the noise patterns. Turned out this was very difficult to tune correctly as it seem to leak into my 5 fold CV and also made the train loss hard to reason about. Basically it made it very easy to overfit to the train data.\n\nI'm also not sure if the way I did the OOF predictions was the best. I would save it after every 5 fold run and then would just average the predictions from all of the past. I think now that this may have increased the confirmation bias and I would have better off with just getting some initial predictions from models that were not blended with new targets and sticking with those.\n\nIn the end it did boost my private test score from .550 to .554. My best blend was 85 epochs starting from all actual targets to 50/50 at epoch 50 and then increasing back in the actual targets. The blend was linear changing by 1% every epoch so old/new: 1/0 -> .5/.5 -> .85/.15\n\nHere was the main paper that I got this idea from. It's on pseudo labeling but I think it applies just as well to dealing with noisy labels. I had mixup augmentation on my todo list as I think it would have greatly enhanced the Noisy label technique, but I never got to try it.\n\n> Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning\n> Eric Arazo, Diego Ortego, Paul Albert, Noel E. O’Connor & Kevin McGuinness\n> https://arxiv.org/pdf/1908.02983.pdf\n\n## Code\nI'm working on cleaning up the code and hope to make it public soon. Thanks for reading!", "votes": 33 }, { "id": 728340, "postDate": "2020-01-24T16:22:31.383Z", "content": "Nice work. Did you by any chance try to take the first k sessions instead of the last k sessions? When I was playing with NNs I interestingly got quite better results by padding in the end instead of beginning.
", "rawMarkdown": "Nice work. Did you by any chance try to take the first k sessions instead of the last k sessions? When I was playing with NNs I interestingly got quite better results by padding in the end instead of beginning.", "votes": 1, "replies": [ { "id": 728374, "postDate": "2020-01-24T16:48:25.823Z", "content": "Interesting. I never tried it. I figured it would be a mess for the installation_ids that had 1000s of game sessions. At one point I did try one single transformer on the last 2500 events. It was surprisingly good but not as good so I dropped it.
\n\nAlso congrats again on the NFL solution! Your solution definitely helped me here. I think the \"Zero Head\" transformer is almost identical to a 1x1 convolution.
", "rawMarkdown": "Interesting. I never tried it. I figured it would be a mess for the installation_ids that had 1000s of game sessions. At one point I did try one single transformer on the last 2500 events. It was surprisingly good but not as good so I dropped it.\n\nAlso congrats again on the NFL solution! Your solution definitely helped me here. I think the \"Zero Head\" transformer is almost identical to a 1x1 convolution." }, { "id": 728414, "postDate": "2020-01-24T17:37:09.747Z", "content": "Thanks! A competing hypothesis would be that the behavior in the beginning of using the app has more impact on the outcome than the latest behavior. You can try it if you have time, I would be curious.
", "rawMarkdown": "Thanks! A competing hypothesis would be that the behavior in the beginning of using the app has more impact on the outcome than the latest behavior. You can try it if you have time, I would be curious." }, { "id": 728640, "postDate": "2020-01-25T02:08:41.537Z", "content": "I also spent some time considering first vs last activities. I came to the following conclusions:\n- Last sessions should be most important since they carry most recent, and presumably most relevant info\n- However, first sessions show how quickly you learn, unbiased by the amount of your experience, so they measure more your general intelligence. Which could also be an important feature
\n\nIn the end, my lstm nn was looking at 50 sessions, so for the majority of sessions I captured both first and last sessions.
", "rawMarkdown": "I also spent some time considering first vs last activities. I came to the following conclusions:\n- Last sessions should be most important since they carry most recent, and presumably most relevant info\n- However, first sessions show how quickly you learn, unbiased by the amount of your experience, so they measure more your *general intelligence*. Which could also be an important feature\n\nIn the end, my lstm nn was looking at 50 sessions, so for the majority of sessions I captured both first and last sessions.", "votes": 1 } ] }, { "id": 728769, "postDate": "2020-01-25T08:24:06.283Z", "content": "congrats! elegant solution!
", "rawMarkdown": "congrats! elegant solution!" }, { "id": 728458, "postDate": "2020-01-24T18:37:42.257Z", "content": "Cool write-up. I am looking forward to check your code architecture!
", "rawMarkdown": "Cool write-up. I am looking forward to check your code architecture!" }, { "id": 728335, "postDate": "2020-01-24T16:16:30.490Z", "content": "Congratulations :) its awesome to see Transformers in this competition 👍
", "rawMarkdown": "Congratulations :) its awesome to see Transformers in this competition 👍 " }, { "id": 728354, "postDate": "2020-01-24T16:32:05.803Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-22T10:53:10Z", "competition_id": 16531, "competition_slug": "data-science-bowl-2019", "team_id": 3934208, "solution_url": "/competitions/data-science-bowl-2019/writeups/zac-warren-single-model-1-5-transformers-31st-plac" }