# 4th private (7th public) place writeup, link to code

Many thanks Booz Allen Hamilton for the great competitions and all participants for interesting ideas and discussions!
We jumped from 7 place public LB on 4th place on private. Best submit we selected is 0.561 private, 0.572 on public LB. It was a blend of neural networks. The second one was a 3-level stack 0.560 private LB, 0.566 public LB.
Some ideas first
The test set can be labeled with the same procedure as a train one and can be used in training.
TfIdf on sequence of events. Each event_id can be coded as title + event_code + correct_flag + incorrect_flag. After that we can look at installation_id history as a sequence of tokens and train tfidf on them. For training procedure we apply transformer on actual sequence of tokens before the current assessment.
Some clips and other titles are very important for estimation of accuracy on the assessment. Maybe order of them is not so important, but RNN architecture can handle their presence in a user history good enough.
We have a small amount of data and stability is much more important than the actual score. If changing column order makes score worse than we are doing something wrong.
Models
Neural network
Tfidf features + RNN on title sequence (last 64) with some additional features:
1. Embedding of title dim=7.
2. Number of correct attempts during the title.
3. Number of incorrect attempts during the title.
4. Ratio of 2 and 3.
5. Log of time in seconds between starts of the titles.
6. Number of correct attempts in the previous title like that.
7. Number of incorrect attempts in the previous title like that.
8. Ratio of 7 and 6.
(9) I've tried a lot to include counters as is in the model. Finally, I gave up, but those models peformed better on the private LB.
Most of single networks were on 0.56+ zone. The most interesting one for me has equal public and private scores:
[https://www.kaggle.com/sergeifironov/bowl-stabilize-coefs-cntrs-all5](https://www.kaggle.com/sergeifironov/bowl-stabilize-coefs-cntrs-all5)
Tree based models
Lightgbm,Xgb, Catboost. (will be soon)
Stack
0 level) NN folds in folds model (5 outer folds, 5 inner folds), lgbm, catboost.
1st level) MLP, Lightgbm.
2nd level) Ridge.
[https://www.kaggle.com/c/data-science-bowl-2019/discussion/127312](https://www.kaggle.com/c/data-science-bowl-2019/discussion/127312)
Validation
I wrote a lot in this topic:
[https://www.kaggle.com/c/data-science-bowl-2019/discussion/125001](https://www.kaggle.com/c/data-science-bowl-2019/discussion/125001)
, but near the end of the competition I gave up to make it correlated with public LB and use the very simple one without installation id groups at all.
What doesn’t work for us
Transformers, GPT-2 and BERT vectors trained on predict event_id, title, title+accuracy_group and so on. They are useless.
Graph NN.
Transformer on a sequence of events. It’s too fat for this small amount of data.