# 4th place submission

Hi, I achieved 4th place so decided that I will share the details of my solution
1. Model/package
LightGBM, because is powerful and very fast operation (with over 7 million rows it will significantly reduce the learning time)
2. Validation
The rows in the test set look as if they were taken from the same set as the training set, but in such a way that each of the auctions concerns a different device. The time relationship did not seem so obvious to me to try to validate over time (the minimum and maximum days in both sets are the same, so learning from old data and validating on new data does not seem to be an ideal strategy for the leaderboard score at first glance).
I tried 3 types of validation:
a) Random division into training and validation set in proportion (70-80% vs. 30-20%),
b) K-fold cross-validation (k between 5 and 10 due to large number of rows)
c) Creating a validation, but ensuring that each auction concerns a different device (as in the test set).
Only the last method provided private results between the public leaderboard and RMSE on the validation set in A/B tests, however the k-fold cross-validation had the lowest values ​​on both sets, so I decided use this method.
3.  Features
Based on raw data (without the use of additional data sources), many features were created and divided into categories:
a) Time features (based on eventTimestamp)
b) Display type (based on unitDisplayType)
c) Popularity (of device, brand and application)
d) Target encoding (by brand, app, devide, c1, c3)
e) Hardware/software
f) Country
g) Connection type
h) Size of ad
i) Sent price
j) Anonymized features
In total there were 41 features diveded to this 10 categories. Full calculation can be find in
[my notebook](https://www.kaggle.com/code/michau96/bid-price-predicition-4th-place-submission)
.
4. Data leaks
Creating features based on the whole "train" file (eg target encoding) from which we then create the training and validation set if cross validation, it will cause that we are dealing with data leaks. There are at least 2 leaks here:
a) To calculate aggregation features (means, medians), the values are also calculated based on validation set, which means that the rows on which the model will validate are counted for aggregating,
b) We ignore time to calculate all aggregates, so future values are also taken into account, although it would be impossible during the production operation of the model.
It was decided to leave these leaks due to:
a) A / B tests have shown that the model behave better with leaks, both on the validation set (which is obvious), but also on the test set (lower RMSE on the public leaderboard),
b) The test set looks like is not divided over time (the same ranges as in the train set), so the aspect of data leakage with time is not crutial for test set results.
Of course, leaving the leaks is a conceptual error, but solving the problem is relatively time-consuming and should not bring such huge changes in the result on the leaderboard in relation to the time spent on it. This does not change the fact that a model trained in this way should not be used thoughtlessly in production.
5. Features selection
The features were selected for the model based on feature importance from lightGBM. Those with zero or very negligible frequency of selecting for tree node were removed from the set. On this basis, around 30 out of 41 features were included in the final model.
6. Hyperparameters
No hyperparameter optimization package was used. Simple A / B were performed where changes in the value of a single parameter were tested. The final set of the most important hyperparameters looks like this:
a) boosting_type: gdbt (better than dart in A / B test)
b) learning_rate: 0.1
c) colsample_bytree: 1 (having a relatively large number of features, different values were tried such as 0.2, 0.6 to create less correlated trees, but the best results were obtained when we take all the features into account when partitioning by node)
d) bagging_fraction: 0.6 (the results for different values between 0.2 and 1 gave very similar results, so the middle option was chosen)
e) max_depth: 12 (as the parameter increases, the RMSE on the validation set decreases, but the RMSE in test set on public leaderboard remains almost unchanged. The middle option was also chosen for fear of overfit when the trees are very complex)
f) n_estimators: 3000 (so that learning does not end too quickly with early stopping)
7. Learning
The model was learned using early stopping set to 40 iterations (if the RMSE on the validation set does not decrease for 40 iterations in a row, we stop learning). All folds had between 1000 and 3000 iterations (a large number due to the low learning rate and max depth).
With the trained model, predictions were made on the test set and a "clip" was applied to the predicted values ​​(all values ​​below 0.01 were assigned 0.01 due to the fact that it is the minimum possible bet value).
8. Best model
After another small A/B tests, the best model had the following parameters: around 30 feature bringing a variety of information,
7-fold cross-validation (7 models were created and the average winBid value from each model was taken for prediction),
relatively low learning rate and maximum depth, and learning with early stopping, hard rule for prediction of values below 0.01.
The final results of the model:
Runtime: Around 90 minutes on Kaggle enviroment
Validation RMSE (mean fold predictions): 7.13
Public leaderboard RMSE: 15.63 (4th place)
Private leaderboard RMSE: 16.23 (4th place)
9. Next step if I would spend more time
Leaderboard model and score can be improved. The next steps that can be taken with more time are:
a) Use of NN models,
b) Model stacking (GBM + NN),
c) Removal of all leaks in the validation set,
d) Getting to know more about the ad recommendation industry and creating new features based on this knowledge.
e) Deeper recognition of timestamp operation in the training and test set along with an attempt to validate over time.
I put all the code in
[a public notebook](https://www.kaggle.com/code/michau96/bid-price-predicition-4th-place-submission)
. Enjoy!
P.S.
[@wrrosa](https://www.kaggle.com/wrrosa)
Congratulations on the 1st place!