Hi, I achieved 4th place so decided that I will share the details of my solution
\n1. Model/package
\nLightGBM, because is powerful and very fast operation (with over 7 million rows it will significantly reduce the learning time)
2. Validation
\nThe rows in the test set look as if they were taken from the same set as the training set, but in such a way that each of the auctions concerns a different device. The time relationship did not seem so obvious to me to try to validate over time (the minimum and maximum days in both sets are the same, so learning from old data and validating on new data does not seem to be an ideal strategy for the leaderboard score at first glance).
I tried 3 types of validation:
\na) Random division into training and validation set in proportion (70-80% vs. 30-20%),
\nb) K-fold cross-validation (k between 5 and 10 due to large number of rows)
\nc) Creating a validation, but ensuring that each auction concerns a different device (as in the test set).
Only the last method provided private results between the public leaderboard and RMSE on the validation set in A/B tests, however the k-fold cross-validation had the lowest values on both sets, so I decided use this method.
\n3. Features
\nBased on raw data (without the use of additional data sources), many features were created and divided into categories:
\na) Time features (based on eventTimestamp)
\nb) Display type (based on unitDisplayType)
\nc) Popularity (of device, brand and application)
\nd) Target encoding (by brand, app, devide, c1, c3)
\ne) Hardware/software
\nf) Country
\ng) Connection type
\nh) Size of ad
\ni) Sent price
\nj) Anonymized features
In total there were 41 features diveded to this 10 categories. Full calculation can be find in my notebook.
\n4. Data leaks
\nCreating features based on the whole \"train\" file (eg target encoding) from which we then create the training and validation set if cross validation, it will cause that we are dealing with data leaks. There are at least 2 leaks here:
\na) To calculate aggregation features (means, medians), the values are also calculated based on validation set, which means that the rows on which the model will validate are counted for aggregating,
\nb) We ignore time to calculate all aggregates, so future values are also taken into account, although it would be impossible during the production operation of the model.
It was decided to leave these leaks due to:
\na) A / B tests have shown that the model behave better with leaks, both on the validation set (which is obvious), but also on the test set (lower RMSE on the public leaderboard),
\nb) The test set looks like is not divided over time (the same ranges as in the train set), so the aspect of data leakage with time is not crutial for test set results.
Of course, leaving the leaks is a conceptual error, but solving the problem is relatively time-consuming and should not bring such huge changes in the result on the leaderboard in relation to the time spent on it. This does not change the fact that a model trained in this way should not be used thoughtlessly in production.
\n5. Features selection
\nThe features were selected for the model based on feature importance from lightGBM. Those with zero or very negligible frequency of selecting for tree node were removed from the set. On this basis, around 30 out of 41 features were included in the final model.
6. Hyperparameters
\nNo hyperparameter optimization package was used. Simple A / B were performed where changes in the value of a single parameter were tested. The final set of the most important hyperparameters looks like this:
\na) boosting_type: gdbt (better than dart in A / B test)
\nb) learning_rate: 0.1
\nc) colsample_bytree: 1 (having a relatively large number of features, different values were tried such as 0.2, 0.6 to create less correlated trees, but the best results were obtained when we take all the features into account when partitioning by node)
\nd) bagging_fraction: 0.6 (the results for different values between 0.2 and 1 gave very similar results, so the middle option was chosen)
\ne) max_depth: 12 (as the parameter increases, the RMSE on the validation set decreases, but the RMSE in test set on public leaderboard remains almost unchanged. The middle option was also chosen for fear of overfit when the trees are very complex)
\nf) n_estimators: 3000 (so that learning does not end too quickly with early stopping)
7. Learning
\nThe model was learned using early stopping set to 40 iterations (if the RMSE on the validation set does not decrease for 40 iterations in a row, we stop learning). All folds had between 1000 and 3000 iterations (a large number due to the low learning rate and max depth).
With the trained model, predictions were made on the test set and a \"clip\" was applied to the predicted values (all values below 0.01 were assigned 0.01 due to the fact that it is the minimum possible bet value).
\n8. Best model
\nAfter another small A/B tests, the best model had the following parameters: around 30 feature bringing a variety of information,
\n7-fold cross-validation (7 models were created and the average winBid value from each model was taken for prediction),
\nrelatively low learning rate and maximum depth, and learning with early stopping, hard rule for prediction of values below 0.01.
The final results of the model:
\nRuntime: Around 90 minutes on Kaggle enviroment
\nValidation RMSE (mean fold predictions): 7.13
\nPublic leaderboard RMSE: 15.63 (4th place)
\nPrivate leaderboard RMSE: 16.23 (4th place)
9. Next step if I would spend more time
\nLeaderboard model and score can be improved. The next steps that can be taken with more time are:
\na) Use of NN models,
\nb) Model stacking (GBM + NN),
\nc) Removal of all leaks in the validation set,
\nd) Getting to know more about the ad recommendation industry and creating new features based on this knowledge.
\ne) Deeper recognition of timestamp operation in the training and test set along with an attempt to validate over time.
I put all the code in a public notebook. Enjoy!
\nP.S. @wrrosa Congratulations on the 1st place!
", "messages": [ { "id": "1969709", "postDate": "10/03/2022 16:35:11", "content": "Hi, I achieved 4th place so decided that I will share the details of my solution
\n1. Model/package
\nLightGBM, because is powerful and very fast operation (with over 7 million rows it will significantly reduce the learning time)
2. Validation
\nThe rows in the test set look as if they were taken from the same set as the training set, but in such a way that each of the auctions concerns a different device. The time relationship did not seem so obvious to me to try to validate over time (the minimum and maximum days in both sets are the same, so learning from old data and validating on new data does not seem to be an ideal strategy for the leaderboard score at first glance).
I tried 3 types of validation:
\na) Random division into training and validation set in proportion (70-80% vs. 30-20%),
\nb) K-fold cross-validation (k between 5 and 10 due to large number of rows)
\nc) Creating a validation, but ensuring that each auction concerns a different device (as in the test set).
Only the last method provided private results between the public leaderboard and RMSE on the validation set in A/B tests, however the k-fold cross-validation had the lowest values on both sets, so I decided use this method.
\n3. Features
\nBased on raw data (without the use of additional data sources), many features were created and divided into categories:
\na) Time features (based on eventTimestamp)
\nb) Display type (based on unitDisplayType)
\nc) Popularity (of device, brand and application)
\nd) Target encoding (by brand, app, devide, c1, c3)
\ne) Hardware/software
\nf) Country
\ng) Connection type
\nh) Size of ad
\ni) Sent price
\nj) Anonymized features
In total there were 41 features diveded to this 10 categories. Full calculation can be find in my notebook.
\n4. Data leaks
\nCreating features based on the whole \"train\" file (eg target encoding) from which we then create the training and validation set if cross validation, it will cause that we are dealing with data leaks. There are at least 2 leaks here:
\na) To calculate aggregation features (means, medians), the values are also calculated based on validation set, which means that the rows on which the model will validate are counted for aggregating,
\nb) We ignore time to calculate all aggregates, so future values are also taken into account, although it would be impossible during the production operation of the model.
It was decided to leave these leaks due to:
\na) A / B tests have shown that the model behave better with leaks, both on the validation set (which is obvious), but also on the test set (lower RMSE on the public leaderboard),
\nb) The test set looks like is not divided over time (the same ranges as in the train set), so the aspect of data leakage with time is not crutial for test set results.
Of course, leaving the leaks is a conceptual error, but solving the problem is relatively time-consuming and should not bring such huge changes in the result on the leaderboard in relation to the time spent on it. This does not change the fact that a model trained in this way should not be used thoughtlessly in production.
\n5. Features selection
\nThe features were selected for the model based on feature importance from lightGBM. Those with zero or very negligible frequency of selecting for tree node were removed from the set. On this basis, around 30 out of 41 features were included in the final model.
6. Hyperparameters
\nNo hyperparameter optimization package was used. Simple A / B were performed where changes in the value of a single parameter were tested. The final set of the most important hyperparameters looks like this:
\na) boosting_type: gdbt (better than dart in A / B test)
\nb) learning_rate: 0.1
\nc) colsample_bytree: 1 (having a relatively large number of features, different values were tried such as 0.2, 0.6 to create less correlated trees, but the best results were obtained when we take all the features into account when partitioning by node)
\nd) bagging_fraction: 0.6 (the results for different values between 0.2 and 1 gave very similar results, so the middle option was chosen)
\ne) max_depth: 12 (as the parameter increases, the RMSE on the validation set decreases, but the RMSE in test set on public leaderboard remains almost unchanged. The middle option was also chosen for fear of overfit when the trees are very complex)
\nf) n_estimators: 3000 (so that learning does not end too quickly with early stopping)
7. Learning
\nThe model was learned using early stopping set to 40 iterations (if the RMSE on the validation set does not decrease for 40 iterations in a row, we stop learning). All folds had between 1000 and 3000 iterations (a large number due to the low learning rate and max depth).
With the trained model, predictions were made on the test set and a \"clip\" was applied to the predicted values (all values below 0.01 were assigned 0.01 due to the fact that it is the minimum possible bet value).
\n8. Best model
\nAfter another small A/B tests, the best model had the following parameters: around 30 feature bringing a variety of information,
\n7-fold cross-validation (7 models were created and the average winBid value from each model was taken for prediction),
\nrelatively low learning rate and maximum depth, and learning with early stopping, hard rule for prediction of values below 0.01.
The final results of the model:
\nRuntime: Around 90 minutes on Kaggle enviroment
\nValidation RMSE (mean fold predictions): 7.13
\nPublic leaderboard RMSE: 15.63 (4th place)
\nPrivate leaderboard RMSE: 16.23 (4th place)
9. Next step if I would spend more time
\nLeaderboard model and score can be improved. The next steps that can be taken with more time are:
\na) Use of NN models,
\nb) Model stacking (GBM + NN),
\nc) Removal of all leaks in the validation set,
\nd) Getting to know more about the ad recommendation industry and creating new features based on this knowledge.
\ne) Deeper recognition of timestamp operation in the training and test set along with an attempt to validate over time.
I put all the code in a public notebook. Enjoy!
\nP.S. @wrrosa Congratulations on the 1st place!
", "rawMarkdown": "Hi, I achieved 4th place so decided that I will share the details of my solution\n\n**1. Model/package**\nLightGBM, because is powerful and very fast operation (with over 7 million rows it will significantly reduce the learning time)\n\n**2. Validation**\nThe rows in the test set look as if they were taken from the same set as the training set, but in such a way that each of the auctions concerns a different device. The time relationship did not seem so obvious to me to try to validate over time (the minimum and maximum days in both sets are the same, so learning from old data and validating on new data does not seem to be an ideal strategy for the leaderboard score at first glance).\n\nI tried 3 types of validation: \na) Random division into training and validation set in proportion (70-80% vs. 30-20%), \nb) K-fold cross-validation (k between 5 and 10 due to large number of rows) \nc) Creating a validation, but ensuring that each auction concerns a different device (as in the test set).\n\nOnly the last method provided private results between the public leaderboard and RMSE on the validation set in A/B tests, however the k-fold cross-validation had the lowest values on both sets, so I decided use this method.\n\n**3. Features**\nBased on raw data (without the use of additional data sources), many features were created and divided into categories:\na) Time features (based on eventTimestamp)\nb) Display type (based on unitDisplayType)\nc) Popularity (of device, brand and application)\nd) Target encoding (by brand, app, devide, c1, c3)\ne) Hardware/software\nf) Country\ng) Connection type\nh) Size of ad\ni) Sent price\nj) Anonymized features\n\nIn total there were 41 features diveded to this 10 categories. Full calculation can be find in [my notebook](https://www.kaggle.com/code/michau96/bid-price-predicition-4th-place-submission).\n\n**4. Data leaks**\nCreating features based on the whole \"train\" file (eg target encoding) from which we then create the training and validation set if cross validation, it will cause that we are dealing with data leaks. There are at least 2 leaks here:\na) To calculate aggregation features (means, medians), the values are also calculated based on validation set, which means that the rows on which the model will validate are counted for aggregating,\nb) We ignore time to calculate all aggregates, so future values are also taken into account, although it would be impossible during the production operation of the model.\n\nIt was decided to leave these leaks due to:\na) A / B tests have shown that the model behave better with leaks, both on the validation set (which is obvious), but also on the test set (lower RMSE on the public leaderboard),\nb) The test set looks like is not divided over time (the same ranges as in the train set), so the aspect of data leakage with time is not crutial for test set results. \n\nOf course, leaving the leaks is a conceptual error, but solving the problem is relatively time-consuming and should not bring such huge changes in the result on the leaderboard in relation to the time spent on it. This does not change the fact that a model trained in this way should not be used thoughtlessly in production.\n\n**5. Features selection**\nThe features were selected for the model based on feature importance from lightGBM. Those with zero or very negligible frequency of selecting for tree node were removed from the set. On this basis, around 30 out of 41 features were included in the final model.\n\n**6. Hyperparameters**\nNo hyperparameter optimization package was used. Simple A / B were performed where changes in the value of a single parameter were tested. The final set of the most important hyperparameters looks like this:\na) boosting_type: gdbt (better than dart in A / B test)\nb) learning_rate: 0.1\nc) colsample_bytree: 1 (having a relatively large number of features, different values were tried such as 0.2, 0.6 to create less correlated trees, but the best results were obtained when we take all the features into account when partitioning by node)\nd) bagging_fraction: 0.6 (the results for different values between 0.2 and 1 gave very similar results, so the middle option was chosen)\ne) max_depth: 12 (as the parameter increases, the RMSE on the validation set decreases, but the RMSE in test set on public leaderboard remains almost unchanged. The middle option was also chosen for fear of overfit when the trees are very complex)\nf) n_estimators: 3000 (so that learning does not end too quickly with early stopping)\n\n**7. Learning**\nThe model was learned using early stopping set to 40 iterations (if the RMSE on the validation set does not decrease for 40 iterations in a row, we stop learning). All folds had between 1000 and 3000 iterations (a large number due to the low learning rate and max depth).\n\nWith the trained model, predictions were made on the test set and a \"clip\" was applied to the predicted values (all values below 0.01 were assigned 0.01 due to the fact that it is the minimum possible bet value).\n\n**8. Best model**\nAfter another small A/B tests, the best model had the following parameters: around 30 feature bringing a variety of information,\n7-fold cross-validation (7 models were created and the average winBid value from each model was taken for prediction),\nrelatively low learning rate and maximum depth, and learning with early stopping, hard rule for prediction of values below 0.01.\n\nThe final results of the model:\nRuntime: Around 90 minutes on Kaggle enviroment\nValidation RMSE (mean fold predictions): 7.13\nPublic leaderboard RMSE: 15.63 (4th place)\nPrivate leaderboard RMSE: 16.23 (4th place)\n\n**9. Next step if I would spend more time**\nLeaderboard model and score can be improved. The next steps that can be taken with more time are:\na) Use of NN models,\nb) Model stacking (GBM + NN),\nc) Removal of all leaks in the validation set,\nd) Getting to know more about the ad recommendation industry and creating new features based on this knowledge.\ne) Deeper recognition of timestamp operation in the training and test set along with an attempt to validate over time.\n\nI put all the code in [a public notebook](https://www.kaggle.com/code/michau96/bid-price-predicition-4th-place-submission). Enjoy!\n\nP.S. @wrrosa Congratulations on the 1st place!", "votes": null }, { "id": "1969717", "postDate": "10/03/2022 16:41:40", "content": "Thanks for sharing your approach @michau96, this is a good reference for non-participants like me who may wish to make a late submission.
\nCongratulations for your final ranking too!
Great write-up @michau96 . thanks!
", "rawMarkdown": "Great write-up @michau96 . thanks!", "votes": null }, { "id": "1971121", "postDate": "10/04/2022 13:23:24", "content": "Thank you for sharing your advice @michau96 👍
", "rawMarkdown": "Thank you for sharing your advice @michau96 👍", "votes": null } ], "comments": [ { "id": 1969717, "author_name": "ravi20076", "author_url": "", "post_date": "10/03/2022 16:41:40", "content": "Thanks for sharing your approach @michau96, this is a good reference for non-participants like me who may wish to make a late submission.
\nCongratulations for your final ranking too!
Great write-up @michau96 . thanks!
", "votes": null, "replies": [] }, { "id": 1971121, "author_name": "rijudhara", "author_url": "", "post_date": "10/04/2022 13:23:24", "content": "Thank you for sharing your advice @michau96 👍
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1969709": "Hi, I achieved 4th place so decided that I will share the details of my solution\n\n**1. Model/package**\nLightGBM, because is powerful and very fast operation (with over 7 million rows it will significantly reduce the learning time)\n\n**2. Validation**\nThe rows in the test set look as if they were taken from the same set as the training set, but in such a way that each of the auctions concerns a different device. The time relationship did not seem so obvious to me to try to validate over time (the minimum and maximum days in both sets are the same, so learning from old data and validating on new data does not seem to be an ideal strategy for the leaderboard score at first glance).\n\nI tried 3 types of validation: \na) Random division into training and validation set in proportion (70-80% vs. 30-20%), \nb) K-fold cross-validation (k between 5 and 10 due to large number of rows) \nc) Creating a validation, but ensuring that each auction concerns a different device (as in the test set).\n\nOnly the last method provided private results between the public leaderboard and RMSE on the validation set in A/B tests, however the k-fold cross-validation had the lowest values on both sets, so I decided use this method.\n\n**3. Features**\nBased on raw data (without the use of additional data sources), many features were created and divided into categories:\na) Time features (based on eventTimestamp)\nb) Display type (based on unitDisplayType)\nc) Popularity (of device, brand and application)\nd) Target encoding (by brand, app, devide, c1, c3)\ne) Hardware/software\nf) Country\ng) Connection type\nh) Size of ad\ni) Sent price\nj) Anonymized features\n\nIn total there were 41 features diveded to this 10 categories. Full calculation can be find in [my notebook](https://www.kaggle.com/code/michau96/bid-price-predicition-4th-place-submission).\n\n**4. Data leaks**\nCreating features based on the whole \"train\" file (eg target encoding) from which we then create the training and validation set if cross validation, it will cause that we are dealing with data leaks. There are at least 2 leaks here:\na) To calculate aggregation features (means, medians), the values are also calculated based on validation set, which means that the rows on which the model will validate are counted for aggregating,\nb) We ignore time to calculate all aggregates, so future values are also taken into account, although it would be impossible during the production operation of the model.\n\nIt was decided to leave these leaks due to:\na) A / B tests have shown that the model behave better with leaks, both on the validation set (which is obvious), but also on the test set (lower RMSE on the public leaderboard),\nb) The test set looks like is not divided over time (the same ranges as in the train set), so the aspect of data leakage with time is not crutial for test set results. \n\nOf course, leaving the leaks is a conceptual error, but solving the problem is relatively time-consuming and should not bring such huge changes in the result on the leaderboard in relation to the time spent on it. This does not change the fact that a model trained in this way should not be used thoughtlessly in production.\n\n**5. Features selection**\nThe features were selected for the model based on feature importance from lightGBM. Those with zero or very negligible frequency of selecting for tree node were removed from the set. On this basis, around 30 out of 41 features were included in the final model.\n\n**6. Hyperparameters**\nNo hyperparameter optimization package was used. Simple A / B were performed where changes in the value of a single parameter were tested. The final set of the most important hyperparameters looks like this:\na) boosting_type: gdbt (better than dart in A / B test)\nb) learning_rate: 0.1\nc) colsample_bytree: 1 (having a relatively large number of features, different values were tried such as 0.2, 0.6 to create less correlated trees, but the best results were obtained when we take all the features into account when partitioning by node)\nd) bagging_fraction: 0.6 (the results for different values between 0.2 and 1 gave very similar results, so the middle option was chosen)\ne) max_depth: 12 (as the parameter increases, the RMSE on the validation set decreases, but the RMSE in test set on public leaderboard remains almost unchanged. The middle option was also chosen for fear of overfit when the trees are very complex)\nf) n_estimators: 3000 (so that learning does not end too quickly with early stopping)\n\n**7. Learning**\nThe model was learned using early stopping set to 40 iterations (if the RMSE on the validation set does not decrease for 40 iterations in a row, we stop learning). All folds had between 1000 and 3000 iterations (a large number due to the low learning rate and max depth).\n\nWith the trained model, predictions were made on the test set and a \"clip\" was applied to the predicted values (all values below 0.01 were assigned 0.01 due to the fact that it is the minimum possible bet value).\n\n**8. Best model**\nAfter another small A/B tests, the best model had the following parameters: around 30 feature bringing a variety of information,\n7-fold cross-validation (7 models were created and the average winBid value from each model was taken for prediction),\nrelatively low learning rate and maximum depth, and learning with early stopping, hard rule for prediction of values below 0.01.\n\nThe final results of the model:\nRuntime: Around 90 minutes on Kaggle enviroment\nValidation RMSE (mean fold predictions): 7.13\nPublic leaderboard RMSE: 15.63 (4th place)\nPrivate leaderboard RMSE: 16.23 (4th place)\n\n**9. Next step if I would spend more time**\nLeaderboard model and score can be improved. The next steps that can be taken with more time are:\na) Use of NN models,\nb) Model stacking (GBM + NN),\nc) Removal of all leaks in the validation set,\nd) Getting to know more about the ad recommendation industry and creating new features based on this knowledge.\ne) Deeper recognition of timestamp operation in the training and test set along with an attempt to validate over time.\n\nI put all the code in [a public notebook](https://www.kaggle.com/code/michau96/bid-price-predicition-4th-place-submission). Enjoy!\n\nP.S. @wrrosa Congratulations on the 1st place!", "1969717": "Thanks for sharing your approach @michau96, this is a good reference for non-participants like me who may wish to make a late submission.\nCongratulations for your final ranking too!", "1970566": "Great write-up @michau96 . thanks!", "1971121": "Thank you for sharing your advice @michau96 👍" }, "source": "existing_discussion", "complete": true, "competition_id": 37722, "competition_slug": "digital-turbine-auction-bid-price-prediction", "team_id": 9100516, "solution_url": "/competitions/digital-turbine-auction-bid-price-prediction/writeups/michal-bogacz-4th-place-submission" }