We know that people seem to be getting better scores using sum(log(transactions)) and that it might eventually result in lb shakeup. So I thought it would be good to know what is the best score that people can achieve using log(sum(transactions)), so we know what to aim for as public leaderboard doesn't represent it.
\n\nI'll go first. \nMy best score using log(sum(transactions)) is 1.764.
", "messages": [ { "id": "392369", "postDate": "09/23/2018 16:22:25", "content": "We know that people seem to be getting better scores using sum(log(transactions)) and that it might eventually result in lb shakeup. So I thought it would be good to know what is the best score that people can achieve using log(sum(transactions)), so we know what to aim for as public leaderboard doesn't represent it.
\n\nI'll go first. \nMy best score using log(sum(transactions)) is 1.764.
", "rawMarkdown": "We know that people seem to be getting better scores using sum(log(transactions)) and that it might eventually result in lb shakeup. So I thought it would be good to know what is the best score that people can achieve using log(sum(transactions)), so we know what to aim for as public leaderboard doesn't represent it. \n\nI'll go first. \nMy best score using log(sum(transactions)) is 1.764.", "votes": null }, { "id": "392374", "postDate": "09/23/2018 16:42:01", "content": "Mine is 1.70. i believe that sum(log(transactions)) is a big mistake for this competition, but better results arise when summing the logs.... i believe that we will have really big shakups, considering that the majority uses the summation of logs.\nFurthermore, what is more intresting,is that i have another model with a per user CV: 1.95 with per transaction CV:1.58 and LB of 1.6575.\ni think that the fitting must be done on user level, no mater the LB is
", "rawMarkdown": "Mine is 1.70. i believe that sum(log(transactions)) is a big mistake for this competition, but better results arise when summing the logs.... i believe that we will have really big shakups, considering that the majority uses the summation of logs.\nFurthermore, what is more intresting,is that i have another model with a per user CV: 1.95 with per transaction CV:1.58 and LB of 1.6575.\ni think that the fitting must be done on user level, no mater the LB is", "votes": null }, { "id": "392419", "postDate": "09/23/2018 18:35:44", "content": "All of my submissions have used log(sum(transactions)) and my best score is 1.7193. I guess this means I should try seeing how I do on the public lb with sum(log(transactions))?
", "rawMarkdown": "All of my submissions have used log(sum(transactions)) and my best score is 1.7193. I guess this means I should try seeing how I do on the public lb with sum(log(transactions))?", "votes": null }, { "id": "392421", "postDate": "09/23/2018 18:41:53", "content": "you can try this. Most probably you will have better LB, but this is wrong. I Believe this happens due to the fact that the target variable is inbalanced, the sum(log(transaction)) is lower tha the sum(log(transactions)) and the LB is on 30% of data
", "rawMarkdown": "you can try this. Most probably you will have better LB, but this is wrong. I Believe this happens due to the fact that the target variable is inbalanced, the sum(log(transaction)) is lower tha the sum(log(transactions)) and the LB is on 30% of data", "votes": null }, { "id": "392428", "postDate": "09/23/2018 18:53:15", "content": "It's very interesting. I guess the best way is to try to optimize my user CV (based on the real metric) and not worry too much about the public lb.
", "rawMarkdown": "It's very interesting. I guess the best way is to try to optimize my user CV (based on the real metric) and not worry *too much* about the public lb.", "votes": null }, { "id": "392434", "postDate": "09/23/2018 19:06:37", "content": "yep.\nmy best LB model has CV:_1.5933 on transaction level,but it has CV 1.95 on user level and LB 1.6575, but this model is useless.....
", "rawMarkdown": "yep.\nmy best LB model has CV:_1.5933 on transaction level,but it has CV 1.95 on user level and LB 1.6575, but this model is useless.....", "votes": null }, { "id": "392517", "postDate": "09/23/2018 22:33:48", "content": "Why is your model useless? I do not understand
", "rawMarkdown": "Why is your model useless? I do not understand", "votes": null }, { "id": "392521", "postDate": "09/23/2018 22:49:31", "content": "Well, there are many perspectives for this statement.\n1.) the model is trained/validated on transactional data and the evaluation on this competition is on user level\n2.) the early stopping on training takes place on the transaction level. Hence, the local CV is 1.59, but when using exp(transactions)-1 and then sum the transactions per user and then using log(sum(transactions)), then the local CV is 1.95. So, the LB of 1.6575 is just due to luck.\n3.) the model must be trained on user level. Consider the following example:
\n\na user has 2 transactions =[2000,0]. If the model's predictions are [0,2000] then the rmse is 2000 on transaction level validation evaluation, but on user level is 0. Hence, fitting this model does not optimize the problem that must be solved.
\n\nOn the other hand, fitting the model on a user level the LB is 1.70+...
", "rawMarkdown": "Well, there are many perspectives for this statement.\n1.) the model is trained/validated on transactional data and the evaluation on this competition is on user level\n2.) the early stopping on training takes place on the transaction level. Hence, the local CV is 1.59, but when using exp(transactions)-1 and then sum the transactions per user and then using log(sum(transactions)), then the local CV is 1.95. So, the LB of 1.6575 is just due to luck.\n3.) the model must be trained on user level. Consider the following example:\n\na user has 2 transactions =[2000,0]. If the model's predictions are [0,2000] then the rmse is 2000 on transaction level validation evaluation, but on user level is 0. Hence, fitting this model does not optimize the problem that must be solved. \n\nOn the other hand, fitting the model on a user level the LB is 1.70+...", "votes": null }, { "id": "392531", "postDate": "09/23/2018 23:43:54", "content": "When you say local CV is 1.59, what is your validation? Are you doing RMSE on log(transaction) data for that one?
", "rawMarkdown": "When you say local CV is 1.59, what is your validation? Are you doing RMSE on log(transaction) data for that one?", "votes": null }, { "id": "392532", "postDate": "09/23/2018 23:46:10", "content": "Exactly. The validation is on log(transaction) for 5 shuffled folds.
", "rawMarkdown": "Exactly. The validation is on log(transaction) for 5 shuffled folds.", "votes": null }, { "id": "392580", "postDate": "09/24/2018 02:30:04", "content": "I think for 1 and 3 you can work with both CV scores (transaction level and user level) and try to improve both at the same time
", "rawMarkdown": "I think for 1 and 3 you can work with both CV scores (transaction level and user level) and try to improve both at the same time", "votes": null }, { "id": "392634", "postDate": "09/24/2018 05:54:48", "content": "1) When you are making a model at user level \nWhats your validation scheme ?( 5 folds? )
\n\n2) when we combine at full visitor Id, how are you aggregating the data ?( Mean,max,min?)
", "rawMarkdown": "1) When you are making a model at user level \nWhats your validation scheme ?( 5 folds? )\n\n2) when we combine at full visitor Id, how are you aggregating the data ?( Mean,max,min?)", "votes": null }, { "id": "392648", "postDate": "09/24/2018 06:27:01", "content": "Hi,\nconcerning the model:\n1.) for the user model, the validation is a classic 5 fold\n2.) i have tried several approaches for the aggregation: 2.1) using mean,max,min etc, 2.2) keep some variables that do not change (f.e. city) and using the last 20 visits at a tabular form. The remained visits are reduced with PCA.
", "rawMarkdown": "Hi,\nconcerning the model:\n1.) for the user model, the validation is a classic 5 fold\n2.) i have tried several approaches for the aggregation: 2.1) using mean,max,min etc, 2.2) keep some variables that do not change (f.e. city) and using the last 20 visits at a tabular form. The remained visits are reduced with PCA.", "votes": null } ], "comments": [ { "id": 392374, "author_name": "dkaraflos", "author_url": "", "post_date": "09/23/2018 16:42:01", "content": "Mine is 1.70. i believe that sum(log(transactions)) is a big mistake for this competition, but better results arise when summing the logs.... i believe that we will have really big shakups, considering that the majority uses the summation of logs.\nFurthermore, what is more intresting,is that i have another model with a per user CV: 1.95 with per transaction CV:1.58 and LB of 1.6575.\ni think that the fitting must be done on user level, no mater the LB is
", "votes": null, "replies": [ { "id": 392634, "author_name": "suchith0312", "author_url": "", "post_date": "09/24/2018 05:54:48", "content": "1) When you are making a model at user level \nWhats your validation scheme ?( 5 folds? )
\n\n2) when we combine at full visitor Id, how are you aggregating the data ?( Mean,max,min?)
", "votes": null, "replies": [] }, { "id": 392648, "author_name": "dkaraflos", "author_url": "", "post_date": "09/24/2018 06:27:01", "content": "Hi,\nconcerning the model:\n1.) for the user model, the validation is a classic 5 fold\n2.) i have tried several approaches for the aggregation: 2.1) using mean,max,min etc, 2.2) keep some variables that do not change (f.e. city) and using the last 20 visits at a tabular form. The remained visits are reduced with PCA.
", "votes": null, "replies": [] } ] }, { "id": 392419, "author_name": "nnnnick", "author_url": "", "post_date": "09/23/2018 18:35:44", "content": "All of my submissions have used log(sum(transactions)) and my best score is 1.7193. I guess this means I should try seeing how I do on the public lb with sum(log(transactions))?
", "votes": null, "replies": [ { "id": 392421, "author_name": "dkaraflos", "author_url": "", "post_date": "09/23/2018 18:41:53", "content": "you can try this. Most probably you will have better LB, but this is wrong. I Believe this happens due to the fact that the target variable is inbalanced, the sum(log(transaction)) is lower tha the sum(log(transactions)) and the LB is on 30% of data
", "votes": null, "replies": [] }, { "id": 392428, "author_name": "nnnnick", "author_url": "", "post_date": "09/23/2018 18:53:15", "content": "It's very interesting. I guess the best way is to try to optimize my user CV (based on the real metric) and not worry too much about the public lb.
", "votes": null, "replies": [] }, { "id": 392434, "author_name": "dkaraflos", "author_url": "", "post_date": "09/23/2018 19:06:37", "content": "yep.\nmy best LB model has CV:_1.5933 on transaction level,but it has CV 1.95 on user level and LB 1.6575, but this model is useless.....
", "votes": null, "replies": [] }, { "id": 392517, "author_name": "returnofsputnik", "author_url": "", "post_date": "09/23/2018 22:33:48", "content": "Why is your model useless? I do not understand
", "votes": null, "replies": [] }, { "id": 392521, "author_name": "dkaraflos", "author_url": "", "post_date": "09/23/2018 22:49:31", "content": "Well, there are many perspectives for this statement.\n1.) the model is trained/validated on transactional data and the evaluation on this competition is on user level\n2.) the early stopping on training takes place on the transaction level. Hence, the local CV is 1.59, but when using exp(transactions)-1 and then sum the transactions per user and then using log(sum(transactions)), then the local CV is 1.95. So, the LB of 1.6575 is just due to luck.\n3.) the model must be trained on user level. Consider the following example:
\n\na user has 2 transactions =[2000,0]. If the model's predictions are [0,2000] then the rmse is 2000 on transaction level validation evaluation, but on user level is 0. Hence, fitting this model does not optimize the problem that must be solved.
\n\nOn the other hand, fitting the model on a user level the LB is 1.70+...
", "votes": null, "replies": [] }, { "id": 392531, "author_name": "returnofsputnik", "author_url": "", "post_date": "09/23/2018 23:43:54", "content": "When you say local CV is 1.59, what is your validation? Are you doing RMSE on log(transaction) data for that one?
", "votes": null, "replies": [] }, { "id": 392532, "author_name": "dkaraflos", "author_url": "", "post_date": "09/23/2018 23:46:10", "content": "Exactly. The validation is on log(transaction) for 5 shuffled folds.
", "votes": null, "replies": [] }, { "id": 392580, "author_name": "jsaguiar", "author_url": "", "post_date": "09/24/2018 02:30:04", "content": "I think for 1 and 3 you can work with both CV scores (transaction level and user level) and try to improve both at the same time
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "392369": "We know that people seem to be getting better scores using sum(log(transactions)) and that it might eventually result in lb shakeup. So I thought it would be good to know what is the best score that people can achieve using log(sum(transactions)), so we know what to aim for as public leaderboard doesn't represent it. \n\nI'll go first. \nMy best score using log(sum(transactions)) is 1.764.", "392374": "Mine is 1.70. i believe that sum(log(transactions)) is a big mistake for this competition, but better results arise when summing the logs.... i believe that we will have really big shakups, considering that the majority uses the summation of logs.\nFurthermore, what is more intresting,is that i have another model with a per user CV: 1.95 with per transaction CV:1.58 and LB of 1.6575.\ni think that the fitting must be done on user level, no mater the LB is", "392419": "All of my submissions have used log(sum(transactions)) and my best score is 1.7193. I guess this means I should try seeing how I do on the public lb with sum(log(transactions))?", "392421": "you can try this. Most probably you will have better LB, but this is wrong. I Believe this happens due to the fact that the target variable is inbalanced, the sum(log(transaction)) is lower tha the sum(log(transactions)) and the LB is on 30% of data", "392428": "It's very interesting. I guess the best way is to try to optimize my user CV (based on the real metric) and not worry *too much* about the public lb.", "392434": "yep.\nmy best LB model has CV:_1.5933 on transaction level,but it has CV 1.95 on user level and LB 1.6575, but this model is useless.....", "392517": "Why is your model useless? I do not understand", "392521": "Well, there are many perspectives for this statement.\n1.) the model is trained/validated on transactional data and the evaluation on this competition is on user level\n2.) the early stopping on training takes place on the transaction level. Hence, the local CV is 1.59, but when using exp(transactions)-1 and then sum the transactions per user and then using log(sum(transactions)), then the local CV is 1.95. So, the LB of 1.6575 is just due to luck.\n3.) the model must be trained on user level. Consider the following example:\n\na user has 2 transactions =[2000,0]. If the model's predictions are [0,2000] then the rmse is 2000 on transaction level validation evaluation, but on user level is 0. Hence, fitting this model does not optimize the problem that must be solved. \n\nOn the other hand, fitting the model on a user level the LB is 1.70+...", "392531": "When you say local CV is 1.59, what is your validation? Are you doing RMSE on log(transaction) data for that one?", "392532": "Exactly. The validation is on log(transaction) for 5 shuffled folds.", "392580": "I think for 1 and 3 you can work with both CV scores (transaction level and user level) and try to improve both at the same time", "392634": "1) When you are making a model at user level \nWhats your validation scheme ?( 5 folds? )\n\n2) when we combine at full visitor Id, how are you aggregating the data ?( Mean,max,min?)", "392648": "Hi,\nconcerning the model:\n1.) for the user model, the validation is a classic 5 fold\n2.) i have tried several approaches for the aggregation: 2.1) using mean,max,min etc, 2.2) keep some variables that do not change (f.e. city) and using the last 20 visits at a tabular form. The remained visits are reduced with PCA." }, "source": "meta" }