I found it quite confusing when I'm doing the same feature extraction and training models with same parameters but got different validation score as well as LB score. When using time-based validation set (last three months), I could get a validation score about 1.69 and LB score about 1.687. At the same time, when using 5-fold cross validation, I got an average validation score about 1.589 with LB score 1.676. What's more, I tried another kernel with 5-fold cross validation which got an average validation score 1.62 and LB score 1.66.
\n\nThis bothers me as I don't actually know where the problem lies when my LB score decreases even if I have done much feature extraction work. I would appreciate if you have any suggestions and thanks in advance.
", "messages": [ { "id": 391757, "postDate": "2018-09-22T11:21:33.133Z", "content": "I get very consistent LB - CV with my time split validation on sum of logs.
\n\nedit: not anymore
", "rawMarkdown": "I get very consistent LB - CV with my time split validation on sum of logs. \n\nedit: not anymore", "votes": 3, "replies": [ { "id": 391833, "postDate": "2018-09-22T14:12:33.990Z", "content": "Hi @divrikwicky by time split validation, you mean that after a date threshold you consider that as a validation set?
", "rawMarkdown": "Hi @divrikwicky by time split validation, you mean that after a date threshold you consider that as a validation set?" } ] }, { "id": 391715, "postDate": "2018-09-22T08:52:05.893Z", "content": "I have the same experience as Zijun Yao. I've done tons of feature engineering and got my local CV down to 1.56, but public LB stays consistently in the 1.66-1.67 range (and that's when using sum of logs; when using the proper summarization with log of sums I'm getting results in the 1.75-1.76 range). My best public LB model has a local CV of 1.61. \nI've more or less decided to disregard the public LB scores for now, but the sum-log/log-sum thing annoys me, because local CV on the user level is also significantly better with the log-sum approach than it is with sum-log.
", "rawMarkdown": "I have the same experience as Zijun Yao. I've done tons of feature engineering and got my local CV down to 1.56, but public LB stays consistently in the 1.66-1.67 range (and that's when using sum of logs; when using the proper summarization with log of sums I'm getting results in the 1.75-1.76 range). My best public LB model has a local CV of 1.61. \nI've more or less decided to disregard the public LB scores for now, but the sum-log/log-sum thing annoys me, because local CV on the user level is also significantly better with the log-sum approach than it is with sum-log. ", "votes": 1, "replies": [ { "id": 392553, "postDate": "2018-09-24T01:08:58.670Z", "content": "Wow, what a boost with sum of logs!
", "rawMarkdown": "Wow, what a boost with sum of logs!" }, { "id": 393227, "postDate": "2018-09-25T02:20:57.933Z", "content": "Sorry for a late reply. I've also been annoyed by sum-log/log-sum thing these days trying to figure out where the problem lies but right now it isn't a trouble any more after the update.
", "rawMarkdown": "Sorry for a late reply. I've also been annoyed by sum-log/log-sum thing these days trying to figure out where the problem lies but right now it isn't a trouble any more after the update." } ] }, { "id": 391243, "postDate": "2018-09-21T13:06:19.177Z", "content": "I found it quite confusing when I'm doing the same feature extraction and training models with same parameters but got different validation score as well as LB score. When using time-based validation set (last three months), I could get a validation score about 1.69 and LB score about 1.687. At the same time, when using 5-fold cross validation, I got an average validation score about 1.589 with LB score 1.676. What's more, I tried another kernel with 5-fold cross validation which got an average validation score 1.62 and LB score 1.66.
\n\nThis bothers me as I don't actually know where the problem lies when my LB score decreases even if I have done much feature extraction work. I would appreciate if you have any suggestions and thanks in advance.
", "rawMarkdown": "I found it quite confusing when I'm doing the same feature extraction and training models with same parameters but got different validation score as well as LB score. When using time-based validation set (last three months), I could get a validation score about 1.69 and LB score about 1.687. At the same time, when using 5-fold cross validation, I got an average validation score about 1.589 with LB score 1.676. What's more, I tried another kernel with 5-fold cross validation which got an average validation score 1.62 and LB score 1.66.\n\nThis bothers me as I don't actually know where the problem lies when my LB score decreases even if I have done much feature extraction work. I would appreciate if you have any suggestions and thanks in advance.", "votes": 1 }, { "id": 391313, "postDate": "2018-09-21T15:30:32.300Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 391541, "postDate": "2018-09-22T02:05:14.810Z", "content": "Thanks. I do think so but right now I have to make some blind attempts to figure things out so that I can optimize my entire work. BTW, it's impressive that you are doing so great in this competition!
", "rawMarkdown": "Thanks. I do think so but right now I have to make some blind attempts to figure things out so that I can optimize my entire work. BTW, it's impressive that you are doing so great in this competition!" } ] } ], "comments": [ { "id": 391757, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2018-09-22T11:21:33.133000", "content": "I get very consistent LB - CV with my time split validation on sum of logs.
\n\nedit: not anymore
", "votes": 3, "replies": [ { "id": 391833, "author_name": "maverick", "author_url": "", "post_date": "2018-09-22T14:12:33.990000", "content": "Hi @divrikwicky by time split validation, you mean that after a date threshold you consider that as a validation set?
", "votes": 0, "replies": [] } ] }, { "id": 391715, "author_name": "Kjetil Åmdal-Sævik", "author_url": "", "post_date": "2018-09-22T08:52:05.893000", "content": "I have the same experience as Zijun Yao. I've done tons of feature engineering and got my local CV down to 1.56, but public LB stays consistently in the 1.66-1.67 range (and that's when using sum of logs; when using the proper summarization with log of sums I'm getting results in the 1.75-1.76 range). My best public LB model has a local CV of 1.61. \nI've more or less decided to disregard the public LB scores for now, but the sum-log/log-sum thing annoys me, because local CV on the user level is also significantly better with the log-sum approach than it is with sum-log.
", "votes": 1, "replies": [ { "id": 392553, "author_name": "Nick", "author_url": "", "post_date": "2018-09-24T01:08:58.670000", "content": "Wow, what a boost with sum of logs!
", "votes": 0, "replies": [] }, { "id": 393227, "author_name": "syoya", "author_url": "", "post_date": "2018-09-25T02:20:57.933000", "content": "Sorry for a late reply. I've also been annoyed by sum-log/log-sum thing these days trying to figure out where the problem lies but right now it isn't a trouble any more after the update.
", "votes": 0, "replies": [] } ] }, { "id": 391313, "author_name": "", "author_url": "", "post_date": "2018-09-21T15:30:32.300000", "content": "", "votes": 1, "replies": [ { "id": 391541, "author_name": "syoya", "author_url": "", "post_date": "2018-09-22T02:05:14.810000", "content": "Thanks. I do think so but right now I have to make some blind attempts to figure things out so that I can optimize my entire work. BTW, it's impressive that you are doing so great in this competition!
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "391757": "I get very consistent LB - CV with my time split validation on sum of logs. \n\nedit: not anymore", "391715": "I have the same experience as Zijun Yao. I've done tons of feature engineering and got my local CV down to 1.56, but public LB stays consistently in the 1.66-1.67 range (and that's when using sum of logs; when using the proper summarization with log of sums I'm getting results in the 1.75-1.76 range). My best public LB model has a local CV of 1.61. \nI've more or less decided to disregard the public LB scores for now, but the sum-log/log-sum thing annoys me, because local CV on the user level is also significantly better with the log-sum approach than it is with sum-log. ", "391243": "I found it quite confusing when I'm doing the same feature extraction and training models with same parameters but got different validation score as well as LB score. When using time-based validation set (last three months), I could get a validation score about 1.69 and LB score about 1.687. At the same time, when using 5-fold cross validation, I got an average validation score about 1.589 with LB score 1.676. What's more, I tried another kernel with 5-fold cross validation which got an average validation score 1.62 and LB score 1.66.\n\nThis bothers me as I don't actually know where the problem lies when my LB score decreases even if I have done much feature extraction work. I would appreciate if you have any suggestions and thanks in advance.", "391313": "" } }