It's a bit early, but why not starting this typical thread. I find local cv and LB correlate very well.
\nMy local cv: 0.8540 LB: 0.8556
", "messages": [ { "id": 1806578, "postDate": "2022-05-31T10:04:50.713Z", "content": "It's a bit early, but why not starting this typical thread. I find local cv and LB correlate very well.
\nMy local cv: 0.8540 LB: 0.8556
", "rawMarkdown": "It's a bit early, but why not starting this typical thread. I find local cv and LB correlate very well.\n\nMy local cv: 0.8540 LB: 0.8556", "votes": 21 }, { "id": 1811836, "postDate": "2022-06-05T07:47:29.477Z", "content": "I'm currently using single train-dev split rather than CV to save the running time.
\nWhen I used 1% of the dataset for the dev split (roughly 1000 notebooks, of course removed all forked notebooks) the dev scores are about 0.01-0.02 point higher than LB scores.
\nNow I'm using 10% for the dev (≒10000 notebooks) and the latest LB score became much closer to the dev score, the difference is now about 0.001 point.
Maybe my 1%-size dev split has too many notebooks that are not forked but have copy-and-paste cells from/to notebooks in the train split.
", "rawMarkdown": "I'm currently using single train-dev split rather than CV to save the running time.\nWhen I used 1% of the dataset for the dev split (roughly 1000 notebooks, of course removed all forked notebooks) the dev scores are about 0.01-0.02 point higher than LB scores.\nNow I'm using 10% for the dev (≒10000 notebooks) and the latest LB score became much closer to the dev score, the difference is now about 0.001 point.\n\nMaybe my 1%-size dev split has too many notebooks that are not forked but have copy-and-paste cells from/to notebooks in the train split.", "votes": 4, "replies": [ { "id": 1872321, "postDate": "2022-07-26T20:24:35.070Z", "content": "@yufuin Are you just doing a random split? How are you removing forks?
", "rawMarkdown": "@yufuin Are you just doing a random split? How are you removing forks?" }, { "id": 1872792, "postDate": "2022-07-27T08:33:43.467Z", "content": "When I posted the comment I simply removed all of notebooks that have parent_id, then randomly split.
\nHowever I'm not using the split now. Since it's two weeks before the deadline, I will refrain from sharing my current method in detail. (sorry!)
I was dropping kernels without an ancestor_id and taking only 1 kernel with a unique ancestor_id.
", "rawMarkdown": "I was dropping kernels without an ancestor_id and taking only 1 kernel with a unique ancestor_id." }, { "id": 1872861, "postDate": "2022-07-27T09:25:00.863Z", "content": "@yufuin this is also helpful. Thanks :) Good luck!
", "rawMarkdown": "@yufuin this is also helpful. Thanks :) Good luck!" } ] }, { "id": 1872451, "postDate": "2022-07-27T01:32:31.880Z", "content": "I'm using 8/2 split with group by ansestor id, it seems online is 6-10k lower then offline. But I found there are so many duplicates even grouby by ansestor id is not enough maybe.
", "rawMarkdown": "I'm using 8/2 split with group by ansestor id, it seems online is 6-10k lower then offline. But I found there are so many duplicates even grouby by ansestor id is not enough maybe.", "votes": 1, "replies": [ { "id": 1872687, "postDate": "2022-07-27T07:13:52.903Z", "content": "I was trying to get away with 2.5% validation. But it wasn't the best idea.
\n@goldenlock you are not dropping any notebooks? In training or validation?
", "rawMarkdown": "I was trying to get away with 2.5% validation. But it wasn't the best idea.\n\n@goldenlock you are not dropping any notebooks? In training or validation?" }, { "id": 1874009, "postDate": "2022-07-28T02:37:03.380Z", "content": "@crodoc Yes using all data. Another finding is though online much lower score then offline, my offline improvement reflected well online so not much a problem maybe.
", "rawMarkdown": "@crodoc Yes using all data. Another finding is though online much lower score then offline, my offline improvement reflected well online so not much a problem maybe.", "votes": 1 }, { "id": 1874152, "postDate": "2022-07-28T05:23:51.667Z", "content": "Yes, same thing for us when we used 10% of data. Thanks a lot!
", "rawMarkdown": "Yes, same thing for us when we used 10% of data. Thanks a lot!" } ] }, { "id": 1809614, "postDate": "2022-06-02T21:12:11.843Z", "content": "Maybe because the notebooks are for the same time period, but the private test set will be for future !
", "rawMarkdown": "Maybe because the notebooks are for the same time period, but the private test set will be for future !", "replies": [ { "id": 1809776, "postDate": "2022-06-03T03:46:04.227Z", "content": "The authors of notebooks will be quite different, there will probably be shake up.
", "rawMarkdown": "The authors of notebooks will be quite different, there will probably be shake up.", "votes": 1 } ] } ], "comments": [ { "id": 1811836, "author_name": "yufuin", "author_url": "", "post_date": "2022-06-05T07:47:29.477000", "content": "I'm currently using single train-dev split rather than CV to save the running time.
\nWhen I used 1% of the dataset for the dev split (roughly 1000 notebooks, of course removed all forked notebooks) the dev scores are about 0.01-0.02 point higher than LB scores.
\nNow I'm using 10% for the dev (≒10000 notebooks) and the latest LB score became much closer to the dev score, the difference is now about 0.001 point.
Maybe my 1%-size dev split has too many notebooks that are not forked but have copy-and-paste cells from/to notebooks in the train split.
", "votes": 4, "replies": [ { "id": 1872321, "author_name": "CroDoc", "author_url": "", "post_date": "2022-07-26T20:24:35.070000", "content": "@yufuin Are you just doing a random split? How are you removing forks?
", "votes": 0, "replies": [] }, { "id": 1872792, "author_name": "yufuin", "author_url": "", "post_date": "2022-07-27T08:33:43.467000", "content": "When I posted the comment I simply removed all of notebooks that have parent_id, then randomly split.
\nHowever I'm not using the split now. Since it's two weeks before the deadline, I will refrain from sharing my current method in detail. (sorry!)
I was dropping kernels without an ancestor_id and taking only 1 kernel with a unique ancestor_id.
", "votes": 0, "replies": [] }, { "id": 1872861, "author_name": "CroDoc", "author_url": "", "post_date": "2022-07-27T09:25:00.863000", "content": "@yufuin this is also helpful. Thanks :) Good luck!
", "votes": 0, "replies": [] } ] }, { "id": 1872451, "author_name": "gezi", "author_url": "", "post_date": "2022-07-27T01:32:31.880000", "content": "I'm using 8/2 split with group by ansestor id, it seems online is 6-10k lower then offline. But I found there are so many duplicates even grouby by ansestor id is not enough maybe.
", "votes": 1, "replies": [ { "id": 1872687, "author_name": "CroDoc", "author_url": "", "post_date": "2022-07-27T07:13:52.903000", "content": "I was trying to get away with 2.5% validation. But it wasn't the best idea.
\n@goldenlock you are not dropping any notebooks? In training or validation?
", "votes": 0, "replies": [] }, { "id": 1874009, "author_name": "gezi", "author_url": "", "post_date": "2022-07-28T02:37:03.380000", "content": "@crodoc Yes using all data. Another finding is though online much lower score then offline, my offline improvement reflected well online so not much a problem maybe.
", "votes": 1, "replies": [] }, { "id": 1874152, "author_name": "CroDoc", "author_url": "", "post_date": "2022-07-28T05:23:51.667000", "content": "Yes, same thing for us when we used 10% of data. Thanks a lot!
", "votes": 0, "replies": [] } ] }, { "id": 1809614, "author_name": "Alireza", "author_url": "", "post_date": "2022-06-02T21:12:11.843000", "content": "Maybe because the notebooks are for the same time period, but the private test set will be for future !
", "votes": 0, "replies": [ { "id": 1809776, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2022-06-03T03:46:04.227000", "content": "The authors of notebooks will be quite different, there will probably be shake up.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1806578": "It's a bit early, but why not starting this typical thread. I find local cv and LB correlate very well.\n\nMy local cv: 0.8540 LB: 0.8556", "1811836": "I'm currently using single train-dev split rather than CV to save the running time.\nWhen I used 1% of the dataset for the dev split (roughly 1000 notebooks, of course removed all forked notebooks) the dev scores are about 0.01-0.02 point higher than LB scores.\nNow I'm using 10% for the dev (≒10000 notebooks) and the latest LB score became much closer to the dev score, the difference is now about 0.001 point.\n\nMaybe my 1%-size dev split has too many notebooks that are not forked but have copy-and-paste cells from/to notebooks in the train split.", "1872451": "I'm using 8/2 split with group by ansestor id, it seems online is 6-10k lower then offline. But I found there are so many duplicates even grouby by ansestor id is not enough maybe.", "1809614": "Maybe because the notebooks are for the same time period, but the private test set will be for future !" } }