Hey, sorry if this is a novice question, this is my first Kaggle competition.
\nThere are some extremely long notebooks in the training dataset (e.g. a good proportion of the notebooks has over 100 total cells). However, the 4 test json examples don't contain any extremely long examples, but 4 examples is not that many to get an idea of the test distribution.
\nFrom what it sounds like, our solution kernel should be able to handle arbitrarily long notebooks where the cell count maybe over one hundred, is that a good assumption? Furthermore, it also sounds like we should expect the test distribution to be pretty much like our training distribution where we will encounter very long notebooks, sometimes very large cells, markdown with different languages, perhaps forked notebooks from some parent notebook, and similar things like that.
\nIt could also be the case that the final test set will be somewhat more maincured? For example, there maybe some limits placed on how many cells we may in fact encounter. If there will be some maincuring for the final test set, it would be really good to know!
", "messages": [ { "id": "1801642", "postDate": "05/26/2022 01:41:48", "content": "Hey, sorry if this is a novice question, this is my first Kaggle competition.
\nThere are some extremely long notebooks in the training dataset (e.g. a good proportion of the notebooks has over 100 total cells). However, the 4 test json examples don't contain any extremely long examples, but 4 examples is not that many to get an idea of the test distribution.
\nFrom what it sounds like, our solution kernel should be able to handle arbitrarily long notebooks where the cell count maybe over one hundred, is that a good assumption? Furthermore, it also sounds like we should expect the test distribution to be pretty much like our training distribution where we will encounter very long notebooks, sometimes very large cells, markdown with different languages, perhaps forked notebooks from some parent notebook, and similar things like that.
\nIt could also be the case that the final test set will be somewhat more maincured? For example, there maybe some limits placed on how many cells we may in fact encounter. If there will be some maincuring for the final test set, it would be really good to know!
", "rawMarkdown": "Hey, sorry if this is a novice question, this is my first Kaggle competition.\n\nThere are some extremely long notebooks in the training dataset (e.g. a good proportion of the notebooks has over 100 total cells). However, the 4 test json examples don't contain any extremely long examples, but 4 examples is not that many to get an idea of the test distribution. \n\nFrom what it sounds like, our solution kernel should be able to handle arbitrarily long notebooks where the cell count maybe over one hundred, is that a good assumption? Furthermore, it also sounds like we should expect the test distribution to be pretty much like our training distribution where we will encounter very long notebooks, sometimes very large cells, markdown with different languages, perhaps forked notebooks from some parent notebook, and similar things like that. \n\nIt could also be the case that the final test set will be somewhat more maincured? For example, there maybe some limits placed on how many cells we may in fact encounter. If there will be some maincuring for the final test set, it would be really good to know!", "votes": null }, { "id": "1803124", "postDate": "05/27/2022 13:39:34", "content": "I think all public notebooks deal with this problem by computing the percentage rank instead of the absolute rank, then the rank becomes continuous and restricted to the 0-1 range.
\nFor long markdowns and codes, I think we can truncate them (as well as padding for short ones), just like in texts.
\nFor different languages, I tried xlm-roberta-base, but it was way worse than distilbert-base-uncased (I don't know why, tbh).
", "rawMarkdown": "I think all public notebooks deal with this problem by computing the percentage rank instead of the absolute rank, then the rank becomes continuous and restricted to the 0-1 range.\n\nFor long markdowns and codes, I think we can truncate them (as well as padding for short ones), just like in texts. \n\nFor different languages, I tried xlm-roberta-base, but it was way worse than distilbert-base-uncased (I don't know why, tbh).", "votes": null }, { "id": "1803506", "postDate": "05/27/2022 22:16:22", "content": "Yeah, I agree there are definitely ways to try to make things work.
\nI think my concerns are two-fold.
\nFirst: if we know there is an upper limit to the size of notebooks, then that can help us be aware of things that can affect memory constraints or other inference time constraints. My best guess is that we'll probably need to design something that is capable of arbitrarily long notebooks.
\nSecond: if there will be more manicuring, we should probably try to shape our training/validation sets to reflect more of the things we expect to see in the test set.
", "rawMarkdown": "Yeah, I agree there are definitely ways to try to make things work. \n\nI think my concerns are two-fold. \n\nFirst: if we know there is an upper limit to the size of notebooks, then that can help us be aware of things that can affect memory constraints or other inference time constraints. My best guess is that we'll probably need to design something that is capable of arbitrarily long notebooks. \n\nSecond: if there will be more manicuring, we should probably try to shape our training/validation sets to reflect more of the things we expect to see in the test set.", "votes": null } ], "comments": [ { "id": 1803124, "author_name": "shinomoriaoshi", "author_url": "", "post_date": "05/27/2022 13:39:34", "content": "I think all public notebooks deal with this problem by computing the percentage rank instead of the absolute rank, then the rank becomes continuous and restricted to the 0-1 range.
\nFor long markdowns and codes, I think we can truncate them (as well as padding for short ones), just like in texts.
\nFor different languages, I tried xlm-roberta-base, but it was way worse than distilbert-base-uncased (I don't know why, tbh).
", "votes": null, "replies": [ { "id": 1803506, "author_name": "shypula", "author_url": "", "post_date": "05/27/2022 22:16:22", "content": "Yeah, I agree there are definitely ways to try to make things work.
\nI think my concerns are two-fold.
\nFirst: if we know there is an upper limit to the size of notebooks, then that can help us be aware of things that can affect memory constraints or other inference time constraints. My best guess is that we'll probably need to design something that is capable of arbitrarily long notebooks.
\nSecond: if there will be more manicuring, we should probably try to shape our training/validation sets to reflect more of the things we expect to see in the test set.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1801642": "Hey, sorry if this is a novice question, this is my first Kaggle competition.\n\nThere are some extremely long notebooks in the training dataset (e.g. a good proportion of the notebooks has over 100 total cells). However, the 4 test json examples don't contain any extremely long examples, but 4 examples is not that many to get an idea of the test distribution. \n\nFrom what it sounds like, our solution kernel should be able to handle arbitrarily long notebooks where the cell count maybe over one hundred, is that a good assumption? Furthermore, it also sounds like we should expect the test distribution to be pretty much like our training distribution where we will encounter very long notebooks, sometimes very large cells, markdown with different languages, perhaps forked notebooks from some parent notebook, and similar things like that. \n\nIt could also be the case that the final test set will be somewhat more maincured? For example, there maybe some limits placed on how many cells we may in fact encounter. If there will be some maincuring for the final test set, it would be really good to know!", "1803124": "I think all public notebooks deal with this problem by computing the percentage rank instead of the absolute rank, then the rank becomes continuous and restricted to the 0-1 range.\n\nFor long markdowns and codes, I think we can truncate them (as well as padding for short ones), just like in texts. \n\nFor different languages, I tried xlm-roberta-base, but it was way worse than distilbert-base-uncased (I don't know why, tbh).", "1803506": "Yeah, I agree there are definitely ways to try to make things work. \n\nI think my concerns are two-fold. \n\nFirst: if we know there is an upper limit to the size of notebooks, then that can help us be aware of things that can affect memory constraints or other inference time constraints. My best guess is that we'll probably need to design something that is capable of arbitrarily long notebooks. \n\nSecond: if there will be more manicuring, we should probably try to shape our training/validation sets to reflect more of the things we expect to see in the test set." }, "source": "meta" }