![\"segment-ranking\"](\"https://raw.githubusercontent.com/louis-she/ai4code/master/assets/segment-ranking.png\")
Thank Kaggle for hosting this game, and the community for all the sharing. It's my first NLP competition so there are not much NLP-related sharing in my solution, just task-related tricks.
\nSource code: https://github.com/louis-she/ai4code
\nInference Notebook: https://www.kaggle.com/code/snaker/new-rank/notebook
\nOne deberta-v3-small model is enough, ranking method, subsequences(segment).
\nMy solution only uses the ranking method. Given a markdown cell and a bunch of sorted code cells, predict the position of the markdown cell. The main difference here is that I do not use a whole notebook to rank a markdown cell, but just a part of it, which can be called a segment.
For instance, given a notebook of 40 code cells, we can divide it into 4 sub segments, each segment containing 10 code cells. Then we rank each markdown of the whole notebook in the 10 code cells.
\nInteresting thoughts about this method, if we divided the 40 code cells into 40 segments, each segment contains only 1 code cell, then this method is the same as Pairwise, if do not divide the notebook(or divide it into 1 segment), then it's same as full ranking.
\nThe full ranking method, which is also open sourced here, is to predict a markdown cell's position in the whole code cells. The sequence feed to the model is usually like this: markdown_cell + code_cell + code_cell … code_cell, but many of the sequence is way too long for transformer models(even for longformer), one simple method to handle the long sequence issue is to cut off the length of makdown_cell and code_cell to make the ensembled sequence length is acceptable. But the more cut-off, the more information we lose, so how can we keep the information as much as we can and also get a reasonable output sequence length?
\nTo address the global information and sequence length balance issue, I came up with the Segment Ranking method.
\nA segment is part of a notebook's sorted code cells. If using M for the number of code cells in a notebook, and N for the number of code cells in a segment, then a notebook can be divided into C segments, in which C equals math.ceil(M / N).
Now since we have multiple segments of a notebook, and markdown cells in that notebook can only belong to one of the segments, we have to first decide if a markdown belongs to a segment, and then do the ranking in that segment. So there are two heads of the model, one for classification and one for ranking. The following figure demonstrates the details.
\n
The number of cell in segments(N) here is a super important hyperparameter. A large N will speed up training and inference, small N can bring more context information. I ended up choosing N = 16 and sequence max length = 384.
\nTraining with language modeling
\nThere is actually the third head of my model, which is used as language modeling. The input for that head is <previous_cell_encode + <anchor_cell_encode + <next_cell_encode, positive means that the three is continuous, negative means not continuous, they are just randomly pick from the notebook(but not the previous and next ones).
\npad is a wast
\nI try to keep as much information as possible and reduce tokens. Given the following 5 cells, how can we cut off to keep a length of 200? Instead cut every cell at a fixed length, I used the following method showed by the figure. Or have a look at the test case.
\n![\"sequence-ensembling\"](\"https://raw.githubusercontent.com/louis-she/ai4code/master/assets/sequence-ensembling.png\")
Thanks for reading.
", "messages": [ { "id": 1895564, "postDate": "2022-08-12T08:17:30.637Z", "content": "Thank Kaggle for hosting this game, and the community for all the sharing. It's my first NLP competition so there are not much NLP-related sharing in my solution, just task-related tricks.
\nSource code: https://github.com/louis-she/ai4code
\nInference Notebook: https://www.kaggle.com/code/snaker/new-rank/notebook
\nOne deberta-v3-small model is enough, ranking method, subsequences(segment).
\nMy solution only uses the ranking method. Given a markdown cell and a bunch of sorted code cells, predict the position of the markdown cell. The main difference here is that I do not use a whole notebook to rank a markdown cell, but just a part of it, which can be called a segment.
For instance, given a notebook of 40 code cells, we can divide it into 4 sub segments, each segment containing 10 code cells. Then we rank each markdown of the whole notebook in the 10 code cells.
\nInteresting thoughts about this method, if we divided the 40 code cells into 40 segments, each segment contains only 1 code cell, then this method is the same as Pairwise, if do not divide the notebook(or divide it into 1 segment), then it's same as full ranking.
\nThe full ranking method, which is also open sourced here, is to predict a markdown cell's position in the whole code cells. The sequence feed to the model is usually like this: markdown_cell + code_cell + code_cell … code_cell, but many of the sequence is way too long for transformer models(even for longformer), one simple method to handle the long sequence issue is to cut off the length of makdown_cell and code_cell to make the ensembled sequence length is acceptable. But the more cut-off, the more information we lose, so how can we keep the information as much as we can and also get a reasonable output sequence length?
\nTo address the global information and sequence length balance issue, I came up with the Segment Ranking method.
\nA segment is part of a notebook's sorted code cells. If using M for the number of code cells in a notebook, and N for the number of code cells in a segment, then a notebook can be divided into C segments, in which C equals math.ceil(M / N).
Now since we have multiple segments of a notebook, and markdown cells in that notebook can only belong to one of the segments, we have to first decide if a markdown belongs to a segment, and then do the ranking in that segment. So there are two heads of the model, one for classification and one for ranking. The following figure demonstrates the details.
\n
The number of cell in segments(N) here is a super important hyperparameter. A large N will speed up training and inference, small N can bring more context information. I ended up choosing N = 16 and sequence max length = 384.
\nTraining with language modeling
\nThere is actually the third head of my model, which is used as language modeling. The input for that head is <previous_cell_encode + <anchor_cell_encode + <next_cell_encode, positive means that the three is continuous, negative means not continuous, they are just randomly pick from the notebook(but not the previous and next ones).
\npad is a wast
\nI try to keep as much information as possible and reduce tokens. Given the following 5 cells, how can we cut off to keep a length of 200? Instead cut every cell at a fixed length, I used the following method showed by the figure. Or have a look at the test case.
\n
Thanks for reading.
", "rawMarkdown": "# Segment Ranking(9th on LB)\n\nThank Kaggle for hosting this game, and the community for all the sharing. It's my first NLP competition so there are not much NLP-related sharing in my solution, just task-related tricks.\n\nSource code: https://github.com/louis-she/ai4code\n\nInference Notebook: https://www.kaggle.com/code/snaker/new-rank/notebook\n\n## TL;DR\n\nOne deberta-v3-small model is enough, ranking method, subsequences(segment).\n\nMy solution only uses the ranking method. Given a markdown cell and a bunch of sorted code cells, predict the position of the markdown cell. The main difference here is that I do not use a whole notebook to rank a markdown cell, but just a part of it, which can be called a `segment`.\n\nFor instance, given a notebook of 40 code cells, we can divide it into 4 sub segments, each segment containing 10 code cells. Then we rank each markdown of the whole notebook in the 10 code cells.\n\nInteresting thoughts about this method, if we divided the 40 code cells into 40 segments, each segment contains only 1 code cell, then this method is the same as Pairwise, if do not divide the notebook(or divide it into 1 segment), then it's same as full ranking.\n\n## Segment Ranking\n\nThe full ranking method, which is also open sourced [here](https://www.kaggle.com/competitions/AI4Code/discussion/326970), is to predict a markdown cell's position in the whole code cells. The sequence feed to the model is usually like this: markdown_cell + code_cell + code_cell ... code_cell, but many of the sequence is way too long for transformer models(even for longformer), one simple method to handle the long sequence issue is to cut off the length of makdown_cell and code_cell to make the ensembled sequence length is acceptable. But the more cut-off, the more information we lose, so how can we keep the information as much as we can and also get a reasonable output sequence length?\n\nTo address the global information and sequence length balance issue, I came up with the Segment Ranking method.\n\nA segment is part of a notebook's sorted code cells. If using M for the number of code cells in a notebook, and N for the number of code cells in a segment, then a notebook can be divided into C segments, in which C equals `math.ceil(M / N)`.\n\nNow since we have multiple segments of a notebook, and markdown cells in that notebook can only belong to one of the segments, we have to first decide if a markdown belongs to a segment, and then do the ranking in that segment. So there are two heads of the model, one for classification and one for ranking. The following figure demonstrates the details.\n\n\n\nThe number of cell in segments(N) here is a super important hyperparameter. A large N will speed up training and inference, small N can bring more context information. I ended up choosing N = 16 and sequence max length = 384.\n\n## Other important Task-related tricks\n\n**Training with language modeling**\n\nThere is actually the third head of my model, which is used as language modeling. The input for that head is@snaker you are back? Do you know what happened?
\nGood luck with getting a gold medal! Happy to see you back on LB.
", "rawMarkdown": "@snaker you are back? Do you know what happened?\n\nGood luck with getting a gold medal! Happy to see you back on LB.", "votes": 1, "replies": [ { "id": 1947613, "postDate": "2022-09-20T15:01:54.877Z", "content": "Maybe there are some mistakes(data related probably) the official has made during the 1st rerun.
", "rawMarkdown": "Maybe there are some mistakes(data related probably) the official has made during the 1st rerun.", "votes": 1 }, { "id": 1947694, "postDate": "2022-09-20T15:36:56.577Z", "content": "I'm changing the title back… I got alive in the 2nd rerun..
", "rawMarkdown": "I'm changing the title back... I got alive in the 2nd rerun..", "votes": 2 } ] }, { "id": 1895695, "postDate": "2022-08-12T10:25:21.497Z", "content": "Thank you. This is really interesting. Do you mind telling more about your inspiration for the solution? How have you come to it? Would love to hear the details :)
", "rawMarkdown": "Thank you. This is really interesting. Do you mind telling more about your inspiration for the solution? How have you come to it? Would love to hear the details :)", "votes": 1, "replies": [ { "id": 1895978, "postDate": "2022-08-12T13:37:08.173Z", "content": "I keep thinking about how to bring more global context information. The model can only handle 512 or 1024 length of sequence, but the number of tokens of one notebook is way larger than that. So splitting a notebook into smaller segments is the first and most straightforward way to handle this issue.
", "rawMarkdown": "I keep thinking about how to bring more global context information. The model can only handle 512 or 1024 length of sequence, but the number of tokens of one notebook is way larger than that. So splitting a notebook into smaller segments is the first and most straightforward way to handle this issue.", "votes": 1 } ] }, { "id": 1895575, "postDate": "2022-08-12T08:38:50.647Z", "content": "Thanks for sharing! A clean and interesting solution indeed.
\nHow much improvement did your LM head and your padding strategy bring in terms of CV and/or public LB score? In our earlier experiments, our team briefly tried a padding strategy similar to yours, but it did not seem to improve on the more \"naive\" padding approach (Method 1 as you described).
", "rawMarkdown": "Thanks for sharing! A clean and interesting solution indeed.\n\nHow much improvement did your LM head and your padding strategy bring in terms of CV and/or public LB score? In our earlier experiments, our team briefly tried a padding strategy similar to yours, but it did not seem to improve on the more \"naive\" padding approach (Method 1 as you described).", "votes": 1, "replies": [ { "id": 1895588, "postDate": "2022-08-12T08:54:00.803Z", "content": "The ML head boosts CV is ~ 0.015 and the padding method boost CV is ~ 0.005.
", "rawMarkdown": "The ML head boosts CV is ~ 0.015 and the padding method boost CV is ~ 0.005.", "votes": 2 } ] }, { "id": 1906436, "postDate": "2022-08-19T22:01:41.083Z", "content": "Thanks for sharing your solution, You clearly worked hard on it.
\nCongrats!
The Devastator.
", "rawMarkdown": "Thanks for sharing your solution, You clearly worked hard on it.\nCongrats! \n\nThe Devastator.\n" }, { "id": 1901854, "postDate": "2022-08-17T00:50:19.563Z", "content": "Thanks for your sharing! Very clean solution.
\nHow do you make improvement on performance? Since in your github project says train.py get us a model with 0.87, but how do you improve the score from 0.87 to 0.91.
", "rawMarkdown": "Thanks for your sharing! Very clean solution.\n \nHow do you make improvement on performance? Since in your github project says train.py get us a model with 0.87, but how do you improve the score from 0.87 to 0.91. ", "replies": [ { "id": 1902942, "postDate": "2022-08-17T02:33:09.693Z", "content": "Train one epoch the score will be above 0.87, train more epochs(like 10) the score will be ~ 0.89 to 0.9 on both CV and LB.
\nAdd fold0 and external dataset to the training set and train 10 epochs, will achieve ~0.91 on LB
", "rawMarkdown": "Train one epoch the score will be above 0.87, train more epochs(like 10) the score will be ~ 0.89 to 0.9 on both CV and LB.\n\nAdd fold0 and external dataset to the training set and train 10 epochs, will achieve ~0.91 on LB", "votes": 2 } ] }, { "id": 1898832, "postDate": "2022-08-14T21:33:11.250Z", "content": "This is quite innovative, nice!
", "rawMarkdown": "This is quite innovative, nice!\n" }, { "id": 1897840, "postDate": "2022-08-14T05:47:46.370Z", "content": "great work.
", "rawMarkdown": "great work." }, { "id": 1897704, "postDate": "2022-08-14T02:57:40.730Z", "content": "Thanks for sharing the source code and it's well explained.
", "rawMarkdown": "Thanks for sharing the source code and it's well explained." }, { "id": 1896898, "postDate": "2022-08-13T08:54:48.150Z", "content": "great work
", "rawMarkdown": "great work \n" }, { "id": 1895767, "postDate": "2022-08-12T11:27:02.477Z", "content": "Nice!
\nI did a \"but we have [popular new item] at home\" version of this in the last week, where I had one classification-based output using a codebert base model. I also didn't try out the last solution for handling tokens, alas.
", "rawMarkdown": "Nice! \n\nI did a \"but we have [popular new item] at home\" version of this in the last week, where I had one classification-based output using a codebert base model. I also didn't try out the last solution for handling tokens, alas." }, { "id": 1895641, "postDate": "2022-08-12T09:28:43.550Z", "content": "Segment ranking - V. Original way to handle long sequences instead of using Longformer BigBird etc. Hope you get a well deserved gold.
\nBelow is it something like an auxiliary task you trained on but didnt actually use in inference. Is my understanding right?
\nThere is actually the third head of my model, which is used as language modeling. The input for that head is <previous_cell_encode + <anchor_cell_encode + <next_cell_encode, positive means that the three is continuous, negative means not continuous, they are just randomly pick from the notebook(but not the previous and next ones).
Instead of pretrain with language modeling, I train the language modeling along with the task.
", "rawMarkdown": "Yes it's not used at inference time.\n\nInstead of pretrain with language modeling, I train the language modeling along with the task.", "votes": 1 } ] }, { "id": 1898682, "postDate": "2022-08-14T18:19:28.340Z", "content": "I'm wondering what are the training samples format, since we only know \"[code1] [code2] [code3] …\" and \"[markdown1] [markdown2] …\" in the training dataset, instead of \"[markdown1] [code1] [code2], [markdown2] [code3] [code4]\". So it seems we don't know if a markdown belong to a code cell(or segment). How to do the classification?
", "rawMarkdown": "I'm wondering what are the training samples format, since we only know \"[code1] [code2] [code3] ...\" and \"[markdown1] [markdown2] ...\" in the training dataset, instead of \"[markdown1] [code1] [code2], [markdown2] [code3] [code4]\". So it seems we don't know if a markdown belong to a code cell(or segment). How to do the classification?", "votes": 1, "isDeleted": true, "replies": [ { "id": 1898964, "postDate": "2022-08-15T00:56:12.910Z", "content": "if the real order of a notebook is like this: c1 c2 c3 c4 c5 m1 c6 c7 c8 c9, and divide into 3 segments:
segment1: m1 c1 c2 c3 cls label 0(m1 not in the segment), ranking label: skipped
\nsegment2: m1 c4 c5 c6 cls label 1 (m1 is in the segment), ranking label: 2.5
\nsegments: m1 c7 c8 c9 cls label 0, ranking label: skipped
How to konw the ranking label? It seems all the code cells appear before markdown cells in the train dataset
", "rawMarkdown": "How to konw the ranking label? It seems all the code cells appear before markdown cells in the train dataset", "isDeleted": true }, { "id": 1904127, "postDate": "2022-08-18T01:36:19.670Z", "content": "train_orders.csv has the right order.
The padding method is cool, thanks!
", "rawMarkdown": "The padding method is cool, thanks!", "votes": 1 }, { "id": 1901859, "postDate": "2022-08-17T00:58:32.197Z", "content": "Thanks, for sharing! well explained.
", "rawMarkdown": "Thanks, for sharing! well explained." } ], "comments": [ { "id": 1945913, "author_name": "CroDoc", "author_url": "", "post_date": "2022-09-19T13:37:07.753000", "content": "@snaker you are back? Do you know what happened?
\nGood luck with getting a gold medal! Happy to see you back on LB.
", "votes": 1, "replies": [ { "id": 1947613, "author_name": "gezi", "author_url": "", "post_date": "2022-09-20T15:01:54.877000", "content": "Maybe there are some mistakes(data related probably) the official has made during the 1st rerun.
", "votes": 1, "replies": [] }, { "id": 1947694, "author_name": "Chenglu", "author_url": "", "post_date": "2022-09-20T15:36:56.577000", "content": "I'm changing the title back… I got alive in the 2nd rerun..
", "votes": 2, "replies": [] } ] }, { "id": 1895695, "author_name": "RoBi", "author_url": "", "post_date": "2022-08-12T10:25:21.497000", "content": "Thank you. This is really interesting. Do you mind telling more about your inspiration for the solution? How have you come to it? Would love to hear the details :)
", "votes": 1, "replies": [ { "id": 1895978, "author_name": "Chenglu", "author_url": "", "post_date": "2022-08-12T13:37:08.173000", "content": "I keep thinking about how to bring more global context information. The model can only handle 512 or 1024 length of sequence, but the number of tokens of one notebook is way larger than that. So splitting a notebook into smaller segments is the first and most straightforward way to handle this issue.
", "votes": 1, "replies": [] } ] }, { "id": 1895575, "author_name": "Ern", "author_url": "", "post_date": "2022-08-12T08:38:50.647000", "content": "Thanks for sharing! A clean and interesting solution indeed.
\nHow much improvement did your LM head and your padding strategy bring in terms of CV and/or public LB score? In our earlier experiments, our team briefly tried a padding strategy similar to yours, but it did not seem to improve on the more \"naive\" padding approach (Method 1 as you described).
", "votes": 1, "replies": [ { "id": 1895588, "author_name": "Chenglu", "author_url": "", "post_date": "2022-08-12T08:54:00.803000", "content": "The ML head boosts CV is ~ 0.015 and the padding method boost CV is ~ 0.005.
", "votes": 2, "replies": [] } ] }, { "id": 1906436, "author_name": "The Devastator", "author_url": "", "post_date": "2022-08-19T22:01:41.083000", "content": "Thanks for sharing your solution, You clearly worked hard on it.
\nCongrats!
The Devastator.
", "votes": 0, "replies": [] }, { "id": 1901854, "author_name": "Teemo", "author_url": "", "post_date": "2022-08-17T00:50:19.563000", "content": "Thanks for your sharing! Very clean solution.
\nHow do you make improvement on performance? Since in your github project says train.py get us a model with 0.87, but how do you improve the score from 0.87 to 0.91.
", "votes": 0, "replies": [ { "id": 1902942, "author_name": "Chenglu", "author_url": "", "post_date": "2022-08-17T02:33:09.693000", "content": "Train one epoch the score will be above 0.87, train more epochs(like 10) the score will be ~ 0.89 to 0.9 on both CV and LB.
\nAdd fold0 and external dataset to the training set and train 10 epochs, will achieve ~0.91 on LB
", "votes": 2, "replies": [] } ] }, { "id": 1898832, "author_name": "Tatiana Zinkina", "author_url": "", "post_date": "2022-08-14T21:33:11.250000", "content": "This is quite innovative, nice!
", "votes": 0, "replies": [] }, { "id": 1897840, "author_name": "Salitha Abeysinghe", "author_url": "", "post_date": "2022-08-14T05:47:46.370000", "content": "great work.
", "votes": 0, "replies": [] }, { "id": 1897704, "author_name": "samu2505", "author_url": "", "post_date": "2022-08-14T02:57:40.730000", "content": "Thanks for sharing the source code and it's well explained.
", "votes": 0, "replies": [] }, { "id": 1896898, "author_name": "Bilal Suppal", "author_url": "", "post_date": "2022-08-13T08:54:48.150000", "content": "great work
", "votes": 0, "replies": [] }, { "id": 1895767, "author_name": "happycube", "author_url": "", "post_date": "2022-08-12T11:27:02.477000", "content": "Nice!
\nI did a \"but we have [popular new item] at home\" version of this in the last week, where I had one classification-based output using a codebert base model. I also didn't try out the last solution for handling tokens, alas.
", "votes": 0, "replies": [] }, { "id": 1895641, "author_name": "Allohvk", "author_url": "", "post_date": "2022-08-12T09:28:43.550000", "content": "Segment ranking - V. Original way to handle long sequences instead of using Longformer BigBird etc. Hope you get a well deserved gold.
\nBelow is it something like an auxiliary task you trained on but didnt actually use in inference. Is my understanding right?
\nThere is actually the third head of my model, which is used as language modeling. The input for that head is <previous_cell_encode + <anchor_cell_encode + <next_cell_encode, positive means that the three is continuous, negative means not continuous, they are just randomly pick from the notebook(but not the previous and next ones).
Yes it's not used at inference time.
\nInstead of pretrain with language modeling, I train the language modeling along with the task.
", "votes": 1, "replies": [] } ] }, { "id": 1898682, "author_name": "", "author_url": "", "post_date": "2022-08-14T18:19:28.340000", "content": "I'm wondering what are the training samples format, since we only know \"[code1] [code2] [code3] …\" and \"[markdown1] [markdown2] …\" in the training dataset, instead of \"[markdown1] [code1] [code2], [markdown2] [code3] [code4]\". So it seems we don't know if a markdown belong to a code cell(or segment). How to do the classification?
", "votes": 1, "replies": [ { "id": 1898964, "author_name": "Chenglu", "author_url": "", "post_date": "2022-08-15T00:56:12.910000", "content": "if the real order of a notebook is like this: c1 c2 c3 c4 c5 m1 c6 c7 c8 c9, and divide into 3 segments:
segment1: m1 c1 c2 c3 cls label 0(m1 not in the segment), ranking label: skipped
\nsegment2: m1 c4 c5 c6 cls label 1 (m1 is in the segment), ranking label: 2.5
\nsegments: m1 c7 c8 c9 cls label 0, ranking label: skipped
How to konw the ranking label? It seems all the code cells appear before markdown cells in the train dataset
", "votes": 0, "replies": [] }, { "id": 1904127, "author_name": "Chenglu", "author_url": "", "post_date": "2022-08-18T01:36:19.670000", "content": "train_orders.csv has the right order.
The padding method is cool, thanks!
", "votes": 1, "replies": [] }, { "id": 1901859, "author_name": "Hyunkook Noh", "author_url": "", "post_date": "2022-08-17T00:58:32.197000", "content": "Thanks, for sharing! well explained.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1895564": "# Segment Ranking(9th on LB)\n\nThank Kaggle for hosting this game, and the community for all the sharing. It's my first NLP competition so there are not much NLP-related sharing in my solution, just task-related tricks.\n\nSource code: https://github.com/louis-she/ai4code\n\nInference Notebook: https://www.kaggle.com/code/snaker/new-rank/notebook\n\n## TL;DR\n\nOne deberta-v3-small model is enough, ranking method, subsequences(segment).\n\nMy solution only uses the ranking method. Given a markdown cell and a bunch of sorted code cells, predict the position of the markdown cell. The main difference here is that I do not use a whole notebook to rank a markdown cell, but just a part of it, which can be called a `segment`.\n\nFor instance, given a notebook of 40 code cells, we can divide it into 4 sub segments, each segment containing 10 code cells. Then we rank each markdown of the whole notebook in the 10 code cells.\n\nInteresting thoughts about this method, if we divided the 40 code cells into 40 segments, each segment contains only 1 code cell, then this method is the same as Pairwise, if do not divide the notebook(or divide it into 1 segment), then it's same as full ranking.\n\n## Segment Ranking\n\nThe full ranking method, which is also open sourced [here](https://www.kaggle.com/competitions/AI4Code/discussion/326970), is to predict a markdown cell's position in the whole code cells. The sequence feed to the model is usually like this: markdown_cell + code_cell + code_cell ... code_cell, but many of the sequence is way too long for transformer models(even for longformer), one simple method to handle the long sequence issue is to cut off the length of makdown_cell and code_cell to make the ensembled sequence length is acceptable. But the more cut-off, the more information we lose, so how can we keep the information as much as we can and also get a reasonable output sequence length?\n\nTo address the global information and sequence length balance issue, I came up with the Segment Ranking method.\n\nA segment is part of a notebook's sorted code cells. If using M for the number of code cells in a notebook, and N for the number of code cells in a segment, then a notebook can be divided into C segments, in which C equals `math.ceil(M / N)`.\n\nNow since we have multiple segments of a notebook, and markdown cells in that notebook can only belong to one of the segments, we have to first decide if a markdown belongs to a segment, and then do the ranking in that segment. So there are two heads of the model, one for classification and one for ranking. The following figure demonstrates the details.\n\n\n\nThe number of cell in segments(N) here is a super important hyperparameter. A large N will speed up training and inference, small N can bring more context information. I ended up choosing N = 16 and sequence max length = 384.\n\n## Other important Task-related tricks\n\n**Training with language modeling**\n\nThere is actually the third head of my model, which is used as language modeling. The input for that head is