\n\nIf you are using the preprocessing method from the cool pairwise notebooks shared by @yuanzhezhou , you may face this issue.
\n
For markdown preprocessing, we may have to be careful with some chars, since these chars may conatain \"ordering information\". For example, given the following tipical well-organized notebbook:
\n# Headline1\n\n<code_cells>\n\n## Headline2\n\n<code_cells>\n\n1. xxxxx\n<code cells>\n\n2. xxxxx\n<code_cells>\n\n3. xxxxx\n<code_cells>\n\n### Headline3\n\na. xxxxx\n<code_cells>\n\nb. xxxxx\n<code_cells>\n\nc. xxxxx\n<code_cells>\nIf using the preprocess_text from the notebooks, the output will be:
headline1 code_cells headline2 code_cells xxxxx code cell xxxxx code_cells xxxxx code_cells headline3 xxxxx code_cells xxxxx code_cells xxxxx code_cells\nall the \"ordering information\" is gone. When processing markdown we may want to keep some chars that have the ordering information, such as:
\nfor headlines:
\n#, ##, ###...\nfor ordered list:
\n1. 2. 3.\na. b. c.\nfor non ordered list, not ordered, but also will benefit the result for grouping.
\n*, -\nif you have more of this, please comment.
", "messages": [ { "id": 1817911, "postDate": "2022-06-12T03:15:29.750Z", "content": "\n\nIf you are using the preprocessing method from the cool pairwise notebooks shared by @yuanzhezhou , you may face this issue.
\n
For markdown preprocessing, we may have to be careful with some chars, since these chars may conatain \"ordering information\". For example, given the following tipical well-organized notebbook:
\n# Headline1\n\n<code_cells>\n\n## Headline2\n\n<code_cells>\n\n1. xxxxx\n<code cells>\n\n2. xxxxx\n<code_cells>\n\n3. xxxxx\n<code_cells>\n\n### Headline3\n\na. xxxxx\n<code_cells>\n\nb. xxxxx\n<code_cells>\n\nc. xxxxx\n<code_cells>\nIf using the preprocess_text from the notebooks, the output will be:
headline1 code_cells headline2 code_cells xxxxx code cell xxxxx code_cells xxxxx code_cells headline3 xxxxx code_cells xxxxx code_cells xxxxx code_cells\nall the \"ordering information\" is gone. When processing markdown we may want to keep some chars that have the ordering information, such as:
\nfor headlines:
\n#, ##, ###...\nfor ordered list:
\n1. 2. 3.\na. b. c.\nfor non ordered list, not ordered, but also will benefit the result for grouping.
\n*, -\nif you have more of this, please comment.
", "rawMarkdown": "> If you are using the preprocessing method from the cool pairwise [notebooks](https://www.kaggle.com/competitions/AI4Code/discussion/326473) shared by @yuanzhezhou , you may face this issue.\n\nFor markdown preprocessing, we may have to be careful with some chars, since these chars may conatain \"ordering information\". For example, given the following tipical well-organized notebbook:\n\n```markdown\n# Headline1\n\n\n\n2. xxxxx\n\n\n3. xxxxx\n\n\n### Headline3\n\na. xxxxx\n\n\nb. xxxxx\n\n\nc. xxxxx\n\n\n```\n\nIf using the `preprocess_text` from the notebooks, the output will be:\n\n```\nheadline1 code_cells headline2 code_cells xxxxx code cell xxxxx code_cells xxxxx code_cells headline3 xxxxx code_cells xxxxx code_cells xxxxx code_cells\n```\n\nall the \"ordering information\" is gone. When processing markdown we may want to keep some chars that have the ordering information, such as:\n\nfor headlines:\n```\n#, ##, ###...\n```\n\nfor ordered list:\n\n```\n1. 2. 3.\na. b. c.\n```\n\nfor non ordered list, not ordered, but also will benefit the result for grouping.\n\n```\n*, -\n```\n\nif you have more of this, please comment.\n",
"votes": 28
}
],
"comments": [],
"raw_markdown_by_id": {
"1817911": "> If you are using the preprocessing method from the cool pairwise [notebooks](https://www.kaggle.com/competitions/AI4Code/discussion/326473) shared by @yuanzhezhou , you may face this issue.\n\nFor markdown preprocessing, we may have to be careful with some chars, since these chars may conatain \"ordering information\". For example, given the following tipical well-organized notebbook:\n\n```markdown\n# Headline1\n\n\n\n## Headline2\n\n\n\n1. xxxxx\n\n\n2. xxxxx\n\n\n3. xxxxx\n\n\n### Headline3\n\na. xxxxx\n\n\nb. xxxxx\n\n\nc. xxxxx\n\n\n```\n\nIf using the `preprocess_text` from the notebooks, the output will be:\n\n```\nheadline1 code_cells headline2 code_cells xxxxx code cell xxxxx code_cells xxxxx code_cells headline3 xxxxx code_cells xxxxx code_cells xxxxx code_cells\n```\n\nall the \"ordering information\" is gone. When processing markdown we may want to keep some chars that have the ordering information, such as:\n\nfor headlines:\n```\n#, ##, ###...\n```\n\nfor ordered list:\n\n```\n1. 2. 3.\na. b. c.\n```\n\nfor non ordered list, not ordered, but also will benefit the result for grouping.\n\n```\n*, -\n```\n\nif you have more of this, please comment.\n"
}
}