Has anyone tested if the two have much difference on infering speed? Thanks!
\nI'm using tokeization on the fly and seems changing to pre seg needs big effort not sure if it could help.
Has anyone tested if the two have much difference on infering speed? Thanks!
\nI'm using tokeization on the fly and seems changing to pre seg needs big effort not sure if it could help.
My observation. Tokenization is insanely fast. I didnt find any significant benefit in pre-tokenizing (during training). In fact, in the popular code of point-wise model, the tokenization is done for the context (ie each of the 20 code cells) for each single markup, so there is lot of duplication. I moved the code tokenization to the dataset initialize function, thinking I will save a good amount of time but it didnt make any noticeable difference of even few seconds. In fact I pre-tokenized the whole thing (code+markdown) and even that barely gave a 1% improvement to the training speed.
", "rawMarkdown": "My observation. Tokenization is insanely fast. I didnt find any significant benefit in pre-tokenizing (during training). In fact, in the popular code of point-wise model, the tokenization is done for the context (ie each of the 20 code cells) for each single markup, so there is lot of duplication. I moved the code tokenization to the dataset initialize function, thinking I will save a good amount of time but it didnt make any noticeable difference of even few seconds. In fact I pre-tokenized the whole thing (code+markdown) and even that barely gave a 1% improvement to the training speed.", "votes": null }, { "id": "1872315", "postDate": "07/26/2022 20:08:49", "content": "Thanks so much @allohvk , your advice is great!
", "rawMarkdown": "Thanks so much @allohvk , your advice is great!", "votes": null }, { "id": "1874250", "postDate": "07/28/2022 06:53:22", "content": "Thanks for your kind words @goldenlock . Kindly note that my experience was more on the training aspect. Perhapes if you have huge set of more than dozen models and you have to repeat tokenization again and again during inference time, maybe there \"could\" be slight improvement by doing it only once. Secondly when I mention tokenization, I dont consider pre-processing as part of that. So cleaning/preprocessing can be done beforehand and there maybe some benefits to doing that at during dataset initialization and not at runtime. Of course you may be aware of all this but just documenting the same for the benefit of the larger community in case this is useful to someone.
\nAlso speaking of tokenization speed, one minor irritant that sooner or later one is bound to run into is the following tokenizer parallelization error/warning for which below SOF post may be helpful to the readers… https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning/72926996#72926996
did quite similar thing and no big change of speed or accuracy change when performing cleaning/pre-tokenization to code and markdown cells. Also, if possible, would like to know how could you achieve 90%+! I tried 20+ models and still far away from it. any type of hints are welcome~~
", "rawMarkdown": "did quite similar thing and no big change of speed or accuracy change when performing cleaning/pre-tokenization to code and markdown cells. Also, if possible, would like to know how could you achieve 90%+! I tried 20+ models and still far away from it. any type of hints are welcome~~", "votes": null }, { "id": "1875575", "postDate": "07/29/2022 05:53:50", "content": "also, if you use amp (fp16) (an easy way is based on pytorch_lightning), it can give some speed up so that more models can be ensembled. (the maximum hours was 8 around currently; 10 hours will fail the time-checking)
", "rawMarkdown": "also, if you use amp (fp16) (an easy way is based on pytorch_lightning), it can give some speed up so that more models can be ensembled. (the maximum hours was 8 around currently; 10 hours will fail the time-checking)", "votes": null }, { "id": "1875891", "postDate": "07/29/2022 11:49:16", "content": "@xianchaowu Thanks! I was thought amp could not speedup on kaggle's old gpu. I will have a try!
\nFor the competition I'm using heavy model based on excellent open code solutions, I will share more after close.
@goldenlock so nice of you
", "rawMarkdown": "goldenlock so nice of you~~~ really appreciate and looking forward to learning more from the future nice solution sharing~~", "votes": null }, { "id": "1877571", "postDate": "07/30/2022 19:33:51", "content": "pre tokenization can make a difference if you reuse it. For instance for training for more than one epoch. But for inference I don't really see what you would gain.
", "rawMarkdown": "pre tokenization can make a difference if you reuse it. For instance for training for more than one epoch. But for inference I don't really see what you would gain.", "votes": null } ], "comments": [ { "id": 1872144, "author_name": "allohvk", "author_url": "", "post_date": "07/26/2022 17:42:04", "content": "My observation. Tokenization is insanely fast. I didnt find any significant benefit in pre-tokenizing (during training). In fact, in the popular code of point-wise model, the tokenization is done for the context (ie each of the 20 code cells) for each single markup, so there is lot of duplication. I moved the code tokenization to the dataset initialize function, thinking I will save a good amount of time but it didnt make any noticeable difference of even few seconds. In fact I pre-tokenized the whole thing (code+markdown) and even that barely gave a 1% improvement to the training speed.
", "votes": null, "replies": [ { "id": 1872315, "author_name": "goldenlock", "author_url": "", "post_date": "07/26/2022 20:08:49", "content": "Thanks so much @allohvk , your advice is great!
", "votes": null, "replies": [] }, { "id": 1874250, "author_name": "allohvk", "author_url": "", "post_date": "07/28/2022 06:53:22", "content": "Thanks for your kind words @goldenlock . Kindly note that my experience was more on the training aspect. Perhapes if you have huge set of more than dozen models and you have to repeat tokenization again and again during inference time, maybe there \"could\" be slight improvement by doing it only once. Secondly when I mention tokenization, I dont consider pre-processing as part of that. So cleaning/preprocessing can be done beforehand and there maybe some benefits to doing that at during dataset initialization and not at runtime. Of course you may be aware of all this but just documenting the same for the benefit of the larger community in case this is useful to someone.
\nAlso speaking of tokenization speed, one minor irritant that sooner or later one is bound to run into is the following tokenizer parallelization error/warning for which below SOF post may be helpful to the readers… https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning/72926996#72926996
did quite similar thing and no big change of speed or accuracy change when performing cleaning/pre-tokenization to code and markdown cells. Also, if possible, would like to know how could you achieve 90%+! I tried 20+ models and still far away from it. any type of hints are welcome~~
", "votes": null, "replies": [] }, { "id": 1875575, "author_name": "xianchaowu", "author_url": "", "post_date": "07/29/2022 05:53:50", "content": "also, if you use amp (fp16) (an easy way is based on pytorch_lightning), it can give some speed up so that more models can be ensembled. (the maximum hours was 8 around currently; 10 hours will fail the time-checking)
", "votes": null, "replies": [ { "id": 1875891, "author_name": "goldenlock", "author_url": "", "post_date": "07/29/2022 11:49:16", "content": "@xianchaowu Thanks! I was thought amp could not speedup on kaggle's old gpu. I will have a try!
\nFor the competition I'm using heavy model based on excellent open code solutions, I will share more after close.
@goldenlock so nice of you
", "votes": null, "replies": [] }, { "id": 1877571, "author_name": "cpmpml", "author_url": "", "post_date": "07/30/2022 19:33:51", "content": "pre tokenization can make a difference if you reuse it. For instance for training for more than one epoch. But for inference I don't really see what you would gain.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1872007": "Has anyone tested if the two have much difference on infering speed? Thanks!\nI'm using tokeization on the fly and seems changing to pre seg needs big effort not sure if it could help.", "1872144": "My observation. Tokenization is insanely fast. I didnt find any significant benefit in pre-tokenizing (during training). In fact, in the popular code of point-wise model, the tokenization is done for the context (ie each of the 20 code cells) for each single markup, so there is lot of duplication. I moved the code tokenization to the dataset initialize function, thinking I will save a good amount of time but it didnt make any noticeable difference of even few seconds. In fact I pre-tokenized the whole thing (code+markdown) and even that barely gave a 1% improvement to the training speed.", "1872315": "Thanks so much @allohvk , your advice is great!", "1874250": "Thanks for your kind words @goldenlock . Kindly note that my experience was more on the training aspect. Perhapes if you have huge set of more than dozen models and you have to repeat tokenization again and again during inference time, maybe there \"could\" be slight improvement by doing it only once. Secondly when I mention tokenization, I dont consider pre-processing as part of that. So cleaning/preprocessing can be done beforehand and there maybe some benefits to doing that at during dataset initialization and not at runtime. Of course you may be aware of all this but just documenting the same for the benefit of the larger community in case this is useful to someone. \nAlso speaking of tokenization speed, one minor irritant that sooner or later one is bound to run into is the following tokenizer parallelization error/warning for which below SOF post may be helpful to the readers... https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning/72926996#72926996", "1875402": "did quite similar thing and no big change of speed or accuracy change when performing cleaning/pre-tokenization to code and markdown cells. Also, if possible, would like to know how could you achieve 90%+! I tried 20+ models and still far away from it. any type of hints are welcome~~", "1875575": "also, if you use amp (fp16) (an easy way is based on pytorch_lightning), it can give some speed up so that more models can be ensembled. (the maximum hours was 8 around currently; 10 hours will fail the time-checking)", "1875891": "xianchaowu Thanks! I was thought amp could not speedup on kaggle's old gpu. I will have a try!\nFor the competition I'm using heavy model based on excellent open code solutions, I will share more after close.", "1876540": "goldenlock so nice of you~~~ really appreciate and looking forward to learning more from the future nice solution sharing~~", "1877571": "pre tokenization can make a difference if you reuse it. For instance for training for more than one epoch. But for inference I don't really see what you would gain." }, "source": "meta" }