So lately my score is not increasing much, here are some experiments I tried which run ~5 epochs:
\nSo lately my score is not increasing much, here are some experiments I tried which run ~5 epochs:
\nIt depends on your approach because a pairwise or pointwise model is not going to win. (in isolation at least, it might work in ensembles)
", "rawMarkdown": "It depends on your approach because a pairwise or pointwise model is not going to win. (in isolation at least, it might work in ensembles)", "votes": 1, "replies": [ { "id": 1863068, "postDate": "2022-07-20T06:24:42.680Z", "content": "yeah, and the pairwise is somehow costing much time. I focus in general idea is using the neuron network and the last neuron to predict the rank. Public kernel they ensembles with pairwise and have a boosting in score.
", "rawMarkdown": "yeah, and the pairwise is somehow costing much time. I focus in general idea is using the neuron network and the last neuron to predict the rank. Public kernel they ensembles with pairwise and have a boosting in score. " }, { "id": 1863538, "postDate": "2022-07-20T11:28:30.350Z", "content": "@locbaop, maybe @readoc is talking about the listwise approach… Did you check it?
", "rawMarkdown": "@locbaop, maybe @readoc is talking about the listwise approach... Did you check it? ", "votes": 1 }, { "id": 1863559, "postDate": "2022-07-20T11:39:38.880Z", "content": "@locbaop You can see my listwise approach here: https://www.kaggle.com/code/readoc/ai4code-deberta-v3-large-train-and-infer
\nI can get 0.76 in LB by training deberta-v3-large for ~1/8th of an epoch. Training for longer and especially increasing max_seq_len to 4096 (w. Longformer / BigBird / ETC) will probably lead to higher scores. I am betting that it should beat pairwise and pointwise because it's a superset of both.
", "rawMarkdown": "@locbaop You can see my listwise approach here: https://www.kaggle.com/code/readoc/ai4code-deberta-v3-large-train-and-infer\n\nI can get 0.76 in LB by training deberta-v3-large for ~1/8th of an epoch. Training for longer and especially increasing max_seq_len to 4096 (w. Longformer / BigBird / ETC) will probably lead to higher scores. I am betting that it should beat pairwise and pointwise because it's a superset of both. ", "votes": 1 }, { "id": 1863726, "postDate": "2022-07-20T13:18:25.677Z", "content": "Thanks @readoc and @vad13ir, I gonna try it out. Hope for the best.
", "rawMarkdown": "Thanks @readoc and @vad13ir, I gonna try it out. Hope for the best. ", "votes": 2 } ] }, { "id": 1870901, "postDate": "2022-07-26T00:25:45.553Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1863054, "author_name": "Sarthak Bhatt", "author_url": "", "post_date": "2022-07-20T06:14:48.763000", "content": "It depends on your approach because a pairwise or pointwise model is not going to win. (in isolation at least, it might work in ensembles)
", "votes": 1, "replies": [ { "id": 1863068, "author_name": "Bao Loc Pham", "author_url": "", "post_date": "2022-07-20T06:24:42.680000", "content": "yeah, and the pairwise is somehow costing much time. I focus in general idea is using the neuron network and the last neuron to predict the rank. Public kernel they ensembles with pairwise and have a boosting in score.
", "votes": 0, "replies": [] }, { "id": 1863538, "author_name": "Vadim Irtlach", "author_url": "", "post_date": "2022-07-20T11:28:30.350000", "content": "@locbaop, maybe @readoc is talking about the listwise approach… Did you check it?
", "votes": 1, "replies": [] }, { "id": 1863559, "author_name": "Sarthak Bhatt", "author_url": "", "post_date": "2022-07-20T11:39:38.880000", "content": "@locbaop You can see my listwise approach here: https://www.kaggle.com/code/readoc/ai4code-deberta-v3-large-train-and-infer
\nI can get 0.76 in LB by training deberta-v3-large for ~1/8th of an epoch. Training for longer and especially increasing max_seq_len to 4096 (w. Longformer / BigBird / ETC) will probably lead to higher scores. I am betting that it should beat pairwise and pointwise because it's a superset of both.
", "votes": 1, "replies": [] }, { "id": 1863726, "author_name": "Bao Loc Pham", "author_url": "", "post_date": "2022-07-20T13:18:25.677000", "content": "Thanks @readoc and @vad13ir, I gonna try it out. Hope for the best.
", "votes": 2, "replies": [] } ] }, { "id": 1870901, "author_name": "", "author_url": "", "post_date": "2022-07-26T00:25:45.553000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1862986": "So lately my score is not increasing much, here are some experiments I tried which run ~5 epochs:\n- Deberta v3: Convergent time is to much, and the results per each epoch is much slower than CodeBert.\n- GraphCodebert: Apparently, this is give me slightly better result than CodeBert-base.\n- Increasing max_length: same as deberta, if I set it to 128, 256, the results per each epochs is not better than 64, 96.\n- Using two encoder: I tried with two encoders and then concat two embeddings, from the discussion https://www.kaggle.com/competitions/AI4Code/discussion/336756 , but until now it's not work.\n- Markdown cleaning: I tried with my custom function which still remains the \"ordering information\", the discussion: https://www.kaggle.com/competitions/AI4Code/discussion/330384 , It give me slightly better results.\n- Pretrain CodeBert: In my case, after pretraining for nearly a day and the loss is not decreasing after 1.4 this method only give me 0.01 boosting.\n- Pretrain Tokenizer: I combine this method and pretrain-codebert gives me much worse result.\n- Modify the architecture: this only give me longer time to train but not the score.\n- Change some hyperparameters: I'm still working on it.\nI'm not using pairwise because of inference time is slow. I'm not training on TPU but GPU only on colab pro+ but seem my lack of experience and knowledge is holding me back. \n", "1863054": "It depends on your approach because a pairwise or pointwise model is not going to win. (in isolation at least, it might work in ensembles)", "1870901": "" } }