\n The following are distribution of model trained by MSELoss.
\n
![\"distribution](\"https://i.imgur.com/bwSIsw6.jpg\")
I published here an inference notebook (www.kaggle.com/code/yuanzhezhou/ai4code-distilbert-inference-777) which is scored around 0.777. The improvement made in this notebook is that, I replaced the loss function MSELoss by L1Loss. I did this for the fact that, prediction made by model trained with MSE loss is always normally distributed, which is far from the distribution of our label.
\n The following are distribution of model trained by MSELoss.
\n
Distribution of model trained by L1Loss.
\n![\"L1Loss\"](\"https://i.imgur.com/DASKEJl.jpg\")
Here is the training notebook for obtaining the 0.777 result, or even above 0.78 . changing the NUM_TRAIN to 1000000 should do the job.
\nwww.kaggle.com/code/yuanzhezhou/ai4code-distilbert-training-777
", "messages": [ { "id": "1791089", "postDate": "05/15/2022 16:19:17", "content": "I published here an inference notebook (www.kaggle.com/code/yuanzhezhou/ai4code-distilbert-inference-777) which is scored around 0.777. The improvement made in this notebook is that, I replaced the loss function MSELoss by L1Loss. I did this for the fact that, prediction made by model trained with MSE loss is always normally distributed, which is far from the distribution of our label.
\n The following are distribution of model trained by MSELoss.
\n
Distribution of model trained by L1Loss.
\n
Here is the training notebook for obtaining the 0.777 result, or even above 0.78 . changing the NUM_TRAIN to 1000000 should do the job.
\nwww.kaggle.com/code/yuanzhezhou/ai4code-distilbert-training-777
", "rawMarkdown": "I published here an inference notebook (www.kaggle.com/code/yuanzhezhou/ai4code-distilbert-inference-777) which is scored around 0.777. The improvement made in this notebook is that, I replaced the loss function MSELoss by L1Loss. I did this for the fact that, prediction made by model trained with MSE loss is always normally distributed, which is far from the distribution of our label. \n The following are distribution of model trained by MSELoss. \n\n\n Distribution of model trained by L1Loss. \n\n\nHere is the training notebook for obtaining the 0.777 result, or even above 0.78 . changing the NUM_TRAIN to 1000000 should do the job.\n\nwww.kaggle.com/code/yuanzhezhou/ai4code-distilbert-training-777", "votes": null }, { "id": "1791340", "postDate": "05/15/2022 23:06:12", "content": "Did you use a different framework to build your \"my_own_model_markdown_file_0_0.78169.bin\" file?
", "rawMarkdown": "Did you use a different framework to build your \"my_own_model_markdown_file_0_0.78169.bin\" file?", "votes": null }, { "id": "1791356", "postDate": "05/15/2022 23:53:53", "content": "I've already been there on yours awesome DistilBert.
", "rawMarkdown": "I've already been there on yours awesome DistilBert.", "votes": null }, { "id": "1791381", "postDate": "05/16/2022 01:00:02", "content": "For \"my_own_model_markdown_file_0_0.78169.bin\", I used the weights from huggingface and the code from baseline.
", "rawMarkdown": "For \"my_own_model_markdown_file_0_0.78169.bin\", I used the weights from huggingface and the code from baseline.", "votes": null }, { "id": "1791388", "postDate": "05/16/2022 01:14:24", "content": "Can you show me a link to information on how you produced this bin file?
", "rawMarkdown": "Can you show me a link to information on how you produced this bin file?", "votes": null }, { "id": "1791394", "postDate": "05/16/2022 01:30:57", "content": "to save a model, you need the following lines.
\n\n", "rawMarkdown": "to save a model, you need the following lines. \n\n> output_model_file = f\"./my_own_model_file_{e}_{np.round(score, 5)}.bin\"\nmodel_to_save = model.module if hasattr(model, 'module') else model\ntorch.save(model_to_save.state_dict(), output_model_file)", "votes": null }, { "id": "1791397", "postDate": "05/16/2022 01:36:10", "content": "output_model_file = f\"./my_own_model_file_{e}_{np.round(score, 5)}.bin\"
\n
\n model_to_save = model.module if hasattr(model, 'module') else model
\n torch.save(model_to_save.state_dict(), output_model_file)
have fun 😏
", "rawMarkdown": "have fun 😏", "votes": null }, { "id": "1791786", "postDate": "05/16/2022 11:11:41", "content": "Thanks for sharing, may I ask about the validation metrics of your model? Like how much of them?
", "rawMarkdown": "Thanks for sharing, may I ask about the validation metrics of your model? Like how much of them?", "votes": null }, { "id": "1791874", "postDate": "05/16/2022 12:43:10", "content": "I used Kendall Tau Correlation to validate the dataset. As is shown in the checkpoint, it gets score ~0.78.
", "rawMarkdown": "I used Kendall Tau Correlation to validate the dataset. As is shown in the checkpoint, it gets score ~0.78.", "votes": null }, { "id": "1791895", "postDate": "05/16/2022 13:07:55", "content": "Ah, so this is the valid score, not your LB. I see, thank you!
", "rawMarkdown": "Ah, so this is the valid score, not your LB. I see, thank you!", "votes": null }, { "id": "1792365", "postDate": "05/16/2022 22:38:30", "content": "Thanks to sharing your insight. This is reasonable since the task metric (Kendall's tau) has a linear property with respect to the L1 distance.
\nIn precise, the true task loss (1-Kendall's tau) is proportional to the absolute difference between the correct position and the predicted position of the cell (=L1 loss) under some assumption.
\nI checked the property in the notebook
Did you use a different framework to build your \"my_own_model_markdown_file_0_0.78169.bin\" file?
", "votes": null, "replies": [ { "id": 1791381, "author_name": "yuanzhezhou", "author_url": "", "post_date": "05/16/2022 01:00:02", "content": "For \"my_own_model_markdown_file_0_0.78169.bin\", I used the weights from huggingface and the code from baseline.
", "votes": null, "replies": [] }, { "id": 1791388, "author_name": "", "author_url": "", "post_date": "05/16/2022 01:14:24", "content": "Can you show me a link to information on how you produced this bin file?
", "votes": null, "replies": [] }, { "id": 1791394, "author_name": "yuanzhezhou", "author_url": "", "post_date": "05/16/2022 01:30:57", "content": "to save a model, you need the following lines.
\n\n", "votes": null, "replies": [] } ] }, { "id": 1791356, "author_name": "mpwolke", "author_url": "", "post_date": "05/15/2022 23:53:53", "content": "output_model_file = f\"./my_own_model_file_{e}_{np.round(score, 5)}.bin\"
\n
\n model_to_save = model.module if hasattr(model, 'module') else model
\n torch.save(model_to_save.state_dict(), output_model_file)
I've already been there on yours awesome DistilBert.
", "votes": null, "replies": [ { "id": 1791397, "author_name": "yuanzhezhou", "author_url": "", "post_date": "05/16/2022 01:36:10", "content": "have fun 😏
", "votes": null, "replies": [] } ] }, { "id": 1791786, "author_name": "shinomoriaoshi", "author_url": "", "post_date": "05/16/2022 11:11:41", "content": "Thanks for sharing, may I ask about the validation metrics of your model? Like how much of them?
", "votes": null, "replies": [ { "id": 1791874, "author_name": "yuanzhezhou", "author_url": "", "post_date": "05/16/2022 12:43:10", "content": "I used Kendall Tau Correlation to validate the dataset. As is shown in the checkpoint, it gets score ~0.78.
", "votes": null, "replies": [] }, { "id": 1791895, "author_name": "shinomoriaoshi", "author_url": "", "post_date": "05/16/2022 13:07:55", "content": "Ah, so this is the valid score, not your LB. I see, thank you!
", "votes": null, "replies": [] } ] }, { "id": 1792365, "author_name": "yufuin", "author_url": "", "post_date": "05/16/2022 22:38:30", "content": "Thanks to sharing your insight. This is reasonable since the task metric (Kendall's tau) has a linear property with respect to the L1 distance.
\nIn precise, the true task loss (1-Kendall's tau) is proportional to the absolute difference between the correct position and the predicted position of the cell (=L1 loss) under some assumption.
\nI checked the property in the notebook