Hello everyone,
\nI recently noticed something interesting about working with a laptop: https://www.kaggle.com/code/rktqwe/lb-0-77-linear-accel-tf-bilstm-gru-attention
\nMy code seems to produce different results depending on the GPU used. In particular, I scored 0.77 points on the P100, but only 0.76 points on the T4.
\nAlthough I know that some minor deviations may occur due to differences in equipment, this difference in scores of ~0.01 seems to be quite significant.
\nCan anyone shed some light on why the P100 might perform noticeably better than the T4 in this particular competition, or perhaps with this particular BiLSTM-GRU architecture with Attention?
", "messages": [ { "id": 3229939, "postDate": "2025-06-22T10:33:10.643Z", "content": "Hello everyone,
\nI recently noticed something interesting about working with a laptop: https://www.kaggle.com/code/rktqwe/lb-0-77-linear-accel-tf-bilstm-gru-attention
\nMy code seems to produce different results depending on the GPU used. In particular, I scored 0.77 points on the P100, but only 0.76 points on the T4.
\nAlthough I know that some minor deviations may occur due to differences in equipment, this difference in scores of ~0.01 seems to be quite significant.
\nCan anyone shed some light on why the P100 might perform noticeably better than the T4 in this particular competition, or perhaps with this particular BiLSTM-GRU architecture with Attention?
", "rawMarkdown": "Hello everyone,\n\nI recently noticed something interesting about working with a laptop: https://www.kaggle.com/code/rktqwe/lb-0-77-linear-accel-tf-bilstm-gru-attention\n\nMy code seems to produce different results depending on the GPU used. In particular, I scored 0.77 points on the P100, but only 0.76 points on the T4.\n\nAlthough I know that some minor deviations may occur due to differences in equipment, this difference in scores of ~0.01 seems to be quite significant.\n\nCan anyone shed some light on why the P100 might perform noticeably better than the T4 in this particular competition, or perhaps with this particular BiLSTM-GRU architecture with Attention?", "votes": 8 }, { "id": 3230030, "postDate": "2025-06-22T12:20:35.827Z", "content": "Public LB contains approximately 1500 sequences, which is roughly equivalent to using a 5-fold CV. At least for my models, simply changing the random seed of the model can cause score variations of up to 0.02 on each fold. So, IMO 0.01 is not significant and can easily be just random noise (and of course, given only 2 decimals, the real difference can be even much smaller)
", "rawMarkdown": "Public LB contains approximately 1500 sequences, which is roughly equivalent to using a 5-fold CV. At least for my models, simply changing the random seed of the model can cause score variations of up to 0.02 on each fold. So, IMO 0.01 is not significant and can easily be just random noise (and of course, given only 2 decimals, the real difference can be even much smaller)\n", "votes": 3, "replies": [ { "id": 3230035, "postDate": "2025-06-22T12:26:19.353Z", "content": "But why do completely identical laptops on different video cards give different results? Although random is recorded the same? see in comment here : https://www.kaggle.com/code/rktqwe/lb-0-77-linear-accel-tf-bilstm-gru-attention/comments.
", "rawMarkdown": "But why do completely identical laptops on different video cards give different results? Although random is recorded the same? see in comment here : https://www.kaggle.com/code/rktqwe/lb-0-77-linear-accel-tf-bilstm-gru-attention/comments.", "replies": [ { "id": 3230051, "postDate": "2025-06-22T12:56:41.207Z", "content": "As far as I know, PyTorch doesn't guarantee reproducibility on different GPUs. Same machine, same seed, same results; but otherwise not.
", "rawMarkdown": "As far as I know, PyTorch doesn't guarantee reproducibility on different GPUs. Same machine, same seed, same results; but otherwise not.", "votes": 3 }, { "id": 3230059, "postDate": "2025-06-22T13:11:34.317Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 3230089, "postDate": "2025-06-22T13:23:18.650Z", "content": "It's strange that you are not on the leaderboard, how can your metric jump by 0.02?
", "rawMarkdown": "It's strange that you are not on the leaderboard, how can your metric jump by 0.02?", "replies": [ { "id": 3230115, "postDate": "2025-06-22T13:44:09.587Z", "content": "It's variation of 5 fold local CV (for each fold, using different seeds for the model, but the same data), which contains about the same number of sequences.
", "rawMarkdown": "It's variation of 5 fold local CV (for each fold, using different seeds for the model, but the same data), which contains about the same number of sequences.", "votes": 1 } ] }, { "id": 3230093, "postDate": "2025-06-22T13:26:54.143Z", "content": "I think the 0.01 metric is significant for this competition. I'm more interested in why P100 performs better than T4.
", "rawMarkdown": "I think the 0.01 metric is significant for this competition. I'm more interested in why P100 performs better than T4." }, { "id": 3230198, "postDate": "2025-06-22T15:20:46.497Z", "content": "Not sure why there are downvotes on this comment. I have observed the same thing though slightly smaller standard deviation (around 0.006). As an example running 15 repeats of same experiment single fold scores have a range of 0.02. Min value 0.81, max 0.83 as an example.
\nIf you don't set the seed in pytorch the score will be different on the same machine with exact same setup (I haven't actually tried setting a seed its just always different).
\nDepending on your CV scheme 0.01 might not mean much, and on public lb with ~1500 samples its definitely possible due to seeding.
", "rawMarkdown": "Not sure why there are downvotes on this comment. I have observed the same thing though slightly smaller standard deviation (around 0.006). As an example running 15 repeats of same experiment single fold scores have a range of 0.02. Min value 0.81, max 0.83 as an example.\n\nIf you don't set the seed in pytorch the score will be different on the same machine with exact same setup (I haven't actually tried setting a seed its just always different). \n\nDepending on your CV scheme 0.01 might not mean much, and on public lb with ~1500 samples its definitely possible due to seeding.", "votes": 1, "replies": [ { "id": 3230209, "postDate": "2025-06-22T15:36:19.150Z", "content": "I get the same results if I don't change anything, but if I replace the graphics card, the results change, which is logical. However, I'm more interested in the pattern of why the P100 performs better than the T4, or is it just a coincidence?
", "rawMarkdown": "\nI get the same results if I don't change anything, but if I replace the graphics card, the results change, which is logical. However, I'm more interested in the pattern of why the P100 performs better than the T4, or is it just a coincidence?" }, { "id": 3230214, "postDate": "2025-06-22T15:42:20.127Z", "content": "I strongly suspect it is a coincidence. This competition has a relatively small dataset. Without using repeated kfold cross validation I would not be able to tell if many of the things I try are really helping or not. I recommend trying the same experiment multiple times with different seeds and seeing the score differences. Having a good CV strategy is critical for making progress and from my perspective doing repeated kfold is a must.
", "rawMarkdown": "I strongly suspect it is a coincidence. This competition has a relatively small dataset. Without using repeated kfold cross validation I would not be able to tell if many of the things I try are really helping or not. I recommend trying the same experiment multiple times with different seeds and seeing the score differences. Having a good CV strategy is critical for making progress and from my perspective doing repeated kfold is a must. ", "votes": 2 } ] } ] }, { "id": 3232606, "postDate": "2025-06-26T02:12:09.237Z", "content": "I think is random noise.
", "rawMarkdown": "I think is random noise.", "votes": 1 }, { "id": 3229965, "postDate": "2025-06-22T10:53:26.340Z", "content": "Great observation, I also don't the actual reason but am gonna compare my models too with different accelerators.
\nIs there a way of putting all of the accelerators at work parallelly?
Public LB contains approximately 1500 sequences, which is roughly equivalent to using a 5-fold CV. At least for my models, simply changing the random seed of the model can cause score variations of up to 0.02 on each fold. So, IMO 0.01 is not significant and can easily be just random noise (and of course, given only 2 decimals, the real difference can be even much smaller)
", "votes": 3, "replies": [ { "id": 3230035, "author_name": "rmse_inf", "author_url": "", "post_date": "2025-06-22T12:26:19.353000", "content": "But why do completely identical laptops on different video cards give different results? Although random is recorded the same? see in comment here : https://www.kaggle.com/code/rktqwe/lb-0-77-linear-accel-tf-bilstm-gru-attention/comments.
", "votes": 0, "replies": [ { "id": 3230051, "author_name": "Herra Huu", "author_url": "", "post_date": "2025-06-22T12:56:41.207000", "content": "As far as I know, PyTorch doesn't guarantee reproducibility on different GPUs. Same machine, same seed, same results; but otherwise not.
", "votes": 3, "replies": [] }, { "id": 3230059, "author_name": "", "author_url": "", "post_date": "2025-06-22T13:11:34.317000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 3230089, "author_name": "rmse_inf", "author_url": "", "post_date": "2025-06-22T13:23:18.650000", "content": "It's strange that you are not on the leaderboard, how can your metric jump by 0.02?
", "votes": 0, "replies": [ { "id": 3230115, "author_name": "Herra Huu", "author_url": "", "post_date": "2025-06-22T13:44:09.587000", "content": "It's variation of 5 fold local CV (for each fold, using different seeds for the model, but the same data), which contains about the same number of sequences.
", "votes": 1, "replies": [] } ] }, { "id": 3230093, "author_name": "R^2 negative", "author_url": "", "post_date": "2025-06-22T13:26:54.143000", "content": "I think the 0.01 metric is significant for this competition. I'm more interested in why P100 performs better than T4.
", "votes": 0, "replies": [] }, { "id": 3230198, "author_name": "Devin Anzelmo", "author_url": "", "post_date": "2025-06-22T15:20:46.497000", "content": "Not sure why there are downvotes on this comment. I have observed the same thing though slightly smaller standard deviation (around 0.006). As an example running 15 repeats of same experiment single fold scores have a range of 0.02. Min value 0.81, max 0.83 as an example.
\nIf you don't set the seed in pytorch the score will be different on the same machine with exact same setup (I haven't actually tried setting a seed its just always different).
\nDepending on your CV scheme 0.01 might not mean much, and on public lb with ~1500 samples its definitely possible due to seeding.
", "votes": 1, "replies": [ { "id": 3230209, "author_name": "R^2 negative", "author_url": "", "post_date": "2025-06-22T15:36:19.150000", "content": "I get the same results if I don't change anything, but if I replace the graphics card, the results change, which is logical. However, I'm more interested in the pattern of why the P100 performs better than the T4, or is it just a coincidence?
", "votes": 0, "replies": [] }, { "id": 3230214, "author_name": "Devin Anzelmo", "author_url": "", "post_date": "2025-06-22T15:42:20.127000", "content": "I strongly suspect it is a coincidence. This competition has a relatively small dataset. Without using repeated kfold cross validation I would not be able to tell if many of the things I try are really helping or not. I recommend trying the same experiment multiple times with different seeds and seeing the score differences. Having a good CV strategy is critical for making progress and from my perspective doing repeated kfold is a must.
", "votes": 2, "replies": [] } ] } ] }, { "id": 3232606, "author_name": "Harper", "author_url": "", "post_date": "2025-06-26T02:12:09.237000", "content": "I think is random noise.
", "votes": 1, "replies": [] }, { "id": 3229965, "author_name": "Ammar", "author_url": "", "post_date": "2025-06-22T10:53:26.340000", "content": "Great observation, I also don't the actual reason but am gonna compare my models too with different accelerators.
\nIs there a way of putting all of the accelerators at work parallelly?