Hello Fellow Kagglers ! this is the first time i am participating actively in a kaggle competition, although watch and follow kaggle competitions for the last 5 years . Now I am having 2 difficulties
\nKaggle scoring is taking long time and sometimes fails after 3 hrs with no reason - Logs mention that successfully run in 112s or so. submission.csv is generated and is available in the output tab as well
I am using connvnext , Dino and Siglip in combination - I am learning to model and train - running out of GPUs easily and not able to do one full training and testing in one day - any tips on the model or so ?
Hello Fellow Kagglers ! this is the first time i am participating actively in a kaggle competition, although watch and follow kaggle competitions for the last 5 years . Now I am having 2 difficulties
\nKaggle scoring is taking long time and sometimes fails after 3 hrs with no reason - Logs mention that successfully run in 112s or so. submission.csv is generated and is available in the output tab as well
I am using connvnext , Dino and Siglip in combination - I am learning to model and train - running out of GPUs easily and not able to do one full training and testing in one day - any tips on the model or so ?
Hello! Be careful with being greedy on GPUs. Larger models typically work better, but not always and not if your logical flow is wrong. Focus first on getting thing right with small models, single folds, no ensembling. If this is working, try scaling up one thing at a time and determining if this is an improvement. This will help you train more models faster, allowing better design->test->feedback loops, which should improve both your model and your understanding of your model faster. Once you are 100% confident in your model, scale it up.
\nHave you run your inference script on a local GPU, or the Kaggle GPU's using the training set as a test set (not, your CV here will be high and wrong)? This allows you to stress test your inference with (roughly) as many test images as you get on the hidden dataset.
Are you using small/tiny versions of these? I train a small dinoV2 locally (RTX3070) in <1 hour, sometimes 2 if I am doing strange things. If I use the large/giant, I go oom, or it takes hours/days. So, I will only do these trainings for final models in the last month of the competition, as I can predict the type of improvements I should get from it.
Hope this helps, good luck!
", "rawMarkdown": "Hello! Be careful with being greedy on GPUs. Larger models typically work better, but not always and not if your logical flow is wrong. Focus first on getting thing right with small models, single folds, no ensembling. If this is working, try scaling up one thing at a time and determining if this is an improvement. This will help you train more models faster, allowing better design->test->feedback loops, which should improve both your model and your understanding of your model faster. Once you are 100% confident in your model, scale it up. \n\n1. Have you run your inference script on a local GPU, or the Kaggle GPU's using the training set as a test set (not, your CV here will be high and wrong)? This allows you to stress test your inference with (roughly) as many test images as you get on the hidden dataset.\n\n2. Are you using small/tiny versions of these? I train a small dinoV2 locally (RTX3070) in <1 hour, sometimes 2 if I am doing strange things. If I use the large/giant, I go oom, or it takes hours/days. So, I will only do these trainings for final models in the last month of the competition, as I can predict the type of improvements I should get from it.\n\nHope this helps, good luck!", "votes": null }, { "id": "3340060", "postDate": "11/19/2025 11:34:28", "content": "Thanks Tim for the pointers ! Yes I do run my inference in local and kaggle GPU it runs perfectly but scoring takes a lot of time.
", "rawMarkdown": "Thanks Tim for the pointers ! Yes I do run my inference in local and kaggle GPU it runs perfectly but scoring takes a lot of time.", "votes": null }, { "id": "3340086", "postDate": "11/19/2025 11:45:18", "content": "Ok. I have had some issues with the Kaggle submissions, where a notebook would refuse to save and commit without errors (seemingly due to an environment set up in which our notebooks run during submit) but I have never had a notebook that has run locally (on all 300+ image) fail a submission. My gut feeling is a OOM at some phase that you reach at 3hr, but I have no method of verifying this for you unfortunately.
\nI hope you find an answer!
", "rawMarkdown": "Ok. I have had some issues with the Kaggle submissions, where a notebook would refuse to save and commit without errors (seemingly due to an environment set up in which our notebooks run during submit) but I have never had a notebook that has run locally (on all 300+ image) fail a submission. My gut feeling is a OOM at some phase that you reach at 3hr, but I have no method of verifying this for you unfortunately. \n\nI hope you find an answer!", "votes": null }, { "id": "3340739", "postDate": "11/19/2025 17:14:44", "content": "Regarding time of the run - the logs that you see are ONLY for the run where the single test image that we can see is processed. The real test file has 800 images. So submission run time will be much longer than stated in the log that you can view. We do not get to see the log that would have been created during the actual scoring - logs would be a path that cheaters could use.
\nAs suggested by Tim a key debugging tool is to make the predictions on the train.csv rather than the test.csv. For this competition for all the notebooks I have forked and tried this has been a simple change in one line of code.
\nGPU quota is a bitch - for vision competitions you really need do training on your own machine and use kaggle only for predictions.
\nRegarding Old and Cranky - 79 here - you?
", "rawMarkdown": "Regarding time of the run - the logs that you see are ONLY for the run where the single test image that we can see is processed. The real test file has 800 images. So submission run time will be much longer than stated in the log that you can view. We do not get to see the log that would have been created during the actual scoring - logs would be a path that cheaters could use.\n\nAs suggested by [Tim](https://www.kaggle.com/timkrige) a key debugging tool is to make the predictions on the train.csv rather than the test.csv. For this competition for all the notebooks I have forked and tried this has been a simple change in one line of code. \n\nGPU quota is a bitch - for vision competitions you really need do training on your own machine and use kaggle only for predictions. \n\nRegarding Old and Cranky - 79 here - you?", "votes": null }, { "id": "3340752", "postDate": "11/19/2025 17:21:33", "content": "Wow - Such an inspiration - 41 here :-)
", "rawMarkdown": "Wow - Such an inspiration - 41 here :-)", "votes": null }, { "id": "3340771", "postDate": "11/19/2025 17:28:39", "content": "Ok - you may be cranky but not sure you qualify as Old Yet.
\nYou qualify as old if your early computer programs required a large box full of punched IBM computer cards or a very long spool roll of punched paper tape.
", "rawMarkdown": "Ok - you may be cranky but not sure you qualify as Old Yet.\n\nYou qualify as old if your early computer programs required a large box full of punched IBM computer cards or a very long spool roll of punched paper tape.", "votes": null }, { "id": "3341891", "postDate": "11/20/2025 13:50:16", "content": "So I ran the train/truth data through the test inference pipeline; I could see my predictions (attached file) are fairly close enough however the score isnt anywhere great than 0.50 . Any thoughts on the same
", "rawMarkdown": "So I ran the train/truth data through the test inference pipeline; I could see my predictions (attached file) are fairly close enough however the score isnt anywhere great than 0.50 . Any thoughts on the same", "votes": null }, { "id": "3341907", "postDate": "11/20/2025 14:00:23", "content": "There are some serious discussions about the data. As a quick summary from a beginner: there are data leaks. There are significant data skews, and inherent data mismatches that cause the models to learn patterns that are not helpful at test time. Read the discussions on local scores vs leaderboard scores and they all start to discuss fold strategies.
\nWe all have models that score >0.75 locally, and fail on the hidden data(mine drop as far as 30 points below my local score).
", "rawMarkdown": "There are some serious discussions about the data. As a quick summary from a beginner: there are data leaks. There are significant data skews, and inherent data mismatches that cause the models to learn patterns that are not helpful at test time. Read the discussions on local scores vs leaderboard scores and they all start to discuss fold strategies. \n\nWe all have models that score >0.75 locally, and fail on the hidden data(mine drop as far as 30 points below my local score).", "votes": null } ], "comments": [ { "id": 3339948, "author_name": "timkrige", "author_url": "", "post_date": "11/19/2025 10:17:45", "content": "Hello! Be careful with being greedy on GPUs. Larger models typically work better, but not always and not if your logical flow is wrong. Focus first on getting thing right with small models, single folds, no ensembling. If this is working, try scaling up one thing at a time and determining if this is an improvement. This will help you train more models faster, allowing better design->test->feedback loops, which should improve both your model and your understanding of your model faster. Once you are 100% confident in your model, scale it up.
\nHave you run your inference script on a local GPU, or the Kaggle GPU's using the training set as a test set (not, your CV here will be high and wrong)? This allows you to stress test your inference with (roughly) as many test images as you get on the hidden dataset.
Are you using small/tiny versions of these? I train a small dinoV2 locally (RTX3070) in <1 hour, sometimes 2 if I am doing strange things. If I use the large/giant, I go oom, or it takes hours/days. So, I will only do these trainings for final models in the last month of the competition, as I can predict the type of improvements I should get from it.
Hope this helps, good luck!
", "votes": null, "replies": [ { "id": 3340060, "author_name": "crankyoldmankk", "author_url": "", "post_date": "11/19/2025 11:34:28", "content": "Thanks Tim for the pointers ! Yes I do run my inference in local and kaggle GPU it runs perfectly but scoring takes a lot of time.
", "votes": null, "replies": [ { "id": 3340086, "author_name": "timkrige", "author_url": "", "post_date": "11/19/2025 11:45:18", "content": "Ok. I have had some issues with the Kaggle submissions, where a notebook would refuse to save and commit without errors (seemingly due to an environment set up in which our notebooks run during submit) but I have never had a notebook that has run locally (on all 300+ image) fail a submission. My gut feeling is a OOM at some phase that you reach at 3hr, but I have no method of verifying this for you unfortunately.
\nI hope you find an answer!
", "votes": null, "replies": [] } ] } ] }, { "id": 3340739, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/19/2025 17:14:44", "content": "Regarding time of the run - the logs that you see are ONLY for the run where the single test image that we can see is processed. The real test file has 800 images. So submission run time will be much longer than stated in the log that you can view. We do not get to see the log that would have been created during the actual scoring - logs would be a path that cheaters could use.
\nAs suggested by Tim a key debugging tool is to make the predictions on the train.csv rather than the test.csv. For this competition for all the notebooks I have forked and tried this has been a simple change in one line of code.
\nGPU quota is a bitch - for vision competitions you really need do training on your own machine and use kaggle only for predictions.
\nRegarding Old and Cranky - 79 here - you?
", "votes": null, "replies": [ { "id": 3340752, "author_name": "crankyoldmankk", "author_url": "", "post_date": "11/19/2025 17:21:33", "content": "Wow - Such an inspiration - 41 here :-)
", "votes": null, "replies": [ { "id": 3340771, "author_name": "pcjimmmy", "author_url": "", "post_date": "11/19/2025 17:28:39", "content": "Ok - you may be cranky but not sure you qualify as Old Yet.
\nYou qualify as old if your early computer programs required a large box full of punched IBM computer cards or a very long spool roll of punched paper tape.
", "votes": null, "replies": [] } ] } ] }, { "id": 3341891, "author_name": "crankyoldmankk", "author_url": "", "post_date": "11/20/2025 13:50:16", "content": "So I ran the train/truth data through the test inference pipeline; I could see my predictions (attached file) are fairly close enough however the score isnt anywhere great than 0.50 . Any thoughts on the same
", "votes": null, "replies": [ { "id": 3341907, "author_name": "timkrige", "author_url": "", "post_date": "11/20/2025 14:00:23", "content": "There are some serious discussions about the data. As a quick summary from a beginner: there are data leaks. There are significant data skews, and inherent data mismatches that cause the models to learn patterns that are not helpful at test time. Read the discussions on local scores vs leaderboard scores and they all start to discuss fold strategies.
\nWe all have models that score >0.75 locally, and fail on the hidden data(mine drop as far as 30 points below my local score).
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3339755": "Hello Fellow Kagglers ! this is the first time i am participating actively in a kaggle competition, although watch and follow kaggle competitions for the last 5 years . Now I am having 2 difficulties \n\n1. Kaggle scoring is taking long time and sometimes fails after 3 hrs with no reason - Logs mention that successfully run in 112s or so. submission.csv is generated and is available in the output tab as well \n\n2. I am using connvnext , Dino and Siglip in combination - I am learning to model and train - running out of GPUs easily and not able to do one full training and testing in one day - any tips on the model or so ?", "3339948": "Hello! Be careful with being greedy on GPUs. Larger models typically work better, but not always and not if your logical flow is wrong. Focus first on getting thing right with small models, single folds, no ensembling. If this is working, try scaling up one thing at a time and determining if this is an improvement. This will help you train more models faster, allowing better design->test->feedback loops, which should improve both your model and your understanding of your model faster. Once you are 100% confident in your model, scale it up. \n\n1. Have you run your inference script on a local GPU, or the Kaggle GPU's using the training set as a test set (not, your CV here will be high and wrong)? This allows you to stress test your inference with (roughly) as many test images as you get on the hidden dataset.\n\n2. Are you using small/tiny versions of these? I train a small dinoV2 locally (RTX3070) in <1 hour, sometimes 2 if I am doing strange things. If I use the large/giant, I go oom, or it takes hours/days. So, I will only do these trainings for final models in the last month of the competition, as I can predict the type of improvements I should get from it.\n\nHope this helps, good luck!", "3340060": "Thanks Tim for the pointers ! Yes I do run my inference in local and kaggle GPU it runs perfectly but scoring takes a lot of time.", "3340086": "Ok. I have had some issues with the Kaggle submissions, where a notebook would refuse to save and commit without errors (seemingly due to an environment set up in which our notebooks run during submit) but I have never had a notebook that has run locally (on all 300+ image) fail a submission. My gut feeling is a OOM at some phase that you reach at 3hr, but I have no method of verifying this for you unfortunately. \n\nI hope you find an answer!", "3340739": "Regarding time of the run - the logs that you see are ONLY for the run where the single test image that we can see is processed. The real test file has 800 images. So submission run time will be much longer than stated in the log that you can view. We do not get to see the log that would have been created during the actual scoring - logs would be a path that cheaters could use.\n\nAs suggested by [Tim](https://www.kaggle.com/timkrige) a key debugging tool is to make the predictions on the train.csv rather than the test.csv. For this competition for all the notebooks I have forked and tried this has been a simple change in one line of code. \n\nGPU quota is a bitch - for vision competitions you really need do training on your own machine and use kaggle only for predictions. \n\nRegarding Old and Cranky - 79 here - you?", "3340752": "Wow - Such an inspiration - 41 here :-)", "3340771": "Ok - you may be cranky but not sure you qualify as Old Yet.\n\nYou qualify as old if your early computer programs required a large box full of punched IBM computer cards or a very long spool roll of punched paper tape.", "3341891": "So I ran the train/truth data through the test inference pipeline; I could see my predictions (attached file) are fairly close enough however the score isnt anywhere great than 0.50 . Any thoughts on the same", "3341907": "There are some serious discussions about the data. As a quick summary from a beginner: there are data leaks. There are significant data skews, and inherent data mismatches that cause the models to learn patterns that are not helpful at test time. Read the discussions on local scores vs leaderboard scores and they all start to discuss fold strategies. \n\nWe all have models that score >0.75 locally, and fail on the hidden data(mine drop as far as 30 points below my local score)." }, "source": "meta" }