Thank you to Kaggle and the sponsors for hosting such a wonderful and engaging competition. It was a heated contest, and I am thrilled that we managed to hold our position amidst the fierce competition. This has been an invaluable learning experience for me. Now, let me walk you through our approach:
\nI joined the team midway through the competition and built my work on top of the excellent baseline (LB 0.67) established by my teammates. My teammates will provide a more detailed breakdown of the baseline operations later.
\nMy solution follows a two-stage pipeline: Stage 1 involves initial training, followed by Stage 2, which consists of online pseudo-labeling and Test-Time Training (TTT).
\nSingle Stream Version:\n@shanzhong8 DINOv3-Large + Single-stream: Predicts all five components during both training and inference, using a regression head with Softplus activation. The first 80% of parameters were frozen during the training process. The overall framework follows a shared feature extractor + task-specific heads paradigm, enabling both parameter efficiency and inductive bias sharing across related regression tasks. Input resolution 1024×2048, accommodating ultra–high-resolution imagery. Single model 5-folds without TTA and TTT can reach 0.76 public and 0.65 private LB.
\nDual Stream Version:\n DINOv3-Large + Dual-stream: During training, the model is supervised on all five components. However, at inference time, it only predicts Dry_Green_g, Dry_Dead_g, and Dry_Clover_g, while GDM_g and Dry_Total_g are derived through post-calculation (using Softplus-activated heads). The first 80% of parameters were frozen during the training process.
\nr^2 loss
\nStratifiedGroupKFold was used with 'sampling date' as groups and 'stat' as labels. Since the original three dates were insufficient for a 5-fold split, I subdivided the WA data into seven virtual dates. A multi-threaded brute-force search across 10 million seeds was conducted to identify the split with the most balanced distribution across all five folds.
\nResize(single-stream: 1024*2048; dual-stream: 1024^2), HorizontalFlip, VerticalFlip, RandomBrightnessContrast, and ColorJitter.
\nDue to the lack of model heterogeneity beyond DINOv3 for ensembling and the significant inference time headroom remaining, I decided to implement Test-Time Training (TTT).
\nFirst, I performed an ensemble inference of single-stream and dual-stream models (with or without TTA). To optimize efficiency, I implemented dual-GPU parallel inference, where one GPU processed the 5-fold single-stream models while the other handled the 5-fold dual-stream models. This significantly accelerated the inference process.
\nThe initial predictions then underwent post-processing—for example, multiplying clover by 0.8 and zeroing out any component values below 0.2. These processed results were used as pseudo-labels to train the best single-stream fold (based on LB/CV) using the combined full training and test sets. This TTT stage lasted for 12 epochs, with SWA (Simple Weight Averaging) applied during the final 4 epochs to prevent overfitting and improve generalization. All other training configurations remained consistent with Stage 1.
\nAdditionally, I attempted parallel TTT for both single-stream and dual-stream models on separate GPUs, but this did not yield any noticeable performance gains. The final result was a weighted blend of the Stage 1 and TTT predictions: 0.2 * Stage 1 + 0.8 * Stage 2.
\nWhat didn't work:
\nIt was incredibly difficult to find ideas that provided a stable boost in this competition. Finally, thanks to everyone—it was a pleasure competing with you all!
", "messages": [ { "id": 3398329, "postDate": "2026-01-29T01:38:33.483Z", "content": "Thank you to Kaggle and the sponsors for hosting such a wonderful and engaging competition. It was a heated contest, and I am thrilled that we managed to hold our position amidst the fierce competition. This has been an invaluable learning experience for me. Now, let me walk you through our approach:
\nI joined the team midway through the competition and built my work on top of the excellent baseline (LB 0.67) established by my teammates. My teammates will provide a more detailed breakdown of the baseline operations later.
\nMy solution follows a two-stage pipeline: Stage 1 involves initial training, followed by Stage 2, which consists of online pseudo-labeling and Test-Time Training (TTT).
\nSingle Stream Version:\n@shanzhong8 DINOv3-Large + Single-stream: Predicts all five components during both training and inference, using a regression head with Softplus activation. The first 80% of parameters were frozen during the training process. The overall framework follows a shared feature extractor + task-specific heads paradigm, enabling both parameter efficiency and inductive bias sharing across related regression tasks. Input resolution 1024×2048, accommodating ultra–high-resolution imagery. Single model 5-folds without TTA and TTT can reach 0.76 public and 0.65 private LB.
\nDual Stream Version:\n DINOv3-Large + Dual-stream: During training, the model is supervised on all five components. However, at inference time, it only predicts Dry_Green_g, Dry_Dead_g, and Dry_Clover_g, while GDM_g and Dry_Total_g are derived through post-calculation (using Softplus-activated heads). The first 80% of parameters were frozen during the training process.
\nr^2 loss
\nStratifiedGroupKFold was used with 'sampling date' as groups and 'stat' as labels. Since the original three dates were insufficient for a 5-fold split, I subdivided the WA data into seven virtual dates. A multi-threaded brute-force search across 10 million seeds was conducted to identify the split with the most balanced distribution across all five folds.
\nResize(single-stream: 1024*2048; dual-stream: 1024^2), HorizontalFlip, VerticalFlip, RandomBrightnessContrast, and ColorJitter.
\nDue to the lack of model heterogeneity beyond DINOv3 for ensembling and the significant inference time headroom remaining, I decided to implement Test-Time Training (TTT).
\nFirst, I performed an ensemble inference of single-stream and dual-stream models (with or without TTA). To optimize efficiency, I implemented dual-GPU parallel inference, where one GPU processed the 5-fold single-stream models while the other handled the 5-fold dual-stream models. This significantly accelerated the inference process.
\nThe initial predictions then underwent post-processing—for example, multiplying clover by 0.8 and zeroing out any component values below 0.2. These processed results were used as pseudo-labels to train the best single-stream fold (based on LB/CV) using the combined full training and test sets. This TTT stage lasted for 12 epochs, with SWA (Simple Weight Averaging) applied during the final 4 epochs to prevent overfitting and improve generalization. All other training configurations remained consistent with Stage 1.
\nAdditionally, I attempted parallel TTT for both single-stream and dual-stream models on separate GPUs, but this did not yield any noticeable performance gains. The final result was a weighted blend of the Stage 1 and TTT predictions: 0.2 * Stage 1 + 0.8 * Stage 2.
\nWhat didn't work:
\nIt was incredibly difficult to find ideas that provided a stable boost in this competition. Finally, thanks to everyone—it was a pleasure competing with you all!
", "rawMarkdown": "Thank you to Kaggle and the sponsors for hosting such a wonderful and engaging competition. It was a heated contest, and I am thrilled that we managed to hold our position amidst the fierce competition. This has been an invaluable learning experience for me. Now, let me walk you through our approach:\n- --\n\nI joined the team midway through the competition and built my work on top of the excellent baseline (LB 0.67) established by my teammates. My teammates will provide a more detailed breakdown of the baseline operations later.\n\n- --\n\nMy solution follows a two-stage pipeline: Stage 1 involves initial training, followed by Stage 2, which consists of online pseudo-labeling and Test-Time Training (TTT).\n\n# **Stage1**\n### **model：**\n**Single Stream Version:**\n@shanzhong8 DINOv3-Large + Single-stream: Predicts all five components during both training and inference, using a regression head with Softplus activation. The first 80% of parameters were frozen during the training process. The overall framework follows a shared feature extractor + task-specific heads paradigm, enabling both parameter efficiency and inductive bias sharing across related regression tasks. Input resolution 1024×2048, accommodating ultra–high-resolution imagery. Single model 5-folds without TTA and TTT can reach 0.76 public and 0.65 private LB. \n\n**Dual Stream Version:**\n DINOv3-Large + Dual-stream: During training, the model is supervised on all five components. However, at inference time, it only predicts Dry_Green_g, Dry_Dead_g, and Dry_Clover_g, while GDM_g and Dry_Total_g are derived through post-calculation (using Softplus-activated heads). The first 80% of parameters were frozen during the training process.\n\n### **loss：**\nr^2 loss\n\n### **CV Strategy:**\nStratifiedGroupKFold was used with 'sampling date' as groups and 'stat' as labels. Since the original three dates were insufficient for a 5-fold split, I subdivided the WA data into seven virtual dates. A multi-threaded brute-force search across 10 million seeds was conducted to identify the split with the most balanced distribution across all five folds.\n\n### **Aug (Albumentations):**\nResize(single-stream: 1024*2048; dual-stream: 1024^2), HorizontalFlip, VerticalFlip, RandomBrightnessContrast, and ColorJitter.\n\n### **Training Configuration:**\n1. Optimizer: AdamW with lr: 2e-4 and wd: 1e-2.\n2. For each fold, I maintained the top 3 best-loss checkpoints and top 3 best-score (CV) checkpoints. These 6 models were then combined using SWA (Simple Weight Averaging) to enhance generalization.\n3. For dual-stream models, HFlip or VFlip was applied to the integrated image before splitting it into its respective streams.\n\n### **Inference Configuration:**\n1. A simple average of the single-stream and dual-stream model results.\n2. Only horizontal flip TTA was utilized.\n\n- --\n\n# **Stage2(TTT)**\nDue to the lack of model heterogeneity beyond DINOv3 for ensembling and the significant inference time headroom remaining, I decided to implement Test-Time Training (TTT). \n\nFirst, I performed an ensemble inference of single-stream and dual-stream models (with or without TTA). To optimize efficiency, I implemented dual-GPU parallel inference, where one GPU processed the 5-fold single-stream models while the other handled the 5-fold dual-stream models. This significantly accelerated the inference process. \n\nThe initial predictions then underwent post-processing—for example, multiplying clover by 0.8 and zeroing out any component values below 0.2. These processed results were used as pseudo-labels to train the best single-stream fold (based on LB/CV) using the combined full training and test sets. This TTT stage lasted for 12 epochs, with SWA (Simple Weight Averaging) applied during the final 4 epochs to prevent overfitting and improve generalization. All other training configurations remained consistent with Stage 1. \n\nAdditionally, I attempted parallel TTT for both single-stream and dual-stream models on separate GPUs, but this did not yield any noticeable performance gains. The final result was a weighted blend of the Stage 1 and TTT predictions: 0.2 * Stage 1 + 0.8 * Stage 2.\n\n\n**What didn't work:**\n1. Model architectures other than DINOv3, such as SigLIP-2.\n2. Mixup, Cutmix, and their various modified variants.\n3. Auxiliary tasks and auxiliary losses.\n4. DINO embeddings combined with ML models.\n5. External datasets.\nAnd many other ideas that didn't provide any meaningful gain.\n\n### **infer code：**\n[lb 0.77 | 0.65](https://www.kaggle.com/code/cudacoding/speedup-ttt-fold2-w-o-pp-lb-0-77-0-65)\n\nIt was incredibly difficult to find ideas that provided a stable boost in this competition. Finally, thanks to everyone—it was a pleasure competing with you all!", "votes": 41 }, { "id": 3399100, "postDate": "2026-01-30T07:28:12.157Z", "content": "Bravo guys @cudacoding @shanzhong8 @haokwok2002 @weiganghuang111 @lihaoweicvch
", "rawMarkdown": "Bravo guys @cudacoding @shanzhong8 @haokwok2002 @weiganghuang111 @lihaoweicvch ", "votes": 1 }, { "id": 3398374, "postDate": "2026-01-29T03:30:53.717Z", "content": "amazing work guys!
", "rawMarkdown": "amazing work guys!", "votes": 1, "replies": [ { "id": 3398380, "postDate": "2026-01-29T03:39:09.663Z", "content": "Thanks, you too!
", "rawMarkdown": "Thanks, you too!" } ] }, { "id": 3398347, "postDate": "2026-01-29T02:11:25.910Z", "content": "Thank you for sharing this so quickly！Without Test-Time Training, what is the score for this single mode?
", "rawMarkdown": "Thank you for sharing this so quickly！Without Test-Time Training, what is the score for this single mode?", "votes": 1, "replies": [ { "id": 3398349, "postDate": "2026-01-29T02:17:30.890Z", "content": "Interestingly, we had a single-stream 5-fold model that reached 0.65 on the LB, though it was still slightly lower than the scores achieved with TTT. Aside from that, all other non-TTT models（including the ensemble) consistently hovered around 0.64.
", "rawMarkdown": "Interestingly, we had a single-stream 5-fold model that reached 0.65 on the LB, though it was still slightly lower than the scores achieved with TTT. Aside from that, all other non-TTT models（including the ensemble) consistently hovered around 0.64." } ] }, { "id": 3400766, "postDate": "2026-02-02T08:17:12.010Z", "content": "This approach is packed with clever techniques—I learned a lot from it.
", "rawMarkdown": "This approach is packed with clever techniques—I learned a lot from it." }, { "id": 3398372, "postDate": "2026-01-29T03:28:47.270Z", "content": "Great work🥰. In your solution, I learned about dual-GPU parallel inference and the Test-Time Training (TTT) approach. When using pseudo labels for TTT, my understanding is that you first run inference on the hidden test data, then perform post-processing, and afterward train the best-performing single-stream fold. The entire training lasts 12 epochs, with SWA applied during the last 4 epochs. Is my understanding correct? Additionally, I’m confused: were the rules for generating pseudo labels derived through multiple probes on the LB (leaderboard)? I’m doubtful about the effectiveness of this practice.
", "rawMarkdown": "Great work🥰. In your solution, I learned about dual-GPU parallel inference and the Test-Time Training (TTT) approach. When using pseudo labels for TTT, my understanding is that you first run inference on the hidden test data, then perform post-processing, and afterward train the best-performing single-stream fold. The entire training lasts 12 epochs, with SWA applied during the last 4 epochs. Is my understanding correct? Additionally, I’m confused: were the rules for generating pseudo labels derived through multiple probes on the LB (leaderboard)? I’m doubtful about the effectiveness of this practice.", "replies": [ { "id": 3398379, "postDate": "2026-01-29T03:38:13.617Z", "content": "Thanks😃. For the pseudo-labeling strategy, he just used a simple average of the single-stream and dual-stream models that had the best combined CV/LB performance. (He also used some post-processing like clover * 0.8; honestly, I’m with you—I don’t quite buy it either, it feels more like a gamble haha.) > Finally, regarding the logic of training the best-performing fold on both the full training set and the pseudo-labeled test set: the goal was to let the model capture the nuances between the Ground Truth and the noisy pseudo-labels before it overfits. It’s almost like an adversarial approach, in a way.
", "rawMarkdown": "Thanks😃. For the pseudo-labeling strategy, he just used a simple average of the single-stream and dual-stream models that had the best combined CV/LB performance. (He also used some post-processing like clover * 0.8; honestly, I’m with you—I don’t quite buy it either, it feels more like a gamble haha.) > Finally, regarding the logic of training the best-performing fold on both the full training set and the pseudo-labeled test set: the goal was to let the model capture the nuances between the Ground Truth and the noisy pseudo-labels before it overfits. It’s almost like an adversarial approach, in a way.", "votes": 1, "replies": [ { "id": 3398389, "postDate": "2026-01-29T03:57:35.103Z", "content": "Interesting—your solution is really simple and efficient.🥰
", "rawMarkdown": "Interesting—your solution is really simple and efficient.🥰", "votes": 1 } ] } ] }, { "id": 3398358, "postDate": "2026-01-29T02:56:01.747Z", "content": "Would you like to know what the triggering probability was for your Mixup experiment and what the value of λ was?
", "rawMarkdown": "Would you like to know what the triggering probability was for your Mixup experiment and what the value of λ was?", "replies": [ { "id": 3398360, "postDate": "2026-01-29T03:04:48.540Z", "content": "Are you saying that using mixup successfully boosted both your CV and LB?👍
", "rawMarkdown": "Are you saying that using mixup successfully boosted both your CV and LB?👍", "replies": [ { "id": 3398365, "postDate": "2026-01-29T03:16:20.270Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3398366, "postDate": "2026-01-29T03:16:48.710Z", "content": "I used it, but you said it wasn't effective. I'm not sure now because the score can only be seen in two digits, so I can't tell whether there was an actual improvement or not. At that time, I found through the experiment that the trigger probability has a significant impact on $R^2$, and it should not exceed p=0.5. So I wanted to ask about your parameters.😂
", "rawMarkdown": "I used it, but you said it wasn't effective. I'm not sure now because the score can only be seen in two digits, so I can't tell whether there was an actual improvement or not. At that time, I found through the experiment that the trigger probability has a significant impact on $R^2$, and it should not exceed p=0.5. So I wanted to ask about your parameters.😂" }, { "id": 3398384, "postDate": "2026-01-29T03:45:05.230Z", "content": "I used 0.5 for mixup probability and experimented with alpha values of 0.3 and 0.4, but with no luck. Might just be me, though. Maybe others have made it work.
", "rawMarkdown": "I used 0.5 for mixup probability and experimented with alpha values of 0.3 and 0.4, but with no luck. Might just be me, though. Maybe others have made it work." }, { "id": 3398386, "postDate": "2026-01-29T03:50:58Z", "content": "OK. My triggering probability is the highest at 0.3. The rest are less than 0.3.
", "rawMarkdown": "OK. My triggering probability is the highest at 0.3. The rest are less than 0.3." } ] } ] }, { "id": 3398388, "postDate": "2026-01-29T03:56:39.547Z", "content": "Amazing! Thank you for sharing.
", "rawMarkdown": "Amazing! Thank you for sharing.", "votes": 1 }, { "id": 3398377, "postDate": "2026-01-29T03:33:25.160Z", "content": "Nice solution, thank you for sharing.
", "rawMarkdown": "Nice solution, thank you for sharing.", "votes": 1 } ], "comments": [ { "id": 3399100, "author_name": "Navneet", "author_url": "", "post_date": "2026-01-30T07:28:12.157000", "content": "Bravo guys @cudacoding @shanzhong8 @haokwok2002 @weiganghuang111 @lihaoweicvch
", "votes": 1, "replies": [] }, { "id": 3398374, "author_name": "evnsnclr", "author_url": "", "post_date": "2026-01-29T03:30:53.717000", "content": "amazing work guys!
", "votes": 1, "replies": [ { "id": 3398380, "author_name": "Boredom", "author_url": "", "post_date": "2026-01-29T03:39:09.663000", "content": "Thanks, you too!
", "votes": 0, "replies": [] } ] }, { "id": 3398347, "author_name": "BokingChen", "author_url": "", "post_date": "2026-01-29T02:11:25.910000", "content": "Thank you for sharing this so quickly！Without Test-Time Training, what is the score for this single mode?
", "votes": 1, "replies": [ { "id": 3398349, "author_name": "Boredom", "author_url": "", "post_date": "2026-01-29T02:17:30.890000", "content": "Interestingly, we had a single-stream 5-fold model that reached 0.65 on the LB, though it was still slightly lower than the scores achieved with TTT. Aside from that, all other non-TTT models（including the ensemble) consistently hovered around 0.64.
", "votes": 0, "replies": [] } ] }, { "id": 3400766, "author_name": "sixue yang", "author_url": "", "post_date": "2026-02-02T08:17:12.010000", "content": "This approach is packed with clever techniques—I learned a lot from it.
", "votes": 0, "replies": [] }, { "id": 3398372, "author_name": "PursueML", "author_url": "", "post_date": "2026-01-29T03:28:47.270000", "content": "Great work🥰. In your solution, I learned about dual-GPU parallel inference and the Test-Time Training (TTT) approach. When using pseudo labels for TTT, my understanding is that you first run inference on the hidden test data, then perform post-processing, and afterward train the best-performing single-stream fold. The entire training lasts 12 epochs, with SWA applied during the last 4 epochs. Is my understanding correct? Additionally, I’m confused: were the rules for generating pseudo labels derived through multiple probes on the LB (leaderboard)? I’m doubtful about the effectiveness of this practice.
", "votes": 0, "replies": [ { "id": 3398379, "author_name": "Boredom", "author_url": "", "post_date": "2026-01-29T03:38:13.617000", "content": "Thanks😃. For the pseudo-labeling strategy, he just used a simple average of the single-stream and dual-stream models that had the best combined CV/LB performance. (He also used some post-processing like clover * 0.8; honestly, I’m with you—I don’t quite buy it either, it feels more like a gamble haha.) > Finally, regarding the logic of training the best-performing fold on both the full training set and the pseudo-labeled test set: the goal was to let the model capture the nuances between the Ground Truth and the noisy pseudo-labels before it overfits. It’s almost like an adversarial approach, in a way.
", "votes": 1, "replies": [ { "id": 3398389, "author_name": "PursueML", "author_url": "", "post_date": "2026-01-29T03:57:35.103000", "content": "Interesting—your solution is really simple and efficient.🥰
", "votes": 1, "replies": [] } ] } ] }, { "id": 3398358, "author_name": "water joe", "author_url": "", "post_date": "2026-01-29T02:56:01.747000", "content": "Would you like to know what the triggering probability was for your Mixup experiment and what the value of λ was?
", "votes": 0, "replies": [ { "id": 3398360, "author_name": "Boredom", "author_url": "", "post_date": "2026-01-29T03:04:48.540000", "content": "Are you saying that using mixup successfully boosted both your CV and LB?👍
", "votes": 0, "replies": [ { "id": 3398365, "author_name": "", "author_url": "", "post_date": "2026-01-29T03:16:20.270000", "content": "", "votes": 0, "replies": [] }, { "id": 3398366, "author_name": "water joe", "author_url": "", "post_date": "2026-01-29T03:16:48.710000", "content": "I used it, but you said it wasn't effective. I'm not sure now because the score can only be seen in two digits, so I can't tell whether there was an actual improvement or not. At that time, I found through the experiment that the trigger probability has a significant impact on $R^2$, and it should not exceed p=0.5. So I wanted to ask about your parameters.😂
", "votes": 0, "replies": [] }, { "id": 3398384, "author_name": "Boredom", "author_url": "", "post_date": "2026-01-29T03:45:05.230000", "content": "I used 0.5 for mixup probability and experimented with alpha values of 0.3 and 0.4, but with no luck. Might just be me, though. Maybe others have made it work.
", "votes": 0, "replies": [] }, { "id": 3398386, "author_name": "water joe", "author_url": "", "post_date": "2026-01-29T03:50:58", "content": "OK. My triggering probability is the highest at 0.3. The rest are less than 0.3.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3398388, "author_name": "TT", "author_url": "", "post_date": "2026-01-29T03:56:39.547000", "content": "Amazing! Thank you for sharing.
", "votes": 1, "replies": [] }, { "id": 3398377, "author_name": "Chakkrit Termritthikun", "author_url": "", "post_date": "2026-01-29T03:33:25.160000", "content": "Nice solution, thank you for sharing.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "3398329": "Thank you to Kaggle and the sponsors for hosting such a wonderful and engaging competition. It was a heated contest, and I am thrilled that we managed to hold our position amidst the fierce competition. This has been an invaluable learning experience for me. Now, let me walk you through our approach:\n- --\n\nI joined the team midway through the competition and built my work on top of the excellent baseline (LB 0.67) established by my teammates. My teammates will provide a more detailed breakdown of the baseline operations later.\n\n- --\n\nMy solution follows a two-stage pipeline: Stage 1 involves initial training, followed by Stage 2, which consists of online pseudo-labeling and Test-Time Training (TTT).\n\n# **Stage1**\n### **model：**\n**Single Stream Version:**\n@shanzhong8 DINOv3-Large + Single-stream: Predicts all five components during both training and inference, using a regression head with Softplus activation. The first 80% of parameters were frozen during the training process. The overall framework follows a shared feature extractor + task-specific heads paradigm, enabling both parameter efficiency and inductive bias sharing across related regression tasks. Input resolution 1024×2048, accommodating ultra–high-resolution imagery. Single model 5-folds without TTA and TTT can reach 0.76 public and 0.65 private LB. \n\n**Dual Stream Version:**\n DINOv3-Large + Dual-stream: During training, the model is supervised on all five components. However, at inference time, it only predicts Dry_Green_g, Dry_Dead_g, and Dry_Clover_g, while GDM_g and Dry_Total_g are derived through post-calculation (using Softplus-activated heads). The first 80% of parameters were frozen during the training process.\n\n### **loss：**\nr^2 loss\n\n### **CV Strategy:**\nStratifiedGroupKFold was used with 'sampling date' as groups and 'stat' as labels. Since the original three dates were insufficient for a 5-fold split, I subdivided the WA data into seven virtual dates. A multi-threaded brute-force search across 10 million seeds was conducted to identify the split with the most balanced distribution across all five folds.\n\n### **Aug (Albumentations):**\nResize(single-stream: 1024*2048; dual-stream: 1024^2), HorizontalFlip, VerticalFlip, RandomBrightnessContrast, and ColorJitter.\n\n### **Training Configuration:**\n1. Optimizer: AdamW with lr: 2e-4 and wd: 1e-2.\n2. For each fold, I maintained the top 3 best-loss checkpoints and top 3 best-score (CV) checkpoints. These 6 models were then combined using SWA (Simple Weight Averaging) to enhance generalization.\n3. For dual-stream models, HFlip or VFlip was applied to the integrated image before splitting it into its respective streams.\n\n### **Inference Configuration:**\n1. A simple average of the single-stream and dual-stream model results.\n2. Only horizontal flip TTA was utilized.\n\n- --\n\n# **Stage2(TTT)**\nDue to the lack of model heterogeneity beyond DINOv3 for ensembling and the significant inference time headroom remaining, I decided to implement Test-Time Training (TTT). \n\nFirst, I performed an ensemble inference of single-stream and dual-stream models (with or without TTA). To optimize efficiency, I implemented dual-GPU parallel inference, where one GPU processed the 5-fold single-stream models while the other handled the 5-fold dual-stream models. This significantly accelerated the inference process. \n\nThe initial predictions then underwent post-processing—for example, multiplying clover by 0.8 and zeroing out any component values below 0.2. These processed results were used as pseudo-labels to train the best single-stream fold (based on LB/CV) using the combined full training and test sets. This TTT stage lasted for 12 epochs, with SWA (Simple Weight Averaging) applied during the final 4 epochs to prevent overfitting and improve generalization. All other training configurations remained consistent with Stage 1. \n\nAdditionally, I attempted parallel TTT for both single-stream and dual-stream models on separate GPUs, but this did not yield any noticeable performance gains. The final result was a weighted blend of the Stage 1 and TTT predictions: 0.2 * Stage 1 + 0.8 * Stage 2.\n\n\n**What didn't work:**\n1. Model architectures other than DINOv3, such as SigLIP-2.\n2. Mixup, Cutmix, and their various modified variants.\n3. Auxiliary tasks and auxiliary losses.\n4. DINO embeddings combined with ML models.\n5. External datasets.\nAnd many other ideas that didn't provide any meaningful gain.\n\n### **infer code：**\n[lb 0.77 | 0.65](https://www.kaggle.com/code/cudacoding/speedup-ttt-fold2-w-o-pp-lb-0-77-0-65)\n\nIt was incredibly difficult to find ideas that provided a stable boost in this competition. Finally, thanks to everyone—it was a pleasure competing with you all!", "3399100": "Bravo guys @cudacoding @shanzhong8 @haokwok2002 @weiganghuang111 @lihaoweicvch ", "3398374": "amazing work guys!", "3398347": "Thank you for sharing this so quickly！Without Test-Time Training, what is the score for this single mode?", "3400766": "This approach is packed with clever techniques—I learned a lot from it.", "3398372": "Great work🥰. In your solution, I learned about dual-GPU parallel inference and the Test-Time Training (TTT) approach. When using pseudo labels for TTT, my understanding is that you first run inference on the hidden test data, then perform post-processing, and afterward train the best-performing single-stream fold. The entire training lasts 12 epochs, with SWA applied during the last 4 epochs. Is my understanding correct? Additionally, I’m confused: were the rules for generating pseudo labels derived through multiple probes on the LB (leaderboard)? I’m doubtful about the effectiveness of this practice.", "3398358": "Would you like to know what the triggering probability was for your Mixup experiment and what the value of λ was?", "3398388": "Amazing! Thank you for sharing.", "3398377": "Nice solution, thank you for sharing." } }