I've also upload this writeup as a kaggle dataset. Sorry for this inconvenience, kaggle's markdown editor can't render latex well :(
\nDisclaimer: All codes are generated by AI, mostly by GPT 5.1/5.2 High. This writeup is finished in hurry, so there maybe some mistakes. Please point them out.
", "messages": [ { "id": "3401216", "postDate": "02/03/2026 07:22:53", "content": "I've also upload this writeup as a kaggle dataset. Sorry for this inconvenience, kaggle's markdown editor can't render latex well :(
\nDisclaimer: All codes are generated by AI, mostly by GPT 5.1/5.2 High. This writeup is finished in hurry, so there maybe some mistakes. Please point them out.
", "rawMarkdown": "# Please see this link: https://github.com/LinkinPony/CSIRO/blob/main/docs/writeup.pdf\n\nI've also upload this writeup as a [kaggle dataset](https://www.kaggle.com/datasets/linkinpony/writeup-for-csiro-image2biomass-prediction/). Sorry for this inconvenience, kaggle's markdown editor can't render latex well :(\n\n\nDisclaimer: All codes are generated by AI, mostly by GPT 5.1/5.2 High. This writeup is finished in hurry, so there maybe some mistakes. Please point them out.", "votes": null }, { "id": "3401236", "postDate": "02/03/2026 08:52:42", "content": "Amazing approach sir! Thanks a lot for sharing. Reading your solution was a huge face palm moment for me as I also tried to use multi layer features from Dinov3 but failed.
\nDuring the first month itself I did an ablation study using EVA02 models, and Dinov2 and Dinov3 variants and found out many interesting insights. For none of the targets was the final layer output optimal.
\nI was trying to learn which layers, for which targets are the most useful and when, but due to the constant change in distribution the heads got unstable training and I failed. Using the simplest approach, like training individual models on each layer also would have drastically made a difference.
\nI should have read some papers like Elmo. \nAgain, huge congrats.
", "rawMarkdown": "Amazing approach sir! Thanks a lot for sharing. Reading your solution was a huge face palm moment for me as I also tried to use multi layer features from Dinov3 but failed. \n\nDuring the first month itself I did an ablation study using EVA02 models, and Dinov2 and Dinov3 variants and found out many interesting insights. For none of the targets was the final layer output optimal. \n\nI was trying to learn which layers, for which targets are the most useful and when, but due to the constant change in distribution the heads got unstable training and I failed. Using the simplest approach, like training individual models on each layer also would have drastically made a difference. \n\nI should have read some papers like Elmo. \nAgain, huge congrats.", "votes": null }, { "id": "3401267", "postDate": "02/03/2026 10:45:22", "content": "Some study shows low level layers have low-level infomation like texture, color, etc. While high level layers have semantic info. But in this contest, semantic information seems be more imporntant, for example, you can get good results by just use CLS token. I think (just I think, with no ablation study) using multi-layer is working like some kind of ensembe method rather than utilize different level information.
", "rawMarkdown": "Some study shows low level layers have low-level infomation like texture, color, etc. While high level layers have semantic info. But in this contest, semantic information seems be more imporntant, for example, you can get good results by just use CLS token. I think (just I think, with no ablation study) using multi-layer is working like some kind of ensembe method rather than utilize different level information.", "votes": null }, { "id": "3401279", "postDate": "02/03/2026 11:05:31", "content": "I agree. In fact my approach aligns with what you said. When I used task tokens (query tokens for each target like in DETR), it acted like if having a cls token for each target. In a way, The Global average pool gave the overall magnitude, and each task token had the information about how much of that total magnitude was for each target.
\nMy ablation study was based on GAPs (exactly because as you said, semantic information is indeed more useful here), or GAPs + Cls token. I didn't try using cls token alone. But I can say, GAPs of early layer were more helpful and even the heads converged faster like if getting a direct/better signal compared to last layer.
", "rawMarkdown": "I agree. In fact my approach aligns with what you said. When I used task tokens (query tokens for each target like in DETR), it acted like if having a cls token for each target. In a way, The Global average pool gave the overall magnitude, and each task token had the information about how much of that total magnitude was for each target. \n\nMy ablation study was based on GAPs (exactly because as you said, semantic information is indeed more useful here), or GAPs + Cls token. I didn't try using cls token alone. But I can say, GAPs of early layer were more helpful and even the heads converged faster like if getting a direct/better signal compared to last layer.", "votes": null }, { "id": "3401557", "postDate": "02/04/2026 02:07:41", "content": "Congratulations！ Thanks for sharing. I also use LoRA in early experiment，but I find full fine tune is better. I use lora for all block. I'd like to ask about freezing the last few layers – what yields the best results? Have you ever tried fine-tuning the entire backbone?
", "rawMarkdown": "Congratulations！ Thanks for sharing. I also use LoRA in early experiment，but I find full fine tune is better. I use lora for all block. I'd like to ask about freezing the last few layers – what yields the best results? Have you ever tried fine-tuning the entire backbone?", "votes": null }, { "id": "3401560", "postDate": "02/04/2026 02:23:10", "content": "I didn't try to do full parameter finetune due to limited GPU resource. But I think LoRA for all blocks will get better result (my 0.69 private LB model use LoRA for all blocks). Freezing low-level layers and only do LoRA on high-level just because this can reduce training time cost and then do more grid research.
", "rawMarkdown": "I didn't try to do full parameter finetune due to limited GPU resource. But I think LoRA for all blocks will get better result (my 0.69 private LB model use LoRA for all blocks). Freezing low-level layers and only do LoRA on high-level just because this can reduce training time cost and then do more grid research.", "votes": null }, { "id": "3403890", "postDate": "02/09/2026 13:46:03", "content": "Congratulations on the great result, very innovative and well-thought-out approach!\nI have a question about your workflow and experiment management,\nDo you version your code with Git and only commit \"successful\" or promising experiments? Or do you track most trials and clean things up later?\nI'm curious how you structure your experimentation process and decide what gets saved or pushed
", "rawMarkdown": "Congratulations on the great result, very innovative and well-thought-out approach!\nI have a question about your workflow and experiment management,\nDo you version your code with Git and only commit \"successful\" or promising experiments? Or do you track most trials and clean things up later?\nI'm curious how you structure your experimentation process and decide what gets saved or pushed", "votes": null }, { "id": "3403974", "postDate": "02/09/2026 16:52:13", "content": "I commited all my experiments on this repo. There are some branches here (though named somewhat arbitrarily) tracking key ideas, and some tags as snapshots of best public Leaderboard scores.Main branch is a comprehensive, configurable branch. Train configs can be found at configs/train.yaml in this dataset: https://www.kaggle.com/datasets/linkinpony/dinov3-regressor-weights (You can check versions to play with early versions). I didn't upload all experiment weights because they are very big (about 1TB). But maybe I can upload tensorboard logs later.
Amazing approach sir! Thanks a lot for sharing. Reading your solution was a huge face palm moment for me as I also tried to use multi layer features from Dinov3 but failed.
\nDuring the first month itself I did an ablation study using EVA02 models, and Dinov2 and Dinov3 variants and found out many interesting insights. For none of the targets was the final layer output optimal.
\nI was trying to learn which layers, for which targets are the most useful and when, but due to the constant change in distribution the heads got unstable training and I failed. Using the simplest approach, like training individual models on each layer also would have drastically made a difference.
\nI should have read some papers like Elmo. \nAgain, huge congrats.
", "votes": null, "replies": [ { "id": 3401267, "author_name": "linkinpony", "author_url": "", "post_date": "02/03/2026 10:45:22", "content": "Some study shows low level layers have low-level infomation like texture, color, etc. While high level layers have semantic info. But in this contest, semantic information seems be more imporntant, for example, you can get good results by just use CLS token. I think (just I think, with no ablation study) using multi-layer is working like some kind of ensembe method rather than utilize different level information.
", "votes": null, "replies": [ { "id": 3401279, "author_name": "komilparmar", "author_url": "", "post_date": "02/03/2026 11:05:31", "content": "I agree. In fact my approach aligns with what you said. When I used task tokens (query tokens for each target like in DETR), it acted like if having a cls token for each target. In a way, The Global average pool gave the overall magnitude, and each task token had the information about how much of that total magnitude was for each target.
\nMy ablation study was based on GAPs (exactly because as you said, semantic information is indeed more useful here), or GAPs + Cls token. I didn't try using cls token alone. But I can say, GAPs of early layer were more helpful and even the heads converged faster like if getting a direct/better signal compared to last layer.
", "votes": null, "replies": [] } ] } ] }, { "id": 3401557, "author_name": "forgakki", "author_url": "", "post_date": "02/04/2026 02:07:41", "content": "Congratulations！ Thanks for sharing. I also use LoRA in early experiment，but I find full fine tune is better. I use lora for all block. I'd like to ask about freezing the last few layers – what yields the best results? Have you ever tried fine-tuning the entire backbone?
", "votes": null, "replies": [ { "id": 3401560, "author_name": "linkinpony", "author_url": "", "post_date": "02/04/2026 02:23:10", "content": "I didn't try to do full parameter finetune due to limited GPU resource. But I think LoRA for all blocks will get better result (my 0.69 private LB model use LoRA for all blocks). Freezing low-level layers and only do LoRA on high-level just because this can reduce training time cost and then do more grid research.
", "votes": null, "replies": [] } ] }, { "id": 3403890, "author_name": "nizarcharrada", "author_url": "", "post_date": "02/09/2026 13:46:03", "content": "Congratulations on the great result, very innovative and well-thought-out approach!\nI have a question about your workflow and experiment management,\nDo you version your code with Git and only commit \"successful\" or promising experiments? Or do you track most trials and clean things up later?\nI'm curious how you structure your experimentation process and decide what gets saved or pushed
", "votes": null, "replies": [ { "id": 3403974, "author_name": "linkinpony", "author_url": "", "post_date": "02/09/2026 16:52:13", "content": "I commited all my experiments on this repo. There are some branches here (though named somewhat arbitrarily) tracking key ideas, and some tags as snapshots of best public Leaderboard scores.Main branch is a comprehensive, configurable branch. Train configs can be found at configs/train.yaml in this dataset: https://www.kaggle.com/datasets/linkinpony/dinov3-regressor-weights (You can check versions to play with early versions). I didn't upload all experiment weights because they are very big (about 1TB). But maybe I can upload tensorboard logs later.