Since the competition is ending in 16 hours, I figured that I will start to share things that don't directly affect your current standing, namely, I'm referring to Alex's amazing kernel: https://www.kaggle.com/shonenkov/wbf-over-tta-single-model-efficientdet and https://www.kaggle.com/shonenkov/training-efficientdet .
\n\nYou'd think that by using product, you are effectively doing 16TTA, but if you actually do a hashmap of the outputs, you will realize that in fact half of the images are the same to the other half, mathematically by some group theory tricks. So his code is effectively 8TTA.
You'd also think that Alex's kernel is deterministic since he seeded everything. But remark how he set both cudnn's benchmark and determinism to True. This is technically conflicting because cudnn's benchmarking mechanism is designed in a way that it cannot be used deterministically. So basically what torch did is that it ignored the deterministic option silently. Don't believe it? Write a pseudo-labeling kernel for your EfficientDet model and submit the same commit to the leaderboard twice. You will get different results.
(Fun fact: I turned off benchmarking and the results were both worse and slower. I'm wondering it it's because I trained the model with benchmarking on. This maybe one of those THC magics that we may never fully understand.
", "messages": [ { "id": "957296", "postDate": "08/04/2020 07:52:14", "content": "Since the competition is ending in 16 hours, I figured that I will start to share things that don't directly affect your current standing, namely, I'm referring to Alex's amazing kernel: https://www.kaggle.com/shonenkov/wbf-over-tta-single-model-efficientdet and https://www.kaggle.com/shonenkov/training-efficientdet .
\n\nYou'd think that by using product, you are effectively doing 16TTA, but if you actually do a hashmap of the outputs, you will realize that in fact half of the images are the same to the other half, mathematically by some group theory tricks. So his code is effectively 8TTA.
You'd also think that Alex's kernel is deterministic since he seeded everything. But remark how he set both cudnn's benchmark and determinism to True. This is technically conflicting because cudnn's benchmarking mechanism is designed in a way that it cannot be used deterministically. So basically what torch did is that it ignored the deterministic option silently. Don't believe it? Write a pseudo-labeling kernel for your EfficientDet model and submit the same commit to the leaderboard twice. You will get different results.
(Fun fact: I turned off benchmarking and the results were both worse and slower. I'm wondering it it's because I trained the model with benchmarking on. This maybe one of those THC magics that we may never fully understand.
", "rawMarkdown": "Since the competition is ending in 16 hours, I figured that I will start to share things that don't directly affect your current standing, namely, I'm referring to Alex's amazing kernel: https://www.kaggle.com/shonenkov/wbf-over-tta-single-model-efficientdet and https://www.kaggle.com/shonenkov/training-efficientdet .\n\n1. TTA\n\nYou'd think that by using `product`, you are effectively doing 16TTA, but if you actually do a hashmap of the outputs, you will realize that in fact half of the images are the same to the other half, mathematically by some group theory tricks. So his code is effectively 8TTA.\n\n2. Determinism\n\nYou'd also think that Alex's kernel is deterministic since he seeded everything. But remark how he set both cudnn's benchmark and determinism to `True`. This is technically conflicting because cudnn's benchmarking mechanism is designed in a way that it cannot be used deterministically. So basically what torch did is that it ignored the deterministic option silently. Don't believe it? Write a pseudo-labeling kernel for your EfficientDet model and submit the **same** commit to the leaderboard twice. You will get different results.\n\n(Fun fact: I turned off benchmarking and the results were both worse and slower. I'm wondering it it's because I trained the model with benchmarking on. This maybe one of those THC magics that we may never fully understand.", "votes": null }, { "id": "957310", "postDate": "08/04/2020 08:02:17", "content": "> Determinism... Don't believe it?
\n\nDamn, it's not obvious behavior :( There is no warning.
\n\nI have also noticed that TTA is redundant in Kernel.
", "rawMarkdown": "> Determinism... Don't believe it?\n\nDamn, it's not obvious behavior :( There is no warning.\n\nI have also noticed that TTA is redundant in Kernel.", "votes": null }, { "id": "958296", "postDate": "08/04/2020 23:11:59", "content": "your are totally right on Determinism, so good luck to everyone
", "rawMarkdown": "your are totally right on Determinism, so good luck to everyone", "votes": null } ], "comments": [ { "id": 957310, "author_name": "sergeyzlobin", "author_url": "", "post_date": "08/04/2020 08:02:17", "content": "> Determinism... Don't believe it?
\n\nDamn, it's not obvious behavior :( There is no warning.
\n\nI have also noticed that TTA is redundant in Kernel.
", "votes": null, "replies": [] }, { "id": 958296, "author_name": "creatrol", "author_url": "", "post_date": "08/04/2020 23:11:59", "content": "your are totally right on Determinism, so good luck to everyone
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "957296": "Since the competition is ending in 16 hours, I figured that I will start to share things that don't directly affect your current standing, namely, I'm referring to Alex's amazing kernel: https://www.kaggle.com/shonenkov/wbf-over-tta-single-model-efficientdet and https://www.kaggle.com/shonenkov/training-efficientdet .\n\n1. TTA\n\nYou'd think that by using `product`, you are effectively doing 16TTA, but if you actually do a hashmap of the outputs, you will realize that in fact half of the images are the same to the other half, mathematically by some group theory tricks. So his code is effectively 8TTA.\n\n2. Determinism\n\nYou'd also think that Alex's kernel is deterministic since he seeded everything. But remark how he set both cudnn's benchmark and determinism to `True`. This is technically conflicting because cudnn's benchmarking mechanism is designed in a way that it cannot be used deterministically. So basically what torch did is that it ignored the deterministic option silently. Don't believe it? Write a pseudo-labeling kernel for your EfficientDet model and submit the **same** commit to the leaderboard twice. You will get different results.\n\n(Fun fact: I turned off benchmarking and the results were both worse and slower. I'm wondering it it's because I trained the model with benchmarking on. This maybe one of those THC magics that we may never fully understand.", "957310": "> Determinism... Don't believe it?\n\nDamn, it's not obvious behavior :( There is no warning.\n\nI have also noticed that TTA is redundant in Kernel.", "958296": "your are totally right on Determinism, so good luck to everyone" }, "source": "meta" }