First of all I would like to say thank you to the organizers and all teams!\n It was very interesting and fruitful competition.
\n\nstandard: global optimization when the set of classes was set up using all unique values in target columns over the whole training set (single codebook case)
\n\nadvanced: column-wise optimization when the number of classes were equal to the number of unique values in each target column in training set (multiple codebooks case)
\n\nUsing of \"Optimizer rounder\" standalone (standard and advanced) gave significant LB (+0.02) improvement but using it jointly with \"0,1 mapping\" led to LB decreasing (see scores below). Also it is well-known that \"Optimizer rounder\" is a very risky approach from overfitting point of view. Therefore I decided finally to chose two submissions: with and w/o \"Optimizer rounder\". When private results were published I saw that I was right and \"Optimizer rounder\" made final score lower.
\n\npublic 0.43318 -> private 0.40901 (\"0,1 mapping\")\npublic 0.43050 -> private 0.40635 (\"0,1 mapping\" + \"Optimizer rounder\")\nSolution w/o \"Optimizer rounder\" allowed to push me 30 positions up on private test set
\n\nA lot of time was spent to overcome submission errors:\n- Submission CSV Not Found\n- Submission Scoring Error
\n\nI have met such errors more than 30 times. It was really annoyed. The worst thing is that there is no additional information from the system to understand that was wrong. I experimentally found that \"Submission CSV Not Found\" is usually caused by RAM memory restriction. It occures because private test set is 7 times bigger that public one. \"Submission Scoring Error\" is caused by columns with small amound of non-zero elements. The reason is features of correlation metric calculation.
\n\nI have met the problem with \"load weights\" method in keras when perform blending. It loads model weights to RAM but does not free memory after clear_session(). It is well-known issue that was discussed many times in different resources but nothing changed.\n(for example here https://github.com/keras-team/keras/issues/7271 and here https://github.com/tensorflow/tensorflow/issues/27511). \nUnfortunately the latest version of keras 2.3.1 still has this issue.
\n\nPossible reason according to @robieta is the following:\n\"The issue is with K.clear_session(), which clears the default graph. In 1.x, Keras uses the default tf graph, but in 2.0 Keras manages its own graph. So the clear session call isn't \nactually clearing the keras graph and models are accumulating there and preventing memory from being freed. It should be a simple fix to make clear_session clears the right graph in 2.0\"
\n\nI tried multiple things to overcome it including but not limited to:\n- garbage collector calling (\"del\" for all variables + \"gc.collect()\").\n- multiprocessing strategy where tensorflow and keras were imported in a separate thread as well as models with weights.
\n\nBut nothing helped. It has limited the number of models that I could use in blending. I hope that it will be fixed somewhen. Please let me know if you aware how to fix this bug.
\n\nGreat Solution and Congratulations! \n- Here you go to solve the GPU Mem issue in somewhat a smart way π Discussion
\n\nEdit Updated linkπ
", "votes": 3, "replies": [ { "id": 744274, "author_name": "DN", "author_url": "", "post_date": "2020-02-12T17:35:37.910000", "content": "Thank you! I got it π
", "votes": 0, "replies": [] }, { "id": 744345, "author_name": "Mycroft Holmes", "author_url": "", "post_date": "2020-02-12T18:54:33.693000", "content": "the thread I get is: \"What happened to Connecticut this year?\"
\n\nthis is the correct link
", "votes": 0, "replies": [] }, { "id": 744400, "author_name": "DN", "author_url": "", "post_date": "2020-02-12T19:51:56.800000", "content": "I also already had found correct link via his profile. Anyway thanks for sharing!
", "votes": 0, "replies": [] } ] }, { "id": 744251, "author_name": "Mycroft Holmes", "author_url": "", "post_date": "2020-02-12T17:23:09.263000", "content": "If somebody has managed to solve keras 2.xx memory leak please link to it here or make a new topic. I believe this affected many in this competition.
", "votes": 1, "replies": [] }, { "id": 743383, "author_name": "Ailurophile", "author_url": "", "post_date": "2020-02-12T02:42:57.880000", "content": "Congratulations!\nThanks for Sharing your Approach & Insights!! @dmitrynovikov
", "votes": 1, "replies": [] }, { "id": 743197, "author_name": "Mycroft Holmes", "author_url": "", "post_date": "2020-02-11T21:55:46.293000", "content": "I feel your Pain 2
", "votes": 1, "replies": [ { "id": 743202, "author_name": "DN", "author_url": "", "post_date": "2020-02-11T22:00:13.183000", "content": "Ordinary for tensorflow + keras I use mutiprocessing to fix such issues or to train different CV folds simultaneously on different GPUs. But in jupyter notebook it does not help. Pytorch looks much better here...
", "votes": 0, "replies": [] } ] } ], "comment_count": 8, "messages": [ { "id": 743196, "postDate": "2020-02-11T21:53:10.230Z", "content": "First of all I would like to say thank you to the organizers and all teams!\n It was very interesting and fruitful competition.
\n\nstandard: global optimization when the set of classes was set up using all unique values in target columns over the whole training set (single codebook case)
\n\nadvanced: column-wise optimization when the number of classes were equal to the number of unique values in each target column in training set (multiple codebooks case)
\n\nUsing of \"Optimizer rounder\" standalone (standard and advanced) gave significant LB (+0.02) improvement but using it jointly with \"0,1 mapping\" led to LB decreasing (see scores below). Also it is well-known that \"Optimizer rounder\" is a very risky approach from overfitting point of view. Therefore I decided finally to chose two submissions: with and w/o \"Optimizer rounder\". When private results were published I saw that I was right and \"Optimizer rounder\" made final score lower.
\n\npublic 0.43318 -> private 0.40901 (\"0,1 mapping\")\npublic 0.43050 -> private 0.40635 (\"0,1 mapping\" + \"Optimizer rounder\")\nSolution w/o \"Optimizer rounder\" allowed to push me 30 positions up on private test set
\n\nA lot of time was spent to overcome submission errors:\n- Submission CSV Not Found\n- Submission Scoring Error
\n\nI have met such errors more than 30 times. It was really annoyed. The worst thing is that there is no additional information from the system to understand that was wrong. I experimentally found that \"Submission CSV Not Found\" is usually caused by RAM memory restriction. It occures because private test set is 7 times bigger that public one. \"Submission Scoring Error\" is caused by columns with small amound of non-zero elements. The reason is features of correlation metric calculation.
\n\nI have met the problem with \"load weights\" method in keras when perform blending. It loads model weights to RAM but does not free memory after clear_session(). It is well-known issue that was discussed many times in different resources but nothing changed.\n(for example here https://github.com/keras-team/keras/issues/7271 and here https://github.com/tensorflow/tensorflow/issues/27511). \nUnfortunately the latest version of keras 2.3.1 still has this issue.
\n\nPossible reason according to @robieta is the following:\n\"The issue is with K.clear_session(), which clears the default graph. In 1.x, Keras uses the default tf graph, but in 2.0 Keras manages its own graph. So the clear session call isn't \nactually clearing the keras graph and models are accumulating there and preventing memory from being freed. It should be a simple fix to make clear_session clears the right graph in 2.0\"
\n\nI tried multiple things to overcome it including but not limited to:\n- garbage collector calling (\"del\" for all variables + \"gc.collect()\").\n- multiprocessing strategy where tensorflow and keras were imported in a separate thread as well as models with weights.
\n\nBut nothing helped. It has limited the number of models that I could use in blending. I hope that it will be fixed somewhen. Please let me know if you aware how to fix this bug.
\n\nGreat Solution and Congratulations! \n- Here you go to solve the GPU Mem issue in somewhat a smart way π Discussion
\n\nEdit Updated linkπ
", "rawMarkdown": "Great Solution and Congratulations! \n- Here you go to solve the GPU Mem issue in somewhat a smart way π [Discussion]( https://www.kaggle.com/c/google-quest-challenge/discussion/129963)\n\nEdit Updated linkπ ", "votes": 3, "replies": [ { "id": 744274, "postDate": "2020-02-12T17:35:37.910Z", "content": "Thank you! I got it π
", "rawMarkdown": "Thank you! I got it π " }, { "id": 744345, "postDate": "2020-02-12T18:54:33.693Z", "content": "the thread I get is: \"What happened to Connecticut this year?\"
\n\nthis is the correct link
", "rawMarkdown": "the thread I get is: \"What happened to Connecticut this year?\" \n\nthis is the correct [link](https://www.kaggle.com/c/google-quest-challenge/discussion/129963)" }, { "id": 744400, "postDate": "2020-02-12T19:51:56.800Z", "content": "I also already had found correct link via his profile. Anyway thanks for sharing!
", "rawMarkdown": "I also already had found correct link via his profile. Anyway thanks for sharing!" } ] }, { "id": 744251, "postDate": "2020-02-12T17:23:09.263Z", "content": "If somebody has managed to solve keras 2.xx memory leak please link to it here or make a new topic. I believe this affected many in this competition.
", "rawMarkdown": "If somebody has managed to solve keras 2.xx memory leak please link to it here or make a new topic. I believe this affected many in this competition.", "votes": 1 }, { "id": 743383, "postDate": "2020-02-12T02:42:57.880Z", "content": "Congratulations!\nThanks for Sharing your Approach & Insights!! @dmitrynovikov
", "rawMarkdown": "Congratulations!\nThanks for Sharing your Approach & Insights!! @dmitrynovikov ", "votes": 1 }, { "id": 743197, "postDate": "2020-02-11T21:55:46.293Z", "content": "I feel your Pain 2
", "rawMarkdown": "I feel your Pain 2", "votes": 1, "replies": [ { "id": 743202, "postDate": "2020-02-11T22:00:13.183Z", "content": "Ordinary for tensorflow + keras I use mutiprocessing to fix such issues or to train different CV folds simultaneously on different GPUs. But in jupyter notebook it does not help. Pytorch looks much better here...
", "rawMarkdown": "Ordinary for tensorflow + keras I use mutiprocessing to fix such issues or to train different CV folds simultaneously on different GPUs. But in jupyter notebook it does not help. Pytorch looks much better here..." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:08:21Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4297131, "solution_url": "/competitions/google-quest-challenge/writeups/dn-39th-place-solution-submission-errors-and-memor" }