First Rodeo - Observations from my First Serious Kaggle Competition
\n\nFirst, let me thank the Kaggle organizational team for a very clean and interesting competition. The issue with the leaked kernel was handled swiftly and effectively; it did not seem to have any effect on the final results. I would also like to thank the participants who graciously share a great amount of wisdom and knowledge.
\n\nI have “dabbled” on Kaggle for a few years, but never really worked diligently on a competition. This competition, I set a goal to get a medal - top 10% would be my best effort thus far. I met that goal, with a silver, top 3% solution.
\n\nI started with features from several popular kernels, mainly by Andrew https://www.kaggle.com/artgor
\n\nHis efforts got me a good start on a decent Lgm model. I tried several things - feature reduction, feature augmentation, more iterations, etc. While these efforts yielded modest improvement, I found that a better feature set was required.
\n\nI read several posts by Alexandre Sauve and realized that he had good insight into the ‘right’ approach to feature generation. His work was widely used and is located in these kernels:
\n\nhttps://www.kaggle.com/asauve/dataset-with-number-of-bonds-between-atoms\nhttps://www.kaggle.com/asauve/v7-estimation-of-mulliken-charges-with-open-babel\nhttps://www.kaggle.com/asauve/training-set-molecule-visualization\nhttps://www.kaggle.com/asauve/comparison-of-acsf-built-charges-with-open-babel
\n\nI reached out to him and he was interested in a collaboration. At this point, he had not submitted anything, but had several feature generation kernels in progress. His background in both computer science and astrophysics proved to be an incredible resource in this type of competition.
\n\nWe worked on refining the Lgb model by creating new feature sets, and some serious, careful model tuning. We took diversions into both simple neural networks and CNN. The CNN effort was interesting, because Alexandre developed a special set of ‘image-like’ features which we thought would work well with CNN. However, none of these efforts improved our base Lgm model CV.
\n\nFinal efforts consisted of building a 10-fold modeling of each type. Each model run was based on creating only 2 folds at a time, which were subsequently median merged together. I think we pushed Lgb hard while improving our results gradually throughout the competition. We probably generated almost 100 kernels, each with numerous versions, to test various aspects of our model.
\n\nThe main thing I learned in this competition is how much effort and research goes into creating a viable solution. I also learned the Lgb is a very robust, durable tool that can be tuned well to specific situations. There were a lot of great resources and pointers to scientific research in both physical chemistry and bleeding-edge ML. Many were interesting, but a lot were above my pay grade. The discussion forum and kernels proved to be a valuable source of information of how teams were addressing this very challenging problem. Domain knowledge in this competition definitely would be an advantage. As the leading solutions emerge, I expect that the top tier of teams will be very deep in domain-specific information.
", "messages": [ { "id": "612825", "postDate": "08/30/2019 03:06:07", "content": "First Rodeo - Observations from my First Serious Kaggle Competition
\n\nFirst, let me thank the Kaggle organizational team for a very clean and interesting competition. The issue with the leaked kernel was handled swiftly and effectively; it did not seem to have any effect on the final results. I would also like to thank the participants who graciously share a great amount of wisdom and knowledge.
\n\nI have “dabbled” on Kaggle for a few years, but never really worked diligently on a competition. This competition, I set a goal to get a medal - top 10% would be my best effort thus far. I met that goal, with a silver, top 3% solution.
\n\nI started with features from several popular kernels, mainly by Andrew https://www.kaggle.com/artgor
\n\nHis efforts got me a good start on a decent Lgm model. I tried several things - feature reduction, feature augmentation, more iterations, etc. While these efforts yielded modest improvement, I found that a better feature set was required.
\n\nI read several posts by Alexandre Sauve and realized that he had good insight into the ‘right’ approach to feature generation. His work was widely used and is located in these kernels:
\n\nhttps://www.kaggle.com/asauve/dataset-with-number-of-bonds-between-atoms\nhttps://www.kaggle.com/asauve/v7-estimation-of-mulliken-charges-with-open-babel\nhttps://www.kaggle.com/asauve/training-set-molecule-visualization\nhttps://www.kaggle.com/asauve/comparison-of-acsf-built-charges-with-open-babel
\n\nI reached out to him and he was interested in a collaboration. At this point, he had not submitted anything, but had several feature generation kernels in progress. His background in both computer science and astrophysics proved to be an incredible resource in this type of competition.
\n\nWe worked on refining the Lgb model by creating new feature sets, and some serious, careful model tuning. We took diversions into both simple neural networks and CNN. The CNN effort was interesting, because Alexandre developed a special set of ‘image-like’ features which we thought would work well with CNN. However, none of these efforts improved our base Lgm model CV.
\n\nFinal efforts consisted of building a 10-fold modeling of each type. Each model run was based on creating only 2 folds at a time, which were subsequently median merged together. I think we pushed Lgb hard while improving our results gradually throughout the competition. We probably generated almost 100 kernels, each with numerous versions, to test various aspects of our model.
\n\nThe main thing I learned in this competition is how much effort and research goes into creating a viable solution. I also learned the Lgb is a very robust, durable tool that can be tuned well to specific situations. There were a lot of great resources and pointers to scientific research in both physical chemistry and bleeding-edge ML. Many were interesting, but a lot were above my pay grade. The discussion forum and kernels proved to be a valuable source of information of how teams were addressing this very challenging problem. Domain knowledge in this competition definitely would be an advantage. As the leading solutions emerge, I expect that the top tier of teams will be very deep in domain-specific information.
", "rawMarkdown": "**First Rodeo - Observations from my First Serious Kaggle Competition**\n\nFirst, let me thank the Kaggle organizational team for a very clean and interesting competition. The issue with the leaked kernel was handled swiftly and effectively; it did not seem to have any effect on the final results. I would also like to thank the participants who graciously share a great amount of wisdom and knowledge.\n\nI have “dabbled” on Kaggle for a few years, but never really worked diligently on a competition. This competition, I set a goal to get a medal - top 10% would be my best effort thus far. I met that goal, with a silver, top 3% solution.\n\nI started with features from several popular kernels, mainly by Andrew https://www.kaggle.com/artgor\n\nHis efforts got me a good start on a decent Lgm model. I tried several things - feature reduction, feature augmentation, more iterations, etc. While these efforts yielded modest improvement, I found that a better feature set was required.\n\nI read several posts by Alexandre Sauve and realized that he had good insight into the ‘right’ approach to feature generation. His work was widely used and is located in these kernels:\n\nhttps://www.kaggle.com/asauve/dataset-with-number-of-bonds-between-atoms\nhttps://www.kaggle.com/asauve/v7-estimation-of-mulliken-charges-with-open-babel\nhttps://www.kaggle.com/asauve/training-set-molecule-visualization\nhttps://www.kaggle.com/asauve/comparison-of-acsf-built-charges-with-open-babel\n\n\n\nI reached out to him and he was interested in a collaboration. At this point, he had not submitted anything, but had several feature generation kernels in progress. His background in both computer science and astrophysics proved to be an incredible resource in this type of competition. \n\nWe worked on refining the Lgb model by creating new feature sets, and some serious, careful model tuning. We took diversions into both simple neural networks and CNN. The CNN effort was interesting, because Alexandre developed a special set of ‘image-like’ features which we thought would work well with CNN. However, none of these efforts improved our base Lgm model CV.\n\nFinal efforts consisted of building a 10-fold modeling of each type. Each model run was based on creating only 2 folds at a time, which were subsequently median merged together. I think we pushed Lgb hard while improving our results gradually throughout the competition. We probably generated almost 100 kernels, each with numerous versions, to test various aspects of our model.\n\nThe main thing I learned in this competition is how much effort and research goes into creating a viable solution. I also learned the Lgb is a very robust, durable tool that can be tuned well to specific situations. There were a lot of great resources and pointers to scientific research in both physical chemistry and bleeding-edge ML. Many were interesting, but a lot were above my pay grade. The discussion forum and kernels proved to be a valuable source of information of how teams were addressing this very challenging problem. Domain knowledge in this competition definitely would be an advantage. As the leading solutions emerge, I expect that the top tier of teams will be very deep in domain-specific information.", "votes": null }, { "id": "612844", "postDate": "08/30/2019 03:24:26", "content": "Congratulations on getting your first medal!
", "rawMarkdown": "Congratulations on getting your first medal!", "votes": null }, { "id": "612877", "postDate": "08/30/2019 03:59:38", "content": "And congratulations again on your first gold. Your kernels have been an inspiration to me and many others!
", "rawMarkdown": "And congratulations again on your first gold. Your kernels have been an inspiration to me and many others!", "votes": null }, { "id": "612884", "postDate": "08/30/2019 04:06:13", "content": "Thank you!
", "rawMarkdown": "Thank you!", "votes": null }, { "id": "613581", "postDate": "08/30/2019 16:26:22", "content": "Amazing to see newcomers rising like you @wjholst . It inspires me to do the same. Thanks for the learning lessons and congrats for your medal! Well deserved.
", "rawMarkdown": "Amazing to see newcomers rising like you @wjholst . It inspires me to do the same. Thanks for the learning lessons and congrats for your medal! Well deserved.", "votes": null }, { "id": "613608", "postDate": "08/30/2019 16:51:15", "content": "Thanks @michelml ! The competition was a grind but worth it in the end. Because I had tried several before, I learned that a serious effort is required to actually do well. The competitions are a marathon - it is easy to get discouraged part way through when nothing you try seems to help. At this point, I found it necessary to read and study more kernels and discussions. There are gems throughout which really help out when you hit a wall.
", "rawMarkdown": "Thanks @michelml ! The competition was a grind but worth it in the end. Because I had tried several before, I learned that a serious effort is required to actually do well. The competitions are a marathon - it is easy to get discouraged part way through when nothing you try seems to help. At this point, I found it necessary to read and study more kernels and discussions. There are gems throughout which really help out when you hit a wall.", "votes": null }, { "id": "613633", "postDate": "08/30/2019 17:18:45", "content": "It is so nice to witness this camaradery among Kagglers. It's sincerely beautiful. All learning together.
", "rawMarkdown": "It is so nice to witness this camaradery among Kagglers. It's sincerely beautiful. All learning together.", "votes": null }, { "id": "613635", "postDate": "08/30/2019 17:19:43", "content": "This is what I feel too. I'm participating to the Recursion Pharma competition and felt loads of ups and downs - and there's a month left to go haha.
", "rawMarkdown": "This is what I feel too. I'm participating to the Recursion Pharma competition and felt loads of ups and downs - and there's a month left to go haha.", "votes": null }, { "id": "613708", "postDate": "08/30/2019 18:48:55", "content": "Also, congratulations on your top 15% finish. Well done!
", "rawMarkdown": "Also, congratulations on your top 15% finish. Well done!", "votes": null } ], "comments": [ { "id": 612844, "author_name": "artgor", "author_url": "", "post_date": "08/30/2019 03:24:26", "content": "Congratulations on getting your first medal!
", "votes": null, "replies": [ { "id": 612877, "author_name": "wjholst", "author_url": "", "post_date": "08/30/2019 03:59:38", "content": "And congratulations again on your first gold. Your kernels have been an inspiration to me and many others!
", "votes": null, "replies": [] }, { "id": 612884, "author_name": "artgor", "author_url": "", "post_date": "08/30/2019 04:06:13", "content": "Thank you!
", "votes": null, "replies": [] }, { "id": 613633, "author_name": "michelml", "author_url": "", "post_date": "08/30/2019 17:18:45", "content": "It is so nice to witness this camaradery among Kagglers. It's sincerely beautiful. All learning together.
", "votes": null, "replies": [] } ] }, { "id": 613581, "author_name": "michelml", "author_url": "", "post_date": "08/30/2019 16:26:22", "content": "Amazing to see newcomers rising like you @wjholst . It inspires me to do the same. Thanks for the learning lessons and congrats for your medal! Well deserved.
", "votes": null, "replies": [ { "id": 613608, "author_name": "wjholst", "author_url": "", "post_date": "08/30/2019 16:51:15", "content": "Thanks @michelml ! The competition was a grind but worth it in the end. Because I had tried several before, I learned that a serious effort is required to actually do well. The competitions are a marathon - it is easy to get discouraged part way through when nothing you try seems to help. At this point, I found it necessary to read and study more kernels and discussions. There are gems throughout which really help out when you hit a wall.
", "votes": null, "replies": [] }, { "id": 613635, "author_name": "michelml", "author_url": "", "post_date": "08/30/2019 17:19:43", "content": "This is what I feel too. I'm participating to the Recursion Pharma competition and felt loads of ups and downs - and there's a month left to go haha.
", "votes": null, "replies": [] }, { "id": 613708, "author_name": "wjholst", "author_url": "", "post_date": "08/30/2019 18:48:55", "content": "Also, congratulations on your top 15% finish. Well done!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "612825": "**First Rodeo - Observations from my First Serious Kaggle Competition**\n\nFirst, let me thank the Kaggle organizational team for a very clean and interesting competition. The issue with the leaked kernel was handled swiftly and effectively; it did not seem to have any effect on the final results. I would also like to thank the participants who graciously share a great amount of wisdom and knowledge.\n\nI have “dabbled” on Kaggle for a few years, but never really worked diligently on a competition. This competition, I set a goal to get a medal - top 10% would be my best effort thus far. I met that goal, with a silver, top 3% solution.\n\nI started with features from several popular kernels, mainly by Andrew https://www.kaggle.com/artgor\n\nHis efforts got me a good start on a decent Lgm model. I tried several things - feature reduction, feature augmentation, more iterations, etc. While these efforts yielded modest improvement, I found that a better feature set was required.\n\nI read several posts by Alexandre Sauve and realized that he had good insight into the ‘right’ approach to feature generation. His work was widely used and is located in these kernels:\n\nhttps://www.kaggle.com/asauve/dataset-with-number-of-bonds-between-atoms\nhttps://www.kaggle.com/asauve/v7-estimation-of-mulliken-charges-with-open-babel\nhttps://www.kaggle.com/asauve/training-set-molecule-visualization\nhttps://www.kaggle.com/asauve/comparison-of-acsf-built-charges-with-open-babel\n\n\n\nI reached out to him and he was interested in a collaboration. At this point, he had not submitted anything, but had several feature generation kernels in progress. His background in both computer science and astrophysics proved to be an incredible resource in this type of competition. \n\nWe worked on refining the Lgb model by creating new feature sets, and some serious, careful model tuning. We took diversions into both simple neural networks and CNN. The CNN effort was interesting, because Alexandre developed a special set of ‘image-like’ features which we thought would work well with CNN. However, none of these efforts improved our base Lgm model CV.\n\nFinal efforts consisted of building a 10-fold modeling of each type. Each model run was based on creating only 2 folds at a time, which were subsequently median merged together. I think we pushed Lgb hard while improving our results gradually throughout the competition. We probably generated almost 100 kernels, each with numerous versions, to test various aspects of our model.\n\nThe main thing I learned in this competition is how much effort and research goes into creating a viable solution. I also learned the Lgb is a very robust, durable tool that can be tuned well to specific situations. There were a lot of great resources and pointers to scientific research in both physical chemistry and bleeding-edge ML. Many were interesting, but a lot were above my pay grade. The discussion forum and kernels proved to be a valuable source of information of how teams were addressing this very challenging problem. Domain knowledge in this competition definitely would be an advantage. As the leading solutions emerge, I expect that the top tier of teams will be very deep in domain-specific information.", "612844": "Congratulations on getting your first medal!", "612877": "And congratulations again on your first gold. Your kernels have been an inspiration to me and many others!", "612884": "Thank you!", "613581": "Amazing to see newcomers rising like you @wjholst . It inspires me to do the same. Thanks for the learning lessons and congrats for your medal! Well deserved.", "613608": "Thanks @michelml ! The competition was a grind but worth it in the end. Because I had tried several before, I learned that a serious effort is required to actually do well. The competitions are a marathon - it is easy to get discouraged part way through when nothing you try seems to help. At this point, I found it necessary to read and study more kernels and discussions. There are gems throughout which really help out when you hit a wall.", "613633": "It is so nice to witness this camaradery among Kagglers. It's sincerely beautiful. All learning together.", "613635": "This is what I feel too. I'm participating to the Recursion Pharma competition and felt loads of ups and downs - and there's a month left to go haha.", "613708": "Also, congratulations on your top 15% finish. Well done!" }, "source": "meta" }