First of all thank you to the organizers for hosting this competition! I was very delighted having the opportunity to compete in a SQL competition.\nCongratulations to the LB winners as well as the winners in the prize categories!
\n\nMy personal objective for the competition was to tryout BigQuery (BQ) including the basic ML features. So I restricted myself to SQL-techniques.
\n\nThe solution can briefly be summarized as:\n- Features: turns, intersection clustering, populations per zipcode\n- Model: Detailed and general (for unknown intersections) LinReg model per target (no CV/KFolding)
\n\nDetails about the solution which led to my highest scoring submission (and what didn't work) are all documented in this kernel. I also published two other kernels during the competition and archived them here.
\n\nBesides the technical part I like to share some thoughts about my ranking and the competition.
\n\nAs you might have noticed there is a big difference between my 1st place in the BigQuery ML Models built in SQL category and my 213th Place on the LB. I can think of some possible reasons for that:\n- Few participants have chosen the \"SQL-way\" in the competition\n- I did some beginners mistakes (e.g. not setting up a proper validation set or I didn't go \"all-the-way\" analyzing the data or doing structured feature elimination)\n- Restriction on the available BQML Models (LinReg, Means and LogReg) can't compete against Tree-, Booster- and NN-Models.\n- The quota restrictions in BQ slowed down the pace. For instance, I didn't do grid search for parameters or KFolding because I could only train a few models per week. (to be fair, I need to mention that I relied entirely on the sandbox and I didn't activate the gcp credits which were provided at the beginning of the competition).
\n\nAnyway, I really love processing data with SQL. For me it feels more consistent than pandas (but that's my biased view). I really enjoyed the easily embedding of the models in SQL. I also liked the easily usable geo spatial functions of BQ. I think with the TensorFlow option there are opportunities to build more complex models.
\n\nThis competition nicely sums up my first kaggle year and points me where to go next.\n- retro 2019 - so much to learn and randomly trying out everything (like a child in a candy store)\n- goals 2020 - approaching data science projects in a more structured kind of manner.
\n\nThanks for reading.\nJohannes
", "comments": [ { "id": 711551, "author_name": "", "author_url": "", "post_date": "2020-01-06T08:13:51.413000", "content": "Good work Sir!!
", "votes": 1, "replies": [] }, { "id": 712941, "author_name": "tomigelo", "author_url": "", "post_date": "2020-01-07T18:40:18.187000", "content": "2nd place winner (BQ/SQL category) here 👋
\n\nFirst of all: congrats for the win!
\n\nI was quite surprised seeing my baseline LinReg kernel ending up at 2nd place which makes me also think that the other submissions were not using BQ/SQL at all. I will not complain, those GCP credits will definitely come in handy 😎
", "votes": 2, "replies": [ { "id": 712955, "author_name": "Jo Tom", "author_url": "", "post_date": "2020-01-07T18:55:21.043000", "content": "It is very welcome to have some credits to tackle compute intensive competitions in the future. 😎
", "votes": 1, "replies": [] } ] }, { "id": 752871, "author_name": "Danny", "author_url": "", "post_date": "2020-02-21T13:43:03.373000", "content": "Hello,I met a primary problem. It would really help me out if you could reply，thanks.\nWhy there are 11522010 rows in sample_submission.csv? In my understanding, there are 1921357 rows in test.csv , so submission.csv will have 1921357*6=11528142 rows.
", "votes": 0, "replies": [ { "id": 753457, "author_name": "Jo Tom", "author_url": "", "post_date": "2020-02-22T08:02:48.773000", "content": "Hi @bitmouxin ,
\n\nYou have to predict 6 values ('TotalTimeStopped_p20','TotalTimeStopped_p50','TotalTimeStopped_p80','DistanceToFirstStop_p20','DistanceToFirstStop_p50','DistanceToFirstStop_p80') per Test-rowid. The getting started notboke only shows how to predict one. Have a look at other notebooks on how to predict all six measures.
", "votes": 0, "replies": [] }, { "id": 753714, "author_name": "Danny", "author_url": "", "post_date": "2020-02-22T15:36:04.787000", "content": "@joatom \n Firstly, I am so grateful for your reply.\n I got that each Test-rowid should have 6 values, my question is that the number of rows of result is different between mine and Kaggle's require. \n In my code, there are 1,921,357*6=11,528,142 rows(because there are 1,921,357 rows in test.csv), but Kaggle requires 11,522,010 rows.\n Look forward to your reply. Best wishes.\n@bitmouxin \nThat is strange. If you look at an old EDA notebook e.g. https://www.kaggle.com/dcaichara/feature-engineering-and-lightgbm the shape of the test.csv used to be different. \nSorry can't help you here. Maybe the admins will reply to your other post.
", "votes": 0, "replies": [] }, { "id": 754340, "author_name": "Danny", "author_url": "", "post_date": "2020-02-23T12:30:35.997000", "content": "@joatom \n- I just found that the RowId of train.cs&test.csv is different. And RowIds from 0 to 1,921,356 are rows in test.csv, the others(bigger than 1,921,356) are in train.csv.(due to other's notebook).\n- Perhaps the file on Kaggle right now is wrong, but I can tell them by this way.\n- Thanks a lot for your suggestion. Best wishes.
", "votes": 2, "replies": [] } ] }, { "id": 714173, "author_name": "Panos", "author_url": "", "post_date": "2020-01-09T06:28:47.500000", "content": "Well done! Thanks for the summary!
", "votes": 1, "replies": [] } ], "comment_count": 9, "messages": [ { "id": 710577, "postDate": "2020-01-04T23:14:49.650Z", "content": "First of all thank you to the organizers for hosting this competition! I was very delighted having the opportunity to compete in a SQL competition.\nCongratulations to the LB winners as well as the winners in the prize categories!
\n\nMy personal objective for the competition was to tryout BigQuery (BQ) including the basic ML features. So I restricted myself to SQL-techniques.
\n\nThe solution can briefly be summarized as:\n- Features: turns, intersection clustering, populations per zipcode\n- Model: Detailed and general (for unknown intersections) LinReg model per target (no CV/KFolding)
\n\nDetails about the solution which led to my highest scoring submission (and what didn't work) are all documented in this kernel. I also published two other kernels during the competition and archived them here.
\n\nBesides the technical part I like to share some thoughts about my ranking and the competition.
\n\nAs you might have noticed there is a big difference between my 1st place in the BigQuery ML Models built in SQL category and my 213th Place on the LB. I can think of some possible reasons for that:\n- Few participants have chosen the \"SQL-way\" in the competition\n- I did some beginners mistakes (e.g. not setting up a proper validation set or I didn't go \"all-the-way\" analyzing the data or doing structured feature elimination)\n- Restriction on the available BQML Models (LinReg, Means and LogReg) can't compete against Tree-, Booster- and NN-Models.\n- The quota restrictions in BQ slowed down the pace. For instance, I didn't do grid search for parameters or KFolding because I could only train a few models per week. (to be fair, I need to mention that I relied entirely on the sandbox and I didn't activate the gcp credits which were provided at the beginning of the competition).
\n\nAnyway, I really love processing data with SQL. For me it feels more consistent than pandas (but that's my biased view). I really enjoyed the easily embedding of the models in SQL. I also liked the easily usable geo spatial functions of BQ. I think with the TensorFlow option there are opportunities to build more complex models.
\n\nThis competition nicely sums up my first kaggle year and points me where to go next.\n- retro 2019 - so much to learn and randomly trying out everything (like a child in a candy store)\n- goals 2020 - approaching data science projects in a more structured kind of manner.
\n\nThanks for reading.\nJohannes
", "rawMarkdown": "First of all thank you to the organizers for hosting this competition! I was very delighted having the opportunity to compete in a SQL competition.\nCongratulations to the LB winners as well as the winners in the prize categories!\n\nMy personal objective for the competition was to tryout BigQuery (BQ) including the basic ML features. So I restricted myself to SQL-techniques.\n\nThe solution can briefly be summarized as:\n- *Features*: turns, intersection clustering, populations per zipcode\n- *Model*: Detailed and general (for unknown intersections) LinReg model per target (no CV/KFolding)\n\nDetails about the solution which led to my highest scoring submission (and what didn't work) are all documented in [this kernel](https://www.kaggle.com/joatom/bigquery-geotab-bqml). I also published two other kernels during the competition and archived them [here](https://github.com/joatom/kaggle-2019-BigQuery-Geotab-Intersection-Congestion).\n\nBesides the technical part I like to share some thoughts about my ranking and the competition.\n \nAs you might have noticed there is a big difference between my 1st place in the *BigQuery ML Models built in SQL* category and my 213th Place on the LB. I can think of some possible reasons for that:\n- Few participants have chosen the \"SQL-way\" in the competition\n- I did some beginners mistakes (e.g. not setting up a proper validation set or I didn't go \"all-the-way\" analyzing the data or doing structured feature elimination)\n- Restriction on the available BQML Models (LinReg, Means and LogReg) can't compete against Tree-, Booster- and NN-Models.\n- The quota restrictions in BQ slowed down the pace. For instance, I didn't do grid search for parameters or KFolding because I could only train a few models per week. (to be fair, I need to mention that I relied entirely on the sandbox and I didn't activate the gcp credits which were provided at the beginning of the competition).\n\nAnyway, I really love processing data with SQL. For me it feels more consistent than pandas (but that's my biased view). I really enjoyed the easily embedding of the models in SQL. I also liked the easily usable geo spatial functions of BQ. I think with the TensorFlow option there are opportunities to build more complex models.\n\nThis competition nicely sums up my first kaggle year and points me where to go next.\n- retro 2019 - so much to learn and randomly trying out everything (like a child in a candy store)\n- goals 2020 - approaching data science projects in a more structured kind of manner.\n\nThanks for reading.\nJohannes", "votes": 7 }, { "id": 711551, "postDate": "2020-01-06T08:13:51.413Z", "content": "Good work Sir!!
", "rawMarkdown": "Good work Sir!!", "votes": 1 }, { "id": 712941, "postDate": "2020-01-07T18:40:18.187Z", "content": "2nd place winner (BQ/SQL category) here 👋
\n\nFirst of all: congrats for the win!
\n\nI was quite surprised seeing my baseline LinReg kernel ending up at 2nd place which makes me also think that the other submissions were not using BQ/SQL at all. I will not complain, those GCP credits will definitely come in handy 😎
", "rawMarkdown": "2nd place winner (BQ/SQL category) here 👋 \n\nFirst of all: congrats for the win!\n\nI was quite surprised seeing my baseline LinReg kernel ending up at 2nd place which makes me also think that the other submissions were not using BQ/SQL at all. I will not complain, those GCP credits will definitely come in handy 😎 ", "votes": 2, "replies": [ { "id": 712955, "postDate": "2020-01-07T18:55:21.043Z", "content": "It is very welcome to have some credits to tackle compute intensive competitions in the future. 😎
", "rawMarkdown": "It is very welcome to have some credits to tackle compute intensive competitions in the future. 😎 ", "votes": 1 } ] }, { "id": 752871, "postDate": "2020-02-21T13:43:03.373Z", "content": "Hello,I met a primary problem. It would really help me out if you could reply，thanks.\nWhy there are 11522010 rows in sample_submission.csv? In my understanding, there are 1921357 rows in test.csv , so submission.csv will have 1921357*6=11528142 rows.
", "rawMarkdown": "Hello,I met a primary problem. It would really help me out if you could reply，thanks.\nWhy there are 11522010 rows in sample_submission.csv? In my understanding, there are 1921357 rows in test.csv , so submission.csv will have 1921357*6=11528142 rows.", "replies": [ { "id": 753457, "postDate": "2020-02-22T08:02:48.773Z", "content": "Hi @bitmouxin ,
\n\nYou have to predict 6 values ('TotalTimeStopped_p20','TotalTimeStopped_p50','TotalTimeStopped_p80','DistanceToFirstStop_p20','DistanceToFirstStop_p50','DistanceToFirstStop_p80') per Test-rowid. The getting started notboke only shows how to predict one. Have a look at other notebooks on how to predict all six measures.
", "rawMarkdown": "Hi @bitmouxin ,\n\nYou have to predict 6 values ('TotalTimeStopped_p20','TotalTimeStopped_p50','TotalTimeStopped_p80','DistanceToFirstStop_p20','DistanceToFirstStop_p50','DistanceToFirstStop_p80') per Test-rowid. The getting started notboke only shows how to predict one. Have a look at other notebooks on how to predict all six measures." }, { "id": 753714, "postDate": "2020-02-22T15:36:04.787Z", "content": "@joatom \n Firstly, I am so grateful for your reply.\n I got that each Test-rowid should have 6 values, my question is that the number of rows of result is different between mine and Kaggle's require. \n In my code, there are 1,921,357*6=11,528,142 rows(because there are 1,921,357 rows in test.csv), but Kaggle requires 11,522,010 rows.\n Look forward to your reply. Best wishes.\n@bitmouxin \nThat is strange. If you look at an old EDA notebook e.g. https://www.kaggle.com/dcaichara/feature-engineering-and-lightgbm the shape of the test.csv used to be different. \nSorry can't help you here. Maybe the admins will reply to your other post.
", "rawMarkdown": "@bitmouxin \nThat is strange. If you look at an old EDA notebook e.g. https://www.kaggle.com/dcaichara/feature-engineering-and-lightgbm the shape of the test.csv used to be different. \nSorry can't help you here. Maybe the admins will reply to your other post." }, { "id": 754340, "postDate": "2020-02-23T12:30:35.997Z", "content": "@joatom \n- I just found that the RowId of train.cs&test.csv is different. And RowIds from 0 to 1,921,356 are rows in test.csv, the others(bigger than 1,921,356) are in train.csv.(due to other's notebook).\n- Perhaps the file on Kaggle right now is wrong, but I can tell them by this way.\n- Thanks a lot for your suggestion. Best wishes.
", "rawMarkdown": "@joatom \n- I just found that the RowId of train.cs&test.csv is different. And RowIds from 0 to 1,921,356 are rows in test.csv, the others(bigger than 1,921,356) are in train.csv.(due to other's notebook).\n- Perhaps the file on Kaggle right now is wrong, but I can tell them by this way.\n- Thanks a lot for your suggestion. Best wishes.", "votes": 2 } ] }, { "id": 714173, "postDate": "2020-01-09T06:28:47.500Z", "content": "Well done! Thanks for the summary!
", "rawMarkdown": "Well done! Thanks for the summary!", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T14:24:23Z", "competition_id": 14859, "competition_slug": "bigquery-geotab-intersection-congestion", "team_id": 3658937, "solution_url": "/competitions/bigquery-geotab-intersection-congestion/writeups/jo-tom-bqml-model-1st-place-solution-of-category-b" }