The model that I created 23 days before is doing better than my final submitted one. After that, I tried a lot, but what really works is tuning the threshold that what percent of the smallest value I should to zero out.
\n\nI 'waste' my lot precious time to tuning the threshold. But it can help me nothing when I join the next competition.\n```\ndef zero_out(df, column, percent):\n \"\"\"Zero out all the values that close to 0\"\"\"\n index = df[column].nsmallest(int(percent*len(df))).index\n df.loc[index, column] = 0\n print(column, \" remain \", len(df)-len(index))
\n\nzero_tupe = [\n (\"question_type_spelling\", 0.99),\n (\"question_not_really_a_question\", 0.94),\n (\"question_type_consequence\", 0.92),\n (\"question_type_definition\", 0.91),\n (\"question_type_entity\", 0.88),\n (\"question_type_compare\", 0.96),\n (\"question_type_choice\", 0.44),\n (\"question_conversational\",0.78),\n (\"question_multi_intent\",0.55),\n ]
\n\nfor column, percent in zero_tupe:\n zero_out(df_sub, column, percent)\n```
\n\nThe score compare is following: (PB ,LB)\nWithout zero-out: 0.386 0.409\nWith zero-out (23 days ago): 0.424 0.450\nAfter 23 days 'hard work' 0.424 0.465
\n\nWhat I did later after turning the threshold is actually a harm to the PB score.
", "comments": [ { "id": 741854, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2020-02-11T01:06:52.320000", "content": "A threshold-free solution will be shared the following morning :)
", "votes": 14, "replies": [ { "id": 741870, "author_name": "oohara", "author_url": "", "post_date": "2020-02-11T01:23:02.673000", "content": "Your team didn't use the threshold to post process result? That't pretty amazing! Waiting to see the solution.
", "votes": 0, "replies": [] }, { "id": 741949, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2020-02-11T02:34:56.357000", "content": "One word: Amazing.
", "votes": 0, "replies": [] }, { "id": 741960, "author_name": "stout", "author_url": "", "post_date": "2020-02-11T02:41:37.107000", "content": "Congratulations! Can't wait for your threshold-free solution...
", "votes": 0, "replies": [] }, { "id": 742210, "author_name": "kenmatsu4", "author_url": "", "post_date": "2020-02-11T06:44:23.080000", "content": "Surprising!!! I'm looking forward to reading your solution!
", "votes": 0, "replies": [] } ] }, { "id": 741858, "author_name": "XY", "author_url": "", "post_date": "2020-02-11T01:13:01.800000", "content": "How did you find the optimal threshold ? based on CV ?
", "votes": 1, "replies": [ { "id": 741866, "author_name": "oohara", "author_url": "", "post_date": "2020-02-11T01:20:47.120000", "content": "Based on 5 fold CV predicted values, I then used for i in range(100) to get the percentage that maximize the rho score.
The comp was bad well it's what it's! But I would be interested if people would like to share and dump their CV - VS LB scores without any post processing!
\n\nThat will reveal the actual winner's and the best model (and approach)
", "votes": 1, "replies": [ { "id": 741855, "author_name": "oohara", "author_url": "", "post_date": "2020-02-11T01:09:45.957000", "content": "Score updated.
", "votes": 0, "replies": [] } ] }, { "id": 741967, "author_name": "Firas Baba", "author_url": "", "post_date": "2020-02-11T02:44:43.520000", "content": "Hey Congrats for your gold! this is amazing!\nWe were all thinking that you're someone's bot or a cheater... Thank you for being real
", "votes": 0, "replies": [ { "id": 742297, "author_name": "oohara", "author_url": "", "post_date": "2020-02-11T08:14:53.177000", "content": "Thanks. Maybe I should update my profile to make me more real😏
", "votes": 2, "replies": [] } ] }, { "id": 742851, "author_name": "Zhongkai Shangguan", "author_url": "", "post_date": "2020-02-11T15:40:09.910000", "content": "Congrats! May I know \"After 23 days 'hard work' \", what is your performance without zero-out?
", "votes": 0, "replies": [] }, { "id": 742367, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-02-11T09:05:31.793000", "content": "Our solution is threshold free as well. But I guess your threshold approach was good enough since you jumped quite some places :)
", "votes": 0, "replies": [ { "id": 745183, "author_name": "oohara", "author_url": "", "post_date": "2020-02-13T15:37:06.477000", "content": "0.385, yes, the score drops from 0.386 to 0.385. If I gave up 25 days before, I could have got a better rank. :p
", "votes": 0, "replies": [] } ] }, { "id": 741925, "author_name": "something4kag", "author_url": "", "post_date": "2020-02-11T02:15:19.943000", "content": "With respect to thresholds, there was little information about the way scores were given in the columns to be predicted. However, looking at train it was not continuous probabilities but more like a value of 0, 1, maybe 0.5 so if a question was answered by 3 people and one scored 1 the rest 0 it could be 0.3333, or all 1 it could be 1, or all 0 it could be 0. Since the evaluation was correlation, it seemed something could be done to more correctly approximate or fine tune what the scores should be. The risk was if the private test set was not at all like train or public test.
\n\nWill be very interesting to see no threshold solutions.
", "votes": 0, "replies": [] }, { "id": 741889, "author_name": "Itaegyun", "author_url": "", "post_date": "2020-02-11T01:48:09.907000", "content": "Oh, I didn't know that It was needed thresholds.\nHow did you know this? (thresholds method, not threshold values)
", "votes": 0, "replies": [ { "id": 741896, "author_name": "oohara", "author_url": "", "post_date": "2020-02-11T01:58:00.343000", "content": "I first tired map every value to 0.3 0.5 0.7 like that. But it didn't work. And I saw Khoi Nguyen complained about the column question type spelling (Thanks, Khoi Nguyen.), and I started to zero out column quesiton type spelling. Iit workd! So how about other columns? And I did a lot to experiments to probe the threshold.
Wow, it is admirable!
", "votes": 0, "replies": [] } ] }, { "id": 741849, "author_name": "", "author_url": "", "post_date": "2020-02-11T01:02:47.163000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 19, "messages": [ { "id": 741826, "postDate": "2020-02-11T00:52:09.507Z", "content": "The model that I created 23 days before is doing better than my final submitted one. After that, I tried a lot, but what really works is tuning the threshold that what percent of the smallest value I should to zero out.
\n\nI 'waste' my lot precious time to tuning the threshold. But it can help me nothing when I join the next competition.\n```\ndef zero_out(df, column, percent):\n \"\"\"Zero out all the values that close to 0\"\"\"\n index = df[column].nsmallest(int(percent*len(df))).index\n df.loc[index, column] = 0\n print(column, \" remain \", len(df)-len(index))
\n\nzero_tupe = [\n (\"question_type_spelling\", 0.99),\n (\"question_not_really_a_question\", 0.94),\n (\"question_type_consequence\", 0.92),\n (\"question_type_definition\", 0.91),\n (\"question_type_entity\", 0.88),\n (\"question_type_compare\", 0.96),\n (\"question_type_choice\", 0.44),\n (\"question_conversational\",0.78),\n (\"question_multi_intent\",0.55),\n ]
\n\nfor column, percent in zero_tupe:\n zero_out(df_sub, column, percent)\n```
\n\nThe score compare is following: (PB ,LB)\nWithout zero-out: 0.386 0.409\nWith zero-out (23 days ago): 0.424 0.450\nAfter 23 days 'hard work' 0.424 0.465
\n\nWhat I did later after turning the threshold is actually a harm to the PB score.
", "rawMarkdown": "The model that I created 23 days before is doing better than my final submitted one. After that, I tried a lot, but what really works is tuning the threshold that what percent of the smallest value I should to zero out. \n\n I 'waste' my lot precious time to tuning the threshold. But it can help me nothing when I join the next competition.\n```\ndef zero_out(df, column, percent):\n \"\"\"Zero out all the values that close to 0\"\"\"\n index = df[column].nsmallest(int(percent*len(df))).index\n df.loc[index, column] = 0\n print(column, \" remain \", len(df)-len(index))\n\nzero_tupe = [\n (\"question_type_spelling\", 0.99),\n (\"question_not_really_a_question\", 0.94),\n (\"question_type_consequence\", 0.92),\n (\"question_type_definition\", 0.91),\n (\"question_type_entity\", 0.88),\n (\"question_type_compare\", 0.96),\n (\"question_type_choice\", 0.44),\n (\"question_conversational\",0.78),\n (\"question_multi_intent\",0.55),\n ]\n\nfor column, percent in zero_tupe:\n zero_out(df_sub, column, percent)\n```\n\nThe score compare is following: (PB ,LB)\nWithout zero-out: 0.386 0.409\nWith zero-out (23 days ago): 0.424 0.450\nAfter 23 days 'hard work' 0.424 0.465\n\n What I did later after turning the threshold is actually a harm to the PB score. ", "votes": 32 }, { "id": 741854, "postDate": "2020-02-11T01:06:52.320Z", "content": "A threshold-free solution will be shared the following morning :)
", "rawMarkdown": "A threshold-free solution will be shared the following morning :)", "votes": 14, "replies": [ { "id": 741870, "postDate": "2020-02-11T01:23:02.673Z", "content": "Your team didn't use the threshold to post process result? That't pretty amazing! Waiting to see the solution.
", "rawMarkdown": "Your team didn't use the threshold to post process result? That't pretty amazing! Waiting to see the solution." }, { "id": 741949, "postDate": "2020-02-11T02:34:56.357Z", "content": "One word: Amazing.
", "rawMarkdown": "One word: Amazing." }, { "id": 741960, "postDate": "2020-02-11T02:41:37.107Z", "content": "Congratulations! Can't wait for your threshold-free solution...
", "rawMarkdown": "Congratulations! Can't wait for your threshold-free solution..." }, { "id": 742210, "postDate": "2020-02-11T06:44:23.080Z", "content": "Surprising!!! I'm looking forward to reading your solution!
", "rawMarkdown": "Surprising!!! I'm looking forward to reading your solution!" } ] }, { "id": 741858, "postDate": "2020-02-11T01:13:01.800Z", "content": "How did you find the optimal threshold ? based on CV ?
", "rawMarkdown": "How did you find the optimal threshold ? based on CV ?", "votes": 1, "replies": [ { "id": 741866, "postDate": "2020-02-11T01:20:47.120Z", "content": "Based on 5 fold CV predicted values, I then used for i in range(100) to get the percentage that maximize the rho score.
The comp was bad well it's what it's! But I would be interested if people would like to share and dump their CV - VS LB scores without any post processing!
\n\nThat will reveal the actual winner's and the best model (and approach)
", "rawMarkdown": "The comp was bad well it's what it's! But I would be interested if people would like to share and dump their CV - VS LB scores without any post processing!\n\nThat will reveal the actual winner's and the best model (and approach)", "votes": 1, "replies": [ { "id": 741855, "postDate": "2020-02-11T01:09:45.957Z", "content": "Score updated.
", "rawMarkdown": "Score updated." } ] }, { "id": 741967, "postDate": "2020-02-11T02:44:43.520Z", "content": "Hey Congrats for your gold! this is amazing!\nWe were all thinking that you're someone's bot or a cheater... Thank you for being real
", "rawMarkdown": "Hey Congrats for your gold! this is amazing!\nWe were all thinking that you're someone's bot or a cheater... Thank you for being real ", "replies": [ { "id": 742297, "postDate": "2020-02-11T08:14:53.177Z", "content": "Thanks. Maybe I should update my profile to make me more real😏
", "rawMarkdown": "Thanks. Maybe I should update my profile to make me more real😏 ", "votes": 2 } ] }, { "id": 742851, "postDate": "2020-02-11T15:40:09.910Z", "content": "Congrats! May I know \"After 23 days 'hard work' \", what is your performance without zero-out?
", "rawMarkdown": "Congrats! May I know \"After 23 days 'hard work' \", what is your performance without zero-out?" }, { "id": 742367, "postDate": "2020-02-11T09:05:31.793Z", "content": "Our solution is threshold free as well. But I guess your threshold approach was good enough since you jumped quite some places :)
", "rawMarkdown": "Our solution is threshold free as well. But I guess your threshold approach was good enough since you jumped quite some places :) ", "replies": [ { "id": 745183, "postDate": "2020-02-13T15:37:06.477Z", "content": "0.385, yes, the score drops from 0.386 to 0.385. If I gave up 25 days before, I could have got a better rank. :p
", "rawMarkdown": "0.385, yes, the score drops from 0.386 to 0.385. If I gave up 25 days before, I could have got a better rank. :p" } ] }, { "id": 741925, "postDate": "2020-02-11T02:15:19.943Z", "content": "With respect to thresholds, there was little information about the way scores were given in the columns to be predicted. However, looking at train it was not continuous probabilities but more like a value of 0, 1, maybe 0.5 so if a question was answered by 3 people and one scored 1 the rest 0 it could be 0.3333, or all 1 it could be 1, or all 0 it could be 0. Since the evaluation was correlation, it seemed something could be done to more correctly approximate or fine tune what the scores should be. The risk was if the private test set was not at all like train or public test.
\n\nWill be very interesting to see no threshold solutions.
", "rawMarkdown": "With respect to thresholds, there was little information about the way scores were given in the columns to be predicted. However, looking at train it was not continuous probabilities but more like a value of 0, 1, maybe 0.5 so if a question was answered by 3 people and one scored 1 the rest 0 it could be 0.3333, or all 1 it could be 1, or all 0 it could be 0. Since the evaluation was correlation, it seemed something could be done to more correctly approximate or fine tune what the scores should be. The risk was if the private test set was not at all like train or public test. \n\nWill be very interesting to see no threshold solutions. " }, { "id": 741889, "postDate": "2020-02-11T01:48:09.907Z", "content": "Oh, I didn't know that It was needed thresholds.\nHow did you know this? (thresholds method, not threshold values)
", "rawMarkdown": "Oh, I didn't know that It was needed thresholds.\nHow did you know this? (thresholds method, not threshold values)", "replies": [ { "id": 741896, "postDate": "2020-02-11T01:58:00.343Z", "content": "I first tired map every value to 0.3 0.5 0.7 like that. But it didn't work. And I saw Khoi Nguyen complained about the column question type spelling (Thanks, Khoi Nguyen.), and I started to zero out column quesiton type spelling. Iit workd! So how about other columns? And I did a lot to experiments to probe the threshold.
Wow, it is admirable!
", "rawMarkdown": "Wow, it is admirable!" } ] }, { "id": 741849, "postDate": "2020-02-11T01:02:47.163Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T19:07:20Z", "competition_id": 7968, "competition_slug": "google-quest-challenge", "team_id": 4211367, "solution_url": "/competitions/google-quest-challenge/writeups/conquest-11th-solution-a-threshold-oriented-compet" }