{
  "id": 127304,
  "title": "44 place writeup(Catboost ranking with eventdata)",
  "content": "<p>First, thanks Kaggle and Booz Allen Hamilton for such a great competition.\nIt was an interesting problem with lots of challenges and I learned a lot.</p>\n\n<p>Here is my solution and observations:</p>\n\n<p><strong>Loss function and framework</strong>\nI use ranking loss(<code>PairLogitPairwise:max_pairs=1000000</code>) with CatBoost, depth 6 or 7, training on GPU.\nThe other hyperparameters are default.\nI train the model 5 fold and then blend all 5 models with CatBoost <code>sum_model</code>to produce the average prediction.\nSo I obtain a quasi single model solution.\nI try to predict <code>accuracy_group</code>. My attempts to predict <code>accuracy</code>, or <code>num_correct</code>and <code>num_incorrect</code>as targets didn't work well.</p>\n\n<p><strong>Validation</strong>\n5-fold truncated cross validation where I perform truncation 5 times for each fold and average the score.</p>\n\n<p><strong>Threshold selection</strong>\nAfter the model is blended I predict the whole training set and optimize the threshold to maximize the Kappa. \nI do it 5 times and then take the median value for each sample.</p>\n\n<p><strong>Features generation</strong>\nI've generated about 1000 features. Among them are:\n1. Overall accumulated counters of event_codes and event_ids.\n2. Various accumulated accuracy statistics.\n3. Timestamp month and hour\n4. Linear extrapolation of accuracy.\n5. Features extracted from event data:\n    a. Overall sum and mean value for each key that has numeric value except coordinates.\n    b. The same statistics groupped by event title.</p>\n\n<p>Here is the few top features by SHAP importatance to illustrate the idea:\n- <code>lastAssessmentTitle</code>\n- <code>misses_mean</code>\n- <code>Bird Measurer (Assessment)_stage_number_mean</code>\n- <code>accuracy_mean</code>\n- <code>4070_count</code>\n- <code>Sandcastle Builder (Activity)_total_duration_mean</code>\n- <code>IsAssessmentAttemptSuccessfull_Chest Sorter (Assessment)</code>\n- <code>Clip_count</code>\n- <code>6bf9e3e1_count</code></p>\n\n<p><strong>Feature selection</strong>\nFeatures of group 5b (like <code>Bird Measurer (Assessment)_stage_number_mean</code>) lead to a heavy overfitting for training set. To mitigate that two approaches work:\n1. Select top 150-200 features by shap.\n2. Drop features using truncated adversarial validation untill ROC AUC becomes ~0.5. That leaves 863 features.</p>\n\n<p><strong>Submission selection</strong>\nI've submitted the most stable blend of 3 models that vary by the selected features and produce 0.555-0.56 at public LB. That produced .552 private and 44 place.\nI have few single model and blend submissions for .553 and .554, so my final submission was quite close to optimal and I get a fair Silver.</p>",
  "comments": [
    {
      "id": 727052,
      "author_name": "Ivan Bagmut",
      "author_url": "",
      "post_date": "2020-01-23T12:11:38.460000",
      "content": "<p>Congrats! \"Select top 150-200 features by shap.\" - new useful library for me</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 726900,
      "author_name": "Hanjoon Choe",
      "author_url": "",
      "post_date": "2020-01-23T10:03:49.397000",
      "content": "<p>Congratulations! ;-)</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 726898,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-01-23T10:01:53.390000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 726864,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-01-23T09:36:37.260000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 4,
  "messages": [
    {
      "id": 726807,
      "postDate": "2020-01-23T08:46:51.187Z",
      "content": "<p>First, thanks Kaggle and Booz Allen Hamilton for such a great competition.\nIt was an interesting problem with lots of challenges and I learned a lot.</p>\n\n<p>Here is my solution and observations:</p>\n\n<p><strong>Loss function and framework</strong>\nI use ranking loss(<code>PairLogitPairwise:max_pairs=1000000</code>) with CatBoost, depth 6 or 7, training on GPU.\nThe other hyperparameters are default.\nI train the model 5 fold and then blend all 5 models with CatBoost <code>sum_model</code>to produce the average prediction.\nSo I obtain a quasi single model solution.\nI try to predict <code>accuracy_group</code>. My attempts to predict <code>accuracy</code>, or <code>num_correct</code>and <code>num_incorrect</code>as targets didn't work well.</p>\n\n<p><strong>Validation</strong>\n5-fold truncated cross validation where I perform truncation 5 times for each fold and average the score.</p>\n\n<p><strong>Threshold selection</strong>\nAfter the model is blended I predict the whole training set and optimize the threshold to maximize the Kappa. \nI do it 5 times and then take the median value for each sample.</p>\n\n<p><strong>Features generation</strong>\nI've generated about 1000 features. Among them are:\n1. Overall accumulated counters of event_codes and event_ids.\n2. Various accumulated accuracy statistics.\n3. Timestamp month and hour\n4. Linear extrapolation of accuracy.\n5. Features extracted from event data:\n    a. Overall sum and mean value for each key that has numeric value except coordinates.\n    b. The same statistics groupped by event title.</p>\n\n<p>Here is the few top features by SHAP importatance to illustrate the idea:\n- <code>lastAssessmentTitle</code>\n- <code>misses_mean</code>\n- <code>Bird Measurer (Assessment)_stage_number_mean</code>\n- <code>accuracy_mean</code>\n- <code>4070_count</code>\n- <code>Sandcastle Builder (Activity)_total_duration_mean</code>\n- <code>IsAssessmentAttemptSuccessfull_Chest Sorter (Assessment)</code>\n- <code>Clip_count</code>\n- <code>6bf9e3e1_count</code></p>\n\n<p><strong>Feature selection</strong>\nFeatures of group 5b (like <code>Bird Measurer (Assessment)_stage_number_mean</code>) lead to a heavy overfitting for training set. To mitigate that two approaches work:\n1. Select top 150-200 features by shap.\n2. Drop features using truncated adversarial validation untill ROC AUC becomes ~0.5. That leaves 863 features.</p>\n\n<p><strong>Submission selection</strong>\nI've submitted the most stable blend of 3 models that vary by the selected features and produce 0.555-0.56 at public LB. That produced .552 private and 44 place.\nI have few single model and blend submissions for .553 and .554, so my final submission was quite close to optimal and I get a fair Silver.</p>",
      "rawMarkdown": "First, thanks Kaggle and Booz Allen Hamilton for such a great competition.\nIt was an interesting problem with lots of challenges and I learned a lot.\n\nHere is my solution and observations:\n\n**Loss function and framework**\nI use ranking loss(`PairLogitPairwise:max_pairs=1000000`) with CatBoost, depth 6 or 7, training on GPU.\nThe other hyperparameters are default.\nI train the model 5 fold and then blend all 5 models with CatBoost `sum_model `to produce the average prediction.\nSo I obtain a quasi single model solution.\nI try to predict `accuracy_group`. My attempts to predict `accuracy`, or `num_correct `and `num_incorrect `as targets didn't work well.\n\n**Validation**\n5-fold truncated cross validation where I perform truncation 5 times for each fold and average the score.\n\n**Threshold selection**\nAfter the model is blended I predict the whole training set and optimize the threshold to maximize the Kappa. \nI do it 5 times and then take the median value for each sample.\n\n**Features generation**\nI've generated about 1000 features. Among them are:\n1. Overall accumulated counters of event_codes and event_ids.\n2. Various accumulated accuracy statistics.\n3. Timestamp month and hour\n4. Linear extrapolation of accuracy.\n5. Features extracted from event data:\n    a. Overall sum and mean value for each key that has numeric value except coordinates.\n    b. The same statistics groupped by event title.\n    \nHere is the few top features by SHAP importatance to illustrate the idea:\n- `lastAssessmentTitle`\n- `misses_mean`\n- `Bird Measurer (Assessment)_stage_number_mean`\n- `accuracy_mean`\n- `4070_count`\n- `Sandcastle Builder (Activity)_total_duration_mean`\n- `IsAssessmentAttemptSuccessfull_Chest Sorter (Assessment)`\n- `Clip_count`\n- `6bf9e3e1_count`\n\n**Feature selection**\nFeatures of group 5b (like `Bird Measurer (Assessment)_stage_number_mean`) lead to a heavy overfitting for training set. To mitigate that two approaches work:\n1. Select top 150-200 features by shap.\n2. Drop features using truncated adversarial validation untill ROC AUC becomes ~0.5. That leaves 863 features.\n\n**Submission selection**\nI've submitted the most stable blend of 3 models that vary by the selected features and produce 0.555-0.56 at public LB. That produced .552 private and 44 place.\nI have few single model and blend submissions for .553 and .554, so my final submission was quite close to optimal and I get a fair Silver.",
      "votes": 17
    },
    {
      "id": 727052,
      "postDate": "2020-01-23T12:11:38.460Z",
      "content": "<p>Congrats! \"Select top 150-200 features by shap.\" - new useful library for me</p>",
      "rawMarkdown": "Congrats! \"Select top 150-200 features by shap.\" - new useful library for me"
    },
    {
      "id": 726900,
      "postDate": "2020-01-23T10:03:49.397Z",
      "content": "<p>Congratulations! ;-)</p>",
      "rawMarkdown": "Congratulations! ;-)"
    },
    {
      "id": 726898,
      "postDate": "2020-01-23T10:01:53.390Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 726864,
      "postDate": "2020-01-23T09:36:37.260Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-23T23:48:56Z",
  "competition_id": 16531,
  "competition_slug": "data-science-bowl-2019",
  "team_id": 3867765,
  "solution_url": "/competitions/data-science-bowl-2019/writeups/victor-zaguskin-44-place-writeup-catboost-ranking-"
}