{
  "id": 82375,
  "title": "sharing idea and result",
  "url": "/competitions/elo-merchant-category-recommendation/writeups/skynet-sharing-idea-and-result",
  "author_name": "",
  "post_date": "2019-03-01T08:48:34.590Z",
  "votes": 11,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Thank you for my teammates and all kagglers.</p>\n\n<p>I want to share some ideas used in my model (mainly feature engineering) and result.</p>\n\n<h2><strong>Feature Engineering</strong></h2>\n\n<p><strong>1. Split data</strong></p>\n\n<p>I think the most important part in this competition is to derive useful features from chunk of data split by various standards.</p>\n\n<p>I split data based on : </p>\n\n<ol>\n<li>Historical / New</li>\n<li>month_lag (each month_lag, range of month_lag, rank..)</li>\n<li>purchase_date</li>\n<li>authorized_flag</li>\n<li>categorical columns (category_1, category_3, category_4)</li>\n</ol>\n\n<p>For some cases, i split data using two or three standards simultaneously (ex. with category_1 = 1 at month_lag = 0).\nIn my model, features from month_lag and category_1 are most powerful.</p>\n\n<p>One of interesting methods i tried is using ranking method for month_lag and purchase_date.</p>\n\n<pre><code>df_his_tran['month_rank'] = df_his_tran.groupby(['card_id'])['month_lag'].rank(method='dense',ascending=False)\ndf_his_tran['date_rank'] = df_his_tran.groupby(['card_id'])['purchase_date'].rank(method='dense',ascending=False)\n</code></pre>\n\n<p>Features based on these rank numbers can give you compact information for the latest purchase of each card_id or vice versa. They improved both my CV  and LB.</p>\n\n<p>BTW, i also tried to rank purchase_amount grouped by card_id, so i can use information  such as \"the gap between the date with highest purchase_amount and now (or reference date)\". but it didn't work.</p>\n\n<p><strong>2. Adding true categorical features to each card_id</strong></p>\n\n<p>Since each card_id has multiple transactions, categorical features in transactions such as category_1, 2, 3, 4, and identifier columns(merchant_id, subsector_id, ..etc) are represented as aggregated values (mean, sum, nunique..) in normal ways.</p>\n\n<p>But, we can obtain true categorical features by getting representative category values which have  highest frequency in transactions of each card_id. I got improvement by using category_2 and subsector_id in this way.</p>\n\n<p><strong>3. Using <em>merchants</em>.csv</strong></p>\n\n<p>I tried basic aggregations of all columns in <em>merchants</em>.csv.\nOnly category_4 and  merchant_group_id worked for me.\nFYI,  you should be careful when you use <em>merchants</em>.csv because it has duplicate merhcant_id.</p>\n\n<p><strong>4. What didn't work (based on LB)</strong></p>\n\n<ul>\n<li>Tfidf, CountVectorizer for identifiers</li>\n<li>Categorical embedding for identifiers</li>\n<li>Target encoding (naive way, leave-one-out, ...)</li>\n<li>FFM on categorical features</li>\n</ul>\n\n<p><strong>Total 93 features selected by GBM importances, RFE, manual works.</strong> </p>\n\n<h2><strong>Models</strong></h2>\n\n<ul>\n<li>Lightgbm</li>\n<li>Xgboost</li>\n<li>Catboost</li>\n<li>NN (for categorical embedding / FFM)</li>\n</ul>\n\n<p>Like most of teams, i employed both single model method and 3-model method with outlier handling.\nAll of three GBM algorithms showed some potential for improvement by blending. But I only used Lightgbm as my final model for our team's ensemble result.</p>\n\n<h2>Best score</h2>\n\n<ul>\n<li>CV        : 3.621</li>\n<li>Public  : 3.666</li>\n<li>Private : 3.601</li>\n</ul>",
  "messages": [
    {
      "id": "481082",
      "postDate": "03/01/2019 03:07:38",
      "content": "<p>Thank you for my teammates and all kagglers.</p>\n\n<p>I want to share some ideas used in my model (mainly feature engineering) and result.</p>\n\n<h2><strong>Feature Engineering</strong></h2>\n\n<p><strong>1. Split data</strong></p>\n\n<p>I think the most important part in this competition is to derive useful features from chunk of data split by various standards.</p>\n\n<p>I split data based on : </p>\n\n<ol>\n<li>Historical / New</li>\n<li>month_lag (each month_lag, range of month_lag, rank..)</li>\n<li>purchase_date</li>\n<li>authorized_flag</li>\n<li>categorical columns (category_1, category_3, category_4)</li>\n</ol>\n\n<p>For some cases, i split data using two or three standards simultaneously (ex. with category_1 = 1 at month_lag = 0).\nIn my model, features from month_lag and category_1 are most powerful.</p>\n\n<p>One of interesting methods i tried is using ranking method for month_lag and purchase_date.</p>\n\n<pre><code>df_his_tran['month_rank'] = df_his_tran.groupby(['card_id'])['month_lag'].rank(method='dense',ascending=False)\ndf_his_tran['date_rank'] = df_his_tran.groupby(['card_id'])['purchase_date'].rank(method='dense',ascending=False)\n</code></pre>\n\n<p>Features based on these rank numbers can give you compact information for the latest purchase of each card_id or vice versa. They improved both my CV  and LB.</p>\n\n<p>BTW, i also tried to rank purchase_amount grouped by card_id, so i can use information  such as \"the gap between the date with highest purchase_amount and now (or reference date)\". but it didn't work.</p>\n\n<p><strong>2. Adding true categorical features to each card_id</strong></p>\n\n<p>Since each card_id has multiple transactions, categorical features in transactions such as category_1, 2, 3, 4, and identifier columns(merchant_id, subsector_id, ..etc) are represented as aggregated values (mean, sum, nunique..) in normal ways.</p>\n\n<p>But, we can obtain true categorical features by getting representative category values which have  highest frequency in transactions of each card_id. I got improvement by using category_2 and subsector_id in this way.</p>\n\n<p><strong>3. Using <em>merchants</em>.csv</strong></p>\n\n<p>I tried basic aggregations of all columns in <em>merchants</em>.csv.\nOnly category_4 and  merchant_group_id worked for me.\nFYI,  you should be careful when you use <em>merchants</em>.csv because it has duplicate merhcant_id.</p>\n\n<p><strong>4. What didn't work (based on LB)</strong></p>\n\n<ul>\n<li>Tfidf, CountVectorizer for identifiers</li>\n<li>Categorical embedding for identifiers</li>\n<li>Target encoding (naive way, leave-one-out, ...)</li>\n<li>FFM on categorical features</li>\n</ul>\n\n<p><strong>Total 93 features selected by GBM importances, RFE, manual works.</strong> </p>\n\n<h2><strong>Models</strong></h2>\n\n<ul>\n<li>Lightgbm</li>\n<li>Xgboost</li>\n<li>Catboost</li>\n<li>NN (for categorical embedding / FFM)</li>\n</ul>\n\n<p>Like most of teams, i employed both single model method and 3-model method with outlier handling.\nAll of three GBM algorithms showed some potential for improvement by blending. But I only used Lightgbm as my final model for our team's ensemble result.</p>\n\n<h2>Best score</h2>\n\n<ul>\n<li>CV        : 3.621</li>\n<li>Public  : 3.666</li>\n<li>Private : 3.601</li>\n</ul>",
      "rawMarkdown": "Thank you for my teammates and all kagglers.\n\nI want to share some ideas used in my model (mainly feature engineering) and result.\n\n**Feature Engineering**\n-----------------------\n\n**1. Split data**\n\nI think the most important part in this competition is to derive useful features from chunk of data split by various standards.\n\nI split data based on : \n\n 1. Historical / New\n 2. month_lag (each month_lag, range of month_lag, rank..)\n 3. purchase_date\n 4. authorized_flag\n 5. categorical columns (category_1, category_3, category_4)\n\nFor some cases, i split data using two or three standards simultaneously (ex. with category_1 = 1 at month_lag = 0).\nIn my model, features from month_lag and category_1 are most powerful.\n\nOne of interesting methods i tried is using ranking method for month_lag and purchase_date.\n\n    df_his_tran['month_rank'] = df_his_tran.groupby(['card_id'])['month_lag'].rank(method='dense',ascending=False)\n    df_his_tran['date_rank'] = df_his_tran.groupby(['card_id'])['purchase_date'].rank(method='dense',ascending=False)\n\nFeatures based on these rank numbers can give you compact information for the latest purchase of each card_id or vice versa. They improved both my CV  and LB.\n\nBTW, i also tried to rank purchase_amount grouped by card_id, so i can use information  such as \"the gap between the date with highest purchase_amount and now (or reference date)\". but it didn't work.\n\n\n**2. Adding true categorical features to each card_id**\n\nSince each card_id has multiple transactions, categorical features in transactions such as category_1, 2, 3, 4, and identifier columns(merchant_id, subsector_id, ..etc) are represented as aggregated values (mean, sum, nunique..) in normal ways.\n\nBut, we can obtain true categorical features by getting representative category values which have  highest frequency in transactions of each card_id. I got improvement by using category_2 and subsector_id in this way.\n\n\n**3. Using *merchants*.csv**\n\nI tried basic aggregations of all columns in *merchants*.csv.\nOnly category_4 and  merchant_group_id worked for me.\nFYI,  you should be careful when you use *merchants*.csv because it has duplicate merhcant_id.\n\n\n\n**4. What didn't work (based on LB)**\n\n - Tfidf, CountVectorizer for identifiers\n - Categorical embedding for identifiers\n - Target encoding (naive way, leave-one-out, ...)\n - FFM on categorical features\n\n**Total 93 features selected by GBM importances, RFE, manual works.** \n\n**Models**\n-----------------------\n\n - Lightgbm\n - Xgboost\n - Catboost\n - NN (for categorical embedding / FFM)\n\nLike most of teams, i employed both single model method and 3-model method with outlier handling.\nAll of three GBM algorithms showed some potential for improvement by blending. But I only used Lightgbm as my final model for our team's ensemble result.\n\nBest score\n----------\n\n - CV        : 3.621\n - Public  : 3.666\n - Private : 3.601",
      "votes": null
    },
    {
      "id": "481086",
      "postDate": "03/01/2019 03:18:51",
      "content": "<p>Congratulations and thanks for your sharing! Your Feature Engineering skill is excellent!</p>",
      "rawMarkdown": "Congratulations and thanks for your sharing! Your Feature Engineering skill is excellent!",
      "votes": null
    },
    {
      "id": "481090",
      "postDate": "03/01/2019 03:23:41",
      "content": "<p>Good job!</p>",
      "rawMarkdown": "Good job!",
      "votes": null
    },
    {
      "id": "481117",
      "postDate": "03/01/2019 04:02:08",
      "content": "<p>Thanks for your sharing</p>",
      "rawMarkdown": "Thanks for your sharing",
      "votes": null
    },
    {
      "id": "481234",
      "postDate": "03/01/2019 07:25:59",
      "content": "<p>Congratulations and thanks for sharing your summary</p>",
      "rawMarkdown": "Congratulations and thanks for sharing your summary",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 481086,
      "author_name": "jiazhuang",
      "author_url": "",
      "post_date": "03/01/2019 03:18:51",
      "content": "<p>Congratulations and thanks for your sharing! Your Feature Engineering skill is excellent!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 481090,
      "author_name": "codlife",
      "author_url": "",
      "post_date": "03/01/2019 03:23:41",
      "content": "<p>Good job!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 481117,
      "author_name": "yrowe483",
      "author_url": "",
      "post_date": "03/01/2019 04:02:08",
      "content": "<p>Thanks for your sharing</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 481234,
      "author_name": "karthik7395",
      "author_url": "",
      "post_date": "03/01/2019 07:25:59",
      "content": "<p>Congratulations and thanks for sharing your summary</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "481082": "Thank you for my teammates and all kagglers.\n\nI want to share some ideas used in my model (mainly feature engineering) and result.\n\n**Feature Engineering**\n-----------------------\n\n**1. Split data**\n\nI think the most important part in this competition is to derive useful features from chunk of data split by various standards.\n\nI split data based on : \n\n 1. Historical / New\n 2. month_lag (each month_lag, range of month_lag, rank..)\n 3. purchase_date\n 4. authorized_flag\n 5. categorical columns (category_1, category_3, category_4)\n\nFor some cases, i split data using two or three standards simultaneously (ex. with category_1 = 1 at month_lag = 0).\nIn my model, features from month_lag and category_1 are most powerful.\n\nOne of interesting methods i tried is using ranking method for month_lag and purchase_date.\n\n    df_his_tran['month_rank'] = df_his_tran.groupby(['card_id'])['month_lag'].rank(method='dense',ascending=False)\n    df_his_tran['date_rank'] = df_his_tran.groupby(['card_id'])['purchase_date'].rank(method='dense',ascending=False)\n\nFeatures based on these rank numbers can give you compact information for the latest purchase of each card_id or vice versa. They improved both my CV  and LB.\n\nBTW, i also tried to rank purchase_amount grouped by card_id, so i can use information  such as \"the gap between the date with highest purchase_amount and now (or reference date)\". but it didn't work.\n\n\n**2. Adding true categorical features to each card_id**\n\nSince each card_id has multiple transactions, categorical features in transactions such as category_1, 2, 3, 4, and identifier columns(merchant_id, subsector_id, ..etc) are represented as aggregated values (mean, sum, nunique..) in normal ways.\n\nBut, we can obtain true categorical features by getting representative category values which have  highest frequency in transactions of each card_id. I got improvement by using category_2 and subsector_id in this way.\n\n\n**3. Using *merchants*.csv**\n\nI tried basic aggregations of all columns in *merchants*.csv.\nOnly category_4 and  merchant_group_id worked for me.\nFYI,  you should be careful when you use *merchants*.csv because it has duplicate merhcant_id.\n\n\n\n**4. What didn't work (based on LB)**\n\n - Tfidf, CountVectorizer for identifiers\n - Categorical embedding for identifiers\n - Target encoding (naive way, leave-one-out, ...)\n - FFM on categorical features\n\n**Total 93 features selected by GBM importances, RFE, manual works.** \n\n**Models**\n-----------------------\n\n - Lightgbm\n - Xgboost\n - Catboost\n - NN (for categorical embedding / FFM)\n\nLike most of teams, i employed both single model method and 3-model method with outlier handling.\nAll of three GBM algorithms showed some potential for improvement by blending. But I only used Lightgbm as my final model for our team's ensemble result.\n\nBest score\n----------\n\n - CV        : 3.621\n - Public  : 3.666\n - Private : 3.601",
    "481086": "Congratulations and thanks for your sharing! Your Feature Engineering skill is excellent!",
    "481090": "Good job!",
    "481117": "Thanks for your sharing",
    "481234": "Congratulations and thanks for sharing your summary"
  },
  "source": "meta"
}