{
  "id": 82055,
  "title": "7th Place Solution (Updated Late Submission Finding)",
  "url": "/competitions/elo-merchant-category-recommendation/writeups/you-ll-never-overfitting-alone-7th-place-solution-",
  "author_name": "",
  "post_date": "2019-03-14T03:20:45.733Z",
  "votes": 151,
  "comment_count": 85,
  "views": 0,
  "content": "<p>Congrats to all winner teams and new masters,experts.It's a very hard competition,I think every team work hard to try everything and fight with overfitting every day,no one fails who does his best.\nI started at very early stage so maybe I just did more work than others,let me share my solution briefly.</p>\n\n<h1><strong>Preprocessing</strong></h1>\n\n<p>sort the historical and new data by timestamp then create 8 kinds of dataset for different features.</p>\n\n<ol>\n<li>Only historical data</li>\n<li>Only historical data with authorized_flag=1</li>\n<li>Only new data </li>\n<li>Merge of historical data with authorized_flag=1 and new data </li>\n<li>Merge of historical data  and new data </li>\n<li>Merge of historical data and merchants data</li>\n<li>Merge of new data and merchants data</li>\n<li>Merge of historical data  and new data and merchants data</li>\n</ol>\n\n<h1><strong>Feature Engineering</strong></h1>\n\n<h2>Aggregation</h2>\n\n<p>I think there are not big difference with everyone's aggregation features, I want to thank raddar's magic that purchase_amount_new can bring additional features beside of original purchase_amount.</p>\n\n<h2>Interval</h2>\n\n<p>days interval(the difference from current purchase_date to next purchase_date)\npurchase_amount interval(the difference from current purchase_amount to next purchase_amount)</p>\n\n<h2>Interaction</h2>\n\n<p>new.purchase_date.max() / historical.purchase_date.max() is very strong</p>\n\n<h2>SVD</h2>\n\n<p>dimession reducing from original card_id's merchant|merchant_category transaction sequence to 5 \n```\ndef svd_feature(prefix, df, traintest, groupby, target,n_comp):\n    tfidf_vec = TfidfVectorizer(ngram_range=(1,1), max_features=None)\n    df_bag = pd.DataFrame(df[[groupby, target]])\n    df_bag = df_bag.groupby(groupby, as_index=False)[target].agg({'list':(lambda x: list(x))}).reset_index()\n    df_bag[target + '_list']=df_bag['list'].apply(lambda x: str(x).replace('[','').replace(']','').replace(',',' '))</p>\n\n<pre><code>df_bag = df_bag.merge(traintest,on=groupby,how='left')\ndf_bag_train = df_bag[df_bag['target'].notnull()].reset_index(drop=True)\ndf_bag_test = df_bag[df_bag['target'].isnull()].reset_index(drop=True)\n\ntfidf_full_vector = tfidf_vec.fit_transform(df_bag[target + '_list'])\ntfidf_train_vector = tfidf_vec.transform(df_bag_train[target + '_list'])\ntfidf_test_vector = tfidf_vec.transform(df_bag_test[target + '_list'])\n\nsvd_vec = TruncatedSVD(n_components=5, algorithm='arpack')\nsvd_vec.fit(tfidf_full_vector)\nsvd_train = pd.DataFrame(svd_vec.transform(tfidf_train_vector))\nsvd_test = pd.DataFrame(svd_vec.transform(tfidf_test_vector))\n\nsvd_train.columns = ['svd_%s_%s_%d'%(prefix,target,x) for x in range(n_comp)]\nsvd_train[groupby] = df_bag_train[groupby]\nsvd_test.columns = ['svd_%s_%s_%d'%(prefix,target,x) for x in range(n_comp)]\nsvd_test[groupby] = df_bag_test[groupby]\n#df_svd = pd.concat([svd_train,svd_test],axis=0)\nprint ('svd_train:' + str(svd_train.shape))\nprint ('svd_test:' + str(svd_test.shape))\nreturn svd_train,svd_test\n</code></pre>\n\n<p>```</p>\n\n<h2>Word2vec</h2>\n\n<p>create each merchant_id's , merchant_category_id's , purchase_date's ... word embedding,then group by card_id's min/max/mean/std\n<code>\ndef word2vec_feature(prefix, df, groupby, target,size):\n    df_bag = pd.DataFrame(df[[groupby, target]])\n    df_bag[target] = df_bag[target].astype(str)\n    df_bag[target].fillna('NAN', inplace=True)\n    df_bag = df_bag.groupby(groupby, as_index=False)[target].agg({'list':(lambda x: list(x))}).reset_index()\n    doc_list = list(df_bag['list'].values)\n    w2v = Word2Vec(doc_list, size=size, window=3, min_count=1, workers=32)\n    vocab_keys = list(w2v.wv.vocab.keys())\n    w2v_array = []\n    for v in vocab_keys :\n        w2v_array.append(list(w2v.wv[v]))\n    df_w2v = pd.DataFrame()\n    df_w2v['vocab_keys'] = vocab_keys <br>\n    df_w2v = pd.concat([df_w2v, pd.DataFrame(w2v_array)], axis=1)\n    df_w2v.columns = [target] + ['w2v_%s_%s_%d'%(prefix,target,x) for x in range(size)]\n    print ('df_w2v:' + str(df_w2v.shape))\n    return df_w2v\n</code></p>\n\n<h2>Transaction-based Model</h2>\n\n<p>use above 6,7,8 dataset, merge train's target to each transaction row,then create a lightgbm to output each transaction's meta feature , then groupby card_id's min/sum to create card_id's meta feature , it can boost 0.005 ~ 0.006  both on cv and lb.</p>\n\n<h2>Feature Selection</h2>\n\n<p>at early stage I add new features one by one,sometimes group by group,at late stage I use target permutation learned from <a href=\"https://www.kaggle.com/ogrellier/feature-selection-with-null-importances\">https://www.kaggle.com/ogrellier/feature-selection-with-null-importances</a>\nI created thousands of features,finally get 12 different feature sets with number from 200 ~ 700</p>\n\n<h2>Top 20 Feature Importance</h2>\n\n<p><code>\n('reborn_new_min_submodel',                         3212145.126355231)\n('reborn_hist_new_min_submodel',                    2897647.0672301054)\n('reborn_hist_min_submodel',                        2571620.7657690495)\n('reborn_hist_sum_submodel',                        1556242.6892476082)\n('reborn_hist_new_sum_submodel',                    1495420.6608867645)\n('card_purchase_date_max_ratio',                    1176560.0546371937)\n('reborn_new_sum_submodel',                         988300.9342432022)\n('hist_new_purchase_date_max_ratio',                751098.8485645056)\n('hist_new_sum_category_1_1_purchase_amount',       634349.3938679695)\n('hist_sum_category_1_1_purchase_amount',           626576.9190101624)\n('new_card_sum_purchase_amount_new',                573281.6908314228)\n('hist_card0_max_month_diff',                       539578.2926783562)\n('new_card_var_day_diff',                           537170.215294838)\n('new_card_last_day',                               532604.059334755)\n('hist_card_sum_month_lag_0_purchase_amount_new',   530102.9405331612)\n('hist_card0_mean_month_diff',                      506467.2120089531)\n('svd_hist_new_merchant_id4',                       454442.183167696)\n('hist_card0_sum_month_diff',                       445077.63865828514)\n('hist_card_std_merchant_category_id_w2v_4',        436928.8732688427)\n('hist_new_card_std_merchant_category_id_w2v_1',    434849.51894950867)\n</code></p>\n\n<h1><strong>Single Model</strong></h1>\n\n<h2>Lightgbm</h2>\n\n<p>My best single model is a lightgbm has 385 features with CV:3.6144,LB:3.668,PB:3.593,maybe it can reach to 3th place.It also happened at homecredit competition,best single model is much better than stacking,I will doubt if stacking is a good choice.</p>\n\n<h2>Nueral Netowork</h2>\n\n<p>My best nn has CV:3.6407,LB:3.680,PB:3.605,it's just a simple 3layers-dense network.</p>\n\n<p><code>\ndef ann(input_shape):\n    model = Sequential()\n    model.add(Dense(2 ** 10, input_dim = input_shape, init='random_uniform', activation='relu'))\n    model.add(Dropout(0.25)) <br>\n    model.add(BatchNormalization())\n    model.add(Dense(2 ** 9, init='random_uniform', activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.25)) \n    model.add(Dense(2 ** 5, init='random_uniform', activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.25)) <br>\n    model.add(Dense(1))\n    model.compile(loss='mean_squared_error', optimizer='adam') \n    return model\n</code></p>\n\n<h2>Others</h2>\n\n<p>Xgboost and Catboost got not bad results but almost no improvement to stacking.\nAlso Tried Ridge,KNN,FFM...no success to me.</p>\n\n<h1><strong>Ensemble</strong></h1>\n\n<p>I built 12 lightgbm models and 40 nn models for Stacking stage1,then use lightgbm and nn as stage2,finally blend them with average.CV can reach to 3.603,but lb got worse,and final pb also got worse.</p>\n\n<h1><strong>Postprocessing</strong></h1>\n\n<p>Isotonic regression give me 0.005 ~ 0.006 boost both on cv and lb but pb score become worse,that's the reason I shakedown.\nCombination with a non-outliers model using binary model's best threshold give me 0.001 ~ 0.002 boost both on cv and lb and pb.</p>\n\n<h1><strong>Leaderboard Probing</strong></h1>\n\n<p>I used 100 subs to probe the lb,that let me grasp 24 public-test outliers and 6 non-outliers.\nI tried Pseudo-Labelling but no boost,finally it just can be used to decide a \"perfect threshold\" to override the outliers but failed.</p>\n\n<h1><strong>Other tricks</strong></h1>\n\n<p>at very early stage I tried target transformation,using minmaxscaler to transform from 0 ~ 1, then use xentropy boosting type,finally reversed to original value range,it didn't improve cv and lb, but I found pb is very good, a very eary stage lightgbm model is enough to get a gold medal.</p>\n\n<h1><strong>Late Submission Founding</strong></h1>\n\n<p>Isotonic regression overfit both on cv and lb, should be careful.\nStacking can work if stage-2 model is nn or ridge ,lgb and extratree overfit too much.</p>",
  "messages": [
    {
      "id": "479196",
      "postDate": "02/27/2019 02:31:14",
      "content": "<p>Congrats to all winner teams and new masters,experts.It's a very hard competition,I think every team work hard to try everything and fight with overfitting every day,no one fails who does his best.\nI started at very early stage so maybe I just did more work than others,let me share my solution briefly.</p>\n\n<h1><strong>Preprocessing</strong></h1>\n\n<p>sort the historical and new data by timestamp then create 8 kinds of dataset for different features.</p>\n\n<ol>\n<li>Only historical data</li>\n<li>Only historical data with authorized_flag=1</li>\n<li>Only new data </li>\n<li>Merge of historical data with authorized_flag=1 and new data </li>\n<li>Merge of historical data  and new data </li>\n<li>Merge of historical data and merchants data</li>\n<li>Merge of new data and merchants data</li>\n<li>Merge of historical data  and new data and merchants data</li>\n</ol>\n\n<h1><strong>Feature Engineering</strong></h1>\n\n<h2>Aggregation</h2>\n\n<p>I think there are not big difference with everyone's aggregation features, I want to thank raddar's magic that purchase_amount_new can bring additional features beside of original purchase_amount.</p>\n\n<h2>Interval</h2>\n\n<p>days interval(the difference from current purchase_date to next purchase_date)\npurchase_amount interval(the difference from current purchase_amount to next purchase_amount)</p>\n\n<h2>Interaction</h2>\n\n<p>new.purchase_date.max() / historical.purchase_date.max() is very strong</p>\n\n<h2>SVD</h2>\n\n<p>dimession reducing from original card_id's merchant|merchant_category transaction sequence to 5 \n```\ndef svd_feature(prefix, df, traintest, groupby, target,n_comp):\n    tfidf_vec = TfidfVectorizer(ngram_range=(1,1), max_features=None)\n    df_bag = pd.DataFrame(df[[groupby, target]])\n    df_bag = df_bag.groupby(groupby, as_index=False)[target].agg({'list':(lambda x: list(x))}).reset_index()\n    df_bag[target + '_list']=df_bag['list'].apply(lambda x: str(x).replace('[','').replace(']','').replace(',',' '))</p>\n\n<pre><code>df_bag = df_bag.merge(traintest,on=groupby,how='left')\ndf_bag_train = df_bag[df_bag['target'].notnull()].reset_index(drop=True)\ndf_bag_test = df_bag[df_bag['target'].isnull()].reset_index(drop=True)\n\ntfidf_full_vector = tfidf_vec.fit_transform(df_bag[target + '_list'])\ntfidf_train_vector = tfidf_vec.transform(df_bag_train[target + '_list'])\ntfidf_test_vector = tfidf_vec.transform(df_bag_test[target + '_list'])\n\nsvd_vec = TruncatedSVD(n_components=5, algorithm='arpack')\nsvd_vec.fit(tfidf_full_vector)\nsvd_train = pd.DataFrame(svd_vec.transform(tfidf_train_vector))\nsvd_test = pd.DataFrame(svd_vec.transform(tfidf_test_vector))\n\nsvd_train.columns = ['svd_%s_%s_%d'%(prefix,target,x) for x in range(n_comp)]\nsvd_train[groupby] = df_bag_train[groupby]\nsvd_test.columns = ['svd_%s_%s_%d'%(prefix,target,x) for x in range(n_comp)]\nsvd_test[groupby] = df_bag_test[groupby]\n#df_svd = pd.concat([svd_train,svd_test],axis=0)\nprint ('svd_train:' + str(svd_train.shape))\nprint ('svd_test:' + str(svd_test.shape))\nreturn svd_train,svd_test\n</code></pre>\n\n<p>```</p>\n\n<h2>Word2vec</h2>\n\n<p>create each merchant_id's , merchant_category_id's , purchase_date's ... word embedding,then group by card_id's min/max/mean/std\n<code>\ndef word2vec_feature(prefix, df, groupby, target,size):\n    df_bag = pd.DataFrame(df[[groupby, target]])\n    df_bag[target] = df_bag[target].astype(str)\n    df_bag[target].fillna('NAN', inplace=True)\n    df_bag = df_bag.groupby(groupby, as_index=False)[target].agg({'list':(lambda x: list(x))}).reset_index()\n    doc_list = list(df_bag['list'].values)\n    w2v = Word2Vec(doc_list, size=size, window=3, min_count=1, workers=32)\n    vocab_keys = list(w2v.wv.vocab.keys())\n    w2v_array = []\n    for v in vocab_keys :\n        w2v_array.append(list(w2v.wv[v]))\n    df_w2v = pd.DataFrame()\n    df_w2v['vocab_keys'] = vocab_keys <br>\n    df_w2v = pd.concat([df_w2v, pd.DataFrame(w2v_array)], axis=1)\n    df_w2v.columns = [target] + ['w2v_%s_%s_%d'%(prefix,target,x) for x in range(size)]\n    print ('df_w2v:' + str(df_w2v.shape))\n    return df_w2v\n</code></p>\n\n<h2>Transaction-based Model</h2>\n\n<p>use above 6,7,8 dataset, merge train's target to each transaction row,then create a lightgbm to output each transaction's meta feature , then groupby card_id's min/sum to create card_id's meta feature , it can boost 0.005 ~ 0.006  both on cv and lb.</p>\n\n<h2>Feature Selection</h2>\n\n<p>at early stage I add new features one by one,sometimes group by group,at late stage I use target permutation learned from <a href=\"https://www.kaggle.com/ogrellier/feature-selection-with-null-importances\">https://www.kaggle.com/ogrellier/feature-selection-with-null-importances</a>\nI created thousands of features,finally get 12 different feature sets with number from 200 ~ 700</p>\n\n<h2>Top 20 Feature Importance</h2>\n\n<p><code>\n('reborn_new_min_submodel',                         3212145.126355231)\n('reborn_hist_new_min_submodel',                    2897647.0672301054)\n('reborn_hist_min_submodel',                        2571620.7657690495)\n('reborn_hist_sum_submodel',                        1556242.6892476082)\n('reborn_hist_new_sum_submodel',                    1495420.6608867645)\n('card_purchase_date_max_ratio',                    1176560.0546371937)\n('reborn_new_sum_submodel',                         988300.9342432022)\n('hist_new_purchase_date_max_ratio',                751098.8485645056)\n('hist_new_sum_category_1_1_purchase_amount',       634349.3938679695)\n('hist_sum_category_1_1_purchase_amount',           626576.9190101624)\n('new_card_sum_purchase_amount_new',                573281.6908314228)\n('hist_card0_max_month_diff',                       539578.2926783562)\n('new_card_var_day_diff',                           537170.215294838)\n('new_card_last_day',                               532604.059334755)\n('hist_card_sum_month_lag_0_purchase_amount_new',   530102.9405331612)\n('hist_card0_mean_month_diff',                      506467.2120089531)\n('svd_hist_new_merchant_id4',                       454442.183167696)\n('hist_card0_sum_month_diff',                       445077.63865828514)\n('hist_card_std_merchant_category_id_w2v_4',        436928.8732688427)\n('hist_new_card_std_merchant_category_id_w2v_1',    434849.51894950867)\n</code></p>\n\n<h1><strong>Single Model</strong></h1>\n\n<h2>Lightgbm</h2>\n\n<p>My best single model is a lightgbm has 385 features with CV:3.6144,LB:3.668,PB:3.593,maybe it can reach to 3th place.It also happened at homecredit competition,best single model is much better than stacking,I will doubt if stacking is a good choice.</p>\n\n<h2>Nueral Netowork</h2>\n\n<p>My best nn has CV:3.6407,LB:3.680,PB:3.605,it's just a simple 3layers-dense network.</p>\n\n<p><code>\ndef ann(input_shape):\n    model = Sequential()\n    model.add(Dense(2 ** 10, input_dim = input_shape, init='random_uniform', activation='relu'))\n    model.add(Dropout(0.25)) <br>\n    model.add(BatchNormalization())\n    model.add(Dense(2 ** 9, init='random_uniform', activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.25)) \n    model.add(Dense(2 ** 5, init='random_uniform', activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.25)) <br>\n    model.add(Dense(1))\n    model.compile(loss='mean_squared_error', optimizer='adam') \n    return model\n</code></p>\n\n<h2>Others</h2>\n\n<p>Xgboost and Catboost got not bad results but almost no improvement to stacking.\nAlso Tried Ridge,KNN,FFM...no success to me.</p>\n\n<h1><strong>Ensemble</strong></h1>\n\n<p>I built 12 lightgbm models and 40 nn models for Stacking stage1,then use lightgbm and nn as stage2,finally blend them with average.CV can reach to 3.603,but lb got worse,and final pb also got worse.</p>\n\n<h1><strong>Postprocessing</strong></h1>\n\n<p>Isotonic regression give me 0.005 ~ 0.006 boost both on cv and lb but pb score become worse,that's the reason I shakedown.\nCombination with a non-outliers model using binary model's best threshold give me 0.001 ~ 0.002 boost both on cv and lb and pb.</p>\n\n<h1><strong>Leaderboard Probing</strong></h1>\n\n<p>I used 100 subs to probe the lb,that let me grasp 24 public-test outliers and 6 non-outliers.\nI tried Pseudo-Labelling but no boost,finally it just can be used to decide a \"perfect threshold\" to override the outliers but failed.</p>\n\n<h1><strong>Other tricks</strong></h1>\n\n<p>at very early stage I tried target transformation,using minmaxscaler to transform from 0 ~ 1, then use xentropy boosting type,finally reversed to original value range,it didn't improve cv and lb, but I found pb is very good, a very eary stage lightgbm model is enough to get a gold medal.</p>\n\n<h1><strong>Late Submission Founding</strong></h1>\n\n<p>Isotonic regression overfit both on cv and lb, should be careful.\nStacking can work if stage-2 model is nn or ridge ,lgb and extratree overfit too much.</p>",
      "rawMarkdown": "Congrats to all winner teams and new masters,experts.It's a very hard competition,I think every team work hard to try everything and fight with overfitting every day,no one fails who does his best.\nI started at very early stage so maybe I just did more work than others,let me share my solution briefly.\n\n#**Preprocessing**\nsort the historical and new data by timestamp then create 8 kinds of dataset for different features.\n\n 1.  Only historical data\n 2. Only historical data with authorized_flag=1\n 3. Only new data \n 4. Merge of historical data with authorized_flag=1 and new data \n 5. Merge of historical data  and new data \n 6. Merge of historical data and merchants data\n 7. Merge of new data and merchants data\n 8. Merge of historical data  and new data and merchants data\n\n#**Feature Engineering**\n## Aggregation\nI think there are not big difference with everyone's aggregation features, I want to thank raddar's magic that purchase_amount_new can bring additional features beside of original purchase_amount.\n\n## Interval\ndays interval(the difference from current purchase_date to next purchase_date)\npurchase_amount interval(the difference from current purchase_amount to next purchase_amount)\n\n## Interaction\nnew.purchase_date.max() / historical.purchase_date.max() is very strong\n\n## SVD\ndimession reducing from original card_id's merchant|merchant_category transaction sequence to 5 \n```\ndef svd_feature(prefix, df, traintest, groupby, target,n_comp):\n    tfidf_vec = TfidfVectorizer(ngram_range=(1,1), max_features=None)\n    df_bag = pd.DataFrame(df[[groupby, target]])\n    df_bag = df_bag.groupby(groupby, as_index=False)[target].agg({'list':(lambda x: list(x))}).reset_index()\n    df_bag[target + '_list']=df_bag['list'].apply(lambda x: str(x).replace('[','').replace(']','').replace(',',' '))\n    \n    df_bag = df_bag.merge(traintest,on=groupby,how='left')\n    df_bag_train = df_bag[df_bag['target'].notnull()].reset_index(drop=True)\n    df_bag_test = df_bag[df_bag['target'].isnull()].reset_index(drop=True)\n    \n    tfidf_full_vector = tfidf_vec.fit_transform(df_bag[target + '_list'])\n    tfidf_train_vector = tfidf_vec.transform(df_bag_train[target + '_list'])\n    tfidf_test_vector = tfidf_vec.transform(df_bag_test[target + '_list'])\n    \n    svd_vec = TruncatedSVD(n_components=5, algorithm='arpack')\n    svd_vec.fit(tfidf_full_vector)\n    svd_train = pd.DataFrame(svd_vec.transform(tfidf_train_vector))\n    svd_test = pd.DataFrame(svd_vec.transform(tfidf_test_vector))\n\n    svd_train.columns = ['svd_%s_%s_%d'%(prefix,target,x) for x in range(n_comp)]\n    svd_train[groupby] = df_bag_train[groupby]\n    svd_test.columns = ['svd_%s_%s_%d'%(prefix,target,x) for x in range(n_comp)]\n    svd_test[groupby] = df_bag_test[groupby]\n    #df_svd = pd.concat([svd_train,svd_test],axis=0)\n    print ('svd_train:' + str(svd_train.shape))\n    print ('svd_test:' + str(svd_test.shape))\n    return svd_train,svd_test\n```\n\n## Word2vec\ncreate each merchant_id's , merchant_category_id's , purchase_date's ... word embedding,then group by card_id's min/max/mean/std\n```\ndef word2vec_feature(prefix, df, groupby, target,size):\n    df_bag = pd.DataFrame(df[[groupby, target]])\n    df_bag[target] = df_bag[target].astype(str)\n    df_bag[target].fillna('NAN', inplace=True)\n    df_bag = df_bag.groupby(groupby, as_index=False)[target].agg({'list':(lambda x: list(x))}).reset_index()\n    doc_list = list(df_bag['list'].values)\n    w2v = Word2Vec(doc_list, size=size, window=3, min_count=1, workers=32)\n    vocab_keys = list(w2v.wv.vocab.keys())\n    w2v_array = []\n    for v in vocab_keys :\n        w2v_array.append(list(w2v.wv[v]))\n    df_w2v = pd.DataFrame()\n    df_w2v['vocab_keys'] = vocab_keys    \n    df_w2v = pd.concat([df_w2v, pd.DataFrame(w2v_array)], axis=1)\n    df_w2v.columns = [target] + ['w2v_%s_%s_%d'%(prefix,target,x) for x in range(size)]\n    print ('df_w2v:' + str(df_w2v.shape))\n    return df_w2v\n```\n\n## Transaction-based Model\nuse above 6,7,8 dataset, merge train's target to each transaction row,then create a lightgbm to output each transaction's meta feature , then groupby card_id's min/sum to create card_id's meta feature , it can boost 0.005 ~ 0.006  both on cv and lb.\n\n## Feature Selection\nat early stage I add new features one by one,sometimes group by group,at late stage I use target permutation learned from https://www.kaggle.com/ogrellier/feature-selection-with-null-importances\nI created thousands of features,finally get 12 different feature sets with number from 200 ~ 700\n\n## Top 20 Feature Importance \n```\n('reborn_new_min_submodel', \t\t\t\t\t\t3212145.126355231)\n('reborn_hist_new_min_submodel', \t\t\t\t\t2897647.0672301054)\n('reborn_hist_min_submodel', \t\t\t\t\t\t2571620.7657690495)\n('reborn_hist_sum_submodel', \t\t\t\t\t\t1556242.6892476082)\n('reborn_hist_new_sum_submodel', \t\t\t\t\t1495420.6608867645)\n('card_purchase_date_max_ratio', \t\t\t\t\t1176560.0546371937)\n('reborn_new_sum_submodel', \t\t\t\t\t\t988300.9342432022)\n('hist_new_purchase_date_max_ratio', \t\t\t\t751098.8485645056)\n('hist_new_sum_category_1_1_purchase_amount', \t\t634349.3938679695)\n('hist_sum_category_1_1_purchase_amount', \t\t\t626576.9190101624)\n('new_card_sum_purchase_amount_new', \t\t\t\t573281.6908314228)\n('hist_card0_max_month_diff', \t\t\t\t\t\t539578.2926783562)\n('new_card_var_day_diff', \t\t\t\t\t\t\t537170.215294838)\n('new_card_last_day', \t\t\t\t\t\t\t\t532604.059334755)\n('hist_card_sum_month_lag_0_purchase_amount_new', \t530102.9405331612)\n('hist_card0_mean_month_diff', \t\t\t\t\t\t506467.2120089531)\n('svd_hist_new_merchant_id4', \t\t\t\t\t\t454442.183167696)\n('hist_card0_sum_month_diff', \t\t\t\t\t\t445077.63865828514)\n('hist_card_std_merchant_category_id_w2v_4', \t\t436928.8732688427)\n('hist_new_card_std_merchant_category_id_w2v_1', \t434849.51894950867)\n```\n\n# **Single Model**\n## Lightgbm\nMy best single model is a lightgbm has 385 features with CV:3.6144,LB:3.668,PB:3.593,maybe it can reach to 3th place.It also happened at homecredit competition,best single model is much better than stacking,I will doubt if stacking is a good choice.\n## Nueral Netowork\nMy best nn has CV:3.6407,LB:3.680,PB:3.605,it's just a simple 3layers-dense network.\n\n```\ndef ann(input_shape):\n    model = Sequential()\n    model.add(Dense(2 ** 10, input_dim = input_shape, init='random_uniform', activation='relu'))\n    model.add(Dropout(0.25))    \n    model.add(BatchNormalization())\n    model.add(Dense(2 ** 9, init='random_uniform', activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.25)) \n    model.add(Dense(2 ** 5, init='random_uniform', activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.25))      \n    model.add(Dense(1))\n    model.compile(loss='mean_squared_error', optimizer='adam') \n    return model\n```\n\n## Others\nXgboost and Catboost got not bad results but almost no improvement to stacking.\nAlso Tried Ridge,KNN,FFM...no success to me.\n\n\n# **Ensemble**\nI built 12 lightgbm models and 40 nn models for Stacking stage1,then use lightgbm and nn as stage2,finally blend them with average.CV can reach to 3.603,but lb got worse,and final pb also got worse.\n\n# **Postprocessing**\nIsotonic regression give me 0.005 ~ 0.006 boost both on cv and lb but pb score become worse,that's the reason I shakedown.\nCombination with a non-outliers model using binary model's best threshold give me 0.001 ~ 0.002 boost both on cv and lb and pb.\n\n# **Leaderboard Probing**\nI used 100 subs to probe the lb,that let me grasp 24 public-test outliers and 6 non-outliers.\nI tried Pseudo-Labelling but no boost,finally it just can be used to decide a \"perfect threshold\" to override the outliers but failed.\n\n#**Other tricks**\nat very early stage I tried target transformation,using minmaxscaler to transform from 0 ~ 1, then use xentropy boosting type,finally reversed to original value range,it didn't improve cv and lb, but I found pb is very good, a very eary stage lightgbm model is enough to get a gold medal.\n\n#**Late Submission Founding**\nIsotonic regression overfit both on cv and lb, should be careful.\nStacking can work if stage-2 model is nn or ridge ,lgb and extratree overfit too much.",
      "votes": null
    },
    {
      "id": "479206",
      "postDate": "02/27/2019 02:46:58",
      "content": "<p>Congrats on your solo gold! GM on the way! </p>",
      "rawMarkdown": "Congrats on your solo gold! GM on the way!",
      "votes": null
    },
    {
      "id": "479207",
      "postDate": "02/27/2019 02:47:03",
      "content": "<p>Congratulations and thanks for you sharing! What's the features of your best single model?</p>",
      "rawMarkdown": "Congratulations and thanks for you sharing! What's the features of your best single model?",
      "votes": null
    },
    {
      "id": "479210",
      "postDate": "02/27/2019 02:47:34",
      "content": "<p>Amazing work man, thank you</p>",
      "rawMarkdown": "Amazing work man, thank you",
      "votes": null
    },
    {
      "id": "479216",
      "postDate": "02/27/2019 02:52:34",
      "content": "<p>Congratulations and thanks for you sharing! </p>",
      "rawMarkdown": "Congratulations and thanks for you sharing!",
      "votes": null
    },
    {
      "id": "479246",
      "postDate": "02/27/2019 03:09:56",
      "content": "<p>Amazing work！thank you for sharing!</p>",
      "rawMarkdown": "Amazing work！thank you for sharing!",
      "votes": null
    },
    {
      "id": "479269",
      "postDate": "02/27/2019 03:27:44",
      "content": "<p>about 300 features</p>",
      "rawMarkdown": "about 300 features",
      "votes": null
    },
    {
      "id": "479270",
      "postDate": "02/27/2019 03:28:16",
      "content": "<p>Come with me</p>",
      "rawMarkdown": "Come with me",
      "votes": null
    },
    {
      "id": "479272",
      "postDate": "02/27/2019 03:28:39",
      "content": "<p>Congratulations your first gold medal.</p>",
      "rawMarkdown": "Congratulations your first gold medal.",
      "votes": null
    },
    {
      "id": "479273",
      "postDate": "02/27/2019 03:29:14",
      "content": "<p>Congratulations your first silver medal.</p>",
      "rawMarkdown": "Congratulations your first silver medal.",
      "votes": null
    },
    {
      "id": "479275",
      "postDate": "02/27/2019 03:29:39",
      "content": "<p>your team is very strong</p>",
      "rawMarkdown": "your team is very strong",
      "votes": null
    },
    {
      "id": "479276",
      "postDate": "02/27/2019 03:30:18",
      "content": "<p>Congratulations <a href=\"/senkin13\">@senkin13</a> on your solo gold and thanks for sharing your solution overview. I also used that Olivier's kernel plus manual methods for my feature selection. Here is my post on feature selection techniques\n<a href=\"https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052\">https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052</a></p>\n\n<blockquote>\n  <p>finally get 12 different feature sets with number from 200 ~ 700</p>\n</blockquote>\n\n<p>Is 200 ~ 700 the split numbers from your null hypothesis result?</p>",
      "rawMarkdown": "Congratulations @senkin13 on your solo gold and thanks for sharing your solution overview. I also used that Olivier's kernel plus manual methods for my feature selection. Here is my post on feature selection techniques\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052\n\n&gt;finally get 12 different feature sets with number from 200 ~ 700\n\nIs 200 ~ 700 the split numbers from your null hypothesis result?",
      "votes": null
    },
    {
      "id": "479279",
      "postDate": "02/27/2019 03:36:16",
      "content": "<p>Thanks, I think Olivier is deserved every kagglers' upvote</p>",
      "rawMarkdown": "Thanks, I think Olivier is deserved every kagglers' upvote",
      "votes": null
    },
    {
      "id": "479286",
      "postDate": "02/27/2019 03:41:17",
      "content": "<p>I manully start different features combination then use target permutation,choose the best importance threshold,the threshold is from 200 ~ 700</p>",
      "rawMarkdown": "I manully start different features combination then use target permutation,choose the best importance threshold,the threshold is from 200 ~ 700",
      "votes": null
    },
    {
      "id": "479298",
      "postDate": "02/27/2019 03:51:57",
      "content": "<p>Congrats man on your gold. I learned so much from your approach. Thanks for this detailed explanation :)</p>",
      "rawMarkdown": "Congrats man on your gold. I learned so much from your approach. Thanks for this detailed explanation :)",
      "votes": null
    },
    {
      "id": "479302",
      "postDate": "02/27/2019 03:57:08",
      "content": "<p>Edit: Didn't refresh the page and therefore missed your explanation above, sorry about that. Please ignore.</p>\n\n<p>Thank you very much for your sharing! May I know how you get your 12 different feature sets after doing feature selection? I also thought about dividing my features into sets (to train models for stacking) after reading your team's sharing for the Home Credit Default Risk competition, but didn't have enough time to think through how exactly to do it.</p>\n\n<p>I understand that after performing target permutation following Olivier's kernel, we can get a more realistic importance for each feature. May I know how you proceeded from there? (Initially I planned to try using top 10 features in each feature set and then randomly draw 190 more from the top 11-500 features to add to each set, so each set has 10+190=200 features).</p>",
      "rawMarkdown": "Edit: Didn't refresh the page and therefore missed your explanation above, sorry about that. Please ignore.\n\nThank you very much for your sharing! May I know how you get your 12 different feature sets after doing feature selection? I also thought about dividing my features into sets (to train models for stacking) after reading your team's sharing for the Home Credit Default Risk competition, but didn't have enough time to think through how exactly to do it.\n\nI understand that after performing target permutation following Olivier's kernel, we can get a more realistic importance for each feature. May I know how you proceeded from there? (Initially I planned to try using top 10 features in each feature set and then randomly draw 190 more from the top 11-500 features to add to each set, so each set has 10+190=200 features).",
      "votes": null
    },
    {
      "id": "479303",
      "postDate": "02/27/2019 03:58:16",
      "content": "<p>Congratulations and thanks for sharing! Another GM is coming!</p>",
      "rawMarkdown": "Congratulations and thanks for sharing! Another GM is coming!",
      "votes": null
    },
    {
      "id": "479304",
      "postDate": "02/27/2019 03:58:28",
      "content": "<p>Congratulations and thanks for you sharing!</p>",
      "rawMarkdown": "Congratulations and thanks for you sharing!",
      "votes": null
    },
    {
      "id": "479339",
      "postDate": "02/27/2019 04:37:43",
      "content": "<p>Great post. Congratulations <a href=\"/senkin13\">@senkin13</a>. I've so much to learn.</p>\n\n<p>I want to understand the below piece from you approach. Can you explain a bit more on this?</p>\n\n<p><strong>dimession reducing from original cardid's merchant|merchantcategory transaction sequence to 5</strong></p>\n\n<p>Thanks.</p>",
      "rawMarkdown": "Great post. Congratulations @senkin13. I've so much to learn.\n\nI want to understand the below piece from you approach. Can you explain a bit more on this?\n\n**dimession reducing from original cardid's merchant|merchantcategory transaction sequence to 5**\n\nThanks.",
      "votes": null
    },
    {
      "id": "479357",
      "postDate": "02/27/2019 05:00:17",
      "content": "<p>Congratulations your first silver medal.</p>",
      "rawMarkdown": "Congratulations your first silver medal.",
      "votes": null
    },
    {
      "id": "479361",
      "postDate": "02/27/2019 05:02:33",
      "content": "<p>Thanks for writing this up, this is quite exhaustive.. </p>",
      "rawMarkdown": "Thanks for writing this up, this is quite exhaustive..",
      "votes": null
    },
    {
      "id": "479365",
      "postDate": "02/27/2019 05:07:13",
      "content": "<p>when new features can not improve the best model,I start to build another model with those features and add new from different idea that give me 12 different feature sets.\nI didn't spend too much time to choose features, just try some thresholds to choose top N important features.</p>",
      "rawMarkdown": "when new features can not improve the best model,I start to build another model with those features and add new from different idea that give me 12 different feature sets.\nI didn't spend too much time to choose features, just try some thresholds to choose top N important features.",
      "votes": null
    },
    {
      "id": "479386",
      "postDate": "02/27/2019 05:28:41",
      "content": "<p>I add  source code to post</p>",
      "rawMarkdown": "I add  source code to post",
      "votes": null
    },
    {
      "id": "479388",
      "postDate": "02/27/2019 05:29:00",
      "content": "<p>I saw your mail,I have added  source code to post</p>",
      "rawMarkdown": "I saw your mail,I have added  source code to post",
      "votes": null
    },
    {
      "id": "479398",
      "postDate": "02/27/2019 05:34:11",
      "content": "<p>thanks, the door is half opened</p>",
      "rawMarkdown": "thanks, the door is half opened",
      "votes": null
    },
    {
      "id": "479401",
      "postDate": "02/27/2019 05:35:18",
      "content": "<p>Also congratulations your first silver medal.</p>",
      "rawMarkdown": "Also congratulations your first silver medal.",
      "votes": null
    },
    {
      "id": "479405",
      "postDate": "02/27/2019 05:37:47",
      "content": "<p>thanks</p>",
      "rawMarkdown": "thanks",
      "votes": null
    },
    {
      "id": "479414",
      "postDate": "02/27/2019 05:49:20",
      "content": "<p>I agree Olivier deserves our gratitude and upvotes for that kernel and many more.</p>\n\n<p>&gt; ('reborn_new_min_submodel',                         3212145.126355231)\n('reborn_hist_new_min_submodel',                     2897647.0672301054)\n('reborn_hist_min_submodel',                         2571620.7657690495)</p>\n\n<p>So you meant the gain value of 200 ~ 700. So those numbers you have in front of the features is not the gain?</p>",
      "rawMarkdown": "I agree Olivier deserves our gratitude and upvotes for that kernel and many more.\n\n&gt; ('reborn_new_min_submodel',                         3212145.126355231)\n('reborn_hist_new_min_submodel',                     2897647.0672301054)\n('reborn_hist_min_submodel',                         2571620.7657690495)\n\nSo you meant the gain value of 200 ~ 700. So those numbers you have in front of the features is not the gain?",
      "votes": null
    },
    {
      "id": "479420",
      "postDate": "02/27/2019 05:52:39",
      "content": "<p>no it's feature numbers, we use \n<code>scores_df = pd.DataFrame(feature_scores, columns=['feature', 'split_score', 'gain_score']) \n</code>\nthe gain_score is 1.0 or 1.5 or 0 due to your cv performance</p>",
      "rawMarkdown": "no it's feature numbers, we use \n```scores_df = pd.DataFrame(feature_scores, columns=['feature', 'split_score', 'gain_score']) \n```\nthe gain_score is 1.0 or 1.5 or 0 due to your cv performance",
      "votes": null
    },
    {
      "id": "479427",
      "postDate": "02/27/2019 06:03:14",
      "content": "<p>Congrats. Next GM is coming. Thanks for your sharing.</p>",
      "rawMarkdown": "Congrats. Next GM is coming. Thanks for your sharing.",
      "votes": null
    },
    {
      "id": "479430",
      "postDate": "02/27/2019 06:08:22",
      "content": "<p>Gongratulations, and special thanks to your sharing. And could you please ellaborate on what is <code>transactions meta feature</code>, and what is <code>cardids meta feature</code>? Is it used in stacking?</p>\n\n<blockquote>\n  <p>use above 6,7,8 dataset, merge train's target to each transaction\n  row,then create a lightgbm to output each transaction's meta feature ,\n  then groupby cardid's min/sum to create cardid's meta feature , it can\n  boost 0.005 ~ 0.006 both on cv and lb.</p>\n</blockquote>",
      "rawMarkdown": "Gongratulations, and special thanks to your sharing. And could you please ellaborate on what is `transactions meta feature`, and what is `cardids meta feature`? Is it used in stacking?\n&gt; use above 6,7,8 dataset, merge train's target to each transaction\n&gt; row,then create a lightgbm to output each transaction's meta feature ,\n&gt; then groupby cardid's min/sum to create cardid's meta feature , it can\n&gt; boost 0.005 ~ 0.006 both on cv and lb.",
      "votes": null
    },
    {
      "id": "479438",
      "postDate": "02/27/2019 06:18:34",
      "content": "<p>Congratulations! And thanks for you sharing.</p>",
      "rawMarkdown": "Congratulations! And thanks for you sharing.",
      "votes": null
    },
    {
      "id": "479452",
      "postDate": "02/27/2019 06:37:32",
      "content": "<p>similar with this post from homecredit solution\n<a href=\"https://www.kaggle.com/c/home-credit-default-risk/discussion/64503\">https://www.kaggle.com/c/home-credit-default-risk/discussion/64503</a></p>",
      "rawMarkdown": "similar with this post from homecredit solution\nhttps://www.kaggle.com/c/home-credit-default-risk/discussion/64503",
      "votes": null
    },
    {
      "id": "479453",
      "postDate": "02/27/2019 06:38:49",
      "content": "<p>your team did well</p>",
      "rawMarkdown": "your team did well",
      "votes": null
    },
    {
      "id": "479455",
      "postDate": "02/27/2019 06:40:15",
      "content": "<p>Congratulations your first silver medal.</p>",
      "rawMarkdown": "Congratulations your first silver medal.",
      "votes": null
    },
    {
      "id": "479462",
      "postDate": "02/27/2019 06:54:59",
      "content": "<p>I see. Thanks for the reply. And thank you again for your sharing!</p>",
      "rawMarkdown": "I see. Thanks for the reply. And thank you again for your sharing!",
      "votes": null
    },
    {
      "id": "479467",
      "postDate": "02/27/2019 06:57:38",
      "content": "<p>Got it top 200 ~ 700 features. Thanks.</p>",
      "rawMarkdown": "Got it top 200 ~ 700 features. Thanks.",
      "votes": null
    },
    {
      "id": "479557",
      "postDate": "02/27/2019 08:42:18",
      "content": "<p>Congrats to your gold metal !  Thank you for sharing! \nThe final trick (using minmaxscaler to transform from 0 ~ 1 ...) is interesting.\nI learned so much from this competition and your sharing!</p>",
      "rawMarkdown": "Congrats to your gold metal !  Thank you for sharing! \nThe final trick (using minmaxscaler to transform from 0 ~ 1 ...) is interesting.\nI learned so much from this competition and your sharing!",
      "votes": null
    },
    {
      "id": "479566",
      "postDate": "02/27/2019 08:49:22",
      "content": "<p>Thanks for sharing and Congratulations.  Learned a lot from your solution. Will try to replicate your solution.</p>",
      "rawMarkdown": "Thanks for sharing and Congratulations.  Learned a lot from your solution. Will try to replicate your solution.",
      "votes": null
    },
    {
      "id": "479584",
      "postDate": "02/27/2019 09:01:16",
      "content": "<p>I like it too,unfortunately I stop to deepdive </p>",
      "rawMarkdown": "I like it too,unfortunately I stop to deepdive",
      "votes": null
    },
    {
      "id": "479589",
      "postDate": "02/27/2019 09:04:45",
      "content": "<p>Congrats and thanks for sharing!</p>",
      "rawMarkdown": "Congrats and thanks for sharing!",
      "votes": null
    },
    {
      "id": "479595",
      "postDate": "02/27/2019 09:14:55",
      "content": "<p>Hi senkin13!\n  Can you share how many boost with word2vec and svd features. i tried svd for card_id -merchant-category matrix, but failed to boost the public lb. thank you!</p>",
      "rawMarkdown": "Hi senkin13!\n  Can you share how many boost with word2vec and svd features. i tried svd for card_id -merchant-category matrix, but failed to boost the public lb. thank you!",
      "votes": null
    },
    {
      "id": "479602",
      "postDate": "02/27/2019 09:18:53",
      "content": "<p>Basicly I don't select features due to LB, I think I got 0.002 cv boost from svd features when my cv is 3.632,I combinate only w2v features to train.csv ,can boost from 3.84 to 3.67</p>",
      "rawMarkdown": "Basicly I don't select features due to LB, I think I got 0.002 cv boost from svd features when my cv is 3.632,I combinate only w2v features to train.csv ,can boost from 3.84 to 3.67",
      "votes": null
    },
    {
      "id": "479603",
      "postDate": "02/27/2019 09:19:38",
      "content": "<p>Thanks</p>",
      "rawMarkdown": "Thanks",
      "votes": null
    },
    {
      "id": "479605",
      "postDate": "02/27/2019 09:20:32",
      "content": "<p>yeah, reproduce solution is good way to learn.</p>",
      "rawMarkdown": "yeah, reproduce solution is good way to learn.",
      "votes": null
    },
    {
      "id": "479610",
      "postDate": "02/27/2019 09:26:28",
      "content": "<p>3.67 mean local cv?</p>",
      "rawMarkdown": "3.67 mean local cv?",
      "votes": null
    },
    {
      "id": "479624",
      "postDate": "02/27/2019 09:43:01",
      "content": "<p>yes,local cv.I merged all the features seems good to create a best single model.</p>",
      "rawMarkdown": "yes,local cv.I merged all the features seems good to create a best single model.",
      "votes": null
    },
    {
      "id": "479688",
      "postDate": "02/27/2019 10:43:36",
      "content": "<p>Congrats for yet another solo gold!</p>",
      "rawMarkdown": "Congrats for yet another solo gold!",
      "votes": null
    },
    {
      "id": "479697",
      "postDate": "02/27/2019 10:52:14",
      "content": "<p>Congrats and thanks for detailed sharing!  </p>",
      "rawMarkdown": "Congrats and thanks for detailed sharing!",
      "votes": null
    },
    {
      "id": "479707",
      "postDate": "02/27/2019 11:02:02",
      "content": "<p>Very unique Feature Engineering!!\nGreat ideas!\nThanks for sharing</p>",
      "rawMarkdown": "Very unique Feature Engineering!!\nGreat ideas!\nThanks for sharing",
      "votes": null
    },
    {
      "id": "479781",
      "postDate": "02/27/2019 12:24:58",
      "content": "<p>@senkins Your method about start to build another model inspired me ! as a new kaggler, i'm confused  that time whatever features i add the public lb get worse base my 3.672 public lb single model.</p>",
      "rawMarkdown": "senkins Your method about start to build another model inspired me ! as a new kaggler, i'm confused  that time whatever features i add the public lb get worse base my 3.672 public lb single model.",
      "votes": null
    },
    {
      "id": "479819",
      "postDate": "02/27/2019 12:59:35",
      "content": "<p>Congratulations and thanks for sharing your solution.\nFor several weeks I was so frustrated not being able to close the gap with top competitors. It was clear I was missing some key ideas. Reading your post helped me understand what I was missing.</p>",
      "rawMarkdown": "Congratulations and thanks for sharing your solution.\nFor several weeks I was so frustrated not being able to close the gap with top competitors. It was clear I was missing some key ideas. Reading your post helped me understand what I was missing.",
      "votes": null
    },
    {
      "id": "479847",
      "postDate": "02/27/2019 13:47:08",
      "content": "<p>Congrats!!</p>\n\n<p>Thank you for explaining the solution. This clearly shows all the hard work and thinking you have put into the competition.</p>",
      "rawMarkdown": "Congrats!!\n\nThank you for explaining the solution. This clearly shows all the hard work and thinking you have put into the competition.",
      "votes": null
    },
    {
      "id": "479856",
      "postDate": "02/27/2019 13:59:10",
      "content": "<p>Congratulations! Very impressive decision! Do I understand you right, your lgb single model is better than your stacking? </p>",
      "rawMarkdown": "Congratulations! Very impressive decision! Do I understand you right, your lgb single model is better than your stacking?",
      "votes": null
    },
    {
      "id": "479883",
      "postDate": "02/27/2019 14:31:03",
      "content": "<p>thanks,your 4 days journey is impressive</p>",
      "rawMarkdown": "thanks,your 4 days journey is impressive",
      "votes": null
    },
    {
      "id": "479887",
      "postDate": "02/27/2019 14:32:53",
      "content": "<p>I read your github and learned something useful :)</p>",
      "rawMarkdown": "I read your github and learned something useful :)",
      "votes": null
    },
    {
      "id": "479890",
      "postDate": "02/27/2019 14:35:53",
      "content": "<p>normal features worked unwell that have to try new ideas</p>",
      "rawMarkdown": "normal features worked unwell that have to try new ideas",
      "votes": null
    },
    {
      "id": "479892",
      "postDate": "02/27/2019 14:39:22",
      "content": "<p>yes,kaggle is the best place to learn pratical skills</p>",
      "rawMarkdown": "yes,kaggle is the best place to learn pratical skills",
      "votes": null
    },
    {
      "id": "479897",
      "postDate": "02/27/2019 14:44:40",
      "content": "<p>Thanks, trial and error are necessary</p>",
      "rawMarkdown": "Thanks, trial and error are necessary",
      "votes": null
    },
    {
      "id": "479899",
      "postDate": "02/27/2019 14:46:38",
      "content": "<p>yes, single lgb is better than stacking on private score,so I think stacking  overfit too much.</p>",
      "rawMarkdown": "yes, single lgb is better than stacking on private score,so I think stacking  overfit too much.",
      "votes": null
    },
    {
      "id": "480212",
      "postDate": "02/27/2019 23:33:51",
      "content": "<p>Congrats! senkin13! <br>\nThank you for writing in detail the solution method filled with your ideas. <br>\nI have a question about your NN. <br>\nThe loss function is MSE instead of RMSE. Did MSE go better?\n(btw I also had single LGB in gold zone. robust model is difficult.)</p>",
      "rawMarkdown": "Congrats! senkin13!  \nThank you for writing in detail the solution method filled with your ideas.  \nI have a question about your NN.  \nThe loss function is MSE instead of RMSE. Did MSE go better?\n(btw I also had single LGB in gold zone. robust model is difficult.)",
      "votes": null
    },
    {
      "id": "480227",
      "postDate": "02/28/2019 00:15:59",
      "content": "<p>So creative of applying NLP here. Congrats.</p>",
      "rawMarkdown": "So creative of applying NLP here. Congrats.",
      "votes": null
    },
    {
      "id": "480260",
      "postDate": "02/28/2019 01:42:03",
      "content": "<p>No, I think best mse equal to best rmse ,isn't it?\nI never have shakeup experience,so I think should make the single model strong enough to prevent huge shakedown.</p>",
      "rawMarkdown": "No, I think best mse equal to best rmse ,isn't it?\nI never have shakeup experience,so I think should make the single model strong enough to prevent huge shakedown.",
      "votes": null
    },
    {
      "id": "480264",
      "postDate": "02/28/2019 01:50:07",
      "content": "<p>Thanks, tabular data competitions are so attractive  that you never know which technology can work which not</p>",
      "rawMarkdown": "Thanks, tabular data competitions are so attractive  that you never know which technology can work which not",
      "votes": null
    },
    {
      "id": "480268",
      "postDate": "02/28/2019 02:05:32",
      "content": "<p>Yes, I think that is the same. I was on the same RMSE as the evaluation, so I asked a question as to whether there is meaning in the different thing. Thank you for reply.\nI may choose a single model as a candidate next.</p>",
      "rawMarkdown": "Yes, I think that is the same. I was on the same RMSE as the evaluation, so I asked a question as to whether there is meaning in the different thing. Thank you for reply.\nI may choose a single model as a candidate next.",
      "votes": null
    },
    {
      "id": "480537",
      "postDate": "02/28/2019 10:08:35",
      "content": "<p>Thanks for sharing! It seems to be one of the most complete feature engineering. And it is a valuable post I will refer and try in the  future.</p>",
      "rawMarkdown": "Thanks for sharing! It seems to be one of the most complete feature engineering. And it is a valuable post I will refer and try in the  future.",
      "votes": null
    },
    {
      "id": "480545",
      "postDate": "02/28/2019 10:15:28",
      "content": "<p>Thanks , I am waiting for your team's solution.</p>",
      "rawMarkdown": "Thanks , I am waiting for your team's solution.",
      "votes": null
    },
    {
      "id": "480631",
      "postDate": "02/28/2019 13:03:54",
      "content": "<p>Haha, I will ask our teammates. One member from Japan contributed the best sub. Some of the feature names are in Japanese, which I am still understanding and reproducing.😓</p>",
      "rawMarkdown": "Haha, I will ask our teammates. One member from Japan contributed the best sub. Some of the feature names are in Japanese, which I am still understanding and reproducing.😓",
      "votes": null
    },
    {
      "id": "480812",
      "postDate": "02/28/2019 17:48:07",
      "content": "<p>Congrads a looot! I’m curious about isotonic regression you mentioned in post processing. I though it was not useful unless distribution of target is mixed and unclear. Why do you think isotonic regression of true labels from your ensembled results worked so well here?</p>",
      "rawMarkdown": "Congrads a looot! I’m curious about isotonic regression you mentioned in post processing. I though it was not useful unless distribution of target is mixed and unclear. Why do you think isotonic regression of true labels from your ensembled results worked so well here?",
      "votes": null
    },
    {
      "id": "480880",
      "postDate": "02/28/2019 20:14:01",
      "content": "<p>Great work! Congradulations! Thanks for sharing!</p>\n\n<p>I've used tfidf merchants and Ridge to precict target and when add it like feature, it get 0.01+ . Also I tried to predict new_merchants purchase with help SVD and calculate \"difference\" between predictions and new_merchant_transactions marchants. Dont know what exactly work, but i get my first bronze)))</p>",
      "rawMarkdown": "Great work! Congradulations! Thanks for sharing!\n\nI've used tfidf merchants and Ridge to precict target and when add it like feature, it get 0.01+ . Also I tried to predict new_merchants purchase with help SVD and calculate \"difference\" between predictions and new_merchant_transactions marchants. Dont know what exactly work, but i get my first bronze)))",
      "votes": null
    },
    {
      "id": "480982",
      "postDate": "03/01/2019 00:19:41",
      "content": "<p>Congratulations</p>",
      "rawMarkdown": "Congratulations",
      "votes": null
    },
    {
      "id": "481347",
      "postDate": "03/01/2019 09:39:17",
      "content": "<p>you are right,after testing late submission,I found isotonic regression overfit too much,that's why I shakedown.</p>",
      "rawMarkdown": "you are right,after testing late submission,I found isotonic regression overfit too much,that's why I shakedown.",
      "votes": null
    },
    {
      "id": "481360",
      "postDate": "03/01/2019 09:59:36",
      "content": "<p>Congrats! nice idea of using Ridge and tfidf,I will try to use that at next comp.</p>",
      "rawMarkdown": "Congrats! nice idea of using Ridge and tfidf,I will try to use that at next comp.",
      "votes": null
    },
    {
      "id": "481362",
      "postDate": "03/01/2019 09:59:54",
      "content": "<p>Same to you.</p>",
      "rawMarkdown": "Same to you.",
      "votes": null
    },
    {
      "id": "481947",
      "postDate": "03/02/2019 04:54:23",
      "content": "<p>Thanks sharing and Congratulations Gold medal!!! </p>\n\n<p>I have some question. I don't understand <code>Transaction Base Model</code>. it is right? Make a 6,7,8 (don't groupby card_id) -&gt; Merge target to each 6, 7, 8 -&gt;  lgb transaction's meta feature(?)</p>\n\n<p>sorry. I have no idea about this part.</p>",
      "rawMarkdown": "Thanks sharing and Congratulations Gold medal!!! \n\nI have some question. I don't understand `Transaction Base Model`. it is right? Make a 6,7,8 (don't groupby card_id) -&gt; Merge target to each 6, 7, 8 -&gt;  lgb transaction's meta feature(?)\n\nsorry. I have no idea about this part.",
      "votes": null
    },
    {
      "id": "482250",
      "postDate": "03/02/2019 15:28:31",
      "content": "<p>don't use groupby features,like this\n<code>historical_train = historical.merge(train,on='card_id',how='right')\nhistorical_test = historical.merge(test,on='card_id',how='right')</code></p>",
      "rawMarkdown": "don't use groupby features,like this\n```historical_train = historical.merge(train,on='card_id',how='right')\nhistorical_test = historical.merge(test,on='card_id',how='right')```",
      "votes": null
    },
    {
      "id": "482317",
      "postDate": "03/02/2019 18:04:35",
      "content": "<p><a href=\"/senkin13\">@senkin13</a> Could you explain your word2vec function more? I didnt understand this - \"then group by card_id's min/max/mean/std\". How do you perform this grouping</p>",
      "rawMarkdown": "senkin13 Could you explain your word2vec function more? I didnt understand this - \"then group by card_id's min/max/mean/std\". How do you perform this grouping",
      "votes": null
    },
    {
      "id": "482436",
      "postDate": "03/03/2019 01:07:56",
      "content": "<p>Thanks for sharing!</p>",
      "rawMarkdown": "Thanks for sharing!",
      "votes": null
    },
    {
      "id": "482529",
      "postDate": "03/03/2019 06:42:25",
      "content": "<p>Thanks for sharing.\nMay I ask you a question?  What's mean of \"Interaction:  new.purchasedate.max() / historical.purchasedate.max() \" ? </p>",
      "rawMarkdown": "Thanks for sharing.\nMay I ask you a question?  What's mean of \"Interaction:  new.purchasedate.max() / historical.purchasedate.max() \" ?",
      "votes": null
    },
    {
      "id": "483113",
      "postDate": "03/04/2019 06:45:13",
      "content": "<p>```\nfor f in [\n          'new_card_purchase_date_max',\n          'hist_card_purchase_date_max',]:\n    df[f +'_int'] = df[f].astype(np.int64) * 1e-9</p>\n\n<p>df['card_purchase_date_max_ratio'] = df['new_card_purchase_date_max_int'] / df['hist_card_purchase_date_max_int']\n```</p>",
      "rawMarkdown": "```\nfor f in [\n          'new_card_purchase_date_max',\n          'hist_card_purchase_date_max',]:\n    df[f +'_int'] = df[f].astype(np.int64) * 1e-9\n\ndf['card_purchase_date_max_ratio'] = df['new_card_purchase_date_max_int'] / df['hist_card_purchase_date_max_int']\n```",
      "votes": null
    },
    {
      "id": "483115",
      "postDate": "03/04/2019 06:50:01",
      "content": "<p>after trained a word2vec model,one merchant_id should has a vector like [w2v_merchant_1,w2v_merchant_2,w2v_merchant_3,w2v_merchant_4,w2v_merchant_5],then you can use historical or new data to create aggregation features like groupby(['card_id'])['w2v_merchant_1'].mean()</p>",
      "rawMarkdown": "after trained a word2vec model,one merchant_id should has a vector like [w2v_merchant_1,w2v_merchant_2,w2v_merchant_3,w2v_merchant_4,w2v_merchant_5],then you can use historical or new data to create aggregation features like groupby(['card_id'])['w2v_merchant_1'].mean()",
      "votes": null
    },
    {
      "id": "483116",
      "postDate": "03/04/2019 06:50:21",
      "content": "<p>thanks</p>",
      "rawMarkdown": "thanks",
      "votes": null
    },
    {
      "id": "483943",
      "postDate": "03/05/2019 11:32:44",
      "content": "<p>Thank you for your reply. I am interested in this feature. I will try it for testing. </p>",
      "rawMarkdown": "Thank you for your reply. I am interested in this feature. I will try it for testing.",
      "votes": null
    },
    {
      "id": "736424",
      "postDate": "02/04/2020 06:58:32",
      "content": "<p>thanks!  learned a lot from this </p>",
      "rawMarkdown": "thanks!  learned a lot from this",
      "votes": null
    },
    {
      "id": "739463",
      "postDate": "02/07/2020 21:04:30",
      "content": "<p><a href=\"/senkin13\">@senkin13</a>, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (<a href=\"https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques\">https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques</a>)</p>",
      "rawMarkdown": "senkin13, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques)",
      "votes": null
    },
    {
      "id": "1526662",
      "postDate": "09/28/2021 09:16:27",
      "content": "<p>Thanks for the excellent insights.<br>\nMay I know what are being referred in the features with “reborn… submodel” :</p>\n<p>('reborn_new_min_submodel',                         3212145.126355231)<br>\n('reborn_hist_new_min_submodel',                     2897647.0672301054)<br>\n('reborn_hist_min_submodel',                         2571620.7657690495)<br>\n('reborn_hist_sum_submodel',                         1556242.6892476082)<br>\n('reborn_hist_new_sum_submodel', …)</p>",
      "rawMarkdown": "Thanks for the excellent insights.\nMay I know what are being referred in the features with “reborn... submodel” :\n\n('reborn_new_min_submodel',                         3212145.126355231)\n('reborn_hist_new_min_submodel',                     2897647.0672301054)\n('reborn_hist_min_submodel',                         2571620.7657690495)\n('reborn_hist_sum_submodel',                         1556242.6892476082)\n('reborn_hist_new_sum_submodel', ...)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1526662,
      "author_name": "samaujs",
      "author_url": "",
      "post_date": "09/28/2021 09:16:27",
      "content": "<p>Thanks for the excellent insights.<br>\nMay I know what are being referred in the features with “reborn… submodel” :</p>\n<p>('reborn_new_min_submodel',                         3212145.126355231)<br>\n('reborn_hist_new_min_submodel',                     2897647.0672301054)<br>\n('reborn_hist_min_submodel',                         2571620.7657690495)<br>\n('reborn_hist_sum_submodel',                         1556242.6892476082)<br>\n('reborn_hist_new_sum_submodel', …)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 479206,
      "author_name": "naivelamb",
      "author_url": "",
      "post_date": "02/27/2019 02:46:58",
      "content": "<p>Congrats on your solo gold! GM on the way! </p>",
      "votes": null,
      "replies": [
        {
          "id": 479270,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 03:28:16",
          "content": "<p>Come with me</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479207,
      "author_name": "jiazhuang",
      "author_url": "",
      "post_date": "02/27/2019 02:47:03",
      "content": "<p>Congratulations and thanks for you sharing! What's the features of your best single model?</p>",
      "votes": null,
      "replies": [
        {
          "id": 479269,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 03:27:44",
          "content": "<p>about 300 features</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479210,
      "author_name": "codlife",
      "author_url": "",
      "post_date": "02/27/2019 02:47:34",
      "content": "<p>Amazing work man, thank you</p>",
      "votes": null,
      "replies": [
        {
          "id": 479272,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 03:28:39",
          "content": "<p>Congratulations your first gold medal.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479216,
      "author_name": "clown9804",
      "author_url": "",
      "post_date": "02/27/2019 02:52:34",
      "content": "<p>Congratulations and thanks for you sharing! </p>",
      "votes": null,
      "replies": [
        {
          "id": 479273,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 03:29:14",
          "content": "<p>Congratulations your first silver medal.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479246,
      "author_name": "chizhu2018",
      "author_url": "",
      "post_date": "02/27/2019 03:09:56",
      "content": "<p>Amazing work！thank you for sharing!</p>",
      "votes": null,
      "replies": [
        {
          "id": 479275,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 03:29:39",
          "content": "<p>your team is very strong</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479276,
      "author_name": "sheriytm",
      "author_url": "",
      "post_date": "02/27/2019 03:30:18",
      "content": "<p>Congratulations <a href=\"/senkin13\">@senkin13</a> on your solo gold and thanks for sharing your solution overview. I also used that Olivier's kernel plus manual methods for my feature selection. Here is my post on feature selection techniques\n<a href=\"https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052\">https://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052</a></p>\n\n<blockquote>\n  <p>finally get 12 different feature sets with number from 200 ~ 700</p>\n</blockquote>\n\n<p>Is 200 ~ 700 the split numbers from your null hypothesis result?</p>",
      "votes": null,
      "replies": [
        {
          "id": 479279,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 03:36:16",
          "content": "<p>Thanks, I think Olivier is deserved every kagglers' upvote</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 479286,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 03:41:17",
          "content": "<p>I manully start different features combination then use target permutation,choose the best importance threshold,the threshold is from 200 ~ 700</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 479414,
          "author_name": "sheriytm",
          "author_url": "",
          "post_date": "02/27/2019 05:49:20",
          "content": "<p>I agree Olivier deserves our gratitude and upvotes for that kernel and many more.</p>\n\n<p>&gt; ('reborn_new_min_submodel',                         3212145.126355231)\n('reborn_hist_new_min_submodel',                     2897647.0672301054)\n('reborn_hist_min_submodel',                         2571620.7657690495)</p>\n\n<p>So you meant the gain value of 200 ~ 700. So those numbers you have in front of the features is not the gain?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 479420,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 05:52:39",
          "content": "<p>no it's feature numbers, we use \n<code>scores_df = pd.DataFrame(feature_scores, columns=['feature', 'split_score', 'gain_score']) \n</code>\nthe gain_score is 1.0 or 1.5 or 0 due to your cv performance</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 479467,
          "author_name": "sheriytm",
          "author_url": "",
          "post_date": "02/27/2019 06:57:38",
          "content": "<p>Got it top 200 ~ 700 features. Thanks.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479298,
      "author_name": "rahulin",
      "author_url": "",
      "post_date": "02/27/2019 03:51:57",
      "content": "<p>Congrats man on your gold. I learned so much from your approach. Thanks for this detailed explanation :)</p>",
      "votes": null,
      "replies": [
        {
          "id": 479357,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 05:00:17",
          "content": "<p>Congratulations your first silver medal.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 479388,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 05:29:00",
          "content": "<p>I saw your mail,I have added  source code to post</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479302,
      "author_name": "tomutm124",
      "author_url": "",
      "post_date": "02/27/2019 03:57:08",
      "content": "<p>Edit: Didn't refresh the page and therefore missed your explanation above, sorry about that. Please ignore.</p>\n\n<p>Thank you very much for your sharing! May I know how you get your 12 different feature sets after doing feature selection? I also thought about dividing my features into sets (to train models for stacking) after reading your team's sharing for the Home Credit Default Risk competition, but didn't have enough time to think through how exactly to do it.</p>\n\n<p>I understand that after performing target permutation following Olivier's kernel, we can get a more realistic importance for each feature. May I know how you proceeded from there? (Initially I planned to try using top 10 features in each feature set and then randomly draw 190 more from the top 11-500 features to add to each set, so each set has 10+190=200 features).</p>",
      "votes": null,
      "replies": [
        {
          "id": 479365,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 05:07:13",
          "content": "<p>when new features can not improve the best model,I start to build another model with those features and add new from different idea that give me 12 different feature sets.\nI didn't spend too much time to choose features, just try some thresholds to choose top N important features.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 479462,
          "author_name": "tomutm124",
          "author_url": "",
          "post_date": "02/27/2019 06:54:59",
          "content": "<p>I see. Thanks for the reply. And thank you again for your sharing!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 479781,
          "author_name": "codlife",
          "author_url": "",
          "post_date": "02/27/2019 12:24:58",
          "content": "<p>@senkins Your method about start to build another model inspired me ! as a new kaggler, i'm confused  that time whatever features i add the public lb get worse base my 3.672 public lb single model.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479303,
      "author_name": "cczaixian",
      "author_url": "",
      "post_date": "02/27/2019 03:58:16",
      "content": "<p>Congratulations and thanks for sharing! Another GM is coming!</p>",
      "votes": null,
      "replies": [
        {
          "id": 479398,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 05:34:11",
          "content": "<p>thanks, the door is half opened</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479304,
      "author_name": "process531",
      "author_url": "",
      "post_date": "02/27/2019 03:58:28",
      "content": "<p>Congratulations and thanks for you sharing!</p>",
      "votes": null,
      "replies": [
        {
          "id": 479401,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 05:35:18",
          "content": "<p>Also congratulations your first silver medal.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479339,
      "author_name": "htts001",
      "author_url": "",
      "post_date": "02/27/2019 04:37:43",
      "content": "<p>Great post. Congratulations <a href=\"/senkin13\">@senkin13</a>. I've so much to learn.</p>\n\n<p>I want to understand the below piece from you approach. Can you explain a bit more on this?</p>\n\n<p><strong>dimession reducing from original cardid's merchant|merchantcategory transaction sequence to 5</strong></p>\n\n<p>Thanks.</p>",
      "votes": null,
      "replies": [
        {
          "id": 479386,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 05:28:41",
          "content": "<p>I add  source code to post</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479361,
      "author_name": "viswanathravindran",
      "author_url": "",
      "post_date": "02/27/2019 05:02:33",
      "content": "<p>Thanks for writing this up, this is quite exhaustive.. </p>",
      "votes": null,
      "replies": [
        {
          "id": 479405,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 05:37:47",
          "content": "<p>thanks</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479427,
      "author_name": "chenxin1991",
      "author_url": "",
      "post_date": "02/27/2019 06:03:14",
      "content": "<p>Congrats. Next GM is coming. Thanks for your sharing.</p>",
      "votes": null,
      "replies": [
        {
          "id": 479453,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 06:38:49",
          "content": "<p>your team did well</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479430,
      "author_name": "liewmanchoi",
      "author_url": "",
      "post_date": "02/27/2019 06:08:22",
      "content": "<p>Gongratulations, and special thanks to your sharing. And could you please ellaborate on what is <code>transactions meta feature</code>, and what is <code>cardids meta feature</code>? Is it used in stacking?</p>\n\n<blockquote>\n  <p>use above 6,7,8 dataset, merge train's target to each transaction\n  row,then create a lightgbm to output each transaction's meta feature ,\n  then groupby cardid's min/sum to create cardid's meta feature , it can\n  boost 0.005 ~ 0.006 both on cv and lb.</p>\n</blockquote>",
      "votes": null,
      "replies": [
        {
          "id": 479452,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 06:37:32",
          "content": "<p>similar with this post from homecredit solution\n<a href=\"https://www.kaggle.com/c/home-credit-default-risk/discussion/64503\">https://www.kaggle.com/c/home-credit-default-risk/discussion/64503</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479438,
      "author_name": "moonyknight",
      "author_url": "",
      "post_date": "02/27/2019 06:18:34",
      "content": "<p>Congratulations! And thanks for you sharing.</p>",
      "votes": null,
      "replies": [
        {
          "id": 479455,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 06:40:15",
          "content": "<p>Congratulations your first silver medal.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479557,
      "author_name": "cywwayne",
      "author_url": "",
      "post_date": "02/27/2019 08:42:18",
      "content": "<p>Congrats to your gold metal !  Thank you for sharing! \nThe final trick (using minmaxscaler to transform from 0 ~ 1 ...) is interesting.\nI learned so much from this competition and your sharing!</p>",
      "votes": null,
      "replies": [
        {
          "id": 479584,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 09:01:16",
          "content": "<p>I like it too,unfortunately I stop to deepdive </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479566,
      "author_name": "chandureddyvari",
      "author_url": "",
      "post_date": "02/27/2019 08:49:22",
      "content": "<p>Thanks for sharing and Congratulations.  Learned a lot from your solution. Will try to replicate your solution.</p>",
      "votes": null,
      "replies": [
        {
          "id": 479605,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 09:20:32",
          "content": "<p>yeah, reproduce solution is good way to learn.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479589,
      "author_name": "gpamoukoff",
      "author_url": "",
      "post_date": "02/27/2019 09:04:45",
      "content": "<p>Congrats and thanks for sharing!</p>",
      "votes": null,
      "replies": [
        {
          "id": 479603,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 09:19:38",
          "content": "<p>Thanks</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479595,
      "author_name": "codlife",
      "author_url": "",
      "post_date": "02/27/2019 09:14:55",
      "content": "<p>Hi senkin13!\n  Can you share how many boost with word2vec and svd features. i tried svd for card_id -merchant-category matrix, but failed to boost the public lb. thank you!</p>",
      "votes": null,
      "replies": [
        {
          "id": 479602,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 09:18:53",
          "content": "<p>Basicly I don't select features due to LB, I think I got 0.002 cv boost from svd features when my cv is 3.632,I combinate only w2v features to train.csv ,can boost from 3.84 to 3.67</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 479610,
          "author_name": "codlife",
          "author_url": "",
          "post_date": "02/27/2019 09:26:28",
          "content": "<p>3.67 mean local cv?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 479624,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 09:43:01",
          "content": "<p>yes,local cv.I merged all the features seems good to create a best single model.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479688,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "02/27/2019 10:43:36",
      "content": "<p>Congrats for yet another solo gold!</p>",
      "votes": null,
      "replies": [
        {
          "id": 479883,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 14:31:03",
          "content": "<p>thanks,your 4 days journey is impressive</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479697,
      "author_name": "chizhu2018",
      "author_url": "",
      "post_date": "02/27/2019 10:52:14",
      "content": "<p>Congrats and thanks for detailed sharing!  </p>",
      "votes": null,
      "replies": [
        {
          "id": 479887,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 14:32:53",
          "content": "<p>I read your github and learned something useful :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479707,
      "author_name": "tpthegreat",
      "author_url": "",
      "post_date": "02/27/2019 11:02:02",
      "content": "<p>Very unique Feature Engineering!!\nGreat ideas!\nThanks for sharing</p>",
      "votes": null,
      "replies": [
        {
          "id": 479890,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 14:35:53",
          "content": "<p>normal features worked unwell that have to try new ideas</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479819,
      "author_name": "seuguh",
      "author_url": "",
      "post_date": "02/27/2019 12:59:35",
      "content": "<p>Congratulations and thanks for sharing your solution.\nFor several weeks I was so frustrated not being able to close the gap with top competitors. It was clear I was missing some key ideas. Reading your post helped me understand what I was missing.</p>",
      "votes": null,
      "replies": [
        {
          "id": 479892,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 14:39:22",
          "content": "<p>yes,kaggle is the best place to learn pratical skills</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479847,
      "author_name": "sreeedevi",
      "author_url": "",
      "post_date": "02/27/2019 13:47:08",
      "content": "<p>Congrats!!</p>\n\n<p>Thank you for explaining the solution. This clearly shows all the hard work and thinking you have put into the competition.</p>",
      "votes": null,
      "replies": [
        {
          "id": 479897,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 14:44:40",
          "content": "<p>Thanks, trial and error are necessary</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 479856,
      "author_name": "xonti1",
      "author_url": "",
      "post_date": "02/27/2019 13:59:10",
      "content": "<p>Congratulations! Very impressive decision! Do I understand you right, your lgb single model is better than your stacking? </p>",
      "votes": null,
      "replies": [
        {
          "id": 479899,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/27/2019 14:46:38",
          "content": "<p>yes, single lgb is better than stacking on private score,so I think stacking  overfit too much.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 480212,
      "author_name": "yryrgogo",
      "author_url": "",
      "post_date": "02/27/2019 23:33:51",
      "content": "<p>Congrats! senkin13! <br>\nThank you for writing in detail the solution method filled with your ideas. <br>\nI have a question about your NN. <br>\nThe loss function is MSE instead of RMSE. Did MSE go better?\n(btw I also had single LGB in gold zone. robust model is difficult.)</p>",
      "votes": null,
      "replies": [
        {
          "id": 480260,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/28/2019 01:42:03",
          "content": "<p>No, I think best mse equal to best rmse ,isn't it?\nI never have shakeup experience,so I think should make the single model strong enough to prevent huge shakedown.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 480268,
          "author_name": "yryrgogo",
          "author_url": "",
          "post_date": "02/28/2019 02:05:32",
          "content": "<p>Yes, I think that is the same. I was on the same RMSE as the evaluation, so I asked a question as to whether there is meaning in the different thing. Thank you for reply.\nI may choose a single model as a candidate next.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 480227,
      "author_name": "kingychiu",
      "author_url": "",
      "post_date": "02/28/2019 00:15:59",
      "content": "<p>So creative of applying NLP here. Congrats.</p>",
      "votes": null,
      "replies": [
        {
          "id": 480264,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/28/2019 01:50:07",
          "content": "<p>Thanks, tabular data competitions are so attractive  that you never know which technology can work which not</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 480537,
      "author_name": "xiaojiu1414",
      "author_url": "",
      "post_date": "02/28/2019 10:08:35",
      "content": "<p>Thanks for sharing! It seems to be one of the most complete feature engineering. And it is a valuable post I will refer and try in the  future.</p>",
      "votes": null,
      "replies": [
        {
          "id": 480545,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "02/28/2019 10:15:28",
          "content": "<p>Thanks , I am waiting for your team's solution.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 480631,
          "author_name": "xiaojiu1414",
          "author_url": "",
          "post_date": "02/28/2019 13:03:54",
          "content": "<p>Haha, I will ask our teammates. One member from Japan contributed the best sub. Some of the feature names are in Japanese, which I am still understanding and reproducing.😓</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 480812,
      "author_name": "billxubf",
      "author_url": "",
      "post_date": "02/28/2019 17:48:07",
      "content": "<p>Congrads a looot! I’m curious about isotonic regression you mentioned in post processing. I though it was not useful unless distribution of target is mixed and unclear. Why do you think isotonic regression of true labels from your ensembled results worked so well here?</p>",
      "votes": null,
      "replies": [
        {
          "id": 481347,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "03/01/2019 09:39:17",
          "content": "<p>you are right,after testing late submission,I found isotonic regression overfit too much,that's why I shakedown.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 480880,
      "author_name": "dmitrkozlovsk",
      "author_url": "",
      "post_date": "02/28/2019 20:14:01",
      "content": "<p>Great work! Congradulations! Thanks for sharing!</p>\n\n<p>I've used tfidf merchants and Ridge to precict target and when add it like feature, it get 0.01+ . Also I tried to predict new_merchants purchase with help SVD and calculate \"difference\" between predictions and new_merchant_transactions marchants. Dont know what exactly work, but i get my first bronze)))</p>",
      "votes": null,
      "replies": [
        {
          "id": 481360,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "03/01/2019 09:59:36",
          "content": "<p>Congrats! nice idea of using Ridge and tfidf,I will try to use that at next comp.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 480982,
      "author_name": "kei1989",
      "author_url": "",
      "post_date": "03/01/2019 00:19:41",
      "content": "<p>Congratulations</p>",
      "votes": null,
      "replies": [
        {
          "id": 481362,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "03/01/2019 09:59:54",
          "content": "<p>Same to you.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 481947,
      "author_name": "chocozzz",
      "author_url": "",
      "post_date": "03/02/2019 04:54:23",
      "content": "<p>Thanks sharing and Congratulations Gold medal!!! </p>\n\n<p>I have some question. I don't understand <code>Transaction Base Model</code>. it is right? Make a 6,7,8 (don't groupby card_id) -&gt; Merge target to each 6, 7, 8 -&gt;  lgb transaction's meta feature(?)</p>\n\n<p>sorry. I have no idea about this part.</p>",
      "votes": null,
      "replies": [
        {
          "id": 482250,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "03/02/2019 15:28:31",
          "content": "<p>don't use groupby features,like this\n<code>historical_train = historical.merge(train,on='card_id',how='right')\nhistorical_test = historical.merge(test,on='card_id',how='right')</code></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 482317,
      "author_name": "aditya1702",
      "author_url": "",
      "post_date": "03/02/2019 18:04:35",
      "content": "<p><a href=\"/senkin13\">@senkin13</a> Could you explain your word2vec function more? I didnt understand this - \"then group by card_id's min/max/mean/std\". How do you perform this grouping</p>",
      "votes": null,
      "replies": [
        {
          "id": 483115,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "03/04/2019 06:50:01",
          "content": "<p>after trained a word2vec model,one merchant_id should has a vector like [w2v_merchant_1,w2v_merchant_2,w2v_merchant_3,w2v_merchant_4,w2v_merchant_5],then you can use historical or new data to create aggregation features like groupby(['card_id'])['w2v_merchant_1'].mean()</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 482436,
      "author_name": "cmswdd",
      "author_url": "",
      "post_date": "03/03/2019 01:07:56",
      "content": "<p>Thanks for sharing!</p>",
      "votes": null,
      "replies": [
        {
          "id": 483116,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "03/04/2019 06:50:21",
          "content": "<p>thanks</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 482529,
      "author_name": "moromoromoro",
      "author_url": "",
      "post_date": "03/03/2019 06:42:25",
      "content": "<p>Thanks for sharing.\nMay I ask you a question?  What's mean of \"Interaction:  new.purchasedate.max() / historical.purchasedate.max() \" ? </p>",
      "votes": null,
      "replies": [
        {
          "id": 483113,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "03/04/2019 06:45:13",
          "content": "<p>```\nfor f in [\n          'new_card_purchase_date_max',\n          'hist_card_purchase_date_max',]:\n    df[f +'_int'] = df[f].astype(np.int64) * 1e-9</p>\n\n<p>df['card_purchase_date_max_ratio'] = df['new_card_purchase_date_max_int'] / df['hist_card_purchase_date_max_int']\n```</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 483943,
          "author_name": "moromoromoro",
          "author_url": "",
          "post_date": "03/05/2019 11:32:44",
          "content": "<p>Thank you for your reply. I am interested in this feature. I will try it for testing. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 736424,
      "author_name": "meistermorxrc",
      "author_url": "",
      "post_date": "02/04/2020 06:58:32",
      "content": "<p>thanks!  learned a lot from this </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 739463,
      "author_name": "vbmokin",
      "author_url": "",
      "post_date": "02/07/2020 21:04:30",
      "content": "<p><a href=\"/senkin13\">@senkin13</a>, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (<a href=\"https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques\">https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques</a>)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "479196": "Congrats to all winner teams and new masters,experts.It's a very hard competition,I think every team work hard to try everything and fight with overfitting every day,no one fails who does his best.\nI started at very early stage so maybe I just did more work than others,let me share my solution briefly.\n\n#**Preprocessing**\nsort the historical and new data by timestamp then create 8 kinds of dataset for different features.\n\n 1.  Only historical data\n 2. Only historical data with authorized_flag=1\n 3. Only new data \n 4. Merge of historical data with authorized_flag=1 and new data \n 5. Merge of historical data  and new data \n 6. Merge of historical data and merchants data\n 7. Merge of new data and merchants data\n 8. Merge of historical data  and new data and merchants data\n\n#**Feature Engineering**\n## Aggregation\nI think there are not big difference with everyone's aggregation features, I want to thank raddar's magic that purchase_amount_new can bring additional features beside of original purchase_amount.\n\n## Interval\ndays interval(the difference from current purchase_date to next purchase_date)\npurchase_amount interval(the difference from current purchase_amount to next purchase_amount)\n\n## Interaction\nnew.purchase_date.max() / historical.purchase_date.max() is very strong\n\n## SVD\ndimession reducing from original card_id's merchant|merchant_category transaction sequence to 5 \n```\ndef svd_feature(prefix, df, traintest, groupby, target,n_comp):\n    tfidf_vec = TfidfVectorizer(ngram_range=(1,1), max_features=None)\n    df_bag = pd.DataFrame(df[[groupby, target]])\n    df_bag = df_bag.groupby(groupby, as_index=False)[target].agg({'list':(lambda x: list(x))}).reset_index()\n    df_bag[target + '_list']=df_bag['list'].apply(lambda x: str(x).replace('[','').replace(']','').replace(',',' '))\n    \n    df_bag = df_bag.merge(traintest,on=groupby,how='left')\n    df_bag_train = df_bag[df_bag['target'].notnull()].reset_index(drop=True)\n    df_bag_test = df_bag[df_bag['target'].isnull()].reset_index(drop=True)\n    \n    tfidf_full_vector = tfidf_vec.fit_transform(df_bag[target + '_list'])\n    tfidf_train_vector = tfidf_vec.transform(df_bag_train[target + '_list'])\n    tfidf_test_vector = tfidf_vec.transform(df_bag_test[target + '_list'])\n    \n    svd_vec = TruncatedSVD(n_components=5, algorithm='arpack')\n    svd_vec.fit(tfidf_full_vector)\n    svd_train = pd.DataFrame(svd_vec.transform(tfidf_train_vector))\n    svd_test = pd.DataFrame(svd_vec.transform(tfidf_test_vector))\n\n    svd_train.columns = ['svd_%s_%s_%d'%(prefix,target,x) for x in range(n_comp)]\n    svd_train[groupby] = df_bag_train[groupby]\n    svd_test.columns = ['svd_%s_%s_%d'%(prefix,target,x) for x in range(n_comp)]\n    svd_test[groupby] = df_bag_test[groupby]\n    #df_svd = pd.concat([svd_train,svd_test],axis=0)\n    print ('svd_train:' + str(svd_train.shape))\n    print ('svd_test:' + str(svd_test.shape))\n    return svd_train,svd_test\n```\n\n## Word2vec\ncreate each merchant_id's , merchant_category_id's , purchase_date's ... word embedding,then group by card_id's min/max/mean/std\n```\ndef word2vec_feature(prefix, df, groupby, target,size):\n    df_bag = pd.DataFrame(df[[groupby, target]])\n    df_bag[target] = df_bag[target].astype(str)\n    df_bag[target].fillna('NAN', inplace=True)\n    df_bag = df_bag.groupby(groupby, as_index=False)[target].agg({'list':(lambda x: list(x))}).reset_index()\n    doc_list = list(df_bag['list'].values)\n    w2v = Word2Vec(doc_list, size=size, window=3, min_count=1, workers=32)\n    vocab_keys = list(w2v.wv.vocab.keys())\n    w2v_array = []\n    for v in vocab_keys :\n        w2v_array.append(list(w2v.wv[v]))\n    df_w2v = pd.DataFrame()\n    df_w2v['vocab_keys'] = vocab_keys    \n    df_w2v = pd.concat([df_w2v, pd.DataFrame(w2v_array)], axis=1)\n    df_w2v.columns = [target] + ['w2v_%s_%s_%d'%(prefix,target,x) for x in range(size)]\n    print ('df_w2v:' + str(df_w2v.shape))\n    return df_w2v\n```\n\n## Transaction-based Model\nuse above 6,7,8 dataset, merge train's target to each transaction row,then create a lightgbm to output each transaction's meta feature , then groupby card_id's min/sum to create card_id's meta feature , it can boost 0.005 ~ 0.006  both on cv and lb.\n\n## Feature Selection\nat early stage I add new features one by one,sometimes group by group,at late stage I use target permutation learned from https://www.kaggle.com/ogrellier/feature-selection-with-null-importances\nI created thousands of features,finally get 12 different feature sets with number from 200 ~ 700\n\n## Top 20 Feature Importance \n```\n('reborn_new_min_submodel', \t\t\t\t\t\t3212145.126355231)\n('reborn_hist_new_min_submodel', \t\t\t\t\t2897647.0672301054)\n('reborn_hist_min_submodel', \t\t\t\t\t\t2571620.7657690495)\n('reborn_hist_sum_submodel', \t\t\t\t\t\t1556242.6892476082)\n('reborn_hist_new_sum_submodel', \t\t\t\t\t1495420.6608867645)\n('card_purchase_date_max_ratio', \t\t\t\t\t1176560.0546371937)\n('reborn_new_sum_submodel', \t\t\t\t\t\t988300.9342432022)\n('hist_new_purchase_date_max_ratio', \t\t\t\t751098.8485645056)\n('hist_new_sum_category_1_1_purchase_amount', \t\t634349.3938679695)\n('hist_sum_category_1_1_purchase_amount', \t\t\t626576.9190101624)\n('new_card_sum_purchase_amount_new', \t\t\t\t573281.6908314228)\n('hist_card0_max_month_diff', \t\t\t\t\t\t539578.2926783562)\n('new_card_var_day_diff', \t\t\t\t\t\t\t537170.215294838)\n('new_card_last_day', \t\t\t\t\t\t\t\t532604.059334755)\n('hist_card_sum_month_lag_0_purchase_amount_new', \t530102.9405331612)\n('hist_card0_mean_month_diff', \t\t\t\t\t\t506467.2120089531)\n('svd_hist_new_merchant_id4', \t\t\t\t\t\t454442.183167696)\n('hist_card0_sum_month_diff', \t\t\t\t\t\t445077.63865828514)\n('hist_card_std_merchant_category_id_w2v_4', \t\t436928.8732688427)\n('hist_new_card_std_merchant_category_id_w2v_1', \t434849.51894950867)\n```\n\n# **Single Model**\n## Lightgbm\nMy best single model is a lightgbm has 385 features with CV:3.6144,LB:3.668,PB:3.593,maybe it can reach to 3th place.It also happened at homecredit competition,best single model is much better than stacking,I will doubt if stacking is a good choice.\n## Nueral Netowork\nMy best nn has CV:3.6407,LB:3.680,PB:3.605,it's just a simple 3layers-dense network.\n\n```\ndef ann(input_shape):\n    model = Sequential()\n    model.add(Dense(2 ** 10, input_dim = input_shape, init='random_uniform', activation='relu'))\n    model.add(Dropout(0.25))    \n    model.add(BatchNormalization())\n    model.add(Dense(2 ** 9, init='random_uniform', activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.25)) \n    model.add(Dense(2 ** 5, init='random_uniform', activation='relu'))\n    model.add(BatchNormalization())\n    model.add(Dropout(0.25))      \n    model.add(Dense(1))\n    model.compile(loss='mean_squared_error', optimizer='adam') \n    return model\n```\n\n## Others\nXgboost and Catboost got not bad results but almost no improvement to stacking.\nAlso Tried Ridge,KNN,FFM...no success to me.\n\n\n# **Ensemble**\nI built 12 lightgbm models and 40 nn models for Stacking stage1,then use lightgbm and nn as stage2,finally blend them with average.CV can reach to 3.603,but lb got worse,and final pb also got worse.\n\n# **Postprocessing**\nIsotonic regression give me 0.005 ~ 0.006 boost both on cv and lb but pb score become worse,that's the reason I shakedown.\nCombination with a non-outliers model using binary model's best threshold give me 0.001 ~ 0.002 boost both on cv and lb and pb.\n\n# **Leaderboard Probing**\nI used 100 subs to probe the lb,that let me grasp 24 public-test outliers and 6 non-outliers.\nI tried Pseudo-Labelling but no boost,finally it just can be used to decide a \"perfect threshold\" to override the outliers but failed.\n\n#**Other tricks**\nat very early stage I tried target transformation,using minmaxscaler to transform from 0 ~ 1, then use xentropy boosting type,finally reversed to original value range,it didn't improve cv and lb, but I found pb is very good, a very eary stage lightgbm model is enough to get a gold medal.\n\n#**Late Submission Founding**\nIsotonic regression overfit both on cv and lb, should be careful.\nStacking can work if stage-2 model is nn or ridge ,lgb and extratree overfit too much.",
    "479206": "Congrats on your solo gold! GM on the way!",
    "479207": "Congratulations and thanks for you sharing! What's the features of your best single model?",
    "479210": "Amazing work man, thank you",
    "479216": "Congratulations and thanks for you sharing!",
    "479246": "Amazing work！thank you for sharing!",
    "479269": "about 300 features",
    "479270": "Come with me",
    "479272": "Congratulations your first gold medal.",
    "479273": "Congratulations your first silver medal.",
    "479275": "your team is very strong",
    "479276": "Congratulations @senkin13 on your solo gold and thanks for sharing your solution overview. I also used that Olivier's kernel plus manual methods for my feature selection. Here is my post on feature selection techniques\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/82052\n\n&gt;finally get 12 different feature sets with number from 200 ~ 700\n\nIs 200 ~ 700 the split numbers from your null hypothesis result?",
    "479279": "Thanks, I think Olivier is deserved every kagglers' upvote",
    "479286": "I manully start different features combination then use target permutation,choose the best importance threshold,the threshold is from 200 ~ 700",
    "479298": "Congrats man on your gold. I learned so much from your approach. Thanks for this detailed explanation :)",
    "479302": "Edit: Didn't refresh the page and therefore missed your explanation above, sorry about that. Please ignore.\n\nThank you very much for your sharing! May I know how you get your 12 different feature sets after doing feature selection? I also thought about dividing my features into sets (to train models for stacking) after reading your team's sharing for the Home Credit Default Risk competition, but didn't have enough time to think through how exactly to do it.\n\nI understand that after performing target permutation following Olivier's kernel, we can get a more realistic importance for each feature. May I know how you proceeded from there? (Initially I planned to try using top 10 features in each feature set and then randomly draw 190 more from the top 11-500 features to add to each set, so each set has 10+190=200 features).",
    "479303": "Congratulations and thanks for sharing! Another GM is coming!",
    "479304": "Congratulations and thanks for you sharing!",
    "479339": "Great post. Congratulations @senkin13. I've so much to learn.\n\nI want to understand the below piece from you approach. Can you explain a bit more on this?\n\n**dimession reducing from original cardid's merchant|merchantcategory transaction sequence to 5**\n\nThanks.",
    "479357": "Congratulations your first silver medal.",
    "479361": "Thanks for writing this up, this is quite exhaustive..",
    "479365": "when new features can not improve the best model,I start to build another model with those features and add new from different idea that give me 12 different feature sets.\nI didn't spend too much time to choose features, just try some thresholds to choose top N important features.",
    "479386": "I add  source code to post",
    "479388": "I saw your mail,I have added  source code to post",
    "479398": "thanks, the door is half opened",
    "479401": "Also congratulations your first silver medal.",
    "479405": "thanks",
    "479414": "I agree Olivier deserves our gratitude and upvotes for that kernel and many more.\n\n&gt; ('reborn_new_min_submodel',                         3212145.126355231)\n('reborn_hist_new_min_submodel',                     2897647.0672301054)\n('reborn_hist_min_submodel',                         2571620.7657690495)\n\nSo you meant the gain value of 200 ~ 700. So those numbers you have in front of the features is not the gain?",
    "479420": "no it's feature numbers, we use \n```scores_df = pd.DataFrame(feature_scores, columns=['feature', 'split_score', 'gain_score']) \n```\nthe gain_score is 1.0 or 1.5 or 0 due to your cv performance",
    "479427": "Congrats. Next GM is coming. Thanks for your sharing.",
    "479430": "Gongratulations, and special thanks to your sharing. And could you please ellaborate on what is `transactions meta feature`, and what is `cardids meta feature`? Is it used in stacking?\n&gt; use above 6,7,8 dataset, merge train's target to each transaction\n&gt; row,then create a lightgbm to output each transaction's meta feature ,\n&gt; then groupby cardid's min/sum to create cardid's meta feature , it can\n&gt; boost 0.005 ~ 0.006 both on cv and lb.",
    "479438": "Congratulations! And thanks for you sharing.",
    "479452": "similar with this post from homecredit solution\nhttps://www.kaggle.com/c/home-credit-default-risk/discussion/64503",
    "479453": "your team did well",
    "479455": "Congratulations your first silver medal.",
    "479462": "I see. Thanks for the reply. And thank you again for your sharing!",
    "479467": "Got it top 200 ~ 700 features. Thanks.",
    "479557": "Congrats to your gold metal !  Thank you for sharing! \nThe final trick (using minmaxscaler to transform from 0 ~ 1 ...) is interesting.\nI learned so much from this competition and your sharing!",
    "479566": "Thanks for sharing and Congratulations.  Learned a lot from your solution. Will try to replicate your solution.",
    "479584": "I like it too,unfortunately I stop to deepdive",
    "479589": "Congrats and thanks for sharing!",
    "479595": "Hi senkin13!\n  Can you share how many boost with word2vec and svd features. i tried svd for card_id -merchant-category matrix, but failed to boost the public lb. thank you!",
    "479602": "Basicly I don't select features due to LB, I think I got 0.002 cv boost from svd features when my cv is 3.632,I combinate only w2v features to train.csv ,can boost from 3.84 to 3.67",
    "479603": "Thanks",
    "479605": "yeah, reproduce solution is good way to learn.",
    "479610": "3.67 mean local cv?",
    "479624": "yes,local cv.I merged all the features seems good to create a best single model.",
    "479688": "Congrats for yet another solo gold!",
    "479697": "Congrats and thanks for detailed sharing!",
    "479707": "Very unique Feature Engineering!!\nGreat ideas!\nThanks for sharing",
    "479781": "senkins Your method about start to build another model inspired me ! as a new kaggler, i'm confused  that time whatever features i add the public lb get worse base my 3.672 public lb single model.",
    "479819": "Congratulations and thanks for sharing your solution.\nFor several weeks I was so frustrated not being able to close the gap with top competitors. It was clear I was missing some key ideas. Reading your post helped me understand what I was missing.",
    "479847": "Congrats!!\n\nThank you for explaining the solution. This clearly shows all the hard work and thinking you have put into the competition.",
    "479856": "Congratulations! Very impressive decision! Do I understand you right, your lgb single model is better than your stacking?",
    "479883": "thanks,your 4 days journey is impressive",
    "479887": "I read your github and learned something useful :)",
    "479890": "normal features worked unwell that have to try new ideas",
    "479892": "yes,kaggle is the best place to learn pratical skills",
    "479897": "Thanks, trial and error are necessary",
    "479899": "yes, single lgb is better than stacking on private score,so I think stacking  overfit too much.",
    "480212": "Congrats! senkin13!  \nThank you for writing in detail the solution method filled with your ideas.  \nI have a question about your NN.  \nThe loss function is MSE instead of RMSE. Did MSE go better?\n(btw I also had single LGB in gold zone. robust model is difficult.)",
    "480227": "So creative of applying NLP here. Congrats.",
    "480260": "No, I think best mse equal to best rmse ,isn't it?\nI never have shakeup experience,so I think should make the single model strong enough to prevent huge shakedown.",
    "480264": "Thanks, tabular data competitions are so attractive  that you never know which technology can work which not",
    "480268": "Yes, I think that is the same. I was on the same RMSE as the evaluation, so I asked a question as to whether there is meaning in the different thing. Thank you for reply.\nI may choose a single model as a candidate next.",
    "480537": "Thanks for sharing! It seems to be one of the most complete feature engineering. And it is a valuable post I will refer and try in the  future.",
    "480545": "Thanks , I am waiting for your team's solution.",
    "480631": "Haha, I will ask our teammates. One member from Japan contributed the best sub. Some of the feature names are in Japanese, which I am still understanding and reproducing.😓",
    "480812": "Congrads a looot! I’m curious about isotonic regression you mentioned in post processing. I though it was not useful unless distribution of target is mixed and unclear. Why do you think isotonic regression of true labels from your ensembled results worked so well here?",
    "480880": "Great work! Congradulations! Thanks for sharing!\n\nI've used tfidf merchants and Ridge to precict target and when add it like feature, it get 0.01+ . Also I tried to predict new_merchants purchase with help SVD and calculate \"difference\" between predictions and new_merchant_transactions marchants. Dont know what exactly work, but i get my first bronze)))",
    "480982": "Congratulations",
    "481347": "you are right,after testing late submission,I found isotonic regression overfit too much,that's why I shakedown.",
    "481360": "Congrats! nice idea of using Ridge and tfidf,I will try to use that at next comp.",
    "481362": "Same to you.",
    "481947": "Thanks sharing and Congratulations Gold medal!!! \n\nI have some question. I don't understand `Transaction Base Model`. it is right? Make a 6,7,8 (don't groupby card_id) -&gt; Merge target to each 6, 7, 8 -&gt;  lgb transaction's meta feature(?)\n\nsorry. I have no idea about this part.",
    "482250": "don't use groupby features,like this\n```historical_train = historical.merge(train,on='card_id',how='right')\nhistorical_test = historical.merge(test,on='card_id',how='right')```",
    "482317": "senkin13 Could you explain your word2vec function more? I didnt understand this - \"then group by card_id's min/max/mean/std\". How do you perform this grouping",
    "482436": "Thanks for sharing!",
    "482529": "Thanks for sharing.\nMay I ask you a question?  What's mean of \"Interaction:  new.purchasedate.max() / historical.purchasedate.max() \" ?",
    "483113": "```\nfor f in [\n          'new_card_purchase_date_max',\n          'hist_card_purchase_date_max',]:\n    df[f +'_int'] = df[f].astype(np.int64) * 1e-9\n\ndf['card_purchase_date_max_ratio'] = df['new_card_purchase_date_max_int'] / df['hist_card_purchase_date_max_int']\n```",
    "483115": "after trained a word2vec model,one merchant_id should has a vector like [w2v_merchant_1,w2v_merchant_2,w2v_merchant_3,w2v_merchant_4,w2v_merchant_5],then you can use historical or new data to create aggregation features like groupby(['card_id'])['w2v_merchant_1'].mean()",
    "483116": "thanks",
    "483943": "Thank you for your reply. I am interested in this feature. I will try it for testing.",
    "736424": "thanks!  learned a lot from this",
    "739463": "senkin13, congratulations! Thanks for sharing your approaches and magic! I added your post to my collection of the best kernels and posts of winners of Kaggle Prize Competitions: [Data Science for tabular data: Advanced Techniques] (https://www.kaggle.com/vbmokin/data-science-for-tabular-data-advanced-techniques)",
    "1526662": "Thanks for the excellent insights.\nMay I know what are being referred in the features with “reborn... submodel” :\n\n('reborn_new_min_submodel',                         3212145.126355231)\n('reborn_hist_new_min_submodel',                     2897647.0672301054)\n('reborn_hist_min_submodel',                         2571620.7657690495)\n('reborn_hist_sum_submodel',                         1556242.6892476082)\n('reborn_hist_new_sum_submodel', ...)"
  },
  "source": "meta"
}