{
  "id": 20258,
  "title": "#3 place solution",
  "content": "<p>I think our solution may sound too plain compared to what the top 2 teams did :) .</p>\n\n<p>We spent quite some time reading forum threads and applied many of things discussed with some success like :</p>\n\n<p>1) rounding features to remove noise\n2) finding the correct denominator of features to get them back to the initial integer state\n3) converting categorical variables to likelihoods</p>\n\n<p>They all helped  a little bit but the most important thing for us (and thanks to clobber for finding it! ) was interacting the categorical variables together , especially  v22 with all the rest categorical variables in 2way and even 3way manner. </p>\n\n<p>Another thing that helped was rounding strongly ( 2 decimals) all the numerical features and creating interactions (as categorical &quot;numericalrounded_catgeorical&quot; with some of the categorical ones.) </p>\n\n<p>Stans was kinda of our features'factory machine! - apart from likelihood and other target-based feature transformations on categorical data, he did a lot of extra work in creating group-by type of aggregate features (like averages) of v50 and other variables (up to 3 way). Although we never understood exactly why this works, stans was able to capture some very useful patterns here .</p>\n\n<p>Our best single xgboost would have finished 4th (0.4242 in private) and was consisted of more than 7.5 K features (mostly interactions).</p>\n\n<p>We also produced many different base level models without much Feature engineering, just different input format types (like load all categorical variables as counts, or as onehot encoding etc).</p>\n\n<p>Our ensemble was consisted of 223 models. Faron did a lot of work in removing noise and discarding many of these in order to get to our bets score with a lvl2 ensemble of geomean weights between an ET ,  2NN and 2 Xgmodels. </p>\n\n<p>We burnt many cores in case you might have questions about it!</p>",
  "comments": [
    {
      "id": 115888,
      "author_name": "Μαριος Μιχαηλιδης KazAnova",
      "author_url": "",
      "post_date": "2016-04-20T16:08:35.673000",
      "content": "<p>I feel keen to mention that our team's name is inspired by:</p>\n\n<p><a href=\"http://www.explodingkittens.com/\">http://www.explodingkittens.com/</a></p>\n\n<p>(A card game for those who like both kittens and explosions!)</p>\n\n<p><img src=\"http://www.explodingkittens.com/img/defuse.png\" alt=\"Exploding Kittens\" title></p>",
      "votes": 10,
      "replies": []
    },
    {
      "id": 115689,
      "author_name": "Μαριος Μιχαηλιδης KazAnova",
      "author_url": "",
      "post_date": "2016-04-19T16:16:47.163000",
      "content": "<p>[quote=Shubin;115688]</p>\n\n<p>congrats! Could you elaborate more on nn structure? besides, you only apply nn on the second level instead of the first?  if not, what would be the strcuture look like for the first level nn?</p>\n\n<p>[/quote]</p>\n\n<p>Thank you :) . I dont feel I am a great nn-tunner-structure-master and there are many different structures that could work I guess. </p>\n\n<p>I have tried nn in both 1st and end levels, although in 2nd levels they were much more useful. </p>\n\n<p>for second level I used  </p>\n\n<pre><code>models = Sequential()\nmodels.add(Dense(100, input_dim=input_dim, init='uniform', W_regularizer=l2(0.00001)))\nmodels.add(PReLU())\nmodels.add(BatchNormalization())\nmodels.add(Dropout(0.7))\nmodels.add(Dense(output_dim, init='uniform'))\nmodels.add(Activation('softmax'))\nopt = optimizers.Adagrad(lr=lrs)\nmodels.compile(loss='binary_crossentropy', optimizer=opt)\n</code></pre>\n\n<p>epocs = 50</p>\n\n<p>and for 1st level</p>\n\n<pre><code>models = Sequential()\nmodels.add(Dense(150, input_dim=input_dim, init='uniform', W_regularizer=l2(0.00001), activation='relu'))\n#models.add(PReLU())\nmodels.add(BatchNormalization())\nmodels.add(Dropout(0.8))\nmodels.add(Dense(output_dim, init='uniform'))\nmodels.add(Activation('softmax'))\nopt = optimizers.Adagrad(lr=lrs)\nmodels.compile(loss='binary_crossentropy', optimizer=opt)\n</code></pre>\n\n<p>also epoc = 50.</p>\n\n<p>Both in Keras.</p>\n\n<p>Hope it helps. I think other people did much better job- as said I am not good with nns. </p>",
      "votes": 10,
      "replies": []
    },
    {
      "id": 115683,
      "author_name": "Μαριος Μιχαηλιδης KazAnova",
      "author_url": "",
      "post_date": "2016-04-19T15:54:26.260000",
      "content": "<p>[quote=MadScientist;115676]</p>\n\n<p>First of all congrats.  Well done!</p>\n\n<p>[/quote]</p>\n\n<p>Thank u!</p>\n\n<p>[quote=MadScientist;115676]</p>\n\n<p>Then of course your reward is to be peppered with questions! =)</p>\n\n<p>[/quote]</p>\n\n<p>Thank u x2!</p>\n\n<p>[quote=MadScientist;115676]</p>\n\n<p>I'm not sure I follow what you are saying--could you expand upon this?</p>\n\n<p>[/quote]</p>\n\n<p>for example max(v50) , mean(v50), min(v50)\ngroup by v22\nWe did that for many categorical variables.  and also group by 2 cats or even 3 cats.\nps. when I say &quot;we&quot; , I mean Stans - haha.</p>\n\n<p>[quote=MadScientist;115676]</p>\n\n<p>.Am I understsanding that for 1 &quot;model&quot; you leave all category variables as encoded integers, then for a second model you one hot encode. </p>\n\n<p>[/quote]</p>\n\n<p>Yes that's it . Also , trying different algorithm in the same data input is also considered a different model (at least in my slang!) </p>",
      "votes": 6,
      "replies": []
    },
    {
      "id": 115934,
      "author_name": "Faron",
      "author_url": "",
      "post_date": "2016-04-20T21:36:12.817000",
      "content": "<p>[quote=Andreu;115878]</p>\n\n<p>After detecting a 3-way interaction, what processes do  you follow to exploit that? I am new to this kind of analysis but as far as I can tell this interaction tells that the variables involved should not be removed from the data set. </p>\n\n<p>[/quote]</p>\n\n<p>Just treat it as a new feature and process it with the same methods used for other features.</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 115729,
      "author_name": "Faron",
      "author_url": "",
      "post_date": "2016-04-19T18:38:18.730000",
      "content": "<p>[quote=Shubin;115697]</p>\n\n<p>you have trained 233 models on different data sets with different feature engineering. therefore, you have 233 features(do you have any idea how strong these each individual models? </p>\n\n<p>[/quote]</p>\n\n<p>The base models performances range from 0.4272 @public LB to something around 0.48x</p>\n\n<p>[quote=Shubin;115697]</p>\n\n<p>have you consider incorporating some of your useful features along with these 233 features to produce the output?  </p>\n\n<p>[/quote]</p>\n\n<p>Yes, but did not help us here.</p>\n\n<p>[quote=Shubin;115697]</p>\n\n<p>in addition, do you have any suggestion regarding code management? i would imagine it is indeed a skill to manage this process and do proper cv.  i always mess up.\nthanks again! </p>\n\n<p>[/quote]</p>\n\n<ul>\n<li>build some sort of your own kaggle coding conventions</li>\n<li>proper local folder structure</li>\n<li>keep track of what you are doing</li>\n<li>usage of svn or git</li>\n<li>write highly modularized, reusable code</li>\n</ul>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 115688,
      "author_name": "Shubin",
      "author_url": "",
      "post_date": "2016-04-19T16:08:16.867000",
      "content": "<p>congrats! Could you elaborate more on nn structure? besides, you only apply nn on the second level instead of the first?  if not, what would be the strcuture look like for the first level nn?</p>\n\n<p>thanks again!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 115676,
      "author_name": "Ronin",
      "author_url": "",
      "post_date": "2016-04-19T15:34:07.463000",
      "content": "<p>First of all congrats.  Well done!</p>\n\n<p>Then of course your reward is to be peppered with questions! =)</p>\n\n<p>[quote=&#924;&#945;&#961;&#953;&#959;&#962; &#924;&#953;&#967;&#945;&#951;&#955;&#953;&#948;&#951;&#962; KazAnova;115664]</p>\n\n<p>he did a lot of extra work in creating group-by type of aggregate features (like averages) of v50 and other variables (up to 3 way).</p>\n\n<p>[/quote]</p>\n\n<p>I'm not sure I follow what you are saying--could you expand upon this?</p>\n\n<p>[quote]\nOur ensemble was consisted of 223 models.\n[/quote]</p>\n\n<p>What do you consider a different model.  You did mention things like one hot encoding.  Am I understsanding that for 1 &quot;model&quot; you leave all category variables as encoded integers, then for a second model you one hot encode.  Basically small variations of the feature set consist of a model?</p>\n\n<p>Many thanks in advance</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 115673,
      "author_name": "Scirpus",
      "author_url": "",
      "post_date": "2016-04-19T15:26:45.447000",
      "content": "<p>Excellent Work - I was struggling memory-wise with 56 features which ate up most of my 6Gb - and that was using i3 on linux to reduce the memory of the windows manager!</p>\n\n<p>Many congratulations to you all!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 115932,
      "author_name": "Rodolphe Lampe",
      "author_url": "",
      "post_date": "2016-04-20T21:33:33.510000",
      "content": "<p>Andreu, let say you know v1 and v2 have an important interaction, then you can for all values (a,b) taken by (v1,v2) compute the target's mean for this condition (v1 = a and v2 = b). Then, for each rows of your dataframe that verify this condition, you add the target's mean value. You do that for all (a,b).\nThis script does just that : <a href=\"https://www.kaggle.com/rsakata/bnp-paribas-cardif-claims-management/xgboost-with-combination-of-factors\">https://www.kaggle.com/rsakata/bnp-paribas-cardif-claims-management/xgboost-with-combination-of-factors</a>\nYou also have to pay attention to leakage so you need to not use the information of the rows your adding the new variable to (look at the script).</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 133476,
      "author_name": "lll",
      "author_url": "",
      "post_date": "2016-09-02T13:24:14.097000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 133427,
      "author_name": "lll",
      "author_url": "",
      "post_date": "2016-09-02T05:01:33.687000",
      "content": "<p>[quote=RodolpheLampe;115932]</p>\n\n<p>Andreu, let say you know v1 and v2 have an important interaction, then you can for all values (a,b) taken by (v1,v2) compute the target's mean for this condition (v1 = a and v2 = b). Then, for each rows of your dataframe that verify this condition, you add the target's mean value. You do that for all (a,b).\nThis script does just that : <a href=\"https://www.kaggle.com/rsakata/bnp-paribas-cardif-claims-management/xgboost-with-combination-of-factors\">https://www.kaggle.com/rsakata/bnp-paribas-cardif-claims-management/xgboost-with-combination-of-factors</a>\nYou also have to pay attention to leakage so you need to not use the information of the rows your adding the new variable to (look at the script).</p>\n\n<p>[/quote]\nHey, thanks for explaining!\nCould you please post some references here about why you exploit interactions this way?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 115878,
      "author_name": "Andreu Sancho-Asensio",
      "author_url": "",
      "post_date": "2016-04-20T15:24:37.157000",
      "content": "<p>[quote=Faron;115854]</p>\n\n<p>Thank you @Andreu!</p>\n\n<p>I'm afraid that I don't get your question. What exactly do you mean by &quot;how did you manage variable interactions&quot;?</p>\n\n<p>[/quote]</p>\n\n<p>After detecting a 3-way interaction, what processes do  you follow to exploit that? I am new to this kind of analysis but as far as I can tell this interaction tells that the variables involved should not be removed from the data set. </p>\n\n<p>Thanks for your time a congratulations again!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 115854,
      "author_name": "Faron",
      "author_url": "",
      "post_date": "2016-04-20T13:40:16.520000",
      "content": "<p>Thank you @Andreu!</p>\n\n<p>I'm afraid that I don't get your question. What exactly do you mean by &quot;how did you manage variable interactions&quot;?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 115737,
      "author_name": "Andreu Sancho-Asensio",
      "author_url": "",
      "post_date": "2016-04-19T20:00:06.887000",
      "content": "<p>Many congratulations and thanks for sharing! Just one question: how did you manage variable interactions? In the forum I often read about 2-, 3-, K-way interactions, but I don't exactly know what to do with these except for the fact that it is not recommended to remove variables that are in relations even when these don't seem to add (useful) information about the class label.</p>\n\n<p>Thank you and many congratulations!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 115697,
      "author_name": "Shubin",
      "author_url": "",
      "post_date": "2016-04-19T17:10:36.163000",
      "content": "<p>thank you for sharing the code. please correct me if my understanding is not correct. you have trained 233 models on different data sets with different feature engineering. therefore, you have 233 features(do you have any idea how strong these each individual models? your nn at first level does not look very complicated) . in the second level,  you apply ET , 2NN and 2 Xgmodels on this 233 features and ensemble the result.</p>\n\n<p>have you consider incorporating some of your useful features along with these 233 features to produce the output?  </p>\n\n<p>in addition, do you have any suggestion regarding code management? i would imagine it is indeed a skill to manage this process and do proper cv.  i always mess up.\nthanks again! </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 115973,
      "author_name": "Andreu Sancho-Asensio",
      "author_url": "",
      "post_date": "2016-04-21T05:49:44.100000",
      "content": "<p>Thank you!</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "comment_count": 16,
  "messages": [
    {
      "id": 115664,
      "postDate": "2016-04-19T14:53:24.130Z",
      "content": "<p>I think our solution may sound too plain compared to what the top 2 teams did :) .</p>\n\n<p>We spent quite some time reading forum threads and applied many of things discussed with some success like :</p>\n\n<p>1) rounding features to remove noise\n2) finding the correct denominator of features to get them back to the initial integer state\n3) converting categorical variables to likelihoods</p>\n\n<p>They all helped  a little bit but the most important thing for us (and thanks to clobber for finding it! ) was interacting the categorical variables together , especially  v22 with all the rest categorical variables in 2way and even 3way manner. </p>\n\n<p>Another thing that helped was rounding strongly ( 2 decimals) all the numerical features and creating interactions (as categorical &quot;numericalrounded_catgeorical&quot; with some of the categorical ones.) </p>\n\n<p>Stans was kinda of our features'factory machine! - apart from likelihood and other target-based feature transformations on categorical data, he did a lot of extra work in creating group-by type of aggregate features (like averages) of v50 and other variables (up to 3 way). Although we never understood exactly why this works, stans was able to capture some very useful patterns here .</p>\n\n<p>Our best single xgboost would have finished 4th (0.4242 in private) and was consisted of more than 7.5 K features (mostly interactions).</p>\n\n<p>We also produced many different base level models without much Feature engineering, just different input format types (like load all categorical variables as counts, or as onehot encoding etc).</p>\n\n<p>Our ensemble was consisted of 223 models. Faron did a lot of work in removing noise and discarding many of these in order to get to our bets score with a lvl2 ensemble of geomean weights between an ET ,  2NN and 2 Xgmodels. </p>\n\n<p>We burnt many cores in case you might have questions about it!</p>",
      "rawMarkdown": "I think our solution may sound too plain compared to what the top 2 teams did :) .\r\n\r\nWe spent quite some time reading forum threads and applied many of things discussed with some success like :\r\n\r\n1) rounding features to remove noise\r\n2) finding the correct denominator of features to get them back to the initial integer state\r\n3) converting categorical variables to likelihoods\r\n\r\nThey all helped  a little bit but the most important thing for us (and thanks to clobber for finding it! ) was interacting the categorical variables together , especially  v22 with all the rest categorical variables in 2way and even 3way manner. \r\n\r\nAnother thing that helped was rounding strongly ( 2 decimals) all the numerical features and creating interactions (as categorical \"numericalrounded_catgeorical\" with some of the categorical ones.) \r\n\r\nStans was kinda of our features'factory machine! - apart from likelihood and other target-based feature transformations on categorical data, he did a lot of extra work in creating group-by type of aggregate features (like averages) of v50 and other variables (up to 3 way). Although we never understood exactly why this works, stans was able to capture some very useful patterns here .\r\n\r\nOur best single xgboost would have finished 4th (0.4242 in private) and was consisted of more than 7.5 K features (mostly interactions).\r\n\r\nWe also produced many different base level models without much Feature engineering, just different input format types (like load all categorical variables as counts, or as onehot encoding etc).\r\n\r\nOur ensemble was consisted of 223 models. Faron did a lot of work in removing noise and discarding many of these in order to get to our bets score with a lvl2 ensemble of geomean weights between an ET ,  2NN and 2 Xgmodels. \r\n\r\nWe burnt many cores in case you might have questions about it!\r\n\r\n\r\n\r\n\r\n\r\n\r\n",
      "votes": 52
    },
    {
      "id": 115888,
      "postDate": "2016-04-20T16:08:35.673Z",
      "content": "<p>I feel keen to mention that our team's name is inspired by:</p>\n\n<p><a href=\"http://www.explodingkittens.com/\">http://www.explodingkittens.com/</a></p>\n\n<p>(A card game for those who like both kittens and explosions!)</p>\n\n<p><img src=\"http://www.explodingkittens.com/img/defuse.png\" alt=\"Exploding Kittens\" title></p>",
      "rawMarkdown": "I feel keen to mention that our team's name is inspired by:\r\n\r\nhttp://www.explodingkittens.com/\r\n\r\n(A card game for those who like both kittens and explosions!)\r\n\r\n![Exploding Kittens][1]\r\n\r\n\r\n  [1]: http://www.explodingkittens.com/img/defuse.png",
      "votes": 10
    },
    {
      "id": 115689,
      "postDate": "2016-04-19T16:16:47.163Z",
      "content": "<p>[quote=Shubin;115688]</p>\n\n<p>congrats! Could you elaborate more on nn structure? besides, you only apply nn on the second level instead of the first?  if not, what would be the strcuture look like for the first level nn?</p>\n\n<p>[/quote]</p>\n\n<p>Thank you :) . I dont feel I am a great nn-tunner-structure-master and there are many different structures that could work I guess. </p>\n\n<p>I have tried nn in both 1st and end levels, although in 2nd levels they were much more useful. </p>\n\n<p>for second level I used  </p>\n\n<pre><code>models = Sequential()\nmodels.add(Dense(100, input_dim=input_dim, init='uniform', W_regularizer=l2(0.00001)))\nmodels.add(PReLU())\nmodels.add(BatchNormalization())\nmodels.add(Dropout(0.7))\nmodels.add(Dense(output_dim, init='uniform'))\nmodels.add(Activation('softmax'))\nopt = optimizers.Adagrad(lr=lrs)\nmodels.compile(loss='binary_crossentropy', optimizer=opt)\n</code></pre>\n\n<p>epocs = 50</p>\n\n<p>and for 1st level</p>\n\n<pre><code>models = Sequential()\nmodels.add(Dense(150, input_dim=input_dim, init='uniform', W_regularizer=l2(0.00001), activation='relu'))\n#models.add(PReLU())\nmodels.add(BatchNormalization())\nmodels.add(Dropout(0.8))\nmodels.add(Dense(output_dim, init='uniform'))\nmodels.add(Activation('softmax'))\nopt = optimizers.Adagrad(lr=lrs)\nmodels.compile(loss='binary_crossentropy', optimizer=opt)\n</code></pre>\n\n<p>also epoc = 50.</p>\n\n<p>Both in Keras.</p>\n\n<p>Hope it helps. I think other people did much better job- as said I am not good with nns. </p>",
      "rawMarkdown": "[quote=Shubin;115688]\r\n\r\ncongrats! Could you elaborate more on nn structure? besides, you only apply nn on the second level instead of the first?  if not, what would be the strcuture look like for the first level nn?\r\n\r\n[/quote]\r\n\r\nThank you :) . I dont feel I am a great nn-tunner-structure-master and there are many different structures that could work I guess. \r\n\r\nI have tried nn in both 1st and end levels, although in 2nd levels they were much more useful. \r\n\r\nfor second level I used  \r\n\r\n    models = Sequential()\r\n    models.add(Dense(100, input_dim=input_dim, init='uniform', W_regularizer=l2(0.00001)))\r\n    models.add(PReLU())\r\n    models.add(BatchNormalization())\r\n    models.add(Dropout(0.7))\r\n    models.add(Dense(output_dim, init='uniform'))\r\n    models.add(Activation('softmax'))\r\n    opt = optimizers.Adagrad(lr=lrs)\r\n    models.compile(loss='binary_crossentropy', optimizer=opt)\r\n\r\nepocs = 50\r\n\r\nand for 1st level\r\n\r\n    models = Sequential()\r\n    models.add(Dense(150, input_dim=input_dim, init='uniform', W_regularizer=l2(0.00001), activation='relu'))\r\n    #models.add(PReLU())\r\n    models.add(BatchNormalization())\r\n    models.add(Dropout(0.8))\r\n    models.add(Dense(output_dim, init='uniform'))\r\n    models.add(Activation('softmax'))\r\n    opt = optimizers.Adagrad(lr=lrs)\r\n    models.compile(loss='binary_crossentropy', optimizer=opt)\r\n\r\nalso epoc = 50.\r\n\r\nBoth in Keras.\r\n\r\nHope it helps. I think other people did much better job- as said I am not good with nns. ",
      "votes": 10
    },
    {
      "id": 115683,
      "postDate": "2016-04-19T15:54:26.260Z",
      "content": "<p>[quote=MadScientist;115676]</p>\n\n<p>First of all congrats.  Well done!</p>\n\n<p>[/quote]</p>\n\n<p>Thank u!</p>\n\n<p>[quote=MadScientist;115676]</p>\n\n<p>Then of course your reward is to be peppered with questions! =)</p>\n\n<p>[/quote]</p>\n\n<p>Thank u x2!</p>\n\n<p>[quote=MadScientist;115676]</p>\n\n<p>I'm not sure I follow what you are saying--could you expand upon this?</p>\n\n<p>[/quote]</p>\n\n<p>for example max(v50) , mean(v50), min(v50)\ngroup by v22\nWe did that for many categorical variables.  and also group by 2 cats or even 3 cats.\nps. when I say &quot;we&quot; , I mean Stans - haha.</p>\n\n<p>[quote=MadScientist;115676]</p>\n\n<p>.Am I understsanding that for 1 &quot;model&quot; you leave all category variables as encoded integers, then for a second model you one hot encode. </p>\n\n<p>[/quote]</p>\n\n<p>Yes that's it . Also , trying different algorithm in the same data input is also considered a different model (at least in my slang!) </p>",
      "rawMarkdown": "[quote=MadScientist;115676]\r\n\r\nFirst of all congrats.  Well done!\r\n\r\n[/quote]\r\n\r\nThank u!\r\n\r\n[quote=MadScientist;115676]\r\n\r\nThen of course your reward is to be peppered with questions! =)\r\n\r\n[/quote]\r\n\r\nThank u x2!\r\n\r\n[quote=MadScientist;115676]\r\n\r\nI'm not sure I follow what you are saying--could you expand upon this?\r\n\r\n[/quote]\r\n\r\nfor example max(v50) , mean(v50), min(v50)\r\ngroup by v22\r\nWe did that for many categorical variables.  and also group by 2 cats or even 3 cats.\r\nps. when I say \"we\" , I mean Stans - haha.\r\n\r\n[quote=MadScientist;115676]\r\n\r\n.Am I understsanding that for 1 \"model\" you leave all category variables as encoded integers, then for a second model you one hot encode. \r\n\r\n[/quote]\r\n\r\nYes that's it . Also , trying different algorithm in the same data input is also considered a different model (at least in my slang!) \r\n\r\n",
      "votes": 6
    },
    {
      "id": 115934,
      "postDate": "2016-04-20T21:36:12.817Z",
      "content": "<p>[quote=Andreu;115878]</p>\n\n<p>After detecting a 3-way interaction, what processes do  you follow to exploit that? I am new to this kind of analysis but as far as I can tell this interaction tells that the variables involved should not be removed from the data set. </p>\n\n<p>[/quote]</p>\n\n<p>Just treat it as a new feature and process it with the same methods used for other features.</p>",
      "rawMarkdown": "[quote=Andreu;115878]\r\n\r\nAfter detecting a 3-way interaction, what processes do  you follow to exploit that? I am new to this kind of analysis but as far as I can tell this interaction tells that the variables involved should not be removed from the data set. \r\n\r\n[/quote]\r\n\r\nJust treat it as a new feature and process it with the same methods used for other features.",
      "votes": 3
    },
    {
      "id": 115729,
      "postDate": "2016-04-19T18:38:18.730Z",
      "content": "<p>[quote=Shubin;115697]</p>\n\n<p>you have trained 233 models on different data sets with different feature engineering. therefore, you have 233 features(do you have any idea how strong these each individual models? </p>\n\n<p>[/quote]</p>\n\n<p>The base models performances range from 0.4272 @public LB to something around 0.48x</p>\n\n<p>[quote=Shubin;115697]</p>\n\n<p>have you consider incorporating some of your useful features along with these 233 features to produce the output?  </p>\n\n<p>[/quote]</p>\n\n<p>Yes, but did not help us here.</p>\n\n<p>[quote=Shubin;115697]</p>\n\n<p>in addition, do you have any suggestion regarding code management? i would imagine it is indeed a skill to manage this process and do proper cv.  i always mess up.\nthanks again! </p>\n\n<p>[/quote]</p>\n\n<ul>\n<li>build some sort of your own kaggle coding conventions</li>\n<li>proper local folder structure</li>\n<li>keep track of what you are doing</li>\n<li>usage of svn or git</li>\n<li>write highly modularized, reusable code</li>\n</ul>",
      "rawMarkdown": "[quote=Shubin;115697]\r\n\r\nyou have trained 233 models on different data sets with different feature engineering. therefore, you have 233 features(do you have any idea how strong these each individual models? \r\n\r\n[/quote]\r\n\r\nThe base models performances range from 0.4272 @public LB to something around 0.48x\r\n\r\n[quote=Shubin;115697]\r\n\r\nhave you consider incorporating some of your useful features along with these 233 features to produce the output?  \r\n\r\n[/quote]\r\n\r\nYes, but did not help us here.\r\n\r\n[quote=Shubin;115697]\r\n\r\nin addition, do you have any suggestion regarding code management? i would imagine it is indeed a skill to manage this process and do proper cv.  i always mess up.\r\nthanks again! \r\n\r\n[/quote]\r\n\r\n - build some sort of your own kaggle coding conventions\r\n - proper local folder structure\r\n - keep track of what you are doing\r\n - usage of svn or git\r\n - write highly modularized, reusable code\r\n\r\n\r\n\r\n",
      "votes": 4
    },
    {
      "id": 115688,
      "postDate": "2016-04-19T16:08:16.867Z",
      "content": "<p>congrats! Could you elaborate more on nn structure? besides, you only apply nn on the second level instead of the first?  if not, what would be the strcuture look like for the first level nn?</p>\n\n<p>thanks again!</p>",
      "rawMarkdown": "congrats! Could you elaborate more on nn structure? besides, you only apply nn on the second level instead of the first?  if not, what would be the strcuture look like for the first level nn?\r\n\r\nthanks again!",
      "votes": 1
    },
    {
      "id": 115676,
      "postDate": "2016-04-19T15:34:07.463Z",
      "content": "<p>First of all congrats.  Well done!</p>\n\n<p>Then of course your reward is to be peppered with questions! =)</p>\n\n<p>[quote=&#924;&#945;&#961;&#953;&#959;&#962; &#924;&#953;&#967;&#945;&#951;&#955;&#953;&#948;&#951;&#962; KazAnova;115664]</p>\n\n<p>he did a lot of extra work in creating group-by type of aggregate features (like averages) of v50 and other variables (up to 3 way).</p>\n\n<p>[/quote]</p>\n\n<p>I'm not sure I follow what you are saying--could you expand upon this?</p>\n\n<p>[quote]\nOur ensemble was consisted of 223 models.\n[/quote]</p>\n\n<p>What do you consider a different model.  You did mention things like one hot encoding.  Am I understsanding that for 1 &quot;model&quot; you leave all category variables as encoded integers, then for a second model you one hot encode.  Basically small variations of the feature set consist of a model?</p>\n\n<p>Many thanks in advance</p>",
      "rawMarkdown": "First of all congrats.  Well done!\r\n\r\n\r\nThen of course your reward is to be peppered with questions! =)\r\n\r\n[quote=Μαριος Μιχαηλιδης KazAnova;115664]\r\n\r\nhe did a lot of extra work in creating group-by type of aggregate features (like averages) of v50 and other variables (up to 3 way).\r\n\r\n[/quote]\r\n\r\nI'm not sure I follow what you are saying--could you expand upon this?\r\n\r\n[quote]\r\nOur ensemble was consisted of 223 models.\r\n[/quote]\r\n\r\nWhat do you consider a different model.  You did mention things like one hot encoding.  Am I understsanding that for 1 \"model\" you leave all category variables as encoded integers, then for a second model you one hot encode.  Basically small variations of the feature set consist of a model?\r\n\r\nMany thanks in advance",
      "votes": 1
    },
    {
      "id": 115673,
      "postDate": "2016-04-19T15:26:45.447Z",
      "content": "<p>Excellent Work - I was struggling memory-wise with 56 features which ate up most of my 6Gb - and that was using i3 on linux to reduce the memory of the windows manager!</p>\n\n<p>Many congratulations to you all!</p>",
      "rawMarkdown": "Excellent Work - I was struggling memory-wise with 56 features which ate up most of my 6Gb - and that was using i3 on linux to reduce the memory of the windows manager!\r\n\r\nMany congratulations to you all!",
      "votes": 1
    },
    {
      "id": 115932,
      "postDate": "2016-04-20T21:33:33.510Z",
      "content": "<p>Andreu, let say you know v1 and v2 have an important interaction, then you can for all values (a,b) taken by (v1,v2) compute the target's mean for this condition (v1 = a and v2 = b). Then, for each rows of your dataframe that verify this condition, you add the target's mean value. You do that for all (a,b).\nThis script does just that : <a href=\"https://www.kaggle.com/rsakata/bnp-paribas-cardif-claims-management/xgboost-with-combination-of-factors\">https://www.kaggle.com/rsakata/bnp-paribas-cardif-claims-management/xgboost-with-combination-of-factors</a>\nYou also have to pay attention to leakage so you need to not use the information of the rows your adding the new variable to (look at the script).</p>",
      "rawMarkdown": "Andreu, let say you know v1 and v2 have an important interaction, then you can for all values (a,b) taken by (v1,v2) compute the target's mean for this condition (v1 = a and v2 = b). Then, for each rows of your dataframe that verify this condition, you add the target's mean value. You do that for all (a,b).\r\nThis script does just that : https://www.kaggle.com/rsakata/bnp-paribas-cardif-claims-management/xgboost-with-combination-of-factors\r\nYou also have to pay attention to leakage so you need to not use the information of the rows your adding the new variable to (look at the script).",
      "votes": 2
    },
    {
      "id": 133476,
      "postDate": "2016-09-02T13:24:14.097Z",
      "rawMarkdown": ""
    },
    {
      "id": 133427,
      "postDate": "2016-09-02T05:01:33.687Z",
      "content": "<p>[quote=RodolpheLampe;115932]</p>\n\n<p>Andreu, let say you know v1 and v2 have an important interaction, then you can for all values (a,b) taken by (v1,v2) compute the target's mean for this condition (v1 = a and v2 = b). Then, for each rows of your dataframe that verify this condition, you add the target's mean value. You do that for all (a,b).\nThis script does just that : <a href=\"https://www.kaggle.com/rsakata/bnp-paribas-cardif-claims-management/xgboost-with-combination-of-factors\">https://www.kaggle.com/rsakata/bnp-paribas-cardif-claims-management/xgboost-with-combination-of-factors</a>\nYou also have to pay attention to leakage so you need to not use the information of the rows your adding the new variable to (look at the script).</p>\n\n<p>[/quote]\nHey, thanks for explaining!\nCould you please post some references here about why you exploit interactions this way?</p>",
      "rawMarkdown": "[quote=RodolpheLampe;115932]\r\n\r\nAndreu, let say you know v1 and v2 have an important interaction, then you can for all values (a,b) taken by (v1,v2) compute the target's mean for this condition (v1 = a and v2 = b). Then, for each rows of your dataframe that verify this condition, you add the target's mean value. You do that for all (a,b).\r\nThis script does just that : https://www.kaggle.com/rsakata/bnp-paribas-cardif-claims-management/xgboost-with-combination-of-factors\r\nYou also have to pay attention to leakage so you need to not use the information of the rows your adding the new variable to (look at the script).\r\n\r\n[/quote]\r\nHey, thanks for explaining!\r\nCould you please post some references here about why you exploit interactions this way?"
    },
    {
      "id": 115878,
      "postDate": "2016-04-20T15:24:37.157Z",
      "content": "<p>[quote=Faron;115854]</p>\n\n<p>Thank you @Andreu!</p>\n\n<p>I'm afraid that I don't get your question. What exactly do you mean by &quot;how did you manage variable interactions&quot;?</p>\n\n<p>[/quote]</p>\n\n<p>After detecting a 3-way interaction, what processes do  you follow to exploit that? I am new to this kind of analysis but as far as I can tell this interaction tells that the variables involved should not be removed from the data set. </p>\n\n<p>Thanks for your time a congratulations again!</p>",
      "rawMarkdown": "[quote=Faron;115854]\r\n\r\nThank you @Andreu!\r\n\r\nI'm afraid that I don't get your question. What exactly do you mean by \"how did you manage variable interactions\"?\r\n\r\n[/quote]\r\n\r\nAfter detecting a 3-way interaction, what processes do  you follow to exploit that? I am new to this kind of analysis but as far as I can tell this interaction tells that the variables involved should not be removed from the data set. \r\n\r\nThanks for your time a congratulations again!\r\n\r\n"
    },
    {
      "id": 115854,
      "postDate": "2016-04-20T13:40:16.520Z",
      "content": "<p>Thank you @Andreu!</p>\n\n<p>I'm afraid that I don't get your question. What exactly do you mean by &quot;how did you manage variable interactions&quot;?</p>",
      "rawMarkdown": "Thank you @Andreu!\r\n\r\nI'm afraid that I don't get your question. What exactly do you mean by \"how did you manage variable interactions\"?"
    },
    {
      "id": 115737,
      "postDate": "2016-04-19T20:00:06.887Z",
      "content": "<p>Many congratulations and thanks for sharing! Just one question: how did you manage variable interactions? In the forum I often read about 2-, 3-, K-way interactions, but I don't exactly know what to do with these except for the fact that it is not recommended to remove variables that are in relations even when these don't seem to add (useful) information about the class label.</p>\n\n<p>Thank you and many congratulations!</p>",
      "rawMarkdown": "Many congratulations and thanks for sharing! Just one question: how did you manage variable interactions? In the forum I often read about 2-, 3-, K-way interactions, but I don't exactly know what to do with these except for the fact that it is not recommended to remove variables that are in relations even when these don't seem to add (useful) information about the class label.\r\n\r\nThank you and many congratulations!"
    },
    {
      "id": 115697,
      "postDate": "2016-04-19T17:10:36.163Z",
      "content": "<p>thank you for sharing the code. please correct me if my understanding is not correct. you have trained 233 models on different data sets with different feature engineering. therefore, you have 233 features(do you have any idea how strong these each individual models? your nn at first level does not look very complicated) . in the second level,  you apply ET , 2NN and 2 Xgmodels on this 233 features and ensemble the result.</p>\n\n<p>have you consider incorporating some of your useful features along with these 233 features to produce the output?  </p>\n\n<p>in addition, do you have any suggestion regarding code management? i would imagine it is indeed a skill to manage this process and do proper cv.  i always mess up.\nthanks again! </p>",
      "rawMarkdown": "thank you for sharing the code. please correct me if my understanding is not correct. you have trained 233 models on different data sets with different feature engineering. therefore, you have 233 features(do you have any idea how strong these each individual models? your nn at first level does not look very complicated) . in the second level,  you apply ET , 2NN and 2 Xgmodels on this 233 features and ensemble the result.\r\n\r\nhave you consider incorporating some of your useful features along with these 233 features to produce the output?  \r\n\r\nin addition, do you have any suggestion regarding code management? i would imagine it is indeed a skill to manage this process and do proper cv.  i always mess up.\r\nthanks again! \r\n\r\n"
    },
    {
      "id": 115973,
      "postDate": "2016-04-21T05:49:44.100Z",
      "content": "<p>Thank you!</p>",
      "rawMarkdown": "Thank you!\r\n",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T12:28:38Z",
  "competition_id": 4852,
  "competition_slug": "bnp-paribas-cardif-claims-management",
  "team_id": 286469,
  "solution_url": "/competitions/bnp-paribas-cardif-claims-management/writeups/exploding-kittens-3-place-solution"
}