{
  "id": 11379,
  "title": "Rare feature removal effective?",
  "url": "/competitions/avazu-ctr-prediction/discussion/11379",
  "author_name": "",
  "post_date": "2014-12-29T04:54:58.740Z",
  "votes": null,
  "comment_count": 19,
  "views": 6079,
  "content": "<p>Hi Forum,&nbsp;</p>\n<p>After one-hot code encoding, more than half of the features only appeared once. &nbsp;Based on insights from previous competitions, removing rare features could improve prediction by reducing noise. &nbsp;I tried to follow this advice on tinrtgu's FTRL code (thank you tinrtgu, it is a great piece of code for us newbies to learn), but removing the weights for all features that only appeared once in the training set actually reduced my validation score from 0.399 to 0.465. &nbsp;I am puzzled by this: are there other fellow Kagglers pursued similar approach on this dataset and willing to shed light on it?</p>\n<p><span style=\"line-height: 1.4\">Thanks in advance!</span></p>",
  "messages": [
    {
      "id": "61992",
      "postDate": "12/29/2014 04:54:58",
      "content": "<p>Hi Forum,&nbsp;</p>\n<p>After one-hot code encoding, more than half of the features only appeared once. &nbsp;Based on insights from previous competitions, removing rare features could improve prediction by reducing noise. &nbsp;I tried to follow this advice on tinrtgu's FTRL code (thank you tinrtgu, it is a great piece of code for us newbies to learn), but removing the weights for all features that only appeared once in the training set actually reduced my validation score from 0.399 to 0.465. &nbsp;I am puzzled by this: are there other fellow Kagglers pursued similar approach on this dataset and willing to shed light on it?</p>\n<p><span style=\"line-height: 1.4\">Thanks in advance!</span></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "61994",
      "postDate": "12/29/2014 06:04:18",
      "content": "<p>I tried it and didn't see any improvement, but certainly nothing as bad as 0.399 -&gt; 0.465. I'd guess that you've got a bug.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62011",
      "postDate": "12/29/2014 14:18:10",
      "content": "<p>I believe in FTRL the L1 term takes care of this problem for you automatically.<br>A&nbsp;value needs to reach a certain &quot;occurrance&quot; threshold before it add anything to the outcome.</p>\n<p>But I'm not Google or Trintu.. so you should ask them to be sure :)</p>\n<p>0.465 seems too much..</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62015",
      "postDate": "12/29/2014 15:25:30",
      "content": "<p>0.442 is what I got from LB score when I removed all the features, aka, constant CTR. So it is possible that you have removed all the features, instead of only the rare features</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62020",
      "postDate": "12/29/2014 16:57:52",
      "content": "<p>[quote=DerekZH;61992]</p>\n<p>Hi Forum,&nbsp;</p>\n<p>After one-hot code encoding, more than half of the features only appeared once. &nbsp;Based on insights from previous competitions, removing rare features could improve prediction by reducing noise. &nbsp;I tried to follow this advice on tinrtgu's FTRL code (thank you tinrtgu, it is a great piece of code for us newbies to learn), but removing the weights for all features that only appeared once in the training set actually reduced my validation score from 0.399 to 0.465. &nbsp;I am puzzled by this: are there other fellow Kagglers pursued similar approach on this dataset and willing to shed light on it?</p>\n<p><span style=\"line-height: 1.4\">Thanks in advance!</span></p>\n<p>[/quote]</p>\n<p>I tried it. There was some improvement in my validation score (~ 0.0001). What puzzles me is what people did to get better score than 0.398. No matter what I try with the tinrtgu's code ~0.398 is the best what I can get from it. I tried various values for learning rates, regularization parameters and number of epochs. I also tried to separate sites and applications it actually improved my validation score by 0.00001 Well, at least something. (Splitting training set using other variables made things much worse). I am slowly running out of ideas...I am training on 9 days and using the last day for validation. I need to think really hard to come up with something that I didn't try:)&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62030",
      "postDate": "12/29/2014 19:18:02",
      "content": "<p>Try training on all 10 days, and see if that improves your score.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62049",
      "postDate": "12/29/2014 21:15:43",
      "content": "<p>[quote=inversion;62030]</p>\n<p>Try training on all 10 days, and see if that improves your score.</p>\n<p>[/quote]</p>\n<p>There are 10 days in the training set. If I train on all 10 days how am I supposed to do validation??? For instance if I use every 100th example for validation and train on 10 days then the validation score will get inflated because the algorithm can adjust to parameters of a certain day. So in my opinion it is not a valid score. Am I missing something here? I am just wondering if people managed to get scores better than ~0.398 predicting values for the 10th day (using FTRL benchmark code). Most likely it is just me who don't understand Machine Learning though.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62051",
      "postDate": "12/29/2014 21:25:18",
      "content": "<p>[quote=bolo;62049]</p>\n<p>If I train on all 10 days how am I supposed to do validation???&nbsp;</p>\n<p>[/quote]</p>\n<p>The leader board score becomes your validation.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62055",
      "postDate": "12/29/2014 21:37:49",
      "content": "<p>[quote=inversion;62051]</p>\n<p>[quote=bolo;62049]</p>\n<p>If I train on all 10 days how am I supposed to do validation???&nbsp;</p>\n<p>[/quote]</p>\n<p>The leader board score becomes your validation.</p>\n<p>[/quote]</p>\n<p>The leader board doesn't exist for me. In real life you don't have any leader boards you just do cross-validation using only training data that is available to you to tune your algorithm. So if I cannot get a good CV score then I will never submit my results. CV is one of the most important parts of Machine Learning, it cannot be skipped.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62075",
      "postDate": "12/30/2014 01:30:37",
      "content": "<p>[quote=bolo;62055]</p>\n<p>I am afraid that it is not a right way to approach a problem. The leader board doesn't exist for me. In real life you don't have any leader boards you just do cross-validation using only training data that is available to you to tune your algorithm. So if I cannot get a good CV score then I will never submit my results. CV is one of the most important parts of Machine Learning, it cannot be skipped.</p>\n<p>[/quote]</p>\n<p>I'm not suggesting you skip CV all together.</p>\n<p>What if there are features in day 10 that aren't in days 1-9?</p>\n<p>If including day 10 helps your LB score, perhaps it should be used in training and a different CV approach used.</p>\n<p>I'm not saying one way or another. I'm just suggesting some ways to think about the problem.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62081",
      "postDate": "12/30/2014 04:14:58",
      "content": "<p>[quote=Nicholas Guttenberg;61994]</p>\n<p>I tried it and didn't see any improvement, but certainly nothing as bad as 0.399 -&gt; 0.465. I'd guess that you've got a bug.</p>\n<p>[/quote]</p>\n<p>Thanks for sharing, Nicholas and Birchwood. &nbsp;It makes sense that validation loss of 0.465 is too much&nbsp;given the all-0.5 benchmark. &nbsp;I am re-examining my code to spot a bug.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62082",
      "postDate": "12/30/2014 04:25:32",
      "content": "<p>[quote=Julian de Wit;62011]</p>\n<p>I believe in FTRL the L1 term takes care of this problem for you automatically.<br>A&nbsp;value needs to reach a certain &quot;occurrance&quot; threshold before it add anything to the outcome.</p>\n<p>[/quote]</p>\n\n<p>Thank you for pointing out the effect of L1 regularization for the FTRL algorithm, Julian. &nbsp;There is much more to learn about regularization! :-)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62084",
      "postDate": "12/30/2014 04:58:11",
      "content": "<p>[quote=bolo;62049]</p>\n<p>[quote=inversion;62030]</p>\n<p>Try training on all 10 days, and see if that improves your score.</p>\n<p>[/quote]</p>\n<p>There are 10 days in the training set. If I train on all 10 days how am I supposed to do validation??? For instance if I use every 100th example for validation and train on 10 days then the validation score will get inflated because the algorithm can adjust to parameters of a certain day. So in my opinion it is not a valid score. Am I missing something here? I am just wondering if people managed to get scores better than ~0.398 predicting values for the 10th day (using FTRL benchmark code). Most likely it is just me who don't understand Machine Learning though.</p>\n<p>[/quote]</p>\n\n<p>validating on just one day is not reliable. there are many differences between the days. a better way would be to use folds,like this:</p>\n<p>http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.KFold.html</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62102",
      "postDate": "12/30/2014 17:14:50",
      "content": "<p>[quote=laserwolf;62084]</p>\n<p>[quote=bolo;62049]</p>\n<p>[quote=inversion;62030]</p>\n<p>Try training on all 10 days, and see if that improves your score.</p>\n<p>[/quote]</p>\n<p>There are 10 days in the training set. If I train on all 10 days how am I supposed to do validation??? For instance if I use every 100th example for validation and train on 10 days then the validation score will get inflated because the algorithm can adjust to parameters of a certain day. So in my opinion it is not a valid score. Am I missing something here? I am just wondering if people managed to get scores better than ~0.398 predicting values for the 10th day (using FTRL benchmark code). Most likely it is just me who don't understand Machine Learning though.</p>\n<p>[/quote]</p>\n<p>validating on just one day is not reliable. there are many differences between the days. a better way would be to use folds,like this:</p>\n<p>http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.KFold.html</p>\n<p>[/quote]</p>\n<p>Of course KFold is better but it is rarely applied to big datasets (computationaly expensive). At least it is not afordable for me taking into account my hardware that is about 15 years old. For me ~6GB is extremely huge dataset.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62103",
      "postDate": "12/30/2014 17:45:59",
      "content": "<p>[quote=bolo;62102]</p>\n<p>Of course KFold is better but it is rarely applied to big datasets (computationaly expensive). At least it is not afordable for me taking into account my hardware that is about 15 years old. For me ~6GB is extremely huge dataset.</p>\n<p>[/quote]</p>\n<p>Have you tried holdout validation? It's worked nicely for me.</p>\n<p><img src=\"http://i.imgur.com/oapKDX7.png\" alt width=\"567\" height=\"380\"></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62105",
      "postDate": "12/30/2014 18:06:23",
      "content": "<p>[quote=inversion;62103]</p>\n<p>[quote=bolo;62102]</p>\n<p>Of course KFold is better but it is rarely applied to big datasets (computationaly expensive). At least it is not afordable for me taking into account my hardware that is about 15 years old. For me ~6GB is extremely huge dataset.</p>\n<p>[/quote]</p>\n<p>Have you tried holdout validation? It's worked nicely for me.</p>\n<p><img src=\"http://i.imgur.com/oapKDX7.png\" alt width=\"567\" height=\"380\"></p>\n<p>[/quote]</p>\n<p>Thanks for the chart! I tried to validate using every 100th example but it is gives a strange logloss of ~0.35 after I applied several experiments that reduced it from ~0.385 (I think one of the experiments led to severe overfitting that was not detected by holdout). At the same time validation on the 10th day improved only by ~0.0005. So it seems that when you do holdout it somehow adapts to the current day and not able to detect overfitting. Our goal is to predict values for a new day where will not have any information about the current click rate for example. Also it is worth noticing that you might experience the same pattern with me: after you reach a certain value for holdout loss your LB score stopped to go down (around ~0.383 in your case).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62108",
      "postDate": "12/30/2014 20:00:46",
      "content": "<p>[quote=inversion;62103]</p>\n<p>[quote=bolo;62102]</p>\n<p>Of course KFold is better but it is rarely applied to big datasets (computationaly expensive). At least it is not afordable for me taking into account my hardware that is about 15 years old. For me ~6GB is extremely huge dataset.</p>\n<p>[/quote]</p>\n<p>Have you tried holdout validation? It's worked nicely for me.</p>\n<p><img src=\"http://i.imgur.com/oapKDX7.png\" alt width=\"567\" height=\"380\"></p>\n<p>[/quote]</p>\n<p>Could you try to train on 9 days and validate on the 10th day? In my case more or less reliable scores were ~0.398 for the 10th day and ~0.385 for holdout (every 100th sample). All other parameters (regularization, learning rates) were the same.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62234",
      "postDate": "01/03/2015 17:04:20",
      "content": "<p>I'm amazed at how linear the relationship is between your holdout logloss and LB score. &nbsp; It hasn't been anywhere close for me.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62287",
      "postDate": "01/05/2015 04:45:02",
      "content": "<p>I recoded values from training set that are not in test per variables.</p>\n<p>LB: 0.400x (FTRL)</p>\n<p>But I'm sure there is a way to make it works! another modeling approach probably?</p>\n<p>[quote=DerekZH;61992]</p>\n<p>Hi Forum,&nbsp;</p>\n<p>After one-hot code encoding, more than half of the features only appeared once. &nbsp;Based on insights from previous competitions, removing rare features could improve prediction by reducing noise. &nbsp;I tried to follow this advice on tinrtgu's FTRL code (thank you tinrtgu, it is a great piece of code for us newbies to learn), but removing the weights for all features that only appeared once in the training set actually reduced my validation score from 0.399 to 0.465. &nbsp;I am puzzled by this: are there other fellow Kagglers pursued similar approach on this dataset and willing to shed light on it?</p>\n<p><span style=\"line-height: 1.4\">Thanks in advance!</span></p>\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62774",
      "postDate": "01/16/2015 03:13:28",
      "content": "<p>[quote=bolo;62020]</p>\n<p>[quote=DerekZH;61992]</p>\n<p>Hi Forum,&nbsp;</p>\n<p>After one-hot code encoding, more than half of the features only appeared once. &nbsp;Based on insights from previous competitions, removing rare features could improve prediction by reducing noise. &nbsp;I tried to follow this advice on tinrtgu's FTRL code (thank you tinrtgu, it is a great piece of code for us newbies to learn), but removing the weights for all features that only appeared once in the training set actually reduced my validation score from 0.399 to 0.465. &nbsp;I am puzzled by this: are there other fellow Kagglers pursued similar approach on this dataset and willing to shed light on it?</p>\n<p><span style=\"line-height: 1.4\">Thanks in advance!</span></p>\n<p>[/quote]</p>\n<p>I tried it. There was some improvement in my validation score (~ 0.0001). What puzzles me is what people did to get better score than 0.398. No matter what I try with the tinrtgu's code ~0.398 is the best what I can get from it. I tried various values for learning rates, regularization parameters and number of epochs. I also tried to separate sites and applications it actually improved my validation score by 0.00001 Well, at least something. (Splitting training set using other variables made things much worse). I am slowly running out of ideas...I am training on 9 days and using the last day for validation. I need to think really hard to come up with something that I didn't try:)&nbsp;</p>\n<p>[/quote]</p>\n<p>Actually you may not need a 0.398 validation score to reach a 0.398 LB score. &nbsp;But&nbsp;you can absolutely improve your validation score better than 0.398. I have made a 0.3979 validation loss by parameter tuning and adding some interactions by hand. You may give it a try.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 61994,
      "author_name": "ngutten",
      "author_url": "",
      "post_date": "12/29/2014 06:04:18",
      "content": "<p>I tried it and didn't see any improvement, but certainly nothing as bad as 0.399 -&gt; 0.465. I'd guess that you've got a bug.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62011,
      "author_name": "juliandewit",
      "author_url": "",
      "post_date": "12/29/2014 14:18:10",
      "content": "<p>I believe in FTRL the L1 term takes care of this problem for you automatically.<br>A&nbsp;value needs to reach a certain &quot;occurrance&quot; threshold before it add anything to the outcome.</p>\n<p>But I'm not Google or Trintu.. so you should ask them to be sure :)</p>\n<p>0.465 seems too much..</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62015,
      "author_name": "jialunhe",
      "author_url": "",
      "post_date": "12/29/2014 15:25:30",
      "content": "<p>0.442 is what I got from LB score when I removed all the features, aka, constant CTR. So it is possible that you have removed all the features, instead of only the rare features</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62020,
      "author_name": "kolobok007",
      "author_url": "",
      "post_date": "12/29/2014 16:57:52",
      "content": "<p>[quote=DerekZH;61992]</p>\n<p>Hi Forum,&nbsp;</p>\n<p>After one-hot code encoding, more than half of the features only appeared once. &nbsp;Based on insights from previous competitions, removing rare features could improve prediction by reducing noise. &nbsp;I tried to follow this advice on tinrtgu's FTRL code (thank you tinrtgu, it is a great piece of code for us newbies to learn), but removing the weights for all features that only appeared once in the training set actually reduced my validation score from 0.399 to 0.465. &nbsp;I am puzzled by this: are there other fellow Kagglers pursued similar approach on this dataset and willing to shed light on it?</p>\n<p><span style=\"line-height: 1.4\">Thanks in advance!</span></p>\n<p>[/quote]</p>\n<p>I tried it. There was some improvement in my validation score (~ 0.0001). What puzzles me is what people did to get better score than 0.398. No matter what I try with the tinrtgu's code ~0.398 is the best what I can get from it. I tried various values for learning rates, regularization parameters and number of epochs. I also tried to separate sites and applications it actually improved my validation score by 0.00001 Well, at least something. (Splitting training set using other variables made things much worse). I am slowly running out of ideas...I am training on 9 days and using the last day for validation. I need to think really hard to come up with something that I didn't try:)&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62030,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "12/29/2014 19:18:02",
      "content": "<p>Try training on all 10 days, and see if that improves your score.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62049,
      "author_name": "kolobok007",
      "author_url": "",
      "post_date": "12/29/2014 21:15:43",
      "content": "<p>[quote=inversion;62030]</p>\n<p>Try training on all 10 days, and see if that improves your score.</p>\n<p>[/quote]</p>\n<p>There are 10 days in the training set. If I train on all 10 days how am I supposed to do validation??? For instance if I use every 100th example for validation and train on 10 days then the validation score will get inflated because the algorithm can adjust to parameters of a certain day. So in my opinion it is not a valid score. Am I missing something here? I am just wondering if people managed to get scores better than ~0.398 predicting values for the 10th day (using FTRL benchmark code). Most likely it is just me who don't understand Machine Learning though.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62051,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "12/29/2014 21:25:18",
      "content": "<p>[quote=bolo;62049]</p>\n<p>If I train on all 10 days how am I supposed to do validation???&nbsp;</p>\n<p>[/quote]</p>\n<p>The leader board score becomes your validation.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62055,
      "author_name": "kolobok007",
      "author_url": "",
      "post_date": "12/29/2014 21:37:49",
      "content": "<p>[quote=inversion;62051]</p>\n<p>[quote=bolo;62049]</p>\n<p>If I train on all 10 days how am I supposed to do validation???&nbsp;</p>\n<p>[/quote]</p>\n<p>The leader board score becomes your validation.</p>\n<p>[/quote]</p>\n<p>The leader board doesn't exist for me. In real life you don't have any leader boards you just do cross-validation using only training data that is available to you to tune your algorithm. So if I cannot get a good CV score then I will never submit my results. CV is one of the most important parts of Machine Learning, it cannot be skipped.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62075,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "12/30/2014 01:30:37",
      "content": "<p>[quote=bolo;62055]</p>\n<p>I am afraid that it is not a right way to approach a problem. The leader board doesn't exist for me. In real life you don't have any leader boards you just do cross-validation using only training data that is available to you to tune your algorithm. So if I cannot get a good CV score then I will never submit my results. CV is one of the most important parts of Machine Learning, it cannot be skipped.</p>\n<p>[/quote]</p>\n<p>I'm not suggesting you skip CV all together.</p>\n<p>What if there are features in day 10 that aren't in days 1-9?</p>\n<p>If including day 10 helps your LB score, perhaps it should be used in training and a different CV approach used.</p>\n<p>I'm not saying one way or another. I'm just suggesting some ways to think about the problem.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62081,
      "author_name": "derekzh",
      "author_url": "",
      "post_date": "12/30/2014 04:14:58",
      "content": "<p>[quote=Nicholas Guttenberg;61994]</p>\n<p>I tried it and didn't see any improvement, but certainly nothing as bad as 0.399 -&gt; 0.465. I'd guess that you've got a bug.</p>\n<p>[/quote]</p>\n<p>Thanks for sharing, Nicholas and Birchwood. &nbsp;It makes sense that validation loss of 0.465 is too much&nbsp;given the all-0.5 benchmark. &nbsp;I am re-examining my code to spot a bug.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62082,
      "author_name": "derekzh",
      "author_url": "",
      "post_date": "12/30/2014 04:25:32",
      "content": "<p>[quote=Julian de Wit;62011]</p>\n<p>I believe in FTRL the L1 term takes care of this problem for you automatically.<br>A&nbsp;value needs to reach a certain &quot;occurrance&quot; threshold before it add anything to the outcome.</p>\n<p>[/quote]</p>\n\n<p>Thank you for pointing out the effect of L1 regularization for the FTRL algorithm, Julian. &nbsp;There is much more to learn about regularization! :-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62084,
      "author_name": "vvarkey",
      "author_url": "",
      "post_date": "12/30/2014 04:58:11",
      "content": "<p>[quote=bolo;62049]</p>\n<p>[quote=inversion;62030]</p>\n<p>Try training on all 10 days, and see if that improves your score.</p>\n<p>[/quote]</p>\n<p>There are 10 days in the training set. If I train on all 10 days how am I supposed to do validation??? For instance if I use every 100th example for validation and train on 10 days then the validation score will get inflated because the algorithm can adjust to parameters of a certain day. So in my opinion it is not a valid score. Am I missing something here? I am just wondering if people managed to get scores better than ~0.398 predicting values for the 10th day (using FTRL benchmark code). Most likely it is just me who don't understand Machine Learning though.</p>\n<p>[/quote]</p>\n\n<p>validating on just one day is not reliable. there are many differences between the days. a better way would be to use folds,like this:</p>\n<p>http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.KFold.html</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62102,
      "author_name": "kolobok007",
      "author_url": "",
      "post_date": "12/30/2014 17:14:50",
      "content": "<p>[quote=laserwolf;62084]</p>\n<p>[quote=bolo;62049]</p>\n<p>[quote=inversion;62030]</p>\n<p>Try training on all 10 days, and see if that improves your score.</p>\n<p>[/quote]</p>\n<p>There are 10 days in the training set. If I train on all 10 days how am I supposed to do validation??? For instance if I use every 100th example for validation and train on 10 days then the validation score will get inflated because the algorithm can adjust to parameters of a certain day. So in my opinion it is not a valid score. Am I missing something here? I am just wondering if people managed to get scores better than ~0.398 predicting values for the 10th day (using FTRL benchmark code). Most likely it is just me who don't understand Machine Learning though.</p>\n<p>[/quote]</p>\n<p>validating on just one day is not reliable. there are many differences between the days. a better way would be to use folds,like this:</p>\n<p>http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.KFold.html</p>\n<p>[/quote]</p>\n<p>Of course KFold is better but it is rarely applied to big datasets (computationaly expensive). At least it is not afordable for me taking into account my hardware that is about 15 years old. For me ~6GB is extremely huge dataset.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62103,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "12/30/2014 17:45:59",
      "content": "<p>[quote=bolo;62102]</p>\n<p>Of course KFold is better but it is rarely applied to big datasets (computationaly expensive). At least it is not afordable for me taking into account my hardware that is about 15 years old. For me ~6GB is extremely huge dataset.</p>\n<p>[/quote]</p>\n<p>Have you tried holdout validation? It's worked nicely for me.</p>\n<p><img src=\"http://i.imgur.com/oapKDX7.png\" alt width=\"567\" height=\"380\"></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62105,
      "author_name": "kolobok007",
      "author_url": "",
      "post_date": "12/30/2014 18:06:23",
      "content": "<p>[quote=inversion;62103]</p>\n<p>[quote=bolo;62102]</p>\n<p>Of course KFold is better but it is rarely applied to big datasets (computationaly expensive). At least it is not afordable for me taking into account my hardware that is about 15 years old. For me ~6GB is extremely huge dataset.</p>\n<p>[/quote]</p>\n<p>Have you tried holdout validation? It's worked nicely for me.</p>\n<p><img src=\"http://i.imgur.com/oapKDX7.png\" alt width=\"567\" height=\"380\"></p>\n<p>[/quote]</p>\n<p>Thanks for the chart! I tried to validate using every 100th example but it is gives a strange logloss of ~0.35 after I applied several experiments that reduced it from ~0.385 (I think one of the experiments led to severe overfitting that was not detected by holdout). At the same time validation on the 10th day improved only by ~0.0005. So it seems that when you do holdout it somehow adapts to the current day and not able to detect overfitting. Our goal is to predict values for a new day where will not have any information about the current click rate for example. Also it is worth noticing that you might experience the same pattern with me: after you reach a certain value for holdout loss your LB score stopped to go down (around ~0.383 in your case).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62108,
      "author_name": "kolobok007",
      "author_url": "",
      "post_date": "12/30/2014 20:00:46",
      "content": "<p>[quote=inversion;62103]</p>\n<p>[quote=bolo;62102]</p>\n<p>Of course KFold is better but it is rarely applied to big datasets (computationaly expensive). At least it is not afordable for me taking into account my hardware that is about 15 years old. For me ~6GB is extremely huge dataset.</p>\n<p>[/quote]</p>\n<p>Have you tried holdout validation? It's worked nicely for me.</p>\n<p><img src=\"http://i.imgur.com/oapKDX7.png\" alt width=\"567\" height=\"380\"></p>\n<p>[/quote]</p>\n<p>Could you try to train on 9 days and validate on the 10th day? In my case more or less reliable scores were ~0.398 for the 10th day and ~0.385 for holdout (every 100th sample). All other parameters (regularization, learning rates) were the same.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62234,
      "author_name": "evanvanness",
      "author_url": "",
      "post_date": "01/03/2015 17:04:20",
      "content": "<p>I'm amazed at how linear the relationship is between your holdout logloss and LB score. &nbsp; It hasn't been anywhere close for me.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62287,
      "author_name": "herimanitra",
      "author_url": "",
      "post_date": "01/05/2015 04:45:02",
      "content": "<p>I recoded values from training set that are not in test per variables.</p>\n<p>LB: 0.400x (FTRL)</p>\n<p>But I'm sure there is a way to make it works! another modeling approach probably?</p>\n<p>[quote=DerekZH;61992]</p>\n<p>Hi Forum,&nbsp;</p>\n<p>After one-hot code encoding, more than half of the features only appeared once. &nbsp;Based on insights from previous competitions, removing rare features could improve prediction by reducing noise. &nbsp;I tried to follow this advice on tinrtgu's FTRL code (thank you tinrtgu, it is a great piece of code for us newbies to learn), but removing the weights for all features that only appeared once in the training set actually reduced my validation score from 0.399 to 0.465. &nbsp;I am puzzled by this: are there other fellow Kagglers pursued similar approach on this dataset and willing to shed light on it?</p>\n<p><span style=\"line-height: 1.4\">Thanks in advance!</span></p>\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62774,
      "author_name": "yejiming",
      "author_url": "",
      "post_date": "01/16/2015 03:13:28",
      "content": "<p>[quote=bolo;62020]</p>\n<p>[quote=DerekZH;61992]</p>\n<p>Hi Forum,&nbsp;</p>\n<p>After one-hot code encoding, more than half of the features only appeared once. &nbsp;Based on insights from previous competitions, removing rare features could improve prediction by reducing noise. &nbsp;I tried to follow this advice on tinrtgu's FTRL code (thank you tinrtgu, it is a great piece of code for us newbies to learn), but removing the weights for all features that only appeared once in the training set actually reduced my validation score from 0.399 to 0.465. &nbsp;I am puzzled by this: are there other fellow Kagglers pursued similar approach on this dataset and willing to shed light on it?</p>\n<p><span style=\"line-height: 1.4\">Thanks in advance!</span></p>\n<p>[/quote]</p>\n<p>I tried it. There was some improvement in my validation score (~ 0.0001). What puzzles me is what people did to get better score than 0.398. No matter what I try with the tinrtgu's code ~0.398 is the best what I can get from it. I tried various values for learning rates, regularization parameters and number of epochs. I also tried to separate sites and applications it actually improved my validation score by 0.00001 Well, at least something. (Splitting training set using other variables made things much worse). I am slowly running out of ideas...I am training on 9 days and using the last day for validation. I need to think really hard to come up with something that I didn't try:)&nbsp;</p>\n<p>[/quote]</p>\n<p>Actually you may not need a 0.398 validation score to reach a 0.398 LB score. &nbsp;But&nbsp;you can absolutely improve your validation score better than 0.398. I have made a 0.3979 validation loss by parameter tuning and adding some interactions by hand. You may give it a try.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "61992": "",
    "61994": "",
    "62011": "",
    "62015": "",
    "62020": "",
    "62030": "",
    "62049": "",
    "62051": "",
    "62055": "",
    "62075": "",
    "62081": "",
    "62082": "",
    "62084": "",
    "62102": "",
    "62103": "",
    "62105": "",
    "62108": "",
    "62234": "",
    "62287": "",
    "62774": ""
  },
  "source": "meta"
}