{
  "id": 10921,
  "title": "logloss go up",
  "url": "/competitions/avazu-ctr-prediction/discussion/10921",
  "author_name": "",
  "post_date": "2014-11-13T14:08:20.653Z",
  "votes": 1,
  "comment_count": 8,
  "views": 3030,
  "content": "<p>Hello all,</p>\n<p>i'm nearly a newbie, sorry if my questions sounds na&#238;ve. But i think maybe someone have the same question as mine</p>\n<p>I use the online version of Logistic regression proposed by @Abhishek. But i get my logloss go up! i tried run small learning rate, but i didn't help.</p>\n<p>And thanks for @ACS69 for noticing me that perhaps i should shuffle the data before running the algo.&nbsp;</p>\n<p>i have the following questions:</p>\n<p>1. how to effecientlly shuffle the data with huge size like this ( unzipped file 8G). I seperate them into small chunk then used 'shuf' command in bash to shuffle each chunk and then combine them. a&nbsp;simpler way?</p>\n<p>2. do we indeed need to shuffle the data in the fashion of online learning? (i didn't run test in a shuffled data yet), but as i see in Criteo challenge, the logloss goes down without shuffle and also the data has sequential order like this data.&nbsp;</p>\n<p>Does this means each time we shuffle the data, we get a different result?</p>\n<p>3.&nbsp;for large set like this, how we do we do cross validation and prevent overfitting.</p>\n<p>ordinary k-fold validation is too costy i think. And leave one observation out at certain position (like %10000==0) is good? Or seprate as train, validation.</p>\n<p>And more, how to define the number of pass.....</p>\n<p>All suggestions are grateful.&nbsp;</p>\n<p>update: original learning rate 0.1, log_loss go up. change to&nbsp;.00001, log_loss go down without permutation</p>",
  "messages": [
    {
      "id": "57989",
      "postDate": "11/13/2014 14:08:20",
      "content": "<p>Hello all,</p>\n<p>i'm nearly a newbie, sorry if my questions sounds na&#238;ve. But i think maybe someone have the same question as mine</p>\n<p>I use the online version of Logistic regression proposed by @Abhishek. But i get my logloss go up! i tried run small learning rate, but i didn't help.</p>\n<p>And thanks for @ACS69 for noticing me that perhaps i should shuffle the data before running the algo.&nbsp;</p>\n<p>i have the following questions:</p>\n<p>1. how to effecientlly shuffle the data with huge size like this ( unzipped file 8G). I seperate them into small chunk then used 'shuf' command in bash to shuffle each chunk and then combine them. a&nbsp;simpler way?</p>\n<p>2. do we indeed need to shuffle the data in the fashion of online learning? (i didn't run test in a shuffled data yet), but as i see in Criteo challenge, the logloss goes down without shuffle and also the data has sequential order like this data.&nbsp;</p>\n<p>Does this means each time we shuffle the data, we get a different result?</p>\n<p>3.&nbsp;for large set like this, how we do we do cross validation and prevent overfitting.</p>\n<p>ordinary k-fold validation is too costy i think. And leave one observation out at certain position (like %10000==0) is good? Or seprate as train, validation.</p>\n<p>And more, how to define the number of pass.....</p>\n<p>All suggestions are grateful.&nbsp;</p>\n<p>update: original learning rate 0.1, log_loss go up. change to&nbsp;.00001, log_loss go down without permutation</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57991",
      "postDate": "11/13/2014 14:52:56",
      "content": "<p>Ad 1. If you are using an online algorithm shuffling the data is a bad move in my opinion.&nbsp;&nbsp;Don't compare this problem to Criteo. You cannot expect to copy the code from another competition and wait for it to create similar results.</p>\n<p>If I understand it correctly Abhishek's code uses something called adaptive learning&nbsp;rate. It works by adapting the learning rate to the strength of the <strong>past</strong> changes in the variable. I suppose the order in which the data is presented matters. TLDR; Yesterdays data matters more for today's prediction - don't shuffle.</p>\n<p>Ad 2. See ad 1</p>\n<p>Ad 3. For this size of the data holding&nbsp;out the 10th day as validation seems reasonable.&nbsp;So you train on 1-9 and predict 10th. Of course you can change it (train on 1-8 predict 9)</p>\n<p>Ad update smaller learning rate creates a more biased model - your log loss decreases but it can be not &quot;quick&quot;&nbsp;enough to adapt to the changes in the data.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57998",
      "postDate": "11/13/2014 15:34:22",
      "content": "<p>thanks for&nbsp;your answer,</p>\n<p>for the learning rate: yes, he reduce the initial rate by sqrt(n) , n as the iteration. ( i think sklearn has an option like this in SGDclassifier).</p>\n<p>but &nbsp;you said:</p>\n<p>&quot;Ad update smaller learning rate creates a more biased model - your log loss decreases but it can be not &quot;quick&quot; enough to adapt to the changes in the data.&quot;</p>\n<p>ah, ok, i finally understand you point. the idea is that for this adaptive rate, finally this rate will be small enough to give a small adaptation each iteration. we shoud set the rate large enough to let the parameter get around the right range rapidely.&nbsp;</p>\n<p>am i right?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58001",
      "postDate": "11/13/2014 15:45:53",
      "content": "<p>&gt;&nbsp;i'm not totally agree. so you mean that the logloss going up or down hasn't a lot meaning then?</p>\n<p>It has meaning and you should experiment with different settings of the learning rate. It should be large&nbsp;enough to catch some strong trends and not as big to cause these &quot;up and downs&quot;.&nbsp;</p>\n<p>To validate your learning rate you can create an experiment. Create 10 models: train 1st day / predict 2nd day; train 2nd day / predict 3rd day etc. Try different learning rates.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58005",
      "postDate": "11/13/2014 16:07:02",
      "content": "<p>&gt;&nbsp;i'm not totally agree. so you mean that the logloss going up or down hasn't a lot meaning then?</p>\n<p>It has meaning and you should experiment with different settings of the learning rate. It should be large&nbsp;enough to catch some strong trends and not as big to cause these &quot;up and downs&quot;.&nbsp;</p>\n<p>To validate your learning rate you can create an experiment. Create 10 models: train 1st day / predict 2nd day; train 2nd day / predict 3rd day etc. Try different learning rates.</p>\n<p>[/quote]</p>\n<p>yes, this is what suddenly came to me after i post the response, then i edited it, you confirmed me :).&nbsp;so the learning rate for you is better considered as a hyper-parameter for tuning? or we just set a sounds good one, as i think it's hard to tune it precisely.</p>\n<p>thanks again!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58394",
      "postDate": "11/19/2014 12:28:41",
      "content": "<p>[quote]</p>\n<p>...Don't compare this problem to Criteo. You cannot expect to copy the code from another competition and wait for it to create similar results...</p>\n<p>[/quote]</p>\n<p>Sorry to bring back an old topic, but this sentence, which is probably true, is really bad for the real purpose of Kaggle.</p>\n<p><br>If we cannot even reuse the code from a CTR competition to another CTR competition, because it's not applicable anymore, I think it&#8217;s really disappointing. It turns out that the only purpose of Kaggle is to help the specific companies that post competitions, while improving the skills of kagglers so they can keep building adhoc solutions.</p>\n<p><br>Of course, if the reason that is not reusable is because one company or the other decided to hide some important parameter, then what I would question is the usefulness of this competition to Avazu or Criteo.</p>\n<p><br>Maybe I&#8217;m too naive but I thought this whole Kaggle thing could serve a bigger purpose. One way to improve this situation,&nbsp;could be for Kaggle not to accept competitions with big amounts of anonymized parameters, which makes winner algorithms not suitable to be reused.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58398",
      "postDate": "11/19/2014 13:16:03",
      "content": "<p>Actually I may have been wrong here. You can use large parts of the solution as it turns out - I'm seeing a lot of people from Criteo doing ok in this competiton so I'm assuming they are using similar techniques.</p>\n<p>What I meant originally is generally linked to &quot;no free lunch&quot; theorem (http://en.wikipedia.org/wiki/No_free_lunch_theorem).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58454",
      "postDate": "11/19/2014 23:01:43",
      "content": "<p>To my understanding, with an online algorithm that makes one pass though the data, the order in which the data is presented affects your weight vector which subsequently affects your logloss. I know this to be true for the perceptron at the very least and the logistic regression model presented is essentially the same in terms of how the processes data.</p>\n<p>That being said, lowering your learning rate essentially decreases the perturbation of the weight vector so it is less likely to &quot;overshoot&quot; in dimensions and mess up on a point. Therefore, a small change results in smaller amounts of error when it shifts but also means that your algorithm converges more slowly.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "58503",
      "postDate": "11/20/2014 09:28:08",
      "content": "<p>[quote=jw592;58454]</p>\n<p>To my understanding, with an online algorithm that makes one pass though the data, the order in which the data is presented affects your weight vector which subsequently affects your logloss. I know this to be true for the perceptron at the very least and the logistic regression model presented is essentially the same in terms of how the processes data.</p>\n<p>[/quote]</p>\n<p>1.yes. that's true, so i agree what Pawel said that we should chose a learning rate not too small in order to catch the trends and let the algo converge fast but not too big as not to step too far away.</p>\n<p><strong>But it's hard to know cause so many choice, right?</strong></p>\n<p><strong>[0.1,0.01,0.001] -&gt; [0.1,0.05,0.01,0.005,0.001]</strong></p>\n<p><strong>does there existe some fast common choice for hyper parameter search? like golden search?&nbsp;</strong></p>\n<p><em>2. should we shuffle the data to get a more robust result?</em></p>\n<p>this is what i look the problem:</p>\n<p>if the order of data contains some information, llike different hour has different influence to the prediction. If we totaly draw this information from the data and add it to an attribute, then the of the data influence the prediction less.</p>\n<p>so for checking if we get the info,we can compare result of shuffled and non shuffled data, if it's big, so there are information that we don't draw out.<strong> Or</strong>&nbsp;LogisticModel does not model the data corretely (like they are in non-linear relation)</p>\n<p><strong>So we should model in order, but check in shuffle :) ?</strong></p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 57991,
      "author_name": "paweljankiewicz",
      "author_url": "",
      "post_date": "11/13/2014 14:52:56",
      "content": "<p>Ad 1. If you are using an online algorithm shuffling the data is a bad move in my opinion.&nbsp;&nbsp;Don't compare this problem to Criteo. You cannot expect to copy the code from another competition and wait for it to create similar results.</p>\n<p>If I understand it correctly Abhishek's code uses something called adaptive learning&nbsp;rate. It works by adapting the learning rate to the strength of the <strong>past</strong> changes in the variable. I suppose the order in which the data is presented matters. TLDR; Yesterdays data matters more for today's prediction - don't shuffle.</p>\n<p>Ad 2. See ad 1</p>\n<p>Ad 3. For this size of the data holding&nbsp;out the 10th day as validation seems reasonable.&nbsp;So you train on 1-9 and predict 10th. Of course you can change it (train on 1-8 predict 9)</p>\n<p>Ad update smaller learning rate creates a more biased model - your log loss decreases but it can be not &quot;quick&quot;&nbsp;enough to adapt to the changes in the data.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57998,
      "author_name": "betterplace",
      "author_url": "",
      "post_date": "11/13/2014 15:34:22",
      "content": "<p>thanks for&nbsp;your answer,</p>\n<p>for the learning rate: yes, he reduce the initial rate by sqrt(n) , n as the iteration. ( i think sklearn has an option like this in SGDclassifier).</p>\n<p>but &nbsp;you said:</p>\n<p>&quot;Ad update smaller learning rate creates a more biased model - your log loss decreases but it can be not &quot;quick&quot; enough to adapt to the changes in the data.&quot;</p>\n<p>ah, ok, i finally understand you point. the idea is that for this adaptive rate, finally this rate will be small enough to give a small adaptation each iteration. we shoud set the rate large enough to let the parameter get around the right range rapidely.&nbsp;</p>\n<p>am i right?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58001,
      "author_name": "paweljankiewicz",
      "author_url": "",
      "post_date": "11/13/2014 15:45:53",
      "content": "<p>&gt;&nbsp;i'm not totally agree. so you mean that the logloss going up or down hasn't a lot meaning then?</p>\n<p>It has meaning and you should experiment with different settings of the learning rate. It should be large&nbsp;enough to catch some strong trends and not as big to cause these &quot;up and downs&quot;.&nbsp;</p>\n<p>To validate your learning rate you can create an experiment. Create 10 models: train 1st day / predict 2nd day; train 2nd day / predict 3rd day etc. Try different learning rates.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58005,
      "author_name": "betterplace",
      "author_url": "",
      "post_date": "11/13/2014 16:07:02",
      "content": "<p>&gt;&nbsp;i'm not totally agree. so you mean that the logloss going up or down hasn't a lot meaning then?</p>\n<p>It has meaning and you should experiment with different settings of the learning rate. It should be large&nbsp;enough to catch some strong trends and not as big to cause these &quot;up and downs&quot;.&nbsp;</p>\n<p>To validate your learning rate you can create an experiment. Create 10 models: train 1st day / predict 2nd day; train 2nd day / predict 3rd day etc. Try different learning rates.</p>\n<p>[/quote]</p>\n<p>yes, this is what suddenly came to me after i post the response, then i edited it, you confirmed me :).&nbsp;so the learning rate for you is better considered as a hyper-parameter for tuning? or we just set a sounds good one, as i think it's hard to tune it precisely.</p>\n<p>thanks again!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58394,
      "author_name": "fernandoconstantino",
      "author_url": "",
      "post_date": "11/19/2014 12:28:41",
      "content": "<p>[quote]</p>\n<p>...Don't compare this problem to Criteo. You cannot expect to copy the code from another competition and wait for it to create similar results...</p>\n<p>[/quote]</p>\n<p>Sorry to bring back an old topic, but this sentence, which is probably true, is really bad for the real purpose of Kaggle.</p>\n<p><br>If we cannot even reuse the code from a CTR competition to another CTR competition, because it's not applicable anymore, I think it&#8217;s really disappointing. It turns out that the only purpose of Kaggle is to help the specific companies that post competitions, while improving the skills of kagglers so they can keep building adhoc solutions.</p>\n<p><br>Of course, if the reason that is not reusable is because one company or the other decided to hide some important parameter, then what I would question is the usefulness of this competition to Avazu or Criteo.</p>\n<p><br>Maybe I&#8217;m too naive but I thought this whole Kaggle thing could serve a bigger purpose. One way to improve this situation,&nbsp;could be for Kaggle not to accept competitions with big amounts of anonymized parameters, which makes winner algorithms not suitable to be reused.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58398,
      "author_name": "paweljankiewicz",
      "author_url": "",
      "post_date": "11/19/2014 13:16:03",
      "content": "<p>Actually I may have been wrong here. You can use large parts of the solution as it turns out - I'm seeing a lot of people from Criteo doing ok in this competiton so I'm assuming they are using similar techniques.</p>\n<p>What I meant originally is generally linked to &quot;no free lunch&quot; theorem (http://en.wikipedia.org/wiki/No_free_lunch_theorem).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58454,
      "author_name": "jasonwang592",
      "author_url": "",
      "post_date": "11/19/2014 23:01:43",
      "content": "<p>To my understanding, with an online algorithm that makes one pass though the data, the order in which the data is presented affects your weight vector which subsequently affects your logloss. I know this to be true for the perceptron at the very least and the logistic regression model presented is essentially the same in terms of how the processes data.</p>\n<p>That being said, lowering your learning rate essentially decreases the perturbation of the weight vector so it is less likely to &quot;overshoot&quot; in dimensions and mess up on a point. Therefore, a small change results in smaller amounts of error when it shifts but also means that your algorithm converges more slowly.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 58503,
      "author_name": "betterplace",
      "author_url": "",
      "post_date": "11/20/2014 09:28:08",
      "content": "<p>[quote=jw592;58454]</p>\n<p>To my understanding, with an online algorithm that makes one pass though the data, the order in which the data is presented affects your weight vector which subsequently affects your logloss. I know this to be true for the perceptron at the very least and the logistic regression model presented is essentially the same in terms of how the processes data.</p>\n<p>[/quote]</p>\n<p>1.yes. that's true, so i agree what Pawel said that we should chose a learning rate not too small in order to catch the trends and let the algo converge fast but not too big as not to step too far away.</p>\n<p><strong>But it's hard to know cause so many choice, right?</strong></p>\n<p><strong>[0.1,0.01,0.001] -&gt; [0.1,0.05,0.01,0.005,0.001]</strong></p>\n<p><strong>does there existe some fast common choice for hyper parameter search? like golden search?&nbsp;</strong></p>\n<p><em>2. should we shuffle the data to get a more robust result?</em></p>\n<p>this is what i look the problem:</p>\n<p>if the order of data contains some information, llike different hour has different influence to the prediction. If we totaly draw this information from the data and add it to an attribute, then the of the data influence the prediction less.</p>\n<p>so for checking if we get the info,we can compare result of shuffled and non shuffled data, if it's big, so there are information that we don't draw out.<strong> Or</strong>&nbsp;LogisticModel does not model the data corretely (like they are in non-linear relation)</p>\n<p><strong>So we should model in order, but check in shuffle :) ?</strong></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "57989": "",
    "57991": "",
    "57998": "",
    "58001": "",
    "58005": "",
    "58394": "",
    "58398": "",
    "58454": "",
    "58503": ""
  },
  "source": "meta"
}