{
  "id": 12526,
  "title": "Self training does not work?",
  "url": "/competitions/datasciencebowl/discussion/12526",
  "author_name": "",
  "post_date": "2015-02-16T03:54:07.803Z",
  "votes": 4,
  "comment_count": 10,
  "views": 3312,
  "content": "<p>As the testing set is 4+ times larger than the training set, I tried to utilize the unlabeled testing set to help the performance. A straight forward and the most scalable approach is self-training. It is reported in some papers that self-training will help the performance when the training set is not sufficient large. I have tried several settings but none of them helped. I am not sure if this approach simply does not work, or I failed to find an appropriate way to make it work. The followings are some settings I have tried.</p>\n<p>1. the way to assign pseudo labels to the testing data</p>\n<p>a. hard assign and add&nbsp;the top 20% high confidence samples into training set.</p>\n<p>b. hard assign&nbsp;and add&nbsp;the top 40% high confidence samples into training set.</p>\n<p>c. soft assign: randomly assign a label to a testing sample according its predicted softmax probs, for each sample I run such soft assign for 5 times.</p>\n<p>2. the mixture ratio of true training data and pseudo training data.</p>\n<p>a. I did not carefully study this parameter, I simply set 1:1 ratio in all experiments.</p>",
  "messages": [
    {
      "id": "64306",
      "postDate": "02/16/2015 03:54:07",
      "content": "<p>As the testing set is 4+ times larger than the training set, I tried to utilize the unlabeled testing set to help the performance. A straight forward and the most scalable approach is self-training. It is reported in some papers that self-training will help the performance when the training set is not sufficient large. I have tried several settings but none of them helped. I am not sure if this approach simply does not work, or I failed to find an appropriate way to make it work. The followings are some settings I have tried.</p>\n<p>1. the way to assign pseudo labels to the testing data</p>\n<p>a. hard assign and add&nbsp;the top 20% high confidence samples into training set.</p>\n<p>b. hard assign&nbsp;and add&nbsp;the top 40% high confidence samples into training set.</p>\n<p>c. soft assign: randomly assign a label to a testing sample according its predicted softmax probs, for each sample I run such soft assign for 5 times.</p>\n<p>2. the mixture ratio of true training data and pseudo training data.</p>\n<p>a. I did not carefully study this parameter, I simply set 1:1 ratio in all experiments.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64308",
      "postDate": "02/16/2015 04:40:46",
      "content": "<p>When you say it doesn't work, do you mean it doesn't work in local cross-validation, or on the leaderboard?</p>\n<p>As you know, the LB test data is corrupt with large numbers of inter-correlated samples, which could potentially cause self-training to fail (through overfitting).</p>\n<p>My personal experience with self-training (on other problems, I haven't tried it on this dataset) has been that although it generally helps a little, the improvement is often negligible. My take is that any slight improvement to your data augmentation, your preprocessing or your model itself will be worth 10x times what you could get through self-training, so that's where to invest the most efforts.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64314",
      "postDate": "02/16/2015 05:04:43",
      "content": "<p>@fchollet</p>\n<p>Thanks for your comments, It helps!</p>\n\n<p>I did not submitted the self-training result on the LB. As it worsen the local validation score, I do not expect that it will get better results on LB.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64367",
      "postDate": "02/16/2015 20:07:30",
      "content": "<p>Are their any generic papers about self-learning (the theoretical way it works) ?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64370",
      "postDate": "02/16/2015 22:11:31",
      "content": "<p>@xvnv do you mean unsupervised learning (like autoencoders) or stuff like &quot;pseudo-label&quot;?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64420",
      "postDate": "02/17/2015 20:18:44",
      "content": "<p>@old-ufo I mean &quot;pseudo-label&quot;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64453",
      "postDate": "02/18/2015 02:07:45",
      "content": "<p>Hi,xudong,&nbsp;</p>\n<p>I've tried semi-supervise learning since the very first commits in this competition, as there is a rather large testing set and &nbsp;small, biased training set. But the result is pretty the same as yours, method I've tried include pseudo label and pseudo ensemble, with both dynamic labeling or leave out top samples as you did. In my experiments, SS learning with a lower weight for unlabel samples does not harm the classification accuracy on validation set, but the log loss in both val/test set are damaged (since it's not quite the same as accuracy). I think it's may be the biased distribution of sample set that makes it not working, and I'm still runing tests on it.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64483",
      "postDate": "02/18/2015 10:19:16",
      "content": "<p>Hi has any one ever tried&nbsp;auto encoders (for self training) whose weights can be used later for fine tuning - in a conv neural net ? I have seem some papers but the architecture of such a self training network is very hazy. Will be great to hear from some of you.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64490",
      "postDate": "02/18/2015 13:30:56",
      "content": "<p>For me self-training helped a little, maybe it gets vanished by having stronger CNN models, but to get an idea of improvement:&nbsp;</p>\n<p>currently, I have 3 models in a linear blend.</p>\n<p>#1: nn+augmentation&nbsp;1.14078</p>\n<p>#2: nn: 1.64873</p>\n<p>#3: nn on 30-dim features: 1.38158 &nbsp;(30-dim features comes from an autoencoder, trained unsupervised with augmentations on train+test)</p>\n<p>blend: #1 + #2: 1.0988</p>\n<p>blend: #1 + #2 +#3:&nbsp;1.06544 (so self-training gave me a 0.033 improvement).</p>\n<p>All numbers are logloss on a 20% random validation set. Leaderboard numbers are slightly but consistently&nbsp;better.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64851",
      "postDate": "02/25/2015 07:48:46",
      "content": "<p>[quote=dreamtreee;64453]</p>\n<p>Hi,xudong,&nbsp;</p>\n<p>I've tried semi-supervise learning since the very first commits in this competition, as there is a rather large testing set and &nbsp;small, biased training set. But the result is pretty the same as yours, method I've tried include pseudo label and pseudo ensemble, with both dynamic labeling or leave out top samples as you did. In my experiments, SS learning with a lower weight for unlabel samples does not harm the classification accuracy on validation set, but the log loss in both val/test set are damaged (since it's not quite the same as accuracy). I think it's may be the biased distribution of sample set that makes it not working, and I'm still runing tests on it.</p>\n<p>[/quote]</p>\n\n\n<p>Are you Wang Fan?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "64897",
      "postDate": "02/26/2015 00:37:53",
      "content": "<p>[quote=xudong cao;64851]</p>\n<p>[quote=dreamtreee;64453]</p>\n<p>Hi,xudong,&nbsp;</p>\n<p>I've tried semi-supervise learning since the very first commits in this competition, as there is a rather large testing set and &nbsp;small, biased training set. But the result is pretty the same as yours, method I've tried include pseudo label and pseudo ensemble, with both dynamic labeling or leave out top samples as you did. In my experiments, SS learning with a lower weight for unlabel samples does not harm the classification accuracy on validation set, but the log loss in both val/test set are damaged (since it's not quite the same as accuracy). I think it's may be the biased distribution of sample set that makes it not working, and I'm still runing tests on it.</p>\n<p>[/quote]</p>\n<p>Are you Wang Fan?</p>\n<p>[/quote]</p>\n<p>Ha, long time no see:)</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 64308,
      "author_name": "fchollet",
      "author_url": "",
      "post_date": "02/16/2015 04:40:46",
      "content": "<p>When you say it doesn't work, do you mean it doesn't work in local cross-validation, or on the leaderboard?</p>\n<p>As you know, the LB test data is corrupt with large numbers of inter-correlated samples, which could potentially cause self-training to fail (through overfitting).</p>\n<p>My personal experience with self-training (on other problems, I haven't tried it on this dataset) has been that although it generally helps a little, the improvement is often negligible. My take is that any slight improvement to your data augmentation, your preprocessing or your model itself will be worth 10x times what you could get through self-training, so that's where to invest the most efforts.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64314,
      "author_name": "xudongcao",
      "author_url": "",
      "post_date": "02/16/2015 05:04:43",
      "content": "<p>@fchollet</p>\n<p>Thanks for your comments, It helps!</p>\n\n<p>I did not submitted the self-training result on the LB. As it worsen the local validation score, I do not expect that it will get better results on LB.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64367,
      "author_name": "oneoff",
      "author_url": "",
      "post_date": "02/16/2015 20:07:30",
      "content": "<p>Are their any generic papers about self-learning (the theoretical way it works) ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64370,
      "author_name": "oldufo",
      "author_url": "",
      "post_date": "02/16/2015 22:11:31",
      "content": "<p>@xvnv do you mean unsupervised learning (like autoencoders) or stuff like &quot;pseudo-label&quot;?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64420,
      "author_name": "oneoff",
      "author_url": "",
      "post_date": "02/17/2015 20:18:44",
      "content": "<p>@old-ufo I mean &quot;pseudo-label&quot;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64453,
      "author_name": "dreamtree1990",
      "author_url": "",
      "post_date": "02/18/2015 02:07:45",
      "content": "<p>Hi,xudong,&nbsp;</p>\n<p>I've tried semi-supervise learning since the very first commits in this competition, as there is a rather large testing set and &nbsp;small, biased training set. But the result is pretty the same as yours, method I've tried include pseudo label and pseudo ensemble, with both dynamic labeling or leave out top samples as you did. In my experiments, SS learning with a lower weight for unlabel samples does not harm the classification accuracy on validation set, but the log loss in both val/test set are damaged (since it's not quite the same as accuracy). I think it's may be the biased distribution of sample set that makes it not working, and I'm still runing tests on it.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64483,
      "author_name": "rightfit",
      "author_url": "",
      "post_date": "02/18/2015 10:19:16",
      "content": "<p>Hi has any one ever tried&nbsp;auto encoders (for self training) whose weights can be used later for fine tuning - in a conv neural net ? I have seem some papers but the architecture of such a self training network is very hazy. Will be great to hear from some of you.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64490,
      "author_name": "mjahrer",
      "author_url": "",
      "post_date": "02/18/2015 13:30:56",
      "content": "<p>For me self-training helped a little, maybe it gets vanished by having stronger CNN models, but to get an idea of improvement:&nbsp;</p>\n<p>currently, I have 3 models in a linear blend.</p>\n<p>#1: nn+augmentation&nbsp;1.14078</p>\n<p>#2: nn: 1.64873</p>\n<p>#3: nn on 30-dim features: 1.38158 &nbsp;(30-dim features comes from an autoencoder, trained unsupervised with augmentations on train+test)</p>\n<p>blend: #1 + #2: 1.0988</p>\n<p>blend: #1 + #2 +#3:&nbsp;1.06544 (so self-training gave me a 0.033 improvement).</p>\n<p>All numbers are logloss on a 20% random validation set. Leaderboard numbers are slightly but consistently&nbsp;better.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64851,
      "author_name": "xudongcao",
      "author_url": "",
      "post_date": "02/25/2015 07:48:46",
      "content": "<p>[quote=dreamtreee;64453]</p>\n<p>Hi,xudong,&nbsp;</p>\n<p>I've tried semi-supervise learning since the very first commits in this competition, as there is a rather large testing set and &nbsp;small, biased training set. But the result is pretty the same as yours, method I've tried include pseudo label and pseudo ensemble, with both dynamic labeling or leave out top samples as you did. In my experiments, SS learning with a lower weight for unlabel samples does not harm the classification accuracy on validation set, but the log loss in both val/test set are damaged (since it's not quite the same as accuracy). I think it's may be the biased distribution of sample set that makes it not working, and I'm still runing tests on it.</p>\n<p>[/quote]</p>\n\n\n<p>Are you Wang Fan?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 64897,
      "author_name": "dreamtree1990",
      "author_url": "",
      "post_date": "02/26/2015 00:37:53",
      "content": "<p>[quote=xudong cao;64851]</p>\n<p>[quote=dreamtreee;64453]</p>\n<p>Hi,xudong,&nbsp;</p>\n<p>I've tried semi-supervise learning since the very first commits in this competition, as there is a rather large testing set and &nbsp;small, biased training set. But the result is pretty the same as yours, method I've tried include pseudo label and pseudo ensemble, with both dynamic labeling or leave out top samples as you did. In my experiments, SS learning with a lower weight for unlabel samples does not harm the classification accuracy on validation set, but the log loss in both val/test set are damaged (since it's not quite the same as accuracy). I think it's may be the biased distribution of sample set that makes it not working, and I'm still runing tests on it.</p>\n<p>[/quote]</p>\n<p>Are you Wang Fan?</p>\n<p>[/quote]</p>\n<p>Ha, long time no see:)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "64306": "",
    "64308": "",
    "64314": "",
    "64367": "",
    "64370": "",
    "64420": "",
    "64453": "",
    "64483": "",
    "64490": "",
    "64851": "",
    "64897": ""
  },
  "source": "meta"
}