{
  "id": 12669,
  "title": "Idea for model merging",
  "url": "/competitions/datasciencebowl/discussion/12669",
  "author_name": "alberto julian",
  "post_date": "2015-03-03T11:53:10.197000",
  "votes": 2,
  "comment_count": 0,
  "views": 842,
  "content": "<p>I've decided to leave the contest; my last effort has been looking for good algorithms to consistently improve CXXNET output and I failed.</p>\n<p>However my approach might be of interest for those who want to merge two models (example in attached code):</p>\n<ul>\n<li><span style=\"line-height: 1.4\">I take CXXNET output, where the input is the train images set; therefore, the real class to which each image belongs is known</span></li>\n<li>by analyzing the output (through some sort of confusion matrix), I identify clusters of classes whose separation is confusing among them, but clearly separated from the rest of classes; an example are classes 10, 11, 12 (the 3&nbsp;chaetognath classes)</li>\n<li>from the CXXNET probs output, I filter those rows (images) whose real class is in the class cluster</li>\n<li>I score the partial loss function for the train filtered list</li>\n<li>I train an alternate classifier method with features from this filtered image list</li>\n<li>if the partial output of the second classifier is better than that of the first:</li>\n<ul>\n<li>I filter from CXXNET test output those images whose predicted class is in the class cluster</li>\n<li>I predict the probabilities of the filtered list with the second classifier</li>\n<li>I replace the values in CXXNET test output with second classifier output for the filtered images and classes in the cluster</li>\n</ul>\n</ul>\n<p>I have attached code with an example; the numbers are:</p>\n<ul>\n<li><span style=\"line-height: 1.4\">I have a CXXNET as-is, with no augmentation or tuning;&nbsp;</span>scores 1.39 with the test images</li>\n<li><span style=\"line-height: 1.4\">I have a Random Forest trained with&nbsp;maxRegion ratios of the original images; scores 1.8 with the test images</span></li>\n<li><span style=\"line-height: 1.4\">in spite of CXXNET better global performance, RF performs better for class cluster 10, 11, 12</span></li>\n<li><span style=\"line-height: 1.4\">as a result, global loss function is (slightly) lowered</span></li>\n<li><span style=\"line-height: 1.4\">I applied it to the test images and improved my LB score</span></li>\n</ul>\n<p>After that, I tried with SURF, SIFT, ... but would need some time to tune this (and CXXNET augmentation, and GPU AWS execution, and...).&nbsp;</p>\n<p>The numbers are not as important as the idea: a worse global classifier can improve the output of a better one for a subset of images.</p>\n<p>I'm sure others have better model merging ideas that this, but there is very few in the forum about it, so I hope it helps someone.</p>\n<p>Have a nice contest end!</p>",
  "messages": [
    {
      "id": 65325,
      "postDate": "2015-03-03T11:53:10.197Z",
      "content": "<p>I've decided to leave the contest; my last effort has been looking for good algorithms to consistently improve CXXNET output and I failed.</p>\n<p>However my approach might be of interest for those who want to merge two models (example in attached code):</p>\n<ul>\n<li><span style=\"line-height: 1.4\">I take CXXNET output, where the input is the train images set; therefore, the real class to which each image belongs is known</span></li>\n<li>by analyzing the output (through some sort of confusion matrix), I identify clusters of classes whose separation is confusing among them, but clearly separated from the rest of classes; an example are classes 10, 11, 12 (the 3&nbsp;chaetognath classes)</li>\n<li>from the CXXNET probs output, I filter those rows (images) whose real class is in the class cluster</li>\n<li>I score the partial loss function for the train filtered list</li>\n<li>I train an alternate classifier method with features from this filtered image list</li>\n<li>if the partial output of the second classifier is better than that of the first:</li>\n<ul>\n<li>I filter from CXXNET test output those images whose predicted class is in the class cluster</li>\n<li>I predict the probabilities of the filtered list with the second classifier</li>\n<li>I replace the values in CXXNET test output with second classifier output for the filtered images and classes in the cluster</li>\n</ul>\n</ul>\n<p>I have attached code with an example; the numbers are:</p>\n<ul>\n<li><span style=\"line-height: 1.4\">I have a CXXNET as-is, with no augmentation or tuning;&nbsp;</span>scores 1.39 with the test images</li>\n<li><span style=\"line-height: 1.4\">I have a Random Forest trained with&nbsp;maxRegion ratios of the original images; scores 1.8 with the test images</span></li>\n<li><span style=\"line-height: 1.4\">in spite of CXXNET better global performance, RF performs better for class cluster 10, 11, 12</span></li>\n<li><span style=\"line-height: 1.4\">as a result, global loss function is (slightly) lowered</span></li>\n<li><span style=\"line-height: 1.4\">I applied it to the test images and improved my LB score</span></li>\n</ul>\n<p>After that, I tried with SURF, SIFT, ... but would need some time to tune this (and CXXNET augmentation, and GPU AWS execution, and...).&nbsp;</p>\n<p>The numbers are not as important as the idea: a worse global classifier can improve the output of a better one for a subset of images.</p>\n<p>I'm sure others have better model merging ideas that this, but there is very few in the forum about it, so I hope it helps someone.</p>\n<p>Have a nice contest end!</p>",
      "votes": 2
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "65325": ""
  }
}