Congratulations again to all the Winners!! I believe that top winners will give many valuable insights soon. :)
\n\nIn this post I would like to share some of our findings and experience in NLP, feature extraction and feature selection tasks, and also my ensemble architecture. Feature engineering helped us a lot in improving the performance of our different models.
\n\nHere I write only about my own parts and I think @Darragh (Pearson correlation analysis to submissions) and @CodingNeo would share their amazing parts later.
\n\nDue to the limitation of Web editor, I cannot show my post in well format here. So please check the following link to download and see full post:
\n\nhttps://drive.google.com/file/d/0B-20iuF5gIcyQWtpOVlnTTREX3M/view?usp=sharing
\n\nR example code for RFE and Boruta feature selection can be downloaded from the attachment.
\n\nThis is my first time to share the details of my feature engineering and ensemble architecture in Kaggle. Please feel free to give any comments.
\n\nHope it can help other kagglers to some extent. :)
\n\nBest regards,
\n\nMark Peng
", "comments": [ { "id": 85627, "author_name": "Mark Peng", "author_url": "", "post_date": "2015-07-16T02:03:56.097000", "content": "[quote=Jing Lu;85549]
\n\nHi Mark,
\n\nThanks for sharing! Your approach is very interesting, but somehow also complex. Do you split data for layer 0 and layer 1, or just use the same training data? So, for each layer, you do a feature selection with RFE. Why do you choose 200 features for layer 0 and 150 features for layer 1? I was thinking feature selection between layer 0 and 1 may help eliminate some data leaking during training..
\n\nThanks,\nJing
\n\n[/quote]
\n\nThanks for the comment!\nRegarding ensemble, I followed the stacked generalization process mentioned in @Triskelion's great post about ensemble learning.
\n\n\n \n\n\n
Let’s say you want to do 2-fold stacking:\n\nSplit the train set in 2 parts: train_a and train_b\nFit a first-stage model on train_a and create predictions for train_b\nFit the same model on train_b and create predictions for train_a\nFinally fit the model on the entire train set and create predictions for the test set.\nNow train a second-stage stacker model on the probabilities from the first-stage model(s).\n\nA stacker model gets more information on the problem space by using the first-stage predictions as features, than if it was trained in isolation.\nI used the same 2-fold CV process to generate predictions as meta-features for layer 1. Feature selection is also done by 3-fold CV (repeats 3 times) using caret package in R. So 200 raw features for layer 0 and 150 meta-features for layer 1 were chosen by 3-fold CV with RFE. Therefore, the chance to data leaking during training should be very small. Similar ensemble process is mentioned in Otto competition winner's interview as well.
\n\nThanks,
\n\nMark
", "votes": 1, "replies": [] }, { "id": 85549, "author_name": "Jing Lu", "author_url": "", "post_date": "2015-07-15T15:37:35.633000", "content": "Hi Mark,
\n\nThanks for sharing! Your approach is very interesting, but somehow also complex. Do you split data for layer 0 and layer 1, or just use the same training data? So, for each layer, you do a feature selection with RFE. Why do you choose 200 features for layer 0 and 150 features for layer 1? I was thinking feature selection between layer 0 and 1 may help eliminate some data leaking during training..
\n\nThanks,\nJing
", "votes": 1, "replies": [] }, { "id": 84144, "author_name": "Mark Nagelberg", "author_url": "", "post_date": "2015-07-11T16:15:24.050000", "content": "Awesome! Very useful to see your features/models in such clear detail, and I didn't know about GloVe, RFE, or Boruta - all look like really useful tools. Also liked seeing your feature selection process as mine could use some serious work (i.e. I didn't really use a formal feature selection process at all :S). I also didn't know about using multiple "layers" when ensembling.
\n\nThanks Mark!
", "votes": 2, "replies": [] }, { "id": 85628, "author_name": "Mark Peng", "author_url": "", "post_date": "2015-07-16T02:07:03.963000", "content": "[quote=Mark Peng;85627]
\n\n[quote=Jing Lu;85549]
\n\nHi Mark,
\n\nThanks for sharing! Your approach is very interesting, but somehow also complex. Do you split data for layer 0 and layer 1, or just use the same training data? So, for each layer, you do a feature selection with RFE. Why do you choose 200 features for layer 0 and 150 features for layer 1? I was thinking feature selection between layer 0 and 1 may help eliminate some data leaking during training..
\n\nThanks,\nJing
\n\n[/quote]
\n\nThanks for the comment!\nRegarding ensemble, I followed the stacked generalization process mentioned in @Triskelion's great post about ensemble learning. Quoted here:
\n\n\n \n\n\n
Let’s say you want to do 2-fold stacking:\n\nSplit the train set in 2 parts: train_a and train_b\nFit a first-stage model on train_a and create predictions for train_b\nFit the same model on train_b and create predictions for train_a\nFinally fit the model on the entire train set and create predictions for the test set.\nNow train a second-stage stacker model on the probabilities from the first-stage model(s).\n\nA stacker model gets more information on the problem space by using the first-stage predictions as features, than if it was trained in isolation.\nI used the same 2-fold CV process to generate predictions as meta-features for layer 1. Feature selection is also done by 3-fold CV (repeats 3 times) using caret package in R. So 200 raw features for layer 0 and 150 meta-features for layer 1 were chosen by 3-fold CV with RFE. Therefore, the chance to data leaking during training should be very small. Similar ensemble process is mentioned in Otto competition winner's interview as well.
\n\nThanks,
\n\nMark
\n\n[/quote]
", "votes": 0, "replies": [] }, { "id": 85485, "author_name": "Mark Peng", "author_url": "", "post_date": "2015-07-15T03:54:25.547000", "content": "[quote=Khanna;84422]
\n\nNice!\nAlas! I had a similar brainstorming , but kudos you covered a vast array of intuitive cases.\nI couldnt get any more than 0.56+ on the leaderboard with my similar heuristics.\nGreat job. At once when I read on other kaggle scripts, I almost lost faith in such feature engineering.
\n\n[/quote]
\n\nThanks Khanna!\nI had several years of experience in information extraction and search engine applications, so I knew that those heuristics should work! \nGloVe deep learning vectors are also very helpful, in addition to pattern-based features.\nYou can download all my features from the document to see if you can find better ways to use them than mine. :)
\n\nThanks,
\n\nMark
", "votes": 0, "replies": [] }, { "id": 84422, "author_name": "Khanna", "author_url": "", "post_date": "2015-07-14T17:16:06.133000", "content": "Nice!\nAlas! I had a similar brainstorming , but kudos you covered a vast array of intuitive cases.\nI couldnt get any more than 0.56+ on the leaderboard with my similar heuristics.\nGreat job. At once when I read on other kaggle scripts, I almost lost faith in such feature engineering.
", "votes": 0, "replies": [] }, { "id": 84189, "author_name": "Mark Peng", "author_url": "", "post_date": "2015-07-12T04:33:56.027000", "content": "[quote=Mark Nagelberg;84144]
\n\nAwesome! Very useful to see your features/models in such clear detail, and I didn't know about GloVe, RFE, or Boruta - all look like really useful tools. Also liked seeing your feature selection process as mine could use some serious work (i.e. I didn't really use a formal feature selection process at all :S). I also didn't know about using multiple "layers" when ensembling.
\n\nThanks Mark!
\n\n[/quote]
\n\nHi Mark,
\n\nI am glad to know that it is helpful to you. :)\nActually I spent more than 12 hours in the past several weekends for this competition.\nMost of the work were done for feature engineering!\nIt took me two days in full time to compete this write up, because I really would like share my ideas with you!
\n\nI also learned a lot from the posts by top kagglers in previous competitions regarding ensemble learning too.
\n\nThanks,
\n\nMark
", "votes": 0, "replies": [] } ], "comment_count": 7, "messages": [ { "id": 84136, "postDate": "2015-07-11T14:24:35.713Z", "content": "Congratulations again to all the Winners!! I believe that top winners will give many valuable insights soon. :)
\n\nIn this post I would like to share some of our findings and experience in NLP, feature extraction and feature selection tasks, and also my ensemble architecture. Feature engineering helped us a lot in improving the performance of our different models.
\n\nHere I write only about my own parts and I think @Darragh (Pearson correlation analysis to submissions) and @CodingNeo would share their amazing parts later.
\n\nDue to the limitation of Web editor, I cannot show my post in well format here. So please check the following link to download and see full post:
\n\nhttps://drive.google.com/file/d/0B-20iuF5gIcyQWtpOVlnTTREX3M/view?usp=sharing
\n\nR example code for RFE and Boruta feature selection can be downloaded from the attachment.
\n\nThis is my first time to share the details of my feature engineering and ensemble architecture in Kaggle. Please feel free to give any comments.
\n\nHope it can help other kagglers to some extent. :)
\n\nBest regards,
\n\nMark Peng
", "rawMarkdown": "Congratulations again to all the Winners!! I believe that top winners will give many valuable insights soon. :)\r\n\r\nIn this post I would like to share some of our findings and experience in NLP, feature extraction and feature selection tasks, and also my ensemble architecture. Feature engineering helped us a lot in improving the performance of our different models.\r\n\r\nHere I write only about my own parts and I think @Darragh (Pearson correlation analysis to submissions) and @CodingNeo would share their amazing parts later.\r\n\r\nDue to the limitation of Web editor, I cannot show my post in well format here. So please check the following link to download and see full post:\r\n\r\nhttps://drive.google.com/file/d/0B-20iuF5gIcyQWtpOVlnTTREX3M/view?usp=sharing\r\n\r\n\r\n----------\r\n\r\n\r\nR example code for RFE and Boruta feature selection can be downloaded from the attachment.\r\n\r\n\r\n\r\nThis is my first time to share the details of my feature engineering and ensemble architecture in Kaggle. Please feel free to give any comments.\r\n\r\nHope it can help other kagglers to some extent. :)\r\n\r\n\r\n\r\nBest regards,\r\n\r\nMark Peng\r\n\r\n\r\n [1]: https://drive.google.com/file/d/0B-20iuF5gIcyaXI1YnVUalp6b28/view?usp=sharing", "votes": 10 }, { "id": 85627, "postDate": "2015-07-16T02:03:56.097Z", "content": "[quote=Jing Lu;85549]
\n\nHi Mark,
\n\nThanks for sharing! Your approach is very interesting, but somehow also complex. Do you split data for layer 0 and layer 1, or just use the same training data? So, for each layer, you do a feature selection with RFE. Why do you choose 200 features for layer 0 and 150 features for layer 1? I was thinking feature selection between layer 0 and 1 may help eliminate some data leaking during training..
\n\nThanks,\nJing
\n\n[/quote]
\n\nThanks for the comment!\nRegarding ensemble, I followed the stacked generalization process mentioned in @Triskelion's great post about ensemble learning.
\n\n\n \n\n\n
Let’s say you want to do 2-fold stacking:\n\nSplit the train set in 2 parts: train_a and train_b\nFit a first-stage model on train_a and create predictions for train_b\nFit the same model on train_b and create predictions for train_a\nFinally fit the model on the entire train set and create predictions for the test set.\nNow train a second-stage stacker model on the probabilities from the first-stage model(s).\n\nA stacker model gets more information on the problem space by using the first-stage predictions as features, than if it was trained in isolation.\nI used the same 2-fold CV process to generate predictions as meta-features for layer 1. Feature selection is also done by 3-fold CV (repeats 3 times) using caret package in R. So 200 raw features for layer 0 and 150 meta-features for layer 1 were chosen by 3-fold CV with RFE. Therefore, the chance to data leaking during training should be very small. Similar ensemble process is mentioned in Otto competition winner's interview as well.
\n\nThanks,
\n\nMark
", "rawMarkdown": "[quote=Jing Lu;85549]\r\n\r\nHi Mark,\r\n\r\nThanks for sharing! Your approach is very interesting, but somehow also complex. Do you split data for layer 0 and layer 1, or just use the same training data? So, for each layer, you do a feature selection with RFE. Why do you choose 200 features for layer 0 and 150 features for layer 1? I was thinking feature selection between layer 0 and 1 may help eliminate some data leaking during training..\r\n\r\n\r\nThanks,\r\nJing\r\n\r\n[/quote]\r\n\r\n\r\nThanks for the comment!\r\nRegarding ensemble, I followed the stacked generalization process mentioned in @Triskelion's [great post][1] about ensemble learning.\r\n\r\n> \r\n\r\n Let’s say you want to do 2-fold stacking:\r\n\r\n Split the train set in 2 parts: train_a and train_b\r\n Fit a first-stage model on train_a and create predictions for train_b\r\n Fit the same model on train_b and create predictions for train_a\r\n Finally fit the model on the entire train set and create predictions for the test set.\r\n Now train a second-stage stacker model on the probabilities from the first-stage model(s).\r\n\r\n A stacker model gets more information on the problem space by using the first-stage predictions as features, than if it was trained in isolation.\r\n\r\nI used the same 2-fold CV process to generate predictions as meta-features for layer 1. Feature selection is also done by 3-fold CV (repeats 3 times) using *caret* package in R. So 200 **raw features** for layer 0 and 150 **meta-features** for layer 1 were chosen by 3-fold CV with RFE. Therefore, the chance to data leaking during training should be very small. Similar ensemble process is mentioned in [Otto competition winner's interview][2] as well.\r\n\r\nThanks,\r\n\r\nMark\r\n\r\n\r\n [1]: http://mlwave.com/kaggle-ensembling-guide/\r\n [2]: http://blog.kaggle.com/2015/06/09/otto-product-classification-winners-interview-2nd-place-alexander-guschin/", "votes": 1 }, { "id": 85549, "postDate": "2015-07-15T15:37:35.633Z", "content": "Hi Mark,
\n\nThanks for sharing! Your approach is very interesting, but somehow also complex. Do you split data for layer 0 and layer 1, or just use the same training data? So, for each layer, you do a feature selection with RFE. Why do you choose 200 features for layer 0 and 150 features for layer 1? I was thinking feature selection between layer 0 and 1 may help eliminate some data leaking during training..
\n\nThanks,\nJing
", "rawMarkdown": "Hi Mark,\r\n\r\nThanks for sharing! Your approach is very interesting, but somehow also complex. Do you split data for layer 0 and layer 1, or just use the same training data? So, for each layer, you do a feature selection with RFE. Why do you choose 200 features for layer 0 and 150 features for layer 1? I was thinking feature selection between layer 0 and 1 may help eliminate some data leaking during training..\r\n\r\n\r\nThanks,\r\nJing", "votes": 1 }, { "id": 84144, "postDate": "2015-07-11T16:15:24.050Z", "content": "Awesome! Very useful to see your features/models in such clear detail, and I didn't know about GloVe, RFE, or Boruta - all look like really useful tools. Also liked seeing your feature selection process as mine could use some serious work (i.e. I didn't really use a formal feature selection process at all :S). I also didn't know about using multiple "layers" when ensembling.
\n\nThanks Mark!
", "rawMarkdown": "Awesome! Very useful to see your features/models in such clear detail, and I didn't know about GloVe, RFE, or Boruta - all look like really useful tools. Also liked seeing your feature selection process as mine could use some serious work (i.e. I didn't really use a formal feature selection process at all :S). I also didn't know about using multiple \"layers\" when ensembling. \r\n\r\nThanks Mark!", "votes": 2 }, { "id": 85628, "postDate": "2015-07-16T02:07:03.963Z", "content": "[quote=Mark Peng;85627]
\n\n[quote=Jing Lu;85549]
\n\nHi Mark,
\n\nThanks for sharing! Your approach is very interesting, but somehow also complex. Do you split data for layer 0 and layer 1, or just use the same training data? So, for each layer, you do a feature selection with RFE. Why do you choose 200 features for layer 0 and 150 features for layer 1? I was thinking feature selection between layer 0 and 1 may help eliminate some data leaking during training..
\n\nThanks,\nJing
\n\n[/quote]
\n\nThanks for the comment!\nRegarding ensemble, I followed the stacked generalization process mentioned in @Triskelion's great post about ensemble learning. Quoted here:
\n\n\n \n\n\n
Let’s say you want to do 2-fold stacking:\n\nSplit the train set in 2 parts: train_a and train_b\nFit a first-stage model on train_a and create predictions for train_b\nFit the same model on train_b and create predictions for train_a\nFinally fit the model on the entire train set and create predictions for the test set.\nNow train a second-stage stacker model on the probabilities from the first-stage model(s).\n\nA stacker model gets more information on the problem space by using the first-stage predictions as features, than if it was trained in isolation.\nI used the same 2-fold CV process to generate predictions as meta-features for layer 1. Feature selection is also done by 3-fold CV (repeats 3 times) using caret package in R. So 200 raw features for layer 0 and 150 meta-features for layer 1 were chosen by 3-fold CV with RFE. Therefore, the chance to data leaking during training should be very small. Similar ensemble process is mentioned in Otto competition winner's interview as well.
\n\nThanks,
\n\nMark
\n\n[/quote]
", "rawMarkdown": "[quote=Mark Peng;85627]\r\n\r\n[quote=Jing Lu;85549]\r\n\r\nHi Mark,\r\n\r\nThanks for sharing! Your approach is very interesting, but somehow also complex. Do you split data for layer 0 and layer 1, or just use the same training data? So, for each layer, you do a feature selection with RFE. Why do you choose 200 features for layer 0 and 150 features for layer 1? I was thinking feature selection between layer 0 and 1 may help eliminate some data leaking during training..\r\n\r\n\r\nThanks,\r\nJing\r\n\r\n[/quote]\r\n\r\n\r\nThanks for the comment!\r\nRegarding ensemble, I followed the stacked generalization process mentioned in @Triskelion's [great post][1] about ensemble learning. Quoted here:\r\n\r\n> \r\n\r\n Let’s say you want to do 2-fold stacking:\r\n\r\n Split the train set in 2 parts: train_a and train_b\r\n Fit a first-stage model on train_a and create predictions for train_b\r\n Fit the same model on train_b and create predictions for train_a\r\n Finally fit the model on the entire train set and create predictions for the test set.\r\n Now train a second-stage stacker model on the probabilities from the first-stage model(s).\r\n\r\n A stacker model gets more information on the problem space by using the first-stage predictions as features, than if it was trained in isolation.\r\n\r\nI used the same 2-fold CV process to generate predictions as meta-features for layer 1. Feature selection is also done by 3-fold CV (repeats 3 times) using *caret* package in R. So 200 **raw features** for layer 0 and 150 **meta-features** for layer 1 were chosen by 3-fold CV with RFE. Therefore, the chance to data leaking during training should be very small. Similar ensemble process is mentioned in [Otto competition winner's interview][2] as well.\r\n\r\nThanks,\r\n\r\nMark\r\n\r\n\r\n [1]: http://mlwave.com/kaggle-ensembling-guide/\r\n [2]: http://blog.kaggle.com/2015/06/09/otto-product-classification-winners-interview-2nd-place-alexander-guschin/\r\n\r\n[/quote]\r\n" }, { "id": 85485, "postDate": "2015-07-15T03:54:25.547Z", "content": "[quote=Khanna;84422]
\n\nNice!\nAlas! I had a similar brainstorming , but kudos you covered a vast array of intuitive cases.\nI couldnt get any more than 0.56+ on the leaderboard with my similar heuristics.\nGreat job. At once when I read on other kaggle scripts, I almost lost faith in such feature engineering.
\n\n[/quote]
\n\nThanks Khanna!\nI had several years of experience in information extraction and search engine applications, so I knew that those heuristics should work! \nGloVe deep learning vectors are also very helpful, in addition to pattern-based features.\nYou can download all my features from the document to see if you can find better ways to use them than mine. :)
\n\nThanks,
\n\nMark
", "rawMarkdown": "[quote=Khanna;84422]\r\n\r\nNice!\r\nAlas! I had a similar brainstorming , but kudos you covered a vast array of intuitive cases.\r\nI couldnt get any more than 0.56+ on the leaderboard with my similar heuristics.\r\nGreat job. At once when I read on other kaggle scripts, I almost lost faith in such feature engineering.\r\n\r\n[/quote]\r\n\r\nThanks Khanna!\r\nI had several years of experience in information extraction and search engine applications, so I knew that those heuristics should work! \r\nGloVe deep learning vectors are also very helpful, in addition to pattern-based features.\r\nYou can download all my features from the document to see if you can find better ways to use them than mine. :)\r\n\r\nThanks,\r\n\r\nMark\r\n\r\n\r\n\r\n\r\n" }, { "id": 84422, "postDate": "2015-07-14T17:16:06.133Z", "content": "Nice!\nAlas! I had a similar brainstorming , but kudos you covered a vast array of intuitive cases.\nI couldnt get any more than 0.56+ on the leaderboard with my similar heuristics.\nGreat job. At once when I read on other kaggle scripts, I almost lost faith in such feature engineering.
", "rawMarkdown": "Nice!\r\nAlas! I had a similar brainstorming , but kudos you covered a vast array of intuitive cases.\r\nI couldnt get any more than 0.56+ on the leaderboard with my similar heuristics.\r\nGreat job. At once when I read on other kaggle scripts, I almost lost faith in such feature engineering." }, { "id": 84189, "postDate": "2015-07-12T04:33:56.027Z", "content": "[quote=Mark Nagelberg;84144]
\n\nAwesome! Very useful to see your features/models in such clear detail, and I didn't know about GloVe, RFE, or Boruta - all look like really useful tools. Also liked seeing your feature selection process as mine could use some serious work (i.e. I didn't really use a formal feature selection process at all :S). I also didn't know about using multiple "layers" when ensembling.
\n\nThanks Mark!
\n\n[/quote]
\n\nHi Mark,
\n\nI am glad to know that it is helpful to you. :)\nActually I spent more than 12 hours in the past several weekends for this competition.\nMost of the work were done for feature engineering!\nIt took me two days in full time to compete this write up, because I really would like share my ideas with you!
\n\nI also learned a lot from the posts by top kagglers in previous competitions regarding ensemble learning too.
\n\nThanks,
\n\nMark
", "rawMarkdown": "[quote=Mark Nagelberg;84144]\r\n\r\nAwesome! Very useful to see your features/models in such clear detail, and I didn't know about GloVe, RFE, or Boruta - all look like really useful tools. Also liked seeing your feature selection process as mine could use some serious work (i.e. I didn't really use a formal feature selection process at all :S). I also didn't know about using multiple \"layers\" when ensembling. \r\n\r\nThanks Mark!\r\n\r\n[/quote]\r\n\r\nHi Mark,\r\n\r\nI am glad to know that it is helpful to you. :)\r\nActually I spent more than 12 hours in the past several weekends for this competition.\r\nMost of the work were done for feature engineering!\r\nIt took me two days in full time to compete this write up, because I really would like share my ideas with you!\r\n\r\nI also learned a lot from the posts by top kagglers in previous competitions regarding ensemble learning too.\r\n\r\nThanks,\r\n\r\nMark\r\n\r\n\r\n" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T20:59:17Z", "competition_id": 4407, "competition_slug": "crowdflower-search-relevance", "team_id": 181569, "solution_url": "/competitions/crowdflower-search-relevance/writeups/ydm-feature-engineering-and-ensemble-architecture-" }