I am from a team which removed by the Admin, I must make a apology to everyone at first.
\n\nI will share my story and make a summary in this post.
\n\nI am from a little group of guys interesting in machine learning and deep learning,we take part in this competition for a simple and a little crazy idea: Why can not we built a framework to generate and find features and train the model automatically on large dataset?
\nSo we start the competition in our group with each other as a game (and a little bet on the result) it is a very good dataset, it is large, we need to predict ahead two week.
In the first week, my score is very pool, 600+ on LB, after some very very hard work, I try to generated 1000+ features and can evaluate on my 32G RAM computer, find it very effective, in last two weeks of the before the end of the competition, I can get 0.430 score on LB, I found the guys in our group can manage large features can get 0.428 score and even better 0.427 in a single model (perhaps top 3-4 in private LB).
\n\nFrom that point, We begin analyze the rule of the Kaggle, we found we must team up, but we found the total submissions is exceed the max times allowed. If we team up into 2 or 3 team, we still be treated as multi accounts holder.
\n\nSo, I refuse the team up request from others, we don’t want affect them, and I realize It is unfair to keep us all at the top of LB, so I select the model I think will be drop to 200+ in last minutes, but some of us refuse to do so, because they think they get the score independently, they want to see which position will be on private LB. Perhaps, they are wrong, although we did not exchange codes and “features” even now.
\n\nAfter the final result came out, We discussed the rule of the kaggle carefully, and refer to other competitions, we decide not contact compliance team, because we checked the submission files of us, I find the correlation is so high(0.95, 0.99+), we can not prove we do the work independently. After I try to train a deep learning model use my feature vectors, I find the correlation is very very low. We realize we have no hope. Let alone we submit on some computers with same IP even in the same browser……………such a mess! We must obey the rule!
\n\nI train two models:
\nthe first one: use 3-8 weeks data to train week 9 and predict 10 and 11.
\nthe second one: use 3-7 weeks to train 8-9 and predict 10 and 11.
We fount that if we generate more features for selection, the correlation of results increase, even we use different code use different data (model 1 and model 2).
\n\nAnd I tried to test it on other competition which has been finished in these weeks such as Santander customer satisfaction (3000+ features automatically), we find we can get high score and the submission is highly correlated…….I guess, perhaps,our idea and algorithm grasped the some hidden magic of the dataset automatically as deep learning… If the features are good , they must be generated and selected by the model.
\n\nIt is really a very hard work, 40+ days and nights…. thousands lines of codes, exciting time and sorry time…
\n\nApologies to all of kagglers and thanks for the kaggle.com and bimbo provide such a good competition, I learned a lot from this competition.
\n\nCongratulations to the winners!
", "messages": [ { "id": "133312", "postDate": "09/01/2016 13:22:46", "content": "I am from a team which removed by the Admin, I must make a apology to everyone at first.
\n\nI will share my story and make a summary in this post.
\n\nI am from a little group of guys interesting in machine learning and deep learning,we take part in this competition for a simple and a little crazy idea: Why can not we built a framework to generate and find features and train the model automatically on large dataset?
\nSo we start the competition in our group with each other as a game (and a little bet on the result) it is a very good dataset, it is large, we need to predict ahead two week.
In the first week, my score is very pool, 600+ on LB, after some very very hard work, I try to generated 1000+ features and can evaluate on my 32G RAM computer, find it very effective, in last two weeks of the before the end of the competition, I can get 0.430 score on LB, I found the guys in our group can manage large features can get 0.428 score and even better 0.427 in a single model (perhaps top 3-4 in private LB).
\n\nFrom that point, We begin analyze the rule of the Kaggle, we found we must team up, but we found the total submissions is exceed the max times allowed. If we team up into 2 or 3 team, we still be treated as multi accounts holder.
\n\nSo, I refuse the team up request from others, we don’t want affect them, and I realize It is unfair to keep us all at the top of LB, so I select the model I think will be drop to 200+ in last minutes, but some of us refuse to do so, because they think they get the score independently, they want to see which position will be on private LB. Perhaps, they are wrong, although we did not exchange codes and “features” even now.
\n\nAfter the final result came out, We discussed the rule of the kaggle carefully, and refer to other competitions, we decide not contact compliance team, because we checked the submission files of us, I find the correlation is so high(0.95, 0.99+), we can not prove we do the work independently. After I try to train a deep learning model use my feature vectors, I find the correlation is very very low. We realize we have no hope. Let alone we submit on some computers with same IP even in the same browser……………such a mess! We must obey the rule!
\n\nI train two models:
\nthe first one: use 3-8 weeks data to train week 9 and predict 10 and 11.
\nthe second one: use 3-7 weeks to train 8-9 and predict 10 and 11.
We fount that if we generate more features for selection, the correlation of results increase, even we use different code use different data (model 1 and model 2).
\n\nAnd I tried to test it on other competition which has been finished in these weeks such as Santander customer satisfaction (3000+ features automatically), we find we can get high score and the submission is highly correlated…….I guess, perhaps,our idea and algorithm grasped the some hidden magic of the dataset automatically as deep learning… If the features are good , they must be generated and selected by the model.
\n\nIt is really a very hard work, 40+ days and nights…. thousands lines of codes, exciting time and sorry time…
\n\nApologies to all of kagglers and thanks for the kaggle.com and bimbo provide such a good competition, I learned a lot from this competition.
\n\nCongratulations to the winners!
", "rawMarkdown": "I am from a team which removed by the Admin, I must make a apology to everyone at first.\r\n\r\nI will share my story and make a summary in this post.\r\n\r\nI am from a little group of guys interesting in machine learning and deep learning,we take part in this competition for a simple and a little crazy idea: Why can not we built a framework to generate and find features and train the model automatically on large dataset?Thank you for the post! So the three top 10 teams are all from your group? \nAnd one thing i don't understand, why submit so many times? Did you guys use public lb for 2nd week validation? Otherwise I don't think public lb feedback is very helpful.
", "rawMarkdown": "Thank you for the post! So the three top 10 teams are all from your group? \r\nAnd one thing i don't understand, why submit so many times? Did you guys use public lb for 2nd week validation? Otherwise I don't think public lb feedback is very helpful.", "votes": null }, { "id": "133328", "postDate": "09/01/2016 14:42:00", "content": "By the way, may i ask what deep learning model you are using?
", "rawMarkdown": "By the way, may i ask what deep learning model you are using?", "votes": null }, { "id": "133330", "postDate": "09/01/2016 15:00:07", "content": "[quote=rcarson;133322]
\n\nThank you for the post! So the three top 10 teams are all from your group? \nAnd one thing i don't understand, why submit so many times? Did you guys use public lb for 2nd week validation? Otherwise I don't think public lb feedback is very helpful.
\n\n[/quote]\nAs I know, not all of three teams are removed are from us. I did not consider 2nd week prediction until last week…. I try to tune our algorithms and codes on public LB , we have no experiences ,we made so many submissions unnecessary…we submit only for update our local LB everyday(Yes ,we DO update the LB of us every night 0:00 AM …… if some one have better score ,we try to exceed him next day … by the way for a long time ,your team is far ahead of us ,I can not image get close to you ……0.429x) if If someone with experiences in our team ,perhaps we can control the number submissions under 200,so we can team up… tears… ( I guess someone want to know ,a regression problem with 6 millions rows can befit from more submission times ,I think the answer is NO.) After I realize I should consider 11 week,4 days left only ,I tried to generate a lot of new features(1000+) including ones with prediction info ,and finished 4hours before the deadline( and I add more RAM to my machine,a little crazy. I found the week10 data can be used to validate week 11. My two models almost have same ability to predict week 11.
", "rawMarkdown": "[quote=rcarson;133322]\r\n\r\nThank you for the post! So the three top 10 teams are all from your group? \r\nAnd one thing i don't understand, why submit so many times? Did you guys use public lb for 2nd week validation? Otherwise I don't think public lb feedback is very helpful.\r\n\r\n[/quote]\r\nAs I know, not all of three teams are removed are from us. I did not consider 2nd week prediction until last week…. I try to tune our algorithms and codes on public LB , we have no experiences ,we made so many submissions unnecessary…we submit only for update our local LB everyday(Yes ,we DO update the LB of us every night 0:00 AM …… if some one have better score ,we try to exceed him next day … by the way for a long time ,your team is far ahead of us ,I can not image get close to you ……0.429x) if If someone with experiences in our team ,perhaps we can control the number submissions under 200,so we can team up… tears… ( I guess someone want to know ,a regression problem with 6 millions rows can befit from more submission times ,I think the answer is NO.) After I realize I should consider 11 week,4 days left only ,I tried to generate a lot of new features(1000+) including ones with prediction info ,and finished 4hours before the deadline( and I add more RAM to my machine,a little crazy. I found the week10 data can be used to validate week 11. My two models almost have same ability to predict week 11.", "votes": null }, { "id": "133333", "postDate": "09/01/2016 15:11:46", "content": "[quote=rcarson;133328]
\n\nBy the way, may i ask what deep learning model you are using?
\n\n[/quote]
\n\nI did not submit my deeplearning model,because it is not that good,the public score is 0.436x also..
\n\nI design the model a little funny,most of them are 1*1 ConvLayer like this ：
\n1 conv1_0 1x1x1024
\n2 conv1_0 1x1x1023
\n3 conv1_0 1x1x1022
\n4 1x1x3069
1024 is the feature number
I want this model can add non-linear to the features,I hope it can grasp ratio,time series info in the data different filter size for lags
\n\nbut I did not go further ,because other guys score was 0.434 also,and the deep learning model need more hardware support....
", "rawMarkdown": "[quote=rcarson;133328]\r\n\r\nBy the way, may i ask what deep learning model you are using?\r\n\r\n[/quote]\r\n\r\nI did not submit my deeplearning model,because it is not that good,the public score is 0.436x also..\r\n\r\nI design the model a little funny,most of them are 1*1 ConvLayer like this ：[quote=rcarson;133322]
\n\nThank you for the post! So the three top 10 teams are all from your group? \nAnd one thing i don't understand, why submit so many times? Did you guys use public lb for 2nd week validation? Otherwise I don't think public lb feedback is very helpful.
\n\n[/quote]
\n\nWe do use the week 10 for 2-week-ahead validation (thus by withholding the week 9). We did some experiments. There does exist some discrepancies between one-week-ahead and two-week-ahead if you are ignoring the effect. The difference is estimated to be about 0.002 (if I remember correctly). That probably explains why half of the teams in top10 comes from rank10-30.
\n\nI am also wondering what deep learning methods they are using?
\n\nJianbo
", "rawMarkdown": "[quote=rcarson;133322]\r\n\r\nThank you for the post! So the three top 10 teams are all from your group? \r\nAnd one thing i don't understand, why submit so many times? Did you guys use public lb for 2nd week validation? Otherwise I don't think public lb feedback is very helpful.\r\n\r\n[/quote]\r\n\r\nWe do use the week 10 for 2-week-ahead validation (thus by withholding the week 9). We did some experiments. There does exist some discrepancies between one-week-ahead and two-week-ahead if you are ignoring the effect. The difference is estimated to be about 0.002 (if I remember correctly). That probably explains why half of the teams in top10 comes from rank10-30.\r\n\r\nI am also wondering what deep learning methods they are using?\r\n\r\nJianbo", "votes": null }, { "id": "133337", "postDate": "09/01/2016 15:29:45", "content": "[quote=Ellioz;133333]
\n\n[quote=rcarson;133328]
\n\nBy the way, may i ask what deep learning model you are using?
\n\n[/quote]
\n\nI did not submit my deeplearning model,because it is not that good,the public score is 0.436x also..
\n\nI design the model a little funny,most of them are 1*1 ConvLayer like this ：\n1 conv1_0 1x1x1024\n2 conv1_0 1x1x1023\n3 conv1_0 1x1x1022\n4 1x1x3069
\n\n1024 is the feature number\n[/quote]
\n\nFirst of all, 0.436 public lb is not bad at all. It is actually better than any of our single models.
\n\nSo you are using 1 by 1 convolution? Wow, that's interesting! so all the feature maps are 1x1 in x-y dimension? If that's the case it seems it is equivalent to a MLP with several dense layers. What activation function do you use? I didn't know what a proper activation function fits regression problem, especially for the last layer.
", "rawMarkdown": "[quote=Ellioz;133333]\r\n\r\n[quote=rcarson;133328]\r\n\r\nBy the way, may i ask what deep learning model you are using?\r\n\r\n[/quote]\r\n\r\nI did not submit my deeplearning model,because it is not that good,the public score is 0.436x also..\r\n\r\nI design the model a little funny,most of them are 1*1 ConvLayer like this ：\r\n1 conv1_0 1x1x1024\r\n2 conv1_0 1x1x1023\r\n3 conv1_0 1x1x1022\r\n4 1x1x3069\r\n\r\n1024 is the feature number\r\n[/quote]\r\n\r\nFirst of all, 0.436 public lb is not bad at all. It is actually better than any of our single models.\r\n\r\nSo you are using 1 by 1 convolution? Wow, that's interesting! so all the feature maps are 1x1 in x-y dimension? If that's the case it seems it is equivalent to a MLP with several dense layers. What activation function do you use? I didn't know what a proper activation function fits regression problem, especially for the last layer.", "votes": null }, { "id": "133346", "postDate": "09/01/2016 16:01:29", "content": "[quote=rcarson;133337]
\n\n[quote=Ellioz;133333]
\n\n[quote=rcarson;133328]
\n\nBy the way, may i ask what deep learning model you are using?
\n\n[/quote]
\n\nI did not submit my deeplearning model,because it is not that good,the public score is 0.436x also..
\n\nI design the model a little funny,most of them are 1*1 ConvLayer like this ：
\n1 conv1_0 1x1x1024
\n2 conv1_0 1x1x1023
\n3 conv1_0 1x1x1022
\n4 1x1x3069
1024 is the feature number
\n[/quote]
First of all, 0.436 public lb is not bad at all. It is actually better than any of our single models.
\n\nSo you are using 1 by 1 convolution? Wow, that's interesting! so all the feature maps are 1x1 in x-y dimension? If that's the case it seems it is equivalent to a MLP with several dense layers. What activation function do you use? I didn't know what a proper activation function fits regression problem, especially for the last layer.
\n\n[/quote]
\n\nI check my code,my submission is almost 0.437.I tried a lot of structures in three days,the simple one is like the following:
\nl_0=Conv2DLayer(l,name='conv1_0',num_filters=1,filter_size=(1,2))
\nl_1=Conv2DLayer(l,name='conv1_1',num_filters=1,filter_size=(1,3))
\nl_2=Conv2DLayer(l,name='conv1_2',num_filters=1,filter_size=(1,4))
\nl = ConcatLayer([l_0,l_1,l_2],axis=3)
\nl=Conv2DLayer(l,name='conv2',num_filters=4,filter_size=(1,1))
\nl=DenseLayer(l,num_units=128,name='dense1')
\nl=DenseLayer(l,num_units=48,name='dense2')
\nl_out=DenseLayer(l,num_units=1,nonlinearity=None,name='output')
\nif I want to train the data with two week,just modify the num_units to 2.
\nI read a lot of papers before the competition,and they invite me to join,so I want to have a try… \nThe activation function is Relu…
I think the model is different from MLP,some papers related to Imagenet discussed 1*1 filter.
\n\nAs to regression problem,some one said that: In the fully connected layers s-shaped relu units are used. I did not try.
\n\nAnd the model suffer from overfitting, so,if I want to use it ,a lot of things need to do.
", "rawMarkdown": "[quote=rcarson;133337]\r\n\r\n[quote=Ellioz;133333]\r\n\r\n[quote=rcarson;133328]\r\n\r\nBy the way, may i ask what deep learning model you are using?\r\n\r\n[/quote]\r\n\r\nI did not submit my deeplearning model,because it is not that good,the public score is 0.436x also..\r\n\r\nI design the model a little funny,most of them are 1*1 ConvLayer like this ：Thank you for the reply!
\n\nYeah, I tried SRELU, except the last layer. I am wondering what is the dimension of your input feature?
", "rawMarkdown": "Thank you for the reply!\r\n\r\nYeah, I tried SRELU, except the last layer. I am wondering what is the dimension of your input feature?", "votes": null }, { "id": "133422", "postDate": "09/02/2016 04:04:03", "content": "Can you tell us more about the feature generation? How do you generate so many features?
", "rawMarkdown": "Can you tell us more about the feature generation? How do you generate so many features?", "votes": null }, { "id": "134026", "postDate": "09/04/2016 18:45:50", "content": "sorry for late replay!\nI use 100 also features to train the model, at that stage, I just used only a part of features due to memory limit. But I believe you can be solved easily if you have more features and more time.\nIf you will try the solution and have good result, please let me know. I hope I will have time to try other features with more layers and other activation functions.
", "rawMarkdown": "sorry for late replay!\r\nI use 100 also features to train the model, at that stage, I just used only a part of features due to memory limit. But I believe you can be solved easily if you have more features and more time.\r\nIf you will try the solution and have good result, please let me know. I hope I will have time to try other features with more layers and other activation functions.", "votes": null }, { "id": "134027", "postDate": "09/04/2016 18:46:27", "content": "[quote=airxiechao;133422]
\n\nCan you tell us more about the feature generation? How do you generate so many features?
\n\n[/quote]
\n\nAs to features, the main ideas is:
\n1, We suppose have very little knowledge about he data and try to list all possible relation. I did not have detail investigation on the BIMBO dataset and did not read data analysis post in forum until now. As you know, if you process image in deep learning, you don't need know much information about the image.
\n2, I designed a lot of filters to select the useful features guide by the loss function.
\n3, It’s large dataset, if we have very large number of features, we must subsample it and make sure we did not loss important information. Some of us use only 1.66M samples train the model and get decent score. (0.433 also,he told me he want to find some theory support mathematically or machine learning theory).
\nHope it will be helpful .
Thank you @Ellioz, it's a good idea
", "rawMarkdown": "Thank you @Ellioz, it's a good idea", "votes": null } ], "comments": [ { "id": 133322, "author_name": "jiweiliu", "author_url": "", "post_date": "09/01/2016 13:59:31", "content": "Thank you for the post! So the three top 10 teams are all from your group? \nAnd one thing i don't understand, why submit so many times? Did you guys use public lb for 2nd week validation? Otherwise I don't think public lb feedback is very helpful.
", "votes": null, "replies": [] }, { "id": 133328, "author_name": "jiweiliu", "author_url": "", "post_date": "09/01/2016 14:42:00", "content": "By the way, may i ask what deep learning model you are using?
", "votes": null, "replies": [] }, { "id": 133330, "author_name": "ellioz", "author_url": "", "post_date": "09/01/2016 15:00:07", "content": "[quote=rcarson;133322]
\n\nThank you for the post! So the three top 10 teams are all from your group? \nAnd one thing i don't understand, why submit so many times? Did you guys use public lb for 2nd week validation? Otherwise I don't think public lb feedback is very helpful.
\n\n[/quote]\nAs I know, not all of three teams are removed are from us. I did not consider 2nd week prediction until last week…. I try to tune our algorithms and codes on public LB , we have no experiences ,we made so many submissions unnecessary…we submit only for update our local LB everyday(Yes ,we DO update the LB of us every night 0:00 AM …… if some one have better score ,we try to exceed him next day … by the way for a long time ,your team is far ahead of us ,I can not image get close to you ……0.429x) if If someone with experiences in our team ,perhaps we can control the number submissions under 200,so we can team up… tears… ( I guess someone want to know ,a regression problem with 6 millions rows can befit from more submission times ,I think the answer is NO.) After I realize I should consider 11 week,4 days left only ,I tried to generate a lot of new features(1000+) including ones with prediction info ,and finished 4hours before the deadline( and I add more RAM to my machine,a little crazy. I found the week10 data can be used to validate week 11. My two models almost have same ability to predict week 11.
", "votes": null, "replies": [] }, { "id": 133333, "author_name": "ellioz", "author_url": "", "post_date": "09/01/2016 15:11:46", "content": "[quote=rcarson;133328]
\n\nBy the way, may i ask what deep learning model you are using?
\n\n[/quote]
\n\nI did not submit my deeplearning model,because it is not that good,the public score is 0.436x also..
\n\nI design the model a little funny,most of them are 1*1 ConvLayer like this ：
\n1 conv1_0 1x1x1024
\n2 conv1_0 1x1x1023
\n3 conv1_0 1x1x1022
\n4 1x1x3069
1024 is the feature number
I want this model can add non-linear to the features,I hope it can grasp ratio,time series info in the data different filter size for lags
\n\nbut I did not go further ,because other guys score was 0.434 also,and the deep learning model need more hardware support....
", "votes": null, "replies": [] }, { "id": 133334, "author_name": "amazingbob", "author_url": "", "post_date": "09/01/2016 15:12:09", "content": "[quote=rcarson;133322]
\n\nThank you for the post! So the three top 10 teams are all from your group? \nAnd one thing i don't understand, why submit so many times? Did you guys use public lb for 2nd week validation? Otherwise I don't think public lb feedback is very helpful.
\n\n[/quote]
\n\nWe do use the week 10 for 2-week-ahead validation (thus by withholding the week 9). We did some experiments. There does exist some discrepancies between one-week-ahead and two-week-ahead if you are ignoring the effect. The difference is estimated to be about 0.002 (if I remember correctly). That probably explains why half of the teams in top10 comes from rank10-30.
\n\nI am also wondering what deep learning methods they are using?
\n\nJianbo
", "votes": null, "replies": [] }, { "id": 133337, "author_name": "jiweiliu", "author_url": "", "post_date": "09/01/2016 15:29:45", "content": "[quote=Ellioz;133333]
\n\n[quote=rcarson;133328]
\n\nBy the way, may i ask what deep learning model you are using?
\n\n[/quote]
\n\nI did not submit my deeplearning model,because it is not that good,the public score is 0.436x also..
\n\nI design the model a little funny,most of them are 1*1 ConvLayer like this ：\n1 conv1_0 1x1x1024\n2 conv1_0 1x1x1023\n3 conv1_0 1x1x1022\n4 1x1x3069
\n\n1024 is the feature number\n[/quote]
\n\nFirst of all, 0.436 public lb is not bad at all. It is actually better than any of our single models.
\n\nSo you are using 1 by 1 convolution? Wow, that's interesting! so all the feature maps are 1x1 in x-y dimension? If that's the case it seems it is equivalent to a MLP with several dense layers. What activation function do you use? I didn't know what a proper activation function fits regression problem, especially for the last layer.
", "votes": null, "replies": [] }, { "id": 133346, "author_name": "ellioz", "author_url": "", "post_date": "09/01/2016 16:01:29", "content": "[quote=rcarson;133337]
\n\n[quote=Ellioz;133333]
\n\n[quote=rcarson;133328]
\n\nBy the way, may i ask what deep learning model you are using?
\n\n[/quote]
\n\nI did not submit my deeplearning model,because it is not that good,the public score is 0.436x also..
\n\nI design the model a little funny,most of them are 1*1 ConvLayer like this ：
\n1 conv1_0 1x1x1024
\n2 conv1_0 1x1x1023
\n3 conv1_0 1x1x1022
\n4 1x1x3069
1024 is the feature number
\n[/quote]
First of all, 0.436 public lb is not bad at all. It is actually better than any of our single models.
\n\nSo you are using 1 by 1 convolution? Wow, that's interesting! so all the feature maps are 1x1 in x-y dimension? If that's the case it seems it is equivalent to a MLP with several dense layers. What activation function do you use? I didn't know what a proper activation function fits regression problem, especially for the last layer.
\n\n[/quote]
\n\nI check my code,my submission is almost 0.437.I tried a lot of structures in three days,the simple one is like the following:
\nl_0=Conv2DLayer(l,name='conv1_0',num_filters=1,filter_size=(1,2))
\nl_1=Conv2DLayer(l,name='conv1_1',num_filters=1,filter_size=(1,3))
\nl_2=Conv2DLayer(l,name='conv1_2',num_filters=1,filter_size=(1,4))
\nl = ConcatLayer([l_0,l_1,l_2],axis=3)
\nl=Conv2DLayer(l,name='conv2',num_filters=4,filter_size=(1,1))
\nl=DenseLayer(l,num_units=128,name='dense1')
\nl=DenseLayer(l,num_units=48,name='dense2')
\nl_out=DenseLayer(l,num_units=1,nonlinearity=None,name='output')
\nif I want to train the data with two week,just modify the num_units to 2.
\nI read a lot of papers before the competition,and they invite me to join,so I want to have a try… \nThe activation function is Relu…
I think the model is different from MLP,some papers related to Imagenet discussed 1*1 filter.
\n\nAs to regression problem,some one said that: In the fully connected layers s-shaped relu units are used. I did not try.
\n\nAnd the model suffer from overfitting, so,if I want to use it ,a lot of things need to do.
", "votes": null, "replies": [] }, { "id": 133352, "author_name": "jiweiliu", "author_url": "", "post_date": "09/01/2016 16:35:27", "content": "Thank you for the reply!
\n\nYeah, I tried SRELU, except the last layer. I am wondering what is the dimension of your input feature?
", "votes": null, "replies": [] }, { "id": 133422, "author_name": "airxiechao", "author_url": "", "post_date": "09/02/2016 04:04:03", "content": "Can you tell us more about the feature generation? How do you generate so many features?
", "votes": null, "replies": [] }, { "id": 134026, "author_name": "ellioz", "author_url": "", "post_date": "09/04/2016 18:45:50", "content": "sorry for late replay!\nI use 100 also features to train the model, at that stage, I just used only a part of features due to memory limit. But I believe you can be solved easily if you have more features and more time.\nIf you will try the solution and have good result, please let me know. I hope I will have time to try other features with more layers and other activation functions.
", "votes": null, "replies": [] }, { "id": 134027, "author_name": "ellioz", "author_url": "", "post_date": "09/04/2016 18:46:27", "content": "[quote=airxiechao;133422]
\n\nCan you tell us more about the feature generation? How do you generate so many features?
\n\n[/quote]
\n\nAs to features, the main ideas is:
\n1, We suppose have very little knowledge about he data and try to list all possible relation. I did not have detail investigation on the BIMBO dataset and did not read data analysis post in forum until now. As you know, if you process image in deep learning, you don't need know much information about the image.
\n2, I designed a lot of filters to select the useful features guide by the loss function.
\n3, It’s large dataset, if we have very large number of features, we must subsample it and make sure we did not loss important information. Some of us use only 1.66M samples train the model and get decent score. (0.433 also,he told me he want to find some theory support mathematically or machine learning theory).
\nHope it will be helpful .
Thank you @Ellioz, it's a good idea
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "133312": "I am from a team which removed by the Admin, I must make a apology to everyone at first.\r\n\r\nI will share my story and make a summary in this post.\r\n\r\nI am from a little group of guys interesting in machine learning and deep learning,we take part in this competition for a simple and a little crazy idea: Why can not we built a framework to generate and find features and train the model automatically on large dataset?