i have been testing neural network for this problem. however, i think in each grid, the label category is too large relative to the sample size. so i can only achieve around 0.3 range. anyone has any success on neural network? thank you!
", "messages": [ { "id": "126163", "postDate": "07/06/2016 22:48:34", "content": "i have been testing neural network for this problem. however, i think in each grid, the label category is too large relative to the sample size. so i can only achieve around 0.3 range. anyone has any success on neural network? thank you!
", "rawMarkdown": "i have been testing neural network for this problem. however, i think in each grid, the label category is too large relative to the sample size. so i can only achieve around 0.3 range. anyone has any success on neural network? thank you!", "votes": null }, { "id": "126184", "postDate": "07/07/2016 00:44:57", "content": "I briefly tested neural networks. I trained a 5-layer fully-connected NN but only obtained a score 0.18 ...\nBased on what I observed from my several trials, I found that using the first 25000000 data for training and the rest for validation, I could not make training loss lower than validation loss ... This indicates that deep neural networks if large enough should have the potential to achieve the best score on this dataset.
", "rawMarkdown": "I briefly tested neural networks. I trained a 5-layer fully-connected NN but only obtained a score 0.18 ...\r\nBased on what I observed from my several trials, I found that using the first 25000000 data for training and the rest for validation, I could not make training loss lower than validation loss ... This indicates that deep neural networks if large enough should have the potential to achieve the best score on this dataset.", "votes": null }, { "id": "126201", "postDate": "07/07/2016 01:37:36", "content": "My NN model using Keras got score 0.58234 on LB.
\n\nit is a straightforward model, 20x40 grid, 4-layer, 50% dropout and following simple features wih sklearn.preprocessing.standardscaler:
\n\nx\ny\naccuracy\nhour\nweekday\nday_of_month\nIt takes 2 days to get result with titan-X.
", "rawMarkdown": "My NN model using Keras got score 0.58234 on LB.\r\n\r\nit is a straightforward model, 20x40 grid, 4-layer, 50% dropout and following simple features wih sklearn.preprocessing.standardscaler:\r\n\r\n x\r\n y\r\n accuracy\r\n hour\r\n weekday\r\n day_of_month\r\n\r\nIt takes 2 days to get result with titan-X.", "votes": null }, { "id": "126203", "postDate": "07/07/2016 01:40:33", "content": "I tried NN at very early stage of the competition, but abandoned it because it was way too slow on my laptop, compared to other methods like random forest, extra trees, and of course knn. The score from my NN model was about 0.57, which is an okay score at that time. The network structure and parameters were arbitrarily pick. If i had time, I would probably fine tune the parameters and include NN models into the ensemble model.
", "rawMarkdown": "I tried NN at very early stage of the competition, but abandoned it because it was way too slow on my laptop, compared to other methods like random forest, extra trees, and of course knn. The score from my NN model was about 0.57, which is an okay score at that time. The network structure and parameters were arbitrarily pick. If i had time, I would probably fine tune the parameters and include NN models into the ensemble model.", "votes": null }, { "id": "126204", "postDate": "07/07/2016 01:55:01", "content": "I'm a little surprised that you all used grid-based approach. I acutually trained on the whole training set (excluding validation set) only one NN model, as I believe NNs are the few strong models that could scale well with such an enormous amount of data and classes. May be I went the wrong direction ...
", "rawMarkdown": "I'm a little surprised that you all used grid-based approach. I acutually trained on the whole training set (excluding validation set) only one NN model, as I believe NNs are the few strong models that could scale well with such an enormous amount of data and classes. May be I went the wrong direction ...", "votes": null }, { "id": "126205", "postDate": "07/07/2016 01:56:01", "content": "@ Guanshuo Xu thanks for sharing.
\n\n@cute do you have drop out after each layer? how many epochs do you run? if you do not mind, how many nodes at each layer? i have been trying hard on this and impressed by your result. also, congrats on a top 10 finish
\n\n@Leon do you mind sharing with me the structure of your NN? like layers and nodes at each grid? i am really curious to know how to get 0.57. thanks
", "rawMarkdown": "Guanshuo Xu thanks for sharing. \r\n\r\n@cute do you have drop out after each layer? how many epochs do you run? if you do not mind, how many nodes at each layer? i have been trying hard on this and impressed by your result. also, congrats on a top 10 finish\r\n\r\n@Leon do you mind sharing with me the structure of your NN? like layers and nodes at each grid? i am really curious to know how to get 0.57. thanks", "votes": null }, { "id": "126210", "postDate": "07/07/2016 02:11:10", "content": "@Shubin
\n\nIt has 2 hidden layers, first one with 128 neurons and second one with 512 neurons. Input features are simply "x", "y", "accuracy", "hour of day", and "day of week". The hidden layers have batch normalization and dropout.
\n\n0.57 was an estimation from my local CV score, which seems to be pretty reliable. It was so slow that I didn't try a full training and testing.
", "rawMarkdown": "Shubin\r\n\r\nIt has 2 hidden layers, first one with 128 neurons and second one with 512 neurons. Input features are simply \"x\", \"y\", \"accuracy\", \"hour of day\", and \"day of week\". The hidden layers have batch normalization and dropout.\r\n\r\n0.57 was an estimation from my local CV score, which seems to be pretty reliable. It was so slow that I didn't try a full training and testing.", "votes": null }, { "id": "126213", "postDate": "07/07/2016 02:31:23", "content": "@Shubin
\n\nThank you.\nYes, drop out after each layer, 90 nb_epoch, 650 hidden_layer(same nodes for each layer).
\n\nFollowing is a portion of the code.
\n\n model = Sequential()\n\n model.add(Dense(hidden_layer, input_shape=(dimensions,)))\n model.add(PReLU())\n model.add(BatchNormalization())\n model.add(Dropout(dropout))\n\n model.add(Dense(hidden_layer))\n model.add(PReLU())\n model.add(BatchNormalization())\n model.add(Dropout(dropout))\n\n model.add(Dense(hidden_layer))\n model.add(PReLU())\n model.add(BatchNormalization())\n model.add(Dropout(dropout))\n\n model.add(Dense(hidden_layer))\n model.add(PReLU())\n model.add(BatchNormalization())\n model.add(Dropout(dropout))\n\n model.add(Dense(nb_classes))\n model.add(Activation('softmax'))\n model.compile(loss='categorical_crossentropy', optimizer="adam")\n model.fit(X, y, nb_epoch=nb_epoch, batch_size=128)\n pred = model.predict_proba(X_test)\nI briefly tested neural networks. I trained a 5-layer fully-connected NN but only obtained a score 0.18 ...\nBased on what I observed from my several trials, I found that using the first 25000000 data for training and the rest for validation, I could not make training loss lower than validation loss ... This indicates that deep neural networks if large enough should have the potential to achieve the best score on this dataset.
", "votes": null, "replies": [] }, { "id": 126201, "author_name": "its7171", "author_url": "", "post_date": "07/07/2016 01:37:36", "content": "My NN model using Keras got score 0.58234 on LB.
\n\nit is a straightforward model, 20x40 grid, 4-layer, 50% dropout and following simple features wih sklearn.preprocessing.standardscaler:
\n\nx\ny\naccuracy\nhour\nweekday\nday_of_month\nIt takes 2 days to get result with titan-X.
", "votes": null, "replies": [] }, { "id": 126203, "author_name": "leonlu", "author_url": "", "post_date": "07/07/2016 01:40:33", "content": "I tried NN at very early stage of the competition, but abandoned it because it was way too slow on my laptop, compared to other methods like random forest, extra trees, and of course knn. The score from my NN model was about 0.57, which is an okay score at that time. The network structure and parameters were arbitrarily pick. If i had time, I would probably fine tune the parameters and include NN models into the ensemble model.
", "votes": null, "replies": [] }, { "id": 126204, "author_name": "wowfattie", "author_url": "", "post_date": "07/07/2016 01:55:01", "content": "I'm a little surprised that you all used grid-based approach. I acutually trained on the whole training set (excluding validation set) only one NN model, as I believe NNs are the few strong models that could scale well with such an enormous amount of data and classes. May be I went the wrong direction ...
", "votes": null, "replies": [] }, { "id": 126205, "author_name": "lishubin", "author_url": "", "post_date": "07/07/2016 01:56:01", "content": "@ Guanshuo Xu thanks for sharing.
\n\n@cute do you have drop out after each layer? how many epochs do you run? if you do not mind, how many nodes at each layer? i have been trying hard on this and impressed by your result. also, congrats on a top 10 finish
\n\n@Leon do you mind sharing with me the structure of your NN? like layers and nodes at each grid? i am really curious to know how to get 0.57. thanks
", "votes": null, "replies": [] }, { "id": 126210, "author_name": "leonlu", "author_url": "", "post_date": "07/07/2016 02:11:10", "content": "@Shubin
\n\nIt has 2 hidden layers, first one with 128 neurons and second one with 512 neurons. Input features are simply "x", "y", "accuracy", "hour of day", and "day of week". The hidden layers have batch normalization and dropout.
\n\n0.57 was an estimation from my local CV score, which seems to be pretty reliable. It was so slow that I didn't try a full training and testing.
", "votes": null, "replies": [] }, { "id": 126213, "author_name": "its7171", "author_url": "", "post_date": "07/07/2016 02:31:23", "content": "@Shubin
\n\nThank you.\nYes, drop out after each layer, 90 nb_epoch, 650 hidden_layer(same nodes for each layer).
\n\nFollowing is a portion of the code.
\n\n model = Sequential()\n\n model.add(Dense(hidden_layer, input_shape=(dimensions,)))\n model.add(PReLU())\n model.add(BatchNormalization())\n model.add(Dropout(dropout))\n\n model.add(Dense(hidden_layer))\n model.add(PReLU())\n model.add(BatchNormalization())\n model.add(Dropout(dropout))\n\n model.add(Dense(hidden_layer))\n model.add(PReLU())\n model.add(BatchNormalization())\n model.add(Dropout(dropout))\n\n model.add(Dense(hidden_layer))\n model.add(PReLU())\n model.add(BatchNormalization())\n model.add(Dropout(dropout))\n\n model.add(Dense(nb_classes))\n model.add(Activation('softmax'))\n model.compile(loss='categorical_crossentropy', optimizer="adam")\n model.fit(X, y, nb_epoch=nb_epoch, batch_size=128)\n pred = model.predict_proba(X_test)\n