Hi all,
\n\nis someone trying to use Keras or another NeuralNetwork framework for this challenge?
\n\nI just tried it and got 0.356...\n20*20 Grids and as Features: x,y,accuracy, hour, weekday, month, year
\n\nIt was a second try (first try was just x and y as features and 50*50 grids).
\n\nAt all, I would keep working on that - but it costs so much time to compute (20-40hours).\nSo my question, is someone else is trying something similar here? :)
\n\nOr is it in general better to use some RandomForest?
", "messages": [ { "id": "122611", "postDate": "06/05/2016 20:00:15", "content": "Hi all,
\n\nis someone trying to use Keras or another NeuralNetwork framework for this challenge?
\n\nI just tried it and got 0.356...\n20*20 Grids and as Features: x,y,accuracy, hour, weekday, month, year
\n\nIt was a second try (first try was just x and y as features and 50*50 grids).
\n\nAt all, I would keep working on that - but it costs so much time to compute (20-40hours).\nSo my question, is someone else is trying something similar here? :)
\n\nOr is it in general better to use some RandomForest?
", "rawMarkdown": "Hi all,\r\n\r\nis someone trying to use Keras or another NeuralNetwork framework for this challenge?\r\n\r\nI just tried it and got 0.356...\r\n20*20 Grids and as Features: x,y,accuracy, hour, weekday, month, year\r\n\r\nIt was a second try (first try was just x and y as features and 50*50 grids).\r\n\r\nAt all, I would keep working on that - but it costs so much time to compute (20-40hours).\r\nSo my question, is someone else is trying something similar here? :)\r\n\r\nOr is it in general better to use some RandomForest?", "votes": null }, { "id": "122620", "postDate": "06/05/2016 21:46:22", "content": "Is that on GPU or CPU? How many layers? What activation functions do you use? How did you determine the number of training rounds? Did you do any sort of CV, what was your local score?
", "rawMarkdown": "Is that on GPU or CPU? How many layers? What activation functions do you use? How did you determine the number of training rounds? Did you do any sort of CV, what was your local score?", "votes": null }, { "id": "122636", "postDate": "06/06/2016 01:29:38", "content": "I wouldn't attempt methods over and over using the full training and test data sets provided by the competition. As you said, it can take potentially multiple days to go through that process.
\n\nI've been doing my own local cross validation on different chunks of the data. Usually I'll take a 1% block of the training data set and train/test using that.
\n\nI haven't submitted a full document yet, just because the run time isn't worth it. I've only submitted on ~40% of the data. I figure I just multiply that score by 2.5 then I'll have a rough estimate of what I'd get if I used the full data set. Oddly, I was getting scores as high as ~.62 for my local cross validation, but it looks like my submission would only receive around a ~.55 on the leaderboards.
\n\nThe name of the game right now is iterations and not place on leaderboards.
", "rawMarkdown": "I wouldn't attempt methods over and over using the full training and test data sets provided by the competition. As you said, it can take potentially multiple days to go through that process.\r\n\r\nI've been doing my own local cross validation on different chunks of the data. Usually I'll take a 1% block of the training data set and train/test using that.\r\n\r\nI haven't submitted a full document yet, just because the run time isn't worth it. I've only submitted on ~40% of the data. I figure I just multiply that score by 2.5 then I'll have a rough estimate of what I'd get if I used the full data set. Oddly, I was getting scores as high as ~.62 for my local cross validation, but it looks like my submission would only receive around a ~.55 on the leaderboards.\r\n\r\nThe name of the game right now is iterations and not place on leaderboards.", "votes": null }, { "id": "122656", "postDate": "06/06/2016 06:20:07", "content": "MathandBikes do you split your train set by time for CV purposes?
", "rawMarkdown": "MathandBikes do you split your train set by time for CV purposes?", "votes": null }, { "id": "122672", "postDate": "06/06/2016 10:49:29", "content": "@MathandBikes,\nso you train and test just on a small set of data, optimizing your network, before you go on the full data set?\nI think, that's what I'll do next.
\n\n@DennisSakva,\non GPU (nvidia 730m, I just have a laptop.. ). I can add the exact configuration later - or maybe upload my chaotic code :D\nI'm actually not at the my own laptop. But I use prelu layers, categorical_crossentropie and adam...
", "rawMarkdown": "MathandBikes,\r\nso you train and test just on a small set of data, optimizing your network, before you go on the full data set?\r\nI think, that's what I'll do next.\r\n\r\n@DennisSakva,\r\non GPU (nvidia 730m, I just have a laptop.. ). I can add the exact configuration later - or maybe upload my chaotic code :D\r\nI'm actually not at the my own laptop. But I use prelu layers, categorical_crossentropie and adam...", "votes": null }, { "id": "122854", "postDate": "06/07/2016 19:17:34", "content": "[quote=AndreSe;122672]
\n\n@MathandBikes,\nso you train and test just on a small set of data, optimizing your network, before you go on the full data set?\nI think, that's what I'll do next.
\n\n@DennisSakva,\non GPU (nvidia 730m, I just have a laptop.. ). I can add the exact configuration later - or maybe upload my chaotic code :D\nI'm actually not at the my own laptop. But I use prelu layers, categorical_crossentropie and adam...
\n\n[/quote]
\n\n@AndreSe Those are some promising results! If you are willing to share your code on Kaggle, then I will run it on some TITAN Xs for you ;)
", "rawMarkdown": "[quote=AndreSe;122672]\r\n\r\n@MathandBikes,\r\nso you train and test just on a small set of data, optimizing your network, before you go on the full data set?\r\nI think, that's what I'll do next.\r\n\r\n@DennisSakva,\r\non GPU (nvidia 730m, I just have a laptop.. ). I can add the exact configuration later - or maybe upload my chaotic code :D\r\nI'm actually not at the my own laptop. But I use prelu layers, categorical_crossentropie and adam...\r\n\r\n[/quote]\r\n\r\n@AndreSe Those are some promising results! If you are willing to share your code on Kaggle, then I will run it on some TITAN Xs for you ;)", "votes": null }, { "id": "122909", "postDate": "06/08/2016 08:05:19", "content": "Sure, when I get some freetime, I'll post the main code here (it's actually on my laptop at home, a bit far away :x )
\n\nAt all, I think the biggest problem is the huge amount of different possible classes/place_id's
\n\nSo cutting the the data down into small grids is necessary and you also need to encode the place_id's. to_categorical can't really work on this big place_id numbers...
\n\nI'm pretty new to keras, and machine learning, so still learning a lot on my self. :)
", "rawMarkdown": "Sure, when I get some freetime, I'll post the main code here (it's actually on my laptop at home, a bit far away :x )\r\n\r\nAt all, I think the biggest problem is the huge amount of different possible classes/place_id's\r\n\r\nSo cutting the the data down into small grids is necessary and you also need to encode the place_id's. to_categorical can't really work on this big place_id numbers... \r\n\r\nI'm pretty new to keras, and machine learning, so still learning a lot on my self. :)", "votes": null }, { "id": "123018", "postDate": "06/09/2016 07:15:06", "content": "The code generates one csv-file for every grid, so you need to concat them in the end manually (or with another script). So I can pause in between... :)\nMaybe a little bit confusing code, but I'm just testing around in my freetime... ;)
\n\nMore epochs would get better results I think, but also costs much more time...
\n\nYou don't need all the imports... and the LB 0.356 was just with one place_id each row_id, not MAP@3.
\n\nimport numpy as np\nimport seaborn as sns\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport time\nfrom keras.layers.core import Dense, Activation, Dropout\nfrom keras.layers.recurrent import LSTM\nfrom keras.models import Sequential\nfrom keras.utils.np_utils import to_categorical\nfrom sklearn import neighbors\nfrom keras.layers.normalization import BatchNormalization\nfrom keras.callbacks import EarlyStopping\nfrom keras.layers.advanced_activations import PReLU\n\n\n#Treshold\nth = 5\n\nprint "Loading Data..."\ntrain = pd.read_csv('train.csv')\ntest = pd.read_csv('test.csv')\n\ntrain['hour'] = (train['time']//60)%24+1\ntrain['weekday'] = (train['time']//1440)%7+1\ntrain['month'] = (train['time']//43200)%12+1\ntrain['year'] = (train['time']//525600)+1\n\ntest['hour'] = (test['time']//60)%24+1\ntest['weekday'] = (test['time']//1440)%7+1\ntest['month'] = (test['time']//43200)%12+1\ntest['year'] = (test['time']//525600)+1\n\n\ndef loaddata(dt, df,y_min,y_max,x_min, x_max,datarange,all=False):\n print "Data loaded. Formatting..."\n\n df_train = train[(train['x'] >= x_min) &\n (train['x'] < x_max) &\n (train['y'] >= y_min) &\n (train['y'] < y_max)]\n\n df_test = test[(test['x'] >= x_min) &\n (test['x'] < x_max) &\n (test['y'] >= y_min) &\n (test['y'] < y_max)]\n\n\n time_id = df_test['row_id']\n\n place_counts = df_train.place_id.value_counts()\n mask = (place_counts[df_train.place_id.values] >= th).values\n df_train = df_train.loc[mask]\n\n\n #List of possible place_ids\n places = list(set(df_train['place_id'].values.tolist()))\n\n buf = df_train['place_id']\n place_ids = []\n for val in buf:\n place_ids.append(places.index(val))\n y_train = np.array(place_ids)\n\n X_train = df_train[['x','y','accuracy','hour','weekday', 'month', 'year']].as_matrix()\n\n print len(y_train)\n y_train_cat = to_categorical(y_train)\n\n print X_train.shape\n print y_train.shape\n\n X_test = df_test[['x','y','accuracy','hour','weekday', 'month', 'year']].as_matrix()\n\n clf = None\n\n return [X_train, y_train_cat, X_test, time_id, len(places),clf,places]\n\ndef build_model(output):\n\n model = Sequential()\n model.add(Dense(input_dim=9, output_dim=512))\n model.add(PReLU())\n model.add(BatchNormalization())\n model.add(Dropout(0.5))\n\n model.add(Dense(output_dim=256))\n model.add(PReLU())\n model.add(BatchNormalization())\n model.add(Dropout(0.5))\n\n model.add(Dense(output_dim=128))\n model.add(PReLU())\n model.add(BatchNormalization())\n model.add(Dropout(0.5))\n\n model.add(Dense(output_dim=output))\n model.add(Activation('softmax'))\n start = time.time()\n model.compile(loss='categorical_crossentropy', optimizer="adam")\n print "Compilation Time : ", time.time() - start\n return model\n\ndef run_network(dt,df,y_min=0,y_max=0,x_min=0,x_max=0,model=None, data=None):\n\n X_train, y_train, X_test,time_id,output,clf,places = loaddata(dt,df,y_min,y_max,x_min,x_max,1000,True)\n\n model = build_model(output)\n\n try:\n #early_stopping = EarlyStopping(monitor='val_loss', patience=15)\n model.fit(\n X_train, y_train,\n batch_size=128, nb_epoch=20, validation_split=0.15) #,callbacks=[early_stopping]\n predicted = np.argsort(model.predict_proba(X_test), axis=1)[:,::-1][:,:3]\n except KeyboardInterrupt:\n print 'Training duration (s) : ', time.time() - global_start_time\n return model, y_test, 0\n\n print 'generating output'\n predicted = np.array([str(places[x[0]]) + ' ' + str(places[x[1]]) + ' ' + str(places[x[2]]) for x in predicted])\n predicted = pd.DataFrame(predicted.reshape(len(predicted),1),index = time_id)\n predicted.to_csv('result_' + str(y_min)+ '_' + str(y_max)+ '_'+ str(x_min)+ '_' + str(x_max)+'.csv', sep=',', index_label='row_id', header=['place_id'])\n\n\nsize = 10.0;\n\nx_step = 0.5\ny_step = 0.2\n\nfor i in range((int)(size/x_step)):\n x_min = x_step * i\n x_max = x_step * (i+1)\n x_min = round(x_min, 4)\n x_max = round(x_max, 4)\n start_time_row = time.time()\n if x_max == size:\n x_max = x_max + 0.001\n\n for j in range((int)(size/y_step)):\n #start_time_cell = time.time()\n y_min = y_step * j\n y_max = y_step * (j+1)\n y_min = round(y_min, 4)\n y_max = round(y_max, 4)\n if y_max == size:\n y_max = y_max + 0.001\n\n run_network(train,test,y_min, y_max, x_min, x_max)\nIn a next approach I wanted to test LSTM networks, what do you think about that?\nIt's good for timeseries, and maybe it could work in some way - with the time-parameter... and the data in "time-series-similar" representation?
", "rawMarkdown": "In a next approach I wanted to test LSTM networks, what do you think about that?\r\nIt's good for timeseries, and maybe it could work in some way - with the time-parameter... and the data in \"time-series-similar\" representation?", "votes": null }, { "id": "123190", "postDate": "06/10/2016 07:04:08", "content": "My actual approach scored 0.44095 LB\nAnd it just took around 12hours.
", "rawMarkdown": "My actual approach scored 0.44095 LB\r\nAnd it just took around 12hours.", "votes": null }, { "id": "123396", "postDate": "06/11/2016 16:18:47", "content": "0.44095 LB is obtained by three prediction for each test sample right? Did you do any in-sample cross validation? What is the in-sample validation score?
\n\nThanks!
", "rawMarkdown": "0.44095 LB is obtained by three prediction for each test sample right? Did you do any in-sample cross validation? What is the in-sample validation score?\r\n\r\nThanks!", "votes": null }, { "id": "123526", "postDate": "06/12/2016 14:37:15", "content": "Yes, its for 3 predictions for each. Didn't done any in-sample cross validation yet.\nWhen I get some more time, i wanted to optimize on just a small part (just a small amount of grids), with cross-validation and so.\nactually i have to write my masterthesis... so it's just a question of time >_<
", "rawMarkdown": "Yes, its for 3 predictions for each. Didn't done any in-sample cross validation yet.\r\nWhen I get some more time, i wanted to optimize on just a small part (just a small amount of grids), with cross-validation and so.\r\nactually i have to write my masterthesis... so it's just a question of time >_<", "votes": null }, { "id": "123593", "postDate": "06/13/2016 00:33:06", "content": "Hi I am new to ML how do u use GPU instead CPU in eclipse pydev.
", "rawMarkdown": "Hi I am new to ML how do u use GPU instead CPU in eclipse pydev.", "votes": null }, { "id": "123971", "postDate": "06/14/2016 18:24:51", "content": "hi AndreSe
\n\ni have some question
\n\nwhy you use BN after activation layar? is it before right?
\n\nwhy you use BN with dropout? i know BN has regularization
", "rawMarkdown": "hi AndreSe\r\n\r\ni have some question\r\n\r\nwhy you use BN after activation layar? is it before right?\r\n\r\nwhy you use BN with dropout? i know BN has regularization", "votes": null }, { "id": "123973", "postDate": "06/14/2016 18:51:21", "content": "BN: "Normalize the activations of the previous layer at each batch"\nand the dropout helps prevent overfitting.
\n\nSo, BN and dropout are two different layers. ;)
", "rawMarkdown": "BN: \"Normalize the activations of the previous layer at each batch\"\r\nand the dropout helps prevent overfitting.\r\n\r\nSo, BN and dropout are two different layers. ;)", "votes": null }, { "id": "124595", "postDate": "06/20/2016 07:33:56", "content": "I used some more features and a different amount of hidden layers per grid... it took too much time (60-80hours) and I got 0.48 on the LB. But I think the biggest problem here, was overfitting... (maybe).
\n\nDo you guys have any better approach?
", "rawMarkdown": "I used some more features and a different amount of hidden layers per grid... it took too much time (60-80hours) and I got 0.48 on the LB. But I think the biggest problem here, was overfitting... (maybe).\r\n\r\nDo you guys have any better approach?", "votes": null } ], "comments": [ { "id": 122620, "author_name": "sakvaua", "author_url": "", "post_date": "06/05/2016 21:46:22", "content": "Is that on GPU or CPU? How many layers? What activation functions do you use? How did you determine the number of training rounds? Did you do any sort of CV, what was your local score?
", "votes": null, "replies": [] }, { "id": 122636, "author_name": "mathandbikes", "author_url": "", "post_date": "06/06/2016 01:29:38", "content": "I wouldn't attempt methods over and over using the full training and test data sets provided by the competition. As you said, it can take potentially multiple days to go through that process.
\n\nI've been doing my own local cross validation on different chunks of the data. Usually I'll take a 1% block of the training data set and train/test using that.
\n\nI haven't submitted a full document yet, just because the run time isn't worth it. I've only submitted on ~40% of the data. I figure I just multiply that score by 2.5 then I'll have a rough estimate of what I'd get if I used the full data set. Oddly, I was getting scores as high as ~.62 for my local cross validation, but it looks like my submission would only receive around a ~.55 on the leaderboards.
\n\nThe name of the game right now is iterations and not place on leaderboards.
", "votes": null, "replies": [] }, { "id": 122656, "author_name": "sakvaua", "author_url": "", "post_date": "06/06/2016 06:20:07", "content": "MathandBikes do you split your train set by time for CV purposes?
", "votes": null, "replies": [] }, { "id": 122672, "author_name": "antsek", "author_url": "", "post_date": "06/06/2016 10:49:29", "content": "@MathandBikes,\nso you train and test just on a small set of data, optimizing your network, before you go on the full data set?\nI think, that's what I'll do next.
\n\n@DennisSakva,\non GPU (nvidia 730m, I just have a laptop.. ). I can add the exact configuration later - or maybe upload my chaotic code :D\nI'm actually not at the my own laptop. But I use prelu layers, categorical_crossentropie and adam...
", "votes": null, "replies": [] }, { "id": 122854, "author_name": "anokas", "author_url": "", "post_date": "06/07/2016 19:17:34", "content": "[quote=AndreSe;122672]
\n\n@MathandBikes,\nso you train and test just on a small set of data, optimizing your network, before you go on the full data set?\nI think, that's what I'll do next.
\n\n@DennisSakva,\non GPU (nvidia 730m, I just have a laptop.. ). I can add the exact configuration later - or maybe upload my chaotic code :D\nI'm actually not at the my own laptop. But I use prelu layers, categorical_crossentropie and adam...
\n\n[/quote]
\n\n@AndreSe Those are some promising results! If you are willing to share your code on Kaggle, then I will run it on some TITAN Xs for you ;)
", "votes": null, "replies": [] }, { "id": 122909, "author_name": "antsek", "author_url": "", "post_date": "06/08/2016 08:05:19", "content": "Sure, when I get some freetime, I'll post the main code here (it's actually on my laptop at home, a bit far away :x )
\n\nAt all, I think the biggest problem is the huge amount of different possible classes/place_id's
\n\nSo cutting the the data down into small grids is necessary and you also need to encode the place_id's. to_categorical can't really work on this big place_id numbers...
\n\nI'm pretty new to keras, and machine learning, so still learning a lot on my self. :)
", "votes": null, "replies": [] }, { "id": 123018, "author_name": "antsek", "author_url": "", "post_date": "06/09/2016 07:15:06", "content": "The code generates one csv-file for every grid, so you need to concat them in the end manually (or with another script). So I can pause in between... :)\nMaybe a little bit confusing code, but I'm just testing around in my freetime... ;)
\n\nMore epochs would get better results I think, but also costs much more time...
\n\nYou don't need all the imports... and the LB 0.356 was just with one place_id each row_id, not MAP@3.
\n\nimport numpy as np\nimport seaborn as sns\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport time\nfrom keras.layers.core import Dense, Activation, Dropout\nfrom keras.layers.recurrent import LSTM\nfrom keras.models import Sequential\nfrom keras.utils.np_utils import to_categorical\nfrom sklearn import neighbors\nfrom keras.layers.normalization import BatchNormalization\nfrom keras.callbacks import EarlyStopping\nfrom keras.layers.advanced_activations import PReLU\n\n\n#Treshold\nth = 5\n\nprint "Loading Data..."\ntrain = pd.read_csv('train.csv')\ntest = pd.read_csv('test.csv')\n\ntrain['hour'] = (train['time']//60)%24+1\ntrain['weekday'] = (train['time']//1440)%7+1\ntrain['month'] = (train['time']//43200)%12+1\ntrain['year'] = (train['time']//525600)+1\n\ntest['hour'] = (test['time']//60)%24+1\ntest['weekday'] = (test['time']//1440)%7+1\ntest['month'] = (test['time']//43200)%12+1\ntest['year'] = (test['time']//525600)+1\n\n\ndef loaddata(dt, df,y_min,y_max,x_min, x_max,datarange,all=False):\n print "Data loaded. Formatting..."\n\n df_train = train[(train['x'] >= x_min) &\n (train['x'] < x_max) &\n (train['y'] >= y_min) &\n (train['y'] < y_max)]\n\n df_test = test[(test['x'] >= x_min) &\n (test['x'] < x_max) &\n (test['y'] >= y_min) &\n (test['y'] < y_max)]\n\n\n time_id = df_test['row_id']\n\n place_counts = df_train.place_id.value_counts()\n mask = (place_counts[df_train.place_id.values] >= th).values\n df_train = df_train.loc[mask]\n\n\n #List of possible place_ids\n places = list(set(df_train['place_id'].values.tolist()))\n\n buf = df_train['place_id']\n place_ids = []\n for val in buf:\n place_ids.append(places.index(val))\n y_train = np.array(place_ids)\n\n X_train = df_train[['x','y','accuracy','hour','weekday', 'month', 'year']].as_matrix()\n\n print len(y_train)\n y_train_cat = to_categorical(y_train)\n\n print X_train.shape\n print y_train.shape\n\n X_test = df_test[['x','y','accuracy','hour','weekday', 'month', 'year']].as_matrix()\n\n clf = None\n\n return [X_train, y_train_cat, X_test, time_id, len(places),clf,places]\n\ndef build_model(output):\n\n model = Sequential()\n model.add(Dense(input_dim=9, output_dim=512))\n model.add(PReLU())\n model.add(BatchNormalization())\n model.add(Dropout(0.5))\n\n model.add(Dense(output_dim=256))\n model.add(PReLU())\n model.add(BatchNormalization())\n model.add(Dropout(0.5))\n\n model.add(Dense(output_dim=128))\n model.add(PReLU())\n model.add(BatchNormalization())\n model.add(Dropout(0.5))\n\n model.add(Dense(output_dim=output))\n model.add(Activation('softmax'))\n start = time.time()\n model.compile(loss='categorical_crossentropy', optimizer="adam")\n print "Compilation Time : ", time.time() - start\n return model\n\ndef run_network(dt,df,y_min=0,y_max=0,x_min=0,x_max=0,model=None, data=None):\n\n X_train, y_train, X_test,time_id,output,clf,places = loaddata(dt,df,y_min,y_max,x_min,x_max,1000,True)\n\n model = build_model(output)\n\n try:\n #early_stopping = EarlyStopping(monitor='val_loss', patience=15)\n model.fit(\n X_train, y_train,\n batch_size=128, nb_epoch=20, validation_split=0.15) #,callbacks=[early_stopping]\n predicted = np.argsort(model.predict_proba(X_test), axis=1)[:,::-1][:,:3]\n except KeyboardInterrupt:\n print 'Training duration (s) : ', time.time() - global_start_time\n return model, y_test, 0\n\n print 'generating output'\n predicted = np.array([str(places[x[0]]) + ' ' + str(places[x[1]]) + ' ' + str(places[x[2]]) for x in predicted])\n predicted = pd.DataFrame(predicted.reshape(len(predicted),1),index = time_id)\n predicted.to_csv('result_' + str(y_min)+ '_' + str(y_max)+ '_'+ str(x_min)+ '_' + str(x_max)+'.csv', sep=',', index_label='row_id', header=['place_id'])\n\n\nsize = 10.0;\n\nx_step = 0.5\ny_step = 0.2\n\nfor i in range((int)(size/x_step)):\n x_min = x_step * i\n x_max = x_step * (i+1)\n x_min = round(x_min, 4)\n x_max = round(x_max, 4)\n start_time_row = time.time()\n if x_max == size:\n x_max = x_max + 0.001\n\n for j in range((int)(size/y_step)):\n #start_time_cell = time.time()\n y_min = y_step * j\n y_max = y_step * (j+1)\n y_min = round(y_min, 4)\n y_max = round(y_max, 4)\n if y_max == size:\n y_max = y_max + 0.001\n\n run_network(train,test,y_min, y_max, x_min, x_max)\nIn a next approach I wanted to test LSTM networks, what do you think about that?\nIt's good for timeseries, and maybe it could work in some way - with the time-parameter... and the data in "time-series-similar" representation?
", "votes": null, "replies": [] }, { "id": 123190, "author_name": "antsek", "author_url": "", "post_date": "06/10/2016 07:04:08", "content": "My actual approach scored 0.44095 LB\nAnd it just took around 12hours.
", "votes": null, "replies": [] }, { "id": 123396, "author_name": "caixiabi", "author_url": "", "post_date": "06/11/2016 16:18:47", "content": "0.44095 LB is obtained by three prediction for each test sample right? Did you do any in-sample cross validation? What is the in-sample validation score?
\n\nThanks!
", "votes": null, "replies": [] }, { "id": 123526, "author_name": "antsek", "author_url": "", "post_date": "06/12/2016 14:37:15", "content": "Yes, its for 3 predictions for each. Didn't done any in-sample cross validation yet.\nWhen I get some more time, i wanted to optimize on just a small part (just a small amount of grids), with cross-validation and so.\nactually i have to write my masterthesis... so it's just a question of time >_<
", "votes": null, "replies": [] }, { "id": 123593, "author_name": "gcshekar08", "author_url": "", "post_date": "06/13/2016 00:33:06", "content": "Hi I am new to ML how do u use GPU instead CPU in eclipse pydev.
", "votes": null, "replies": [] }, { "id": 123971, "author_name": "ishuca", "author_url": "", "post_date": "06/14/2016 18:24:51", "content": "hi AndreSe
\n\ni have some question
\n\nwhy you use BN after activation layar? is it before right?
\n\nwhy you use BN with dropout? i know BN has regularization
", "votes": null, "replies": [] }, { "id": 123973, "author_name": "antsek", "author_url": "", "post_date": "06/14/2016 18:51:21", "content": "BN: "Normalize the activations of the previous layer at each batch"\nand the dropout helps prevent overfitting.
\n\nSo, BN and dropout are two different layers. ;)
", "votes": null, "replies": [] }, { "id": 124595, "author_name": "antsek", "author_url": "", "post_date": "06/20/2016 07:33:56", "content": "I used some more features and a different amount of hidden layers per grid... it took too much time (60-80hours) and I got 0.48 on the LB. But I think the biggest problem here, was overfitting... (maybe).
\n\nDo you guys have any better approach?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "122611": "Hi all,\r\n\r\nis someone trying to use Keras or another NeuralNetwork framework for this challenge?\r\n\r\nI just tried it and got 0.356...\r\n20*20 Grids and as Features: x,y,accuracy, hour, weekday, month, year\r\n\r\nIt was a second try (first try was just x and y as features and 50*50 grids).\r\n\r\nAt all, I would keep working on that - but it costs so much time to compute (20-40hours).\r\nSo my question, is someone else is trying something similar here? :)\r\n\r\nOr is it in general better to use some RandomForest?", "122620": "Is that on GPU or CPU? How many layers? What activation functions do you use? How did you determine the number of training rounds? Did you do any sort of CV, what was your local score?", "122636": "I wouldn't attempt methods over and over using the full training and test data sets provided by the competition. As you said, it can take potentially multiple days to go through that process.\r\n\r\nI've been doing my own local cross validation on different chunks of the data. Usually I'll take a 1% block of the training data set and train/test using that.\r\n\r\nI haven't submitted a full document yet, just because the run time isn't worth it. I've only submitted on ~40% of the data. I figure I just multiply that score by 2.5 then I'll have a rough estimate of what I'd get if I used the full data set. Oddly, I was getting scores as high as ~.62 for my local cross validation, but it looks like my submission would only receive around a ~.55 on the leaderboards.\r\n\r\nThe name of the game right now is iterations and not place on leaderboards.", "122656": "MathandBikes do you split your train set by time for CV purposes?", "122672": "MathandBikes,\r\nso you train and test just on a small set of data, optimizing your network, before you go on the full data set?\r\nI think, that's what I'll do next.\r\n\r\n@DennisSakva,\r\non GPU (nvidia 730m, I just have a laptop.. ). I can add the exact configuration later - or maybe upload my chaotic code :D\r\nI'm actually not at the my own laptop. But I use prelu layers, categorical_crossentropie and adam...", "122854": "[quote=AndreSe;122672]\r\n\r\n@MathandBikes,\r\nso you train and test just on a small set of data, optimizing your network, before you go on the full data set?\r\nI think, that's what I'll do next.\r\n\r\n@DennisSakva,\r\non GPU (nvidia 730m, I just have a laptop.. ). I can add the exact configuration later - or maybe upload my chaotic code :D\r\nI'm actually not at the my own laptop. But I use prelu layers, categorical_crossentropie and adam...\r\n\r\n[/quote]\r\n\r\n@AndreSe Those are some promising results! If you are willing to share your code on Kaggle, then I will run it on some TITAN Xs for you ;)", "122909": "Sure, when I get some freetime, I'll post the main code here (it's actually on my laptop at home, a bit far away :x )\r\n\r\nAt all, I think the biggest problem is the huge amount of different possible classes/place_id's\r\n\r\nSo cutting the the data down into small grids is necessary and you also need to encode the place_id's. to_categorical can't really work on this big place_id numbers... \r\n\r\nI'm pretty new to keras, and machine learning, so still learning a lot on my self. :)", "123018": "The code generates one csv-file for every grid, so you need to concat them in the end manually (or with another script). So I can pause in between... :)\r\nMaybe a little bit confusing code, but I'm just testing around in my freetime... ;)\r\n\r\nMore epochs would get better results I think, but also costs much more time...\r\n\r\nYou don't need all the imports... and the LB 0.356 was just with one place_id each row_id, not MAP@3.\r\n\r\n\r\n import numpy as np\r\n import seaborn as sns\r\n import pandas as pd\r\n import matplotlib.pyplot as plt\r\n import time\r\n from keras.layers.core import Dense, Activation, Dropout\r\n from keras.layers.recurrent import LSTM\r\n from keras.models import Sequential\r\n from keras.utils.np_utils import to_categorical\r\n from sklearn import neighbors\r\n from keras.layers.normalization import BatchNormalization\r\n from keras.callbacks import EarlyStopping\r\n from keras.layers.advanced_activations import PReLU\r\n \r\n \r\n #Treshold\r\n th = 5\r\n \r\n print \"Loading Data...\"\r\n train = pd.read_csv('train.csv')\r\n test = pd.read_csv('test.csv')\r\n \r\n train['hour'] = (train['time']//60)%24+1\r\n train['weekday'] = (train['time']//1440)%7+1\r\n train['month'] = (train['time']//43200)%12+1\r\n train['year'] = (train['time']//525600)+1\r\n \r\n test['hour'] = (test['time']//60)%24+1\r\n test['weekday'] = (test['time']//1440)%7+1\r\n test['month'] = (test['time']//43200)%12+1\r\n test['year'] = (test['time']//525600)+1\r\n \r\n \r\n def loaddata(dt, df,y_min,y_max,x_min, x_max,datarange,all=False):\r\n print \"Data loaded. Formatting...\"\r\n \r\n df_train = train[(train['x'] >= x_min) &\r\n (train['x'] < x_max) &\r\n (train['y'] >= y_min) &\r\n (train['y'] < y_max)]\r\n \r\n df_test = test[(test['x'] >= x_min) &\r\n (test['x'] < x_max) &\r\n (test['y'] >= y_min) &\r\n (test['y'] < y_max)]\r\n \r\n \r\n time_id = df_test['row_id']\r\n \r\n place_counts = df_train.place_id.value_counts()\r\n mask = (place_counts[df_train.place_id.values] >= th).values\r\n df_train = df_train.loc[mask]\r\n \r\n \r\n #List of possible place_ids\r\n places = list(set(df_train['place_id'].values.tolist()))\r\n \r\n buf = df_train['place_id']\r\n place_ids = []\r\n for val in buf:\r\n place_ids.append(places.index(val))\r\n y_train = np.array(place_ids)\r\n \r\n X_train = df_train[['x','y','accuracy','hour','weekday', 'month', 'year']].as_matrix()\r\n \r\n print len(y_train)\r\n y_train_cat = to_categorical(y_train)\r\n \r\n print X_train.shape\r\n print y_train.shape\r\n \r\n X_test = df_test[['x','y','accuracy','hour','weekday', 'month', 'year']].as_matrix()\r\n \r\n clf = None\r\n \r\n return [X_train, y_train_cat, X_test, time_id, len(places),clf,places]\r\n \r\n def build_model(output):\r\n \r\n model = Sequential()\r\n model.add(Dense(input_dim=9, output_dim=512))\r\n model.add(PReLU())\r\n model.add(BatchNormalization())\r\n model.add(Dropout(0.5))\r\n \r\n model.add(Dense(output_dim=256))\r\n model.add(PReLU())\r\n model.add(BatchNormalization())\r\n model.add(Dropout(0.5))\r\n \r\n model.add(Dense(output_dim=128))\r\n model.add(PReLU())\r\n model.add(BatchNormalization())\r\n model.add(Dropout(0.5))\r\n \r\n model.add(Dense(output_dim=output))\r\n model.add(Activation('softmax'))\r\n start = time.time()\r\n model.compile(loss='categorical_crossentropy', optimizer=\"adam\")\r\n print \"Compilation Time : \", time.time() - start\r\n return model\r\n \r\n def run_network(dt,df,y_min=0,y_max=0,x_min=0,x_max=0,model=None, data=None):\r\n \r\n X_train, y_train, X_test,time_id,output,clf,places = loaddata(dt,df,y_min,y_max,x_min,x_max,1000,True)\r\n \r\n model = build_model(output)\r\n \r\n try:\r\n #early_stopping = EarlyStopping(monitor='val_loss', patience=15)\r\n model.fit(\r\n X_train, y_train,\r\n batch_size=128, nb_epoch=20, validation_split=0.15) #,callbacks=[early_stopping]\r\n predicted = np.argsort(model.predict_proba(X_test), axis=1)[:,::-1][:,:3]\r\n except KeyboardInterrupt:\r\n print 'Training duration (s) : ', time.time() - global_start_time\r\n return model, y_test, 0\r\n \r\n print 'generating output'\r\n predicted = np.array([str(places[x[0]]) + ' ' + str(places[x[1]]) + ' ' + str(places[x[2]]) for x in predicted])\r\n predicted = pd.DataFrame(predicted.reshape(len(predicted),1),index = time_id)\r\n predicted.to_csv('result_' + str(y_min)+ '_' + str(y_max)+ '_'+ str(x_min)+ '_' + str(x_max)+'.csv', sep=',', index_label='row_id', header=['place_id'])\r\n \r\n \r\n size = 10.0;\r\n \r\n x_step = 0.5\r\n y_step = 0.2\r\n \r\n for i in range((int)(size/x_step)):\r\n x_min = x_step * i\r\n x_max = x_step * (i+1)\r\n x_min = round(x_min, 4)\r\n x_max = round(x_max, 4)\r\n start_time_row = time.time()\r\n if x_max == size:\r\n x_max = x_max + 0.001\r\n \r\n for j in range((int)(size/y_step)):\r\n #start_time_cell = time.time()\r\n y_min = y_step * j\r\n y_max = y_step * (j+1)\r\n y_min = round(y_min, 4)\r\n y_max = round(y_max, 4)\r\n if y_max == size:\r\n y_max = y_max + 0.001\r\n \r\n run_network(train,test,y_min, y_max, x_min, x_max)", "123035": "In a next approach I wanted to test LSTM networks, what do you think about that?\r\nIt's good for timeseries, and maybe it could work in some way - with the time-parameter... and the data in \"time-series-similar\" representation?", "123190": "My actual approach scored 0.44095 LB\r\nAnd it just took around 12hours.", "123396": "0.44095 LB is obtained by three prediction for each test sample right? Did you do any in-sample cross validation? What is the in-sample validation score?\r\n\r\nThanks!", "123526": "Yes, its for 3 predictions for each. Didn't done any in-sample cross validation yet.\r\nWhen I get some more time, i wanted to optimize on just a small part (just a small amount of grids), with cross-validation and so.\r\nactually i have to write my masterthesis... so it's just a question of time >_<", "123593": "Hi I am new to ML how do u use GPU instead CPU in eclipse pydev.", "123971": "hi AndreSe\r\n\r\ni have some question\r\n\r\nwhy you use BN after activation layar? is it before right?\r\n\r\nwhy you use BN with dropout? i know BN has regularization", "123973": "BN: \"Normalize the activations of the previous layer at each batch\"\r\nand the dropout helps prevent overfitting.\r\n\r\nSo, BN and dropout are two different layers. ;)", "124595": "I used some more features and a different amount of hidden layers per grid... it took too much time (60-80hours) and I got 0.48 on the LB. But I think the biggest problem here, was overfitting... (maybe).\r\n\r\nDo you guys have any better approach?" }, "source": "meta" }