{ "id": 12690, "title": "Validation Accuracy is 70% ranked 800 in LB", "url": "/competitions/datasciencebowl/discussion/12690", "author_name": "", "post_date": "2015-03-05T10:40:18.500Z", "votes": null, "comment_count": 13, "views": 1557, "content": "

Hi guys. I just come in this competition recent few days. While I deploy a CNN with Nolearn + Lasagne with a single GPU in some cloud computing platform. I left 10% training data as my validation data using sklearn.train_test_split. And I get my validation accuracy nearly 70% within an hour. However, when I submit my result, my logloss is about 8 ranked 800 in LB. Does anything go wrong? And I guess maybe the row or column mixing problem?

\n

Any suggestions are welcomed. Thank you ! 

", "messages": [ { "id": "65486", "postDate": "03/05/2015 10:40:18", "content": "

Hi guys. I just come in this competition recent few days. While I deploy a CNN with Nolearn + Lasagne with a single GPU in some cloud computing platform. I left 10% training data as my validation data using sklearn.train_test_split. And I get my validation accuracy nearly 70% within an hour. However, when I submit my result, my logloss is about 8 ranked 800 in LB. Does anything go wrong? And I guess maybe the row or column mixing problem?

\n

Any suggestions are welcomed. Thank you ! 

", "rawMarkdown": "", "votes": null }, { "id": "65488", "postDate": "03/05/2015 10:59:05", "content": "

Seems like a bug in your submission file or classification process, a logloss of 8 is very bad, like random. An accuracy of ~70% should get you closer to a logloss of 1.

", "rawMarkdown": "", "votes": null }, { "id": "65489", "postDate": "03/05/2015 11:01:16", "content": "

@npetitclerc, does the row of submission file should be ordered like 1.jpg, 2.jpg, ... ?

", "rawMarkdown": "", "votes": null }, { "id": "65490", "postDate": "03/05/2015 11:17:03", "content": "

Check the order of the columns for the label. I had this problem too, and solved after checking the order of the columns in the submission file

", "rawMarkdown": "", "votes": null }, { "id": "65491", "postDate": "03/05/2015 11:24:55", "content": "

@ Elena Cuoco, do the column names should comply with the training folder names?

", "rawMarkdown": "", "votes": null }, { "id": "65492", "postDate": "03/05/2015 11:33:23", "content": "

Column names is the same as folder names in train folder. I think that you are mixing columns in submission file (For example in acantharia_protist column you write acantharia_protist_big_center predictions).

", "rawMarkdown": "", "votes": null }, { "id": "65493", "postDate": "03/05/2015 11:40:49", "content": "

So, row order does not matter?

", "rawMarkdown": "", "votes": null }, { "id": "65494", "postDate": "03/05/2015 12:01:54", "content": "

Row order is irrelevant, as long as you've put the right data next to the right filename on each row.

\n

I think column order is irrelevant too, as long as the labels in the header are ordered the same as the classes in your data. Can someone confirm? Or does the order have to exactly the same as in the sample submission?

", "rawMarkdown": "", "votes": null }, { "id": "65499", "postDate": "03/05/2015 12:22:51", "content": "

@Nicolay Korslund you are right. Order does not matter until class corresponds to class title. 

", "rawMarkdown": "", "votes": null }, { "id": "65500", "postDate": "03/05/2015 12:22:52", "content": "

[quote=PengPai;65491]

\n

@ Elena Cuoco, do the column names should comply with the training folder names?

\n

[/quote]

\n

Yes, the labels correspond to the folder names. So be sure to make label and name ordered in the same way. As other say, row order does not matter.

", "rawMarkdown": "", "votes": null }, { "id": "65504", "postDate": "03/05/2015 13:03:10", "content": "

Hi PengPai

\n

which cloud service you are using?

\n\n

is there any tutorial or other resources help learning this method?

", "rawMarkdown": "", "votes": null }, { "id": "65520", "postDate": "03/05/2015 15:20:09", "content": "

Thank you guys. I just re-rank my submissions and get a reasonable LB-rank.

", "rawMarkdown": "", "votes": null }, { "id": "65578", "postDate": "03/06/2015 08:54:04", "content": "

You can use below code for generating your submission.csv

\n\n

######################################################################
#Submission
######################################################################

\n

header = "acantharia_protist_big_center,acantharia_protist_halo,acantharia_protist,amphipods,appendicularian_fritillaridae,appendicularian_s_shape,appendicularian_slight_curve,appendicularian_straight,artifacts_edge,artifacts,chaetognath_non_sagitta,chaetognath_other,chaetognath_sagitta,chordate_type1,copepod_calanoid_eggs,copepod_calanoid_eucalanus,copepod_calanoid_flatheads,copepod_calanoid_frillyAntennae,copepod_calanoid_large_side_antennatucked,copepod_calanoid_large,copepod_calanoid_octomoms,copepod_calanoid_small_longantennae,copepod_calanoid,copepod_cyclopoid_copilia,copepod_cyclopoid_oithona_eggs,copepod_cyclopoid_oithona,copepod_other,crustacean_other,ctenophore_cestid,ctenophore_cydippid_no_tentacles,ctenophore_cydippid_tentacles,ctenophore_lobate,decapods,detritus_blob,detritus_filamentous,detritus_other,diatom_chain_string,diatom_chain_tube,echinoderm_larva_pluteus_brittlestar,echinoderm_larva_pluteus_early,echinoderm_larva_pluteus_typeC,echinoderm_larva_pluteus_urchin,echinoderm_larva_seastar_bipinnaria,echinoderm_larva_seastar_brachiolaria,echinoderm_seacucumber_auricularia_larva,echinopluteus,ephyra,euphausiids_young,euphausiids,fecal_pellet,fish_larvae_deep_body,fish_larvae_leptocephali,fish_larvae_medium_body,fish_larvae_myctophids,fish_larvae_thin_body,fish_larvae_very_thin_body,heteropod,hydromedusae_aglaura,hydromedusae_bell_and_tentacles,hydromedusae_h15,hydromedusae_haliscera_small_sideview,hydromedusae_haliscera,hydromedusae_liriope,hydromedusae_narco_dark,hydromedusae_narco_young,hydromedusae_narcomedusae,hydromedusae_other,hydromedusae_partial_dark,hydromedusae_shapeA_sideview_small,hydromedusae_shapeA,hydromedusae_shapeB,hydromedusae_sideview_big,hydromedusae_solmaris,hydromedusae_solmundella,hydromedusae_typeD_bell_and_tentacles,hydromedusae_typeD,hydromedusae_typeE,hydromedusae_typeF,invertebrate_larvae_other_A,invertebrate_larvae_other_B,jellies_tentacles,polychaete,protist_dark_center,protist_fuzzy_olive,protist_noctiluca,protist_other,protist_star,pteropod_butterfly,pteropod_theco_dev_seq,pteropod_triangle,radiolarian_chain,radiolarian_colony,shrimp_caridean,shrimp_sergestidae,shrimp_zoea,shrimp-like_other,siphonophore_calycophoran_abylidae,siphonophore_calycophoran_rocketship_adult,siphonophore_calycophoran_rocketship_young,siphonophore_calycophoran_sphaeronectes_stem,siphonophore_calycophoran_sphaeronectes_young,siphonophore_calycophoran_sphaeronectes,siphonophore_other_parts,siphonophore_partial,siphonophore_physonect_young,siphonophore_physonect,stomatopod,tornaria_acorn_worm_larvae,trichodesmium_bowtie,trichodesmium_multiple,trichodesmium_puff,trichodesmium_tuft,trochophore_larvae,tunicate_doliolid_nurse,tunicate_doliolid,tunicate_partial,tunicate_salp_chains,tunicate_salp,unknown_blobs_and_smudges,unknown_sticks,unknown_unclassified".split(',')

\n

labels = map(lambda s: s.split('/')[-1], namesClasses)

\n

#get the total test images
fnames = glob.glob(os.path.join("competition_data", "test", "*.jpg"))
numberofTestImages = len(fnames)

\n

print(numberofTestImages)

\n

X_test = np.zeros((numberofTestImages, num_features), dtype=float)

\n

images = map(lambda fileName: fileName.split('/')[-1], fnames)

\n

i = 0
# report progress for each 5% done
report = [int((j+1)*numberofTestImages/20.) for j in range(20)]
for fileName in fnames:
# Read in the images and create the features
image = imread(fileName, as_grey=True)
(axisratio,orientation,convex_area,extent,fscore) = getFeatures(image)
image = resize(image, (maxPixel, maxPixel))

\n

# Store the rescaled image pixels and the axis ratio
X_test[i, 0:imageSize] = np.reshape(image, (1, imageSize))
X_test[i, imageSize] = axisratio
X_test[i, imageSize+1] = orientation
X_test[i, imageSize+2] = convex_area
X_test[i, imageSize+3] = extent
X_test[i, imageSize+4] = fscore

\n

i += 1
if i in report: print np.ceil(i *100.0 / numberofTestImages), "% done"

\n

y_pred = clf.predict_proba(X_test)

\n

y_pred.shape

\n

df = pd.DataFrame(y_pred, columns=labels, index=images)
df.index.name = 'image'
df = df[header]
df.to_csv('competition_data/submission.csv')

", "rawMarkdown": "", "votes": null }, { "id": "65584", "postDate": "03/06/2015 09:57:27", "content": "

@data-lover, I am using terminal.com. Actually there have been a lot of cloud computing services around. Amazons' are popular as well. For CNN tutorial, check deeplearning.net

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 65488, "author_name": "npetitclerc", "author_url": "", "post_date": "03/05/2015 10:59:05", "content": "

Seems like a bug in your submission file or classification process, a logloss of 8 is very bad, like random. An accuracy of ~70% should get you closer to a logloss of 1.

", "votes": null, "replies": [] }, { "id": 65489, "author_name": "pengpai", "author_url": "", "post_date": "03/05/2015 11:01:16", "content": "

@npetitclerc, does the row of submission file should be ordered like 1.jpg, 2.jpg, ... ?

", "votes": null, "replies": [] }, { "id": 65490, "author_name": "elenacuoco", "author_url": "", "post_date": "03/05/2015 11:17:03", "content": "

Check the order of the columns for the label. I had this problem too, and solved after checking the order of the columns in the submission file

", "votes": null, "replies": [] }, { "id": 65491, "author_name": "pengpai", "author_url": "", "post_date": "03/05/2015 11:24:55", "content": "

@ Elena Cuoco, do the column names should comply with the training folder names?

", "votes": null, "replies": [] }, { "id": 65492, "author_name": "pikquu", "author_url": "", "post_date": "03/05/2015 11:33:23", "content": "

Column names is the same as folder names in train folder. I think that you are mixing columns in submission file (For example in acantharia_protist column you write acantharia_protist_big_center predictions).

", "votes": null, "replies": [] }, { "id": 65493, "author_name": "pengpai", "author_url": "", "post_date": "03/05/2015 11:40:49", "content": "

So, row order does not matter?

", "votes": null, "replies": [] }, { "id": 65494, "author_name": "nicolaykorslund", "author_url": "", "post_date": "03/05/2015 12:01:54", "content": "

Row order is irrelevant, as long as you've put the right data next to the right filename on each row.

\n

I think column order is irrelevant too, as long as the labels in the header are ordered the same as the classes in your data. Can someone confirm? Or does the order have to exactly the same as in the sample submission?

", "votes": null, "replies": [] }, { "id": 65499, "author_name": "oldufo", "author_url": "", "post_date": "03/05/2015 12:22:51", "content": "

@Nicolay Korslund you are right. Order does not matter until class corresponds to class title. 

", "votes": null, "replies": [] }, { "id": 65500, "author_name": "elenacuoco", "author_url": "", "post_date": "03/05/2015 12:22:52", "content": "

[quote=PengPai;65491]

\n

@ Elena Cuoco, do the column names should comply with the training folder names?

\n

[/quote]

\n

Yes, the labels correspond to the folder names. So be sure to make label and name ordered in the same way. As other say, row order does not matter.

", "votes": null, "replies": [] }, { "id": 65504, "author_name": "", "author_url": "", "post_date": "03/05/2015 13:03:10", "content": "

Hi PengPai

\n

which cloud service you are using?

\n\n

is there any tutorial or other resources help learning this method?

", "votes": null, "replies": [] }, { "id": 65520, "author_name": "pengpai", "author_url": "", "post_date": "03/05/2015 15:20:09", "content": "

Thank you guys. I just re-rank my submissions and get a reasonable LB-rank.

", "votes": null, "replies": [] }, { "id": 65578, "author_name": "saquibkhan", "author_url": "", "post_date": "03/06/2015 08:54:04", "content": "

You can use below code for generating your submission.csv

\n\n

######################################################################
#Submission
######################################################################

\n

header = "acantharia_protist_big_center,acantharia_protist_halo,acantharia_protist,amphipods,appendicularian_fritillaridae,appendicularian_s_shape,appendicularian_slight_curve,appendicularian_straight,artifacts_edge,artifacts,chaetognath_non_sagitta,chaetognath_other,chaetognath_sagitta,chordate_type1,copepod_calanoid_eggs,copepod_calanoid_eucalanus,copepod_calanoid_flatheads,copepod_calanoid_frillyAntennae,copepod_calanoid_large_side_antennatucked,copepod_calanoid_large,copepod_calanoid_octomoms,copepod_calanoid_small_longantennae,copepod_calanoid,copepod_cyclopoid_copilia,copepod_cyclopoid_oithona_eggs,copepod_cyclopoid_oithona,copepod_other,crustacean_other,ctenophore_cestid,ctenophore_cydippid_no_tentacles,ctenophore_cydippid_tentacles,ctenophore_lobate,decapods,detritus_blob,detritus_filamentous,detritus_other,diatom_chain_string,diatom_chain_tube,echinoderm_larva_pluteus_brittlestar,echinoderm_larva_pluteus_early,echinoderm_larva_pluteus_typeC,echinoderm_larva_pluteus_urchin,echinoderm_larva_seastar_bipinnaria,echinoderm_larva_seastar_brachiolaria,echinoderm_seacucumber_auricularia_larva,echinopluteus,ephyra,euphausiids_young,euphausiids,fecal_pellet,fish_larvae_deep_body,fish_larvae_leptocephali,fish_larvae_medium_body,fish_larvae_myctophids,fish_larvae_thin_body,fish_larvae_very_thin_body,heteropod,hydromedusae_aglaura,hydromedusae_bell_and_tentacles,hydromedusae_h15,hydromedusae_haliscera_small_sideview,hydromedusae_haliscera,hydromedusae_liriope,hydromedusae_narco_dark,hydromedusae_narco_young,hydromedusae_narcomedusae,hydromedusae_other,hydromedusae_partial_dark,hydromedusae_shapeA_sideview_small,hydromedusae_shapeA,hydromedusae_shapeB,hydromedusae_sideview_big,hydromedusae_solmaris,hydromedusae_solmundella,hydromedusae_typeD_bell_and_tentacles,hydromedusae_typeD,hydromedusae_typeE,hydromedusae_typeF,invertebrate_larvae_other_A,invertebrate_larvae_other_B,jellies_tentacles,polychaete,protist_dark_center,protist_fuzzy_olive,protist_noctiluca,protist_other,protist_star,pteropod_butterfly,pteropod_theco_dev_seq,pteropod_triangle,radiolarian_chain,radiolarian_colony,shrimp_caridean,shrimp_sergestidae,shrimp_zoea,shrimp-like_other,siphonophore_calycophoran_abylidae,siphonophore_calycophoran_rocketship_adult,siphonophore_calycophoran_rocketship_young,siphonophore_calycophoran_sphaeronectes_stem,siphonophore_calycophoran_sphaeronectes_young,siphonophore_calycophoran_sphaeronectes,siphonophore_other_parts,siphonophore_partial,siphonophore_physonect_young,siphonophore_physonect,stomatopod,tornaria_acorn_worm_larvae,trichodesmium_bowtie,trichodesmium_multiple,trichodesmium_puff,trichodesmium_tuft,trochophore_larvae,tunicate_doliolid_nurse,tunicate_doliolid,tunicate_partial,tunicate_salp_chains,tunicate_salp,unknown_blobs_and_smudges,unknown_sticks,unknown_unclassified".split(',')

\n

labels = map(lambda s: s.split('/')[-1], namesClasses)

\n

#get the total test images
fnames = glob.glob(os.path.join("competition_data", "test", "*.jpg"))
numberofTestImages = len(fnames)

\n

print(numberofTestImages)

\n

X_test = np.zeros((numberofTestImages, num_features), dtype=float)

\n

images = map(lambda fileName: fileName.split('/')[-1], fnames)

\n

i = 0
# report progress for each 5% done
report = [int((j+1)*numberofTestImages/20.) for j in range(20)]
for fileName in fnames:
# Read in the images and create the features
image = imread(fileName, as_grey=True)
(axisratio,orientation,convex_area,extent,fscore) = getFeatures(image)
image = resize(image, (maxPixel, maxPixel))

\n

# Store the rescaled image pixels and the axis ratio
X_test[i, 0:imageSize] = np.reshape(image, (1, imageSize))
X_test[i, imageSize] = axisratio
X_test[i, imageSize+1] = orientation
X_test[i, imageSize+2] = convex_area
X_test[i, imageSize+3] = extent
X_test[i, imageSize+4] = fscore

\n

i += 1
if i in report: print np.ceil(i *100.0 / numberofTestImages), "% done"

\n

y_pred = clf.predict_proba(X_test)

\n

y_pred.shape

\n

df = pd.DataFrame(y_pred, columns=labels, index=images)
df.index.name = 'image'
df = df[header]
df.to_csv('competition_data/submission.csv')

", "votes": null, "replies": [] }, { "id": 65584, "author_name": "pengpai", "author_url": "", "post_date": "03/06/2015 09:57:27", "content": "

@data-lover, I am using terminal.com. Actually there have been a lot of cloud computing services around. Amazons' are popular as well. For CNN tutorial, check deeplearning.net

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "65486": "", "65488": "", "65489": "", "65490": "", "65491": "", "65492": "", "65493": "", "65494": "", "65499": "", "65500": "", "65504": "", "65520": "", "65578": "", "65584": "" }, "source": "meta" }