![\"\"](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/3978/media/plankton%20schmorgasborg.jpg\")
In total, Oregon State University’s Hatfield Marine Science Center has captured nearly 50 million plankton images over an 18-day period. This is more than 80 terabytes of data! They need your help creating an automated classification process to better understand the image contents.
\r\nFor this competition, Hatfield scientists have prepared a large collection of labeled images, approximately 30k of which are provided as a training set. Each raw image was run through an automatic process to extract regions of interest, resulting in smaller images that contain a single organism/entity. You must create an algorithm that assigns class probabilities to a given image. Several characteristics of this problem make this classification difficult:
\r\n
The file train.zip contains labeled images for training. Each folder represents a class. Images within the folder belong to that class. Classes were chosen by the Hatfield experts and represent scientifically meaningful groupings of organisms and objects (see the below FAQs). You will notice that it is possible for different classes to contain the same underlying organism. In these cases, they have been separated based on other factors of interest (e.g. one may represent an organism in motion vs. one that is still). You are permitted to use domain knowledge about the class relationships in your methodology. The official list of classes is the following:
\r\nacantharia_protist_big_center\r\n
acantharia_protist_halo
acantharia_protist
amphipods
appendicularian_fritillaridae
appendicularian_s_shape
appendicularian_slight_curve
appendicularian_straight
artifacts_edge
artifacts
chaetognath_non_sagitta
chaetognath_other
chaetognath_sagitta
chordate_type1
copepod_calanoid_eggs
copepod_calanoid_eucalanus
copepod_calanoid_flatheads
copepod_calanoid_frillyAntennae
copepod_calanoid_large_side_antennatucked
copepod_calanoid_large
copepod_calanoid_octomoms
copepod_calanoid_small_longantennae
copepod_calanoid
copepod_cyclopoid_copilia
copepod_cyclopoid_oithona_eggs
copepod_cyclopoid_oithona
copepod_other
crustacean_other
ctenophore_cestid
ctenophore_cydippid_no_tentacles
ctenophore_cydippid_tentacles
ctenophore_lobate
decapods
detritus_blob
detritus_filamentous
detritus_other
diatom_chain_string
diatom_chain_tube
echinoderm_larva_pluteus_brittlestar
echinoderm_larva_pluteus_early
echinoderm_larva_pluteus_typeC
echinoderm_larva_pluteus_urchin
echinoderm_larva_seastar_bipinnaria
echinoderm_larva_seastar_brachiolaria
echinoderm_seacucumber_auricularia_larva
echinopluteus
ephyra
euphausiids_young
euphausiids
fecal_pellet
fish_larvae_deep_body
fish_larvae_leptocephali
fish_larvae_medium_body
fish_larvae_myctophids
fish_larvae_thin_body
fish_larvae_very_thin_body
heteropod
hydromedusae_aglaura
hydromedusae_bell_and_tentacles
hydromedusae_h15
hydromedusae_haliscera_small_sideview
hydromedusae_haliscera
hydromedusae_liriope
hydromedusae_narco_dark
hydromedusae_narco_young
hydromedusae_narcomedusae
hydromedusae_other
hydromedusae_partial_dark
hydromedusae_shapeA_sideview_small
hydromedusae_shapeA
hydromedusae_shapeB
hydromedusae_sideview_big
hydromedusae_solmaris
hydromedusae_solmundella
hydromedusae_typeD_bell_and_tentacles
hydromedusae_typeD
hydromedusae_typeE
hydromedusae_typeF
invertebrate_larvae_other_A
invertebrate_larvae_other_B
jellies_tentacles
polychaete
protist_dark_center
protist_fuzzy_olive
protist_noctiluca
protist_other
protist_star
pteropod_butterfly
pteropod_theco_dev_seq
pteropod_triangle
radiolarian_chain
radiolarian_colony
shrimp_caridean
shrimp_sergestidae
shrimp_zoea
shrimp-like_other
siphonophore_calycophoran_abylidae
siphonophore_calycophoran_rocketship_adult
siphonophore_calycophoran_rocketship_young
siphonophore_calycophoran_sphaeronectes_stem
siphonophore_calycophoran_sphaeronectes_young
siphonophore_calycophoran_sphaeronectes
siphonophore_other_parts
siphonophore_partial
siphonophore_physonect_young
siphonophore_physonect
stomatopod
tornaria_acorn_worm_larvae
trichodesmium_bowtie
trichodesmium_multiple
trichodesmium_puff
trichodesmium_tuft
trochophore_larvae
tunicate_doliolid_nurse
tunicate_doliolid
tunicate_partial
tunicate_salp_chains
tunicate_salp
unknown_blobs_and_smudges
unknown_sticks
unknown_unclassified
The file test.zip contains the test set images, for which you must predict class probabilities. To deter hand labeling of images, Kaggle has supplemented the test set with additional \"ignored\" images. These are not counted in the scoring.
\r\nsampleSubmission.csv provides the appropriate format for generating a valid submission.
\r\nplankton_identification.pdf is provided as a rough guide to understand relationships between the classes. The tree-like diagram indicates morphological and biological connections between groups. Dashed lines indicate a weak(er) relationship and solid lines a stronger relationship.
\r\nwritten by Jessica Luo, PhD Candidate, Hatfield Marine Science Center
\r\nWe collected the images between May-June 2014 in the Straits of Florida using the In Situ Ichthyoplankton Imaging System (ISIIS), a towed, underwater imaging system using shadowgraph imagery with a line-scan camera.
\r\nA high-resolution continuous image is parsed into 2048 x 2048 pixel frames, which were corrected using flat-fielding. Frames were then thresholded and segmented into regions of interest (ROIs). These ROI segments are the images provided for this competition.
\r\nWe made every effort to include a representation of real world data in this dataset. In other words, we did not cherry-pick for the best and clearest images, but used images spanning the gamut from blurry to clear, and tiny to big.
\r\nNo. Because of the shadowgraph imagery technique, organisms will be the same size regardless of distance to the camera.
\r\nPartial images in our dataset occur due to imperfect segmentation. If an organism is highly transparent (like a jellyfish), then sometimes the thresholding and segmentation process will cut off part of the image. This is fairly rare; typically our segmentation process works very well.
\r\nYes, there are relationships between the categories. Some of the relationships are taxonomic, some are behavioral / ecological, and some are based on shape. We diagrammed all of these relationships in the plankton relationships document, which can be accessed along with the dataset. Red boxes indicate our categories, blue boxes indicate major groups, solid lines indicate direct relationships, and dashed lines indicate minor relationships or shape similarities.
\r\nIn some cases, we chose to partition organisms into different categories based on shape. For example, in the appendicularians, the shape indicates a behavior (tail beating). Furthermore, appendicularians are often confused with a whole host of other things - most notably, fish larvae - so we have traditionally separated them out by shape to increase classification accuracy.
\r\nWe have a trained team and have cross-validated the classifications, such that certain images were checked by up to five people. That said, there is an intrinsic amount of variability in classifying ambiguous images. For example, Culverhouse et al. (2003) found that experts are able to maintain 84-95% self-consistency in labeling difficult images. A spot check on the self-consistency in this dataset was on the high end of that range. We believe that this is the best possible dataset of these kinds of plankton images currently in existence.
\r\nCitation: PF Culverhouse, Williams R, Reguera B, Herry V, González-Gil S. (2003) \"Do experts make mistakes? A comparison of human and machine identification of dinoflagellates.\" Mar. Ecol. Prog. Ser. 247:17-25.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Description", "content": "Plankton are critically important to our ecosystem, accounting for more than half the primary productivity on earth and nearly half the total carbon fixed in the global carbon cycle. They form the foundation of aquatic food webs including those of large, important fisheries. Loss of plankton populations could result in ecological upheaval as well as negative societal impacts, particularly in indigenous cultures and the developing world. Plankton’s global significance makes their population levels an ideal measure of the health of the world’s oceans and ecosystems.
\r\n![\"\"](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/3978/media/Plankton-Diagram3-lg.png\")
Traditional methods for measuring and monitoring plankton populations are time consuming and cannot scale to the granularity or scope necessary for large-scale studies. Improved approaches are needed. One such approach is through the use of an underwater imagery sensor. This towed, underwater camera system captures microscopic, high-resolution images over large study areas. The images can then be analyzed to assess species populations and distributions.
\r\nManual analysis of the imagery is infeasible – it would take a year or more to manually analyze the imagery volume captured in a single day. Automated image classification using machine learning tools is an alternative to the manual approach. Analytics will allow analysis at speeds and scales previously thought impossible. The automated system will have broad applications for assessment of ocean and ecosystem health.
\r\nThe National Data Science Bowl challenges you to build an algorithm to automate the image identification process. Scientists at the Hatfield Marine Science Center and beyond will use the algorithms you create to study marine food webs, fisheries, ocean conservation, and more. This is your chance to contribute to the health of the world’s oceans, one plankton at a time.
\r\nThe National Data Science Bowl is presented by![](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/3978/media/booz_kaggle.png\")
with data provided by the Hatfield Marine Science Center at Oregon State University.
Submissions are evaluated using the multi-class logarithmic loss. Each image has been labeled with one true class. For each image, you must submit a set of predicted probabilities (one for every class). The formula is then,
\r\n$$log loss = -\\frac{1}{N}\\sum_{i=1}^N\\sum_{j=1}^My_{ij}\\log(p_{ij}),$$
\r\nwhere N is the number of images in the test set, M is the number of class labels, \\\\(log\\\\) is the natural logarithm, \\\\(y_{ij}\\\\) is 1 if observation \\\\(i\\\\) is in class \\\\(j\\\\) and 0 otherwise, and \\\\(p_{ij}\\\\) is the predicted probability that observation \\\\(i\\\\) belongs to class \\\\(j\\\\).
\r\nThe submitted probabilities for a given image are not required to sum to one because they are rescaled prior to being scored (each row is divided by the row sum). In order to avoid the extremes of the log function, predicted probabilities are replaced with \\\\(max(min(p,1-10^{-15}),10^{-15})\\\\).
\r\nYou must submit a csv file with the image name, all candidate class names, and a probability for each class. The order of the rows does not matter. The file must have a header and should look like the following:
\r\nimage,acantharia_protist_big_center,...,unknown_unclassified", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Prizes", "content": "
1.jpg,0.00826446,...,0.00826446
10.jpg,0.00826446,...,0.00826446
...
etc.
The total prize pool for this competition is $175,000, distributed as follows:
\r\nIn addition, a $15,000 prize will be awarded to the highest ranked team comprised entirely of participants currently enrolled in a U.S. college*.
\r\n* Each student must be actively enrolled in at least one course, for the 2014-2015 academic year, at a college geographically located in one of the 50 United States, the District of Columbia, or Puerto Rico.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "rules", "content": "\r\n\t\r\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\r\n\r\n Privately sharing code or data outside of teams is not permitted.\r\n It's okay to share code if made available to all participants on the forums.\r\n
\r\n\r\n Team mergers are allowed and can be performed by the team leader. In order to merge, the combined team must have a total submission count less than or equal to the maximum allowed as of the merge date. The maximum allowed is the number of submissions per day multiplied by the number of days the competition has been running.
\r\n
There is no maximum team size.
\r\nYou may submit a maximum of 5 entries per day.
\r\nYou may select up to 2 final submissions for judging.
\r\nStart Date: 12/15/2014 2:00 PM UTC
\r\nMerger Deadline: 3/9/2015 11:59 PM UTC
\r\nEntry Deadline: 3/9/2015 11:59 PM UTC
\r\nEnd Date: 3/16/2015 11:59 PM UTC
\r\n\t\r\nCOMPETITION NAME (the 'Competition'): National Data Science Bowl
\r\nCOMPETITION SPONSOR: Oregon State University and Booz Allen Hamilton Inc.
\r\nCOMPETITION WEBSITE: https://www.kaggle.com/c/datasciencebowl
\r\nPRIZES: As specified on https://www.kaggle.com/c/datasciencebowl/details/prizes
\r\nWINNER LICENSE TYPE: Non-exclusive License
\r\nMAXIMUM TEAM SIZE: None
\r\nMAXIMUM SUBMISSIONS PER DAY: 5
\r\nThese are the complete, official rules for the Competition (the 'Competition Rules') and incorporate by reference the contents of the Competition Website listed above.
\r\nBy downloading a dataset linked from the Competition Website, submitting an entry to this Competition, or joining a Team in this Competition, you are agreeing to be bound by these Competition Rules which constitute a binding agreement between you and the Competition Co-Sponsors and or its designee.
\r\nThe Competition is sponsored by the Competition Co-Sponsors listed above and hosted on the Sponsors’ behalf by Kaggle Inc. ('Kaggle'). The Competition will run according to the dates listed on the Competition Website.
\r\nEach registered individual or Team is referred to as a Participant. You may only participate using a single, unique Kaggle account registered at http://www.kaggle.com. Participating using more than one Kaggle account per individual Participant is a breach of these Competition Rules and Competition Co-Sponsors reserves the right to disqualify any Participant (or Team including Participant) who is found to breach these Competition Rules.
\r\nThe Competition is open to all individuals over the age of 18 at the time of entry and to all validly formed legal entities that have not declared or been declared in bankruptcy.
\r\nOfficers, directors, employees and advisory board members (and their immediate families and members of the same household) of the Competition Co-Sponsors, Kaggle and their respective affiliates, subsidiaries, contractors (with the express exception of Kaggle's authorized Kaggle Community Evangelists), agents, judges and advertising and promotion agencies are not eligible to win a prize in the Competition.
\r\nYou are not eligible to receive any Prize in the Competition if you are a resident of a country designated by the United States Treasury’s Office of Foreign Assets Control (see <http://www.treasury.gov/resource-center/sanctions/SDN-List/Pages/default.aspx> for additional information).
\r\n'Submission' means the material submitted by you in the manner and format specified on the Website via the Submission form on the Website. You may submit up to the maximum number of Submissions per day as specified above. All Submissions must be uploaded to the Website in the manner and format specified on the Website. Submissions must be received prior to the Competition deadline and adhere to the guidelines for Submissions specified on the Website.
\r\nSubmissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.
\r\nIf the Competition is a multi-stage Competition with temporally separate training data and/or leaderboard data, one or more valid Submissions must be made and selected during each stage of the Competition in the manner described on the Competition Website.
\r\n
'Data' means the Data or Datasets linked from the Competition Website for the purpose of use by Participants in the Competition. For the avoidance of doubt, Data is deemed for the purpose of these Competition Rules to include any prototype or executable code provided to Participants by Kaggle or Competition Co-Sponsors via the Website. Participants must use the Data only as permitted by these Competition Rules and any associated data use rules specified on the Competition Website.
Unless otherwise permitted by the terms of the Competition Website, Participants must use the Data solely for the purpose and duration of the Competition, including but not limited to reading and learning from the Data, analyzing the Data, modifying the Data and generally preparing your Submission and any underlying models and participating in forum discussions on the Website. Participants agree to use suitable measures to prevent persons who have not formally agreed to these Competition Rules from gaining access to the Data and agree not to transmit, duplicate, publish, redistribute or otherwise provide or make available the Data to any party not participating in the Competition. Participants agree to notify Kaggle immediately upon learning of any possible unauthorized transmission or unauthorized access of the Data and agree to work with Kaggle to rectify any unauthorized transmission. Participants agree that participation in the Competition shall not be construed as having or being granted a license (expressly, by implication, estoppel, or otherwise) under, or any right of ownership in, any of the Data.
\r\nUnless otherwise expressly stated on the Competition Website, Participants must not use data other than the Data to develop and test their models and Submissions. Competition Co-Sponsors reserves the right in its sole discretion to disqualify any Participant who Competition Co-Sponsors discovers has undertaken or attempted to undertake the use of data other than the Data, or who uses the Data other than as permitted according to the Competition Website and in these Competition Rules, in the course of the Competition.
\r\nParticipants are prohibited from privately sharing source or executable code developed in connection with or based upon the Data, and any such sharing is a breach of these Competition Rules and may result in disqualification.
\r\nParticipants are permitted to publicly share source or executable code developed in connection with or based upon the Data, or otherwise relevant to the Competition, provided that such sharing does not violate the intellectual property rights of any third party. By so sharing, the sharing Participant is thereby deemed to have licensed the shared code under the MIT License (an open source software license commonly described at <http://opensource.org/licenses/MIT>).
\r\nA Submission will be ineligible to win a prize if it was developed using code containing or depending on software licensed under an open source license:
\r\n* other than an Open Source Initiative-approved license (see <http://opensource.org/>); or
* an open source license that prohibits commercial use.
This Competition is a challenge of skill and the final results are determined solely by leaderboard ranking on the private leaderboard (subject to compliance with these Competition Rules). Participants' scores and ranks on the Competition Website at any given stage of the Competition will be based on the evaluation metric described on the Competition Website, as determined by applying the predictions in the Submission to the ground truth of a validation dataset whose instances were a fixed set sampled from the Data.
\r\nThe evaluation metric used for scoring and ranking Submissions will be displayed on the Competition Website.
\r\nPrize awards are subject to verification of eligibility and compliance with these Competition Rules. All decisions of the Competition Co-Sponsors and judges will be final and binding on all matters relating to this Competition. Competition Co-Sponsors reserves the right to examine the Submission and any associated code or documentation for compliance with these Competition Rules. In the event that the Submission demonstrates a breach of these Competition Rules, Competition Co-Sponsors may at its discretion take either of the following actions:
\r\n• disqualify your Submission(s); or
• require that you remediate within one week all issues identified in your Submission(s) (including, without limitation, the resolution of license conflicts, the fulfillment of all obligations required by software licenses, and the removal of any software that violates the software restrictions).
A tie between two or more valid and identically ranked submissions will be resolved in favour of the tied submission that was submitted first.
\r\nA Participant may decline to be nominated as a Winner by notifying Kaggle directly within one week following the Competition deadline, in which case the declining Participant forgoes any prize or other features associated with winning the Competition. Kaggle reserves the right to disqualify a Participant who so declines at Kaggle's sole discretion if Kaggle deems disqualification appropriate.
\r\nAs a condition of receipt of the Prize, the Prize winner(s) must deliver the final model’s software code as used to generate the winning Submission and associated documentation (consistent with the winning model documentation template available on the Kaggle wiki at https://www.kaggle.com/wiki/WinningModelDocumentationTemplate) to the Competition Co-Sponsors. The delivered software code must be capable of generating the winning Submission and contain a description of resources required to build and/or run the executable code successfully.
\r\nIf a potential winning Participant or Team member is a U.S. citizen, potential winner must also sign and return an IRS W-9 form, or if a foreign resident, an IRS W-8BEN form, within the stated time in order to claim the prize.
\r\nPotential winners will be disqualified and the prize may be awarded to an alternate winner (the next-ranked qualified Participant on the leaderboard) if
\r\n• the required documentation is not returned within 14 days after receipt of any request to provide documentation;
• prize notification letter/email or prize is returned as undeliverable; or
• potential winner or winning Team member is disqualified for any reason.
By accepting any Prize, each Participant receiving a Prize thereby agrees to use of his/her name, address, likeness and/or Prize information by Kaggle and Competition Co-Sponsors for promotional purposes in any medium without additional compensation.
\r\nAs a further condition of receipt of a Prize, each winning Participant thereby licenses their winning Submission and the source code used to generate the Submission according to the Winner License Type specified above (note: if no Winner License Type is specified above, the Winner License Type is deemed to be Non-Exclusive License).
\r\nIf the Winner License Type for the Competition (see Winner License Type above) is a Non-Exclusive License then each Winner by accepting a Prize thereby:
\r\n• grants to Competition Co-Sponsors and its designees a worldwide, non-exclusive, sub-licensable, transferable, fully paid-up, royalty-free, perpetual, irrevocable right to use, not use, reproduce, distribute, create derivative works of, publicly perform, publicly display, digitally perform, make, have made, sell, offer for sale and/or import their winning Submission and the source code used to generate the Submission, in any media now known or hereafter developed, for any purpose whatsoever, commercial or otherwise, without further approval by or payment to Participant; and
• represents that he/she/it has the unrestricted right to grant that license.
If the Winner License Type for the Competition (see Winner License Type above) is Open Source License, then each Winner by accepting a Prize thereby:
\r\n* licenses their winning Submission and the source code used to generate the Submission under the MIT License (an open source software license commonly described at <http://opensource.org/licenses/MIT>), unless another open source license is chosen explicitly from the list of those approved by the Open Source Initiative at <http://opensource.org/licenses/>; and
* represents that he/she/it has the unrestricted right to grant that license.
Participating using more than one Kaggle account is deemed cheating and, if discovered, will result in disqualification from the Competition and any other affected Competitions and may result in banning or deactivation of affected Kaggle accounts.
\r\nAfter verification of eligibility, each Prize winner will receive the prize in the form of a check or wire transfer made out to the Prize winner (if an individual, or to the individual Team members if a Team). Allow 30 days from final confirmation for Prize delivery. Any winners who are U.S. citizens will receive an IRS 1099 form in the amount of their prize at the appropriate time. Prize winners are responsible for any taxes, fees or other liability resulting from their receipt of a Prize.
\r\nMultiple individuals or entities may collaborate as a team (“Team”). You may not participate on more than one Team. Each Team member must be a single individual operating a separate Kaggle account. You must register individually for the Competition before joining a Team. You must confirm your Team membership to make it official by responding to the Team notification message which will be sent to your Account.
\r\nTeam membership may not exceed the Maximum Team Size.
\r\nTeams may be permitted to merge at Kaggle’s discretion, so long as the merged Team meets all requirements in these Competition Rules. Kaggle will review any merger request and will either approve or reject the request within three business days. Merger requests may be rejected if the combined number of Entries made by the merging Teams exceeds the number of Entries permissible at the date of the merger request. Team merger requests will not be permitted within seven days of any deadline listed on the Competition Website.
\r\nIf a Team wins a monetary Prize, Competition Co-Sponsors will allocate the Prize money in even shares between the Team members unless the Team unanimously contacts Kaggle via the Competition Website within three business days following the Submission deadline to request an alternative prize distribution.
\r\nBy registering, you agree that (a) your Account is complete, correct and accurate and (b) your registration may be rejected or terminated and all Entries submitted by you and/or your Team may be disqualified if any of the information in your Account is (or Competition Co-Sponsors has reasonable grounds to believe it is) incomplete, incorrect or inaccurate. You are solely responsible for your Account. All registration information is deemed collected in the United States.
\r\nParticipation is subject to all federal, state and local laws and regulations. Void where prohibited or restricted by law. You are responsible for checking applicable laws and regulations in your jurisdiction before participating in the Competition to make sure that your participation is legal. You are responsible for all taxes and reporting related to any award that you may receive as part of the Competition. You are responsible for abiding by your employer's policies regarding participation in the Competition. Competition Co-Sponsors disclaims any and all liability or responsibility for disputes arising between you and your employer related to this Competition.
\r\nEach Participant is solely responsible for all equipment, including but not necessarily limited to a computer and internet connection necessary to access the Website and to develop and upload any Submission, and any telephone, data, hosting or other service fees associated with such access, as well as all costs incurred by or behalf of the Entrant in participating in the Competition.
\r\nBy entering a Submission, you represent and warrant that all information you enter on the Website is true and complete to the best of your knowledge, that you have the right and authority to make the Submission (including any underlying code and model) on your own behalf or on behalf of the persons and entities that you specify within the Submission, and that your Submission:
\r\n• is your own original work, or is used by permission, in which case full and proper credit and identify is given and the third party contributions are clearly identified within your Submission;
• does not contain confidential information or trade secrets and is not the subject of a registered patent or pending patent application;
• does not violate or infringe upon the patent rights, industrial design rights, copyrights, trademarks, rights of privacy, publicity or other intellectual property or other rights of any person or entity;
• does not contain malicious code, such as viruses, timebombs, cancelbots, worms, Trojan horses or other potentially harmful programs or other material or information;
• does not and will not violate any applicable law, statute, ordinance, rule or regulation;
• does not trigger any reporting or royalty obligation to any third party; and
• was not previously published and has not won any other prize/award.
A breach of any of these warranties will result in the corresponding Submission being invalid.
By participating in the Competition, each Participant agrees to release, indemnify and hold harmless Competition Co-Sponsors, Kaggle, their respective affiliates, subsidiaries, advertising and promotions agencies, as applicable, and each of their respective agents, representatives, officers, directors, shareholders, and employees from and against any injuries, losses, damages, claims, actions and any liability of any kind resulting from or arising out of your participation in or association with the Competition. Competition Co-Sponsors is not responsible for any miscommunications such as technical failures related to computer, telephone, cable, and unavailable network or server connections, related technical failures, or other failures related to hardware, software or virus, or incomplete, late or misdirected Submissions. Competition Co-Sponsors reserves the right to cancel, modify or suspend the Competition should any computer virus, bug or other technical difficulty or other causes beyond the control of Competition Co-Sponsors corrupt the administration, security or proper play of the Competition, and to determine winners from among Submission not affected by the corruption, if any, in its sole discretion.
\r\nNeither Kaggle nor Competition Co-Sponsors are responsible for (a) late, lost, stolen, damaged, garbled, incomplete, incorrect or misdirected Entries or other communications, (b) errors, omissions, interruptions, deletions, defects, or delays in operations or transmission of information, in each case whether arising by way of technical or other failures or malfunctions of computer hardware, software, communications devices, or transmission lines, or (c) data corruption, theft, destruction, unauthorized access to or alteration of Submission materials, loss or otherwise. Neither Kaggle nor Competition Co-Sponsors are responsible for electronic communications or emails which are undeliverable as a result of any form of active or passive filtering of any kind, or insufficient space in any email account to receive email messages. Competition Co-Sponsors disclaims any liability for damage to any computer system resulting from participation in, or accessing or downloading information in connection with, the Competition.
\r\nCompetition Co-Sponsors reserves the right to modify, remove or add Data to the Website upon notice on the forum on the Competition Website or via email using the email address associated with their Accounts. NEITHER KAGGLE NOR COMPETITION CO-SPONSORS ARE RESPONSIBLE FOR ANY FAILURE OF A PARTICIPANT TO RECEIVE DATA CHANGES.
\r\nThe invalidity or unenforceability of any provision of these Competition Rules shall not affect the validity or enforceability of any other provision. In the event that any provision is determined to be invalid or otherwise unenforceable or illegal, these Competition Rules shall otherwise remain in effect and be construed in accordance with their terms as if the invalid or illegal provision was not contained herein.
\r\nYou agree that these terms and the relationship between you and Competition Co-Sponsors shall be governed by the laws of the State of California and the United States of America.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Timeline", "content": "All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The organizers reserve the right to update the contest timeline if they deem it necessary.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "About The Ndsb", "content": "Are you interested in solving the world’s toughest analytic challenges? Do you have what it takes to be a top data scientist? Join the data science community and compete for top honors in the National Data Science Bowl.
\r\nFor us, data science is more than a skill or profession. It is a calling and a way of life. It rewards grit as much as talent. Failure, curiosity, and small successes lead to larger advances. Data science grants the power of discovery to the individual.
\r\nBecome part of this global movement. Enter the first-ever National Data Science Bowl. See if you have what it takes to observe the right patterns, ask the right questions and, in turn, make our beautiful and complex world a little easier to understand and a better place to live. Whether a seasoned veteran or a newcomer, the NDSB provides an opportunity to advance the art of data science, benefit society, and inspire change across the globe. The competition will be challenging but it will also be incredibly rewarding. At stake? The very health of our oceans.
\r\nAre you ready to change the world?
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Tutorial", "content": "written by Aaron Sander, Data Scientist, Booz Allen Hamilton
\r\nIn this tutorial, we will go step-by-step through a simple model to distinguish different types of plankton and demonstrate some tools for exploring the image dataset. We will start by going through an example of one image to show how you could choose to develop a metric based on the shape of the object within the image. First, we import the necessary modules from scikit-image, matplotlib, scikit-learn, and numpy. If you don't currently have python installed, you can get the Anaconda distribution that includes all of the referenced packages below.
\r\nIn [1]:
\r\n#Import libraries for doing image analysis\r\n
from skimage.io import imread
from skimage.transform import resize
from sklearn.ensemble import RandomForestClassifier as RF
import glob
import os
from sklearn import cross_validation
from sklearn.cross_validation import StratifiedKFold as KFold
from sklearn.metrics import classification_report
from matplotlib import pyplot as plt
from matplotlib import colors
from pylab import cm
from skimage import segmentation
from skimage.morphology import watershed
from skimage import measure
from skimage import morphology
import numpy as np
import pandas as pd
from scipy import ndimage
from skimage.feature import peak_local_max
# make graphics inline
%matplotlib inline
In [2]:
\r\nimport warnings\r\n
warnings.filterwarnings(\"ignore\")
The training data is organized in a series of subdirectories that contain examples for the each class of interest. We will store the list of directory names to aid in labelling the data classes for training and testing purposes.
\r\nIn [3]:
\r\n# get the classnames from the directory structure\r\n
directory_names = list(set(glob.glob(os.path.join(\"competition_data\",\"train\", \"*\"))\\
).difference(set(glob.glob(os.path.join(\"competition_data\",\"train\",\"*.*\")))))
We will develop our feature on one image example and examine each step before calculating the feature across the distribution of classes.
\r\nIn [4]:
\r\n# Example image\r\n
# This example was chosen for because it has two noncontinguous pieces
# that will make the segmentation example more illustrative
example_file = glob.glob(os.path.join(directory_names[5],\"*.jpg\"))[9]
print example_file
im = imread(example_file, as_grey=True)
plt.imshow(im, cmap=cm.gray)
plt.show()
competition_data/train/acantharia_protist/101574.jpg
\r\n![](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/3978/media/plot1.png\")
To create the features of interest, we will need to prepare the images by completing a few preprocessing procedures. We will step through some common image preprocessing actions: thresholding the images, segmenting the images, and extracting region properties. Using the region properties, we will create features based on the intrinsic properties of the classes, which we expect will allow us discriminate between them. Let's walk through the process of adding one such feature for the ratio of the width by length of the object of interest.
\r\nFirst, we begin by thresholding the image on the the mean value. This will reduce some of the noise in the image. Then, we apply a three step segmentation process: first we dilate the image to connect neighboring pixels, then we calculate the labels for connected regions, and finally we apply the original threshold to the labels to label the original, undilated regions.
\r\nIn [5]:
\r\n# First we threshold the image by only taking values greater than the mean to reduce noise in the image\r\n
# to use later as a mask
f = plt.figure(figsize=(12,3))
imthr = im.copy()
imthr = np.where(im > np.mean(im),0.,1.0)
sub1 = plt.subplot(1,4,1)
plt.imshow(im, cmap=cm.gray)
sub1.set_title(\"Original Image\")
sub2 = plt.subplot(1,4,2)
plt.imshow(imthr, cmap=cm.gray_r)
sub2.set_title(\"Thresholded Image\")
imdilated = morphology.dilation(imthr, np.ones((4,4)))
sub3 = plt.subplot(1, 4, 3)
plt.imshow(imdilated, cmap=cm.gray_r)
sub3.set_title(\"Dilated Image\")
labels = measure.label(imdilated)
labels = imthr*labels
labels = labels.astype(int)
sub4 = plt.subplot(1, 4, 4)
sub4.set_title(\"Labeled Image\")
plt.imshow(labels)
![](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/3978/media/plot2.png\")
With the image segmented into different parts, we would like to choose the largest non-background part to compute our metric. We would like to select the largest segment as the likely object of interest for classification purposes. We loop through the available regions and select the one with the largest area. There are many properties available within the regions that you can explore for creating new features. Look at the documentation for regionprops for inspiration.
\r\nIn [6]:
\r\n# calculate common region properties for each region within the segmentation\r\nregions = measure.regionprops(labels)\r\n# find the largest nonzero region\r\ndef getLargestRegion(props=regions, labelmap=labels, imagethres=imthr):\r\n regionmaxprop = None\r\n for regionprop in props:\r\n # check to see if the region is at least 50% nonzero\r\n if sum(imagethres[labelmap == regionprop.label])*1.0/regionprop.area < 0.50:\r\n continue\r\n if regionmaxprop is None:\r\n regionmaxprop = regionprop\r\n if regionmaxprop.filled_area < regionprop.filled_area:\r\n regionmaxprop = regionprop\r\n return regionmaxprop\r\n\r\n
The results for our test image are shown below. The segmentation picked one region and we use that region to calculate our ratio metric.
\r\nIn [7]:
\r\nregionmax = getLargestRegion()\r\n
plt.imshow(np.where(labels == regionmax.label,1.0,0.0))
plt.show()
![](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/3978/media/plot3.png\")
In [8]:
\r\nprint regionmax.minor_axis_length/regionmax.major_axis_length\r\n
0.144141699631
\r\nNow, we collect the previous steps together in a function to make it easily repeatable.
\r\nIn [9]:
\r\ndef getMinorMajorRatio(image):\r\n image = image.copy()\r\n # Create the thresholded image to eliminate some of the background\r\n imagethr = np.where(image > np.mean(image),0.,1.0)\r\n\r\n #Dilate the image\r\n imdilated = morphology.dilation(imagethr, np.ones((4,4)))\r\n\r\n # Create the label list\r\n label_list = measure.label(imdilated)\r\n label_list = imagethr*label_list\r\n label_list = label_list.astype(int)\r\n \r\n region_list = measure.regionprops(label_list)\r\n maxregion = getLargestRegion(region_list, label_list, imagethr)\r\n \r\n # guard against cases where the segmentation fails by providing zeros\r\n ratio = 0.0\r\n if ((not maxregion is None) and (maxregion.major_axis_length != 0.0)):\r\n ratio = 0.0 if maxregion is None else maxregion.minor_axis_length*1.0 / maxregion.major_axis_length\r\n return ratio\r\n\r\n
With our code for the ratio of minor to major axis, let's add the raw pixel values to the list of features for our dataset. In order to use the pixel values in a model for our classifier, we need a fixed length feature vector, so we will rescale the images to be constant size and add the fixed number of pixels to the feature vector.
\r\nTo create the feature vectors, we will loop through each of the directories in our training data set and then loop over each image within that class. For each image, we will rescale it to 25 x 25 pixels and then add the rescaled pixel values to a feature vector, X. The last feature we include will be our width-to-length ratio. We will also create the class label in the vector y, which will have the true class label for each row of the feature vector, X.
\r\nIn [10]:
\r\n# Rescale the images and create the combined metrics and training labels\r\n\r\n#get the total training images\r\nnumberofImages = 0\r\nfor folder in directory_names:\r\n for fileNameDir in os.walk(folder): \r\n for fileName in fileNameDir[2]:\r\n # Only read in the images\r\n if fileName[-4:] != \".jpg\":\r\n continue\r\n numberofImages += 1\r\n\r\n# We'll rescale the images to be 25x25\r\nmaxPixel = 25\r\nimageSize = maxPixel * maxPixel\r\nnum_rows = numberofImages # one row for each image in the training dataset\r\nnum_features = imageSize + 1 # for our ratio\r\n\r\n# X is the feature vector with one row of features per image\r\n# consisting of the pixel values and our metric\r\nX = np.zeros((num_rows, num_features), dtype=float)\r\n# y is the numeric class label \r\ny = np.zeros((num_rows))\r\n\r\nfiles = []\r\n# Generate training data\r\ni = 0 \r\nlabel = 0\r\n# List of string of class names\r\nnamesClasses = list()\r\n\r\nprint \"Reading images\"\r\n# Navigate through the list of directories\r\nfor folder in directory_names:\r\n # Append the string class name for each class\r\n currentClass = folder.split(os.pathsep)[-1]\r\n namesClasses.append(currentClass)\r\n for fileNameDir in os.walk(folder): \r\n for fileName in fileNameDir[2]:\r\n # Only read in the images\r\n if fileName[-4:] != \".jpg\":\r\n continue\r\n \r\n # Read in the images and create the features\r\n nameFileImage = \"{0}{1}{2}\".format(fileNameDir[0], os.sep, fileName) \r\n image = imread(nameFileImage, as_grey=True)\r\n files.append(nameFileImage)\r\n axisratio = getMinorMajorRatio(image)\r\n image = resize(image, (maxPixel, maxPixel))\r\n \r\n # Store the rescaled image pixels and the axis ratio\r\n X[i, 0:imageSize] = np.reshape(image, (1, imageSize))\r\n X[i, imageSize] = axisratio\r\n \r\n # Store the classlabel\r\n y[i] = label\r\n i += 1\r\n # report progress for each 5% done \r\n report = [int((j+1)*num_rows/20.) for j in range(20)]\r\n if i in report: print np.ceil(i *100.0 / num_rows), \"% done\"\r\n label += 1\r\n\r\nReading images
5.0 % done
10.0 % done
15.0 % done
20.0 % done
25.0 % done
30.0 % done
35.0 % done
40.0 % done
45.0 % done
50.0 % done
55.0 % done
60.0 % done
65.0 % done
70.0 % done
75.0 % done
80.0 % done
85.0 % done
90.0 % done
95.0 % done
100.0 % done
Now that we have calculated the width-to-length ratio metric for all the images, we can look at the class separation to see how well our feature performs. We'll compare pairs of the classes' distributions by plotting each pair of classes. While this will not cover the whole space of hundreds of possible combinations, it will give us a feel for how similar or dissimilar different classes are in this feature, and the class distributions should be comparable across subplots.
\r\nIn [12]:
\r\n# Loop through the classes two at a time and compare their distributions of the Width/Length Ratio\r\n\r\n#Create a DataFrame object to make subsetting the data on the class \r\ndf = pd.DataFrame({\"class\": y[:], \"ratio\": X[:, num_features-1]})\r\n\r\nf = plt.figure(figsize=(30, 20))\r\n#we suppress zeros and choose a few large classes to better highlight the distributions.\r\ndf = df.loc[df[\"ratio\"] > 0]\r\nminimumSize = 20 \r\ncounts = df[\"class\"].value_counts()\r\nlargeclasses = [int(x) for x in list(counts.loc[counts > minimumSize].index)]\r\n# Loop through 40 of the classes \r\nfor j in range(0,40,2):\r\n subfig = plt.subplot(4, 5, j/2 +1)\r\n # Plot the normalized histograms for two classes\r\n classind1 = largeclasses[j]\r\n classind2 = largeclasses[j+1]\r\n n, bins,p = plt.hist(df.loc[df[\"class\"] == classind1][\"ratio\"].values,\\\r\n alpha=0.5, bins=[x*0.01 for x in range(100)], \\\r\n label=namesClasses[classind1].split(os.sep)[-1], normed=1)\r\n\r\n n2, bins,p = plt.hist(df.loc[df[\"class\"] == (classind2)][\"ratio\"].values,\\\r\n alpha=0.5, bins=bins, label=namesClasses[classind2].split(os.sep)[-1],normed=1)\r\n subfig.set_ylim([0.,10.])\r\n plt.legend(loc='upper right')\r\n plt.xlabel(\"Width/Length Ratio\")\r\n\r\n![](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/3978/media/plot4.png\")
From the (truncated) figure above, you will see some cases where the classes are well separated and others were they are not. It is typical that one single feature will not allow you to completely separate more than thirty distinct classes. You will need to be creative in coming up with additional metrics to discriminate between all the classes.
\r\nWe choose a random forest model to classify the images. Random forests perform well in many classification tasks and have robust default settings. We will give a brief description of a random forest model so that you can understand its two main free parameters: n_estimators and max_features.
\r\nA random forest model is an ensemble model of n_estimators number of decision trees. During the training process, each decision tree is grown automatically by making a series of conditional splits on the data. At each split in the decision tree, a random sample of max_features number of features is chosen and used to make a conditional decision on which of the two nodes that the data will be grouped in. The best condition for the split is determined by the split that maximizes the class purity of the nodes directly below. The tree continues to grow by making additional splits until the leaves are pure or the leaves have less than the minimum number of samples for a split (in sklearn default for min_samples_split is two data points). The final majority class purity of the terminal nodes of the decision tree are used for making predictions on what class a new data point will belong. Then, the aggregate vote across the forest determines the class prediction for new samples.
\r\nWith our training data consisting of the feature vector X and the class label vector y, we will now calculate some class metrics for the performance of our model, by class and overall. First, we train the random forest on all the available data and let it perform the 5-fold cross validation. Then we perform the cross validation using the KFold method, which splits the data into train and test sets, and a classification report. The classification report provides a useful list of performance metrics for your classifier vs. the internal metrics of the random forest module.
\r\nIn [19]:
\r\nprint \"Training\"\r\n
# n_estimators is the number of decision trees
# max_features also known as m_try is set to the default value of the square root of the number of features
clf = RF(n_estimators=100, n_jobs=3);
scores = cross_validation.cross_val_score(clf, X, y, cv=5, n_jobs=1);
print \"Accuracy of all classes\"
print np.mean(scores)
Training
Accuracy of all classes
0.446073202468
In [14]:
\r\nkf = KFold(y, n_folds=5)\r\ny_pred = y * 0\r\nfor train, test in kf:\r\n X_train, X_test, y_train, y_test = X[train,:], X[test,:], y[train], y[test]\r\n clf = RF(n_estimators=100, n_jobs=3)\r\n clf.fit(X_train, y_train)\r\n y_pred[test] = clf.predict(X_test)\r\nprint classification_report(y, y_pred, target_names=namesClasses)\r\n
precision recall f1-score support
\r\ncompetition_data/train/appendicularian_slight_curve 0.32 0.43 0.37 532
competition_data/train/hydromedusae_shapeB 0.33 0.08 0.13 150
competition_data/train/hydromedusae_shapeA 0.44 0.80 0.57 412
competition_data/train/siphonophore_other_parts 0.00 0.00 0.00 29
competition_data/train/tunicate_doliolid_nurse 0.24 0.07 0.11 417
competition_data/train/acantharia_protist 0.40 0.85 0.54 889
competition_data/train/hydromedusae_narco_young 0.23 0.08 0.12 336
competition_data/train/fish_larvae_deep_body 0.00 0.00 0.00 10
competition_data/train/hydromedusae_haliscera_small_sideview 0.00 0.00 0.00 9
competition_data/train/echinoderm_larva_pluteus_early 0.46 0.26 0.33 92
competition_data/train/siphonophore_calycophoran_rocketship_young 0.33 0.23 0.27 483
competition_data/train/chaetognath_sagitta 0.38 0.20 0.26 694
competition_data/train/shrimp-like_other 0.00 0.00 0.00 52
competition_data/train/hydromedusae_liriope 0.00 0.00 0.00 19
competition_data/train/heteropod 0.00 0.00 0.00 10
competition_data/train/appendicularian_straight 0.50 0.04 0.07 242
competition_data/train/chaetognath_non_sagitta 0.57 0.74 0.65 815
competition_data/train/trochophore_larvae 0.00 0.00 0.00 29
competition_data/train/copepod_cyclopoid_oithona_eggs 0.54 0.80 0.64 1189
competition_data/train/ctenophore_cestid 0.00 0.00 0.00 113
competition_data/train/pteropod_butterfly 0.60 0.08 0.15 108
competition_data/train/hydromedusae_sideview_big 0.00 0.00 0.00 76
competition_data/train/siphonophore_calycophoran_rocketship_adult 0.36 0.04 0.07 135
competition_data/train/shrimp_caridean 0.68 0.27 0.38 49
competition_data/train/hydromedusae_typeD 0.00 0.00 0.00 43
competition_data/train/appendicularian_s_shape 0.37 0.54 0.44 696
competition_data/train/crustacean_other 0.25 0.11 0.15 201
competition_data/train/fish_larvae_myctophids 0.55 0.54 0.54 114
competition_data/train/hydromedusae_partial_dark 0.66 0.27 0.38 190
competition_data/train/copepod_calanoid_eggs 0.80 0.19 0.31 173
competition_data/train/stomatopod 0.00 0.00 0.00 24
competition_data/train/siphonophore_physonect_young 0.00 0.00 0.00 21
competition_data/train/hydromedusae_solmundella 0.77 0.16 0.27 123
competition_data/train/ephyra 0.00 0.00 0.00 14
competition_data/train/hydromedusae_bell_and_tentacles 0.00 0.00 0.00 75
competition_data/train/pteropod_triangle 0.00 0.00 0.00 65
competition_data/train/hydromedusae_h15 0.47 0.20 0.28 35
competition_data/train/diatom_chain_string 0.55 0.91 0.69 519
competition_data/train/hydromedusae_narcomedusae 0.00 0.00 0.00 132
competition_data/train/copepod_calanoid_large 0.48 0.41 0.44 286
competition_data/train/radiolarian_colony 0.44 0.23 0.30 158
competition_data/train/tunicate_partial 0.61 0.95 0.75 352
competition_data/train/invertebrate_larvae_other_B 0.00 0.00 0.00 24
competition_data/train/invertebrate_larvae_other_A 0.00 0.00 0.00 14
competition_data/train/echinoderm_larva_pluteus_brittlestar 0.60 0.08 0.15 36
competition_data/train/siphonophore_calycophoran_abylidae 0.21 0.04 0.07 212
competition_data/train/euphausiids_young 0.00 0.00 0.00 38
competition_data/train/hydromedusae_aglaura 0.00 0.00 0.00 127
competition_data/train/protist_dark_center 0.00 0.00 0.00 108
competition_data/train/trichodesmium_bowtie 0.44 0.69 0.54 708
competition_data/train/radiolarian_chain 0.58 0.07 0.12 287
competition_data/train/protist_fuzzy_olive 0.75 0.78 0.76 372
competition_data/train/polychaete 0.75 0.02 0.04 131
competition_data/train/copepod_calanoid 0.39 0.55 0.46 681
competition_data/train/amphipods 0.00 0.00 0.00 49
competition_data/train/acantharia_protist_big_center 0.00 0.00 0.00 13
competition_data/train/copepod_calanoid_octomoms 0.00 0.00 0.00 49
competition_data/train/protist_other 0.36 0.68 0.47 1172
competition_data/train/hydromedusae_other 0.00 0.00 0.00 12
competition_data/train/tunicate_salp 0.59 0.75 0.66 236
competition_data/train/siphonophore_calycophoran_sphaeronectes_stem 0.00 0.00 0.00 57
competition_data/train/trichodesmium_puff 0.71 0.93 0.80 1979
competition_data/train/artifacts 0.54 0.85 0.66 393
competition_data/train/fish_larvae_leptocephali 0.00 0.00 0.00 31
competition_data/train/echinoderm_larva_seastar_bipinnaria 0.54 0.56 0.55 385
competition_data/train/chordate_type1 0.52 0.56 0.54 77
competition_data/train/shrimp_zoea 0.54 0.20 0.29 174
competition_data/train/fish_larvae_very_thin_body 0.00 0.00 0.00 16
competition_data/train/ctenophore_cydippid_no_tentacles 0.00 0.00 0.00 42
competition_data/train/appendicularian_fritillaridae 0.00 0.00 0.00 16
competition_data/train/siphonophore_physonect 0.00 0.00 0.00 128
competition_data/train/trichodesmium_tuft 0.37 0.46 0.41 678
competition_data/train/fecal_pellet 0.32 0.29 0.31 511
competition_data/train/hydromedusae_shapeA_sideview_small 0.37 0.10 0.16 274
competition_data/train/siphonophore_calycophoran_sphaeronectes 0.44 0.11 0.17 179
competition_data/train/artifacts_edge 0.89 0.75 0.81 170
competition_data/train/ctenophore_cydippid_tentacles 0.00 0.00 0.00 53
competition_data/train/copepod_cyclopoid_oithona 0.50 0.61 0.55 899
competition_data/train/siphonophore_partial 0.00 0.00 0.00 30
competition_data/train/tunicate_doliolid 0.26 0.17 0.20 439
competition_data/train/copepod_other 0.00 0.00 0.00 24
competition_data/train/unknown_blobs_and_smudges 0.27 0.13 0.18 317
competition_data/train/shrimp_sergestidae 0.72 0.08 0.15 153
competition_data/train/hydromedusae_solmaris 0.40 0.51 0.44 703
competition_data/train/copepod_calanoid_flatheads 0.50 0.02 0.04 178
competition_data/train/echinoderm_larva_seastar_brachiolaria 0.61 0.76 0.68 536
competition_data/train/copepod_calanoid_eucalanus 0.88 0.07 0.13 96
competition_data/train/ctenophore_lobate 0.74 0.45 0.56 38
competition_data/train/detritus_filamentous 0.12 0.01 0.01 394
competition_data/train/jellies_tentacles 0.40 0.01 0.03 141
competition_data/train/detritus_blob 0.19 0.06 0.09 363
competition_data/train/chaetognath_other 0.43 0.74 0.54 1934
competition_data/train/copepod_cyclopoid_copilia 0.00 0.00 0.00 30
competition_data/train/copepod_calanoid_large_side_antennatucked 0.50 0.23 0.31 106
competition_data/train/trichodesmium_multiple 1.00 0.07 0.14 54
competition_data/train/fish_larvae_thin_body 0.00 0.00 0.00 64
competition_data/train/diatom_chain_tube 0.39 0.41 0.40 500
competition_data/train/tunicate_salp_chains 0.00 0.00 0.00 73
competition_data/train/protist_star 0.84 0.58 0.68 113
competition_data/train/fish_larvae_medium_body 0.49 0.32 0.39 85
competition_data/train/hydromedusae_narco_dark 0.00 0.00 0.00 23
competition_data/train/hydromedusae_haliscera 0.54 0.40 0.46 229
competition_data/train/hydromedusae_typeE 0.00 0.00 0.00 14
competition_data/train/hydromedusae_typeF 0.54 0.21 0.31 61
competition_data/train/echinoderm_larva_pluteus_urchin 0.59 0.26 0.36 88
competition_data/train/siphonophore_calycophoran_sphaeronectes_young 0.38 0.04 0.07 247
competition_data/train/protist_noctiluca 0.57 0.50 0.53 625
competition_data/train/copepod_calanoid_frillyAntennae 1.00 0.03 0.06 63
competition_data/train/echinoderm_seacucumber_auricularia_larva 0.00 0.00 0.00 96
competition_data/train/tornaria_acorn_worm_larvae 0.84 0.42 0.56 38
competition_data/train/detritus_other 0.26 0.33 0.29 914
competition_data/train/unknown_sticks 0.43 0.06 0.10 175
competition_data/train/unknown_unclassified 0.14 0.01 0.02 425
competition_data/train/pteropod_theco_dev_seq 0.00 0.00 0.00 13
competition_data/train/acantharia_protist_halo 1.00 0.03 0.05 71
competition_data/train/hydromedusae_typeD_bell_and_tentacles 0.75 0.11 0.19 56
competition_data/train/echinoderm_larva_pluteus_typeC 0.71 0.15 0.25 80
competition_data/train/decapods 0.00 0.00 0.00 55
competition_data/train/copepod_calanoid_small_longantennae 0.70 0.08 0.14 87
competition_data/train/euphausiids 0.55 0.09 0.15 136
competition_data/train/echinopluteus 0.00 0.00 0.00 27
avg / total 0.43 0.47 0.41 30336
\r\n
The current model, while somewhat accurate overall, doesn't do well for all classes, including the shrimp caridean, stomatopod, or hydromedusae tentacles classes. For others it does quite well, getting many of the correct classifications for trichodesmium_puff and copepod_oithona_eggs classes. The metrics shown above for measuring model performance include precision, recall, and f1-score. The precision metric gives probability that a chosen class is correct, (true positives / (true positive + false positives)), while recall measures the ability of the model correctly classify examples of a given class, (true positives / (false negatives + true positives)). The F1 score is the geometric average of the precision and recall.
The competition scoring uses a multiclass log-loss metric to compute your overall score. In the next steps, we define the multiclass log-loss function and compute your estimated score on the training dataset.
\r\nIn [16]:
\r\ndef multiclass_log_loss(y_true, y_pred, eps=1e-15):\r\n \"\"\"Multi class version of Logarithmic Loss metric.\r\n https://www.kaggle.com/wiki/MultiClassLogLoss\r\n\r\n Parameters\r\n ----------\r\n y_true : array, shape = [n_samples]\r\n true class, intergers in [0, n_classes - 1)\r\n y_pred : array, shape = [n_samples, n_classes]\r\n\r\n Returns\r\n -------\r\n loss : float\r\n \"\"\"\r\n predictions = np.clip(y_pred, eps, 1 - eps)\r\n\r\n # normalize row sums to 1\r\n predictions /= predictions.sum(axis=1)[:, np.newaxis]\r\n\r\n actual = np.zeros(y_pred.shape)\r\n n_samples = actual.shape[0]\r\n actual[np.arange(n_samples), y_true.astype(int)] = 1\r\n vectsum = np.sum(actual * np.log(predictions))\r\n loss = -1.0 / n_samples * vectsum\r\n return loss\r\n\r\n
In [17]:
\r\n# Get the probability predictions for computing the log-loss function\r\nkf = KFold(y, n_folds=5)\r\n# prediction probabilities number of samples, by number of classes\r\ny_pred = np.zeros((len(y),len(set(y))))\r\nfor train, test in kf:\r\n X_train, X_test, y_train, y_test = X[train,:], X[test,:], y[train], y[test]\r\n clf = RF(n_estimators=100, n_jobs=3)\r\n clf.fit(X_train, y_train)\r\n y_pred[test] = clf.predict_proba(X_test)\r\n
In [18]:
\r\nmulticlass_log_loss(y, y_pred)\r\n
3.7390475458333374
\r\nThe multiclass log loss function is an classification error metric that heavily penalizes you for being both confident (either predicting very high or very low class probability) and wrong. Throughout the competition you will want to check that your model improvements are driving this loss metric lower.
\r\nNow that you've made a simple metric, created a model, and examined the model's performance on the training data, the next step is to make improvements to your model to make it more competitive. The random forest model we created does not perform evenly across all classes and in some cases fails completely. By creating new features and looking at some of your distributions for the problem classes directly, you can identify features that specifically help separate those classes from the others. You can add new metrics by considering other image properties, stratified sampling, transformations, or other models for the classification.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false } ] }