# National Data Science Bowl — Data

来源：https://www.kaggle.com/competitions/datasciencebowl/data

本地 Meta Kaggle 快照；不是本次实时页面。

# Dataset Description

<p>In total, Oregon State University’s Hatfield Marine Science Center has captured nearly 50 million plankton images over an 18-day period. This is more than 80 terabytes of data! They need your help creating an automated classification process to better understand the image contents.</p>
<p>For this competition, Hatfield scientists have prepared a large collection of labeled images, approximately 30k of which are provided as a training set. Each raw image was run through an automatic process to extract regions of interest, resulting in smaller images that contain a single organism/entity. You must create an algorithm that assigns class probabilities to a given image. Several characteristics of this problem make this classification difficult:</p>
<ul>
<li><span style="font-size: 1em; line-height: 1.5em;">There are many different species, ranging from the smallest single-celled protists to copepods, larval fish, and larger jellies.</span></li>
<li><span style="font-size: 1em; line-height: 1.5em;">Representatives from each taxon can have any orientation within 3-D space.</span></li>
<li>The ocean is replete with detritus (often decomposing plant or animal matter that scientists like to call “whale snot”) and fecal pellets that have no taxonomic identification but are important in other marine processes.</li>
<li>Some images are so noisy or ambiguous that experts have a difficult time labeling them. Some amount of noise in the ground truth is thus inevitable.</li>
<li>The presence of "unknown" classes require models to handle the special cases of unidentifiable objects.</li>
</ul>
<p><img src="https://storage.googleapis.com/kaggle-media/competitions/kaggle/3978/media/plankton%20schmorgasborg.jpg" alt="" width="650" height="538" /></p>
<p>The file <strong>train.zip</strong> contains labeled images for training. Each folder represents a class. Images within the folder belong to that class. Classes were chosen by the Hatfield experts and represent scientifically meaningful groupings of organisms and objects (see the below FAQs). You will notice that it is possible for different classes to contain the same underlying organism. In these cases, they have been separated based on other factors of interest (e.g. one may represent an organism in motion vs. one that is still). You are permitted to use domain knowledge about the class relationships in your methodology. The official list of classes is the following:</p>
<pre>acantharia_protist_big_center<br />acantharia_protist_halo<br />acantharia_protist<br />amphipods<br />appendicularian_fritillaridae<br />appendicularian_s_shape<br />appendicularian_slight_curve<br />appendicularian_straight<br />artifacts_edge<br />artifacts<br />chaetognath_non_sagitta<br />chaetognath_other<br />chaetognath_sagitta<br />chordate_type1<br />copepod_calanoid_eggs<br />copepod_calanoid_eucalanus<br />copepod_calanoid_flatheads<br />copepod_calanoid_frillyAntennae<br />copepod_calanoid_large_side_antennatucked<br />copepod_calanoid_large<br />copepod_calanoid_octomoms<br />copepod_calanoid_small_longantennae<br />copepod_calanoid<br />copepod_cyclopoid_copilia<br />copepod_cyclopoid_oithona_eggs<br />copepod_cyclopoid_oithona<br />copepod_other<br />crustacean_other<br />ctenophore_cestid<br />ctenophore_cydippid_no_tentacles<br />ctenophore_cydippid_tentacles<br />ctenophore_lobate<br />decapods<br />detritus_blob<br />detritus_filamentous<br />detritus_other<br />diatom_chain_string<br />diatom_chain_tube<br />echinoderm_larva_pluteus_brittlestar<br />echinoderm_larva_pluteus_early<br />echinoderm_larva_pluteus_typeC<br />echinoderm_larva_pluteus_urchin<br />echinoderm_larva_seastar_bipinnaria<br />echinoderm_larva_seastar_brachiolaria<br />echinoderm_seacucumber_auricularia_larva<br />echinopluteus<br />ephyra<br />euphausiids_young<br />euphausiids<br />fecal_pellet<br />fish_larvae_deep_body<br />fish_larvae_leptocephali<br />fish_larvae_medium_body<br />fish_larvae_myctophids<br />fish_larvae_thin_body<br />fish_larvae_very_thin_body<br />heteropod<br />hydromedusae_aglaura<br />hydromedusae_bell_and_tentacles<br />hydromedusae_h15<br />hydromedusae_haliscera_small_sideview<br />hydromedusae_haliscera<br />hydromedusae_liriope<br />hydromedusae_narco_dark<br />hydromedusae_narco_young<br />hydromedusae_narcomedusae<br />hydromedusae_other<br />hydromedusae_partial_dark<br />hydromedusae_shapeA_sideview_small<br />hydromedusae_shapeA<br />hydromedusae_shapeB<br />hydromedusae_sideview_big<br />hydromedusae_solmaris<br />hydromedusae_solmundella<br />hydromedusae_typeD_bell_and_tentacles<br />hydromedusae_typeD<br />hydromedusae_typeE<br />hydromedusae_typeF<br />invertebrate_larvae_other_A<br />invertebrate_larvae_other_B<br />jellies_tentacles<br />polychaete<br />protist_dark_center<br />protist_fuzzy_olive<br />protist_noctiluca<br />protist_other<br />protist_star<br />pteropod_butterfly<br />pteropod_theco_dev_seq<br />pteropod_triangle<br />radiolarian_chain<br />radiolarian_colony<br />shrimp_caridean<br />shrimp_sergestidae<br />shrimp_zoea<br />shrimp-like_other<br />siphonophore_calycophoran_abylidae<br />siphonophore_calycophoran_rocketship_adult<br />siphonophore_calycophoran_rocketship_young<br />siphonophore_calycophoran_sphaeronectes_stem<br />siphonophore_calycophoran_sphaeronectes_young<br />siphonophore_calycophoran_sphaeronectes<br />siphonophore_other_parts<br />siphonophore_partial<br />siphonophore_physonect_young<br />siphonophore_physonect<br />stomatopod<br />tornaria_acorn_worm_larvae<br />trichodesmium_bowtie<br />trichodesmium_multiple<br />trichodesmium_puff<br />trichodesmium_tuft<br />trochophore_larvae<br />tunicate_doliolid_nurse<br />tunicate_doliolid<br />tunicate_partial<br />tunicate_salp_chains<br />tunicate_salp<br />unknown_blobs_and_smudges<br />unknown_sticks<br />unknown_unclassified</pre>
<p>The file <strong>test.zip</strong> contains the test set images, for which you must predict class probabilities. To deter hand labeling of images, Kaggle has supplemented the test set with additional "ignored" images. These are not counted in the scoring.</p>
<p><strong>sampleSubmission.csv</strong> provides the appropriate format for generating a valid submission.</p>
<p><strong>plankton_identification.pdf</strong> is provided as a rough guide to understand relationships between the classes. The tree-like diagram indicates morphological and biological connections between groups. Dashed lines indicate a weak(er) relationship and solid lines a stronger relationship.</p>
<h2>Frequently Asked Questions</h2>
<p><em>written by Jessica Luo, PhD Candidate, Hatfield Marine Science Center</em></p>
<h5><strong>Q. How did you collect the images?</strong></h5>
<p>We collected the images between May-June 2014 in the Straits of Florida using the In Situ Ichthyoplankton Imaging System (ISIIS), a towed, underwater imaging system using shadowgraph imagery with a line-scan camera.</p>
<p>A high-resolution continuous image is parsed into 2048 x 2048 pixel frames, which were corrected using flat-fielding. Frames were then thresholded and segmented into regions of interest (ROIs). These ROI segments are the images provided for this competition.</p>
<p>We made every effort to include a representation of real world data in this dataset. In other words, we did not cherry-pick for the best and clearest images, but used images spanning the gamut from blurry to clear, and tiny to big.</p>
<h5><strong>Q. Do organisms appear smaller if they are further away from the camera?</strong></h5>
<p>No. Because of the shadowgraph imagery technique, organisms will be the same size regardless of distance to the camera.</p>
<h5><strong>Q. Can you explain why you would get partial images of organisms then?</strong></h5>
<p>Partial images in our dataset occur due to imperfect segmentation. If an organism is highly transparent (like a jellyfish), then sometimes the thresholding and segmentation process will cut off part of the image. This is fairly rare; typically our segmentation process works very well.</p>
<h5><strong>Q. Are there relationships between the categories?</strong></h5>
<p>Yes, there are relationships between the categories. Some of the relationships are taxonomic, some are behavioral / ecological, and some are based on shape. We diagrammed all of these relationships in the plankton relationships document, which can be accessed along with the dataset. Red boxes indicate our categories, blue boxes indicate major groups, solid lines indicate direct relationships, and dashed lines indicate minor relationships or shape similarities.</p>
<h5><strong>Q. Why are there non-taxonomic distinctions in categories?</strong></h5>
<p>In some cases, we chose to partition organisms into different categories based on shape. For example, in the appendicularians, the shape indicates a behavior (tail beating). Furthermore, appendicularians are often confused with a whole host of other things - most notably, fish larvae - so we have traditionally separated them out by shape to increase classification accuracy.</p>
<h5><strong>Q. Are these classifications reliable?</strong></h5>
<p>We have a trained team and have cross-validated the classifications, such that certain images were checked by up to five people. That said, there is an intrinsic amount of variability in classifying ambiguous images. For example, Culverhouse et al. (2003) found that experts are able to maintain 84-95% self-consistency in labeling difficult images. A spot check on the self-consistency in this dataset was on the high end of that range. We believe that this is the best possible dataset of these kinds of plankton images currently in existence.</p>
<p>Citation: PF Culverhouse, Williams R, Reguera B, Herry V, González-Gil S. (2003) "Do experts make mistakes? A comparison of human and machine identification of dinoflagellates." Mar. Ecol. Prog. Ser. 247:17-25.</p>