![\"\"](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/6004/media/Sample%20MRI%20Slice.png\")
NOTE: due to data set usage restrictions, the data for this competition is no longer available for download.
\nIn this dataset, you are given over a thousand low-dose CT images from high-risk patients in DICOM format. Each image contains a series with multiple axial slices of the chest cavity. Each image has a variable number of 2D slices, which can vary based on the machine taking the scan and patient.
\nThe DICOM files have a header that contains the necessary information about the patient id, as well as scan parameters such as the slice thickness.
\n
The competition task is to create an automated method capable of determining whether or not the patient will be diagnosed with lung cancer within one year of the date the scan was taken. The ground truth labels were confirmed by pathology diagnosis.
\nThe images in this dataset come from many sources and will vary in quality. For example, older scans were imaged with less sophisticated equipment. You should expect the stage 2 data to be, on the whole, more recent and higher quality than the stage 1 data (generally having thinner slice thickness). Ideally, your algorithm should perform well across a range of image quality.
\nEach patient id has an associated directory of DICOM files. The patient id is found in the DICOM header and is identical to the patient name. The exact number of images will differ from case to case, varying according in the number of slices. Images were compressed as .7z files due to the large size of the dataset.
\nThe DICOM standard is complex and there are a number of different tools to work with DICOM files. You may find the following resources helpful for managing the competition data:
\nIn the United States, lung cancer strikes 225,000 people every year, and accounts for $12 billion in health care costs. Early detection is critical to give patients the best chance at recovery and survival.
\nOne year ago, the office of the U.S. Vice President spearheaded a bold new initiative, the Cancer Moonshot, to make a decade's worth of progress in cancer prevention, diagnosis, and treatment in just 5 years.
\nIn 2017, the Data Science Bowl will be a critical milestone in support of the Cancer Moonshot by convening the data science and medical communities to develop lung cancer detection algorithms.
\nUsing a data set of thousands of high-resolution lung scans provided by the National Cancer Institute, participants will develop algorithms that accurately determine when lesions in the lungs are cancerous. This will dramatically reduce the false positive rate that plagues the current detection technology, get patients earlier access to life-saving interventions, and give radiologists more time to spend with their patients.
\nThis year, the Data Science Bowl will award $1 million in prizes to those who observe the right patterns, ask the right questions, and in turn, create unprecedented impact around cancer screening care and prevention. The funds for the prize purse will be provided by the Laura and John Arnold Foundation.
\nVisit DataScienceBowl.com to:
• Sign up to receive news about the competition
• Learn about the history of the Data Science Bowl and past competitions
• Read our latest insights on emerging analytics techniques
![\"BAH](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/6004/media/illustration-lung-3.5.jpg\")
The Data Science Bowl is presented by
\n![\"BAH](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/3978/media/booz_kaggle.png\")
Laura and John Arnold Foundation
Cancer Imaging Program of the National Cancer Institute
American College of Radiology
Amazon Web Services
NVIDIA
National Lung Screening Trial
The Cancer Imaging Archive
Diagnostic Image Analysis Group, Radboud University
Lahey Hospital & Medical Center
Copenhagen University Hospital
Bayes Impact
Black Data Processng Associates
Code the Change
Data Community DC
DataKind
Galvanize
Great Minds in STEM
Hortonworks
INFORMS
Lesbians Who Tech
NSBE
Society of Asian Scientists & Engineers
Society of Women Engineers
University of Texas Austin, Business Analytics Program,
McCombs School of Business
US Dept. of Health and Human Services
US Food and Drug Administration
Women in Technology
Women of Cyberjutsu
Submissions are scored on the log loss:
\r\n$$
\\textrm{LogLoss} = - \\frac{1}{n} \\sum_{i=1}^n \\left[ y_i \\log(\\hat{y}_i) + (1 - y_i) \\log(1 - \\hat{y}_i)\\right],
$$
where
\r\nNote: the actual submitted predicted probabilities are replaced with \\\\(max(min(p,1-10^{-15}),10^{-15})\\\\). A smaller log loss is better.
\r\nFor each patient id in the test set, you must submit a probability. The file should have a header and be in the following format:
\r\nid,cancer", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Prizes", "content": "
01e349d34c02410e1da273add27be25c,0.5
05a20caf6ab6df4643644c923f06a5eb,0.5
0d12f1c627df49eb223771c28548350e,0.5
...
In addition to the $1 million in cash prizes awarded by the Arnold Foundation to the winners of this competition, NVIDIA and Amazon have made the following prizes available.
\r\nYou cannot sign up to Kaggle from multiple accounts and therefore you cannot submit from multiple accounts.
\nPrivately sharing code or data outside of teams is not permitted. It's okay to share code if made available to all participants on the forums.
\nTeam mergers are allowed and can be performed by the team leader. In order to merge, the combined team must have a total submission count less than or equal to the maximum allowed as of the merge date. The maximum allowed is the number of submissions per day multiplied by the number of days the competition has been running.
\nThere is no maximum team size.
\nYou may submit a maximum of 3 entries per day.
\nYou may select up to 2 final submissions for judging.
\nStart Date: 1/12/2017 2:00 PM UTC
\nMerger Deadline: 3/31/2017 11:59 PM UTC
\nEntry Deadline: 3/31/2017 11:59 PM UTC
\nEnd Date: 4/12/2017 11:59 PM UTC
\n\nCOMPETITION NAME (the 'Competition'): Data Science Bowl 2017
\nCOMPETITION SPONSOR: Booz Allen Hamilton
\nCOMPETITION WEBSITE: https://www.kaggle.com/c/data-science-bowl-2017
\nPRIZES: $1,000,000 total prizes
\nWINNER LICENSE TYPE: Open Source License
\nMAXIMUM TEAM SIZE: as listed above
\nMAXIMUM SUBMISSIONS PER DAY: as listed above
\nMembers of the following organizations are invited to participate in the competition, however they will not be eligible to win a cash prize:
\nAdditionally, any individual determined to have access to the test data prior to the Competition will not be eligible for a cash prize.
\nThese are the complete, official rules for the Competition (the 'Competition Rules') and incorporate by reference the contents of the Competition Website listed above.
\nBy downloading a dataset linked from the Competition Website, submitting an entry to this Competition, or joining a Team in this Competition, you are agreeing to be bound by these Competition Rules which constitute a binding agreement between you and the Competition Sponsor.
\nThe Competition is sponsored by the Competition Sponsor listed above and hosted on the Sponsor's behalf by Kaggle Inc ('Kaggle').
\nThe Competition will run from Start Date to End Date as listed on the Competition Website. Note that Competition deadlines are subject to change and additional hurdle deadlines may be introduced during the Competition. Any additional or altered deadlines not already described in these Competition Rules will be publicized to the Participants via the Competition Website. It is the Participant's responsibility in this Competition to check the website regularly throughout the Competition to stay informed of any new or updated deadlines. Neither Kaggle nor Competition Sponsor are responsible for any Participant's failure to do so.
\nEach registered individual or Team is referred to as a Participant. You may only participate using a single, unique Kaggle account registered at http://www.kaggle.com. Participating using more than one Kaggle account per individual Participant is a breach of these Competition Rules and Competition Sponsor reserves the right to disqualify any Participant (or Team including Participant) who is found to breach these Competition Rules.
\nThe Competition is open to all individuals over the age of 18 at the time of entry and to all validly formed legal entities that have not declared or been declared in bankruptcy.
\nOfficers, directors, employees and advisory board members (and their immediate families and members of the same household) of the Competition Sponsor, Kaggle and their respective affiliates, subsidiaries, contractors (with the express exception of Kaggle's authorized Kaggle Community Evangelists), agents, judges and advertising and promotion agencies are not eligible to receive any Prize in the Competition.
\nYou are not eligible to receive any Prize in the Competition if you are a resident of a country designated by the United States Treasury’s Office of Foreign Assets Control (see <http://www.treasury.gov/resource-center/sanctions/SDN-List/Pages/default.aspx> for additional information).
\n'Submission' means the material submitted by you in the manner and format specified on the Website via the Submission form on the Website. You (or if you are part of a Team, your Team) may submit up to the maximum number of Submissions per day as specified above. All Submissions must be uploaded to the Website in the manner and format specified on the Website. Submissions must be received prior to the Competition deadline and adhere to the guidelines for Submissions specified on the Website.
\nSubmissions may not use or incorporate information from hand labeling or human prediction of the validation dataset or test data records.
\nIf the Competition is a multi-stage Competition with temporally separate training data and/or leaderboard data, one or more valid Submissions must be made and selected during each stage of the Competition in the manner described on the Competition Website.
\n'Data' means the Data or Datasets linked from the Competition Website for the purpose of use by Participants in the Competition. For the avoidance of doubt, Data is deemed for the purpose of these Competition Rules to include any prototype or executable code provided to Participants by Kaggle or Competition Sponsor via the Website. Participants must use the Data only as permitted by these Competition Rules and any associated data use rules specified on the Competition Website.
\nUnless otherwise permitted by the terms of the Competition Website, Participants must use the Data solely for the purpose and duration of the Competition, including but not limited to reading and learning from the Data, analyzing the Data, modifying the Data and generally preparing your Submission and any underlying models and participating in forum discussions on the Website. Participants agree to use suitable measures to prevent persons who have not formally agreed to these Competition Rules from gaining access to the Data and agree not to transmit, duplicate, publish, redistribute or otherwise provide or make available the Data to any party not participating in the Competition. Participants agree to notify Kaggle immediately upon learning of any possible unauthorized transmission or unauthorized access of the Data and agree to work with Kaggle to rectify any unauthorized transmission. Participants agree that participation in the Competition shall not be construed as having or being granted a license (expressly, by implication, estoppel, or otherwise) under, or any right of ownership in, any of the Data.
\nUnless otherwise expressly stated on the Competition Website, Participants must not use data other than the Data to develop and test their models and Submissions. Competition Sponsor reserves the right in its sole discretion to disqualify any Participant who Competition Sponsor discovers has undertaken or attempted to undertake the use of data other than the Data, or who uses the Data other than as permitted according to the Competition Website and in these Competition Rules, in the course of the Competition.
\nParticipants are prohibited from privately sharing source or executable code developed in connection with or based upon the Data, and any such sharing is a breach of these Competition Rules and may result in disqualification.
\nParticipants are permitted to publicly share source or executable code developed in connection with or based upon the Data, or otherwise relevant to the Competition, provided that such sharing does not violate the intellectual property rights of any third party. By so sharing, the sharing Participant is thereby deemed to have licensed the shared code under the MIT License (an open source software license commonly described at <http://opensource.org/licenses/MIT>).
\nA Submission will be ineligible to win a prize if it was developed using code containing or depending on software licensed under an open source license:
\n* other than an Open Source Initiative-approved license (see <http://opensource.org/>); or
* an open source license that prohibits commercial use.
This Competition is a challenge of skill and the final results are determined solely by leaderboard ranking on the private leaderboard (subject to compliance with these Competition Rules). Participants' scores and ranks on the Competition Website at any given stage of the Competition will be based on the evaluation metric described on the Competition Website, as determined by applying the predictions in the Submission to the ground truth of a validation dataset whose instances were a fixed set sampled from the Data.
\nThe evaluation metric used for scoring and ranking Submissions will be displayed on the Competition Website.
\nPrize awards are subject to verification of eligibility and compliance with these Competition Rules. All decisions of the Competition Sponsor and judges will be final and binding on all matters relating to this Competition. Competition Sponsor reserves the right to examine the Submission and any associated code or documentation for compliance with these Competition Rules. In the event that the Submission demonstrates a breach of these Competition Rules, Competition Sponsor may at its discretion take either of the following actions:
\n* disqualify your Submission(s); or
* require that you remediate within one week all issues identified in your Submission(s) (including, without limitation, the resolution of license conflicts, the fulfillment of all obligations required by software licenses, and the removal of any software that violates the software restrictions).
A tie between two or more valid and identically ranked submissions will be resolved in favour of the tied submission that was submitted first.
\nA Participant may decline to be nominated as a Winner by notifying Kaggle directly within one week following the Competition deadline, in which case the declining Participant forgoes any prize or other features associated with winning the Competition. Kaggle reserves the right to disqualify a Participant who so declines at Kaggle's sole discretion if Kaggle deems disqualification appropriate.
\nAs a condition of receipt of the Prize, the Prize winner must deliver the final model’s software code as used to generate the winning Submission and associated documentation (consistent with the winning model documentation template available on the kaggle wiki at https://www.kaggle.com/wiki/WinningModelDocumentationTemplate) to the Competition Sponsor. The delivered software code must be capable of generating the winning Submission and contain a description of resources required to build and/or run the executable code successfully.
\nIf a potential winning Participant or Team member is a U.S. citizen, potential winner must also sign and return an IRS W-9 form, or if a foreign resident, an IRS W-8BEN form, within the stated time in order to claim the prize.
\nPotential winners will be disqualified and the prize may be awarded to an alternate winner (the next-ranked qualified Participant on the leaderboard) if
\n* the required documentation is not returned within 14 days after receipt of any request to provide documentation;
* prize notification letter/email or prize is returned as undeliverable; or
* potential winner or winning Team member is disqualified for any reason.
By accepting any Prize, each Participant receiving a Prize thereby agrees to use of his/her name, address, likeness and/or Prize information by Kaggle and Competition Sponsor for promotional purposes in any medium without additional compensation.
\nAs a further condition of receipt of a Prize, each winning Participant thereby licenses their winning Submission and the source code used to generate the Submission according to the Winner License Type specified above (note: if no Winner License Type is specified above, the Winner License Type is deemed to be Non-Exclusive License).
\nIf the Winner License Type for the Competition (see Winner License Type above) is a Non-Exclusive License then each Winner by accepting a Prize thereby:
\n* grants to Competition Sponsor and its designees a worldwide, non-exclusive, sub-licensable, transferable, fully paid-up, royalty-free, perpetual, irrevocable right to use, not use, reproduce, distribute, create derivative works of, publicly perform, publicly display, digitally perform, make, have made, sell, offer for sale and import their winning Submission and the source code used to generate the Submission, in any media now known or hereafter developed, for any purpose whatsoever, commercial or otherwise, without further approval by or payment to Participant; and
* represents that he/she/it has the unrestricted right to grant that license.
If the Winner License Type for the Competition (see Winner License Type above) is Open Source License, then each Winner by accepting a Prize thereby:
\n* licenses their winning Submission and the source code used to generate the Submission under the MIT License (an open source software license commonly described at <http://opensource.org/licenses/MIT>), unless another open source license is chosen explicitly from the list of those approved by the Open Source Initiative at <http://opensource.org/licenses/>; and
* represents that he/she/it has the unrestricted right to grant that license.
If the Competition is classed above as a Recruiting Competition, Participant agrees to grant and hereby grants to Competition Sponsor a perpetual, irrevocable, worldwide, royalty-free, transferable, and sublicenseable right to use, review, reproduce, and internally distribute the Submission (including any and all submitted source code) in connection with evaluating Participant's suitability for employment, provided that the Submission licensed as set forth above will not be used for Competition Host’s commercial purposes except as may otherwise be set forth in this Agreement.
\nParticipating using more than one Kaggle account is deemed cheating and, if discovered, will result in disqualification from the Competition and any other affected Competitions and may result in banning or deactivation of affected Kaggle accounts.
\nAfter verification of eligibility, each Prize winner will receive the prize in the form of a check or wire transfer made out to the Prize winner (if an individual, or to the individual Team members if a Team). Allow 30 days from final confirmation for Prize delivery. Any winners who are U.S. citizens will receive an IRS 1099 form in the amount of their prize at the appropriate time. Prize winners are responsible for any taxes, fees or other liability resulting from their receipt of a Prize.
\nMultiple individuals or entities may collaborate as a team (“Team”). You may not participate on more than one Team. Each Team member must be a single individual operating a separate Kaggle account. You must register individually for the Competition before joining a Team. You must confirm your Team membership to make it official by responding to the Team notification message which will be sent to your Account.
\nTeam membership may not exceed the Maximum Team Size.
\nTeams may be permitted to merge at Kaggle’s discretion, so long as the merged Team meets all requirements in these Competition Rules. Kaggle will review any merger request and will either approve or reject the request within three business days. Merger requests may be rejected if the combined number of Entries made by the merging Teams exceeds the number of Entries permissible at the date of the merger request. Team merger requests will not be permitted within seven days of any deadline listed on the Competition Website.
\nIf a Team wins a monetary Prize, Competition Sponsor will allocate the Prize money in even shares between the Team members unless the Team unanimously contacts Kaggle via the Competition Website within three business days following the Submission deadline to request an alternative prize distribution.
\nBy registering, you agree that (a) your Account is complete, correct and accurate and (b) your registration may be rejected or terminated and all Entries submitted by you and/or your Team may be disqualified if any of the information in your Account is (or Competition Sponsor has reasonable grounds to believe it is) incomplete, incorrect or inaccurate. You are solely responsible for your Account. All registration information is deemed collected in the United States.
\nParticipation is subject to all federal, state and local laws and regulations. Void where prohibited or restricted by law. You are responsible for checking applicable laws and regulations in your jurisdiction before participating in the Competition to make sure that your participation is legal. You are responsible for all taxes and reporting related to any award that you may receive as part of the Competition. You are responsible for abiding by your employer's policies regarding participation in the Competition. Competition Sponsor disclaims any and all liability or responsibility for disputes arising between you and your employer related to this Competition.
\nEach Participant is solely responsible for all equipment, including but not necessarily limited to a computer and internet connection necessary to access the Website and to develop and upload any Submission, and any telephone, data, hosting or other service fees associated with such access, as well as all costs incurred by or behalf of the Entrant in participating in the Competition.
\nBy entering a Submission, you represent and warrant that all information you enter on the Website is true and complete to the best of your knowledge, that you have the right and authority to make the Submission (including any underlying code and model) on your own behalf or on behalf of the persons and entities that you specify within the Submission, and that your Submission:
\n* is your own original work, or is used by permission, in which case full and proper credit and identify is given and the third party contributions are clearly identified within your Submission;
* does not contain confidential information or trade secrets and is not the subject of a registered patent or pending patent application;
* does not violate or infringe upon the patent rights, industrial design rights, copyrights, trademarks, rights of privacy, publicity or other intellectual property or other rights of any person or entity;
* does not contain malicious code, such as viruses, timebombs, cancelbots, worms, Trojan horses or other potentially harmful programs or other material or information;
* does not and will not violate any applicable law, statute, ordinance, rule or regulation;
* does not trigger any reporting or royalty obligation to any third party; and
* was not previously published and has not won any other prize/award.
A breach of any of these warranties will result in the corresponding Submission being invalid.
By participating in the Competition, each Participant agrees to release, indemnify and hold harmless Competition Sponsor, Kaggle, their respective affiliates, subsidiaries, advertising and promotions agencies, as applicable, and each of their respective agents, representatives, officers, directors, shareholders, and employees from and against any injuries, losses, damages, claims, actions and any liability of any kind resulting from or arising out of your participation in or association with the Competition. Competition Sponsor is not responsible for any miscommunications such as technical failures related to computer, telephone, cable, and unavailable network or server connections, related technical failures, or other failures related to hardware, software or virus, or incomplete, late or misdirected Submissions. Competition Sponsor reserves the right to cancel, modify or suspend the Competition should any computer virus, bug or other technical difficulty or other causes beyond the control of Competition Sponsor corrupt the administration, security or proper play of the Competition, and to determine winners from among Submission not affected by the corruption, if any, in its sole discretion.
\nNeither Kaggle nor Competition Sponsor are responsible for (a) late, lost, stolen, damaged, garbled, incomplete, incorrect or misdirected Entries or other communications, (b) errors, omissions, interruptions, deletions, defects, or delays in operations or transmission of information, in each case whether arising by way of technical or other failures or malfunctions of computer hardware, software, communications devices, or transmission lines, or (c) data corruption, theft, destruction, unauthorized access to or alteration of Submission materials, loss or otherwise. Neither Kaggle nor Competition Sponsor are responsible for electronic communications or emails which are undeliverable as a result of any form of active or passive filtering of any kind, or insufficient space in any email account to receive email messages. Competition Sponsor disclaims any liability for damage to any computer system resulting from participation in, or accessing or downloading information in connection with, the Competition.
\nCompetition Sponsor reserves the right to modify the Start Date, End Date, and the right to modify, add or remove additional Competition deadlines for any reason. Any changes will be communicated through the Competition Website. Competition Sponsor also reserves the right to modify, remove or add Data to the Website upon notice via the Competition Website. NEITHER KAGGLE NOR COMPETITION SPONSOR ARE RESPONSIBLE FOR ANY FAILURE OF A PARTICIPANT TO RECEIVE DATA CHANGES.
\nThe invalidity or unenforceability of any provision of these Competition Rules shall not affect the validity or enforceability of any other provision. In the event that any provision is determined to be invalid or otherwise unenforceable or illegal, these Competition Rules shall otherwise remain in effect and be construed in accordance with their terms as if the invalid or illegal provision was not contained herein.
\nYou agree that these terms and the relationship between you and Competition Sponsor shall be governed by the laws of the State of California and the United States of America.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Timeline", "content": "All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "About", "content": "The Data Science Bowl, presented by Booz Allen and Kaggle, is the world’s premier data science for social good competition. It convenes data scientists, technologists, domain experts, and organizations to take on the world’s challenges with data and technology. It’s a platform through which individuals can harness their passion, unleash their curiosity, and amplify their impact to effect change on a global scale.
\r\nDuring a 90-day period, participants, either alone or working in teams, gain access to unique data sets to develop algorithms that address a specific challenge. And each year, the competition awards hundreds of thousands of dollars in prize money to top teams.
\r\nIn 2014/2015, participants examined more than 100,000 underwater images, provided by the Hatfield Marine Science Center, to assess ocean health at a massive speed and scale.
\r\nIn 2015/2016, they applied analytics in cardiology, transforming the practice of assessing heart function.
\r\nIn 2017, we’ll turn machine intelligence against lung cancer, and work to end the disease as we know it.
\r\n![\"DSB](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/6004/media/DSB_Zone_Graphic_kids-01.png\")
Visit DataScienceBowl.com to learn more about the competition, get insights into emerging analytics techniques, and sign up for news alerts.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Tutorial", "content": "written by Jonathan Mulholland and Aaron Sander, Booz Allen Hamilton
\r\nIn this tutorial, we will show one approach to predicting whether a CT scan is of a patient who either has or will develop cancer within the next 12 months or not.
\r\nWe will train a network to segment out potentially cancerous nodules and then use the characteristics of that segmentation to make predictions about the diagnosis of the scanned patient within a 12 month time frame.
\r\nThe code for this tutorial can be found at https://github.com/booz-allen-hamilton/DSB3Tutorial
\r\nThis tutorial uses python and has the following dependencies
\r\nNote: Keras allows multiple backends for use when training. We chose a GPU enabled installation of tensorflow as the backend of Keras.
\r\nIn order to identify regions with nodules, we will use a U-Net style convolutional network which was designed for segmenting neuronal structures. https://arxiv.org/abs/1505.04597
\r\nOur code for the network was based on a turorial posted by Marko Jocic on the Kaggle forum for the Ultrasound Nerve Segmentation Challenge. https://www.kaggle.com/c/ultrasound-nerve-segmentation/forums/t/21358/0-57-deep-learning-keras-tutorial
\r\nThe images we'll be predicting cancer diagnoses on are scans from low-dose helical computed tomography (CT). The appearance on nodules within the CT scan indicate the possibility of cancer, and we will need training examples with marked nodules in order train the U Net to find these nodules. Rather than hand label images, we turn to the Lung Nodule Analysis 2016 (LUNA2016) challenge which has made available CT images with annotated nodule locations. We will first use the LUNA data set to generate an appropriate training set for our U-Net. We will use these examples to train our supervised segmenter.
\r\nWe are going to use the nodule locations as given in annotations.csv and extract three transverse slices that contain the largest nodule from each patient scan. Masks will be created for those slices based on the nodule dimensions given in annotations.csv. The output of this file will be two files for each patient scan: a set of images and a set of corresponding nodule masks. The data from the LUNA 2016 challenge can be found at https://luna16.grand-challenge.org/
\r\nFirst we import the necessary tools and find the largest nodule in the patient scan. There are multiple nodule listings for some patients in annotations.csv. We're using a pandas DataFrame named df_node to keep track of the case numbers and the node information. The node information is an (x,y,z) coordinate in mm using a coordinate system defined in the .mhd file.
The following snippets of code are from LUNA_mask_extraction.py:
\r\nimport SimpleITK as sitk\r\nimport numpy as np\r\nimport csv\r\nfrom glob import glob\r\nimport pandas as pd\r\nfile_list=glob(luna_subset_path+\"*.mhd\")\r\n#####################\r\n#\r\n# Helper function to get rows in data frame associated \r\n# with each file\r\ndef get_filename(case):\r\n global file_list\r\n for f in file_list:\r\n if case in f:\r\n return(f)\r\n#\r\n# The locations of the nodes\r\ndf_node = pd.read_csv(luna_path+\"annotations.csv\")\r\ndf_node[\"file\"] = df_node[\"seriesuid\"].apply(get_filename)\r\ndf_node = df_node.dropna()\r\n#####\r\n#\r\n# Looping over the image files\r\n#\r\nfcount = 0\r\nfor img_file in file_list:\r\n print \"Getting mask for image file %s\" % img_file.replace(luna_subset_path,\"\")\r\n mini_df = df_node[df_node[\"file\"]==img_file] #get all nodules associate with file\r\n if len(mini_df)>0: # some files may not have a nodule--skipping those \r\n biggest_node = np.argsort(mini_df[\"diameter_mm\"].values)[-1] # just using the biggest node\r\n node_x = mini_df[\"coordX\"].values[biggest_node]\r\n node_y = mini_df[\"coordY\"].values[biggest_node]\r\n node_z = mini_df[\"coordZ\"].values[biggest_node]\r\n diam = mini_df[\"diameter_mm\"].values[biggest_node]\r\nThe nodule locations are given in terms of millimeters relative to a coordinate system defined by the CT scanner. The image data is given as a varying length stack of 512 X 512 arrays. In order to translate the voxel position to the world coordinate system, one needs to know the real world position of the [0,0,0] voxel and the voxel spacing in mm.
\r\nTo find the voxel coordinates of a nodule, given its real world position, we use the GetOrigin() and GetSpacing() method of the itk image object:
itk_img = sitk.ReadImage(img_file) \r\nimg_array = sitk.GetArrayFromImage(itk_img) # indexes are z,y,x (notice the ordering)\r\ncenter = np.array([node_x,node_y,node_z]) # nodule center\r\norigin = np.array(itk_img.GetOrigin()) # x,y,z Origin in world coordinates (mm)\r\nspacing = np.array(itk_img.GetSpacing()) # spacing of voxels in world coor. (mm)\r\nv_center =np.rint((center-origin)/spacing) # nodule center in voxel space (still x,y,z ordering)\r\nThe center of the nodule is located in the v_center[2] slice of the img_array. We pass the node information to the make_mask() function and copy the generated masks and the image for the v_center[2]slice and the slice above and below it.
i = 0\r\nfor i_z in range(int(v_center[2])-1,int(v_center[2])+2):\r\n mask = make_mask(center,diam,i_z*spacing[2]+origin[2],width,height,spacing,origin)\r\n masks[i] = mask\r\n imgs[i] = matrix2int16(img_array[i_z])\r\n i+=1\r\nnp.save(output_path+\"images_%d.npy\" % (fcount) ,imgs)\r\nnp.save(output_path+\"masks_%d.npy\" % (fcount) ,masks)\r\nIn the make_mask() function it is worth noting that the mask coordinates have to match the ordering of the array coordinates. The x and y ordering is flipped. See the next to last line in the below code:
def make_mask(center,diam,z,width,height,spacing,origin):\r\n ...\r\n for v_x in v_xrange:\r\n for v_y in v_yrange:\r\n p_x = spacing[0]*v_x + origin[0]\r\n p_y = spacing[1]*v_y + origin[1]\r\n if np.linalg.norm(center-np.array([p_x,p_y,z]))<=diam:\r\n mask[int((p_y-origin[1])/spacing[1]),int((p_x-origin[0])/spacing[0])] = 1.0\r\n return(mask)\r\nShould we collect more slices from each scan?
\r\nSince the nodule locations are defined in terms of spheres, and the nodules are irregularly shaped, slices near the edges of the spheres may contain no nodule tissue. Using such slices would contaminate the training set with false positives. For this segmentation project, there is probably an optimal number of slices through a nodule that one should incorporate. For simplicity, we stick to 3 and only pull the slices centered on the largest nodule.
\r\nCheck to make sure the nodule masks look as expected:
\r\nimport matplotlib.pyplot as plt\r\nimgs = np.load(output_path+'images_0.npy')\r\nmasks = np.load(output_path+'masks_0.npy')\r\nfor i in range(len(imgs)):\r\n print \"image %d\" % i\r\n fig,ax = plt.subplots(2,2,figsize=[8,8])\r\n ax[0,0].imshow(imgs[i],cmap='gray')\r\n ax[0,1].imshow(masks[i],cmap='gray')\r\n ax[1,0].imshow(imgs[i]*masks[i],cmap='gray')\r\n plt.show()\r\n raw_input(\"hit enter to cont : \")\r\nThe image on the top left is the scan slice. The image on the top right is the node mask. The image on the bottom left is the masked slice, highlighting the node.
\r\nClose up on the nodule :
\r\nThe node masks seem to be constructed properly. The next step is to isolate the lungs in the images. We'll need to import some skimage image processing modules for this step. The general strategy is to threshold the image to isolate the regions within the image, and then to identify which of those regions are the lungs. The lungs have a high constrast with the surrounding tissue, so the thresholding is fairly straightforward. We use some ad-hoc criteria for eliminating the non-lung regions from the image which do not apply equally to all data sets.
\r\nIn addition to our previous imports, we'll make use of ...
\r\n![\"Example](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/6004/media/example_luna_mask.png\")
![\"Example](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/6004/media/example_luna_mask_detail.png\")
from skimage import morphology\r\nfrom skimage import measure\r\nfrom sklearn.cluster import KMeans\r\nfrom skimage.transform import resize\r\nThese steps are found in LUNA_segment_lung_ROI.py
\r\nThe arrays were loaded as dtype = np.float64 because KMeans in sklearn has a bug related to the precision of the input to KMeans.
We'll walk through the steps of isolating the lung ROI with img, which is a 512 X 512 slice from the set we extracted from the LUNA 2016 data, which looks like:
Our first step is to standardize the pixel values and take a look at the intensity distribution
\r\n![\"ROI](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/6004/media/ROI_step1.png\")
img = imgs_to_process[i]\r\n#Standardize the pixel values\r\nmean = np.mean(img)\r\nstd = np.std(img)\r\nimg = img-mean\r\nimg = img/std\r\nplt.hist(img.flatten(),bins=200)\r\nThe underflow peak near -1.5 is the black out-of-scanner part of the image. The peaks around 0.0 are the background and lung interior and the wide clumps from 1.0 to 2.0 are the non-lung-tissue and bone. The structure of this histogram varies throughout the data set. Two images are shown below that are typical of the data set. The one on the left has the same black background around a grey circular region of scanner data as is present in img. That black background is not present in the image on the right, making for a very different pixel value histogram.
We have to make sure that we set our threshold between the lung pixel values and the denser tissue pixel values. To do this, we reset the pixels with the minimum value to the average pixel value near the center of the picture and perform kmeans clustering with k=2. This seems to work well for both scenarios.
\r\n![\"ROI](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/6004/media/ROI_step1_hist.png\")
![\"ROI](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/6004/media/ROI_hist_diff.png\")
middle = img[100:400,100:400] \r\nmean = np.mean(middle) \r\nmax = np.max(img)\r\nmin = np.min(img)\r\n#move the underflow bins\r\nimg[img==max]=mean\r\nimg[img==min]=mean\r\nkmeans = KMeans(n_clusters=2).fit(np.reshape(middle,[np.prod(middle.shape),1]))\r\ncenters = sorted(kmeans.cluster_centers_.flatten())\r\nthreshold = np.mean(centers)\r\nthresh_img = np.where(img<threshold,1.0,0.0) # threshold the image\r\nWhich produces a satisfactory separation of regions for both types of images and eliminates the black halo in the one on the left
\r\nWe then use an erosion and dilation to fill in the incursions into the lungs region by radio-opaque tissue, followed by a selection of the regions based on the bounding box sizes of each region. The initial set of regions looks like
\r\n![\"ROI](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/6004/media/ROI_step2.png\")
eroded = morphology.erosion(thresh_img,np.ones([4,4]))\r\ndilation = morphology.dilation(eroded,np.ones([10,10]))\r\nlabels = measure.label(dilation)\r\nlabel_vals = np.unique(labels)\r\nplt.imshow(labels)\r\nThe cuts applied to each region bounding box were determined empirically and seem to work well for the LUNA data, but may not be generally applicable
\r\n![\"ROI](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/6004/media/ROI_step3.png\")
labels = measure.label(dilation)\r\nlabel_vals = np.unique(labels)\r\nregions = measure.regionprops(labels)\r\ngood_labels = []\r\nfor prop in regions:\r\n B = prop.bbox\r\n if B[2]-B[0]<475 and B[3]-B[1]<475 and B[0]>40 and B[2]<472:\r\n good_labels.append(prop.label)\r\nmask = np.ndarray([512,512],dtype=np.int8)\r\nmask[:] = 0\r\n#\r\n# The mask here is the mask for the lungs--not the nodes\r\n# After just the lungs are left, we do another large dilation\r\n# in order to fill in and out the lung mask \r\n#\r\nfor N in good_labels:\r\n mask = mask + np.where(labels==N,1,0)\r\nmask = morphology.dilation(mask,np.ones([10,10])) # one last dilation\r\nplt.imshow(mask,cmap='gray')\r\nThe next step in LUNA_segment_lung_ROI.py is applying the mask of the lung ROI to each of the images, cropping down to the bounding square of the lungs ROI, and then resizing the resulting image to 512 X 512.
\r\n![\"ROI](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/6004/media/ROI_step4.png\")
masks = np.load(working_path+\"lungmask_0.py\")\r\nimgs = np.load(working_path+\"images_0.py\")\r\nimgs = masks*imgs\r\n...crop to bounding square and resize to 512 X 512...
\r\nThen we perform some final pixel normalization. This is because the mask sends the non ROI area in the picture to 0, and that operation is not sensitive to the pixel value distribution. To fix this, we get the mean and standard deviation of the masked region and send the background (now zero) to the lower end of the pixel distribution (-1.2*stdev, which was chosen empirically).
#\r\n# renormalizing the masked image (in the mask region)\r\n#\r\nnew_mean = np.mean(img[mask>0]) \r\nnew_std = np.std(img[mask>0])\r\n#\r\n# Pushing the background color up to the lower end\r\n# of the pixel range for the lungs\r\n#\r\nold_min = np.min(img) # background color\r\nimg[img==old_min] = new_mean-1.2*new_std # resetting backgound color\r\nimg = img-new_mean\r\nimg = img/new_std\r\nThe final product is a set of lungs that is ready to be compiled into our training example set.
\r\nThese images and the correspondingly trimmed and rescaled masks are randomized and sent to a single file that contains a numpy array of dimension [<num_images>,1,512,512]. The 1 is important as the U-net is enabled for multiple channels.
![\"ROI](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/6004/media/ROI_step5.png\")
#\r\n# Writing out images and masks as 1 channel arrays for input into network\r\n#\r\nfinal_images = np.ndarray([num_images,1,512,512],dtype=np.float32)\r\nfinal_masks = np.ndarray([num_images,1,512,512],dtype=np.float32)\r\nfor i in range(num_images):\r\n final_images[i,0] = out_images[i]\r\n final_masks[i,0] = out_nodemasks[i]\r\nrand_i = np.random.choice(range(num_images),size=num_images,replace=False)\r\ntest_i = int(0.2*num_images)\r\nnp.save(working_path+\"trainImages.npy\",final_images[rand_i[test_i:]])\r\nnp.save(working_path+\"trainMasks.npy\",final_masks[rand_i[test_i:]])\r\nnp.save(working_path+\"testImages.npy\",final_images[rand_i[:test_i]])\r\nnp.save(working_path+\"testMasks.npy\",final_masks[rand_i[:test_i]])\r\nYou can check the ROI isolationg by examining the lung masks and the original files:
\r\nimgs = np.load(working path+'images_0.npy')\r\nlungmask = np.load(working_path+'lungmask_0.npy')\r\nfor i in range(len(imgs)):\r\n print \"image %d\" % i\r\n fig,ax = plt.subplots(2,2,figsize=[8,8])\r\n ax[0,0].imshow(imgs[i],cmap='gray')\r\n ax[0,1].imshow(lungmask[i],cmap='gray')\r\n ax[1,0].imshow(imgs[i]*lungmask[i],cmap='gray')\r\n plt.show()\r\n raw_input(\"hit enter to cont : \")\r\nThe network we'll be using is the U-net linked to the beginning of the tutorial which uses the keras framework. The loss function is the dice coefficient https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient comparing the predicted and actual node mask.
\r\nThe following code snippets are all taken from LUNA_train_unet.py
\r\nThe loss function is as follows:
\r\nsmooth = 1.\r\n# Tensorflow version for the model\r\ndef dice_coef(y_true, y_pred):\r\n y_true_f = K.flatten(y_true)\r\n y_pred_f = K.flatten(y_pred)\r\n intersection = K.sum(y_true_f * y_pred_f)\r\n return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\r\ndef dice_coef_loss(y_true, y_pred):\r\n return -dice_coef(y_true, y_pred)\r\nWhich is similar to the metric used to evaluate the Ultrasound Nerve Segmentation challenge that this network was originally written for (once again, see the link at the beginning of this tutorial).
\r\nThe function call
\r\nmodel = get_unet() \r\nmodel_checkpoint = ModelCheckpoint('unet.hdf5', monitor='loss', save_best_only=True)\r\nwill compile and return the model and tell keras to save the model weights during checkpoints. If you want to load the best weights from a previous training session or use the weights included in this tutorial's repo, load the weight file with the line
\r\nmodel.load_weights('unet.hdf5')\r\nCalling LUNA_train_unet.py from the command line will automatically attempt to load a unet.hdf5 file from the current directory, train it according to the parameters set by the line
\r\nmodel.fit(imgs_train, imgs_mask_train, batch_size=2, nb_epoch=20, \r\n verbose=1, shuffle=True,callbacks=[model_checkpoint])\r\nin the script, and make predictions on the test set.
\r\nnum_test = len(imgs_test)\r\nimgs_mask_test = np.ndarray([num_test,1,512,512],dtype=np.float32)\r\nfor i in range(num_test):\r\n imgs_mask_test[i] = model.predict([imgs_test[i:i+1]], verbose=0)[0]\r\nnp.save('masksTestPredicted.npy', imgs_mask_test)\r\nmean = 0.0\r\nfor i in range(num_test):\r\n mean+=dice_coef_np(imgs_mask_test_true[i,0], imgs_mask_test[i,0])\r\nmean/=num_test\r\nprint(\"Mean Dice Coeff : \",mean)\r\nThe model.predict() function can take more than one case at a time, but that can quickly overload a GPU, so we are looping over individual cases.
The final results for this tutorial were produced using a multi-GPU machine using TitanX's. For a home GPU computation benchmark, a personal set up with a GTX970 we were able to run 20 epochs with a training set size of 320 and batch size of 2 in about an hour. We started obtaing reasonable nodule mask predictions after about 3 hours of training once the reported loss value approached 0.3.
\r\nAn example segmentation is given here for the three slices taken from a patient scan. The perfect circle is the \"true\" node mask from the LUNA annotationc.csv file, and the red is the predicted node region from the segmenter. The original image is given in the top right.
\r\nNow we are ready to begin training a classifier using our image segmentation from the previous sections to generate features.
\r\nThe Data Science Bowl training data set must be fed through the segmenter, which can be done largely by reusing the code used to treat the LUNA data. There are two points where this deviates.
\r\nFirst of all, the DSB data is in dicom format, which can be read using the pydicom module
\r\n![\"Example](\"https://storage.googleapis.com/kaggle-media/competitions/kaggle/6004/media/example_segmentation.png\")
import dicom\r\ndc = dicom.read_file(filename)\r\nimg = dc.pixel_array\r\nSecondly, in order to locate nodes in the scans, every layer of the scan must be run through the segmenter, and thus every layer must also be subject to the image processing to mask off the ROI. This can be a very time consuming process.
\r\nWe start by characterizing some of the features of the nodule maps and putting them into a feature vector that we can use for classification purposes. Our list of features is by no means exhaustive and is meant to illustrate the process of developing some meterics that could be used to characterize the segmented regions where nodules are likely.
\r\nWe encourage you to play with adding some new features, explore convolutional models for extracting features from the region of interest directly. We have included some features about the average size, morphology, and position within the image for use in model building.
\r\ndef getRegionMetricRow(fname = \"nodules.npy\"):\r\n seg = np.load(fname)\r\n nslices = seg.shape[0]\r\n #metrics\r\n totalArea = 0.\r\n avgArea = 0.\r\n maxArea = 0.\r\n avgEcc = 0.\r\n avgEquivlentDiameter = 0.\r\n stdEquivlentDiameter = 0.\r\n weightedX = 0.\r\n weightedY = 0.\r\n numNodes = 0.\r\n numNodesperSlice = 0.\r\n # do not allow any nodes to be larger than 10% of the pixels to eliminate background regions\r\n maxAllowedArea = 0.10 * 512 * 512 \r\n areas = []\r\n eqDiameters = []\r\n for slicen in range(nslices):\r\n regions = getRegionFromMap(seg[slicen,0,:,:])\r\n for region in regions:\r\n if region.area > maxAllowedArea:\r\n continue\r\n totalArea += region.area\r\n areas.append(region.area)\r\n avgEcc += region.eccentricity\r\n avgEquivlentDiameter += region.equivalent_diameter\r\n eqDiameters.append(region.equivalent_diameter)\r\n weightedX += region.centroid[0]*region.area\r\n weightedY += region.centroid[1]*region.area\r\n numNodes += 1\r\n weightedX = weightedX / totalArea \r\n weightedY = weightedY / totalArea\r\n avgArea = totalArea / numNodes\r\n avgEcc = avgEcc / numNodes\r\n avgEquivlentDiameter = avgEquivlentDiameter / numNodes\r\n stdEquivlentDiameter = np.std(eqDiameters)\r\n maxArea = max(areas)\r\n numNodesperSlice = numNodes*1. / nslices\r\n return np.array([avgArea,maxArea,avgEcc,avgEquivlentDiameter,\\\r\n stdEquivlentDiameter, weightedX, weightedY, numNodes, numNodesperSlice])\r\ndef getRegionFromMap(slice_npy):\r\n thr = np.where(slice_npy > np.mean(slice_npy),0.,1.0)\r\n label_image = label(thr)\r\n labels = label_image.astype(int)\r\n regions = regionprops(labels)\r\n return regions\r\nimport pickle\r\ndef createFeatureDataset(nodfiles=None):\r\n if nodfiles == None:\r\n noddir = \"nodulesdir/\"\r\n nodfiles = glob(noddir +\"*npy\")\r\n # dict with mapping between truth and \r\n truthdata = pickle.load(open(\"truthdict.pkl\",'r'))\r\n numfeatures = 9\r\n feature_array = np.zeros((len(nodfiles),numfeatures))\r\n truth_metric = np.zeros((len(nodfiles)))\r\n for i,nodfile in enumerate(nodfiles):\r\n patID = nodfile.split(\"_\")[2]\r\n truth_metric[i] = truthdata[int(patID)]\r\n feature_array[i] = getRegionMetricRow(nodfile)\r\n np.save(\"dataY.npy\", truth_metric)\r\n np.save(\"dataX.npy\", feature_array)\r\nOnce we've created our feature dataset, we can create a simple classifer based on the featureset to determine the cancer diagnosis. We've chose to illustrate two popular classifiers Random Forest and XGBoost, because they are both robust to overfitting and quick to train. The common python implementation of XGBoost also allows for easy modification of the objective function and reweighting the class importance, so that we can reweight the optimization problem to give more weight to cancer cases.
\r\nfrom sklearn import cross_validation\r\nfrom sklearn.cross_validation import StratifiedKFold as KFold\r\nfrom sklearn.metrics import classification_report\r\nfrom sklearn.ensemble import RandomForestClassifier as RF\r\nimport xgboost as xgb\r\nX = np.load(\"dataX.npy\")\r\nY = np.load(\"dataY.npy\")\r\n#Random Forest\r\nkf = KFold(Y, n_folds=3)\r\ny_pred = Y * 0\r\ny_pred_prob = Y * 0\r\nfor train, test in kf:\r\n X_train, X_test, y_train, y_test = X[train,:], X[test,:], Y[train], Y[test]\r\n clf = RF(n_estimators=100, n_jobs=3)\r\n clf.fit(X_train, y_train)\r\n y_pred[test] = clf.predict(X_test)\r\n y_pred_prob[test] = clf.predict_proba(X_test)[:,1]\r\nprint ('Random Forest')\r\nprint classification_report(Y, y_pred, target_names=[\"No Cancer\", \"Cancer\"])\r\nprint(\"logloss\",logloss(Y, y_pred_prob))\r\n#XGBoost\r\nprint (\"XGBoost\")\r\nkf = KFold(Y, n_folds=3)\r\ny_pred = Y * 0\r\ny_pred_prob = Y * 0\r\nfor train, test in kf:\r\n X_train, X_test, y_train, y_test = X[train,:], X[test,:], Y[train], Y[test]\r\n clf = xgb.XGBClassifier(objective=\"binary:logistic\", scale_pos_weight=3 )\r\n clf.fit(X_train, y_train)\r\n y_pred[test] = clf.predict(X_test)\r\n y_pred_prob[test] = clf.predict_proba(X_test)[:,1]\r\nprint classification_report(Y, y_pred, target_names=[\"No Cancer\", \"Cancer\"])\r\nprint(\"logloss\",logloss(Y, y_pred_prob))\r\n# All Cancer\r\nprint \"Predicting all positive\"\r\ny_pred = np.ones(Y.shape)\r\nprint classification_report(Y, y_pred, target_names=[\"No Cancer\", \"Cancer\"])\r\nprint(\"logloss\",logloss(Y, y_pred))\r\n# No Cancer\r\nprint \"Predicting all negative\"\r\ny_pred = Y*0\r\nprint classification_report(Y, y_pred, target_names=[\"No Cancer\", \"Cancer\"])\r\nprint(\"logloss\",logloss(Y, y_pred))\r\nWe compare the results for Random forest, XGBoost, and two models consisting of only predicting cancer and only predicting no cancer.
\r\nRandom Forest\r\n precision recall f1-score support\r\n No Cancer 0.81 0.98 0.89 463\r\n Cancer 0.17 0.02 0.03 107\r\navg / total 0.69 0.80 0.73 570\r\n('logloss', 0.52600332670816652)\r\nXGBoost\r\n precision recall f1-score support\r\n No Cancer 0.83 0.86 0.84 463\r\n Cancer 0.27 0.21 0.24 107\r\navg / total 0.72 0.74 0.73 570\r\n('logloss', 0.5700685138621493)\r\nPredicting all positive\r\n precision recall f1-score support\r\n No Cancer 0.00 0.00 0.00 463\r\n Cancer 0.19 1.00 0.32 107\r\navg / total 0.04 0.19 0.06 570\r\n('logloss', 28.055831025357818)\r\nPredicting all negative\r\n precision recall f1-score support\r\n No Cancer 0.81 1.00 0.90 463\r\n Cancer 0.00 0.00 0.00 107\r\navg / total 0.66 0.81 0.73 570\r\n('logloss', 6.4835948671148085)\r\nWe've given you a framework for approaching this problem that combines a deep learning based segmentation approach with an older computer vision approach of hand designed features. From here you have lots of ways to improve the u-net model with more data, or additional preprocessing. The classification piece could be replaced by a another convolutional nueral net or you could take more advantage of the 3d nature of the nodules that we've mostly considered as independant 2d slices here.
\r\nAWS research and technical computing website: www.aws.amazon.com/rtc
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false }, { "name": "Engagement Contest", "content": "The Data Science Bowl Engagement contest is open to all individuals over the age of 18 at the time of entry and to all validly formed legal entities that have not declared or been declared in bankruptcy. The contest will contain three phases, each with a separate mini challenge and instructions on how to participate. The instructions on how to participate in the first mini-challenge/phase are listed below.
\n\n\n\nMini-Challenge # 1: February 1 – 15 3:00 p.m. ET; Prize: $2,500
\n\nMini-Challenge #2: February 15 – 28 3:00 p.m. ET; Prize: $2,500
\n\nMini-Challenge #3: March 1 – 15 3:00 p.m. ET; Prize: $5,000
\n\nUsing Instagram and/or Twitter, share a post using the hashtags #DataSciBowl AND #Data4Good. Start sharing NOW until February 28th to qualify to win $2,500!
\n\nWe want you to share your commitment to the Data Science Bowl (participating, watching, promoting, other something else) by writing a pledge to #Data4Good. What will #DataSciBowl mean to you? You might choose to pledge your passion, your best guess, your undivided attention. Or the challenge might inspire you to pledge a donation or volunteer with a nonprofit. Show us how the power of data for good inspires you.
\n\nSample Posts:
\nELIGIBILITY NOTE: Members of the following organizations are invited to participate in the competition, however, they will not be eligible to win a cash prize: Booz Allen Hamilton, Amazon Web Services, NVIDIA, NCI Division of Cancer Treatment and Diagnosis, NCI Division of Cancer Prevention, Functional Image Analysis group, Radboud University, The Cancer Imaging Archive, Lahey Hospital & Medical Center, and Copenhagen University Hospital. Additionally, anyone who has previously accessed any of the data sources outside the purview of this competition will not be eligible to win a cash prize. Officers, directors, employees and advisory board members (and their immediate families and members of the same household) of the Competition Sponsor, Kaggle and their respective affiliates, subsidiaries, contractors (with the express exception of Kaggle’s authorized Kaggle Community Evangelists), agents, judges and advertising and promotion agencies are not eligible for prizes. Residents of a country designated by the United States Treasury’s Office of Foreign Assets Control (see http://www.treasury.gov/resource-center/sanctions/SDN-List/Pages/default.aspx for additional information) are not eligible to receive prizes.
", "isPublished": false, "hasIsPublished": false, "postTitle": "", "hasPostTitle": false, "mimeType": "", "hasMimeType": false } ] }