![\"ROC\"](\"https://retinopaty.blob.core.windows.net/ndsb3/roc_public_lb2.png\" "\\"\\"")
Hello, here a very quick 1st summary on my part our 2nd place solution. Of course a real writeup and code will follow.
\n\nTools.
\nI used Keras + Tensorflow on windows 64.\nIt just came out and wanted to give it a go.\nFor visualisation + debugging I used a specially built viewer.\nI used that one for checking labels, predictions etc.
General approach.
\nThe main insight for me (and as a coincidence Daniel) was to use the malignancy information from the LIDC database as labeled by the doctors and combine it with the LUNA16 negative labels.\nI trained up a 32x32x32 3d convnet to detect nodules AND predict malignancy in one go.\nI did not use segmented lungs but rather trained on the full images to not miss any nodules located near the edges.
Furthermore it was appearantly extremely important you not let yourself get too distracted by the leaderboard. \nAs Daniel said \"In CV we trust\".
\n\nPreprocessing
\nScale everything to 1x1x1 mm. Make sure that all patients have the same orientation in both LUNA and NDSB.\nConnect the LIDC data to the LUNA data. I expressed all coordinates as 0.0 to 1.0 so that it was scale independent.\nI used my viewer to check if all positive and negative labels.
Labels
\nI used ALL positive LIDC doctor assessments as positive, malignancy label info. Even when only one doctor saw something.\nThe though was that the malignancy info was still very valuable. Then I used the negative entries of candidatesv2.csv from LUNA16 as negatives labels.\nNote that some overlapped with nodules that were tagged by only one doctor.\nTo train on the full images I samples extra negative labels on non-lung tissue. I found that tissue by taking the segmenter from the forum and sampling around the mask edges.\nThen I trained a first network and predicted on the LUNA set. All false positives I added as extra negative examples.\nAll in all ca. 400.000 candidates, 100.000 non lung tissue, 10.000 false positives, 2000 LIDC positives.
For my 2nd model I predicted on the NDSB data and then played \"doctor julian\" and manually annotated positive nodules and false positives and added those to the trainset.
\n\nAnother important observation was that in LUNA16 nodules > 3cm (masses) were NOT labeled. So I excluded negative labels that overlapped these masses since it migt confuse the network.
\n\n3D convnet nodule, malignancy detector.
\nI trained a 32x32x32 3D convnet on extracted, labeled crops . The architecture was a butchered version of the C3D architecture, which is basically VGG like.\nI spent very (too?) little time optimizing the network architecture.
Somehow I made it so to do multitask learning. So I trained on Nodule Y/N, Malignancy in one go. This worked best for me but Daniel\ndid it in 2 stages.
\n\nThe labels were very unbalanced so I upsampled the positive cases onto 1:20. Heavy (lossless) augmentation somehow worked.\nI'm still a bit surprised how few positive cases were needed.
\n\nUsing the trained network I prediced on the NDSB data using a 12 mm stepsize which gave a lattice with nodule malignancies.\nFor every patient I only used the max malignancy and the z-location. I did this at 3 zoom levels to also be able to detect bigger nodules.
\n\nStrange tissue detector.
\nA small part of the solution was a \"strange tissue\" detector. I basically used some examples of strange tissue from LUNA, NDSB, a few axamples from the extra data thread.\nI labeled them with the same program I used for the 2nd NDSB. Then I trained a small U-Net. This net gave per patient the amount of strange tissue.\nI also tried the same trick for overal holistic info and empysema but they did not have any significant impact.
Final classifier.
\nI used Xgboost on the max malignancy at 3 zoom levels, the z-location and the amount of strange tissue.\nI used min-childweight of 60 to prevent overfitting.\nLocal CV was around 0.39 but the LB had a will of its own and I had a very hard time finding improvements that improved both local CV and LB.
Daniels models.
\nI joined with Daniel Hammack and we both were convinced we had an original idea. Turned out that we both used malignancy info from LIDC.\nHowever, his network architecture was 64x64x64, resnet like and used 2 stages. Also he used extra nodule info from LIDC. So in the end our models were still quite complimentary.\nMore info on his stuff will be in his writeup. My strategy was to average the models into one final solution.
My thanks to Daniel.
\nHe really kept his cool when the LB was at complete chaos with all the overfitting taking place.\nYou are extremely bright and much more sophisticated than me in your ML approach :)
Really good point about trusting in CV and not over-fitting the leaderboard.
", "votes": 3, "replies": [] }, { "id": 174934, "author_name": "Fréderic Godin", "author_url": "", "post_date": "2017-04-13T08:40:16.960000", "content": "Congratz Julian & Daniel!
", "votes": 3, "replies": [] }, { "id": 176145, "author_name": "SHU-JU TU", "author_url": "", "post_date": "2017-04-19T03:58:57.727000", "content": "Hi
\n\nCongratulate on your great work!
\n\nI notice that your log-loss is around 0.4.\nIs there a way that you can convert this metric to something like true-positive rate, false-positive rate, true-negative rate, or false-negative rate?
\n\nWarm regards,\nShu-Ju
", "votes": 2, "replies": [ { "id": 176193, "author_name": "Julian de Wit", "author_url": "", "post_date": "2017-04-19T10:33:35.330000", "content": "Hello, thanks!
\nFor stage 1. public LB. Logloss 0.43 AUC was 0.85.
\nFor stage2 (0.40) we do not have the labels..
Cool mindset on handling such competitive ML subject!
", "votes": 2, "replies": [] }, { "id": 175118, "author_name": "jsardinha", "author_url": "", "post_date": "2017-04-13T20:57:39.603000", "content": "Congrats Julian and Daniel and thanks for sharing.
\n\nDo you know how big of a difference the strange tissue detector made to your overall score?
", "votes": 2, "replies": [ { "id": 175176, "author_name": "Julian de Wit", "author_url": "", "post_date": "2017-04-14T06:03:32.430000", "content": "Not much but I'd like to think that it just made the difference.\nThere are about 10 cases in the trainset with a lot of strange tissue and no or small nodules.\nOf those 10 cases about 5 of them are cancer. So with strange tissue the chance of being cancer is higher. Instead of a hard negative the classifier with raise the chance a bit. So instead of a big logloss penalty you get a more mild penalty.
\n\nHere an example.\n![\"enter](\"https://retinopaty.blob.core.windows.net/ndsb3/tissue.png\" "\\"\\"")
Thanks a lot for sharing! Do you mind sharing what kind of hardware you were using? Also, is your visualisation tool publicly available?
", "votes": 2, "replies": [ { "id": 175178, "author_name": "Julian de Wit", "author_url": "", "post_date": "2017-04-14T06:08:56.467000", "content": "I trained on backup computer with a GTX 980.
\nSince I was on 32x32x32 you did not need a heavy card.
\nI think having SSD harddrives was more imprortant for high throughput.
The viewer is a hasty job and depends on specific folder structures.\nIt would be a pain to run.
", "votes": 0, "replies": [] } ] }, { "id": 174932, "author_name": "sh1ng", "author_url": "", "post_date": "2017-04-13T08:29:26.633000", "content": "Big thanks for your sharing and congratulations!
\n\nCould you elaborate a little more about loss function in each case. Was it cross entropy, dice or something else?
\n\nBy predicting nodule and malignancy do you mean two masks with probability per pixel(1x1x1 cube) or per whole patch?
", "votes": 2, "replies": [] }, { "id": 175976, "author_name": "Georgios Sarantitis", "author_url": "", "post_date": "2017-04-18T10:39:56.847000", "content": "Thanks for sharing! Great job, it was really helpful to me since I am just starting my journey on image recognition competitions..!
", "votes": 1, "replies": [] }, { "id": 175465, "author_name": "Dex De Guzman", "author_url": "", "post_date": "2017-04-15T16:01:37.487000", "content": "congrats julian!! when r u planning on posting the code?
", "votes": 1, "replies": [ { "id": 175469, "author_name": "Julian de Wit", "author_url": "", "post_date": "2017-04-15T16:53:22.227000", "content": "Thanks,
\nYes, we'll publish the code.
\nEverybody that wants to win a prize has to if I understood correctly.
\nJust have to clean up a bit. :)
Congrats Julian and Daniel! Thank you for sharing your approach!
", "votes": 1, "replies": [] }, { "id": 175271, "author_name": "Junjie Bai", "author_url": "", "post_date": "2017-04-14T17:24:59.767000", "content": "Congratulations, Julian & Daniel! Thanks for sharing. I am wondering did you use convNet raw features (likely fully-connected layer input) from max malignancy locations? Or did you just use the scalar max malignancy probability?
\n\nAlso, does you final submission still have a big gap between local CV and stage 1 LB? We also had a very difficult time getting both improving at the same time.
", "votes": 1, "replies": [ { "id": 175291, "author_name": "dhammack", "author_url": "", "post_date": "2017-04-14T18:51:49.290000", "content": "\n\n\ndid you use convNet raw features (likely fully-connected layer input) from max malignancy locations? Or did you just use the scalar max malignancy probability?
\n
We did not chop off the last layer of the network if that is what you are asking. We both used the actual malignancy predictions to generate some 'hand crafted' features like:
\n\nIt may be possible to slightly improve the score by doing what you suggest and using the inputs to the final layer at the most malignant nodule as features. I tried a variant of this earlier in the competition without success but I never tried it again later.
\n\nAlso, does you final submission still have a big gap between local CV and stage 1 LB?
\n\nYes. For example, my 'second ensemble' had a local CV score of 0.390 and a stage 1 LB score of 0.451. It was very hard to ignore the stage 1 LB and trust our CV score! Ultimately I ended up basically ignoring the stage 1 LB altogether and just focusing on improving CV as CV had 1400 scans compared to the 200 in the public LB.
", "votes": 2, "replies": [] }, { "id": 175320, "author_name": "Junjie Bai", "author_url": "", "post_date": "2017-04-14T22:01:44.300000", "content": "Thank you very much for the detailed sharing, Daniel!
\n\nI am deeply impressed by how well your results are without including the \"black art\" convNet features. Even though convNet is so popular on image tasks right now, it's good to know that we don't have to rely on such black features to achieve top results (or at least just keep the black features inside the black box itself).
\n\nFor the big gap between local CV and LB, it's really courageous to ignore LB altogether and just focus on local CV! I was too scared to do this, but guess I could be braver next time.
", "votes": 0, "replies": [] } ] }, { "id": 175010, "author_name": "Jun Shi", "author_url": "", "post_date": "2017-04-13T13:28:03.967000", "content": "Congrats! Thanks for sharing.\nWhat do you mean by \"3 zoom levels\"? 3 different 3d-filter sizes (receptive field) at the first level of your 3d-cnn?
", "votes": 1, "replies": [ { "id": 175020, "author_name": "Julian de Wit", "author_url": "", "post_date": "2017-04-13T13:44:46.040000", "content": "I was not clear about that.
\nI trained the net on 1x1x1 mm per pixel.
\nSince the receptive field size was 32x32x32 it sometimes missed larger nodules.
\nSo during prediction I predicted on 3 scales
\nI used the same network every time.
\nIt was a cheeky fix but somehow it worked.
I see. You resampled test scans three times, but using the same network instead. Thanks for sharing.
", "votes": 0, "replies": [] } ] }, { "id": 174974, "author_name": "Aleś", "author_url": "", "post_date": "2017-04-13T10:39:01.687000", "content": "Dear all, I was trying to find the malignancy labels for LIDC data, but didn't quite manage to find them. Could you give a hint where those labels are stored?
", "votes": 1, "replies": [ { "id": 174976, "author_name": "Julian de Wit", "author_url": "", "post_date": "2017-04-13T10:44:38.540000", "content": "This link
\nThen detailed description.
\nThen \"XML File Documentation\"
thanks!
", "votes": 0, "replies": [] } ] }, { "id": 174962, "author_name": "Julian de Wit", "author_url": "", "post_date": "2017-04-13T10:01:30.800000", "content": "@sh1ng\nLoss for nodule yes/no: binary cross entropy\nLoss for malignancy: Squared Error\nLoss for tissue segmenter: Dice\nPredictions were not per 1mm but I used a stride of 12mm.\nSo a 240x240x240 patient scan gave 20x20x20 predictions.\nYes one for malignancy and one for nodule yes/no..
\n\nI will post some screenshots..
\n\n@FangzouLiao\nYes.. 32x32x32 was a bit small but combined with different zoom levels it was quite adequate.\nThe 0.429 LB was a result of blending our models. Best single model was 0.441 LB, 0.385 CV.\nSomehow blending Daniel's stuff with mine did good on LB.\nHowever.. I really struggled with LB. Better CV usually gave worse LB and vise versa.\nI looked for directional improvements for both at the same time. Daniel's mantra was \"In CV we trust\" :)
", "votes": 1, "replies": [ { "id": 174967, "author_name": "Fangzhou Liao", "author_url": "", "post_date": "2017-04-13T10:21:07.360000", "content": "Yes, I finally realized that the LB is full of outliers and is not trustworthy, so I asked you the question in your last post. Your confirmation gives me the confidence to my CV. Thank you for that
", "votes": 1, "replies": [] } ] }, { "id": 174954, "author_name": "Argmen", "author_url": "", "post_date": "2017-04-13T09:31:38.700000", "content": "Ok..I noticed the malignancy info but do not have much time to try that - -
", "votes": 1, "replies": [] }, { "id": 174935, "author_name": "Fangzhou Liao", "author_url": "", "post_date": "2017-04-13T08:43:59.923000", "content": "Cool! 32x32x32? This model architecture is much simpler than mine! Does it work well?
\n\nAnd how do you (and other guys) achieved 0.42 in stage 1? I can not achieve this score even after fine tuning the model on LB dataset.
", "votes": 1, "replies": [] }, { "id": 828728, "author_name": "karnabharath", "author_url": "", "post_date": "2020-05-01T08:46:20.790000", "content": "Hi,
\n\nAny links for the final code ?
\n\nThnx
", "votes": 0, "replies": [] }, { "id": 175065, "author_name": "PhilGreenwood", "author_url": "", "post_date": "2017-04-13T16:51:15.223000", "content": "Thanks very much for the detail
", "votes": 1, "replies": [] }, { "id": 175061, "author_name": "Jie Yang", "author_url": "", "post_date": "2017-04-13T16:39:51.430000", "content": "Congrats! Thanks for sharing!
", "votes": 1, "replies": [] }, { "id": 174928, "author_name": "Keyur K. Mithawala", "author_url": "", "post_date": "2017-04-13T07:56:04.150000", "content": "Thank you for the walk-through Julian!
", "votes": 1, "replies": [] } ], "comment_count": 31, "messages": [ { "id": 174900, "postDate": "2017-04-13T06:20:22.873Z", "content": "Hello, here a very quick 1st summary on my part our 2nd place solution. Of course a real writeup and code will follow.
\n\nTools.
\nI used Keras + Tensorflow on windows 64.\nIt just came out and wanted to give it a go.\nFor visualisation + debugging I used a specially built viewer.\nI used that one for checking labels, predictions etc.
General approach.
\nThe main insight for me (and as a coincidence Daniel) was to use the malignancy information from the LIDC database as labeled by the doctors and combine it with the LUNA16 negative labels.\nI trained up a 32x32x32 3d convnet to detect nodules AND predict malignancy in one go.\nI did not use segmented lungs but rather trained on the full images to not miss any nodules located near the edges.
Furthermore it was appearantly extremely important you not let yourself get too distracted by the leaderboard. \nAs Daniel said \"In CV we trust\".
\n\nPreprocessing
\nScale everything to 1x1x1 mm. Make sure that all patients have the same orientation in both LUNA and NDSB.\nConnect the LIDC data to the LUNA data. I expressed all coordinates as 0.0 to 1.0 so that it was scale independent.\nI used my viewer to check if all positive and negative labels.
Labels
\nI used ALL positive LIDC doctor assessments as positive, malignancy label info. Even when only one doctor saw something.\nThe though was that the malignancy info was still very valuable. Then I used the negative entries of candidatesv2.csv from LUNA16 as negatives labels.\nNote that some overlapped with nodules that were tagged by only one doctor.\nTo train on the full images I samples extra negative labels on non-lung tissue. I found that tissue by taking the segmenter from the forum and sampling around the mask edges.\nThen I trained a first network and predicted on the LUNA set. All false positives I added as extra negative examples.\nAll in all ca. 400.000 candidates, 100.000 non lung tissue, 10.000 false positives, 2000 LIDC positives.
For my 2nd model I predicted on the NDSB data and then played \"doctor julian\" and manually annotated positive nodules and false positives and added those to the trainset.
\n\nAnother important observation was that in LUNA16 nodules > 3cm (masses) were NOT labeled. So I excluded negative labels that overlapped these masses since it migt confuse the network.
\n\n3D convnet nodule, malignancy detector.
\nI trained a 32x32x32 3D convnet on extracted, labeled crops . The architecture was a butchered version of the C3D architecture, which is basically VGG like.\nI spent very (too?) little time optimizing the network architecture.
Somehow I made it so to do multitask learning. So I trained on Nodule Y/N, Malignancy in one go. This worked best for me but Daniel\ndid it in 2 stages.
\n\nThe labels were very unbalanced so I upsampled the positive cases onto 1:20. Heavy (lossless) augmentation somehow worked.\nI'm still a bit surprised how few positive cases were needed.
\n\nUsing the trained network I prediced on the NDSB data using a 12 mm stepsize which gave a lattice with nodule malignancies.\nFor every patient I only used the max malignancy and the z-location. I did this at 3 zoom levels to also be able to detect bigger nodules.
\n\nStrange tissue detector.
\nA small part of the solution was a \"strange tissue\" detector. I basically used some examples of strange tissue from LUNA, NDSB, a few axamples from the extra data thread.\nI labeled them with the same program I used for the 2nd NDSB. Then I trained a small U-Net. This net gave per patient the amount of strange tissue.\nI also tried the same trick for overal holistic info and empysema but they did not have any significant impact.
Final classifier.
\nI used Xgboost on the max malignancy at 3 zoom levels, the z-location and the amount of strange tissue.\nI used min-childweight of 60 to prevent overfitting.\nLocal CV was around 0.39 but the LB had a will of its own and I had a very hard time finding improvements that improved both local CV and LB.
Daniels models.
\nI joined with Daniel Hammack and we both were convinced we had an original idea. Turned out that we both used malignancy info from LIDC.\nHowever, his network architecture was 64x64x64, resnet like and used 2 stages. Also he used extra nodule info from LIDC. So in the end our models were still quite complimentary.\nMore info on his stuff will be in his writeup. My strategy was to average the models into one final solution.
My thanks to Daniel.
\nHe really kept his cool when the LB was at complete chaos with all the overfitting taking place.\nYou are extremely bright and much more sophisticated than me in your ML approach :)
Really good point about trusting in CV and not over-fitting the leaderboard.
", "rawMarkdown": "Really good point about trusting in CV and not over-fitting the leaderboard.", "votes": 3 }, { "id": 174934, "postDate": "2017-04-13T08:40:16.960Z", "content": "Congratz Julian & Daniel!
", "rawMarkdown": "Congratz Julian & Daniel!", "votes": 3 }, { "id": 176145, "postDate": "2017-04-19T03:58:57.727Z", "content": "Hi
\n\nCongratulate on your great work!
\n\nI notice that your log-loss is around 0.4.\nIs there a way that you can convert this metric to something like true-positive rate, false-positive rate, true-negative rate, or false-negative rate?
\n\nWarm regards,\nShu-Ju
", "rawMarkdown": "Hi\n\nCongratulate on your great work!\n\nI notice that your log-loss is around 0.4.\nIs there a way that you can convert this metric to something like true-positive rate, false-positive rate, true-negative rate, or false-negative rate?\n\nWarm regards,\nShu-Ju\n\n ", "votes": 2, "replies": [ { "id": 176193, "postDate": "2017-04-19T10:33:35.330Z", "content": "Hello, thanks!
\nFor stage 1. public LB. Logloss 0.43 AUC was 0.85.
\nFor stage2 (0.40) we do not have the labels..
Cool mindset on handling such competitive ML subject!
", "rawMarkdown": "Cool mindset on handling such competitive ML subject!", "votes": 2 }, { "id": 175118, "postDate": "2017-04-13T20:57:39.603Z", "content": "Congrats Julian and Daniel and thanks for sharing.
\n\nDo you know how big of a difference the strange tissue detector made to your overall score?
", "rawMarkdown": "Congrats Julian and Daniel and thanks for sharing. \n\nDo you know how big of a difference the **strange tissue detector** made to your overall score? ", "votes": 2, "replies": [ { "id": 175176, "postDate": "2017-04-14T06:03:32.430Z", "content": "Not much but I'd like to think that it just made the difference.\nThere are about 10 cases in the trainset with a lot of strange tissue and no or small nodules.\nOf those 10 cases about 5 of them are cancer. So with strange tissue the chance of being cancer is higher. Instead of a hard negative the classifier with raise the chance a bit. So instead of a big logloss penalty you get a more mild penalty.
\n\nHere an example.\n
Thanks a lot for sharing! Do you mind sharing what kind of hardware you were using? Also, is your visualisation tool publicly available?
", "rawMarkdown": "Thanks a lot for sharing! Do you mind sharing what kind of hardware you were using? Also, is your visualisation tool publicly available?", "votes": 2, "replies": [ { "id": 175178, "postDate": "2017-04-14T06:08:56.467Z", "content": "I trained on backup computer with a GTX 980.
\nSince I was on 32x32x32 you did not need a heavy card.
\nI think having SSD harddrives was more imprortant for high throughput.
The viewer is a hasty job and depends on specific folder structures.\nIt would be a pain to run.
", "rawMarkdown": "I trained on backup computer with a GTX 980.Big thanks for your sharing and congratulations!
\n\nCould you elaborate a little more about loss function in each case. Was it cross entropy, dice or something else?
\n\nBy predicting nodule and malignancy do you mean two masks with probability per pixel(1x1x1 cube) or per whole patch?
", "rawMarkdown": "Big thanks for your sharing and congratulations!\n\n Could you elaborate a little more about loss function in each case. Was it cross entropy, dice or something else?\n\nBy predicting nodule and malignancy do you mean two masks with probability per pixel(1x1x1 cube) or per whole patch? ", "votes": 2 }, { "id": 175976, "postDate": "2017-04-18T10:39:56.847Z", "content": "Thanks for sharing! Great job, it was really helpful to me since I am just starting my journey on image recognition competitions..!
", "rawMarkdown": "Thanks for sharing! Great job, it was really helpful to me since I am just starting my journey on image recognition competitions..!", "votes": 1 }, { "id": 175465, "postDate": "2017-04-15T16:01:37.487Z", "content": "congrats julian!! when r u planning on posting the code?
", "rawMarkdown": "congrats julian!! when r u planning on posting the code? ", "votes": 1, "replies": [ { "id": 175469, "postDate": "2017-04-15T16:53:22.227Z", "content": "Thanks,
\nYes, we'll publish the code.
\nEverybody that wants to win a prize has to if I understood correctly.
\nJust have to clean up a bit. :)
Congrats Julian and Daniel! Thank you for sharing your approach!
", "rawMarkdown": "Congrats Julian and Daniel! Thank you for sharing your approach!", "votes": 1 }, { "id": 175271, "postDate": "2017-04-14T17:24:59.767Z", "content": "Congratulations, Julian & Daniel! Thanks for sharing. I am wondering did you use convNet raw features (likely fully-connected layer input) from max malignancy locations? Or did you just use the scalar max malignancy probability?
\n\nAlso, does you final submission still have a big gap between local CV and stage 1 LB? We also had a very difficult time getting both improving at the same time.
", "rawMarkdown": "Congratulations, Julian & Daniel! Thanks for sharing. I am wondering did you use convNet raw features (likely fully-connected layer input) from max malignancy locations? Or did you just use the scalar max malignancy probability? \n\nAlso, does you final submission still have a big gap between local CV and stage 1 LB? We also had a very difficult time getting both improving at the same time. ", "votes": 1, "replies": [ { "id": 175291, "postDate": "2017-04-14T18:51:49.290Z", "content": "\n\n\ndid you use convNet raw features (likely fully-connected layer input) from max malignancy locations? Or did you just use the scalar max malignancy probability?
\n
We did not chop off the last layer of the network if that is what you are asking. We both used the actual malignancy predictions to generate some 'hand crafted' features like:
\n\nIt may be possible to slightly improve the score by doing what you suggest and using the inputs to the final layer at the most malignant nodule as features. I tried a variant of this earlier in the competition without success but I never tried it again later.
\n\nAlso, does you final submission still have a big gap between local CV and stage 1 LB?
\n\nYes. For example, my 'second ensemble' had a local CV score of 0.390 and a stage 1 LB score of 0.451. It was very hard to ignore the stage 1 LB and trust our CV score! Ultimately I ended up basically ignoring the stage 1 LB altogether and just focusing on improving CV as CV had 1400 scans compared to the 200 in the public LB.
", "rawMarkdown": ">did you use convNet raw features (likely fully-connected layer input) from max malignancy locations? Or did you just use the scalar max malignancy probability?\n\nWe did not chop off the last layer of the network if that is what you are asking. We both used the actual malignancy predictions to generate some 'hand crafted' features like:\n\n* max (nodule malignancy predictions)\n* stdev( nodule malignancy predictions)\n* location of most malignant nodule, etc.\n\nIt may be possible to slightly improve the score by doing what you suggest and using the inputs to the final layer at the most malignant nodule as features. I tried a variant of this earlier in the competition without success but I never tried it again later.\n\nAlso, does you final submission still have a big gap between local CV and stage 1 LB?\n\nYes. For example, my 'second ensemble' had a local CV score of 0.390 and a stage 1 LB score of 0.451. It was very hard to ignore the stage 1 LB and trust our CV score! Ultimately I ended up basically ignoring the stage 1 LB altogether and just focusing on improving CV as CV had 1400 scans compared to the 200 in the public LB.\n\n", "votes": 2 }, { "id": 175320, "postDate": "2017-04-14T22:01:44.300Z", "content": "Thank you very much for the detailed sharing, Daniel!
\n\nI am deeply impressed by how well your results are without including the \"black art\" convNet features. Even though convNet is so popular on image tasks right now, it's good to know that we don't have to rely on such black features to achieve top results (or at least just keep the black features inside the black box itself).
\n\nFor the big gap between local CV and LB, it's really courageous to ignore LB altogether and just focus on local CV! I was too scared to do this, but guess I could be braver next time.
", "rawMarkdown": "Thank you very much for the detailed sharing, Daniel! \n\nI am deeply impressed by how well your results are without including the \"black art\" convNet features. Even though convNet is so popular on image tasks right now, it's good to know that we don't have to rely on such black features to achieve top results (or at least just keep the black features inside the black box itself). \n\nFor the big gap between local CV and LB, it's really courageous to ignore LB altogether and just focus on local CV! I was too scared to do this, but guess I could be braver next time. " } ] }, { "id": 175010, "postDate": "2017-04-13T13:28:03.967Z", "content": "Congrats! Thanks for sharing.\nWhat do you mean by \"3 zoom levels\"? 3 different 3d-filter sizes (receptive field) at the first level of your 3d-cnn?
", "rawMarkdown": "Congrats! Thanks for sharing.\nWhat do you mean by \"3 zoom levels\"? 3 different 3d-filter sizes (receptive field) at the first level of your 3d-cnn?", "votes": 1, "replies": [ { "id": 175020, "postDate": "2017-04-13T13:44:46.040Z", "content": "I was not clear about that.
\nI trained the net on 1x1x1 mm per pixel.
\nSince the receptive field size was 32x32x32 it sometimes missed larger nodules.
\nSo during prediction I predicted on 3 scales
\nI used the same network every time.
\nIt was a cheeky fix but somehow it worked.
I see. You resampled test scans three times, but using the same network instead. Thanks for sharing.
", "rawMarkdown": "I see. You resampled test scans three times, but using the same network instead. Thanks for sharing." } ] }, { "id": 174974, "postDate": "2017-04-13T10:39:01.687Z", "content": "Dear all, I was trying to find the malignancy labels for LIDC data, but didn't quite manage to find them. Could you give a hint where those labels are stored?
", "rawMarkdown": "Dear all, I was trying to find the malignancy labels for LIDC data, but didn't quite manage to find them. Could you give a hint where those labels are stored?", "votes": 1, "replies": [ { "id": 174976, "postDate": "2017-04-13T10:44:38.540Z", "content": "This link
\nThen detailed description.
\nThen \"XML File Documentation\"
thanks!
", "rawMarkdown": "thanks!" } ] }, { "id": 174962, "postDate": "2017-04-13T10:01:30.800Z", "content": "@sh1ng\nLoss for nodule yes/no: binary cross entropy\nLoss for malignancy: Squared Error\nLoss for tissue segmenter: Dice\nPredictions were not per 1mm but I used a stride of 12mm.\nSo a 240x240x240 patient scan gave 20x20x20 predictions.\nYes one for malignancy and one for nodule yes/no..
\n\nI will post some screenshots..
\n\n@FangzouLiao\nYes.. 32x32x32 was a bit small but combined with different zoom levels it was quite adequate.\nThe 0.429 LB was a result of blending our models. Best single model was 0.441 LB, 0.385 CV.\nSomehow blending Daniel's stuff with mine did good on LB.\nHowever.. I really struggled with LB. Better CV usually gave worse LB and vise versa.\nI looked for directional improvements for both at the same time. Daniel's mantra was \"In CV we trust\" :)
", "rawMarkdown": "@sh1ng\nLoss for nodule yes/no: binary cross entropy\nLoss for malignancy: Squared Error\nLoss for tissue segmenter: Dice\nPredictions were not per 1mm but I used a stride of 12mm.\nSo a 240x240x240 patient scan gave 20x20x20 predictions.\nYes one for malignancy and one for nodule yes/no..\n\nI will post some screenshots..\n\n@FangzouLiao\nYes.. 32x32x32 was a bit small but combined with different zoom levels it was quite adequate.\nThe 0.429 LB was a result of blending our models. Best single model was 0.441 LB, 0.385 CV.\nSomehow blending Daniel's stuff with mine did good on LB.\nHowever.. I really struggled with LB. Better CV usually gave worse LB and vise versa.\nI looked for directional improvements for both at the same time. Daniel's mantra was \"In CV we trust\" :)\n\n\n\n\n\n\n\n\n\n", "votes": 1, "replies": [ { "id": 174967, "postDate": "2017-04-13T10:21:07.360Z", "content": "Yes, I finally realized that the LB is full of outliers and is not trustworthy, so I asked you the question in your last post. Your confirmation gives me the confidence to my CV. Thank you for that
", "rawMarkdown": "Yes, I finally realized that the LB is full of outliers and is not trustworthy, so I asked you the question in your last post. Your confirmation gives me the confidence to my CV. Thank you for that", "votes": 1 } ] }, { "id": 174954, "postDate": "2017-04-13T09:31:38.700Z", "content": "Ok..I noticed the malignancy info but do not have much time to try that - -
", "rawMarkdown": "Ok..I noticed the malignancy info but do not have much time to try that - -", "votes": 1 }, { "id": 174935, "postDate": "2017-04-13T08:43:59.923Z", "content": "Cool! 32x32x32? This model architecture is much simpler than mine! Does it work well?
\n\nAnd how do you (and other guys) achieved 0.42 in stage 1? I can not achieve this score even after fine tuning the model on LB dataset.
", "rawMarkdown": "Cool! 32x32x32? This model architecture is much simpler than mine! Does it work well?\n\nAnd how do you (and other guys) achieved 0.42 in stage 1? I can not achieve this score even after fine tuning the model on LB dataset.\n\n", "votes": 1 }, { "id": 828728, "postDate": "2020-05-01T08:46:20.790Z", "content": "Hi,
\n\nAny links for the final code ?
\n\nThnx
", "rawMarkdown": "Hi,\n\nAny links for the final code ?\n\nThnx" }, { "id": 175065, "postDate": "2017-04-13T16:51:15.223Z", "content": "Thanks very much for the detail
", "rawMarkdown": "Thanks very much for the detail", "votes": 1 }, { "id": 175061, "postDate": "2017-04-13T16:39:51.430Z", "content": "Congrats! Thanks for sharing!
", "rawMarkdown": "Congrats! Thanks for sharing!", "votes": 1 }, { "id": 174928, "postDate": "2017-04-13T07:56:04.150Z", "content": "Thank you for the walk-through Julian!
", "rawMarkdown": "Thank you for the walk-through Julian! ", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-22T09:15:39Z", "competition_id": 6004, "competition_slug": "data-science-bowl-2017", "team_id": 517341, "solution_url": "/competitions/data-science-bowl-2017/writeups/julian-de-wit-daniel-hammack-very-quick-1st-summar" }