{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"> #  **Deep Dive into RSNA Breast Cancer Detection**","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\n\n# Load the image\nimage_path = \"/kaggle/input/breast-cancer-image/dataset-cover.png\"\nimage = mpimg.imread(image_path)\n\n# Display the image\nplt.imshow(image)\nplt.axis('off')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-15T11:22:34.070217Z","iopub.execute_input":"2023-07-15T11:22:34.070599Z","iopub.status.idle":"2023-07-15T11:22:34.296237Z","shell.execute_reply.started":"2023-07-15T11:22:34.070573Z","shell.execute_reply":"2023-07-15T11:22:34.295382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Introduction:\nThe field of medical imaging plays a crucial role in the early detection and diagnosis of breast cancer, a prevalent and potentially life-threatening disease affecting millions of women worldwide. In the pursuit of improving screening methods, the Kaggle community came together in the \"Screening Mammography Breast Cancer Detection\" (by Radiological Society of North America) competition. This competition was started on 28 November 2022 and its last date was 27 February 2023. There were a total of 1687 teams in it.\n\nThis essay aims to analyze the top-performing 3 notebooks from this competition, delving into the methodologies, techniques, and insights that contributed to their success.","metadata":{}},{"cell_type":"markdown","source":"# Evaluation Criteria:\nBefore diving into the analysis, it is essential to understand the evaluation criteria used in the competition. Participants were likely assessed based on their ability to predict breast cancer outcomes and submissions are evaluated using the probabilistic F1 score (pF1). This extension of the traditional F score accepts probabilities instead of binary classifications","metadata":{}},{"cell_type":"code","source":"# Load the image\nimage_path = \"/kaggle/input/optimization-image/model img.jpg\"\nimage = mpimg.imread(image_path)\n\n# Display the image\nplt.imshow(image)\nplt.axis('off')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-15T11:22:34.297817Z","iopub.execute_input":"2023-07-15T11:22:34.298522Z","iopub.status.idle":"2023-07-15T11:22:34.430380Z","shell.execute_reply.started":"2023-07-15T11:22:34.298489Z","shell.execute_reply":"2023-07-15T11:22:34.429076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model, Training and Optimization:\n\n## First Place notebook\nThe analysis used four different models: Eff-B2, Eff-B4, Effv2-s, and 4xConvnextV1-small. The Convnextv1-small model had a higher CV score. Five external datasets were used for training: VinDr-Mammo, MiniDDSM, CMMD, CDD-CESM, and BMCD. ROI crop was used, and the ROI detector used was YOLOX-nano at a resolution of 416x416. Min-Max normalization was applied. 521 images were used for training, and 50 images were used for validation, including all 47 images from the original external dataset. Hyperparameters were not used. In case of YOLOX's miss-detection, a simpler Otsu thresholding + findCountours() approach was used to find the breast bounding box. If both methods missed the breast box, the entire image was used without cropping. Multiple metrics were used, including PR_AUC, ROC_AUC, best_PF1 (binarized), and best_threshold. Data augmentation techniques such as HorizontalFlip + VerticalFlip, RandomBrightnessContrast, and upsampling positive samples were used for each epoch. The competition data was split into 4 folds, and one fold was used for validation while the rest of the folds and external data were used for training. Each split contained around 5560/75400 positive cases (~7.38%). The training started with the first two folds: fold 0 and fold 1, using the specified configuration. The model used was timm's convnext_small.fb_in22k_ft_in1k_384, with an input size of 2048x1024. The loss function used was vanilla BCE (without class weight). The sampler was set to upsample positive samples per epoch to pos/neg = 1/7, making sure each batch contained at least 1 positive sample. The batch size was 8. Other settings included Automatic Mixed Precision (AMP) and Model EMA. Global pooling was set to max, and the soft positive label was set to 0.8. The optimizer used was SGD with momentum=0.9, and the scheduler used was Cosine lr decay (epoch = 24, lr = 1e-3, min_lr = 1e-5) along with linear warmup (warmup_lr = 1e-5, warmup_epoch = 4). The drop rate was 0.5, and the drop path rate was 0.2. Once training on fold 0 finished, the results were not as expected. So fold 2 and fold 3 were trained with some changes: longer training with a larger learning rate and a reduced soft positive label. The scheduler for these folds was Cosine lr decay (epoch = 30, lr = 3e-3, min_lr = 5e-5) along with linear warmup (warmup_lr = 3e-5, warmup_epoch = 4). The soft positive label was set to 0.9. The results on fold 2 and fold 3 seemed to be better. Therefore, it was decided to fine-tune fold 0 and fold 1 with the same value of soft_positive_label = 0.9. For each fold, the 3-7 best checkpoints were manually selected based on multiple metrics. A grid search was then performed over all possible combinations of each fold's checkpoints (3x5x7x5 = 525 combinations in this case). The Out-Of-Fold (OOF) results were computed for each combination. The best combination was selected based on the highest binarized pF1. The best combination had a local OOF pF1 of 0.5187, while the worst combination had a pF1 of 0.4951.\n\n## Second place notebook\nThe analysis used two models, Convnextv1-small and R-CNN. They pretrained a single view model with a resolution of 1280x1280 using an external dataset called DDSM. Then, they finetuned this model without using the external dataset, with a resolution of 1536x1536. The finetuned single view model was further finetuned to create a dual view model and a four view model. They manually annotated the bounding box for the target breast in approximately 300 images. After that, they trained a basic Faster R-CNN model to crop all the breast regions for subsequent stages of training. Specifically, they minimized the bounding box to exclude the nipple and other unwanted body parts. They used the trained Faster R-CNN to crop an external dataset. They also used randomFlip, RandAugment, and RandomErasing from the mmcls library. The training optimizer used was AdamW, with a learning rate of 0.00015. The scheduler used was CosineAnnealingLR, and the training was done for 24 epochs. The batch size was set to 192 with gradient accumulation. They applied Test-Time Augmentation (TTA) using diagonal flip and used Exponential Moving Average (EMA). They only focused on the 0 fold. The single view model performed strongly with a Private Leaderboard.\n\n## Third place notebook\nThe third-place notebook used two types of classification models: CNN (EfficientNet or Convnext) and a multi-view model based on CNN+LSTM. They used ROI (Region of Interest) with a fixed aspect ratio of 1.6:1 using YOLOX. The ROI was fed into different classification models, and the results from these models were fused using average weighting. They used the CBIS-DDSM + CMMD dataset, and the other models used all four external datasets. They had 1,000 annotated images, with 500 labeled using open-source annotations. They trained YOLOX-nano and YOLOX-x using the BCD_all dataset and used YOLOX-nano for final online submissions and YOLOX-x for ROI cropping on external datasets. They cropped images based on the detection boxes and resized them to a 1.6:1 aspect ratio (1536960 or 1280800) with padding. They performed data augmentation using various techniques like HorizontalFlip, VerticalFlip, RandomBrightnessContrast, ShiftScaleRotate, MedianBlur, GaussianBlur, GaussNoise, ElasticTransform, GridDistortion, OpticalDistortion, CoarseDropout, and Mixup. They encoded site and view using One-Hot encoding. The image-level models were trained using a 4-fold split, while the LSTM models were trained using a 5-fold split. The final submission focused on LSTM and included 7 LSTM models and 4 image-level models. They optimized the threshold for combining these two types of models using the best cross-validation threshold obtained from the out-of-fold data.","metadata":{}},{"cell_type":"markdown","source":"# Comparing the three notebooks\nIt is evident that the first-place notebook performed the best with an accuracy score of 0.55. The Convnextv1-small model, along with appropriate data augmentation techniques, and 4 types external data proved to be a winning combination. The second-place notebook, using the Convnextv1-small and R-CNN models, performed well with an accuracy score of 0.53. Finally, the third-place notebook, despite its comprehensive approach and utilization of multiple models, achieved a comparatively lower accuracy score of 0.53.","metadata":{}},{"cell_type":"markdown","source":"# Conclusion\nThe choice of model architecture, data augmentation techniques, and external datasets play crucial roles in achieving high classification accuracy. The Convnextv1-small model, used in the first and second-place notebook, demonstrated its effectiveness in achieving accurate results. However, it is important to note that the success of notebooks depends on various factors which already described.","metadata":{}},{"cell_type":"markdown","source":"# limitations and areas for further improvement\nDuring the Breast Cancer Detection Competition on Kaggle, several limitations and areas for further improvement may have been identified. Some of these potential limitations and areas for improvement include:\n\n\n1. **Class imbalance:** Class imbalance occurs when the number of samples from each class (e.g., malignant or benign) is significantly different. In breast cancer detection, there is often an imbalance in the number of malignant and benign cases. Addressing class imbalance could help improve the performance of the models, ensuring they are not biased towards the majority class.\n\n2. **Noise and variability in data:** Medical imaging data can be noisy and subject to variability due to various factors such as image acquisition techniques, equipment variations, and inter-observer variability. Developing robust models that are less sensitive to noise and variability could enhance the accuracy and reliability of breast cancer detection.\n\n3. **Interpretability of models:** Deep learning models, in particular, can be highly complex and difficult to interpret. Although these models often achieve high accuracy, it is important to understand the reasoning behind their decisions. Improving the interpretability of models can help build trust and confidence in their predictions, especially in medical applications.\n\n4. **Integration with clinical workflows:** To make breast cancer detection models more useful in real-world clinical settings, seamless integration with existing clinical workflows and systems is crucial. This involves addressing technical challenges, ensuring interoperability, and considering the needs and preferences of healthcare professionals.\n\n5. **Generalizability to diverse populations:** It is essential to ensure that breast cancer detection models perform well across different demographics, including diverse populations. This requires collecting data from various ethnicities, age groups, and geographical locations to account for potential variations in breast cancer characteristics.\n\nBy addressing these limitations and areas for improvement, future breast cancer detection competitions can lead to advancements in the field, improving the accuracy and effectiveness of early diagnosis and treatment.","metadata":{}},{"cell_type":"markdown","source":"# References:\n1. Kaggle, “Meta Kaggle.” https://www.kaggle.com/datasets/kaggle/meta-kaggle\n2. First place notebook's writeup https://www.kaggle.com/c/39272/discussion/392449\n3. Second place notebook's writeup https://www.kaggle.com/c/39272/discussion/391676\n4. Third place notebook's writeup https://www.kaggle.com/c/39272/discussion/391725","metadata":{}},{"cell_type":"code","source":"import pandas as pd\n# Assign values to the variables\nessay_category = 'kaggle_writeups'\nessay_url = 'https://www.kaggle.com/umesalma/evolution-of-knowledge-insights-gained-from-kaggel/'\nfeedback1_url = 'https://www.kaggle.com/code/deepthiaj/meta-kaggle-analytics-exploring-ai-trends-21-23/comments'\nfeedback2_url = 'https://www.kaggle.com/code/soniadsilva/ai-at-the-cutting-edge-of-medical-imaging/comments'\nfeedback3_url = 'https://www.kaggle.com/code/lorresprz/applying-ai-ml-to-theoretical-physics/comments#2330973'\n\n# Create the submission.csv file\nsubmission_data = {\n    'type': ['essay_category', 'essay_url', 'feedback1_url', 'feedback2_url', 'feedback3_url'],\n    'value': [essay_category, essay_url, feedback1_url, feedback2_url, feedback3_url]\n}\nsubmission_df = pd.DataFrame(submission_data)\n\n# Save the submission dataframe as submission.csv\nsubmission_df.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-07-15T11:22:34.431992Z","iopub.execute_input":"2023-07-15T11:22:34.432502Z","iopub.status.idle":"2023-07-15T11:22:34.443473Z","shell.execute_reply.started":"2023-07-15T11:22:34.432463Z","shell.execute_reply":"2023-07-15T11:22:34.442368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}