{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction","metadata":{}},{"cell_type":"markdown","source":"## Reference\nhttps://www.kaggle.com/tarunpaparaju/plant-pathology-2020-eda-models/notebook\n\nhttps://www.kaggle.com/pestipeti/eda-plant-pathology-2020","metadata":{}},{"cell_type":"markdown","source":"# something might help\n\n0. Seed everything ---[reproducible](https://pytorch.org/docs/stable/notes/randomness.html)\n1. Get know your data, Do data laundry, Take care of the imbalance dataset ---[dealing with imbalanced Classification ](https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/)\n2. 合适的 augmentation method ---[Fast AutoAugment](https://github.com/kakaobrain/fast-autoaugment) [RandAugment](https://arxiv.org/abs/1909.13719)\n3. 合适的 lr schedule 方法 ---[LR_schedule](https://www.kaggle.com/isbhargav/guide-to-pytorch-learning-rate-scheduling)\n4. APEX ---[Automatic Mixed Precision](https://pytorch.org/blog/accelerating-training-on-nvidia-gpus-with-pytorch-automatic-mixed-precision/)\n4. Model Selection ---[TIMM](https://paperswithcode.com/lib/timm)\n5. Cross-validation, TTA ---[Stratified k-fold](https://scikit-learn.org/stable/modules/cross_validation.html#:~:text=3.1.2.2.1.-,Stratified%20k-fold,-%C2%B6) [When and Why TTA](https://arxiv.org/abs/2011.11156)\n5. 合适的 Metric method ---[quantifying the quality of predictions](https://scikit-learn.org/stable/modules/model_evaluation.html#classification-metrics)","metadata":{}},{"cell_type":"markdown","source":"## Content\n* [EDA](#section-ONE)\n    - [Import Dataset](#subsection-import)\n    - [Image Meta Data](#subsection-meta)\n    - [Visualize The Data](#subsection-visualize)\n    - [Noisy in Data](#subsection-noisy)\n    - [Canny edge detection](#subsection-canny)\n* [Image Augmentation](#section-TWO)\n    - [Albumentations](#subsection-albumentations)\n    - [AutoAlbument](#subsection-autoaug)\n    - [RangomAug](#subsection-randomaug)\n* [Model](#section-THREE)\n    - [Model Collection](#subsection-collection)\n* [Training Process](#section-FOUR)\n    - [Config](#subsection-Config)\n    - [Prepare Data](#section-PData)\n    - [Model](#section-Model)\n    - [Train Loop](#section-Loop)\n* [Inference Process](#section-FIVE)\n    \n GL&HF Kaggling :D\n    ","metadata":{}},{"cell_type":"markdown","source":"<a id=\"section-ONE\"></a>\n# EDA","metadata":{}},{"cell_type":"markdown","source":"<a id=\"subsection-import\"></a>\n## Import Dataset","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport cv2\nimport hashlib\nimport matplotlib.pyplot as plt\n\nimport PIL\nfrom PIL import Image\nfrom PIL import ImageStat\n\nfrom tqdm.notebook import tqdm\n\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport plotly.figure_factory as ff","metadata":{"execution":{"iopub.status.busy":"2021-06-23T05:58:36.154840Z","iopub.execute_input":"2021-06-23T05:58:36.155335Z","iopub.status.idle":"2021-06-23T05:58:39.775506Z","shell.execute_reply.started":"2021-06-23T05:58:36.155278Z","shell.execute_reply":"2021-06-23T05:58:39.774590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install -U albumentations\nimport albumentations\nalbumentations.__version__","metadata":{"execution":{"iopub.status.busy":"2021-06-23T05:58:39.776820Z","iopub.execute_input":"2021-06-23T05:58:39.777188Z","iopub.status.idle":"2021-06-23T05:58:49.022501Z","shell.execute_reply.started":"2021-06-23T05:58:39.777156Z","shell.execute_reply":"2021-06-23T05:58:49.021557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"CFG = {\n    'rootdir': '../input/classify-leaves/'\n}","metadata":{"execution":{"iopub.status.busy":"2021-06-23T05:58:49.024242Z","iopub.execute_input":"2021-06-23T05:58:49.024545Z","iopub.status.idle":"2021-06-23T05:58:49.028768Z","shell.execute_reply.started":"2021-06-23T05:58:49.024509Z","shell.execute_reply":"2021-06-23T05:58:49.027537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv(CFG['rootdir'] + 'train.csv')\ntest = pd.read_csv(CFG['rootdir'] + 'test.csv')\nsubmission_sample = pd.read_csv(CFG['rootdir'] + 'sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2021-06-23T05:58:49.030639Z","iopub.execute_input":"2021-06-23T05:58:49.031172Z","iopub.status.idle":"2021-06-23T05:58:49.105918Z","shell.execute_reply.started":"2021-06-23T05:58:49.031124Z","shell.execute_reply":"2021-06-23T05:58:49.104976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.sample(10)","metadata":{"execution":{"iopub.status.busy":"2021-06-23T05:58:49.107029Z","iopub.execute_input":"2021-06-23T05:58:49.107304Z","iopub.status.idle":"2021-06-23T05:58:49.129028Z","shell.execute_reply.started":"2021-06-23T05:58:49.107276Z","shell.execute_reply":"2021-06-23T05:58:49.128018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.sample(10)","metadata":{"execution":{"iopub.status.busy":"2021-06-23T05:58:49.130322Z","iopub.execute_input":"2021-06-23T05:58:49.130707Z","iopub.status.idle":"2021-06-23T05:58:49.139520Z","shell.execute_reply.started":"2021-06-23T05:58:49.130661Z","shell.execute_reply":"2021-06-23T05:58:49.138838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train.shape)\nprint(test.shape)","metadata":{"execution":{"iopub.status.busy":"2021-06-23T05:58:49.140536Z","iopub.execute_input":"2021-06-23T05:58:49.140819Z","iopub.status.idle":"2021-06-23T05:58:49.153269Z","shell.execute_reply.started":"2021-06-23T05:58:49.140793Z","shell.execute_reply":"2021-06-23T05:58:49.152176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.label.value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-06-23T05:58:49.155298Z","iopub.execute_input":"2021-06-23T05:58:49.155652Z","iopub.status.idle":"2021-06-23T05:58:49.169963Z","shell.execute_reply.started":"2021-06-23T05:58:49.155597Z","shell.execute_reply":"2021-06-23T05:58:49.169168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"176 labes in dataset","metadata":{}},{"cell_type":"code","source":"submission_sample.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-23T05:58:49.171118Z","iopub.execute_input":"2021-06-23T05:58:49.171574Z","iopub.status.idle":"2021-06-23T05:58:49.187402Z","shell.execute_reply.started":"2021-06-23T05:58:49.171544Z","shell.execute_reply":"2021-06-23T05:58:49.186578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Lets disply a single image","metadata":{}},{"cell_type":"code","source":"fig = px.imshow(cv2.imread(CFG['rootdir']+'{}'.format(train.sample(1)['image'].values[0])))\nfig.show()","metadata":{"tags":[],"execution":{"iopub.status.busy":"2021-06-23T05:58:55.527825Z","iopub.execute_input":"2021-06-23T05:58:55.528176Z","iopub.status.idle":"2021-06-23T05:58:56.606048Z","shell.execute_reply.started":"2021-06-23T05:58:55.528143Z","shell.execute_reply":"2021-06-23T05:58:56.603979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"subsection-meta\"></a>\n## Image Meta Data","metadata":{}},{"cell_type":"code","source":"def get_hash(image):\n    md5 = hashlib.md5()\n    md5.update(np.array(image).tobytes())\n    return md5.hexdigest()\n\ndef get_image_meta(image_id, image_src, dataset = 'train'):\n    img = Image.open(image_src)\n    extrema = img.getextrema()\n    stat = ImageStat.Stat(img)\n    \n    meta = {\n        'image': image_id,\n        'dataset': dataset,\n        'hash': get_hash(img),\n        'R_min': extrema[0][0],\n        'R_max': extrema[0][1],\n        'G_min': extrema[1][0],\n        'G_max': extrema[1][1],\n        'B_min': extrema[2][0],\n        'B_max': extrema[2][1],\n        'R_avg': stat.mean[0],\n        'G_avg': stat.mean[1],\n        'B_avg': stat.mean[2],\n        'height': img.height,\n        'width': img.width,\n        'format': img.format,\n        'mode': img.mode\n    }\n    return meta","metadata":{"execution":{"iopub.status.busy":"2021-06-23T05:59:00.778671Z","iopub.execute_input":"2021-06-23T05:59:00.779027Z","iopub.status.idle":"2021-06-23T05:59:00.786575Z","shell.execute_reply.started":"2021-06-23T05:59:00.778992Z","shell.execute_reply":"2021-06-23T05:59:00.785111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def image_metadata(df, rootdir, mark='train'):\n    img_data = []\n    for i, image_id in enumerate(tqdm(df['image'], total=df.shape[0])):\n        img_data.append(get_image_meta(image_id, (CFG['rootdir'] + '{}'.format(image_id)), mark))\n    \n    meta_pd = pd.DataFrame(img_data)\n    return meta_pd","metadata":{"execution":{"iopub.status.busy":"2021-06-23T05:59:13.474957Z","iopub.execute_input":"2021-06-23T05:59:13.475493Z","iopub.status.idle":"2021-06-23T05:59:13.480144Z","shell.execute_reply.started":"2021-06-23T05:59:13.475432Z","shell.execute_reply":"2021-06-23T05:59:13.479503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_train = image_metadata(df=train, rootdir = CFG['rootdir'], mark='train')\nmeta_train.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-23T05:59:22.139253Z","iopub.execute_input":"2021-06-23T05:59:22.139591Z","iopub.status.idle":"2021-06-23T06:01:04.538580Z","shell.execute_reply.started":"2021-06-23T05:59:22.139563Z","shell.execute_reply":"2021-06-23T06:01:04.537604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_test = image_metadata(test, rootdir = CFG['rootdir'], mark='test')\nmeta_test.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:04.539989Z","iopub.execute_input":"2021-06-23T06:01:04.540298Z","iopub.status.idle":"2021-06-23T06:01:53.009654Z","shell.execute_reply.started":"2021-06-23T06:01:04.540267Z","shell.execute_reply":"2021-06-23T06:01:53.008725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"subsection-visualize\"></a>\n## Visualize The Data","metadata":{}},{"cell_type":"code","source":"fig = px.histogram(train, 'label', marginal='violin', hover_data=train.columns)\nfig.update_layout(title_text=\"Distribution of Classes\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:53.011648Z","iopub.execute_input":"2021-06-23T06:01:53.011921Z","iopub.status.idle":"2021-06-23T06:01:53.391856Z","shell.execute_reply.started":"2021-06-23T06:01:53.011894Z","shell.execute_reply":"2021-06-23T06:01:53.390788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we can see that the distribution seems like a long-tail dataset, and our testing data set might based on 176 x 50 = 88800 (with an even distribution). \n\n这里可以看到，我们的训练集看起来有那么一点点像long-tail的数据集, 并且因为目标测试集应该是按照 176 X 50 = 8800(按照均匀分布).\n所以我们取sampler的时候需要注意，还有cross validation, 也要注意每个fold的分法.","metadata":{}},{"cell_type":"code","source":"fig = ff.create_distplot([meta_train['R_avg']], group_labels=[\"ChannelsR\"], colors=[\"RED\"])\nfig.update_layout(showlegend=False, template=\"simple_white\")\nfig.update_layout(title_text=\"Distribution of Channel Values\")\nfig.data[0].marker.line.color = 'rgb(0, 0, 0)'\nfig.data[0].marker.line.width = 0.5\nfig","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:53.393547Z","iopub.execute_input":"2021-06-23T06:01:53.393842Z","iopub.status.idle":"2021-06-23T06:01:53.944888Z","shell.execute_reply.started":"2021-06-23T06:01:53.393811Z","shell.execute_reply":"2021-06-23T06:01:53.943856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = ff.create_distplot([meta_train['R_avg'], meta_train['G_avg'], meta_train['B_avg']], group_labels=['R','G','B'], colors=[\"RED\",'GREEN','BLUE'])\nfig.update_layout(showlegend=False, template=\"simple_white\")\nfig.update_layout(title_text=\"Distribution of Channel Values\")\nfig.data[0].marker.line.color = 'rgb(0, 0, 0)'\nfig.data[0].marker.line.width = 0.5\nfig.data[1].marker.line.color = 'rgb(0, 0, 0)'\nfig.data[1].marker.line.width = 0.5\nfig.data[2].marker.line.color = 'rgb(0, 0, 0)'\nfig.data[2].marker.line.width = 0.5\nfig","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:53.946172Z","iopub.execute_input":"2021-06-23T06:01:53.946495Z","iopub.status.idle":"2021-06-23T06:01:55.182923Z","shell.execute_reply.started":"2021-06-23T06:01:53.946462Z","shell.execute_reply":"2021-06-23T06:01:55.178664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = go.Figure()\n\nfor idx, values in enumerate([meta_train['R_avg'], meta_train['G_avg'], meta_train['B_avg']]):\n    if idx == 0:\n        color = \"RED\"\n    if idx == 1:\n        color = \"GREEN\"\n    if idx == 2:\n        color = \"BLUE\"\n    fig.add_trace(go.Box(x=[color]*len(values), y=values, name=color, marker=dict(color=color.lower())))\n    \nfig.update_layout(yaxis_title=\"Mean Value\", xaxis_title=\"Color Channel\",\n                  title=\"Mean Value vs. Color Channel\", template=\"plotly_white\")","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:55.183970Z","iopub.execute_input":"2021-06-23T06:01:55.184350Z","iopub.status.idle":"2021-06-23T06:01:55.663332Z","shell.execute_reply.started":"2021-06-23T06:01:55.184320Z","shell.execute_reply":"2021-06-23T06:01:55.662388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that RGB channel plot are mostly over lapped, since there are a lot of white space in out image data. The white channel [255,255,255] raised a lot of our average level. A boundbox might can help us reduce those useless pixels.(TBD)\n\n通过查看RGB的channel图, 可以看到到RGB通道大致上都非常接近. 主要原因是图片集中有太多的空白, 所以[255,255,255]的数值过多,导致平均数值也很高.\n也许可以考虑通过加一个boundbox来缩小搜索范围.","metadata":{}},{"cell_type":"markdown","source":"<a id=\"subsection-noisy\"></a>\n## Noisy in Data","metadata":{}},{"cell_type":"code","source":"def get_duplicate(dt, mark = 'train'):\n    dup = dt.groupby(by = 'hash')[['dataset']].count().reset_index()\n    dup = dup[dup['dataset'] > 1]\n    dup.reset_index(drop = True, inplace = True)\n    dup = dup.merge(dt[['image', 'hash']], on = 'hash')\n    dup['dup_number'] = dup['dataset']\n    dup.loc[:,'dataset'] = mark\n    \n    return dup","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:55.664414Z","iopub.execute_input":"2021-06-23T06:01:55.664735Z","iopub.status.idle":"2021-06-23T06:01:55.670397Z","shell.execute_reply.started":"2021-06-23T06:01:55.664706Z","shell.execute_reply":"2021-06-23T06:01:55.669432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dup_train = get_duplicate(meta_train, 'train')\ndup_train","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:55.672388Z","iopub.execute_input":"2021-06-23T06:01:55.672702Z","iopub.status.idle":"2021-06-23T06:01:55.752519Z","shell.execute_reply.started":"2021-06-23T06:01:55.672673Z","shell.execute_reply":"2021-06-23T06:01:55.751594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"duptest = get_duplicate(meta_test, 'test')\nduptest","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:55.754117Z","iopub.execute_input":"2021-06-23T06:01:55.754378Z","iopub.status.idle":"2021-06-23T06:01:55.796005Z","shell.execute_reply.started":"2021-06-23T06:01:55.754352Z","shell.execute_reply":"2021-06-23T06:01:55.792498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"duptest['dup_number'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:55.797732Z","iopub.execute_input":"2021-06-23T06:01:55.798101Z","iopub.status.idle":"2021-06-23T06:01:55.808484Z","shell.execute_reply.started":"2021-06-23T06:01:55.798061Z","shell.execute_reply":"2021-06-23T06:01:55.806986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Add label to our dup dataset","metadata":{}},{"cell_type":"code","source":"dup_train = dup_train.merge(train[['image','label']], on = 'image')\ndup_train","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:55.810604Z","iopub.execute_input":"2021-06-23T06:01:55.810986Z","iopub.status.idle":"2021-06-23T06:01:55.844951Z","shell.execute_reply.started":"2021-06-23T06:01:55.810946Z","shell.execute_reply":"2021-06-23T06:01:55.843349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tester = dup_train.copy()\n# we will drop those rows both have the same hash and label which just duplicated data, not noisy confusion data.\n#把 hash和 label都相同的 drop掉\ntester = tester.drop_duplicates(subset=['hash','label'], keep=False)","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:55.846372Z","iopub.execute_input":"2021-06-23T06:01:55.846751Z","iopub.status.idle":"2021-06-23T06:01:55.855888Z","shell.execute_reply.started":"2021-06-23T06:01:55.846711Z","shell.execute_reply":"2021-06-23T06:01:55.854775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tester['dup_number'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:55.857319Z","iopub.execute_input":"2021-06-23T06:01:55.857713Z","iopub.status.idle":"2021-06-23T06:01:55.873081Z","shell.execute_reply.started":"2021-06-23T06:01:55.857674Z","shell.execute_reply":"2021-06-23T06:01:55.872202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tester.head(10)","metadata":{"tags":[],"execution":{"iopub.status.busy":"2021-06-23T06:01:55.874229Z","iopub.execute_input":"2021-06-23T06:01:55.874565Z","iopub.status.idle":"2021-06-23T06:01:55.895125Z","shell.execute_reply.started":"2021-06-23T06:01:55.874531Z","shell.execute_reply":"2021-06-23T06:01:55.893977Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_dup(df, number):\n    fig, ax = plt.subplots(int(number/2), 2, figsize=(10, 30))\n    ax = ax.flatten()\n\n    for i in range(0, min(tester.shape[0], number), 2):\n        image_i = cv2.imread(CFG['rootdir'] + '{}'.format(df.iloc[i, 2]), cv2.IMREAD_COLOR)\n        image_i = cv2.cvtColor(image_i, cv2.COLOR_BGR2RGB)\n        ax[i].set_axis_off()\n        ax[i].imshow(image_i)\n        ax[i].set_title(df.iloc[i, 2] + '\\n' + df.iloc[i, 4])\n\n        image_i_1 = cv2.imread(CFG['rootdir'] + '{}'.format(df.iloc[i + 1, 2]), cv2.IMREAD_COLOR)\n        image_i_1 = cv2.cvtColor(image_i_1, cv2.COLOR_BGR2RGB)\n        ax[i + 1].set_axis_off()\n        ax[i + 1].imshow(image_i_1)\n        ax[i + 1].set_title(df.iloc[i + 1, 2] + '\\n' + df.iloc[i + 1,4])","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:55.896209Z","iopub.execute_input":"2021-06-23T06:01:55.896633Z","iopub.status.idle":"2021-06-23T06:01:55.905644Z","shell.execute_reply.started":"2021-06-23T06:01:55.896588Z","shell.execute_reply":"2021-06-23T06:01:55.904550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#plot_dup(dup_train, 12)","metadata":{"execution":{"iopub.execute_input":"2021-06-21T13:19:22.694642Z","iopub.status.busy":"2021-06-21T13:19:22.694183Z","iopub.status.idle":"2021-06-21T13:19:23.562793Z","shell.execute_reply":"2021-06-21T13:19:23.561864Z","shell.execute_reply.started":"2021-06-21T13:19:22.694597Z"},"tags":[]},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_dup(tester, 12)","metadata":{"tags":[],"execution":{"iopub.status.busy":"2021-06-23T06:01:55.907071Z","iopub.execute_input":"2021-06-23T06:01:55.907479Z","iopub.status.idle":"2021-06-23T06:01:56.952628Z","shell.execute_reply.started":"2021-06-23T06:01:55.907420Z","shell.execute_reply":"2021-06-23T06:01:56.951780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, we will drop the duplicated row. \n\n这里可以选择考虑把dup的数据全都drop掉, 或者也可以加上soft label.","metadata":{}},{"cell_type":"code","source":"train_clean = train[~train['image'].isin(tester['image'])].reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:56.953786Z","iopub.execute_input":"2021-06-23T06:01:56.954068Z","iopub.status.idle":"2021-06-23T06:01:56.961422Z","shell.execute_reply.started":"2021-06-23T06:01:56.954041Z","shell.execute_reply":"2021-06-23T06:01:56.960715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_clean.nunique()","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:56.962445Z","iopub.execute_input":"2021-06-23T06:01:56.962868Z","iopub.status.idle":"2021-06-23T06:01:56.990351Z","shell.execute_reply.started":"2021-06-23T06:01:56.962826Z","shell.execute_reply":"2021-06-23T06:01:56.989412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#save cleaned data\ntrain_clean.to_csv('train_clean.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:05:33.165646Z","iopub.execute_input":"2021-06-23T06:05:33.166016Z","iopub.status.idle":"2021-06-23T06:05:33.202215Z","shell.execute_reply.started":"2021-06-23T06:05:33.165982Z","shell.execute_reply":"2021-06-23T06:05:33.201502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"subsection-canny\"></a>\n## Canny edge detection","metadata":{}},{"cell_type":"markdown","source":"https://docs.opencv.org/3.4/da/d22/tutorial_py_canny.html\n\nHere we use canny algorithm to search the edge, then we use the bound of edge to estimate our bounding.\n\n这里使用的方法是用canny找出轮廓后，然后根据轮廓边界画出bounding box","metadata":{}},{"cell_type":"code","source":"def edge_and_cut(img_src):\n    img = cv2.imread(img_src)\n    emb_img = img.copy()\n    edges = cv2.Canny(img, 100, 200)\n    edge_coors = []\n    for i in range(edges.shape[0]):\n        for j in range(edges.shape[1]):\n            if edges[i][j] != 0:\n                edge_coors.append((i, j))\n    \n    row_min = edge_coors[np.argsort([coor[0] for coor in edge_coors])[0]][0]\n    row_max = edge_coors[np.argsort([coor[0] for coor in edge_coors])[-1]][0]\n    col_min = edge_coors[np.argsort([coor[1] for coor in edge_coors])[0]][1]\n    col_max = edge_coors[np.argsort([coor[1] for coor in edge_coors])[-1]][1]\n    new_img = img[row_min:row_max, col_min:col_max]\n    \n    emb_img[row_min-5:row_min+5, col_min:col_max] = [255, 0, 0]\n    emb_img[row_max-5:row_max+5, col_min:col_max] = [255, 0, 0]\n    emb_img[row_min:row_max, col_min-5:col_min+5] = [255, 0, 0]\n    emb_img[row_min:row_max, col_max-5:col_max+5] = [255, 0, 0]\n    \n    fig, ax = plt.subplots(nrows=1, ncols=3, figsize=(30, 20))\n    ax[0].imshow(img)\n    ax[0].set_title('Original Image', fontsize=24)\n    ax[1].imshow(edges, cmap='gray')\n    ax[1].set_title('Canny Edges', fontsize=24)\n    ax[2].imshow(emb_img, cmap='gray')\n    ax[2].set_title('Bounding Box', fontsize=24)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:56.991750Z","iopub.execute_input":"2021-06-23T06:01:56.992061Z","iopub.status.idle":"2021-06-23T06:01:57.003377Z","shell.execute_reply.started":"2021-06-23T06:01:56.992033Z","shell.execute_reply":"2021-06-23T06:01:57.002224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"edge_and_cut('../input/classify-leaves/images/233.jpg')\nedge_and_cut('../input/classify-leaves/images/2333.jpg')\nedge_and_cut('../input/classify-leaves/images/25261.jpg')\n# edge_and_cut('../input/classify-leaves/images/25909.jpg')\n# edge_and_cut('../input/classify-leaves/images/26634.jpg')\n# edge_and_cut('../input/classify-leaves/images/26723.jpg')","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:57.004674Z","iopub.execute_input":"2021-06-23T06:01:57.005023Z","iopub.status.idle":"2021-06-23T06:01:59.363917Z","shell.execute_reply.started":"2021-06-23T06:01:57.004990Z","shell.execute_reply":"2021-06-23T06:01:59.362911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-TWO\"></a>\n# Image Augmentation","metadata":{}},{"cell_type":"markdown","source":"有关Augmentation, 我们可以用pytorch自带的，也可以考虑使用albumentations lib.","metadata":{}},{"cell_type":"markdown","source":"The augmentation method can be divided as follows:\n1. geometric transformations,  \n2. color space transformations, \n3. kernel filters, \n4. mixing images,\n5. random erasing\n\nWe should wisely choose and combine our augmentation method. Since some methods will increase your search space, \nand even dmage your performance.\n\n\n\n通常图片的数据增强主要分为\n1. 几何变换 2.颜色变换 3.滤镜化 4. 图片组合 5. 随机擦除\n\n我们需要根据自己的目标数据集，合理的选择适合我们的数据增强方法，有些时候一些数据增强方法甚至会损害结果.\n\n推荐大家看看这一篇 [A survey on Image Data Augmentation for Deep Learning](https://journalofbigdata.springeropen.com/articles/10.1186/s40537-019-0197-0#Fun)\n\nAbout mixup and cutmix https://www.kaggle.com/cdeotte/cutmix-and-mixup-on-gpu-tpu","metadata":{}},{"cell_type":"markdown","source":"<a id=\"subsection-albumentations\"></a>\n## Albumentations","metadata":{}},{"cell_type":"code","source":"from albumentations import (\n    HorizontalFlip, VerticalFlip, Rotate, ShiftScaleRotate, RandomBrightnessContrast, Perspective, CLAHE, \n    Transpose, Blur, OpticalDistortion, GridDistortion, HueSaturationValue, ColorJitter, GaussNoise, MotionBlur, MedianBlur,\n    Emboss, Sharpen, Flip, OneOf, SomeOf, Compose, Normalize, CoarseDropout, CenterCrop, GridDropout, Resize\n)","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:59.365021Z","iopub.execute_input":"2021-06-23T06:01:59.365277Z","iopub.status.idle":"2021-06-23T06:01:59.370071Z","shell.execute_reply.started":"2021-06-23T06:01:59.365251Z","shell.execute_reply":"2021-06-23T06:01:59.369076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# helper function\ndef get_img(imgsrc):\n    im_bgr = cv2.imread(imgsrc)\n    im_rgb = cv2.cvtColor(im_bgr, cv2.COLOR_BGR2RGB)\n    return im_rgb\n\ndef visualize(image):\n    plt.figure(figsize=(5, 5))\n    plt.axis('off')\n    plt.imshow(image)","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:59.371253Z","iopub.execute_input":"2021-06-23T06:01:59.371552Z","iopub.status.idle":"2021-06-23T06:01:59.385098Z","shell.execute_reply.started":"2021-06-23T06:01:59.371520Z","shell.execute_reply":"2021-06-23T06:01:59.384141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":" def plot_aug(img_src, method):\n    img = get_img(img_src)\n    augmented_img = method(p=1)(image=img)['image']\n    fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(10, 10))\n    ax[0].imshow(img)\n    ax[0].set_title('Original Image', fontsize=24)\n    ax[1].imshow(augmented_img)\n    ax[1].set_title(method.__name__, fontsize=24)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:59.388258Z","iopub.execute_input":"2021-06-23T06:01:59.388646Z","iopub.status.idle":"2021-06-23T06:01:59.401580Z","shell.execute_reply.started":"2021-06-23T06:01:59.388599Z","shell.execute_reply":"2021-06-23T06:01:59.400823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_aug('../input/classify-leaves/images/233.jpg', HorizontalFlip)\nplot_aug('../input/classify-leaves/images/233.jpg', VerticalFlip)\nplot_aug('../input/classify-leaves/images/233.jpg', Transpose)\nplot_aug('../input/classify-leaves/images/233.jpg', ShiftScaleRotate)\nplot_aug('../input/classify-leaves/images/233.jpg', Blur)\nplot_aug('../input/classify-leaves/images/233.jpg', GaussNoise)\nplot_aug('../input/classify-leaves/images/233.jpg', HueSaturationValue)\nplot_aug('../input/classify-leaves/images/233.jpg', CoarseDropout)\nplot_aug('../input/classify-leaves/images/233.jpg', GridDistortion)\nplot_aug('../input/classify-leaves/images/233.jpg', Sharpen)\nplot_aug('../input/classify-leaves/images/233.jpg', CLAHE)","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:01:59.402715Z","iopub.execute_input":"2021-06-23T06:01:59.403071Z","iopub.status.idle":"2021-06-23T06:02:03.060340Z","shell.execute_reply.started":"2021-06-23T06:01:59.403044Z","shell.execute_reply":"2021-06-23T06:02:03.059261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Common pipeline for augmentation\n\nhttps://albumentations.readthedocs.io/en/latest/probabilities.html\n\nhttps://albumentations.ai/docs/getting_started/setting_probabilities/\n\n如果是使用 albumentations的话, 可以考虑合理使用 someof 和 oneof 和不同的 probability rate 来组合数据增强方法, 可以适当的减少搜索空间.","metadata":{}},{"cell_type":"code","source":"transform = Compose([\n        SomeOf([\n            Flip(),\n            GridDistortion(),\n            Transpose(),\n            ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=45, p=0.2),\n        ], n=2, p=1),\n        OneOf([\n            MotionBlur(p=.2),\n            MedianBlur(blur_limit=3, p=0.1),\n            Blur(blur_limit=3, p=0.1),\n            GaussNoise(p=0.2),\n        ], p=0.2),\n        OneOf([\n            CLAHE(clip_limit=2),\n            Sharpen(),\n            Emboss(),\n            RandomBrightnessContrast(),            \n        ], p=0.3),\n        OneOf([\n            CoarseDropout(),\n            GridDropout(),\n        ], p=0.2),\n        HueSaturationValue(p=0.3),\n    ])\n\nimg = get_img('../input/classify-leaves/images/233.jpg')\n\nvisualize(transform(image=img)['image'])\nvisualize(transform(image=img)['image'])\nvisualize(transform(image=img)['image'])","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:02:03.061932Z","iopub.execute_input":"2021-06-23T06:02:03.062311Z","iopub.status.idle":"2021-06-23T06:02:03.412788Z","shell.execute_reply.started":"2021-06-23T06:02:03.062272Z","shell.execute_reply":"2021-06-23T06:02:03.411800Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most of the time, we use the normalization from ImageNet's [mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225), max_pixel_value=255.0]\nbut we can also make our own normalization data from our dataset.\n\n\n大多数时候的normalization都使用Imagenet的就行了，但是我们也可以考虑自己计算 mean和 std. 或者使用其他的方法来做normalization \n\n比如：albumentations.augmentations.domain_adaptation.HistogramMatching","metadata":{}},{"cell_type":"markdown","source":"这里简单提一下两种自动选择augmentation方法.","metadata":{}},{"cell_type":"markdown","source":"<a id=\"subsection-autoaug\"></a>\n## AutoAlbument\n","metadata":{}},{"cell_type":"markdown","source":"Fast AutoAugment \nhttps://github.com/kakaobrain/fast-autoaugment\n\npip install -U autoalbument\n\nhttps://github.com/albumentations-team/autoalbument","metadata":{}},{"cell_type":"markdown","source":"<a id=\"subsection-randomaug\"></a>\n## RandomAug","metadata":{}},{"cell_type":"markdown","source":"[RandAugment](https://arxiv.org/abs/1909.13719)","metadata":{}},{"cell_type":"code","source":"#!pip install git+https://github.com/ildoonet/pytorch-randaugment","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#from RandAugment import RandAugment","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from RandAugment import RandAugment\n\n# N = 5\n# M = 5\n\n# train_transforms = Compose([\n#             RandAugment(N, M),\n#             Transpose(p=0.5),\n#             VerticalFlip(p=0.5),\n#             HorizontalFlip(p=0.5),\n#             ShiftScaleRotate(shift_limit=0.1, scale_limit=0.1, rotate_limit=15, border_mode=0, p=0.85),\n#             Resize(img_size[0], img_size[1], p=1.0),\n#             #A.Normalize(),\n#             #ToTensorV2(p=1.0),\n#         ], p=1.0)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-THREE\"></a>\n# Model","metadata":{}},{"cell_type":"markdown","source":"<a id=\"subsection-collection\"></a>\n## Model Collection","metadata":{}},{"cell_type":"code","source":"import os\nimport time\nfrom glob import glob\nimport random\n\nimport pandas as pd\nimport numpy as np\n\nimport torch\nfrom torch import nn\n\nfrom torch.utils.data import Dataset,DataLoader\nfrom torch.cuda.amp import autocast, GradScaler\n\nimport sklearn\nfrom sklearn import preprocessing\nfrom sklearn.metrics import roc_auc_score, log_loss\nfrom sklearn.model_selection import StratifiedKFold\n\nimport cv2\n\nfrom tqdm import tqdm\n\n#import timm","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:02:03.413965Z","iopub.execute_input":"2021-06-23T06:02:03.414246Z","iopub.status.idle":"2021-06-23T06:02:04.552850Z","shell.execute_reply.started":"2021-06-23T06:02:03.414217Z","shell.execute_reply":"2021-06-23T06:02:04.551693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install timm\n!pip install torchinfo","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:02:04.554320Z","iopub.execute_input":"2021-06-23T06:02:04.554766Z","iopub.status.idle":"2021-06-23T06:02:19.195750Z","shell.execute_reply.started":"2021-06-23T06:02:04.554719Z","shell.execute_reply":"2021-06-23T06:02:19.194632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import timm\nfrom torchinfo import summary","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:02:19.197324Z","iopub.execute_input":"2021-06-23T06:02:19.197632Z","iopub.status.idle":"2021-06-23T06:02:20.036946Z","shell.execute_reply.started":"2021-06-23T06:02:19.197599Z","shell.execute_reply":"2021-06-23T06:02:20.036094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"通常我们可以选择使用[torchvision.models](https://pytorch.org/vision/stable/models.html)来选择主流的模型. \n\n这里推荐使用timm库, 相比pytorch的官方库, 模型选择更多,也更新，可以查看很多模型参数细节. \n\n具体可以参考下面的链接. 可以通过torchinfo，辅助了解选择的模型.\n\nhttps://paperswithcode.com/lib/timm\n","metadata":{}},{"cell_type":"code","source":"model_batch_size = 32","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:31:01.227699Z","iopub.execute_input":"2021-06-23T06:31:01.228076Z","iopub.status.idle":"2021-06-23T06:31:01.232748Z","shell.execute_reply.started":"2021-06-23T06:31:01.228037Z","shell.execute_reply":"2021-06-23T06:31:01.231439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eff_model = timm.create_model('tf_efficientnet_b4', pretrained=True)\nsummary(eff_model, input_size = (model_batch_size, 3, 224, 224))","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:31:04.705726Z","iopub.execute_input":"2021-06-23T06:31:04.706070Z","iopub.status.idle":"2021-06-23T06:31:10.344541Z","shell.execute_reply.started":"2021-06-23T06:31:04.706038Z","shell.execute_reply":"2021-06-23T06:31:10.343513Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eff_model_2 = timm.create_model('tf_efficientnet_b5', pretrained=True)\nsummary(eff_model, input_size = (model_batch_size, 3, 224, 224))","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:31:10.346241Z","iopub.execute_input":"2021-06-23T06:31:10.346643Z","iopub.status.idle":"2021-06-23T06:31:15.527284Z","shell.execute_reply.started":"2021-06-23T06:31:10.346607Z","shell.execute_reply":"2021-06-23T06:31:15.526221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"res_model = timm.create_model('resnest50d', pretrained=True)\nres_model.fc = nn.Linear(2048, 176)\nsummary(res_model, input_size = (model_batch_size, 3, 224, 224))","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:31:15.529270Z","iopub.execute_input":"2021-06-23T06:31:15.529704Z","iopub.status.idle":"2021-06-23T06:31:38.859419Z","shell.execute_reply.started":"2021-06-23T06:31:15.529658Z","shell.execute_reply":"2021-06-23T06:31:38.858531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del eff_model\ndel eff_model_2\ndel res_model","metadata":{"execution":{"iopub.status.busy":"2021-06-23T06:03:12.122585Z","iopub.execute_input":"2021-06-23T06:03:12.122834Z","iopub.status.idle":"2021-06-23T06:03:12.127860Z","shell.execute_reply.started":"2021-06-23T06:03:12.122809Z","shell.execute_reply":"2021-06-23T06:03:12.127134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can use summary to estimate our GPU RAM consumption with different settings\n","metadata":{}},{"cell_type":"markdown","source":"<a id=\"section-FOUR\"></a>\n# Train Loop","metadata":{}},{"cell_type":"markdown","source":"## Config","metadata":{}},{"cell_type":"code","source":"CFG = {\n    'rootdir': '../input/classify-leaves/',\n    'fold_num': 5,\n    'seed': 123,\n    'model_arch': 'tf_efficientnet_b5',\n    'img_size': 224,\n    'epochs': 10,\n    'train_bs': 32,\n    'valid_bs': 32,\n    'T_0': 10, # Number of iterations for the first restart\n    'lr': 3e-4,\n    'min_lr': 1e-6,\n    'weight_decay':1e-6,\n    'num_workers': 8,\n    'accum_iter': 2, # suppoprt to do batch accumulation for backprop with effectively larger batch size\n    'verbose_step': 1, # 0 (silent),1 (progress bar), 2 (one line per epoch)\n    'device': 'cuda:0'\n}","metadata":{"execution":{"iopub.status.busy":"2021-06-22T07:39:23.289524Z","iopub.execute_input":"2021-06-22T07:39:23.289953Z","iopub.status.idle":"2021-06-22T07:39:23.30014Z","shell.execute_reply.started":"2021-06-22T07:39:23.289914Z","shell.execute_reply":"2021-06-22T07:39:23.299243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helper Function","metadata":{}},{"cell_type":"code","source":"# labelencoder for dataset\nt_labelencoder = preprocessing.LabelEncoder()\nt_labelencoder.fit(train['label'])","metadata":{"execution":{"iopub.status.busy":"2021-06-22T07:33:57.763342Z","iopub.execute_input":"2021-06-22T07:33:57.763662Z","iopub.status.idle":"2021-06-22T07:33:57.774039Z","shell.execute_reply.started":"2021-06-22T07:33:57.763634Z","shell.execute_reply":"2021-06-22T07:33:57.77301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_clean['label'] = t_labelencoder.transform(train_clean['label'])","metadata":{"execution":{"iopub.status.busy":"2021-06-22T07:34:02.727957Z","iopub.execute_input":"2021-06-22T07:34:02.728319Z","iopub.status.idle":"2021-06-22T07:34:02.739276Z","shell.execute_reply.started":"2021-06-22T07:34:02.728289Z","shell.execute_reply":"2021-06-22T07:34:02.73825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_clean","metadata":{"tags":[],"execution":{"iopub.status.busy":"2021-06-22T07:34:05.026775Z","iopub.execute_input":"2021-06-22T07:34:05.027127Z","iopub.status.idle":"2021-06-22T07:34:05.037975Z","shell.execute_reply.started":"2021-06-22T07:34:05.027097Z","shell.execute_reply":"2021-06-22T07:34:05.037175Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    torch.backends.cudnn.benchmark = True\n    \ndef get_img(path):\n    im_bgr = cv2.imread(path)\n    im_rgb = im_bgr[:, :, ::-1]\n    #print(im_rgb)\n    return im_rgb","metadata":{"execution":{"iopub.status.busy":"2021-06-22T07:34:09.890901Z","iopub.execute_input":"2021-06-22T07:34:09.891258Z","iopub.status.idle":"2021-06-22T07:34:09.897528Z","shell.execute_reply.started":"2021-06-22T07:34:09.891228Z","shell.execute_reply":"2021-06-22T07:34:09.896348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prepare Data","metadata":{}},{"cell_type":"code","source":"class MyDataset(Dataset):\n    def __init__(self, df, data_root, transforms=None, output_label=True):\n        \n        super().__init__()\n        self.df = df.reset_index(drop=True).copy()\n        self.transforms = transforms\n        self.data_root = data_root\n        \n        self.output_label = output_label\n        \n        if output_label == True:\n            self.labels = self.df['label'].values\n            \n    def __len__(self):\n        return self.df.shape[0]\n    \n    def __getitem__(self, index: int):\n        # get labels\n        if self.output_label:\n            target = self.labels[index]\n          \n        img  = get_img(\"{}/{}\".format(self.data_root, self.df.loc[index]['image']))\n\n        if self.transforms:\n            img = self.transforms(image=img)['image']\n        \n        if self.output_label == True:\n            return img, target\n        else:\n            return img","metadata":{"execution":{"iopub.status.busy":"2021-06-22T07:34:18.008741Z","iopub.execute_input":"2021-06-22T07:34:18.009166Z","iopub.status.idle":"2021-06-22T07:34:18.016308Z","shell.execute_reply.started":"2021-06-22T07:34:18.009135Z","shell.execute_reply":"2021-06-22T07:34:18.01548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Augmentation","metadata":{}},{"cell_type":"code","source":"from albumentations import (\n    HorizontalFlip, VerticalFlip, Rotate, ShiftScaleRotate, RandomBrightnessContrast, Perspective, CLAHE, \n    Transpose, Blur, OpticalDistortion, GridDistortion, HueSaturationValue, ColorJitter, GaussNoise, MotionBlur, MedianBlur,\n    Emboss, Sharpen, Flip, OneOf, SomeOf, Compose, Normalize, CoarseDropout, CenterCrop, GridDropout, Resize\n)\nfrom albumentations.pytorch import ToTensorV2\n\ndef get_train_transforms():\n    return Compose([\n            OneOf([\n            CoarseDropout(p=0.5),\n            GaussNoise(),\n            ], p=0.5),\n            SomeOf([\n            Transpose(p=0.5),\n            HorizontalFlip(p=0.5),\n            VerticalFlip(p=0.5),\n            ShiftScaleRotate(p=0.5),\n            HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n            RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n            ], n=3, p=0.6),\n            Resize(CFG['img_size'], CFG['img_size']),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.0)\n  \n        \ndef get_valid_transforms():\n    return Compose([\n            Resize(CFG['img_size'], CFG['img_size']),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.)","metadata":{"execution":{"iopub.status.busy":"2021-06-22T07:34:50.141713Z","iopub.execute_input":"2021-06-22T07:34:50.142089Z","iopub.status.idle":"2021-06-22T07:34:50.156843Z","shell.execute_reply.started":"2021-06-22T07:34:50.142054Z","shell.execute_reply":"2021-06-22T07:34:50.155983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def prepare_dataloader(df, trn_idx, val_idx, data_root='../classify-leaves/'):\n\n    train_ = df.loc[trn_idx,:].reset_index(drop=True)\n    valid_ = df.loc[val_idx,:].reset_index(drop=True)\n        \n    train_ds = MyDataset(train_, data_root, transforms=get_train_transforms(), output_label=True)\n    valid_ds = MyDataset(valid_, data_root, transforms=get_valid_transforms(), output_label=True)\n    \n    train_loader = torch.utils.data.DataLoader(\n        train_ds,\n        batch_size=CFG['train_bs'],\n        pin_memory=True,\n        drop_last=False,\n        shuffle=True,        \n        num_workers=CFG['num_workers'],\n    )\n    val_loader = torch.utils.data.DataLoader(\n        valid_ds, \n        batch_size=CFG['valid_bs'],\n        num_workers=CFG['num_workers'],\n        shuffle=False,\n        pin_memory=True,\n    )\n    return train_loader, val_loader","metadata":{"execution":{"iopub.status.busy":"2021-06-22T07:35:03.632363Z","iopub.execute_input":"2021-06-22T07:35:03.632683Z","iopub.status.idle":"2021-06-22T07:35:03.639765Z","shell.execute_reply.started":"2021-06-22T07:35:03.632653Z","shell.execute_reply":"2021-06-22T07:35:03.638654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model","metadata":{}},{"cell_type":"code","source":"class EfficientModel(nn.Module):\n    def __init__(self, model_arch, n_class, pretrained=False):\n        super().__init__()\n        self.model = timm.create_model(model_arch, pretrained=pretrained)\n        n_features = self.model.classifier.in_features\n        self.model.classifier = nn.Linear(n_features, n_class)\n\n    def forward(self, x):\n        x = self.model(x)\n        return x\n    \nclass ResnetModel(nn.Module):\n    def __init__(self, model_arch, n_class, pretrained=False):\n        super().__init__()\n        self.model = timm.create_model(model_arch, pretrained=pretrained)\n        n_features = self.model.fc.in_features\n        self.model.fc = nn.Linear(n_features, n_class)\n        '''\n        self.model.classifier = nn.Sequential(\n            nn.Dropout(0.3),\n            #nn.Linear(n_features, hidden_size,bias=True), nn.ELU(),\n            nn.Linear(n_features, n_class, bias=True)\n        )\n        '''\n    def forward(self, x):\n        x = self.model(x)\n        return x","metadata":{"execution":{"iopub.status.busy":"2021-06-22T07:35:11.481572Z","iopub.execute_input":"2021-06-22T07:35:11.481891Z","iopub.status.idle":"2021-06-22T07:35:11.490481Z","shell.execute_reply.started":"2021-06-22T07:35:11.481863Z","shell.execute_reply":"2021-06-22T07:35:11.488737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training","metadata":{}},{"cell_type":"code","source":"def train_one_epoch(epoch, model, loss_fn, optimizer, train_loader, device, scheduler=None):\n    model.train()\n\n    running_loss = None\n\n    pbar = tqdm(enumerate(train_loader), total=len(train_loader))\n    for step, (imgs, image_labels) in pbar:\n        imgs = imgs.to(device).float()\n        image_labels = image_labels.to(device).long()\n\n        with autocast():\n            image_preds = model(imgs)\n\n            loss = loss_fn(image_preds, image_labels)\n            scaler.scale(loss).backward()\n\n            if running_loss is None:\n                running_loss = loss.item()\n            else:\n                running_loss = running_loss * .99 + loss.item() * .01\n\n            if ((step + 1) %  CFG['accum_iter'] == 0) or ((step + 1) == len(train_loader)):\n                scaler.step(optimizer)\n                scaler.update()\n                optimizer.zero_grad() \n\n            if ((step + 1) % CFG['verbose_step'] == 0) or ((step + 1) == len(train_loader)):\n                description = f'epoch {epoch} loss: {running_loss:.4f}'\n                \n                pbar.set_description(description)\n                \n    if scheduler is not None:\n        scheduler.step()\n        \n    return running_loss\n        \ndef valid_one_epoch(epoch, model, loss_fn, val_loader, device, scheduler=None):\n    model.eval()\n\n    loss_sum = 0\n    sample_num = 0\n    image_preds_all = []\n    image_targets_all = []\n    val_loss = None\n    \n    pbar = tqdm(enumerate(val_loader), total=len(val_loader))\n    for step, (imgs, image_labels) in pbar:\n        imgs = imgs.to(device).float()\n        image_labels = image_labels.to(device).long()\n        \n        image_preds = model(imgs)\n        image_preds_all += [torch.argmax(image_preds, 1).detach().cpu().numpy()]\n        image_targets_all += [image_labels.detach().cpu().numpy()]\n        \n        loss = loss_fn(image_preds, image_labels)\n        \n        loss_sum += loss.item()*image_labels.shape[0]\n        sample_num += image_labels.shape[0]\n\n        if ((step + 1) % CFG['verbose_step'] == 0) or ((step + 1) == len(val_loader)):\n            val_loss = loss_sum/sample_num\n            description = f'epoch {epoch} loss: {val_loss:.4f}'\n            pbar.set_description(description)\n    \n    image_preds_all = np.concatenate(image_preds_all)\n    image_targets_all = np.concatenate(image_targets_all)\n    \n    val_acc = (image_preds_all==image_targets_all).mean()\n    print('validation multi-class accuracy = {:.4f}'.format(val_acc))\n    \n    if scheduler is not None:\n        scheduler.step()\n        \n    return val_acc, val_loss","metadata":{"execution":{"iopub.status.busy":"2021-06-22T07:35:18.164791Z","iopub.execute_input":"2021-06-22T07:35:18.165134Z","iopub.status.idle":"2021-06-22T07:35:18.180736Z","shell.execute_reply.started":"2021-06-22T07:35:18.165104Z","shell.execute_reply":"2021-06-22T07:35:18.179701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Main Loop","metadata":{}},{"cell_type":"code","source":"if __name__ == '__main__':\n\n    seed_everything(CFG['seed'])\n    \n    folds = StratifiedKFold(n_splits=CFG['fold_num'], shuffle=True, random_state=CFG['seed']).split(np.arange(train_clean.shape[0]), train_clean.label.values)\n    train_loss_all = []\n    val_loss_all = []\n    val_acc_all = []\n    for fold, (trn_idx, val_idx) in enumerate(folds):\n        start_time = time.time()\n        # 测试两个fold\n        if fold > 1:\n            break \n            \n        print('Training with {} started'.format(fold))\n\n        print(len(trn_idx), len(val_idx))\n        train_loader, val_loader = prepare_dataloader(train_clean, trn_idx, val_idx, data_root=CFG['rootdir'])\n\n        device = torch.device(CFG['device'])\n        \n        model = EfficientModel(CFG['model_arch'], train_clean.label.nunique(), pretrained=True).to(device)\n        scaler = GradScaler()   \n        optimizer = torch.optim.AdamW(model.parameters(), lr=CFG['lr'], weight_decay=CFG['weight_decay'])\n        scheduler = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizer, T_0=CFG['T_0'], T_mult=1, eta_min=CFG['min_lr'], last_epoch=-1)\n        #scheduler = torch.optim.lr_scheduler.StepLR(optimizer, gamma=0.1, step_size=CFG['epochs']-1)\n        \n        loss_tr = nn.CrossEntropyLoss().to(device)\n        loss_fn = nn.CrossEntropyLoss().to(device)\n        train_loss_temp = []\n        val_loss_temp = [] \n        val_acc_temp = []\n        \n        for epoch in range(CFG['epochs']):\n            train_loss_temp = train_one_epoch(epoch, model, loss_tr, optimizer, train_loader, device, scheduler=scheduler)\n            train_loss_all.append([fold, epoch, train_loss_temp])\n            with torch.no_grad():\n                val_acc_temp, val_loss_temp = valid_one_epoch(epoch, model, loss_fn, val_loader, device, scheduler=None)\n                \n                val_loss_all.append([fold, epoch, val_loss_temp])\n                val_acc_all.append([fold, epoch, val_acc_temp])\n            if epoch > CFG['epochs'] - 4:  #save last three models\n                torch.save(model.state_dict(),'{}_fold_{}_{}'.format(CFG['model_arch'], fold, epoch))\n                \n        del model, optimizer, train_loader, val_loader, scaler, scheduler\n        torch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2021-06-22T07:39:43.038187Z","iopub.execute_input":"2021-06-22T07:39:43.038504Z","iopub.status.idle":"2021-06-22T09:32:16.893569Z","shell.execute_reply.started":"2021-06-22T07:39:43.038476Z","shell.execute_reply":"2021-06-22T09:32:16.892579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plot the result","metadata":{}},{"cell_type":"code","source":"train_loss_all_df = pd.DataFrame(np.array(train_loss_all), columns = ['fold','epoch','train_loss'])\nval_loss_all_df = pd.DataFrame(np.array(val_loss_all), columns = ['fold','epoch','val_loss'])\nval_acc_all_df = pd.DataFrame(np.array(val_acc_all), columns = ['fold','epoch','val_acc'])","metadata":{"execution":{"iopub.status.busy":"2021-06-22T09:32:45.350362Z","iopub.execute_input":"2021-06-22T09:32:45.350688Z","iopub.status.idle":"2021-06-22T09:32:45.356987Z","shell.execute_reply.started":"2021-06-22T09:32:45.350656Z","shell.execute_reply":"2021-06-22T09:32:45.35617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig1 = px.line(train_loss_all_df, x=\"epoch\", y=\"train_loss\", color='fold')\nfig2 = px.line(val_loss_all_df, x=\"epoch\", y=\"val_loss\", color='fold')\nfig3 = px.line(val_acc_all_df, x=\"epoch\", y=\"val_acc\", color='fold')\nfig4 = go.Figure(data=fig1.data + fig2.data + fig3.data)\nfig4.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-22T09:33:40.586957Z","iopub.execute_input":"2021-06-22T09:33:40.587321Z","iopub.status.idle":"2021-06-22T09:33:40.747894Z","shell.execute_reply.started":"2021-06-22T09:33:40.587288Z","shell.execute_reply":"2021-06-22T09:33:40.746953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-FIVE\"></a>\n# Inference Loop","metadata":{}},{"cell_type":"code","source":"CFG['weights'] = [0.8,1,0.9] # weight for out model\nCFG['tta'] = 3 # set TTA times\nCFG['used_epochs'] = [7,8,9] # choose the model\n\nCFG","metadata":{"execution":{"iopub.status.busy":"2021-06-22T09:35:37.677079Z","iopub.execute_input":"2021-06-22T09:35:37.677395Z","iopub.status.idle":"2021-06-22T09:35:37.685182Z","shell.execute_reply.started":"2021-06-22T09:35:37.677365Z","shell.execute_reply":"2021-06-22T09:35:37.68437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_inference_transforms():\n    return Compose([\n            Resize(CFG['img_size'], CFG['img_size']),\n            Transpose(p=0.5),\n            HorizontalFlip(p=0.5),\n            VerticalFlip(p=0.5),\n            #HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n            #RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n            Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n            ToTensorV2(p=1.0),\n        ], p=1.)","metadata":{"execution":{"iopub.status.busy":"2021-06-22T09:35:49.467699Z","iopub.execute_input":"2021-06-22T09:35:49.468029Z","iopub.status.idle":"2021-06-22T09:35:49.473183Z","shell.execute_reply.started":"2021-06-22T09:35:49.467978Z","shell.execute_reply":"2021-06-22T09:35:49.472344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def inference_one_epoch(model, data_loader, device):\n    model.eval()\n\n    image_preds_all = []\n    \n    pbar = tqdm(enumerate(data_loader), total=len(data_loader))\n    for step, (imgs) in pbar:\n        imgs = imgs.to(device).float()\n        \n        image_preds = model(imgs)\n        image_preds_all += [torch.softmax(image_preds, 1).detach().cpu().numpy()]\n        \n    image_preds_all = np.concatenate(image_preds_all, axis=0)\n    return image_preds_all","metadata":{"execution":{"iopub.status.busy":"2021-06-22T09:35:57.682744Z","iopub.execute_input":"2021-06-22T09:35:57.683069Z","iopub.status.idle":"2021-06-22T09:35:57.689106Z","shell.execute_reply.started":"2021-06-22T09:35:57.68304Z","shell.execute_reply":"2021-06-22T09:35:57.687885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if __name__ == '__main__':\n    \n    seed_everything(CFG['seed'])\n    \n    folds = StratifiedKFold(n_splits=CFG['fold_num']).split(np.arange(train_clean.shape[0]), train_clean.label.values)\n    tst_preds_all = []\n    for fold, (trn_idx, val_idx) in enumerate(folds):\n        \n        if fold > 1:\n            break\n\n        print('Inference fold {} started'.format(fold))\n        \n        valid_ = train_clean.loc[val_idx,:].reset_index(drop=True)\n        valid_ds = MyDataset(valid_, CFG['rootdir'], transforms=get_inference_transforms(), output_label=False)\n\n        test = pd.read_csv( CFG['rootdir'] + 'test.csv')\n        test_ds = MyDataset(test, CFG['rootdir'], transforms=get_inference_transforms(), output_label=False)\n\n        val_loader = torch.utils.data.DataLoader(\n            valid_ds, \n            batch_size=CFG['valid_bs'],\n            num_workers=CFG['num_workers'],\n            shuffle=False,\n            pin_memory=True,\n        )\n\n        tst_loader = torch.utils.data.DataLoader(\n            test_ds, \n            batch_size=CFG['valid_bs'],\n            num_workers=CFG['num_workers'],\n            shuffle=False,\n            pin_memory=True,\n        )\n        \n        device = torch.device(CFG['device'])\n        model = EfficientModel(CFG['model_arch'], train.label.nunique()).to(device)\n        \n        val_preds = []\n        tst_preds = []\n        \n        for i, epoch in enumerate(CFG['used_epochs']):    \n            model.load_state_dict(torch.load('{}_fold_{}_{}'.format(CFG['model_arch'], fold, epoch)))\n            \n            with torch.no_grad():\n                for _ in range(CFG['tta']):\n                    val_preds += [CFG['weights'][i]/sum(CFG['weights'])/CFG['tta']*inference_one_epoch(model, val_loader, device)]\n                    tst_preds += [CFG['weights'][i]/sum(CFG['weights'])/CFG['tta']*inference_one_epoch(model, tst_loader, device)]\n\n                    \n        val_preds = np.mean(val_preds, axis=0) \n        tst_preds = np.mean(tst_preds, axis=0) \n        tst_preds_all.append(tst_preds)\n        \n        print('fold {} validation loss = {:.5f}'.format(fold, log_loss(valid_.label.values, val_preds)))\n        print('fold {} validation accuracy = {:.5f}'.format(fold, (valid_.label.values==np.argmax(val_preds, axis=1)).mean()))\n        del model\n        torch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2021-06-22T10:22:26.869986Z","iopub.execute_input":"2021-06-22T10:22:26.870376Z","iopub.status.idle":"2021-06-22T10:38:26.898316Z","shell.execute_reply.started":"2021-06-22T10:22:26.870337Z","shell.execute_reply":"2021-06-22T10:38:26.897438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# calculate the average result from all folds\navg_tst = np.mean(tst_preds_all, axis=0)","metadata":{"execution":{"iopub.status.busy":"2021-06-22T10:38:43.281053Z","iopub.execute_input":"2021-06-22T10:38:43.281387Z","iopub.status.idle":"2021-06-22T10:38:43.292314Z","shell.execute_reply.started":"2021-06-22T10:38:43.281351Z","shell.execute_reply":"2021-06-22T10:38:43.291357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test['label'] = np.argmax(avg_tst, axis=1)\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2021-06-22T10:38:46.357165Z","iopub.execute_input":"2021-06-22T10:38:46.357499Z","iopub.status.idle":"2021-06-22T10:38:46.373258Z","shell.execute_reply.started":"2021-06-22T10:38:46.35747Z","shell.execute_reply":"2021-06-22T10:38:46.372489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# inverse our label\ntest['label'] = t_labelencoder.inverse_transform(test['label'])","metadata":{"execution":{"iopub.status.busy":"2021-06-22T10:38:56.269459Z","iopub.execute_input":"2021-06-22T10:38:56.269777Z","iopub.status.idle":"2021-06-22T10:38:56.275434Z","shell.execute_reply.started":"2021-06-22T10:38:56.269748Z","shell.execute_reply":"2021-06-22T10:38:56.274373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-06-22T10:39:00.877083Z","iopub.execute_input":"2021-06-22T10:39:00.877409Z","iopub.status.idle":"2021-06-22T10:39:00.912829Z","shell.execute_reply.started":"2021-06-22T10:39:00.877379Z","shell.execute_reply":"2021-06-22T10:39:00.91207Z"},"trusted":true},"execution_count":null,"outputs":[]}]}