{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# 0. 알아 둘 사항들 \n * tensorflow 2.x 버전에서 안돌아가요. tensorflow-gpu==1.13.1 . keras-2.1.5\n * keras-yolo3 폴더가 홈아래에 있어서 object detection 수행할 때 귀찮지만 홈디렉토리를 변경 한 후 해야합니다.\n * 프로젝트 구조 \n\n```\n--/kaggle/working/\n  |-YoloV3.ipynb\n  |-data/cropped_train/cropped_train/\n  |-data/cropped_train/cropped_test/\n  |-keras-yolo3\n    |-train.py\n    |-class.txt (label 파일)\n    |-train.txt (yolo에서 훈련할 수 있는 포맷으로 만든 훈련파일 목록\n    |-model_data\n      |-yolo_tiny.h5 (convert.py 로 빌드해 만든 weigth 파일)\n      |-voc_classes.txt(label파일의 사본)\n    |-logs\n      |-000\n        |-checkpoint files .. (checkpoint 파일 저장 경로)\n```\n# 1. pre-requsite\n- 1.1. tiny yolov3 pretrained weights 다운로드\n- 1.2 kears-yolo3 다운로드 \n- 1.3 convert darknet model to Keras \n- 1.4 테스트 \n\n# 2. yolo 적용 \n- 2.1 환경 체크, 데이터 불러오기\n- 2.2 yolo훈련용 이미지 리사이징 \n- 2.3 훈련 파일 목록 train.txt 생성 \n- 2.4 라벨 파일 목록 class.txt 생성 \n- 2.5 train.py 소스 수정\n- 2.6 train.py 훈련 \n                 \n# 1.Pre-requsite\n## 1.1 tiny yolov3 pretrained weights(모바일용 경량화 된 yolo) 다운로드","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!wget https://pjreddie.com/media/files/yolov3-tiny.weights","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.2 keras-yolo3 다운로드","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!git clone https://github.com/larsien/keras-yolo3","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls keras-yolo3","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.3 convert darknet model to Keras \n\n케라스용 모델로 컨버트","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import os \nos.chdir('keras-yolo3')\n!ls ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!python convert.py yolov3-tiny.cfg ../yolov3-tiny.weights model_data/yolo_tiny.h5","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.4 테스트 \n\n잘 돌아가는지 이미지 한개 테스트 해보기. Yolo는 입력 이미지를 일정 크기의 그리드로 나눠서 오브젝트를 탐색합니다. 여기서 입력 이미지는 32의 배수여야 합니다. yolo.py 파일에서 확인 가능.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 1.4.1 이미지 리사이즈","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!unzip /kaggle/input/2019-3rd-ml-month-with-kakr/train.zip -d /kaggle/working/data/train/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#현재 위치는 ./kaggle/working\nfrom PIL import Image\n\ndef get_resized_img(file, file_name='test.jpg', \n                    save_path='./', \n                    image_size=(320,320)):\n    Image.open(file).resize(image_size).save(save_path+file_name)\n    return Image.open(save_path+file_name)\n\ntest_img = get_resized_img('/kaggle/working/data/train/train_00036.jpg')\ntest_img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.chdir('./keras-yolo3')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#에러 난다면 tensorflow 버전 확인 필요\nfrom yolo import YOLO\ndef objectDetection(path, model_path, class_path):\n    yolo = YOLO(model_path = model_path, classes_path = class_path, anchors_path = 'model_data/tiny_yolo_anchors.txt')\n    result_image, label = yolo.detect_image(Image.open(path))\n    display(result_image)\n    return result_image\n    \n# os.chdir('./keras-yolo3')\ntest_result = objectDetection('../test.jpg', 'model_data/yolo_tiny.h5','model_data/coco_classes.txt')\ntest_result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. yolo 적용 \n## 2.1 환경 체크, 데이터 불러오기","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# 환경마다 삭제해야 할 라이브러리가 조금 씩 다름. 에러나는 것들 지우고 텐서플로우 설치 \n!pip uninstall tensorflow tensorflow-estimator tensorflow-probability keras fancyimpute catalyst --yes\n!pip uninstall tensorflow-gpu tensorflow-tensorboard --yes \n\n!pip install tensorflow-gpu==1.13.1\n!pip install keras==2.1.5\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport PIL\nfrom sklearn.model_selection import StratifiedKFold, KFold\nfrom tensorflow.keras import backend as K\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.applications import Xception\nfrom tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint\nfrom tensorflow.keras import layers, models, optimizers\nfrom tensorflow.keras.utils import multi_gpu_model\nimport keras \nimport warnings \nimport tensorflow as tf\nwarnings.filterwarnings('ignore')\n\nprint(K.image_data_format()) # 이미지 데이터 포맷 방식 확인. first, last 2가지. 이미지를 처음부터 읽을지 마지막부터 읽을지 \nprint(keras.__version__)\nprint(tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import gc\nimport os\nimport glob\nimport zipfile\nimport warnings\nimport numpy as np \nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n\nimport cv2\nimport PIL\nfrom PIL import ImageOps, ImageFilter, ImageDraw\nimport pandas as pd\nimport os\n\nDATA_PATH = os.getcwd()+'/data/'\nTRAIN_IMG_PATH = os.path.join(DATA_PATH, 'train')\nTEST_IMG_PATH = os.path.join(DATA_PATH, 'test')\n\n# CSV 파일 경로\ndf_train = pd.read_csv(os.path.join(DATA_PATH, 'train.csv'))\ndf_test = pd.read_csv(os.path.join(DATA_PATH, 'test.csv'))\ndf_class = pd.read_csv(os.path.join(DATA_PATH, 'class.csv'))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.head()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.2 yolo훈련용 이미지 리사이징 \n\n32의 배수만 사용 가능 . 320x320, 288x288등. 전 320으로 고 ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_SIZE = 320\n\ndef crop_resize_boxing_img(img_name, margin=16, size=(288,288)):\n    PATH = ''\n    if 'train_' in img_name:\n        PATH = TRAIN_IMG_PATH\n        data = df_train\n    elif 'test_' in img_name:\n        PATH = TEST_IMG_PATH\n        data = df_test\n    else:\n        print(f'wrong file input {img_name}')\n        return\n    img = PIL.Image.open(os.path.join(PATH, img_name))\n#     print(data['img_file'] )\n    pos = data.loc[data['img_file'] == img_name, ['bbox_x1','bbox_y1','bbox_x2','bbox_y2']].values.reshape(-1)\n#     print(pos[:10])\n    width, height =  img.size\n    x1 = max(0, pos[0] - margin)\n    y1 = max(0, pos[1] - margin)\n    x2 = min(pos[2] + margin, width)\n    y2 = min(pos[3] + margin, height)\n    return img.crop((x1, y1, x2, y2)).resize(size)\n\nTRAIN_CROP_PATH = './data/cropped_train'\nTEST_CROP_PATH = './data/cropped_test'\n\nif (os.path.isdir(TRAIN_CROP_PATH) == False):\n    os.mkdir(TRAIN_CROP_PATH)\n    for i, row in df_train.iterrows():\n        cropped = crop_resize_boxing_img(row['img_file'])\n        cropped.save(f\"{TRAIN_CROP_PATH}/{row['img_file']}\")\n\nif (os.path.isdir(TEST_CROP_PATH) == False):\n    os.mkdir(TEST_CROP_PATH)\n    for i, row in df_test.iterrows():\n        cropped = crop_resize_boxing_img(row['img_file'])\n        cropped.save(f\"{TEST_CROP_PATH}/{row['img_file']}\")\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display(PIL.Image.open(os.path.join(TRAIN_IMG_PATH, df_train.loc[0,'img_file'])))\ndisplay(PIL.Image.open(os.path.join(TRAIN_CROP_PATH,df_train.loc[0, 'img_file'])))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.3 훈련 파일 목록 train.txt 생성 \n\n* 형식 : 파일 경로 x1,y1,x2,y2,class x1,y1,x2,y2,class\n\n* ex) /KaggleKorea3rd_2/data/train_00001.jpg 1,80,641,461,108 \n\n* yolo는 label이 0 부터 시작 해야 함. 캐글 훈련 셋의 label이 1부터 시작하기 때문에 조정해야 함. \n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train.to_csv('sample.txt', index=False, header=None, sep=',')\n\nwith open('sample.txt', 'r') as f, open('./keras-yolo3/train.txt', 'w') as w:\n    # train_00001.jpg,1,80,641,461,108\n    lines = f.readlines()\n    for line in lines:\n        w.write(f\"{line.split(',')[0].replace('train_',home+'/data/cropped_train/train_')} 0,0,320,320,{int(line.split(',')[-1]) -1}\\n\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.4 라벨 파일 class.txt 생성\n* keras-yolo3/model_data/voc_classes.txt 보면 형식이 다음과 같음. classes.csv 의 형식 변경\n\n```python\naeroplane\nbicycle\nbird\n```\n* 'class id must be less than num_classes' train셋의 라벨 갯수와 class.txt 파일에 라벨 갯수 맞춰야 함 ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class_df = pd.read_csv('./data/class.csv')\nclass_df = class_df.loc[:, 'name']\nclass_df.to_frame().to_csv('./keras-yolo3/class.txt',index=None, header=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.5 train.py 소스 수정 \n* train.py의 다음 부분을 우리가 만든 train.txt와 class.txt 변경 \n\n* before \n```python\nclasses_path = 'model_data/voc_classes.txt'\nanchors_path = 'model_data/yolo_anchors.txt' \nmodel = create_tiny_model(input_shape, anchors, num_classes,\n            freeze_body=2, weights_path='model_data/tiny_yolo_weights.h5')\n```\n* after\n``` python\nclasses_path = 'class.txt'\nanchors_path = 'model_data/tiny_yolo_anchors.txt'\nmodel = create_tiny_model(input_shape, anchors, num_classes,\n            freeze_body=2, weights_path='model_data/yolo_tiny.h5')\n```\n","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"* model_checkpoint 재사용을 위한 코드를 train.py에 추가 ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def is_history_exist(log_dir):\n    history_files = []\n    for file in os.listdir(log_dir):\n        if '.h5' in file:\n            history_files.append(file)\n    \n    if len(history_files) == 0:\n        return False\n    else : \n        return True\n    \ndef get_minimum_loss_weight(log_dir):\n    history_files = []\n    for file in os.listdir(log_dir):\n        if '.h5' in file:\n            history_files.append(file)\n    minimum_loss = float(9999999)\n    minimum_loss_file = ''\n    for file in history_files:\n        loss = float(file.split('-')[1].replace('loss',''))\n        if loss < minimum_loss:\n            minimum_loss = loss\n            minimum_loss_file = file\n    print(f'Minimum loss file : {minimum_loss_file}, minimum loss : {minimum_loss}, ')\n    epoch = int(minimum_loss_file.split('-')[0].replace('ep', ''))\n    return os.path.join(log_dir,minimum_loss_file), epoch\n\nlog_dir = 'logs/000'\nprint(is_history_exist('./test'))\nprint(get_minimum_loss_weight(log_dir))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.6 train.py 훈련 \n* 훈련 시 home 디렉토리 변경 후 해야 함. \n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#현재 위치는 keras-yolo \n!python ./train.py","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.7 훈련된 모델 테스트 ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(os.getcwd())\nimport os\n# os.chdir('./keras-yolo3')\nfrom PIL import Image\nfrom yolo import  YOLO \nimg, i2 = objectDetection('../data/cropped_test/test_00002.jpg', 'logs/000/ep100-loss8.058-val_loss8.383.h5', 'model_data/voc_classes.txt')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Trouble shooting\n1. cause : AttributeError: module 'keras.backend' has no attribute 'control_flow_ops'\n* solve : hange K.control_flow_ops.while_loop to tf.while_loop\n* https://github.com/qqwweee/keras-yolo3/issues/544#issuecomment-526825314\n2. Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.\n* solve : \nimport keras\nimport tensorflow as tf\n\nconfig = tf.ConfigProto()\nconfig.gpu_options.per_process_gpu_memory_fraction = 0.9\nkeras.backend.tensorflow_backend.set_session(tf.Session(config=config))\n\nand \n\nchanged gpu counts from 2 to 1 \n\n* https://github.com/tensorflow/tensorflow/issues/24828#issuecomment-609426655\n\n3. AssertionError: class id must be less than num_classes\ntrain.txt 에 들어갈 label 값이 0 부터 시작해 총 갯수가 class.txt 갯수와 맞아야 함. 캐글3rd데이터셋 경우 label이 1부터 시작했음. ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 2.8 결과 제출 ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def objects_detection(path, model_path, class_path):\n    yolo = YOLO(model_path = model_path, classes_path = class_path, anchors_path = 'model_data/tiny_yolo_anchors.txt')\n    result_image, score = yolo.detect_image(Image.open(path))\n    print(score)\n    display(result_image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df = df_test.loc[:,'img_file'].to_frame()\nfor i, img_path in enumerate(submission_df.loc[:,'img_file']):\n    objectDetection(os.path.join(TEST_CROP_PATH, img_path), 'model_data/yolo_tiny.h5', 'class.txt')\n\nsubmission_df['class'] = pred\nsubmission_df.to_csv('suvmission.csv', index = False)\nsubmission_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}