{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Summary -> \n\nWe generated TFRecords Dataset Multi-parts using quick_ml for Dog Breed Identification. <br>"},{"metadata":{"papermill":{"duration":0.006974,"end_time":"2020-09-10T03:30:47.573251","exception":false,"start_time":"2020-09-10T03:30:47.566277","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Installation & Imports"},{"metadata":{},"cell_type":"markdown","source":"Install quick_ml & do the necessary imports. Please maintain the order of imports after the installation (TF then quick_ml)"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2020-09-10T03:30:47.591722Z","iopub.status.busy":"2020-09-10T03:30:47.590887Z","iopub.status.idle":"2020-09-10T03:32:10.550594Z","shell.execute_reply":"2020-09-10T03:32:10.549735Z"},"papermill":{"duration":82.971014,"end_time":"2020-09-10T03:32:10.550753","exception":false,"start_time":"2020-09-10T03:30:47.579739","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"!pip install -q quick-ml","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-10T03:32:10.570232Z","iopub.status.busy":"2020-09-10T03:32:10.569462Z","iopub.status.idle":"2020-09-10T03:32:22.947507Z","shell.execute_reply":"2020-09-10T03:32:22.946699Z"},"papermill":{"duration":12.391164,"end_time":"2020-09-10T03:32:22.947658","exception":false,"start_time":"2020-09-10T03:32:10.556494","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nimport quick_ml","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.005377,"end_time":"2020-09-10T03:32:22.958783","exception":false,"start_time":"2020-09-10T03:32:22.953406","status":"completed"},"tags":[]},"cell_type":"markdown","source":"### Making TFRecords Data"},{"metadata":{},"cell_type":"markdown","source":"Classification Image Data is majorly in two formats. Either it is the main data directory and sub-directories or sub-folders contain various classes and the images or there is a CSV file and there is no bifurcation in the main directory. The CSV file contains the image list and it's class labeling. <br>\n#### In this case, we've Images Data & CSV File containing the file name and it's class label."},{"metadata":{"execution":{"iopub.execute_input":"2020-09-10T03:32:22.976811Z","iopub.status.busy":"2020-09-10T03:32:22.975731Z","iopub.status.idle":"2020-09-10T03:32:22.979582Z","shell.execute_reply":"2020-09-10T03:32:22.978952Z"},"papermill":{"duration":0.015183,"end_time":"2020-09-10T03:32:22.979752","exception":false,"start_time":"2020-09-10T03:32:22.964569","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-10T03:32:23.000047Z","iopub.status.busy":"2020-09-10T03:32:22.999221Z","iopub.status.idle":"2020-09-10T03:32:23.038982Z","shell.execute_reply":"2020-09-10T03:32:23.038102Z"},"papermill":{"duration":0.05344,"end_time":"2020-09-10T03:32:23.039116","exception":false,"start_time":"2020-09-10T03:32:22.985676","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"df = pd.read_csv('../input/dog-breed-identification/labels.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Our Dataset matches the Type 2 of the two types of data format supported by quick_ml. To learn more, click [here](https://antoreepjana.wixsite.com/quick-ml/making-custom-datasets-tfrecords) & scroll down to \"Creating Custom Labeled TFRecords Dataset with CSV File\" <br>\n#### Let's bring our Dataframe, df, in the same format as demanded by quick_ml. <br>\n* Filenames must have .jpg or .jpeg\n* Column Names should be ['Image','Id']"},{"metadata":{"execution":{"iopub.execute_input":"2020-09-10T03:32:23.070493Z","iopub.status.busy":"2020-09-10T03:32:23.06938Z","iopub.status.idle":"2020-09-10T03:32:23.072556Z","shell.execute_reply":"2020-09-10T03:32:23.07321Z"},"papermill":{"duration":0.02824,"end_time":"2020-09-10T03:32:23.073374","exception":false,"start_time":"2020-09-10T03:32:23.045134","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"df['id'] = df['id'].apply(lambda x : x + '.jpg')","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-10T03:32:23.100571Z","iopub.status.busy":"2020-09-10T03:32:23.099336Z","iopub.status.idle":"2020-09-10T03:32:23.110256Z","shell.execute_reply":"2020-09-10T03:32:23.110852Z"},"papermill":{"duration":0.031541,"end_time":"2020-09-10T03:32:23.111023","exception":false,"start_time":"2020-09-10T03:32:23.079482","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"df","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-10T03:32:23.130077Z","iopub.status.busy":"2020-09-10T03:32:23.129268Z","iopub.status.idle":"2020-09-10T03:32:23.132928Z","shell.execute_reply":"2020-09-10T03:32:23.132158Z"},"papermill":{"duration":0.015531,"end_time":"2020-09-10T03:32:23.133055","exception":false,"start_time":"2020-09-10T03:32:23.117524","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"df.columns = ['Image', 'Id']","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-10T03:32:23.159817Z","iopub.status.busy":"2020-09-10T03:32:23.158684Z","iopub.status.idle":"2020-09-10T03:32:23.163747Z","shell.execute_reply":"2020-09-10T03:32:23.163138Z"},"papermill":{"duration":0.024422,"end_time":"2020-09-10T03:32:23.163884","exception":false,"start_time":"2020-09-10T03:32:23.139462","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"df","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-10T03:32:23.187142Z","iopub.status.busy":"2020-09-10T03:32:23.185893Z","iopub.status.idle":"2020-09-10T03:32:23.654303Z","shell.execute_reply":"2020-09-10T03:32:23.653518Z"},"papermill":{"duration":0.483838,"end_time":"2020-09-10T03:32:23.654451","exception":false,"start_time":"2020-09-10T03:32:23.170613","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"df.to_csv('train.csv', index = False)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-10T03:32:23.673976Z","iopub.status.busy":"2020-09-10T03:32:23.673179Z","iopub.status.idle":"2020-09-10T03:32:23.7042Z","shell.execute_reply":"2020-09-10T03:32:23.703515Z"},"papermill":{"duration":0.042786,"end_time":"2020-09-10T03:32:23.704348","exception":false,"start_time":"2020-09-10T03:32:23.661562","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"train = pd.read_csv('train.csv')\ntrain","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After processing our CSV File, we can begin generating TFRecords Files from CSV. <br>\n<br>\n##### We'll be using quick_ml's create_split_tfrecords_from_csv\n\ncreate_split_tfrecords_from_csv generates TFRecords Dataset if the dataset has a CSV file of the same format as demanded by quick_ml. To learn about the parameters or arguments needed, please visit this [site](https://antoreepjana.wixsite.com/quick-ml/making-custom-datasets-tfrecords) & search/scroll down to create_split_tfrecords_from_csv."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","execution":{"iopub.execute_input":"2020-09-10T03:32:23.724348Z","iopub.status.busy":"2020-09-10T03:32:23.723546Z","iopub.status.idle":"2020-09-10T03:32:23.727187Z","shell.execute_reply":"2020-09-10T03:32:23.726536Z"},"papermill":{"duration":0.015519,"end_time":"2020-09-10T03:32:23.727312","exception":false,"start_time":"2020-09-10T03:32:23.711793","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"from quick_ml.tfrecords_maker import create_split_tfrecords_from_csv\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##### Parameters -> \n\n* DATA_DIR -> The image directory\n* csv_path -> path of the updated or processed csv file to be used by quick_ml. (That's why we saved the output in the above lines)\n* outfile1name -> Name of the training files you'd like to obtain\n* outfile2name -> Name of the validation files you'd like to obtain\n* split_size_ratio -> ratio in which the dataset has to be divided between the above two files\n* IMAGE_SIZE -> Output Image Size in the TFRecords Dataset."},{"metadata":{"execution":{"iopub.execute_input":"2020-09-10T03:32:23.74902Z","iopub.status.busy":"2020-09-10T03:32:23.748196Z","iopub.status.idle":"2020-09-10T03:33:32.033361Z","shell.execute_reply":"2020-09-10T03:33:32.032187Z"},"papermill":{"duration":68.299037,"end_time":"2020-09-10T03:33:32.033637","exception":false,"start_time":"2020-09-10T03:32:23.7346","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"DATA_DIR = '../input/dog-breed-identification/train'\ncsv_path = './train.csv'\noutfile1name = 'train.tfrecords'\noutfile2name = 'val.tfrecords'\nsplit_size_ratio = 0.7\n\ncreate_split_tfrecords_from_csv(DATA_DIR,csv_path,  outfile1name, outfile2name, split_size_ratio, IMAGE_SIZE = (192,192))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Post-TFRecords Dataset Generation\n\nPlease ensure the following steps have been followed after obtaining TFRecords files:-\n\n* Create a New Public Dataset on Kaggle\n* In the description of the dataset, mention the class encodings. (find it in the output)\n* Mention the Labeled TFRecord Format in the description\n* Mention the Image Size in the TFRecords File (in our case, it would be 192x192)"},{"metadata":{"papermill":{"duration":0.009535,"end_time":"2020-09-10T03:33:32.055011","exception":false,"start_time":"2020-09-10T03:33:32.045476","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}