{"cells":[{"metadata":{},"cell_type":"markdown","source":"# 1. Motivation","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"As noted by [@Peter](https://www.kaggle.com/pestipeti) in this [notebook](http://www.kaggle.com/pestipeti/jigsaw-puzzle-solved), the images from train dataset are 1024x1024 squared patches of the original images. For some classes we can reconstruct the original images by solving a jigsaw puzzle. For other classes, we have some patches missing se we can't reconstruct all the original images but we can get most of them.\n\nThe mentioned notebook by @Peter doesn't work for the (3x2) and for arvalis_3 class because all the images of this class are too similar to each other. My notebook works for all the classes including (3x2) and arvalis_3.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Parameters","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"My approach is configurable to some extent. You can run the script with different parameters to maybe get better results.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Outputs","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"There is an output with the merges images and the corresponding dataset with the bounding boxes. Since version 9 of the kernel, the bounding boxes of adjacent images that have large edges in common are merged.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Known issues","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"1. After a visual inspection, all the results are correct except for two images in arvalis_3 and one in ethz_1. We must remove those images manually.\n2. For arvalis_3, arvalis_1 and ethz_1, there are patches that couldn't be linked to any group. This is normal because we don't have all the patches for those two classes. Maybe with a better tuning of the parameters other groups could be found.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# 2. Imports and definitions","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nfrom matplotlib import patches\nimport ast\nimport os\nimport random\nfrom IPython.display import clear_output\nfrom datetime import datetime\nfrom tqdm.notebook import tnrange\nfrom shutil import copyfile","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"INPUT_DIR = '/kaggle/input/global-wheat-detection/'\nOUTPUT_DIR = '/kaggle/working/global-wheat-detection/'\nTEST_DIR = INPUT_DIR+'test/'\nTRAIN_DIR = INPUT_DIR+'train/'\nTRAIN_LABELS_FILE = INPUT_DIR+'train.csv'\nPUZZLE_IMAGES_TRAIN_DIR = OUTPUT_DIR+'train_puzzle/'\nPUZZLE_IMAGES_TRAIN_LABELS_FILE = OUTPUT_DIR+'train_puzzle.csv'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_ax(row=1, col=1, size=16, figsize=None):\n    if figsize is None:\n        _, ax = plt.subplots(row,col,figsize=(size,size))\n    else:\n        _, ax = plt.subplots(row,col,figsize=figsize)\n    return ax\n\ndef display_instance(filename, bbox, ax):\n    im = Image.open(filename)\n    color=[\"red\",\"yellow\",\"blue\",\"purple\",\"green\",\"black\",\"white\"]\n    if type(bbox)!=list:\n        lbbox=[bbox]\n    else:\n        lbbox=bbox\n    for idx, bbox_list in enumerate(lbbox):\n        for str_box in bbox_list:\n            if type(str_box)==np.ndarray:\n                box = str_box\n            else:\n                box = ast.literal_eval(str_box)\n            #print(box)\n            x1, y1, w, h = box\n            p = patches.Rectangle((x1, y1), w, h, linewidth=2,\n                                  alpha=0.7, linestyle=\"solid\",\n                                  edgecolor=color[idx%len(color)], facecolor='none')\n            ax.add_patch(p)\n    ax.imshow(im)\n    ax.set_title(filename)\n\ndef merge_images(candidate):\n    shape = (2,3) if len(candidate)==6 else (1,2) if len(candidate)==2 else (2,2)\n    im = np.zeros((1024*shape[0],1024*shape[1],3),dtype='uint8')\n    for rows in range(shape[0]):\n        for cols in range(shape[1]):\n            im[rows*1024:(rows+1)*1024, cols*1024:(cols+1)*1024] = np.array(Image.open(TRAIN_DIR+candidate[rows*shape[1]+cols]+'.jpg'))\n    return im\n\ndef get_images_in_group(solved):\n    linked = []\n    for candidate in solved:\n        shift = parameters['shift'][candidate['dir_type']]\n        linked.append([images[n] for n in np.array(candidate['nodes'])[shift]])\n    return linked","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Data Frames","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels=pd.read_csv(TRAIN_LABELS_FILE)\nbboxes = [ast.literal_eval(str_box) for str_box in train_labels['bbox'].values]\ntrain_labels['x'] = [i[0] for i in bboxes]\ntrain_labels['y'] = [i[1] for i in bboxes]\ntrain_labels['w'] = [i[2] for i in bboxes]\ntrain_labels['h'] = [i[3] for i in bboxes]\ndel train_labels['bbox']\ndel train_labels['width']\ndel train_labels['height']\ntrain_labels.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_all_images = pd.DataFrame({'count': train_labels.groupby(['image_id','source'])['image_id'].count()}).reset_index()\ntrain_all_images = train_all_images.join(pd.DataFrame({'image_id':[f[:-4] for f in os.listdir(TRAIN_DIR)]}).set_index('image_id'), how='outer', on='image_id').reset_index(drop=True)\ntrain_all_images['source'] = train_all_images['source'].fillna('empty')\ntrain_all_images['count'] = train_all_images['count'].fillna(0)\ntrain_all_images.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Jigsaw Solver","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Set_config","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"In config we have the parameters used by each of the classes. You can modify config['params'] to try to get better results. Do not modify the rest of the parameters.\n\nConfig['params'] is a list of tuples. We will run the algorithm for each tuple in the list. Each tuple contains:\n* Threshold: to define two borders as neighbors the distance between them must be less than this value.\n* Similarity acceptance: For the moment I always use 0.2. For a candidate to be accepted, the distance between edges must be 20% lower than for the second best candidate","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def set_config(group_name):\n    config = {}\n    if group_name == 'rres_1':\n        config['params'] = [(0.1,0.2),(0.2,0.2)]\n    elif group_name == 'usask_1':\n        config['params'] = [(0.2,0.2)]\n    elif group_name == 'ethz_1':\n        config['params'] = [(0.2,0.2),(0.25,0.2)]\n    elif group_name == 'inrae_1':\n        config['params'] = [(0.2,0.2)]\n    elif group_name == 'arvalis_3':\n        config['params'] = [(0.05,0.2),(0.1,0.2),(0.15,0.2),(0.2,0.2)]\n    elif group_name == 'arvalis_2':\n        config['params'] = [(0.15,0.2),(0.25,0.2)]\n    elif group_name == 'arvalis_1':\n        config['params'] = [(0.1,0.2),(0.15,0.2)]\n\n    if group_name in ['arvalis_1', 'rres_1']:\n        config['shape'] = (2,3)\n        config['list_borders'] = [[0,0,1,2,2,3],[0,1,2,2,3,0],[1,2,2,3,0,0],[2,2,3,0,0,1],[2,3,0,0,1,2],[3,0,0,1,2,2]]\n        config['shortcuts'] = [[[-1],[-1],[-1],[-1],[3,1],[3,0]], [[-1],[-1],[-1],[3,0],[-1],[0,0]], [[-1],[-1],[-1],[-1],[-1],[0,0,1,2]], [[-1],[-1],[-1],[-1],[1,1],[1,0]], [[-1],[-1],[-1],[1,0],[-1],[2,0]], [[-1],[-1],[-1],[-1],[-1],[2,0,3,2]]]\n        config['nb_steps'] = 5\n        config['shift'] = [[3,4,5,2,1,0],[2,3,4,1,0,5],[1,2,3,0,5,4],[0,1,2,5,4,3],[5,0,1,4,3,2],[4,5,0,3,2,1]]\n    elif group_name in ['arvalis_2', 'arvalis_3', 'inrae_1']:\n        config['shape'] = (2,2)\n        config['list_borders'] = [[0,1,2,3],[1,2,3,0],[2,3,0,1],[3,0,1,2]]\n        config['shortcuts'] = [[[-1],[-1],[-1],[3,0]], [[-1],[-1],[-1],[0,0]], [[-1],[-1],[-1],[1,0]], [[-1],[-1],[-1],[2,0]]]\n        config['nb_steps'] = 3\n        config['shift'] = [[2,3,1,0],[1,2,0,3],[0,1,3,2],[3,0,2,1]]\n    elif group_name in ['usask_1', 'ethz_1']:\n        config['shape'] = (1,2)\n        config['list_borders'] = [[0],[2]]\n        config['shortcuts'] = [[[-1],[-1]],[[-1],[-1]]]\n        config['nb_steps'] = 1\n        config['shift'] = [[1,0],[0,1]]\n    return config","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Read Images","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I read all the images of a group and return the following information:\n* color_signature: np_array with the colors of each of the edges of the image. We can set nb_avg to x get the average of the x pixels near the edge.\n* images: list with the name of the images\n* cache: a dictionary that I will use to avoid repeating calculation of distance between edges\n* used: an array with the images that were already linked into a group","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_images(df, group_name):\n    nb_avg = 1\n    images = df[df['source']==group_name]['image_id'].values\n    color_signature = np.zeros((4,len(images),1024,3))\n    for idx, image_id in enumerate(images):\n        image_filename = TRAIN_DIR+image_id+'.jpg'\n        im_pil = Image.open(image_filename)\n\n        #enhancer = ImageEnhance.Brightness(im_pil)\n        #enhancer = ImageEnhance.Color(im_pil)\n        #im_pil = enhancer.enhance(1.6)\n        #enhancer = ImageEnhance.Brightness(im_pil)\n        #im_pil = enhancer.enhance(1.1)\n\n        im = np.array(im_pil)\n        color_signature[0,idx] = im[:,:nb_avg].mean(axis=1)   # left border\n        color_signature[1,idx] = im[:nb_avg,:].mean(axis=0)   # top border\n        color_signature[2,idx] = im[:,-nb_avg:].mean(axis=1)  # right border\n        color_signature[3,idx] = im[-nb_avg:,:].mean(axis=0)  # bottom border\n    cache = {'cache': np.zeros((4,len(images),len(images))), 'is_cached':np.zeros((4,len(images)))}\n    used = np.zeros(len(images), dtype='uint8')\n    \n    return color_signature, images, cache, used","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Border distance","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"With an image and a direction, calculate the distance to all the other images of the group. If destination_node is not null, return the distance with another image, otherwise, return an array of distances with all the other images.\nI defined two distances measures: euclidean and selection of the channels (rgb) with max difference for each point and then euclidean. Any other distance can be used, like cosine distance. You just need to add another elif for distance_type","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def border_distance(image_nb, direction, destination_node=None):\n    distance_type = 'euclidean'\n    if cache['is_cached'][direction,image_nb]==1:\n        if destination_node is None:\n            return cache['cache'][direction,image_nb]\n        else:\n            return cache['cache'][direction,image_nb,destination_node]\n    else:\n        border_1, border_2 = color_signature[direction,image_nb:image_nb+1], color_signature[(direction+2)%4,:]\n        if distance_type=='euclidean':\n            distance = np.sqrt(np.sum((border_1[None, :, :] - border_2[:, None, :])**2,axis=(-2,-1))[:,0]/(3*1024))/256\n        elif distance_type=='max_channel+euclidean':\n            distance = np.sqrt(np.sum(np.max((border_1[None, :, :] - border_2[:, None, :])**2,axis=-1),axis=-1)[:,0]/1024)/256\n        cache['cache'][direction,image_nb] = distance\n        cache['is_cached'][direction,image_nb] = 1\n        if destination_node is None:\n            return distance\n        else:\n            return distance[destination_node]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Get Neighbors","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Given a candidate, get the distance with the five neighbors closest to it if this distance is lower than the accepted threshold","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_neighbors(candidate, param_config):\n    NB_NEIGHBOAR_MAX = 5\n    image_nb = candidate['nodes'][-1]\n    direction = candidate['next_direction']\n    distance = border_distance(image_nb, direction)\n    args = np.argsort(distance)\n    args = np.array([x for x in args if x not in candidate['nodes']])\n    dist_threshold = np.where(np.logical_and(distance[args]<param_config[0],used[args]==0))[0][:NB_NEIGHBOAR_MAX]\n    neigh_distance = distance[args][dist_threshold]\n    neigh_node = args[dist_threshold]\n    return [{'distance': neigh_distance[i], 'node': neigh_node[i]} for i in range(len(dist_threshold))]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Check shortcuts","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Check if two images are close enough to each other. This is useful for the (3x2) images to validate that the two images in the center are close enough.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def check_shortcuts(candidate, param_config):\n    shortcuts = parameters['shortcuts']\n    original_node = candidate['nodes'][-1]\n    candidate_pos = len(candidate['nodes'])-1\n    direction = shortcuts[candidate['dir_type']][candidate_pos][0]\n    if direction>=0:\n        destination_node = candidate['nodes'][shortcuts[candidate['dir_type']][candidate_pos][1]]\n        distance = border_distance(original_node, direction, destination_node)\n        candidate_res = candidate.copy()\n        candidate_res['distance_acc'] += distance\n        if len(shortcuts[candidate['dir_type']][candidate_pos])==2 or distance>=param_config[0]:\n            return distance<param_config[0], candidate_res\n        else:\n            direction = shortcuts[candidate['dir_type']][candidate_pos][2]\n            destination_node = candidate['nodes'][shortcuts[candidate['dir_type']][candidate_pos][3]]\n            distance = border_distance(original_node, direction, destination_node)\n            candidate_res['distance_acc'] += distance\n            return distance<param_config[0], candidate_res\n    else:\n        return True, candidate","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Main process","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"1. We iterate over the list of parameters (*for config in parameters['params']*)\n2. For each config, we iterate over all the images of the group (*for idx in tnrange(....*)\n3. We build a list of candidates with all the images with a distance lower than the threshold\n4. If at the end we get only one candidate we take it as a final image. If there are more than one, we take the best if the distance with the second one is big enough and we move to the next image","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def solve_puzzle(group_name):\n    print(\"Processing images for group\",group_name)\n    start_time = datetime.now()\n    global parameters\n    global color_signature\n    global images\n    global cache\n    global used\n    parameters = set_config(group_name)\n    color_signature, images, cache, used = read_images(train_all_images,group_name)\n    process_time = (datetime.now()-start_time).total_seconds()\n    print(\"Start solving puzzle\")\n    solved=[]\n    candidates = []\n    nb_groups_found = 0\n    start_time = datetime.now()\n    #steps = 0\n    for config in parameters['params']:\n        threshold = config[0]\n        for idx in tnrange(len(images),desc=\"threshold \"+str(threshold),leave=False):\n            if used[idx]==1:\n                continue\n            candidates = [{'dir_type': i, 'threshold': config[0], 'steps': [], 'distance_acc': 0.0, 'nodes': [idx], 'next_direction':parameters['list_borders'][i][0]} for i in range(len(parameters['list_borders']))]\n            for i in range(parameters['nb_steps']):\n                #steps += 1\n                cur_candidates = candidates.copy()\n                candidates = []\n                for cur_candidate in cur_candidates:\n                    neighbors = get_neighbors(cur_candidate, config)\n                    for n in neighbors:\n                        next_steps = cur_candidate['steps'].copy()\n                        next_steps.append(cur_candidate['next_direction'])\n                        next_nodes = cur_candidate['nodes'].copy()\n                        next_nodes.append(n['node'])\n                        new_candidate = {\n                            'dir_type': cur_candidate['dir_type'], \n                            'threshold': cur_candidate['threshold'],\n                            'steps': next_steps,\n                            'distance_acc': cur_candidate['distance_acc']+n['distance'],\n                            'nodes': next_nodes,\n                            'next_direction': parameters['list_borders'][cur_candidate['dir_type']][len(cur_candidate['nodes'])] if len(cur_candidate['nodes'])<parameters['nb_steps'] else -1\n                        }\n                        shortcut_ok, new_candidate = check_shortcuts(new_candidate, config)\n                        if shortcut_ok:\n                            candidates.append(new_candidate)\n            if len(candidates)==1:\n                candidates = candidates[0]\n                used[candidates['nodes']] = 1\n            elif len(candidates)>1:\n                candidates.sort(key=lambda x:x['distance_acc'])\n                if candidates[0]['distance_acc']*(1+config[1])<candidates[1]['distance_acc']:\n                    candidates = candidates[0]\n                    used[candidates['nodes']] = 1\n                else:\n                    candidates = []\n            if not candidates == []:\n                solved.append(candidates)\n            \n    group_list = get_images_in_group(solved)\n    clear_output(wait=True)\n    #print(\"Images processed in\",process_time,\"seconds\")\n    #print(\"Puzzle solved in\",(datetime.now()-start_time).total_seconds(),\"seconds\")\n    #print(\"Groups found:\",len(solved))\n    #print(\"Images linked to a group:\",len(solved)*parameters['shape'][0]*parameters['shape'][1],\"out of\",len(images))\n    \n    return group_list\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Run the puzzle solver for all the classes","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"groups = {}\nclasses = ['rres_1','inrae_1','ethz_1','arvalis_1','arvalis_2','arvalis_3','usask_1']\nfor c in classes:\n    groups[c] = solve_puzzle(c)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for g in groups.keys():\n    print(\"source:  \", g, \"\\t  length:\", len(groups[g]), \"\\texamples:\", groups[g][:2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If group is 'arvalis_3' or 'ethz_1', we remove the images we know that were incorrectly linked","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for group_name in groups.keys():\n    incorrect_groups = []\n    if group_name == 'arvalis_3':\n        incorrect_groups = [['88f3e7313','47a1184e4','f8d848989','218a99bee'],['dbd433d29','d688932d4','fabaeac81','2ad7fa68e']]\n    elif group_name == 'ethz_1':\n        incorrect_groups = [['8a702e7da', '02d662fa8']]\n    groups[group_name] = [g for g in groups[group_name] if g not in incorrect_groups]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for g in groups.keys():\n    print(\"source:  \", g, \"\\t  length:\", len(groups[g]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Add non grouped images","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"for g in groups.keys():\n    images = train_all_images[train_all_images['source']==g]['image_id'].values\n    grouped_images = [im for gg in groups[g] for im in gg]\n    non_grouped_images = [im for im in images if im not in grouped_images]\n    for im in non_grouped_images:\n        groups[g].append([im])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for g in groups.keys():\n    print(\"source:  \", g, \"\\t  length:\", len(groups[g]), \"\\t grouped:\", len([a for a in groups[g] if len(a)>1]), \"\\t non-grouped:\", len([a for a in groups[g] if len(a)==1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Save images and dataset","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"if not os.path.exists(PUZZLE_IMAGES_TRAIN_DIR):\n    os.makedirs(PUZZLE_IMAGES_TRAIN_DIR)\nIMG_SIZE=1024\nnew_images = []\nfor k2 in tnrange(len(groups.keys()),leave=False):\n    k = list(groups.keys())[k2]\n    for g2 in tnrange(len(groups[k]),leave=False):\n        g = groups[k][g2]\n        if len(g)==1:\n            new_images.append((k,g,g[0],g[0],1,0,0))\n            copyfile(TRAIN_DIR+g[0]+'.jpg', PUZZLE_IMAGES_TRAIN_DIR+g[0]+'.jpg')\n        else:\n            if len(g)==2:\n                im_np = np.zeros((IMG_SIZE,IMG_SIZE*2,3),dtype='uint8')\n            elif len(g)==4:\n                im_np = np.zeros((IMG_SIZE*2,IMG_SIZE*2,3),dtype='uint8')\n            else:\n                im_np = np.zeros((IMG_SIZE*2,IMG_SIZE*3,3),dtype='uint8')\n            for idx, im in enumerate(g):\n                if len(g)==2:\n                    start_x, start_y = 1024*idx, 0\n                elif len(g)==4:\n                    start_x, start_y = 1024*(idx%2), 1024*(idx//2)\n                else:\n                    start_x, start_y = 1024*(idx%3), 1024*(idx//3)\n                im_np[start_y:start_y+1024,start_x:start_x+1024] = np.array(Image.open(TRAIN_DIR+im+'.jpg'))\n                new_images.append((k,g,\"\".join(g),im,len(g),start_x,start_y))\n            Image.fromarray(im_np).save(PUZZLE_IMAGES_TRAIN_DIR+\"\".join(g)+'.jpg')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(PUZZLE_IMAGES_TRAIN_DIR)[:10]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_puzzle = pd.DataFrame(new_images, columns = ['source', 'image_group', 'final_image_id', 'image_id', 'image_len', 'start_x', 'start_y'])\ndf_puzzle.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels_big_images = pd.merge(df_puzzle, train_labels, on=['source','image_id'])\ntrain_labels_big_images['x'] += train_labels_big_images['start_x']\ntrain_labels_big_images['y'] += train_labels_big_images['start_y']\ntrain_labels_big_images['bbox_source']='original'\ndel train_labels_big_images['start_x']\ndel train_labels_big_images['start_y']\ntrain_labels_big_images.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### display some random images with boxes","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ax = get_ax(10,2,figsize=(30,120))\nall_images = train_labels_big_images['final_image_id'].values\nfor i in range(20):\n    random_image = np.random.choice(all_images)\n    bboxes = train_labels_big_images[train_labels_big_images['final_image_id']==random_image][['x','y','w','h']].values\n    display_instance(PUZZLE_IMAGES_TRAIN_DIR+random_image+'.jpg',bboxes,ax[i//2,i%2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Merge boxes in adjacent patches","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def merge_boxes(bbox_1, bbox_2, axis):\n    if axis==0: \n        offset=1\n    else: \n        offset=0\n    \n    if bbox_1[offset]>=(bbox_2[offset]+bbox_2[offset+2]):\n        return None\n    if (bbox_1[offset]+bbox_1[offset+2])<=bbox_2[offset]:\n        return None\n    intersection = min(bbox_1[offset]+bbox_1[2+offset],bbox_2[offset]+bbox_2[2+offset])-max(bbox_1[offset],bbox_2[offset])\n    union = max(bbox_1[offset]+bbox_1[2+offset],bbox_2[offset]+bbox_2[2+offset])-min(bbox_1[offset],bbox_2[offset])\n    if (intersection/union)<0.6:\n        return None\n    return [min(bbox_1[0],bbox_2[0]),min(bbox_1[1],bbox_2[1]),max(bbox_2[0]+bbox_2[2]-bbox_1[0],bbox_1[0]+bbox_1[2]-bbox_2[0]),max(bbox_2[1]+bbox_2[3]-bbox_1[1],bbox_1[1]+bbox_1[3]-bbox_2[1])]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is the function to merge boxes. The coding is awful with lots of embeded loops. Probably it could be improved a lot, but since the dataset is small, it takes a reasonable time to finish.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def df_merge_boxes(df,axis, img_names):\n    rows = []\n    for img_name in img_names:\n        nb_images = df[df['final_image_id']==img_name][['image_len']].values[0][0]\n        row = df[df['final_image_id']==img_name].values.tolist()\n        bboxes = df[df['final_image_id']==img_name][['x','y','w','h']].values.tolist()\n        merged_boxes = []\n        if nb_images == 1 or (nb_images==2 and axis==1):\n            rows += row\n        else:\n            if nb_images==6 and axis==0:\n                edges = [1024,2048]\n            elif nb_images==6 and axis==1:\n                edges = [1024]\n            elif nb_images==4:\n                edges = [1024]\n            elif nb_images==2:\n                edges = [1024]\n            for idx, edge in enumerate(edges):\n                lbox_1 = [b for b in bboxes if b[axis]+b[axis+2]==edge]\n                lbox_2 = [b for b in bboxes if b[axis]==edge]\n                #print(axis,idx,edge,len(lbox_1),len(lbox_2))\n                for box1 in lbox_1:\n                    for box2 in lbox_2:\n                        b = merge_boxes(box1,box2,axis)\n                        if b is not None:\n                            row = [r for r in row if not (box1[0]==r[5] and box1[1]==r[6] and box1[2]==r[7] and box1[3]==r[8]) and not (box2[0]==r[5] and box2[1]==r[6] and box2[2]==r[7] and box2[3]==r[8])]\n                            merged_boxes.append(row[0][:3]+[None]+[row[0][4]]+b+['merged'])\n            rows += row\n            rows += merged_boxes\n    return pd.DataFrame(rows,columns = ['source','image_group','final_image_id','image_id','image_len','x','y','w','h','bbox_source'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Fix boxes in vertical axis and then in horizontal axis. Save the results in a dataset.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"img_names = np.unique(train_labels_big_images['final_image_id'].values)\n\ntrain_labels_big_images = df_merge_boxes(train_labels_big_images,0, img_names)\ntrain_labels_big_images = df_merge_boxes(train_labels_big_images,1, img_names)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Display the results. Boxes in yellow are merged boxes. In red the original ones.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"ax = get_ax(2,2,figsize=(60,60))\nall_images = np.random.choice(np.unique(train_labels_big_images[train_labels_big_images['image_len']>1]['final_image_id'].values),4)\nprint(all_images)\nfor i in range(len(all_images)):\n    random_image = all_images[i]\n    bboxes_original = train_labels_big_images[(train_labels_big_images['final_image_id']==random_image) & (train_labels_big_images['bbox_source']=='original')][['x','y','w','h']].values\n    bboxes_merged = train_labels_big_images[(train_labels_big_images['final_image_id']==random_image) & (train_labels_big_images['bbox_source']=='merged')][['x','y','w','h']].values\n    display_instance(PUZZLE_IMAGES_TRAIN_DIR+random_image+'.jpg',[bboxes_original,bboxes_merged],ax[i//2,i%2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Export dataframe as csv","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_labels_big_images.to_csv(PUZZLE_IMAGES_TRAIN_LABELS_FILE)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}