{"cells":[{"metadata":{},"cell_type":"markdown","source":"# kNN features\nI'd like to share some of the features of my current kernel with you. \nI'd be happy to read your comments or suggestions (it's still a py/pandas beginners code)."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nfrom sklearn.decomposition import PCA\nfrom sklearn.metrics import pairwise_distances\nfrom sklearn.neighbors import NearestNeighbors\n\nimport os\nimport warnings  \nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Structures data"},{"metadata":{"trusted":true},"cell_type":"code","source":"structures = pd.read_csv('../input/structures.csv') \n\n# uncomment for debugging \nstructures = structures.head(n=100)\n\nstructures.head(n=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The nn_feateres() function extracts the atom names, distances and coordinates of k nearest neighbors. I'm using 4 neighbors.\nBecause my primary skills are SQL and I'm fairly new to py and pandas I've tried to build the function to use it in a SQL-like \"..nn_features() over(partition by molecule_name)\" kind of manner. I therefor used pd.transform to pass the indexes of a molecule to the function and look up its atoms. If you know of a better or faster way please let me know."},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef nn_features(l):\n    ''' l: indexed pd.Series of a molecule '''\n    \n    # number of nearest neighbors +1\n    k = 4+1\n    \n    # lookup coordinates of atoms in molecule \n    x=np.array(structures.loc[l.index,'x'])\n    y=np.array(structures.loc[l.index,'y'])\n    z=np.array(structures.loc[l.index,'z'])\n    coord = np.append(np.append(x,y),z).reshape((l.size,3),order='F')\n    \n    # NN calculations\n    nbrs = NearestNeighbors(n_neighbors=min(len(coord),k), algorithm='ball_tree').fit(coord)\n    distances, indices = nbrs.kneighbors(coord)\n    \n    \n    if indices.shape != (1,1):\n        # PCA - not relevant for nn, but nice feature anyway\n        pca = PCA(n_components=2)\n        p=pca.fit_transform(coord)\n        \n        # NN id and NN distance\n        atm = np.pad(indices[:,1:l.size],((0,0),(0, max(0, k-l.size))), 'constant', constant_values=(999, 999))\n        dst = np.pad(distances[:,1:l.size], ((0,0),(0,max(0,k-l.size))), 'constant', constant_values=(0, 0))\n        \n        # LookUps for atom name and x,y,z, default value N/A or 0\n        lu = np.append(np.array(structures.loc[l.index,'atom']),np.array('N/A'))\n        lu_x = np.append(np.array(structures.loc[l.index,'x']),np.array(0))\n        lu_y = np.append(np.array(structures.loc[l.index,'y']),np.array(0))\n        lu_z = np.append(np.array(structures.loc[l.index,'z']),np.array(0))\n        \n        # for each nn look up coordinates and atom name \n        nn_x = np.take(lu_x, atm, mode = 'clip') \n        nn_y = np.take(lu_y, atm, mode = 'clip') \n        nn_z = np.take(lu_z, atm, mode = 'clip') \n        atm = np.take(lu, atm, mode = 'clip')\n    else: \n        # in case the molecule contains only 1 atom (e.g. while debugging a small dataset)\n        p = np.ones((1, 2))*(999)\n        atm = np.ones((1, max(0, k-l.size)))*(999) \n        dst = np.ones((1, max(0, k-l.size)))*(999)\n        nn_x = np.ones((1, max(0, k-l.size)))*(999)\n        nn_y = np.ones((1, max(0, k-l.size)))*(999)\n        nn_z = np.ones((1, max(0, k-l.size)))*(999)\n    \n    # put together atom names, distances, coordinates of nnearest neighbors and pca\n    out = np.append(np.append(np.append(np.append(np.append(atm,dst,axis=1),nn_x, axis=1),nn_y, axis=1),nn_z, axis=1) ,p, axis=1)\n    \n    return [i for i in out]\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the hole structures dataset it takes about 12 minutes to calculate the features of 4 nearest neighbors."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nwarnings.filterwarnings('ignore')\n\nstructures['nearestn'] = structures.groupby('molecule_name')['x'].transform(nn_features)\n\nstructures.head(n=10)\n#11mi 12s","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Split the list of nn features. (30 sec)"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# atom name of nn\nstructures['nn_1'] = structures['nearestn'].apply(lambda x: x[0])\nstructures['nn_2'] = structures['nearestn'].apply(lambda x: x[1])\nstructures['nn_3'] = structures['nearestn'].apply(lambda x: x[2])\nstructures['nn_4'] = structures['nearestn'].apply(lambda x: x[3])\n\n# eucledian distances to nn\nstructures['nn_1_dist'] = structures['nearestn'].apply(lambda x: x[4])\nstructures['nn_2_dist'] = structures['nearestn'].apply(lambda x: x[5])\nstructures['nn_3_dist'] = structures['nearestn'].apply(lambda x: x[6])\nstructures['nn_4_dist'] = structures['nearestn'].apply(lambda x: x[7])\n\n# x,y,z distances to nn\nstructures['nn_dx_1'] = structures['nearestn'].apply(lambda x: x[8])  - structures['x']\nstructures['nn_dx_2'] = structures['nearestn'].apply(lambda x: x[9])  - structures['x']\nstructures['nn_dx_3'] = structures['nearestn'].apply(lambda x: x[10])  - structures['x']\nstructures['nn_dx_4'] = structures['nearestn'].apply(lambda x: x[11])  - structures['x']\n\nstructures['nn_dy_1'] = structures['nearestn'].apply(lambda x: x[12])  - structures['y']\nstructures['nn_dy_2'] = structures['nearestn'].apply(lambda x: x[13])  - structures['y']\nstructures['nn_dy_3'] = structures['nearestn'].apply(lambda x: x[14])  - structures['y']\nstructures['nn_dy_4'] = structures['nearestn'].apply(lambda x: x[15])  - structures['y']\n\nstructures['nn_dz_1'] = structures['nearestn'].apply(lambda x: x[16])  - structures['z']\nstructures['nn_dz_2'] = structures['nearestn'].apply(lambda x: x[17])  - structures['z']\nstructures['nn_dz_3'] = structures['nearestn'].apply(lambda x: x[18])  - structures['z']\nstructures['nn_dz_4'] = structures['nearestn'].apply(lambda x: x[19])  - structures['z']\n\n# 2 dim pca\nstructures['pca_x'] = structures['nearestn'].apply(lambda x: x[20])\nstructures['pca_y'] = structures['nearestn'].apply(lambda x: x[21])\n\nstructures = structures.drop(columns='nearestn',axis=0)\nstructures.head(n=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}