{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Background","metadata":{}},{"cell_type":"markdown","source":"## Self-Organizing maps for Anomaly/Fraud Detection\n\n#### There are several different methods for anomaly detection. You can use both supervised and unsupervised machine learning Models. The Advantage of using self organizing maps is that you only need data from the new condition of the machine. For purely supervised methods, you need data from new machines and data from previous failures. But sensor data of machine failure is rare and expensive, so unsupervised models are often cheaper and faster to start with.\n\n
\n\n\n\n#### With SOMs, two approaches could be taken: both approaches involve fitting the SOM onto the training data, which will not include the labels (Unsupervised Learning). \n\n#### Approach 1: Labels can be assigned to each neuron in the resulting 2d-lattice of neurons obtained by weight adjustment in training, by iterating through all training transactions and assigning their class labels to the most similar neuron in the lattice based on weight euclidean similarity. If most training samples assigned to a neuron are labelled fraudulent, this neuron will represent fraudulent behavior. Any new transaction hence found to be similar, in its weight vector, to a neuron that is assigned \"fraudulent/abnormal\" class can be considered fraudulent.\n\n#### Approach 2: Labels are ignored. It is known that neurons themselves form clusters on the map. Adjacent neurons must show similar behaviors. Any neuron, belonging to any cluster, may be evaluated for abnormal behavior as compared to neighbouring neurons. Though this neuron may not show fraudulent behavior as clustered by Approach 1, it must, due to cooperative learning, have similar behaviors to adjacent neurons. Neurons in lattice with the highest interneuron distance to their neighbours can be considered to be fraudulent or abnormal, as they have most resisted cooperative learning.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nfrom numpy.ma.core import ceil\nfrom scipy.spatial import distance #distance calculation\nfrom sklearn.preprocessing import MinMaxScaler #normalization\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score #scoring\nfrom sklearn.metrics import confusion_matrix\nimport matplotlib.pyplot as plt\nfrom matplotlib import animation, colors\nimport pandas as pd","metadata":{"id":"RFEUC3Yx8vIY","execution":{"iopub.status.busy":"2023-05-25T08:56:06.270725Z","iopub.execute_input":"2023-05-25T08:56:06.271181Z","iopub.status.idle":"2023-05-25T08:56:06.277649Z","shell.execute_reply.started":"2023-05-25T08:56:06.271151Z","shell.execute_reply":"2023-05-25T08:56:06.276310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Helper Functions","metadata":{"id":"aQMSI4ygCyUb"}},{"cell_type":"code","source":"# Euclidean distance - used to check for higher dimensional similarity\ndef e_distance(x,y):\n return distance.euclidean(x,y)\n\n# Manhattan distance - used to check for adjacency in 2d neuron grid\ndef m_distance(x,y):\n return distance.cityblock(x,y)\n\n# Best Matching Unit search\ndef winning_neuron(data, t, som, num_rows, num_cols):\n winner = [0,0]\n shortest_distance = np.sqrt(data.shape[1]) # initialise with max distance\n input_data_instance = data[t]\n for row in range(num_rows):\n for col in range(num_cols):\n distance = e_distance(som[row][col], input_data_instance) # som - 2d lattice of neurons\n if distance < shortest_distance: \n shortest_distance = distance\n winner = [row,col] # coordinates of winner neuron in som\n return winner\n\n# Learning rate and neighbourhood range calculation for a given step\ndef decay(step, max_steps,max_learning_rate,max_m_distance):\n coefficient = 1.0 - (np.float64(step)/max_steps)\n learning_rate = coefficient*max_learning_rate # as num_steps for an iteration increases, learning rate decreases\n neighbourhood_range = ceil(coefficient * max_m_distance) # similarly, neighbourhood range also decreases\n return learning_rate, neighbourhood_range","metadata":{"id":"o1NgYEkyCxrr","execution":{"iopub.status.busy":"2023-05-25T08:56:06.279965Z","iopub.execute_input":"2023-05-25T08:56:06.280291Z","iopub.status.idle":"2023-05-25T08:56:06.292714Z","shell.execute_reply.started":"2023-05-25T08:56:06.280263Z","shell.execute_reply":"2023-05-25T08:56:06.291937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Preprocessing","metadata":{"id":"3PGITDnZBw0c"}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/fraud-detection-competition/train_transaction.csv')\ndf.isnull().sum() ","metadata":{"id":"fCWgiMvD9gxw","outputId":"4f3de8bb-2c8e-4a44-9799-1223a94ae7c6","execution":{"iopub.status.busy":"2023-05-25T08:56:06.294276Z","iopub.execute_input":"2023-05-25T08:56:06.294808Z","iopub.status.idle":"2023-05-25T08:56:17.546719Z","shell.execute_reply.started":"2023-05-25T08:56:06.294782Z","shell.execute_reply":"2023-05-25T08:56:17.545575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2023-05-25T08:56:17.548179Z","iopub.execute_input":"2023-05-25T08:56:17.548498Z","iopub.status.idle":"2023-05-25T08:56:17.573063Z","shell.execute_reply.started":"2023-05-25T08:56:17.548468Z","shell.execute_reply":"2023-05-25T08:56:17.571840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df.fillna(df.mean())\n\ndf = df.drop(['R_emaildomain'], axis=1) # RAM Constraint\ndf = df.drop(df[~ (df['card4'] == 'visa') & ~ (df['card4'] == 'mastercard')].index)\ndf = df.drop(df[~ (df['P_emaildomain'] == 'gmail.com') & ~ (df['P_emaildomain'] == 'yahoo.com')].index)\n\ndata_x = df.drop(['isFraud'], axis =1)\ndata_y = df.loc[:,'isFraud']","metadata":{"id":"XR61bKSaBLIp","outputId":"97c88e10-8dc0-4ff8-c514-0f5e880e6e58","execution":{"iopub.status.busy":"2023-05-25T08:56:17.575941Z","iopub.execute_input":"2023-05-25T08:56:17.576510Z","iopub.status.idle":"2023-05-25T08:56:52.318218Z","shell.execute_reply.started":"2023-05-25T08:56:17.576479Z","shell.execute_reply":"2023-05-25T08:56:52.317071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_y.hist()","metadata":{"execution":{"iopub.status.busy":"2023-05-25T08:56:52.319319Z","iopub.execute_input":"2023-05-25T08:56:52.319647Z","iopub.status.idle":"2023-05-25T08:56:52.496103Z","shell.execute_reply.started":"2023-05-25T08:56:52.319625Z","shell.execute_reply":"2023-05-25T08:56:52.495268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Label Encoding","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nle = LabelEncoder()\n\nle.fit([\"T\",\"F\",'n'])\n\nfor i,k in enumerate(['M1','M2','M3','M5','M6','M7','M8','M9']):\n data_x[k] = le.transform(data_x[k])","metadata":{"id":"dezMYnPeGxpq","execution":{"iopub.status.busy":"2023-05-25T08:56:52.497460Z","iopub.execute_input":"2023-05-25T08:56:52.498005Z","iopub.status.idle":"2023-05-25T08:56:53.349680Z","shell.execute_reply.started":"2023-05-25T08:56:52.497971Z","shell.execute_reply":"2023-05-25T08:56:53.348784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### One Hot Encoding","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import OneHotEncoder\nfrom sklearn.compose import ColumnTransformer\n\n# queriedColumns = ['ProductCD', 'card4', 'card6', 'P_emaildomain', 'R_emaildomain', 'M4'] # RAM Constraint\nqueriedColumns = ['ProductCD', 'card4', 'card6', 'P_emaildomain', 'M4'] # RAM Constraint\n\nindexes = []\nfor i, k in enumerate(queriedColumns): \n indexes.append(data_x.columns.get_loc(k))\n\nct = ColumnTransformer(transformers=[('encoder', OneHotEncoder(sparse_output=False) , indexes)], remainder = 'passthrough')\ndata_x = np.array(ct.fit_transform(data_x)).astype('float32')","metadata":{"id":"-ELoX3ekF7Wz","execution":{"iopub.status.busy":"2023-05-25T08:56:53.350712Z","iopub.execute_input":"2023-05-25T08:56:53.350951Z","iopub.status.idle":"2023-05-25T08:56:54.168782Z","shell.execute_reply.started":"2023-05-25T08:56:53.350930Z","shell.execute_reply":"2023-05-25T08:56:54.167576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Note: train_y (labels) is used only for label collection in Approach 1: Correct labels assigned to clusters. It is ignored in Approach 2: Unsupervised Clustering.","metadata":{}},{"cell_type":"code","source":"train_x, val_x, train_y, val_y = train_test_split(data_x, data_y, test_size=0.5, random_state=42)\nprint(f'Shapes: {train_x.shape} | {train_y.shape} | {val_x.shape} | {val_y.shape}') # check the shap\n\nsc = MinMaxScaler(feature_range=(0,1))\ntrain_x_scaled = sc.fit_transform(train_x) # normalization\nval_x_scaled = sc.transform(val_x)","metadata":{"id":"zwo3LYrZKGpb","outputId":"a71c30ec-3eda-4574-8f8e-e8d8f3a16c3b","execution":{"iopub.status.busy":"2023-05-25T08:57:42.002687Z","iopub.execute_input":"2023-05-25T08:57:42.003032Z","iopub.status.idle":"2023-05-25T08:57:42.331510Z","shell.execute_reply.started":"2023-05-25T08:57:42.003005Z","shell.execute_reply":"2023-05-25T08:57:42.330077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training","metadata":{"id":"uZEITpUQE9Qg"}},{"cell_type":"markdown","source":"### Hyperparameters","metadata":{"id":"CH9r5OrEEzkZ"}},{"cell_type":"code","source":"num_rows = 25 # ~ sqrt(sqrt(train_x.shape[0]))\nnum_cols = 25 # ~ sqrt(sqrt(train_x.shape[0]))\nmax_m_distance = 15\nmax_learning_rate = 0.5\nmax_steps = int(300000) # ~ 500 * num_rows * num_cols","metadata":{"id":"kYZwGPOVEzG-","execution":{"iopub.status.busy":"2023-05-25T08:57:47.466193Z","iopub.execute_input":"2023-05-25T08:57:47.466555Z","iopub.status.idle":"2023-05-25T08:57:47.472314Z","shell.execute_reply.started":"2023-05-25T08:57:47.466505Z","shell.execute_reply":"2023-05-25T08:57:47.470933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Additional Helper Function:\n\n#### Adjusting Neighbours' Weights\n\n\n$ W(t + 1) = W(t) + \\Theta(t) * \\lambda(t) * (V(t) - W(t)) $\n\n\n$ \\Theta = exp(-\\frac{dist^{2}}{2 r^{2} (t)}) $ \n$ \\Theta $ depends on the time and the distance of the neuron to the BMU &arr; Ensures that the weights are adjusted particularly strongly for neurons close to the BMU\n\n##### r = neighbourhood range / radius\n##### t = time / steps \n##### dist = distance of given neighbour neuron from BMU \n\n$ \\lambda $ is the learning rate and can be freely chosen","metadata":{"id":"da951o-rYwjp"}},{"cell_type":"code","source":"import math\n\ndef neighbourFunction(dist, r, t):\n funcOutput = math.exp(0-(dist*dist)/(2*r*r*t))\n return funcOutput ","metadata":{"id":"JtFJPDiTZGl3","execution":{"iopub.status.busy":"2023-05-25T08:57:47.474447Z","iopub.execute_input":"2023-05-25T08:57:47.474987Z","iopub.status.idle":"2023-05-25T08:57:47.488718Z","shell.execute_reply.started":"2023-05-25T08:57:47.474956Z","shell.execute_reply":"2023-05-25T08:57:47.487747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_dims = train_x_scaled.shape[1] # numnber of dimensions in the input data\n\nnp.random.seed(2)\n\nsom = np.random.random_sample(size=(num_rows, num_cols, num_dims)) # neuron 2d lattice - map construction\n\nfor step in range(max_steps):\n if (step+1) % 10000 == 0:\n print(\"Iteration: \", step+1) # print out the current iteration for every 10k\n\n learning_rate, neighbourhood_range = decay(step, max_steps,max_learning_rate,max_m_distance)\n\n t = np.random.randint(0,high=train_x_scaled.shape[0]) # random index of training data\n winner = winning_neuron(train_x_scaled, t, som, num_rows, num_cols)\n\n for row in range(num_rows):\n for col in range(num_cols):\n if m_distance([row,col],winner) <= neighbourhood_range:\n if(row==winner[0] and col == winner[1]):\n som[row][col] += learning_rate*(train_x_scaled[t]-som[row][col]) # update BMU's weights\n else:\n neighbourFunc = neighbourFunction(m_distance([row,col],winner), neighbourhood_range, (step+1))\n som[row][col] += neighbourFunc * learning_rate*(train_x_scaled[t]-som[row][col]) # update neighbours' weights","metadata":{"id":"2sK37qXdE-tK","outputId":"6424617a-ccac-4399-9dae-08564892a7b2","execution":{"iopub.status.busy":"2023-05-25T08:57:47.490147Z","iopub.execute_input":"2023-05-25T08:57:47.490795Z","iopub.status.idle":"2023-05-25T09:43:05.142221Z","shell.execute_reply.started":"2023-05-25T08:57:47.490762Z","shell.execute_reply":"2023-05-25T09:43:05.140821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Approach 1: Correct Labels Assigned to Clusters. \n\n#### Clusters are already formed in the self organizing map. Labels can be assigned to each neuron in the map, and inputs can be assigned to each neuron to determine their class (Fraud/Not Fraud)","metadata":{}},{"cell_type":"markdown","source":"> ## Collecting Labels","metadata":{}},{"cell_type":"code","source":"label_data = train_y.values\nmap = np.empty(shape=(num_rows, num_cols), dtype=object)\n\nfor row in range(num_rows):\n for col in range(num_cols):\n map[row][col] = [] # em","metadata":{"execution":{"iopub.status.busy":"2023-05-25T09:43:05.144964Z","iopub.execute_input":"2023-05-25T09:43:05.145310Z","iopub.status.idle":"2023-05-25T09:43:05.151820Z","shell.execute_reply.started":"2023-05-25T09:43:05.145274Z","shell.execute_reply":"2023-05-25T09:43:05.150713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for t in range(train_x_scaled.shape[0]):\n if (t+1) % 10000 == 0:\n print(\"sample data: \", t+1)\n winner = winning_neuron(train_x_scaled, t, som, num_rows, num_cols)\n map[winner[0]][winner[1]].append(label_data[t])","metadata":{"execution":{"iopub.status.busy":"2023-05-25T09:58:47.759308Z","iopub.execute_input":"2023-05-25T09:58:47.759675Z","iopub.status.idle":"2023-05-25T10:07:33.424210Z","shell.execute_reply.started":"2023-05-25T09:58:47.759648Z","shell.execute_reply":"2023-05-25T10:07:33.422426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import statistics\nimport matplotlib.colors as mcolors\nimport matplotlib.cm as cm\n\nlabel_map = np.zeros(shape=(num_rows, num_cols),dtype=np.float32)\n\nfor row in range(num_rows):\n for col in range(num_cols):\n label_list = map[row][col]\n if len(label_list)==0:\n label = 0 # Identified Classes Included\n else:\n label = label_list.count(1)/len(label_list)\n label_map[row][col] = label","metadata":{"execution":{"iopub.status.busy":"2023-05-25T10:07:33.426125Z","iopub.execute_input":"2023-05-25T10:07:33.426469Z","iopub.status.idle":"2023-05-25T10:07:33.468733Z","shell.execute_reply.started":"2023-05-25T10:07:33.426438Z","shell.execute_reply":"2023-05-25T10:07:33.467598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"title = ('Class-wise cluster' )\n\nfrom PIL import Image\nimport requests\nfrom io import BytesIO\n\ndef truncate_colormap(cmap, minval=0.0, maxval=1.0, n=100):\n new_cmap = colors.LinearSegmentedColormap.from_list(\n 'trunc({n},{a:.2f},{b:.2f})'.format(n=cmap.name, a=minval, b=maxval),\n cmap(np.linspace(minval, maxval, n)))\n return new_cmap\n\ncmap = truncate_colormap(plt.get_cmap('jet'), 0.45, 1.0)\n\nplt.title(title)\n\nplt.imshow(label_map, cmap=cmap)\n\nclb=plt.colorbar()\nclb.ax.tick_params(labelsize=8) \nclb.set_label('(1 : Fraud) (0 : Not fraud / Unidentified Class)',loc='bottom', rotation=0)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-25T10:07:33.469890Z","iopub.execute_input":"2023-05-25T10:07:33.470175Z","iopub.status.idle":"2023-05-25T10:07:33.688689Z","shell.execute_reply.started":"2023-05-25T10:07:33.470152Z","shell.execute_reply":"2023-05-25T10:07:33.687312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Approach 2: Unsupervised Clustering. \n\n#### Clusters are already formed in the self organizing map. However, we can ignore train_y, and consider the scenario where we don't have access to data labels (no information of whether a given transaction is fraud/not-fraud).\n\n#### The Self Organizing Map is an unsupervised clustering technique. Interneuron distance of every neuron to neighbouring neurons may be computed to search for members of a cluster that show the most abnormal behavior / difference in weights. Any data instance (transaction) assigned to a neuron/BMU with the highest interneuron distance may be considered to be fraudulent.","metadata":{}},{"cell_type":"code","source":"def fast_norm(x):\n \"\"\"Returns norm-2 of a 1-D numpy array.\n \"\"\"\n return math.sqrt(np.dot(x, x.T))\n\ndef distance_map(map):\n \"\"\"Returns the distance map of the weights.\n Each cell is the normalised sum of\n the distances between a neuron and its neighbours. Note that this\n method uses the euclidean distance. *Partly copied from JustGlowing/Minisom Github project* \"\"\"\n \n ii = [[0, -1, -1, -1, 0, 1, 1, 1]]*2\n jj = [[-1, -1, 0, 1, 1, 1, 0, -1]]*2\n \n um = np.nan * np.zeros((som.shape[0],\n som.shape[1],\n 8)) \n \n for x in range(som.shape[0]):\n for y in range(som.shape[1]):\n neuron2 = som[x, y]\n e = y % 2 == 0 # only used on hexagonal topology\n for k, (i, j) in enumerate(zip(ii[e], jj[e])):\n if (x+i >= 0 and x+i < som.shape[0] and\n y+j >= 0 and y+j < som.shape[1]):\n neuron1 = som[x+i, y+j]\n um[x, y, k] = fast_norm(neuron2-neuron1)\n \n um = np.nansum(um, axis=2)\n \n return um/um.max()","metadata":{"execution":{"iopub.status.busy":"2023-05-25T10:07:33.691908Z","iopub.execute_input":"2023-05-25T10:07:33.692304Z","iopub.status.idle":"2023-05-25T10:07:33.702486Z","shell.execute_reply.started":"2023-05-25T10:07:33.692259Z","shell.execute_reply":"2023-05-25T10:07:33.701062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"title = ('Inter-Neuron Distance (IND) Map' )\n\ndist_map = distance_map(som)\nplt.title(title)\n\ncmap = cm.get_cmap('Greys_r')\n\nplt.imshow(dist_map, cmap=cmap)\n\nclb=plt.colorbar()\nclb.ax.tick_params(labelsize=8) \nclb.set_label('(1 : Greater IND) (0 : Lesser IND)', rotation=0, loc = 'bottom')\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-25T10:07:33.704190Z","iopub.execute_input":"2023-05-25T10:07:33.704683Z","iopub.status.idle":"2023-05-25T10:07:33.941283Z","shell.execute_reply.started":"2023-05-25T10:07:33.704649Z","shell.execute_reply":"2023-05-25T10:07:33.939815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Testing both approaches","metadata":{}},{"cell_type":"markdown","source":"### Approach 1:","metadata":{}},{"cell_type":"markdown","source":"#### Every neuron has been assigned mean class value of range 0-1. Threshold for fraudulent will be considered > 0.75","metadata":{}},{"cell_type":"code","source":"true_labels = val_y\npredicted_labels = []\n\nfor i, k in enumerate(val_x_scaled):\n winner = winning_neuron(val_x_scaled, i, som, 16, 16)\n winner_class = label_map[winner[0]][winner[1]]\n predicted_labels.append(1 if winner_class > 0.5 else 0)","metadata":{"execution":{"iopub.status.busy":"2023-05-25T10:07:33.943513Z","iopub.execute_input":"2023-05-25T10:07:33.943917Z","iopub.status.idle":"2023-05-25T10:11:01.601824Z","shell.execute_reply.started":"2023-05-25T10:07:33.943883Z","shell.execute_reply":"2023-05-25T10:11:01.600680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\n\nconf_matrix = confusion_matrix(y_true=true_labels, y_pred=predicted_labels)\n\nfig, ax = plt.subplots(figsize=(7.5, 7.5))\nax.matshow(conf_matrix, cmap=cm.Blues, alpha=0.3)\n\nfor i in range(conf_matrix.shape[0]):\n for j in range(conf_matrix.shape[1]):\n ax.text(x=j, y=i,s=conf_matrix[i, j], va='center', ha='center', size='xx-large')\n \nplt.xlabel('Predictions', fontsize=18)\nplt.ylabel('Actuals', fontsize=18)\nplt.title('Confusion Matrix', fontsize=18)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-25T10:11:01.603267Z","iopub.execute_input":"2023-05-25T10:11:01.603655Z","iopub.status.idle":"2023-05-25T10:11:01.802743Z","shell.execute_reply.started":"2023-05-25T10:11:01.603625Z","shell.execute_reply":"2023-05-25T10:11:01.801630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import accuracy_score\n\nprint(f'Accuracy Score: {round(accuracy_score(y_true=true_labels, y_pred=predicted_labels),2)*100}%')","metadata":{"execution":{"iopub.status.busy":"2023-05-25T10:11:01.804133Z","iopub.execute_input":"2023-05-25T10:11:01.804981Z","iopub.status.idle":"2023-05-25T10:11:01.837512Z","shell.execute_reply.started":"2023-05-25T10:11:01.804931Z","shell.execute_reply":"2023-05-25T10:11:01.836590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Approach 2:","metadata":{}},{"cell_type":"markdown","source":"#### Every unit has been assigned normalized IND of range 0-1. Threshold for fraudulent will be considered > 0.75","metadata":{}},{"cell_type":"code","source":"true_labels = val_y\npredicted_labels = []\n\nfor i, k in enumerate(val_x_scaled):\n winner = winning_neuron(val_x_scaled, i, som, 16, 16)\n winner_class = dist_map[winner[0]][winner[1]]\n predicted_labels.append(1 if winner_class > 0.65 else 0)","metadata":{"execution":{"iopub.status.busy":"2023-05-25T10:27:13.327670Z","iopub.execute_input":"2023-05-25T10:27:13.328051Z","iopub.status.idle":"2023-05-25T10:30:41.605261Z","shell.execute_reply.started":"2023-05-25T10:27:13.328022Z","shell.execute_reply":"2023-05-25T10:30:41.604266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\n\nconf_matrix = confusion_matrix(y_true=true_labels, y_pred=predicted_labels)\n\nfig, ax = plt.subplots(figsize=(7.5, 7.5))\nax.matshow(conf_matrix, cmap=cm.Blues, alpha=0.3)\n\nfor i in range(conf_matrix.shape[0]):\n for j in range(conf_matrix.shape[1]):\n ax.text(x=j, y=i,s=conf_matrix[i, j], va='center', ha='center', size='xx-large')\n \nplt.xlabel('Predictions', fontsize=18)\nplt.ylabel('Actuals', fontsize=18)\nplt.title('Confusion Matrix', fontsize=18)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-05-25T10:30:41.606785Z","iopub.execute_input":"2023-05-25T10:30:41.607085Z","iopub.status.idle":"2023-05-25T10:30:41.845100Z","shell.execute_reply.started":"2023-05-25T10:30:41.607060Z","shell.execute_reply":"2023-05-25T10:30:41.843941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import accuracy_score\n\nprint(f'Accuracy Score: {round(accuracy_score(y_true=true_labels, y_pred=predicted_labels),2)*100}%')","metadata":{"execution":{"iopub.status.busy":"2023-05-25T10:30:41.846796Z","iopub.execute_input":"2023-05-25T10:30:41.847137Z","iopub.status.idle":"2023-05-25T10:30:41.889840Z","shell.execute_reply.started":"2023-05-25T10:30:41.847109Z","shell.execute_reply":"2023-05-25T10:30:41.888619Z"},"trusted":true},"execution_count":null,"outputs":[]}]}