{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":34426,"databundleVersionId":3218632,"sourceType":"competition"},{"sourceId":1111676,"sourceType":"datasetVersion","datasetId":623289}],"dockerImageVersionId":30747,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true},"colab":{"provenance":[]}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n\"\"\"\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\"\"\"\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\"\n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","id":"a5NhpRVA7lYZ","outputId":"963430dc-81d2-423a-a07f-ed40b21f3f37","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Cat and Dog Classification using ResNet-KAN Method\n\n### Description of Kolmogorov-Arnold Network (KAN)\n\nThe paper: https://arxiv.org/abs/2404.19756\n\nThe Kolmogorov-Arnold Network (KAN) is a type of artificial neural network based on the Kolmogorov-Arnold Representation Theorem. This network has distinct characteristics in its structure and operation, differing from Multi-Layer Perceptrons (MLPs). KAN features the following:\n\n1. **Kolmogorov-Arnold Representation Theorem**:\n \n ![image.png](attachment:4d97a307-2dcc-4058-ac94-e4717c8f2965.png)\n \n2. **Learnable Activations on Edges**:\n - Unlike MLPs, which have fixed activations on nodes, KAN has learnable activations on edges.\n\n3. **Sum Operation on Nodes**:\n - KAN combines the results of several sum operations performed on nodes.\n\n### Comparison Between KAN and MLP\n\n#### Similarities\n- **Both models** can be used for universal approximation of continuous functions.\n- **Both models** consist of layers that can be combined to form deep networks.\n\n#### Differences\n1. **Activation Structure**:\n - **MLP**: Uses fixed activation functions on nodes.\n - **KAN**: Uses learnable activation functions on edges.\n\n2. **Learning and Operation Locations**:\n - **MLP**: Learnable weights are on the edges between layers, with fixed activation functions.\n - **KAN**: The activation functions themselves are on the edges and learnable, with sum operations on nodes.\n\n3. **Mathematical Formulas**:\n\n ![image.png](attachment:3cd32c62-f6ae-4f5a-b2de-ae644c100045.png)\n\n4. **Model Visualization**:\n - **MLP Shallow**: Depicted as a network with a single layer of fixed activations on nodes and weights on edges.\n - **KAN Shallow**: Depicted as a network with learnable functions on edges and sum operations on nodes.\n - **MLP Deep**: A network with multiple layers, each layer having weights and fixed activation functions.\n - **KAN Deep**: A network with multiple layers, but the learnable functions are on the edges.\n\n### Conclusion\nKAN and MLP are two different approaches to building neural networks with universal approximation capabilities. KAN offers higher flexibility in terms of having learnable activation functions on edges compared to MLPs, which use fixed activation functions on nodes. This makes KAN more adaptive in some cases, although with additional complexity in learning.","metadata":{"id":"oj67aH2b7lYe"}},{"cell_type":"markdown","source":"\n\n

\n \n

","metadata":{"id":"h-uK_bXc7lYv"}},{"cell_type":"markdown","source":"# Step 1: Load and Prepare the Flickr8k Dataset\n","metadata":{"id":"KsizsGZX-PlJ"}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport pandas as pd\nimport string\nfrom tqdm import tqdm\nimport glob\nfrom textwrap import wrap\nimport matplotlib.pyplot as plt\nfrom tensorflow.keras.preprocessing.text import Tokenizer\nfrom tensorflow.keras.preprocessing.sequence import pad_sequences\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras.preprocessing.image import load_img, img_to_array\nfrom tensorflow.keras.applications.vgg16 import VGG16, preprocess_input\nfrom tensorflow.keras.models import Model, load_model\nfrom tensorflow.keras.layers import Input, Dense, LSTM, Embedding, Dropout, add, Activation\nfrom nltk.translate.bleu_score import corpus_bleu\nfrom pickle import dump, load","metadata":{"id":"b9Y_rxAR7lYx","execution":{"iopub.status.busy":"2024-07-19T14:05:51.532699Z","iopub.execute_input":"2024-07-19T14:05:51.533061Z","iopub.status.idle":"2024-07-19T14:06:05.64352Z","shell.execute_reply.started":"2024-07-19T14:05:51.533031Z","shell.execute_reply":"2024-07-19T14:06:05.642707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load and prepare the Flickr8k dataset\n","metadata":{"id":"cLpl0H_N-S4q"}},{"cell_type":"code","source":"dataset_text = '/kaggle/input/flickr8k/captions.txt'\ndataset_images = '/kaggle/input/flickr8k/Images'","metadata":{"id":"VpyWxllC759X","execution":{"iopub.status.busy":"2024-07-19T14:05:46.395172Z","iopub.execute_input":"2024-07-19T14:05:46.395559Z","iopub.status.idle":"2024-07-19T14:05:46.400209Z","shell.execute_reply.started":"2024-07-19T14:05:46.395527Z","shell.execute_reply":"2024-07-19T14:05:46.399288Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load the dataset\ndata = pd.read_csv(dataset_text)\nprint(data.head())","metadata":{"id":"FmadERjICMgh","outputId":"5a0a4ec2-d307-4fba-8932-00be3ac6f614","execution":{"iopub.status.busy":"2024-07-19T14:05:48.317474Z","iopub.execute_input":"2024-07-19T14:05:48.318289Z","iopub.status.idle":"2024-07-19T14:05:48.343266Z","shell.execute_reply.started":"2024-07-19T14:05:48.318255Z","shell.execute_reply":"2024-07-19T14:05:48.342192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def text_preprocessing(data):\n data['caption'] = data['caption'].apply(lambda x: x.lower())\n data['caption'] = data['caption'].str.replace(\"[^A-Za-z]\", \"\")\n data['caption'] = data['caption'].str.replace(\"\\s+\", \" \")\n data['caption'] = data['caption'].apply(lambda x: \" \".join([word for word in x.split() if len(word) > 1]))\n data['caption'] = \"startseq \" + data['caption'] + \" endseq\"\n return data","metadata":{"execution":{"iopub.status.busy":"2024-07-19T13:49:56.252588Z","iopub.execute_input":"2024-07-19T13:49:56.252957Z","iopub.status.idle":"2024-07-19T13:49:56.2622Z","shell.execute_reply.started":"2024-07-19T13:49:56.252928Z","shell.execute_reply":"2024-07-19T13:49:56.261227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = text_preprocessing(data)\ncaptions = data['caption'].tolist()\n\n# Tokenizer\ntokenizer = Tokenizer()\ntokenizer.fit_on_texts(captions)\nvocab_size = len(tokenizer.word_index) + 1\nmax_length = max(len(caption.split()) for caption in captions)\n\n# Split dataset\nimages = data['image'].unique().tolist()\nnimages = len(images)\nsplit_index = round(0.85 * nimages)\ntrain_images = images[:split_index]\nval_images = images[split_index:]\n\ntrain = data[data['image'].isin(train_images)]\ntest = data[data['image'].isin(val_images)]\n\ntrain.reset_index(inplace=True, drop=True)\ntest.reset_index(inplace=True, drop=True)\n\n# Function to read and preprocess images\ndef readImage(path, img_size=224):\n if not os.path.exists(path):\n print(f\"Image path {path} does not exist.\")\n return None\n img = load_img(path, color_mode='rgb', target_size=(img_size, img_size))\n img = img_to_array(img)\n img = img / 255.0\n return img\n\n# Function to display images with captions\ndef display_images(temp_df, num_images=15):\n temp_df = temp_df.reset_index(drop=True)\n plt.figure(figsize=(20, 20))\n n = 0\n for i in range(min(num_images, len(temp_df))):\n n += 1\n plt.subplot(5, 5, n)\n plt.subplots_adjust(hspace=0.7, wspace=0.3)\n image_path = f\"{dataset_images}/{temp_df.image[i]}\"\n image = readImage(image_path)\n if image is not None:\n plt.imshow(image)\n plt.title(\"\\n\".join(wrap(temp_df.caption[i], 20)))\n plt.axis(\"off\")\n else:\n plt.title(\"Image not found\")\n plt.show()\n\n# Display a few images from the training set\ndisplay_images(train)","metadata":{"id":"Ib7Ms5jDCQLh","execution":{"iopub.status.busy":"2024-07-19T14:06:47.952219Z","iopub.execute_input":"2024-07-19T14:06:47.95352Z","iopub.status.idle":"2024-07-19T14:06:48.448945Z","shell.execute_reply.started":"2024-07-19T14:06:47.953484Z","shell.execute_reply":"2024-07-19T14:06:48.447732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocess captions\n\n","metadata":{"id":"nernuSrn7lYy"}},{"cell_type":"markdown","source":"### 1. Definition of KANLinear\n - This definition follows the provided source and is used in the CNN-KAN model.\n\n

\n \n

\n\nKAN vs MLP Complexity (please correct if there are any errors):\n\nBig O notation measures the time or space complexity of an algorithm in terms of its input. To compare the memory and performance requirements between the Kolmogorov-Arnold Network (KAN) and Multi-Layer Perceptron (MLP), we can look at the components that affect memory usage and computational time for both algorithms.\n\n### Multi-Layer Perceptron (MLP)\n\nMLP usually consists of several layers of neurons, where each neuron in one layer is connected to every neuron in the next layer.\n\n- **Time Complexity (Training):** O(N * M * L * E)\n - N: Number of samples in the dataset\n - M: Number of neurons per layer\n - L: Number of layers\n - E: Number of epochs\n\n- **Space Complexity:** O(M * L + N * F)\n - M: Number of neurons per layer\n - L: Number of layers\n - N: Number of samples in the dataset\n - F: Dimension of input features\n\n### Kolmogorov-Arnold Network (KAN)\n\nKAN uses learnable activation functions at each edge and summation operations at nodes, adding complexity to the model.\n\n- **Time Complexity (Training):** O(N * (M^2) * L * E)\n - N: Number of samples in the dataset\n - M: Number of neurons per layer (since the activation functions are learnable at each edge, there is additional complexity here)\n - L: Number of layers\n - E: Number of epochs\n\n- **Space Complexity:** O(M^2 * L + N * F)\n - M: Number of neurons per layer (with learnable activation functions at each edge, memory requirements increase)\n - L: Number of layers\n - N: Number of samples in the dataset\n - F: Dimension of input features\n\n### Conclusion\n\nFrom the analysis above, it can be concluded that KAN has higher complexity compared to MLP in terms of both time and space.\n\n1. **Time Complexity:**\n - MLP: O(N * M * L * E)\n - KAN: O(N * M^2 * L * E)\n\n2. **Space Complexity:**\n - MLP: O(M * L + N * F)\n - KAN: O(M^2 * L + N * F)\n\nIt can be seen that KAN requires more memory (RAM) during training because it has learnable activation functions at each edge, which significantly increases the number of parameters that need to be stored and processed.","metadata":{"id":"vbIV5bZz7lY4"}},{"cell_type":"markdown","source":"# Step 2: Define the KAN Layers\n\n\n[1] Reference: KAN: https://github.com/Blealtan/efficient-kan/blob/master/src/efficient_kan/kan.py","metadata":{"id":"cOhfLRcM7lY5"}},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\n\nclass SplineActivation(nn.Module):\n def __init__(self, knots, coefficients, interval):\n super(SplineActivation, self).__init__()\n self.knots = knots\n self.coefficients = coefficients\n self.interval = interval\n\n def forward(self, x):\n result = torch.zeros_like(x)\n for i in range(len(self.knots) - 1):\n mask = (x >= self.knots[i]) & (x < self.knots[i + 1])\n result += mask.float() * (self.coefficients[i] * (x - self.knots[i]))\n mask = (x >= self.knots[-1])\n result += mask.float() * (self.coefficients[-1] * (x - self.knots[-1]))\n return result\n\nclass HybridActivation(nn.Module):\n def __init__(self, linear_weight, spline_knots, spline_coefficients, spline_interval):\n super(HybridActivation, self).__init__()\n self.linear_weight = linear_weight\n self.spline = SplineActivation(spline_knots, spline_coefficients, spline_interval)\n\n def forward(self, x):\n return self.linear_weight * x + self.spline(x)","metadata":{"id":"CWcQMNuL7lY5","execution":{"iopub.status.busy":"2024-07-19T13:50:30.44182Z","iopub.execute_input":"2024-07-19T13:50:30.442537Z","iopub.status.idle":"2024-07-19T13:50:34.570328Z","shell.execute_reply.started":"2024-07-19T13:50:30.442506Z","shell.execute_reply":"2024-07-19T13:50:34.569349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Model CNNKAN\n\nReference: CNN-KAN: https://github.com/jakariaemon/CNN-KAN\n\n # Basics CNN model for Cat and Dog Classifications with KANLinear\n \n class CNNKAN(nn.Module):\n def __init__(self):\n super(CNNKAN, self).__init__()\n self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1) \n self.pool1 = nn.MaxPool2d(2)\n self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)\n self.pool2 = nn.MaxPool2d(2)\n self.kan1 = KANLinear(64 * 8 * 8, 256) \n self.kan2 = KANLinear(256, 100)\n\n def forward(self, x):\n x = F.selu(self.conv1(x))\n x = self.pool1(x)\n x = F.selu(self.conv2(x))\n x = self.pool2(x)\n x = x.view(x.size(0), -1)\n x = self.kan1(x)\n x = self.kan2(x)\n return x","metadata":{"id":"sX1O7Ry_7lY7"}},{"cell_type":"markdown","source":"# Step 3: Extract Features using VGG16\n","metadata":{"id":"ZJ6MkJZU-hsD"}},{"cell_type":"markdown","source":"### Model VGG16-KAN\n\nReference KANVGG16 architectures from: https://www.linkedin.com/posts/damienbenveniste_this-is-a-pure-kolmogorovarnold-kan-convolutional-activity-7193403017008680960-FOHW?utm_source=share&utm_medium=member_android","metadata":{"id":"7EFscVxM7lY7"}},{"cell_type":"code","source":"# Step 3: Extract Features using VGG16\ndef extract_features(directory):\n model = VGG16()\n model = Model(inputs=model.inputs, outputs=model.layers[-2].output)\n print(model.summary())\n features = {}\n for img in tqdm(glob.glob(directory + '/*.jpg')):\n img_id = img.split('/')[-1].split('.')[0]\n image = load_img(img, target_size=(224, 224))\n image = img_to_array(image)\n image = np.expand_dims(image, axis=0)\n image = preprocess_input(image)\n feature = model.predict(image, verbose=0)\n features[img_id] = feature\n return features\n\n# Extract features and save to file\nfeatures = extract_features(dataset_images)\ndump(features, open('features.pkl', 'wb'))\n","metadata":{"id":"pMK5Q4eA8Sni","outputId":"871d4511-749b-4ed6-96bc-83d0e149b4cc","execution":{"iopub.status.busy":"2024-07-19T13:50:42.252376Z","iopub.execute_input":"2024-07-19T13:50:42.253307Z","iopub.status.idle":"2024-07-19T14:01:15.214547Z","shell.execute_reply.started":"2024-07-19T13:50:42.253272Z","shell.execute_reply":"2024-07-19T14:01:15.213729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 4: Tokenizer and Data Preparation","metadata":{"id":"xCaYM1gm7lY9"}},{"cell_type":"code","source":"# Prepare tokenizer\ndef create_tokenizer(captions):\n tokenizer = Tokenizer()\n tokenizer.fit_on_texts(captions)\n return tokenizer\n\ntokenizer = create_tokenizer(captions)\ndump(tokenizer, open('tokenizer.pkl', 'wb'))\n\nvocab_size = len(tokenizer.word_index) + 1\nprint('Vocabulary Size: %d' % vocab_size)\n\n# Create sequences of images, input sequences, and output words for training\ndef create_sequences(tokenizer, max_length, captions, photos):\n X1, X2, y = [], [], []\n for i, caption in enumerate(captions):\n seq = tokenizer.texts_to_sequences([caption])[0]\n for j in range(1, len(seq)):\n in_seq, out_seq = seq[:j], seq[j]\n in_seq = pad_sequences([in_seq], maxlen=max_length)[0]\n out_seq = to_categorical([out_seq], num_classes=vocab_size)[0]\n X1.append(photos[i])\n X2.append(in_seq)\n y.append(out_seq)\n return np.array(X1), np.array(X2), np.array(y)\n\n# Determine the maximum sequence length\nmax_length = max(len(caption.split()) for caption in captions)\nprint('Description Length: %d' % max_length)\n\n\n# Create the data generator\ndef data_generator(captions, photos, tokenizer, max_length):\n while True:\n for key, desc_list in captions.items():\n for desc in desc_list:\n seq = tokenizer.texts_to_sequences([desc])[0]\n for i in range(1, len(seq)):\n in_seq, out_seq = seq[:i], seq[i]\n in_seq = pad_sequences([in_seq], maxlen=max_length)[0]\n out_seq = to_categorical([out_seq], num_classes=vocab_size)[0]\n yield [[photos[key][0], in_seq], out_seq]","metadata":{"id":"H7_JnAkF7lY9","execution":{"iopub.status.busy":"2024-07-19T14:01:24.258633Z","iopub.execute_input":"2024-07-19T14:01:24.259336Z","iopub.status.idle":"2024-07-19T14:01:25.025509Z","shell.execute_reply.started":"2024-07-19T14:01:24.2593Z","shell.execute_reply":"2024-07-19T14:01:25.024594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 5: Define the Image Captioning Model with KAN\n","metadata":{"id":"syc1SEG6_FeH"}},{"cell_type":"code","source":"from keras.layers import Activation, Dense, Embedding, LSTM, Input, Dropout, add\nfrom keras.models import Model\nfrom keras.utils import plot_model\n\ndef define_model_with_kan(vocab_size, max_length):\n # Feature extractor model\n inputs1 = Input(shape=(4096,))\n fe1 = Dropout(0.5)(inputs1)\n fe2 = Dense(256)(fe1)\n fe2 = Activation(HybridActivation(linear_weight=0.5,\n spline_knots=[-1.0, 0.0, 1.0],\n spline_coefficients=[1.0, 2.0, 3.0],\n spline_interval=1.0))(fe2)\n\n # Sequence model\n inputs2 = Input(shape=(max_length,))\n se1 = Embedding(vocab_size, 256, mask_zero=True)(inputs2)\n se2 = Dropout(0.5)(se1)\n se3 = LSTM(256)(se2)\n\n # Decoder model\n decoder1 = add([fe2, se3])\n decoder2 = Dense(256)(decoder1)\n decoder2 = Activation(HybridActivation(linear_weight=0.5,\n spline_knots=[-1.0, 0.0, 1.0],\n spline_coefficients=[1.0, 2.0, 3.0],\n spline_interval=1.0))(decoder2)\n outputs = Dense(vocab_size, activation='softmax')(decoder2)\n\n # Tie it together [image, seq] [word]\n model = Model(inputs=[inputs1, inputs2], outputs=outputs)\n model.compile(loss='categorical_crossentropy', optimizer='adam')\n\n print(model.summary())\n return model\n\nmodel = define_model_with_kan(vocab_size, max_length)","metadata":{"id":"suoz0rGG7lY-","execution":{"iopub.status.busy":"2024-07-19T14:01:28.934931Z","iopub.execute_input":"2024-07-19T14:01:28.935261Z","iopub.status.idle":"2024-07-19T14:01:29.241866Z","shell.execute_reply.started":"2024-07-19T14:01:28.935236Z","shell.execute_reply":"2024-07-19T14:01:29.240984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 6: Train the Model\n","metadata":{"id":"6WBvIV9m7lY-"}},{"cell_type":"code","source":"# Load features and tokenizer\nfeatures = load(open('features.pkl', 'rb'))\ntokenizer = load(open('tokenizer.pkl', 'rb'))\n\n# Create sequences\ntrain_captions = train['caption'].tolist()\ntrain_images = [features[img_id.split('.')[0]] for img_id in train['image']]\nX1, X2, y = create_sequences(tokenizer, max_length, train_captions, train_images)\n\n# Define data generator\ndef data_generator(X1, X2, y, batch_size):\n while True:\n for i in range(0, len(X1), batch_size):\n yield [X1[i:i+batch_size], X2[i:i+batch_size]], y[i:i+batch_size]\n\nbatch_size = 16 # Adjust batch size for quicker testing\nepochs = 5 # Adjust epochs for quicker testing\nsteps = len(X1) // batch_size\n\n# Add callback for monitoring training progress\nfrom tensorflow.keras.callbacks import LambdaCallback\n\nprint_callback = LambdaCallback(on_epoch_end=lambda epoch, logs: print(f\"Epoch {epoch+1} - Loss: {logs['loss']}\"))\n\n# Train the model with progress monitoring\nmodel.fit(data_generator(X1, X2, y, batch_size), epochs=epochs, steps_per_epoch=steps, verbose=1, callbacks=[print_callback])\n\n# Save the model\nmodel.save('image_captioning_model_with_kan.h5')","metadata":{"id":"i5L9G34l7lZA","execution":{"iopub.status.busy":"2024-07-19T14:01:35.495931Z","iopub.execute_input":"2024-07-19T14:01:35.496324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 7: Evaluate and Generate Captions\n","metadata":{"id":"Y0VLNMgT_Pyj"}},{"cell_type":"code","source":"from keras.models import load_model\nfrom nltk.translate.bleu_score import corpus_bleu\n\n# Load the model\nmodel = load_model('image_captioning_model_with_kan.h5')\n\n# Map an integer to a word\ndef word_for_id(integer, tokenizer):\n for word, index in tokenizer.word_index.items():\n if index == integer:\n return word\n return None\n\n# Generate a caption for an image\ndef generate_desc(model, tokenizer, photo, max_length):\n in_text = 'startseq'\n for i in range(max_length):\n sequence = tokenizer.texts_to_sequences([in_text])[0]\n sequence = pad_sequences([sequence], maxlen=max_length)\n yhat = model.predict([photo, sequence], verbose=0)\n yhat = np.argmax(yhat)\n word = word_for_id(yhat, tokenizer)\n if word is None:\n break\n in_text += ' ' + word\n if word == 'endseq':\n break\n return in_text\n\n# Evaluate the model\ndef evaluate_model(model, captions, photos, tokenizer, max_length):\n actual, predicted = [], []\n for key, desc_list in captions.items():\n yhat = generate_desc(model, tokenizer, photos[key], max_length)\n references = [d.split() for d in desc_list]\n actual.append(references)\n predicted.append(yhat.split())\n print('BLEU-1: %f' % corpus_bleu(actual, predicted, weights=(1.0, 0, 0, 0)))\n print('BLEU-2: %f' % corpus_bleu(actual, predicted, weights=(0.5, 0.5, 0, 0)))\n print('BLEU-3: %f' % corpus_bleu(actual, predicted, weights=(0.3, 0.3, 0.3, 0)))\n print('BLEU-4: %f' % corpus_bleu(actual, predicted, weights=(0.25, 0.25, 0.25, 0.25)))\n\nevaluate_model(model, captions, features, tokenizer, max_length)","metadata":{"id":"_SRWzhqI7lZA","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Step 8: Making Predictions\n\nCode Colab: https://colab.research.google.com/drive/1h-a2_EvDZYKawcH847a3kj7gyujWsQPR?usp=sharing","metadata":{"id":"ukcx_5Ec7laf"}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# Function to extract features from a single image using VGG16\ndef extract_feature(image_path, model):\n image = load_img(image_path, target_size=(224, 224))\n image = img_to_array(image)\n image = np.expand_dims(image, axis=0)\n image = preprocess_input(image)\n feature = model.predict(image, verbose=0)\n return feature\n\n# Function to generate a caption for an image\ndef generate_caption(model, tokenizer, photo, max_length):\n in_text = 'startseq'\n for i in range(max_length):\n sequence = tokenizer.texts_to_sequences([in_text])[0]\n sequence = pad_sequences([sequence], maxlen=max_length)\n yhat = model.predict([photo, sequence], verbose=0)\n yhat = np.argmax(yhat)\n word = word_for_id(yhat, tokenizer)\n if word is None:\n break\n in_text += ' ' + word\n if word == 'endseq':\n break\n return in_text\n\n# Load the VGG16 model for feature extraction\nvgg_model = VGG16()\nvgg_model = Model(inputs=vgg_model.inputs, outputs=vgg_model.layers[-2].output)\n\n# Path to the image to be captioned\nimage_path = '//kaggle/input/flickr8k/Images/1000268201_693b08cb0e.jpg'\n\n# Extract features from the image\nphoto = extract_feature(image_path, vgg_model)\n\n# Generate caption for the image\ncaption = generate_caption(model, tokenizer, photo, max_length)\n\n# Display the image and its caption\ndef display_image_with_caption(image_path, caption):\n image = load_img(image_path)\n plt.imshow(image)\n plt.axis('off')\n plt.title(caption)\n plt.show()\n\n# Display the image and generated caption\ndisplay_image_with_caption(image_path, caption)","metadata":{"id":"zumfDhbL7laf","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Result prediction in colab\n\n![image.png](attachment:85b03500-0689-4050-9275-ce2a4a923da6.png)","metadata":{"id":"U1qUu63L7lag"}},{"cell_type":"code","source":"# Generate Flow Diagram using Graphviz\nimport graphviz\n\n# Create a Digraph object\ndot = graphviz.Digraph(comment='Image Captioning with VGG16 and KAN')\n\n# Add nodes for each step in the process and their outputs\ndot.node('A', 'Load and Prepare Dataset\\nOutput: Cleaned Captions, Images')\ndot.node('B', 'Define KAN Layers\\nOutput: Spline and Hybrid Activation Layers')\ndot.node('C', 'Extract Features using VGG16\\nOutput: Image Features')\ndot.node('D', 'Tokenizer and Data Preparation\\nOutput: Tokenizer, Sequences')\ndot.node('E', 'Define Image Captioning Model with KAN\\nOutput: Model Architecture')\ndot.node('F', 'Train the Model\\nOutput: Trained Model')\ndot.node('G', 'Evaluate and Generate Captions\\nOutput: Evaluation Metrics, Captions')\ndot.node('H', 'Making Predictions\\nOutput: Generated Captions for New Images')\n\n# Add edges to represent the flow\ndot.edges(['AB', 'BC', 'CD', 'DE', 'EF', 'FG', 'GH'])\n\n# Add a title\ndot.attr(label='Image Captioning with VGG16 and KAN', fontsize='20', labelloc='t')\n\n# Render the graph\ndot.render('image_captioning_flow_diagram_with_outputs', format='png', cleanup=True)\n\n# Display the diagram\nfrom IPython.display import Image\nImage(filename='image_captioning_flow_diagram_with_outputs.png')","metadata":{"id":"HvIoWHCj7lah","trusted":true},"execution_count":null,"outputs":[]}]}