{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**📲 Imports.**\n\nThis cell contains a list of Python libraries and modules that are being imported for use in the notebook. These imports include both standard Python libraries (such as 📁 io, 📁 os, 📁 csv, and ➗ math) and popular third-party libraries (such as 📷 cv2, 🧮 numpy, 🐼 pandas, 🔥 torch, 🤖 sklearn, 🤖 models, 🤖 transforms, 🧑‍🔬 optim, and 🧠 nn).\n\nSome of the libraries are used for data manipulation and visualization (such as 🧮 numpy, and 🐼 pandas), while others are used for machine learning and deep learning tasks (such as 🔥 torch, 🤖 sklearn, 🤖 models, 🤖 transforms, 🧑‍🔬 optim, and 🧠 nn). The 📷 cv2 library is specifically used for computer vision tasks","metadata":{}},{"cell_type":"code","source":"#Define libraries\nimport io\nimport requests\nimport copy\nimport time \nimport numpy as np\nimport pandas as pd\nimport torchvision\nimport torch\nfrom torchvision.utils import make_grid\nfrom torch.utils.data import Dataset, DataLoader\nfrom torchvision import datasets, models\nimport torch.nn as nn\nimport torch.optim as optim\nimport torchvision.transforms as transforms\nfrom torch.utils.data import DataLoader\nfrom torchvision.datasets import ImageFolder\nimport torch.utils.data\nimport torch.nn.functional as F\nfrom torch.optim import lr_scheduler\nfrom glob import glob\nimport json\nimport os\nimport sys\nfrom tqdm.notebook import tqdm\nimport argparse\nfrom sklearn.preprocessing import OneHotEncoder, LabelEncoder\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score\nimport math, random\nimport pathlib\nimport cv2 as cv\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nimport random\nfrom collections import defaultdict\nimport shutil\nimport logging\nimport argparse\nimport torchvision\nimport seaborn as sb\nimport torch.nn as nn\nfrom PIL import Image\nfrom tqdm import tqdm\nimport torch.optim as optim\nfrom sklearn import datasets\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import trange\nfrom statistics import mean, stdev\nfrom sklearn.model_selection import train_test_split, StratifiedKFold, StratifiedShuffleSplit, KFold\nimport easyocr\nimport matplotlib.patches as patches\nfrom pprint import pprint\nfrom pathlib import Path\nfrom PIL import Image\n","metadata":{"execution":{"iopub.status.busy":"2023-06-05T15:27:48.968776Z","iopub.execute_input":"2023-06-05T15:27:48.969317Z","iopub.status.idle":"2023-06-05T15:27:48.982994Z","shell.execute_reply.started":"2023-06-05T15:27:48.969290Z","shell.execute_reply":"2023-06-05T15:27:48.981982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**✔ Checking Pytorch and Torchvision Versions.**\n\nThis cell 🔍 prints out the current versions of the 🔥 PyTorch and 🔍 Torchvision libraries that are installed in the notebook's environment. 🔥 PyTorch is a popular machine learning library, while 🔍 Torchvision is a package that provides image and video datasets and models for 🔥 PyTorch.\n\nChecking the versions of these libraries is important for ensuring that the notebook is using the correct and up-to-date versions of the libraries, which can affect the performance and compatibility of the code. The print function is used to display the 🔥 PyTorch and 🔍 Torchvision versions, and the 🔥 torch and 🔍 torchvision modules are imported at the beginning of the notebook to access their version numbers. 🔬","metadata":{}},{"cell_type":"code","source":"print(\"PyTorch Version: \",torch.__version__)\nprint(\"Torchvision Version: \",torchvision.__version__)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T15:27:52.894509Z","iopub.execute_input":"2023-06-05T15:27:52.894934Z","iopub.status.idle":"2023-06-05T15:27:52.901158Z","shell.execute_reply.started":"2023-06-05T15:27:52.894898Z","shell.execute_reply":"2023-06-05T15:27:52.900264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**setting the random seed**\n\nSetting the random seed ensures that these sources of randomness are consistent across different runs of the experiment, allowing for reproducibility of results, and that the results of any random operations are consistent across different machine","metadata":{}},{"cell_type":"code","source":"def seed_everything(seed):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n\nseed = 1999\nseed_everything(seed)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T15:27:56.201602Z","iopub.execute_input":"2023-06-05T15:27:56.201958Z","iopub.status.idle":"2023-06-05T15:27:56.212872Z","shell.execute_reply.started":"2023-06-05T15:27:56.201928Z","shell.execute_reply":"2023-06-05T15:27:56.211745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**⌨ Defining some parameters for the model.**\n\nThis cell sets several important parameters that will be used to configure and train a machine learning model. The (num_classes, batch_size, and fold_train) variables specify the number of classes in the dataset, the batch size for training, and the number of epochs to train for, respectively. These are common parameters in machine learning tasks and can affect the performance and training time of the mode. fold_train parameter refers to the usage of the first fold of the data for traing and The remaining folds will be used for validation, This helps to prevent the model from overfitting to the training data.","metadata":{}},{"cell_type":"code","source":"class CFG:\n    isOneHot = False\n    label_map = {'dot': 0, 'horizontal_bar' : 1, 'vertical_bar': 2, 'line': 3, 'scatter': 4}\n    num_classes = 5\n    batchSize = 32\n    fold_train = 0 #  the first fold of the data will be used for training","metadata":{"execution":{"iopub.status.busy":"2023-06-05T15:27:58.826193Z","iopub.execute_input":"2023-06-05T15:27:58.826529Z","iopub.status.idle":"2023-06-05T15:27:58.834517Z","shell.execute_reply.started":"2023-06-05T15:27:58.826500Z","shell.execute_reply":"2023-06-05T15:27:58.833667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Data Loading**\n\n\nonly images that are at least 9 pixels wide and 9 pixels high will be loaded from the dataset. The first 10000 images that meet this criteria will be loaded.","metadata":{}},{"cell_type":"code","source":"INPUT_DIR = '/kaggle/input/benetech-making-graphs-accessible'\ntrain_image_paths = sorted(glob(os.path.join(INPUT_DIR, 'train/images/*.jpg')))\ntrain_annotation_paths = sorted(glob(os.path.join(INPUT_DIR, 'train/annotations/*.json')))\nprint('train images:', len(train_image_paths))\nprint('train annotations:', len(train_annotation_paths))\nargs = argparse.Namespace(\n    sampling_size = 10000,\n    thresholds = [9, 128]\n)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T15:28:01.036998Z","iopub.execute_input":"2023-06-05T15:28:01.037545Z","iopub.status.idle":"2023-06-05T15:28:02.722713Z","shell.execute_reply.started":"2023-06-05T15:28:01.037506Z","shell.execute_reply":"2023-06-05T15:28:02.721729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#load a random image from the training directory\nplt.figure(figsize=(14, 6))\nfile_name = random.choice(train_image_paths)\nimage = plt.imread(file_name)\nplt.imshow(image)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-05T15:28:04.300079Z","iopub.execute_input":"2023-06-05T15:28:04.300449Z","iopub.status.idle":"2023-06-05T15:28:04.604083Z","shell.execute_reply.started":"2023-06-05T15:28:04.300418Z","shell.execute_reply":"2023-06-05T15:28:04.603206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#load the annotations for the above graph\na = file_name.split(\"/\")[-1]\n\nsplit_file_path = file_name.split(\"/\")\nsplit_file_name = a.split(\".\")\njson_file_path = INPUT_DIR+\"/train/annotations/\"+split_file_name[0]+\".json\"\nannot = open(json_file_path)\nannot = json.load(annot)\nannot\n","metadata":{"execution":{"iopub.status.busy":"2023-06-05T15:28:07.801885Z","iopub.execute_input":"2023-06-05T15:28:07.802258Z","iopub.status.idle":"2023-06-05T15:28:07.824140Z","shell.execute_reply.started":"2023-06-05T15:28:07.802226Z","shell.execute_reply":"2023-06-05T15:28:07.823132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# loading a random image from the test directory\nplt.figure(figsize=(14, 6))\nfile_name = random.choice(os.listdir(INPUT_DIR+\"/test/images/\"))\nimage = plt.imread(INPUT_DIR+\"/test/images/\"+file_name)\nplt.imshow(image)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-05T15:28:11.243334Z","iopub.execute_input":"2023-06-05T15:28:11.243749Z","iopub.status.idle":"2023-06-05T15:28:11.621963Z","shell.execute_reply.started":"2023-06-05T15:28:11.243710Z","shell.execute_reply":"2023-06-05T15:28:11.621095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**creating a dataframe with images and annotations for classification**\n\nit sets the hight and length of the plot bounding box","metadata":{}},{"cell_type":"code","source":"#creating a dataframe with images and annotations for classification\ntrain_df = pd.DataFrame({'image_path':train_image_paths})\n\ntrain_df['train_annotations'] = train_df['image_path'].str.replace('images', 'annotations')\ntrain_df['train_annotations'] = train_df['train_annotations'].str.replace('jpg', 'json')\n\nchart_type, source, height, width = [], [], [], []\nfor annotations in tqdm(train_df['train_annotations']):\n    annot = open(annotations)\n    annot = json.load(annot)\n    chart_type.append(annot['chart-type'])\n    source.append(annot['source'])\n    height.append(annot['plot-bb']['height'])\n    width.append(annot['plot-bb']['width'])\n\ntrain_df['chart_type'] = chart_type\ntrain_df['source'] = source\ntrain_df['height'] = height\ntrain_df['width'] = width","metadata":{"execution":{"iopub.status.busy":"2023-06-05T15:28:14.463600Z","iopub.execute_input":"2023-06-05T15:28:14.463957Z","iopub.status.idle":"2023-06-05T15:33:53.850503Z","shell.execute_reply.started":"2023-06-05T15:28:14.463923Z","shell.execute_reply":"2023-06-05T15:33:53.848268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T15:34:15.984594Z","iopub.execute_input":"2023-06-05T15:34:15.984952Z","iopub.status.idle":"2023-06-05T15:34:16.003172Z","shell.execute_reply.started":"2023-06-05T15:34:15.984921Z","shell.execute_reply":"2023-06-05T15:34:16.002185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.chart_type.value_counts()","metadata":{"execution":{"iopub.status.busy":"2023-06-05T15:34:19.400421Z","iopub.execute_input":"2023-06-05T15:34:19.400774Z","iopub.status.idle":"2023-06-05T15:34:19.427164Z","shell.execute_reply.started":"2023-06-05T15:34:19.400745Z","shell.execute_reply":"2023-06-05T15:34:19.425642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"🔧 Transformation Configurations.\n\nThis cell discusses how to configure PyTorch transforms for image preprocessing 🖼️, which is an important step in developing deep learning models for image classification tasks 🤖. The code shows how to define different sets of transforms for training and testing data 🔧, including normalization 📊. The use of these transforms can improve the model's performance by increasing the variability and diversity of the training data 💪. These transforms are used to convert images from their original format to a format that can be used by a machine learning model. transform normalizes the pixel values of an image to have a mean of 0.5 and a standard deviation of 0.5.","metadata":{}},{"cell_type":"code","source":"#data transformation\ntransforms_train = transforms.Compose(\n    [transforms.ToTensor(),\n     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\ntransforms_test = transforms.Compose(\n    [transforms.ToTensor(),\n     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])","metadata":{"execution":{"iopub.status.busy":"2023-06-05T16:35:07.483898Z","iopub.execute_input":"2023-06-05T16:35:07.484292Z","iopub.status.idle":"2023-06-05T16:35:07.490729Z","shell.execute_reply.started":"2023-06-05T16:35:07.484261Z","shell.execute_reply":"2023-06-05T16:35:07.489793Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" The image is then converted from the BGR color space to RGB. The image is then resized to 64x128 pixels. The image is then converted to a NumPy array and normalized to have a range of [0, 1]. The label is then extracted from the Pandas DataFrame","metadata":{}},{"cell_type":"code","source":"class ImageCharts(torch.utils.data.Dataset):\n    def __init__(self, df, transforms  = None): ### phase = train/test\n        self.df = df\n        self.transforms = transforms\n\n    def __len__(self): #return the num of images in the class\n        return len(self.df)\n\n    def __getitem__(self,idx):\n        name_img = self.df.iloc[idx]['image_path']\n        img = cv.imread(name_img)\n        img = cv.cvtColor(img,cv.COLOR_BGR2RGB)\n        img = cv.resize(img,(64,128))\n        img = img.astype(np.float32)/255.0\n\n        label = np.array(self.df.iloc[idx]['label'])\n\n        if(self.transforms is not None):\n            img = self.transforms(img)\n        return img, label","metadata":{"execution":{"iopub.status.busy":"2023-06-05T16:35:10.267090Z","iopub.execute_input":"2023-06-05T16:35:10.267461Z","iopub.status.idle":"2023-06-05T16:35:10.274987Z","shell.execute_reply.started":"2023-06-05T16:35:10.267431Z","shell.execute_reply":"2023-06-05T16:35:10.274067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Cross Validation**\n\nIt is a technique used to evaluate the performance of a model. it splits the data into folds. The lambda function converts the chart type to the corresponding label. The StratifiedKFold object ensures that each fold has the same distribution of labels as the training data as a whole. This helps to ensure that the model is not overfit to any particular subset of the data.","metadata":{}},{"cell_type":"code","source":"def cv_split(Xtrain, ytrain, n_folds, seed):\n    kfold = StratifiedKFold(n_splits = n_folds, shuffle = True, random_state = seed)\n    for num, (train_index, val_index) in enumerate(kfold.split(Xtrain, ytrain)):\n        Xtrain.loc[val_index, 'fold'] = int(num)\n    Xtrain['fold'] = Xtrain['fold'].astype(int)\n    return Xtrain\n\nmeta_df = cv_split(train_df,train_df['chart_type'], 5, 42)\nmeta_df['label'] = meta_df['chart_type'].apply(lambda x: CFG.label_map[x])\nmeta_df.head(10)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T16:35:13.808969Z","iopub.execute_input":"2023-06-05T16:35:13.809940Z","iopub.status.idle":"2023-06-05T16:35:14.008964Z","shell.execute_reply.started":"2023-06-05T16:35:13.809898Z","shell.execute_reply":"2023-06-05T16:35:14.008006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"DataLoaders\n\ncreates a mini-batch iterator for the dataset which is more efficient than training and evaluating the model on the entire dataset at once.","metadata":{}},{"cell_type":"code","source":"#dataloaders\ntrain_ds = ImageCharts(meta_df[meta_df.fold != CFG.fold_train].reset_index(), transforms=transforms_train)\nval_ds = ImageCharts(meta_df[meta_df.fold == CFG.fold_train].reset_index(), transforms=transforms_train)\ntrain_dl = torch.utils.data.DataLoader(train_ds, batch_size= CFG.batchSize, shuffle=True) #specify the size of each mini batch\nval_dl = torch.utils.data.DataLoader(val_ds, batch_size= CFG.batchSize, shuffle=True)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T16:35:17.079688Z","iopub.execute_input":"2023-06-05T16:35:17.080064Z","iopub.status.idle":"2023-06-05T16:35:17.107525Z","shell.execute_reply.started":"2023-06-05T16:35:17.080033Z","shell.execute_reply":"2023-06-05T16:35:17.106614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"this cell dicusses to get the first batch of data from the training dataset","metadata":{}},{"cell_type":"code","source":"a,b = next(iter(train_dl)) #a: images in the batch b:labels\nprint(a.shape)\nprint(b.shape)\nprint('chart_type:',list(CFG.label_map)[b[0].detach().numpy()])\nplt.imshow(a[0].permute(1,2,0))","metadata":{"execution":{"iopub.status.busy":"2023-06-05T16:35:30.102896Z","iopub.execute_input":"2023-06-05T16:35:30.103282Z","iopub.status.idle":"2023-06-05T16:35:30.447898Z","shell.execute_reply.started":"2023-06-05T16:35:30.103249Z","shell.execute_reply":"2023-06-05T16:35:30.447033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Defing the model we will use which is \"ResNet50\" and load it into gpu**","metadata":{}},{"cell_type":"code","source":"model = torchvision.models.resnet50(pretrained=False)\n\n\nnum_features = model.fc.in_features\nmodel.fc = nn.Linear(num_features, CFG.num_classes)\ndevice = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\nmyModel = model.to(device)\nnext(myModel.parameters()).device","metadata":{"execution":{"iopub.status.busy":"2023-06-05T16:35:35.592565Z","iopub.execute_input":"2023-06-05T16:35:35.592926Z","iopub.status.idle":"2023-06-05T16:35:36.021866Z","shell.execute_reply.started":"2023-06-05T16:35:35.592896Z","shell.execute_reply":"2023-06-05T16:35:36.020937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model","metadata":{"execution":{"iopub.status.busy":"2023-06-05T16:35:38.782395Z","iopub.execute_input":"2023-06-05T16:35:38.782751Z","iopub.status.idle":"2023-06-05T16:35:38.808078Z","shell.execute_reply.started":"2023-06-05T16:35:38.782713Z","shell.execute_reply":"2023-06-05T16:35:38.806307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**📝 Training Loop (The definition of the function).**\n\n\nThe following code defines a Python function called \"training\" that implements the training loop for a given model architecture, dataset, loss function, optimizer, and learning rate scheduler.\n\nThe function takes five input arguments:\n\n\"model\" refers to the model architecture that needs to be trained. \"dataloaders\" refers to a dictionary containing the training and validation data loaders. \"criterion\" refers to the loss function that needs to be optimized. \"optimizer\" refers to the optimization algorithm that updates the model parameters. \"scheduler\" refers to the learning rate scheduler that dynamically adjusts the learning rate during training.The scheduler is used to adjust the learning rate of the optimizer over time.. The \"num_epochs\" parameter specifies the number of times the model will iterate over the entire dataset during training.\n\nDuring each epoch, the training loop iterates over the training and validation data loaders, computing the loss and accuracy on each batch of data. The model parameters are updated based on the gradients of the loss function with respect to the model parameters. The learning rate scheduler adjusts the learning rate at each epoch based on a predefined schedule.\n\nThe function also saves the model with the best validation accuracy and the final model parameters in two different files for later use.\n\nThe training loop's progress is printed out for each epoch, including the loss, accuracy, and learning rate for both the training and validation phases.\n\nThe function returns the trained model, along with four lists that contain the training and validation accuracy and loss history.\n","metadata":{}},{"cell_type":"code","source":"def training(model, train_dl, val_dl, num_epochs):\n    \n  # Loss Function, Optimizer and Scheduler\n    criterion = nn.CrossEntropyLoss()\n    optimizer = torch.optim.Adam(model.parameters(),lr=0.0001)\n    scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=0.001,\n                                                steps_per_epoch=int(len(train_dl)),\n                                                epochs=num_epochs,\n                                                anneal_strategy='linear')\n\n  # Repeat for each epoch\n    best_acc = -1\n    for epoch in range(num_epochs):\n        \n        running_loss = 0.0\n        correct_prediction = 0\n        total_prediction = 0\n        \n\n    # Repeat for each batch in the training set\n        for i, data in enumerate(train_dl):\n            \n            \n        # Get the input features and target labels, and put them on the GPU\n            inputs, labels = data[0].to(device), data[1].to(device)\n        # Normalize the inputs\n            inputs_m, inputs_s = inputs.mean(), inputs.std()\n            inputs = (inputs - inputs_m) / inputs_s\n\n        # Zero the parameter gradients\n            optimizer.zero_grad()\n\n        # forward + backward + optimize\n            outputs = model(inputs)\n            loss = criterion(outputs, labels)\n            loss.backward()\n            optimizer.step()\n            scheduler.step()\n\n        # Keep stats for Loss and Accuracy\n            running_loss += loss.item()\n\n        # Get the predicted class with the highest score\n            _, prediction = torch.max(outputs,1)\n        # Count of predictions that matched the target label\n            correct_prediction += (prediction == labels).sum().item()\n            total_prediction += prediction.shape[0]\n            if (i + 1) % 50 == 0:\n                # print every 10 mini-batches\n                print('Epoch [{}/{}], Step [{}/{}], Loss : {:.4f}'\n               .format(epoch + 1, num_epochs, i + 1, len(train_dl), running_loss/(i + 1)))\n    \n    # Print stats at the end of the epoch\n    num_batches = len(train_dl)\n    avg_loss = running_loss / num_batches\n    acc = correct_prediction/total_prediction\n    print(f'Epoch: {epoch + 1}, Loss: {avg_loss:.4f}, Accuracy: {acc:.4f}')\n\n    gt = []\n    pred = []\n\n    with torch.no_grad():\n        correct = 0\n        total = 0\n        val_loss = 0 \n        for idx, data_ in enumerate(val_dl):\n            inputs, labels = data_[0].to(device), data_[1].to(device)\n\n            # Normalize the inputs\n            inputs_m, inputs_s = inputs.mean(), inputs.std()\n            inputs = (inputs - inputs_m) / inputs_s\n\n            # forward + backward + optimize\n            outputs = model(inputs)\n            loss = criterion(outputs, labels)\n            \n            # Keep stats for Loss and Accuracy\n            val_loss += loss.item()\n\n            # Get the predicted class with the highest score\n            _, prediction = torch.max(outputs,1)\n            predi = torch.softmax(outputs, dim = -1)\n            gt.append(labels)\n            pred.append(predi[:,1])\n            # Count of predictions that matched the target label\n            correct += (prediction == labels).sum().item()\n            total += prediction.shape[0]\n        print('Accuracy of the network val: {:.4f} %'.format(100 * correct / total))\n\n        final_score = 100 * correct / total\n\n\n        if (best_acc < final_score):\n            best_acc = final_score\n            print(\"Saving best model!\")\n            torch.save(model.state_dict(), f'Benetech _ResNet50_fold0.pth')\n    print('Finished Training')\nnum_epochs = 4\ntraining(myModel, train_dl, val_dl, num_epochs)\n\n","metadata":{"execution":{"iopub.status.busy":"2023-06-05T16:36:00.576025Z","iopub.execute_input":"2023-06-05T16:36:00.576394Z","iopub.status.idle":"2023-06-05T16:56:31.806178Z","shell.execute_reply.started":"2023-06-05T16:36:00.576362Z","shell.execute_reply":"2023-06-05T16:56:31.805108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Object Character Recognition**","metadata":{}},{"cell_type":"code","source":"reader = easyocr.Reader(['en']) # initialize OCR\nresult = reader.readtext(image) # input image\n\nfor i in range(3):\n    print(result[i], \"\\n\")","metadata":{"execution":{"iopub.status.busy":"2023-06-05T17:01:55.185790Z","iopub.execute_input":"2023-06-05T17:01:55.186171Z","iopub.status.idle":"2023-06-05T17:01:58.683048Z","shell.execute_reply.started":"2023-06-05T17:01:55.186135Z","shell.execute_reply":"2023-06-05T17:01:58.681940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"annotations = Path(\"/kaggle/input/benetech-making-graphs-accessible/train/annotations\")\nimages = Path(\"/kaggle/input/benetech-making-graphs-accessible/train/images\")\nall_annotations = list(annotations.iterdir())","metadata":{"execution":{"iopub.status.busy":"2023-06-05T17:02:01.036002Z","iopub.execute_input":"2023-06-05T17:02:01.036662Z","iopub.status.idle":"2023-06-05T17:02:01.400760Z","shell.execute_reply.started":"2023-06-05T17:02:01.036628Z","shell.execute_reply":"2023-06-05T17:02:01.399781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Plot graphs with annotations","metadata":{}},{"cell_type":"code","source":"def plot_with_annotations(filename, ax):\n    with open(filename, \"r\") as f:\n        row = pd.json_normalize(json.load(f)).iloc[0] #first row of the df\n        \n    img = Image.open(images / filename.with_suffix(\".jpg\").name)\n    \n    ax.imshow(img)\n    bbox = patches.Rectangle(\n        (row[\"plot-bb.x0\"], row[\"plot-bb.y0\"]), \n        row[\"plot-bb.width\"], \n        row[\"plot-bb.height\"], \n        linewidth=1, \n        edgecolor=\"r\", \n        facecolor=\"none\"\n    )\n    ax.add_patch(bbox)\n    \n    role2color = {\n        'axis_title': \"blue\",\n        'chart_title':\"green\",\n        'legend_label':\"yellow\",\n        'other':\"grey\",\n        'tick_grouping':\"brown\",\n        'tick_label':\"purple\",\n    }\n    for text in row[\"text\"]:\n        polygon = text[\"polygon\"]\n        xs = np.array([v for k, v in polygon.items() if \"x\" in k])\n        ys = np.array([v for k, v in polygon.items() if \"y\" in k])\n        xy = np.vstack([xs, ys]).T\n        text_polygon = patches.Polygon(xy, linewidth=1, edgecolor=role2color[text[\"role\"]], facecolor=\"none\")\n        ax.add_patch(text_polygon)\n\n    \n    ax.axis(\"off\")","metadata":{"execution":{"iopub.status.busy":"2023-06-05T17:02:04.151250Z","iopub.execute_input":"2023-06-05T17:02:04.151600Z","iopub.status.idle":"2023-06-05T17:02:04.160852Z","shell.execute_reply.started":"2023-06-05T17:02:04.151570Z","shell.execute_reply":"2023-06-05T17:02:04.159958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nrows, ncols = 3, 3\nfig, ax = plt.subplots(nrows, ncols, figsize=(15, 10))\nfig.tight_layout()\nax = ax.flatten()\n\nidxs = np.random.randint(low=0, high=len(all_annotations), size=nrows*ncols)\n\nfor i, idx in enumerate(idxs):\n    plot_with_annotations(all_annotations[idx], ax[i])\n    \nplt.savefig(\"annotated_examples.pdf\", bbox_inches=\"tight\")","metadata":{"execution":{"iopub.status.busy":"2023-06-05T17:02:08.363930Z","iopub.execute_input":"2023-06-05T17:02:08.364642Z","iopub.status.idle":"2023-06-05T17:02:11.170667Z","shell.execute_reply.started":"2023-06-05T17:02:08.364605Z","shell.execute_reply":"2023-06-05T17:02:11.168909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ANNOTATION_PATH = \"/kaggle/input/benetech-making-graphs-accessible/train/annotations/0000ae6cbdb1.json\"\n","metadata":{"execution":{"iopub.status.busy":"2023-06-05T17:02:15.125178Z","iopub.execute_input":"2023-06-05T17:02:15.125878Z","iopub.status.idle":"2023-06-05T17:02:15.132066Z","shell.execute_reply.started":"2023-06-05T17:02:15.125844Z","shell.execute_reply":"2023-06-05T17:02:15.131107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Predict Bounding Boxes**","metadata":{}},{"cell_type":"code","source":"with open(ANNOTATION_PATH) as f:\n    annotation = json.loads(f.read())\n\nmasks = []\nfor i in range(2):\n    x0 = annotation[\"text\"][i][\"polygon\"][\"x0\"]\n    y0 = annotation[\"text\"][i][\"polygon\"][\"y0\"]\n    H = annotation[\"text\"][i][\"polygon\"][\"x1\"] - x0\n    W = annotation[\"text\"][i][\"polygon\"][\"y2\"] - y0\n    mask = image[y0:y0+W, x0:x0+H]\n    masks.append(mask)\n\nreader = easyocr.Reader(['en']) # this needs to run only once to load the model into memory\n\nresults = []\nfor mask in masks:\n    result = reader.readtext(mask)\n    results.append(result)\n\nprint(results)","metadata":{"execution":{"iopub.status.busy":"2023-06-05T17:02:17.037788Z","iopub.execute_input":"2023-06-05T17:02:17.038173Z","iopub.status.idle":"2023-06-05T17:02:19.912999Z","shell.execute_reply.started":"2023-06-05T17:02:17.038133Z","shell.execute_reply":"2023-06-05T17:02:19.912043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nimport io\nimport json\nimport requests\nimport cv2\nimport os\n\ndef ocr_space(image_path, api_key):\n    url_api = \"https://api.ocr.space/parse/image\"\n    image = cv2.imread(image_path)\n    _, compressedimage = cv2.imencode(\".jpg\", image, [1, 90])\n    file_bytes = io.BytesIO(compressedimage.tobytes())\n\n    response = requests.post(\n        url_api,\n        files={\"/kaggle/input/benetech-making-graphs-accessible/train/images/0000ae6cbdb1.jpg\": file_bytes},\n        data={\"apikey\":\"K82246663188957\" , \"language\": \"eng\"},\n    )\n    return response.json()\n\ndataset_dir = \"/kaggle/input/benetech-making-graphs-accessible/train/images\"  # replace with the path to your dataset directory\napi_key = \"K82246663188957\"  # replace with your OCR.space API key\n\n# Get the list of image file paths\nimage_files = [os.path.join(dataset_dir, f) for f in os.listdir(dataset_dir) if f.endswith(\".jpg\") or f.endswith(\".png\")]\n\n# Perform OCR on each image and store the results\nocr_results = {}\nfor image_path in image_files:\n    print(f\"Processing {image_path}\")\n    ocr_results[image_path] = ocr_space(image_path, api_key)\n\nprint(ocr_results)'''","metadata":{"execution":{"iopub.status.busy":"2023-06-03T21:45:15.496418Z","iopub.execute_input":"2023-06-03T21:45:15.496795Z","iopub.status.idle":"2023-06-03T21:45:15.507468Z","shell.execute_reply.started":"2023-06-03T21:45:15.496763Z","shell.execute_reply":"2023-06-03T21:45:15.506462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nimport os\nimport pytesseract\n\ndef ocr_on_images(images):\n    \n\n    text_list = []\n    for image_path in os.listdir(images):\n        \n        img = pytesseract.image_to_string(os.path.join(images, image_path))\n        text_list.append(img)\n\n    return text_list\nif __name__ == \"__main__\":\n    \n        \n    dataset_path = \"/path/to/dataset\"\n    text_list = ocr_on_images(images)\n\n    for text in text_list:\n        \n        print(text)\n'''","metadata":{"execution":{"iopub.status.busy":"2023-06-05T18:09:54.779186Z","iopub.execute_input":"2023-06-05T18:09:54.779550Z","iopub.status.idle":"2023-06-05T18:09:54.787580Z","shell.execute_reply.started":"2023-06-05T18:09:54.779524Z","shell.execute_reply":"2023-06-05T18:09:54.786551Z"},"trusted":true},"execution_count":null,"outputs":[]}]}