{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# MNIST OCR with Fast.ai using a RESNET18 model","metadata":{"execution":{"iopub.status.busy":"2023-08-11T12:28:57.817144Z","iopub.execute_input":"2023-08-11T12:28:57.818016Z","iopub.status.idle":"2023-08-11T12:28:57.877249Z","shell.execute_reply.started":"2023-08-11T12:28:57.817974Z","shell.execute_reply":"2023-08-11T12:28:57.876074Z"}}},{"cell_type":"markdown","source":"

ResNets

\n\nA Residual Network (ResNet) is a deep learning Convolutional Neural Network (CNN) model used for computer vision applications, which is designed to overcome the “vanishing gradient” problem, a phenomenon that occurs during the training of deep learning models, where the gradients used to update the model become extremely small or \"vanish\" as they are backpropogated from the output layers to the earlier layers.\n\nBy inherently overcoming the vanishing gradient problem, the ResNet architecture can be used for constructing deep learning models with up to thousands of convolutional layers.\n\nIn this notebook, we use a ResNet 18 model.\n\n

Convolutional Neural Networks (CNN)

\n\nA CNN is composed of an input layer, an output layer, and many hidden layers in between. These layers perform operations that alter the data with the aim of learning features in the input images such as lines, gradients, circles and their combinations, making it good for Computer Vision.\n\n

Fast.ai

\n\nFast.ai is an open source deep learning Python library which provides a high-level API that simplifies the process of building, training, and evaluating deep learning models. The API is designed as an extension of PyTorch, enabling better and faster use of Python and PyTorch for Deep Learning and providing optimized implementations of a vast number of deep learning techniques, ready to use with minimal coding.\n\n

Transfer Learning

\nInstead of training a new model, we are going to apply Transfer learning, which is also used for further speeding up model development by leveraging from a pretrained model which has learned features related to our problem domain, Computer Vision.","metadata":{"execution":{"iopub.status.busy":"2023-08-11T12:28:21.831325Z","iopub.execute_input":"2023-08-11T12:28:21.831746Z","iopub.status.idle":"2023-08-11T12:28:21.898696Z","shell.execute_reply.started":"2023-08-11T12:28:21.831714Z","shell.execute_reply":"2023-08-11T12:28:21.896962Z"}}},{"cell_type":"code","source":"# perform necessary imports\n\nimport warnings\nwarnings.simplefilter('ignore') # hide warnings\n%matplotlib inline\n%reload_ext autoreload\n%autoreload 2\n\nimport pandas as pd\nimport numpy as np\nimport os\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nfrom pathlib import Path\nimport torch\nimport fastai\nfrom fastai import *\nfrom fastai.vision import *\nfrom fastai.vision.all import *\n# from fastai.callbacks import *\nfrom fastai.vision.data import ImageDataLoaders","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:04:04.976080Z","iopub.execute_input":"2023-08-15T08:04:04.977008Z","iopub.status.idle":"2023-08-15T08:04:10.237259Z","shell.execute_reply.started":"2023-08-15T08:04:04.976967Z","shell.execute_reply":"2023-08-15T08:04:10.236302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"Version checks: \\nPyTorch:{torch.__version__}\")\nprint(f\"Fastai:\\t{fastai.__version__} \\nNumpy:\\t{np.__version__} \\nPandas:\\t{pd.__version__}\")","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:04:10.239469Z","iopub.execute_input":"2023-08-15T08:04:10.239818Z","iopub.status.idle":"2023-08-15T08:04:10.302695Z","shell.execute_reply.started":"2023-08-15T08:04:10.239786Z","shell.execute_reply":"2023-08-15T08:04:10.301531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"WORKING_PATH = Path(\"/kaggle/working/\")\nTRAIN_PATH = os.path.join(WORKING_PATH,\"train\")\nTEST_PATH = os.path.join(WORKING_PATH, \"test\")\nprint(f\"Train Path: {TRAIN_PATH}\")\nprint(f\"Test Path: {TEST_PATH}\")","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:04:10.304161Z","iopub.execute_input":"2023-08-15T08:04:10.304495Z","iopub.status.idle":"2023-08-15T08:04:10.367580Z","shell.execute_reply.started":"2023-08-15T08:04:10.304464Z","shell.execute_reply":"2023-08-15T08:04:10.366699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Using Kaggle CSV Dataset","metadata":{}},{"cell_type":"markdown","source":"We start off by performing data wrangling, i.e. converting raw data into the correct, usable format, which in our case is a format that the Fast.ai API accepts, i.e. image files, since we're wroking on a Computer Vision problem.\n\nOut data is in CSV format. Each row of our dataset represents an image and its associated label. We have to create images and folders from row entries in the CSV data.","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\nprint(train_df.shape)\n\ntest_df = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")\nprint(test_df.shape)","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:04:10.369442Z","iopub.execute_input":"2023-08-15T08:04:10.370151Z","iopub.status.idle":"2023-08-15T08:04:15.797861Z","shell.execute_reply.started":"2023-08-15T08:04:10.370092Z","shell.execute_reply":"2023-08-15T08:04:15.796879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Create train and test folders","metadata":{}},{"cell_type":"markdown","source":"We define methods that will create the folder structure and generate image files. Here, we need all training images in the training folder, with subfolders for each type or class label associated with the images; and all the test images in the test folder - without the labels subfolders.\n\nThis is done in order to be able to use Fast.ai's ImageDataLoaders to load our data, which expects actual images when working with data for Computer Vision models.","metadata":{"execution":{"iopub.status.busy":"2023-08-11T12:32:15.560289Z","iopub.execute_input":"2023-08-11T12:32:15.561315Z","iopub.status.idle":"2023-08-11T12:32:15.625328Z","shell.execute_reply.started":"2023-08-11T12:32:15.561257Z","shell.execute_reply":"2023-08-11T12:32:15.623752Z"}}},{"cell_type":"code","source":"def create_train_test_folders():\n for index in range(10):\n try:\n os.makedirs(os.path.join(TRAIN_PATH,str(index)))\n print(f\"Folder created successfully:{os.path.join(TRAIN_PATH,str(index))}\")\n except:\n print(f\"Folder creation not successful: {os.path.join(TRAIN_PATH,str(index))}\")\n\n try:\n os.makedirs(TEST_PATH)\n print(f\"Folder created successfully:{TEST_PATH}\")\n except:\n print(f\"Folder creation not successful: {TEST_PATH}\")\n\ndef save_digit_image(digit, file_path):\n digit = digit.reshape(28,28)\n digit = digit.astype(np.uint8)\n\n img = Image.fromarray(digit)\n img.save(file_path)\n\ndef save_digit_images(source_df, folder):\n source_df = source_df\n for index, row in source_df.iterrows():\n label,digit = row[0], row[1:]\n file_name = f\"{index}.jpg\"\n if folder == TRAIN_PATH:\n sub_folder = os.path.join(TRAIN_PATH,str(label))\n file_path = os.path.join(sub_folder,file_name)\n else: # folder = TEST_PATH\n digit = row \n file_path = os.path.join(folder,file_name)\n digit = digit.values\n save_digit_image(digit, file_path)\n \nimport shutil\ndef get_csv_data():\n print(f\"Working/output folder({WORKING_PATH}): {os.listdir(WORKING_PATH)}\")\n create_train_test_folders()\n save_digit_images(train_df, TRAIN_PATH)\n save_digit_images(test_df, TEST_PATH)","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:04:15.801034Z","iopub.execute_input":"2023-08-15T08:04:15.801547Z","iopub.status.idle":"2023-08-15T08:04:15.870230Z","shell.execute_reply.started":"2023-08-15T08:04:15.801518Z","shell.execute_reply":"2023-08-15T08:04:15.869286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Working folder clean-up, if necessary","metadata":{}},{"cell_type":"code","source":"# shutil.rmtree(TRAIN_PATH)\n# shutil.rmtree(TEST_PATH)","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:04:15.871970Z","iopub.execute_input":"2023-08-15T08:04:15.872305Z","iopub.status.idle":"2023-08-15T08:04:15.935156Z","shell.execute_reply.started":"2023-08-15T08:04:15.872275Z","shell.execute_reply":"2023-08-15T08:04:15.934076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_csv_data()","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:04:15.936426Z","iopub.execute_input":"2023-08-15T08:04:15.936741Z","iopub.status.idle":"2023-08-15T08:04:37.494980Z","shell.execute_reply.started":"2023-08-15T08:04:15.936710Z","shell.execute_reply":"2023-08-15T08:04:37.493882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Display images

\n\nTo verify that the images are in the folders as expected and to get a general idea of what sort of images we're working with, we show a few randomly selected images from each folder.","metadata":{}},{"cell_type":"code","source":"images = os.listdir(os.path.join(TRAIN_PATH,str(9)))\nlength = len(images)\nprint(length)","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:04:37.496370Z","iopub.execute_input":"2023-08-15T08:04:37.497222Z","iopub.status.idle":"2023-08-15T08:04:37.566274Z","shell.execute_reply.started":"2023-08-15T08:04:37.497181Z","shell.execute_reply":"2023-08-15T08:04:37.565226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import random\ndef add_random_subplot(plot_figure, num_rows, num_cols, index_position, path):\n\n # The subplot will take the on a grid with rows and columns.\n\n images = os.listdir(path)\n random_num = random.randint(0, len(images)-1)\n image = Image.open(os.path.join(path,images[random_num])) \n\n ax = plot_figure.add_subplot(num_rows, num_cols, index_position)\n ax.axis(\"off\") \n plt.imshow(image, cmap='gray')\n\ndef show_sample_data_plots(num_rows, num_cols, path):\n plot_figure = plt.figure(figsize=(num_cols, num_rows)) #figsize: (width, height)\n if path == TRAIN_PATH:\n print(f'Training data samples: {path}')\n index_position = 1\n for row_index in range(0, num_rows):\n folders = os.listdir(path)\n for col_index in range(1, num_cols+1):\n sub_folder = str(row_index)\n sub_folder_path = os.path.join(path,sub_folder)\n add_random_subplot(plot_figure, num_rows, num_cols, index_position, sub_folder_path)\n index_position += 1\n else: #TEST\n print(f'Test data samples: {path}')\n index_position = 1\n for row_index in range(1, num_rows*num_cols+1):\n add_random_subplot(plot_figure, num_rows, num_cols, index_position, path)\n index_position += 1\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:04:37.567643Z","iopub.execute_input":"2023-08-15T08:04:37.568399Z","iopub.status.idle":"2023-08-15T08:04:37.639337Z","shell.execute_reply.started":"2023-08-15T08:04:37.568354Z","shell.execute_reply":"2023-08-15T08:04:37.638307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_sample_data_plots(10, 5, TRAIN_PATH)\n\n# by specifying num_rows as 10, we're want to see a sample for each of the 10 digits","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:04:37.641746Z","iopub.execute_input":"2023-08-15T08:04:37.642448Z","iopub.status.idle":"2023-08-15T08:04:39.492518Z","shell.execute_reply.started":"2023-08-15T08:04:37.642409Z","shell.execute_reply":"2023-08-15T08:04:39.491459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"At this point we have no labels for the test dataset, so we can only show randomly selected images and hope to see at least one for each digit.","metadata":{}},{"cell_type":"code","source":"show_sample_data_plots(4, 5, TEST_PATH) ","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:04:39.493548Z","iopub.execute_input":"2023-08-15T08:04:39.494643Z","iopub.status.idle":"2023-08-15T08:04:40.505709Z","shell.execute_reply.started":"2023-08-15T08:04:39.494609Z","shell.execute_reply":"2023-08-15T08:04:40.504793Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Create a learner

\nFor simplicity, we start off by using the Resnet18 model, specifying that we want to keep track of both the error rate and accuracy metrics as the model trains on the MNIST dataset. The model that we're using is pretrained..","metadata":{}},{"cell_type":"code","source":"data = ImageDataLoaders.from_folder(\n path = TRAIN_PATH,\n test = TEST_PATH,\n valid_pct = 0.2,\n bs = 32,\n size = 28,\n shuffle = True,\n batch_tfms=Normalize(imagenet_stats)\n)","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:04:40.506996Z","iopub.execute_input":"2023-08-15T08:04:40.507437Z","iopub.status.idle":"2023-08-15T08:04:47.388663Z","shell.execute_reply.started":"2023-08-15T08:04:40.507400Z","shell.execute_reply":"2023-08-15T08:04:47.387721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn = cnn_learner(\n data,resnet18,\n metrics=[error_rate, accuracy],\n)","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:04:47.390058Z","iopub.execute_input":"2023-08-15T08:04:47.390428Z","iopub.status.idle":"2023-08-15T08:04:48.142860Z","shell.execute_reply.started":"2023-08-15T08:04:47.390390Z","shell.execute_reply":"2023-08-15T08:04:48.141919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Learning rate finder

\nThe learning rate, a pre-determined step-size used for updating the weights of a Machine learning model during backpropagation, is an important variable that controls how much to change the model in response to the estimated error each time the model weights are updated. It also influences how long a model will take to train and whether the model will actually learn.\n\nFastai's learning rate finder function can be used for determining the ideal learning rate. The graph plotted by the learning rate finder is divided into zones like \"minimum, steep, valley and slide\", which presents us with better options for selecting a learning rate. From the learning rate finder, we select the steepest point as a base learning rate.","metadata":{}},{"cell_type":"code","source":"lrs = learn.lr_find(suggest_funcs=(minimum, steep, valley, slide))","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:04:48.147092Z","iopub.execute_input":"2023-08-15T08:04:48.147379Z","iopub.status.idle":"2023-08-15T08:05:00.366563Z","shell.execute_reply.started":"2023-08-15T08:04:48.147354Z","shell.execute_reply":"2023-08-15T08:05:00.365406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"torch.manual_seed(42)\nlearn.fit_one_cycle(3,slice(lrs.valley,lrs.slide))","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:05:00.368407Z","iopub.execute_input":"2023-08-15T08:05:00.368780Z","iopub.status.idle":"2023-08-15T08:09:01.060971Z","shell.execute_reply.started":"2023-08-15T08:05:00.368731Z","shell.execute_reply":"2023-08-15T08:09:01.059921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)\ninterp.plot_top_losses(9, figsize=(7, 7))","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:09:01.062666Z","iopub.execute_input":"2023-08-15T08:09:01.064164Z","iopub.status.idle":"2023-08-15T08:09:14.021966Z","shell.execute_reply.started":"2023-08-15T08:09:01.064124Z","shell.execute_reply":"2023-08-15T08:09:14.020823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To summarize the performance of our model, we use a confusion matrix, which is used to understand how confused the model is when doing predictions. The X-axis f the confusion matrix shows the predicted classes and the Y-axis shows the actual classes, as follows:\n\nTrue positive (TP): Observation is predicted positive and is actually positive.\n\nFalse positive (FP): Observation is predicted positive but is actually negative.\n\nTrue negative (TN): Observation is predicted negative and is actually negative.\n\nFalse negative (FN): Observation is predicted negative but is actually positive.\n\nThe ideal confusion matrix is the closest to an identity matrix, which is only possible if working with a dataset that's perfect for the classification problem.","metadata":{}},{"cell_type":"code","source":"interp.plot_confusion_matrix()","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:09:14.024038Z","iopub.execute_input":"2023-08-15T08:09:14.024739Z","iopub.status.idle":"2023-08-15T08:09:25.783896Z","shell.execute_reply.started":"2023-08-15T08:09:14.024702Z","shell.execute_reply":"2023-08-15T08:09:25.782807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After visual analysis of the top losses and the confusion matrix, we can determine if we need to make any changes to our data preprocessing or data augmentation, thereafter we can proceed training further. Our model has now shows very small losses and a decent accuracy above 98%. If we want a better accuracy than this, we need to perform training over more epochs while applying data augmentation and either keeping track of the best performing model by using checkpoints.\n\nWe then plot top losses in order to evaluate our results. When we visually explore these, we can determine whether the images were in the first place obvious to classify or not. For some, it may be difficult for a human to look at and confidently classify them.","metadata":{}},{"cell_type":"code","source":"learn.unfreeze()\nlrs = learn.lr_find(suggest_funcs=(minimum, steep, valley, slide))","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:09:25.785743Z","iopub.execute_input":"2023-08-15T08:09:25.786392Z","iopub.status.idle":"2023-08-15T08:09:33.435564Z","shell.execute_reply.started":"2023-08-15T08:09:25.786355Z","shell.execute_reply":"2023-08-15T08:09:33.433213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since we no longer have random weights in our last layer, the graph that is a little different looking than in the first lr_find run.\n\nHere we use Python slice function to ensure that the learning rates are incremented linearly going from the first to last layer, ensuring the early layers are trained at a lower learning rate when compared to later layers.\n\nA good exercise at this point is to observe whether doing further intensive training on the model leads to generalization or not. We also need to avoid too much exposure of our model to the test data. These observations will help us determine if we need explicit control on Data Augmentation.","metadata":{}},{"cell_type":"code","source":"EPOCHS = 1\nlearn.fit_one_cycle(EPOCHS,slice(lrs.valley,lrs.slide))","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:32:53.665457Z","iopub.execute_input":"2023-08-15T08:32:53.665972Z","iopub.status.idle":"2023-08-15T08:34:42.693692Z","shell.execute_reply.started":"2023-08-15T08:32:53.665930Z","shell.execute_reply":"2023-08-15T08:34:42.692545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"interp.plot_confusion_matrix()\nlearn.recorder.plot_loss()","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:17:49.120517Z","iopub.execute_input":"2023-08-15T08:17:49.122158Z","iopub.status.idle":"2023-08-15T08:18:01.027092Z","shell.execute_reply.started":"2023-08-15T08:17:49.122115Z","shell.execute_reply":"2023-08-15T08:18:01.025822Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Making predictions

\nIn order to get the predictions on the test set, our learner returns a propability distribution over all possible classes for every given image. We use argmax to get the index of the class with the highest probability:","metadata":{}},{"cell_type":"code","source":"#Prediction on test set\npath = Path('test')\ntest_files = get_image_files(path)\ntest_dl = data.test_dl(test_files)\ntest_dl.show_batch()","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:18:01.028683Z","iopub.execute_input":"2023-08-15T08:18:01.029083Z","iopub.status.idle":"2023-08-15T08:18:02.063760Z","shell.execute_reply.started":"2023-08-15T08:18:01.029046Z","shell.execute_reply":"2023-08-15T08:18:02.062254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"class_score, _, pred_classes = learn.get_preds(dl=test_dl, with_decoded=True)\nprobabilities = class_score[0].tolist()\n[f\"{index}: {probabilities[index]}\" for index in range(len(probabilities))]","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:18:02.065420Z","iopub.execute_input":"2023-08-15T08:18:02.065947Z","iopub.status.idle":"2023-08-15T08:18:30.232859Z","shell.execute_reply.started":"2023-08-15T08:18:02.065906Z","shell.execute_reply":"2023-08-15T08:18:30.231623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class_score = np.argmax(class_score, axis=1)","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:18:30.234589Z","iopub.execute_input":"2023-08-15T08:18:30.235083Z","iopub.status.idle":"2023-08-15T08:18:30.322233Z","shell.execute_reply.started":"2023-08-15T08:18:30.235049Z","shell.execute_reply":"2023-08-15T08:18:30.321252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submission","metadata":{}},{"cell_type":"code","source":"submission = pd.read_csv('/kaggle/input/digit-recognizer/sample_submission.csv', index_col = 'ImageId')\nsubmission['Label'] = class_score\n# submission.to_csv('submission.csv')","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:18:30.323911Z","iopub.execute_input":"2023-08-15T08:18:30.324294Z","iopub.status.idle":"2023-08-15T08:18:30.487690Z","shell.execute_reply.started":"2023-08-15T08:18:30.324258Z","shell.execute_reply":"2023-08-15T08:18:30.486550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Save model for future use or retraining","metadata":{}},{"cell_type":"code","source":"# learn.save(os.path.join(WORKING_PATH,'MNIST-RESNET-Model'))","metadata":{"execution":{"iopub.status.busy":"2023-08-15T08:27:50.729333Z","iopub.execute_input":"2023-08-15T08:27:50.729789Z","iopub.status.idle":"2023-08-15T08:27:51.243086Z","shell.execute_reply.started":"2023-08-15T08:27:50.729751Z","shell.execute_reply":"2023-08-15T08:27:51.242062Z"},"trusted":true},"execution_count":null,"outputs":[]}]}