{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":" Optimization Algorithms in Deep Learning
\n\n Category: Other
","metadata":{}},{"cell_type":"markdown","source":"# Contents\n\n\n* [1 Introduction](#1)\n* [2 Gradient Descent](#2)\n* [3 Stochastic Gradient Descent and Mini-Batch Gradient Descent](#3)\n* [4 SGD with Momentum and Nesterov Momentum](#4)\n* [5 Adaptive Gradient Descent Algorithms](#5)\n* [6 MoMo: Momentum Models for Adaptive Learning Rates](#6)\n* [7 Sophia: Scalable Stochastic Second-Order Optimizer](#7)\n* [8 Conclusion](#8)\n* [9 References](#9)","metadata":{}},{"cell_type":"markdown","source":" \n# 1 Introduction","metadata":{}},{"cell_type":"markdown","source":"Optimization problems appear in different areas, so they have not lost their relevance so far. Optimization methods are algorithms that are used to minimize or maximize a function. In machine learning, optimization methods (optimizers) are used to minimize the model prediction's error, or loss function, by modifying model's parameters in such a way that the model has the most accurate results. Loss function shows how the model's output is close to the expected value. Therefore, optimization is a crucial part in training a model.\n\nNeural network training is one of the most difficult optimization problems because neural networks can be large and can have a tremendous number of parameters. A landscape of a loss function is usually defined in a high-dimensional space due to the high number of parameters. \n","metadata":{}},{"cell_type":"markdown","source":" \n\n# 2 Gradient Descent","metadata":{}},{"cell_type":"markdown","source":"Most neural networks use optimizers that are based on a gradient descent algorithm that has a simple idea behind it. First, we have to choose random point as an initial step of the process. Then we calculate the gradient of the function. Gradient indicates the direction of the steepest ascent. For this purpose, we have to take partial derivatives of the function. Since we are looking for a minimum value, we have to take the gradient with a negative sign and apply it to modify the parameters. The algorithm works iteratively and updates the parameters on each iteration. The iteration step is defined by learning rate that can be changed. After each iteration, the algorithm moves to the next point that is located closer to the minimum. \n\n\nAn illustration on a simple curve may help understand the idea. Let the graph of a function $(x-2)^2$ below represent the loss function:","metadata":{}},{"cell_type":"code","source":"from scipy.misc import derivative\nfrom scipy import optimize\nimport matplotlib.pyplot as plt\nimport pandas as pd\nimport numpy as np\nfrom numpy.linalg import norm\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nplt.style.use('ggplot')\n\n\ndef f(x): return (x-2)**2 \n\nx = range(-15,25)\ny = [f(xn) for xn in x]\n\nplt.plot(x, y, color='dodgerblue')\nplt.title(r'f(x)=(x-2)^2')\nplt.xlabel('x')\nplt.ylabel('f(x)');","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T20:38:18.854309Z","iopub.execute_input":"2023-07-16T20:38:18.854675Z","iopub.status.idle":"2023-07-16T20:38:19.160265Z","shell.execute_reply.started":"2023-07-16T20:38:18.854648Z","shell.execute_reply":"2023-07-16T20:38:19.159434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us choose a point on a curve that corresponds to $x=-10$ as a starting point. Let learning rate be equal to 0.1 and $\\epsilon=0.01$. Points where the gradient is equal to zero are called stationary. Gradient in these points does not give any information about the function, and therefore the iteration process can be stopped as soon as the algorithm achieves such point. The difference between the value on the current algorithm step and the updated value will therefore be less than epsilon value that is to be defined beforehand. ","metadata":{}},{"cell_type":"code","source":"x_prev = -10.0\nstep = 1e-1\neps = 1e-2\n\nx_vals, y_vals = [], []\n\nx_vals.append(x_prev)\ny_vals.append(f(x_prev))\n\nwhile True:\n x_current = x_prev - step*derivative(f, x_prev)\n \n x_vals.append(x_current)\n y_vals.append(f(x_current))\n \n if(x_current - x_prev) \n\n# 3 Stochastic Gradient Descent and Mini-Batch Gradient Descent","metadata":{}},{"cell_type":"markdown","source":"Stochastic gradient descent (SGD) is one of the modifications of the traditional gradient descent algorithm (GD). In contrast to GD, SGD does not use the entire data at each step; instead, it picks a random sample at each iteration. The advantage of SGD is that it requires less memory compared to GD, making it more useful for large datasets. In addition, its randomness adds noise to the learning process, which helps avoid some local minimum points.\n\n\nIf SGD randomly picks a subsets of data instead of one point at each iteration, then algorithm is called Mini-Batch Gradient Descent. On each subset the algorithm computes the gradient and moves in the direction opposite to the gradient's direction. Although stochastic approach reduces the computational cost per learning step, it may lead to noisy gradients, causing oscillations near the minima [[1]](#9.1),[[2]](#9.2).\n\n\nIn order to illustrate the difference between SGD and Mini-Batch SGD, let us generate some random data to plot possible gradient paths. The chart below has two subplots that demonstrate the paths SGD and Mini-Batch SGD may take towards minima. Each subplot represents loss function coutour lines that are often used to display a 3D surface on a 2D plot. \n","metadata":{}},{"cell_type":"code","source":"def generate_plot(batch_size, title, ax=None):\n \n if ax is None:\n ax = plt.gca()\n \n np.random.seed(2)\n target_point = np.random.normal(4, 2, 2)\n current_point = np.random.normal(6, 2, 2)\n X = np.random.normal(2, 3, (100, 2))\n y = np.dot(X, target_point) + np.random.normal(0, 1, 100)\n \n eps = 1e-2\n lr = 1e-2\n history = []\n \n history.append(current_point.copy())\n prev_point = current_point + 2 * eps\n\n while norm(current_point - prev_point) > eps:\n prev_point = current_point.copy()\n\n indices = np.random.choice(X.shape[0], batch_size, replace=False)\n X_batch, y_batch = X[indices], y[indices]\n\n current_point -= 2 * lr * np.dot(X_batch.T, np.dot(X_batch, current_point) - y_batch) / y_batch.shape[0]\n history.append(current_point.copy())\n \n \n D1, D2 = np.meshgrid(np.linspace(-1, 10, 100), np.linspace(-1, 10, 100))\n D = np.dstack((D1, D2))\n Z = np.zeros_like(D1)\n \n for i in range(D1.shape[0]):\n for j in range(D1.shape[1]):\n Z[i, j] = np.mean(np.power(np.dot(X, D[i][j]) - y, 2))\n\n \n ax.set_title(str(title))\n ax.set_xlim((-1, 8))\n ax.set_ylim((0, 10))\n\n ax.contour(D1, D2, Z, levels=np.linspace(0, 1000, 30), cmap=plt.cm.plasma)\n ax.plot(target_point[0], target_point[1], 'rx', markersize=12)\n ax.plot(np.array(history)[:, 0], np.array(history)[:, 1], \n marker='<', color='dodgerblue', markersize=10, \n markeredgecolor='purple', linewidth=4)\n \n ax.annotate('Starting point', xy=(-0.5, 9), size=15, \n bbox=dict(boxstyle='round', fc='lightblue', ec='gray'))\n \n ax.set_xlabel('Weight_1')\n ax.set_ylabel('Weight_2');\n return ax\n \n\nfig, axes = plt.subplots(1,2, figsize=(16,6), sharey=True)\n\ntitles = ['SGD', 'Mini-Batch SGD, batch size=10']\nfor i, ax in enumerate(fig.axes):\n ax=generate_plot(batch_size=np.power(10, i), title=titles[i], ax=ax)\n \nplt.tight_layout(w_pad=2)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T20:38:19.509888Z","iopub.execute_input":"2023-07-16T20:38:19.510636Z","iopub.status.idle":"2023-07-16T20:38:20.642440Z","shell.execute_reply.started":"2023-07-16T20:38:19.510605Z","shell.execute_reply":"2023-07-16T20:38:20.641014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since SGD uses a random point to calculate the gradient, it can show oscillations. In contrast, Mini-Batch SGD is less noisy because the gradient is calculated on a random group of points. Nevertheless, both approaches may still require a huge number of iterations and may be very sensitive to the learning rate value.","metadata":{}},{"cell_type":"markdown","source":" \n\n# 4 SGD with Momentum and Nesterov Momentum","metadata":{}},{"cell_type":"markdown","source":"Another modification of gradient descent is Stochastic Gradient Descent with Momentum (SGDM). \n\n\nImagine a ball rolling down a mountain. While moving down, the ball accumulates momentum that helps it continue moving even on plateau or uphill.\n\n\nSince gradients can be noisy, the mini-batch SGD algorithm may not always continue moving in the optimal direction. Momentum helps alleviate this problem, potentially preventing from getting stuck in local minima and contributing to escaping saddle points. In SGDM, the gradient is calculated at the current step. However, the value of the accumulated gradient may cease to be relevant after a shift in momentum. As a result, the algorithm can overshoot the minimum. Nesterov Momentum, as shown in the Figure 1, addresses this issue by calculating the gradient at the approximate next position, which can make gradient calculation more accurate and can lead to faster convergence (the ability to find minimum value). ","metadata":{}},{"cell_type":"markdown","source":"\n\n\n\nFigure 1: Illustration of SGDM and Nesterov Momentum updates [[3]](#9.3).","metadata":{}},{"cell_type":"markdown","source":"Kidambi and his colleagues demonstrated that in certain problems, momentum does not increase the convergence rate of SGD and that the advantage of momentum is attributable to exploiting mini-batches [[4]](#9.4). However, in [[5]](#9.5) the authors showed that for both strongly convex and nonconvex functions SGDM can be as fast as SGD. In addition, the authors of [[6]](#9.6) demonstrated that with a large batch size, mini-batch SGDM can converge faster than mini-batch SGD for strongly convex and smooth objective functions. \n\n\nThe momentum approach in neural network optimization has been scrutinized in many papers, but there is still room for improvement momentum-based algorithms [[7]](#9.7), [[8]](#9.8), [[9]](#9.9). The authors of [[10]](#9.10) proposed a modified stochastic gradient descent with momentum that showed good results, performing better than SGDM in certain problems and better than Adam (will be mentioned further) in convolutional networks. According to authors, a new method is as simple as SGDM, but it employs momentum term more efficiently. In contrast to SGDM, older changes loose their relevance at a lower rate. In other words, a model forgets older changes not as fast as it would with SGDM. The authors also claim that the method is suitable for usage on large scale data sets and non-convex optimization problems.","metadata":{}},{"cell_type":"markdown","source":" \n\n# 5 Adaptive Gradient Descent Algorithms","metadata":{}},{"cell_type":"markdown","source":"In complex optimization problems with non-trivial objective landscape, gradients may vary across different dimensions, impeding convergence. Since a gradient can be large in one dimension (for one parameter) and small in another dimension (for a different parameter), the algorithm's converge rate may decrease due to the oscillations near minima. Adaptive methods may be useful in such cases, since they allow using different learning rates for each dimension during the learning process.\n\n\n\nIn its implementation, **AdaGrad** (Adaptive Gradient Descent) [[11]](#9.11) accumulates squared gradients, which helps adjust the learning rate of each parameter. In directions where gradient values have high variance, the learning rate will be lower than in directions with low variance. As an analogy, we can think of an example of driving a car on a broken and on a new road. In the first case we have to drive slowly and carefully, avoiding potholes, while in the second case a car can go faster. This approach is effective on steep and plateau-like areas where we want to move slower and faster respectively. However, since squared gradients are accumulated from the beginning of the learning process, the learning rate may eventually start to decrease significantly. As a result, AdaGrad can get stuck near a minimum.\n\n\n**RMSProp** [[12]](#9.12) uses the same idea as AdaGrad, but it also modifies the accumulated gradient. The algorithm limits the oscillations in the vertical direction and therefore can use a larger learning rate. Unlike Adagrad, RMSProp does not consider past accumulated gradients and usually has a higher convergence rate.\n\n\nA new method **NRMSProp**, proposed in [[13]](#9.13), is a modification of RMSProp that incorporates Nestrov approach as well. This method keeps knowledge about the previous gradients and calculates exponential moving average that helps algorithm be more responsive to recent changes, which helps it perform better than the original RMSProp. According to the authors, NRMSProp shows high accuracy and fast convergence without significant complexity.\n\n\nAdaptive and momentum approaches do not contradict each other, and thus **Adam** (Adaptive Moment Estimation) [[14]](#9.14) integrates the advantages of both RMSProp and AdaGrad. With certain hyperparameters, Adam is equivalent to AdaGrad. Like AdaGrad, it uses different learning rates, but it also incorporates the momentum approach. This method has shown fast convergence, and is considered one of the most popular algorithms. Until recently, there has not been shown whether Adam performs better than SGD [[15]](#9.15). However, the authors of [[15]](#9.15) demonstrated that Adam converges faster than SGD.\n\n\n\nThere are a lot of other gradient-based optimizers that are not considered in the current essay.","metadata":{}},{"cell_type":"markdown","source":" \n\n# 6 MoMo: Momentum Models for Adaptive Learning Rates","metadata":{}},{"cell_type":"markdown","source":"In [[16]](#9.16), the authors addressed the problem of hyperparameters (tunable parameters) tuning caused by computing complexity and proposed adaptive learning rates that can be used for momentum-based models. According to authors and [[17]](#9.17), choosing an appropriate learning rate is as important as choosing an optimizer. In their approach, the authors of [[16]](#9.16) suggest dealing with the model instead of dealing with the loss function, which can reduce the complexity of the task. Their method is supposed to be more versatile compared to other momentum-based methods, since it simplifies the model, which helps converge smarter and be more adaptable. Getting back to the rolling ball analogy, we can say that this method helps the ball estimate further landscape and adjust its route to roll down faster and smarter based on what the ball has already learned. \n\nThe authors demonstrated that with lower learning rate values, MoMo models' performance is comparable with the SGDM's and Adam's performance, but with larger values, MoMo models perform better (Figures 2-5).","metadata":{}},{"cell_type":"markdown","source":"![\"Drawing\"](\"https://github.com/fabian-sp/MoMo/blob/main/png/cifar10_resnet20.png?raw=true\")
\n\nFigure 2: ResNet20 for CIFAR10 validation accuracy for different learning rates on a fixed number of epochs [[18]](#9.18).\n","metadata":{}},{"cell_type":"markdown","source":"![\"Drawing\"](\"https://github.com/fabian-sp/MoMo/blob/main/png/cifar100_resnet110.png?raw=true\")
\n\nFigure 3: ResNet110 for CIFAR100 validation accuracy for different learning rates on a fixed number of epochs [[18]](#9.18).\n","metadata":{}},{"cell_type":"markdown","source":"![\"Drawing\"](\"https://github.com/fabian-sp/MoMo/blob/main/png/cifar10_resnet20_training.png?raw=true\")
\n\nFigure 4: ResNet20 for CIFAR10 validation accuracy over training comparison for the three learning rate values that lead to the best validation scores from Figures 2-3 [[18]](#9.18).\n","metadata":{}},{"cell_type":"markdown","source":"![\"Drawing\"](\"https://github.com/fabian-sp/MoMo/blob/main/png/cifar100_resnet110_training.png?raw=true\")
\n\nFigure 5: ResNet110 for CIFAR100 validation accuracy over training comparison for the three learning rate values that lead to the best validation scores from Figures 2-3 [[18]](#9.18).\n","metadata":{}},{"cell_type":"markdown","source":"The authors maintain that there were two interesting observations: 1. MoMo makes a learning decay automatically, so a user is not supposed to make any choice regarding learning rate schedule; 2. MoMo starts slowly, with a low learning rate, and then increases it. In the context of rolling ball that means that the ball starts to move slowly and carefully, but as it gets a better understanding of the landscape, it increases the speed. These factors contribute to a more efficient learning. However, the convergence guarantee of this approach is yet to be shown [[16]](#9.16).\n","metadata":{}},{"cell_type":"markdown","source":" \n\n# 7 Sophia: Scalable Stochastic Second-Order Optimizer","metadata":{}},{"cell_type":"markdown","source":"Another popular group of neural network optimizers is second-order methods. These methods employ second derivatives that provide information about curvature, which reduces the number of training steps and contributes to a higher convergence rate. Neural networks usually deal with complex functions that have a variety of second derivatives. The advantage of using second derivatives in neural network optimization is that second derivatives can help determine whether a landscape point is an extremum or a saddle point [[19]](#9.19). While second-order methods tend to converge more effectively than first-order methods, they are computationally expensive and therefore are rarely used. \n\nThe authors of [[20]](#9.20) proposed a new second-order clipped stochastic optimization (Sophia) that performs better than many popular methods. Sophia algorithm only estimates the diagonal of the Hessian matrix (a collection of second derivatives), which makes it manageable and suitable for a variety of problems. Moreover, there is also a clipping mechanism that imposes restrictions on model's update. In contrast to other second-order methods ([[21]](#9.21), [[22]](#9.22)) that update derivatives at each step, Sophia does so every 10 steps, which contributes to its efficiency. In its algorithm, Sophia uses exponential moving average of the gradient that is divided by exponential moving average of the Hessian's diagonal estimation. Since a gradient provides information about how loss function changes when we update model's parameters and Hessian provides information about how the gradient changes, this trick gives more information about the landscape. Although Sophia is based on a simple idea, it has shown efficiency on large data sets. According to authors, the proposed method is faster and more efficient than Adam in certain problems. In addition, it does not require a large amount memory, which is another advantage [[20]](#9.20).","metadata":{}},{"cell_type":"markdown","source":" \n\n# 8 Conclusion","metadata":{}},{"cell_type":"markdown","source":"In this essay, we have discussed several popular gradient-based optimizers and some novel techniques. The progress in neural network optimization should lead to meaningful improvements in training AI models, making the training process faster and less expensive. Although some empirical results are not conclusive and some findings may require further investigation, the community has made a huge step in this area. It should be noted that development and modifications of optimization methods remain an important problem in the machine learning community. New optimizers can help models exhibit better generalization ability faster. Problems with tremendous datasets may also benefit from new and more effective optimizers. Therefore, scientists should continue working on improving existing and creating fundamentally new optimization problems, which undoubtedly makes a great contribution to the machine learning community. \n","metadata":{}},{"cell_type":"markdown","source":" \n\n# 9 References\n\n \n[1] Bottou, L. (2010). Large-Scale Machine Learning with Stochastic Gradient Descent. In: Lechevallier, Y., Saporta, G. (eds) Proceedings of COMPSTAT'2010. Physica-Verlag HD. https://doi.org/10.1007/978-3-7908-2604-3_16\n\n \n[2] Liu Ziyin, Kangqiao Liu, Takashi Mori, Masahito Ueda. Strength of Minibatch Noise in SGD. ICLR 2022 https://doi.org/10.48550/arXiv.2102.05375\n\n \n[3] Stanford University CS231n: Deep Learning for Computer Vision. https://cs231n.github.io/neural-networks-3/\n\n \n[4] Rahul Kidambi, Praneeth Netrapalli, Prateek Jain, Sham M. Kakade. On the insufficiency of existing momentum schemes for Stochastic Optimization (2018) https://doi.org/10.48550/arXiv.1803.05591\n\n \n[5] Yanli Liu, Yuan Gao, Wotao Yin. An Improved Analysis of Stochastic Gradient Descent with Momentum (2020) https://doi.org/10.48550/arXiv.2007.07989 \n\n \n[6] Kejie Tang, Weidong Liu, Yichen Zhang. Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normality (2023) https://doi.org/10.48550/arXiv.2305.17665\n\n \n[7] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. Proceedings of the 30th International Conference on International Conference on Machine Learning. JMLR:W&CP volume 28, 2013 https://proceedings.mlr.press/v28/sutskever13.html\n\n \n[8] Todd K. Leen and Genevieve B. Orr. Optimal Stochastic Search and Adaptive Momentum. Advances in Neural Information Processing Systems 6 (NIPS 1993) https://proceedings.neurips.cc/paper_files/paper/1993/hash/e57c6b956a6521b28495f2886ca0977a-Abstract.html\n\n \n[9] Polyak, B.T. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, Volume 4, Issue 5, 1964, Pages 1-17 https://www.sciencedirect.com/science/article/abs/pii/0041555364901375\n\n \n[10] Liang Liu, Xiaopeng Luo. A New Accelerated Stochastic Gradient Method with Momentum (2020) https://doi.org/10.48550/arXiv.2006.00423 \n\n \n[11] John Duchi, et al. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. (2011) https://www.researchgate.net/publication/220320677_Adaptive_Subgradient_Methods_for_Online_Learning_and_Stochastic_Optimization\n\n \n[12] Hinton, G., Srivastava, N., & Swersky, K. Lecture 6a: Overview of Mini-Batch Gradient Descent.\nhttps://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf\n\n \n[13] Reham Elshamy, Osama Abu-Elnasr, Mohamed Elhoseny & Samir Elmougy. Improving the efficiency of RMSProp optimizer by utilizing Nestrove in deep learning (2023) https://www.nature.com/articles/s41598-023-35663-x\n\n \n[14] Diederik P. Kingma, Jimmy Ba Adam: A Method for Stochastic Optimization (2014) https://doi.org/10.48550/arXiv.1412.6980\n\n \n[15] Bohan Wang, Yushun Zhang, Huishuai Zhang, Qi Meng, Zhi-Ming Ma, Tie-Yan Liu, Wei Chen. Provable Adaptivity in Adam (2022) https://doi.org/10.48550/arXiv.2208.09900\n\n \n[16] Fabian Schaipp, Ruben Ohana, Michael Eickenberg, Aaron Defazio, Robert M. Gower. MoMo: Momentum Models for Adaptive Learning Rates (2023) https://doi.org/10.48550/arXiv.2305.07583\n\n \n[17] Robin M Schmidt, Frank Schneider, and Philipp Hennig. Descending through a crowded valley - benchmarking deep learning optimizers. Proceedings of the 38th International Conference on Machine Learning, PMLR 139:9367-9376, 2021 http://proceedings.mlr.press/v139/schmidt21a.html\n\n \n[18] https://github.com/fabian-sp/MoMo/tree/main/png\n\n \n[19] Ian Goodfellow, Aaron Courville, Yoshua Bengio. Deep Learning. MIT Press (2016)\n\n \n[20] Hong Liu, Zhiyuan Li, David Hall, Percy Liang, Tengyu Ma. Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training (2023) https://doi.org/10.48550/arXiv.2305.14342\n\n \n[21] Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, Michael W. Mahoney. Adahessian: An Adaptive Second Order Optimizer for Machine Learning (2021) https://doi.org/10.48550/arXiv.2006.00719\n\n \n[22] Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, Yoram Singer. Scalable Second Order Optimization for Deep Learning (2021) https://doi.org/10.48550/arXiv.2002.09018\n","metadata":{}},{"cell_type":"code","source":"submission=pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/sample_submission.csv\")\n\nsubmission.loc[0]['value']='Other'\nsubmission.loc[1]['value']='https://www.kaggle.com/pluvias/kaggle-ai-report-optimization'\nsubmission.loc[2]['value']='https://www.kaggle.com/code/constantinruchkin/variational-approach-in-semi-supervised-learning/comments#2334513'\nsubmission.loc[3]['value']='https://www.kaggle.com/code/ahsuna123/kaggle-ai-report-healthcare-surge/comments#2335363'\nsubmission.loc[4]['value']='https://www.kaggle.com/code/ojaekomo/kaggle-essay-audio-data/comments#2335804'\n\nsubmission.to_csv('submission.csv', index=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-07-16T20:38:20.644842Z","iopub.execute_input":"2023-07-16T20:38:20.645271Z","iopub.status.idle":"2023-07-16T20:38:20.657781Z","shell.execute_reply.started":"2023-07-16T20:38:20.645233Z","shell.execute_reply":"2023-07-16T20:38:20.656810Z"},"trusted":true},"execution_count":null,"outputs":[]}]}