{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":50947,"databundleVersionId":5741538,"sourceType":"competition"},{"sourceId":64148,"databundleVersionId":7669720,"sourceType":"competition"},{"sourceId":11371,"sourceType":"modelInstanceVersion","modelInstanceId":5171}],"dockerImageVersionId":30664,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#

Gemma LLM Fine-Tuning/LoRA:Q&A Summaries

\n\nPublished on March 3, 2024\n \nAuthor: **Jocelyn C. Dumlao**\n","metadata":{}},{"cell_type":"markdown","source":"![Google launches Gemma: new open source AI tool for developers - ReadWrite](https://readwrite.com/wp-content/uploads/2024/02/Google-launches-Gemma-new-open-source-AI-tool-for-developers-825x500.png)\n\nImage Source:[Google launches Gemma: new open source AI tool for developers - ReadWrite](https://www.google.com/url?sa=i&url=https%3A%2F%2Freadwrite.com%2Fgoogle-launches-gemma-new-open-source-ai-tool-for-developers%2F&psig=AOvVaw0zjVPbRd-4SHL4U9PvhAIa&ust=1709621238763000&source=images&cd=vfe&opi=89978449&ved=0CBMQjRxqFwoTCMDHqseB2oQDFQAAAAAdAAAAABAF)\n\n\n\n","metadata":{}},{"cell_type":"markdown","source":"#

What is Gemma LLM?

\n\n**Gemma, the remarkable family of open-source large language models (LLMs) by Google, extends an invitation to developers and researchers, unlocking the vast potential of AI at their disposal. Delving into the world of Gemma involves understanding how to seamlessly install and run the models. Furthermore, you can leverage the power of pre-trained models, accessing a wealth of knowledge and linguistic proficiency. Embark on a journey to explore Gemma's capabilities, empowering yourself to build bespoke applications that harness the diverse strengths of these impressive language models.**","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-04T22:28:53.519855Z","iopub.execute_input":"2024-03-04T22:28:53.520237Z","iopub.status.idle":"2024-03-04T22:28:54.104566Z","shell.execute_reply.started":"2024-03-04T22:28:53.520200Z","shell.execute_reply":"2024-03-04T22:28:54.103085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Install Dependencies

\n\n ✨ Install Keras and KerasNLP\n****","metadata":{}},{"cell_type":"code","source":"%%capture\n!pip install -U keras-nlp\n!pip install -U keras>=3","metadata":{"execution":{"iopub.status.busy":"2024-03-04T22:29:00.994508Z","iopub.execute_input":"2024-03-04T22:29:00.996068Z","iopub.status.idle":"2024-03-04T22:29:39.969438Z","shell.execute_reply.started":"2024-03-04T22:29:00.996001Z","shell.execute_reply":"2024-03-04T22:29:39.967486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" ✨ Select backend\n****","metadata":{}},{"cell_type":"code","source":"import os\nos.environ[\"KERAS_BACKEND\"] = \"jax\" \nos.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"]=\"1.00\"\n","metadata":{"execution":{"iopub.status.busy":"2024-03-04T22:30:03.606400Z","iopub.execute_input":"2024-03-04T22:30:03.606967Z","iopub.status.idle":"2024-03-04T22:30:03.615892Z","shell.execute_reply.started":"2024-03-04T22:30:03.606925Z","shell.execute_reply":"2024-03-04T22:30:03.613535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Import packages

\n","metadata":{}},{"cell_type":"code","source":"import keras\nimport keras_nlp\nimport numpy as np","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-04T22:30:08.458351Z","iopub.execute_input":"2024-03-04T22:30:08.458934Z","iopub.status.idle":"2024-03-04T22:30:26.063438Z","shell.execute_reply.started":"2024-03-04T22:30:08.458893Z","shell.execute_reply":"2024-03-04T22:30:26.062131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Load Dataset

","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv', nrows =1000)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-04T22:31:18.930637Z","iopub.execute_input":"2024-03-04T22:31:18.931601Z","iopub.status.idle":"2024-03-04T22:31:19.064423Z","shell.execute_reply.started":"2024-03-04T22:31:18.931556Z","shell.execute_reply":"2024-03-04T22:31:19.063202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" ✨ Let’s have a look at the content under Write-up column.\n****","metadata":{}},{"cell_type":"code","source":"# Display only the 'Writeup' column\nwriteup_df = df[['Writeup']]\n\n# Display the DataFrame\nwriteup_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-04T22:31:23.787237Z","iopub.execute_input":"2024-03-04T22:31:23.787679Z","iopub.status.idle":"2024-03-04T22:31:23.811107Z","shell.execute_reply.started":"2024-03-04T22:31:23.787647Z","shell.execute_reply":"2024-03-04T22:31:23.809577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['Writeup'][0]","metadata":{"execution":{"iopub.status.busy":"2024-03-04T22:31:27.456651Z","iopub.execute_input":"2024-03-04T22:31:27.457202Z","iopub.status.idle":"2024-03-04T22:31:27.468365Z","shell.execute_reply.started":"2024-03-04T22:31:27.457156Z","shell.execute_reply":"2024-03-04T22:31:27.466558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Before initiating the fine-tuning process with Gemma, it's essential to ensure proper formatting of the data. This step involves preparing the input data in a structured and compatible format that aligns with Gemma's requirements. By adhering to a predefined format, the fine-tuning process becomes more seamless and effective, optimizing the collaboration between the data and Gemma for enhanced model refinement.**","metadata":{}},{"cell_type":"code","source":"# Initialize an empty list to store the formatted instructions and responses\nformatted_data = []\n\n# Iterate over each row of the DataFrame\nfor index, row in df.iterrows():\n # Extract messages\n messages = row['Writeup']\n\n # Check if 'messages' is a float (type float('nan') is also considered)\n if isinstance(messages, float):\n # Handle the case where 'messages' is a float, for example, by converting it to an empty string\n messages = str(messages)\n \n # Assuming 'Writeup' contains HTML content, you can use BeautifulSoup to parse it\n from bs4 import BeautifulSoup\n\n # Parse the HTML content\n soup = BeautifulSoup(messages, 'html.parser')\n\n # Extract content from the first message if it exists, otherwise set it to an empty string\n instruction_elements = soup.find_all('p')\n instruction_content = instruction_elements[0].get_text() if len(instruction_elements) > 0 else ''\n\n # Extract content from the second message if it exists, otherwise set it to an empty string\n response_elements = soup.find_all('p')\n response_content = response_elements[1].get_text() if len(response_elements) > 1 else ''\n\n # Create the formatted string\n formatted_string = f'Instruction:\\n{instruction_content}\\n\\nResponse:\\n{response_content}'\n\n # Append the formatted string to the list\n formatted_data.append(formatted_string)\n","metadata":{"execution":{"iopub.status.busy":"2024-03-04T22:31:31.271754Z","iopub.execute_input":"2024-03-04T22:31:31.272299Z","iopub.status.idle":"2024-03-04T22:31:33.420010Z","shell.execute_reply.started":"2024-03-04T22:31:31.272254Z","shell.execute_reply":"2024-03-04T22:31:33.418226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" ✨ Let’s have a look at data after it’s formatted\n****","metadata":{}},{"cell_type":"code","source":"# Print the formatted data\nfor item in formatted_data:\n print(item)\n break","metadata":{"execution":{"iopub.status.busy":"2024-03-04T22:31:37.561234Z","iopub.execute_input":"2024-03-04T22:31:37.562176Z","iopub.status.idle":"2024-03-04T22:31:37.568538Z","shell.execute_reply.started":"2024-03-04T22:31:37.562130Z","shell.execute_reply":"2024-03-04T22:31:37.567157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**For the purpose of learning and fine-tuning, I have decided to utilize a limited set of 200 instances. This deliberate choice of a smaller dataset, specifically for tuning, allows for a more focused and efficient learning experience. By concentrating on a subset of instances, I can streamline the tuning process, making it more manageable and conducive to the learning objectives at hand.**\n","metadata":{}},{"cell_type":"code","source":"formatted_data = formatted_data[:200]","metadata":{"execution":{"iopub.status.busy":"2024-03-04T22:31:41.887140Z","iopub.execute_input":"2024-03-04T22:31:41.887631Z","iopub.status.idle":"2024-03-04T22:31:41.893334Z","shell.execute_reply.started":"2024-03-04T22:31:41.887596Z","shell.execute_reply":"2024-03-04T22:31:41.892370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Create a Model

\n\n\n**In KerasNLP, you can leverage various pre-implemented model architectures. One such model is GemmaCausalLM, designed for end-to-end Gemma-based causal language modeling. A causal language model anticipates the next token in a sequence by considering the context of preceding tokens.**\n\n**To create a model using GemmaCausalLM, you can utilize the 'from_preset' method. This method streamlines the model creation process, allowing you to quickly set up and configure your GemmaCausalLM model for effective causal language modeling.**\n\n","metadata":{}},{"cell_type":"code","source":"gemma_model = keras_nlp.models.GemmaCausalLM.from_preset(\"gemma_2b_en\")","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-04T22:31:45.799752Z","iopub.execute_input":"2024-03-04T22:31:45.800969Z","iopub.status.idle":"2024-03-04T22:33:40.974122Z","shell.execute_reply.started":"2024-03-04T22:31:45.800921Z","shell.execute_reply":"2024-03-04T22:33:40.972938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Inference Before Fine Tuning

\n\n**Inference Before Fine Tuning\" refers to using a machine learning model for predictions or outputs after its initial training (pre-training) but before any task-specific adjustments (fine-tuning). This stage assesses the model's performance on new data before refining it for a specific task.**\n\n**The exploration involves querying the model with a diverse array of prompts to observe and analyze its responses. This approach allows for a comprehensive understanding of the model's behavior, shedding light on how it interprets and generates text in response to different inputs. By engaging in this interactive process, users can gain valuable insights into the model's capabilities and nuances in handling varied prompts.**\n","metadata":{}},{"cell_type":"markdown","source":" ✨ Teach concepts from Kaggle competition solution write-ups\n\n ✨ Model Architecture and Hyperparameter Tuning:\n\n* Describe the choice of the model architecture in the Kaggle solution. What hyperparameter tuning strategies were employed to optimize the model's performance?\n****\n","metadata":{}},{"cell_type":"code","source":"prompt = \"Describe the choice of the model architecture in the Kaggle solution. What hyperparameter tuning strategies were employed to optimize the model's performance?\"\nprint(gemma_model.generate(prompt, max_length=64))","metadata":{"execution":{"iopub.status.busy":"2024-03-04T22:33:46.150771Z","iopub.execute_input":"2024-03-04T22:33:46.151220Z","iopub.status.idle":"2024-03-04T22:35:24.269613Z","shell.execute_reply.started":"2024-03-04T22:33:46.151185Z","shell.execute_reply":"2024-03-04T22:35:24.268184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" ✨ Transfer Learning and External Data Integration:\n****\n\n* Explore whether the Kaggle solution leveraged transfer learning from pre-trained models or integrated external datasets. What considerations were taken into account when combining information from different sources?\n","metadata":{}},{"cell_type":"code","source":"prompt = \"Explore whether the Kaggle solution leveraged transfer learning from pre-trained models or integrated external datasets.\"\nprint(gemma_model.generate(prompt, max_length=64))","metadata":{"execution":{"iopub.status.busy":"2024-03-04T22:35:35.721977Z","iopub.execute_input":"2024-03-04T22:35:35.722589Z","iopub.status.idle":"2024-03-04T22:37:15.088285Z","shell.execute_reply.started":"2024-03-04T22:35:35.722548Z","shell.execute_reply":"2024-03-04T22:37:15.086775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prompt = \"what considerations were taken into account when combining information from different sources?\"\nprint(gemma_model.generate(prompt, max_length=256))","metadata":{"execution":{"iopub.status.busy":"2024-03-04T22:38:44.289249Z","iopub.execute_input":"2024-03-04T22:38:44.289734Z","iopub.status.idle":"2024-03-04T22:48:09.386348Z","shell.execute_reply.started":"2024-03-04T22:38:44.289679Z","shell.execute_reply":"2024-03-04T22:48:09.385129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

Fine-Tuning

\n\n ✨ Use summary to get more info about the model:\n****\n","metadata":{}},{"cell_type":"code","source":"# Enable LoRA for the model and set the LoRA rank to 3.\ngemma_model.backbone.enable_lora(rank=3)\ngemma_model.summary()","metadata":{"execution":{"iopub.status.busy":"2024-03-04T22:48:13.443905Z","iopub.execute_input":"2024-03-04T22:48:13.444373Z","iopub.status.idle":"2024-03-04T22:48:14.111156Z","shell.execute_reply.started":"2024-03-04T22:48:13.444337Z","shell.execute_reply":"2024-03-04T22:48:14.110092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"References: \n\nhttps://ai.google.dev/gemma/docs/get_started\n\nhttps://keras.io/getting_started/\n","metadata":{}},{"cell_type":"markdown","source":"
\"Your positive feedback and upvote mean a lot! It motivates me to create more valuable content and helps others discover it too. Let's build a thriving community of knowledge-sharing. Thank you for your support! 😊\"
","metadata":{}}]}