{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":50947,"databundleVersionId":5741538,"sourceType":"competition"},{"sourceId":64148,"databundleVersionId":7669720,"sourceType":"competition"},{"sourceId":11371,"sourceType":"modelInstanceVersion","modelInstanceId":5171},{"sourceId":11372,"sourceType":"modelInstanceVersion","modelInstanceId":5388}],"dockerImageVersionId":30646,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Published on February 24, 2024. By Marília Prata, mpwolke.","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nimport re\npd.set_option('display.max_colwidth', 256)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-02-24T17:03:45.953959Z","iopub.execute_input":"2024-02-24T17:03:45.954252Z","iopub.status.idle":"2024-02-24T17:03:47.050945Z","shell.execute_reply.started":"2024-02-24T17:03:45.954226Z","shell.execute_reply":"2024-02-24T17:03:47.050027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRNpcu4rQmpRKcjQig6yljTEVt-D4HHzL2U7ufXfswfLlv7hRWct_z46HVvXtIVperfkpBxkVgAyC8xhv5By2V3bXTevAayOEAhsU_qLliDKeYVn-bFW3pn8Pxr3pwa7ncmiWoU6XiOpcbkyQ_GdurR4dWsJ42pCp2SlydX3deLn6RYcagsRHjF3ZOGQo/s16000-rw/1708520569161.webp)Times of AI","metadata":{}},{"cell_type":"markdown","source":"#Load a file (That you choose)","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('../input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv', nrows =1000)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:03:53.963411Z","iopub.execute_input":"2024-02-24T17:03:53.964361Z","iopub.status.idle":"2024-02-24T17:03:54.076891Z","shell.execute_reply.started":"2024-02-24T17:03:53.964330Z","shell.execute_reply":"2024-02-24T17:03:54.076014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Till now I will need 2 columns: Questions/instructions and Answers/Responses \n\nI would like to keep all the columns above though I don't know how to manage the questions that would be made next. Any clue? ","metadata":{}},{"cell_type":"code","source":"cols_to_drop=['Competition Launch Date','Competition URL', 'Title of Competition', 'Date of Writeup', 'Writeup URL']\ndf=df.drop(cols_to_drop,axis=1)\ndf.tail()","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2024-02-24T17:04:01.224231Z","iopub.execute_input":"2024-02-24T17:04:01.224576Z","iopub.status.idle":"2024-02-24T17:04:01.242354Z","shell.execute_reply.started":"2024-02-24T17:04:01.224549Z","shell.execute_reply":"2024-02-24T17:04:01.241361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Install Keras","metadata":{}},{"cell_type":"code","source":"# Install Keras 3 last. See https://keras.io/getting_started/ for more details.\n!pip install -q -U keras-nlp\n!pip install -q -U keras>=3\n\nimport os\n\nos.environ[\"KERAS_BACKEND\"] = \"jax\"  # Or \"torch\" or \"tensorflow\".\n# Avoid memory fragmentation on JAX backend.\nos.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"]=\"1.00\"\n\nimport keras\nimport keras_nlp","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:04:07.614633Z","iopub.execute_input":"2024-02-24T17:04:07.614991Z","iopub.status.idle":"2024-02-24T17:04:51.071901Z","shell.execute_reply.started":"2024-02-24T17:04:07.614962Z","shell.execute_reply":"2024-02-24T17:04:51.071105Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Gemma_2b_en","metadata":{}},{"cell_type":"code","source":"%%time\n\ngemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(\"gemma_2b_en\")","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:04:55.530964Z","iopub.execute_input":"2024-02-24T17:04:55.531615Z","iopub.status.idle":"2024-02-24T17:05:53.353732Z","shell.execute_reply.started":"2024-02-24T17:04:55.531565Z","shell.execute_reply":"2024-02-24T17:05:53.352774Z"},"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Ask one random question\n\nNot from the dataset and one that is from the dataset. If Gemma isn't communicating yet with the csv file the model obviously won't provide a good question. Since we didn't feed it.","metadata":{}},{"cell_type":"code","source":"%%time\nprint(gemma_lm.generate(\"What are Kaggle Competition Writeups?\", max_length=256))","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:06:30.823082Z","iopub.execute_input":"2024-02-24T17:06:30.823826Z","iopub.status.idle":"2024-02-24T17:07:11.833442Z","shell.execute_reply.started":"2024-02-24T17:06:30.823796Z","shell.execute_reply":"2024-02-24T17:07:11.832488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Let's try to load in the Q&A (questions/answers) dataset","metadata":{}},{"cell_type":"code","source":"#By York Yong https://www.kaggle.com/code/yorkyong/gemma-trial-ace-a-data-science-interview\n\nQNA_dataset = []\n    \nfor index, row in df.iterrows():\n    question, answer = row['Title of Writeup'], row['Writeup']\n    template = (f\"Title of Writeup:\\n{question}\\n\\nWriteup:\\n{answer}\")\n    QNA_dataset.append(template)","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:07:20.663210Z","iopub.execute_input":"2024-02-24T17:07:20.663612Z","iopub.status.idle":"2024-02-24T17:07:20.741000Z","shell.execute_reply.started":"2024-02-24T17:07:20.663582Z","shell.execute_reply":"2024-02-24T17:07:20.740068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Enable LoRA","metadata":{}},{"cell_type":"code","source":"# Enable LoRA for the model and set the LoRA rank to 64.\ngemma_lm.backbone.enable_lora(rank=64)","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:07:25.733826Z","iopub.execute_input":"2024-02-24T17:07:25.734728Z","iopub.status.idle":"2024-02-24T17:07:26.137567Z","shell.execute_reply.started":"2024-02-24T17:07:25.734695Z","shell.execute_reply":"2024-02-24T17:07:26.136757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Control Memory usage","metadata":{}},{"cell_type":"code","source":"# Limit the input sequence length to 512 (to control memory usage).\ngemma_lm.preprocessor.sequence_length = 512\n# Use AdamW (a common optimizer for transformer models).\noptimizer = keras.optimizers.AdamW(\n    learning_rate=5e-5,\n    weight_decay=0.01,\n)\n# Exclude layernorm and bias terms from decay.\noptimizer.exclude_from_weight_decay(var_names=[\"bias\", \"scale\"])\n\ngemma_lm.compile(\n    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n    optimizer=optimizer,\n    weighted_metrics=[keras.metrics.SparseCategoricalAccuracy()],\n)","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:07:31.018763Z","iopub.execute_input":"2024-02-24T17:07:31.019127Z","iopub.status.idle":"2024-02-24T17:07:31.100564Z","shell.execute_reply.started":"2024-02-24T17:07:31.019100Z","shell.execute_reply":"2024-02-24T17:07:31.099741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Now wait for the Only One Epoch Reading Gemma's Order : )\n\nI think it was 30 minutes of waiting below.","metadata":{}},{"cell_type":"markdown","source":"![](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQDskblSM4cEF8lJvSv_bj1KSSTpkx2bOo4EUGZEE5E7N1yAi5IIr6mcockOhARVU6qnGE&usqp=CAU)Tik Tok","metadata":{}},{"cell_type":"code","source":"%%time\n\ngemma_lm.fit(QNA_dataset, epochs=1, batch_size=1)","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:07:36.185003Z","iopub.execute_input":"2024-02-24T17:07:36.185768Z","iopub.status.idle":"2024-02-24T17:31:46.994107Z","shell.execute_reply.started":"2024-02-24T17:07:36.185734Z","shell.execute_reply":"2024-02-24T17:31:46.993238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Which are the most frequent words on Title of Writeup?\n\nDatetime answers???","metadata":{}},{"cell_type":"code","source":"%%time\nprint(gemma_lm.generate(\"Which are the most frequent words on Title of Writeup?\", max_length=256))","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:32:22.831641Z","iopub.execute_input":"2024-02-24T17:32:22.832011Z","iopub.status.idle":"2024-02-24T17:32:47.295075Z","shell.execute_reply.started":"2024-02-24T17:32:22.831984Z","shell.execute_reply":"2024-02-24T17:32:47.294087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#The most common titles on Title of Writeup?\n\nOne kaggler answer? It was about the titles of the Writeups.","metadata":{}},{"cell_type":"code","source":"print(gemma_lm.generate(\"Which are the most common titles on Title of Writeup?\", max_length=256))","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:33:05.900145Z","iopub.execute_input":"2024-02-24T17:33:05.900874Z","iopub.status.idle":"2024-02-24T17:33:09.766036Z","shell.execute_reply.started":"2024-02-24T17:33:05.900839Z","shell.execute_reply":"2024-02-24T17:33:09.765044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Which are the most frequent words on Kaggle Writeups?\n\nGemma wrote a snippet/helper function?","metadata":{}},{"cell_type":"code","source":"%%time\nprint(gemma_lm.generate(\"Which are the most frequent words on Kaggle Writeups?\", max_length=256))","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:33:26.823101Z","iopub.execute_input":"2024-02-24T17:33:26.824000Z","iopub.status.idle":"2024-02-24T17:33:38.777482Z","shell.execute_reply.started":"2024-02-24T17:33:26.823963Z","shell.execute_reply":"2024-02-24T17:33:38.776476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#How many times The word Medal appeared on Writeups?\n\n150 times?\n\n\"The medal is not a big thing, but it’s a big thing in the context of the competition.\" \nYes, tell that for those that lose competitions medals for a tiny difference.","metadata":{}},{"cell_type":"code","source":"print(gemma_lm.generate(\"How many times the word medal appeared on Writeup?\", max_length=256))","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:33:50.749912Z","iopub.execute_input":"2024-02-24T17:33:50.750275Z","iopub.status.idle":"2024-02-24T17:34:02.763415Z","shell.execute_reply.started":"2024-02-24T17:33:50.750245Z","shell.execute_reply":"2024-02-24T17:34:02.762252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Which was the 1st Kaggle Writeup?\n\nGemma answered about someone's 1st place and not about the 1st Writeup. Though, I don't know if the model has access to the other columns that I've dropped.","metadata":{}},{"cell_type":"code","source":"print(gemma_lm.generate(\"Which was the 1st Kaggle Writeup?\", max_length=256))","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:34:25.987366Z","iopub.execute_input":"2024-02-24T17:34:25.988323Z","iopub.status.idle":"2024-02-24T17:34:35.255302Z","shell.execute_reply.started":"2024-02-24T17:34:25.988286Z","shell.execute_reply":"2024-02-24T17:34:35.254253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Which Kaggler wrote more Writeups? The funnier answer:\n\n\"This is not a competition. I am not trying to find out.\"\n\n\"I have a feeling that there are some Kaggler who have written more writeups but they are very private. So I am not trying to get their names.\"\n\nUnfortunately, Gemma for your information, it's an Analytical competition. Besides, Writeups aren't private. Kagglers publish their solutions, so Writeups are Public Not Private as you said. ","metadata":{}},{"cell_type":"code","source":"print(gemma_lm.generate(\"Which Kaggler wrote more Writeups?\", max_length=256))","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:35:15.577745Z","iopub.execute_input":"2024-02-24T17:35:15.578155Z","iopub.status.idle":"2024-02-24T17:35:19.232967Z","shell.execute_reply.started":"2024-02-24T17:35:15.578122Z","shell.execute_reply":"2024-02-24T17:35:19.231932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Yes, it's a Competition Gemma. Writeups aren't private Gemma!","metadata":{}},{"cell_type":"markdown","source":"#How many 4th place solution on Title of Writeup?\n\nIs there a value_counts for models? I'm sorry but \"Thank you for the kind words\" isn't the correct answer for \"how many anything\"","metadata":{}},{"cell_type":"code","source":"print(gemma_lm.generate(\"How many 4th place solution on Title of Writeup?\", max_length=256))","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:39:46.659107Z","iopub.execute_input":"2024-02-24T17:39:46.659852Z","iopub.status.idle":"2024-02-24T17:39:50.966395Z","shell.execute_reply.started":"2024-02-24T17:39:46.659810Z","shell.execute_reply":"2024-02-24T17:39:50.965474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#How many 'thanks to the hosts' on Writeups?\n\nAnother fun answer:\n\n\"I've got some 'thanks to the host' in 2nd and 3rd place, I'm happy with that.\nBut it is strange to me that 5th and 6th place have no thanks to the hosts.\nMaybe they were not able to read the writeup. Anyway, congratulations. Thanks a lot.\"","metadata":{}},{"cell_type":"code","source":"print(gemma_lm.generate(\"How many 'thanks to the hosts' on Writeups?\", max_length=256))","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:42:23.609832Z","iopub.execute_input":"2024-02-24T17:42:23.610772Z","iopub.status.idle":"2024-02-24T17:42:33.230002Z","shell.execute_reply.started":"2024-02-24T17:42:23.610735Z","shell.execute_reply":"2024-02-24T17:42:33.228859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Why they share Public and Private scores on Writeups?\n\n\"I have seen some players that have a public score and a private score, but the private one has a higher level than the public one. I think this is just a mistake. I've checked it and it's just a mistake.\"\n\nI've No experience to comment that answer.","metadata":{}},{"cell_type":"code","source":"print(gemma_lm.generate(\"Why they share Public and Private scores on Writeups?\", max_length=256))","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:45:16.792303Z","iopub.execute_input":"2024-02-24T17:45:16.792772Z","iopub.status.idle":"2024-02-24T17:45:20.655084Z","shell.execute_reply.started":"2024-02-24T17:45:16.792739Z","shell.execute_reply":"2024-02-24T17:45:20.654099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Gemma made a GPU question! Are we changing places Gemma? Lol\n\n\"I have a question, what is a difference between 2080ti, 3080, and 6090?\" Check below","metadata":{}},{"cell_type":"markdown","source":"#Have competitors mentioned GPUs on Writeups? Gemma asking question!!!\n\n\"Yes, it was discussed in a lot of places, but it seems to be not a big deal for most of people.\"  Speak for yourself Gemma, Accelerators are a Big/huge deal for competitors.\n\n\"I have a question, what is a difference between 2080ti, 3080, and 6090?\"\n\nNow, you're changing place with the humam Gemma.","metadata":{}},{"cell_type":"code","source":"print(gemma_lm.generate(\"Have competitors mentioned GPUs on Writeups?\", max_length=256))","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:48:54.601776Z","iopub.execute_input":"2024-02-24T17:48:54.602403Z","iopub.status.idle":"2024-02-24T17:49:06.696948Z","shell.execute_reply.started":"2024-02-24T17:48:54.602363Z","shell.execute_reply":"2024-02-24T17:49:06.695829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Have competitors complained on Writeups?\n\n\"I've seen this happen in the past, but I'd like to hear from people who are still active in this forum.\"\n\n\"It'd be great if you can share what happened, what you did to resolve the issue, and how you avoided it in your next project.\"\n\nKagglers are often complaining about something. (Kaggle team will agree with that) Many times, the complaint is valid. Others have no justification at all. ","metadata":{}},{"cell_type":"code","source":"print(gemma_lm.generate(\"Have competitors complained on Writeups?\", max_length=256))","metadata":{"execution":{"iopub.status.busy":"2024-02-24T17:51:28.425677Z","iopub.execute_input":"2024-02-24T17:51:28.426067Z","iopub.status.idle":"2024-02-24T17:51:31.867506Z","shell.execute_reply.started":"2024-02-24T17:51:28.426041Z","shell.execute_reply":"2024-02-24T17:51:31.866488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Thanks for Delivering my Order Gemma. Now I'm quite satisfied : )","metadata":{}},{"cell_type":"markdown","source":"#Acknowledgements:\n\nFine-tune Gemma models in Keras using LoRA https://ai.google.dev/gemma/docs/lora_tuning#load_dataset\n\nYork Yong https://www.kaggle.com/code/yorkyong/gemma-trial-ace-a-data-science-interview\n\nNilay Chauhan https://www.kaggle.com/code/nilaychauhan/fine-tune-gemma-models-in-keras-using-lora\n\nCarl McBride Ellis https://www.kaggle.com/code/carlmcbrideellis/gemma-i-am-replacing-myself-with-an-llm","metadata":{}}]}