{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"tpu1vmV38","dataSources":[{"sourceId":50947,"databundleVersionId":5741538,"sourceType":"competition"},{"sourceId":64148,"databundleVersionId":7669720,"sourceType":"competition"},{"sourceId":7986194,"sourceType":"datasetVersion","datasetId":9},{"sourceId":11372,"sourceType":"modelInstanceVersion","modelInstanceId":5388},{"sourceId":21690,"sourceType":"modelInstanceVersion","modelInstanceId":17963}],"dockerImageVersionId":30646,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 1. Introduction\n\nIn this notebook, Gemma 2B is integrated with LangChain to summarize Kaggle solution write-ups.<br>\nTo accomplish this, the Keras model of gemma_2b is wrapped as a custom model within LangChain.\n\nWe also demonstrate that Gemma 2B can be used for creating overall summary and comparison matrix.<br>\nAdditionally, my fine-tune variant of Gemma 2B \"MindGemmap\" is used to create mind maps which can visualize the concept of solutions.","metadata":{}},{"cell_type":"markdown","source":"# 2. Install packages\n\nFirst of all, necesarry packages are installed as follows,","metadata":{}},{"cell_type":"code","source":"!pip install -q tensorflow-cpu\n!pip install -q -U keras-nlp tensorflow-hub\n!pip install -q -U keras>=3\n!pip install -q -U langchain\n!pip install -q -U graphviz","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-31T10:07:54.545821Z","iopub.execute_input":"2024-03-31T10:07:54.546046Z","iopub.status.idle":"2024-03-31T10:08:13.223093Z","shell.execute_reply.started":"2024-03-31T10:07:54.546021Z","shell.execute_reply":"2024-03-31T10:08:13.221987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!apt update\n!apt install -y -q graphviz","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-31T10:08:13.225307Z","iopub.execute_input":"2024-03-31T10:08:13.225626Z","iopub.status.idle":"2024-03-31T10:08:15.339535Z","shell.execute_reply.started":"2024-03-31T10:08:13.225597Z","shell.execute_reply":"2024-03-31T10:08:15.338390Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Prepare Gemma Model\n## 3.1 Device Setting\nIf multiple accelerators can be used, set up them as follows,","metadata":{}},{"cell_type":"code","source":"import os\n\nos.environ[\"KERAS_BACKEND\"] = \"jax\"  # Or \"torch\" or \"tensorflow\".\n# Avoid memory fragmentation on JAX backend.\nos.environ[\"XLA_PYTHON_CLIENT_MEM_FRACTION\"]=\"1.00\"\n\nimport jax\nimport keras\nimport keras_nlp","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:08:15.341047Z","iopub.execute_input":"2024-03-31T10:08:15.341407Z","iopub.status.idle":"2024-03-31T10:08:18.768902Z","shell.execute_reply.started":"2024-03-31T10:08:15.341365Z","shell.execute_reply":"2024-03-31T10:08:18.768179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Note we use jax.devices() instead of keras.distribution.list_devices()\ndef initialize_device_mesh(\n    shape: tuple[int, int] = (1, 8), \n    batch_axis_name: str = \"batch\",\n    model_axis_name: str = \"model\"\n) -> keras.distribution.DeviceMesh:\n    \"\"\"Initializes and returns a DeviceMesh for distributing computation across devices.\n\n    Args:\n        shape: A tuple specifying the shape of the overall `DeviceMesh` \n            - `(8,)` for a data parallel only distribution,\n            - `(4, 2)` for a model+data parallel distribution.\n        batch_axis_name: A string indicating the axis name for the batch axis for DeviceMesh\n        model_axis_name: The logical name of the model axis for the `DeviceMesh`\n\n    Returns:\n        A configured DeviceMesh instance. \n            - Defaults to (1, 8) shape so that the weights are sharded across all 8 TPUs (v3-8).\n        NOTE: This API is aligned with `jax.sharding.Mesh` and `tf.dtensor.Mesh`\n            - i.e. It represents the computation devices in the global context.\n    \"\"\"\n    return keras.distribution.DeviceMesh(\n        shape=shape, \n        axis_names=[batch_axis_name, model_axis_name], \n        devices=jax.devices()\n    )\n\n\ndef configure_layout_map(\n    device_mesh: keras.distribution.DeviceMesh,\n    model_axis_name: str = \"model\"\n) -> keras.distribution.LayoutMap:\n    \"\"\"Configures and returns a LayoutMap for model weight distribution.\n\n    Args:\n        device_mesh: The `DeviceMesh` that is used to populate the `TensorLayout.device_mesh`\n        axis_name: The logical name of the model axis for the `DeviceMesh`\n\n    Returns:\n        A LayoutMap instance with predefined sharding configurations.\n    \"\"\"\n\n    # A dict-like object that maps string to `TensorLayout` instances.\n    layout_map = keras.distribution.LayoutMap(device_mesh)\n\n    # Weights that match 'token_embedding/embeddings' will be sharded on 8 TPUs\n    layout_map[\"token_embedding/embeddings\"] = (None, model_axis_name)\n    \n    # Regex to match against the query, key and value matrices in the decoder attention layers\n    layout_map[\"decoder_block.*attention.*(query|key|value).*kernel\"] = (None, model_axis_name, None)\n    \n    # etc.\n    layout_map[\"decoder_block.*attention_output.*kernel\"] = (None, None, model_axis_name)\n    layout_map[\"decoder_block.*ffw_gating.*kernel\"] = (model_axis_name, None)\n    layout_map[\"decoder_block.*ffw_linear.*kernel\"] = (None, model_axis_name)\n    \n    return layout_map\n\n\ndef update_distribution_strategy(\n    device_mesh: keras.distribution.DeviceMesh, \n    layout_map: keras.distribution.LayoutMap\n) -> None:\n    \"\"\"Loads a distributed Gemma model based on the provided device mesh and layout map.\n\n    Args:\n        model_name: The name of the Gemma model to load.\n        device_mesh: A DeviceMesh instance for model distribution.\n        layout_map: A LayoutMap instance defining how to distribute model weights.\n\n    Returns:\n        None; The keras backend is updated with the appropriate distribution strategy\n    \"\"\"\n    \n    # Shard across devices in the mesh and update distribution strategy accordingly\n    model_parallel = keras.distribution.ModelParallel(device_mesh, layout_map, batch_dim_name=\"batch\")\n    keras.distribution.set_distribution(model_parallel)\n\n\ndef do_gemma_prep(return_device_mesh=True, return_layout_map=True):\n    \"\"\"Does the necessary steps so that we can instantiate a model properly\n    \n    Args:\n        return_*: Whether to return the respective object as part of a dictionary\n            - The key is the name (*) and the value is the object itself\n    \"\"\"\n    device_mesh = initialize_device_mesh()\n    layout_map = configure_layout_map(device_mesh)\n    update_distribution_strategy(device_mesh, layout_map)\n    \n    return_map = {}\n    if not (return_device_mesh or return_layout_map):\n        pass\n    else:\n        if return_device_mesh:\n            return_map[\"device_mesh\"] = device_mesh\n        if return_layout_map:\n            return_map[\"layout_map\"] = layout_map\n    return return_map\n\n\nsetup_objects = do_gemma_prep()\nprint(f\"SETUP ARTIFACTS: \\n{setup_objects}\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-03-31T10:08:18.770152Z","iopub.execute_input":"2024-03-31T10:08:18.770732Z","iopub.status.idle":"2024-03-31T10:08:21.392206Z","shell.execute_reply.started":"2024-03-31T10:08:18.770685Z","shell.execute_reply":"2024-03-31T10:08:21.390987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.2 Import Model\nIn this notebook, Gemma Model is used locally. Gemma model can be imported as a keras Model.","metadata":{}},{"cell_type":"code","source":"%%time\ngemma_lm = keras_nlp.models.GemmaCausalLM.from_preset(\"gemma_instruct_2b_en\")","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:08:21.394570Z","iopub.execute_input":"2024-03-31T10:08:21.394855Z","iopub.status.idle":"2024-03-31T10:09:15.778543Z","shell.execute_reply.started":"2024-03-31T10:08:21.394826Z","shell.execute_reply":"2024-03-31T10:09:15.777547Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sampler = keras_nlp.samplers.GreedySampler()\ngemma_lm.compile(sampler=sampler)","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:09:15.779446Z","iopub.execute_input":"2024-03-31T10:09:15.779698Z","iopub.status.idle":"2024-03-31T10:09:15.787901Z","shell.execute_reply.started":"2024-03-31T10:09:15.779673Z","shell.execute_reply":"2024-03-31T10:09:15.787176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.3 Run gemma model directly\nGemma models can generate sentences as follows,","metadata":{}},{"cell_type":"code","source":"%%time\nprint(gemma_lm.generate(\"hi, how are you doing?\", max_length=256))","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:09:15.788737Z","iopub.execute_input":"2024-03-31T10:09:15.788968Z","iopub.status.idle":"2024-03-31T10:09:31.253194Z","shell.execute_reply.started":"2024-03-31T10:09:15.788942Z","shell.execute_reply":"2024-03-31T10:09:31.252336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.4 Define the custom model for LangChain\n\nSince keras_nlp is not compatible with LangChain, we need to define a custom model.\n\nThe details are described on the page below.<br>\nhttps://python.langchain.com/docs/modules/model_io/llms/custom_llm","metadata":{}},{"cell_type":"code","source":"from typing import Any, Optional, List, Mapping\nfrom langchain_core.callbacks.manager import CallbackManagerForLLMRun\nfrom langchain_core.language_models.llms import LLM\n\nclass GemmaLC(LLM):\n\n    model: Any = None\n    n: int = None\n\n    def __init__(self, keras_model, n):\n        super(GemmaLC, self).__init__()\n        self.model = keras_model\n        self.n = n\n\n    @property\n    def _llm_type(self) -> str:\n        return \"Gemma\"\n\n    def _call(self, prompt: str, stop: Optional[List[str]] = None, **kwargs) -> str:\n\n        generated = self.model.generate(prompt, max_length=self.n)\n\n        # post-processing to extract the result of summarization\n        split_string = generated.split(\"SUMMARY:\", 1)\n        if len(split_string) > 1:            \n            return split_string[1].lstrip('\\n')\n        else:\n            return generated.lstrip('\\n')\n\n    @property\n    def _identifying_params(self) -> Mapping[str, Any]:\n        \"\"\"Get the identifying parameters.\"\"\"\n        return {\"n\": self.n}","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:09:31.254313Z","iopub.execute_input":"2024-03-31T10:09:31.254640Z","iopub.status.idle":"2024-03-31T10:09:31.576337Z","shell.execute_reply.started":"2024-03-31T10:09:31.254611Z","shell.execute_reply":"2024-03-31T10:09:31.575497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gemma_lc = GemmaLC(gemma_lm, 1024)","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:09:31.577364Z","iopub.execute_input":"2024-03-31T10:09:31.577656Z","iopub.status.idle":"2024-03-31T10:09:31.586025Z","shell.execute_reply.started":"2024-03-31T10:09:31.577626Z","shell.execute_reply":"2024-03-31T10:09:31.585312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Create a summarizer\n## 4.1 Prompt Engneering\nSince solution write-ups include many technical terms, similar to scientific papers, the default prompt tends to yield outputs like \"The paper introduces ...\".<br>\nTo address this, I've created the following prompt that processes summarization with the consideration that the document is a part of Kaggle write-up.\n\nIn addition, we used a prompt which specifies the number of tokens to control the length of final summary.","metadata":{}},{"cell_type":"code","source":"from langchain_core.prompts import PromptTemplate\n\nmap_prompt_template = \"\"\"Write a concise summary of the following kaggle solution writeup:\n\n\n\"{text}\"\n\n\nCONCISE SUMMARY:\"\"\"\nMAP_PROMPT = PromptTemplate(template=map_prompt_template, input_variables=[\"text\"])\n\ncombine_prompt_template = \"\"\"Write a 300 letters summary of the following kaggle solution writeup:\n\n\n\"{text}\"\n\n\n300 LETTERS SUMMARY:\"\"\"\nCOMBINE_PROMPT = PromptTemplate(template=combine_prompt_template, input_variables=[\"text\"])","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:09:31.586887Z","iopub.execute_input":"2024-03-31T10:09:31.587169Z","iopub.status.idle":"2024-03-31T10:09:31.825311Z","shell.execute_reply.started":"2024-03-31T10:09:31.587141Z","shell.execute_reply":"2024-03-31T10:09:31.824517Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2 summarization chain\nTo summarize documents using LangChain, a summarization chain needs to be defined.\n\nThe details are described on the following page:<br>\nhttps://python.langchain.com/docs/use_cases/summarization\n\nMake sure that the maximum token length for gemma_2b is 1024.","metadata":{}},{"cell_type":"code","source":"from langchain.chains.summarize import load_summarize_chain\n\nchain = load_summarize_chain(\n    gemma_lc, chain_type=\"map_reduce\",\n    map_prompt=MAP_PROMPT,\n    combine_prompt=COMBINE_PROMPT,\n    token_max=1024\n)","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:09:31.826261Z","iopub.execute_input":"2024-03-31T10:09:31.826563Z","iopub.status.idle":"2024-03-31T10:09:31.876068Z","shell.execute_reply.started":"2024-03-31T10:09:31.826514Z","shell.execute_reply":"2024-03-31T10:09:31.875346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from langchain import OpenAI, PromptTemplate, LLMChain\nfrom langchain.chains.mapreduce import MapReduceChain\nfrom langchain.prompts import PromptTemplate\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\nfrom langchain.docstore.document import Document\n\n\ndef summarize(text):\n    text_splitter = RecursiveCharacterTextSplitter(chunk_size = 1000, chunk_overlap=50)\n    texts = text_splitter.split_text(text)\n    paged_docs = [Document(page_content=t) for t in texts]\n    return chain.invoke(paged_docs)[\"output_text\"]","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:09:31.876859Z","iopub.execute_input":"2024-03-31T10:09:31.877121Z","iopub.status.idle":"2024-03-31T10:09:31.921126Z","shell.execute_reply.started":"2024-03-31T10:09:31.877093Z","shell.execute_reply":"2024-03-31T10:09:31.920477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Prepare solution write-ups\n\nI'm following [Darek's way](https://www.kaggle.com/datasets/thedrcat/kaggle-winning-solutions-methods/) to prepare solution write-ups, which scrapes the list of solutions from [Kaggle Solutions](https://farid.one/kaggle-solutions/) and merges with kaggle-ai-report.\n\nI found some areas for improvement to gather more solutions, so I made a few modifications as shown below.\n\n## 5.1 Scraping\n\nThere's already a wealth of information collected on the amazing [Kaggle Solutions](https://farid.one/kaggle-solutions/) website, so let's start there - we'll scrape all the top solution description links along with the competition data. ","metadata":{}},{"cell_type":"code","source":"from bs4 import BeautifulSoup\nimport requests\n\nurl = \"https://farid.one/kaggle-solutions/\"\n\nresponse = requests.get(url)\nhtml = response.text\n\nsoup = BeautifulSoup(html, 'html.parser')\n\nlinks = []\nfor li_tag in soup.find_all('li', class_='secondary'):\n    link = li_tag.find('a')\n    if link is None:\n        continue\n\n    if 'discussion' in link['href']:\n        medal_img = li_tag.find('img')\n        if medal_img is None:\n            continue\n        if medal_img.next_sibling is not None:\n            # case:1 <img />\n            place = medal_img.next_sibling.strip().split(' ')[0]\n        else:\n            # case:2 <img></img>\n            place = list(li_tag.find('img').children)[0].strip().split(' ')[0]\n\n        competition_name = li_tag.find_previous('b').text.split('. ')[-1].strip()\n\n        info_tuple = (link['href'], int(place[:-2]), place, competition_name)\n        links.append(info_tuple)\n\nprint(len(links))","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-31T10:09:31.921892Z","iopub.execute_input":"2024-03-31T10:09:31.922109Z","iopub.status.idle":"2024-03-31T10:09:33.832079Z","shell.execute_reply.started":"2024-03-31T10:09:31.922085Z","shell.execute_reply":"2024-03-31T10:09:33.831241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nscraped = pd.DataFrame(data=links, columns=['link', 'place', 'place_order', 'competition_name'])\nscraped.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:09:33.834812Z","iopub.execute_input":"2024-03-31T10:09:33.835068Z","iopub.status.idle":"2024-03-31T10:09:33.850776Z","shell.execute_reply.started":"2024-03-31T10:09:33.835042Z","shell.execute_reply":"2024-03-31T10:09:33.849965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.2 Combining with Kaggle Writeups\n\nKaggle doesn't like scraping discussion posts, but fortunately they provided us a bunch of writeups in the competition data. Let's merge it with the scraped data. ","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\ndf = pd.read_csv('../input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv')\n\nscraped['nm'] = scraped.link.apply(lambda x: x.split('/')[-1])\ndf['nm'] = df['Writeup URL'].apply(lambda x: x.split('/')[-1])\nwriteup_mapping = dict(zip(df.nm, df['Writeup']))\nscraped['writeup'] = scraped.nm.apply(lambda x: writeup_mapping[x] if x in writeup_mapping else np.nan)\n\nwriteups = scraped.dropna()\nwriteups.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:09:33.851686Z","iopub.execute_input":"2024-03-31T10:09:33.851936Z","iopub.status.idle":"2024-03-31T10:09:34.041444Z","shell.execute_reply.started":"2024-03-31T10:09:33.851908Z","shell.execute_reply":"2024-03-31T10:09:34.040612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.3 Combining with Meta Kaggle\nTo add the data of teams and private/public scores, use meta-kaggle and merge with the writeup data.","metadata":{}},{"cell_type":"code","source":"teams = pd.read_csv('../input/meta-kaggle/Teams.csv')\nteams = teams.dropna(axis=0, subset=['PrivateLeaderboardRank'])\nint_columns = ['TeamLeaderId', 'PrivateLeaderboardSubmissionId', 'PrivateLeaderboardRank']\nteams[int_columns] = teams[int_columns].astype(int)\nteams = teams.loc[:, ['Id', 'CompetitionId', 'TeamName'] + int_columns]\nteams.head()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-31T10:09:34.042487Z","iopub.execute_input":"2024-03-31T10:09:34.042760Z","iopub.status.idle":"2024-03-31T10:09:45.100586Z","shell.execute_reply.started":"2024-03-31T10:09:34.042732Z","shell.execute_reply":"2024-03-31T10:09:45.099561Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"users = pd.read_csv('../input/meta-kaggle/Users.csv')\nusers = users.loc[:, ['Id', 'UserName', 'DisplayName']]\nusers.columns = ['UserId', 'UserName', 'DisplayName']\nusers.head()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-31T10:09:45.101614Z","iopub.execute_input":"2024-03-31T10:09:45.101908Z","iopub.status.idle":"2024-03-31T10:10:10.113305Z","shell.execute_reply.started":"2024-03-31T10:09:45.101878Z","shell.execute_reply":"2024-03-31T10:10:10.112183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"competitions = pd.read_csv('../input/meta-kaggle/Competitions.csv')\ncompetitions = competitions[['Id', 'Title']]\ncompetitions.columns = ['CompetitionId', 'competition_name']\ncompetitions.head()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-31T10:10:10.114253Z","iopub.execute_input":"2024-03-31T10:10:10.114547Z","iopub.status.idle":"2024-03-31T10:10:10.172976Z","shell.execute_reply.started":"2024-03-31T10:10:10.114518Z","shell.execute_reply":"2024-03-31T10:10:10.172079Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submissions = pd.read_csv('../input/meta-kaggle/Submissions.csv')\nsubmissions = submissions[['Id', 'PublicScoreLeaderboardDisplay', 'PrivateScoreLeaderboardDisplay']]\nsubmissions.columns = ['PrivateLeaderboardSubmissionId', 'PublicScoreLeaderboardDisplay', 'PrivateScoreLeaderboardDisplay']\nsubmissions.head()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-31T10:10:10.173984Z","iopub.execute_input":"2024-03-31T10:10:10.174264Z","iopub.status.idle":"2024-03-31T10:10:26.188683Z","shell.execute_reply.started":"2024-03-31T10:10:10.174227Z","shell.execute_reply":"2024-03-31T10:10:26.187857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Loaded datasets are merged for each teams.","metadata":{}},{"cell_type":"code","source":"teams = pd.merge(teams, competitions, how='left', on='CompetitionId')\nteams = pd.merge(teams, users, how='left', left_on='TeamLeaderId', right_on='UserId')\nteams = pd.merge(teams, submissions, how='left', on='PrivateLeaderboardSubmissionId')\nteams.head()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-03-31T10:10:26.189613Z","iopub.execute_input":"2024-03-31T10:10:26.189891Z","iopub.status.idle":"2024-03-31T10:10:33.671997Z","shell.execute_reply.started":"2024-03-31T10:10:26.189848Z","shell.execute_reply":"2024-03-31T10:10:33.671088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, the writeup data is merged with the teams data.","metadata":{}},{"cell_type":"code","source":"writeups = pd.merge(writeups, teams, how='left',\n                    left_on=['place', 'competition_name'],\n                    right_on=['PrivateLeaderboardRank', 'competition_name'])\nwriteups.head()","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2024-03-31T10:10:33.673073Z","iopub.execute_input":"2024-03-31T10:10:33.673400Z","iopub.status.idle":"2024-03-31T10:10:33.915935Z","shell.execute_reply.started":"2024-03-31T10:10:33.673367Z","shell.execute_reply":"2024-03-31T10:10:33.915123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Define summarization format\n\nIn this step, some functions prepared to define summarization format.\n\nBasically, each solution is summarized by the summarization chain.\n\n**Overall Summary** and **Comparision Table** are also created by Gemma 2B.<br>\nAdditionally, **Mind Maps** are created by MindGemmap which is a fine-tuned variant of Gemma 2B.\n\nRegarding MindGemmap, further detailes are explained in the [model card](https://www.kaggle.com/models/toshik/mindgemmap).\n\n## 6.1 Overall Summary","metadata":{}},{"cell_type":"code","source":"def create_overall_summary(solutions_md):\n    overall_summary_md = gemma_lm.generate(f\"\"\"\nCreate an overall summary of the kaggle competition from the following solution writeups:\n\n{solutions_md}\n\nOVERALL SUMMARY:\"\"\", max_length=2048).split(\"SUMMARY:\", 1)[1]\n    overall_summary_md = \"### Overall Summary\" + overall_summary_md\n    return overall_summary_md","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:10:33.916946Z","iopub.execute_input":"2024-03-31T10:10:33.917228Z","iopub.status.idle":"2024-03-31T10:10:33.921758Z","shell.execute_reply.started":"2024-03-31T10:10:33.917200Z","shell.execute_reply":"2024-03-31T10:10:33.920973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6.2 Comparison Table","metadata":{}},{"cell_type":"code","source":"def create_comparision_table(solutions_md):\n    comparison_table_md = gemma_lm.generate(f\"\"\"\nCreate an approach comparison table by Markdown format from the following kaggle solutions (each row corresponds to each solution):\n\n{solutions_md}\n\nAPPROACH COMPARISON TABLE:\"\"\", max_length=2048).split(\"TABLE:\", 1)[1]\n    comparison_table_md = \"### Comparison Table\" + comparison_table_md\n    return comparison_table_md","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:10:33.922747Z","iopub.execute_input":"2024-03-31T10:10:33.923043Z","iopub.status.idle":"2024-03-31T10:10:33.933374Z","shell.execute_reply.started":"2024-03-31T10:10:33.923012Z","shell.execute_reply":"2024-03-31T10:10:33.932604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6.3 Mind Map","metadata":{}},{"cell_type":"code","source":"mind_gemmap = keras_nlp.models.GemmaCausalLM.from_preset(\"gemma_instruct_2b_en\")\nmind_gemmap.load_weights('/kaggle/input/mindgemmap/keras/v1/1/MindGemmap.weights.h5')","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:10:33.934141Z","iopub.execute_input":"2024-03-31T10:10:33.934413Z","iopub.status.idle":"2024-03-31T10:12:34.184741Z","shell.execute_reply.started":"2024-03-31T10:10:33.934376Z","shell.execute_reply":"2024-03-31T10:12:34.183836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sampler = keras_nlp.samplers.GreedySampler()\nmind_gemmap.compile(sampler=sampler)","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:12:34.185900Z","iopub.execute_input":"2024-03-31T10:12:34.186163Z","iopub.status.idle":"2024-03-31T10:12:34.194728Z","shell.execute_reply.started":"2024-03-31T10:12:34.186137Z","shell.execute_reply":"2024-03-31T10:12:34.193899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from IPython.display import display\nimport graphviz\n\ndef create_mindmap(solution_md):\n    dot_file = mind_gemmap.generate(f\"\"\"Generate a dot file to create a mindmap from the following text:\n\nTEXT: {solution_md}\n\nDOTFILE:\n\"\"\", max_length=4096).split(\"DOTFILE:\", 1)[1]\n    try:\n        graph = graphviz.Source(dot_file, format=\"png\")\n        display(graph)\n    except:\n        print('Failed to generate the mind map for this solution :(')\n    return","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:12:34.195693Z","iopub.execute_input":"2024-03-31T10:12:34.195943Z","iopub.status.idle":"2024-03-31T10:12:34.217594Z","shell.execute_reply.started":"2024-03-31T10:12:34.195917Z","shell.execute_reply":"2024-03-31T10:12:34.216834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6.4 Rendering function","metadata":{}},{"cell_type":"code","source":"from IPython.display import display, Markdown\n\n\ndef top_solutions(writeups, competition_name):\n    '''summarize top10 solutions'''\n\n    # filter target rows\n    df = writeups.query('competition_name == @competition_name').drop_duplicates(subset=['place'])\n    df = df.query('place <= 10').reset_index(inplace=False)\n\n    solutions_md = ''\n    for i, row in df.iterrows():\n\n        solution_title = f'{row[\"place_order\"]} place solution'\n        solution_txt = summarize(row[\"writeup\"])\n        solution_md = f'''### {solution_title}\n{solution_txt}\n'''\n        solutions_md += solution_md\n\n        if row[\"place\"] <= 3:\n            # display raw summaries for top 3 solutions\n            display_solution_md = f'''### [{solution_title}]({row[\"link\"]})\n#### Team: {row[\"TeamName\"]} &nbsp; Leader: [{row[\"DisplayName\"]}](https://www.kaggle.com/{row[\"UserName\"]}) &nbsp; Public Score: {row[\"PublicScoreLeaderboardDisplay\"]} &nbsp; Private Score: {row[\"PrivateScoreLeaderboardDisplay\"]}\n{solution_txt}\n'''\n            # display raw summaries for top 3 solutions\n            display(Markdown(display_solution_md))\n\n            # display mind map for top 3 solutions\n            create_mindmap(solution_md)\n\n    # display the overall summary\n    overall_summary_md = create_overall_summary(solutions_md)\n    display(Markdown(overall_summary_md))\n\n    # display the comparision table\n    comparison_table_md = create_comparision_table(solutions_md)\n    display(Markdown(comparison_table_md))","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:12:34.218366Z","iopub.execute_input":"2024-03-31T10:12:34.218584Z","iopub.status.idle":"2024-03-31T10:12:34.225135Z","shell.execute_reply.started":"2024-03-31T10:12:34.218560Z","shell.execute_reply":"2024-03-31T10:12:34.224370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 7. Summarize solution write-ups\nNow, we can summarize kaggle solultion write-ups.\nSome exapmles are shown as follows,\n\n## 7.1 OTTO – Multi-Objective Recommender System","metadata":{}},{"cell_type":"code","source":"%%time\ntop_solutions(writeups, \"OTTO – Multi-Objective Recommender System\")","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:12:34.225978Z","iopub.execute_input":"2024-03-31T10:12:34.226211Z","iopub.status.idle":"2024-03-31T10:14:40.897094Z","shell.execute_reply.started":"2024-03-31T10:12:34.226186Z","shell.execute_reply":"2024-03-31T10:14:40.896081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 7.2 Foursquare - Location Matching","metadata":{}},{"cell_type":"code","source":"%%time\ntop_solutions(writeups, \"Foursquare - Location Matching\")","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"execution":{"iopub.status.busy":"2024-03-31T10:14:40.898384Z","iopub.execute_input":"2024-03-31T10:14:40.898696Z","iopub.status.idle":"2024-03-31T10:15:29.219009Z","shell.execute_reply.started":"2024-03-31T10:14:40.898666Z","shell.execute_reply":"2024-03-31T10:15:29.217966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 7.3 G2Net Detecting Continuous Gravitational Waves","metadata":{}},{"cell_type":"code","source":"%%time\ntop_solutions(writeups, \"G2Net Detecting Continuous Gravitational Waves\")","metadata":{"execution":{"iopub.status.busy":"2024-03-31T10:15:29.220209Z","iopub.execute_input":"2024-03-31T10:15:29.220515Z","iopub.status.idle":"2024-03-31T10:16:12.972380Z","shell.execute_reply.started":"2024-03-31T10:15:29.220484Z","shell.execute_reply":"2024-03-31T10:16:12.971367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 8. Conclusion\n\nIn this notebook, we demonstrate that LangChain integrated with Gemma 2B aggregates and summarizes the winning solutions on a per competition basis.\n\nGemma 2B can also generate overall summary and comparison table.<br>\nAdditionally, my fine-tuned variant of Gemma 2B \"MindGemmap\" can generate mind maps which was impossible for original Gemma 2B.\n\nThank you for reading this to the end. Please don't forget to upvote :)","metadata":{}}]}