{"metadata":{"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":50947,"databundleVersionId":5741538,"sourceType":"competition"},{"sourceId":64148,"databundleVersionId":7669720,"sourceType":"competition"},{"sourceId":5134565,"sourceType":"datasetVersion","datasetId":2593672},{"sourceId":8149368,"sourceType":"datasetVersion","datasetId":4819562},{"sourceId":11261,"sourceType":"modelInstanceVersion","modelInstanceId":8332},{"sourceId":26154,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":22015}],"dockerImageVersionId":30648,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.13"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Gemma v1.1  Summarizes Kaggle Competition Writeups\n\nHow can we use Gemma to summarize Kaggle competition writeups, or can we? Well, lets see.\n\nThis notebook is to trial Gemmas summarization skills on Kaggle competition writeups.\n\n- v1: first trials with summarization\n- v2: attempt at parsing the writeup placement for prioritization basis\n- v3: trying to summarize each writeup first, and from those all writeups per competition\n- v4: tuning prompt for better summary of key points, exploring Gemma's ability for overall summarization of multiple writeups across the competition, adding my own summary on Gemma learnings\n- v5: optimized by clipping context to 4k on Kaggle to avoid GPU OOM. runs on P100 now much faster than 2xT4\n- v6: after trialing these summaries as inputs with [a RAG notebook](https://www.kaggle.com/code/donkeys/qa-solutions-from-past-competitions), now added section on summarizing large writeup sets per competition in parts and learnings from that\n- v7: analysis of larger set of writeups and summaries for different subset sizes.\n- v8: updated model to Gemma 1.1, did some basic comparison between the two Gemma versions.","metadata":{}},{"cell_type":"markdown","source":"# Executive Summary of (my) Gemma Learning\n\n- Gemma is a bit talkative, at least the instruction tuned version. It seems to wish to tell me more than I asked for.\n- Gemma does a decent job figuring out what position the solution described in a writeup finished in, if that information is in the writeup or its title. But can also miss it even if it is straight in the title.\n- Due to talkative nature and unreliable position identification it is a bit difficult to use for sorting the writeups by their competition scores\n- Gemma does not seem very good at distinguishing different parts of a longer prompt, such as competition description vs the solution writeup. It seems to mix potential approaches from the descriptions in the writeup.\n- With little information (e.g., almost empty writeups), summaries seem to hallucinate the competition description into the writeup summary. With large number of writeups, Gemma sticks to the \"facts\" much better.\n- When asked for a summary of key points, Gemma seems to want to always write its own extra opinion piece in the end about the summary.\n- Asking simply for a list of key points without \"summary\" works better to avoid this opinion piece. Perhaps it is something to do with its training in relation to summarization.\n- Gemma seems to produce summaries in the range of 200-300 tokens, regardless of the input context size. Again, perhaps something about how it has been trained?\n- When a good prompt format and approach is found, Gemma seems to give quite good summaries. Probably quite common for models this size.\n- When summarizing multiple parts, such as partial summaries of large writeup sets, subsections are needed or Gemma seems to lose it.\n- When re-summarizing larger numbers of subsets, Gemma seems to more easily lose it. Some ways to mitigate that should be considered if using such hierarchical summarization approaches.\n- Gemma 1.1 does better summaries, capturing overall concepts better than 1.0. Both seem to hallucinate for very short writeups with no real data, and sometimes drop it for long sets of writeup summaries. 1.1 seems to do this more often.","metadata":{}},{"cell_type":"code","source":"#have to update these first, or otherwise it seems the libraries might not update and load older versions\n!pip install -q -U transformers accelerate bitsandbytes lxml\n#flash attention does not work on Kaggle GPU's, too old\n#!pip install flash-attn --no-build-isolation\n","metadata":{"execution":{"iopub.status.busy":"2024-04-20T13:16:21.771461Z","iopub.execute_input":"2024-04-20T13:16:21.771850Z","iopub.status.idle":"2024-04-20T13:16:55.433711Z","shell.execute_reply.started":"2024-04-20T13:16:21.771820Z","shell.execute_reply":"2024-04-20T13:16:55.432407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom IPython.display import HTML, Markdown, display\nfrom tqdm.notebook import tqdm\n\nfrom transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n\non_kaggle = False\nif os.path.exists(\"/kaggle/input\"):\n    on_kaggle = True\n\nif on_kaggle:\n    for dirname, _, filenames in os.walk('/kaggle/input'):\n        for filename in filenames:\n            print(os.path.join(dirname, filename))\n\n\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.status.busy":"2024-04-20T13:16:59.429635Z","iopub.execute_input":"2024-04-20T13:16:59.430374Z","iopub.status.idle":"2024-04-20T13:17:06.177951Z","shell.execute_reply.started":"2024-04-20T13:16:59.430335Z","shell.execute_reply":"2024-04-20T13:17:06.176928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!nvidia-smi","metadata":{"execution":{"iopub.status.busy":"2024-04-20T13:17:06.179963Z","iopub.execute_input":"2024-04-20T13:17:06.180918Z","iopub.status.idle":"2024-04-20T13:17:07.304692Z","shell.execute_reply.started":"2024-04-20T13:17:06.180880Z","shell.execute_reply":"2024-04-20T13:17:07.303355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Gemma Model and Try it with Poetry Prompt\n\nThe poetry prompt is actually taken from the [Huggingface Gemma page](https://huggingface.co/google/gemma-7b-it).\n\nI use it here just as a smoke test to see the code and model works.","metadata":{}},{"cell_type":"code","source":"#https://www.kaggle.com/code/minhsienweng/create-ai-generated-essays-gemma/notebook\nimport torch\nDEVICE = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\nDEVICE = \"auto\"\nprint(f\"Device: {DEVICE}\")\nprint(f\"CUDA Version: {torch.version.cuda}\")\nprint(f\"Pytorch {torch.__version__}\")","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:26:49.757795Z","iopub.execute_input":"2024-04-19T21:26:49.758248Z","iopub.status.idle":"2024-04-19T21:26:49.789375Z","shell.execute_reply.started":"2024-04-19T21:26:49.758208Z","shell.execute_reply":"2024-04-19T21:26:49.788097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"thanks to @minhsienweng for [helping](https://www.kaggle.com/competitions/data-assistants-with-gemma/discussion/479773#2670394) to get quantization working on Kaggle\n\n","metadata":{"execution":{"iopub.execute_input":"2024-02-28T05:30:12.155611Z","iopub.status.busy":"2024-02-28T05:30:12.155237Z","iopub.status.idle":"2024-02-28T05:30:12.162600Z","shell.execute_reply":"2024-02-28T05:30:12.161278Z","shell.execute_reply.started":"2024-02-28T05:30:12.155577Z"}}},{"cell_type":"code","source":"from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig\n\nif on_kaggle:\n    #model_path = \"/kaggle/input/gemma/transformers/7b-it/1\"\n    model_path = \"/kaggle/input/gemma/transformers/1.1-7b-it/1\"\n    \nelse:\n    model_path = \"/mystuff/llm/gemma-7b-it\"\n#    model_path = \"/mystuff/llm/gemma-7b\"\n    \n\ntokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)\n\n#Gemma has 8096 context length:\n#https://huggingface.co/google/gemma-7b-it/discussions/27\n\ncompute_dtype = getattr(torch, \"float16\")\nquantization_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_use_double_quant=False,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_compute_dtype=torch.float16,\n)\n\ng_model = AutoModelForCausalLM.from_pretrained(model_path, \n                                               device_map=DEVICE, \n                                               quantization_config=quantization_config,\n                                               #attn_implementation=\"flash_attention_2\",\n                                              )\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:26:49.791934Z","iopub.execute_input":"2024-04-19T21:26:49.792303Z","iopub.status.idle":"2024-04-19T21:27:06.401617Z","shell.execute_reply.started":"2024-04-19T21:26:49.792271Z","shell.execute_reply":"2024-04-19T21:27:06.400716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So next to try the prompt from HF model page as the base smoke test on model working:","metadata":{}},{"cell_type":"code","source":"%%time\ninput_text = \"Write me a poem about Machine Learning.\"\ninput_ids = tokenizer(input_text, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = g_model.generate(**input_ids, max_new_tokens=1000)\nprint(tokenizer.decode(outputs[0]))","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:06.402789Z","iopub.execute_input":"2024-04-19T21:27:06.403240Z","iopub.status.idle":"2024-04-19T21:27:21.065755Z","shell.execute_reply.started":"2024-04-19T21:27:06.403213Z","shell.execute_reply":"2024-04-19T21:27:21.064733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<p></p>\n\nThere you go. An interesting poem :) So the model now works, also on Kaggle.","metadata":{}},{"cell_type":"markdown","source":"# Load Datasets\n\nI use data from a set of Kaggle competition writeups, and competition descriptions.\n\nIn this notebook, I try out Gemma as a summarizer for those writeups. And the descriptions are there to help build a bit of context for the writeups and their summaries.","metadata":{}},{"cell_type":"code","source":"if on_kaggle:\n    df_writeups = pd.read_csv(f\"/kaggle/input/2023-kaggle-ai-report/kaggle_writeups_20230510.csv\")\n    df_comp_meta = pd.read_csv(f\"/kaggle/input/kaggles-all-completed-competition-dataset/kaggle comp_submission.csv\")\nelse:\n    df_writeups = pd.read_csv(f\"/mystuff/data/kaggle_writeups_20230510.csv\")\n    df_comp_meta = pd.read_csv(f\"/mystuff/data/kaggle_comp_submission.csv\")\n\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:21.067112Z","iopub.execute_input":"2024-04-19T21:27:21.067697Z","iopub.status.idle":"2024-04-19T21:27:21.281191Z","shell.execute_reply.started":"2024-04-19T21:27:21.067644Z","shell.execute_reply":"2024-04-19T21:27:21.280245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some of the writeups have a lot of HTML in them, causing the Gemma model sometimes to overflow its context window, and lose the text in the HTML markup. This shows in the Writeup column here:","metadata":{}},{"cell_type":"code","source":"df_writeups.head(2)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:21.282749Z","iopub.execute_input":"2024-04-19T21:27:21.283426Z","iopub.status.idle":"2024-04-19T21:27:21.300835Z","shell.execute_reply.started":"2024-04-19T21:27:21.283387Z","shell.execute_reply":"2024-04-19T21:27:21.299809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This `strip_html()` will remove HTML formatting from a writeup and only leave the actual text:","metadata":{}},{"cell_type":"code","source":"from lxml import html\n\ndef strip_html(html_text):\n    tree = html.fromstring(html_text)\n    clean_text = tree.text_content()\n    return clean_text\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:21.302042Z","iopub.execute_input":"2024-04-19T21:27:21.302348Z","iopub.status.idle":"2024-04-19T21:27:21.323093Z","shell.execute_reply.started":"2024-04-19T21:27:21.302321Z","shell.execute_reply":"2024-04-19T21:27:21.322111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 2 writeups with NAN values, dropping them allows processing the column at once, and removes invalid rows.","metadata":{}},{"cell_type":"code","source":"print(f\"                 Initial set of writeups: {df_writeups.shape}\")\ndf_writeups = df_writeups.dropna(subset=['Writeup'])\nprint(f\"After removing rows with nan for writeup: {df_writeups.shape}\")\n# this shows there were 2 rows with nan writeup, 3127->3125","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:21.324149Z","iopub.execute_input":"2024-04-19T21:27:21.324781Z","iopub.status.idle":"2024-04-19T21:27:21.334239Z","shell.execute_reply.started":"2024-04-19T21:27:21.324755Z","shell.execute_reply":"2024-04-19T21:27:21.333238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Initially, I tried simply apply `strip_html()` directly to the `Writeup` column for all rows. However, this made printing any example writeups in the notebook harder to read. So I added the stripped writeup as its own column `writeup_clean`.","metadata":{}},{"cell_type":"code","source":"#df_writeups.loc[:, \"Writeup\"] = df_writeups[\"Writeup\"].apply(strip_html)\ndf_writeups[\"writeup_clean\"] = df_writeups[\"Writeup\"].apply(strip_html)\ndf_writeups = df_writeups.rename(columns={'Writeup': 'writeup'})\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:21.340300Z","iopub.execute_input":"2024-04-19T21:27:21.340888Z","iopub.status.idle":"2024-04-19T21:27:21.886670Z","shell.execute_reply.started":"2024-04-19T21:27:21.340862Z","shell.execute_reply":"2024-04-19T21:27:21.885623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The `df_comp_meta` dataset contains competition metadata such as competition description.","metadata":{}},{"cell_type":"code","source":"df_comp_meta.head(2)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:21.887930Z","iopub.execute_input":"2024-04-19T21:27:21.888221Z","iopub.status.idle":"2024-04-19T21:27:21.903779Z","shell.execute_reply.started":"2024-04-19T21:27:21.888196Z","shell.execute_reply":"2024-04-19T21:27:21.902782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data stats\n\nThere appear to be 310 different competitions in the writeup data. There are more writeups (3125) than competitions because multiple participants did a writeup on single competitions from their own perspective.","metadata":{}},{"cell_type":"code","source":"df_writeups[\"Title of Competition\"].nunique()","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:21.905322Z","iopub.execute_input":"2024-04-19T21:27:21.905867Z","iopub.status.idle":"2024-04-19T21:27:21.914625Z","shell.execute_reply.started":"2024-04-19T21:27:21.905835Z","shell.execute_reply":"2024-04-19T21:27:21.913648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The first writeup in the dataset appears to be for a chess rating competition:","metadata":{}},{"cell_type":"code","source":"competitions = df_writeups[\"Title of Competition\"].unique()\ncomp0 = competitions[0]\ncomp0","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:21.915951Z","iopub.execute_input":"2024-04-19T21:27:21.916302Z","iopub.status.idle":"2024-04-19T21:27:21.926797Z","shell.execute_reply.started":"2024-04-19T21:27:21.916269Z","shell.execute_reply":"2024-04-19T21:27:21.925704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This competition appears to have 5 writeups:","metadata":{}},{"cell_type":"code","source":"df_comp0 = df_writeups[df_writeups[\"Title of Competition\"]==comp0]\nprint(f\"Chess competition dataframe shape: {df_comp0.shape}\")\ndf_comp0","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:21.928251Z","iopub.execute_input":"2024-04-19T21:27:21.928561Z","iopub.status.idle":"2024-04-19T21:27:21.948021Z","shell.execute_reply.started":"2024-04-19T21:27:21.928537Z","shell.execute_reply":"2024-04-19T21:27:21.947023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Example Writeup\n\nWhat does a writeup look like?","metadata":{}},{"cell_type":"code","source":"writeup0=df_comp0.iloc[1][\"writeup\"]\nMarkdown(writeup0)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:21.949207Z","iopub.execute_input":"2024-04-19T21:27:21.949498Z","iopub.status.idle":"2024-04-19T21:27:21.956095Z","shell.execute_reply.started":"2024-04-19T21:27:21.949471Z","shell.execute_reply":"2024-04-19T21:27:21.955111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Attempt 1 - Ask Gemma for a useful summary for a Kaggle learner:\n\nTry with the first writeup collected above:","metadata":{}},{"cell_type":"code","source":"input_text = f\"Summarize the following Kaggle competition writeup.\" \\\n f\"Make it useful for someone trying to learn and apply a similar \" \\\n f\"approach in a different competition: {writeup0}\"\ninput_ids = tokenizer(input_text, return_tensors=\"pt\").to(\"cuda\")\ninput_token_count = len(input_ids[\"input_ids\"][0])\n#this will be the length of the prompt including the writeup, in Gemma tokens\n#since Gemma max context length is 8092, there is still plenty of room for processing most writeups \n#(one by one at least)\ninput_token_count","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:21.957639Z","iopub.execute_input":"2024-04-19T21:27:21.958002Z","iopub.status.idle":"2024-04-19T21:27:21.974043Z","shell.execute_reply.started":"2024-04-19T21:27:21.957971Z","shell.execute_reply":"2024-04-19T21:27:21.973105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"outputs = g_model.generate(**input_ids, max_new_tokens=1000, do_sample=False)\nnew_tokens = outputs[0][input_token_count:]\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:21.975309Z","iopub.execute_input":"2024-04-19T21:27:21.975922Z","iopub.status.idle":"2024-04-19T21:27:40.124255Z","shell.execute_reply.started":"2024-04-19T21:27:21.975890Z","shell.execute_reply":"2024-04-19T21:27:40.123327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Writeup Summary:","metadata":{}},{"cell_type":"code","source":"summary1 = tokenizer.decode(new_tokens)\nMarkdown(summary1)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:40.125416Z","iopub.execute_input":"2024-04-19T21:27:40.125739Z","iopub.status.idle":"2024-04-19T21:27:40.132359Z","shell.execute_reply.started":"2024-04-19T21:27:40.125712Z","shell.execute_reply":"2024-04-19T21:27:40.131477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Attempt 2: Add more context to Gemma input\n\nCompared to Gemma 1.0, the above summary is better in providing clearer key points and less focus on unrelated and hard to grasp details. However, it still lacks a bit of the overall context due to writeups typically assuming knowledge about the competition it is for. Maybe Gemma can improve the summary with a bit of knowledge about what kind of competition the writeup is for.\n\nSo trying here with adding writeup title, competition name, and competition description to the prompt. Lets see.","metadata":{}},{"cell_type":"code","source":"writeup0_title = df_comp0[\"Title of Writeup\"].iloc[0]\nwriteup0_title","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:40.133879Z","iopub.execute_input":"2024-04-19T21:27:40.134654Z","iopub.status.idle":"2024-04-19T21:27:40.146803Z","shell.execute_reply.started":"2024-04-19T21:27:40.134616Z","shell.execute_reply":"2024-04-19T21:27:40.145754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"comp0_name = df_comp0[\"Title of Competition\"].iloc[0]\ncomp0_name","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:40.148462Z","iopub.execute_input":"2024-04-19T21:27:40.148782Z","iopub.status.idle":"2024-04-19T21:27:40.159696Z","shell.execute_reply.started":"2024-04-19T21:27:40.148756Z","shell.execute_reply":"2024-04-19T21:27:40.158736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"comp0_desc = df_comp_meta[df_comp_meta[\"comp_name\"] == comp0_name][\"desc\"].values[0]\nMarkdown(comp0_desc)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:40.160927Z","iopub.execute_input":"2024-04-19T21:27:40.161267Z","iopub.status.idle":"2024-04-19T21:27:40.172784Z","shell.execute_reply.started":"2024-04-19T21:27:40.161235Z","shell.execute_reply":"2024-04-19T21:27:40.171719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"But first a helper method to avoid copy-pasting the generation code all the time:","metadata":{}},{"cell_type":"code","source":"def gemma_generate(prompt, debug=False):\n    prompt = prompt.replace(\"<eos>\", \"\")\n    input_ids = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n    input_token_count = len(input_ids[\"input_ids\"][0])\n\n    outputs = g_model.generate(**input_ids, max_new_tokens=1000, do_sample=False)\n\n    output_token_count = len(outputs[0])\n    if debug:\n        print(f\"input tokens: {input_token_count}, output tokens: {output_token_count}\")\n    \n    new_tokens = outputs[0][input_token_count:]\n    output = tokenizer.decode(new_tokens)\n    output_md = Markdown(output)\n    return output, output_md","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:40.174039Z","iopub.execute_input":"2024-04-19T21:27:40.174360Z","iopub.status.idle":"2024-04-19T21:27:40.182624Z","shell.execute_reply.started":"2024-04-19T21:27:40.174334Z","shell.execute_reply":"2024-04-19T21:27:40.181735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And now, try to generate a writeup summary from the competition description + a writeup:","metadata":{}},{"cell_type":"code","source":"prompt = f\"given the following kaggle competition description and solution writeup, \" \\\n         f\"summarize the solution for someone interested in applying a similar solution in a different context:\\n\\n\" \\\n         f\"description: {comp0_desc}\\n\\n\" \\\n         f\"writeup: {writeup0_title}: {writeup0}\"","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:40.183795Z","iopub.execute_input":"2024-04-19T21:27:40.184108Z","iopub.status.idle":"2024-04-19T21:27:40.197126Z","shell.execute_reply.started":"2024-04-19T21:27:40.184071Z","shell.execute_reply":"2024-04-19T21:27:40.196390Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nsummary2, summary2_md = gemma_generate(prompt)\nMarkdown(summary2)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:27:40.198228Z","iopub.execute_input":"2024-04-19T21:27:40.198517Z","iopub.status.idle":"2024-04-19T21:28:02.235778Z","shell.execute_reply.started":"2024-04-19T21:27:40.198494Z","shell.execute_reply":"2024-04-19T21:28:02.234749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Better ..\n\nWith Gemma 1.1, and comparing to the 1.0 version summary, this does seem much better. The key points are clearer, and much easier to use as a basis for understanding the potential use of these for new applications. With 1.0 it was quite vague on this update.\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Attempt 3: Summarize the Description First\n\nAs the details in the description seem to confuse Gemma vs the writeup itself (more so in Gemma 1.0 than 1.1, but lets see anyway), I decided to try to summarize the competition description first, to give basic context but with less details for Gemma to get confused.","metadata":{}},{"cell_type":"code","source":"prompt = f\"Summarize the following kaggle competition description in a very concise way. \"\\\n         f\"Omit references to specific algorithms or known solutions, focus on overview of the contest topic and goals.\\n\\n\" \\\n         f\"description: {comp0_desc}\\n\\n\" \\\n         f\"summary: \"","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:28:02.237140Z","iopub.execute_input":"2024-04-19T21:28:02.237521Z","iopub.status.idle":"2024-04-19T21:28:02.242510Z","shell.execute_reply.started":"2024-04-19T21:28:02.237487Z","shell.execute_reply":"2024-04-19T21:28:02.241592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First a summary of the competition description itself:","metadata":{}},{"cell_type":"code","source":"%%time\ncomp_summary, comp_summary_md = gemma_generate(prompt)\nMarkdown(comp_summary)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:28:02.243611Z","iopub.execute_input":"2024-04-19T21:28:02.244857Z","iopub.status.idle":"2024-04-19T21:28:08.542870Z","shell.execute_reply.started":"2024-04-19T21:28:02.244826Z","shell.execute_reply":"2024-04-19T21:28:08.541503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Writeup Summary from Description Summary and the Writeup itself\n\nNow that we have the competition description summary, we can give it and the writeup to Gemma and see how well it manages to summarize them together (vs full competition description + writeup above):","metadata":{}},{"cell_type":"code","source":"prompt = f\"you are to answer as a helpful assistant helping understand questions about \" \\\n         f\"a kaggle competition writeup, and the solutions the writeup describes. \" \\\n         f\"a short description of the competition and its goals, and a solution writeup follows. \" \\\n         f\"description: {comp_summary}\\n\\n\" \\\n         f\"writeup: {writeup0_title}:\\n {writeup0}\\n\\n\" \\\n         f\"question: summarize the solution in the above writeup, focusing on used data analysis methods. \" \\\n         f\"Focus on the key points of the solutions and how they might have helped achieving better score in the competition.\\n\\n\" \\\n         f\"answer: \"","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:28:08.544427Z","iopub.execute_input":"2024-04-19T21:28:08.544976Z","iopub.status.idle":"2024-04-19T21:28:08.551684Z","shell.execute_reply.started":"2024-04-19T21:28:08.544941Z","shell.execute_reply":"2024-04-19T21:28:08.550227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nsummary3, summary3_md = gemma_generate(prompt)\nMarkdown(summary3)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:28:08.565045Z","iopub.execute_input":"2024-04-19T21:28:08.565541Z","iopub.status.idle":"2024-04-19T21:28:38.017746Z","shell.execute_reply.started":"2024-04-19T21:28:08.565498Z","shell.execute_reply":"2024-04-19T21:28:38.016735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Its better, I think. Point 1 seems a bit concerning though, as it seems to have picked the \"simultaneous calculation\" from the competition description summary it made itself earlier. On the other hand, the overall summary is (in my view) more understandable than it was with Gemma 1.0. And looking at the writeup itself printed above, it seems the input is quite messy as well.\n\nBut there are multiple writeups typically for a competition, so finding the best signal from the noise would be nice. That is, first to select which writeups to show first, or how to rank them for the user.","metadata":{}},{"cell_type":"markdown","source":"# Try to Rank the Solutions in a Competition\n\nBecause it could be helpful to show someone who wants a competition solution writeup summary the best scoring solutions first.\n\nSo, first a method to perform competition summarization all in once to apply across different competitions. See how I tried to trick people tend to give for prompting as an option, asking it to think step by step :).","metadata":{}},{"cell_type":"code","source":"def find_writeup_ranking(title, print_prompt=False, n_process=None):\n    df_comp = df_writeups[df_writeups[\"Title of Competition\"]==title]\n    display(df_comp)\n    for i in range(0, df_comp.shape[0]):\n        writeup_title = df_comp.iloc[i][\"Title of Writeup\"]\n        writeup = df_comp.iloc[i][\"writeup_clean\"]\n        prompt = f\"Given the following competition solution writeup, \" \\\n                 f\"what is the placement of the described solution in the competition? \"\\\n                 f\"Think step by step. First check if the title contains the placement, then check if the writeup does. \" \\\n                 f\"If you cannot find the placement, answer only with 'insufficient information'.\\n\\n\" \\\n                 f\"title: {writeup_title}:\\n\" \\\n                 f\"writeup: {writeup}\"\n        #prompt = f\"{writeup_title}:\\n\" \\\n        #         f\"{writeup}.\\n\\nBased on my analysis, the above write solution in the competition ranked at position: \"\n        if print_prompt:\n            print(prompt)\n        ranking, ranking_md = gemma_generate(prompt)\n        print(f\"--------------- {writeup_title} -------------------\")\n        display(ranking_md)\n        if n_process is not None and i >= n_process-1:\n            break","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:28:38.019292Z","iopub.execute_input":"2024-04-19T21:28:38.019945Z","iopub.status.idle":"2024-04-19T21:28:38.028454Z","shell.execute_reply.started":"2024-04-19T21:28:38.019900Z","shell.execute_reply":"2024-04-19T21:28:38.027340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"comp0 = competitions[0]\ncomp0","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:28:38.029771Z","iopub.execute_input":"2024-04-19T21:28:38.030044Z","iopub.status.idle":"2024-04-19T21:28:38.041490Z","shell.execute_reply.started":"2024-04-19T21:28:38.030014Z","shell.execute_reply":"2024-04-19T21:28:38.040618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From reading Kaggle writeups in the past, I recall seeing those mostly posted for high-ranking notebooks. The following shows how the title often states the ranking of the writeup solution in the competition, and how well Gemma does with identifying that information:","metadata":{}},{"cell_type":"code","source":"find_writeup_ranking(comp0)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:28:38.042519Z","iopub.execute_input":"2024-04-19T21:28:38.042808Z","iopub.status.idle":"2024-04-19T21:29:00.987826Z","shell.execute_reply.started":"2024-04-19T21:28:38.042782Z","shell.execute_reply":"2024-04-19T21:29:00.986856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well, that is quite a fail. Not better than 1.0, looks even worse. Giving wrong information (21st vs 2nd), and missing the obvious information in the title.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As a sidenote, some of the collected writeups in this dataset do now even seem to be actual solutions, just random looking forum posts that may have been caught \"by accident\" in the dataset, or that just refer to some notebook with a few words in the forum post.\n\nWell, I guess all datasets end up with random noise. Thats why data cleaning is such a big job.. \n\nHere is one example, it just seems to congratulate the winners:","metadata":{}},{"cell_type":"code","source":"comp3 = competitions[3]\nfind_writeup_ranking(comp3, True, 1)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:29:00.989153Z","iopub.execute_input":"2024-04-19T21:29:00.989442Z","iopub.status.idle":"2024-04-19T21:29:03.494067Z","shell.execute_reply.started":"2024-04-19T21:29:00.989417Z","shell.execute_reply":"2024-04-19T21:29:03.493040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this case, Gemma 1.1 at least sticks more to answering the actual question. With 1.0 it was giving some descriptions of solutions as well, even if nothing like that was requested. In this case, the missing information to give answer is also correct.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Attempt 4: Summarize Multiple Solutions\n\nOK, so we have multiple writeups per competition, and automated ranking identification was so far not very successful. How about summarizing multiple writeups from the competition to give an idea of what was kind of solutions people generally used?","metadata":{}},{"cell_type":"markdown","source":"### Try with a single writeup first to experiment prompts\n\nThis prompt still asks for summary with that exact word.","metadata":{}},{"cell_type":"code","source":"prompt_writeup_and_desc = \\\n         f\"you are to answer as a helpful assistant helping understand questions about \" \\\n         f\"a kaggle competition writeup, and the solutions the writeup describes. \" \\\n         f\"a short description of the competition and its goals, and a solution writeup follows. \" \\\n         f\"description: {comp_summary}\\n\\n\" \\\n         f\"writeup: {writeup0_title}:\\n {writeup0}\\n\\n\" \\\n         f\"question: summarize the solution in the above writeup, focusing on used data analysis methods. \" \\\n         f\"Focus on the key points of the solutions and how they might have helped achieving better score in the competition.\\n\\n\" \\\n         f\"answer: \"","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:29:03.495126Z","iopub.execute_input":"2024-04-19T21:29:03.495391Z","iopub.status.idle":"2024-04-19T21:29:03.500353Z","shell.execute_reply.started":"2024-04-19T21:29:03.495369Z","shell.execute_reply":"2024-04-19T21:29:03.499377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nsummary3, summary3_md = gemma_generate(prompt_writeup_and_desc)\nMarkdown(summary3)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:29:03.501687Z","iopub.execute_input":"2024-04-19T21:29:03.501993Z","iopub.status.idle":"2024-04-19T21:29:32.956646Z","shell.execute_reply.started":"2024-04-19T21:29:03.501968Z","shell.execute_reply":"2024-04-19T21:29:32.955704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Change Prompt to Avoid \"Summarize\" term\n\nWith Gemma 1.0, it used to like to add its \"opinion\" to the end of the summary, even if not asked for one. With Gemma 1.1 it seems to be less prone to this. \n\nSo I tried to avoid this with 1.0, by asking for the key points and avoiding the term \"summarize\". It might also drive 1.1 towards a better summary. Lets see:","metadata":{}},{"cell_type":"code","source":"def summarize_description(description):\n    #print(\"summarizing description\")\n    prompt = f\"Summarize the following kaggle competition description in a very concise way. \"\\\n         f\"Omit references to specific algorithms or known solutions, focus on overview of the contest topic and goals.\\n\\n\" \\\n         f\"description: {description}\\n\\n\" \\\n         f\"summary: \"\n    summary, summary_md = gemma_generate(prompt)\n    return summary","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:29:32.957978Z","iopub.execute_input":"2024-04-19T21:29:32.958272Z","iopub.status.idle":"2024-04-19T21:29:32.963440Z","shell.execute_reply.started":"2024-04-19T21:29:32.958247Z","shell.execute_reply":"2024-04-19T21:29:32.962237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now for the modified summarization prompt. Here I do not use the term \"summarize\", but rather as Gemma to just list the key points (from datascience application viewpoint):","metadata":{}},{"cell_type":"code","source":"%%time\n\ndef summarize_writeup_and_desc(df_comp_meta, comp_name, writeup_title, writeup):\n    comp_desc = df_comp_meta[df_comp_meta[\"comp_name\"] == comp_name][\"desc\"].values[0]\n    desc_summary = summarize_description(comp_desc)\n    prompt_writeup_and_desc = \\\n         f\"You are Gemma, a helpful assistant for Kaggle competitors to understand their datascience problems \" \\\n         f\"and propose approaches to solve them. \" \\\n         f\"Your points should have enough details to be useful for similar problem solving, \" \\\n         f\"but not excessively focused on the specific approach in writeup. \" \\\n         f\"More in the sense of overall application lessons for similar problems. \" \\\n         f\"here is a short summary of the competition now being analyzed: \" \\\n         f\"description: {desc_summary}\\n\\n\" \\\n         f\"following will be a writeup of someone who participated in this competition.\" \\\n         f\"list the key points of this from the datascience application viewpoint for similar problem solving.\\n\" \\\n         f\"writeup:\\n {writeup_title}:\\n {writeup}\\n\\n\" \\\n         f\"answer: \\n\"\n    final_summary, final_summary_md = gemma_generate(prompt_writeup_and_desc)\n    return final_summary, final_summary_md\n\ncompetitions = df_writeups[\"Title of Competition\"].unique()\ncomp0 = competitions[0]\ndf_comp0 = df_writeups[df_writeups[\"Title of Competition\"]==comp0]\nwriteup0_title = df_comp0[\"Title of Writeup\"].iloc[0]\ncomp0_name = df_comp0[\"Title of Competition\"].iloc[0]\nwriteup0=df_comp0.iloc[1][\"writeup_clean\"]\nfinal_summary, final_summary_md = summarize_writeup_and_desc(df_comp_meta, comp0_name, writeup0_title, writeup0)\ndisplay(final_summary_md)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:29:32.964598Z","iopub.execute_input":"2024-04-19T21:29:32.964904Z","iopub.status.idle":"2024-04-19T21:30:09.827168Z","shell.execute_reply.started":"2024-04-19T21:29:32.964881Z","shell.execute_reply":"2024-04-19T21:30:09.826253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Gemma 1.0 was a bit more direct in its list of key points at this point. Gemma 1.1 seems to have moved on to also categorizing items under different main categories, and looking for specific and perhaps more generic lessons. A bit different approach, but I like it.\n\nNow it remains to be seen how it can combine with 1.1 these categories in addition to simple list of key points.","metadata":{}},{"cell_type":"markdown","source":"## Try to summarize competition writeups together\n\nTo make it a bit quicker and concise, I summarize only a subset of the competitions here. First a set of helpers to summarize writeups for a competition together:","metadata":{}},{"cell_type":"code","source":"competitions = df_writeups[\"Title of Competition\"].unique()\nwriteup_summaries = []\nwriteup_summaries_md = []\noverall_summaries = []\nprocessed_titles = set()\nprocessed_titles_list = []\nskipped_titles = set()\nskipped_titles_list = []\noverall_summary_prompts = []\nprompts = []\nwriteups = []\nprompt_lengths = []\nwriteup_lengths = []\nsummary_lengths = []\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:30:09.828233Z","iopub.execute_input":"2024-04-19T21:30:09.828493Z","iopub.status.idle":"2024-04-19T21:30:09.834978Z","shell.execute_reply.started":"2024-04-19T21:30:09.828471Z","shell.execute_reply":"2024-04-19T21:30:09.833955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def summarize_competition_writeups(comps_to_summarize, df_writeups_here, model_tokenizer, max_comps=None):\n    skip_count = 0\n    for idx, competition in tqdm(enumerate(comps_to_summarize), total=len(comps_to_summarize)):\n        df_comp = df_writeups_here[df_writeups_here[\"Title of Competition\"] == competition]\n\n        comp_name = df_comp[\"Title of Competition\"].iloc[0]\n        # have to skip overly long writeups. luckily not too many of those\n        max_len = 8192 - 1001 #1000 here is requested count for max new tokens\n        if on_kaggle:\n            max_len = 4096 # Kaggle GPU seems unable to handle long sequence due to memory limit\n        memory_error_msg = \"skipped due to token limit (on Kaggle need shorter due to GPU memory limit)\"\n        #memory_skip = False\n\n        # list_idx is to print position in saved lists, to be able to pick one later\n        list_idx = len(overall_summaries)\n        print(f\"{list_idx}: competition {comp_name}, writeups {df_comp.shape[0]}\")\n        #here filter only the writeups for this competition\n        intermediate = df_comp_meta[df_comp_meta[\"comp_name\"] == comp_name]\n        if intermediate.shape[0] == 0:\n            # the two datasets I use here are not fully in sync, have to skip full mismatches\n            print(\"competition metadata not found, skipping\")\n            skipped_titles.add(competition)\n            skipped_titles_list.append(competition)\n            skip_count += 1\n            continue\n        # on Kaggle it runs a bit slow at time, so had to implement some extra capping support\n        if max_comps is not None and idx - skip_count >= max_comps:\n            print(\"stopping due to maximum summaries reached\")\n            break\n\n        comp_desc = df_comp_meta[df_comp_meta[\"comp_name\"] == comp_name][\"desc\"].values[0]\n        desc_summary = summarize_description(comp_desc)\n\n        comp_writeup_summaries = []\n        comp_writeup_summaries_md = []\n        comp_prompts = []\n        comp_writeups = []\n        comp_prompt_lengths = []\n        comp_writeup_lengths = []\n        comp_summary_lengths = []\n        prompts.append(comp_prompts)\n        writeups.append(comp_writeups)\n        writeup_summaries.append(comp_writeup_summaries)\n        writeup_summaries_md.append(comp_writeup_summaries_md)\n        prompt_lengths.append(comp_prompt_lengths)\n        writeup_lengths.append(comp_writeup_lengths)\n        summary_lengths.append(comp_summary_lengths)\n\n        prompt_overall_summary = \\\n            f\"The following gives a summary of a Kaggle competition description, \" \\\n            f\"and a set of one or more writeups on solutions used in that competition, separated by ======.\\n\\n\" \\\n            f\"Use these to summarize a set of guidelines for ideas on how to approach \" \\\n            f\"a given Kaggle data science competition. \\n\\n\" \\\n            f\"Competition description summary: {desc_summary}\\n\\n\"\n\n        # loop all writeups for each competition\n        for i, row in tqdm(df_comp.iterrows(), total=df_comp.shape[0]):\n            writeup_title = row[\"Title of Writeup\"] \n            writeup = row[\"writeup_clean\"]\n            #print(i)\n\n            prompt_writeup_and_desc = \\\n                 f\"You are Gemma, a helpful assistant for Kaggle competitors to understand their datascience problems \" \\\n                 f\"and propose approaches to solve them. \" \\\n                 f\"Your points should have enough details to be useful for similar problem solving, \" \\\n                 f\"but not excessively focused on the specific approach in writeup. \" \\\n                 f\"More in the sense of overall application lessons for similar problems. \" \\\n                 f\"Here is a short summary of the competition now being analyzed:\\n\\n \" \\\n                 f\"description:\\n {desc_summary}\\n\\n\" \\\n                 f\"Following will be a writeup of someone who participated in this competition. \" \\\n                 f\"List the key points of this from the datascience application viewpoint for similar problem solving.\\n\" \\\n                 f\"Writeup:\\n {writeup_title}:\\n {writeup}\\n\\n\" \\\n                 f\"Answer: \\n\"\n\n            input_ids = model_tokenizer(prompt_writeup_and_desc, return_tensors=\"pt\")\n            prompt_len = len(input_ids[\"input_ids\"][0])\n\n            comp_prompts.append(prompt_writeup_and_desc)\n            comp_writeups.append(writeup)\n            comp_prompt_lengths.append(prompt_len)\n            input_ids = model_tokenizer(writeup_title+\":\\n\"+writeup, return_tensors=\"pt\")\n            writeup_len = len(input_ids[\"input_ids\"][0])\n            comp_writeup_lengths.append(writeup_len)\n            #print(f\"prompt length: {prompt_len}, writeup length: {writeup_len}\")\n\n            if prompt_len > max_len:\n                print(f\"skipping writeup {writeup_title} for competition {competition} due to too high length {prompt_len}.\")\n                #continue\n                summary, summary_md = memory_error_msg, memory_error_msg\n                memory_skip = True\n            else:\n                summary, summary_md = gemma_generate(prompt_writeup_and_desc)\n\n            # Store or process the 'summary' as needed\n            #print(f\"Competition: {comp_name}\")\n            #print(f\"Writeup Title: {writeup_title}\")\n            #print(f\"Summary: {summary}\")\n            #print(\"----------------------\")\n            comp_writeup_summaries.append(summary)\n            comp_writeup_summaries_md.append(summary_md)\n            input_ids = model_tokenizer(summary, return_tensors=\"pt\")\n            summary_len = len(input_ids[\"input_ids\"][0])\n            comp_summary_lengths.append(summary_len)\n\n            input_ids = model_tokenizer(prompt_overall_summary, return_tensors=\"pt\")\n            prompt_overall_summary_len = len(input_ids[\"input_ids\"][0])\n\n            if prompt_overall_summary_len < max_len:\n                prompt_overall_summary += f\"\\n\\n======\\n\\n writeup summary:\\n {summary}\\n\\n\"\n            else:\n                print(f\"skipping adding to overall prompt due to reaching max limit set: {prompt_overall_summary_len} > {max_len}\")\n\n        if competition not in processed_titles:\n            processed_titles.add(competition)\n            processed_titles_list.append(competition)\n\n        prompt_overall_summary += \\\n            f\"\\n\\n======\\n\\n Focus on the key points of the writeups and how they might have helped achieving better score in the competition.\" \\\n            f\"Extract specifically used data analysis methods, and summarize how they are related across writeups.\\n\\n\" \\\n            f\"answer: \"\n        input_ids = model_tokenizer(prompt_overall_summary, return_tensors=\"pt\")\n        overall_prompt_len = len(input_ids[\"input_ids\"][0])\n\n        if overall_prompt_len > max_len+500:\n            print(f\"skipping overall summary for competition {competition} due to too high length {overall_prompt_len}.\")\n            overall_summary, overall_summary_md = memory_error_msg, memory_error_msg\n        else:\n            overall_summary, overall_summary_md = gemma_generate(prompt_overall_summary)\n\n        input_ids = model_tokenizer(overall_summary, return_tensors=\"pt\")\n        overall_summary_len = len(input_ids[\"input_ids\"][0])\n        \n        #print(overall_summary)\n        print(f\"     prompt lengths={comp_prompt_lengths}\")\n        print(f\"    writeup lengths={comp_writeup_lengths}\")\n        print(f\"    summary lengths={comp_summary_lengths}\")\n        print(f\"    total writeups length:{sum(comp_writeup_lengths)}, overall prompt length: {overall_prompt_len}, overall summary length: {overall_summary_len}\")\n        overall_summaries.append(overall_summary_md)\n        overall_summary_prompts.append(prompt_overall_summary)\n    print(f\"total of {len(prompts)} competitions processed\")\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:30:09.836571Z","iopub.execute_input":"2024-04-19T21:30:09.836965Z","iopub.status.idle":"2024-04-19T21:30:09.861561Z","shell.execute_reply.started":"2024-04-19T21:30:09.836922Z","shell.execute_reply":"2024-04-19T21:30:09.860714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With the above method, the following notebook cells will create overall summaries of competition writeups. This approach first uses Gemma to summarize each writeup for a competition separately, takes these summaries together, and finally uses Gemma to summarize the key points from all these writeup summaries as one. It prints out the following information:\n\n- writeup count: how many writeups are there for the competition in the dataset.\n- prompt lengths: how long is the separate writeup summary prompt used as counted in Gemma tokens. this prompt includes the writeup.\n- writeup lengths: length of a separate writeup alone, without the rest of the prompt.\n- summary lengths: length of the Gemma built summaries (list of key points) for each writeup\n- total writeup length: length of all writeups for the competition, in Gemma tokens. if we concatenate the raw writeups in full\n- overall prompt length: the summary prompt length using summaries of each writeup as input and not full writeups.\n- overall summary length: length of the final overall summary generated by Gemma for overall prompt.","metadata":{}},{"cell_type":"code","source":"%%time\nsummarize_competition_writeups(competitions[:6], df_writeups, tokenizer)\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:30:09.862829Z","iopub.execute_input":"2024-04-19T21:30:09.863184Z","iopub.status.idle":"2024-04-19T21:36:26.546854Z","shell.execute_reply.started":"2024-04-19T21:30:09.863151Z","shell.execute_reply":"2024-04-19T21:36:26.545714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As shown above, most of these competitions actually just have one writeup in this dataset. Many of them are also very short, with the shortest being only 15 tokens. A few competitions had more than one writeup, and some had longer writeups. So lets take a look at some of these different types in a bit more detail.","metadata":{}},{"cell_type":"markdown","source":"### Overall summary, competition 1\n\nThe first competition has the most writeups in this set, 5 writeups. Lets see how all of them get summarized together.\n","metadata":{}},{"cell_type":"code","source":"overall_summaries[0]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.548771Z","iopub.execute_input":"2024-04-19T21:36:26.549266Z","iopub.status.idle":"2024-04-19T21:36:26.555839Z","shell.execute_reply.started":"2024-04-19T21:36:26.549230Z","shell.execute_reply":"2024-04-19T21:36:26.554841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With Gemma 1.0, it used to just provide a single list and it was not so clear how the writeups were related. Here it seems to do much better in finding both common analysis methods and interconnections, along with specific solution approaches. Looks good. If I was developing the model, I guess I would still dig deeper into comparisons, but I am quite happy with the results here for 1.1 compared to 1.0.\n\nIt also now is missing its opinion pieces as I already noted above. Which is good, since I was not looking for those.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Overall summary, competition 2\n\nAs seen in the above processing, competition 2 had a single writeup, that had the shortest set of 15 tokens. Lets see what Gemma has to say about that. \n\nBut first, lets check what is such as super-short writeup anyway:\n","metadata":{}},{"cell_type":"code","source":"writeups[1]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.557060Z","iopub.execute_input":"2024-04-19T21:36:26.557364Z","iopub.status.idle":"2024-04-19T21:36:26.570103Z","shell.execute_reply.started":"2024-04-19T21:36:26.557338Z","shell.execute_reply":"2024-04-19T21:36:26.569161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This looks even a bit suspicious, as if something is left out of the writeup. Let's see if the problem is in what I picked from the dataframe, in the collection method of the writeups, or maybe the writeup is just this :).","metadata":{}},{"cell_type":"code","source":"df_writeups[df_writeups[\"Title of Competition\"] == \"RTA Freeway Travel Time Prediction\"]\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.571553Z","iopub.execute_input":"2024-04-19T21:36:26.572156Z","iopub.status.idle":"2024-04-19T21:36:26.591178Z","shell.execute_reply.started":"2024-04-19T21:36:26.572124Z","shell.execute_reply":"2024-04-19T21:36:26.590118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_writeups[df_writeups[\"Title of Competition\"] == \"RTA Freeway Travel Time Prediction\"][\"writeup\"]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.592586Z","iopub.execute_input":"2024-04-19T21:36:26.592913Z","iopub.status.idle":"2024-04-19T21:36:26.605300Z","shell.execute_reply.started":"2024-04-19T21:36:26.592886Z","shell.execute_reply":"2024-04-19T21:36:26.604140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"At least it seems that this is the actual writeup in the data I am using. \n\nSo what does Gemma say when it tries to summarize this writeup that does not actually say much of anything?","metadata":{}},{"cell_type":"code","source":"Markdown(writeup_summaries[1][0])","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.606964Z","iopub.execute_input":"2024-04-19T21:36:26.607488Z","iopub.status.idle":"2024-04-19T21:36:26.618113Z","shell.execute_reply.started":"2024-04-19T21:36:26.607450Z","shell.execute_reply":"2024-04-19T21:36:26.616803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The actual input I used for Gemma is the competition description summary + competition title + writeup title + the writeup, so the actual prompt was this: ","metadata":{}},{"cell_type":"code","source":"Markdown(prompts[1][0])","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.619762Z","iopub.execute_input":"2024-04-19T21:36:26.620057Z","iopub.status.idle":"2024-04-19T21:36:26.628356Z","shell.execute_reply.started":"2024-04-19T21:36:26.620014Z","shell.execute_reply":"2024-04-19T21:36:26.627417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well, much like Gemma 1.0, 1.1 seems to have hallucinated complete solutions when given none, laced with some information from the competition description. Very bad for that again, I would say.\n\n\nSo how does Gemma summarize this summary itself? Remember, the approach I trial at this point is to first summarize all the writeups for a competition, and from these a single overall summary. Since there is just one writeup in this competition, Gemma just tries to re-summarize its own summary now. \n\nLet's see what it says:\n","metadata":{}},{"cell_type":"code","source":"overall_summaries[1]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.629784Z","iopub.execute_input":"2024-04-19T21:36:26.630066Z","iopub.status.idle":"2024-04-19T21:36:26.638690Z","shell.execute_reply.started":"2024-04-19T21:36:26.630042Z","shell.execute_reply":"2024-04-19T21:36:26.637750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well, at least it seems to do better at summarizing its own hallucinations. I guess it would be important to make sure we have some proper input before asking for a summary. Since we dont usually manually check everything in a big set of data that would be useful to remember in testing and implementing something like this.\n\nIn this regard it is quite similar betweeb Gemma 1.0 and 1.1.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Overall summary, competition 3\n\nThis 3rd competition in the dataset has 2 writeups. One very short (20 tokens), and one longer (561 tokens). \n\nLets see what these two writeups are, and how Gemma handles them:","metadata":{}},{"cell_type":"code","source":"writeups[2][0]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.640117Z","iopub.execute_input":"2024-04-19T21:36:26.640869Z","iopub.status.idle":"2024-04-19T21:36:26.649679Z","shell.execute_reply.started":"2024-04-19T21:36:26.640836Z","shell.execute_reply":"2024-04-19T21:36:26.648700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Markdown(writeups[2][1])","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.650814Z","iopub.execute_input":"2024-04-19T21:36:26.651416Z","iopub.status.idle":"2024-04-19T21:36:26.660030Z","shell.execute_reply.started":"2024-04-19T21:36:26.651390Z","shell.execute_reply":"2024-04-19T21:36:26.659054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, the first writeup seems to just refer to some attachment. I did now know people could do that on Kaggle. \n\nThe second one has some content. So lets see how Gemma handles this combination.\n\nFirst the separate summary for the first writeup:","metadata":{}},{"cell_type":"code","source":"Markdown(writeup_summaries[2][0])","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.661551Z","iopub.execute_input":"2024-04-19T21:36:26.662018Z","iopub.status.idle":"2024-04-19T21:36:26.672175Z","shell.execute_reply.started":"2024-04-19T21:36:26.661984Z","shell.execute_reply":"2024-04-19T21:36:26.671231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With Gemma 1.0, it was more correct in stating that for the first writeup there was no solution provided. Here with Gemma 1.1 it again hallucinated a solution. I guess in general it is just best to remember to check the input first if using Gemma this way.\n\nThis is what the prompt with the competition description summary and the writeup looked like in this case:","metadata":{}},{"cell_type":"code","source":"Markdown(prompts[2][0])","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.673199Z","iopub.execute_input":"2024-04-19T21:36:26.673456Z","iopub.status.idle":"2024-04-19T21:36:26.683239Z","shell.execute_reply.started":"2024-04-19T21:36:26.673433Z","shell.execute_reply":"2024-04-19T21:36:26.682234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Going from that to the above summary is quite a hallucination.\n\nHere is the summary for the second, longer writeup, in this competition:","metadata":{}},{"cell_type":"code","source":"Markdown(writeup_summaries[2][1])","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.684488Z","iopub.execute_input":"2024-04-19T21:36:26.684781Z","iopub.status.idle":"2024-04-19T21:36:26.694268Z","shell.execute_reply.started":"2024-04-19T21:36:26.684758Z","shell.execute_reply":"2024-04-19T21:36:26.693303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And the overall summary from the combination of these two writeup summaries:","metadata":{}},{"cell_type":"code","source":"overall_summaries[2]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.695300Z","iopub.execute_input":"2024-04-19T21:36:26.695583Z","iopub.status.idle":"2024-04-19T21:36:26.706752Z","shell.execute_reply.started":"2024-04-19T21:36:26.695557Z","shell.execute_reply":"2024-04-19T21:36:26.705578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This overall summary seems to be quite accurate. At least it is not picking up any hallucinations from the very short one. Probably the accuracy and lack of hallucination in the short summary helps too.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Summaries for Competitions with Longer and Larger Set of Writeups\n\nMost of the writeups in the above set are quite similar to the three I checked above. Most have only one writeup, and at most there are three writeups, some of which are usually quite short as well. \n\nBesides these few and short writeups, an interesting combination to check is longer writeups, and competitions with a larger set of writeups.\n\nThe writeups in this dataset appear to be in chronological ordering, so the older ones (starting from year 2010) are first in the list/dataframe. The newest ones (seems to be around 2022) are last. In general it seems that over time there are more writeups for the later competitions, and often with more text. Perhaps more people on Kaggle, more writing on them, or people gaming the discussion badges more. \n\nWhatever the reason, here are the summaries for the last 20 competitions in the dataset, similar to the above first ones from the dataset (with few and shorter writeups):","metadata":{}},{"cell_type":"code","source":"for competition in tqdm(competitions[-20:]):\n    df_comp = df_writeups[df_writeups[\"Title of Competition\"] == competition]\n    comp_name = df_comp[\"Title of Competition\"].iloc[0]\n    comp_meta = df_comp_meta[df_comp_meta[\"comp_name\"] == comp_name]\n    found = comp_meta.shape[0] > 0\n    if found:\n        print(f\"competition {comp_name}, writeups {df_comp.shape[0]}\")\n    else:\n        print(f\"competition {comp_name}, writeups {df_comp.shape[0]}, no metadata\")\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.708065Z","iopub.execute_input":"2024-04-19T21:36:26.708356Z","iopub.status.idle":"2024-04-19T21:36:26.767803Z","shell.execute_reply.started":"2024-04-19T21:36:26.708332Z","shell.execute_reply":"2024-04-19T21:36:26.766726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"With the amount of writeups these have, it takes a bit of a long time to run them all. Especially on Kaggle with the slower GPU's and limited GPU memory. So I selected a few to run here after initially trying all the 20 listed above:\n- Feedback prize: 19 writeups, the second highest in the list that has metadata\n- America Express: 31 writeups, the most writeups in list with metadata\n- Mayo Clinit: 10 writeups, least in the set with metadata\n- AI Village: 18 writeups, but with many very short ones\n\nIt seems to be a decent set to compliment the earlier one with few writeups and many short ones.","metadata":{}},{"cell_type":"code","source":"selected_competitions = [\n                         \"Feedback Prize - Predicting Effective Arguments\",\n                         \"American Express - Default Prediction\",\n                         \"Mayo Clinic - STRIP AI\",\n                         \"AI Village Capture the Flag @ DEFCON\",\n                        ]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.769029Z","iopub.execute_input":"2024-04-19T21:36:26.769316Z","iopub.status.idle":"2024-04-19T21:36:26.774011Z","shell.execute_reply.started":"2024-04-19T21:36:26.769291Z","shell.execute_reply":"2024-04-19T21:36:26.772913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"NOTE: following have capped context on Kaggle due to GPU memory limit","metadata":{}},{"cell_type":"code","source":"#summarize_competition_writeups(competitions[-20:])\nsummarize_competition_writeups(selected_competitions, df_writeups, tokenizer)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T21:36:26.775154Z","iopub.execute_input":"2024-04-19T21:36:26.775427Z","iopub.status.idle":"2024-04-19T22:06:53.921343Z","shell.execute_reply.started":"2024-04-19T21:36:26.775396Z","shell.execute_reply":"2024-04-19T22:06:53.920303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All the skipping due to too long context is due to having to limit the prompt on Kaggle with the limited GPU VRAM. But it still captures a large enough set for this experiment.\n\nIt also seems that the competition metadata I use does not have metadata for all the latest competition writeups. Its not a big deal since I can still explore the writeups with Gemma just fine.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Overall about Overall Summarization\n\nAs I noted, it seems the number of writeups and their length increases quite a lot over time. Here there are 10-31 writeups for the ones that I had also the metadata for. \n\nThe longest combination of the writeups I checked above (American Express Default Prediction, 31 writeups) is 30k Gemma tokens in length, meaning it would not fit into Gemma's 8k window.\n\nWhile the 30k tokens would be too much for Gemma context window of 8k, as shown above, the combined summaries of those writeups fit in 8k (just had to cap it on Kaggle to 4k). The longest (American Express) going from 30k too 7.1k tokens with using the combined write summaries as input vs the raw writeups. \n\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"First, a look at the first of the resulting overall summaries. This one had 19 writeups (or their summaries) as input as seen above:","metadata":{}},{"cell_type":"code","source":"overall_summaries[5]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:06:53.922785Z","iopub.execute_input":"2024-04-19T22:06:53.923084Z","iopub.status.idle":"2024-04-19T22:06:53.929286Z","shell.execute_reply.started":"2024-04-19T22:06:53.923058Z","shell.execute_reply":"2024-04-19T22:06:53.928361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is followed by the longest one (American Express), which had 31 writeups, with the writeup summaries together amounting to over 7k tokens (it not capped as I had to do on Kaggle):","metadata":{}},{"cell_type":"code","source":"overall_summaries[6]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:06:53.930517Z","iopub.execute_input":"2024-04-19T22:06:53.930843Z","iopub.status.idle":"2024-04-19T22:06:53.941387Z","shell.execute_reply.started":"2024-04-19T22:06:53.930819Z","shell.execute_reply":"2024-04-19T22:06:53.940423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And one more for the road, this time the one with fewest writeups, meaning 10 in this case:","metadata":{}},{"cell_type":"code","source":"overall_summaries[7]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:06:53.942546Z","iopub.execute_input":"2024-04-19T22:06:53.942894Z","iopub.status.idle":"2024-04-19T22:06:53.952518Z","shell.execute_reply.started":"2024-04-19T22:06:53.942863Z","shell.execute_reply":"2024-04-19T22:06:53.951565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, I can see one more interesting one in the above set. The \"AI Village Capture the Flag @ DEFCON\" competition has multiple writeups that are very short (under 100 tokens), along with a similar count of longer ones. Let's see how that looks in an overall summary:","metadata":{}},{"cell_type":"code","source":"overall_summaries[8]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:06:53.953754Z","iopub.execute_input":"2024-04-19T22:06:53.954019Z","iopub.status.idle":"2024-04-19T22:06:53.962848Z","shell.execute_reply.started":"2024-04-19T22:06:53.953996Z","shell.execute_reply":"2024-04-19T22:06:53.961979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I found this a bit confusing as it seem to discuss topics related to image processing as well as many other topics. But checking the competition description, it seems to have been a security related competition with multiple challenges related to math, natural language, and image challenges. So the overall summary seems to make sense.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The set of smaller writeups mostly seem to be about referencing solutions in notebooks or otherwise. Without spamming this notebook too much, here is one example:","metadata":{}},{"cell_type":"code","source":"Markdown(writeups[8][0])","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:06:53.964016Z","iopub.execute_input":"2024-04-19T22:06:53.964320Z","iopub.status.idle":"2024-04-19T22:06:53.973060Z","shell.execute_reply.started":"2024-04-19T22:06:53.964296Z","shell.execute_reply":"2024-04-19T22:06:53.972261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As I noted before, perhaps it would be useful to filter out the very short writeups with very little content. In this case, along with a larger number of better writeups and their summaries it seems to have little effect though. A working ranking-based selection might also help to filter such larger sets.\n\nWith Gemma 1.0, all the summaries tended to be around 100-300 tokens, and have a list of key points. Here with Gemma 1.1, the overall summary seems to be around 400 tokens, and thus quite similar size again. And also summarizing in categories now, along with more specific and generic categorization of methods, as compared to the plainer key points from Gemma 1.0.\n","metadata":{}},{"cell_type":"markdown","source":"# Summarizing in Sub-Sets vs All at Once\n\nUsing the initial results from this notebook, I built a [second one](https://www.kaggle.com/code/donkeys/qa-solutions-from-past-competitions) to experiment a bit with a RAG approach to help a user find interesting competitions related to the one they wish to participate in, get a summary of any writeups for that competition, and ask questions about solutions described in those writeups, using the summaries as a basis for questions and writeups as basis for more detailed answers.\n\nIn trying that, I realized I wanted to have all writeups summarized regardless of how many and long writeups the competition had. In my above trials, I capped the max context length at 4k for Kaggle, and excluded some writeups if the total became too long. For example, in the above parts of this notebook it shows how the _American Express - Default Prediction_ competition had 31 writeups, and some of the were skipped beceause they did not fit in the 4k token limit I had to use on Kaggle. And even with the full 8k Gemma context, some of the ones with a really large writeup sets still get cut off on my desktop setup with more VRAM.\n\nTo get even the ones with a large number of writeups summarized, I tried summarizing all writeups in several smaller sets per competition, and from there to a final summary. If this works fine, it should also work for all the writeup counts.","metadata":{}},{"cell_type":"markdown","source":"## Filter to Only Competitions with only N Writeups\n\nFirst to find all competitions with at least some writeups, to form a basis for what we are going to summarize all together. Well, it also allows trialing with a minimum set of writeups if wanting the see the effects better.","metadata":{}},{"cell_type":"code","source":"writeups = []\nused_names = []\nskipped_names_empty = []\nskipped_names_few = []\n# writeups need metadata to have a description that I use for the summary\nskipped_names_nometa = []\n\npbar_selected = tqdm(total=0, desc=\"Selected Writeups\")\npbar_skipped_empty = tqdm(total=0, desc=\"Skipped (No Writeups)\")\npbar_skipped_few = tqdm(total=0, desc=\"Skipped (Too Few Writeups)\")\npbar_skipped_nometa = tqdm(total=0, desc=\"Skipped (No Metadata)\")\n\n#if on_kaggle:\n#    comps_to_find = 20\n#else:\n#    # should be all\n#    comps_to_find = 10000\n\nfor idx, name in enumerate(competitions):\n    comp_writeups = df_writeups[df_writeups[\"Title of Competition\"] == name]\n    print(f\"processing {name}, writeups: {comp_writeups.shape[0]}\")\n    if len(comp_writeups) == 0:\n        print(f\"skipping {name} due to no writeups\")\n        skipped_names_empty.append(name)\n        pbar_skipped_empty.update(1)\n        continue\n    comp_meta = df_comp_meta[df_comp_meta[\"comp_name\"] == name]\n    found = comp_meta.shape[0] > 0\n    if not found:\n        print(f\"skipping {name} due to no metadata\")\n        skipped_names_nometa.append(name)\n        pbar_skipped_nometa.update(1)\n        continue\n        \n    #if len(comp_writeups) < 5:\n    #    #print(f\"skipping {name} due to too few writeups\")\n    #    skipped_names_few.append(name)\n    #    pbar_skipped_few.update(1)\n    #    continue    \n    used_names.append(name)\n    writeups.append(comp_writeups)\n    pbar_selected.update(1)\n#    if len(used_names) >= comps_to_find:\n#        break","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:06:53.974279Z","iopub.execute_input":"2024-04-19T22:06:53.974556Z","iopub.status.idle":"2024-04-19T22:06:54.498425Z","shell.execute_reply.started":"2024-04-19T22:06:53.974523Z","shell.execute_reply":"2024-04-19T22:06:54.497439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The above illustrates also how the latter competitions have a larger set of writeups each.\n\n## Split Large Writeup Sets to Smaller Ones\n\nFirst, a helper function to split a large set of writeups into somewhat even size subsets. So we dont end up with subsets with sizes like [10, 10, 10, 1]. Rather in such case [10, 10, 6, 5] so there is enough summarizing for everyone.","metadata":{}},{"cell_type":"code","source":"def split_list(data, max_sublist_size):\n  sublists = []\n  current_sublist = []\n  for idx, item in data.iterrows():\n    if len(current_sublist) < max_sublist_size:\n      current_sublist.append(item)\n    else:\n      sublists.append(current_sublist)\n      current_sublist = [item]\n  # Add the last sublist, even if it's shorter than max_sublist_size\n  sublists.append(current_sublist)\n\n  # Ensure the last sublists are more evenly distributed\n  last_sublists = sublists[-2:]  # Get the last two sublists\n  total_length = sum(len(sublist) for sublist in last_sublists)\n  ideal_length = total_length // len(last_sublists)\n\n  # Distribute elements until a balanced state is achieved\n  while len(last_sublists[0]) > ideal_length and len(last_sublists) > 1:\n      #print(len(last_sublists[0]))\n      last_sublists[1].append(last_sublists[0].pop(0))\n\n  # If there are still more elements in the second-to-last sublist, distribute to previous ones\n  if len(last_sublists) > 1 and len(last_sublists[1]) > ideal_length:\n    for i in range(len(sublists) - 2, 0, -1):\n      if len(sublists[i]) < max_sublist_size:\n        sublists[i].append(last_sublists[1].pop(0))\n        break\n\n  return sublists\n\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:06:54.499564Z","iopub.execute_input":"2024-04-19T22:06:54.499845Z","iopub.status.idle":"2024-04-19T22:06:54.509362Z","shell.execute_reply.started":"2024-04-19T22:06:54.499821Z","shell.execute_reply":"2024-04-19T22:06:54.508355Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next use the above helper to actually split the writeups into more even size subsets:","metadata":{}},{"cell_type":"code","source":"writeup_dataframes = []\n\nN = 5\n#N = 10\n#N = \"ALL\"\n\nsplit_size = N\n\n# ALL just refers to summarizing all together, so no need for subsets then\n# it is what I used for the desktop env where I ran full summaries for comparison\nif N == \"ALL\":\n    N2 = 10000\nelse:\n    N2 = N\nif True:\n    for comp_writeups in writeups:\n        writeups_sublists = split_list(comp_writeups, N2)\n        #print([len(sub) for sub in writeups_sublists])\n        writeups_subframes = [pd.DataFrame(sub) for sub in writeups_sublists]\n        #print([subframe.shape[0] for subframe in writeups_subframes])\n        writeup_dataframes.append(writeups_subframes)\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:06:54.510778Z","iopub.execute_input":"2024-04-19T22:06:54.511061Z","iopub.status.idle":"2024-04-19T22:06:55.050393Z","shell.execute_reply.started":"2024-04-19T22:06:54.511037Z","shell.execute_reply":"2024-04-19T22:06:55.049463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Collect Writeups for Subsets","metadata":{}},{"cell_type":"code","source":"# create new list of writeup dataframes filtering out all writeups for competitions in skipped_titles\nwriteup_dataframes_filtered = []\nnames_filtered = []\nfor idx, comp_dataframes in enumerate(writeup_dataframes):\n    comp_name = comp_dataframes[0][\"Title of Competition\"].iloc[0]\n    if comp_name in skipped_titles:\n        continue\n    names_filtered.append(comp_name)\n    writeup_dataframes_filtered.append(comp_dataframes)\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:06:55.051612Z","iopub.execute_input":"2024-04-19T22:06:55.051968Z","iopub.status.idle":"2024-04-19T22:06:55.077730Z","shell.execute_reply.started":"2024-04-19T22:06:55.051938Z","shell.execute_reply":"2024-04-19T22:06:55.076873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# concatenate all dataframes in the writeup_dataframes list into a single dataframe\n# just useful when summarizing all together for a competition\ndf_writeups_all = pd.concat([df for sublist in writeup_dataframes_filtered for df in sublist])","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:06:55.079138Z","iopub.execute_input":"2024-04-19T22:06:55.079390Z","iopub.status.idle":"2024-04-19T22:06:55.114530Z","shell.execute_reply.started":"2024-04-19T22:06:55.079368Z","shell.execute_reply":"2024-04-19T22:06:55.113693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(skipped_names_nometa)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:06:55.115489Z","iopub.execute_input":"2024-04-19T22:06:55.115768Z","iopub.status.idle":"2024-04-19T22:06:55.121379Z","shell.execute_reply.started":"2024-04-19T22:06:55.115745Z","shell.execute_reply":"2024-04-19T22:06:55.120548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# these are just to check up the filtering is OK. \n# Previous filtering has given 310-27=283 competitions left, so thats the number needed here\nprint(len(writeup_dataframes_filtered))\nprint(df_writeups_all.shape)\nprint(df_writeups_all[\"Title of Competition\"].nunique())","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:06:55.122734Z","iopub.execute_input":"2024-04-19T22:06:55.123001Z","iopub.status.idle":"2024-04-19T22:06:55.131110Z","shell.execute_reply.started":"2024-04-19T22:06:55.122979Z","shell.execute_reply":"2024-04-19T22:06:55.130185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Summarize the Selected Writeups as Input for Subsets","metadata":{}},{"cell_type":"code","source":"# first have to reset the lists to not pollute with previous experiments in this notebook\nwriteup_summaries = []\nwriteup_summaries_md = []\noverall_summaries = []\nprocessed_titles = set()\nprocessed_titles_list = []\nskipped_titles = set()\nskipped_titles_list = []\noverall_summary_prompts = []\nprompts = []\nwriteups = []\nprompt_lengths = []\nwriteup_lengths = []\nsummary_lengths = []\nfinal_names = []\n\n# subset is how many competitions are used here, limiting it on Kaggle for GPU time\nif on_kaggle:\n    subset = 5\nelse:\n    subset = len(used_names)\n\nif N != \"ALL\":\n    # taking the latest competitions subset to have competitions with more writeups\n    start_idx = len(used_names) - subset\n    for x in tqdm(range(start_idx, start_idx+subset)):\n        print(x)\n        name = used_names[x]\n        final_names.append(name)\n        # writeup_dataframes holds the writeups for the competition name at same index\n        for df_subframe in writeup_dataframes_filtered[x]:\n            summarize_competition_writeups([name], df_subframe, tokenizer)\nelse:\n    # this was on local desktop with more resources and GPU time\n    # in this case there are splits or subsets, all comp writeup summaries are summarized as one\n    summarize_competition_writeups(used_names, df_writeups_all, tokenizer)\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:06:55.132236Z","iopub.execute_input":"2024-04-19T22:06:55.132528Z","iopub.status.idle":"2024-04-19T22:40:50.892508Z","shell.execute_reply.started":"2024-04-19T22:06:55.132504Z","shell.execute_reply":"2024-04-19T22:40:50.891503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Build the Subsets","metadata":{"execution":{"iopub.execute_input":"2024-04-13T07:13:42.007830Z","iopub.status.busy":"2024-04-13T07:13:42.006962Z","iopub.status.idle":"2024-04-13T07:13:42.011354Z","shell.execute_reply":"2024-04-13T07:13:42.010480Z","shell.execute_reply.started":"2024-04-13T07:13:42.007799Z"}}},{"cell_type":"markdown","source":"When we come here, we have writeup summaries in a single flat list. Now need to re-group them so that each subset is its own list of writeup summaries for that subset.","metadata":{}},{"cell_type":"code","source":"def recombine_summaries(summaries, sublist_lengths):\n    result = []\n    start_index = 0\n    for idx, length in enumerate(sublist_lengths):\n        end_index = start_index + length\n        sublist = summaries[start_index:end_index]\n        if len(sublist) == 0:\n            print(f\"stopping recombo due to end of list at idx {idx}\")\n            break\n        result.append(sublist)\n        start_index = end_index\n\n    return result \n\n# Example usage\nsummaries = [\"summary1_1\", \"summary1_2\", \"summary2_1\", \"summary2_2\", \"summary3_1\"]\nsublist_lengths = [2, 2, 1]\n\nrecombine_summaries(summaries, sublist_lengths)\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:40:50.893867Z","iopub.execute_input":"2024-04-19T22:40:50.894149Z","iopub.status.idle":"2024-04-19T22:40:50.903784Z","shell.execute_reply.started":"2024-04-19T22:40:50.894125Z","shell.execute_reply":"2024-04-19T22:40:50.902893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# this should produce 5 lists of subsets, since I capped it at 5 for Kaggle above\nsublist_lengths = [len(sub) for sub in writeup_dataframes_filtered[-subset:]]\nlen(sublist_lengths)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:40:50.905133Z","iopub.execute_input":"2024-04-19T22:40:50.905512Z","iopub.status.idle":"2024-04-19T22:40:50.924773Z","shell.execute_reply.started":"2024-04-19T22:40:50.905478Z","shell.execute_reply":"2024-04-19T22:40:50.923741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if N != \"ALL\":\n    combined = recombine_summaries(overall_summaries, sublist_lengths)\nelse:\n    combined = None","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:40:50.926101Z","iopub.execute_input":"2024-04-19T22:40:50.926786Z","iopub.status.idle":"2024-04-19T22:40:50.941819Z","shell.execute_reply.started":"2024-04-19T22:40:50.926752Z","shell.execute_reply":"2024-04-19T22:40:50.940904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Summarize the Subsets Separately\n\nNow for summarizing all the summaries so far together into one set of key points per competition.\n","metadata":{}},{"cell_type":"code","source":"def summarize_subsummaries(sublist, print_prompt=False, prompt_style=\"simple\"):\n    #print(f\"summarizing: {sublist}\")\n    writeup_summaries = [summary.data for summary in sublist]\n    if prompt_style == \"simple\":\n        points = \"\\n\\n\".join(writeup_summaries)\n    elif prompt_style == \"subtitle writeup\":\n        points = \"\"\n        for idx, writeup_summary in enumerate(writeup_summaries):\n            points += \"\\n\\n\"\n            points += f\"writeup {idx}\\n\"\n            points += writeup_summary\n    else:\n        points = \"\"\n        for idx, writeup_summary in enumerate(writeup_summaries):\n            points += \"\\n\\n\"\n            points += f\"writeups summary {idx}\\n\"\n            points += writeup_summary\n    prompt = f\"\"\"collect the key points listed in these writeup summaries into one list of key points.\n    include the descriptions given for these key points.\n    writeup summaries:\n    {points}\n    \"\"\"\n    # trying to remove <eos> just in case Gemma thinks the sequence ends in the middle\n    prompt = prompt.replace(\"<eos>\", \"\")\n    summary, summary_md = gemma_generate(prompt, print_prompt)\n    if print_prompt:\n        # this printing is just for debugging, because Gemma is giving some \n        # rather strange output at times it seems\n        input_ids = tokenizer(prompt, return_tensors=\"pt\")\n        prompt_len = len(input_ids[\"input_ids\"][0])\n        print(f\"------- PROMPT ------- {prompt_len} tokens ------\")\n        print(prompt)\n        input_ids = tokenizer(summary, return_tensors=\"pt\")\n        summary_len = len(input_ids[\"input_ids\"][0])\n        print(f\"------ SUMMARY: ------ {summary_len} tokens ------\")\n        print(summary)\n    return summary, summary_md\n\n#print(type(tokenizer))\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:40:50.942991Z","iopub.execute_input":"2024-04-19T22:40:50.943328Z","iopub.status.idle":"2024-04-19T22:40:50.954078Z","shell.execute_reply.started":"2024-04-19T22:40:50.943296Z","shell.execute_reply":"2024-04-19T22:40:50.953150Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Summarize Competition Subset Summaries to One Per Competition\n\nCompetition writeup summaries -> summaries for subsets of those -> one summary per competition.","metadata":{}},{"cell_type":"code","source":"if N != \"ALL\":\n    len(combined)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:40:50.955238Z","iopub.execute_input":"2024-04-19T22:40:50.955510Z","iopub.status.idle":"2024-04-19T22:40:50.965164Z","shell.execute_reply.started":"2024-04-19T22:40:50.955487Z","shell.execute_reply":"2024-04-19T22:40:50.964234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"resummarized = None\nresummarized_md = None\nresummarized_simple = []\nresummarized_md_simple = []\nresummarized_writeup = []\nresummarized_md_writeup = []\nresummarized_writeup_summary = []\nresummarized_md_writeup_summary = []\n\nif N != \"ALL\":\n    for idx, combo in tqdm(enumerate(combined), total=len(combined)):\n        combo_filtered = [x for x in combo if not isinstance(x, str)]\n        if len(combo_filtered) == 0:\n            print(f\"no data, skipping idx: {idx}\")\n            continue\n        title = final_names[idx]#comp_dataframes[0][\"Title of Competition\"].iloc[0]\n        #print(title)\n        #if on_kaggle:\n        #    # just to keep the story on kaggle consistent :)\n        #    summary, summary_md = summarize_subsummaries(combo_filtered)\n        #    resummarized_simple.append(summary)\n        #    resummarized_md_simple.append(summary_md)\n        #else: #TODO: here the other summary types\n        summary, summary_md = summarize_subsummaries(combo_filtered)\n        resummarized_simple.append(summary)\n        resummarized_md_simple.append(summary_md)\n\n        summary, summary_md = summarize_subsummaries(combo_filtered, False, \"subtitle writeup\")\n        resummarized_writeup.append(summary)\n        resummarized_md_writeup.append(summary_md)\n\n        summary, summary_md = summarize_subsummaries(combo_filtered, False, \"writeup summaries title\")\n        resummarized_writeup_summary.append(summary)\n        resummarized_md_writeup_summary.append(summary_md)\n\nelse:\n    plain_summaries = []\n    for idx, summary in enumerate(overall_summaries):\n        if isinstance(summary, str):\n            print(f\"string: {idx}:{summary}\")\n            plain_summaries.append(summary)\n        else:\n            plain_summaries.append(summary.data)\n            \n    #plain_summaries = [summary.data for summary in overall_summaries]\n    resummarized_simple.extend(plain_summaries)\n    resummarized_md_simple.extend(overall_summaries)\n    resummarized_comps = final_names","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:40:50.966448Z","iopub.execute_input":"2024-04-19T22:40:50.966978Z","iopub.status.idle":"2024-04-19T22:42:24.134890Z","shell.execute_reply.started":"2024-04-19T22:40:50.966946Z","shell.execute_reply":"2024-04-19T22:42:24.133838Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"OK, so that is for the 5 competitions I filtered above for Kaggle. Now lets look at them and what do they say, because these latter ones are ones where Gemma seems to lose the context for some reason:","metadata":{}},{"cell_type":"code","source":"print(len(resummarized_simple))\nprint(len(resummarized_writeup))\nprint(len(resummarized_writeup_summary))\n\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:24.136203Z","iopub.execute_input":"2024-04-19T22:42:24.136535Z","iopub.status.idle":"2024-04-19T22:42:24.142043Z","shell.execute_reply.started":"2024-04-19T22:42:24.136509Z","shell.execute_reply":"2024-04-19T22:42:24.141077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Markdown(resummarized_simple[0])","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:24.143256Z","iopub.execute_input":"2024-04-19T22:42:24.143543Z","iopub.status.idle":"2024-04-19T22:42:24.157854Z","shell.execute_reply.started":"2024-04-19T22:42:24.143518Z","shell.execute_reply":"2024-04-19T22:42:24.157029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### It all Looks Rather Lost\n\nOK, so with Gemma 1.0, this resulted in a list of generic points that had no real content. In this case, it seems to have simply resulted in a one-liner \"Note: This list includes key points from both writeup summaries provided.\". So all is indeed totally lost here.\n\nLets take a closer look.\n\n## Closer look at Selected Competition 1 and its Final Summary vs Parts\n\nLets try to verify that I did not mess it up in giving wrong inputs or misinterpret the output. The first competition from that list of 5 selected above, and its subset summaries:","metadata":{}},{"cell_type":"code","source":"#lets first check how many subsets and thus subset summaries the selected competition 1 has\nif N != \"ALL\":\n    len(combined[0])\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:24.159019Z","iopub.execute_input":"2024-04-19T22:42:24.159255Z","iopub.status.idle":"2024-04-19T22:42:24.166989Z","shell.execute_reply.started":"2024-04-19T22:42:24.159235Z","shell.execute_reply":"2024-04-19T22:42:24.166082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# the first subset summary for the first competition in the list\nif N != \"ALL\":\n    print(combined[0][0])\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:24.168143Z","iopub.execute_input":"2024-04-19T22:42:24.169019Z","iopub.status.idle":"2024-04-19T22:42:24.177833Z","shell.execute_reply.started":"2024-04-19T22:42:24.168993Z","shell.execute_reply":"2024-04-19T22:42:24.176973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# the second subset summary for the first competition in the list\n#combined[0][1]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:24.178867Z","iopub.execute_input":"2024-04-19T22:42:24.179164Z","iopub.status.idle":"2024-04-19T22:42:24.187978Z","shell.execute_reply.started":"2024-04-19T22:42:24.179141Z","shell.execute_reply":"2024-04-19T22:42:24.187200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# the third and final subset summary for the first competition in the list\n#combined[0][2]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:24.189008Z","iopub.execute_input":"2024-04-19T22:42:24.189335Z","iopub.status.idle":"2024-04-19T22:42:24.198226Z","shell.execute_reply.started":"2024-04-19T22:42:24.189302Z","shell.execute_reply":"2024-04-19T22:42:24.197356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Rerun the Final Summary for Competition 1 with Debug Info\n\nThe three subset summaries shown above seem fine. So lets see what the input is actually for the final summarization step, and what the output is. Just to see there is no obvious issue with the input and how it is run.\n\nI put the debug/print_prompt flag in my helper functions above to print the prompt and token counts just for this purpose.","metadata":{}},{"cell_type":"code","source":"if N!=\"ALL\":\n    summary, summary_md = summarize_subsummaries(combined[0], True, \"simple\")","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:24.199112Z","iopub.execute_input":"2024-04-19T22:42:24.199341Z","iopub.status.idle":"2024-04-19T22:42:27.797965Z","shell.execute_reply.started":"2024-04-19T22:42:24.199321Z","shell.execute_reply":"2024-04-19T22:42:27.796975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Look at the Comp 1 Prompt and Output\n\nHere, Gemma 1.1 seems to say it has summarized the prompt key points, yet provides none. Gemma 1.0 was just as bad at times, other times it provided some key points in different runs. So it is quite similar for both models in this regard, sometimes just dropping it completely. ","metadata":{}},{"cell_type":"markdown","source":"## Rerun the Final Summary for Competition 2 with Debug Info\n\nNow lets check similarly, the second competition in the selected set of 5. It has 2 subsets, which the debug info will also print with the prompt below.\n","metadata":{"execution":{"iopub.execute_input":"2024-04-13T07:58:45.719314Z","iopub.status.busy":"2024-04-13T07:58:45.718647Z","iopub.status.idle":"2024-04-13T07:58:45.726182Z","shell.execute_reply":"2024-04-13T07:58:45.725032Z","shell.execute_reply.started":"2024-04-13T07:58:45.719283Z"}}},{"cell_type":"code","source":"# number of subset summaries for this competition\nif N != \"ALL\":\n    print(len(combined[1]))\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:27.811827Z","iopub.execute_input":"2024-04-19T22:42:27.812378Z","iopub.status.idle":"2024-04-19T22:42:27.817202Z","shell.execute_reply.started":"2024-04-19T22:42:27.812349Z","shell.execute_reply":"2024-04-19T22:42:27.816230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if N != \"ALL\":\n    summary, summary_md = summarize_subsummaries(combined[1], True)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:27.818643Z","iopub.execute_input":"2024-04-19T22:42:27.819064Z","iopub.status.idle":"2024-04-19T22:42:42.106277Z","shell.execute_reply.started":"2024-04-19T22:42:27.819026Z","shell.execute_reply":"2024-04-19T22:42:42.105285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Look at the Comp 2 Prompt and Output\n\nThis is quite similar to what Gemma 1.0 gave, but I find the key points to be better described and perhaps represent better the kind of higher-level concepts that I would like to see from such large set of writeups. This applies also the the way Gemma 1.1 has summarized the input writeups. So I would say overall it is better in 1.1.","metadata":{}},{"cell_type":"markdown","source":"## Changing the Prompt\n\nIn the above debug prints, we see that my prompting might confuse Gemma a bit in asking to summarize multiple writeup summaries, but just concatenating them all together. So lets try giving it a clue where the next writeup summary starts.","metadata":{}},{"cell_type":"code","source":"if N != \"ALL\":\n    summary, summary_md = summarize_subsummaries(combined[0], True, \"subtitle writeup\")","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:42.107590Z","iopub.execute_input":"2024-04-19T22:42:42.107908Z","iopub.status.idle":"2024-04-19T22:42:46.070507Z","shell.execute_reply.started":"2024-04-19T22:42:42.107883Z","shell.execute_reply":"2024-04-19T22:42:46.069464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Did it Help?\n\nSometimes it helped in different runs, not not so much. In this case, it has simply stated that there are no details about the competition or the dataset.. Perhaps it is trying to hint at wanting the competition description. But I believe the input summaries should provide a good set of input already. Cannot say much from this output about 1.1 vs 1.0. This is just bad output in general.","metadata":{}},{"cell_type":"code","source":"if N != \"ALL\":\n    summary_md","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:46.071772Z","iopub.execute_input":"2024-04-19T22:42:46.072079Z","iopub.status.idle":"2024-04-19T22:42:46.077048Z","shell.execute_reply.started":"2024-04-19T22:42:46.072052Z","shell.execute_reply":"2024-04-19T22:42:46.075984Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### And Again a Slightly Different prompt\n\nInstead of subtitling the subset summaries as \"writeup X\", lets try a more correct \"writeups summary X\" and see if it has any effect:","metadata":{}},{"cell_type":"code","source":"if N != \"ALL\":\n    summary, summary_md = summarize_subsummaries(combined[0], True, \"writeup summaries title\")","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:46.078258Z","iopub.execute_input":"2024-04-19T22:42:46.078542Z","iopub.status.idle":"2024-04-19T22:42:50.177609Z","shell.execute_reply.started":"2024-04-19T22:42:46.078517Z","shell.execute_reply":"2024-04-19T22:42:50.176583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Any Difference?\n\nAgain, even more short answers. It seems to refer to some guidelines that do not exist. I only instructed it in the beginning to give me the list of key points. In this regard, Gemma 1.1 seems to do worse than 1.0, since 1.0 most of the time at least tried to collect some points. Both above runs now just miss it all in both formats.","metadata":{}},{"cell_type":"code","source":"#if N != \"ALL\":\nsummary_md","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.179076Z","iopub.execute_input":"2024-04-19T22:42:50.179530Z","iopub.status.idle":"2024-04-19T22:42:50.184322Z","shell.execute_reply.started":"2024-04-19T22:42:50.179493Z","shell.execute_reply":"2024-04-19T22:42:50.183098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Subset Summarization Conclusions So Far\n\nYou need to subtitle your subsets in the input context. What more can I say.. :)","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Save the Data to Load Later","metadata":{"execution":{"iopub.status.busy":"2024-04-17T21:35:41.073069Z","iopub.execute_input":"2024-04-17T21:35:41.073771Z","iopub.status.idle":"2024-04-17T21:35:41.085003Z","shell.execute_reply.started":"2024-04-17T21:35:41.073732Z","shell.execute_reply":"2024-04-17T21:35:41.084022Z"}}},{"cell_type":"code","source":"import pickle\n\n# List of all your lists and their corresponding file names\ndata = {\n    \"writeup_summaries\": writeup_summaries,\n    \"writeup_summaries_md\": writeup_summaries_md,\n    \"overall_summaries\": overall_summaries,\n    \"processed_titles\": processed_titles,\n    \"processed_titles_list\": processed_titles_list,\n    \"skipped_titles\": skipped_titles,\n#    \"skipped_titles_list\": skipped_titles_list,\n    \"overall_summary_prompts\": overall_summary_prompts,\n    \"prompts\": prompts,\n    \"writeups\": writeups,\n    \"prompt_lengths\": prompt_lengths,\n    \"writeup_lengths\": writeup_lengths,\n    \"summary_lengths\": summary_lengths,\n    \"resummarized_simple\": resummarized_simple,\n    \"resummarized_md_simple\": resummarized_md_simple,\n    \"resummarized_writeup\": resummarized_writeup,\n    \"resummarized_md_writeup\": resummarized_md_writeup,\n    \"resummarized_writeup_summary\": resummarized_writeup_summary,\n    \"resummarized_md_writeup_summary\": resummarized_md_writeup_summary,\n}","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.185770Z","iopub.execute_input":"2024-04-19T22:42:50.186502Z","iopub.status.idle":"2024-04-19T22:42:50.196194Z","shell.execute_reply.started":"2024-04-19T22:42:50.186437Z","shell.execute_reply":"2024-04-19T22:42:50.195349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"split_size","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.197209Z","iopub.execute_input":"2024-04-19T22:42:50.197483Z","iopub.status.idle":"2024-04-19T22:42:50.211887Z","shell.execute_reply.started":"2024-04-19T22:42:50.197461Z","shell.execute_reply":"2024-04-19T22:42:50.211059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pickle\n\n# Assuming 'data' is your dictionary\nwith open(f'data_{split_size}.pickle', 'wb') as f:\n    pickle.dump(data, f)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.212975Z","iopub.execute_input":"2024-04-19T22:42:50.213246Z","iopub.status.idle":"2024-04-19T22:42:50.223459Z","shell.execute_reply.started":"2024-04-19T22:42:50.213202Z","shell.execute_reply":"2024-04-19T22:42:50.222639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load the data back in\nwith open(f'data_{split_size}.pickle', 'rb') as f:\n    data = pickle.load(f)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.224510Z","iopub.execute_input":"2024-04-19T22:42:50.224820Z","iopub.status.idle":"2024-04-19T22:42:50.233212Z","shell.execute_reply.started":"2024-04-19T22:42:50.224779Z","shell.execute_reply":"2024-04-19T22:42:50.232375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Import pandas\nimport pandas as pd\n\n# Calculate the number of writeups per competition\nwriteup_counts = df_writeups.groupby(\"Title of Competition\").size()\n\n# Create a DataFrame for writeup counts\ndf_writeup_counts = pd.DataFrame({\n    'Competition': writeup_counts.index,\n    'Number of Writeups': writeup_counts.values\n})\n\n# Rename the 'comp_name' column in df_comp_meta to 'Competition' for the merge\ndf_comp_meta_renamed = df_comp_meta.rename(columns={'comp_name': 'Competition'})\n\n# Merge the two DataFrames on the 'Competition' column\ndf_summary = pd.merge(df_writeup_counts, df_comp_meta_renamed, on='Competition', how='outer')\n\n# Replace NaN values in 'Number of Writeups' with 0\ndf_summary['Number of Writeups'] = df_summary['Number of Writeups'].fillna(0)\n\n# Create a new column 'Metadata Available' that is True if 'desc' is not NaN\ndf_summary['Metadata Available'] = ~df_summary['desc'].isna()\n\n# Reset the index\ndf_summary.reset_index(drop=True, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.234409Z","iopub.execute_input":"2024-04-19T22:42:50.234806Z","iopub.status.idle":"2024-04-19T22:42:50.252459Z","shell.execute_reply.started":"2024-04-19T22:42:50.234771Z","shell.execute_reply":"2024-04-19T22:42:50.251716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# find all column names in df_summary that are type float\nfloat_cols = df_summary.select_dtypes(include=['float']).columns\nfloat_cols","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.253869Z","iopub.execute_input":"2024-04-19T22:42:50.254132Z","iopub.status.idle":"2024-04-19T22:42:50.260597Z","shell.execute_reply.started":"2024-04-19T22:42:50.254109Z","shell.execute_reply":"2024-04-19T22:42:50.259634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fill NaN values with -1 for columns in float_cols\ndf_summary[float_cols] = df_summary[float_cols].fillna(-1)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.261852Z","iopub.execute_input":"2024-04-19T22:42:50.262135Z","iopub.status.idle":"2024-04-19T22:42:50.273810Z","shell.execute_reply.started":"2024-04-19T22:42:50.262111Z","shell.execute_reply":"2024-04-19T22:42:50.272940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# convert all columns in float_cols to integer type\ndf_summary[float_cols] = df_summary[float_cols].astype(int)\ndf_summary.head()","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.274996Z","iopub.execute_input":"2024-04-19T22:42:50.275314Z","iopub.status.idle":"2024-04-19T22:42:50.296713Z","shell.execute_reply.started":"2024-04-19T22:42:50.275289Z","shell.execute_reply":"2024-04-19T22:42:50.295706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"too_long_indices = []\noverall_summaries_raw = []\n\nfor idx, sum in enumerate(overall_summaries):\n    # check if sum is string\n    if isinstance(sum, str):\n        overall_summaries_raw.append(sum)\n        too_long_indices.append(idx)\n    else:\n        overall_summaries_raw.append(sum.data)\n\ntoo_long_indices","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.297972Z","iopub.execute_input":"2024-04-19T22:42:50.298506Z","iopub.status.idle":"2024-04-19T22:42:50.305983Z","shell.execute_reply.started":"2024-04-19T22:42:50.298474Z","shell.execute_reply":"2024-04-19T22:42:50.305046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# combine compatition names and resummarized summaries into a dataframe\nif N != \"ALL\":\n    df_resummarized = pd.DataFrame({\n        'Competition': processed_titles_list,\n    #    'Resummarized Summary': overall_summaries_raw\n        'Resummarized Summary Simple': data[\"resummarized_simple\"],\n        'Resummarized Summary Writeup': data[\"resummarized_writeup\"],\n        'Resummarized Summary Writeup Summary': data[\"resummarized_writeup_summary\"]\n    })\nelse:\n    df_resummarized = pd.DataFrame({\n        'Competition': processed_titles_list,\n        'Overall Summary': data[\"resummarized_simple\"],\n    })\n\ndf_resummarized.head()","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.307286Z","iopub.execute_input":"2024-04-19T22:42:50.307552Z","iopub.status.idle":"2024-04-19T22:42:50.320277Z","shell.execute_reply.started":"2024-04-19T22:42:50.307522Z","shell.execute_reply":"2024-04-19T22:42:50.319368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#merge df_resummarized with df_summary\ndf_final = pd.merge(df_summary, df_resummarized, on='Competition', how='outer')\ndf_final.head(5)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.321503Z","iopub.execute_input":"2024-04-19T22:42:50.321815Z","iopub.status.idle":"2024-04-19T22:42:50.350934Z","shell.execute_reply.started":"2024-04-19T22:42:50.321782Z","shell.execute_reply":"2024-04-19T22:42:50.350107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final.to_csv(f\"combined_df_{split_size}.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.352106Z","iopub.execute_input":"2024-04-19T22:42:50.352375Z","iopub.status.idle":"2024-04-19T22:42:50.407631Z","shell.execute_reply.started":"2024-04-19T22:42:50.352352Z","shell.execute_reply":"2024-04-19T22:42:50.406911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Initialize an empty dictionary\ncombined_dict = {}\n\nidx = 0\n# Iterate over processed_titles, writeup_dataframes, and combined simultaneously\n#for title, dataframe, combine in zip(processed_titles_list, writeup_dataframes_filtered, combined):\nfor title, dataframe, combine in zip(processed_titles_list, writeup_dataframes_filtered, overall_summaries):\n    # For each title, create a new dictionary with keys \"writeup_dataframes\" and \"combined\"\n    combined_dict[title] = {\"writeup_dataframes\": dataframe, \"combined\": combine}\n    if title == \"2019 Data Science Bowl\":\n        print(idx)\n    idx += 1","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.408614Z","iopub.execute_input":"2024-04-19T22:42:50.408890Z","iopub.status.idle":"2024-04-19T22:42:50.414572Z","shell.execute_reply.started":"2024-04-19T22:42:50.408868Z","shell.execute_reply":"2024-04-19T22:42:50.413607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# write the combined_dict to a pickle file\nwith open(f'combined_dict_{split_size}.pickle', 'wb') as f:\n    pickle.dump(combined_dict, f)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.415901Z","iopub.execute_input":"2024-04-19T22:42:50.416595Z","iopub.status.idle":"2024-04-19T22:42:50.424878Z","shell.execute_reply.started":"2024-04-19T22:42:50.416551Z","shell.execute_reply":"2024-04-19T22:42:50.424060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with open(f'combined_dict_{split_size}.pickle', 'rb') as f:\n    combined_dict = pickle.load(f)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.426002Z","iopub.execute_input":"2024-04-19T22:42:50.426330Z","iopub.status.idle":"2024-04-19T22:42:50.434615Z","shell.execute_reply.started":"2024-04-19T22:42:50.426297Z","shell.execute_reply":"2024-04-19T22:42:50.433797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Compare Subsets vs All-in-One\n\nI ran bigger sets of summaries on my local desktop and collected them into a dataset for look here at Kaggle without spending too much GPU time. This is still for Gemma 1.0, since I have not yet re-run it for 1.1. Perhaps later. So lets load that 1.0 set up and see what it looks like:","metadata":{}},{"cell_type":"code","source":"def load_and_filter_df(split_size, more_than=10):\n    # Load the combined dataframe\n    if on_kaggle:\n        df = pd.read_csv(f\"/kaggle/input/gemma-summarized-kaggle-writeups/gemma_summaries_{split_size}.csv\")\n    else:\n        df = pd.read_csv(f\"combined_df_{split_size}.csv\")\n    print(f\"Loaded combined dataframe with shape {df.shape}\")\n    # Filter the dataframe for competitions with more than 10 writeups (or more_than param)\n    df_filtered = df[df[\"Number of Writeups\"] > more_than]\n    print(f\"Filtered dataframe has shape {df_filtered.shape}\")\n    # drop compos that had no metadata and were thus not processed\n    df_filtered = df_filtered[df_filtered[\"comp_Reward\"].notna()]\n    print(f\"Filtered dataframe has shape {df_filtered.shape}\")\n    return df_filtered\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.435751Z","iopub.execute_input":"2024-04-19T22:42:50.436107Z","iopub.status.idle":"2024-04-19T22:42:50.445595Z","shell.execute_reply.started":"2024-04-19T22:42:50.436084Z","shell.execute_reply":"2024-04-19T22:42:50.444706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_5 = load_and_filter_df(5, 0)\ndf_10 = load_and_filter_df(10, 0)\ndf_all = load_and_filter_df(\"all\", 0)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.446705Z","iopub.execute_input":"2024-04-19T22:42:50.447014Z","iopub.status.idle":"2024-04-19T22:42:50.561378Z","shell.execute_reply.started":"2024-04-19T22:42:50.446990Z","shell.execute_reply":"2024-04-19T22:42:50.560349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with open(f'/kaggle/input/gemma-summarized-kaggle-writeups/data_5.pickle', 'rb') as f:\n    data_5 = pickle.load(f)\nwith open(f'/kaggle/input/gemma-summarized-kaggle-writeups/data_10.pickle', 'rb') as f:\n    data_10 = pickle.load(f)\nwith open(f'/kaggle/input/gemma-summarized-kaggle-writeups/data_all.pickle', 'rb') as f:\n    data_all = pickle.load(f)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.562634Z","iopub.execute_input":"2024-04-19T22:42:50.563052Z","iopub.status.idle":"2024-04-19T22:42:50.741451Z","shell.execute_reply.started":"2024-04-19T22:42:50.563014Z","shell.execute_reply":"2024-04-19T22:42:50.740648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_5[\"Number of Writeups\"].unique()","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.742869Z","iopub.execute_input":"2024-04-19T22:42:50.743178Z","iopub.status.idle":"2024-04-19T22:42:50.750011Z","shell.execute_reply.started":"2024-04-19T22:42:50.743152Z","shell.execute_reply":"2024-04-19T22:42:50.749128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Competition with 1 Writeup\n\nThis one has only a single writeup, so the summarization should be simple.","metadata":{}},{"cell_type":"code","source":"df_5[df_5[\"Number of Writeups\"]==1].iloc[0]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.751276Z","iopub.execute_input":"2024-04-19T22:42:50.751606Z","iopub.status.idle":"2024-04-19T22:42:50.765576Z","shell.execute_reply.started":"2024-04-19T22:42:50.751580Z","shell.execute_reply":"2024-04-19T22:42:50.764614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Subsets of 5\n\nSplitting 1 writeup into subsets of 5 results in just a single subset with the single writeup. So not much to look at, but lets look for having a complete process.","metadata":{}},{"cell_type":"code","source":"comp1 = df_5[df_5[\"Number of Writeups\"]==1][\"Competition\"].iloc[0]\nsummy = df_5[df_5[\"Competition\"]==comp1][\"Resummarized Summary Simple\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.766668Z","iopub.execute_input":"2024-04-19T22:42:50.767001Z","iopub.status.idle":"2024-04-19T22:42:50.777490Z","shell.execute_reply.started":"2024-04-19T22:42:50.766977Z","shell.execute_reply":"2024-04-19T22:42:50.776472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Subsets of 10\n\nFor 1 writeup this is the same as subset 5. But lets see again..","metadata":{}},{"cell_type":"code","source":"summy = df_10[df_10[\"Competition\"]==comp1][\"Resummarized Summary Simple\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.778455Z","iopub.execute_input":"2024-04-19T22:42:50.778748Z","iopub.status.idle":"2024-04-19T22:42:50.790082Z","shell.execute_reply.started":"2024-04-19T22:42:50.778724Z","shell.execute_reply":"2024-04-19T22:42:50.789224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## All in one, no subsets\n\nAgain, about the same but missing the extra step of re-summarizing the subset summaries. Because it already summarizes all writeup summaries in one step. This is why it has the same points practically but the structuring and wording differs.","metadata":{}},{"cell_type":"code","source":"summy = df_all[df_all[\"Competition\"]==comp1][\"Overall Summary\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.791260Z","iopub.execute_input":"2024-04-19T22:42:50.791583Z","iopub.status.idle":"2024-04-19T22:42:50.802178Z","shell.execute_reply.started":"2024-04-19T22:42:50.791552Z","shell.execute_reply":"2024-04-19T22:42:50.801284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Odd...\n\nThe above \"all\" summary talks about \"both\" summaries, when there is only a single writeup. So a quick look at where does this come from:","metadata":{}},{"cell_type":"code","source":"comp1_idx = data_all[\"processed_titles_list\"].index(comp1)\nMarkdown(data_all[\"overall_summary_prompts\"][comp1_idx])","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.803291Z","iopub.execute_input":"2024-04-19T22:42:50.803569Z","iopub.status.idle":"2024-04-19T22:42:50.813209Z","shell.execute_reply.started":"2024-04-19T22:42:50.803545Z","shell.execute_reply":"2024-04-19T22:42:50.812347Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well, it seems to get confused by the separator ====== I used. Guess this model at this size has some training to take. Or need to be aware of these limitations.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5 Writeups","metadata":{}},{"cell_type":"code","source":"df_5[df_5[\"Number of Writeups\"]==5].iloc[0]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.814320Z","iopub.execute_input":"2024-04-19T22:42:50.814929Z","iopub.status.idle":"2024-04-19T22:42:50.826571Z","shell.execute_reply.started":"2024-04-19T22:42:50.814903Z","shell.execute_reply":"2024-04-19T22:42:50.825723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Subsets of 5\n\nIt is a single subset, and seems to summarize quite fine.","metadata":{}},{"cell_type":"code","source":"comp5 = df_5[df_5[\"Number of Writeups\"]==5][\"Competition\"].iloc[0]\nsummy = df_5[df_5[\"Competition\"]==comp5][\"Resummarized Summary Simple\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.827753Z","iopub.execute_input":"2024-04-19T22:42:50.828077Z","iopub.status.idle":"2024-04-19T22:42:50.838970Z","shell.execute_reply.started":"2024-04-19T22:42:50.828046Z","shell.execute_reply":"2024-04-19T22:42:50.837987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Subsets of 10\n\nPractically the same as above subsets of 5. No wonder since the 5 writeups fit into both as a single subset.","metadata":{}},{"cell_type":"code","source":"summy = df_10[df_10[\"Competition\"]==comp5][\"Resummarized Summary Simple\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.839897Z","iopub.execute_input":"2024-04-19T22:42:50.840146Z","iopub.status.idle":"2024-04-19T22:42:50.855717Z","shell.execute_reply.started":"2024-04-19T22:42:50.840123Z","shell.execute_reply":"2024-04-19T22:42:50.854726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## No Subsets, All in One\n\nThe results are about the same but it lacks on step in re-summarization, so the result is a bit more \"raw\" and focused (the \"additional points\" start is not there):","metadata":{}},{"cell_type":"code","source":"summy = df_all[df_all[\"Competition\"]==comp5][\"Overall Summary\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.856874Z","iopub.execute_input":"2024-04-19T22:42:50.857654Z","iopub.status.idle":"2024-04-19T22:42:50.866655Z","shell.execute_reply.started":"2024-04-19T22:42:50.857622Z","shell.execute_reply":"2024-04-19T22:42:50.865728Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 10 Writeups\n\nThis should be 2 subsets for size of 5, and all in one for size 10 and \"ALL\".","metadata":{}},{"cell_type":"code","source":"df_5[df_5[\"Number of Writeups\"]==10].iloc[0]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.867913Z","iopub.execute_input":"2024-04-19T22:42:50.868362Z","iopub.status.idle":"2024-04-19T22:42:50.879437Z","shell.execute_reply.started":"2024-04-19T22:42:50.868328Z","shell.execute_reply":"2024-04-19T22:42:50.878462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Subsets of 5\n\nThis loses the key point details already:","metadata":{}},{"cell_type":"code","source":"comp10 = df_5[df_5[\"Number of Writeups\"]==10][\"Competition\"].iloc[0]\nsummy = df_5[df_5[\"Competition\"]==comp10][\"Resummarized Summary Simple\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.880565Z","iopub.execute_input":"2024-04-19T22:42:50.880921Z","iopub.status.idle":"2024-04-19T22:42:50.891001Z","shell.execute_reply.started":"2024-04-19T22:42:50.880897Z","shell.execute_reply":"2024-04-19T22:42:50.890096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Subsets of 10\n\nThis only still has a single subset to summarize, and the result does seem better.","metadata":{}},{"cell_type":"code","source":"summy = df_10[df_10[\"Competition\"]==comp10][\"Resummarized Summary Simple\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.892242Z","iopub.execute_input":"2024-04-19T22:42:50.893190Z","iopub.status.idle":"2024-04-19T22:42:50.903304Z","shell.execute_reply.started":"2024-04-19T22:42:50.893164Z","shell.execute_reply":"2024-04-19T22:42:50.902439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## No Subsets, All in One\n\nAbout the same as above subsets of 10, just minor variation in wording of the key points: ","metadata":{}},{"cell_type":"code","source":"summy = df_all[df_all[\"Competition\"]==comp10][\"Overall Summary\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.904564Z","iopub.execute_input":"2024-04-19T22:42:50.905112Z","iopub.status.idle":"2024-04-19T22:42:50.913916Z","shell.execute_reply.started":"2024-04-19T22:42:50.905085Z","shell.execute_reply":"2024-04-19T22:42:50.913022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 15 Writeups\n\nThis would be 3 subsets for subset size 5, 2 subsets for size 10, and a single set for the \"all\" version:","metadata":{}},{"cell_type":"code","source":"df_5[df_5[\"Number of Writeups\"]==15].iloc[0]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.915091Z","iopub.execute_input":"2024-04-19T22:42:50.915383Z","iopub.status.idle":"2024-04-19T22:42:50.926059Z","shell.execute_reply.started":"2024-04-19T22:42:50.915360Z","shell.execute_reply":"2024-04-19T22:42:50.925085Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Subsets of 5\n\nThis one starts overly pushing it all together, and seems to have lost a lot of key points and structure already:","metadata":{}},{"cell_type":"code","source":"comp15 = df_5[df_5[\"Number of Writeups\"]==15][\"Competition\"].iloc[0]\nsummy = df_5[df_5[\"Competition\"]==comp15][\"Resummarized Summary Simple\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.927332Z","iopub.execute_input":"2024-04-19T22:42:50.927634Z","iopub.status.idle":"2024-04-19T22:42:50.938053Z","shell.execute_reply.started":"2024-04-19T22:42:50.927609Z","shell.execute_reply":"2024-04-19T22:42:50.937275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Subsets of 10\n\nHere we are still good:","metadata":{}},{"cell_type":"code","source":"summy = df_10[df_10[\"Competition\"]==comp15][\"Resummarized Summary Simple\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.939384Z","iopub.execute_input":"2024-04-19T22:42:50.939728Z","iopub.status.idle":"2024-04-19T22:42:50.950128Z","shell.execute_reply.started":"2024-04-19T22:42:50.939697Z","shell.execute_reply":"2024-04-19T22:42:50.949087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## No Subsets, All in One\n\nAbout the same as above subsets of 10, but perhas a bit of differentiation between what seems to be slightly more generic topics, and a list of key points: ","metadata":{}},{"cell_type":"code","source":"summy = df_all[df_all[\"Competition\"]==comp15][\"Overall Summary\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.951415Z","iopub.execute_input":"2024-04-19T22:42:50.951776Z","iopub.status.idle":"2024-04-19T22:42:50.960896Z","shell.execute_reply.started":"2024-04-19T22:42:50.951753Z","shell.execute_reply":"2024-04-19T22:42:50.960077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 36 Writeups\n\nNow lets go for one of the biggest sets of writeups we have. First a look at the unique writeup counts:","metadata":{}},{"cell_type":"code","source":"counts = df_5[\"Number of Writeups\"].unique()\nnp.sort(counts)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.962057Z","iopub.execute_input":"2024-04-19T22:42:50.962599Z","iopub.status.idle":"2024-04-19T22:42:50.971570Z","shell.execute_reply.started":"2024-04-19T22:42:50.962567Z","shell.execute_reply":"2024-04-19T22:42:50.970771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Highest counts here are 36, 37 and 41. Unfortunately 41 and 37 were too much fit the writeup summaries into Gemma 8k context, so I will go with the 36. This is both non-divisible by 5, and also as close to the top height as possible.\n\nThe too long lists:","metadata":{}},{"cell_type":"code","source":"comp41 = df_all[df_all[\"Number of Writeups\"]==41][\"Competition\"].iloc[0]\nsummy = df_all[df_all[\"Competition\"]==comp41][\"Overall Summary\"].iloc[0]\nMarkdown(summy)\n","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.972920Z","iopub.execute_input":"2024-04-19T22:42:50.973422Z","iopub.status.idle":"2024-04-19T22:42:50.985372Z","shell.execute_reply.started":"2024-04-19T22:42:50.973375Z","shell.execute_reply":"2024-04-19T22:42:50.984446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"comp37 = df_all[df_all[\"Number of Writeups\"]==37][\"Competition\"].iloc[0]\nsummy = df_all[df_all[\"Competition\"]==comp37][\"Overall Summary\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.986672Z","iopub.execute_input":"2024-04-19T22:42:50.987420Z","iopub.status.idle":"2024-04-19T22:42:50.996725Z","shell.execute_reply.started":"2024-04-19T22:42:50.987385Z","shell.execute_reply":"2024-04-19T22:42:50.995836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The 36 writeup version did fit into the 8k context window, and it also has a number that is not divisible by 5, so a bit more interesting than exact subsets of 5. Lets see:","metadata":{}},{"cell_type":"code","source":"df_5[df_5[\"Number of Writeups\"]==36].iloc[0]","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:50.997784Z","iopub.execute_input":"2024-04-19T22:42:50.998462Z","iopub.status.idle":"2024-04-19T22:42:51.009333Z","shell.execute_reply.started":"2024-04-19T22:42:50.998436Z","shell.execute_reply":"2024-04-19T22:42:51.008492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Subsets of 5\n\nThis splitting produces the generic pointless summary that Gemma often seems to produce, losing all of the real information:","metadata":{}},{"cell_type":"code","source":"comp36 = df_5[df_5[\"Number of Writeups\"]==36][\"Competition\"].iloc[0]\nsummy = df_5[df_5[\"Competition\"]==comp36][\"Resummarized Summary Simple\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:51.010423Z","iopub.execute_input":"2024-04-19T22:42:51.011264Z","iopub.status.idle":"2024-04-19T22:42:51.022248Z","shell.execute_reply.started":"2024-04-19T22:42:51.011236Z","shell.execute_reply":"2024-04-19T22:42:51.021342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Subsets of 10\n\nThis one is not quite the shortest one-liner above, but a few points. Unfortunately considering it consists of 36 writeups and summarizing their key points into one final summary, it is very bad. Loses all details, and lists a few generic data science points:","metadata":{}},{"cell_type":"code","source":"comp36 = df_10[df_10[\"Number of Writeups\"]==36][\"Competition\"].iloc[0]\nsummy = df_10[df_10[\"Competition\"]==comp36][\"Resummarized Summary Simple\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:51.023664Z","iopub.execute_input":"2024-04-19T22:42:51.024264Z","iopub.status.idle":"2024-04-19T22:42:51.035125Z","shell.execute_reply.started":"2024-04-19T22:42:51.024228Z","shell.execute_reply":"2024-04-19T22:42:51.034298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## No Subsets, All in One\n\nThis one has the key points collected, as there is no subsplitting before:","metadata":{}},{"cell_type":"code","source":"summy = df_all[df_all[\"Competition\"]==comp36][\"Overall Summary\"].iloc[0]\nMarkdown(summy)","metadata":{"execution":{"iopub.status.busy":"2024-04-19T22:42:51.036147Z","iopub.execute_input":"2024-04-19T22:42:51.036404Z","iopub.status.idle":"2024-04-19T22:42:51.046570Z","shell.execute_reply.started":"2024-04-19T22:42:51.036382Z","shell.execute_reply":"2024-04-19T22:42:51.045548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusions About Subsets\n\nWell, Gemma seems to do quite fine up to a point. But when I start to summarize larger number of subsets, it seems to give a very bland and generic, overly short answer. Without much of real content. So I would say it would be best to trial some more and perhaps think of ways to show the initial summaries for the user as input for further insights. Or other ways to support the user in their task, in this case analyzing Kaggle writeups. Such as in my second [RAG notebook](https://www.kaggle.com/code/donkeys/qa-solutions-from-past-competitions), where I tried to use these summaries as input to the user to drive their RAG-based question formulation and help figure out the details starting from this high-level summary picture.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}