{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":61542,"databundleVersionId":6888007,"sourceType":"competition"},{"sourceId":6868189,"sourceType":"datasetVersion","datasetId":3937441},{"sourceId":7072072,"sourceType":"datasetVersion","datasetId":4010765}],"dockerImageVersionId":30558,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"![](\"https://i.imgur.com/1lX7uZl.png\")
\n\n Detect AI Generated text
\n- Classify using DistilBERT -
\n\n> ๐ **Competition Scope**: the goal of the competition is to determine whether or not essay was generated by an LLM.\n\n### โ References\n* Additional generated train data from Radek's dataset: [๐ค LLM Generated Essays for the Detect AI Comp!](https://www.kaggle.com/datasets/radek1/llm-generated-essays/)\n* [Big Vocab Words](https://www.berlitz.com/blog/big-words-to-sound-smart)\n* [Transitional Words](https://www.touro.edu/departments/writing-center/tutorials/transitional-words/#:~:text=and%2C%20again%2C%20and%20then%2C,first%20(second%2C%20etc.))\n\n### โ Libraries","metadata":{}},{"cell_type":"code","source":"# libraries\nimport os\nimport gc\nimport wandb\nimport random\nimport json\nimport string\nimport re\nfrom tqdm import tqdm\nfrom time import time\nimport warnings\nimport pandas as pd\nimport numpy as np\nfrom collections import Counter\nimport string\n\n# visuals\nimport seaborn as sns\nimport matplotlib.patches as patches\nimport matplotlib.pyplot as plt\nplt.rcParams.update({'font.size': 18})\n\n# env check\nwarnings.filterwarnings('ignore')\nos.environ[\"WANDB_SILENT\"] = \"true\"\n\nCOMP_ID = '2023_detect_gen_text'\nCONFIG = {'competition': COMP_ID, '_wandb_kernel': 'aot', \"source_type\": \"artifact\"}\n\n# color\nclass clr:\n S = '\\033[1m' + '\\033[90m'\n E = '\\033[0m'\nmy_colors = [\"#A63D40\", \"#E9B872\", \"#90A959\", \"#6494AA\", \"#1C6E8C\"]\n\nprint(clr.S+\"Notebook Color Schemes:\"+clr.E)\nsns.palplot(sns.color_palette(my_colors))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:33:07.851399Z","iopub.execute_input":"2023-11-29T16:33:07.852255Z","iopub.status.idle":"2023-11-29T16:33:09.781777Z","shell.execute_reply.started":"2023-11-29T16:33:07.852219Z","shell.execute_reply":"2023-11-29T16:33:09.780327Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ๐ W&B Fork & Run\n\nIn order to run this notebook you will need to input your own **secret API key** within the `! wandb login $secret_value_0` line. \n\n๐**How do you get your own API key?**\n\nSuper simple! Go to **https://wandb.ai/site** -> Login -> Click on your profile in the top right corner -> Settings -> Scroll down to API keys -> copy your very own key (for more info check [this amazing notebook for ML Experiment Tracking on Kaggle](https://www.kaggle.com/ayuraj/experiment-tracking-with-weights-and-biases)).\n\n![](\"https://i.imgur.com/fFccmoS.png\")
","metadata":{}},{"cell_type":"code","source":"# ๐ secrets\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nWANDB_KEY = user_secrets.get_secret(\"wandb\")\n\n! wandb login $WANDB_KEY","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:33:09.784869Z","iopub.execute_input":"2023-11-29T16:33:09.785940Z","iopub.status.idle":"2023-11-29T16:33:14.262529Z","shell.execute_reply.started":"2023-11-29T16:33:09.785884Z","shell.execute_reply":"2023-11-29T16:33:14.261427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### โ Helper Functions Below","metadata":{}},{"cell_type":"code","source":"def calculate_repetitiveness(text):\n # clean text\n translator = str.maketrans(\"\", \"\", string.punctuation)\n text = text.translate(translator).lower()\n\n # tokenize\n words = text.split()\n\n # count the occurrences of each word\n word_counts = Counter(words)\n\n # compute repetitiveness\n total_words = len(words)\n unique_words = len(word_counts)\n repetitiveness = 1 - (unique_words / total_words) if total_words > 0 else 0\n\n return repetitiveness\n\n\ndef calculate_vocabulary_score(text, vocab):\n # clean text\n translator = str.maketrans(\"\", \"\", string.punctuation)\n text = text.translate(translator).lower()\n\n # tokenize\n words = text.split()\n\n # count the occurrences of each word\n word_counts = Counter(words)\n\n # compute score based on vocabulary\n total_words = len(words)\n vocab_occurrences = sum(word_counts[word] for word in vocab)\n score = vocab_occurrences / total_words if total_words > 0 else 0\n\n return score\n\n\n# === plots ===\ndef show_values_on_bars(axs, h_v=\"v\", space=0.4):\n '''Plots the value at the end of the a seaborn barplot.\n axs: the ax of the plot\n h_v: weather or not the barplot is vertical/ horizontal'''\n \n def _show_on_single_plot(ax):\n if h_v == \"v\":\n for p in ax.patches:\n _x = p.get_x() + p.get_width() / 2\n _y = p.get_y() + p.get_height()\n value = int(p.get_height())\n ax.text(_x, _y, format(value, ','), ha=\"center\") \n elif h_v == \"h\":\n for p in ax.patches:\n _x = p.get_x() + p.get_width() + float(space)\n _y = p.get_y() + p.get_height()\n value = int(p.get_width())\n ax.text(_x, _y, format(value, ','), ha=\"left\")\n\n if isinstance(axs, np.ndarray):\n for idx, ax in np.ndenumerate(axs):\n _show_on_single_plot(ax)\n else:\n _show_on_single_plot(axs)\n \n \n# === ๐ w&b ===\ndef save_dataset_artifact(run_name, artifact_name, path, data_type=\"dataset\"):\n '''Saves dataset to W&B Artifactory.\n run_name: name of the experiment\n artifact_name: under what name should the dataset be stored\n path: path to the dataset'''\n \n run = wandb.init(project=COMP_ID, \n name=run_name, \n config=CONFIG)\n artifact = wandb.Artifact(name=artifact_name, \n type=data_type)\n artifact.add_file(path)\n\n wandb.log_artifact(artifact)\n wandb.finish()\n print(f\"๐Artifact {artifact_name} has been saved successfully.\")\n \n \ndef create_wandb_plot(x_data=None, y_data=None, x_name=None, y_name=None, title=None, log=None, plot=\"line\"):\n '''Create and save lineplot/barplot in W&B Environment.\n x_data & y_data: Pandas Series containing x & y data\n x_name & y_name: strings containing axis names\n title: title of the graph\n log: string containing name of log'''\n \n data = [[label, val] for (label, val) in zip(x_data, y_data)]\n table = wandb.Table(data=data, columns = [x_name, y_name])\n \n if plot == \"line\":\n wandb.log({log : wandb.plot.line(table, x_name, y_name, title=title)})\n elif plot == \"bar\":\n wandb.log({log : wandb.plot.bar(table, x_name, y_name, title=title)})\n elif plot == \"scatter\":\n wandb.log({log : wandb.plot.scatter(table, x_name, y_name, title=title)})\n \n \ndef create_wandb_hist(x_data=None, x_name=None, title=None, log=None):\n '''Create and save histogram in W&B Environment.\n x_data: Pandas Series containing x values\n x_name: strings containing axis name\n title: title of the graph\n log: string containing name of log'''\n \n data = [[x] for x in x_data]\n table = wandb.Table(data=data, columns=[x_name])\n wandb.log({log : wandb.plot.histogram(table, x_name, title=title)})","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-29T16:33:14.264029Z","iopub.execute_input":"2023-11-29T16:33:14.264344Z","iopub.status.idle":"2023-11-29T16:33:14.285310Z","shell.execute_reply.started":"2023-11-29T16:33:14.264314Z","shell.execute_reply":"2023-11-29T16:33:14.284415Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ๐\nrun = wandb.init(project=COMP_ID, name='cover', config=CONFIG)\ncover = plt.imread(\"/kaggle/input/english-vocabulary-miscellaneous/cover.png\")\nwandb.log({\"example\": wandb.Image(cover)})\nwandb.finish()","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-29T16:33:14.286464Z","iopub.execute_input":"2023-11-29T16:33:14.286752Z","iopub.status.idle":"2023-11-29T16:34:20.891491Z","shell.execute_reply.started":"2023-11-29T16:33:14.286728Z","shell.execute_reply":"2023-11-29T16:34:20.890699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Data Explore\n\n**๐ TL;DR:**\n* As the comp data has (almost) only texts made by pupils and only 3 generated I am combining the data from Radek\n* We end up with 703 generated texts and 1375 real texts\n* The texts are evenly distributed between the 2 prompts within this comp.","metadata":{}},{"cell_type":"code","source":"# retrieve comp data\ntrain_comp = pd.read_csv(\"/kaggle/input/llm-detect-ai-generated-text/train_essays.csv\")\ntrain_prompts = pd.read_csv(\"/kaggle/input/llm-detect-ai-generated-text/train_prompts.csv\")\n\n# retrieve additional data\ntrain_gen1 = pd.read_csv(\"/kaggle/input/llm-generated-essays/ai_generated_train_essays.csv\")\ntrain_gen2 = pd.read_csv(\"/kaggle/input/llm-generated-essays/ai_generated_train_essays_gpt-4.csv\")\ntrain_gen = pd.concat([train_gen1, train_gen2], axis=0)\\\n .reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:20.894633Z","iopub.execute_input":"2023-11-29T16:34:20.895015Z","iopub.status.idle":"2023-11-29T16:34:21.061337Z","shell.execute_reply.started":"2023-11-29T16:34:20.894980Z","shell.execute_reply":"2023-11-29T16:34:21.060298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_comp.head(3)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:21.062607Z","iopub.execute_input":"2023-11-29T16:34:21.062920Z","iopub.status.idle":"2023-11-29T16:34:21.079716Z","shell.execute_reply.started":"2023-11-29T16:34:21.062895Z","shell.execute_reply":"2023-11-29T16:34:21.078876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_prompts.head()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:21.080966Z","iopub.execute_input":"2023-11-29T16:34:21.081242Z","iopub.status.idle":"2023-11-29T16:34:21.090951Z","shell.execute_reply.started":"2023-11-29T16:34:21.081219Z","shell.execute_reply":"2023-11-29T16:34:21.090067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_gen.head(3)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:21.092204Z","iopub.execute_input":"2023-11-29T16:34:21.092580Z","iopub.status.idle":"2023-11-29T16:34:21.105625Z","shell.execute_reply.started":"2023-11-29T16:34:21.092549Z","shell.execute_reply":"2023-11-29T16:34:21.104734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# data shape\nprint(clr.S+\"-> Number of total unique IDs in comp train data: \"+clr.E,\n train_comp.id.nunique())\n\n# plot\nprint(clr.S+\"-> Our dataset has only 3 generated values and the rest 1375 texts are written by pupils.\"+clr.E,\n \"\\n\")\nplt.figure(figsize=(20, 10))\nfigure = sns.countplot(data=train_comp,\n x=\"generated\",palette=my_colors)\nshow_values_on_bars(figure, h_v=\"v\", space=0.4)\nplt.title('[Comp Data] Not Generated vs Generated distribution',\n weight=\"bold\", size=20)\n\nplt.xlabel(\"Generated\", size = 18, weight=\"bold\")\nplt.ylabel(\"Count\", size = 18, weight=\"bold\")\n \nsns.despine(right=True, top=True, left=True)\nplt.show();","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-29T16:34:21.107172Z","iopub.execute_input":"2023-11-29T16:34:21.107499Z","iopub.status.idle":"2023-11-29T16:34:21.439977Z","shell.execute_reply.started":"2023-11-29T16:34:21.107468Z","shell.execute_reply":"2023-11-29T16:34:21.439015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# prompt analysis\nprint(clr.S+\"-> The 2 types of prompts that were asked in this comp are:\"+clr.E)\nfor k in range(len(train_prompts)):\n info = train_prompts.loc[k, :]\n print(clr.S+f\"Prompt {k}\"+clr.E, \"\\n\",\n clr.S+\"prompt_name:\"+clr.E, info.prompt_name, \"\\n\",\n clr.S+\"instructions:\"+clr.E, info.instructions, \"\\n\",\n clr.S+\"source_text:\"+clr.E, info.source_text[:40], \"...\", \"\\n\")\n \n# plot\nprint(clr.S+\"-> The types of responses based on prompt ID is fairly evenly distributed.\"+clr.E,\n \"\\n\")\nplt.figure(figsize=(20, 10))\nfigure = sns.countplot(data=train_comp,\n x=\"prompt_id\",palette=my_colors[2:])\nshow_values_on_bars(figure, h_v=\"v\", space=0.4)\nplt.title('[Comp Data] No. of texts based on prompt ID',\n weight=\"bold\", size=20)\n\nplt.xlabel(\"Prompt ID\", size = 18, weight=\"bold\")\nplt.ylabel(\"Count\", size = 18, weight=\"bold\")\n \nsns.despine(right=True, top=True, left=True)\nplt.show();","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-29T16:34:21.441181Z","iopub.execute_input":"2023-11-29T16:34:21.441456Z","iopub.status.idle":"2023-11-29T16:34:21.748143Z","shell.execute_reply.started":"2023-11-29T16:34:21.441431Z","shell.execute_reply":"2023-11-29T16:34:21.747116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot 1\nprint(clr.S+\"-> We have 700 more generated data from Radek.\"+clr.E,\n \"\\n\")\nplt.figure(figsize=(20, 10))\nfigure = sns.countplot(data=train_gen,\n x=\"generated\",palette=my_colors)\nshow_values_on_bars(figure, h_v=\"v\", space=0.4)\nplt.title('[Gen Data] Not Generated vs Generated distribution',\n weight=\"bold\", size=20)\n\nplt.xlabel(\"Generated\", size = 18, weight=\"bold\")\nplt.ylabel(\"Count\", size = 18, weight=\"bold\")\n \nsns.despine(right=True, top=True, left=True)\nplt.show();\n\n# plot 2\nprint(clr.S+\"-> And the prompts are evenly distributed! We can get to work :)\"+clr.E,\n \"\\n\")\nplt.figure(figsize=(20, 10))\nfigure = sns.countplot(data=train_gen,\n x=\"prompt_id\",palette=my_colors[2:])\nshow_values_on_bars(figure, h_v=\"v\", space=0.4)\nplt.title('[Gen Data] No. of texts based on prompt ID',\n weight=\"bold\", size=20)\n\nplt.xlabel(\"Prompt ID\", size = 18, weight=\"bold\")\nplt.ylabel(\"Count\", size = 18, weight=\"bold\")\n \nsns.despine(right=True, top=True, left=True)\nplt.show();","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-29T16:34:21.749315Z","iopub.execute_input":"2023-11-29T16:34:21.749613Z","iopub.status.idle":"2023-11-29T16:34:22.264477Z","shell.execute_reply.started":"2023-11-29T16:34:21.749587Z","shell.execute_reply":"2023-11-29T16:34:22.263562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# combine the datasets\ntrain_final = pd.concat([train_comp, train_gen], axis=0)\\\n .reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:22.265895Z","iopub.execute_input":"2023-11-29T16:34:22.266667Z","iopub.status.idle":"2023-11-29T16:34:22.273724Z","shell.execute_reply.started":"2023-11-29T16:34:22.266631Z","shell.execute_reply":"2023-11-29T16:34:22.272730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Comparison between Gen & Not Gen\n\nHere are some [traditional ways to spot if a text was generated by AI](https://goldpenguin.org/blog/check-for-ai-content/#:~:text=Repetition%20of%20words%20and%20phrases,know%20what%20it's%20talking%20about):\n1. *Watch out for Transitional Words* -> like Furthermore, Additionally, Moreover, Consequently, Hence etc.\n2. *Big vocab words are suspicious* -> like Utilized, Implemented, Leveraged, Elucidated, and Ascertained\n3. *Repetition of words and phrases* -> look for repetition of words and phrases (aka โ it doesn't really know what it's talking about)\n\nLet us test these claims in our text to see if they are true:\n* transitional words - for the selected trans words vocab it looks like the 2 distributions are the same -> aka no difference spotted.\n* big words vocab - no conclusion, as there are too fiew vocab words found within both generated and not generated text\n* repetitive words - it looks like it's the exact opposite from the initial reasoning, as there is more repetition in the non generated text vs generated one\n * I would conclude this is the case because the non generated text is written by children and teens - hence their vocabulary and reasoning is still developing, so their repetition is higher\n \n**๐ TL;DR:**\n* we might want to include `repetition_score` within our model, as it may contain valuable info. ","metadata":{}},{"cell_type":"code","source":"# import premade dictionaries\ntrans_words = json.load(open(\"/kaggle/input/english-vocabulary-miscellaneous/transitional_words.json\"))\nbig_words = json.load(open(\"/kaggle/input/english-vocabulary-miscellaneous/big_vocab_words.json\"))\n\n\n# compute scores\ntrain_final[\"transitional_score\"] = train_final.text.apply(lambda x: \n calculate_vocabulary_score(x, trans_words))\ntrain_final[\"big_vocab_score\"] = train_final.text.apply(lambda x: \n calculate_vocabulary_score(x, big_words))\ntrain_final[\"repetition_score\"] = train_final.text.apply(lambda x: \n calculate_repetitiveness(x))","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:22.275000Z","iopub.execute_input":"2023-11-29T16:34:22.275372Z","iopub.status.idle":"2023-11-29T16:34:23.226691Z","shell.execute_reply.started":"2023-11-29T16:34:22.275341Z","shell.execute_reply":"2023-11-29T16:34:23.225361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20, 10))\nsns.histplot(data=train_final, x=\"transitional_score\", \n hue=\"generated\",\n element=\"step\",\n palette=my_colors[1:], \n lw=2,)\n\nplt.xlabel(\"Transitional Score\", size = 18, weight=\"bold\")\nplt.ylabel(\"Count\", size = 18, weight=\"bold\")\n\nplt.title(\"[transitional words] Not Gen vs Gen Text\", weight=\"bold\", size=25)\nsns.despine(right=True, top=True, left=True);","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-29T16:34:23.231983Z","iopub.execute_input":"2023-11-29T16:34:23.232452Z","iopub.status.idle":"2023-11-29T16:34:23.719111Z","shell.execute_reply.started":"2023-11-29T16:34:23.232409Z","shell.execute_reply":"2023-11-29T16:34:23.718032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20, 10))\nsns.histplot(data=train_final, x=\"big_vocab_score\", \n hue=\"generated\",\n element=\"step\",\n palette=my_colors[1:], \n lw=2,)\n\nplt.xlabel(\"Big Vocab Score\", size = 18, weight=\"bold\")\nplt.ylabel(\"Count\", size = 18, weight=\"bold\")\n\nplt.title(\"[big vocab words] Not Gen vs Gen Text\", weight=\"bold\", size=25)\nsns.despine(right=True, top=True, left=True);","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-29T16:34:23.720446Z","iopub.execute_input":"2023-11-29T16:34:23.720820Z","iopub.status.idle":"2023-11-29T16:34:24.161125Z","shell.execute_reply.started":"2023-11-29T16:34:23.720787Z","shell.execute_reply":"2023-11-29T16:34:24.160225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(20, 10))\nsns.histplot(data=train_final, x=\"repetition_score\", \n hue=\"generated\",\n element=\"step\",\n palette=my_colors[1:], \n lw=2,)\n\nplt.xlabel(\"Repetition Score\", size = 18, weight=\"bold\")\nplt.ylabel(\"Count\", size = 18, weight=\"bold\")\n\nplt.title(\"[repetitive words] Not Gen vs Gen Text\", weight=\"bold\", size=25)\nsns.despine(right=True, top=True, left=True);","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2023-11-29T16:34:24.162525Z","iopub.execute_input":"2023-11-29T16:34:24.163218Z","iopub.status.idle":"2023-11-29T16:34:24.602786Z","shell.execute_reply.started":"2023-11-29T16:34:24.163185Z","shell.execute_reply":"2023-11-29T16:34:24.601814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Parameters Initialization\n\n* https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html\n* https://www.sabrepc.com/blog/Deep-Learning-and-AI/Hugging-Face-Text-Classification-Tutorial-PyTorch\n\n### โ Initial Setup","metadata":{}},{"cell_type":"code","source":"from nltk.corpus import stopwords \nfrom collections import Counter\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score\n\nimport torch\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.optim import Adam\nimport torch.nn.functional as F\nimport torch.nn as nn\n\n# BERT\nfrom transformers import BertTokenizer, BertModel, DistilBertTokenizer, DistilBertModel\nimport transformers\ntransformers.logging.set_verbosity_error()\n\n# reproducibility\nSEED = 1234\ntorch.manual_seed(SEED)\ntorch.backends.cudnn.deterministic = True\n\n\nSTOPWORDS = set(stopwords.words('english'))\nDEVICE = torch.device('cuda:0')\nprint(clr.S+\"Device: \"+clr.E, DEVICE)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:24.603966Z","iopub.execute_input":"2023-11-29T16:34:24.604247Z","iopub.status.idle":"2023-11-29T16:34:37.423102Z","shell.execute_reply.started":"2023-11-29T16:34:24.604222Z","shell.execute_reply":"2023-11-29T16:34:37.422135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### โ Global Parameters","metadata":{}},{"cell_type":"code","source":"# ~~~~~~~~~~~~~ PARAMS ~~~~~~~~~~~~~\nSEED = 24\nMAX_LEN = 512\nLAYER_SIZE = 768\nNVALID = 0.3\nLR = 0.005\nBATCH_SIZE = 16 # 32 gets OOO in this environment :)\nEPOCHS = 2\n\nBERT_MODEL = 'bert-base-uncased'\n# BERT_MODEL = 'distilbert-base-uncased'\n# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:37.424225Z","iopub.execute_input":"2023-11-29T16:34:37.424802Z","iopub.status.idle":"2023-11-29T16:34:37.429612Z","shell.execute_reply.started":"2023-11-29T16:34:37.424773Z","shell.execute_reply":"2023-11-29T16:34:37.428664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Text Cleanup","metadata":{}},{"cell_type":"code","source":"def clean_text(input_text):\n # remove punctuation\n translator = str.maketrans('', '', string.punctuation)\n text_no_punct = input_text.translate(translator)\n # remove multiple spaces (replace with a single space)\n cleaned_text = re.sub(r'\\s+', ' ', text_no_punct)\n # lower case\n cleaned_text = cleaned_text.lower()\n\n return cleaned_text","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:37.430782Z","iopub.execute_input":"2023-11-29T16:34:37.431072Z","iopub.status.idle":"2023-11-29T16:34:37.440415Z","shell.execute_reply.started":"2023-11-29T16:34:37.431040Z","shell.execute_reply":"2023-11-29T16:34:37.439703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### โ How the function works:","metadata":{}},{"cell_type":"code","source":"print(clr.S+ \"Original:\" +clr.E)\nprint(train_final[\"text\"][0][:1000],\"\\n\")\n\nprint(clr.S+ \"Cleaned:\" +clr.E)\nprint(clean_text(train_final[\"text\"][0][:1000]))","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:37.441444Z","iopub.execute_input":"2023-11-29T16:34:37.441731Z","iopub.status.idle":"2023-11-29T16:34:37.451046Z","shell.execute_reply.started":"2023-11-29T16:34:37.441702Z","shell.execute_reply":"2023-11-29T16:34:37.450154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Dataset","metadata":{}},{"cell_type":"code","source":"class GenDataset(Dataset):\n \n def __init__(self, data, max_len):\n super().__init__()\n \n# self.tokenizer = DistilBertTokenizer.from_pretrained(BERT_MODEL)\n self.tokenizer = BertTokenizer.from_pretrained(BERT_MODEL)\n self.max_len = max_len\n self.data = data\n \n def __len__(self):\n return self.data.shape[0]\n\n def __getitem__(self, index):\n # get data by index\n row = self.data.iloc[index]\n \n # clean and tokenize\n text = clean_text(row.text)\n inputs = self.tokenizer(text,\n max_length=self.max_len,\n padding=\"max_length\",\n truncation=True)\n \n ids = torch.tensor(inputs['input_ids'], dtype=torch.long)\n mask = torch.tensor(inputs['attention_mask'], dtype=torch.long)\n \n target = torch.tensor(row.generated, dtype=torch.float)\n \n # return ids, mask, target for batch\n return {\"input_ids\" : ids,\n \"attention_mask\" : mask,\n \"target\" : target}","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:37.452072Z","iopub.execute_input":"2023-11-29T16:34:37.452354Z","iopub.status.idle":"2023-11-29T16:34:37.461722Z","shell.execute_reply.started":"2023-11-29T16:34:37.452315Z","shell.execute_reply":"2023-11-29T16:34:37.460856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# automatiocally save data to cuda\ndef data_to_device(data):\n \n ids, mask, target = data.values()\n return ids.to(DEVICE), mask.to(DEVICE), target.to(DEVICE)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:37.462650Z","iopub.execute_input":"2023-11-29T16:34:37.462933Z","iopub.status.idle":"2023-11-29T16:34:37.474966Z","shell.execute_reply.started":"2023-11-29T16:34:37.462909Z","shell.execute_reply":"2023-11-29T16:34:37.474161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### โ Sanity check","metadata":{}},{"cell_type":"code","source":"ex_data = GenDataset(train_final.sample(n=6), MAX_LEN)\nex_loader = DataLoader(ex_data, batch_size=3, shuffle=False)\n\nfor k, data in enumerate(ex_loader):\n ids, mask, target = data_to_device(data)\n print(clr.S + f\"Batch: {k}\" + clr.E, \"\\n\" +\n clr.S + \"ids:\" + clr.E, ids, \"\\n\" +\n clr.S + \"target:\" + clr.E, target, \"\\n\" +\n \"=\"*50)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:37.475982Z","iopub.execute_input":"2023-11-29T16:34:37.476237Z","iopub.status.idle":"2023-11-29T16:34:43.881817Z","shell.execute_reply.started":"2023-11-29T16:34:37.476215Z","shell.execute_reply":"2023-11-29T16:34:43.880810Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Model","metadata":{}},{"cell_type":"code","source":"class TransformersModel(nn.Module):\n \n def __init__(self, bert_model, layer_size):\n super(TransformersModel, self).__init__()\n \n# self.Model = DistilBertModel.from_pretrained(bert_model)\n self.Model = BertModel.from_pretrained(bert_model)\n self.ln1 = nn.Linear(768, layer_size)\n self.ln2 = nn.Linear(layer_size, 1)\n \n def forward(self, ids, mask, verbose=False):\n '''A forward pass of this network.\n Use `verbose=True` if you want to see output shape at each pass.'''\n \n text = self.Model(ids, mask)[0]\n out_ln1 = self.ln1(text)\n out_ln2 = self.ln2(out_ln1)\n out = out_ln2[:, 0, :] \n \n if verbose:\n print(\"===============\")\n print(clr.S+\"Text Out Shape:\"+clr.E, text.shape)\n print(clr.S+\"After LN1 Shape:\"+clr.E, out_ln1.shape)\n print(clr.S+\"After LN2 Shape:\"+clr.E, out_ln2.shape)\n print(clr.S+\"Output Shape:\"+clr.E, out.shape)\n \n return out","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:43.882949Z","iopub.execute_input":"2023-11-29T16:34:43.883247Z","iopub.status.idle":"2023-11-29T16:34:43.891480Z","shell.execute_reply.started":"2023-11-29T16:34:43.883220Z","shell.execute_reply":"2023-11-29T16:34:43.890507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### โ Sanity check","metadata":{}},{"cell_type":"code","source":"ex_model = TransformersModel(BERT_MODEL, layer_size=200).to(DEVICE)\nex_model.train()\n\n# we'll use the dataloader from earlier\nfor k, data in enumerate(ex_loader):\n ids, mask, target = data_to_device(data)\n break\n \nprint(clr.S+\"Input data shape:\"+clr.E, len(ids), \"texts.\", \"\\n\")\n\n# Make a prediction\nout = ex_model(ids, mask, verbose=True)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:43.892548Z","iopub.execute_input":"2023-11-29T16:34:43.892828Z","iopub.status.idle":"2023-11-29T16:34:48.815191Z","shell.execute_reply.started":"2023-11-29T16:34:43.892804Z","shell.execute_reply":"2023-11-29T16:34:48.814220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 7. Train Functions\n\n### โ Loader Function","metadata":{}},{"cell_type":"code","source":"def get_loaders(train, valid, batch_size):\n\n train_dataset = GenDataset(train, MAX_LEN)\n valid_dataset = GenDataset(valid, MAX_LEN)\n\n train_loader = DataLoader(train_dataset, batch_size=batch_size, \n num_workers=8, shuffle=True)\n valid_loader = DataLoader(valid_dataset, batch_size=batch_size, \n num_workers=8, shuffle=False)\n \n return train_loader, valid_loader","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:48.816267Z","iopub.execute_input":"2023-11-29T16:34:48.816532Z","iopub.status.idle":"2023-11-29T16:34:48.822178Z","shell.execute_reply.started":"2023-11-29T16:34:48.816510Z","shell.execute_reply":"2023-11-29T16:34:48.821235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### โ Model Trainer","metadata":{}},{"cell_type":"code","source":"def model_trainer():\n '''Function to train the model.'''\n \n # ๐ W&B configs\n config_defaults = {\"model\": BERT_MODEL,\n \"batch_size\": BATCH_SIZE,\n \"layer_size\" : LAYER_SIZE,\n \"lr\" : LR,\n \"epochs\" : EPOCHS}\n config_defaults.update(CONFIG)\n \n with wandb.init(project=COMP_ID, config=config_defaults):\n config = wandb.config\n \n # loaders\n train_loader, valid_loader = get_loaders(train, valid, \n batch_size=config.batch_size)\n\n # model\n model = TransformersModel(BERT_MODEL, layer_size=config.layer_size).to(DEVICE)\n optimizer = Adam(model.parameters(), lr=config.lr)\n criterion = nn.MSELoss()\n\n print(clr.S + f\"--- lr {config.lr} | epochs {config.epochs} ---\" + clr.E)\n print(clr.S + f\"--- batch {config.batch_size} | layer {config.layer_size} ---\" + clr.E)\n\n for e in range(config.epochs):\n print(clr.S + f\"- Epoch {e} -\" + clr.E)\n\n # -Train the Model-\n model.train()\n\n losses = []\n for k, data in enumerate(train_loader):\n ids, mask, target = data_to_device(data)\n\n optimizer.zero_grad()\n out = model(ids, mask)\n loss = criterion(out, target)\n loss.backward()\n optimizer.step()\n\n losses.append(loss.cpu().detach().numpy().tolist())\n\n # log training loss\n train_loss = np.mean(losses)\n wandb.log({\"mean_train_loss\": np.float(train_loss)}, step=e)\n print(\"Mean Train Loss:\", train_loss)\n\n\n # -Evaluate the Model-\n model.eval() \n\n valid_preds, valid_targets = [], []\n with torch.no_grad():\n for k, data in enumerate(valid_loader):\n ids, mask, target = data_to_device(data)\n\n out = model(ids, mask)\n\n valid_preds.append(out.detach().cpu().numpy().ravel())\n valid_targets.append(target.detach().cpu().numpy().ravel())\n\n\n # save results\n valid_preds = np.concatenate(valid_preds)\n valid_targets = np.concatenate(valid_targets)\n roc = roc_auc_score(valid_targets, valid_preds)\n\n wandb.log({\"valid_ROC\": np.float(roc),\n \"epoch\": e})\n print(f\"Valid ROC: {roc}\")","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:48.823429Z","iopub.execute_input":"2023-11-29T16:34:48.823771Z","iopub.status.idle":"2023-11-29T16:34:48.840875Z","shell.execute_reply.started":"2023-11-29T16:34:48.823740Z","shell.execute_reply":"2023-11-29T16:34:48.839732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 8. Training ...","metadata":{}},{"cell_type":"code","source":"# split\ntrain, valid = train_test_split(train_final, test_size=0.3, random_state=SEED)\n\nprint(clr.S+\"Train:\"+clr.E, \"\\n\",\n train[\"generated\"].value_counts() / train[\"generated\"].value_counts().sum())\nprint(clr.S+\"Valid:\"+clr.E, \"\\n\",\n valid[\"generated\"].value_counts() / valid[\"generated\"].value_counts().sum())","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:48.842197Z","iopub.execute_input":"2023-11-29T16:34:48.842581Z","iopub.status.idle":"2023-11-29T16:34:48.859887Z","shell.execute_reply.started":"2023-11-29T16:34:48.842546Z","shell.execute_reply":"2023-11-29T16:34:48.858999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_trainer()","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:34:48.861096Z","iopub.execute_input":"2023-11-29T16:34:48.861435Z","iopub.status.idle":"2023-11-29T16:38:29.996487Z","shell.execute_reply.started":"2023-11-29T16:34:48.861392Z","shell.execute_reply":"2023-11-29T16:38:29.995492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 9. [๐wandb] Hyperparameter Tuning\n\n> (0.53) Sweep image below available [here](https://wandb.ai/andrada/2023_detect_gen_text/sweeps/hfz90b19?workspace=user-andrada).\n\n![](\"https://i.imgur.com/R9A86pz.png\")
\n\nOther sweeps:\n* distilbert (0.54): https://wandb.ai/andrada/2023_detect_gen_text/sweeps/hvpodn5n?workspace=user-andrada","metadata":{}},{"cell_type":"code","source":"# Sweep Config\nsweep_config = {\n \"method\": \"random\",\n \"metric\": {\n \"name\": \"valid_ROC\",\n \"goal\": \"maximize\" \n },\n \"parameters\": {\n \"batch_size\": {\n \"values\": [8, 16]\n },\n \"layer_size\": {\n \"values\": [100, 200, 300, 768, 987]\n },\n \"lr\": {\n 'distribution': 'uniform',\n 'max': 0.1,\n 'min': 0\n },\n \"epochs\": {\n \"values\": [1, 3, 5, 8]\n }\n }\n}\n\nprint(sweep_config)\nsweep_id = wandb.sweep(sweep_config, project=COMP_ID)","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:38:29.998010Z","iopub.execute_input":"2023-11-29T16:38:29.998295Z","iopub.status.idle":"2023-11-29T16:38:46.912621Z","shell.execute_reply.started":"2023-11-29T16:38:29.998268Z","shell.execute_reply":"2023-11-29T16:38:46.911661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ๐ sweep\nstart = time()\n\n# count = the number of trials/experiments to run\nwandb.agent(sweep_id, model_trainer, count=9)\nprint(\"Sweeping took:\", round((time()-start)/60, 1), \"mins\")","metadata":{"execution":{"iopub.status.busy":"2023-11-29T16:38:46.914059Z","iopub.execute_input":"2023-11-29T16:38:46.914437Z","iopub.status.idle":"2023-11-29T17:44:12.493121Z","shell.execute_reply.started":"2023-11-29T16:38:46.914396Z","shell.execute_reply":"2023-11-29T17:44:12.492106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**TODO**:\n* make infer notebook\n* add metadata\n* change model","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ๐ [my W&B dash](https://wandb.ai/andrada/2023_detect_gen_text?workspace=user-andrada)\n \n![](\"https://i.imgur.com/OhHsTWv.png\")
\n\n------\n\n![](\"https://i.imgur.com/knxTRkO.png\")
\n\n### My Specs\n\n* ๐ฅ Z8 G4 Workstation\n* ๐พ 2 CPUs & 96GB Memory\n* ๐ฎ 2x NVIDIA A6000\n* ๐ป Zbook Studio G9 on the go","metadata":{}}]}