{"metadata":{"kernelspec":{"name":"ir","display_name":"R","language":"R"},"language_info":{"name":"R","codemirror_mode":"r","pygments_lexer":"r","mimetype":"text/x-r-source","file_extension":".r","version":"4.4.0"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceType":"competition","sourceId":121150,"databundleVersionId":15745837},{"sourceType":"datasetVersion","sourceId":14857597,"datasetId":9503932,"databundleVersionId":15718440}],"dockerImageVersionId":30749,"isInternetEnabled":true,"language":"r","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🏛️ Project Opening: The Manifesto\nPlace this at the very beginning of your notebook in a Markdown cell.\n\n## Project Title: \nDeep Past Initiative – Machine Translation (Akkadian to English)\n\n## Scientific Preface:\nThis project represents a rigorous application of Statistical Natural Language Processing (NLP) to the challenging domain of Ancient Akkadian translation. While many might default to generic black-box models, this notebook leverages the pure, raw power of the R Programming Language.\n\nWe have engineered a high-performance Vector Space Model (VSM) from the ground up, utilizing sparse matrix computation and surgical text preprocessing. In this environment, every byte is optimized, and every token is strategically weighted.\n\n## A Final Warning:\nData Science is a battlefield of precision. When it comes to handling complex linguistic structures with mathematical elegance: No one messes with R.\n\nCore Technologies: tidyverse, text2vec, proxy, tm.","metadata":{}},{"cell_type":"code","source":"################################################################################\n# STEP 1: GLOBAL CONFIGURATION AND LIBRARY INITIALIZATION\n################################################################################\n\n# Suppressing warnings and startup messages for a clean environment\noptions(warn = -1)\n\n# Loading the primary NLP and Data Science toolkits\n# Using suppressPackageStartupMessages to prevent console clutter\nsuppressPackageStartupMessages({\n  library(tidyverse)   # Core data manipulation and string handling\n  library(tm)          # Text mining infrastructure\n  library(proxy)       # Statistical distance and similarity measures\n  library(text2vec)    # High-performance NLP vectorization engine\n  library(wordcloud)   # Frequency-based visual analytics\n  library(moments)     # Higher-order statistical analysis (Skewness/Kurtosis)\n  library(formattable) # Professional data representation\n  library(IRdisplay)   # Image rendering for Kaggle/Jupyter kernels\n})\n\n################################################################################\n# STEP 2: DATA ACQUISITION\n################################################################################\n\n# Importing the primary training and testing datasets from Kaggle's input directory\n# train_data: Contains historical Akkadian transliterations and English labels\n# test_data: Contains the target transliterations for competition inference\ntrain_data <- read.csv('/kaggle/input/deep-past-initiative-machine-translation/train.csv')\ntest_data <- read.csv('/kaggle/input/deep-past-initiative-machine-translation/test.csv')\n\n# Verification of successful loading\ncat(\"Environment initialized successfully.\\n\")\ncat(\"Training records loaded:\", nrow(train_data), \"\\n\")\ncat(\"Testing records loaded:\", nrow(test_data), \"\\n\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-25T19:32:56.198766Z","iopub.execute_input":"2026-02-25T19:32:56.200717Z","iopub.status.idle":"2026-02-25T19:32:59.077806Z","shell.execute_reply":"2026-02-25T19:32:59.076339Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"################################################################################\n# STEP 3: TEXT PREPROCESSING AND STATISTICAL OUTLIER FILTERING\n################################################################################\n\n# Defining the \"Surgical Cleaning\" function to normalize Akkadian transliterations\n# This function targets noise (digits, punctuation, and short tokens)\nclean_surgery <- function(text_vec) {\n    text_vec <- str_to_lower(text_vec)                        # Case normalization\n    text_vec <- str_replace_all(text_vec, '[[:digit:]]', '')  # Remove numerical noise\n    text_vec <- str_replace_all(text_vec, '[[:punct:]]', ' ') # Remove symbols\n    text_vec <- str_replace_all(text_vec, \"\\\\b[a-z]{1,2}\\\\b\", \" \") # Filter out 1-2 char tokens\n    text_vec <- str_squish(text_vec)                          # Eliminate redundant whitespace\n    return(text_vec)\n}\n\n# Feature Engineering: Creating clean text and calculating character length\ntrain_data <- train_data %>%\n    mutate(akd_clean = clean_surgery(transliteration),\n           akd_len = nchar(akd_clean))\n\n# Statistical Outlier Detection using the Interquartile Range (IQR) Method\n# This step treats the 'Bad Skew' in sentence length distribution\nQ1 <- quantile(train_data$akd_len, 0.25)\nQ3 <- quantile(train_data$akd_len, 0.75)\nIQR_val <- Q3 - Q1\n\n# Filtering the dataset to keep only structurally representative samples\ntrain_filtered <- train_data %>%\n    filter(akd_len >= (Q1 - 1.5 * IQR_val) & akd_len <= (Q3 + 1.5 * IQR_val))\n\n################################################################################\n# STEP 4: PRE-PROCESSING AUDIT\n################################################################################\n\n# Outputting statistical results to verify the data distribution improvement\ncat(\"Original Skewness:\", skewness(train_data$akd_len), \"\\n\")\ncat(\"Filtered Skewness:\", skewness(train_filtered$akd_len), \"\\n\")\ncat(\"Total samples retained after filtering:\", nrow(train_filtered), \"\\n\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-25T19:32:59.080974Z","iopub.execute_input":"2026-02-25T19:32:59.121078Z","iopub.status.idle":"2026-02-25T19:32:59.314668Z","shell.execute_reply":"2026-02-25T19:32:59.313148Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"################################################################################\n# STEP 5: VOCABULARY ENGINEERING AND VECTOR SPACE MODELING\n################################################################################\n\n# Initializing the iterator for tokenization\n# The 'itoken' function prepares the cleaned text for efficient processing\nit_train <- itoken(train_filtered$akd_clean, progressbar = FALSE)\n\n# Vocabulary Construction:\n# We identify unique tokens across the dataset and apply 'Pruning'\n# prune_vocabulary(term_count_min = 3) reduces dimensionality by removing \n# infrequent tokens that likely represent noise or rare dialectal variations.\nvocab <- create_vocabulary(it_train) %>% \n  prune_vocabulary(term_count_min = 3)\n\n# Initializing the Vectorizer:\n# The vectorizer maps words to specific indices in a high-dimensional space\nvectorizer <- vocab_vectorizer(vocab)\n\n# Document-Term Matrix (DTM) Construction:\n# This creates a Sparse Matrix where rows represent sentences and columns \n# represent the 196+ core Akkadian terms. This is the final input for the AI.\ndtm_train <- create_dtm(it_train, vectorizer)\n\n################################################################################\n# STEP 6: VOCABULARY AUDIT\n################################################################################\ncat(\"Total unique tokens identified:\", nrow(vocab), \"\\n\")\ncat(\"Dimensions of the Training Matrix (DTM):\", dim(dtm_train)[1], \"x\", dim(dtm_train)[2], \"\\n\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-25T19:32:59.317834Z","iopub.execute_input":"2026-02-25T19:32:59.318918Z","iopub.status.idle":"2026-02-25T19:32:59.527905Z","shell.execute_reply":"2026-02-25T19:32:59.526366Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"################################################################################\n# STEP 7: LINGUISTIC VISUALIZATION (HIGH-RESOLUTION WORD CLOUD)\n################################################################################\n\n# 1. Data Preparation\n# Selecting the top 100 most frequent terms from the pruned vocabulary\nword_freq <- vocab %>% \n  arrange(desc(term_count)) %>% \n  head(100)\n\n# 2. Graphics Device Initialization\n# Opening a high-resolution PNG device to ensure professional visual quality\n# 'res=150' enhances the clarity of the text for final report presentation\npng(\"wordcloud_large.png\", width=1000, height=1000, res=150)\n\n# 3. Word Cloud Generation\n# Utilizing a Reproducible Seed (123) to ensure consistent layout\n# The 'scale' parameter is optimized for maximum readability of dominant terms\nset.seed(123)\nwordcloud(words = word_freq$term,\n          freq = word_freq$term_count, \n          scale = c(6, 2),\n          min.freq = 2,\n          max.words = 100,\n          random.order = FALSE, \n          colors = brewer.pal(8, \"Dark2\"))\n\n# 4. Finalizing Output\n# Closing the graphics device to save the file to the local directory\ndev.off()\n\n# 5. Display Rendering\n# Visualizing the final output within the notebook interface\ndisplay_png(file = \"wordcloud_large.png\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-25T19:32:59.531142Z","iopub.execute_input":"2026-02-25T19:32:59.532261Z","iopub.status.idle":"2026-02-25T19:33:00.299987Z","shell.execute_reply":"2026-02-25T19:33:00.297607Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"################################################################################\n# STEP 8: QUANTITATIVE LINGUISTIC ANALYSIS (TOP 15 TERMS)\n################################################################################\n\n# Visualizing the distribution of the most frequent Akkadian tokens\n# This provides a precise statistical rank of the model's primary features\nword_freq %>% \n    head(15) %>%\n    ggplot(aes(x = reorder(term, term_count), y = term_count, fill = term_count)) + \n    \n    # Utilizing a gradient bar chart to represent frequency intensity\n    geom_col() + \n    \n    # Flipping coordinates for better readability of linguistic tokens\n    coord_flip() + \n    \n    # Custom color palette: 'Gold to Brown' representing ancient parchment tones\n    scale_fill_gradient(low = 'gold', high = 'brown') + \n    \n    # Clean, professional theme for academic presentation\n    theme_minimal() + \n    \n    # Labels and titles (kept minimal for integration into larger dashboards)\n    labs(title = \"Top 15 Most Frequent Akkadian Tokens\",\n         x = \"Akkadian Term\",\n         y = \"Frequency Count\",\n         fill = \"Total Count\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-25T19:33:00.303563Z","iopub.execute_input":"2026-02-25T19:33:00.305415Z","iopub.status.idle":"2026-02-25T19:33:00.692806Z","shell.execute_reply":"2026-02-25T19:33:00.691298Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"################################################################################\n# STEP 9: TEST DATA FEATURIZATION AND VECTOR ALIGNMENT\n################################################################################\n\n# 1. Surgical Cleaning of Test Transliterations\n# Applying the same 'clean_surgery' function used on the training set \n# to ensure consistent tokenization and noise reduction.\ntest_data$akd_clean <- clean_surgery(test_data$transliteration)\n\n# 2. Test Set Iteration\n# Preparing the cleaned test text for the vectorization process.\nit_test <- itoken(test_data$akd_clean, progressbar = FALSE)\n\n# 3. Test DTM Projection\n# Projecting the test data into the previously built Vector Space.\n# Important: We use the 'vectorizer' from the training phase to ensure\n# that the column indices (words) match perfectly between Train and Test.\ndtm_test <- create_dtm(it_test, vectorizer)\n\n################################################################################\n# STEP 10: INFERENCE AUDIT\n################################################################################\ncat(\"Test Matrix Dimensions:\", dim(dtm_test)[1], \"x\", dim(dtm_test)[2], \"\\n\")\ncat(\"Consistency Check: Do column counts match? \", ncol(dtm_test) == ncol(dtm_train), \"\\n\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-25T19:33:00.694977Z","iopub.execute_input":"2026-02-25T19:33:00.696150Z","iopub.status.idle":"2026-02-25T19:33:00.725733Z","shell.execute_reply":"2026-02-25T19:33:00.724345Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"################################################################################\n# STEP 11: SEMANTIC MATCHING AND TRANSLATION INFERENCE\n################################################################################\n\n# 1. Similarity Computation\n# Executing Cosine Similarity between the Test DTM and Training DTM.\n# This measures the angular distance between sentence vectors to find the \n# nearest linguistic neighbor.\nsim_matrix <- as.matrix(proxy::simil(as.matrix(dtm_test), \n                                    as.matrix(dtm_train), \n                                    method = \"cosine\"))\n\n# 2. Prediction Engine with Structural Constraints\n# Initializing the results container with original metadata for auditing\nfinal_results <- data.frame(\n  id = test_data$id, \n  Akkadian_Original = test_data$transliteration, \n  translation = NA, \n  confidence = NA\n)\n\n# Inference Loop: Mapping the best training match to each test case\nfor(i in 1:nrow(sim_matrix)) {\n  # Identifying the index of the highest similarity score\n  best_idx <- which.max(sim_matrix[i,])\n  full_text <- train_filtered$translation[best_idx]\n  \n  # Applying 'Single Sentence Constraint' for BLEU Optimization:\n  # We split the matched text at the first period to ensure a concise output.\n  single_sentence <- str_trim(str_split(full_text, \"\\\\.\")[[1]][1])\n  \n  # Final formatting: Re-inserting the period and storing the confidence score\n  final_results$translation[i] <- paste0(single_sentence, \".\")\n  final_results$confidence[i] <- sim_matrix[i, best_idx]\n}\n\n################################################################################\n# STEP 12: MODEL PERFORMANCE REVIEW & VISUAL AUDIT\n################################################################################\n\ncat(\"\\n--- Final Translation Inference Review ---\\n\")\n\n# Formatting the results for professional presentation\n# Includes 'Match_Rate' as a percentage and visual color tiles for confidence\nview_table <- final_results %>%\n  mutate(\n    Match_Rate = percent(confidence),\n    # Truncating original text for dashboard clarity\n    Akkadian_Original = str_trunc(Akkadian_Original, 50)\n  ) %>%\n  select(id, Akkadian_Original, translation, Match_Rate)\n\n# Rendering the Interactive Results Table\nformattable(view_table, list(\n  # Color gradient: Visualizing the strength of the match (White to Orange)\n  Match_Rate = color_tile(\"white\", \"orange\"),\n  # Highlighting the predicted translation in Bold Dark Blue\n  translation = formatter(\"span\", \n                          style = x ~ style(color = \"darkblue\", \n                                            font.weight = \"bold\"))\n))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-25T19:33:00.727963Z","iopub.execute_input":"2026-02-25T19:33:00.729199Z","iopub.status.idle":"2026-02-25T19:33:00.793835Z","shell.execute_reply":"2026-02-25T19:33:00.792340Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"################################################################################\n# STEP 13: FINAL EXPORT AND KAGGLE SUBMISSION GENERATION\n################################################################################\n\n# 1. Data Sanitization and Formatting\n# We perform a final trim of leading/trailing whitespaces to ensure \n# the 'translation' strings are clean for metric evaluation (BLEU/chrF++).\nsubmission_file <- final_results %>%\n    mutate(\n        translation = str_trim(translation)\n    ) %>%\n    # Selecting strictly the required columns: 'id' and 'translation'\n    select(id, translation)\n\n# 2. File Export Configuration\n# Saving the dataframe as 'submission.csv'. \n# 'row.names = FALSE' is critical to prevent index columns from corrupting the format.\nwrite.csv(submission_file, 'submission.csv', row.names = FALSE)\n\n################################################################################\n# STEP 14: SUBMISSION VERIFICATION\n################################################################################\n\n# Final system check to confirm file creation and row count\ncat(\"====================================================\\n\")\ncat(\"SUCCESS: 'submission.csv' has been generated.\\n\")\ncat(\"Total Inference Records: \", nrow(submission_file), \"\\n\")\ncat(\"Submission ready for Kaggle Leaderboard upload.\\n\")\ncat(\"====================================================\\n\")\n\n# Preview of the final output format\nprint(head(submission_file))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-02-25T19:33:00.796093Z","iopub.execute_input":"2026-02-25T19:33:00.797265Z","iopub.status.idle":"2026-02-25T19:33:00.833774Z","shell.execute_reply":"2026-02-25T19:33:00.832391Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# 🏁 Project Conclusion: The Final Verdict\nPlace this at the very end of your notebook, after your submission code.\n\n## Final Analysis & Inference Summary:\nThe pipeline is now complete. We have successfully navigated through surgical data cleaning, statistical outlier mitigation (via IQR filtering), and high-dimensional semantic matching using Cosine Similarity.\n\n## Results & Submission:\nThe generated submission.csv follows the exact architectural constraints required by the competition—optimized for BLEU and chrF++ metrics. By prioritizing linguistic integrity over noise, we have built a model that is both interpretable and robust.\n\n## The Takeaway:\nThis project proves that with the right statistical framework and the unmatched vectorization capabilities of this ecosystem, we can bridge the gap between ancient history and modern AI.\n\n## Mission Accomplished.\nAnd remember: Never underestimate the engine. No one messes with R.","metadata":{}}]}