{"cells":[{"cell_type":"markdown","metadata":{},"source":"# Grupo Bimbo: sampling-aware LightGBM + FM features\n\nI came back to Grupo Bimbo as a late-submission exercise. The model is one LightGBM per test week on lag and group-mean features, with the predictions of a small factorization machine and a hashed linear model added as features. The part that cost me the most, and the reason for the title, is that both test weeks are row samples of roughly a third of a normal week. Any feature that counts rows inside the week being predicted has to be computed the same way in training, otherwise the model looks fine on a holdout and falls apart on the leaderboard.\n\nLate-submission scores (RMSLE):\n\n| version | change | public | private |\n|---|---|---|---|\n| v1 | lags, group means, in-week row counts | 0.47554 | 0.50436 |\n| v2 | in-week counts removed | 0.45300 | 0.46794 |\n| v3 | counts back, computed on a simulated 33.6% sample in training | 0.44323 | 0.45862 |\n| v4 | + FM, hashed linear and target-encoding features | 0.43243 | 0.44415 |\n| v5 | + product-family and new-pair features | 0.42953 | 0.44397 |\n\nThis notebook reproduces v5 with `USE_NAME_FEATURES = True` and v4 with `False`. The submitted v4 and v5 averaged two LightGBM seeds; here I use one to stay inside the Kaggle time limit, so expect a slightly worse score. My week-9 holdout for v1 said 0.445, so the gap to 0.504 private was the first thing to explain.","id":"6a492cde"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"import os, gc, glob, re, time, unicodedata, zipfile\n\nN_THREADS = int(os.environ.get(\"N_THREADS\", 4))\nos.environ.setdefault(\"POLARS_MAX_THREADS\", str(N_THREADS))\n\nimport numpy as np\nimport polars as pl\nimport lightgbm as lgb\n\nDATA_DIR = os.environ.get(\"DATA_DIR\", \"/kaggle/input\")   # the competition folder is searched recursively\nSUBSAMPLE = os.environ.get(\"SUBSAMPLE\", \"0\") == \"1\"      # clients with Cliente_ID % 4 == 0 only; for quick local runs\nUSE_NAME_FEATURES = True                                 # True = v5, False = v4\nRUN_VALIDATION = os.environ.get(\"RUN_VALIDATION\", \"0\") == \"1\"   # train on week 8, early-stop on week 9\nWORK_DIR = os.environ.get(\"WORK_DIR\", \"/tmp/bimbo\" + (\"_sub\" if SUBSAMPLE else \"\"))\nos.makedirs(WORK_DIR, exist_ok=True)\n\nSAMPLE_P = 0.336             # share of rows kept in the test weeks\nN_LAGS = {1: 5, 2: 4}        # min_lag=1 is the week-10 model, min_lag=2 the week-11 model\nSEEDS = [0]\n# Boosting rounds for the final fit on weeks 8+9 = 1.15 x best iteration of the week-8 -> week-9 run.\n# 1.15 x best iteration from the full-data week-9 runs (v4: 409 / 541, v5: 470 / 752)\nROUNDS = {False: {1: 470, 2: 622}, True: {1: 540, 2: 865}}[USE_NAME_FEATURES]\n\nPARAMS = dict(objective=\"regression\", learning_rate=0.1, num_leaves=255, min_data_in_leaf=200,\n              feature_fraction=0.8, bagging_fraction=0.7, bagging_freq=1, max_bin=255,\n              verbose=-1, num_threads=N_THREADS)\nKEYS = [\"Agencia_ID\", \"Canal_ID\", \"Ruta_SAK\", \"Cliente_ID\", \"Producto_ID\"]\nT0 = time.time()\n\ndef log(*a):\n    print(f\"[{(time.time() - T0) / 60:6.1f} min]\", *a, flush=True)","id":"abdc151a"},{"cell_type":"markdown","metadata":{},"source":"## Data\n\nDepending on how the competition files are mounted they come as plain CSV or as `.csv.zip`. Zipped files are extracted to local disk once, then read with polars. All IDs fit in int32, which keeps the 74M training rows at about 3 GB.","id":"f07c234b"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"def find_csv(name):\n    for p in [os.path.join(DATA_DIR, name), *glob.glob(os.path.join(DATA_DIR, \"**\", name), recursive=True)]:\n        if os.path.isfile(p):\n            return p\n    zips = [os.path.join(DATA_DIR, name + \".zip\"), *glob.glob(os.path.join(DATA_DIR, \"**\", name + \".zip\"), recursive=True)]\n    for z in zips:\n        if os.path.isfile(z):\n            out = os.path.join(WORK_DIR, \"raw\")\n            if not os.path.isfile(os.path.join(out, name)):\n                with zipfile.ZipFile(z) as zf:\n                    member = [m for m in zf.namelist() if m.endswith(name)][0]\n                    zf.extract(member, out)\n                    if member != name:\n                        os.replace(os.path.join(out, member), os.path.join(out, name))\n            return os.path.join(out, name)\n    raise FileNotFoundError(f\"{name} not found under {DATA_DIR}: {sorted(glob.glob(os.path.join(DATA_DIR, '**'), recursive=True))[:40]}\")\n\n\nI32 = pl.Int32\nID_TYPES = {k: I32 for k in KEYS}\ntr = pl.read_csv(find_csv(\"train.csv\"),\n                 columns=[\"Semana\", *KEYS, \"Venta_uni_hoy\", \"Dev_uni_proxima\", \"Demanda_uni_equil\"],\n                 schema_overrides={\"Semana\": pl.Int8, **ID_TYPES, \"Venta_uni_hoy\": I32,\n                                   \"Dev_uni_proxima\": I32, \"Demanda_uni_equil\": I32})\nte = pl.read_csv(find_csv(\"test.csv\"), schema_overrides={\"id\": I32, \"Semana\": pl.Int8, **ID_TYPES})\nALL_TEST_IDS = te[\"id\"]\nif SUBSAMPLE:\n    tr = tr.filter(pl.col(\"Cliente_ID\") % 4 == 0)\n    te = te.filter(pl.col(\"Cliente_ID\") % 4 == 0)\ntr = tr.with_columns(y=pl.col(\"Demanda_uni_equil\").log1p().cast(pl.Float32)).drop(\"Demanda_uni_equil\")\n\nproducts = pl.read_csv(find_csv(\"producto_tabla.csv\"), schema_overrides={\"Producto_ID\": I32})\nclients = pl.read_csv(find_csv(\"cliente_tabla.csv\"), schema_overrides={\"Cliente_ID\": I32})\ntowns = pl.read_csv(find_csv(\"town_state.csv\"), schema_overrides={\"Agencia_ID\": I32})\nlog(f\"train {tr.shape}, test {te.shape}, {tr.estimated_size('gb'):.1f} GB\")","id":"9dee1e87"},{"cell_type":"markdown","metadata":{},"source":"## The test weeks are samples\n\nWeek 9 in train has about 10.4M rows, weeks 10 and 11 in test about 3.5M each. The ratio of rows per agency between week 10 and week 9 sits in a narrow band around 0.34, while per client it varies a lot. That is what independent random sampling of rows looks like; dropping whole clients or routes would look different.","id":"bcd599ed"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"weeks = pl.concat([tr.group_by(\"Semana\").len(), te.group_by(\"Semana\").len()]).sort(\"Semana\")\nprint(\"rows per week:\", dict(weeks.iter_rows()))\n\nfor g in [\"Agencia_ID\", \"Cliente_ID\"]:\n    r = (te.filter(pl.col(\"Semana\") == 10).group_by(g).len()\n           .join(tr.filter(pl.col(\"Semana\") == 9).group_by(g).len(), on=g, suffix=\"_w9\")\n           .select((pl.col(\"len\") / pl.col(\"len_w9\")).alias(\"ratio\"))[\"ratio\"])\n    print(f\"rows per {g}, week 10 / week 9: median {r.median():.3f}, \"\n          f\"10-90% {r.quantile(0.1):.2f}-{r.quantile(0.9):.2f}\")","id":"2e78ebc2"},{"cell_type":"markdown","metadata":{},"source":"v1 had four features counting rows of the target week itself: rows for the same client, route x product, client x product and agency x product. They were strong on the holdout, and in the test they are about 2.7 times smaller than anything the model saw in training. On a client subsample (`Cliente_ID % 4 == 0`) I recomputed the week-9 holdout with the counts taken from a 34% sample of week 9. The same week-10 style model went from 0.4468 to 0.4763, which is about the size of the leaderboard gap.\n\n| week-9 holdout, subsample | week-10 model | week-11 model |\n|---|---|---|\n| full counts in train and holdout (what my validation reported) | 0.4468 | 0.4540 |\n| same model, test-like counts in the holdout | 0.4763 | 0.4912 |\n| same model, test-like counts divided by 0.336 | 0.4582 | 0.4644 |\n| no count features | 0.4516 | 0.4613 |\n| counts on a simulated sample in training too | 0.4482 | 0.4556 |\n\nThe last row is what v3 onwards does. For every training week I draw a Bernoulli(0.336) mask over its rows and give each row a count of itself plus the other kept rows in its group. All rows stay in training, only the count changes. In the test the counts are taken from the file as it is. Rescaling alone is not enough because a small group's count in the test is 1 + Binomial, not a shrunken copy of the full count.","id":"d0aec965"},{"cell_type":"markdown","metadata":{},"source":"## Lag and group-mean features\n\nNothing unusual here; the 1st place writeup (The Slippery Appraisals) describes a much larger version of the same thing. Log demand of the same client x product x route for the last 4 or 5 available weeks, the return rate for those weeks, and the mean log demand and row count per week for nine groupings over the last four available weeks. `min_lag` is how far back the most recent usable week is: 1 for week 10, 2 for week 11.","id":"a22da7bd"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"AGG_GROUPS = [[\"Producto_ID\"], [\"Cliente_ID\"], [\"Cliente_ID\", \"Producto_ID\"], [\"Ruta_SAK\", \"Producto_ID\"],\n              [\"Agencia_ID\", \"Producto_ID\"], [\"Ruta_SAK\", \"Cliente_ID\", \"Producto_ID\"], [\"Canal_ID\", \"Producto_ID\"],\n              [\"Ruta_SAK\"], [\"Agencia_ID\"]]\nCOUNT_GROUPS = [[\"Cliente_ID\"], [\"Producto_ID\", \"Ruta_SAK\"], [\"Cliente_ID\", \"Producto_ID\"], [\"Agencia_ID\", \"Producto_ID\"]]\nAGG_WINDOW = 4\n\n\ndef short(g):\n    return \"_\".join(k.split(\"_\")[0][:4] for k in g)\n\n\ndef base_product_table():\n    name = pl.col(\"NombreProducto\")\n    grams = name.str.extract(r\"(\\d+)\\s?[gG]\\b\", 1).cast(pl.Float32)\n    kilos = name.str.extract(r\"(\\d+(?:\\.\\d+)?)\\s?[kK][gG]\\b\", 1).cast(pl.Float32) * 1000\n    return products.select(\n        \"Producto_ID\",\n        weight=pl.coalesce(grams, kilos),\n        pieces=name.str.extract(r\"(\\d+)p\\b\", 1).cast(pl.Float32),\n        brand=name.str.extract(r\"\\s([A-Z0-9]{2,5})\\s\\d+$\", 1).cast(pl.Categorical).to_physical().cast(pl.Int32),\n    ).unique(\"Producto_ID\")\n\n\nBASE_PRODUCTS = base_product_table()\n\n\ndef build_base(tr, df, T, min_lag, n_lags, sample_p=None):\n    \"\"\"df: rows of week T. Uses only weeks <= T - min_lag of tr.\"\"\"\n    df = df.with_columns(pl.lit(T).cast(pl.Int8).alias(\"Semana\"))\n    last = T - min_lag\n    ck = [\"Cliente_ID\", \"Producto_ID\", \"Ruta_SAK\", \"Agencia_ID\"]\n    lag_src = (tr.filter(pl.col(\"Semana\").is_between(last - n_lags + 1, last))\n                 .group_by([*ck, \"Semana\"])\n                 .agg(pl.col(\"y\").mean(), pl.col(\"Venta_uni_hoy\").sum(), pl.col(\"Dev_uni_proxima\").sum()))\n    for i in range(n_lags):\n        lag = min_lag + i\n        part = lag_src.filter(pl.col(\"Semana\") == last - i).select(\n            *ck, pl.col(\"y\").alias(f\"lag{lag}\"),\n            (pl.col(\"Dev_uni_proxima\") / (pl.col(\"Venta_uni_hoy\") + 1)).cast(pl.Float32).alias(f\"ret_rate{lag}\"))\n        df = df.join(part, on=ck, how=\"left\")\n    del lag_src\n    lag_cols = [f\"lag{min_lag + i}\" for i in range(n_lags)]\n    df = df.with_columns(lag_mean=pl.mean_horizontal(lag_cols),\n                         lag_cnt=pl.sum_horizontal([pl.col(c).is_not_null() for c in lag_cols]))\n\n    agg_src = tr.filter(pl.col(\"Semana\").is_between(last - AGG_WINDOW + 1, last)).select([*KEYS, \"y\"])\n    for g in AGG_GROUPS:\n        a = agg_src.group_by(g).agg(pl.col(\"y\").mean().cast(pl.Float32).alias(f\"m_{short(g)}\"),\n                                    (pl.len() / AGG_WINDOW).cast(pl.Float32).alias(f\"n_{short(g)}\"))\n        df = df.join(a, on=g, how=\"left\")\n    del agg_src\n\n    # rows of the target week itself; in training they are counted on a simulated test-like sample\n    if sample_p is not None:\n        keep = np.random.default_rng(T * 10 + min_lag).random(df.height) < sample_p\n        df = df.with_columns(_keep=pl.Series(keep))\n    for g in COUNT_GROUPS:\n        if sample_p is None:\n            cnt = pl.len().over(g)\n        else:\n            cnt = pl.col(\"_keep\").sum().over(g) - pl.col(\"_keep\").cast(pl.UInt32) + 1\n        df = df.with_columns(cnt.cast(pl.Float32).alias(\"cur_n_\" + short(g)))\n    if sample_p is not None:\n        df = df.drop(\"_keep\")\n    return df.join(BASE_PRODUCTS, on=\"Producto_ID\", how=\"left\")","id":"a69125fd"},{"cell_type":"markdown","metadata":{},"source":"## FM, hashed linear and target-encoding features\n\nFeeding the predictions of linear and factorization models on sparse ID interactions into a tree model is the \"deep features\" idea from the 3rd place writeup (Team Mystic). They used VW, FTRL, libFM and libFFM. I wrote a small PyTorch version instead:\n\n1. A factorization machine (k=16) over eight fields: client, product, route, agency, channel, the first two words of the product name, the brand suffix of the name, and the state. Exported: its prediction, the client and product biases, and the client-product embedding dot product. Each ID is replaced by its field's \"unknown\" slot with probability 0.02 during training, so clients and products that never appeared still get a trained vector.\n2. A linear model on the same eight fields plus all 28 pairwise crosses, hashed into 2^22 or 2^24 buckets.\n3. Shrunk target means along product -> agency x product -> route x product -> client x product, and a client offset relative to its route x product means.\n\nBefore fitting, history rows are collapsed to one row per key combination with a recency-weighted mean (0.8 per week of age) and a weight. Everything is fit only on weeks up to T - min_lag and each row's features depend on its own keys only, so the sampling of the test weeks does not affect them. I checked this by building the features for a 34% sample of week 9 and for the full week: identical values.\n\nFour fits are needed, one per history cutoff: week 6 (target 8, week-11 model), 7 (target 8 week-10 model and target 9 week-11 model), 8 (target 9, week-10 model) and 9 (both test weeks).\n\nTuning notes from the client subsample: more epochs, a higher learning rate or k=32 all lowered training error and raised next-week error. I read that as the model following week-to-week drift, not classical overfitting. A low learning rate (1e-3) with 8 epochs and the 0.8 recency weight was the best of what I tried.","id":"07963b55"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"DEEP_CFG = dict(fm_k=16, fm_epochs=8, fm_lr=1e-3, fm_l2=1e-5, id_dropout=0.02, batch=4096,\n                lin_epochs=4, lin_lr=1e-3, lin_l2=1e-6, te_m=(3.0, 5.0, 5.0, 3.0), recency=0.8, seed=0)\nDEEP_COLS = [\"fm_pred\", \"fm_bias_c\", \"fm_bias_p\", \"fm_cp\", \"fm_unk\", \"lin_pred\",\n             \"te_ap\", \"te_rp\", \"te_cp\", \"te_coff\", \"te_rp_coff\"]\nFIELDS = [\"Cliente_ID\", \"Producto_ID\", \"Ruta_SAK\", \"Agencia_ID\", \"Canal_ID\", \"pshort\", \"pbrand\", \"state\"]\n\n_name = pl.col(\"NombreProducto\")\nSIDE_PRODUCT = products.unique(\"Producto_ID\").select(\n    \"Producto_ID\",\n    pshort=_name.str.replace(r\"\\s\\d.*$\", \"\").str.to_lowercase().str.split(\" \").list.slice(0, 2).list.join(\" \"),\n    pbrand=_name.str.extract(r\"\\s([A-Z0-9]{2,5})\\s\\d+$\", 1).fill_null(\"?\"))\nSIDE_STATE = towns.select(\"Agencia_ID\", state=pl.col(\"State\").str.to_uppercase().str.strip_chars())\n\n\ndef with_side(df):\n    return (df.with_row_index(\"_r\")\n              .join(SIDE_PRODUCT, on=\"Producto_ID\", how=\"left\")\n              .join(SIDE_STATE, on=\"Agencia_ID\", how=\"left\")\n              .sort(\"_r\").drop(\"_r\")\n              .with_columns(pl.col(\"pshort\", \"pbrand\", \"state\").fill_null(\"?\")))\n\n\ndef encode(train_side, *frames):\n    \"\"\"Per-field index, 0 = not seen in history. Returns local indices and each field's offset.\"\"\"\n    maps, offsets, off = {}, [], 0\n    for f in FIELDS:\n        u = train_side.select(pl.col(f).unique().sort()).to_series()\n        maps[f] = pl.DataFrame({f: u, \"_ix\": np.arange(1, len(u) + 1, dtype=np.int32)})\n        offsets.append(off)\n        off += len(u) + 1\n    out = []\n    for fr in frames:\n        cols = []\n        base = fr.select(FIELDS).with_row_index(\"_r\")\n        for f in FIELDS:\n            cols.append(base.select(\"_r\", f).join(maps[f], on=f, how=\"left\").sort(\"_r\")[\"_ix\"].fill_null(0).to_numpy())\n        out.append(np.stack(cols, 1).astype(np.int32))\n    return out, np.array(offsets, dtype=np.int32), off\n\n\ndef hash_cross(loc, bits):\n    \"\"\"Hashed bucket of every single field and every pair of fields.\"\"\"\n    F = loc.shape[1]\n    u = loc.astype(np.uint64)\n    out = np.empty((len(loc), F + F * (F - 1) // 2), dtype=np.int32)\n    mask = np.uint64((1 << bits) - 1)\n    c = 0\n    with np.errstate(over=\"ignore\"):\n        for i in range(F):\n            out[:, c] = ((u[:, i] * np.uint64(0x9E3779B97F4A7C15) + np.uint64(i + 1) * np.uint64(0xC2B2AE3D27D4EB4F))\n                         >> np.uint64(20)) & mask\n            c += 1\n        for i in range(F):\n            for j in range(i + 1, F):\n                h = (u[:, i] * np.uint64(0x9E3779B97F4A7C15)) ^ (u[:, j] * np.uint64(0xD6E8FEB86659FD93)) \\\n                    ^ (np.uint64(100 + i * F + j) * np.uint64(0xC2B2AE3D27D4EB4F))\n                h ^= h >> np.uint64(29)\n                h *= np.uint64(0xBF58476D1CE4E5B9)\n                out[:, c] = (h >> np.uint64(20)) & mask\n                c += 1\n    return out\n\n\ndef shrunk_target_means(hist, keys):\n    m1, m2, m3, m4 = DEEP_CFG[\"te_m\"]\n    y = pl.col(\"y\")\n    glob_mean = float(hist[\"y\"].mean())\n    out = keys.with_row_index(\"_r\")\n    prod = hist.group_by(\"Producto_ID\").agg(y.sum().alias(\"_s\"), pl.len().alias(\"_n\"))\n    out = out.join(prod, on=\"Producto_ID\", how=\"left\").with_columns(\n        te_p=(pl.col(\"_s\").fill_null(0) + m1 * glob_mean) / (pl.col(\"_n\").fill_null(0) + m1)).drop(\"_s\", \"_n\")\n    for g, name, parent, m in [([\"Agencia_ID\", \"Producto_ID\"], \"te_ap\", \"te_p\", m1),\n                               ([\"Ruta_SAK\", \"Producto_ID\"], \"te_rp\", \"te_ap\", m2),\n                               ([\"Cliente_ID\", \"Producto_ID\"], \"te_cp\", \"te_rp\", m3)]:\n        a = hist.group_by(g).agg(y.sum().alias(\"_s\"), pl.len().alias(\"_n\"))\n        out = out.join(a, on=g, how=\"left\").with_columns(\n            ((pl.col(\"_s\").fill_null(0) + m * pl.col(parent)) / (pl.col(\"_n\").fill_null(0) + m)).alias(name)\n        ).drop(\"_s\", \"_n\")\n    rp = hist.group_by([\"Ruta_SAK\", \"Producto_ID\"]).agg(y.mean().alias(\"_rp\"))\n    cr = hist.join(rp, on=[\"Ruta_SAK\", \"Producto_ID\"]).group_by(\"Cliente_ID\").agg(\n        (y - pl.col(\"_rp\")).sum().alias(\"_s\"), pl.len().alias(\"_n\"))\n    out = out.join(cr, on=\"Cliente_ID\", how=\"left\").with_columns(\n        te_coff=pl.col(\"_s\").fill_null(0) / (pl.col(\"_n\").fill_null(0) + m4))\n    out = out.with_columns(te_rp_coff=pl.col(\"te_rp\") + pl.col(\"te_coff\"))\n    return out.sort(\"_r\").select([pl.col(c).cast(pl.Float32) for c in [\"te_ap\", \"te_rp\", \"te_cp\", \"te_coff\", \"te_rp_coff\"]])","id":"bd0a3f50"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"def fit_torch_models(hist, keys):\n    \"\"\"FM + hashed linear model fitted on `hist`, scored on `keys`. torch is only needed here.\"\"\"\n    import torch\n    cfg = DEEP_CFG\n    torch.set_num_threads(N_THREADS)\n    torch.manual_seed(cfg[\"seed\"])\n\n    rw = pl.lit(cfg[\"recency\"]) ** (pl.lit(hist[\"Semana\"].max()) - pl.col(\"Semana\")).cast(pl.Float64)\n    agg = with_side(hist.with_columns(_rw=rw).group_by(KEYS).agg(\n        ((pl.col(\"y\") * pl.col(\"_rw\")).sum() / pl.col(\"_rw\").sum()).cast(pl.Float32).alias(\"y\"),\n        pl.col(\"_rw\").sum().cast(pl.Float32).alias(\"w\")).sort(KEYS))\n    y = agg[\"y\"].to_numpy().astype(np.float32)\n    w = agg[\"w\"].to_numpy().astype(np.float32)\n    (loc, tloc), offsets, vocab = encode(agg, agg, with_side(keys))\n    del agg\n    gc.collect()\n    w0 = float((y * w).sum() / w.sum())\n    bits = 22 if len(y) < 8_000_000 else 24\n    log(f\"  {len(y):,} history keys, vocab {vocab:,}, hash bits {bits}\")\n\n    class FM(torch.nn.Module):\n        def __init__(self, n, k):\n            super().__init__()\n            self.b, self.v = torch.nn.Embedding(n, 1), torch.nn.Embedding(n, k)\n            torch.nn.init.zeros_(self.b.weight)\n            torch.nn.init.normal_(self.v.weight, std=0.01)\n            self.w0 = torch.nn.Parameter(torch.tensor(w0))\n\n        def forward(self, x):\n            v = self.v(x)\n            inter = 0.5 * (v.sum(1).pow(2) - v.pow(2).sum(1)).sum(1)\n            return self.w0 + self.b(x).squeeze(-1).sum(1) + inter, v\n\n    class Lin(torch.nn.Module):\n        def __init__(self, n):\n            super().__init__()\n            self.w = torch.nn.Embedding(n, 1)\n            torch.nn.init.zeros_(self.w.weight)\n            self.w0 = torch.nn.Parameter(torch.tensor(w0))\n\n        def forward(self, x):\n            e = self.w(x)\n            return self.w0 + e.squeeze(-1).sum(1), e\n\n    def fit(model, X, epochs, lr, l2, drop, batch):\n        g = torch.Generator().manual_seed(cfg[\"seed\"])\n        opt = torch.optim.Adam(model.parameters(), lr=lr)\n        Xt, yt, wt = torch.from_numpy(X), torch.from_numpy(y), torch.from_numpy(w)\n        for ep in range(epochs):\n            perm, tot, tw = torch.randperm(len(yt), generator=g), 0.0, 0.0\n            for s in range(0, len(yt), batch):\n                ix = perm[s:s + batch]\n                pred, emb = model(drop(Xt[ix].long(), g))\n                wb = wt[ix]\n                mse = (wb * (pred - yt[ix]) ** 2).sum() / wb.sum()\n                loss = mse + l2 * emb.pow(2).sum(-1).mean()\n                opt.zero_grad(set_to_none=True)\n                loss.backward()\n                opt.step()\n                tot += float(mse.detach()) * float(wb.sum())\n                tw += float(wb.sum())\n            log(f\"  {type(model).__name__} epoch {ep + 1}/{epochs}  train rmse {np.sqrt(tot / tw):.4f}\")\n        return model\n\n    def predict(model, X, batch=200_000):\n        with torch.no_grad():\n            return np.concatenate([model(torch.from_numpy(X[s:s + batch]).long())[0].numpy()\n                                   for s in range(0, len(X), batch)]).astype(np.float32)\n\n    p = cfg[\"id_dropout\"]\n    offs = torch.from_numpy(offsets.astype(np.int64))\n    drop_fm = lambda xb, g: torch.where(torch.rand(xb.shape, generator=g) < p, offs.expand_as(xb), xb)\n    drop_lin = lambda xb, g: torch.where(torch.rand(xb.shape, generator=g) < p, torch.zeros_like(xb), xb)\n\n    out = {}\n    fm = fit(FM(vocab, cfg[\"fm_k\"]), loc + offsets, cfg[\"fm_epochs\"], cfg[\"fm_lr\"], cfg[\"fm_l2\"], drop_fm, cfg[\"batch\"])\n    X = tloc + offsets\n    out[\"fm_pred\"] = predict(fm, X)\n    with torch.no_grad():\n        Xt = torch.from_numpy(X).long()\n        b = fm.b(Xt).squeeze(-1).numpy()\n        out[\"fm_bias_c\"], out[\"fm_bias_p\"] = b[:, 0].astype(np.float32), b[:, 1].astype(np.float32)\n        out[\"fm_cp\"] = (fm.v(Xt[:, 0]) * fm.v(Xt[:, 1])).sum(-1).numpy().astype(np.float32)\n    out[\"fm_unk\"] = (tloc[:, 0] == 0).astype(np.float32) + 2 * (tloc[:, 1] == 0).astype(np.float32)\n    del fm, Xt, X, b\n    gc.collect()\n\n    lin = fit(Lin(1 << bits), hash_cross(loc, bits), cfg[\"lin_epochs\"], cfg[\"lin_lr\"], cfg[\"lin_l2\"], drop_lin, cfg[\"batch\"] * 2)\n    out[\"lin_pred\"] = predict(lin, hash_cross(tloc, bits))\n    del lin, loc, tloc, y, w\n    gc.collect()\n    return pl.DataFrame(out)","id":"77ed76f3"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"def target_rows(T):\n    src = tr if T <= 9 else te\n    cols = [\"_rid\", *KEYS, \"y\"] if T <= 9 else [\"_rid\", \"id\", *KEYS]\n    return src.filter(pl.col(\"Semana\") == T).with_row_index(\"_rid\").select(cols)\n\n\ndef deep_path(T, L):\n    return os.path.join(WORK_DIR, f\"deep_T{T}_L{L}.parquet\")\n\n\n# history cutoff -> (target week, min_lag) pairs scored with that fit\nCUTOFFS = {6: [(8, 2)], 7: [(8, 1), (9, 2)], 8: [(9, 1)], 9: [(10, 1), (11, 2)]}\nfor last, items in CUTOFFS.items():\n    if all(os.path.exists(deep_path(T, L)) for T, L in items):\n        continue\n    log(f\"deep features, history <= week {last}\")\n    keys = pl.concat([target_rows(T).select([\"_rid\", *KEYS]).with_columns(_T=pl.lit(T), _L=pl.lit(L)) for T, L in items])\n    hist = tr.filter(pl.col(\"Semana\") <= last).select([\"Semana\", *KEYS, \"y\"])\n    feats = keys.select(\"_rid\", \"_T\", \"_L\").with_columns(*fit_torch_models(hist, keys.select(KEYS)).get_columns(),\n                                                        *shrunk_target_means(hist, keys.select(KEYS)).get_columns())\n    for T, L in items:\n        feats.filter((pl.col(\"_T\") == T) & (pl.col(\"_L\") == L)).select([\"_rid\", *DEEP_COLS]).write_parquet(deep_path(T, L))\n    del keys, hist, feats\n    gc.collect()\nlog(\"deep features ready\")","id":"a0ccd6c5"},{"cell_type":"markdown","metadata":{},"source":"On the client subsample these features moved the week-9 holdout from 0.4488 to 0.4395 for the week-10 model and from 0.4556 to 0.4471 for the week-11 model. About half of the week-10 gain and two thirds of the week-11 gain came from client x product pairs with no history in the last four weeks. That is where a factorized model helps: it can combine a client's overall level with the product's level even if the two never met. On its own the FM scored 0.474 on that holdout, about 0.025 behind the full baseline LightGBM.","id":"e5c3b210"},{"cell_type":"markdown","metadata":{},"source":"## Product families and new pairs\n\nProducts get re-coded constantly. The same loaf shows up with a new pack code, a promo box or a slightly different weight, and gets a new Producto_ID. So many \"new\" client x product pairs are a product line the client already buys. I parse the product name into brand (the last token before the ID), a short name (words before the first token with a digit, without pack and channel codes like MTA, CU or Prom), a family word and the first two words, then aggregate the client's recent demand by product line.\n\nThe second group describes the client's size: its mean residual against the route x product average, overall and within the product family. `exp_pair` = route x product mean + the client's family residual ended up as the top feature of the model that used it.\n\nOn the subsample, without the in-week counts, these two groups gave -0.0074 for the week-10 model and -0.0110 for the week-11 model, mostly on new pairs. I also tried a rule-based client type from the client names (OXXO and other chains, schools, abarrotes, person names, ...) and town/state x product means. Both gave less than 0.001, which is inside the variation I saw just from rebuilding features in a fresh process (about 0.0006). The client-type block stays in because v5 was scored with it; the town/state means are out.","id":"cabbe808"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"CHAINS = [\n    (\"CEDIS_REMISION\", r\"\\bREMISION\\b|\\bCEDIS\\b\"), (\"OXXO\", r\"\\bOXXO\\b\"),\n    (\"SEVEN_ELEVEN\", r\"\\b7 ELEVEN\\b|\\bSEVEN ELEVEN\\b|\\bSEVEN\\b\"), (\"CIRCULO_K\", r\"\\bCIRCULO K\\b\"),\n    (\"EXTRA\", r\"^EXTRA\\b|\\bTIENDA EXTRA\\b\"), (\"AMPM\", r\"\\bAM ?PM\\b\"), (\"SIX\", r\"\\bSIX\\b\"),\n    (\"MODELORAMA\", r\"\\bMODELORAMA\\b\"), (\"KIOSKO\", r\"^KIOSKO\\b\"), (\"DICONSA\", r\"\\bDICONSA\\b|\\bCONASUPO\\b\"),\n    (\"ISSSTE\", r\"\\bISSSTE\\b\"), (\"BODEGA_AURRERA\", r\"\\bBOD(EGA)? AURRERA\\b|\\bAURRERA\\b|\\bMI BODEGA\\b\"),\n    (\"WALMART\", r\"\\bWAL ?MART\\b|\\bSUPERCENTER\\b|^SC\\b\"), (\"SUPERAMA\", r\"\\bSUPERAMA\\b\"),\n    (\"SAMS_COSTCO\", r\"\\bSAMS\\b|\\bCOSTCO\\b|\\bCITY CLUB\\b\"), (\"CHEDRAUI\", r\"\\bCHEDRAUI\\b\"), (\"SORIANA\", r\"\\bSORIANA\\b\"),\n    (\"COMERCIAL_MEXICANA\", r\"\\bCOMERCIAL MEXICANA\\b|^MEGA\\b|\\bBODEGA COMERCIAL\\b\"), (\"HEB\", r\"\\bHEB\\b|\\bH E B\\b\"),\n    (\"GIGANTE\", r\"\\bGIGANTE\\b\"),\n    (\"REGIONAL_SUPER\", r\"\\bCASA LEY\\b|\\bCALIMAX\\b|\\bALSUPER\\b|\\bMERZA\\b|\\bSMART\\b|\\bSUPER CITY\\b|\\bLA COMER\\b\"),\n    (\"FARMACIA_CHAIN\", r\"\\bBENAVIDES\\b|\\bFARMACIAS GUADALAJARA\\b|\\bSIMILARES\\b|\\bFARMACIAS DEL AHORRO\\b\"),\n    (\"BURGER_KING\", r\"\\bBURGER KING\\b\"), (\"VIPS_SANBORNS\", r\"\\bVIPS\\b|\\bSANBORNS\\b|\\bTOKS\\b\"),\n    (\"CINEMA\", r\"\\bCINEPOLIS\\b|\\bCINEMEX\\b\"), (\"BOTTLER\", r\"\\bCOCA COLA\\b|\\bEMBOTELLADORA\\b|\\bBEBIDAS\\b|\\bPEPSI\\b\"),\n]\nCHAIN_TO_TYPE = {\"CEDIS_REMISION\": \"DISTRIBUTOR\", \"BOTTLER\": \"DISTRIBUTOR\", \"DICONSA\": \"GOVERNMENT_STORE\",\n                 \"ISSSTE\": \"GOVERNMENT_STORE\", \"FARMACIA_CHAIN\": \"PHARMACY\", \"BURGER_KING\": \"FAST_FOOD\",\n                 \"VIPS_SANBORNS\": \"RESTAURANT\", \"CINEMA\": \"RESTAURANT\",\n                 **{c: \"CONVENIENCE\" for c in [\"OXXO\", \"SEVEN_ELEVEN\", \"CIRCULO_K\", \"EXTRA\", \"AMPM\", \"SIX\", \"MODELORAMA\", \"KIOSKO\"]},\n                 **{c: \"SUPERMARKET\" for c in [\"BODEGA_AURRERA\", \"WALMART\", \"SUPERAMA\", \"SAMS_COSTCO\", \"CHEDRAUI\", \"SORIANA\",\n                                               \"COMERCIAL_MEXICANA\", \"HEB\", \"GIGANTE\", \"REGIONAL_SUPER\"]}}\nTYPES = [\n    (\"NO_ID\", r\"^NO IDENTIFICADO\\b|^SIN NOMBRE\\b|^$\"),\n    (\"SCHOOL\", r\"\\bESC(UELA)?\\b|\\bPRIMARIA\\b|\\bSECUNDARIA\\b|\\bTELESECUNDARIA\\b|\\bCOLEGIO\\b|\\bPREPA(RATORIA)?\\b\"\n               r\"|\\bUNIVERSIDAD\\b|\\bINSTITUTO\\b|\\bKINDER\\b|\\bJARDIN DE NI|\\bCBTIS?\\b|\\bCONALEP\\b|\\bCECYT|\\bBACHILLER\"\n               r\"|\\bTECNOLOGICO\\b|\\bCOOPERATIVA\\b|\\bGUARDERIA\\b|\\bUNAM\\b|\\bIPN\\b|\\bUAEM\\b|\\bFACULTAD\\b|\\bCOBACH|\\bCETIS\\b\"),\n    (\"HOSPITAL\", r\"\\bHOSPITAL\\b|\\bCLINICA\\b|\\bIMSS\\b|\\bSANATORIO\\b|\\bCRUZ ROJA\\b|\\bMEDIC\"),\n    (\"PHARMACY\", r\"\\bFARMACIA\"),\n    (\"HOTEL\", r\"\\bHOTEL\\b|\\bMOTEL\\b|\\bSUITES\\b\"),\n    (\"FAST_FOOD\", r\"\\bHAMBURGUESA|\\bHOT ?DOGS?\\b|\\bBURGER|\\bTORTAS?\\b|\\bTAQUERIA\\b|\\bTACOS\\b|\\bPIZZA|\\bPOLLOS?\\b\"\n                  r\"|\\bALITAS\\b|\\bKFC\\b|\\bCARLS\\b|\\bSNACK\\b|\\bANTOJITOS\\b|\\bSANDWICH|\\bLONCHERIA\\b|\\bBIRRIA\\b|\\bJUGOS\\b\"),\n    (\"RESTAURANT\", r\"\\bRESTAURANTE?\\b|\\bREST\\b|\\bCAFETERIA\\b|\\bCAFE\\b|\\bCOMEDOR\\b|\\bCOCINA\\b|\\bFONDA\\b|\\bBANQUETES\\b\"\n                   r\"|\\bBAR\\b|\\bCANTINA\\b|\\bMARISCOS\\b|\\bBUFFET\\b\"),\n    (\"GAS_STATION\", r\"\\bGASOLINERA\\b|\\bPEMEX\\b|\\bSERVICIO\\b\"),\n    (\"SPECIALTY\", r\"\\bPAPELERIA\\b|\\bCARNICERIA\\b|\\bFRUTERIA\\b|\\bDULCERIA\\b|\\bDULCES\\b|\\bVINOS\\b|\\bLICORES\\b\"\n                  r\"|\\bTORTILLERIA\\b|\\bPANADERIA\\b|\\bPOLLERIA\\b|\\bRECAUDERIA\\b|\\bVERDULERIA\\b|\\bCREMERIA\\b|\\bSALCHICHONERIA\\b\"\n                  r\"|\\bNEVERIA\\b|\\bPALETERIA\\b|\\bCERVECERIA\\b|\\bCERVEZA|\\bSEMILLAS\\b|\\bFORRAJE\"),\n    (\"KIOSK_STAND\", r\"\\bPUESTO\\b|\\bCASETA\\b|\\bKIOSKO\\b|\\bCARRITO\\b|\\bLOCAL\\b|\\bESTANQUILLO\\b|\\bMODULO\\b|\\bSTAND\\b\"),\n    (\"MINISUPER\", r\"\\bMINI ?SUPER\\b|\\bSUPER\\b|\\bAUTOSERVICIO\\b|\\bMERCADO\\b|\\bBODEGA\\b|\\bBODEGON\\b|\\bCENTRO COMERCIAL\\b\"),\n    (\"GROCERY\", r\"\\bABARROTES\\b|\\bABTS\\b|\\bABARR|\\bMISC(ELAN[EI]A)?\\b|\\bTIENDA\\b|\\bTIENDITA\\b|\\bDEPOSITO\\b\"\n                r\"|\\bEXPENDIO\\b|\\bCOMERCIAL\\b|\\bCONSUMO\\b|\\bCOMODIN\\b|\\bSUPERETTE\\b\"),\n    (\"COMPANY\", r\"\\bS ?A DE C ?V\\b|\\bS DE R ?L\\b|\\bSA\\b|\\bCV\\b|\\bGRUPO\\b|\\bCORPORATIVO\\b|\\bDISTRIBUIDORA\\b|\\bINDUSTRIA\"\n                r\"|\\bSERVICIOS\\b|\\bCOMPANIA\\b|\\bFMA\\d\"),\n]\nGIVEN = set(\"\"\"MARIA JOSE JUAN LUIS JESUS CARLOS FRANCISCO ANTONIO MIGUEL MANUEL PEDRO JORGE JAVIER RAFAEL RAMON ALBERTO\nMARIO VICTOR SALVADOR FERNANDO RICARDO ROBERTO ALEJANDRO SERGIO DAVID DANIEL ARTURO EDUARDO ENRIQUE OSCAR RAUL GERARDO\nHECTOR JAIME ALFREDO ANGEL ARMANDO GUILLERMO MARTIN ISMAEL IGNACIO ADRIAN GABRIEL RUBEN ALFONSO ERNESTO FELIPE ANDRES\nPABLO RODOLFO AGUSTIN GUADALUPE JOAQUIN JULIO CESAR EFRAIN ABEL LORENZO SANTIAGO TOMAS VICENTE EDGAR OSWALDO NOE ISRAEL\nANA ROSA CARMEN MARTHA LUZ ELENA TERESA JUANA MARGARITA ISABEL ALICIA GLORIA PATRICIA ROSARIO LAURA SILVIA BLANCA LETICIA\nJOSEFINA IRMA CLAUDIA YOLANDA LUISA ESTHER DOLORES NORMA BERTHA VERONICA ADRIANA ELIZABETH ALMA LOURDES SOCORRO REFUGIO\nALEJANDRA GABRIELA SANDRA MONICA LILIA GUILLERMINA ANGELICA ANGELES CONCEPCION CRISTINA OFELIA ROCIO SUSANA LORENA\nREYNA REINA EVA LIDIA MARISOL GRACIELA ESPERANZA ANTONIA FRANCISCA JULIA ELVIA MIRIAM KARINA BEATRIZ EUGENIA ARACELI\nVIRGINIA HILDA LUCIA RAQUEL AURORA AMPARO CATALINA DELIA ELVIRA EMMA ENRIQUETA FLORENCIA IRENE JUDITH MARICELA MA\"\"\".split())\nSURNAMES = set(\"\"\"GARCIA HERNANDEZ GONZALEZ MARTINEZ LOPEZ RODRIGUEZ PEREZ SANCHEZ RAMIREZ CRUZ FLORES GOMEZ TORRES DIAZ\nREYES VAZQUEZ GUTIERREZ JIMENEZ MORALES RUIZ CHAVEZ MENDOZA AGUILAR ALVAREZ JUAREZ MORENO ORTIZ RIVERA CASTILLO\nROMERO RAMOS MEDINA GUZMAN HERRERA VARGAS CASTRO MENDEZ ESPINOZA CONTRERAS ESTRADA VELAZQUEZ ORTEGA SALAZAR ROJAS\nSANDOVAL NAVARRO DOMINGUEZ CARRILLO SOTO LARA ALVARADO RIOS OROZCO AVILA IBARRA CERVANTES PADILLA CARDENAS GUERRERO\nLEON LUNA SANTIAGO SANTOS SILVA DELGADO VEGA CAMPOS NUNEZ MALDONADO CABRERA RIVAS FUENTES ACOSTA ROSALES SOLIS\nMARQUEZ MIRANDA TREJO VALDEZ BAUTISTA CORTES ZAMORA ESCOBAR MOLINA PACHECO ROSAS CAMACHO SALINAS GALLEGOS\nBECERRA ALONSO VILLANUEVA PENA MEJIA CORONA ARELLANO FRANCO ESPINOSA ACEVEDO HUERTA TAPIA DURAN MEZA\"\"\".split())\nNICKNAMES = GIVEN | {\"LUPITA\", \"MARY\", \"LUPE\", \"GABY\", \"ROSY\", \"DANY\", \"PATY\", \"LULU\"}\n\n\ndef norm_name(s):\n    s = unicodedata.normalize(\"NFKD\", str(s or \"\")).encode(\"ascii\", \"ignore\").decode().upper()\n    return re.sub(r\"\\s+\", \" \", re.sub(r\"[^A-Z0-9 ]\", \" \", s)).strip()\n\n\ndef classify_client(name):\n    chain = next((label for label, pat in CHAINS if re.search(pat, name)), None)\n    ctype = CHAIN_TO_TYPE.get(chain) if chain else None\n    if ctype is None:\n        ctype = next((label for label, pat in TYPES if re.search(pat, name)), None)\n    toks = name.split()\n    n_named = sum(t in GIVEN for t in toks) + sum(t in SURNAMES for t in toks)\n    is_person = int(ctype is None and len(toks) >= 2 and n_named >= max(2, len(toks) - 1))\n    if ctype is None:\n        if is_person:\n            ctype = \"PERSON\"\n        elif len(toks) == 1 and toks[0] in NICKNAMES:\n            ctype = \"NICKNAME_STORE\"\n        elif toks and toks[0] in {\"LA\", \"EL\", \"LOS\", \"LAS\", \"MI\", \"DON\", \"DONA\", \"SAN\", \"SANTA\"}:\n            ctype = \"NAMED_STORE\"\n        else:\n            ctype = \"OTHER\"\n    return chain or \"NONE\", ctype, is_person\n\n\ndef add_codes(t, cols):\n    for col in cols:\n        cats = sorted(t[col].unique().to_list())\n        t = t.with_columns(pl.col(col).replace_strict({c: i for i, c in enumerate(cats)}, return_dtype=pl.Int32).alias(f\"{col}_id\"))\n    return t\n\n\ndef client_table():\n    c = clients.with_columns(nm=pl.col(\"NombreCliente\").map_elements(norm_name, return_dtype=pl.Utf8))\n    # some IDs have several names; prefer an identified one\n    c = (c.with_columns(noid=(pl.col(\"nm\") == \"NO IDENTIFICADO\").cast(pl.Int8)).sort([\"Cliente_ID\", \"noid\"])\n          .group_by(\"Cliente_ID\", maintain_order=True).agg(pl.col(\"nm\").first(), pl.len().alias(\"n_names\")))\n    c = c.with_columns(name_share=pl.len().over(\"nm\"))\n    rows = [classify_client(n) for n in c[\"nm\"].to_list()]\n    c = c.with_columns(chain=pl.Series([r[0] for r in rows]), client_type=pl.Series([r[1] for r in rows]),\n                       is_person=pl.Series([r[2] for r in rows], dtype=pl.Int8),\n                       name_ntok=pl.col(\"nm\").str.split(\" \").list.len().cast(pl.Int16))\n    return add_codes(c, [\"client_type\", \"chain\"])\n\n\nCODE_TOKENS = {\"MTA\", \"MTB\", \"TAB\", \"TNB\", \"CU\", \"SP\", \"ME\", \"CR1\", \"CR2\", \"CR3\", \"CJM\", \"DPY\", \"TIRA\", \"INST\", \"MG\",\n               \"LT\", \"FS\", \"KC\", \"CJ\", \"PROM\", \"PROMO\", \"PROMOCAJA\", \"CONSUMOS\", \"MUESTREO\", \"TWIN\", \"DUO\", \"TRI\",\n               \"PACK\", \"MM\", \"BK\", \"NES\", \"WB\", \"EXH\", \"SS\", \"CHAROLA\", \"PALLET\", \"GDE\", \"MED\", \"CHICO\", \"MINI\",\n               \"JUMBO\", \"FAMILIAR\", \"REG\", \"RW\", \"LTA\", \"BCO\"}\nPROMO_TOKENS = {\"PROM\", \"PROMO\", \"PROMOCAJA\", \"MUESTREO\", \"REGALO\", \"GRATIS\", \"OFERTA\", \"NAVIDAD\", \"SNVALENTIN\"}\nMULTI_TOKENS = {\"TWIN\", \"DUO\", \"TRI\", \"PACK\", \"2PQ\", \"3PQ\", \"CJ\", \"CAJA\", \"CHAROLA\", \"PALLET\", \"CUBETA\", \"GRANEL\"}\nFOOD_SERVICE = {\"BK\", \"MCD\", \"KFC\", \"WB\", \"FS\", \"QUARTER\"}\nPKG_CODES = {\"MTA\", \"MTB\", \"TAB\", \"TNB\", \"CU\", \"SP\", \"ME\", \"CR1\", \"CR2\", \"CR3\", \"CJM\"}\nFAMILY_STOP = {\"mi\", \"super\", \"el\", \"la\", \"los\", \"las\", \"gran\", \"big\", \"nuevo\", \"new\", \"doble\", \"div\", \"tri\", \"pan\"}\n\n\ndef parse_product(name):\n    toks = name.split()\n    if toks and toks[-1].isdigit():\n        toks = toks[:-1]\n    brand = toks[-1].upper() if len(toks) >= 2 else \"NONE\"\n    body = toks[:-1] if len(toks) >= 2 else toks\n    txt = \" \".join(body)\n    m = re.search(r\"(\\d+(?:[.\\s]\\d+)?)\\s?(kg|g|ml|l)\\b\", txt, re.I) or re.search(r\"(\\d+)\\s?(kg|g|ml)(?=[A-Za-z])\", txt, re.I)\n    weight = None\n    if m:\n        weight = float(m.group(1).replace(\" \", \".\")) * (1000 if m.group(2).lower() in (\"kg\", \"l\") else 1)\n    mp = re.search(r\"\\b(\\d+)\\s?p\\b\", txt, re.I) or re.search(r\"\\b(\\d+)p(?=\\d)\", txt, re.I)\n    mr = re.search(r\"(\\d+)\\s?Reb\", txt, re.I)\n    up = {t.upper() for t in body}\n    short_name = []\n    for t in body:\n        if re.search(r\"\\d\", t):\n            break\n        if t.upper() not in CODE_TOKENS:\n            short_name.append(t.lower())\n    short_name = short_name or [t.lower() for t in body[:1]] or [\"none\"]\n    return dict(\n        brand=brand, short_name=\" \".join(short_name), short2=\" \".join(short_name[:2]),\n        family=next((w for w in short_name if w not in FAMILY_STOP), short_name[0]),\n        weight=weight, pieces=float(mp.group(1)) if mp else None, slices=float(mr.group(1)) if mr else None,\n        is_promo=int(bool(up & PROMO_TOKENS) or \"prom\" in txt.lower()),\n        is_multi=int(bool(up & MULTI_TOKENS) or bool(re.search(r\"\\d\\s?pq\\b\", txt, re.I))),\n        is_foodservice=int(bool(up & FOOD_SERVICE)), is_mini=int(\"MINI\" in up or \"MG\" in up),\n        pkg_code=next((t.upper() for t in body if t.upper() in PKG_CODES), \"NONE\"),\n        n_codes=sum(t.upper() in CODE_TOKENS for t in body),\n        is_liquid=int(bool(re.search(r\"\\d\\s?ml\\b|\\blata\\b\", txt, re.I))),\n    )\n\n\ndef name_product_table():\n    p = products.unique(\"Producto_ID\", keep=\"first\").sort(\"Producto_ID\")\n    t = pl.DataFrame([parse_product(n) for n in p[\"NombreProducto\"].to_list()],\n                     schema_overrides={\"weight\": pl.Float32, \"pieces\": pl.Float32, \"slices\": pl.Float32})\n    t = p.select(\"Producto_ID\").with_columns(*t.get_columns()).with_columns(\n        w_per_piece=(pl.col(\"weight\") / pl.col(\"pieces\")).cast(pl.Float32))\n    return add_codes(t, [\"brand\", \"family\", \"short2\", \"short_name\", \"pkg_code\"])\n\n\ndef town_table():\n    t = towns.with_columns(state=pl.col(\"State\").map_elements(norm_name, return_dtype=pl.Utf8),\n                           town=pl.col(\"Town\").map_elements(norm_name, return_dtype=pl.Utf8))\n    return add_codes(t, [\"state\", \"town\"]).select(\"Agencia_ID\", \"state_id\", \"town_id\")\n\n\nCLIENT_ATTRS = [\"client_type_id\", \"chain_id\", \"is_person\", \"name_ntok\", \"n_names\", \"name_share\"]\nPRODUCT_ATTRS = [\"family_id\", \"short2_id\", \"short_name_id\", \"brand_id\", \"pkg_code_id\", \"slices\", \"w_per_piece\",\n                 \"is_promo\", \"is_multi\", \"is_foodservice\", \"is_mini\", \"n_codes\", \"is_liquid\"]\nif USE_NAME_FEATURES:\n    CT, PT, TT = client_table(), name_product_table(), town_table()\n    NONE_CHAIN = CT.filter(pl.col(\"chain\") == \"NONE\")[\"chain_id\"][0]\n    log(f\"name tables: {CT.height:,} clients, {PT.height:,} products\")","id":"c304536a"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"def agg_join(src, df, g, name, stats):\n    exprs = []\n    if \"mean\" in stats:\n        exprs.append(pl.col(\"y\").mean().cast(pl.Float32).alias(f\"m_{name}\"))\n    if \"n\" in stats:\n        exprs.append(pl.len().cast(pl.Float32).alias(f\"n_{name}\"))\n    if \"res\" in stats:\n        exprs.append(pl.col(\"res\").mean().cast(pl.Float32).alias(f\"r_{name}\"))\n    return df.join(src.group_by(g).agg(exprs), on=g, how=\"left\")\n\n\ndef add_name_features(tr, df, last, window=4):\n    src = (tr.filter(pl.col(\"Semana\").is_between(last - window + 1, last)).select(\"Semana\", *KEYS, \"y\")\n             .join(CT.select(\"Cliente_ID\", \"client_type_id\", \"chain_id\"), on=\"Cliente_ID\", how=\"left\")\n             .join(PT.select(\"Producto_ID\", \"family_id\", \"short2_id\", \"brand_id\"), on=\"Producto_ID\", how=\"left\")\n             .join(TT, on=\"Agencia_ID\", how=\"left\"))\n    out = (df.join(CT.select(\"Cliente_ID\", *CLIENT_ATTRS), on=\"Cliente_ID\", how=\"left\")\n             .join(PT.select(\"Producto_ID\", *PRODUCT_ATTRS), on=\"Producto_ID\", how=\"left\")\n             .join(TT, on=\"Agencia_ID\", how=\"left\"))\n\n    # client type / chain x product\n    for g, nm in [([\"client_type_id\", \"Producto_ID\"], \"ctype_Prod\"), ([\"chain_id\", \"Producto_ID\"], \"chain_Prod\"),\n                  ([\"Agencia_ID\", \"client_type_id\", \"Producto_ID\"], \"Agen_ctype_Prod\"),\n                  ([\"Ruta_SAK\", \"client_type_id\", \"Producto_ID\"], \"Ruta_ctype_Prod\"), ([\"client_type_id\"], \"ctype\")]:\n        out = agg_join(src, out, g, nm, (\"mean\", \"n\"))\n    out = out.with_columns(pl.when(pl.col(\"chain_id\") == NONE_CHAIN).then(None).otherwise(pl.col(\"m_chain_Prod\")).alias(\"m_chain_Prod\"))\n\n    # product lines\n    for g, nm in [([\"family_id\"], \"fam\"), ([\"short2_id\"], \"sh2\"), ([\"brand_id\"], \"brand\"),\n                  ([\"Ruta_SAK\", \"family_id\"], \"Ruta_fam\"), ([\"Ruta_SAK\", \"short2_id\"], \"Ruta_sh2\"),\n                  ([\"Ruta_SAK\", \"brand_id\"], \"Ruta_brand\"), ([\"Agencia_ID\", \"short2_id\"], \"Agen_sh2\"),\n                  ([\"Cliente_ID\", \"family_id\"], \"Clie_fam\"), ([\"Cliente_ID\", \"short2_id\"], \"Clie_sh2\"),\n                  ([\"Cliente_ID\", \"brand_id\"], \"Clie_brand\")]:\n        out = agg_join(src, out, g, nm, (\"mean\", \"n\"))\n    seen = src.select(\"Producto_ID\").unique().with_columns(_seen=pl.lit(1, pl.Int8))\n    out = out.join(seen, on=\"Producto_ID\", how=\"left\").with_columns(\n        prod_new=(1 - pl.col(\"_seen\").fill_null(0)).cast(pl.Int8)).drop(\"_seen\")\n\n    # client size relative to its route, and priors for pairs never seen\n    rp = src.group_by([\"Ruta_SAK\", \"Producto_ID\"]).agg(pl.col(\"y\").mean().alias(\"_mrp\"))\n    ap = src.group_by([\"Agencia_ID\", \"Producto_ID\"]).agg(pl.col(\"y\").mean().alias(\"_map\"))\n    res = src.join(rp, on=[\"Ruta_SAK\", \"Producto_ID\"], how=\"left\").with_columns(res=pl.col(\"y\") - pl.col(\"_mrp\"))\n    for g, nm in [([\"Cliente_ID\"], \"Clie\"), ([\"Cliente_ID\", \"family_id\"], \"Clie_fam\"), ([\"Cliente_ID\", \"brand_id\"], \"Clie_brand\"),\n                  ([\"client_type_id\", \"Producto_ID\"], \"ctype_Prod\"), ([\"chain_id\", \"Producto_ID\"], \"chain_Prod\"),\n                  ([\"Canal_ID\", \"Producto_ID\"], \"Cana_Prod\")]:\n        out = agg_join(res, out, g, nm, (\"res\",))\n    del res\n    breadth = src.group_by(\"Cliente_ID\").agg(clie_nprod=pl.col(\"Producto_ID\").n_unique().cast(pl.Float32),\n                                             clie_nfam=pl.col(\"family_id\").n_unique().cast(pl.Float32),\n                                             clie_nweeks=pl.col(\"Semana\").n_unique().cast(pl.Float32))\n    base = pl.coalesce(\"_mrp\", \"_map\")\n    out = (out.join(breadth, on=\"Cliente_ID\", how=\"left\")\n              .join(rp, on=[\"Ruta_SAK\", \"Producto_ID\"], how=\"left\").join(ap, on=[\"Agencia_ID\", \"Producto_ID\"], how=\"left\")\n              .with_columns(exp_pair=(base + pl.coalesce(\"r_Clie_fam\", \"r_Clie\", pl.lit(0.0))).cast(pl.Float32),\n                            exp_pair2=(base + pl.coalesce(\"r_ctype_Prod\", pl.lit(0.0)) + pl.coalesce(\"r_Clie\", pl.lit(0.0))).cast(pl.Float32))\n              .drop(\"_mrp\", \"_map\"))\n    seen_cr = src.select(\"Cliente_ID\", \"Ruta_SAK\").unique().with_columns(clie_at_route=pl.lit(1, pl.Int8))\n    seen_pr = src.select(\"Producto_ID\", \"Ruta_SAK\").unique().with_columns(prod_at_route=pl.lit(1, pl.Int8))\n    return (out.join(seen_cr, on=[\"Cliente_ID\", \"Ruta_SAK\"], how=\"left\").join(seen_pr, on=[\"Producto_ID\", \"Ruta_SAK\"], how=\"left\")\n               .with_columns(pl.col(\"clie_at_route\").fill_null(0), pl.col(\"prod_at_route\").fill_null(0)))","id":"c3dae598"},{"cell_type":"markdown","metadata":{},"source":"## Two models, trained on weeks 8 and 9\n\nWeek 10 gets a model whose most recent lag is the previous week (min_lag=1) with five lag weeks; five was 0.0027 better than four on the subsample. Week 11 gets a separate direct model with min_lag=2. Both are trained on target weeks 8 and 9. Adding week 7 as a third target week helped by less than 0.001.\n\nTo keep memory under control the feature frame for each week is written straight into one preallocated float32 matrix and dropped, and the LightGBM Dataset is built before the next step.","id":"646e699c"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"DROP = {\"id\", \"_rid\", \"Semana\", \"y\", \"Cliente_ID\", \"Ruta_SAK\"}\nWEEK_ROWS = dict(tr.group_by(\"Semana\").len().iter_rows())\n\n\ndef make_frame(T, L, sampled):\n    df = build_base(tr, target_rows(T), T, L, N_LAGS[L], sample_p=SAMPLE_P if sampled else None)\n    if USE_NAME_FEATURES:\n        df = add_name_features(tr, df, T - L)\n    return df.join(pl.read_parquet(deep_path(T, L)), on=\"_rid\", how=\"left\")\n\n\ndef feature_matrix(weeks, L, sampled=True, cols=None):\n    n = sum(WEEK_ROWS[T] for T in weeks)\n    X, y, off = None, np.empty(n, dtype=np.float32), 0\n    for T in weeks:\n        df = make_frame(T, L, sampled)\n        if cols is None:\n            cols = [c for c in df.columns if c not in DROP and df[c].dtype.is_numeric()]\n        if X is None:\n            X = np.empty((n, len(cols)), dtype=np.float32)\n        for j, c in enumerate(cols):\n            X[off:off + df.height, j] = df[c].cast(pl.Float32).to_numpy()\n        y[off:off + df.height] = df[\"y\"].to_numpy()\n        off += df.height\n        del df\n        gc.collect()\n        log(f\"  week {T}, min_lag={L}: matrix {X.shape}, {X.nbytes / 1e9:.1f} GB\")\n    return X, y, cols\n\n\ndef rmse(a, b):\n    return float(np.sqrt(np.mean((a - b) ** 2)))","id":"2f654ccb"},{"cell_type":"markdown","metadata":{},"source":"The validation run below (train on week 8, early-stop on week 9, counts sampled on both) is how I picked the number of rounds. It is off by default here to save time and memory; the final fit uses 1.15 x the best iteration from those runs.","id":"a6d856cf"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"if RUN_VALIDATION:\n    for L in (1, 2):\n        Xa, ya, cols = feature_matrix([8], L)\n        dtr = lgb.Dataset(Xa, ya, feature_name=cols, free_raw_data=True, params=PARAMS).construct()\n        del Xa\n        Xv, yv, _ = feature_matrix([9], L, cols=cols)\n        dva = lgb.Dataset(Xv, yv, reference=dtr).construct()\n        m = lgb.train(PARAMS, dtr, 3000, valid_sets=[dva], callbacks=[lgb.early_stopping(30), lgb.log_evaluation(100)])\n        pred = np.clip(m.predict(Xv, num_iteration=m.best_iteration), 0, None)\n        ROUNDS[L] = int(m.best_iteration * 1.15)\n        log(f\"min_lag={L}: week-9 RMSLE {rmse(pred, yv):.5f}, best iteration {m.best_iteration}, final rounds {ROUNDS[L]}\")\n        del dtr, dva, m, Xv, yv, pred\n        gc.collect()","id":"43dad011"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"preds = []\nfor L, T_test in [(1, 10), (2, 11)]:\n    X, y, cols = feature_matrix([8, 9], L)\n    dtrain = lgb.Dataset(X, y, feature_name=cols, free_raw_data=True, params=PARAMS).construct()\n    del X, y\n    gc.collect()\n    tst = make_frame(T_test, L, sampled=False)\n    Xt = np.empty((tst.height, len(cols)), dtype=np.float32)\n    for j, c in enumerate(cols):\n        Xt[:, j] = tst[c].cast(pl.Float32).to_numpy()\n    ids = tst[\"id\"].to_numpy()\n    del tst\n    gc.collect()\n    p = np.zeros(len(ids))\n    for s in SEEDS:\n        log(f\"training min_lag={L}, seed {s}, {ROUNDS[L]} rounds, {len(cols)} features\")\n        m = lgb.train(dict(PARAMS, seed=s), dtrain, ROUNDS[L])\n        p += np.clip(m.predict(Xt), 0, None) / len(SEEDS)\n    imp = sorted(zip(cols, m.feature_importance(\"gain\")), key=lambda t: -t[1])\n    total = sum(v for _, v in imp)\n    print(\"top features:\", \", \".join(f\"{c} {v / total:.1%}\" for c, v in imp[:12]))\n    preds.append(pl.DataFrame({\"id\": ids, \"Demanda_uni_equil\": np.round(np.expm1(p), 4)}))\n    log(f\"week {T_test} done, {len(ids):,} rows\")\n    del dtrain, Xt, m, p, ids\n    gc.collect()","id":"430ee202"},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":"sub = pl.concat(preds).sort(\"id\")\nif SUBSAMPLE:\n    # rows of clients left out by the subsample get a flat value so the file still has every id\n    fill = float(np.round(np.expm1(tr.filter(pl.col(\"Semana\") == 9)[\"y\"].mean()), 4))\n    sub = (pl.DataFrame({\"id\": ALL_TEST_IDS}).join(sub, on=\"id\", how=\"left\")\n             .with_columns(pl.col(\"Demanda_uni_equil\").fill_null(fill)).sort(\"id\"))\nassert sub.height == len(ALL_TEST_IDS) == 6_999_251 and sub[\"id\"].n_unique() == sub.height\nsub.write_csv(\"submission.csv\")\nlog(\"submission.csv written\")\nprint(sub.describe())","id":"ba06062b"},{"cell_type":"markdown","metadata":{},"source":"## Things that did not make it\n\nRecursive week 11. The usual trick is to predict week 10 and use that as lag 1 for week 11. Here it cannot work well: week 10 in the test is a 34% sample, so only about 22% of week-11 rows find a predicted week-10 value (between two complete training weeks the share is 67%). A model trained with 67% lag coverage reads a missing lag as \"did not buy last week\", which is wrong most of the time. Simulated on week 9 of the subsample, recursive scored 0.4627 against 0.4558 for the direct model. Blending the two gained up to 0.0009, but averaging two seeds of the direct model alone gained 0.0010, so the blend was ordinary ensembling. The 4th/6th place writeup also said the lag-1 question for week 11 was worth very little, around 0.0007.\n\nExtra aggregates. Min/max/std of demand by group, unit prices and short trends each moved the subsample holdout by less than 0.001, so I left them out.\n\nClient types from names and town/state means. Covered above; both below the noise level. Most client names are small shops whose type does not separate demand, and big chains are already separated by channel and by their own history.\n\nThings I have not tried yet: declaring the ID-like columns as categorical in LightGBM, and a second model family for blending.","id":"f5d14666"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","pygments_lexer":"ipython3"}},"nbformat":4,"nbformat_minor":5}