{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.13.15","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"title":"H&M Fashion Intelligence and Recommendations"},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"94a706e4","cell_type":"markdown","source":"# H&M | Where does personalization help most?\n\nI test whether recommendations based on a customer's own purchase history beat a plain list of the most popular products. The local evaluation (16–22 September 2020) points to a clear gain for customers who shop regularly. For customers with little history I do not see evidence that is as strong.\n\n**What the notebook does**\n1. Looks at demand, customers, products, prices and trends.\n2. Builds candidate articles from recent popularity, each customer's own purchases and items bought together.\n3. Compares four stages on the same buyers: popularity, + repurchase, + item-to-item similarity, + LightGBM LambdaRank.\n4. Checks which extra features help, which customer segments gain, and how much uncertainty is left.\n5. Turns the findings into business next steps and an A/B test design.\n\nAll numbers (MAP@12, confidence intervals, candidate recall, segment results) are computed when the cells run. The final summary at the bottom collects them. They come from a sample of active customers and one evaluation week, so they are not comparable with the Kaggle leaderboard.\n\nData: [H&M Kaggle](https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/data)","metadata":{}},{"id":"fa8fc3c6","cell_type":"markdown","source":"## 0. Setup\n\nH&M red is the accent color so the charts read quickly. Every metric and chart below is computed from the data when the cell runs.","metadata":{}},{"id":"eef104c2","cell_type":"code","source":"import os, gc, math, warnings, time, json\nfrom pathlib import Path\nfrom collections import Counter, defaultdict\nfrom itertools import combinations\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport matplotlib.ticker as mticker\nimport seaborn as sns\nfrom IPython.display import display, Markdown\nfrom sklearn.metrics import ndcg_score\nimport lightgbm as lgb\n\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)\nSEED = 42\nrng = np.random.default_rng(SEED)\n\nDATA_DIR = Path('/kaggle/input/competitions/h-and-m-personalized-fashion-recommendations')\nif not DATA_DIR.exists():\n DATA_DIR = Path('./h-and-m-personalized-fashion-recommendations')\nOUTPUT_DIR = Path('/kaggle/working/hm_portfolio') if Path('/kaggle/working').exists() else Path('./hm_portfolio')\nOUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n\n# customers are sampled from those active before the first cutoff\nDEV_CUSTOMERS = 16000 # Increase to 40000+ if compute permits\nTRAIN_WEEKS = pd.to_datetime(['2020-08-26', '2020-09-02', '2020-09-09'])\nVALID_WEEK = pd.Timestamp('2020-09-16')\nTEST_WEEK = pd.Timestamp('2020-09-23')\nCANDIDATE_LIMIT = 65\nPOPULAR_N = 24\nREORDER_N = 18\nSIMILAR_N = 18\nBUILD_NEIGHBORS_MAX_USERS = 65000\nNEIGHBORS_PER_ITEM = 10\nFINAL_CUSTOMER_BATCH = 4000\nRUN_ABLATION = True\n\nHM = dict(primary='#CC071E', dark='#99172B', soft='#E86B78', muted='#E8DADD',\n grey='#C7C3C5', text='#282527', secondary='#777176', bg='#FFFFFF', grid='#EBE6E8')\nsns.set_theme(style='white', context='notebook')\nplt.rcParams.update({'figure.figsize':(11,5),'figure.dpi':125,\n 'figure.facecolor':'white','axes.facecolor':'white',\n 'axes.titlesize':14,'axes.titleweight':'bold',\n 'axes.labelcolor':HM['text'],'text.color':HM['text'],\n 'xtick.color':HM['secondary'],'ytick.color':HM['secondary'],\n 'axes.spines.top':False,'axes.spines.right':False,\n 'font.family':'DejaVu Sans'})\n\ndef finish_plot(ax, title, subtitle=None, xlabel=None, ylabel=None, save=None, grid='y'):\n ax.set_title(title, loc='left', pad=23 if subtitle else 14, fontweight='bold')\n if subtitle: ax.text(0,1.035, subtitle, transform=ax.transAxes, color=HM['secondary'], fontsize=9)\n ax.grid(axis=grid, alpha=.6, linewidth=.65, color=HM['grid'])\n ax.set_axisbelow(True)\n if xlabel is not None: ax.set_xlabel(xlabel)\n if ylabel is not None: ax.set_ylabel(ylabel)\n plt.tight_layout()\n if save: plt.savefig(OUTPUT_DIR/save, dpi=200, bbox_inches='tight')\n plt.show()\n\ndef hm_bar(series, title, subtitle='', top=12, horizontal=True, save=None):\n s = series.sort_values(ascending=False).head(top)\n fig,ax=plt.subplots(figsize=(11,max(4, .40*len(s)+1.3)))\n colors=[HM['primary']]+[HM['grey']]*(len(s)-1)\n if horizontal:\n ax.barh(s.index.astype(str)[::-1], s.values[::-1], color=colors[::-1])\n ax.xaxis.set_major_formatter(mticker.StrMethodFormatter('{x:,.0f}'))\n else:\n ax.bar(s.index.astype(str), s.values, color=colors)\n ax.tick_params(axis='x', rotation=45)\n finish_plot(ax,title,subtitle,save=save,grid='x' if horizontal else 'y')\n\nprint('LightGBM:', lgb.__version__)\nprint('Data folder:', DATA_DIR)\nprint('Output folder:', OUTPUT_DIR)\n","metadata":{"execution":{"iopub.status.busy":"2026-10-16T08:28:14.897414Z","iopub.execute_input":"2026-10-16T08:28:14.897823Z","iopub.status.idle":"2026-10-16T08:28:14.926591Z","shell.execute_reply.started":"2026-10-16T08:28:14.897791Z","shell.execute_reply":"2026-10-16T08:28:14.925131Z"},"trusted":true},"outputs":[],"execution_count":null},{"id":"94b6374a","cell_type":"markdown","source":"## 1. Loading and checking the data\n\nThe transaction file is read in chunks to keep memory under control. `customer_id` becomes an integer code for the analysis, and `article_id` stays numeric. Identical transaction rows are kept, because they can be several units of the same article. For the submission, article IDs go back to 10 digits.","metadata":{}},{"id":"8dab849e","cell_type":"code","source":"required = ['articles.csv','customers.csv','transactions_train.csv','sample_submission.csv']\nmissing = [p for p in required if not (DATA_DIR/p).exists()]\nif missing:\n raise FileNotFoundError(f'Missing {missing}. Add the H&M competition data to the Kaggle notebook.')\n\ncustomers = pd.read_csv(DATA_DIR/'customers.csv',\n dtype={'customer_id':'string','age':'float32', 'FN':'float32','Active':'float32'})\ncustomers['uid'] = np.arange(len(customers), dtype=np.int32)\ncustomers['age_group'] = pd.cut(customers['age'],bins=[0,24,34,44,54,64,110],\n labels=['≤24','25–34','35–44','45–54','55–64','65+'])\narticles = pd.read_csv(DATA_DIR/'articles.csv', low_memory=False)\narticles['article_id'] = articles['article_id'].astype('int32')\nkeep_articles=['article_id','prod_name','product_type_no','product_type_name','product_group_name',\n 'graphical_appearance_name',\n 'colour_group_code','colour_group_name','index_group_no','index_group_name',\n 'garment_group_no','garment_group_name','section_name','detail_desc']\narticles=articles[[c for c in keep_articles if c in articles.columns]].copy()\nfor col in ['prod_name', 'product_type_name', 'colour_group_name', 'product_group_name']:\n if col not in articles.columns: articles[col] = pd.NA\n\n# integer codes instead of customer hashes\ncustomer_dtype = pd.CategoricalDtype(categories=customers['customer_id'].astype(str), ordered=False)\nchunks=[]\nfor i,chunk in enumerate(pd.read_csv(DATA_DIR/'transactions_train.csv',\n usecols=['t_dat','customer_id','article_id','price','sales_channel_id'],\n dtype={'customer_id':'string','article_id':'int32','price':'float32','sales_channel_id':'uint8'},\n chunksize=2_000_000)):\n u = pd.Categorical(chunk['customer_id'].astype(str), dtype=customer_dtype).codes\n if (u<0).any(): raise ValueError('Unknown customer_id in transaction file')\n chunk = pd.DataFrame({'date':pd.to_datetime(chunk['t_dat']),\n 'uid':u.astype('int32'),\n 'article_id':chunk['article_id'].to_numpy(dtype='int32'),\n 'price':chunk['price'].to_numpy(dtype='float32'),\n 'sales_channel_id':chunk['sales_channel_id'].to_numpy(dtype='uint8')})\n chunks.append(chunk)\n if i%5==0: print(f'Rows read: {sum(len(c) for c in chunks):,} baris')\ntx=pd.concat(chunks,ignore_index=True)\ndel chunks,chunk,customer_dtype,u\ngc.collect()\ntx.sort_values('date',inplace=True,kind='stable')\ntx.reset_index(drop=True,inplace=True)\nprint('Shape:',tx.shape,'\\nMemory (GiB):',round(tx.memory_usage(deep=True).sum()/2**30,2))\nprint('Date range:',tx.date.min().date(),'to',tx.date.max().date())\nassert tx.date.max() == pd.Timestamp('2020-09-22'), 'Review the split dates for this dataset version'\nassert customers.uid.is_unique and articles.article_id.is_unique\nassert tx.article_id.isin(articles.article_id).all()\n","metadata":{"execution":{"iopub.status.busy":"2026-10-16T08:28:14.928578Z","iopub.execute_input":"2026-10-16T08:28:14.929083Z","iopub.status.idle":"2026-10-16T08:29:16.164004Z","shell.execute_reply.started":"2026-10-16T08:28:14.929041Z","shell.execute_reply":"2026-10-16T08:29:16.162937Z"},"trusted":true},"outputs":[],"execution_count":null},{"id":"5380804c","cell_type":"code","source":"# identical rows are not necessarily duplicates\nstruct = pd.DataFrame({\n 'table':['Transactions','Customers','Articles'],\n 'rows':[len(tx),len(customers),len(articles)],\n 'columns':[tx.shape[1],customers.shape[1],articles.shape[1]],\n 'missing_cells':[int(tx.isna().sum().sum()),int(customers.isna().sum().sum()),int(articles.isna().sum().sum())]\n})\ndisplay(struct)\nprint('Identical transaction tuples (possibly multi-unit purchases):',\n int(tx.duplicated(['date','uid','article_id','price','sales_channel_id']).sum()))\nprint('Transaction coverage: customers =',tx.uid.nunique(),'articles =',tx.article_id.nunique())\n\nna = (customers.isna().mean()*100).sort_values(ascending=False)\nfig,ax=plt.subplots(figsize=(11,4))\nax.bar(na.index.astype(str),na.values,color=[HM['primary'] if v>20 else HM['grey'] for v in na.values])\nax.tick_params(axis='x', rotation=35)\nfinish_plot(ax,f'{na.index[0]} is the customer field with the most missing values ({na.iloc[0]:.1f}%)',\n 'Share of missing values per customer column',ylabel='Missing (%)',save='eda_01_missingness.png')\n","metadata":{"execution":{"iopub.status.busy":"2026-10-16T08:29:16.165189Z","iopub.execute_input":"2026-10-16T08:29:16.165453Z","iopub.status.idle":"2026-10-16T08:29:29.116716Z","shell.execute_reply.started":"2026-10-16T08:29:16.16543Z","shell.execute_reply":"2026-10-16T08:29:29.115061Z"},"trusted":true},"outputs":[],"execution_count":null},{"id":"b004c705","cell_type":"markdown","source":"## 2. How demand moves over time\n\nI start with units bought, the busiest days and shifts between categories. One row is one unit, not an order I can identify, and the recorded price is not net revenue.","metadata":{}},{"id":"bda3b23d","cell_type":"code","source":"daily=tx.groupby('date',sort=True).agg(units=('article_id','size'),customers=('uid','nunique'),\n purchase_value=('price','sum'))\ndaily['units_28d']=daily.units.rolling(28,min_periods=7).mean()\nfig,ax=plt.subplots(figsize=(12,4.5))\nax.plot(daily.index,daily.units,color=HM['muted'],lw=.8,alpha=.65,label='Daily units')\nax.plot(daily.index,daily.units_28d,color=HM['primary'],lw=2,label='28-day average')\nax.legend(frameon=False)\nfinish_plot(ax,f'Daily purchases peaked at {int(daily.units.max()):,} units on {daily.units.idxmax().date()}',\n 'Units per day with a 28-day moving average',ylabel='Purchased units',save='eda_02_demand.png')\n\ndow=tx.groupby(tx.date.dt.dayofweek).agg(units=('uid','size'),buyers=('uid','nunique'))\ndow.index=['Mon','Tue','Wed','Thu','Fri','Sat','Sun']\nhm_bar(dow['units'],f'{dow.units.idxmax()} is the busiest day of the week',\n 'Units by day of week',horizontal=False,save='eda_03_dayofweek.png')\n","metadata":{"execution":{"iopub.status.busy":"2026-10-16T08:29:29.119154Z","iopub.execute_input":"2026-10-16T08:29:29.119646Z","iopub.status.idle":"2026-10-16T08:29:38.64879Z","shell.execute_reply.started":"2026-10-16T08:29:29.119567Z","shell.execute_reply":"2026-10-16T08:29:38.647639Z"},"trusted":true},"outputs":[],"execution_count":null},{"id":"bb55d09a","cell_type":"code","source":"recent180=tx[tx.date >= tx.date.max()-pd.Timedelta(days=179)].copy()\narticle_groups = articles[['article_id','product_group_name','colour_group_name','product_type_no','colour_group_code']]\nretail=recent180.merge(article_groups,on='article_id',how='left',validate='many_to_one')\nproduct_counts=retail.product_group_name.value_counts()\nhm_bar(product_counts,f'{product_counts.index[0]} makes up {product_counts.iloc[0]/product_counts.sum():.1%} of units in the last 180 days',\n 'Total units by product group',save='eda_04_groups.png')\n\nretail['month']=retail.date.dt.to_period('M').astype(str)\nmonthly_group=retail.groupby(['month','product_group_name']).size().unstack(fill_value=0)\nshare=monthly_group.div(monthly_group.sum(axis=1),axis=0)\nleading=product_counts.head(7).index\nfig,ax=plt.subplots(figsize=(12,5))\n(sns.heatmap(share.reindex(columns=leading)*100,cmap='Reds',annot=True,fmt='.1f',\n linewidths=.4, cbar_kws={'label':'Share of units (%)'}, ax=ax))\nmonthly_swing=(share[leading].max()-share[leading].min()).sort_values(ascending=False)\nfinish_plot(ax,f'The share of {monthly_swing.index[0]} moves by {monthly_swing.iloc[0]*100:.1f} points between months',\n 'Monthly share of units by product group',xlabel='Product group',ylabel='Month',\n save='eda_05_category_mix.png',grid='x')\n","metadata":{"execution":{"iopub.status.busy":"2026-10-16T08:29:38.649963Z","iopub.execute_input":"2026-10-16T08:29:38.650232Z","iopub.status.idle":"2026-10-16T08:29:48.153682Z","shell.execute_reply.started":"2026-10-16T08:29:38.650211Z","shell.execute_reply":"2026-10-16T08:29:48.152722Z"},"trusted":true},"outputs":[],"execution_count":null},{"id":"12a05ffc","cell_type":"code","source":"# demand summary computed from the data\nlead_group=product_counts.index[0]\nlead_share=product_counts.iloc[0]/product_counts.sum()\npeak_day=daily.units.idxmax().date()\npeak_units=int(daily.units.max())\ndisplay(Markdown(\n f'Over the last 180 days, **{lead_group}** is **{lead_share:.1%}** of recorded units, '\n f'and the busiest single day sold **{peak_units:,} units** ({peak_day}). '\n 'This category is where I would check stock and demand forecasts first, '\n 'after confirming what availability data exists.'\n))","metadata":{"execution":{"iopub.status.busy":"2026-10-16T08:29:48.156149Z","iopub.execute_input":"2026-10-16T08:29:48.156457Z","iopub.status.idle":"2026-10-16T08:29:48.165465Z","shell.execute_reply.started":"2026-10-16T08:29:48.156426Z","shell.execute_reply":"2026-10-16T08:29:48.164583Z"},"trusted":true},"outputs":[],"execution_count":null},{"id":"40c73080","cell_type":"markdown","source":"## 3. Customer behavior\n\nHere I compare purchase counts, gaps between shopping days and activity by the first month a customer appears. For the model features I only use history from before the target week.","metadata":{}},{"id":"b78ba802","cell_type":"code","source":"# customer activity over the last 90 days\ncut_end=tx.date.max()+pd.Timedelta(days=1)\nbehavior=tx[tx.date>=cut_end-pd.Timedelta(days=90)]\ncustomer_rfm=behavior.groupby('uid').agg(units=('article_id','size'),\n unique_articles=('article_id','nunique'),\n median_price=('price','median'),last_purchase=('date','max'),\n active_dates=('date','nunique'))\ncustomer_rfm['recency']=(cut_end-customer_rfm.last_purchase).dt.days\ncustomer_rfm['repeat_units']=customer_rfm['units']-customer_rfm['unique_articles']\nfig,ax=plt.subplots(figsize=(11,4.5))\nc=customer_rfm['units'].clip(upper=40)\nax.hist(c,bins=np.arange(1,42),color=HM['soft'],edgecolor='white')\nax.axvline(c.median(),color=HM['dark'],linestyle='--',label=f'Median: {c.median():.0f}')\nax.legend(frameon=False)\nfinish_plot(ax,f'The median active customer bought {customer_rfm.units.median():.0f} units in 90 days',\n 'Values above 40 are pooled into the last bin',\n xlabel='Units purchased (display capped)',ylabel='Customers',save='eda_06_purchase_intensity.png')\n\nfig,ax=plt.subplots(figsize=(11,4.5))\nax.scatter(customer_rfm.recency.sample(min(len(customer_rfm),16000),random_state=SEED),\n customer_rfm.units.sample(min(len(customer_rfm),16000),random_state=SEED).clip(upper=60),\n s=8,alpha=.15,color=HM['primary'])\nfinish_plot(ax,f'Recency vs. frequency for a sample of {min(len(customer_rfm),16000):,} customers',\n 'Days since last purchase against units bought',\n xlabel='Days since last purchase',ylabel='Units purchased (capped)',save='eda_07_recency_frequency.png')\n","metadata":{"execution":{"iopub.status.busy":"2026-10-16T08:29:48.166928Z","iopub.execute_input":"2026-10-16T08:29:48.167305Z","iopub.status.idle":"2026-10-16T08:29:50.959758Z","shell.execute_reply.started":"2026-10-16T08:29:48.167276Z","shell.execute_reply":"2026-10-16T08:29:50.958266Z"},"trusted":true},"outputs":[],"execution_count":null},{"id":"90cd4af0","cell_type":"code","source":"month_tx=tx[['uid','date']].copy()\nmonth_tx['month']=month_tx.date.dt.to_period('M')\nunique_months=month_tx[['uid','month']].drop_duplicates()\nfirst_month=unique_months.groupby('uid',sort=False)['month'].min().rename('cohort')\ncohort=unique_months.join(first_month,on='uid')\ncohort['offset']=(cohort['month'].dt.year-cohort['cohort'].dt.year)*12+(cohort['month'].dt.month-cohort['cohort'].dt.month)\ncohort=cohort[(cohort.cohort.astype(str)>='2019-10')&(cohort.offset<=8)]\nret=cohort.groupby(['cohort','offset']).uid.nunique().unstack(fill_value=0)\nret=ret.div(ret[0],axis=0)*100\nfig,ax=plt.subplots(figsize=(12,5))\nsns.heatmap(ret,ax=ax,cmap='Reds',annot=True,fmt='.1f',linewidths=.4,\n cbar_kws={'label':'Observed monthly return (%)'})\nfinish_plot(ax,'Share of each monthly cohort that comes back to buy',\n 'Cohorts by the first month a customer appears in the data',\n xlabel='Months since first observed activity',ylabel='First observed month',save='eda_08_cohorts.png',grid='x')\ndel month_tx,unique_months,cohort,ret\n","metadata":{"execution":{"iopub.status.busy":"2026-10-16T08:29:50.96132Z","iopub.execute_input":"2026-10-16T08:29:50.96288Z","iopub.status.idle":"2026-10-16T08:31:26.035859Z","shell.execute_reply.started":"2026-10-16T08:29:50.962845Z","shell.execute_reply":"2026-10-16T08:31:26.034587Z"},"trusted":true},"outputs":[],"execution_count":null},{"id":"cde1f1e5","cell_type":"code","source":"# gaps between shopping days, not between transaction rows\nshopping_dates=behavior[['uid','date']].drop_duplicates().sort_values(['uid','date'])\nshopping_dates['gap']=shopping_dates.groupby('uid').date.diff().dt.days\npurchase_gap=shopping_dates.gap.dropna()\nfig,ax=plt.subplots(figsize=(11,4.5))\nax.hist(purchase_gap.clip(upper=45),bins=46,color=HM['soft'],edgecolor='white')\nfinish_plot(ax,f'Returning customers shop again after a median of {purchase_gap.median():.0f} days',\n 'Gaps longer than 45 days are pooled into the last bin',\n xlabel='Days between purchases (capped)',ylabel='Intervals',save='eda_09_cadence.png')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:31:26.037425Z","iopub.execute_input":"2026-10-16T08:31:26.037848Z","iopub.status.idle":"2026-10-16T08:31:27.363128Z","shell.execute_reply.started":"2026-10-16T08:31:26.037811Z","shell.execute_reply":"2026-10-16T08:31:27.36167Z"}},"outputs":[],"execution_count":null},{"id":"95718bed","cell_type":"code","source":"median_units=float(customer_rfm.units.median())\nmedian_gap=float(purchase_gap.median()) if len(purchase_gap) else float('nan')\nfrequent_pct=float((customer_rfm.units>=15).mean())\ndisplay(Markdown(\n f'The median active customer bought **{median_units:.0f} units** in 90 days, and '\n f'**{frequent_pct:.1%}** bought at least 15. Customers who shop more than once '\n f'come back after a median of **{median_gap:.0f} days**. I use these patterns to '\n 'separate customer groups, not to set a promotion frequency without testing.'\n))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:31:27.368037Z","iopub.execute_input":"2026-10-16T08:31:27.36851Z","iopub.status.idle":"2026-10-16T08:31:27.396237Z","shell.execute_reply.started":"2026-10-16T08:31:27.368479Z","shell.execute_reply":"2026-10-16T08:31:27.395077Z"}},"outputs":[],"execution_count":null},{"id":"38bec0bc","cell_type":"markdown","source":"## 4. Products, prices and trends\n\nI want to know how concentrated demand is, which colors sell most, and how fast the popular list changes. These findings guide how I pick candidates. Whether they help MAP@12 is tested separately.","metadata":{}},{"id":"23b63612","cell_type":"code","source":"purchases=tx.groupby('article_id').size().sort_values(ascending=False)\npareto=purchases.cumsum()/purchases.sum()\nfig,ax=plt.subplots(figsize=(11,4.5))\nax.plot(np.arange(1,len(pareto)+1)/len(pareto)*100,pareto.values*100,lw=2,color=HM['primary'])\nax.axhline(80,color=HM['grey'],linestyle='--')\nax.axvline(20,color=HM['grey'],linestyle='--')\nfinish_plot(ax,f'The top 20% of articles account for {pareto.iloc[max(0,math.ceil(len(pareto)*.2)-1)]:.1%} of purchases',\n 'Articles that sold at least once, ranked by units',\n xlabel='Share of observed articles, ranked by demand (%)',ylabel='Cumulative unit share (%)',\n save='eda_10_pareto.png')\n\ncolor_units=retail.groupby('colour_group_name').size().sort_values(ascending=False)\nhm_bar(color_units, f'{color_units.index[0]} is the most-bought color ({color_units.iloc[0]/color_units.sum():.1%} of units)',\n 'Units by color group over the last 180 days',save='eda_11_colors.png')\n\nfig,ax=plt.subplots(figsize=(11,4.5))\nq99=float(retail.price.quantile(.99))\nax.hist(retail.price.clip(upper=q99),bins=60,color=HM['soft'])\nfinish_plot(ax,f'The median recorded price is {retail.price.median():.4f} (dataset units)',\n 'The top 1% of prices is capped so the distribution stays readable',\n xlabel='Recorded price (top 1% capped)',ylabel='Purchased units',save='eda_12_prices.png')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:31:27.397711Z","iopub.execute_input":"2026-10-16T08:31:27.398179Z","iopub.status.idle":"2026-10-16T08:31:30.796987Z","shell.execute_reply.started":"2026-10-16T08:31:27.398134Z","shell.execute_reply":"2026-10-16T08:31:30.795908Z"}},"outputs":[],"execution_count":null},{"id":"e597f866","cell_type":"code","source":"# use shares so differences in monthly volume do not dominate\ncolor_top=retail.colour_group_name.value_counts().head(8).index\nmonthly_color=retail.groupby(['month','colour_group_name']).size().unstack(fill_value=0)\nmonthly_color=monthly_color.div(monthly_color.sum(axis=1),axis=0)*100\nfig,ax=plt.subplots(figsize=(12,5))\nsns.heatmap(monthly_color.reindex(columns=color_top),cmap='Reds',annot=True,fmt='.1f',\n linewidths=.35,ax=ax,cbar_kws={'label':'Share of all purchased units (%)'})\ncolor_swing=(monthly_color[color_top].max()-monthly_color[color_top].min()).sort_values(ascending=False)\nfinish_plot(ax,f'The share of {color_swing.index[0]} moves by {color_swing.iloc[0]:.1f} points between months',\n 'The eight most-bought colors; percentage of all units',\n xlabel='Color',ylabel='Month',save='eda_13_color_mix.png',grid='x')\n\nbychannel=tx.sales_channel_id.value_counts(normalize=True).sort_index().mul(100)\nhm_bar(bychannel,f'Channel {int(bychannel.idxmax())} accounts for {bychannel.max():.1f}% of recorded units',\n 'Channel codes 1 and 2 as in the raw data',\n top=2,horizontal=False,save='eda_14_channel.png')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:31:30.798304Z","iopub.execute_input":"2026-10-16T08:31:30.798777Z","iopub.status.idle":"2026-10-16T08:31:33.8957Z","shell.execute_reply.started":"2026-10-16T08:31:30.798748Z","shell.execute_reply":"2026-10-16T08:31:33.894507Z"}},"outputs":[],"execution_count":null},{"id":"9f98247a","cell_type":"code","source":"w=tx[tx.date>=pd.Timestamp('2020-07-29')].copy()\nw['week']=w.date.dt.to_period('W-TUE').dt.start_time\nweekly=w.groupby(['week','article_id']).size().rename('cnt').reset_index()\nweeks=sorted(weekly.week.unique())\nmovement=[]\nfor prev,cur in zip(weeks[:-1],weeks[1:]):\n a=weekly[weekly.week==prev].nlargest(100,'cnt').article_id\n b=weekly[weekly.week==cur].nlargest(100,'cnt').article_id\n movement.append((pd.Timestamp(cur),len(set(a)&set(b))/100))\nif movement:\n m=pd.DataFrame(movement,columns=['week','top100_overlap'])\n fig,ax=plt.subplots(figsize=(11,4))\n ax.plot(m.week,m.top100_overlap,marker='o',color=HM['primary'],lw=2)\n ax.set_ylim(0,1)\n finish_plot(ax,f'On average {m.top100_overlap.mean():.1%} of the top 100 stay in the top 100 the next week',\n 'Overlap between consecutive weekly top-100 lists',\n ylabel='Top-100 overlap',save='eda_15_product_turnover.png')\ndel w,weekly\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:31:33.896956Z","iopub.execute_input":"2026-10-16T08:31:33.897359Z","iopub.status.idle":"2026-10-16T08:31:35.177904Z","shell.execute_reply.started":"2026-10-16T08:31:33.897312Z","shell.execute_reply":"2026-10-16T08:31:35.175318Z"}},"outputs":[],"execution_count":null},{"id":"aa8c5586","cell_type":"code","source":"top20_share=float(pareto.iloc[max(0,math.ceil(len(pareto)*.2)-1)])\nmean_top_overlap=float(m.top100_overlap.mean()) if movement else float('nan')\ndisplay(Markdown(\n f'The top 20% of articles bring in **{top20_share:.1%}** of units, yet only '\n f'**{mean_top_overlap:.1%}** of the weekly top 100 survives to the next week. '\n 'A popular-items list therefore needs frequent refreshing, and I would check '\n 'that high-demand products are actually in stock before promoting them.'\n))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:31:35.179841Z","iopub.execute_input":"2026-10-16T08:31:35.180892Z","iopub.status.idle":"2026-10-16T08:31:35.192004Z","shell.execute_reply.started":"2026-10-16T08:31:35.180846Z","shell.execute_reply":"2026-10-16T08:31:35.190504Z"}},"outputs":[],"execution_count":null},{"id":"c59bd2b7","cell_type":"markdown","source":"### 4A. Seeing the products behind the numbers\n\nBesides counting units, I show product photos for the best sellers, the dominant colors and the articles that are picking up. The photo path follows the `article_id` mapping.\n\nPhotos are read from `images/{first_three_digits}/{article_id_10_digits}.jpg`. Some articles appear in transactions without a photo; they stay in every number and are only missing from the galleries. If the image folder is not mounted, the galleries are skipped and the notebook keeps running.","metadata":{}},{"id":"e0e1a82b","cell_type":"code","source":"from PIL import Image, ImageOps\nfrom matplotlib.patches import Rectangle\nimport textwrap\n\nIMAGE_ROOT = DATA_DIR / 'images'\nHAS_PRODUCT_IMAGES = IMAGE_ROOT.is_dir()\nIMAGE_IDS = ({int(p.stem) for p in IMAGE_ROOT.glob('*/*.jpg') if p.stem.isdigit()}\n if HAS_PRODUCT_IMAGES else set())\n\n\ndef article_photo(article_id):\n \"\"\"Path of the photo for a 10-digit article ID.\"\"\"\n key = f'{int(article_id):010d}'\n return IMAGE_ROOT / key[:3] / f'{key}.jpg'\n\n\ndef hm_product_gallery(rows, title, subtitle='', label_col='label',\n metric_col='metric', n_cols=4, save=None):\n \"\"\"Draw photos with captions in separate panels.\"\"\"\n if not HAS_PRODUCT_IMAGES:\n print(f'Gallery skipped because the photo folder is missing: {IMAGE_ROOT}')\n return 0\n rows = rows.loc[rows.article_id.isin(IMAGE_IDS)].drop_duplicates('article_id').copy()\n if rows.empty:\n print(f'No matching photos for this gallery: {title}')\n return 0\n\n n = len(rows)\n ncols = min(int(n_cols), n)\n nrows = math.ceil(n / ncols)\n # photo and caption get separate panels\n fig = plt.figure(figsize=(ncols * 3.35, nrows * 4.45 + .8), facecolor='white')\n outer = fig.add_gridspec(nrows, ncols, left=.035, right=.965,\n top=.875, bottom=.025, wspace=.18, hspace=.25)\n fig.text(.035, .975, title, fontsize=16, fontweight='bold',\n ha='left', va='top', color=HM['text'])\n if subtitle:\n fig.text(.035, .936, textwrap.shorten(subtitle, 155, placeholder='…'),\n fontsize=9, ha='left', va='top', color=HM['secondary'])\n\n for pos, (_, item) in enumerate(rows.iterrows()):\n r, c = divmod(pos, ncols)\n inner = outer[r, c].subgridspec(2, 1, height_ratios=[3.30, 1.20], hspace=.025)\n ax_img = fig.add_subplot(inner[0])\n ax_cap = fig.add_subplot(inner[1])\n ax_img.axis('off')\n ax_cap.axis('off')\n try:\n with Image.open(article_photo(item.article_id)) as raw:\n pic = ImageOps.exif_transpose(raw).convert('RGB')\n pic.thumbnail((540, 640))\n ax_img.imshow(pic)\n except (OSError, ValueError):\n ax_img.set_facecolor('#F5F1F2')\n ax_img.text(.5, .5, 'Photo not available', va='center', ha='center',\n transform=ax_img.transAxes, color=HM['secondary'])\n ax_img.add_patch(Rectangle((0, 0), 1, 1, transform=ax_img.transAxes,\n fill=False, edgecolor=HM['grid'], linewidth=.8))\n\n value = item.get(label_col, '')\n label = str(value) if pd.notna(value) else 'Unnamed article'\n label = textwrap.shorten(' '.join(label.split()), width=44, placeholder='…')\n lines = textwrap.wrap(label, width=27, break_long_words=False,\n break_on_hyphens=False)[:2] or ['Unnamed article']\n metric = item.get(metric_col, '')\n metric = str(metric) if pd.notna(metric) else ''\n metric = textwrap.shorten(' '.join(metric.split()), width=43, placeholder='…')\n ax_cap.text(.02, .94, '\\n'.join(lines), fontsize=9.5, fontweight='semibold',\n transform=ax_cap.transAxes, ha='left', va='top', color=HM['text'])\n ax_cap.text(.02, .36, metric, fontsize=8.6, transform=ax_cap.transAxes,\n ha='left', va='top', color=HM['primary'])\n ax_cap.text(.02, .09, f'Article ID {int(item.article_id):010d}',\n fontsize=8, transform=ax_cap.transAxes, ha='left', va='bottom',\n color=HM['secondary'], family='monospace')\n if save:\n fig.savefig(OUTPUT_DIR / save, dpi=185, facecolor='white')\n plt.show()\n plt.close(fig)\n return n\n\n\nprint(f'Image folder: {IMAGE_ROOT}')\nprint(f'Photos found: {len(IMAGE_IDS):,}; articles: {len(articles):,}')\nif not HAS_PRODUCT_IMAGES:\n print('Skipping the galleries because the images/ folder is not available.')\nelse:\n photo_audit = articles[['article_id','product_group_name']].copy()\n photo_audit['has_photo'] = photo_audit.article_id.isin(IMAGE_IDS)\n group_coverage = photo_audit.groupby('product_group_name', dropna=False).agg(\n catalog_articles=('article_id','size'), photo_coverage=('has_photo','mean'))\n display(group_coverage.sort_values('catalog_articles', ascending=False).head(12))\n ten = group_coverage.nlargest(10, 'catalog_articles').sort_values('photo_coverage')\n fig, ax = plt.subplots(figsize=(11,5))\n ax.barh(ten.index.astype(str), ten.photo_coverage*100,\n color=[HM['primary'] if v == ten.photo_coverage.min() else HM['muted']\n for v in ten.photo_coverage])\n ax.set_xlim(0, 105)\n finish_plot(ax, f'{photo_audit.has_photo.mean():.1%} of catalog articles have a photo',\n 'Photo coverage for the ten largest product groups',\n xlabel='Articles with images (%)', save='eda_24_image_coverage.png', grid='x')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:31:35.194069Z","iopub.execute_input":"2026-10-16T08:31:35.194475Z","iopub.status.idle":"2026-10-16T08:31:36.927786Z","shell.execute_reply.started":"2026-10-16T08:31:35.194434Z","shell.execute_reply":"2026-10-16T08:31:36.925288Z"}},"outputs":[],"execution_count":null},{"id":"6e649333","cell_type":"markdown","source":"#### Best-selling products\n\nThe 12 articles with the most units that also have a photo. Units come from transactions, not from how popular a picture is in the catalog.","metadata":{}},{"id":"5921fb1d","cell_type":"code","source":"top_looks = purchases.rename('units').reset_index()\ntop_looks.columns = ['article_id', 'units']\ntop_looks = top_looks.head(250).merge(\n articles[['article_id','prod_name','product_type_name','colour_group_name']],\n on='article_id', how='left', validate='one_to_one'\n)\ntop_looks = top_looks.loc[top_looks.article_id.isin(IMAGE_IDS)].head(12).copy()\ntop_looks['label'] = top_looks['prod_name'].fillna(top_looks['product_type_name']).fillna('Fashion article')\ntop_looks['metric'] = top_looks.units.map(lambda x: f'{int(x):,} units sold')\ncount = hm_product_gallery(top_looks, 'Top 12 best-selling articles with a photo',\n 'Ranked by units across all transactions',\n save='eda_25_top_sellers_gallery.png')\nif count:\n print(f'Showing {count} products; {len(purchases):,} articles have sold at least once.')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:31:36.929419Z","iopub.execute_input":"2026-10-16T08:31:36.929884Z","iopub.status.idle":"2026-10-16T08:31:39.676779Z","shell.execute_reply.started":"2026-10-16T08:31:36.929828Z","shell.execute_reply":"2026-10-16T08:31:39.675797Z"}},"outputs":[],"execution_count":null},{"id":"e1ed7da0","cell_type":"markdown","source":"#### Example products for the main categories and colors\n\nA category name alone can be abstract, so for each major product group and color I show one frequently bought article.","metadata":{}},{"id":"b162cf6a","cell_type":"code","source":"recent_article_units = retail.groupby('article_id').size().rename('units')\nrepresentatives = articles[['article_id','prod_name','product_type_name',\n 'product_group_name','colour_group_name']].join(\n recent_article_units, on='article_id', how='inner'\n)\nrepresentatives = representatives.loc[representatives.article_id.isin(IMAGE_IDS)].copy()\n\ndef representative_board(attribute, popular_labels, n=8):\n subset = representatives.loc[representatives[attribute].isin(popular_labels)]\n if subset.empty:\n return pd.DataFrame(columns=['article_id', 'label', 'metric'])\n rows = (subset.sort_values(['units','article_id'], ascending=[False,True])\n .drop_duplicates(attribute)\n .set_index(attribute)\n .reindex([v for v in popular_labels if v in subset[attribute].values])\n .dropna(subset=['article_id'])\n .head(n).reset_index())\n rows['label'] = rows[attribute].astype(str)\n rows['metric'] = rows.apply(lambda x: f'{int(x.units):,} units · {str(x.product_type_name)[:17]}', axis=1)\n return rows\n\ncat_labels = retail.product_group_name.value_counts().head(12).index.tolist()\ncat_looks = representative_board('product_group_name', cat_labels)\nhm_product_gallery(cat_looks, 'Example products from the busiest product groups',\n 'One article with a photo for each main product group',\n save='eda_26_category_gallery.png')\n\ncolor_labels = retail.colour_group_name.value_counts().head(12).index.tolist()\ncolor_looks = representative_board('colour_group_name', color_labels)\n_ = hm_product_gallery(color_looks, 'What the most-bought colors look like',\n 'One article with a photo for each main color group',\n save='eda_27_color_gallery.png')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:31:39.678184Z","iopub.execute_input":"2026-10-16T08:31:39.679454Z","iopub.status.idle":"2026-10-16T08:31:44.63568Z","shell.execute_reply.started":"2026-10-16T08:31:39.679403Z","shell.execute_reply":"2026-10-16T08:31:44.634328Z"}},"outputs":[],"execution_count":null},{"id":"0f41eeb3","cell_type":"markdown","source":"#### Products that are picking up\n\nI compare units in the last seven days with the weekly average of the 28 days before. This surfaces articles that just started to rise, not only the ones that have always been popular.","metadata":{}},{"id":"d5363c06","cell_type":"code","source":"observed_end = tx.date.max() + pd.Timedelta(days=1)\nlookback_7 = tx.loc[(tx.date >= observed_end - pd.Timedelta(days=7)) &\n (tx.date < observed_end)].groupby('article_id').size().rename('units_7d')\nlookback_prev28 = tx.loc[(tx.date >= observed_end - pd.Timedelta(days=35)) &\n (tx.date < observed_end - pd.Timedelta(days=7))].groupby('article_id').size().rename('units_prev28')\nmomentum = pd.concat([lookback_7, lookback_prev28], axis=1).fillna(0)\nmomentum['previous_weekly_avg'] = momentum.units_prev28 / 4\nmomentum['extra_units'] = momentum.units_7d - momentum.previous_weekly_avg\nmomentum['weekly_ratio'] = (momentum.units_7d + 1) / (momentum.previous_weekly_avg + 1)\nmin_recent_volume = max(10, int(momentum.units_7d.quantile(.80)))\nrising = (momentum.loc[(momentum.units_7d >= min_recent_volume) &\n (momentum.extra_units > 0)]\n .sort_values(['extra_units','units_7d'], ascending=False)\n .reset_index())\nif rising.empty:\n print('No article meets the minimum sales-growth threshold.')\nelse:\n rising = rising.loc[rising.article_id.isin(IMAGE_IDS)].head(12).merge(\n articles[['article_id','prod_name','product_type_name']], on='article_id', how='left')\n rising['label'] = rising.prod_name.fillna(rising.product_type_name).fillna('Fashion article')\n rising['metric'] = rising.apply(lambda x:\n f'+{x.extra_units:,.0f} units/week · {x.weekly_ratio:.1f}× earlier avg', axis=1)\n hm_product_gallery(rising, 'Articles with the biggest jump in purchases last week',\n f'Last 7 days vs. weekly average of the previous 28 days · at least {min_recent_volume:,} units',\n save='eda_28_rising_products_gallery.png')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:31:44.637199Z","iopub.execute_input":"2026-10-16T08:31:44.637687Z","iopub.status.idle":"2026-10-16T08:31:48.150537Z","shell.execute_reply.started":"2026-10-16T08:31:44.637642Z","shell.execute_reply":"2026-10-16T08:31:48.148753Z"}},"outputs":[],"execution_count":null},{"id":"c5f5d792","cell_type":"markdown","source":"#### Examples from each price band\n\nArticles are split into quartiles by median transaction price, and I show a few best sellers from each band. Prices stay in the dataset's own units.","metadata":{}},{"id":"2a2b0285","cell_type":"code","source":"article_median_price = tx.groupby('article_id').price.median().rename('median_price')\nprice_story = (purchases.rename('units').to_frame()\n .join(article_median_price)\n .reset_index()\n .merge(articles[['article_id','prod_name','product_type_name']],\n on='article_id', how='left', validate='one_to_one'))\nif len(price_story) >= 4:\n price_story['price_band'] = pd.qcut(\n price_story.median_price.rank(method='first'), q=4,\n labels=['Q1 · Lower', 'Q2', 'Q3', 'Q4 · Higher'])\n price_story = price_story.loc[price_story.article_id.isin(IMAGE_IDS)].copy()\n price_looks = (price_story.sort_values(['units','article_id'], ascending=[False,True])\n .groupby('price_band', observed=True, sort=True)\n .head(3).sort_values(['price_band','units'], ascending=[True,False])\n .copy())\n if price_looks.empty:\n price_looks['label'] = pd.Series(dtype='string')\n price_looks['metric'] = pd.Series(dtype='string')\n else:\n price_looks['label'] = price_looks.apply(lambda x:\n f'{x.price_band}: {str(x.prod_name) if pd.notna(x.prod_name) else str(x.product_type_name)}',axis=1)\n price_looks['metric'] = price_looks.apply(lambda x:\n f'Median price {x.median_price:.4f} · {int(x.units):,} units',axis=1)\n hm_product_gallery(price_looks, 'Best sellers in each of four price bands',\n 'Three products per price quartile · dataset price units',\n n_cols=4, save='eda_29_price_quartiles_gallery.png')\nelse:\n print('Not enough articles to split into four price bands.')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:31:48.15209Z","iopub.execute_input":"2026-10-16T08:31:48.153241Z","iopub.status.idle":"2026-10-16T08:31:53.711325Z","shell.execute_reply.started":"2026-10-16T08:31:48.153207Z","shell.execute_reply":"2026-10-16T08:31:53.710067Z"}},"outputs":[],"execution_count":null},{"id":"d6f7f192","cell_type":"markdown","source":"The photos help me recognize the products behind categories and rankings, but they are not a feature of the model. I do not publish competition photos in public material; for presentations I only use aggregate charts.","metadata":{}},{"id":"7979f81c","cell_type":"code","source":"# summary of the gallery results\nif HAS_PRODUCT_IMAGES:\n top_name=str(top_looks.iloc[0].get('label','Unnamed')) if len(top_looks) else 'Not available'\n top_units=int(top_looks.iloc[0].units) if len(top_looks) else 0\n mover=str(rising.iloc[0].get('label','Unnamed')) if len(rising) else 'Not available'\n mover_delta=float(rising.iloc[0].extra_units) if len(rising) else 0\n display(Markdown(\n f'Among articles with a photo, **{top_name}** is the best seller with **{top_units:,} units**. '\n f'The fastest riser is **{mover}**, up **{mover_delta:,.0f} units per week** against its '\n 'average over the previous four weeks. I would track best sellers and risers separately, '\n 'so new articles are not always outranked by historical popularity.'\n ))\nelse:\n display(Markdown('The photo folder is not available. The analysis continues from transaction data and article attributes.'))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:31:53.713231Z","iopub.execute_input":"2026-10-16T08:31:53.713689Z","iopub.status.idle":"2026-10-16T08:31:53.724845Z","shell.execute_reply.started":"2026-10-16T08:31:53.713645Z","shell.execute_reply":"2026-10-16T08:31:53.723756Z"}},"outputs":[],"execution_count":null},{"id":"0c83d2ab","cell_type":"markdown","source":"### Customer groups and article age\n\nNext: category mix by age group, price spread across product groups, how varied each customer's purchases are, and how old an article is since its first recorded sale. That age is not an official launch date.","metadata":{}},{"id":"f99ce449","cell_type":"code","source":"retail_age=retail[['uid','product_group_name']].merge(customers[['uid','age_group']],on='uid',how='left')\nage_category=pd.crosstab(retail_age.age_group,retail_age.product_group_name)\ntop_age_groups=age_category.sum().nlargest(7).index\nage_share=age_category.reindex(columns=top_age_groups).div(age_category.sum(axis=1),axis=0)*100\nfig,ax=plt.subplots(figsize=(11,4.8))\nsns.heatmap(age_share,cmap='Reds',annot=True,fmt='.1f',ax=ax,linewidths=.4,\n cbar_kws={'label':'Share of age-group units (%)'})\nfinish_plot(ax,f'The share of {top_age_groups[0]} differs by age group',\n 'Share of units per age group (customers with a known age)',\n xlabel='Product group',ylabel='Age band',save='eda_18_age_category.png',grid='x')\ndel retail_age,age_category,age_share\n\nmajor=retail.product_group_name.value_counts().head(7).index\npr=retail[retail.product_group_name.isin(major)][['product_group_name','price']]\nif len(pr)>70000:pr=pr.sample(70000,random_state=SEED)\nfig,ax=plt.subplots(figsize=(11,5))\nsns.boxplot(data=pr,x='price',y='product_group_name',order=list(major),\n showfliers=False,palette=[HM['primary']]+[HM['muted']]*6,ax=ax)\nfinish_plot(ax,f'{pr.groupby(\"product_group_name\").price.median().idxmax()} has the highest median price',\n 'Distribution of transaction prices by main product group',\n xlabel='Price (native dataset unit)',ylabel='Product group',save='eda_19_group_prices.png',grid='x')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:31:53.72672Z","iopub.execute_input":"2026-10-16T08:31:53.72723Z","iopub.status.idle":"2026-10-16T08:31:59.484693Z","shell.execute_reply.started":"2026-10-16T08:31:53.727189Z","shell.execute_reply":"2026-10-16T08:31:59.48348Z"}},"outputs":[],"execution_count":null},{"id":"a7276e99","cell_type":"code","source":"variety_uids=rng.choice(behavior.uid.unique(),size=min(50000,behavior.uid.nunique()),replace=False)\ndivers=behavior[behavior.uid.isin(variety_uids)][['uid','article_id']].merge(\n articles[['article_id','product_type_no']],on='article_id',how='left')\ncategory_freq=divers.groupby(['uid','product_type_no']).size().rename('n').reset_index()\ncategory_freq['total']=category_freq.groupby('uid').n.transform('sum')\nprob=category_freq.n/category_freq.total\ncategory_freq['contribution']=-prob*np.log(np.maximum(prob,1e-12))\nentropy=category_freq.groupby('uid').agg(entropy=('contribution','sum'),\n categories=('product_type_no','nunique'))\nsummary=entropy.join(customer_rfm[['units','repeat_units']],how='inner')\nfig,ax=plt.subplots(figsize=(11,4.8))\nax.scatter(summary.entropy,summary.units.clip(upper=50),s=7,alpha=.18,color=HM['primary'])\nfinish_plot(ax,f'Category variety for {len(summary):,} sampled customers',\n 'Product-type diversity against units bought in 90 days',\n xlabel='Category entropy (nats)',ylabel='Purchased units (capped at 50)',\n save='eda_20_entropy.png')\n\nfig,ax=plt.subplots(figsize=(11,4.4))\nax.hist(summary.categories.clip(upper=15),bins=np.arange(.5,16.5),color=HM['soft'],edgecolor='white')\nfinish_plot(ax,f'The median customer bought {summary.categories.median():.0f} product types in 90 days',\n 'Distinct product types bought per customer',\n xlabel='Distinct product types (capped)',ylabel='Customers',save='eda_21_diversity.png')\ndel divers,category_freq,summary,entropy\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:31:59.48604Z","iopub.execute_input":"2026-10-16T08:31:59.486663Z","iopub.status.idle":"2026-10-16T08:32:01.251187Z","shell.execute_reply.started":"2026-10-16T08:31:59.486618Z","shell.execute_reply":"2026-10-16T08:32:01.25038Z"}},"outputs":[],"execution_count":null},{"id":"2796261c","cell_type":"code","source":"first_seen=tx.groupby('article_id').date.min()\nage_units=retail[['article_id','date']].join(first_seen.rename('first_seen'),on='article_id')\nage_units['days_since_first_sale']=(age_units.date-age_units.first_seen).dt.days\nfig,ax=plt.subplots(figsize=(11,4.8))\nax.hist(age_units.days_since_first_sale.clip(upper=365),bins=45,color=HM['soft'])\nfinish_plot(ax,f'{(age_units.days_since_first_sale>=180).mean():.1%} of units come from articles first sold 180+ days ago',\n 'Age is measured from the first sale visible in the data',\n xlabel='Days since first observed sale',ylabel='Purchased units',save='eda_22_product_age.png')\ndel age_units\n\nweekly_scope=tx.loc[tx.date>=tx.date.max()-pd.Timedelta(days=89),['article_id','date']].copy()\nweekly_scope['week']=weekly_scope.date.dt.to_period('W-TUE').dt.start_time\ncatalog=weekly_scope.groupby('week').article_id.nunique()\nfig,ax=plt.subplots(figsize=(11,4.6))\nax.plot(catalog.index,catalog.values,marker='o',lw=2,color=HM['primary'])\nfinish_plot(ax,f'Between {catalog.min():,} and {catalog.max():,} distinct articles sell each week',\n 'Distinct articles bought, not catalog stock',\n ylabel='Distinct sold articles',save='eda_23_weekly_catalog.png')\ndel weekly_scope\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:32:01.252328Z","iopub.execute_input":"2026-10-16T08:32:01.252661Z","iopub.status.idle":"2026-10-16T08:32:05.641894Z","shell.execute_reply.started":"2026-10-16T08:32:01.252584Z","shell.execute_reply":"2026-10-16T08:32:05.640776Z"}},"outputs":[],"execution_count":null},{"id":"c5438dc0","cell_type":"markdown","source":"## 5. Product relationships and recommendation readiness\n\nI look at which categories are bought by the same customers. There is no `order_id`, so this reads as shared preference, not as proof that two products were in one basket. Then I check how much of the target week's buying is a repeat purchase or an article not seen before.","metadata":{}},{"id":"efed2870","cell_type":"code","source":"sample_uids=rng.choice(behavior.uid.unique(),size=min(45000,behavior.uid.nunique()),replace=False)\naff=behavior.loc[behavior.uid.isin(sample_uids),['uid','article_id']].merge(\n articles[['article_id','product_group_name']],on='article_id',how='left')\ntop_groups=aff.product_group_name.value_counts().head(8).index\ncust_group=aff.loc[aff.product_group_name.isin(top_groups),['uid','product_group_name']].drop_duplicates()\nmat=pd.crosstab(cust_group.uid,cust_group.product_group_name).astype('uint8')\ncounts=mat.T.dot(mat.astype('int32'))\nbase=mat.sum(axis=0)\nlift=counts.div(base,axis=0).div(base/len(mat),axis=1)\nnp.fill_diagonal(lift.values,np.nan)\nfig,ax=plt.subplots(figsize=(10,7))\nsns.heatmap(lift,annot=True,fmt='.2f',cmap='Reds',ax=ax,vmin=0,\n cbar_kws={'label':'Observed customer co-preference lift'})\nfinish_plot(ax,f'The strongest category pair has a lift of {np.nanmax(lift.to_numpy()):.2f}',\n 'Categories bought by the same customer within 90 days',\n save='eda_16_affinity.png',grid='x')\ndel aff,mat,counts,cust_group\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:32:05.644367Z","iopub.execute_input":"2026-10-16T08:32:05.644802Z","iopub.status.idle":"2026-10-16T08:32:08.436306Z","shell.execute_reply.started":"2026-10-16T08:32:05.64477Z","shell.execute_reply":"2026-10-16T08:32:08.434904Z"}},"outputs":[],"execution_count":null},{"id":"911adaee","cell_type":"code","source":"last_week=tx[(tx.date>=VALID_WEEK)&(tx.date=VALID_WEEK-pd.Timedelta(days=90)]\nbuyer_gt=last_week[['uid','article_id']].drop_duplicates()\nrepeat_hit=buyer_gt.merge(previous_90[['uid','article_id']].drop_duplicates(),\n on=['uid','article_id'],how='inner')\nseen_articles=set(past.article_id.unique())\nprint('Customer–article pairs bought in the target week:',len(buyer_gt))\nprint('Pairs that also appeared in the previous 90 days:',f'{len(repeat_hit)/max(len(buyer_gt),1):.2%}')\nprint('Pairs with an article that had never sold before:',\n f'{(~buyer_gt.article_id.isin(seen_articles)).mean():.2%}')\nprint('Buyers with no earlier transactions:',\n f'{(~buyer_gt.uid.isin(past.uid.unique())).groupby(buyer_gt.uid).all().mean():.2%}',\n '(share of customers)')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:32:08.438523Z","iopub.execute_input":"2026-10-16T08:32:08.43917Z","iopub.status.idle":"2026-10-16T08:32:12.822461Z","shell.execute_reply.started":"2026-10-16T08:32:08.439136Z","shell.execute_reply":"2026-10-16T08:32:12.820895Z"}},"outputs":[],"execution_count":null},{"id":"c4797f9d","cell_type":"code","source":"repeat_share=len(repeat_hit)/max(len(buyer_gt),1)\nunseen_share=float((~buyer_gt.article_id.isin(seen_articles)).mean())\ndisplay(Markdown(\n f'Only **{repeat_share:.2%}** of the customer–article pairs bought in the target week '\n f'were also bought in the previous 90 days, and **{unseen_share:.2%}** involve an article '\n 'with no earlier sale at all. Repurchases alone are not enough, so popular and '\n 'similar items stay in the candidate list. An article that is new to the data is not '\n 'necessarily a new product.'\n))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:32:12.824516Z","iopub.execute_input":"2026-10-16T08:32:12.825199Z","iopub.status.idle":"2026-10-16T08:32:12.917723Z","shell.execute_reply.started":"2026-10-16T08:32:12.82515Z","shell.execute_reply":"2026-10-16T08:32:12.916645Z"}},"outputs":[],"execution_count":null},{"id":"56c6aabd","cell_type":"markdown","source":"## 6. Splitting the data by time\n\nThe calibration week is kept apart from the evaluation week. If the number of boosting rounds were chosen on the evaluation week, the reported score would be too optimistic.\n\n| Purpose | Target week |\n|---|---|\n| Train the calibration model | 26 August and 2 September 2020 |\n| Choose the number of rounds (early stopping) | 9–15 September 2020 |\n| Local evaluation | 16–22 September 2020 |\n| Kaggle submission period | 23–29 September 2020 |\n\nAfter calibration, the local model is trained on the three target weeks before the evaluation week. Every feature uses only transactions older than the week being predicted. The development customers are drawn from people already active before the first cutoff, so this sample does not represent all H&M customers.\n\nMAP@12 is computed only for customers who bought in the target week, as in the competition metric, and differences between models are judged on the same buyers. The bootstrap intervals measure uncertainty between buyers within this one week. They do not replace a test on other weeks.","metadata":{}},{"id":"fd7594c7","cell_type":"code","source":"first_cutoff=TRAIN_WEEKS.min()\neligible=tx[(tx.date>=first_cutoff-pd.Timedelta(days=90))&(tx.date0:\n scores.append(ap_at_12(truth[uid],predictions.get(uid,[])))\n return {'MAP@12':float(np.mean(scores)) if scores else np.nan,\n 'scored_buyers':len(scores)}\n\ndef ground_truth(start, selected_uids=None):\n target=tx.loc[(tx.date>=start)&(tx.date=cutoff-pd.Timedelta(days=90)]\n groups=[]\n for days in (7,28,90):\n x=past90.loc[past90.date>=cutoff-pd.Timedelta(days=days)]\n groups.append(x.groupby('article_id').size().rename(f'pop_{days}'))\n a=pd.concat(groups,axis=1).fillna(0)\n last=past90.groupby('article_id').date.max()\n a['article_recency']=(cutoff-last).dt.days\n a['article_price']=past90.groupby('article_id').price.median()\n a['trend_ratio']=(a['pop_7']+1)/(a['pop_28']/4+1)\n a['article_id']=a.index.astype('int32')\n return a.reset_index(drop=True)\n\ndef neighbor_index(history, cutoff, cap_users=BUILD_NEIGHBORS_MAX_USERS):\n \"\"\"Item co-occurrence from prior 45d; bounded users and last 8 distinct items per user.\"\"\"\n recent=history.loc[history.date>=cutoff-pd.Timedelta(days=45),['uid','article_id','date']]\n users=recent.uid.unique()\n if len(users)>cap_users:\n users=np.random.default_rng(SEED).choice(users,cap_users,replace=False)\n recent=recent[recent.uid.isin(users)]\n recent=recent.sort_values(['uid','date']).drop_duplicates(['uid','article_id'],keep='last')\n recent=recent.groupby('uid',sort=False).tail(8)\n pairs=Counter(); degrees=Counter()\n for _,group in recent.groupby('uid',sort=False):\n items=list(map(int,group.article_id.unique()))\n degrees.update(items)\n for a,b in combinations(sorted(items),2): pairs[(a,b)]+=1\n edges=defaultdict(list)\n for (a,b),common in pairs.items():\n if common<2: continue\n score=common/(math.sqrt(degrees[a]*degrees[b])+1e-9)\n edges[a].append((b,score));edges[b].append((a,score))\n result={a:sorted(lst,key=lambda x:(-x[1],x[0]))[:NEIGHBORS_PER_ITEM] for a,lst in edges.items()}\n return result\n\ndef customer_reorders(history, cutoff, uids):\n past=history.loc[(history.date>=cutoff-pd.Timedelta(days=90)) &\n history.uid.isin(uids),['uid','article_id','date']]\n if past.empty: return {}\n last=past.groupby(['uid','article_id']).date.max().rename('last_date')\n freq=past.groupby(['uid','article_id']).size().rename('count')\n rep=pd.concat([freq,last],axis=1).reset_index()\n rep['days_ago']=(cutoff-rep.last_date).dt.days\n rep['heuristic']=rep['count']/(1+rep['days_ago']/7)\n rep.sort_values(['uid','heuristic','days_ago'],ascending=[True,False,True],inplace=True)\n rep=rep.groupby('uid',sort=False).head(REORDER_N)\n return {int(k):list(map(int,v)) for k,v in rep.groupby('uid').article_id}\n\ndef candidate_frame(uids, history, cutoff, include_similarity=True, neighbors=None,\n pop_n=POPULAR_N, limit=CANDIDATE_LIMIT):\n \"\"\"Bounded union of ranked popularity / reorder / item-similarity candidates.\"\"\"\n art_stats=article_signals(history,cutoff)\n top=art_stats.sort_values(['pop_7','pop_28'],ascending=False).head(pop_n)\n pop=list(map(int,top.article_id))\n reorder=customer_reorders(history,cutoff,uids)\n if neighbors is None: neighbors={}\n all_rows=[]\n for uid in uids:\n uid=int(uid)\n bag={}\n def add(a,src,rank):\n if a not in bag: bag[a]=[0,0,0,0,0,0]\n pos={'pop':0,'rep':1,'sim':2}[src]\n bag[a][pos]=1\n if bag[a][pos+3]==0: bag[a][pos+3]=rank\n for rank,a in enumerate(pop,1): add(a,'pop',rank)\n bought=reorder.get(uid,[])\n for rank,a in enumerate(bought,1): add(a,'rep',rank)\n if include_similarity:\n simscores=defaultdict(float)\n for i,a in enumerate(bought[:5]):\n for b,score in neighbors.get(a,[]):\n if b != a: simscores[b]+=score/(i+1)\n similar=sorted(simscores,key=lambda a:(-simscores[a],a))[:SIMILAR_N]\n for rank,a in enumerate(similar,1): add(a,'sim',rank)\n for a,v in list(bag.items())[:limit]:\n all_rows.append((uid,a,*v))\n cols=['uid','article_id','src_pop','src_rep','src_sim','rank_pop','rank_rep','rank_sim']\n c=pd.DataFrame.from_records(all_rows,columns=cols)\n if c.empty: return c,art_stats,pop,reorder\n for z in cols: c[z]=c[z].astype('int32' if z in ['uid','article_id'] else 'int16')\n assert not c.duplicated(['uid','article_id']).any()\n return c,art_stats,pop,reorder\n\ndef heuristic_lists(cand, pop, stage):\n \"\"\"Deterministic top-12 recipes; no label tuning.\"\"\"\n x=cand.copy()\n if stage=='Popularity':\n return {int(uid):pop[:12] for uid in x.uid.unique()}\n if stage=='Popularity + Repurchase':\n x['priority']=np.where(x.src_rep>0,0,1)\n x['rank_order']=np.where(x.src_rep>0,x.rank_rep,x.rank_pop)\n elif stage=='Popularity + Repurchase + Item2Item':\n x['priority']=np.where(x.src_rep>0,0,np.where(x.src_sim>0,1,2))\n x['rank_order']=np.where(x.src_rep>0,x.rank_rep,np.where(x.src_sim>0,x.rank_sim,x.rank_pop))\n else: raise ValueError(stage)\n x.sort_values(['uid','priority','rank_order','article_id'],inplace=True)\n x=x.groupby('uid',sort=False).head(12)\n result={int(k):list(map(int,v)) for k,v in x.groupby('uid',sort=False).article_id}\n return result\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:32:14.06684Z","iopub.execute_input":"2026-10-16T08:32:14.067202Z","iopub.status.idle":"2026-10-16T08:32:14.11293Z","shell.execute_reply.started":"2026-10-16T08:32:14.067174Z","shell.execute_reply":"2026-10-16T08:32:14.109909Z"}},"outputs":[],"execution_count":null},{"id":"3c459574","cell_type":"markdown","source":"## 8. Preparing candidate features\n\nEach training week has its own history. Customer, article and customer–article features are computed from data available before that week.","metadata":{}},{"id":"01e87e68","cell_type":"code","source":"ARTICLE_ATTR=['article_id','product_type_no','colour_group_code','index_group_no']\nARTICLE_META=articles[ARTICLE_ATTR].copy()\nARTICLE_META=ARTICLE_META.fillna(-1)\n\nBASE_FEATURES=['src_pop','src_rep','src_sim','rank_pop','rank_rep','rank_sim',\n 'pop_7','pop_28','pop_90','article_recency',\n 'u_units90','u_distinct90','u_recency','u_n_days90',\n 'pair_count90','pair_recency','cat_count90',\n 'product_type_no','index_group_no','age']\nPRICE_FEATURES=['u_median_price','article_price','price_distance']\nTREND_FEATURES=['trend_ratio']\nCOLOR_FEATURES=['color_count90','category_entropy90','colour_group_code']\nALL_FEATURES=BASE_FEATURES+PRICE_FEATURES+TREND_FEATURES+COLOR_FEATURES\n\n\ndef add_features(c, history, cutoff, art_stats):\n \"\"\"Snapshot feature engineering. All inputs come exclusively from history.date < cutoff.\"\"\"\n h=history[history.date>=cutoff-pd.Timedelta(days=90)]\n stats=h.groupby('uid').agg(u_units90=('article_id','size'),\n u_distinct90=('article_id','nunique'),u_last=('date','max'),\n u_n_days90=('date','nunique'),u_median_price=('price','median'))\n stats['u_recency']=(cutoff-stats.u_last).dt.days\n stats=stats.drop(columns='u_last').reset_index()\n\n pair=h.groupby(['uid','article_id']).agg(pair_count90=('date','size'),pair_last=('date','max')).reset_index()\n pair['pair_recency']=(cutoff-pair.pair_last).dt.days\n pair.drop(columns='pair_last',inplace=True)\n\n hc=h[['uid','article_id']].merge(ARTICLE_META,on='article_id',how='left')\n cat=hc.groupby(['uid','product_type_no']).size().rename('cat_count90').reset_index()\n color=hc.groupby(['uid','colour_group_code']).size().rename('color_count90').reset_index()\n cat_entropy=cat.merge(stats[['uid','u_units90']],on='uid')\n p=cat_entropy.cat_count90/cat_entropy.u_units90\n cat_entropy['_entropy']=-p*np.log(np.clip(p,1e-10,None))\n ent=cat_entropy.groupby('uid')['_entropy'].sum().rename('category_entropy90').reset_index()\n\n x=c.merge(art_stats,on='article_id',how='left',validate='many_to_one')\n x=x.merge(ARTICLE_META,on='article_id',how='left',validate='many_to_one')\n x=x.merge(stats,on='uid',how='left',validate='many_to_one')\n x=x.merge(pair,on=['uid','article_id'],how='left',validate='many_to_one')\n x=x.merge(cat,on=['uid','product_type_no'],how='left',validate='many_to_one')\n x=x.merge(color,on=['uid','colour_group_code'],how='left',validate='many_to_one')\n x=x.merge(ent,on='uid',how='left',validate='many_to_one')\n x=x.merge(customers[['uid','age']],on='uid',how='left',validate='many_to_one')\n x['price_distance']=np.abs(np.log(np.maximum(x.article_price,1e-6)/\n np.maximum(x.u_median_price,1e-6)))\n x['price_distance']=x.price_distance.replace([np.inf,-np.inf],np.nan)\n for col in ALL_FEATURES:\n x[col]=pd.to_numeric(x[col],errors='coerce').astype('float32')\n return x\n\n\ndef build_fold(cutoff,uids, train=True):\n past=tx[tx.date=VALID_WEEK-pd.Timedelta(days=28)) &\n (tx.date=candidate_recall\nprint(f'Base candidate recall: {candidate_recall:.2%}')\nprint(f'With age-group popular items added: {age_recall:.2%} '\n f'(+{(age_recall-candidate_recall)*100:.2f} percentage points, before re-capping the candidate list)')\ndisplay(Markdown('Adding these items makes the candidate list longer. I have not checked whether '\n 'the gain holds when the number of candidates is kept equal and the ranker is retrained.'))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:56:54.685465Z","iopub.execute_input":"2026-10-16T08:56:54.686144Z","iopub.status.idle":"2026-10-16T08:56:55.508696Z","shell.execute_reply.started":"2026-10-16T08:56:54.686108Z","shell.execute_reply":"2026-10-16T08:56:55.507583Z"}},"outputs":[],"execution_count":null},{"id":"fa0e9573","cell_type":"markdown","source":"## 10. LightGBM LambdaRank\n\nCandidates are grouped by `(target_week, uid)`: one customer's list of articles in one week is one ranking group, so the same customer in two different weeks counts as two groups.\n\n**Number of boosting rounds.** I train on the first two target weeks and use early stopping on the 9–15 September calibration week. The best round found there is the number of rounds used for every later model. I do not set a minimum, because a floor chosen by hand is not backed by any test. The choice is still rough: the calibration week has few positive pairs, so the best round can shift from week to week, and the later models see more data (three weeks, then four). Checking this on several calibration weeks would be the next improvement.","metadata":{}},{"id":"9cfa47a0","cell_type":"code","source":"training = pd.concat([f.assign(target_week=i) for i,f in enumerate(folds)], ignore_index=True)\ntraining.sort_values(['target_week','uid','article_id'], inplace=True, kind='stable')\ntraining.reset_index(drop=True, inplace=True)\nval_features = val_features.sort_values(['uid','article_id']).reset_index(drop=True)\ntrain_groups = training.groupby(['target_week','uid'], sort=False).size().to_numpy()\nassert train_groups.sum() == len(training)\nassert len(train_groups) == training[['target_week','uid']].drop_duplicates().shape[0]\n\nLGB_PARAMS = dict(objective='lambdarank', metric='ndcg', ndcg_eval_at=[12],\n learning_rate=.05, num_leaves=31, max_depth=-1,\n min_data_in_leaf=100, feature_fraction=.9,\n bagging_fraction=.85, bagging_freq=1, verbosity=-1,\n seed=SEED, feature_fraction_seed=SEED,\n bagging_seed=SEED, num_threads=4)\n\n\ndef fit_ranker(train_df, cols, val_df=None, n_estimators=240, early_stopping=False):\n ordered = train_df.sort_values(['target_week','uid','article_id'], kind='stable').reset_index(drop=True)\n group = ordered.groupby(['target_week','uid'], sort=False).size().to_numpy()\n assert int(group.sum()) == len(ordered)\n train_set = lgb.Dataset(ordered[cols], label=ordered.label.astype('int8'),\n group=group, free_raw_data=False)\n valid_sets, callbacks = [], []\n if val_df is not None and early_stopping:\n tmp = val_df.loc[val_df.groupby('uid').label.transform('max').gt(0)].copy()\n tmp = tmp.sort_values(['uid','article_id'], kind='stable')\n valid_set = lgb.Dataset(tmp[cols], label=tmp.label.astype('int8'),\n group=tmp.groupby('uid', sort=False).size().to_numpy(),\n reference=train_set)\n valid_sets = [valid_set]\n callbacks = [lgb.early_stopping(30, verbose=False)]\n return lgb.train(LGB_PARAMS, train_set, num_boost_round=int(n_estimators),\n valid_sets=valid_sets, callbacks=callbacks)\n\n# calibration uses the week just before the evaluation week\ncalibration_train = pd.concat([f.assign(target_week=i) for i,f in enumerate(folds[:-1])],\n ignore_index=True)\ncalibration_val = folds[-1]\ncalibration_model = fit_ranker(calibration_train, BASE_FEATURES,\n val_df=calibration_val, n_estimators=300, early_stopping=True)\n# rounds come from early stopping on the calibration week, with no manual minimum\nN_ROUNDS = int(calibration_model.best_iteration or calibration_model.current_iteration())\nprint(f'Early stopping on the 9–15 Sep calibration week chose {N_ROUNDS} rounds; every later model uses the same number.')\ndel calibration_train, calibration_val, calibration_model\ngc.collect()\n\nranker = fit_ranker(training, BASE_FEATURES, n_estimators=N_ROUNDS)\nval_features['ranker_score'] = ranker.predict(val_features[BASE_FEATURES], num_threads=4)\n\n\ndef score_to_predictions(frame, score_col='ranker_score'):\n x = frame.sort_values(['uid',score_col,'article_id'], ascending=[True,False,True])\n x = x.groupby('uid',sort=False).head(12)\n return {int(u):list(map(int,s)) for u,s in x.groupby('uid',sort=False).article_id}\n\npred_ranker = score_to_predictions(val_features)\nbaseline_results['+ LightGBM LambdaRank'] = evaluate_predictions(pred_ranker,truth_valid)['MAP@12']\nperf = pd.DataFrame({'stage':list(baseline_results.keys()),\n 'MAP@12':list(baseline_results.values())})\nperf['delta_previous'] = perf['MAP@12'].diff()\n\nholdout_uids = np.array(sorted(truth_valid), dtype=np.int32)\nholdout_ap = pd.DataFrame(index=holdout_uids)\nfor stage in base_names:\n preds = heuristic_lists(val_features,val_pop,stage)\n holdout_ap[stage] = [ap_at_12(truth_valid[u],preds.get(u,[])) for u in holdout_uids]\nholdout_ap['+ LightGBM LambdaRank'] = [\n ap_at_12(truth_valid[u], pred_ranker.get(u,[])) for u in holdout_uids]\n\nBOOT_N = 2500\nboot_idx = np.random.default_rng(SEED).integers(0,len(holdout_ap),\n size=(BOOT_N,len(holdout_ap)),dtype=np.int32)\n\ndef paired_ci(values, reference=None, alpha=.05):\n \"\"\"Paired bootstrap over the same buyers.\"\"\"\n values = np.asarray(values, dtype=np.float64)\n if reference is not None:\n values = values - np.asarray(reference, dtype=np.float64)\n sampled = values[boot_idx].mean(axis=1)\n return tuple(np.quantile(sampled, [alpha/2,1-alpha/2]))\n\nperf['ci_low'] = [paired_ci(holdout_ap[st])[0] for st in perf.stage]\nperf['ci_high'] = [paired_ci(holdout_ap[st])[1] for st in perf.stage]\nperf['delta_ci_low'] = np.nan\nperf['delta_ci_high'] = np.nan\nfor i in range(1,len(perf)):\n low,high = paired_ci(holdout_ap[perf.stage.iloc[i]],holdout_ap[perf.stage.iloc[i-1]])\n perf.loc[i,['delta_ci_low','delta_ci_high']] = [low,high]\nassert np.allclose(perf['MAP@12'],holdout_ap.mean().to_numpy())\ndisplay(perf)\nperf.to_csv(OUTPUT_DIR/'internal_model_staircase.csv',index=False)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:56:59.415657Z","iopub.execute_input":"2026-10-16T08:56:59.416008Z","iopub.status.idle":"2026-10-16T08:57:03.61544Z","shell.execute_reply.started":"2026-10-16T08:56:59.415981Z","shell.execute_reply":"2026-10-16T08:57:03.614212Z"}},"outputs":[],"execution_count":null},{"id":"fd1023c2","cell_type":"code","source":"fig,ax=plt.subplots(figsize=(12,5.7))\nx=np.arange(len(perf)); yy=perf['MAP@12'].to_numpy()\ncolors=[HM['grey'],HM['soft'],HM['dark'],HM['primary']]\nax.bar(x,yy,color=colors,width=.65)\nax.errorbar(x, yy,\n yerr=[yy-perf.ci_low.to_numpy(), perf.ci_high.to_numpy()-yy],\n fmt='none',ecolor=HM['text'],capsize=4,lw=1.0)\nfor i,y in enumerate(yy):\n ax.text(i,perf.ci_high.iloc[i]+max(yy)*.035,f'{y:.5f}',ha='center',va='bottom',\n weight='bold',fontsize=10)\nax.set_xticks(x,['Recent\\npopularity','+ Repurchase','+ Item-to-item','+ LambdaRank'])\nax.set_ylim(0,max(perf.ci_high)*1.30 if max(perf.ci_high)>0 else 1)\nfinish_plot(ax,('LambdaRank beats popularity on this evaluation sample' if yy[-1]>yy[0] else 'The model stages show no consistent gain'),\n f'95% bootstrap interval · {len(holdout_uids):,} buyers · 16–22 September 2020',\n ylabel='Internal MAP@12',save='slide_01_staircase.png')\nfor _,row in perf.iloc[1:].iterrows():\n display(Markdown(f'After the {row.stage} stage, MAP@12 changes by **{row.delta_previous:+.5f}** '\n f'(95% CI of the difference [{row.delta_ci_low:+.5f}, {row.delta_ci_high:+.5f}]).'))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:57:08.493258Z","iopub.execute_input":"2026-10-16T08:57:08.49408Z","iopub.status.idle":"2026-10-16T08:57:08.995703Z","shell.execute_reply.started":"2026-10-16T08:57:08.494043Z","shell.execute_reply":"2026-10-16T08:57:08.994882Z"}},"outputs":[],"execution_count":null},{"id":"29ff9df4","cell_type":"markdown","source":"## 11. Do the extra features really help?\n\nI compare the core features with three additions: price, time trend, and color plus category entropy. The core model here is exactly the model from the stage comparison above, with the same training data and the same number of rounds.\n\nI read the result through the confidence interval of the difference. If the interval includes zero, there is no evidence that the feature helps, and I keep the simpler core set.","metadata":{}},{"id":"eec08454","cell_type":"code","source":"variants = {\n 'Core retrieval + behavior': BASE_FEATURES,\n '+ Price': BASE_FEATURES+PRICE_FEATURES,\n '+ Price + temporal trend': BASE_FEATURES+PRICE_FEATURES+TREND_FEATURES,\n '+ Price + trend + color/entropy': ALL_FEATURES\n}\nablation_pred = {'Core retrieval + behavior': pred_ranker}\nif RUN_ABLATION:\n for name,cols in list(variants.items())[1:]:\n print('Training:',name,'|',N_ROUNDS,'rounds, same as the core model')\n m=fit_ranker(training,cols,n_estimators=N_ROUNDS)\n tmp=val_features[['uid','article_id']].copy()\n tmp['score']=m.predict(val_features[cols],num_threads=4)\n ablation_pred[name]=score_to_predictions(tmp,'score')\n del m,tmp\n gc.collect()\n\nablation_rows=[]\nablation_per_buyer=pd.DataFrame(index=holdout_uids)\nfor name,pred in ablation_pred.items():\n ap=np.array([ap_at_12(truth_valid[u],pred.get(u,[])) for u in holdout_uids])\n ablation_per_buyer[name]=ap\n low,high=paired_ci(ap)\n dl,dh=paired_ci(ap,ablation_per_buyer['Core retrieval + behavior'].to_numpy())\n ablation_rows.append((name,ap.mean(),low,high,dl,dh))\nablation=pd.DataFrame(ablation_rows,columns=[\n 'variant','MAP@12','ci_low','ci_high','delta_ci_low','delta_ci_high'])\nablation['delta_core']=ablation['MAP@12']-ablation.iloc[0]['MAP@12']\nablation['delta_previous']=ablation['MAP@12'].diff()\nassert np.isclose(ablation.iloc[0]['MAP@12'],perf.iloc[-1]['MAP@12']), 'The core model must give the same predictions in both evaluations'\ndisplay(ablation)\nablation.to_csv(OUTPUT_DIR/'feature_ablation.csv',index=False)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:57:15.120582Z","iopub.execute_input":"2026-10-16T08:57:15.121941Z","iopub.status.idle":"2026-10-16T08:57:20.809689Z","shell.execute_reply.started":"2026-10-16T08:57:15.121902Z","shell.execute_reply":"2026-10-16T08:57:20.80869Z"}},"outputs":[],"execution_count":null},{"id":"f2d39236","cell_type":"code","source":"if len(ablation)>1:\n fig,ax=plt.subplots(figsize=(11,5))\n view=ablation.iloc[1:].copy()\n xx=np.arange(len(view)); y=view.delta_core.to_numpy()\n lo=view.delta_ci_low.to_numpy(); hi=view.delta_ci_high.to_numpy()\n ax.bar(xx,y,color=[HM['primary'] if z>=0 else HM['soft'] for z in y],width=.60)\n ax.errorbar(xx,y,yerr=[y-lo,hi-y],fmt='none',ecolor=HM['text'],capsize=5)\n ax.axhline(0,color=HM['text'],lw=1)\n ax.set_xticks(xx,['+ Price','+ Price + trend','+ Price + trend +\\ncolor/entropy'])\n ymin=min(0,lo.min()); ymax=max(0,hi.max()); pad=max((ymax-ymin)*.20,.0004)\n ax.set_ylim(ymin-pad,ymax+pad)\n for i,v in enumerate(y):\n ax.annotate(f'{v:+.5f}',(i,max(v,hi[i])),xytext=(0,7),\n textcoords='offset points',ha='center',fontsize=9)\n all_cross=bool(((view.delta_ci_low<=0)&(view.delta_ci_high>=0)).all())\n finish_plot(ax,('The extra features show no convincing MAP@12 gain' if all_cross\n else 'Some extra features move MAP@12; confirm on other weeks'),\n f'Difference from the core model · 95% bootstrap · {len(holdout_uids):,} buyers',\n ylabel='Internal Δ MAP@12 vs core',save='slide_02_ablation.png')\n for _,r in view.iterrows():\n evid = ('The CI includes zero, so there is no evidence of a gain' if r.delta_ci_low<=0<=r.delta_ci_high\n else 'The CI excludes zero in this week; it needs a check on other weeks')\n display(Markdown(f'{r.variant}: MAP@12 changes by **{r.delta_core:+.5f}** '\n f'(95% CI [{r.delta_ci_low:+.5f}, {r.delta_ci_high:+.5f}]). {evid}.'))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:57:25.156436Z","iopub.execute_input":"2026-10-16T08:57:25.156942Z","iopub.status.idle":"2026-10-16T08:57:25.6435Z","shell.execute_reply.started":"2026-10-16T08:57:25.156912Z","shell.execute_reply":"2026-10-16T08:57:25.642556Z"}},"outputs":[],"execution_count":null},{"id":"b0c22bf3","cell_type":"markdown","source":"## 12. Understanding the model and where it fails\n\nI check which features the model uses most, then compare scores by customer shopping frequency. I also count how many articles that were actually bought never made it into the candidate list.","metadata":{}},{"id":"45abb9d8","cell_type":"code","source":"importance=pd.Series(ranker.feature_importance(importance_type='gain'),index=BASE_FEATURES).sort_values(ascending=False)\nhm_bar(importance,f'{importance.index[0]} contributes the most gain',\n 'Feature importance of the core model (gain)',top=15,\n save='eda_17_feature_importance.png')\n\nbefore_val=tx[tx.date=VALID_WEEK-pd.Timedelta(days=90)) &\n before_val.uid.isin(DEV_UIDS)]\nseg=seg_hist.groupby('uid').agg(units=('article_id','size'),last=('date','max'))\nseg['recency']=(VALID_WEEK-seg['last']).dt.days\nseg['segment']=np.select([seg.recency>45,seg.units>=15,seg.units>=5],\n ['Dormant 46–90d','High frequency','Mid frequency'],\n default='Low frequency')\nseg=seg[['segment']]\nrows=[]\nall_pred={key:heuristic_lists(val_features,val_pop,key) for key in base_names}\nall_pred['LightGBM LambdaRank']=pred_ranker\nfor u,gt in truth_valid.items():\n tag=seg.loc[u,'segment'] if u in seg.index else 'Cold-start / no recent purchase'\n row={'uid':u,'segment':tag,'n_true_items':len(gt),'candidate_hits':len(possible.get(u,set())>)}\n for name,pred in all_pred.items(): row[name]=ap_at_12(gt,pred.get(u,[]))\n rows.append(row)\nbuyer_level=pd.DataFrame(rows)\nif len(buyer_level):\n segment_scores=buyer_level.groupby('segment').agg(n_buyers=('uid','size'),\n popularity=('Popularity','mean'),\n ranker=('LightGBM LambdaRank','mean'),\n mean_candidate_hits=('candidate_hits','mean'))\n segment_scores['ranker_delta']=segment_scores.ranker-segment_scores.popularity\n display(segment_scores.sort_values('n_buyers',ascending=False))\n segment_scores.to_csv(OUTPUT_DIR/'segment_scores.csv')\n fig,ax=plt.subplots(figsize=(11,4.8))\n y=segment_scores.sort_values('n_buyers',ascending=False)\n xx=np.arange(len(y));width=.36\n ax.bar(xx-width/2,y.popularity,width,label='Popularity',color=HM['grey'])\n ax.bar(xx+width/2,y.ranker,width,label='LightGBM',color=HM['primary'])\n ax.set_xticks(xx,[f'{n}\\n(n={int(k):,})' for n,k in zip(y.index,y.n_buyers)])\n ax.legend(frameon=False)\n finish_plot(ax,f'The largest ranker gain over popularity is in {segment_scores.ranker_delta.idxmax()}',\n 'Segments defined from history before the evaluation week',\n ylabel='Internal MAP@12',save='slide_03_segments.png')\n\nerror_cases=buyer_level.nsmallest(10,'LightGBM LambdaRank')[[\n 'uid','segment','n_true_items','candidate_hits','LightGBM LambdaRank']]\n# the worst cases stay in error_cases and are not printed as a debug table\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:57:33.143465Z","iopub.execute_input":"2026-10-16T08:57:33.144561Z","iopub.status.idle":"2026-10-16T08:57:36.853667Z","shell.execute_reply.started":"2026-10-16T08:57:33.144523Z","shell.execute_reply":"2026-10-16T08:57:36.852786Z"}},"outputs":[],"execution_count":null},{"id":"2848e8e4","cell_type":"code","source":"segment_ci_rows=[]\nfor label,g in buyer_level.groupby('segment'):\n dif=(g['LightGBM LambdaRank']-g['Popularity']).to_numpy()\n B=3000\n idx=np.random.default_rng(SEED).integers(0,len(dif),size=(B,len(dif)))\n lo,hi=np.quantile(dif[idx].mean(axis=1),[.025,.975])\n segment_ci_rows.append((label,len(dif),dif.mean(),lo,hi))\nsegment_uncertainty=pd.DataFrame(segment_ci_rows,columns=[\n 'segment','n_buyers','ranker_minus_popularity','delta_ci_low','delta_ci_high'])\ndisplay(segment_uncertainty.sort_values('n_buyers',ascending=False))\nsegment_uncertainty.to_csv(OUTPUT_DIR/'segment_uncertainty.csv',index=False)\n\n# this fallback was only thought of after seeing the evaluation\nsegments_by_uid=buyer_level.set_index('uid')['segment']\nif 'Dormant 46–90d' in set(buyer_level.segment):\n policy_scores=np.where(buyer_level.segment.eq('Dormant 46–90d'),\n buyer_level.Popularity,buyer_level['LightGBM LambdaRank'])\n estimated_policy_map=float(np.mean(policy_scores))\n display(Markdown(\n f'I tried serving popularity to dormant customers. The combined MAP@12 is '\n f'**{estimated_policy_map:.5f}**, against **{perf.iloc[-1][\"MAP@12\"]:.5f}** for the ranker on all buyers. '\n 'I thought of this rule after seeing the same week\\'s results, so I do not treat '\n 'it as a validated improvement.'\n ))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:57:42.614895Z","iopub.execute_input":"2026-10-16T08:57:42.615219Z","iopub.status.idle":"2026-10-16T08:57:42.682526Z","shell.execute_reply.started":"2026-10-16T08:57:42.615194Z","shell.execute_reply":"2026-10-16T08:57:42.681764Z"}},"outputs":[],"execution_count":null},{"id":"35dad53d","cell_type":"markdown","source":"## 13. What I would do with these results\n\nThe next cells read the segment results through their confidence intervals. The plan follows from what the intervals support: test personalization where the evidence is strong, treat the other segments as open questions, and improve candidate coverage before adding more features.\n\nReal commercial impact still needs stock, margin and recommendation exposure data.","metadata":{}},{"id":"3dc937c2","cell_type":"code","source":"base=float(perf.iloc[0]['MAP@12'])\ncore=float(perf.iloc[-1]['MAP@12'])\nmult=core/base if base>0 else np.nan\nci_main=paired_ci(holdout_ap['+ LightGBM LambdaRank'],holdout_ap['Popularity'])\nsegments=segment_scores.copy()\nroutine_labels=['Mid frequency','High frequency']\nroutine_n=int(segments.loc[segments.index.isin(routine_labels),'n_buyers'].sum())\nstrong=segment_uncertainty.loc[\n segment_uncertainty.delta_ci_low.gt(0),['segment','n_buyers','ranker_minus_popularity']\n]\ndormant=segments.loc['Dormant 46–90d'] if 'Dormant 46–90d' in segments.index else None\nroutine_evidence=', '.join(str(s) for s in strong.segment if s in routine_labels)\nif not routine_evidence:\n routine_evidence='no regular-buyer segment has a fully positive CI'\n\nbusiness_actions=pd.DataFrame([\n ['1. Test', 'Regular customers',\n f'{routine_n:,} of {len(holdout_uids):,} evaluation buyers are mid/high frequency; positive CI: {routine_evidence}',\n 'Compare LambdaRank with popularity for regular customers in a live test.',\n 'Recommendation impressions, purchases, conversion and basket value'],\n ['2. Learn', 'Customers with limited history',\n (f'Dormant: ranker {dormant.ranker:.5f} vs popularity {dormant.popularity:.5f}'\n if dormant is not None else 'Too few dormant buyers to summarize'),\n 'No fallback rule yet; test one on a different period before adopting it.',\n 'Per-segment results with uncertainty intervals'],\n ['3. Improve', 'Candidate coverage',\n f'Mean candidate recall is {candidate_recall:.1%}',\n 'Try extra candidate sources under the same candidate budget.',\n 'Candidate recall, MAP@12, run time and catalog coverage'],\n ['4. Keep', 'Core features',\n (f'Core {core:.5f}; richest variant {float(ablation.iloc[-1][\"MAP@12\"]):.5f}'\n if len(ablation)>1 else f'Core {core:.5f}; ablation not run'),\n 'Keep the core features until another feature proves useful on a different week.',\n 'Multi-week evaluation and compute cost'],\n ['5. Review', 'Stock and best sellers',\n f'The top 20% of articles bring {top20_share:.1%} of units',\n 'Compare top-demand articles with actual stock data.',\n 'Stock, margin, stock-outs, returns and supply schedule']\n], columns=['priority','topic','finding','next_step','data_needed'])\ndisplay(business_actions)\nbusiness_actions.to_csv(OUTPUT_DIR/'business_recommendations.csv',index=False)\n\nexecutive_summary=(\n f'In the local evaluation, MAP@12 goes from {base:.5f} to {core:.5f} '\n f'({mult:.2f}x; 95% CI of the difference [{ci_main[0]:+.5f}, {ci_main[1]:+.5f}]).\\n'\n f'The clearer evidence is in: {routine_evidence} '\n f'({routine_n:,} of {len(holdout_uids):,} evaluation buyers are mid/high frequency).\\n'\n f'Candidate recall is {candidate_recall:.1%}, so candidate retrieval is the first thing to improve.\\n'\n 'The dormant fallback was chosen after seeing the results and is not validated. '\n 'All scores come from a local sample, not the Kaggle leaderboard.'\n)\nprint(executive_summary)\n(OUTPUT_DIR/'slide_notes.txt').write_text(executive_summary,encoding='utf-8')\n\n# summary slide\nfig,ax=plt.subplots(figsize=(13.33,7.5))\nax.axis('off')\nfig.patch.set_facecolor('white')\nfig.text(.06,.88,'PERSONALIZED RECOMMENDATIONS VS. POPULARITY',\n fontsize=21,fontweight='bold',color=HM['text'])\nfig.text(.06,.81,'What the local H&M evaluation shows',fontsize=12,color=HM['secondary'])\nfig.text(.06,.67,f'{mult:.2f}×',fontsize=46,color=HM['primary'],fontweight='bold')\nfig.text(.31,.70,'MAP@12 of LambdaRank compared with popularity',fontsize=13,color=HM['text'])\nfig.text(.31,.64,f'{base:.5f} → {core:.5f} | 95% CI of the difference [{ci_main[0]:+.5f}, {ci_main[1]:+.5f}]',\n fontsize=10,color=HM['secondary'])\nfig.text(.06,.48,f'{candidate_recall:.1%}',fontsize=33,fontweight='bold',color=HM['dark'])\nfig.text(.27,.49,'Average candidate recall per buyer',fontsize=12,color=HM['text'])\nfig.text(.06,.34,f'{routine_n:,}',fontsize=28,fontweight='bold',color=HM['primary'])\nfig.text(.27,.35,'Evaluation buyers in mid/high-frequency segments',fontsize=12,color=HM['text'])\nfig.text(.06,.20,'NEXT STEPS',fontsize=11,fontweight='bold',color=HM['primary'])\nfig.text(.06,.16,'Test on regular customers; widen candidate retrieval; measure conversion in an experiment',\n fontsize=11,color=HM['text'])\nfig.text(.06,.07,'Local MAP@12, 16–22 Sep 2020. Sample of active buyers; not a Kaggle score or a sales impact.',\n fontsize=9,color=HM['secondary'])\nfig.savefig(OUTPUT_DIR/'slide_00_executive_summary.png',dpi=200,facecolor='white')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:57:47.893146Z","iopub.execute_input":"2026-10-16T08:57:47.893474Z","iopub.status.idle":"2026-10-16T08:57:48.317733Z","shell.execute_reply.started":"2026-10-16T08:57:47.893441Z","shell.execute_reply":"2026-10-16T08:57:48.316587Z"}},"outputs":[],"execution_count":null},{"id":"260e5dca","cell_type":"markdown","source":"### Business impact simulation\n\nThe competition data has no recommendation impressions, conversions or margins. So I run three simple scenarios for 100,000 customers who are assumed to see recommendations: a purchase-probability lift of 0.05, 0.10 or 0.20 percentage points means 50, 100 or 200 extra buyers. This is pure assumption; MAP@12 cannot be converted directly into conversion.","metadata":{}},{"id":"ff237318","cell_type":"code","source":"assumed_exposures=100_000 # assumed number of exposed customers\nscenario_pp=[.0005,.001,.002] # assumed lift in purchase probability\nsensitivity=pd.DataFrame({\n 'assumed_incremental_purchase_probability_pp':[x*100 for x in scenario_pp],\n 'assumed_eligible_exposed_customers':[assumed_exposures]*len(scenario_pp),\n 'illustrative_incremental_purchasers':[round(assumed_exposures*x) for x in scenario_pp]\n})\ndisplay(sensitivity)\nsensitivity.to_csv(OUTPUT_DIR/'hypothetical_pilot_scenarios.csv',index=False)\n\ndisplay(Markdown('I cannot put a money value on this from the competition data. I would need '\n 'the number of customers who see recommendations, a measured conversion lift, '\n 'basket value, margin, stock and returns.'))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:57:56.930849Z","iopub.execute_input":"2026-10-16T08:57:56.931209Z","iopub.status.idle":"2026-10-16T08:57:56.947542Z","shell.execute_reply.started":"2026-10-16T08:57:56.931183Z","shell.execute_reply":"2026-10-16T08:57:56.94625Z"}},"outputs":[],"execution_count":null},{"id":"5937371f","cell_type":"markdown","source":"### How I would test the impact in a real business\n\nI would randomly split eligible customers into two groups, one shown popularity-based recommendations and one shown LightGBM recommendations, and compare the share of customers who buy in the same observation period.\n\nBesides conversion I would track units, purchase value where available, article coverage, stock and system response time. I would also read results by segment, especially mid and high frequency. The test length and sample size depend on a real baseline conversion rate; the example below uses an assumed one.","metadata":{}},{"id":"148e8607","cell_type":"code","source":"from scipy.stats import norm\n\ndef approximate_group_size(p0, absolute_mde, power=.80, alpha=.05):\n \"\"\"Approximate sample size per group for a two-group A/B test.\"\"\"\n p1=p0+absolute_mde\n if not(0=TEST_WEEK-pd.Timedelta(days=90)]\nlast90_index=last90.set_index('uid',drop=False).sort_index()\nusers_with_history=set(last90.uid.unique())\nsample_order=pd.read_csv(DATA_DIR/'sample_submission.csv',usecols=['customer_id'],dtype={'customer_id':'string'})\nuser_id_list=pd.Index(customers.customer_id.astype(str)).get_indexer(sample_order.customer_id.astype(str))\nassert (user_id_list>=0).all()\nsubmission_part=OUTPUT_DIR/'submission_stream.csv'\nwith open(submission_part,'w',encoding='utf-8') as fh:\n fh.write('customer_id,prediction\\n')\n for start in range(0,len(user_id_list),FINAL_CUSTOMER_BATCH):\n batch=user_id_list[start:start+FINAL_CUSTOMER_BATCH]\n present=[int(u) for u in batch if int(u) in users_with_history]\n h_batch=last90_index.loc[present].reset_index(drop=True) if present else last90.iloc[:0].copy()\n def batch_candidates(batch_uids, h_custom, pop, neigh):\n reps=customer_reorders(h_custom,TEST_WEEK,batch_uids)\n rows=[]\n for u in batch_uids:\n u=int(u);bag={}\n def add(a,src,pos):\n if a not in bag: bag[a]=[0]*6\n j={'pop':0,'rep':1,'sim':2}[src]\n bag[a][j]=1\n if bag[a][j+3]==0:bag[a][j+3]=pos\n for r,a in enumerate(pop,1):add(a,'pop',r)\n rb=reps.get(u,[])\n for r,a in enumerate(rb,1):add(a,'rep',r)\n scores=defaultdict(float)\n for j,a in enumerate(rb[:5]):\n for b,v in neigh.get(a,[]):\n if b!=a:scores[b]+=v/(j+1)\n for r,a in enumerate(sorted(scores,key=lambda z:(-scores[z],z))[:SIMILAR_N],1):\n add(a,'sim',r)\n rows.extend([(u,a,*v) for a,v in list(bag.items())[:CANDIDATE_LIMIT]])\n return pd.DataFrame.from_records(rows,columns=[\n 'uid','article_id','src_pop','src_rep','src_sim',\n 'rank_pop','rank_rep','rank_sim'])\n cb=batch_candidates(batch,h_batch,final_pop,final_neighbors)\n fb=add_features(cb,h_batch,TEST_WEEK,final_articles)\n fb['score']=final_ranker.predict(fb[chosen],num_threads=4)\n rank=score_to_predictions(fb,'score')\n for u in batch:\n articles12=list(dict.fromkeys(rank.get(int(u),[])+final_pop))[:12]\n assert len(articles12)==12\n ids=' '.join(f'{int(a):010d}' for a in articles12)\n fh.write(f'{customers.iloc[int(u)].customer_id},{ids}\\n')\n if start%(FINAL_CUSTOMER_BATCH*20)==0:\n print(f'Processed {min(start+FINAL_CUSTOMER_BATCH,len(user_id_list)):,}/{len(user_id_list):,} customers')\n del cb,fb,rank,h_batch\n gc.collect()\nprint('Submission file saved to:',submission_part)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T08:58:15.346299Z","iopub.execute_input":"2026-10-16T08:58:15.346651Z","iopub.status.idle":"2026-10-16T09:08:53.18067Z","shell.execute_reply.started":"2026-10-16T08:58:15.346624Z","shell.execute_reply":"2026-10-16T09:08:53.179536Z"}},"outputs":[],"execution_count":null},{"id":"5c67bdbb","cell_type":"markdown","source":"### Checking the submission file\n\nI confirm that the customer order matches Kaggle's sample file and that every row holds 12 unique, 10-digit article IDs. This only proves the file format is right. It says nothing about the competition score.","metadata":{}},{"id":"79c5da93","cell_type":"code","source":"sample=pd.read_csv(DATA_DIR/'sample_submission.csv',dtype={'customer_id':'string','prediction':'string'})\nsubmitted=pd.read_csv(submission_part,dtype={'customer_id':'string','prediction':'string'})\nassert len(submitted)==len(sample)\nassert submitted.customer_id.is_unique\nassert submitted.customer_id.equals(sample.customer_id), 'Sample customer order differs; reindex before submission'\nassert submitted.prediction.notna().all()\nassert submitted.prediction.map(lambda s:len(s.split())==12 and\n len(set(s.split()))==12 and all(len(z)==10 and z.isdigit() for z in s.split())).all()\nprint('Submission format is valid: customer order matches and every row has 12 unique articles.')\nprint('First three rows, shown without customer hashes.'); display(submitted.prediction.head(3).to_frame('sample_12_articles'))\nprint('File to upload to Kaggle:', submission_part)\n\n# files saved for slides\nprint('Saved files:')\nfor p in sorted(OUTPUT_DIR.glob('slide_*.png')):print(' -',p)\nprint(' -',OUTPUT_DIR/'internal_model_staircase.csv')\nprint(' -',OUTPUT_DIR/'slide_notes.txt')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T09:08:53.182643Z","iopub.execute_input":"2026-10-16T09:08:53.183062Z","iopub.status.idle":"2026-10-16T09:09:07.837857Z","shell.execute_reply.started":"2026-10-16T09:08:53.183035Z","shell.execute_reply":"2026-10-16T09:09:07.837018Z"}},"outputs":[],"execution_count":null},{"id":"d8574fb5","cell_type":"markdown","source":"---\n## Final summary\n\nThe last section collects the evaluation scores, candidate coverage, customer segments and the feature test, and notes what still has to be tested before the recommendation is worth deploying.\n\nThe evaluation covers one week and a sample of customers. Without recommendation exposure, stock, margin or return data I cannot measure revenue impact. The Kaggle score for this run can be added in the constants at the top of the last code cell once the new submission file has been uploaded and scored.","metadata":{}},{"id":"96e47a87","cell_type":"code","source":"# final summary built from this run\n# After uploading the new submission file to Kaggle, fill in the scores (leave None until then).\nKAGGLE_PRIVATE=None\nKAGGLE_PUBLIC=None\n\nci_delta=paired_ci(holdout_ap['+ LightGBM LambdaRank'],holdout_ap['Popularity'])\nstrong_segments=segment_uncertainty.loc[\n segment_uncertainty.delta_ci_low.gt(0),'segment'\n].tolist()\nstrong_text=', '.join(str(s) for s in strong_segments) if strong_segments else 'no segment with a fully positive CI'\nuncertain_segments=segment_uncertainty.loc[\n (segment_uncertainty.delta_ci_low<=0) & (segment_uncertainty.delta_ci_high>=0),'segment'\n].tolist()\nuncertain_text=', '.join(str(s) for s in uncertain_segments) if uncertain_segments else 'none in this sample'\nall_ci_cross=(len(ablation)<=1 or\n ((ablation.iloc[1:].delta_ci_low<=0) & (ablation.iloc[1:].delta_ci_high>=0)).all())\nfeature_note=('The CI of the difference includes zero for every extra feature set, so I keep the core features until a test on other weeks shows a benefit.'\n if all_ci_cross else\n 'At least one extra feature set moves MAP@12 with a CI that excludes zero; I would confirm it on other weeks before adopting it.')\nif KAGGLE_PRIVATE is not None and KAGGLE_PUBLIC is not None:\n kaggle_note=(f'Kaggle (late submission): private {KAGGLE_PRIVATE:.5f}, public {KAGGLE_PUBLIC:.5f}. '\n 'This is not comparable with the local MAP@12 above, which uses a sample of active buyers and one week.')\nelse:\n kaggle_note='I have not recorded a Kaggle score for this run yet, and the local MAP@12 is not comparable with the leaderboard.'\nsummary_md=f\"\"\"# What I found\n\nI compared popularity with LightGBM LambdaRank on the same buyers in the week of\n16–22 September 2020. MAP@12 is **{perf.iloc[0]['MAP@12']:.5f}** for popularity and\n**{perf.iloc[-1]['MAP@12']:.5f}** for LambdaRank, a difference with a 95% CI of\n[{ci_delta[0]:+.5f}, {ci_delta[1]:+.5f}]. The ranker uses {N_ROUNDS} boosting rounds, the number picked by\nearly stopping on the calibration week.\n\nAcross the **{len(truth_valid):,} evaluation buyers**, mean candidate recall is only\n**{candidate_recall:.2%}**, so there is a lot of room to improve how products are found\nbefore the model ranks them.\n\nThe clearer gain is in: {strong_text}. Results for {uncertain_text} are not strong\nenough to turn into a separate recommendation rule, and the dormant fallback was chosen after\nseeing this week's results, so I do not treat it as validated.\n\nI tested price, time-trend and color/entropy features. {feature_note}\n\nIn the target week, **{repeat_share:.2%}** of customer–article pairs were repeats from the\nprevious 90 days and **{unseen_share:.2%}** involved an article with no earlier sale. This is why\nrepurchases alone are not enough as a candidate source.\n\nNext steps: widen the candidates under the same candidate budget, evaluate more weeks, and\nmeasure the effect on conversion with an A/B test. I cannot estimate revenue because the data has no\nrecommendation exposure, margin or stock.\n\n*Note:* these results come from a sample of customers in one evaluation week. {kaggle_note}\n\"\"\"\n(OUTPUT_DIR/'executive_takeaways.md').write_text(summary_md,encoding='utf-8')\ndisplay(Markdown(summary_md))\nprint('Summary and result tables saved to:',OUTPUT_DIR)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-10-16T09:09:07.842527Z","iopub.execute_input":"2026-10-16T09:09:07.842889Z","iopub.status.idle":"2026-10-16T09:09:07.879071Z","shell.execute_reply.started":"2026-10-16T09:09:07.842864Z","shell.execute_reply":"2026-10-16T09:09:07.877933Z"}},"outputs":[],"execution_count":null}]}