# 13th place solution

Before going on to our solution, I would like to thank Kaggle staff and H&M team, alongside with all competitors on the leaderboard for competing with us in this competition. And of course, special thanks to
[@A.Sato](https://www.kaggle.com/A.Sato)
,
[@taksai](https://www.kaggle.com/taksai)
, and
[@tomo20180402](https://www.kaggle.com/tomo20180402)
who are my teammates, for their amazing effort. We are honored to win the gold medal!
Overall Strategy
The overall strategy of our solution is based on four steps.
Candidate Selection
Feature Engineering
Ranking
Ensembling
Key Points
We trained three LightGBM ranker models with different candidate selection methodologies, and finally used an ensemble technique to boost our score.
The key idea of our solution was utilizing item2vec embeddings, which is an algorithm inspired by word2vec.
Article embeddings were retrieved using item2vec embeddings as they are, and customer embeddings were created by taking an average among all dimensions of all past purchased articles for each customer.
Step1: Candidate(Article) Selection
In the candidate selection phase, we tried to sample positive and negative samples for each customer. The candidate selection methods were different for each of the three models, and the diversity of candidates between our models leveraged our scores after ensembling.
Model1
The ensemble method for Model1 was a concatenation of 4 strategies. Past 14 weeks were used to train the model.
recent 50 purchased articles for each customer
top 20 products with the same product code of each article in 1.
last week top 100 articles of all customers for each sales_channel_id group
last week top 30 of all customers for each [sales_channel_id, index_group_no] group
Model2
Aside from recently purchased items (and items with the same product code of recently purchased items), my main model solely relies on the local popularity among the nearest 10,000 customers. 1.I obtained user's latent expression by getting the mean of the item2vec embedded expression (32 dimentions) of the items that the customer purchased in the past, 2.then selected 10,000 nearest customers for EACH customer based on the cosine similarity of user latent expression, 3.then picked top 300 last week popular items among 10,000 nearest customers as candidates.
As for single model, this approach got the best score among three models(Private LB 330-ish).
Model3
Firstly, I created 4 types of clusters for each customer based on the techniques bellow.
(Article-based embeddings were averaged by each dimension of past bought articles to create customer embeddings)
Swin-Transformer Image embeddings
Age of each customer
item2vec based article embeddings
postal_code2vec (inspired by item2vec) embeddings
After the clustering process, each customer will belong to 4 different clusters.
For each cluster, I created an article ranking for the past 30 days, and made a weighted overall ranking depending on which clusters each customer belonged to.
Recent bought article_ids and product_codes were weighted heavier according to the customer.
From the overall ranking, Model3 took the top 350 articles per user.
Step2: Feature Engineering
For each of the candidates, we created features as bellow.
deep learning based latent expressions
Articles embedded expresion by BERT4REC
Articles embedded expresion by LightGCN
Detailed description's latent expression by Roberta (plus PCA for dimensionality reduction)
Swin-Transformer based image features
cosine similarity of the article and the customer based on swin-transformer embeddings and item2vec embeddings
number of past purchases/interval_days by the user for the candidate article_id, product_code, etc
number of past purchases/interval_days by the cluster(clustered using item2vec) of the relevant user for the candidate article_id, product_code, etc
article sales forecasts (forecasted by LightGBM regression model) for the corresponding week
Step3: Model
All three models were trained by LightGBMRaker.
Model1 was trained with 14 weeks,and model2/3 were only trained by using candidates from the last 1 week.
Step4: Ensembling
Ensembling was simply based by using the methodology used in many of the public notebooks (with a slight change).
def cust_blend(dt, W = [1,1,1], base=3): # base option is added

    #Create a list of all model predictions
    REC = []
    REC.append(dt['model1'].split())
    REC.append(dt['model2'].split())
    REC.append(dt['model3'].split())


    res = {}
    for M in range(len(REC)):
        if type(REC[M]) == list:
            for n, v in enumerate(REC[M]):
                if v in res:
                    res[v] += (W[M]/(n+base))
                else:
                    res[v] = (W[M]/(n+base))

    res = list(dict(sorted(res.items(), key=lambda item: -item[1])).keys())

    return ' '.join(res[:12])
Failures
All neural network driven approaches discussed in the academic paper failed for us. Recbole and recommenders gave us marginal boost in this competition.
Increasing training weeks worsened the local scores for model2 and 3.
catboost (It's score was worse compared to LightGBM)
general MLP