# 46th place solution

46th place solution (Item2Vec/Transformer)
Introduction
Thank you to H & M, kaggle, and my rivals for the exciting competition.
This post is a brief summary of my solution.
Since it is English written by google translate, I think that there are some parts that are not connected, but if you have any questions, please feel free to ask.
Overview
This time, I designed the model using the 2-tower model using NN.
The 2-tower model provides high performance recommendations by using two parts: the retrieval part, which narrows down candidates with a lightweight model, and the ranking part, which makes predictions with a highly accurate model.
Specifically, the flow of prediction / learning is as follows.
1st stage (retrieval part) Public: 0.02137 Private: 0.02296
Item2Vec is used to narrow down the ranking of the number of articles appearing around the predicted date. (1)
Extract items with a history in the past that have an interaction with another user most recently on a rule basis. (2)
2nd stage (ranking part) Public: 0.2779 Private: 0.02846
Rerank (1) and (2) using Transformer.
Solution
The model will be explained in the following three parts.
Embedding part common to retrieval / ranking
Retrieval part
ranking part
Embedding part
In the Embedding part, the pre-calculated image / natural language model embedding and the category embedding initialized with word2vec of gensim are mixed with
[DCNV2](https://arxiv.org/abs/2008.13535)
.
Each embedding was obtained as follows.
image
swin_large_patch4_window12_384_in22k + TruncatedSVD (1024 dim)
EfficientNetV2L + TruncatedSVD (1024 dim)
Natural language
ELECTRA LARGE (Enter by concatenating detail_desc and category name)
Sentence-T5 st5-11b (Enter by concatenating detail_desc and category name)
BERT Tokenizer + Tf-Idf + PCA (128 dim)
Category
Word2vec by gensim
Table data
price
Total quantity (same products are combined into one record for each day)
How many interactions for that user
Price statistics when compared to other days
Since these features were introduced before the implementation of other methods with greater effects, we have not verified the individual effects, but I feel that they were not very effective.
Retrieval part
In the Retrieval part, I narrowed down by the search vector that took the weighted average of the embedding of the article and the cosine similarity of the embedding of the product.
Model
Target embedding
I put the embedding of the article into several fully connected layers and converted it.
Query embedding
I used a weighted average weighted by time etc. for the embedding of the article.
As for how to add weight, the weight was calculated by
(1/2) ^ relu (edge_feature)
with the feature such as elapsed time as edge_feature, and the weighted average was calculated.
I also considered Transformer using self attention, but it was meaningless in this part.
Loss
We calculated
[SupervisedContrastiveLoss](https://arxiv.org/abs/2004.11362)
with 1 as the one with interaction and 0 as the negative example without it.
At the same time, I used
[AdaCos](https://arxiv.org/abs/1905.00292)
. The margin parameter is 0.
When the margin is greater than 0, all cosine similarity diverges negatively.
Also, I was monitoring the cosine similarity of the regular case, but in the end it was about 0.3.
In terms of ranking, the correct example out of 4096 samples was about 600th, and I felt that this task was difficult.
Subsampling
Batch segmented users by age 10 years, ensured that targets of the same age and same day were in the batch, and trained in chronological order.
During training, I sampled up to 4096 articles from the top 10,000 appearances of articles in the last week of the group segmented by 10 years old.
Sampling used
[Gumbel-max trick](https://stackoverflow.com/questions/43310075/sampling-without-replacement-from-a-given-non-uniform-distribution-in-tensorflow)
.
At the time of inference, we calculated the cosine similarity for 10,000 items that appeared frequently in the last 90 days, which had somebody's purchase history in the last 7 days. (I also tried an approximate neighborhood search like ScaNN, but the time impact was negligible)
Ranking part
In the ranking, rank learning was performed using Transformer.
Model
We further used Transformer with the recommended symmetric article as Query and the user's browsing history as Key & Value. Again, there was no need for a self-attention type Encoder or a Decoder with more than one layer.
Embedding obtained by this and the result of the intermediate layer calculated so far were put into DCNV2 and binary classification was performed.
Loss
loss is based on
[ApproxNDCGLoss](https://www.tensorflow.org/ranking/api_docs/python/tfr/keras/losses/ApproxNDCGLoss)
and uses a unique loss with a penalty for each rank. ..
We compared ApproxNDCGLoss (original), rank difference ** 1, and rank difference ** 2, and the rank difference ** 1 was the best result.
Subsampling
At the time of training, a maximum of 128 negative cases were created by Gumbel-max trick with the ratio of the cube of the cosine similarity as the probability. This gave good feedback to the model as it could take a reasonably weaker negative example than taking the top as it is.
At the time of inference, we predicted the top 128 of the Retrieveval part and the past browsing history.
Machine & Experiments
I used a core i9-12900 CPU, 64GB RAM, and an RTX 3090 GPU.
Of that, CPU and RAM were used about half, learning using all data was 1 epoch 9 hours, and the highest score was 2 epoch.
As an aside, the experiment number on the notebook was 159 at the end, and I feel that I spent a lot of effort.
I will update it from time to time. Feel free to ask questions!
UPDATE
I published a more detailed solution on the technical blog of my company. The explanation for those who did not participate in the competition and the part that I could not write in the discussion due to my lack of English ability are written. If you can read Japanese, please read it.
[H&M Personalized Fashion Recommendations 参加記 (46th/2952)](https://future-architect.github.io/articles/20220602b/)
[image of solution overview](https://future-architect.github.io/images/20220602b/H&M_46th_solution_overview.drawio.png)