{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# State of Content Recommendation Engines in 2023\n\nRecommendation engines are among the most hotly discussed machine learning topics in the research and business community. The early seeds for research in this field were laid as part of the Netflix content recommendation challenge in 2007. While a large number of efforts have been focused on beating contest datasets, several practical problems remain.\n\nThis report is written to address both the practical aspects of building recommendation systems and the aspect that involves performing the best in dataset-based contests. \n  \n# How do you evaluate the performance of recommendation engines?\n\nFor any machine learning problem, it is critical to identify the statistical metric the model is optimizing for. The two key metrics recommendation engine model performances are evaluated are normalized discounted cumulative gain and recall.\n\n### Normalized Discounted Cumulative Gain(nDCG)\n\n\nThe relevancy of an item is directly correlated to the usefulness of said item. Thus, relevancy is very important when we are analyzing recommendation systems.\n\n**Cumulative Gain (CG) is the sum of all the relevance scores within a recommendation set.\n\n\n\n$$CG_{p} = \\sum \\limits _{i=1} ^{p} rel_{i} \\qquad (1) $$                           \n\n**Discounted Cumulative Gain(DCG) improves upon CG by accounting for the position of each score in the recommendation set. Position is a factor in the quality of recommendation sets as it is ideal to have the more relevant documents recommended first. DCG is calculated by the summation of an adjusted relevance score (each relevance score is divided by the logarithm of its position to produce the adjusted relevance score) [1].\n\n\n$$DCG_{p}= \\sum \\limits _{i=1} ^{p} \\frac{rel_{i}}{log_{2}(i + 1)}   =  rel_{1} + \\sum \\limits _{i=2} ^{p} \\frac{rel_{i}}{log_{2}(i + 1)} \\qquad (2) $$           \n\n\n\n$$DCG_{p} = \\sum \\limits _{i=1} ^{p} \\frac{2^{rel_{i}}-1}{log_{2}(i+1)} \\qquad (3) $$\n\nEq. 3 is an alternative computation that prioritizes the position of higher rated scores meaning that If the document with more relevance is rated lower, Eq. 3 penalizes far more severely than Eq.2 does.\n\n\n**Normalized Discounted Cumulative Gain (nDCG) even further improves upon DCG by adding normalization. DCG is insufficient because of the variable quantity of recommendations given to each user based on many parameters. This variation will also lead to DCG varying and not be useful in comparisons across recommendation sets with varying length.\n\nnDCG is the product of the ratio between the DCG of a recommendation set and the Idealized discounted cumulative gain (IDCG) . IDCG is simply the DCG of the adjusted recommendation set where items are ordered by decreasing relevance[1].\n\n$$IDCG_{p} = \\sum \\limits _{i=1} ^{|REL_{p}|} \\frac{rel_{i}}{log_{2}(i+1)}      \\qquad (4) $$\n\n\n\n$$nDCG_{p} = \\frac{DCG_{p}}{IDCG_{p}} \\qquad (5) $$ \n\n\nThe nDCG of a recommendation set is always normalized to the range of 0 to 1. To add, nDCG@k is simply nDCG for the top k ranked elements in a set. Thus, DCG and iDCG are only calculated for the top k elements in the set. To further specify, in computing both DCG and iDCG, the adjusted set refers to the top k ranked elements so in the case of iDCG, it can also be referred to as the top k relevant scores.\n\n### Recall\n\n**Recall**, in regards to recommendation systems, is a metric for relevancy that quantifies the proportion of relevant elements retrieved amongst all relevant elements. \nA general definition, as stated in [2], is dividing the correctly classified positives by the total positive count.In the scope of recommendation systems, the correctly classified positives could be the number of good games recommended and the total positive count would be the total number of good games in this example.\n\nRecall is also often referred to as the true positive rate or the hit rate. \n\n$$ Recall = \\frac{TP}{TP + FN}   \\qquad(6) $$\n\n\nWhere TP denotes a true positive case and FN denotes a false negative.\n\nSimilarly to how nDCG@k is computed, recall@k is the fraction of relevant items present in the top k recommendations.\n\n\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"# Evaluating Recommendation Models in Production\nThere is a difference between the mathematical metrics and the business metrics. The goal of an engineering solution is to drive business. We would talk about the metrics used by businesses to evaluate model performance in production.\nBusiness teams may choose to define different metrics as the **Key Performance Indicators(KPIs)** for a given ML models success or failure. Some of these metrics are highlighted below \n![](https://i0.wp.com/neptune.ai/wp-content/uploads/2022/10/Recommender-Systems-ML-Metrics-vs-Business-Metrics_16.png?resize=790%2C255&ssl=1)\nTo learn more about these metrics [refer to this paper](https://arxiv.org/pdf/1908.08328.pdf) [3] which goes into depth about each of these metrics. \nThere are two types of studies that are run in production by companies to evaluate the success of a model.\n* When a new model is launched for recommendation purposes\n* Evaluating whether the current model is doing what it is supposed to do and do updates to the model add any value to the business\n\n## Launching a new model\nWhen a new model is launched to replace an existing model, companies **run randomized A/B tests**. Under an A/B test, different sample sets of users are shown different experiences. Basis key business metrics captured, decisions are made on whether a model can be launched or not. [4]\n\n## Deploying an improved model\nTo evaluate if a model actually improved business metrics defined to drive the success of an experiment, we rely on **automated monitoring and alerting software engineering solutions**. These systems are built out to capture user actions and use that information to determine how the model is performing. These automated systems alert the stakeholders if business metrics show a negative trend. This continuous evaluation also helps ensure that model performance does not drop.\n","metadata":{}},{"cell_type":"markdown","source":"## Do deep learning models add value?\n\nTraining and deploying deep learning models can be a very costly affair. So, it is important to identify what sort of value addition can these models bring to the table. There are several Learning-to-Rank(LTR) models that were developed at Microsoft Research. **LightGBM** and **LambdaMART** are the most popular ones. \n\nOver the years, as computation power available to researchers have improved, there has been an increased focus on adopting Deep Learning models. However, there are several studies that show that Decision Trees perform just as well for recommendation systems as deep learning models. In fact, they remain a strong baseline for performance for any new models developed for recommendation problems. Even in contests, they are widely used. For instance, the winner of the RecSys 2022 challenge was a model that leveraged the LightGBM model that was trained with item metadata and GNN embeddings.\n\nSome of the key weaknesses neural network solutions face are:\n* **Data Sparsity**: For highly complex neural networks, you require a lot of data for training the model to provide high quality personalisation. Quite often the dataset is sparse to the point that this training is not effective. Data augmentation techniques can be used to increase the size of the dataset. But it is more challenging to come up with the right data augmentation strategy for LTR datasets. It’s not as easy as rotating an image in computer vision. [5]\n* **Generalising to new datasets**: Research presented by Shwartz-Ziv and Armon highlighted the fact that deep learning models perform well on the same dataset but do not generalise well to new datasets. This is a problem in the real world as the dataset is constantly evolving and you need to provide relevant recommendations at all times[6].\n\nWhile deep learning models often report success in the paper, we should take these successes with a grain of salt. There is a significant lack of standardization in the way recommendation systems(and in general tabular datasets) are evaluated. We should be open to the idea that while a model performs well for the dataset being chosen in a given paper, it may not generalise well to a new dataset. The ability of a model to generalise to any given situation is critical for its adoption in the real-world.\n","metadata":{}},{"cell_type":"markdown","source":"# Problems faced by Real World Recommendation Systems\n\nWhen designing recommendation engine solutions at scale, it is important for us to factor in key challenges that all recommendation systems face. The two major challenges are:\n1. **Sparse Datasets**\n\n2. **Cold Start Recommendation Performance**\nBoth these problems are an active area of research and there is no one-stop shop solution for these problems. We would prorvide a bird's eye view into these issues in the next section.","metadata":{}},{"cell_type":"markdown","source":"## Sparse Datasets\n\nThis problem occurs when the model doesn’t have enough data on a given user’s interaction with the content catalog. As a result of a lack of interaction between the user and the content catalog, establishing relationships is more challenging for a model. The definition of Sparsity is highly dependent on the model being used.\n\nHaving sparse data in a dataset hampers the performance of recommendation models. To tackle sparse datasets models like SLIM are employed which have been shown to exhibit better performance for Sparse datasets. We can also leverage decision trees like LightGBM and LambdaMart for getting meaningful rankings for these datasets.\n\nNetflix utilized the **SLIM approach with the Alternating Directions Method of Multipliers (ADMM) optimisation method** for training sparse datasets. ADMM was found to cause training time to be independent of the number of users allowing great room for scalability. \n\nADMM is an algorithmic optimiser due to its ability to divide a problem into smaller sections that are simpler to handle. In the research paper, it is actualised through splitting up the item-item weight matrix (matrix B) into two equal matrices (B, C). The equality constraint is then implemented in the mathematical optimization problem through utilizing a Lagrangian multiplier matrix associated with the equality constraint. [7] \n\n$$ L_{ρ}(B,C, Γ) =  f(B) + g(C) + ⟨Γ, B − C⟩_{F} + \\frac{ρ}{2} · ∥B − C∥_{F} ^{2}   \\qquad (7) $$         \n\nHere:\n\n* Lp is the augmented Lagrangian function.\n* B and C represent the primal variables to be optimised.\n* Γ  denotes the Lagrange multipliers associated with the equality constraint B = C and is the dual variable\n* f (B)  is the objective function.\n* g(C) is the L1 penalty term \n* ⟨Γ, B − C⟩F is the Frobenius inner product of matrices Γ and B − C\n* ρ is a parameter that controls the penalty strength by applying the square difference between B and C\n\n\nThis is the augmented Lagrangian approach and implements a penalty parameter that relates to the square difference between the matrices B and C. During the optimisation process, the objective function is targeted to meet the equality constraint. The approach has iterative updates to solve the constrained optimization-problem. Each of the three matrices (B, C and Γ )  are consecutively updated at every iteration step.[7]\n\n3 model variants of ADMM are used in the approach: \n* ADMM ≥ 0 & L1: ADMM with non-negative weights, s, L1- norm and L2-norm regularisation\n* ADMM ≥ 0: ADMM with non-negative weights and only L2-norm regularisation\n* ADMM L1: ADMM with unrestricted weights, L1-norm and L2-norm regularisation\n\nThe model variants and other baseline models were evaluated on three datasets: ML-20M, Netflix and MSD. The evaluation metrics used were Recall@20, Recall@50 and nDCG@100. The best performing baseline model across the three datasets was MULT-VAEPR  and ADMM L1 was the best performing ADMM model variant. In terms of nDCG@100, ADMM L1 was 1% less than MULT-VAEPR on ML-20M, 3% greater on Netflix and 25% greater on MSD. In terms of both Recall@20 and Recall@50, the same trend follows with ADMM L1 being faintly worse than MULT-VAEPR on ML-20M, somewhat better on Netflix and significantly superior on MSD. Overall, ADMM reduces training time whilst increasing recommendation accuracy via optimisation. It can also be concluded that ADMM beats state-of-the-art methods on Netflix and MSD datasets.","metadata":{}},{"cell_type":"markdown","source":"## The Cold Start Problem\n\nThe biggest problem all content recommendation systems face is providing an appropriate ranking for a newly released piece of content. As we do not have a lot of user feedback data, it is very challenging to rank the show for a user. This is popularly referred to as the cold start problem.\nThe general plan of action to improve performance for cold start scenarios is to build hybrid models that leverage alternative sources of information. There are several ways to build such models. \n\nOne way to improve cold start ranking is to integrate metadata information of the title being watched.  Google’s research paper introduces the idea of combining user history metadata with item metadata to get a more meaningful recommendation for cold start items.\n\nThe model was evaluated for tweet recommendation systems, where the research team was able to identify that performance of the model improved for both linear and non linear metadata adaptation models. The lift was more significant for non linear models. \n\nThe **Neural Feature Combiner approach** is another approach used to improve performance in cold start scenarios. This model utilises feature embeddings pre-learned from a collaborative filtering approach like SLIM for training and then providing more accurat results. The model takes the features of a given item as input and generates a similarity between the pre-existing warm items and the new cold items. Deep learning architectures enable the model to identify and create complex non-linear relationships. There are different variants of the approach which leverage different loss functions and model the cold start problem in slightly different ways. \n\nThese approaches use Mean Squared Error, Cross Entropy and Bayesian Personalized Ranking optimization criterion. You can read about the detailed implementation details in the original paper.\n","metadata":{}},{"cell_type":"markdown","source":"Solutions in production environment follow a different set of rules than that applied in a Kaggle contest. For instance, Netflix did not productionize the winning solution of the Netflix Recommendation Challenge. Hence, this article would be incomplete without talking very briefly about some of the winning Kaggle solutions that have been used across contests.","metadata":{}},{"cell_type":"markdown","source":"## Winning solutions for Kaggle Contests\n\n### Collaborative Filtering \n\nThe content based engine suffers from some severe limitations. It is only capable of suggesting movies which are close to a certain movie. That is, it is not capable of capturing tastes and providing recommendations across genres.\nAlso, the engine that authors built is not really personal in that it doesn't capture the personal tastes and biases of a user. Anyone querying authors’ engines for recommendations based on a movie will receive the same recommendations for that movie, regardless of who she/he is.\n* **User based filtering** recommends products to a user that similar users have liked. For measuring the similarity between two users we can either use pearson correlation or cosine similarity.\n* **Item Based Collaborative Filtering**: Instead of measuring the similarity between users, the item-based CF recommends items based on their similarity with the items that the target user rated. Likewise, the similarity can be computed with Pearson Correlation or Cosine Similarity. The major difference is that, with item-based collaborative filtering, we fill in the blank vertically, as opposed to the horizontal manner that user-based CF does.\nIt successfully avoids the problem posed by dynamic user preference as item-based CF is more static. However, several problems remain for this method. First, the main issue is scalability. The computation grows with both the customer and the product. The worst case complexity is O(mn) with m users and n items. In addition, sparsity is another concern.\n\n### Single Value Decomposition\n\nOne way to handle the scalability and sparsity issue created by CF is to leverage a latent factor model to capture the similarity between users and items. Essentially, authors [8] want to turn the recommendation problem into an optimization problem. Authors can view it as how good authors are in predicting the rating for items given to a user. One common metric is Root Mean Square Error (RMSE). The lower the RMSE, the better the performance.\nLatent factor is a broad idea which describes a property or concept that a user or an item has. For instance, for music, latent factor can refer to the genre that the music belongs to. SVD decreases the dimension of the utility matrix by extracting its latent factors. Essentially, authors map each user and each item into a latent space with dimension r. Therefore, it helps authors better understand the relationship between users and items as they become directly comparable. The below figure illustrates this idea and more info can be found in [this notebook](https://www.kaggle.com/code/ibtesama/getting-started-with-a-movie-recommendation-system).\n\n   ![](https://hackernoon.imgix.net/hn-images/1*GUw90kG2ltTd2k_iv3Vo0Q.png)","metadata":{}},{"cell_type":"markdown","source":"## Large Language Models for recommendation \n\nLLMs have promise for recommendation tasks due to their ability for task generalisation and rationality. Empirical studies and analysis have been carried out on assessing LLMs for this purpose. A particular notable one was carried out by researchers at Renmin University of China and UC San Diego in 2023[9]. By creating prompting templates that direct LLMs to assess applicants based on prior encounters, the study frames recommendation as a conditional ranking assignment. The main results show that LLMs outperform conventional zero-shot recommendation techniques and may use previous behaviours for personalised ranking. They also highlight the significance of quick design in recognising interaction order. However, a major concern surrounding LLMs has been hallucinations and bias. The study does suggest that such biases can be limited through prompting and bootstrapping. \n\n![Figure: LLM-based zero-shot ranking method ](https://github.com/RUCAIBox/LLMRank/raw/master/assets/model.png)\n\n\n\nAnother survey on LLMs for recommendation [10] was conducted by researchers from the University of Science and Technology of China and BOSS Zhipin’s Career Science Lab. LLMs are categorized into two categories for the purpose of the study: **Discriminative LLMs for Recommendation (DLLM4Rec)** and **Generative LLMs for Recommendation (GLLM4Rec)**. In this survey, DLLM4Rec typically refers to BERT series models. DLLM4Rec finetune pre-trained models like BERT with recommendation-specific data and have been implemented on various recommendation tasks including user representation learning, ranking, group recommendation, search/matching, and CTR prediction. GLLM4Rec do not require alignment on the recommendation-domain but rather leverage techniques such as in-context learning, prompt tuning, and instruction tuning to adapt LLMs for recommendation. The survey also further categorises GLLM4Rec into two categories: the non-tuning paradigm and the tuning paradigm.  \n\n* The **non-tuning paradigm** makes the assumption that the LLM is able to directly generate recommendation results and requires prompting as the main technique to improve the execution of the task.  \n\n* The **tuning paradigm** introduces two tuning methods: **prompt tuning** and **instruction tuning**. Prompt tuning involves fine-tuning LLMs or tuning soft prompts on a specific task, such as rating prediction. Contrarily, LLMs are fine-tuned by instruction tuning on a variety of jobs with various sorts of instructions. The study suggests instruction tuning provides greater alignment with human intent and enhancing zero-shot capabilities but also notifies that this notion is not universally agreed upon. \n\nLike the first paper, the survey's findings express the issue of biases; particularly position (ranking), popularity, and fairness bias. Position bias is the impact of item order on rankings and was particularly expressed in GLLM4Rec, whereas popularity bias is the favouring of popular items. Fairness bias concerns attributes based on demographics such as gender or race. Other findings found that evaluating generative recommendation capability LLMs was quite challenging due to a lack of a fully-fledged evaluation criteria. ","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nsubmission=pd.read_csv(\"/kaggle/input/2023-kaggle-ai-report/sample_submission.csv\")\nsubmission.head()","metadata":{"execution":{"iopub.status.busy":"2023-07-05T15:04:47.099789Z","iopub.execute_input":"2023-07-05T15:04:47.100134Z","iopub.status.idle":"2023-07-05T15:04:47.167796Z","shell.execute_reply.started":"2023-07-05T15:04:47.100107Z","shell.execute_reply":"2023-07-05T15:04:47.166368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## References:\n\n[1] Järvelin, K & Kekäläinen, J (2002), 'Cumulated Gain-based Evaluation of IR Techniques', ACM Transactions on Information Systems (ACM TOIS), vol. 20, no. 4, pp. 422-446. https://doi.org/10.1145/582415.582418\n\n[2] Olson, D.L. and Delen, D. (2008) Advanced Data Mining Techniques. Springer, pp. 138\n\n[3] Jannach, D. and Jugovac, M., (2019). Measuring the business value of recommender systems. ACM Transactions on Management Information Systems (TMIS), 10(4).\n\n[4] Netflix Technology Blog (2019) ‘Page Simulation for Better Offline Metrics at Netflix’, Medium. Available at: https://netflixtechblog.com/page-simulator-fa02069fb269.\n\n[5] Qin, Z. et al. (2021) ‘Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees?’, International Conference on Learning Representations (ICLR). https://storage.googleapis.com/pub-tools-public-publication-data/pdf/a7f0822e77e8b6b4c00c879707fe60e3955d4a03.pdf\n\n\n[6] Shwartz-Ziv, R. and Armon, A. (2021) Tabular data: Deep Learning is not all you need. https://arxiv.org/pdf/2106.03253.pdf\n \n\n[7] Steck, H. et al. (2020) ‘Admm Slim’, Proceedings of the 13th International Conference on Web Search and Data Mining [Preprint]. doi:10.1145/3336191.3371774. \n\n  \n[8] Ahmed, I. (2020) ‘Getting Started with a Movie Recommendation System’, Kaggle. Available at: https://www.kaggle.com/code/ibtesama/getting-started-with-a-movie-recommendation-system#Collaborative-Filtering. \n\n\n[9] Hou, Y., Zhang, J., Lin, Z., Lu, H., Xie, R., McAuley, J. and Zhao, W.X., (2023). Large language models are zero-shot rankers for recommender systems. arXiv preprint arXiv:2305.08845. \n\n[10] Wu, L., Zheng, Z., Qiu, Z., Wang, H., Gu, H., Shen, T., Qin, C., Zhu, C., Zhu, H., Liu, Q. and Xiong, H., (2023). A Survey on Large Language Models for Recommendation. arXiv preprint arXiv:2305.19860. \n","metadata":{}},{"cell_type":"code","source":"submission.loc[0]['value']='Tabular and/or time series data'\nsubmission.loc[1]['value']= 'https://www.kaggle.com/code/coder98/state-of-content-recommendation-engines-in-2023'\nsubmission.loc[2]['value']= 'https://www.kaggle.com/code/barbosajaf/jafb-2023-kaggle-ai-report/comments#2340953'\nsubmission.loc[3]['value']= 'https://www.kaggle.com/code/gigikenneth/ethical-implications-of-artificial-intelligence/comments#2340994'\nsubmission.loc[4]['value']= 'https://www.kaggle.com/code/kavisekar/latest-development-in-time-series-analysis-report/comments#2345975'\n\nsubmission.head()\n# peer feedback will be added soon","metadata":{"execution":{"iopub.status.busy":"2023-07-05T15:09:48.995632Z","iopub.execute_input":"2023-07-05T15:09:48.995969Z","iopub.status.idle":"2023-07-05T15:09:49.006426Z","shell.execute_reply.started":"2023-07-05T15:09:48.995937Z","shell.execute_reply":"2023-07-05T15:09:49.005229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('submission.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2023-07-05T15:12:29.224917Z","iopub.execute_input":"2023-07-05T15:12:29.225288Z","iopub.status.idle":"2023-07-05T15:12:29.231513Z","shell.execute_reply.started":"2023-07-05T15:12:29.225262Z","shell.execute_reply":"2023-07-05T15:12:29.230737Z"},"trusted":true},"execution_count":null,"outputs":[]}]}