Well this was fun :)
\r\nCongrats to the winners! GG
\r\nI'd like to start this topic for participants that want to share their solutions. I'm really curious how others approached the problem.
\r\nMy code and comments are here: http://webmining.olariu.org/event-recommendation-contest-on-kaggle
", "messages": [ { "id": "20690", "postDate": "02/21/2013 22:35:30", "content": "Well this was fun :)
\r\nCongrats to the winners! GG
\r\nI'd like to start this topic for participants that want to share their solutions. I'm really curious how others approached the problem.
\r\nMy code and comments are here: http://webmining.olariu.org/event-recommendation-contest-on-kaggle
", "rawMarkdown": "", "votes": null }, { "id": "20700", "postDate": "02/22/2013 07:39:23", "content": "The contest was indeed exciting.. If anything I learnt the perils of overfitting in this contest. Our code is in a bit of a mess.. :) .. I will put it up along with a blog post after we clean it, but here is a summary of what we did.
\r\nWe used regression (random forest, grad boost regressor in scikit) to score each (user, event) pair. A target of 1 if interested and 0 if not. (Funnily everytime we tried to use the 'not interested' column my score decreased hence we ignored it).
\r\nBeing amateurs in programming and python, we didn't know how to handle the 3 million events, but we noticed only 30k odd events featured in any other data file, hence we pruned the rest of the events and made a 13MB file out of the 1.1GB file and only worked\r\n with that.. :)
\r\nAlso we didn't do any clustering, (user or event) we just put all the event details also in the feature vector for the (user, event) pair. The feature vector had three parts : The user part (containing age, sex, locale, no. of events attended etc.) , the\r\n event part (no. of attendees, word freq count etc) and the 'User-Event' part. The main components of User-Event part is detailed below:
\r\n# and fraction of friends who attended the event; friendship with event creator; if event city is a substring of user location; No of `similar' events attended by the user; No. of similar events attended by the friends of the user; Time between event start\r\n and event seen by the user;
\r\n\r\n
After a cross validation and some careful weighting of the regressors, we managed 0.727 (3rd) by the time public leaderboard closed. \r\n
\r\nIn the last one week, we added more RFs/GBRs with different parameters and also added dolaameng's regression results to the already significant number of sub-learners. (Thanks to dolaameng for his code)
\r\nManaged to get 0.707 and 6th place in the final result.
\r\n\r\n
Best,
\r\nHarishgp.
\r\n\r\n
\r\n
\r\n
", "rawMarkdown": "", "votes": null }, { "id": "20702", "postDate": "02/22/2013 08:55:59", "content": "
@Harishgp
\r\n\r\n\r\n(Funnily everytime we tried to use the 'not interested' column my score decreased hence we ignored it).
\r\n
I was puzzled by this one too. Seem to remember doing a query and finding that after taking the funny business about time stamps into account, there weren't any not-interesteds in the remaining data.
\r\n@Andrei
\r\n\r\n\r\n"I didn't manage to get anything out of user age and gender. I'm still wondering if (and how) that info can be used in some useful way."
\r\n
FWIW, I managed to get something out of that by adding up the keyword vectors of all people of a particular gender in a locale's attendances and interests, and taking the cosine similarity with the event in question. Worried now about over fitting (lost\r\n 20 places in the final cut perhaps from this and another feature.) Also my GLM results showed older people were a little less likely to be interested.
", "rawMarkdown": "", "votes": null }, { "id": "20706", "postDate": "02/22/2013 12:36:55", "content": "@Andrei. Excellent solution!
\r\nyour blog says that "I chose a Random Forest (again.. scikit-learn), because it was able to work with missing values. "
\r\nI use sklearn, too. Could you illustrate how to work with missing values in sklearn? thanks in advance.
", "rawMarkdown": "", "votes": null }, { "id": "20707", "postDate": "02/22/2013 13:03:51", "content": "My team built several models and blended them together using logistic regression. We also found approaching this as a classification problem and ignoring 'not interested' worked best. Our models included libFM, svmrank, regularized greedy forests, R's\r\n gbm, and random forest on various subsets of the data. The regularized greedy forests gave the best single model.
\r\nOver fitting was an issue in this competition and we expected to see a shake up in the private leader board. The regularization of the final logistic regression made a big difference in our final score. Specifically, L2 regularization worked better for\r\n us than L1 regularization. Varying the regularization parameter gave a 0.02 range of scores with better scores favoring more regularization.
\r\n", "rawMarkdown": "", "votes": null }, { "id": "20723", "postDate": "02/22/2013 21:45:42", "content": "
Congrats to the winners!
\r\nIt was a lot fun. I wrote a blog post about my solution and why the difference between public and private has so big(from 1th to 15th) and it has nothing to do with overfitting:
\r\nhttp://datalab.lu/blog/2013/02/21/kaggle-event-recommentation-engine/
\r\nCheers,
\r\nDzidas
", "rawMarkdown": "", "votes": null }, { "id": "20725", "postDate": "02/22/2013 22:13:05", "content": "@Matthew - I tried adding the age and gender, as they were or by building an event profile (with the average age and gender of attendants). They were ranked as important by the random trees, but crossvalidation error increased.
\r\n@hellofys - You don't have to do anything. Missing values were represented as None in the feature vectors. The Random Forest Classifier from scikit-learn will work with that. For SVM or Logistic Regression, you'll have to remove those features or use some\r\n averaging technique.
\r\n@Dzidas - I also struggled with those locations. Having an API to get coordinates was so comfortable. But you could get the same results without. There are 1.6 million events with both location string and GPS coordinates. That's a big dataset. I used them\r\n to match the other locations to coordinates. Nonetheless, I also think that this sort of data should be provided already processed or tools that use public information should be allowed.
", "rawMarkdown": "", "votes": null }, { "id": "20728", "postDate": "02/22/2013 22:51:17", "content": "@Andrei - yep, you right that users locations could be derived from events database. I didn't get such idea and it is why my score is lower ;)
", "rawMarkdown": "", "votes": null }, { "id": "20743", "postDate": "02/23/2013 02:17:07", "content": "@Andrei - I try do nothing with missing value. For example,
\r\na = np.array([[1,np.NaN,3],[4,5,6]])\r\n
clf = RandomForestClassifier()
clf.fit(a[:, :2], a[:,2])
It comes with error "ValueError: Array contains NaN or infinity."\r\n
\r\n
@Andrei @hellofys I may shed some light on this - it seems that the Random Forest in the latest relase of scikit-learn doesn't work well with missing values, while previous versions tolerated them. I've updated scikit-learn midway during the challenge,\r\n and suddenly encountered the error @hellofys is describing.
", "rawMarkdown": "", "votes": null }, { "id": "20753", "postDate": "02/23/2013 11:19:23", "content": "Hi guys,
\r\nthis competition was basically all about feature creation. So here is a list of my some of my features:
\r\nIn the end I had about 60-70 features, which were all more or less important.
\r\nWith all these features I tried to predict the "interested"-column and for this I used the probability-output of a gbm. My aim was to minimize the LogLoss-error between my prediction and the actual solution. I never used the MAP-metric, because the lower\r\n the LogLoss-error the higher the MAP.
\r\nThe create a submission I ordered all events using my probability-prediction.
", "rawMarkdown": "", "votes": null }, { "id": "20754", "postDate": "02/23/2013 12:51:27", "content": "[quote=jsf;20753]
\r\n[/quote]
\r\nWe also used the above features written by jsf as well as the proportion of count to the total response, and also did user community detection. Since the number of users are quite large, we first filtered out users to users whose info is available, that\r\n is, who are in users.csv file. We also tried to take into account that how many events each user had and it was actually top ranked in terms of relative inference by gbm, but we reconsidered and omitted from our final model. I think the reason the number of\r\n events is important when you make score in probability is obvious, probability of interested is 1/4, 1/5, 1/6 depending on the number of events 4, 5, 6 respectively, but we couldn't believe that it is relevant and in our experiments, it didn't give any improvement\r\n in MAP metric. We used about 30 features. Our final model was averaged over gbm w/ bernoulli, gbm w/ adaboost, and randomForest.
", "rawMarkdown": "", "votes": null }, { "id": "20759", "postDate": "02/23/2013 15:22:54", "content": "Well my solution wasn't a lot different than the others and I attribute my high ranking mostly to luck. When the public leaderboard was frozen, I was at the 11th position, and since I did not get any significant breakthrough in the remaining week, I was\r\n a bit surprised to find myself in the third place.
\r\n\r\n
Two features that I think are unique in my solution involved directly comparing the events that were suggested to a given user each against the other. The first feature was the distance of each feature from the median of the locations of all the events shown\r\n to the user. This actually worked better than any other distance metric I've used (I didn't try any third party services, but I did leveraged the locations of the events in events.csv to estimate the coordinates of users). The other feature first found out\r\n which event was happening first from all those suggested and then calculated for each event the ratio between its delta and that minimum delta.
\r\n\r\n
You can read about the lessons I've learned here - http://www.rouli.net/2013/02/five-lessons-from-kaggles-event.html, which also details a sad twist in my story.
", "rawMarkdown": "", "votes": null }, { "id": "20761", "postDate": "02/23/2013 16:06:18", "content": "I tried similar features as jsf, but used only "gaussian" gbm.
\r\nI built two type of gbm models.The first model roughly estimated the probability of interestedness, then the second model refined the first model.
\r\nhttps://github.com/tks0123456789/kaggle-event-recommendation
", "rawMarkdown": "", "votes": null }, { "id": "20947", "postDate": "02/28/2013 12:34:18", "content": "Hi,
\r\ncongrats to the winners too!!
\r\nI spent a lot of time comparing various target modeling strategies: interested vs. not interested classif, interested vs not interested vs unknown, learning to rank (ala SVMRank).
\r\nI ended up with the simplest one, interested vs. other classification using GBRTs using prediction score to rank the events. I was quite surprised that pure ranking solution did not work well for this contest. Did anybody also tried ranking strategies, pairwise\r\n or listwise?
\r\nAlso, I am sure you noticed some events had identical descriptions (only id differs in events.csv) but different attendees. I did not manage to tranform this into a informative feature.
", "rawMarkdown": "", "votes": null }, { "id": "20972", "postDate": "03/01/2013 17:52:10", "content": "I'd be really interested to hear what anyone that used clustering successfully did exactly. E.g.
\r\nAlso, I found charting my performance on the public/private leaderboards interesting. Some morals for me:
\r\nThanks for the joys of hindsight all.
\r\n\r\n
PS @Andrei, thanks for the insight.
", "rawMarkdown": "", "votes": null }, { "id": "92227", "postDate": "09/12/2015 03:47:01", "content": "I am building a product recommender system, instead of recommending event i have to recommend products. Please advise on the below question.
\n\nI am treating this problem as a classification problem and for each user-product combination i am predicting the probability of purchase using (GBM etc).
\n\nGiven the class imbalance(6% event rate) & that fact that I need to maximize map@k which is the best loss function & evaluation criteria for evaluating result of a binary classifier( random forest/ gbm etc)?
\n\nI am asking because I can't specify map@k as evaluation criteria in for a binary classifier right?
\n\nDid you use recall or precision or kappa etc as a scoring criteria for tune the hyper parameters of the random forest? Please advise.
\n\nCan someone explain how this statement by Josef Feigl. I didnt understand.
\n\n\n", "rawMarkdown": "I am building a product recommender system, instead of recommending event i have to recommend products. Please advise on the below question.\r\n\r\nI am treating this problem as a classification problem and for each user-product combination i am predicting the probability of purchase using (GBM etc). \r\n\r\nGiven the class imbalance(6% event rate) & that fact that I need to maximize map@k which is the best loss function & evaluation criteria for evaluating result of a binary classifier( random forest/ gbm etc)? \r\n\r\nI am asking because I can't specify map@k as evaluation criteria in for a binary classifier right?\r\n\r\nDid you use recall or precision or kappa etc as a scoring criteria for tune the hyper parameters of the random forest? Please advise.\r\n\r\nCan someone explain how this statement by Josef Feigl. I didnt understand.\r\n\r\n> \"My aim was to minimize the LogLoss-error between my prediction and\r\n> the actual solution. I never used the MAP-metric, because the lower\r\n> the LogLoss-error the higher the MAP.\"", "votes": null } ], "comments": [ { "id": 20700, "author_name": "harishgp", "author_url": "", "post_date": "02/22/2013 07:39:23", "content": ""My aim was to minimize the LogLoss-error between my prediction and\n the actual solution. I never used the MAP-metric, because the lower\n the LogLoss-error the higher the MAP."
\n
The contest was indeed exciting.. If anything I learnt the perils of overfitting in this contest. Our code is in a bit of a mess.. :) .. I will put it up along with a blog post after we clean it, but here is a summary of what we did.
\r\nWe used regression (random forest, grad boost regressor in scikit) to score each (user, event) pair. A target of 1 if interested and 0 if not. (Funnily everytime we tried to use the 'not interested' column my score decreased hence we ignored it).
\r\nBeing amateurs in programming and python, we didn't know how to handle the 3 million events, but we noticed only 30k odd events featured in any other data file, hence we pruned the rest of the events and made a 13MB file out of the 1.1GB file and only worked\r\n with that.. :)
\r\nAlso we didn't do any clustering, (user or event) we just put all the event details also in the feature vector for the (user, event) pair. The feature vector had three parts : The user part (containing age, sex, locale, no. of events attended etc.) , the\r\n event part (no. of attendees, word freq count etc) and the 'User-Event' part. The main components of User-Event part is detailed below:
\r\n# and fraction of friends who attended the event; friendship with event creator; if event city is a substring of user location; No of `similar' events attended by the user; No. of similar events attended by the friends of the user; Time between event start\r\n and event seen by the user;
\r\n\r\n
After a cross validation and some careful weighting of the regressors, we managed 0.727 (3rd) by the time public leaderboard closed. \r\n
\r\nIn the last one week, we added more RFs/GBRs with different parameters and also added dolaameng's regression results to the already significant number of sub-learners. (Thanks to dolaameng for his code)
\r\nManaged to get 0.707 and 6th place in the final result.
\r\n\r\n
Best,
\r\nHarishgp.
\r\n\r\n
\r\n
\r\n
", "votes": null, "replies": [] }, { "id": 20702, "author_name": "matthewpearce", "author_url": "", "post_date": "02/22/2013 08:55:59", "content": "
@Harishgp
\r\n\r\n\r\n(Funnily everytime we tried to use the 'not interested' column my score decreased hence we ignored it).
\r\n
I was puzzled by this one too. Seem to remember doing a query and finding that after taking the funny business about time stamps into account, there weren't any not-interesteds in the remaining data.
\r\n@Andrei
\r\n\r\n\r\n"I didn't manage to get anything out of user age and gender. I'm still wondering if (and how) that info can be used in some useful way."
\r\n
FWIW, I managed to get something out of that by adding up the keyword vectors of all people of a particular gender in a locale's attendances and interests, and taking the cosine similarity with the event in question. Worried now about over fitting (lost\r\n 20 places in the final cut perhaps from this and another feature.) Also my GLM results showed older people were a little less likely to be interested.
", "votes": null, "replies": [] }, { "id": 20706, "author_name": "hellofys", "author_url": "", "post_date": "02/22/2013 12:36:55", "content": "@Andrei. Excellent solution!
\r\nyour blog says that "I chose a Random Forest (again.. scikit-learn), because it was able to work with missing values. "
\r\nI use sklearn, too. Could you illustrate how to work with missing values in sklearn? thanks in advance.
", "votes": null, "replies": [] }, { "id": 20707, "author_name": "chaseos", "author_url": "", "post_date": "02/22/2013 13:03:51", "content": "My team built several models and blended them together using logistic regression. We also found approaching this as a classification problem and ignoring 'not interested' worked best. Our models included libFM, svmrank, regularized greedy forests, R's\r\n gbm, and random forest on various subsets of the data. The regularized greedy forests gave the best single model.
\r\nOver fitting was an issue in this competition and we expected to see a shake up in the private leader board. The regularization of the final logistic regression made a big difference in our final score. Specifically, L2 regularization worked better for\r\n us than L1 regularization. Varying the regularization parameter gave a 0.02 range of scores with better scores favoring more regularization.
\r\n", "votes": null, "replies": [] }, { "id": 20723, "author_name": "dzidas", "author_url": "", "post_date": "02/22/2013 21:45:42", "content": "
Congrats to the winners!
\r\nIt was a lot fun. I wrote a blog post about my solution and why the difference between public and private has so big(from 1th to 15th) and it has nothing to do with overfitting:
\r\nhttp://datalab.lu/blog/2013/02/21/kaggle-event-recommentation-engine/
\r\nCheers,
\r\nDzidas
", "votes": null, "replies": [] }, { "id": 20725, "author_name": "aolariu", "author_url": "", "post_date": "02/22/2013 22:13:05", "content": "@Matthew - I tried adding the age and gender, as they were or by building an event profile (with the average age and gender of attendants). They were ranked as important by the random trees, but crossvalidation error increased.
\r\n@hellofys - You don't have to do anything. Missing values were represented as None in the feature vectors. The Random Forest Classifier from scikit-learn will work with that. For SVM or Logistic Regression, you'll have to remove those features or use some\r\n averaging technique.
\r\n@Dzidas - I also struggled with those locations. Having an API to get coordinates was so comfortable. But you could get the same results without. There are 1.6 million events with both location string and GPS coordinates. That's a big dataset. I used them\r\n to match the other locations to coordinates. Nonetheless, I also think that this sort of data should be provided already processed or tools that use public information should be allowed.
", "votes": null, "replies": [] }, { "id": 20728, "author_name": "dzidas", "author_url": "", "post_date": "02/22/2013 22:51:17", "content": "@Andrei - yep, you right that users locations could be derived from events database. I didn't get such idea and it is why my score is lower ;)
", "votes": null, "replies": [] }, { "id": 20743, "author_name": "hellofys", "author_url": "", "post_date": "02/23/2013 02:17:07", "content": "@Andrei - I try do nothing with missing value. For example,
\r\na = np.array([[1,np.NaN,3],[4,5,6]])\r\n
clf = RandomForestClassifier()
clf.fit(a[:, :2], a[:,2])
It comes with error "ValueError: Array contains NaN or infinity."\r\n
\r\n
@Andrei @hellofys I may shed some light on this - it seems that the Random Forest in the latest relase of scikit-learn doesn't work well with missing values, while previous versions tolerated them. I've updated scikit-learn midway during the challenge,\r\n and suddenly encountered the error @hellofys is describing.
", "votes": null, "replies": [] }, { "id": 20753, "author_name": "joseffeigl", "author_url": "", "post_date": "02/23/2013 11:19:23", "content": "Hi guys,
\r\nthis competition was basically all about feature creation. So here is a list of my some of my features:
\r\nIn the end I had about 60-70 features, which were all more or less important.
\r\nWith all these features I tried to predict the "interested"-column and for this I used the probability-output of a gbm. My aim was to minimize the LogLoss-error between my prediction and the actual solution. I never used the MAP-metric, because the lower\r\n the LogLoss-error the higher the MAP.
\r\nThe create a submission I ordered all events using my probability-prediction.
", "votes": null, "replies": [] }, { "id": 20754, "author_name": "naokazumizuta", "author_url": "", "post_date": "02/23/2013 12:51:27", "content": "[quote=jsf;20753]
\r\n[/quote]
\r\nWe also used the above features written by jsf as well as the proportion of count to the total response, and also did user community detection. Since the number of users are quite large, we first filtered out users to users whose info is available, that\r\n is, who are in users.csv file. We also tried to take into account that how many events each user had and it was actually top ranked in terms of relative inference by gbm, but we reconsidered and omitted from our final model. I think the reason the number of\r\n events is important when you make score in probability is obvious, probability of interested is 1/4, 1/5, 1/6 depending on the number of events 4, 5, 6 respectively, but we couldn't believe that it is relevant and in our experiments, it didn't give any improvement\r\n in MAP metric. We used about 30 features. Our final model was averaged over gbm w/ bernoulli, gbm w/ adaboost, and randomForest.
", "votes": null, "replies": [] }, { "id": 20759, "author_name": "roulinir", "author_url": "", "post_date": "02/23/2013 15:22:54", "content": "Well my solution wasn't a lot different than the others and I attribute my high ranking mostly to luck. When the public leaderboard was frozen, I was at the 11th position, and since I did not get any significant breakthrough in the remaining week, I was\r\n a bit surprised to find myself in the third place.
\r\n\r\n
Two features that I think are unique in my solution involved directly comparing the events that were suggested to a given user each against the other. The first feature was the distance of each feature from the median of the locations of all the events shown\r\n to the user. This actually worked better than any other distance metric I've used (I didn't try any third party services, but I did leveraged the locations of the events in events.csv to estimate the coordinates of users). The other feature first found out\r\n which event was happening first from all those suggested and then calculated for each event the ratio between its delta and that minimum delta.
\r\n\r\n
You can read about the lessons I've learned here - http://www.rouli.net/2013/02/five-lessons-from-kaggles-event.html, which also details a sad twist in my story.
", "votes": null, "replies": [] }, { "id": 20761, "author_name": "tks0123456789", "author_url": "", "post_date": "02/23/2013 16:06:18", "content": "I tried similar features as jsf, but used only "gaussian" gbm.
\r\nI built two type of gbm models.The first model roughly estimated the probability of interestedness, then the second model refined the first model.
\r\nhttps://github.com/tks0123456789/kaggle-event-recommendation
", "votes": null, "replies": [] }, { "id": 20947, "author_name": "jmeynet", "author_url": "", "post_date": "02/28/2013 12:34:18", "content": "Hi,
\r\ncongrats to the winners too!!
\r\nI spent a lot of time comparing various target modeling strategies: interested vs. not interested classif, interested vs not interested vs unknown, learning to rank (ala SVMRank).
\r\nI ended up with the simplest one, interested vs. other classification using GBRTs using prediction score to rank the events. I was quite surprised that pure ranking solution did not work well for this contest. Did anybody also tried ranking strategies, pairwise\r\n or listwise?
\r\nAlso, I am sure you noticed some events had identical descriptions (only id differs in events.csv) but different attendees. I did not manage to tranform this into a informative feature.
", "votes": null, "replies": [] }, { "id": 20972, "author_name": "matthewpearce", "author_url": "", "post_date": "03/01/2013 17:52:10", "content": "I'd be really interested to hear what anyone that used clustering successfully did exactly. E.g.
\r\nAlso, I found charting my performance on the public/private leaderboards interesting. Some morals for me:
\r\nThanks for the joys of hindsight all.
\r\n\r\n
PS @Andrei, thanks for the insight.
", "votes": null, "replies": [] }, { "id": 92227, "author_name": "deepakgeorge", "author_url": "", "post_date": "09/12/2015 03:47:01", "content": "I am building a product recommender system, instead of recommending event i have to recommend products. Please advise on the below question.
\n\nI am treating this problem as a classification problem and for each user-product combination i am predicting the probability of purchase using (GBM etc).
\n\nGiven the class imbalance(6% event rate) & that fact that I need to maximize map@k which is the best loss function & evaluation criteria for evaluating result of a binary classifier( random forest/ gbm etc)?
\n\nI am asking because I can't specify map@k as evaluation criteria in for a binary classifier right?
\n\nDid you use recall or precision or kappa etc as a scoring criteria for tune the hyper parameters of the random forest? Please advise.
\n\nCan someone explain how this statement by Josef Feigl. I didnt understand.
\n\n\n", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "20690": "", "20700": "", "20702": "", "20706": "", "20707": "", "20723": "", "20725": "", "20728": "", "20743": "", "20749": "", "20753": "", "20754": "", "20759": "", "20761": "", "20947": "", "20972": "", "92227": "I am building a product recommender system, instead of recommending event i have to recommend products. Please advise on the below question.\r\n\r\nI am treating this problem as a classification problem and for each user-product combination i am predicting the probability of purchase using (GBM etc). \r\n\r\nGiven the class imbalance(6% event rate) & that fact that I need to maximize map@k which is the best loss function & evaluation criteria for evaluating result of a binary classifier( random forest/ gbm etc)? \r\n\r\nI am asking because I can't specify map@k as evaluation criteria in for a binary classifier right?\r\n\r\nDid you use recall or precision or kappa etc as a scoring criteria for tune the hyper parameters of the random forest? Please advise.\r\n\r\nCan someone explain how this statement by Josef Feigl. I didnt understand.\r\n\r\n> \"My aim was to minimize the LogLoss-error between my prediction and\r\n> the actual solution. I never used the MAP-metric, because the lower\r\n> the LogLoss-error the higher the MAP.\"" }, "source": "existing_discussion", "complete": true, "competition_id": 3288, "competition_slug": "event-recommendation-engine-challenge", "team_id": 33524, "solution_url": "/competitions/event-recommendation-engine-challenge/writeups/andrei-olariu-solutions" }"My aim was to minimize the LogLoss-error between my prediction and\n the actual solution. I never used the MAP-metric, because the lower\n the LogLoss-error the higher the MAP."
\n