Hello everyone! Here i'd like to share my 2-3 weeks experience about outlier predictions.
\n\n2-3 weeks ago i decided that the main key for cracking this competion is a pretty good prediction of outliers. That means that False_Positive + False_Negative should be less than the number of outliers as much as possible - in other words less than aproximatly 2200 outliers on train.
\n\nAssuming that distribution of train and test is aproximatly the same, so i concluded that proportion of outliers would be aproximatly the same - 2200 outlier in train and 1000-1200 outliers in test.
\n\nI figured out that visiting specific merchants in historical_transactions is strongly correlated with outliers (I also figured out that the last and the first date of visiting specific merchants by specific card_id is also influenced on the outliers but less). So I calculated *merchant_outlier_mean* - the mean uniquly visited outlier for every merchant in historical_transactions (not new_merchant_transactions) and then i got the mean of *merchant_outlier_mean* for every card_id and that feature had about abs(0.5) correletion by Pearson.
\n\nBy the way, there are a lot of NOT outliers, which didn't make any transactions in new_merchant_transactions. So i denied the hypothesis that outlers are the card_ids that didn't make any transactions according to Elo recommendation algorithm.
\n\nAfter that I assumed that more a card_id visits \"bad\" merchants - more chance that the card_id is outlier. However there are about 33000 merchants that the card_ids from train never visit, but the card_ids from test visit ones. So I have to make a classifier to predict if a merchant have been ever visited by an outlier to predict further *merchant_outlier_mean* for these merchants. An accuracy of that classifier was pretty bad, but regressor that predict *merchant_outlier_mean* worked very well (0.005 MAE), but i fited regressor on *merchant_outlier_mean>0*, so if the classifier predicts for *merchant_outlier_mean=0* examples that *merchant_outlier_mean>0* for very few visiting merchants, regressor could predict *merchant_outlier_mean* close to 1 or 0.5.
\n\nAfter that, i made a classifier that predicts about 600 False_Positive+False_Negative on all train. The model was very confident about its own predictions. CV gave me the pretty close results as for training fold. \nOn test.csv it predicted about 600 card_id as outliers (not 1000-1200 according to my previous assumtion at the beginning of the topic). After, i combined my outllier model with the very simple model fitted with outliers ( i fitted with ouliers to cover False_Negative errors of my outlier-classifier-model). It gave me about 1.8-2 RMSE error on CV. Howerer, on LB i've got very bad results - about 4.2 RMSE.
\n\nAssumtions and conclusions about results:\n1) There are some merchants that have very few visits and *merchant_outlier_mean*=0 according to train, so the card_id that is actualy outlier in test can visit that \"non-outlier-merchant\".\n2) I have classified pretty bad if *merchant_outlier_mean>0* or not. And this is a big reason for bad results in LB.\n3) There are a lot of outliers in train that visited outlier-merchants according to train, and there are very-very few outliers from test that visited the same outlier-merchants according to train. That fact can be explained by the first assumption(conclusion) - There are a lot merchants that have very-very few visits (<5 for example) and *merchant_outlier_mean*>0 (0.8 for example) according to train, so the card_id that is actualy non-outlier in test can visit that \"outlier-merchant\" according to train and the model mark that test non-outlier as outlier and as a result the numbers of False_Positive and False_Negative are growing in test.
\n\nI'm very upset with my results and not being on the top of LB after a lot of work and sleepless nights (planning how i'll spend 20000$ after winning the competition) xD. Soi decided to write a kernel during a week (if i would have enough time) that will cover all my notes - EDA, FE and so on. But now i've just writen the topic to share my main results.
\n\nFeel free to upvote! Leave your comments bellow. Good luck and best wishes on LB)
", "messages": [ { "id": "465137", "postDate": "02/02/2019 12:25:28", "content": "Hello everyone! Here i'd like to share my 2-3 weeks experience about outlier predictions.
\n\n2-3 weeks ago i decided that the main key for cracking this competion is a pretty good prediction of outliers. That means that False_Positive + False_Negative should be less than the number of outliers as much as possible - in other words less than aproximatly 2200 outliers on train.
\n\nAssuming that distribution of train and test is aproximatly the same, so i concluded that proportion of outliers would be aproximatly the same - 2200 outlier in train and 1000-1200 outliers in test.
\n\nI figured out that visiting specific merchants in historical_transactions is strongly correlated with outliers (I also figured out that the last and the first date of visiting specific merchants by specific card_id is also influenced on the outliers but less). So I calculated *merchant_outlier_mean* - the mean uniquly visited outlier for every merchant in historical_transactions (not new_merchant_transactions) and then i got the mean of *merchant_outlier_mean* for every card_id and that feature had about abs(0.5) correletion by Pearson.
\n\nBy the way, there are a lot of NOT outliers, which didn't make any transactions in new_merchant_transactions. So i denied the hypothesis that outlers are the card_ids that didn't make any transactions according to Elo recommendation algorithm.
\n\nAfter that I assumed that more a card_id visits \"bad\" merchants - more chance that the card_id is outlier. However there are about 33000 merchants that the card_ids from train never visit, but the card_ids from test visit ones. So I have to make a classifier to predict if a merchant have been ever visited by an outlier to predict further *merchant_outlier_mean* for these merchants. An accuracy of that classifier was pretty bad, but regressor that predict *merchant_outlier_mean* worked very well (0.005 MAE), but i fited regressor on *merchant_outlier_mean>0*, so if the classifier predicts for *merchant_outlier_mean=0* examples that *merchant_outlier_mean>0* for very few visiting merchants, regressor could predict *merchant_outlier_mean* close to 1 or 0.5.
\n\nAfter that, i made a classifier that predicts about 600 False_Positive+False_Negative on all train. The model was very confident about its own predictions. CV gave me the pretty close results as for training fold. \nOn test.csv it predicted about 600 card_id as outliers (not 1000-1200 according to my previous assumtion at the beginning of the topic). After, i combined my outllier model with the very simple model fitted with outliers ( i fitted with ouliers to cover False_Negative errors of my outlier-classifier-model). It gave me about 1.8-2 RMSE error on CV. Howerer, on LB i've got very bad results - about 4.2 RMSE.
\n\nAssumtions and conclusions about results:\n1) There are some merchants that have very few visits and *merchant_outlier_mean*=0 according to train, so the card_id that is actualy outlier in test can visit that \"non-outlier-merchant\".\n2) I have classified pretty bad if *merchant_outlier_mean>0* or not. And this is a big reason for bad results in LB.\n3) There are a lot of outliers in train that visited outlier-merchants according to train, and there are very-very few outliers from test that visited the same outlier-merchants according to train. That fact can be explained by the first assumption(conclusion) - There are a lot merchants that have very-very few visits (<5 for example) and *merchant_outlier_mean*>0 (0.8 for example) according to train, so the card_id that is actualy non-outlier in test can visit that \"outlier-merchant\" according to train and the model mark that test non-outlier as outlier and as a result the numbers of False_Positive and False_Negative are growing in test.
\n\nI'm very upset with my results and not being on the top of LB after a lot of work and sleepless nights (planning how i'll spend 20000$ after winning the competition) xD. Soi decided to write a kernel during a week (if i would have enough time) that will cover all my notes - EDA, FE and so on. But now i've just writen the topic to share my main results.
\n\nFeel free to upvote! Leave your comments bellow. Good luck and best wishes on LB)
", "rawMarkdown": "Hello everyone! Here i'd like to share my 2-3 weeks experience about outlier predictions.\n\n2-3 weeks ago i decided that the main key for cracking this competion is a pretty good prediction of outliers. That means that False_Positive + False_Negative should be less than the number of outliers as much as possible - in other words less than aproximatly 2200 outliers on train.\n\nAssuming that distribution of train and test is aproximatly the same, so i concluded that proportion of outliers would be aproximatly the same - 2200 outlier in train and 1000-1200 outliers in test.\n\nI figured out that visiting specific merchants in historical_transactions is strongly correlated with outliers (I also figured out that the last and the first date of visiting specific merchants by specific card_id is also influenced on the outliers but less). So I calculated *merchant_outlier_mean* - the mean uniquly visited outlier for every merchant in historical_transactions (not new_merchant_transactions) and then i got the mean of *merchant_outlier_mean* for every card_id and that feature had about abs(0.5) correletion by Pearson.\n\nBy the way, there are a lot of NOT outliers, which didn't make any transactions in new_merchant_transactions. So i denied the hypothesis that outlers are the card_ids that didn't make any transactions according to Elo recommendation algorithm.\n\nAfter that I assumed that more a card_id visits \"bad\" merchants - more chance that the card_id is outlier. However there are about 33000 merchants that the card_ids from train never visit, but the card_ids from test visit ones. So I have to make a classifier to predict if a merchant have been ever visited by an outlier to predict further *merchant_outlier_mean* for these merchants. An accuracy of that classifier was pretty bad, but regressor that predict *merchant_outlier_mean* worked very well (0.005 MAE), but i fited regressor on *merchant_outlier_mean>0*, so if the classifier predicts for *merchant_outlier_mean=0* examples that *merchant_outlier_mean>0* for very few visiting merchants, regressor could predict *merchant_outlier_mean* close to 1 or 0.5.\n\nAfter that, i made a classifier that predicts about 600 False_Positive+False_Negative on all train. The model was very confident about its own predictions. CV gave me the pretty close results as for training fold. \nOn test.csv it predicted about 600 card_id as outliers (not 1000-1200 according to my previous assumtion at the beginning of the topic). After, i combined my outllier model with the very simple model fitted with outliers ( i fitted with ouliers to cover False_Negative errors of my outlier-classifier-model). It gave me about 1.8-2 RMSE error on CV. Howerer, on LB i've got very bad results - about 4.2 RMSE.\n\nAssumtions and conclusions about results:\n1) There are some merchants that have very few visits and *merchant_outlier_mean*=0 according to train, so the card_id that is actualy outlier in test can visit that \"non-outlier-merchant\".\n2) I have classified pretty bad if *merchant_outlier_mean>0* or not. And this is a big reason for bad results in LB.\n3) There are a lot of outliers in train that visited *outlier-merchants* according to train, and there are very-very few outliers from test that visited the same *outlier-merchants* according to train. That fact can be explained by the first assumption(conclusion) - There are a lot merchants that have very-very few visits (<5 for example) and *merchant_outlier_mean*>0 (0.8 for example) according to train, so the card_id that is actualy *non-outlier* in test can visit that \"outlier-merchant\" according to train and the model mark that test *non-outlier* as outlier and as a result the numbers of False_Positive and False_Negative are growing in test.\n\nI'm very upset with my results and not being on the top of LB after a lot of work and sleepless nights (planning how i'll spend 20000$ after winning the competition) xD. Soi decided to write a kernel during a week (if i would have enough time) that will cover all my notes - EDA, FE and so on. But now i've just writen the topic to share my main results.\n\nFeel free to upvote! Leave your comments bellow. Good luck and best wishes on LB)", "votes": null }, { "id": "465188", "postDate": "02/02/2019 15:20:32", "content": "You guy dive into a good direction. Best wishes!
", "rawMarkdown": "You guy dive into a good direction. Best wishes!", "votes": null }, { "id": "465259", "postDate": "02/02/2019 17:39:24", "content": "One thing that bothers me is that we discuss the issue of outliers as if it was noise, but it might be a part of the signal. We need to predict them it seems, but and regression based models optimizing for RMSE tend to smooth them out unless the model is heavily overfit. Would that mean that we are blindly trying to optimize a metric that causes the problem that we are facing?
", "rawMarkdown": "One thing that bothers me is that we discuss the issue of outliers as if it was noise, but it might be a part of the signal. We need to predict them it seems, but and regression based models optimizing for RMSE tend to smooth them out unless the model is heavily overfit. Would that mean that we are blindly trying to optimize a metric that causes the problem that we are facing?", "votes": null }, { "id": "465285", "postDate": "02/02/2019 18:19:17", "content": "I think that you have a nice piece of information to work, now it's time to concentrate on FE. Probably, you'll achieve better scores by simple take those best public kernels and apply or adapt your outlier treatment. I started to work on the outlier problem yesterday with a semi-supervised neural net encoder approach, get an improved ROC by doing that. Your highlights will certainly inspire my next steps. Thanks for sharing!
", "rawMarkdown": "I think that you have a nice piece of information to work, now it's time to concentrate on FE. Probably, you'll achieve better scores by simple take those best public kernels and apply or adapt your outlier treatment. I started to work on the outlier problem yesterday with a semi-supervised neural net encoder approach, get an improved ROC by doing that. Your highlights will certainly inspire my next steps. Thanks for sharing!", "votes": null }, { "id": "466999", "postDate": "02/06/2019 10:03:56", "content": "Just for understanding, when you were calculating merchantoutliermean, were you using card ids from test valuation set too?
", "rawMarkdown": "Just for understanding, when you were calculating merchantoutliermean, were you using card ids from test valuation set too?", "votes": null }, { "id": "467461", "postDate": "02/07/2019 07:06:37", "content": "No, I didn't use cardId from test, cause i don't know if cardId from test is outlier. But let me show you on one example - let's suppose, that one merchant has 0 outliers cardId of 9 cardId from train, and 1 cardId from test, and we don't know if that cardId from test is outlier or not. So, final merchantOutlierMean would be from 0 upto 0.1, depending on whether cardId from test is outlier or not. Then my model predict for that merchant_id merchantOutlierMean, and final value of merchantOutlierMean would be equal to min(max(predictedValue, 0), 0.1) for that specific example.
\n\nSoon I will have posted my kernel, where probably you find all the anwers for your possible questions about outliers and coresponding to them merchantId
", "rawMarkdown": "No, I didn't use cardId from test, cause i don't know if cardId from test is outlier. But let me show you on one example - let's suppose, that one merchant has 0 outliers cardId of 9 cardId from train, and 1 cardId from test, and we don't know if that cardId from test is outlier or not. So, final merchantOutlierMean would be from 0 upto 0.1, depending on whether cardId from test is outlier or not. Then my model predict for that merchant_id merchantOutlierMean, and final value of merchantOutlierMean would be equal to min(max(predictedValue, 0), 0.1) for that specific example.\n\nSoon I will have posted my kernel, where probably you find all the anwers for your possible questions about outliers and coresponding to them merchantId", "votes": null }, { "id": "467595", "postDate": "02/07/2019 11:49:38", "content": "thanks for explanation! what I actually meant is wheter you used all of train data to calculate the merchantOutlierMean. Or if you have reserved a representable parf of train data (with known outliers) which wasn't used for merchantOutlierMean calculation and was used for valuation of trained model. \nIn my own model I reserved about 20000 card ids (210 outliers) and treat them as if they were from test set, so that no target information from this set can leak into the model. \nI meant to understood that your private RMSE was very low 1.8-2 as compared to LB rank, thats why I was wondering if some target info from valuation set leaked into model.
", "rawMarkdown": "thanks for explanation! what I actually meant is wheter you used all of train data to calculate the merchantOutlierMean. Or if you have reserved a representable parf of train data (with known outliers) which wasn't used for merchantOutlierMean calculation and was used for valuation of trained model. \nIn my own model I reserved about 20000 card ids (210 outliers) and treat them as if they were from test set, so that no target information from this set can leak into the model. \nI meant to understood that your private RMSE was very low 1.8-2 as compared to LB rank, thats why I was wondering if some target info from valuation set leaked into model.", "votes": null }, { "id": "467610", "postDate": "02/07/2019 12:16:52", "content": "Yeah, I used all of train data to calculate the merchantOutlierMean. And probably you are right that there is some leak in validation, although i didn't find it. Probably after posting my kernel (in 1-2 days) you can point out on leak, thanks
", "rawMarkdown": "Yeah, I used all of train data to calculate the merchantOutlierMean. And probably you are right that there is some leak in validation, although i didn't find it. Probably after posting my kernel (in 1-2 days) you can point out on leak, thanks", "votes": null }, { "id": "467617", "postDate": "02/07/2019 12:27:43", "content": "If you use all train data for merchantOutlierMean, that is where target data from valuation set leaks. It would pass target data from valuation set to your model, but this cant be done with real test data, since there is no target info and explains the difference in RMSE. I suggest first to split train data into train/valuation sets and forget/wipe out the target info on valuation set, when it is used for merchantOutlierMean calculation. That should align private score and LB
", "rawMarkdown": "If you use all train data for merchantOutlierMean, that is where target data from valuation set leaks. It would pass target data from valuation set to your model, but this cant be done with real test data, since there is no target info and explains the difference in RMSE. I suggest first to split train data into train/valuation sets and forget/wipe out the target info on valuation set, when it is used for merchantOutlierMean calculation. That should align private score and LB", "votes": null }, { "id": "467628", "postDate": "02/07/2019 12:43:26", "content": "while predicting outliers, i don't use target (i mean target column values from train.csv). Neither in train nor validation.
", "rawMarkdown": "while predicting outliers, i don't use target (i mean target column values from train.csv). Neither in train nor validation.", "votes": null }, { "id": "467632", "postDate": "02/07/2019 12:56:10", "content": "but merchantOutlierMean is probably used for outlier predictions, and target information is contained in merchantOutlierMean
", "rawMarkdown": "but merchantOutlierMean is probably used for outlier predictions, and target information is contained in merchantOutlierMean", "votes": null }, { "id": "467642", "postDate": "02/07/2019 13:17:37", "content": "```\ndef merge_two_df(df1, df2, left_on, right_on):\n return pd.merge(df1, df2, how='left', left_on=left_on, right_on=right_on)
\n\ntrain['outlier'] = (train['target']<-19)*1\n_ = df_hist_trans[df_hist_trans['card_id'].isin(train['card_id'])].groupby(['card_id', 'merchant_id'])['outlier'].max().reset_index()\n_ = merge_two_df(_, _.groupby('merchant_id')['outlier'].mean().reset_index().rename(columns={'outlier': 'outlier_merch_mean'}), ['merchant_id'], ['merchant_id'])\ntrain = merge_two_df(train, _.groupby('card_id')['outlier_merch_mean'].mean().reset_index(), ['card_id'], ['card_id'])\n```\nThat's how I calculated merchantOutlierMean. Outlier - if target is only equal to about -33,..., It's a binary and while calculating outlier_merch_mean, i didn't use 'target' as you can see from the code above
", "rawMarkdown": "```\ndef merge_two_df(df1, df2, left_on, right_on):\n return pd.merge(df1, df2, how='left', left_on=left_on, right_on=right_on)\n\n\ntrain['outlier'] = (train['target']<-19)*1\n_ = df_hist_trans[df_hist_trans['card_id'].isin(train['card_id'])].groupby(['card_id', 'merchant_id'])['outlier'].max().reset_index()\n_ = merge_two_df(_, _.groupby('merchant_id')['outlier'].mean().reset_index().rename(columns={'outlier': 'outlier_merch_mean'}), ['merchant_id'], ['merchant_id'])\ntrain = merge_two_df(train, _.groupby('card_id')['outlier_merch_mean'].mean().reset_index(), ['card_id'], ['card_id'])\n```\nThat's how I calculated merchantOutlierMean. Outlier - if target is only equal to about -33,..., It's a binary and while calculating outlier_merch_mean, i didn't use 'target' as you can see from the code above", "votes": null }, { "id": "467647", "postDate": "02/07/2019 13:26:51", "content": "here outlier information contained in target column is used for computing train[outlier] column and then merchantOutlierMean. Clear not all target info is used, just the outlier info. But this is also main goal of training objective. And outlier info from valuation set also goes into calculation of merchantOutlierMean.\nOne could exclude valuation id from calculation by something like
\n\ntrain['outlier']=(train['target']<-19. and ~train['card_id'].isin(val_set))*1\nIf u use something like target encoding, u should pay carefully about data leakage, try to use CV to generate the variable that encoded outlier information, and add some noises to merchants that have little transactions.
", "rawMarkdown": "If u use something like target encoding, u should pay carefully about data leakage, try to use CV to generate the variable that encoded outlier information, and add some noises to merchants that have little transactions.", "votes": null }, { "id": "471108", "postDate": "02/14/2019 03:31:09", "content": "@eb1c0n what are the \"bad merchants\" and \"good merchants\", curious.
", "rawMarkdown": "eb1c0n what are the \"bad merchants\" and \"good merchants\", curious.", "votes": null } ], "comments": [ { "id": 465188, "author_name": "codlife", "author_url": "", "post_date": "02/02/2019 15:20:32", "content": "You guy dive into a good direction. Best wishes!
", "votes": null, "replies": [] }, { "id": 465259, "author_name": "calabist", "author_url": "", "post_date": "02/02/2019 17:39:24", "content": "One thing that bothers me is that we discuss the issue of outliers as if it was noise, but it might be a part of the signal. We need to predict them it seems, but and regression based models optimizing for RMSE tend to smooth them out unless the model is heavily overfit. Would that mean that we are blindly trying to optimize a metric that causes the problem that we are facing?
", "votes": null, "replies": [] }, { "id": 465285, "author_name": "camaradaancap", "author_url": "", "post_date": "02/02/2019 18:19:17", "content": "I think that you have a nice piece of information to work, now it's time to concentrate on FE. Probably, you'll achieve better scores by simple take those best public kernels and apply or adapt your outlier treatment. I started to work on the outlier problem yesterday with a semi-supervised neural net encoder approach, get an improved ROC by doing that. Your highlights will certainly inspire my next steps. Thanks for sharing!
", "votes": null, "replies": [] }, { "id": 466999, "author_name": "sergejkoenig", "author_url": "", "post_date": "02/06/2019 10:03:56", "content": "Just for understanding, when you were calculating merchantoutliermean, were you using card ids from test valuation set too?
", "votes": null, "replies": [ { "id": 467461, "author_name": "peacemaket", "author_url": "", "post_date": "02/07/2019 07:06:37", "content": "No, I didn't use cardId from test, cause i don't know if cardId from test is outlier. But let me show you on one example - let's suppose, that one merchant has 0 outliers cardId of 9 cardId from train, and 1 cardId from test, and we don't know if that cardId from test is outlier or not. So, final merchantOutlierMean would be from 0 upto 0.1, depending on whether cardId from test is outlier or not. Then my model predict for that merchant_id merchantOutlierMean, and final value of merchantOutlierMean would be equal to min(max(predictedValue, 0), 0.1) for that specific example.
\n\nSoon I will have posted my kernel, where probably you find all the anwers for your possible questions about outliers and coresponding to them merchantId
", "votes": null, "replies": [] }, { "id": 467595, "author_name": "sergejkoenig", "author_url": "", "post_date": "02/07/2019 11:49:38", "content": "thanks for explanation! what I actually meant is wheter you used all of train data to calculate the merchantOutlierMean. Or if you have reserved a representable parf of train data (with known outliers) which wasn't used for merchantOutlierMean calculation and was used for valuation of trained model. \nIn my own model I reserved about 20000 card ids (210 outliers) and treat them as if they were from test set, so that no target information from this set can leak into the model. \nI meant to understood that your private RMSE was very low 1.8-2 as compared to LB rank, thats why I was wondering if some target info from valuation set leaked into model.
", "votes": null, "replies": [] }, { "id": 467610, "author_name": "peacemaket", "author_url": "", "post_date": "02/07/2019 12:16:52", "content": "Yeah, I used all of train data to calculate the merchantOutlierMean. And probably you are right that there is some leak in validation, although i didn't find it. Probably after posting my kernel (in 1-2 days) you can point out on leak, thanks
", "votes": null, "replies": [] }, { "id": 467617, "author_name": "sergejkoenig", "author_url": "", "post_date": "02/07/2019 12:27:43", "content": "If you use all train data for merchantOutlierMean, that is where target data from valuation set leaks. It would pass target data from valuation set to your model, but this cant be done with real test data, since there is no target info and explains the difference in RMSE. I suggest first to split train data into train/valuation sets and forget/wipe out the target info on valuation set, when it is used for merchantOutlierMean calculation. That should align private score and LB
", "votes": null, "replies": [] }, { "id": 467628, "author_name": "peacemaket", "author_url": "", "post_date": "02/07/2019 12:43:26", "content": "while predicting outliers, i don't use target (i mean target column values from train.csv). Neither in train nor validation.
", "votes": null, "replies": [] }, { "id": 467632, "author_name": "sergejkoenig", "author_url": "", "post_date": "02/07/2019 12:56:10", "content": "but merchantOutlierMean is probably used for outlier predictions, and target information is contained in merchantOutlierMean
", "votes": null, "replies": [] }, { "id": 467642, "author_name": "peacemaket", "author_url": "", "post_date": "02/07/2019 13:17:37", "content": "```\ndef merge_two_df(df1, df2, left_on, right_on):\n return pd.merge(df1, df2, how='left', left_on=left_on, right_on=right_on)
\n\ntrain['outlier'] = (train['target']<-19)*1\n_ = df_hist_trans[df_hist_trans['card_id'].isin(train['card_id'])].groupby(['card_id', 'merchant_id'])['outlier'].max().reset_index()\n_ = merge_two_df(_, _.groupby('merchant_id')['outlier'].mean().reset_index().rename(columns={'outlier': 'outlier_merch_mean'}), ['merchant_id'], ['merchant_id'])\ntrain = merge_two_df(train, _.groupby('card_id')['outlier_merch_mean'].mean().reset_index(), ['card_id'], ['card_id'])\n```\nThat's how I calculated merchantOutlierMean. Outlier - if target is only equal to about -33,..., It's a binary and while calculating outlier_merch_mean, i didn't use 'target' as you can see from the code above
", "votes": null, "replies": [] }, { "id": 467647, "author_name": "sergejkoenig", "author_url": "", "post_date": "02/07/2019 13:26:51", "content": "here outlier information contained in target column is used for computing train[outlier] column and then merchantOutlierMean. Clear not all target info is used, just the outlier info. But this is also main goal of training objective. And outlier info from valuation set also goes into calculation of merchantOutlierMean.\nOne could exclude valuation id from calculation by something like
\n\ntrain['outlier']=(train['target']<-19. and ~train['card_id'].isin(val_set))*1\nIf u use something like target encoding, u should pay carefully about data leakage, try to use CV to generate the variable that encoded outlier information, and add some noises to merchants that have little transactions.
", "votes": null, "replies": [] }, { "id": 471108, "author_name": "liwenhuang", "author_url": "", "post_date": "02/14/2019 03:31:09", "content": "@eb1c0n what are the \"bad merchants\" and \"good merchants\", curious.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "465137": "Hello everyone! Here i'd like to share my 2-3 weeks experience about outlier predictions.\n\n2-3 weeks ago i decided that the main key for cracking this competion is a pretty good prediction of outliers. That means that False_Positive + False_Negative should be less than the number of outliers as much as possible - in other words less than aproximatly 2200 outliers on train.\n\nAssuming that distribution of train and test is aproximatly the same, so i concluded that proportion of outliers would be aproximatly the same - 2200 outlier in train and 1000-1200 outliers in test.\n\nI figured out that visiting specific merchants in historical_transactions is strongly correlated with outliers (I also figured out that the last and the first date of visiting specific merchants by specific card_id is also influenced on the outliers but less). So I calculated *merchant_outlier_mean* - the mean uniquly visited outlier for every merchant in historical_transactions (not new_merchant_transactions) and then i got the mean of *merchant_outlier_mean* for every card_id and that feature had about abs(0.5) correletion by Pearson.\n\nBy the way, there are a lot of NOT outliers, which didn't make any transactions in new_merchant_transactions. So i denied the hypothesis that outlers are the card_ids that didn't make any transactions according to Elo recommendation algorithm.\n\nAfter that I assumed that more a card_id visits \"bad\" merchants - more chance that the card_id is outlier. However there are about 33000 merchants that the card_ids from train never visit, but the card_ids from test visit ones. So I have to make a classifier to predict if a merchant have been ever visited by an outlier to predict further *merchant_outlier_mean* for these merchants. An accuracy of that classifier was pretty bad, but regressor that predict *merchant_outlier_mean* worked very well (0.005 MAE), but i fited regressor on *merchant_outlier_mean>0*, so if the classifier predicts for *merchant_outlier_mean=0* examples that *merchant_outlier_mean>0* for very few visiting merchants, regressor could predict *merchant_outlier_mean* close to 1 or 0.5.\n\nAfter that, i made a classifier that predicts about 600 False_Positive+False_Negative on all train. The model was very confident about its own predictions. CV gave me the pretty close results as for training fold. \nOn test.csv it predicted about 600 card_id as outliers (not 1000-1200 according to my previous assumtion at the beginning of the topic). After, i combined my outllier model with the very simple model fitted with outliers ( i fitted with ouliers to cover False_Negative errors of my outlier-classifier-model). It gave me about 1.8-2 RMSE error on CV. Howerer, on LB i've got very bad results - about 4.2 RMSE.\n\nAssumtions and conclusions about results:\n1) There are some merchants that have very few visits and *merchant_outlier_mean*=0 according to train, so the card_id that is actualy outlier in test can visit that \"non-outlier-merchant\".\n2) I have classified pretty bad if *merchant_outlier_mean>0* or not. And this is a big reason for bad results in LB.\n3) There are a lot of outliers in train that visited *outlier-merchants* according to train, and there are very-very few outliers from test that visited the same *outlier-merchants* according to train. That fact can be explained by the first assumption(conclusion) - There are a lot merchants that have very-very few visits (<5 for example) and *merchant_outlier_mean*>0 (0.8 for example) according to train, so the card_id that is actualy *non-outlier* in test can visit that \"outlier-merchant\" according to train and the model mark that test *non-outlier* as outlier and as a result the numbers of False_Positive and False_Negative are growing in test.\n\nI'm very upset with my results and not being on the top of LB after a lot of work and sleepless nights (planning how i'll spend 20000$ after winning the competition) xD. Soi decided to write a kernel during a week (if i would have enough time) that will cover all my notes - EDA, FE and so on. But now i've just writen the topic to share my main results.\n\nFeel free to upvote! Leave your comments bellow. Good luck and best wishes on LB)", "465188": "You guy dive into a good direction. Best wishes!", "465259": "One thing that bothers me is that we discuss the issue of outliers as if it was noise, but it might be a part of the signal. We need to predict them it seems, but and regression based models optimizing for RMSE tend to smooth them out unless the model is heavily overfit. Would that mean that we are blindly trying to optimize a metric that causes the problem that we are facing?", "465285": "I think that you have a nice piece of information to work, now it's time to concentrate on FE. Probably, you'll achieve better scores by simple take those best public kernels and apply or adapt your outlier treatment. I started to work on the outlier problem yesterday with a semi-supervised neural net encoder approach, get an improved ROC by doing that. Your highlights will certainly inspire my next steps. Thanks for sharing!", "466999": "Just for understanding, when you were calculating merchantoutliermean, were you using card ids from test valuation set too?", "467461": "No, I didn't use cardId from test, cause i don't know if cardId from test is outlier. But let me show you on one example - let's suppose, that one merchant has 0 outliers cardId of 9 cardId from train, and 1 cardId from test, and we don't know if that cardId from test is outlier or not. So, final merchantOutlierMean would be from 0 upto 0.1, depending on whether cardId from test is outlier or not. Then my model predict for that merchant_id merchantOutlierMean, and final value of merchantOutlierMean would be equal to min(max(predictedValue, 0), 0.1) for that specific example.\n\nSoon I will have posted my kernel, where probably you find all the anwers for your possible questions about outliers and coresponding to them merchantId", "467595": "thanks for explanation! what I actually meant is wheter you used all of train data to calculate the merchantOutlierMean. Or if you have reserved a representable parf of train data (with known outliers) which wasn't used for merchantOutlierMean calculation and was used for valuation of trained model. \nIn my own model I reserved about 20000 card ids (210 outliers) and treat them as if they were from test set, so that no target information from this set can leak into the model. \nI meant to understood that your private RMSE was very low 1.8-2 as compared to LB rank, thats why I was wondering if some target info from valuation set leaked into model.", "467610": "Yeah, I used all of train data to calculate the merchantOutlierMean. And probably you are right that there is some leak in validation, although i didn't find it. Probably after posting my kernel (in 1-2 days) you can point out on leak, thanks", "467617": "If you use all train data for merchantOutlierMean, that is where target data from valuation set leaks. It would pass target data from valuation set to your model, but this cant be done with real test data, since there is no target info and explains the difference in RMSE. I suggest first to split train data into train/valuation sets and forget/wipe out the target info on valuation set, when it is used for merchantOutlierMean calculation. That should align private score and LB", "467628": "while predicting outliers, i don't use target (i mean target column values from train.csv). Neither in train nor validation.", "467632": "but merchantOutlierMean is probably used for outlier predictions, and target information is contained in merchantOutlierMean", "467642": "```\ndef merge_two_df(df1, df2, left_on, right_on):\n return pd.merge(df1, df2, how='left', left_on=left_on, right_on=right_on)\n\n\ntrain['outlier'] = (train['target']<-19)*1\n_ = df_hist_trans[df_hist_trans['card_id'].isin(train['card_id'])].groupby(['card_id', 'merchant_id'])['outlier'].max().reset_index()\n_ = merge_two_df(_, _.groupby('merchant_id')['outlier'].mean().reset_index().rename(columns={'outlier': 'outlier_merch_mean'}), ['merchant_id'], ['merchant_id'])\ntrain = merge_two_df(train, _.groupby('card_id')['outlier_merch_mean'].mean().reset_index(), ['card_id'], ['card_id'])\n```\nThat's how I calculated merchantOutlierMean. Outlier - if target is only equal to about -33,..., It's a binary and while calculating outlier_merch_mean, i didn't use 'target' as you can see from the code above", "467647": "here outlier information contained in target column is used for computing train[outlier] column and then merchantOutlierMean. Clear not all target info is used, just the outlier info. But this is also main goal of training objective. And outlier info from valuation set also goes into calculation of merchantOutlierMean.\nOne could exclude valuation id from calculation by something like\n\n train['outlier']=(train['target']<-19. and ~train['card_id'].isin(val_set))*1", "469226": "If u use something like target encoding, u should pay carefully about data leakage, try to use CV to generate the variable that encoded outlier information, and add some noises to merchants that have little transactions.", "471108": "eb1c0n what are the \"bad merchants\" and \"good merchants\", curious." }, "source": "meta" }