Thanks to the organizers of the Categorical Feature Encoding Challenge II competition, and to the generous sharing and discussion between the kagglers, which helped me a lot.\nI will try to describe my entire competition process:
\n\nWhen I first participated in this competition, I saw many participants said that simple LogisticRegression works well, so I used the sklearn.linear_model.LogisticRegression model, used hyperopt for hyperparameter search, and then trained the model in 40-fold As a baseline. Later tried catboost, but the effect was not good. I found that in the data set, the discrete classification features accounted for a large proportion, because I was used Wide & deep and DeepFM in other competitions, so I think to use some deep learning for CTR may have better results, and it turned out to be a correct choice.
\n\nPart of the training process I used deepctr package, source:
\n- https://github.com/shenweichen/DeepCTR
\nYou can install it using pip install deepctr
FE: I didn't focus too much on feature engineering, so I just made some normal FE.
\n\npython\ndata_df['num_null'] = data_df.isna().sum(axis=1)\ndata_df['ord_5_0'] = data_df['ord_5'].str[0]\ndata_df['ord_5_1'] = data_df['ord_5'].str[1]\n
At this stage, I rewritten LogisticRegression(batch_size = 8192, optimizer = 'Adam', regularizers = l2) with TensorFlow as the new baseline.
\n\nAfter that, I tried a variety of CTR models.If there is no special instruction, I use hyperopt to perform 5-fold CV on the training set for hyperparameter search, and select the 5 best performing hyperparameter. Finally use 40-fold CV on training set for each hyperparameter and take the average, which is the final score.
\n\nFM: FM without linear, score: 0.78661
\nDeepFM: Use the best 10 hyperparameter combinations, 20-fold cross-validation, score: 0.78922. Thanks to @Siavash wonderful kernel-https://www.kaggle.com/siavrez/deepfm-model, Your idea of using Mish and CyclicLR in training deepfm has greatly influenced me.\nCIN(XdeepFM): Without Deep and Linear component, score: 0.78940
\nCrossNet(Deep&Cross): Without Deep and Linear component, score: 0.78888
\nafm: score: 0.78817
\nautoint: score: 0.78926
The reason why I do not use the Deep and linear component in CIN and Cross is because I want to see the performance of these two modules separately, and I want to stacking the models that perform well in the future, and stacking prefers excellent and Different models.
\n\nIt can be seen that CIN performs best on the training set. After I submitted these results to the LB, I verified my judgment. But it is interesting that when I stacking all models, although the scores on the training set increased, but the score on LB dropped significantly. Considering the possible risk of overfitting, I abandoned the idea of stacking.
\n\nSo what should I trust? The cross-validation score on the training set or the Public Leaderboard?
\n\nI carefully analyzed the competition dataset. The training set has 600k rows of data and the test set has 400k rows of data. The PLB only uses 25% of the test data, which is 100k rows. When I use 5-fold cross-validation, the validation dataset for each fold is 120k rows, which is closer to the amount of data used by the PLB, and the validation score between different folds of the same model often jumps between [0.790 and 0.787]. the variance is relatively large. The most important thing is that when using the same kFold iterator (fixed KFold random seed and the number of fold, shuffle=True), the validation scores of different models have correlation, which means that some samples in the data set are difficult to train on any model, even It is impossible to train. According to this, I guess that the reason why the score of the PLB is lower than the k-fold CV is likely to be that the proportion of this part of the \"difficult sample\" in the PLB is slightly more. The total number of test data rows is similar to the train data, so the final result should be closer to the score of the k-fold CV on training set.
\n\nFinally, I stacking CIN and xdeepfm, and improved the score on training set and PLB. The score in PLB increased to 0.78658(rank 84). But then I believe that the final result will be closer to the score of the train set. It turns out that this guess should correct.
", "messages": [ { "id": "794105", "postDate": "04/01/2020 15:04:34", "content": "Thanks to the organizers of the Categorical Feature Encoding Challenge II competition, and to the generous sharing and discussion between the kagglers, which helped me a lot.\nI will try to describe my entire competition process:
\n\nWhen I first participated in this competition, I saw many participants said that simple LogisticRegression works well, so I used the sklearn.linear_model.LogisticRegression model, used hyperopt for hyperparameter search, and then trained the model in 40-fold As a baseline. Later tried catboost, but the effect was not good. I found that in the data set, the discrete classification features accounted for a large proportion, because I was used Wide & deep and DeepFM in other competitions, so I think to use some deep learning for CTR may have better results, and it turned out to be a correct choice.
\n\nPart of the training process I used deepctr package, source:
\n- https://github.com/shenweichen/DeepCTR
\nYou can install it using pip install deepctr
FE: I didn't focus too much on feature engineering, so I just made some normal FE.
\n\npython\ndata_df['num_null'] = data_df.isna().sum(axis=1)\ndata_df['ord_5_0'] = data_df['ord_5'].str[0]\ndata_df['ord_5_1'] = data_df['ord_5'].str[1]\n
At this stage, I rewritten LogisticRegression(batch_size = 8192, optimizer = 'Adam', regularizers = l2) with TensorFlow as the new baseline.
\n\nAfter that, I tried a variety of CTR models.If there is no special instruction, I use hyperopt to perform 5-fold CV on the training set for hyperparameter search, and select the 5 best performing hyperparameter. Finally use 40-fold CV on training set for each hyperparameter and take the average, which is the final score.
\n\nFM: FM without linear, score: 0.78661
\nDeepFM: Use the best 10 hyperparameter combinations, 20-fold cross-validation, score: 0.78922. Thanks to @Siavash wonderful kernel-https://www.kaggle.com/siavrez/deepfm-model, Your idea of using Mish and CyclicLR in training deepfm has greatly influenced me.\nCIN(XdeepFM): Without Deep and Linear component, score: 0.78940
\nCrossNet(Deep&Cross): Without Deep and Linear component, score: 0.78888
\nafm: score: 0.78817
\nautoint: score: 0.78926
The reason why I do not use the Deep and linear component in CIN and Cross is because I want to see the performance of these two modules separately, and I want to stacking the models that perform well in the future, and stacking prefers excellent and Different models.
\n\nIt can be seen that CIN performs best on the training set. After I submitted these results to the LB, I verified my judgment. But it is interesting that when I stacking all models, although the scores on the training set increased, but the score on LB dropped significantly. Considering the possible risk of overfitting, I abandoned the idea of stacking.
\n\nSo what should I trust? The cross-validation score on the training set or the Public Leaderboard?
\n\nI carefully analyzed the competition dataset. The training set has 600k rows of data and the test set has 400k rows of data. The PLB only uses 25% of the test data, which is 100k rows. When I use 5-fold cross-validation, the validation dataset for each fold is 120k rows, which is closer to the amount of data used by the PLB, and the validation score between different folds of the same model often jumps between [0.790 and 0.787]. the variance is relatively large. The most important thing is that when using the same kFold iterator (fixed KFold random seed and the number of fold, shuffle=True), the validation scores of different models have correlation, which means that some samples in the data set are difficult to train on any model, even It is impossible to train. According to this, I guess that the reason why the score of the PLB is lower than the k-fold CV is likely to be that the proportion of this part of the \"difficult sample\" in the PLB is slightly more. The total number of test data rows is similar to the train data, so the final result should be closer to the score of the k-fold CV on training set.
\n\nFinally, I stacking CIN and xdeepfm, and improved the score on training set and PLB. The score in PLB increased to 0.78658(rank 84). But then I believe that the final result will be closer to the score of the train set. It turns out that this guess should correct.
", "rawMarkdown": "Thanks to the organizers of the Categorical Feature Encoding Challenge II competition, and to the generous sharing and discussion between the kagglers, which helped me a lot.\nI will try to describe my entire competition process: \n\n###Start & early attempts \nWhen I first participated in this competition, I saw many participants said that simple LogisticRegression works well, so I used the sklearn.linear_model.LogisticRegression model, used hyperopt for hyperparameter search, and then trained the model in 40-fold As a baseline. Later tried catboost, but the effect was not good. I found that in the data set, the discrete classification features accounted for a large proportion, because I was used Wide & deep and DeepFM in other competitions, so I think to use some deep learning for CTR may have better results, and it turned out to be a correct choice. \n\n### Try CTR \nPart of the training process I used *deepctr* package, source: \n- https://github.com/shenweichen/DeepCTR \nYou can install it using **pip install deepctr** \n\n\n**FE:** I didn't focus too much on feature engineering, so I just made some normal FE.\n\n```python\ndata_df['num_null'] = data_df.isna().sum(axis=1)\ndata_df['ord_5_0'] = data_df['ord_5'].str[0]\ndata_df['ord_5_1'] = data_df['ord_5'].str[1]\n``` \n\nAt this stage, I rewritten LogisticRegression(batch_size = 8192, optimizer = 'Adam', regularizers = l2) with TensorFlow as the new baseline.\n\nAfter that, I tried a variety of CTR models.If there is no special instruction, I use *hyperopt* to perform 5-fold CV on the training set for hyperparameter search, and select the 5 best performing hyperparameter. Finally use 40-fold CV on training set for each hyperparameter and take the average, which is the final score. \n\n**FM:** FM without linear, score: 0.78661 \n**DeepFM:** Use the best 10 hyperparameter combinations, 20-fold cross-validation, score: 0.78922. Thanks to @Siavash wonderful kernel-https://www.kaggle.com/siavrez/deepfm-model, Your idea of using Mish and CyclicLR in training deepfm has greatly influenced me.\n**CIN(XdeepFM):** Without Deep and Linear component, score: 0.78940 \n**CrossNet(Deep&Cross):** Without Deep and Linear component, score: 0.78888 \n**afm:** score: 0.78817 \n**autoint:** score: 0.78926 \n\nThe reason why I do not use the Deep and linear component in CIN and Cross is because I want to see the performance of these two modules separately, and I want to stacking the models that perform well in the future, and stacking prefers excellent and Different models. \n\nIt can be seen that CIN performs best on the training set. After I submitted these results to the LB, I verified my judgment. But it is interesting that when I stacking all models, although the scores on the training set increased, but the score on LB dropped significantly. Considering the possible risk of overfitting, I abandoned the idea of stacking. \n\n\n### Focus on xDeepFM \n- https://arxiv.org/abs/1803.05170\nI started to focus on training xdeepfm. By the same method as the period, I got a better score: 0.78955, but the problem followed. we can see that xDeepFM performs best on the training set, but when I put the results to the LB, the score did not rise but fell.\n\nSo what should I trust? The cross-validation score on the training set or the Public Leaderboard? \n\nI carefully analyzed the competition dataset. The training set has 600k rows of data and the test set has 400k rows of data. The PLB only uses 25% of the test data, which is 100k rows. When I use 5-fold cross-validation, the validation dataset for each fold is 120k rows, which is closer to the amount of data used by the PLB, and the validation score between different folds of the same model often jumps between [0.790 and 0.787]. the variance is relatively large. The most important thing is that when using the same kFold iterator (fixed KFold random seed and the number of fold, shuffle=True), the validation scores of different models have correlation, which means that some samples in the data set are difficult to train on any model, even It is impossible to train. According to this, I guess that the reason why the score of the PLB is lower than the k-fold CV is likely to be that the proportion of this part of the \"difficult sample\" in the PLB is slightly more. The total number of test data rows is similar to the train data, so the final result should be closer to the score of the k-fold CV on training set. \n\nFinally, I stacking CIN and xdeepfm, and improved the score on training set and PLB. The score in PLB increased to 0.78658(rank 84). But then I believe that the final result will be closer to the score of the train set. It turns out that this guess should correct.", "votes": null }, { "id": "794135", "postDate": "04/01/2020 15:34:53", "content": "A very nice Approach. What's is really xdeep-FM is It Deep learning or Ml approach?
", "rawMarkdown": "A very nice Approach. What's is really xdeep-FM is It Deep learning or Ml approach?", "votes": null }, { "id": "794143", "postDate": "04/01/2020 15:44:10", "content": "Thanks, xdeepfm is a state-of-the-art deep learning model
", "rawMarkdown": "Thanks, xdeepfm is a state-of-the-art deep learning model", "votes": null }, { "id": "794338", "postDate": "04/01/2020 18:37:46", "content": "Thank you for sharing. Really good thought process. I have a question - why did you consider splitting ord5 feature into two separate features but didn't do that with any other feature in the data?
", "rawMarkdown": "Thank you for sharing. Really good thought process. I have a question - why did you consider splitting ord5 feature into two separate features but didn't do that with any other feature in the data?", "votes": null }, { "id": "794685", "postDate": "04/02/2020 02:26:13", "content": "Because ord5 is a concatenation of two letters, I have doubted whether it is something like region code, but I did a word frequency analysis on it and found that no certain letter-combination is significant, so I simply split it, which can reduce the dimensional problems after onehot (I think Embedding in deep learning models is also a kind of onehot ), and let the deep model learn the interaction between them. The reason why I don’t split features such as nom_8 is because values like '7ca8775da' also appear on well-known datasets such as the 'Avazu' dataset. So I'm almost sure that this is a way to anonymize the data. A value like ‘7ca8775da’ can hardly contain other information besides the category, so I didn’t split it.
", "rawMarkdown": "Because ord5 is a concatenation of two letters, I have doubted whether it is something like region code, but I did a word frequency analysis on it and found that no certain letter-combination is significant, so I simply split it, which can reduce the dimensional problems after onehot (I think Embedding in deep learning models is also a kind of onehot ), and let the deep model learn the interaction between them. The reason why I don’t split features such as nom_8 is because values like '7ca8775da' also appear on well-known datasets such as the 'Avazu' dataset. So I'm almost sure that this is a way to anonymize the data. A value like ‘7ca8775da’ can hardly contain other information besides the category, so I didn’t split it.", "votes": null }, { "id": "1827376", "postDate": "06/21/2022 02:40:52", "content": "老哥，CrossNet，有代码分享吗?
", "rawMarkdown": "老哥，CrossNet，有代码分享吗?", "votes": null } ], "comments": [ { "id": 1827376, "author_name": "kgxiao", "author_url": "", "post_date": "06/21/2022 02:40:52", "content": "老哥，CrossNet，有代码分享吗?
", "votes": null, "replies": [] }, { "id": 794135, "author_name": "nasere", "author_url": "", "post_date": "04/01/2020 15:34:53", "content": "A very nice Approach. What's is really xdeep-FM is It Deep learning or Ml approach?
", "votes": null, "replies": [ { "id": 794143, "author_name": "leechh", "author_url": "", "post_date": "04/01/2020 15:44:10", "content": "Thanks, xdeepfm is a state-of-the-art deep learning model
", "votes": null, "replies": [] } ] }, { "id": 794338, "author_name": "hsawhne", "author_url": "", "post_date": "04/01/2020 18:37:46", "content": "Thank you for sharing. Really good thought process. I have a question - why did you consider splitting ord5 feature into two separate features but didn't do that with any other feature in the data?
", "votes": null, "replies": [ { "id": 794685, "author_name": "leechh", "author_url": "", "post_date": "04/02/2020 02:26:13", "content": "Because ord5 is a concatenation of two letters, I have doubted whether it is something like region code, but I did a word frequency analysis on it and found that no certain letter-combination is significant, so I simply split it, which can reduce the dimensional problems after onehot (I think Embedding in deep learning models is also a kind of onehot ), and let the deep model learn the interaction between them. The reason why I don’t split features such as nom_8 is because values like '7ca8775da' also appear on well-known datasets such as the 'Avazu' dataset. So I'm almost sure that this is a way to anonymize the data. A value like ‘7ca8775da’ can hardly contain other information besides the category, so I didn’t split it.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "794105": "Thanks to the organizers of the Categorical Feature Encoding Challenge II competition, and to the generous sharing and discussion between the kagglers, which helped me a lot.\nI will try to describe my entire competition process: \n\n###Start & early attempts \nWhen I first participated in this competition, I saw many participants said that simple LogisticRegression works well, so I used the sklearn.linear_model.LogisticRegression model, used hyperopt for hyperparameter search, and then trained the model in 40-fold As a baseline. Later tried catboost, but the effect was not good. I found that in the data set, the discrete classification features accounted for a large proportion, because I was used Wide & deep and DeepFM in other competitions, so I think to use some deep learning for CTR may have better results, and it turned out to be a correct choice. \n\n### Try CTR \nPart of the training process I used *deepctr* package, source: \n- https://github.com/shenweichen/DeepCTR \nYou can install it using **pip install deepctr** \n\n\n**FE:** I didn't focus too much on feature engineering, so I just made some normal FE.\n\n```python\ndata_df['num_null'] = data_df.isna().sum(axis=1)\ndata_df['ord_5_0'] = data_df['ord_5'].str[0]\ndata_df['ord_5_1'] = data_df['ord_5'].str[1]\n``` \n\nAt this stage, I rewritten LogisticRegression(batch_size = 8192, optimizer = 'Adam', regularizers = l2) with TensorFlow as the new baseline.\n\nAfter that, I tried a variety of CTR models.If there is no special instruction, I use *hyperopt* to perform 5-fold CV on the training set for hyperparameter search, and select the 5 best performing hyperparameter. Finally use 40-fold CV on training set for each hyperparameter and take the average, which is the final score. \n\n**FM:** FM without linear, score: 0.78661 \n**DeepFM:** Use the best 10 hyperparameter combinations, 20-fold cross-validation, score: 0.78922. Thanks to @Siavash wonderful kernel-https://www.kaggle.com/siavrez/deepfm-model, Your idea of using Mish and CyclicLR in training deepfm has greatly influenced me.\n**CIN(XdeepFM):** Without Deep and Linear component, score: 0.78940 \n**CrossNet(Deep&Cross):** Without Deep and Linear component, score: 0.78888 \n**afm:** score: 0.78817 \n**autoint:** score: 0.78926 \n\nThe reason why I do not use the Deep and linear component in CIN and Cross is because I want to see the performance of these two modules separately, and I want to stacking the models that perform well in the future, and stacking prefers excellent and Different models. \n\nIt can be seen that CIN performs best on the training set. After I submitted these results to the LB, I verified my judgment. But it is interesting that when I stacking all models, although the scores on the training set increased, but the score on LB dropped significantly. Considering the possible risk of overfitting, I abandoned the idea of stacking. \n\n\n### Focus on xDeepFM \n- https://arxiv.org/abs/1803.05170\nI started to focus on training xdeepfm. By the same method as the period, I got a better score: 0.78955, but the problem followed. we can see that xDeepFM performs best on the training set, but when I put the results to the LB, the score did not rise but fell.\n\nSo what should I trust? The cross-validation score on the training set or the Public Leaderboard? \n\nI carefully analyzed the competition dataset. The training set has 600k rows of data and the test set has 400k rows of data. The PLB only uses 25% of the test data, which is 100k rows. When I use 5-fold cross-validation, the validation dataset for each fold is 120k rows, which is closer to the amount of data used by the PLB, and the validation score between different folds of the same model often jumps between [0.790 and 0.787]. the variance is relatively large. The most important thing is that when using the same kFold iterator (fixed KFold random seed and the number of fold, shuffle=True), the validation scores of different models have correlation, which means that some samples in the data set are difficult to train on any model, even It is impossible to train. According to this, I guess that the reason why the score of the PLB is lower than the k-fold CV is likely to be that the proportion of this part of the \"difficult sample\" in the PLB is slightly more. The total number of test data rows is similar to the train data, so the final result should be closer to the score of the k-fold CV on training set. \n\nFinally, I stacking CIN and xdeepfm, and improved the score on training set and PLB. The score in PLB increased to 0.78658(rank 84). But then I believe that the final result will be closer to the score of the train set. It turns out that this guess should correct.", "794135": "A very nice Approach. What's is really xdeep-FM is It Deep learning or Ml approach?", "794143": "Thanks, xdeepfm is a state-of-the-art deep learning model", "794338": "Thank you for sharing. Really good thought process. I have a question - why did you consider splitting ord5 feature into two separate features but didn't do that with any other feature in the data?", "794685": "Because ord5 is a concatenation of two letters, I have doubted whether it is something like region code, but I did a word frequency analysis on it and found that no certain letter-combination is significant, so I simply split it, which can reduce the dimensional problems after onehot (I think Embedding in deep learning models is also a kind of onehot ), and let the deep model learn the interaction between them. The reason why I don’t split features such as nom_8 is because values like '7ca8775da' also appear on well-known datasets such as the 'Avazu' dataset. So I'm almost sure that this is a way to anonymize the data. A value like ‘7ca8775da’ can hardly contain other information besides the category, so I didn’t split it.", "1827376": "老哥，CrossNet，有代码分享吗?" }, "source": "meta" }