Hi All,
\n\nI am wondering Denoising AutoEncoders (DAE) are helpful or not in this competition? I have attempted here.
\n\nThey are mostly used for Computer Vision applications but one winning solution of previous Kaggle competition have used it for tabular data.
\n\nWhat kind of structure should one choose for it? And what kind of parameters we need to consider while choosing activations, # of hidden layers, dropouts (if any) of our DAE?
\n\nThe one which I have tried (link), # of layers, activations etc are chosen by trial and error. Some another method for it should be there.
\n\nThanks.
", "messages": [ { "id": "394820", "postDate": "09/27/2018 14:09:12", "content": "Hi All,
\n\nI am wondering Denoising AutoEncoders (DAE) are helpful or not in this competition? I have attempted here.
\n\nThey are mostly used for Computer Vision applications but one winning solution of previous Kaggle competition have used it for tabular data.
\n\nWhat kind of structure should one choose for it? And what kind of parameters we need to consider while choosing activations, # of hidden layers, dropouts (if any) of our DAE?
\n\nThe one which I have tried (link), # of layers, activations etc are chosen by trial and error. Some another method for it should be there.
\n\nThanks.
", "rawMarkdown": "Hi All,\n\nI am wondering Denoising AutoEncoders (DAE) are helpful or not in this competition? I have attempted [here][1].\n\nThey are mostly used for Computer Vision applications but one winning solution of previous Kaggle competition have used it for tabular data.\n\nWhat kind of structure should one choose for it? And what kind of parameters we need to consider while choosing activations, # of hidden layers, dropouts (if any) of our DAE?\n\nThe one which I have tried ([link][2]), # of layers, activations etc are chosen by trial and error. Some another method for it should be there.\n\nThanks. \n\n\n [1]: https://www.kaggle.com/prashantkikani/rstudio-humble-try-for-autoencoders\n [2]: https://www.kaggle.com/prashantkikani/rstudio-humble-try-for-autoencoders", "votes": null }, { "id": "394861", "postDate": "09/27/2018 15:11:33", "content": "I tried using one in an unsupervised fashion as a sort of anomaly detector. The AE returned an array the same dimensions of the input, which I subtracted row wise from the original data to get a score for each visitor. I then stacked the scores onto the original data and ran LightGBM. It didn't help. Still, I think there's something to it and will keep trying.
", "rawMarkdown": "I tried using one in an unsupervised fashion as a sort of anomaly detector. The AE returned an array the same dimensions of the input, which I subtracted row wise from the original data to get a score for each visitor. I then stacked the scores onto the original data and ran LightGBM. It didn't help. Still, I think there's something to it and will keep trying.", "votes": null }, { "id": "394954", "postDate": "09/27/2018 17:57:22", "content": "Same here. It didn't work.\nI think to get benefit from DAE, structure and parameters of it matters a lot. Also some good amount of computation power is required..
", "rawMarkdown": "Same here. It didn't work.\nI think to get benefit from DAE, structure and parameters of it matters a lot. Also some good amount of computation power is required..", "votes": null }, { "id": "395097", "postDate": "09/28/2018 00:49:49", "content": "I haven't tried DAE here yet, but I tried replicating the Porto Seguro winning solution without success. As far as I know, no one has been able to do so.\nI sincerely hope you are successful in your attempt!
", "rawMarkdown": "I haven't tried DAE here yet, but I tried replicating the Porto Seguro winning solution without success. As far as I know, no one has been able to do so.\nI sincerely hope you are successful in your attempt!", "votes": null }, { "id": "395131", "postDate": "09/28/2018 02:33:25", "content": "I understand the anomaly is the sample which has a very large revenue.\nDid your autoencoder detect anomalies(large revenue)?
", "rawMarkdown": "I understand the anomaly is the sample which has a very large revenue.\nDid your autoencoder detect anomalies(large revenue)?", "votes": null }, { "id": "395149", "postDate": "09/28/2018 03:20:47", "content": "I detected a good portion of the anomalies. However, I falsely identified many visitors with 0 revenue. I believe this often happens with imbalanced data when the algorithm underperforms.
", "rawMarkdown": "I detected a good portion of the anomalies. However, I falsely identified many visitors with 0 revenue. I believe this often happens with imbalanced data when the algorithm underperforms.", "votes": null }, { "id": "395610", "postDate": "09/29/2018 00:52:23", "content": "As far as I can see in your code, I think you haven't added any recommended noise to your input apart from the dropout layer which is also a kind of induced noise. Most of the times other kinds of noises such as s&p, gaussian noise seems to give pretty much decent results when I used before on other data sets. What I mean is the kind of noise is also a hyper-parameter. You can give it a try.
", "rawMarkdown": "As far as I can see in your code, I think you haven't added any recommended noise to your input apart from the dropout layer which is also a kind of induced noise. Most of the times other kinds of noises such as s&p, gaussian noise seems to give pretty much decent results when I used before on other data sets. What I mean is the kind of noise is also a hyper-parameter. You can give it a try.", "votes": null }, { "id": "395646", "postDate": "09/29/2018 03:31:40", "content": "Thanks @manu845\nWhat kind of noise should I add to OHE features as they are binary. \nFor numerical, noise means just add or subtract some value or something else?
\n\nIf you have any great source from which I can get hints, can you please share it with us?
\n\nThanks.
", "rawMarkdown": "Thanks @manu845\nWhat kind of noise should I add to OHE features as they are binary. \nFor numerical, noise means just add or subtract some value or something else?\n\nIf you have any great source from which I can get hints, can you please share it with us?\n\nThanks.", "votes": null }, { "id": "395656", "postDate": "09/29/2018 04:24:50", "content": "S&p noise is claimed to work well when categorical data is involved, moreover they tend to work well with the numerical data too. Try to normalize your data before feeding into the network and use cross entropy loss, this tend to give more variability to the representation of features.\nYou may get a different looking set of features but, the pattern and information will be preserved which a model can recognize.
", "rawMarkdown": "S&p noise is claimed to work well when categorical data is involved, moreover they tend to work well with the numerical data too. Try to normalize your data before feeding into the network and use cross entropy loss, this tend to give more variability to the representation of features.\nYou may get a different looking set of features but, the pattern and information will be preserved which a model can recognize.", "votes": null }, { "id": "395667", "postDate": "09/29/2018 05:32:10", "content": "Thanks for sharing! I'm going to use the latent features of autoencoder and experiment the performance. I will update you!
", "rawMarkdown": "Thanks for sharing! I'm going to use the latent features of autoencoder and experiment the performance. I will update you!", "votes": null }, { "id": "395670", "postDate": "09/29/2018 05:36:07", "content": "@youhanlee, \nWhat structure are you going to use for DAE?
", "rawMarkdown": "youhanlee, \nWhat structure are you going to use for DAE?", "votes": null }, { "id": "395674", "postDate": "09/29/2018 05:49:49", "content": "At first, I'll use simple fully-connected AE and see the AUC for classification (high revenue). If It works, I'll add the latent features and calculate performance.
\n\nActually, I'll test many encoders(CNN, RNN, DAE etc.), but, I think fully connected-based autoencoder is a good choice. DAE is one of them.
", "rawMarkdown": "At first, I'll use simple fully-connected AE and see the AUC for classification (high revenue). If It works, I'll add the latent features and calculate performance.\n\nActually, I'll test many encoders(CNN, RNN, DAE etc.), but, I think fully connected-based autoencoder is a good choice. DAE is one of them.", "votes": null }, { "id": "396065", "postDate": "09/30/2018 00:54:28", "content": "IIRC, for noise, this is inspired from the Porto Seguro winning solution:
\n\ndef inputSwap(x, arr, p):\n ### Gives augmented version of the row\n ### Takes x and swaps each element with a probability of p\n ### With an element from arr that corresponds to the same column\n np.random.seed()\n x_tilde = x.copy()\n swap_idx = np.where(np.random.rand(len(x)) < p)[0]\n x_tilde[swap_idx] = np.array([np.random.choice(arr[:,c], 1)[0] for c in swap_idx])\n return (x, x_tilde)
\n\nsource :https://github.com/KeremTurgutlu/deeplearning/blob/master/porto_seguro/DAE.py
\n\nAlso, I don't think any dropout was used.
", "rawMarkdown": "IIRC, for noise, this is inspired from the Porto Seguro winning solution:\n\ndef inputSwap(x, arr, p):\n ### Gives augmented version of the row\n ### Takes x and swaps each element with a probability of p\n ### With an element from arr that corresponds to the same column\n np.random.seed()\n x_tilde = x.copy()\n swap_idx = np.where(np.random.rand(len(x)) < p)[0]\n x_tilde[swap_idx] = np.array([np.random.choice(arr[:,c], 1)[0] for c in swap_idx])\n return (x, x_tilde)\n\nsource :https://github.com/KeremTurgutlu/deeplearning/blob/master/porto_seguro/DAE.py\n\nAlso, I don't think any dropout was used.", "votes": null }, { "id": "396288", "postDate": "09/30/2018 11:53:57", "content": "Thanks @sdoria for the code & link
", "rawMarkdown": "Thanks @sdoria for the code & link", "votes": null } ], "comments": [ { "id": 394861, "author_name": "jpmiller", "author_url": "", "post_date": "09/27/2018 15:11:33", "content": "I tried using one in an unsupervised fashion as a sort of anomaly detector. The AE returned an array the same dimensions of the input, which I subtracted row wise from the original data to get a score for each visitor. I then stacked the scores onto the original data and ran LightGBM. It didn't help. Still, I think there's something to it and will keep trying.
", "votes": null, "replies": [ { "id": 394954, "author_name": "prashantkikani", "author_url": "", "post_date": "09/27/2018 17:57:22", "content": "Same here. It didn't work.\nI think to get benefit from DAE, structure and parameters of it matters a lot. Also some good amount of computation power is required..
", "votes": null, "replies": [] }, { "id": 395131, "author_name": "youhanlee", "author_url": "", "post_date": "09/28/2018 02:33:25", "content": "I understand the anomaly is the sample which has a very large revenue.\nDid your autoencoder detect anomalies(large revenue)?
", "votes": null, "replies": [] }, { "id": 395149, "author_name": "jpmiller", "author_url": "", "post_date": "09/28/2018 03:20:47", "content": "I detected a good portion of the anomalies. However, I falsely identified many visitors with 0 revenue. I believe this often happens with imbalanced data when the algorithm underperforms.
", "votes": null, "replies": [] }, { "id": 395667, "author_name": "youhanlee", "author_url": "", "post_date": "09/29/2018 05:32:10", "content": "Thanks for sharing! I'm going to use the latent features of autoencoder and experiment the performance. I will update you!
", "votes": null, "replies": [] }, { "id": 395670, "author_name": "prashantkikani", "author_url": "", "post_date": "09/29/2018 05:36:07", "content": "@youhanlee, \nWhat structure are you going to use for DAE?
", "votes": null, "replies": [] }, { "id": 395674, "author_name": "youhanlee", "author_url": "", "post_date": "09/29/2018 05:49:49", "content": "At first, I'll use simple fully-connected AE and see the AUC for classification (high revenue). If It works, I'll add the latent features and calculate performance.
\n\nActually, I'll test many encoders(CNN, RNN, DAE etc.), but, I think fully connected-based autoencoder is a good choice. DAE is one of them.
", "votes": null, "replies": [] } ] }, { "id": 395097, "author_name": "sdoria", "author_url": "", "post_date": "09/28/2018 00:49:49", "content": "I haven't tried DAE here yet, but I tried replicating the Porto Seguro winning solution without success. As far as I know, no one has been able to do so.\nI sincerely hope you are successful in your attempt!
", "votes": null, "replies": [] }, { "id": 395610, "author_name": "manu845", "author_url": "", "post_date": "09/29/2018 00:52:23", "content": "As far as I can see in your code, I think you haven't added any recommended noise to your input apart from the dropout layer which is also a kind of induced noise. Most of the times other kinds of noises such as s&p, gaussian noise seems to give pretty much decent results when I used before on other data sets. What I mean is the kind of noise is also a hyper-parameter. You can give it a try.
", "votes": null, "replies": [ { "id": 395646, "author_name": "prashantkikani", "author_url": "", "post_date": "09/29/2018 03:31:40", "content": "Thanks @manu845\nWhat kind of noise should I add to OHE features as they are binary. \nFor numerical, noise means just add or subtract some value or something else?
\n\nIf you have any great source from which I can get hints, can you please share it with us?
\n\nThanks.
", "votes": null, "replies": [] }, { "id": 395656, "author_name": "manu845", "author_url": "", "post_date": "09/29/2018 04:24:50", "content": "S&p noise is claimed to work well when categorical data is involved, moreover they tend to work well with the numerical data too. Try to normalize your data before feeding into the network and use cross entropy loss, this tend to give more variability to the representation of features.\nYou may get a different looking set of features but, the pattern and information will be preserved which a model can recognize.
", "votes": null, "replies": [] }, { "id": 396065, "author_name": "sdoria", "author_url": "", "post_date": "09/30/2018 00:54:28", "content": "IIRC, for noise, this is inspired from the Porto Seguro winning solution:
\n\ndef inputSwap(x, arr, p):\n ### Gives augmented version of the row\n ### Takes x and swaps each element with a probability of p\n ### With an element from arr that corresponds to the same column\n np.random.seed()\n x_tilde = x.copy()\n swap_idx = np.where(np.random.rand(len(x)) < p)[0]\n x_tilde[swap_idx] = np.array([np.random.choice(arr[:,c], 1)[0] for c in swap_idx])\n return (x, x_tilde)
\n\nsource :https://github.com/KeremTurgutlu/deeplearning/blob/master/porto_seguro/DAE.py
\n\nAlso, I don't think any dropout was used.
", "votes": null, "replies": [] }, { "id": 396288, "author_name": "prashantkikani", "author_url": "", "post_date": "09/30/2018 11:53:57", "content": "Thanks @sdoria for the code & link
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "394820": "Hi All,\n\nI am wondering Denoising AutoEncoders (DAE) are helpful or not in this competition? I have attempted [here][1].\n\nThey are mostly used for Computer Vision applications but one winning solution of previous Kaggle competition have used it for tabular data.\n\nWhat kind of structure should one choose for it? And what kind of parameters we need to consider while choosing activations, # of hidden layers, dropouts (if any) of our DAE?\n\nThe one which I have tried ([link][2]), # of layers, activations etc are chosen by trial and error. Some another method for it should be there.\n\nThanks. \n\n\n [1]: https://www.kaggle.com/prashantkikani/rstudio-humble-try-for-autoencoders\n [2]: https://www.kaggle.com/prashantkikani/rstudio-humble-try-for-autoencoders", "394861": "I tried using one in an unsupervised fashion as a sort of anomaly detector. The AE returned an array the same dimensions of the input, which I subtracted row wise from the original data to get a score for each visitor. I then stacked the scores onto the original data and ran LightGBM. It didn't help. Still, I think there's something to it and will keep trying.", "394954": "Same here. It didn't work.\nI think to get benefit from DAE, structure and parameters of it matters a lot. Also some good amount of computation power is required..", "395097": "I haven't tried DAE here yet, but I tried replicating the Porto Seguro winning solution without success. As far as I know, no one has been able to do so.\nI sincerely hope you are successful in your attempt!", "395131": "I understand the anomaly is the sample which has a very large revenue.\nDid your autoencoder detect anomalies(large revenue)?", "395149": "I detected a good portion of the anomalies. However, I falsely identified many visitors with 0 revenue. I believe this often happens with imbalanced data when the algorithm underperforms.", "395610": "As far as I can see in your code, I think you haven't added any recommended noise to your input apart from the dropout layer which is also a kind of induced noise. Most of the times other kinds of noises such as s&p, gaussian noise seems to give pretty much decent results when I used before on other data sets. What I mean is the kind of noise is also a hyper-parameter. You can give it a try.", "395646": "Thanks @manu845\nWhat kind of noise should I add to OHE features as they are binary. \nFor numerical, noise means just add or subtract some value or something else?\n\nIf you have any great source from which I can get hints, can you please share it with us?\n\nThanks.", "395656": "S&p noise is claimed to work well when categorical data is involved, moreover they tend to work well with the numerical data too. Try to normalize your data before feeding into the network and use cross entropy loss, this tend to give more variability to the representation of features.\nYou may get a different looking set of features but, the pattern and information will be preserved which a model can recognize.", "395667": "Thanks for sharing! I'm going to use the latent features of autoencoder and experiment the performance. I will update you!", "395670": "youhanlee, \nWhat structure are you going to use for DAE?", "395674": "At first, I'll use simple fully-connected AE and see the AUC for classification (high revenue). If It works, I'll add the latent features and calculate performance.\n\nActually, I'll test many encoders(CNN, RNN, DAE etc.), but, I think fully connected-based autoencoder is a good choice. DAE is one of them.", "396065": "IIRC, for noise, this is inspired from the Porto Seguro winning solution:\n\ndef inputSwap(x, arr, p):\n ### Gives augmented version of the row\n ### Takes x and swaps each element with a probability of p\n ### With an element from arr that corresponds to the same column\n np.random.seed()\n x_tilde = x.copy()\n swap_idx = np.where(np.random.rand(len(x)) < p)[0]\n x_tilde[swap_idx] = np.array([np.random.choice(arr[:,c], 1)[0] for c in swap_idx])\n return (x, x_tilde)\n\nsource :https://github.com/KeremTurgutlu/deeplearning/blob/master/porto_seguro/DAE.py\n\nAlso, I don't think any dropout was used.", "396288": "Thanks @sdoria for the code & link" }, "source": "meta" }