![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2594941%2Fc2f284c8e30befd559890bf27ea9ba16%2F__results___43_2.png?generation=1579379228688910&alt=media\")
So, I have implemented correlation functions from this article. You can find it in my kernel. So, thats what I have founded:\n
So, as you can see features nom_{5-9} are, somehow, connected. I used Theil's U statistic to find it. Maybe I've got an error. Or they are correlated because of huge amount of unique elements in nom_9 or nom_6. Can somebody explain it to me?
", "messages": [ { "id": "722627", "postDate": "01/18/2020 20:41:38", "content": "So, I have implemented correlation functions from this article. You can find it in my kernel. So, thats what I have founded:\n
So, as you can see features nom_{5-9} are, somehow, connected. I used Theil's U statistic to find it. Maybe I've got an error. Or they are correlated because of huge amount of unique elements in nom_9 or nom_6. Can somebody explain it to me?
", "rawMarkdown": "So, I have implemented correlation functions from [this](https://towardsdatascience.com/the-search-for-categorical-correlation-a1cf7f1888c9) article. You can find it in [my kernel](https://www.kaggle.com/vadbeg/pytorch-nn-with-embeddings-and-catboost). So, thats what I have founded:\n\n\nSo, as you can see features nom\\_{5-9} are, somehow, connected. I used Theil's U statistic to find it. Maybe I've got an error. Or they are correlated because of huge amount of unique elements in nom\\_9 or nom\\_6. Can somebody explain it to me?", "votes": null }, { "id": "723579", "postDate": "01/20/2020 07:20:16", "content": "hmm so the feature counts are correlated for those variables... yeah it could be because they are all close to 1, right?
", "rawMarkdown": "hmm so the feature counts are correlated for those variables... yeah it could be because they are all close to 1, right?", "votes": null }, { "id": "723592", "postDate": "01/20/2020 07:43:14", "content": "What did you mean under feature counts? Features nom_{5-9} consists of hundreds and thousands of unique values. We can use OneHotEncoding for them, and they will be 0 or 1. But I have used LabelEncoding, so hashed features are just numbers. Explain it, please.
", "rawMarkdown": "What did you mean under feature counts? Features nom_{5-9} consists of hundreds and thousands of unique values. We can use OneHotEncoding for them, and they will be 0 or 1. But I have used LabelEncoding, so hashed features are just numbers. Explain it, please.", "votes": null }, { "id": "723713", "postDate": "01/20/2020 11:06:19", "content": "I'm not sure what to do if the nominal values are actually numbers.
\n\nMaybe a brute force method to test effectiveness of different label encodings?
", "rawMarkdown": "I'm not sure what to do if the nominal values are actually numbers.\n\nMaybe a brute force method to test effectiveness of different label encodings?", "votes": null }, { "id": "729780", "postDate": "01/26/2020 16:24:55", "content": "Pretty interesting @vadbeg, this high correlation found are in respect with the same unique element in both columns?
", "rawMarkdown": "Pretty interesting @vadbeg, this high correlation found are in respect with the same unique element in both columns?", "votes": null }, { "id": "729813", "postDate": "01/26/2020 17:04:04", "content": "No, as I remember, in this columns there is no same elements. \n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2594941%2Fd93705a3ab33dcfc89a2dd63adb9de5f%2F1_3Mx7I537OnQybSOMPvgqEw.png?generation=1580058005120051&alt=media\")
This correlation explains how we can determine elements from one column using elements from another one (in this table we can determine every element from X using elements from Y, but not vice versa)
", "rawMarkdown": "No, as I remember, in this columns there is no same elements. \n\n\nThis correlation explains how we can determine elements from one column using elements from another one (in this table we can determine every element from X using elements from Y, but not vice versa)", "votes": null }, { "id": "730318", "postDate": "01/27/2020 10:46:10", "content": "Interesting, maybe there's a bottom to top correlation that must happen.
", "rawMarkdown": "Interesting, maybe there's a bottom to top correlation that must happen.", "votes": null }, { "id": "739177", "postDate": "02/07/2020 14:12:36", "content": "I tried to use them as numbers, I did not get a very good result. In fact I tried different pre-processing strategies and tested them with and without using numbers for nom_5 to nom_9. I always got a worse result when using numbers. I normalized them before of course. Well, in the competition, they say these features are strings, so I suppose we are to use them as such, and not as numbers.
hmm so the feature counts are correlated for those variables... yeah it could be because they are all close to 1, right?
", "votes": null, "replies": [ { "id": 723592, "author_name": "vadbeg", "author_url": "", "post_date": "01/20/2020 07:43:14", "content": "What did you mean under feature counts? Features nom_{5-9} consists of hundreds and thousands of unique values. We can use OneHotEncoding for them, and they will be 0 or 1. But I have used LabelEncoding, so hashed features are just numbers. Explain it, please.
", "votes": null, "replies": [] }, { "id": 723713, "author_name": "rkotwani", "author_url": "", "post_date": "01/20/2020 11:06:19", "content": "I'm not sure what to do if the nominal values are actually numbers.
\n\nMaybe a brute force method to test effectiveness of different label encodings?
", "votes": null, "replies": [] } ] }, { "id": 729780, "author_name": "brunhs", "author_url": "", "post_date": "01/26/2020 16:24:55", "content": "Pretty interesting @vadbeg, this high correlation found are in respect with the same unique element in both columns?
", "votes": null, "replies": [ { "id": 729813, "author_name": "vadbeg", "author_url": "", "post_date": "01/26/2020 17:04:04", "content": "No, as I remember, in this columns there is no same elements. \n
This correlation explains how we can determine elements from one column using elements from another one (in this table we can determine every element from X using elements from Y, but not vice versa)
", "votes": null, "replies": [] }, { "id": 730318, "author_name": "brunhs", "author_url": "", "post_date": "01/27/2020 10:46:10", "content": "Interesting, maybe there's a bottom to top correlation that must happen.
", "votes": null, "replies": [] } ] }, { "id": 739177, "author_name": "catadanna", "author_url": "", "post_date": "02/07/2020 14:12:36", "content": "I tried to use them as numbers, I did not get a very good result. In fact I tried different pre-processing strategies and tested them with and without using numbers for nom_5 to nom_9. I always got a worse result when using numbers. I normalized them before of course. Well, in the competition, they say these features are strings, so I suppose we are to use them as such, and not as numbers.