Hello all,
\n\nI have been thinking about this questions for some time, so I thought it is better to discuss with you.
\n\nI wanted to use KNNImputer to fill the missing values. However, I believe that I have to encode the data first before I can use it. I encode my features using different strategies depending on the column, i.e., I use a mix of one-hot and ordinal encoding.
After encoding all features, I ended up with a data frame consisting of 511 columns. When I attempt to encode that using KNNImputer, it actually never converges--I let it run fro 2 days.
Is my mistake that I am passing the one-hot-encoded columns to the imputer? Should I remove those columns from the dataframe before I impute?
\n\nAny hints would be much appreciated!
", "messages": [ { "id": "768555", "postDate": "03/11/2020 00:48:51", "content": "Hello all,
\n\nI have been thinking about this questions for some time, so I thought it is better to discuss with you.
\n\nI wanted to use KNNImputer to fill the missing values. However, I believe that I have to encode the data first before I can use it. I encode my features using different strategies depending on the column, i.e., I use a mix of one-hot and ordinal encoding.
After encoding all features, I ended up with a data frame consisting of 511 columns. When I attempt to encode that using KNNImputer, it actually never converges--I let it run fro 2 days.
Is my mistake that I am passing the one-hot-encoded columns to the imputer? Should I remove those columns from the dataframe before I impute?
\n\nAny hints would be much appreciated!
", "rawMarkdown": "Hello all,\n\nI have been thinking about this questions for some time, so I thought it is better to discuss with you.\n\nI wanted to use `KNNImputer` to fill the missing values. However, I believe that I have to encode the data first before I can use it. I encode my features using different strategies depending on the column, i.e., I use a mix of one-hot and ordinal encoding. \n\nAfter encoding all features, I ended up with a data frame consisting of 511 columns. When I attempt to encode that using `KNNImputer`, it actually never converges--I let it run fro 2 days.\n\nIs my mistake that I am passing the one-hot-encoded columns to the imputer? Should I remove those columns from the dataframe before I impute?\n\nAny hints would be much appreciated!", "votes": null }, { "id": "769036", "postDate": "03/11/2020 13:05:40", "content": "@tunguz would really appreciate your input here!
", "rawMarkdown": "tunguz would really appreciate your input here!", "votes": null }, { "id": "770563", "postDate": "03/13/2020 05:23:20", "content": "you face with too much computations due to KNN's basic principles. KNN computes distance between every couple of observations. it means that for dataset with M rows causes M^2 distance calculations. therefore i can suggest you to split dataset into several smaller parts and perform KNN imputing independently.\nfor example, you split dataset into 20 parts of M/20 rows, summary you will need M^2/20.
\n\nin case of columns, that's right that you perform ohe and other conversions of categorical features. without this you wouldn't be able to calculate distances. but again, every feature affects distance calculation but with linear complexity. what can you do here? select the most relevant features. how? primary component analysis (pce), eda, statistical tests or even building model and checking feature importance.
\n\ni also suggest you to try target encoding instead of one-hot. it converts categorical features into numerical ones and doesn't cause much more columns.
\n\ni hope i helped you. good luck!
", "rawMarkdown": "you face with too much computations due to KNN's basic principles. KNN computes distance between every couple of observations. it means that for dataset with M rows causes M^2 distance calculations. therefore i can suggest you to split dataset into several smaller parts and perform KNN imputing independently.\nfor example, you split dataset into 20 parts of M/20 rows, summary you will need M^2/20.\n\nin case of columns, that's right that you perform ohe and other conversions of categorical features. without this you wouldn't be able to calculate distances. but again, every feature affects distance calculation but with linear complexity. what can you do here? select the most relevant features. how? primary component analysis (pce), eda, statistical tests or even building model and checking feature importance.\n\ni also suggest you to try target encoding instead of one-hot. it converts categorical features into numerical ones and doesn't cause much more columns.\n\ni hope i helped you. good luck!", "votes": null }, { "id": "770859", "postDate": "03/13/2020 13:41:17", "content": "@fkdplc This is very helpful indeed. You suggestions are great.
\n\nRe imputing batches, I wonder what the impact on accuracy will be. I will give it a try though.
\n\nRe target encoding instead of ohe, I agree it would result in fewer columns. I wanted to see if there is away to still get this giant df imputed.
", "rawMarkdown": "fkdplc This is very helpful indeed. You suggestions are great.\n\nRe imputing batches, I wonder what the impact on accuracy will be. I will give it a try though.\n\nRe target encoding instead of ohe, I agree it would result in fewer columns. I wanted to see if there is away to still get this giant df imputed.", "votes": null } ], "comments": [ { "id": 769036, "author_name": "minimaxer", "author_url": "", "post_date": "03/11/2020 13:05:40", "content": "@tunguz would really appreciate your input here!
", "votes": null, "replies": [] }, { "id": 770563, "author_name": "fkdplc", "author_url": "", "post_date": "03/13/2020 05:23:20", "content": "you face with too much computations due to KNN's basic principles. KNN computes distance between every couple of observations. it means that for dataset with M rows causes M^2 distance calculations. therefore i can suggest you to split dataset into several smaller parts and perform KNN imputing independently.\nfor example, you split dataset into 20 parts of M/20 rows, summary you will need M^2/20.
\n\nin case of columns, that's right that you perform ohe and other conversions of categorical features. without this you wouldn't be able to calculate distances. but again, every feature affects distance calculation but with linear complexity. what can you do here? select the most relevant features. how? primary component analysis (pce), eda, statistical tests or even building model and checking feature importance.
\n\ni also suggest you to try target encoding instead of one-hot. it converts categorical features into numerical ones and doesn't cause much more columns.
\n\ni hope i helped you. good luck!
", "votes": null, "replies": [] }, { "id": 770859, "author_name": "minimaxer", "author_url": "", "post_date": "03/13/2020 13:41:17", "content": "@fkdplc This is very helpful indeed. You suggestions are great.
\n\nRe imputing batches, I wonder what the impact on accuracy will be. I will give it a try though.
\n\nRe target encoding instead of ohe, I agree it would result in fewer columns. I wanted to see if there is away to still get this giant df imputed.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "768555": "Hello all,\n\nI have been thinking about this questions for some time, so I thought it is better to discuss with you.\n\nI wanted to use `KNNImputer` to fill the missing values. However, I believe that I have to encode the data first before I can use it. I encode my features using different strategies depending on the column, i.e., I use a mix of one-hot and ordinal encoding. \n\nAfter encoding all features, I ended up with a data frame consisting of 511 columns. When I attempt to encode that using `KNNImputer`, it actually never converges--I let it run fro 2 days.\n\nIs my mistake that I am passing the one-hot-encoded columns to the imputer? Should I remove those columns from the dataframe before I impute?\n\nAny hints would be much appreciated!", "769036": "tunguz would really appreciate your input here!", "770563": "you face with too much computations due to KNN's basic principles. KNN computes distance between every couple of observations. it means that for dataset with M rows causes M^2 distance calculations. therefore i can suggest you to split dataset into several smaller parts and perform KNN imputing independently.\nfor example, you split dataset into 20 parts of M/20 rows, summary you will need M^2/20.\n\nin case of columns, that's right that you perform ohe and other conversions of categorical features. without this you wouldn't be able to calculate distances. but again, every feature affects distance calculation but with linear complexity. what can you do here? select the most relevant features. how? primary component analysis (pce), eda, statistical tests or even building model and checking feature importance.\n\ni also suggest you to try target encoding instead of one-hot. it converts categorical features into numerical ones and doesn't cause much more columns.\n\ni hope i helped you. good luck!", "770859": "fkdplc This is very helpful indeed. You suggestions are great.\n\nRe imputing batches, I wonder what the impact on accuracy will be. I will give it a try though.\n\nRe target encoding instead of ohe, I agree it would result in fewer columns. I wanted to see if there is away to still get this giant df imputed." }, "source": "meta" }