{
  "id": 548442,
  "title": "Imputation Using KNN",
  "url": "/competitions/brist1d/discussion/548442",
  "author_name": "",
  "post_date": "2024-11-26T21:33:04.273150900Z",
  "votes": 2,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi All,</p>\n<p>I've seen and read about using KNN imputation for missing values and it looks very interesting and powerful, however fairly obviously KNN is not great with large datasets due to its O(N^2) efficiency.</p>\n<p>I have been playing with scikit-learns KNNImputer (<a href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.impute.KNNImputer.html)\" target=\"_blank\">https://scikit-learn.org/1.5/modules/generated/sklearn.impute.KNNImputer.html)</a>, however even when chunking based on participant or other similar methods, I run out of memory or it takes a very long time. It appears to be the transform that takes all the time so I was wondering if this could be parallelised on a gpu or via threading.</p>\n<p>Has anyone managed to get KNN imputation to work and willing to share any helpful hints or ways to get this to work? If not I will continue using xgboost other iterative methods.</p>\n<p>Cheers,<br>\nJamie</p>",
  "messages": [
    {
      "id": "3056352",
      "postDate": "11/26/2024 21:33:04",
      "content": "<p>Hi All,</p>\n<p>I've seen and read about using KNN imputation for missing values and it looks very interesting and powerful, however fairly obviously KNN is not great with large datasets due to its O(N^2) efficiency.</p>\n<p>I have been playing with scikit-learns KNNImputer (<a href=\"https://scikit-learn.org/1.5/modules/generated/sklearn.impute.KNNImputer.html)\" target=\"_blank\">https://scikit-learn.org/1.5/modules/generated/sklearn.impute.KNNImputer.html)</a>, however even when chunking based on participant or other similar methods, I run out of memory or it takes a very long time. It appears to be the transform that takes all the time so I was wondering if this could be parallelised on a gpu or via threading.</p>\n<p>Has anyone managed to get KNN imputation to work and willing to share any helpful hints or ways to get this to work? If not I will continue using xgboost other iterative methods.</p>\n<p>Cheers,<br>\nJamie</p>",
      "rawMarkdown": "Hi All,\n\nI've seen and read about using KNN imputation for missing values and it looks very interesting and powerful, however fairly obviously KNN is not great with large datasets due to its O(N^2) efficiency.\n\nI have been playing with scikit-learns KNNImputer (https://scikit-learn.org/1.5/modules/generated/sklearn.impute.KNNImputer.html), however even when chunking based on participant or other similar methods, I run out of memory or it takes a very long time. It appears to be the transform that takes all the time so I was wondering if this could be parallelised on a gpu or via threading.\n\nHas anyone managed to get KNN imputation to work and willing to share any helpful hints or ways to get this to work? If not I will continue using xgboost other iterative methods.\n\nCheers,\nJamie",
      "votes": null
    },
    {
      "id": "3058140",
      "postDate": "11/29/2024 05:26:23",
      "content": "<p>Apologies for the late reply. You might want to try <a href=\"https://www.kaggle.com/competitions/playground-series-s4e9/discussion/532329\" target=\"_blank\">this much faster implementation</a>.</p>",
      "rawMarkdown": "Apologies for the late reply. You might want to try [this much faster implementation](https://www.kaggle.com/competitions/playground-series-s4e9/discussion/532329).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3058140,
      "author_name": "thomasmeiner",
      "author_url": "",
      "post_date": "11/29/2024 05:26:23",
      "content": "<p>Apologies for the late reply. You might want to try <a href=\"https://www.kaggle.com/competitions/playground-series-s4e9/discussion/532329\" target=\"_blank\">this much faster implementation</a>.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3056352": "Hi All,\n\nI've seen and read about using KNN imputation for missing values and it looks very interesting and powerful, however fairly obviously KNN is not great with large datasets due to its O(N^2) efficiency.\n\nI have been playing with scikit-learns KNNImputer (https://scikit-learn.org/1.5/modules/generated/sklearn.impute.KNNImputer.html), however even when chunking based on participant or other similar methods, I run out of memory or it takes a very long time. It appears to be the transform that takes all the time so I was wondering if this could be parallelised on a gpu or via threading.\n\nHas anyone managed to get KNN imputation to work and willing to share any helpful hints or ways to get this to work? If not I will continue using xgboost other iterative methods.\n\nCheers,\nJamie",
    "3058140": "Apologies for the late reply. You might want to try [this much faster implementation](https://www.kaggle.com/competitions/playground-series-s4e9/discussion/532329)."
  },
  "source": "meta"
}