{
  "id": 547551,
  "title": "How should the input be preprocessed?",
  "url": "/competitions/brist1d/discussion/547551",
  "author_name": "",
  "post_date": "2024-11-22T08:58:58.270806200Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>As a complete beginner, I have no experience in handling this kind of data. Is there anyone who can help me broaden my thinking?</p>",
  "messages": [
    {
      "id": "3052299",
      "postDate": "11/22/2024 08:58:58",
      "content": "<p>As a complete beginner, I have no experience in handling this kind of data. Is there anyone who can help me broaden my thinking?</p>",
      "rawMarkdown": "As a complete beginner, I have no experience in handling this kind of data. Is there anyone who can help me broaden my thinking?",
      "votes": null
    },
    {
      "id": "3052566",
      "postDate": "11/22/2024 14:57:26",
      "content": "<p>Ok so there are a few things here, </p>\n<p>First there are a lot of missing values in multiple columns so you may want to look at dropping some rows if there are too many missing values especially for the bg-{time} columns.</p>\n<p>If there are only a couple values missing you may want to look at imputing these. You could use a knn imputer, but this may be difficult given the size of the dataset, alternatively you could use best judgement and set columns such as calories to 0 if its null and fill columns such as bg using forward or backward fill easily implemented in pandas/polars.</p>\n<p>Next you probably want to look at one hot encoding the categorical data, especially if you plan to use nueral network based approaches. You may want to group certain columns together before doing this as run and running are actually the same activity.</p>\n<p>Finally you would want to normalise the data, probably using min max scaling for nn approaches  before you start training to prevent instability.</p>\n<p>This dataset is pretty big and has loads of missing data so dont worry if you're struggling more than on a normal Kaggle competition. Of course there are loads more you could do but i hope this helps :)</p>",
      "rawMarkdown": "Ok so there are a few things here, \n\nFirst there are a lot of missing values in multiple columns so you may want to look at dropping some rows if there are too many missing values especially for the bg-{time} columns.\n\nIf there are only a couple values missing you may want to look at imputing these. You could use a knn imputer, but this may be difficult given the size of the dataset, alternatively you could use best judgement and set columns such as calories to 0 if its null and fill columns such as bg using forward or backward fill easily implemented in pandas/polars.\n\nNext you probably want to look at one hot encoding the categorical data, especially if you plan to use nueral network based approaches. You may want to group certain columns together before doing this as run and running are actually the same activity.\n\nFinally you would want to normalise the data, probably using min max scaling for nn approaches  before you start training to prevent instability.\n\nThis dataset is pretty big and has loads of missing data so dont worry if you're struggling more than on a normal Kaggle competition. Of course there are loads more you could do but i hope this helps :)",
      "votes": null
    },
    {
      "id": "3052929",
      "postDate": "11/23/2024 02:20:00",
      "content": "<p>Thank you for your insights!</p>",
      "rawMarkdown": "Thank you for your insights!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3052566,
      "author_name": "jamiemilsom",
      "author_url": "",
      "post_date": "11/22/2024 14:57:26",
      "content": "<p>Ok so there are a few things here, </p>\n<p>First there are a lot of missing values in multiple columns so you may want to look at dropping some rows if there are too many missing values especially for the bg-{time} columns.</p>\n<p>If there are only a couple values missing you may want to look at imputing these. You could use a knn imputer, but this may be difficult given the size of the dataset, alternatively you could use best judgement and set columns such as calories to 0 if its null and fill columns such as bg using forward or backward fill easily implemented in pandas/polars.</p>\n<p>Next you probably want to look at one hot encoding the categorical data, especially if you plan to use nueral network based approaches. You may want to group certain columns together before doing this as run and running are actually the same activity.</p>\n<p>Finally you would want to normalise the data, probably using min max scaling for nn approaches  before you start training to prevent instability.</p>\n<p>This dataset is pretty big and has loads of missing data so dont worry if you're struggling more than on a normal Kaggle competition. Of course there are loads more you could do but i hope this helps :)</p>",
      "votes": null,
      "replies": [
        {
          "id": 3052929,
          "author_name": "zhilinzhu68",
          "author_url": "",
          "post_date": "11/23/2024 02:20:00",
          "content": "<p>Thank you for your insights!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3052299": "As a complete beginner, I have no experience in handling this kind of data. Is there anyone who can help me broaden my thinking?",
    "3052566": "Ok so there are a few things here, \n\nFirst there are a lot of missing values in multiple columns so you may want to look at dropping some rows if there are too many missing values especially for the bg-{time} columns.\n\nIf there are only a couple values missing you may want to look at imputing these. You could use a knn imputer, but this may be difficult given the size of the dataset, alternatively you could use best judgement and set columns such as calories to 0 if its null and fill columns such as bg using forward or backward fill easily implemented in pandas/polars.\n\nNext you probably want to look at one hot encoding the categorical data, especially if you plan to use nueral network based approaches. You may want to group certain columns together before doing this as run and running are actually the same activity.\n\nFinally you would want to normalise the data, probably using min max scaling for nn approaches  before you start training to prevent instability.\n\nThis dataset is pretty big and has loads of missing data so dont worry if you're struggling more than on a normal Kaggle competition. Of course there are loads more you could do but i hope this helps :)",
    "3052929": "Thank you for your insights!"
  },
  "source": "meta"
}