{
  "id": 542426,
  "title": "Clarification of missing data",
  "url": "/competitions/gpr-max-deep-learning-challenge-1-gdlc-1/discussion/542426",
  "author_name": "",
  "post_date": "2024-10-24T19:37:41.104251700Z",
  "votes": 1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>According to your <a href=\"https://www.kaggle.com/code/iragia/open-evaluation-dat\" target=\"_blank\">notebook</a>, I find that all the data are missing in the whole column. Is there only one missing form, or are there other missing forms? That needs clarification. Thank you.</p>",
  "messages": [
    {
      "id": "3027421",
      "postDate": "10/24/2024 19:37:41",
      "content": "<p>According to your <a href=\"https://www.kaggle.com/code/iragia/open-evaluation-dat\" target=\"_blank\">notebook</a>, I find that all the data are missing in the whole column. Is there only one missing form, or are there other missing forms? That needs clarification. Thank you.</p>",
      "rawMarkdown": "According to your [notebook](https://www.kaggle.com/code/iragia/open-evaluation-dat), I find that all the data are missing in the whole column. Is there only one missing form, or are there other missing forms? That needs clarification. Thank you.",
      "votes": null
    },
    {
      "id": "3028234",
      "postDate": "10/25/2024 18:29:52",
      "content": "<p>Hello Zhuoqun, yeah you are right there is only one type of missing data. We assume that some traces (columns) might be missing in the real data, therefore the algorithm should be able to deal with those traces. To summarise, in the evaluation set the data will be corrupted by missing columns only. There are no data where for example half a column is missing and the other half is there. The algorithm must be able to deal with missing columns. Hope this was helpful, let me know if further clarifications are needed. </p>",
      "rawMarkdown": "Hello Zhuoqun, yeah you are right there is only one type of missing data. We assume that some traces (columns) might be missing in the real data, therefore the algorithm should be able to deal with those traces. To summarise, in the evaluation set the data will be corrupted by missing columns only. There are no data where for example half a column is missing and the other half is there. The algorithm must be able to deal with missing columns. Hope this was helpful, let me know if further clarifications are needed.",
      "votes": null
    },
    {
      "id": "3028468",
      "postDate": "10/26/2024 06:20:22",
      "content": "<p>Thank you.</p>",
      "rawMarkdown": "Thank you.",
      "votes": null
    },
    {
      "id": "3037025",
      "postDate": "11/05/2024 08:30:36",
      "content": "<p><a href=\"https://www.kaggle.com/iragia\" target=\"_blank\">@iragia</a> Hello Iraklis, I noticed that the value ranges for each B-Scan image vary significantly. I don't understand the underlying principles of B-Scans. Could you please help explain what causes this? Thank you!  <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11486426%2F54043e23e56b2f4be33ca4ba9e9f6249%2F2024-11-05%20162949.png?generation=1730795403876765&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "iragia Hello Iraklis, I noticed that the value ranges for each B-Scan image vary significantly. I don't understand the underlying principles of B-Scans. Could you please help explain what causes this? Thank you!  \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11486426%2F54043e23e56b2f4be33ca4ba9e9f6249%2F2024-11-05%20162949.png?generation=1730795403876765&alt=media)",
      "votes": null
    },
    {
      "id": "3041462",
      "postDate": "11/10/2024 11:50:26",
      "content": "<p>Yeah this is normal. It has to do with how we generate the data. In general in real data we don't expect to have a specific range of values, it differs based on the antenna system that we use. </p>\n<p>The synthetic data that we generated are based on different pulses, with different central frequencies, and different max/min amplitudes. This adds an extra complexity to the problem with the hope to increase the generalisation capabilities of the final model.</p>\n<p>In general, I would advise to normalise each Bscan sedately e.g. divide each Bscan with its maximum absolute value; or some other type of scaling. </p>\n<p>Hope this was helpful, let me know if further clarifications are needed. </p>",
      "rawMarkdown": "Yeah this is normal. It has to do with how we generate the data. In general in real data we don't expect to have a specific range of values, it differs based on the antenna system that we use. \n\nThe synthetic data that we generated are based on different pulses, with different central frequencies, and different max/min amplitudes. This adds an extra complexity to the problem with the hope to increase the generalisation capabilities of the final model.\n\nIn general, I would advise to normalise each Bscan sedately e.g. divide each Bscan with its maximum absolute value; or some other type of scaling. \n\nHope this was helpful, let me know if further clarifications are needed.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3028234,
      "author_name": "iragia",
      "author_url": "",
      "post_date": "10/25/2024 18:29:52",
      "content": "<p>Hello Zhuoqun, yeah you are right there is only one type of missing data. We assume that some traces (columns) might be missing in the real data, therefore the algorithm should be able to deal with those traces. To summarise, in the evaluation set the data will be corrupted by missing columns only. There are no data where for example half a column is missing and the other half is there. The algorithm must be able to deal with missing columns. Hope this was helpful, let me know if further clarifications are needed. </p>",
      "votes": null,
      "replies": [
        {
          "id": 3028468,
          "author_name": "ucas0v0zhuoqunli",
          "author_url": "",
          "post_date": "10/26/2024 06:20:22",
          "content": "<p>Thank you.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 3037025,
          "author_name": "ucas0v0zhuoqunli",
          "author_url": "",
          "post_date": "11/05/2024 08:30:36",
          "content": "<p><a href=\"https://www.kaggle.com/iragia\" target=\"_blank\">@iragia</a> Hello Iraklis, I noticed that the value ranges for each B-Scan image vary significantly. I don't understand the underlying principles of B-Scans. Could you please help explain what causes this? Thank you!  <br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11486426%2F54043e23e56b2f4be33ca4ba9e9f6249%2F2024-11-05%20162949.png?generation=1730795403876765&amp;alt=media\" alt=\"\"></p>",
          "votes": null,
          "replies": [
            {
              "id": 3041462,
              "author_name": "iragia",
              "author_url": "",
              "post_date": "11/10/2024 11:50:26",
              "content": "<p>Yeah this is normal. It has to do with how we generate the data. In general in real data we don't expect to have a specific range of values, it differs based on the antenna system that we use. </p>\n<p>The synthetic data that we generated are based on different pulses, with different central frequencies, and different max/min amplitudes. This adds an extra complexity to the problem with the hope to increase the generalisation capabilities of the final model.</p>\n<p>In general, I would advise to normalise each Bscan sedately e.g. divide each Bscan with its maximum absolute value; or some other type of scaling. </p>\n<p>Hope this was helpful, let me know if further clarifications are needed. </p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3027421": "According to your [notebook](https://www.kaggle.com/code/iragia/open-evaluation-dat), I find that all the data are missing in the whole column. Is there only one missing form, or are there other missing forms? That needs clarification. Thank you.",
    "3028234": "Hello Zhuoqun, yeah you are right there is only one type of missing data. We assume that some traces (columns) might be missing in the real data, therefore the algorithm should be able to deal with those traces. To summarise, in the evaluation set the data will be corrupted by missing columns only. There are no data where for example half a column is missing and the other half is there. The algorithm must be able to deal with missing columns. Hope this was helpful, let me know if further clarifications are needed.",
    "3028468": "Thank you.",
    "3037025": "iragia Hello Iraklis, I noticed that the value ranges for each B-Scan image vary significantly. I don't understand the underlying principles of B-Scans. Could you please help explain what causes this? Thank you!  \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11486426%2F54043e23e56b2f4be33ca4ba9e9f6249%2F2024-11-05%20162949.png?generation=1730795403876765&alt=media)",
    "3041462": "Yeah this is normal. It has to do with how we generate the data. In general in real data we don't expect to have a specific range of values, it differs based on the antenna system that we use. \n\nThe synthetic data that we generated are based on different pulses, with different central frequencies, and different max/min amplitudes. This adds an extra complexity to the problem with the hope to increase the generalisation capabilities of the final model.\n\nIn general, I would advise to normalise each Bscan sedately e.g. divide each Bscan with its maximum absolute value; or some other type of scaling. \n\nHope this was helpful, let me know if further clarifications are needed."
  },
  "source": "meta"
}