{
  "id": 191533,
  "title": "Standardization of Data (Data Cleaning) - to be done.",
  "url": "/competitions/cdp-unlocking-climate-solutions/discussion/191533",
  "author_name": "",
  "post_date": "2020-10-17T02:44:29.788083300Z",
  "votes": 4,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Dear Kagglers,</p>\n<p>If you guys are deep into this competition then 1st thing you have to tackle is  <strong>Standardization of data</strong>. Data is raw and requires both cleaning and standardization.</p>\n<p>That said, to start with in <strong>CITIES</strong>, the Sections and it's Parent's data in files is not in coherence with Questionnaire.</p>\n<pre><code>There are couple of ways to handle it.\n\n1. Either, create a complementary data set.\n2. Or, create something in your notebook.\n</code></pre>\n<pre><code>Should you find more please do share.\n</code></pre>\n<p>Wish you all Good Luck.</p>\n<p>Cheers</p>",
  "messages": [
    {
      "id": "1051869",
      "postDate": "10/17/2020 02:44:29",
      "content": "<p>Dear Kagglers,</p>\n<p>If you guys are deep into this competition then 1st thing you have to tackle is  <strong>Standardization of data</strong>. Data is raw and requires both cleaning and standardization.</p>\n<p>That said, to start with in <strong>CITIES</strong>, the Sections and it's Parent's data in files is not in coherence with Questionnaire.</p>\n<pre><code>There are couple of ways to handle it.\n\n1. Either, create a complementary data set.\n2. Or, create something in your notebook.\n</code></pre>\n<pre><code>Should you find more please do share.\n</code></pre>\n<p>Wish you all Good Luck.</p>\n<p>Cheers</p>",
      "rawMarkdown": "Dear Kagglers,\n\nIf you guys are deep into this competition then 1st thing you have to tackle is  **Standardization of data**. Data is raw and requires both cleaning and standardization.\n\nThat said, to start with in **CITIES**, the Sections and it's Parent's data in files is not in coherence with Questionnaire.\n\n```\nThere are couple of ways to handle it.\n\n1. Either, create a complementary data set.\n2. Or, create something in your notebook.\n```\n\n```\nShould you find more please do share.\n```\nWish you all Good Luck.\n\nCheers",
      "votes": null
    },
    {
      "id": "1051899",
      "postDate": "10/17/2020 03:29:42",
      "content": "<p>As someone with experience working in bureaucracies (aerospace industry &amp; military logistics), I can say that you'll have to determine what responses are truly in earnest and have a solid enough impetus of accountability or self-interest in the truth to provide useful information, and what is simply manipulation and C.Y.A. hogwash. Some outcomes, such as pollution and econometrics, can be accessed from outside (and generally very factual) data, but for measuring the actions of the cities in question you will need the questionnaire info, but you'll also need to figure out what to simply throw away.</p>\n<p>The words used will give you your strongest clue as to which is which: C.Y.A. and politics are generally done with very general, self-inflating, authority-based or fear-based terms that are very difficult or even politically suicidal to attempt to disprove, or with references to internal policies, whereas truly responsible communication will be very technical and specific. In addition, political responses may be excessively polite, emotion-heavy and amicable, whereas factual responses are usually very direct. Although it's common in Natural Language Processing to ignore stop words, you may not wish to in this case as stop words may provide a metric in to how accountable and specific a particular response is. Length may also serve - factual responses will tend to be shorter as more detail and qualifiers make falsification more difficult and undermine accountability. Poppers Principle will serve you well in determining which is which, as anything not committed to the truth will not be falsifiable.</p>",
      "rawMarkdown": "As someone with experience working in bureaucracies (aerospace industry & military logistics), I can say that you'll have to determine what responses are truly in earnest and have a solid enough impetus of accountability or self-interest in the truth to provide useful information, and what is simply manipulation and C.Y.A. hogwash. Some outcomes, such as pollution and econometrics, can be accessed from outside (and generally very factual) data, but for measuring the actions of the cities in question you will need the questionnaire info, but you'll also need to figure out what to simply throw away.\n\nThe words used will give you your strongest clue as to which is which: C.Y.A. and politics are generally done with very general, self-inflating, authority-based or fear-based terms that are very difficult or even politically suicidal to attempt to disprove, or with references to internal policies, whereas truly responsible communication will be very technical and specific. In addition, political responses may be excessively polite, emotion-heavy and amicable, whereas factual responses are usually very direct. Although it's common in Natural Language Processing to ignore stop words, you may not wish to in this case as stop words may provide a metric in to how accountable and specific a particular response is. Length may also serve - factual responses will tend to be shorter as more detail and qualifiers make falsification more difficult and undermine accountability. Poppers Principle will serve you well in determining which is which, as anything not committed to the truth will not be falsifiable.",
      "votes": null
    },
    {
      "id": "1052017",
      "postDate": "10/17/2020 07:39:46",
      "content": "<p>Some cities And organisations will provide links to specific policy papers and documentation in their free text answers as well </p>\n<p>if you can identify where a URL is included in an answer and then scrape the resource it links out to (IF THAT RESOURCE IS <strong>PUBLICLY</strong> AVAILABLE) , then this may be another method to get to more data driven / factual information  as highlighted above </p>\n<p>Interesting take on finding more data driven responses <a href=\"https://www.kaggle.com/ahfretheim\" target=\"_blank\">@ahfretheim</a> </p>",
      "rawMarkdown": "Some cities And organisations will provide links to specific policy papers and documentation in their free text answers as well \n\nif you can identify where a URL is included in an answer and then scrape the resource it links out to (IF THAT RESOURCE IS **PUBLICLY** AVAILABLE) , then this may be another method to get to more data driven / factual information  as highlighted above \n\n Interesting take on finding more data driven responses @ahfretheim",
      "votes": null
    },
    {
      "id": "1052613",
      "postDate": "10/18/2020 02:34:56",
      "content": "<p>That is true, although the one caveat to that response is since these organizations are independent, it is possible that the URL documentation may be the product of a diversity of methodologies and even units of measure and thus any comparisons could be apples to oranges. That being said, there certainly is nothing wrong in using the questionnaire responsones to help you personally locate sources of data that you could then apply to ALL of the cities as part of your analysis.</p>",
      "rawMarkdown": "That is true, although the one caveat to that response is since these organizations are independent, it is possible that the URL documentation may be the product of a diversity of methodologies and even units of measure and thus any comparisons could be apples to oranges. That being said, there certainly is nothing wrong in using the questionnaire responsones to help you personally locate sources of data that you could then apply to ALL of the cities as part of your analysis.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1051899,
      "author_name": "ahfretheim",
      "author_url": "",
      "post_date": "10/17/2020 03:29:42",
      "content": "<p>As someone with experience working in bureaucracies (aerospace industry &amp; military logistics), I can say that you'll have to determine what responses are truly in earnest and have a solid enough impetus of accountability or self-interest in the truth to provide useful information, and what is simply manipulation and C.Y.A. hogwash. Some outcomes, such as pollution and econometrics, can be accessed from outside (and generally very factual) data, but for measuring the actions of the cities in question you will need the questionnaire info, but you'll also need to figure out what to simply throw away.</p>\n<p>The words used will give you your strongest clue as to which is which: C.Y.A. and politics are generally done with very general, self-inflating, authority-based or fear-based terms that are very difficult or even politically suicidal to attempt to disprove, or with references to internal policies, whereas truly responsible communication will be very technical and specific. In addition, political responses may be excessively polite, emotion-heavy and amicable, whereas factual responses are usually very direct. Although it's common in Natural Language Processing to ignore stop words, you may not wish to in this case as stop words may provide a metric in to how accountable and specific a particular response is. Length may also serve - factual responses will tend to be shorter as more detail and qualifiers make falsification more difficult and undermine accountability. Poppers Principle will serve you well in determining which is which, as anything not committed to the truth will not be falsifiable.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1052017,
          "author_name": "callumr22",
          "author_url": "",
          "post_date": "10/17/2020 07:39:46",
          "content": "<p>Some cities And organisations will provide links to specific policy papers and documentation in their free text answers as well </p>\n<p>if you can identify where a URL is included in an answer and then scrape the resource it links out to (IF THAT RESOURCE IS <strong>PUBLICLY</strong> AVAILABLE) , then this may be another method to get to more data driven / factual information  as highlighted above </p>\n<p>Interesting take on finding more data driven responses <a href=\"https://www.kaggle.com/ahfretheim\" target=\"_blank\">@ahfretheim</a> </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1052613,
          "author_name": "ahfretheim",
          "author_url": "",
          "post_date": "10/18/2020 02:34:56",
          "content": "<p>That is true, although the one caveat to that response is since these organizations are independent, it is possible that the URL documentation may be the product of a diversity of methodologies and even units of measure and thus any comparisons could be apples to oranges. That being said, there certainly is nothing wrong in using the questionnaire responsones to help you personally locate sources of data that you could then apply to ALL of the cities as part of your analysis.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1051869": "Dear Kagglers,\n\nIf you guys are deep into this competition then 1st thing you have to tackle is  **Standardization of data**. Data is raw and requires both cleaning and standardization.\n\nThat said, to start with in **CITIES**, the Sections and it's Parent's data in files is not in coherence with Questionnaire.\n\n```\nThere are couple of ways to handle it.\n\n1. Either, create a complementary data set.\n2. Or, create something in your notebook.\n```\n\n```\nShould you find more please do share.\n```\nWish you all Good Luck.\n\nCheers",
    "1051899": "As someone with experience working in bureaucracies (aerospace industry & military logistics), I can say that you'll have to determine what responses are truly in earnest and have a solid enough impetus of accountability or self-interest in the truth to provide useful information, and what is simply manipulation and C.Y.A. hogwash. Some outcomes, such as pollution and econometrics, can be accessed from outside (and generally very factual) data, but for measuring the actions of the cities in question you will need the questionnaire info, but you'll also need to figure out what to simply throw away.\n\nThe words used will give you your strongest clue as to which is which: C.Y.A. and politics are generally done with very general, self-inflating, authority-based or fear-based terms that are very difficult or even politically suicidal to attempt to disprove, or with references to internal policies, whereas truly responsible communication will be very technical and specific. In addition, political responses may be excessively polite, emotion-heavy and amicable, whereas factual responses are usually very direct. Although it's common in Natural Language Processing to ignore stop words, you may not wish to in this case as stop words may provide a metric in to how accountable and specific a particular response is. Length may also serve - factual responses will tend to be shorter as more detail and qualifiers make falsification more difficult and undermine accountability. Poppers Principle will serve you well in determining which is which, as anything not committed to the truth will not be falsifiable.",
    "1052017": "Some cities And organisations will provide links to specific policy papers and documentation in their free text answers as well \n\nif you can identify where a URL is included in an answer and then scrape the resource it links out to (IF THAT RESOURCE IS **PUBLICLY** AVAILABLE) , then this may be another method to get to more data driven / factual information  as highlighted above \n\n Interesting take on finding more data driven responses @ahfretheim",
    "1052613": "That is true, although the one caveat to that response is since these organizations are independent, it is possible that the URL documentation may be the product of a diversity of methodologies and even units of measure and thus any comparisons could be apples to oranges. That being said, there certainly is nothing wrong in using the questionnaire responsones to help you personally locate sources of data that you could then apply to ALL of the cities as part of your analysis."
  },
  "source": "meta"
}